OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
Authors: Hongrui Jia
Jitong Liao
Xi Zhang
Haiyang Xu
Tianbao Xie
Chaoya Jiang
Ming Yan
Si Liu
Wei Ye
Fei Huang
Venue: ICLR 2026 Poster (2026)
Keywords: MCP
Computer-use Agent
LMM
Primary area: datasets and benchmarks

Abstract: With advances in decision-making and reasoning capabilities, multimodal agents show strong potential in computer application scenarios. Past evaluations have mainly assessed GUI interaction skills, while tool invocation abilities, such as those enabled by the Model Context Protocol (MCP), have been largely overlooked. Comparing agents with integrated tool invocation to those evaluated only on GUI interaction is inherently unfair.
We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities in a real-world environment. We design a novel automated code-generation pipeline to create tools and combine them with a curated selection from existing tools. Rigorous manual validation yields 158 high-quality tools (covering 7 common applications), each verified for correct functionality, practical applicability, and versatility.
Extensive evaluations of state-of-the-art multimodal agents on OSWorld-MCP show that MCP tools generally improve task success rates (e.g., from 8.3\% to 17.6\% for OpenAI
o3 at 15 steps, from 38.9\% to 45.0\% for Claude 4 Sonnet at 50 steps), underscoring the importance of assessing tool invocation capabilities. However, even the strongest models have relatively low tool invocation rates, Only 33.3\%, indicating room for improvement and highlighting the benchmark's challenge.
By explicitly measuring MCP tool usage skills, OSWorld-MCP deepens understanding of multimodal agents and sets a new standard for evaluating performance in complex, tool-assisted environments. Our code, environment, and data are publicly available at https://osworld-mcp.github.io.

Published as a conference paper at ICLR 2026
SSMORLL-MCPPBBENCHMARRIIIG MCP TOOL INVOCATION IN COMPUTER-USE AGENTS
Hongrui Jia Jia!.2.t, 12.,, Jitong Liao Xi Zhanga1, Haiyang Xu² Tianbao Xie?,
Chaoya Jiang',
Ming Yan Si Liu", Wei Ye" Fei Huang²
Peking University
Tongyi Lab, Alibaba Group Beijing Zhongguaneun Academy
(jiahcngrui, wyetepku. shuofeng xhy ibaba-inbbbainnccccmm com
ABSTRACT
With advances in decision-making and reasoning capabilities, multimodal agents show strong potential in computer application scenarios. Past evaluations have mainly assessed GUT interaction skills, while tool invocation abilities, such as those enabled by the Model Context Protocol (MCP), have been largely over- looked. Comparing agents with integrated tool invocation to those evaluated only on GUI interaction is inherently unfair. We present OSWorld-MCP, the first com- prehensive and fair benchmark for assessing computer-use agents" tool invoca- tion, GUI operation, and decision-makee making abilities in real-world environment. We design novel automated code-generation pipeline to create tools and com- bine them with a curated selection from existing tools. Rigorous manual valida- tion yields 158 high-quality tools (covering 7 common applications), each verified for correct functionality, practical applicability, and versatility. Extensive evalu- ations of state-of-the- multimodal agents on OSWorld-MCP show that MCP tools generally improve task success rates (c.g., from 8.3% to 17.6% for OpenAl o3 at 15 steps, from 38.9% to 45.0% for Claude 4 Sonnet at 50 steps), underscor- ing the importance of assessing tool invocation capabilities. However, even the strongest models have relatively low tool invocation rates, Only 33.3%, indicating room for improvement and highlighting the benchmark's challenge. By explic- itly measuring MCP tool usage skills, OSWorld-MCT deepens understanding of multimodal agents and sets a new standard for evaluating performance in com- plex, tool-assisted environments. Our code, environment, and data are publicly available at http:://wwwwwodddmiii.itttuueeee
INTRODUCTION
Large Language Models (LLMs) such as GPT-5 (OpenAl 2025a). DeepSeek-R1 (Guo et al. 2025a), and Qwen3 (Yang et al. 2025) have dramatically advanced reasoning and decision-maaee
capabil-
ities. Building on these advances, recent Large Multimodal Models (LMMs) are able to address complex computer-use tasks, which has stimulated considerable research interest (Qin et al.) 2025
Lai et al. 2025; Song et al. [0025] Luo et al. 2025 Lu et al., [ Ye et al., 2025 Wang et al, 2024b 2025a Zhu al. 2025 Wang et al. 2024a). Consequently, how to reliably and robustly
benchmark different LMMs in GUI-driven scenarios has become a key open question. Existing
evaluation frameworks Xie et al. 2024; Abhyankar al. 2025 Xie et al. 2025 Bonatti et al.
2024 Kuntz et al., 2025 Rawles et al. (2024) primarily focus on assessing model's ability to per-
form GUl-based operations, by predefining set of user-interface actions (e.g., click, type, and drag)
and allowing the model to autonomously decide how to complete the task.
Although many benchmarks have been proposed for evaluating GUI agents, most neglect a crucial capability: the ability to invoke external tools such as the Model Context Protocol (MCP) (An- thropic 2024). MCP is an open standard that connects AI applications with external systems. By using MCP, computer-use agents can access diverse resources, including data sources such as local files and databases, and tools such as search engines and calculator, thereby enabling them to obtain critical information and complete tasks more effectively. In many cases, performing task through MCP is more efficient than relying solely on GUI operations. For instance, as illustrated in Fig- ure when agent is instructed install the autoDocstring extension in VS Code, GUI-based

---

Published as a conference paper at ICLR 2026
Step Extenstons
Type autebocstring search bat
Drep- tood
dos Japto weweeeononntttott doriilla ontonsvon,
anstallo oxtonslon or uplates in
Aegel
corsand Pooddo "emjuutoll- actersiinn,, extenstio__
succboss extens entry.
The autuDecuting edani on now installed.
command apeend "eeppe-rol eace""
eupprapess run jocmaand, ctacceTroo)
extenatin"
eccept wubpprocaza wtt.eeeellllleee Galledfroceadterss ax VAc
YAC
dinaspacted Anr
(a) GUI-based approach
(b) MCP tool
Figure 1: Comparison of completing the instruction "Please help me install the autoDocstring ex- tension in VS Code" " via GUI operations (a) and the MCP Tool (b).
approach may require least four steps, whereas an MCP tool can achieve the same result in single step, offering both greater efficiency and higher robustness. Several recent agents (Lai et al., 2025] SSng al.) (025) have already incorporated autonomous tool invocation and have achieved notable performance gains. However, it is inherently inequitable to compare such agents with others that assess only GUT interaction. At present, there remains a lack comprehensive and equitable benchmarks that jointly evaluate GUI operation skills, tool invocation capabilities, and the decision- making competence of computer-use use agents in an integrated framework.
To bridge this gap, introduce OSWorld- the first comprchensive and fair benchmark de- signed to evaluate the tool invocation capabilities of computer-use agents. Our primary motivation is to establish unified standard for fair comparison of tool utilization abilities across different models, addressing the current lack of consistency in tool sets and evaluation metrics. Built upon a widely used real-world computer-use environment OSWorld (Xie et al. 2024). OSWorld-MCP significantly extends its capabilities by incorporating a curated set of 158 high-quality MCP tools. These tools cover common applications such as LibreOffice Writer and VS Code, ensuring diverse and realistic testing environment (Figure 4 A number of 25 non-target tools are included, serving as distractors in the OS World-MCP tasks. Besides, our designed MCP tools are applicable to 250 tasks, accounting for 69% of the entire benchmark, which underscores their broad applicability. Notably, 153 tasks, representing 42% of the benchmark, involve challenging multi-round tool invocations. Even the strongest model, Claude 4 Sonnet, achieves accuracy of O when relying solely on GUI operations for tasks requiring four rounds of tool invocation, and only 19.5% accuracy when MCP tools introduced. These results highlight the challenging of our benchmark and the broad utility of our tools in diverse scenarios.
Another distinguishing feature of OSWorld-MCPin is its dynamic interaction between GUI operations and tool usage. Specifically, at every step of a task, the agent can autonomously choose between our MCP tools and direct GUT actions (e.g., click and type) to interact with the graphical interface. With this setting, OSWorld-MCP can provide a balanced and thorough assessment of LMM capabilities in hybrid -aaiiinnnmmkingggg.. skills. Here, the decision-making involves not only selecting the correct tools, but also choosing the most efficient execution path when both GUI operations and MCP tools are required to accomplish the task. Besides, to provide more nuanced evaluation, we introduce two new metrics alongside task accuracy: Tool Invocation Rate (TIR) and Average Completion Steps (ACS). TIR measures the proportion of tasks successfully completed using MCP tools, offering insights into an agent's 'ss tool utilization propensity, while ACS quantifies task completion efficiency. In conclusion, compared with existing text-based tool-use benchmarks (Liu et al., 2025 Mo et al.) Gao et al. 2025: Wang al. 2025b) and the above GUI relevant benchmarks, OSW has significant ad advantages in evaluating real-world computer-use scenarios. challenges agents to interpret visual GUI information, perform GUI operations, invoke appropriate tools, and effectively chain multiple tools. This combination of features makes OSWorld-MGS WWrrdd-MCP a more comprehensive and realistic benchmark for evaluating computer-use capabilities.
2025
2

---

Published as a conference paper at ICLR 2026
To develop the above mentionde high-quality MCP tools, design an automated code generation pipeline comprising three modules: the Code Generation Module, the Code Filter Module, and the Tool Wrap Module. By leveraging the advanced reasoning capabilities of OpenAl o3 (OpenAI) 2025b), this pipeline produces 72 functional tools. These tools are then combined with those cu- rated from existing MCP servers (Lai al., 2025), followed by fine-grained manual verification procedure to remove functionally redundant items and highly task-specific ones that lack relevance to real-world applications. Through this process, we obtain a curated collection of 158 high-quality tools, each verified to be readily usable and aligned with the designed task difficulty. With this com- prehensive tool set and our WCrr we conduct a detailed experimental study and analysis on range of state-of-the-an art LMMs and multimodal frameworks. Three key findings are identi- fied during experiments: (1) MCP tools enhance agent accuracy and efficiency compared to the GUI-only setting. For example, the success rate of OpenAl o3 increases from 8.3% to 17.6% at 15 steps. (2) The tool invocation rate positively correlates with performance. We also observe that tool invocation rates for multimodal agents remain relatively low, indicating the significant potential in the tool utilization capabilitiess of current LMMs and multimodal frameworks. (3) The compo- sition of multiple tools remains significant challenge. Performance declines complex task involving more tools. In the other hands, agent struggle on selecting the correct tool from a full list.
Our contributions are as follows:
We introduce OSWorld-MCR. a comprehensive, fair, and novel benchmark for evaluating computer-use agents that integrates 158 high-quality MCP tools (covering 7 common apps) with GUI operations in real-world scenarios. It bridges the gap between pure-GUI and text-based tool-use evaluations, offering holistic and realistic assessment of computer-use capabilities.
We propose novel pipeline combining automated code generation with rigorous manual curation to create MCP tools, enhancing our benchmark's s evaluation depth and fairness. We also introduce new metrics (i.e., TIR and ACS) for nuanced assessment of agents' tool utilization propensity.
Our extensive experiments identify three key findings: (1) the effective use of MCP tools signifi- cantly improves the accuracy of LMMs and reduces the ACS; (2) the TIR is positively correlated with accuracy: (3) while MCP tools consistently enhance efficiency and accuracy as task difficulty increases, combining MCP tools remains more challenging than combining GUI operations.
2 RELATED WORK
2.1 BENCHMARKS FOR MULTIMODAL AGENTS
Current benchmarks Deng al.
2023 Lu et al. 2024; Kapoor al. 2024: Zhou et al. 2023 Koh
(ct al. 2024; Drouin et al. 2024 Tian et al. 2024| Bonatti et al. 2024| Xie et al. 2024) for multi- modal agents primarily focus on evaluating their ability to complete tasks through GUI-based opera- tions. Static benchmarks such as Mind2 Web (Deng et al. 2023). WebLinx (Lù et al. 2024). and Om- niAct (Kapoor al., (024) rely on manually collected static datasets assess agent performance. These static benchmarks are built upon pre-defined trajectories of GUI actions, which make them incapable of evaluating alternative action paths that may arise when tool invocation is introduced. In contrast, dynamic interactive benchmarks operate in open- open-ended environments and provide reward signals upon task completion, enabling more flexible and open-ended evaluation of agents. No- table examples of dynamic benchmarks for specific environments include WebArena (Zhou et al., 2023). VisualWebArena (Koh et al.) (024), WorkArena (Drouin al.) 2024). MMInA (Tian et al. 2024 WindowsAgentAreen (Bonatti et al. 2024), and OSWorld (Xie et al. 2024). However, ex- isting dynamic interactive benchmarks typically predefine only GUI actions for the agent to use, and therefore lack a comprehensive and fair evaluation framework that jointly measures multimodal agents' tool invocation, GUI interaction, and decision-makinn capabilities.
2.2 MODEL CONTEXT PROTOCOL
With the rapid development of multimodal agents, increasing attention has been paid to their tool invocation capabilities (Song et al. 2025 Lai et al. 2025). Introduced by Anthropic in November 2024, the Model Context Protocol (MCP) JSON-RPC- for secure context ingestion and structured tool invocation. MCP provides standardized, model- agnostic interface that enables AI applications to connect to external systems such as tools, data
seivet interface designed
3

---

Published as a conference paper at ICLR 2026
Thoughht
Cardig
Virtual Machine Karform
on car contly boode en Cokiiveee
Vitual Roceine
Cacrdinaior
Sarcent
Actions
ball 29:3 REspeco
Manage
OUI
HCP
Tesk Accerscy
OUI Due
Tos Invecis fare
Crapg
Eurque
Typee
Type
VMI
VILAE Mucheno
Simdator
Conciasion
Vrodl Gadtene
NCP ServEs
"ak.erriing" intorii va Cole autodrciiiss
Chirt
GAI
Reward
Figure 2: Overview of the OSWorld- framework.
resources, and workflows, thereby facilitating the integration of large language models with exter- nal data sources and utilities (Anthropic 2024). This standardization addresses challenges arising from fragmented and highly customized integrations. MCP supports flexible plug-and-ple -pay tool-
ing, secure infrastructure integration, and cross-LLM vendor compatibility (Ehtesham et al. 2025)
Several contemporary MCP-related benchmarks (Gao et al. 2025 Liu et 2025 Mo et al. 2025
Wang et al.] 20256). MCPEval (Liu et al., 2025) for LLM evaluation have recently emerged and MCP-Radar (Gao et al. 2025) cover only a limited set of MCP servers, typically no more than a few dozen tools, which restricts task diversity. LiveMCPBench (Mo al. 2025) employs large language model-based evaluation, an approach that is not well suited to tasks requiring real-time knowledge. MCP-Bench (Wang et al.) 2025b) defines its tasks based on available tools. introducing constraints that create a gap between benchmark tasks and truly open-ended real-world problems. Currently, there is no benchmark that comprehensively evaluates multimodal agents in terms of GUI interac- tion, tool invocation, and decision- capabilities within an integrated and fair framework.
3 OSWORLD-MCP BENCH
3.1 OVERVIEW
We propose OSWorld-MCP, a comprehensive benchmark for evaluating computer-use agents. OSWorld-MCP is built upon the OSWorld benchmark, which is widely used dynamic and in- teractive evaluation framework designed to assess the performance of multimodal agents in realistic computing environments, including Ubuntu, Windows, and macOS. OSWorld covers nine applica- tions and consists of a total of 369 real-world tasks that involve interaction through both graphical user interfaces (GUI) and command- line interfaces (CLJ). As illustrated in Figure 2, OSWorld-MCP enables effective assessment of multimodal agents in authentic scenarios, capturing their tool invo- cation capability, GUI operation skills, and decision-making competence. Decision-making making assess- ment includes the ability to choose between GUT and MCP pathways, as well as the ability to select the most appropriate MCP tool for given task. In the following section, we first introduce our automated pipeline for tool generation and the procedures for collecting and filtering tools used to construct OS OSWorld-MCP. We then present a detailed analysis of the MCP tools we produce, demon- strating the high quality and rational design of the OSWorld-MCP tool set. Finally, we describe the evaluation metrics defined for the OSWorld-MCS dataset and outline how these metrics enable comprehensive performance measurement of different models.
3.2 TOOLS GENERATION AND FILTER
Existing MCP servers generally offer tools that are relatively simple and often have overlapping functionalities. Consequently, there is shortage of high-quality MCP tools that can be readily applied in practical scenarios. To address this limitation, we design an automated tool generation pipeline composed of three modules: the Code Generation Module, the Code Filter Module, and the
4

---

Published as a conference paper at ICLR 2026
Whapping Prorgit
cutometcolly Generating
Autamaticaaly fibering
oute us for vise the pookirs
1509
Coce Generation Mocule
Codo Fiter Module
Tool Vrop Hodule
code unage guide into
</>
san as loot
Tasks
Soldiomn
Cookss
(ocign caida Firmatt
Autometicalll Wemppring
Tools
Marsall hituring
poe blact sing uoage qudki Tawer for Kater wooo
MCP Tools
Revlewers
High-quality HCP Tools
(a) The Wrapping Prompt
(b) High-quality tool generation process
Figure 3: Illustration of our tool generation process. (a) Prompt for wrapping code into MCP tools with OpenAl o3. (b) Workflow for generating high-quality tools.
Tool Wrap Module. In the Code Generation Module, the user only needs to specify the target task, and the module automatically generates code to accomplish it. Specifically, we employ OpenAl o3 (OpenAl] (0255) to produce node-based solutions for every task in OoWolld. Following the prompting strategy of CoAct (Song et al. 2025), we develop our own prompt and instruct OpenAI o3 to generate, whenever feasible, functional code solutions capable of completing the tasks. In the Code Filter Module, we use OpenAl I o3 to summarize usable code obtained from multi-turn interactions. This summarized code is then applied to solve the corresponding tasks, and any code successfully completing the tasks is retained. Through this process, we obtain seventy-two verified solutions. In the Tool Wrap Module, we provide OpenAl o3 with a prompt that automates the packaging of these verified code solutions into 72 MCP tools, as illustrated in Figure (a).
In addition, we carefully curate 192 tools from existing MCP servers. Since some generate or im- ported tools are tailored to solve single specific task and thus unsuitable for real-world use, conduct a manual inspection of all 264 collected tools to remove such task-specific items as well as functionality duplicates. Each tool is independently evaluated by at least two reviewers with exten- sive GUI agent development experience, and is retained only if both reviewers deemed it qualified. After this manual filtering process, we obtain 158 high-quality tools that are both applicable and valuable in real-world computing environments.
3.3 TOOLS ANALYSIS
We analyze the composition of the 158 high-quality tools designed for practical real-world use, with their distribution across application scenarios shown in Figure (b)). To further ensure that these tools have a tangible positive impact on task completion in realistic settings, we conduct an addi- tional manual validation and find that 133 tools effectively contribute improving task efficiency, while the remaining 25 tools originate from existing external MCP servers.
To verify that the effective tools can be actively utilized by models, we evaluate five state-of-the- art large multimodal models, including Qwen2.5-vLL-72-Innste
(Bai et al. 2025) and Claude 4 Sonnet (Anthropic 2025), on OSWorld-MCP, allowing them at each step to autonomously choose between performing GUI operations and invoking any of the 158 high-quality tools. The results indicate that 131 tools are invoked at least once during evaluation. The remaining two tools, which are manually re-ver re-verified for usability, are hypothesized to be absent from model usage due to the complexity of the associated tasks, which likely discouraged models from attempting to invoke them. Tool invocation frequencies are presented in Figure 4 (a).
We also conduct a manual analysis of all 361 OSWorld-MCP tasks (8 Google Drive tasks excluded) to annotate the set of available tools for each task and to record the total number of invocations of these tools across evaluations. The distribution
total available tool invocations per task is shown in Figure 4 (c). An available tool is defined as whose invocation
make task execution sub- stantially more efficient. Based on this definition, OSWorld-MCP classified into two subsets: Fool-Beneficia Tasks, which include tasks for which at least one available tool can improve effi-
5

---

Published as a conference paper at ICLR 2026
180
Figure 4: (a) illustrates the total number of times each tool was invoked by the five models single OSWoridMMCP evaluation. (b) depicts the distribution of our 158 high-quality tools across different usage scenarios for various OSWorld environments. Due to software version constraints within World,,, MCP servers were not developed for GIMP and Thunderbird. (c) presents the distribution the total available tool invocations per task.
ciency (250 tasks), and Non-Tool- Tenl---eenfil Tasks, which include tasks for which no available tool improves efficiency (111 tasks).
These findings demonstrate that the tools in OSWorld-MCP are genuinely relevant to real-world needs and are not artificially tailored for specific benchmark tasks. Furthermore, many tasks be completed more efficiently through multiple tool invocations, requiring models to select appropriate tools and exhibit strong decision-making capabilities. This confirms the soundness of our tool design and underscores the challenging nature of the benchmark.
3.4 METRICS
Building on the evaluation of GUI operation capabilities, we introduce three metrics in OSWorld- MCP to assess a multimodal agent's 's tool invocation and decision decision-making abilities: Task Accuracy, Tool Invocation Rate, and Average Completion Steps.
Task Accuracy. Similar to OSWorld, we use accuracy as overall performance indicator to mea- sure how well a multimodal agent completes tasks. This metric jointly reflects the agent's decision- making ability, tool invocation skills, and GUI interaction performance.
Tool Invocation Rate (TIR). As described in Section 3.2 human reviewers classify each task into one of two categories: Tool-Beneficial Tasks Non-Tool-Benefici Tasks.Let N, be the total number of Tool-Benefi Benefficia Tasks, and n, the number of such tasks in which the agent invoked a tool and successfully completed the task during evaluation. Let N₂ be the total number of Non- Tool-Beneficial Tasks, and ny the number of such tasks which the agent did not invoke tool and successfully completed the task. We define TIR
TIR (n, + ng) N₂))
When computing TIR within the TIR ne/N,, which measures the proportion of Tool-Beneficial Tasks where the agent correctly invoked a tool and succeeded. When computing TIR within the Mon-Tool-Benefiche Task subset (N₂ 0), the formula reduces TIR ng/Ng. which measures the proportion of Non-Tool Beneficial Tasks where the agent correctly avoided tool invocation and succeeded. TIR can effec- tively indicate the agent's ability decide when tool should should not be invoked.
Task subset (NV, = 0), the formula reduces to
Average Completion Steps (ACS). This metric measures the average number of steps an agent takes to complete task. For N tasks, the number of execution steps for task i is S,, the Average Completion Steps is computed as:
N
ACS Σ S
(2)

---

Published as a conference paper at ICLR 2026
ACS reflects pocision-making efficiency: the more accurate the decisions, the higher the rate of correct tool usage, and the more frequently the agent selects efficient tools, the lower ACS will be.
4 EXPERIMENTS
4.1 SETUP
We evaluate a series of state-of-the-art Large Multimodal Models (LMMs), including Qwen2.5-
VL-72B-Instruct (Bai et al., 2025), OWen3-VL-Plus (Qwen'Temm) 2025), Seed1.5-VL. (Guo et al., 2025b), Claude 4 Sonnet (Anthropic 2025), OpenAl 03 (OpenAl 2025b), and Semini-2.5-Pro Co- manici al., 2025), by running a computer-use agent on real-world tasks in OSWorld-MCT in- volving nine different software applications. To facilitate comparison of performance differences across models, we standardized our evaluation using the GUI-Owl agent configuration. This may lead to some performance fluctuations for certain models under the original OSWorld configura- tion. Specifically, at each step, the core LMM performs visual perception of the current interface, generates corresponding thought, and proposes the next action along with reasoning summary. This reasoning history informs subsequent planning during task execution. Besides, we also eval- uate multi-agent framework Agent-S2.5 (Simular Research 2025), and adopt OpenAl o3 as the main generation model with UI-TARS 1,5-72B as the grounding model. For each task, the agent is restricted to fixed maximum number of steps either complete the task or determine that it is infeasible. The temperature parameter is set to 1.0.
In the original OSWorld configuration, LMMs may only erations, including key, type, mousee_ovee click, drag, right_clin click,, middle double _click, scroll, wait and terminate, to complete tasks. With the introduction of MCP tools in our OSWorld-MCP the LMM can, autonomously decide whether to invoke any MCP Tool or perform GUI opera- tion each action step. In details, we employ the 158 high-quality MCP tools curated in Section 3.2 Since providing all 158 tools simultaneously would result in excessively long input contexts, we apply Retrieval- AAtmeetl-- Generation (RAG) to select only the tools relevant to the current application. The model then chooses from these filtered MCP tools or GUI operations.
predefined set of 11 basic GUI op-
4.2
MAIN RESULTS
As shown in Table 1 we evaluate six advanced end-to-end models and one agent-based frameworks under both the original OSWorld configuration and OSWorld-MCP with our curated high-quality tools, using maximum step limits of 15 and 50 respectively. Due to fluctuations in the experimental results, we conducted three runs for each model or framework under each configuration. The results reported in Tables T are the averages over these three runs.
Among the end-to-end models, Claude Sonnet achieves the highest accuracy in OSWorld-MCP for both step limits, reaching 36.1 (15 steps) and 45.0 (50 steps) respectively among LMMs. Claude Sonnet also records the highest tool invocation rate in OSWorld-MCP among these models. Qwen3- VL-PLUS achieve the lowest average completion steps (ACS) at 15 and 50 steps respectively, with values of 10.0 and 18.6. Our results also reveal that current large multimodal models generally have low tool invocation rates in OSWorid-MCP: even the highest, achieved by Claude 4 Sonnet, is only 33.3 percent, while the lowest, from Dwen2.5-VL-722 Instruct, is 9.3 percent. This shows that these models still face challenges in invoking tools both correctly and efficiently. For the multi-agent frameworks, Agent-S2.5 achieves an accuracy of 42.1 at the 15-step limit and 49.5 at the 50-step limit, confirming its effectiveness handling challenging tasks.
Impact of Tool Invocation on Accuracy and Efficiency. A comparison with the GUI-only setting shows that, with the exception of Qwen2.5-VI. all six end-to-end models and both agent frameworks achieve higher accuracies and lower ACS after the introduction of MCP tools. Qwen2.5-VL despite achieving a slight accuracy improvement, shows increased ACS. indicates poor tool invocation capability and weak decision-making ability, leading longer average completion times.
Among the remaining models and frameworks, Gemini-2.55 2.5-Pro exhibits the most significant im- provement. At 50 steps, its overall accuracy rises from 13.3 to 25.7, while ACS decreases from 40.0 to 31.0. Furthermore, a comparison of Tool-Beneficia. and that, with the exception of Qwe all models demonstrate substantial accuracy improvement
Tasks shows
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance on OSWorld-MCP.
Tool-Beneficial Tasks
Non- Ton-Too--Bentlcial Tasks Accuracy
Overall
Agent Model
Actions
Steps
Accuracy
TIR
ACS
TIR
ACS
Accuracy
TIR
ACS
Mouelss
GUI
15
10.1
13.0
14.4
12.9
11.4
13.0
Qwen2.5-VT.
MCP
15
6..4
9.5
14.1 1.1f
12.6
11.4
13.7 0,8t
14.5 3.17
10.1
14.0 1.01
GUI
50
12.3
30.6
17.7
30.4
13.9
30.5
MCP
50
12.0
3.5
39.7
23.7 6.01
22.5
37.4
15.6
9.3
39.0 6.31
GUI
24.5
27.3
11.7
25,4
11.6
Qwen3-VL
GUT
MCP
50
15
7.99
18.5
9.61.94
25.5
34.7
38.0
28.2
25.7
10.9
33.8
32.8
21.5
25.6
10.0
MCP
50
37.3
22.5
17.4
44.3
33.9
21.2
39.5 5.71
26.
18.6 7.04
Proprintary Models
GUI
15
6.3
13.6
9.8
14.0
744
13.8
Gemini-2.5.Po
MCP
22.4
16
15.2
10.5
6.3
13.9
17.4 10.t
12.2
11.6
GUI
50
11.5
39.5
17.5
41.3
13.3
40.0
MCP
50
16.6
16.3
26.6
21.3
18.0
40.7
25.7
16.8
31.0 9.04
7.8
OpenAl o3
GUI
MCP
50
15
23.0
23.0
10.7
15.2
43.9
10.8
2.5
17.4
3.3
46.8
14.3
17.6
12.8
11.6
44.8
11.9
MCP
50
27.0 16:31
17.6
27.9 16.04
17.6
12.3
44.4
24.1
16.0
33.0
GUI
15
27.2
10.6
29.3
11.5
27,9
10,9
Secd1.5-VI.
MCP
31,5 3.30
17.7
9.6
28.9 u.44
28.5
11.3 6.24
30.7 2.87
21.0
10.1 0.4
GUT
50
31.1
22.5
40.2
26,8
33,9
23.8
MCP
SMM
3.41
41.4
40.5
0:30
38.2 1.37
22.3 1.34
GUI
15
29.0
11.8
32,9
12.0
30.2
11.9
Claude-4-Sonet
MCP
37.5 3.55 8.st
26.0
10.1
33.0
30.6
11.6
36.1 5.91
27.4
10.5 1.41
GUI
50
36.6
24.7
44.0
25.4
38,9
25.0
MCP
50
45.2
31,5
18.9
44.6
37.5
22.5
6.10
33.3
20.0 6.04
Mudti-agent Models
GUI
15
35.8
11.4
38.6
11.2
36.7
11.3
Agent-S2.5
MCP
42.9 v.a
28.1
9.6
40.4 1.st
34.2
10.9 0.a.
42.1 5.47
10.0 1.34
GUI
50
46.9
21.1
47.3
18.4
47.1
20.2
MCP
49.4
32,9
16.3
49.6
40.5
18.7 a.a
49.5
2.47
35.3
17.0 3.21
in Tool-Beneficia Tasks after the introduction of MCP tools. The largest gain is observed in Ope- nAI o3 at 50 steps, where accuracy increases from 10.7 to 27.0. In addition, accuracy changes for Non-Tool-Beenffici Tasks minor. Two possible factors may explain these results: First, Non-Tool-Bennficin Tasks refer only to tasks without efficieency-nnhaccing tools, yet they may still include tools that, while not improving efficiency, have a positive effect task completion. Such tools can make it easier for the model to solve the problem, thereby increasing Acc. Second, for tasks that contain no tools beneficial out irrelevant solution paths, making it easier to execute the task along the correct path and thus im- proving task Accuracy. This also explains why ACS often decreases on Non- Non-Tool- Tool-Beneficial Tasks. From these results, we derive the following conclusion:
the task at all, the tools provided can help the model rule
Finding 1: MCP tools significantly enhance LMMs' performance in computer-use, improv- ing accuracy and reducing completion steps for most models. The effectiveness varies across different LMMs, indicating disparities tool utilization capabilities.
Impact of Tool Invocation Rate on Accuracy and Average Completion Steps. Our experiments reveal that, across different models, the tool invocation rate (TIR) and accuracy (Acc) generally hibit a positive correlation, whereas Average Completion Steps (ACS) show no obvious correlation with TIR. We computed TIR, Acc, and ACS for each model under varying maximum step limits, across Tool-Beneficial Tasks, Non-Tool-Beneee -eon--Beeficiall Tasks, and the entire task set. These results were aggregated into a single chart (Figure (])) As shown the figure, for a given model, higher TIR values tend to correspond to higher accuracies, indicating a clear positive relationship between tool invocation and task success. Notably, this relationship remains stable regardless of differences in step limits or task sets. This strongly supports the soundness of our MCP tool design. In contrast, ACS does not show a clear correlation with TIR. Further analysis of ACS-TIR patterns across differ- ent task sets and step limits suggests that, under the same settings, an increase in TIR can sometimes coincide with a decrease ACS. We hypothesize two possible reasons for this phenomenon: a). TIR reflects the proportion of correct tool invocations. A higher TIR indicates higher proportion of cor- rect tool usage, which can enable the model to complete tasks more efficiently. b). The complexity of OS WCr tasks varies significantly, and the overall task completion rate remains relatively
8

---

Published as a conference paper at ICLR 2026
(a) Relationship between TIR and Acc, and ACS
(b) Relationship between available tool count and
TIR, Acc, and ACS
60
30
18
20
40
24
AC ALS
Bead The benwees ADc and
AMC
ACC
18,6M
WESS
Figure 5: (a): The relationships between TIR, Acc, and ACS, (b): The performance of Claude-4- Sonnet across task sets with different numbers of available tools. GM: GUI + MCP, G: GUI Only.
Table 2: Performance of Gemini-2.5-Por on OSWorld-MCP under different configurations.
Cuol-Beneficial
Non- Tonl-Beneiiia
Overall TIR
Agent Model
Settings
Accuracy
TIR
ACS
Accuracy
TIR
ACS
Accuracy
ACS
Base (MCP)
22.5
15.2
10.3
7.2
6.3
14.1
17.8
12.5
11.5
Gemini-22 P.o-Pro
Tools Shuffee
22.3 mag
14.8
10.4
13.1
12.5aa
13.9
19.5
14.1 14.1
11.5
16.3
11.2
8.7
74aat
7.2
13.6
10.1
89
low. When the maximum step limit is raised, models tend to make more attempts in solving complex tasks, which can counteract or obscure the efficiency gains that come from correct tool usage.
Tool Invocation Rate (TIR) positively correlates with task accuracy, but its re- lationship with ACS is complex and non-linear, suggesting that the impact of tool use on efficiency depends on various factors including task difficulty and model-specific strategies.
Finding 2:
Impact of the number of available tools. As shown in Figure 5 (b), we conducted experiments using the best- perfforff model from the previous evaluations, Claude 4 Sonnet, on the manually annotated task set described in Section 3.3 The tasks were grouped according to the number of available tools. For each configuration, we computed Acc, TIR, and ACS.
Firstly, in the results for the GUT-only configuration, we observe that as the number of available tools increases, Ace tends to decrease and ACS tends increase. This indicates that task difficulty rises with the number of available tools. Secondly.in the results for the GUI plus MCP Tools config- uration, we found that when the number of available tools was relatively small, Ace increased and ACS decreased. However, as the number of available tools grew, both Acc and TIR dropped sharply, and ACS gradually rose. A possible explanation that with fewer available tools, the tasks are rel- atively easier, and the model is more likely select the tools that are useful for the task, thereby completing it with higher accuracy and efficiency. When the number of available tools increases, the tasks become more complex. Even though often require multiple-tool combinations, making it difficult for the model to accurately select the most relevant tools, leading to reduced task accuracy and higher average completion steps. Thirdly, when comparing the GUI-only configuration with the GUI plus MCP Tools configuration, the latter consistently achieved lower ACS and higher accuracy overall. However, in cases where ACS was similar in both configurations, we found that the GUI plus MCP Tools configuration sometimes sulted in lower accuracy. We suspect that the use of MCP tool combinations more challenging for LMMs than combining GUI operations.
efficient tools might be available, these tasks
Finding 3: MCP tools generally improve performance in complex tasks, but their efficacy diminishes in extremely complex scenarios requiring combining multiple tools more challenging than combining GUI operations.
tool combinations. This indicates that

---

Published as a conference paper at ICLR 2026
4.3
ABLATION STUDY
To more comprehensively evaluate the tool invocation and decision-making capabilities of LMMs, we conducted a series of ablation studies. We selected Gemini-2.5-PP the model with the largest accuracy gain from MCP tools over the GUI-only setting, as the test subject for these experiments.
Impact of the Number of Callable Tools on Model Accuracy. the default OSWorld-MCP setup, the set of tools available at each step is a subset obtained via Retrieval-Augmenter Genera- tion (RAG), filtered to match the current application in use. To investigate the impact of the number of callable tools on model performance, we removed the RAG filtering and allowed the model to choose freely from all 158 tools for each task. As shown in Table 2 removing RAG led to a notice- able performance drop: the overall Acc decreased from 17.8 13.6. We attribute this the fact that descriptions of all 158 tools result in excessively long tool contexts, which markedly reduce the model's s tendency use tools, thereby impairing accurate tool invocation. For Tooe-beneficial Tasks, removing RAG reduced Acc from 22.5 to 16.3 and TIR from 15.2 to 11.2, indicating pro- nounced decline in the model's inclination Tasks, although Acc remained virtually unchanged after removing RAG, TIR increased from 6.3 to 7.2, suggesting that the model became more inclined employ the GUI rather than tools to solve the tasks.
invoke tools for problem solving. In contrast, for
Impact of Tool Description Order in Prompts. In the default OSWorld-MCY configuration, tool descriptions are provided to the LMM in alphabetical order. To investigate the impact of tool de- scription ordering on model performance, we randomly shuffled the descriptions prior to passing them to the LMM and re-evaluated on OSWorld-MCP. As shown in Table 2 random ordering in- creased overall Ace from 17.8 to 19.5. This result indicates that the ordering of tool descriptions in the prompt has a substantial effect on model performance in OSWorld-MCP. While the model's performance on Tool-beneficial Tasks was nearly unchanged before and after shuffling, differences were considerable for tends to invoke the corresponding tool when one is available, whereas in the absence of available tools, the description order may implicitly suggest alternative solution strategies. For consistency in evaluation, tool descriptions are presented to the LMM in lexicographical order in OSWorld-MCP.
Tasks. This may be because, with fewer tools, the model
4.4 CASE STUDY
In order demonstrate the tool invocation capability of the GUI Agent, we present complex example in LibreOffice Cale: Copy the "Revenue" column along with the header to new sheet named "Sheet2". This task requires creating new sheet and copying the specified column. The end- to-end Gemini-2.5-Pro agent accomplishes this by utilizing tools for creating a sheet and copying data, then switching to Sheet2 to verify the operation, as illustrated in Figure g In contrast, the agent without MCP tools fails to select the specific column, demonstrating how tools can serve as valuable complement to the agent's capabilities. More cases are analyzed in Appendix A.
5 CONCLUSION
We introduce OSWorld-MCP, a fair and comprehensive benchmark for evaluating Large Multimodal Models (LMMs) in computer-use scenarios by jointly assessing graphical user interface (GUI) op- eration skills and MCP tool-invocation capabilities. Using an automated pipeline combined with meticulous manual validation, we construct a high-quality and diverse collection of MCP tools that supports realistic and balanced evaluation within the OSWorld framework. Experiments on seven state-of-the-art LMMs and multi-agent frameworks demonstrate that tool invocation can substan- tially improve robustness and efficiency, while also revealing trade-offs between usage frequency and overall performance. Looking ahead, extending OSWorld-MCP to more complex, dynamic, and collaborative environments, as well as incorporating human-centred evaluation metrics, will further advance the development of penerol-purpose, efficient, and trustworthy computer-use agents.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
The OSWorld-MCP dataset is constructed entirely from publicly available software environments and tasks in OSWorld. All MCP tools included in the benchmark are either automatically generated and manually validated by the authors selected from existing open-source MCP servers, ensuring that no proprictary, confidential, or personally identifiable information included. The dataset contains only synthetic interaction records between multimodal agents and computer application environments; no human subject data are collected. We release OSWorld-MCP solely for research and educational purposes advance the evaluation of multimodal agents in realistic computer- use scenarios, Researchers using this dataset should comply with all applicable laws, institutional guidelines, and license terms. The authors bear responsibility for ensuring that the dataset is free of harmful or unethical content and that its use will not compromise privacy or security.
REPRODUCIBILITY STATEMENT
We will release all resources necessary to reproduce our work, including the OSWorld-MCP dataset, the complete set of 158 validated MCP tools, and the automated vode-generation pipeline used to create them. Detailed documentation will be provided to ensure that researchers can replicate experiments under the same conditions, including task definitions, maximum step settings, and all metric implementations (Accuracy, Tool Invocation Rate, and Average Completion Steps). All resources will be hosted in an open-access repository upon publication.
REFERENCES
Reyna Abhyankar, Qi Qi, and Yiying Zhang. Osworld-human: Benchmarking the efficiency of computer-use agents. arXiv preprint arXiv:2506. 16042, 2025.
Anthropic. What the model context protocol (mep)? httpss://noellcnntetttroooo 2024.
1o/docs/ http://eettin---taetddee
Anthropic. Claude Sonnet 4. cessed 31-08-2025].
2025. [Ac-
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 25f.:2500..1922. 2025.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08266 082644 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Oni Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint urXiv:2507.0626/ 2025.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-281 14, 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07778, 2024.
Abul Ehtesham, Aditi Singh, Gaurav Kumar Gupta, and Saket Kumar. A survey of agent interoper- ability protocols: Model context protocol (mep), agent communication protocol (acp), agent-to- agent protocol (a2a), and agent network protocol (anp). arXiv preprint 022229,05.00279, 2025.

---

Published as a conference paper at ICLR 2026
Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. Mcp-radar: A multi-dimensional benchmark for evaluating tool use capabilities in large language models. arXiv preprint arXiv:2505.16700, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025a.
Dong Guo. Faming Wu, Feida Zhu, Fuxing Leng. Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, al. Seedl. 5-v1 technical report. arXiv preprint arXiv:2505.0706, 2025b.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pp. 161- 178. Springer, 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint (tt:24001136449 2024.
Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. Os-harm: A benchmark for measuring safety of computer use agents. arXiv preprint arXiv:2506. 14866, 2025.
Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerrl: Scaling end-to-end online reinforcement learning for computer agents. arXiv preprint arXiv:2508. /4040, 2025.
Zhiwei Liu, Jielin Qiu, Shiyu Wang. Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, et al. Mcpeval: Automatic mep-based deep evaluation for ai agent models. arXiv preprint Xxiv:2507.12806 2025.
Xing Han LU, Zdenčk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi- turn dialogue. arXiv preprint ar arXhv:2402.059300 2024.
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing efficient action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620 2025a.
Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, al. Ui-sl: Advancing gui automation via semi-online reinforcement learning. arXiv preprint arXiv:2509.11545 2025b.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.104 10458, 2025.
Guozhao Mo, Wenliang Zhong, Jiawei Chen, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. Livemepbench: Can agents navigate an ocean of mep tools? arXiv preprint arXiv:2508.01790 2025.
OpenAl. Introducing gpt-5. Technical report, OpenAI, 2025a. URL https://///oe .omimaaa.com///ooommm zh Hans httee///nnnnnninninneeeeeeeeeeee
gpt-5/
OpenAl Openai o3 and 04-mini system card. Technical report, OpenAI, 2025b. URL
System Card.
Yujia Qin, Yining Ye. Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025.
12

---

Published as a conference paper at ICLR 2026
QwenTeam. Qwen3-vl: Sharper vision, deeper thought, broader action.
hctps:
/
Lesearchh.tttss--cvvvcnneeeeees 2025.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Camphell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024.
Simular Research. Agent-s. htttps://gtttbu.cmm//// 2025. [Accessed 23-09-2025].
ai/Agent ltt.ggnntttttteeeaaalllaaaaaaa S/tree/main
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, et al. Coact-l: Computer-using agents with coding actions. arXiv preprint arXiv:2508.0308 2025.
Shulin Tian, Ziniu Zhang, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multi- modal internet agents. arXiv preprint arXiv:2404. 09992, 2024.
Junyang Wang. Haiyang Xu, Haitao Jia, Zhang, Ming Yan, Weizhou Shen, Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. Advances Neural Information Processing Systems, 37:2686-2710, 2024a.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobilc-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024b.
Zhenhailong Wang, Haiyang Xu, Junyang Wang. Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint 2rXiv:2501.11733 2025a.
Zhenting Wang, Qi Chang. Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, et al. Mcp-hench: Benchmarking tool-using 1lm agents with complex real-world tasks via mep servers. arXiv preprint arXiv:2508.20455 2025b.
Tianbao Xie, Danyang Zhang. Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et Osworld: Benehmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52000 52094, 2024.
Tianbao Xie. Mengqi Yuan, Danyang Zhang, Xinzhuang Xiong, Zhennan Shen, Zilong Zhou, Xinyuan Wang, Yanxu Chen, Jiaqi Deng, Junda Chen, Bowen Wang. Haoyuan Wu, Jixuan Chen, Junli Wang, Dunjie Lu, Hao Hu, and Tao Yu. Introducing osworld-verified. xlang.ai, ai, July 2025.
URL http:: ang. a htwll//l//////oooo hsw.://wwrrlo wewwrrld-eeri.. d-verified
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 298.:22005003888. 2025.
Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Foundamental agents for gui automation. arXiv preprint arXiv:2508.15/4
2025.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. arXiv preprini arXiv:2307. 13854, 2023.
Jinguo Zhu, Weiyun Wang Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
MORE CASES
We present and analyze additional tool call examples to demonstrate how tool calls enable agents to complete diverse and complex tasks more efficiently and accurately.
A.1.1 CASE: VSCODE SETTINGS
The goal of the task is "Please help me modify VS Code setting hide all ppycache.... folders in the explorer view". Executed actions are as follows:
1.
swwo:ldmmp..ode..ddd..lee.....
pattern=""*/_bbaachhe"
2. computer.use
"ction="left.click"
x=1131.0
y=712.5
3. computer.use
action="ttermnntee"
status- "success"
Figure 6 showcases executing process. According to the evaluation program, the specific folder is hidden in the explorer view.
Initial state: zeset
Step I: oswer 1 www.code...oooooooommmmoooo add fi ls www.uuddeeeeeeoeooeooooooee Ip...
#
Step 2: wwmputer.use (actionn click",
Step 3: computer.use (ection "cermentttettt"
Figure 6: Please help me modify VS Code setting to hide all pycache..." folders in the explorer view.
A.1.2 CASE: FILESYSTEM OPERATION
The goal of the is "I have image of my receipt located in /home/user. I'm looking to transform it into a PDF file. Can you assist me with this task? Save the resulting PDF as 'receipt.pdf" on the desktop". Executed actions follows:
14

---

Published as a conference paper at ICLR 2026
1. oxvozzddngp.s.cooettt..gg.ee.oe
Ancegpattha///oomm///rrr//epppgp
2. fi flleysstemmisst.dreectrrr
path="/home//ser:
3. oxworr/damggssss..corrrttaaa...p
Imnggppathh//hmmm//ss////11111
oft
4. computer.use
"ction="terminate"
status="success
The agent first utilized the tool to convert an image to PDF format, but the tool response indicated the specified file or directory did not exist. By listing the files in the home directory path, the agent obtained the correct filename. The subsequent conversion attempt successfully produced the desired PDF file from the original image, as demonstrated in Figure
state: reset ()
Step aew.::..d.e..e...rrrr...........
Stop Filesystemmm111 lit.di rectozy (path-" /hom....
Stop 3: wwworldmme..ss.oommm convert wtw.tttiiineeeieeooooooooeoo (...
Step 4: computer.uue.om (action= "terminate", Sta...
Figure 7: have an image of my receipt located in /home/user. I'm looking to transform it into
PDF file. Can you assist me with this task? Save the resulting PDF as "receipt.pdf" on the desktop.
15

---

Published as a conference paper at ICLR 2026
A.1.3 CASE: SLIDE DUPLICATION
The goal of the task is "Please duplicate the last slides and insert the copies in alternating order, so the sequence becomes: original slide A, original slide B, then duplicated slide A, duplicated slide B" Executed actions are follows:
osworld wwp licreofficeeeeeeeee...oc.. hmpprrrrrroiiiiirrssss..so..sss
impress. get slide count
2. oxwor/dmm.lllbrroffii..iirrrsseo
catp:e/emmmmpprssdp.loccctttss wwwpscatppplllcccoocccooo target-position
s1ide.index=24
3. osworld mep libreoffice impress. slide count
4. osworld map libreoffice impress. dupl i cate s slide
slide_index=24
tarrget.position=25
5. ooter/ulaolllporrrr...ppeeeeeeee
slide index=23
target.position=21
6. asxo::/mmg.lbbeofffi....meea.a.d
slide_index=24
target 20s
7. oxworlldmcplllbrrefff..c.....eee hmm..rrrroff..i.mmmme......ee
8, computer.use
action="terminntee
"latuss=succes:
The agent initially determined the positions of the last two slides (indexed 23 and 24) through tool calls and attempted to directly duplicate the slides. However, the operation failed at steps 2 and 4 due to the incorrect parameter of target position. After several attempts, the agent successfully duplicated the final two slides to positions 25 at step 5, and 26 at step 6. As shown in Figure S the agent ultimately completing the task.
16

---

Published as a conference paper at ICLR 2026
Initial state: reset
Step I: Leeeorrddnep..... 11t.e:111bbrrfro impress
MLA 8th Edition Formatting and Style Guide:
MLA 8th Edition Formatting and Style Guide:
Works Cited: The Basics
Works Cited: The Basics
Step 2: oawor libreofrice www.rrroreeeeipeeeeeeeeem dupl.
Step Ld librectrice impress get....
MLA Edition Formatting and Style Cuide:
MLA Edition Formatting and Style Guide:
Works Cited: The Basics
Works Cited: The Basics
Step oewerlsooo.o d.mpp www.elfffeeffee..ooooooooo impresss dupl....
Step 5: osxor d.mep Abrec fficefieee.eee.ccc impress.................oo...... dupl
NOW
MIA Edition Formatting and Style Guide:
Mitebortenn bei
Moaderiaao retitaatiio repartace Shakk pagr
Works Clted: The Basics
Con warh
Worke Cline repertine 033
cruse propatt
DeteativE MLA Morks Clied for cesuunc
Step 6: baworld_nose www.ebfftiie..... htg.:e/eeaeibreooiiiieeeeeeeee dupl...
Step 7: wwworldmmep. wet.ssiiibeefficoc Empresseeeeeeeeeeeeeeeoemcmeeeee
Elode OR2 RAY ASH ViSlcE
The Andde ORL JUNTT AAH WitleR
'ILL Caasas Codde ERTAEL WiTS Clec
"MLL Chiise Orde
ERLAS: Wate Glsd Lart
ixsppciiicciii Acconed
Step 8: computer..ueem (act on "termniinaa"""" ata....
WORKS
The Fonder 033 Sastr Woriee
ahowd Acreved
Betteo Works Clive
arermed lng 2013
Figure& 8: Please duplicate the last two slides and insert the copies in alternating order, so the se- quence becomes: original slide A, original slide B, then duplicated slide A, duplicated slide B.
17

---

Published as a conference paper at ICLR 2026
A.1.4 CASE: SPREADSHEET COPYING
The goal of the task is "Copy the 'Revenue' column along with the header new sheet named 'Sheet2" Executed actions are as follows:
1. osworld mep libreoffice fibrrofficee wwtpbr/effic..aa.cooooooo..e coppycelllcl
source.range-
target target "
tarrget_start.ccel1="1111
2. hawp::/www..bboo........o....o.. het..enshlssbtwweeshhhheeee
source.range=?31::2000
target hww.:sshhetteeeeSSSeeeeee targge_sstart.cca11="11
3. oww.raamw.lbeeerrfrcccccoo.ootoo
4. opw.tpa....ooccccccc.......eeeee
source "our=ee2aang==A11:222"
"argetssheet..amm=="5hettt?
tarrge_sstart.ce1=="11
5. oww.iaam..lpeeffccc.aaaa.aaaaaaa
sheet name="Sheet2"
6. http::/gw..brooffoe...........e.
sheet..ram=="Sheet"""
7. hww.:///wwg...ee.....cl....eoeee
source_raagg==A11::000
target seaettsshettnnceee shethhhhhh=====
target.staat.cc31_"""
8. hww.:o/c.ggglbbeeeff..cc.aa..eed
sheet_rame==Shestt2
9. computer.use
"temint"t
"tatus="success:
According to Figure g the "Revenue" column has been copied to "Sheet2" successfully.
18

---

Published as a conference paper at ICLR 2026
Initial state: reset Q
Step wtwpp///ww.bbbeee..eseeeeeeeeeee
Step 2: oopy.cppyyooooooooooooooooooo... lespyyceellaaooee between s (source range.......
Step get www.wookkbobkooooo ()
Step 4: h.p...eell...btteee.aaaaattt (tourccrragg...
Step S: awitchh.etiiv..hhet (shest.name=" Sheet22))
6: switch.ac. sheet neme- SSeeet 1
Step www.y.ceell..eeeeee..eeeoo (source.range....
Step 8: www.cchaacrive.ahheeeeo (sheat_nahee paamos S
Step computer.useeeeeom (act "tstmmmiatt"""""tttttt" Sta...
Figure 9: Copy the "Revenue" column along with the header new sheet named "Sheer2".
19

---

Published as a conference paper at ICLR 2026
A.2 BAD CASES
We present and analyze additional tool call examples to demonstrate how tool calls enable agents to complete diverse and complex tasks more efficiently and accurately.
1.2.1 CASE: REASONING FAILURE
The goal of the task is "Lately I have changed my English name to Thomas. want update my username. Could you help me change the username in chrome profiles to Thomas?". The Claude-4- Sonnet model successfully open the profile setting in Figure [T]] filling the text box with new name Thomas. However, it immediately report finishing task without clicking anywhere else or press enter, leaving the name unchanged, finally caused the task failed.
Figure 10: Reasoning Failure
In another task, the goal is "Check the names in column "Names with duplicates" and put the unique ones in column "Unique Names". Keep the original order of the first occurrences", as shown in Fig- ure 11 The task requires multiple read-and-wiid operation, which can be efficiently and accurately cww.......ccccc.cccccccccccc.cc. wew.eeccelllvaleeeeeeeeeeeeeeaee
addressed by tool wwwporrddmp..ibbrere
in one step. Instead of utilizing MCP tools, Qwen2.5-VL attempted to call GUI operation only, clicking wrong cell during processing, which finally lead to an inaccurate input position and failed to finish the task.
The resuls demonstrated in Figure 12
Figure 11: Reasoning Failure (before)
20

---

Published as a conference paper at ICLR 2026
Figure 12: Reasoning Failure (after)
A.2.2 CASE: GROUNDING FAILURE
The goal of the task is "Can you make a new folder for me on the bookmarks bar in my inter- net browser? Let's call it 'Favorites". For OpenAl o3 model, it first opened the bookmark page by osworld_mcp. coep.google. bepogle_chromee mark bar for opening the context-menu, shown in [3 However, the model clicked the wrong position and failed to open the context-menu for times, leading to a failure of this task in 13
open_bookma bookma rks, and attempted to right-click the book-
Figure 13: Grounding Failure (before)
A.2.3 CASE: MCP TOOL INVOCATION ERROR
The goal of the task is "On my surface pro whenever I launch Chrome it always opens funbrain.com. don't want this. cleared my cache but it still happens. What should do?". For Qwen2.5- model, the thought of first steps in are: are:
To address the issue of Chrome always opening 'funbrain.com I
need to check if there are any extensions that might be causing this behavior. Opening the extensions page will allow
me to review and disable any suspicious or unwanted the three-dot menu in the top right corner of Chrome and select 'More tools! then "Extensions.'
extensions. \nAction: Anactionn Click
21

---

Published as a conference paper at ICLR 2026
Figure 14: Grounding Failure (after)
However, the following made action osworld wwwoorddddmmooooooggg e_chrome....ee www.hrom..ppeeeeeeeee.eeeeeeeeee open_extens ions, which misaligned with the previous thought. The mismatching between thought and action may cause potential impact.
Google
Figure 15: MCP Tool Invocation Error
A.2.4 CASE: MCP TOOL PARAMETER ERROR
The goal of the task is "So, I've been dabbling with coding Snake game in Python, and finally got up and rumning. It's pretty cool, but it's not without its quirks. The biggest issue I'm facing right now is that the snake can't code so the snake can actually eat the food? Thanks bunch!". For Qwen2.5-VI. model, it firstly checked the code with thought in 16
eat the food, matter what. Could you help tweak the
To ensure the snake "main.py" where the collision between the snake and the food is detected. The current issue might be related to how positions are compared or updated. AnAction: Click on the 'main.py' file in the explorer to review and modify the collision detection logic between the snake and the food.
eat the food, I need to check the logic in
However, raised a tool calling
httwooldd_ep_cooee "parameters": | ("file_path": "/path/to/main.py" "line": 26, "character"; 5)
22

---

Published as a conference paper at ICLR 2026
which passed totally wrong tool name and parameter the MCP server, leading failure of the task.
(
Figure 16: MCP Tool Parameter error
23

---

Published as a conference paper at ICLR 2026
Table 3: Performance on OSWorld-MCP.
Steps
Method
(mean-sstd)
TIR (mccnn+std)
TIR (CD)
(mean)stt))
ACS(CI)
Agent
36.7
(1.5
Agent 52.5
15
GUI+MCP
42.1
(10.4 11.99)
30.0 1.4
(28.431.6)
10.0
(9.8 10.2)
Agest $2.5
S0
GUI Ouly
47.1 + 2.7
(44.0) 50.2)
20.2 1
(196, 20.5)
Agest 52.5
50
GUIL-MP
40.5 + 2.1
(47.2 51
35.3 + 0.6
(34.6,55 35.9)
17.0 + 0.5
(1644177 17.6)
Seed-vl
27.9
(36.4, 29.4)
10.9
Sced.vi
(19.9
Sced-v]
50
GUI Only
33.9
35.8)
23.8 1:
(23.6, 24.0)
Sced-r]
50
GUI-MCP
38.2
(36.5, 39.8)
25.1 1.3
(23,622666)
22.3 1: 0.3
22.6
QwenI-VL
15
GUI Only
25.4
(23.8, 26
11.6 1:
11.77
Qwen3-VI.
GUI--MUP
21.5+11.
(192,
10.2)
Qwent-VI.
Only
25.6
26
Owen.-VL.
50
GUI-MCP
39.5 0.1
(39.1, 39.61
26.1 2.0
(238,
18.5
(18.1, 19.0)
Clande 4 Sonnet
15
GUI Only
302 2 1.7
(28,3, 32.11
11.9 1: 0.1
(1L8, 12.01
Clande Sonnet
15
GUI+MCP
36.1 0.3
(35.7.363)
27.4 0.0
(27:42774)
10.5
(10.3, 10.7)
Clande
Cladde Sonnet
50
(44.4 45.6)
33.3
(33.1,33.5
20.0
(19A,
Gemini-2.5 Pro
15
GUIl Dnly
74 -4++11
(39, 10.99
18.8 0.5
(13.3, 14.3)
Gemini-2.5 Pro
15
GUT MCP
17.4 1.4
(15.9, 19.0)
122 0.6
(11.6 12.8)
11.6 + D.1
(11.,,15)
Gemini- 2.5 Pro
50
GUI Only
13.3 1
(11.3, (5.3)
40.0
(38.1,
(1.33,1833)
(3..7
OpmnAl u3
15
Only
833 3.8
(2.6,
14.0
(13.8 14.2)
OpesA o3
15
GUI--MP
17.6
(1688 18.4)
11.6 + 0.5
(11.0,12.) 12.13
11,9 D.1
(11 12.0)
OpenAl o3
50
GUll Only
12.8 0.44
(12.3 13.2)
44,8 D.4
(44.4, 45.2)
OpenAl nã
50
GUI-MCP
24.1 0.
(5.21
16.0 0.6
(15.3,
(32.2 33.8)
Owen2.5.VL
Owen2.5-VL
15
GUI-MCP
14.5 0.2
14.8)
10.1
19.4, 10.7)
14.0
(13.9 14.0)
V-
S0
GUI Only
13.9 0.6
(13.3. 14.6)
30.5 1: 0.3
(30.2 30.9)
Qwen2.5-VL
50
GUR-MCP
156 044
(15.2, 16.0)
93
(8.7. 10.0)
39.0 4:
(38.4, 39.6)
A.3 VARIANCE MEASURES AND CONFIDENCE INTERVALS
To further accurately evaluate the performance of each agent, we conducted three independent runs for each agent. Table below reports the mean, standard deviation, and 95% confidence interval of Accuracy, TIR, and ACS for each model under different settings.
The experimental results show that in most cases, after introducing MCP, the performance differ- ences between runs under the same settings become smaller, and the models demonstrate greater stability. In certain cases, this may be related to the relatively low Tool Invocation Rate of the model.
A.4 MCP TOoL LIST
We present the complete list of 158 MCP tools as follows.
Table 4: Complete List of MCP Tools
No. MCP Server
Domain
MCP Tool
osworld_mcp
VS Code
add_folder
Adds folder to the last active window in VS-
Code
2
oswo
VS Code
compare. liiparrefille....
Compares two files in VSCode
3 osworld mep
VS Code
disable extensiie
Disables specific extension for the next in- stance of VSCode
4
osworld.mcp
VS Code
goto_file
Opens a file specific line and character po- sition
osworld.mpp
VS Code
install.extent ion Installs an extension or updates it in VSCode launch.vscode
osworld.mcp
VS Code
Launches Visual Studio Code with the speci- fied file path or directory
7
osworld mep
VS Code
list extension. Lists installed extensions in VSCode
24

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
osworld mep
VS Code
remove folder
Removes folder from the last active window VSCode
mep
VS Code
toggle_sync
Toggles synchronization on or off in VSCode
mcp
VS Code
uninstall.exter
Uninstalls am extension from VSCode update extensi ons
11
osworld mcp
VS Code
Updates all installed extensions in VSCode to the latest version
12
osworld.mcp
VS Code
add_f Les-exclude
Add a glob pattern to the Files: Exclude set- ting
13
osworld.mcp
VS Code
replace_ter text
Open VSCode search and replace panel, input search and replacement text, and execute re- placement
14
osw
VS Code
search.text
Open VSCode search panel, input the search term, and execute search
oew
VS Code
set_auto_save, _dyayy
Set the auto save delay in milliseconds set_color cccoorr_hheeeee Change the editor color theme set_focus_editon Set the Dehug: Focus Editor On Break setting to true or false
osworld.mcp
VS Code
osworld.mpp
VS Code
18
oswo
VS Code
set-pythhon hon_hooddooooooooooooooooo agnost i cs_overr ide Override severity for a specific Python analy- sis diagnostic rule
19 osworld.mcp
VS Code
bet_word_wwr .wp.poollooee column Set the number of columns at which the editor will word wrap
20
osworld_mcp
VS Code
aet_wrap_tabs Enable disable wrap tabs in the editor bookmark.page
21
oswo
Google Chrome
Bookmarks the current page in the browser (equivalent to Ctrl+D) bring boingg_ack__a_ back_aacaaassse lacck_lsstttbbbb Restores the last-closed tab in the browser (equivalent Ctrl+Shift++))
22
osworld_mep
Google Chrome
23
osworld_mcp
Google Chrome
appenttabsssetu Opens the entered URL delete browsing data Opens the 'Clear browsing data' dialog in the browser (equivalent to Ctr++ShfttDDll) open.appearance.sett
24
osworld mep
Google Chrome
25
osworld.mcp
Google Chrome
Opens the appearance settings page in the browser
26
osworld mep
Google Chrome
open bookmarks Opens the bookmarks page in the browser open_extenst ons Opens the extensions management page in the browser
27
osworld_mep
Google Chrome
28
osworld.mep
Google Chrome
www.-passswwddettts.ooos setgingst
Opens the password/autofill settings page the browser
25

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
29
osworld mep
Google Chrome
open privacy settings
Opens the privacy settings page in the browser
30
ocp
Google Chrome
open_pro filleeeseett inssttiings ngs
Opens the profile settings page in the browser wwwarcheeninn.eee.
31
osworld.mcp
Google Chrome
Opens the search engine settings page the browser
32
osworld mcp
Google Chrome
print
Opens the print dialog for the current browser page (equivalent Ctrl+P) adjust_co umn_width Adjust the width of specified columns ad jjust_rrw wow.heiee height Adjust the height of specified rows ropy_sheet
33
osworld.mep
LibreOffice Cale
34
osworld mep
LibreOffice Cale
35
osworld_mcp
LibreOffice Cale
Create copy of an existing worksheet in the workbook
36
osworld.mpp
LibreOffice Calc
sreate_chazp
Create chart in the active worksheet based on the specified data range create.p vot_table Create a pivot table in the active worksheet based on data from the source sheet env-info
37
osworld.mcp
LibreOffice Cale
38
msp
LibreOffice Cale
Get content the specified or active sheet, in- cluding its name, headers, and data
39
osworld mep
LibreOffice Calc
export to csv
Export the current document CSV file with the same path and name as the original file export. export.to.pdl tt.tt.ppff Export the current document or specified sheets PDF
40
osworld.mcp
LibreOffice Cale
41
osworld mep
LibreOffice Cale
format_rant .range
Apply formatting to the specified range in the active worksheet
42
ocw
LibreOffice Cale
freeze.pance
Freeze rows and/or columns in the active worksheet get_column. coolumuddatt data Get all data from the specified column get_worke wtt.wwobookkiifooooooo
43
osworld
LibreOffice Cale
44
osworld_mep
LibreOffice Cale
Get workbook information, including file path, file name, sheets and active sheet hi de row data
45
osworld mep
LibreOffice Calc
Hide rows that contain the specified value h gh1 i ight_range Highlight the specified range with the speci- fied color
46
osworld_mep
LibreOffice Cale
47
osworld.mcp
LibreOffice Cale
scrge_cells Merge cells in the specified range rename sheet.
48
osworld mep
LibreOffice Cale
Rename worksheet in the workbook
49
osworld_mcp
LibreOffice Cale
reorder_co umns Reorder the columns in the sheet according the specified order
26

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
50
osworld mep
LibreOffice Cale
reorder sheets
Change the order of worksheets in the work- book
51
oew
LibreOffice Cale
save Save the current workbook to its current loca- tion
52
osworld mep
LibreOffice Cale
set_cell aettcell.valee vaauue
Set a value to a specific cell the active work- sheet
53
osw
LibreOffice Cale
set_col umn_va I lues
Set values to the specified column, cannot be used to set formulas set net_number. forrmbt format Apply a specific number format to a range of cells
54
osworld.mep
LibreOffice Cale
55
osworld_mcp
LibreOffice Calc
set set_zoom_les Level
Adjust the zoom level of the current worksheet sort_col ccolum Sort the data in the specified column in ascend- ing or descending order switch_acd ivv_sli seeeeeet Switch to the specified sheet and make it ac- tive. Creates new sheet if doesn't exist transpose.rane
56
osworld.mpp
LibreOffice Calc
57
oswo
LibreOffice Cale
58
osp
LibreOffice Cale
Transpose the specified range and paste to the target cell
59
osworld mep
LibreOffice Calc
copy cells between sheets Copy cells from specified rectangular source range to another sheet fill.blan ank_down down
60
osworld.mcp
LibreOffice Cale
Forward-fily fills blank cells in specified columns with value from cell above
61
osworld mep
LibreOffice Cale
format hww.ttrnnmbrra..mo to human raaimanneedallleee Convert numeric values to human-readable strings (M for millions, B for billions)
62
mcw
LibreOffice Cale
ww..frrsssshee...... www..neeert....eeccccmom Scales the first sheet to specified pages and ex- ports to PDF save Save the current document to its current loca- tion
63
LibreOffice Writer osworld.mep
64
osworld_mpp
LibreOffice Writer
write.text
Write text the current cursor position in the document
65
osworld mep
LibreOffice Writer
set-color
Changes the color of matched text in the doc- ument Find_and_reple lace
66
mep
LibreOffice Writer
Finds and replaces text in the document set font Changes the font of text in the document set_l: ne_spacci ng Sets the line spacing for specified paragraphs insert-formul. wtt.mullaaa..uusseee.
67
osworld mep
LibreOffice Writer
68
osworld_mep
LibreOffice Writer
69
osworld.mep
LibreOffice Writer
Inserts formula at the current cursor position insert insert_image... www.mmaeea..uureeeeeee
70
osworld.mpp
LibreOffice Writer
Inserts an image at the current cursor position
27

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
71
osworld mep
LibreOffice Writer
set font size
Changes the font size of specified text export_to_pdl Exports the current document PDF format set vet.pparagraph.. all gnment
72
ocp
LibreOffice Writer
73
osworld.mcp
LibreOffice Writer
Sets the text alignment for specified para- graphs
osworld mep
LibreOffice Writer
set www.ultallt...mmm font Sets the default font for new text without changing existing text
75
osworld.mep
LibreOffice Writer
add_page_numberr
Adds page numbers to the document at speci- fied position
76
osworld_mcy mep
LibreOffice Writer
nseer pnsert_pageeee... break page_beeak
Inserts page break at current cursor position env_info
77
osworld_mcp
LibreOffice Writer
Retrieve all paragraphs, truncate each to most 500 characters
78
osworld mep
LibreOffice Impress
configua auto save Enables or disables auto-save functionality Dellet__contett
79
osworld_mcp
LibreOffice Impress
Deletes the specified textbox from a slide dupl 1 iiaate e e_s lide
osworld_mcp
LibreOffice Impress
Creates duplicate of specific slide env_info
81
osworld.mpp
LibreOffice Impress
Get the content of the specified pages export to image Exports the current presentation specific slide to an image file get_sl _o-un
82
osworld mep
LibreOffice Impress
83
osworld.mep
LibreOffice Impress
Gets the total number of slides in the current presentation
84
osworld mep
LibreOffice Impress
go po_tt__lll soo_siide Navigates to a specific slide the presentation insert_file
85
osw
LibreOffice Impress
Insects a video or audio file into the current or specified slide
86
mew
LibreOffice Impress
cnsert_image Inserts image specifie slide position_bs
87
osworld_mcp
LibreOffice Impress
Positions textbox image slide spe- cific location
88
osworld.mpp
LibreOffice Impress
save
Save the current presentation to its current lo- cation
89
osworld_mep
LibreOffice Impress
save_as
Saves the current document to a specified lo- cation
90
osworld.mop
LibreOffice Impress
set bet.backgrronndcolozz
Sets the background color for the specified texthox
91
osworld_mep
LibreOffice Impress
set_sl ide_background
Sets the background color or image for a spe- cific slide or all slides
92
osworld.mop
LibreOffice Impress
set foot
Sets the font style for all text elements spe-
cific slide
28

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
93
osworld mep
LibreOffice Impress
set siide orientation
Changes the orientation of slides between por- trait and landscape
94
LibreOffice Impress oep
set-style
Sets the style properties for the specified textbox
95
osworld mep
LibreOffice Impress
Sets the text color for the specified textbox set_tet__atri kethrough Applies or removes strike-through formatting to text
96
osworld_mcp
LibreOffice Impress
97
osworld mep
LibreOffice Impress
set_ tet_texboox al alignment Sets the text alignment for the specified textbox
98
osworld.mcp
LibreOffice Impress
write.text
writes text to a specific texthox on a slide convert.to.des
99
oep
LibreOffice Impress
Transfers all text slide-by-slide into new Writer document
100
osworld OS
change eee.tteeettt.eeeeeeeeeeee -tett-scallleeeee scale Changes the text-scaling factor and returns the previous value
101 osworld_mcp OS
configure_autee autoollckk
Configures the GNOME automatic-lock be- haviour
102
osworld mep OS
copy match i ng files with hi hierarchy Copies all files matching a pattern preserving directory hierarchy
103
mcp os
get_do _not bot_dist turbstaatu Returns the current Do Not Disturb state get_text.scall
104
osworld.mop mep os
Returns the current GNOME text-scaling fac- tor
osworld.mcp OS
get_trash. rectory
Returns the absolute path to the user Trash 'files' directory
106
osworld mep OS
get_volume
Returns the current output volume (percent- age)
107
osworld_mcp OS
open_shell
Open new terminal directly
108
osworld.mpp OS
Safely renames folder on the local filesystem
109
osworld mep OS
restore file
Restore the specified file from the user Trash back to its original location
110 mcp OS
search-files
Recursively search all files under the given folder
111
osworld mep OS
set default terminal size Persists the given number of columns x rows the default GNOME-TTrminal window size set_do_nos _nt_non_d i sturb
112 osworld_mcp os
Enable disable the GNOME/Ubuntu Do Not Disturb mode
29

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
113
osworld mep OS
set volume
Sets the output volume of the default PulseAu- dio / Wire sink
114
mep
OS
menvert.image.for
Convert an input image file to a specified for- mat
115
osworld mep os
ffmeeg vi video ridee_tt.ogg to..qf
Extract portion of video file and save an animated GIF
116 msw os
git_operation
Execute git command (clone, add, commit, pull, push, etc.)
117 osworld os
rit_set.user.inf
Configure git user.name and user.email remove ammoveimmaee background
118
osworld.mcp os
Remove the background of an input image and save with transparency
119 mcp oS
calculator
Simple calculator.
120
osworld mep
VLC
add to playlist
Adds media file to the VLC playlist rett_cureett.e. time Gets the current playback time position sec- onds
121
osp
VLC
122
msw
VLC
get_media www.diiaaduuatt.. aduuurr ion
Gets the total duration of the currently playing media file in seconds
123
mep
VLC
get_med waw.iiaaflliio a_files Gets the media files for the specified path get-play ay ist
124
osworld.mpp
VLC
Gets the current VLC playlist with track infor- mation
125
osworld
VLC
vet_settings
Gets the current settings of the VLC player next
126
mep
VLC
Switches to the next media item in the VLC
playlist
127 VLC
mcp
pause
Pauses the currently playing media in VLC player
128 VLC
osworld_mep
play
Starts playing the current media in VLC player previous
129
osworld.mpp
VLC
Switches to the previous media item in the VLC playlist
130
osworld_mcp
VLC
set_settings Sets the settings for VLC player
131
osw
VLC
Soggle.fullsrres
Toggles fullscreen mode for the currently playing video
132 filesystem
OS
read file
Read the complete contents of file text (DEPRECATED)
133 filesystem
OS
read.text.file
Read the complete contents of file as text with encoding handling
30

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
134
filesystem
OS
read media file
Read image audio file base64 encoded
data
135
filesystem
OS
geadmmultiple.iil
Read the contents of multiple files simultane- ously
136
filesystem
OS
write_file
Create new file or completely overwrite an existing file
137
filesystem
OS
edit_file
Make line-based edits to text file
138
filesystem
OS
create create directory
Create new directory ensure directory exists
139
filesystem
OS
list_di rect
Get detailed listing of all files and directories in specified path
140
filesystem
OS
list_di rect wwwwithhsiies
Get detailed listing with sizes of all files and directories
141
filesystem
OS
directory_ce reeee
Get recursive tree view of files and directo- ries as JSON
142
filesystem
OS
move_file
Move or rename files and directories
143
filesystem
OS
sea rch files Recursively search for files and directories matching a pattern
144
filesystem
OS
get_file.ints
Retrieve detailed metadata about file or di- rectory
145
filesystem
OS
list_al lowed directories
Returns the list of directories that this server is allowed access
146
git
OS
git_status
Shows the working tree status
147 git
OS
aged
Shows changes in the working directory that
not yet staged
148 git
OS
git.diff.stageee
Shows changes that staged for commit git_diff
149 git
OS
Shows differences between branches or com- mits
150 git
OS
git-committ
Records changes to the repository git_add
151 git
OS
Adds file contents to the staging area
152 git
OS
git.reset
Unstages all staged changes
153 git
OS
git log
Shows the commit logs g ait_create_bbart
154 git
OS
Creates a new branch from an optional base branch
155 git
OS
git.checkout
Switches branches
31

---

Published as a conference paper at ICLR 2026
No.
MCP Server
Domain
MCP Tool
156
git
OS
git show
Shows the contents of commit
157
git
OS
git_init
Initialize new Git repository
158
git
OS
git_branch
List Git branches
B
PROMPT
The following shows the complete prompt send the LLM during task execution, together with the corresponding screenshot. The user's instruction is: Please help me modify VS Code setting to hide all " pycache folders in the explorer view.
System prompt:
You are helpful assistant.
You may call one more functions to assist with the user query.
You provided with function signatures within <tools>
XML tags:
("type": "function", "function": ("name": "computer use", "description": "Use a mouse and keyboard to interact with computer, and take screenshots. \n This an interface desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications. n Some applications may take time to start or process actions, you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and window doesn't I open, try wait and taking another screenshot. |n The screen's resolution is 1280x720. n Whenever you intend to move the cursor to click an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.(n" If you tried clicking load, even after waiting, try adjusting your cursor position so that the tip of the cursor visually falls on the element that you want to click, \n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't 't click boxes on their edges unless asked.", "parameters": {" ("properties": ("action": ("description": "The action to perform. The available actions a 'key": Performs key down presses on the arguments passed in order, then performs key releases in reverse order. n 'type": Input string of text. Use the 'clear' parameter to decide whether to overwrite the existing text, and use the 'enter' parameter to decide whether the enter key should be pressed after typing the text. \n 'mouse_move', Move the cursor to a specified (x. y) pixel coordinate on the screen. n 'click": Click the left mouse button at a specified (x, y) pixel coordinate on the screen. \ 'drag': Click at a specified (x, y) pixel coordinate on the screen, and drag the cursor to another specified (x2, y2) pixel coordinate on the screen. n "right click": Click the right mouse button a specified (x, y) pixel coordinate the screen. n 'middle_click': Click the middle mouse button at a specified (x. y) pixel coordinate on the screen. \n 'double_click" Double-click the left mouse button at a specified (x. the screen. \ 'scroll': Performs scroll of the mouse scroll wheel. \ 'wait": Wait specified seconds for the change to happen. |n 'terminate": Terminate the current task and report its completion status.". "enum": ["key", "type", "mouse move", "click", "drag", "right_click", "middle_click" "double_elic" "string"). "keys": ("description"": "Required only by "action=key""" "type": "array"), "text": ("description": "Required only by "action=type""" "type": "string"). "clear": ("ddescrption": "Assign it to 1 if the text should overwrite the existing text, otherwise assign it to O. Using this argument clears all text in an element. Required only by "action=type"", "type": "number"), "enter": ("description": "Assign to if the enter key should pressed after typing the text, otherwise assign it to O. Required only by "action=type"", "type": "number"). "coordinate": ("description": "(x, y): The (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to.", "type": "array"), "coordinate2": ["description": "(x2, y2):
program or link but it failed to
y) pixel coordinate
"scroll", "wait", "terminate"), "type":
32

---

Published as a conference paper at ICLR 2026
The x2 (pixels from the left edge) and y2 (pixels from the top edge) coordinates to drag the cursor to. Required only by "action=drag""" "type": "array"), "pixels": ""description": "The amount scrolling to perform. Positive values scroll up, negative values scroll down. This value should be between -5 and 5. Required only by "action=scrolll",, "type": "number"), "time": f"description": "The seconds to wait. Required only by "action=wait". "type": "number"]: "status": ("description". "The status of the task. Required only by 'action=terminate "type": "string", "enum": ("success", "failure"]}}, "required": ["action"], "type": "object"]]] ("type": "function", "function": ("name": "osworld mep code.add folder". "description": "Adds a folder to the last active window VSCode", "parameters": ("type": "object", "prop- erties": ("folder": ("type": "string", "description": "The folder path to add")), "required": ["folder"]]]
("type": "function", "function": ("name": "osworld www. www.oee.omaaeeeiiiiiii fille"" "description". "Compares two files in VSCode", "parameters": {"type": "object", "properties": {"filel": ["type": "string", "description": "The path the first file"), "file2": ("type": "string", "description": "The path to the second }. "required": ["flll", "file21))}
("type": "function", "function": ("name": "osworld_mqg" cocp_code.disablle eetpbleeexxtttto""" "descrip- tion": "Disables specific extension for the next instance of VSCode", "parameters": ("type": "object", "properties": (""x"""""eeeea ("extension id": ("type": "string", "description": "The identifier of the extension")], }, "required": rextensioo_id"]] ("type": "function", "function": ("name": "osworld mep code.goto file", "description". "Opens file at specifie line and character position", "parameters"" ("type": "object", "properties": ("file_path": ("type": "string", "description": "The file path to open"). "line": ("type": "integer", "description". "The line number navigate to", "default": 1}, "character": ("type": "integer", "description": "The character position to navigate to", "default": 1}}. "required": f'llle path"]}}}
"descrip- tion": "Installs an extension or updates in VSCode". "parameters": ("type": "'ojject", "properties": ("extension_id" id": ("type": "string", "description": "The identifier of the exten- sion"), "pre_release" ("type": "boolean", "ddscription": "Whether to install the pre-release version", "default": false) }. "required":: ["extension_id"]]]]
("type": "function". "function": f"name": "osworld wwpp www.c/o...nstlleeeeeeeeee
"description": "Launches Visual Studio Code with the specified file path directory", "parameters": ("type": "object", "properties": ("path": ("type": "string", "description": "The file path or directory to open in VS Code")), "required": ["path"]}]]
("type": "function", "function": ("name": "osworld mmcpccodc.launhh
{"type": "function", "function": ("name": "osworld http__mpppccpceeeeoe list. ixsteeexttssons" "descrip- tion": "Lists installed extensions in VSCode", "parameters": {"type": "object", "properties": ("type": "boolean", "description": "Whether to show extension versions". "default": false), "category": f"type": "string". "description": "The category to filter extensions by"}}}}}
("type": "function", "function": ("name": "osworld_mey httdd__ep_codeeeeeeeeee foldeer. "description": "Removes folder from the last active window in VSCode", "parameters": ("type": "object", "properties": ("folder": {"type": "string", "description": "The folder path to remove")),
"required":: ["folder"]]]]
("type": "function", "function": ("name": "osworld fosworld.mep_coml wwwp:/mccppccdcc.eeeoeeme taagletogllesyntnne "description": "Toggles synchronization or off in VSCode", "parameters": f"type": "object", "properties", ("state": ("type": "string", "description": "The state to set ('on' or 'off')", "enum"" ["on", "off')), "required": ["state"]]))
"descrip- tion": "Uninstalls an extension from VSCode", "parameters": ("type": "object", "properties". ("extension id": ("type": "string", "description": "The identifier of the extension") "re- quired":: ["extension id'])}}
("type": "function", "function": f"name": "osworld_mqpp wwwpddddmppccode.ung www.euunnnnnllltteeennt
id'"])
("type": "function", "function": ("name": "osworld_mme wacpd_mmp___epee_e coop_codeuudatee extensions". "descrip- tion": "Updates all installed extensions in VSCode to the latest version". "parameters": ("type": "object", "properties": {}}}
("type": "function". "function": ("name": wwworrldmmp_codee... wttp:/oddeddd.llll.eeeeeeeeee .xcluud", "descrip- tion": "Add glob pattemn to the Files: Exclude setting.". "parameters": ["type": "object", "properties": ("pattern": f"type": "string", "description": "Glob pattern to exclude, e.g.
33

---

Published as a conference paper at ICLR 2026
**% "y/_pycache..."" "required":: ["patemm")]]
("type": "function", "function": {"name": "osworld mep code.replace text", "description". "Open VSCode search and replace panel, input search and replacement text, and execute replacement in either all files current file.", "parameters": ("type": "object", "properties": ("search_text" ("type": "string", "description": "The text to search for.", "replace_text" ("type": "string". "description": "The text to replace matches with."), "all_files": ("type": "boolcan", "description": "True to replace in all files, false to replace only in the current file. Default is True.") "required": ["search "replace "replace.text"]] text"]]}}
f"type": "function", "function": f"name": "osworld wwtp_mcppppop____ www..search........oeee text", "description": "Open VSCode search panel, input the search term, and execute search either files or current file.", "parameters": ("type": "object", "properties": ("text": ("type": "string", "description": "The text to search for."), "all_files": ("type": "boolean", "description". "True to search in all files, false to search only in the current file. Default is True.") ", "required". ["
("type": "function", "function"": ("name": "osworld mep code.set auto save delay", "descrip- tion": "Set the auto save delay in milliseconds.", "parameters": ("type": "object", "properties"; ("delay
: mss:
("type":
"integer", "description": "Delay before auto save in milliseconds."
"required":: ("delay mms"}}}
("type": "function", "function": ("name": "osworld_mep www.:/mppccooooooo wott...cootttemmee "descrip- tion": "Change the editor color theme.", " "parameters": {"type": "object", "properties": ("theme name": ("type": "string", "description": "The name of the theme to apply. egg. DDefault Dark+" ")), "required":: I'theme name"]}
("type": "function", "function": ("name": "osworld_mey sectordddm___oooooe coapppcooessett.eeeeee focus editor on _ate_oo__eeee brebkk"," "description": "Set the Debug: Focus Editor On Break setting to true or false.". "parameters": ("type": "object", "properties": {" ("value": {"type": "boolean", "description": "True to enable focus editor when debugger breaks, false disable.")) "required": ["value"[|)}
("type": "function", "function", ("name": "osworld mep code.set python diagnostics override", "description": "Override severity for a specific Python analysis diagnostic rule.", "parameters": ("type": "object", "properties": ("rule": ("type": "string", "description": "Diagnostic rule ID, e.g. "reportMisinggnpposs..
"severity": ("type": "string". "description": "Severity level, e.g. 'info', 'warning'. 'error', or [uone""}}, "required": ["rule", "severity[]]]
{"type": "function". "function": ("name": "osworld mep code.set word wrap column", "description": "Set the number of columns at which the editor will word wrap.", "parameters": f"type": "object", "properties": f"column". ("type": "integer". "description". "The column number where lines should wrap, e.g. 80 120."}], "required":: ["column"]]]] ("type": "function", "function": ("name": neworld_mep_codde "errepsttwrrappasss "description": "Enable disable wrap tabs in the editor.", "parameters": {"type": "object", "properties": f"value": ("type":: "boolean", "description": "True to enable wrap tabs, false to disable.")] "required":: ["value"]]]]]
</tools>
For each function call, return json object with function name and arguments within <oollccll>><<tool.ccmm tags:
<tool call>
("name": <function-name:: "arguments": <args-json-objett>)
</tol_call>
User Prompt
Please generate the next move according to the UI screenshot, instruction and previous actions. \n Instruction: Please help modify VS Code setting hide all ppyccche_... folders in the explorer view.
Previous actions:
Step 1: I first clicked "Yes, trust the authors" to dismiss the trust dialog, then used the 'add.file www.seeeellueeeee function to add the glob pattern "#s/_pyyachh__. to VS Code's
34

---

Published as a conference paper at ICLR 2026
Files: Exclude setting. This pattern will hide all pycache folders any depth in the project structure from the explorer view.
Tool call result:
Call (content= (TextCoeeeet (type="text' text='true', annotations=None, meta=None)), data=None, is [s_error=False)
Before answering, explain your reasoning step-by-step in and insert them before the <tool call></tool aallxXML tags. (nAteer answering, sum- marize your action in <conclasion><<coonuusiioss
and insert them after the
<tool_call>><< <all>><<too_cal>>>>> CMM.laMMMMMMMM............. tags.
History Images
SCREENSHOOT OF STEP 1:
Youul Studo Code
SCREENSHIOT OF CURRENT STEP:
Studie
iiI
B. USE OF LLMS
Large language models (LLMs) are used solely assist the preparation of this manuscript. They help improve the clarity, coherence, and conciseness of the text, refine phrasing, and ensure that the language conforms academic writing standards. All conceptual content, experimental design, data analysis, and conclusions are developed entirely by the authors.
35


Reviews:
[{"review_id": "hD6NeYuoX1", "reviewer": "Reviewer_8XpA", "summary": "The paper introduces OSWorld-MCP, a new benchmark built on OSWorld for evaluating multimodal computer-use agents that perform both GUI operations and Model Context Protocol (MCP) tool invocations. It extends OSWorld by adding 158 validated MCP tools across seven common applications and supplements existing metrics with two new ones—Tool Invocation Rate (TIR) and Average Completion Steps (ACS)—to measure tool-use propensity and efficiency. An automated pipeline using OpenAI o3 is proposed to generate and curate high-quality tools. Experiments on several state-of-the-art multimodal models demonstrate that MCP tools substantially improve task performance, though multi-tool composition remains challenging. Overall, OSWorld-MCP offers a fair and realistic framework for benchmarking tool-augmented computer-use agents.", "questions": "1.Tool Description Impact: How does the phrasing or length of tool descriptions influence model performance? Have the authors observed any sensitivity to wording or order beyond the shuffle experiment?\n\n2.Failure Case Analysis: Can the authors provide an analysis of common failure cases—specifically, whether models fail by selecting the wrong tool or by invoking the correct tool with incorrect arguments?\n\n3.Metric Interpretation: How should Tool Invocation Rate (TIR) be interpreted when models invoke tools unnecessarily or inefficiently—does a higher TIR always indicate better decision-making quality?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is original in extending OSWorld to evaluate MCP-based tool invocation, a capability overlooked in prior benchmarks. Its methodology is solid, combining automated tool generation with manual validation and extensive experiments on several state-of-the-art models. The presentation is clear and well-structured. The significance is high—OSWorld-MCP offers a realistic, open benchmark that will benefit research on tool-using multimodal agents. Its usefulness will further grow as more GUI applications adopt MCP-compatible tools.", "weaknesses": "The benchmark’s scope is somewhat limited—its 158 tools cover mainly seven desktop apps, leaving out broader web or cross-platform settings. The tool-generation pipeline lacks quantitative evaluation of success and failure cases. Some tasks remain similar to the original OSWorld, reducing novelty in task design. While TIR and ACS are useful, their interpretation could be clearer. Finally, the benchmark’s long-term value depends on wider MCP adoption across GUI applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wy4EUNJsj6", "reviewer": "Reviewer_cXcb", "summary": "This paper introduces OSWorld-MCP, a new benchmark for evaluating AI agents' ability to use both GUI and external tools via the Model Context Protocol (MCP). The authors developed a set of 158 high-quality tools and found that using them generally improves agent performance on various computer tasks. However, the results also show that even the most advanced models still struggle with effectively invoking these tools, indicating significant room for improvement.", "questions": "1. Could you please provide more details on the Retrieval-Augmented Generation (RAG) system? Specifically, what retriever model was used, and how was the query formed at each step to filter the 158 tools?\n2. What was the exact prompt template used to integrate the visual information (screenshot), the task instruction, and the descriptions of the RAG-filtered MCP tools for the LMM to make a decision at each step?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Creation of a High-Quality and Rigorously Validated Toolset**: The paper's primary contribution is the construction of a novel benchmark centered around a \"curated collection of 158 high-quality tools\". The reliability of this toolset is strongly supported by a meticulous creation process, which combines a \"novel automated code-generation pipeline\" with \"rigorous manual validation\". This validation was not superficial; each tool was independently assessed by at least two experienced reviewers to ensure its \"correct functionality, practical applicability, and versatility\", establishing a robust and trustworthy foundation for future agent evaluation.\n2. **Introduction of More Granular and Insightful Evaluation Metrics**: Moving beyond simple task accuracy, the paper introduces two new metrics—Tool Invocation Rate (TIR) and Average Completion Steps (ACS)—to enable a \"more nuanced evaluation\" of agent capabilities. TIR offers crucial insights into an agent's \"tool utilization propensity,\" while ACS directly \"quantifies task completion efficiency\". \n3. **A Unified Benchmark Bridging GUI and Tool-Use (MCP)**: A significant strength is the benchmark's design, which integrates GUI operations with MCP tool invocation in a single, dynamic environment. This approach directly addresses the \"gap between pure-GUI and text-based tool-use evaluations\", creating the first \"comprehensive and fair benchmark\" of its kind. By forcing agents to autonomously choose between GUI actions and MCP tools at each step, OSWorld-MCP provides a \"holistic and realistic assessment\" of an agent's hybrid decision-making skills in complex, real-world computer tasks.", "weaknesses": "1. **Incremental Novelty Built on an Existing Framework**: While the work is a significant engineering effort, its core contribution is presented as an extension to the pre-existing OSWorld benchmark. The paper states it is \"Built upon a widely used... environment OSWorld\" [cite: 077-078]. Consequently, the foundational novelty could be viewed as somewhat limited, as it enhances an established platform rather than introducing a completely new paradigm for agent evaluation. \n2. **Insufficient Depth in Analyzing the Hybrid GUI-MCP Interaction**: The paper's central premise is that evaluating the choice between GUI and MCP is more realistic. However, the analysis could delve deeper into the nuances of this hybrid decision-making. While the results in Table 1 show that adding MCP tools improves outcomes, the paper does not offer a detailed qualitative analysis of the agents' decision-making process itself. For example, it lacks a thorough exploration of common failure modes, such as when an agent illogically prefers a complex GUI sequence over a simple tool, which would offer richer insights into the current reasoning limitations of models.\n3. **Current Limitations in Application and Task Diversity**: The benchmark's scope is currently constrained, as the authors acknowledge that \"MCP servers were not developed for GIMP and Thunderbird\". This gap limits the benchmark's ability to assess agent performance across a truly comprehensive range of real-world activities and narrows the generalizability of its conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wEMc0Zc2PC", "reviewer": "Reviewer_SQsy", "summary": "The paper introduces OSWorld-MCP, a benchmark designed to fairly and comprehensively evaluate multimodal computer-use agents that combine GUI interaction with Model Context Protocol (MCP)–based tool invocation. It extends OSWorld by integrating 158 manually verified MCP tools covering 7 common applications (LibreOffice, VS Code, Chrome, VLC, OS utilities, etc.).", "questions": "1. How was task difficulty balanced between GUI-only and MCP-enhanced versions to ensure fairness?\n\n2. Could tool invocation failures be categorized (e.g., recognition vs. reasoning vs. execution errors)?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The automated pipeline (generation, filtering, wrapping) for tool creation is well-structured and uses both LLM-based and manual verification steps.\n\n- Covers multiple proprietary and open models under consistent settings, providing valuable comparative insights.", "weaknesses": "- While benchmark creation idea is strong, analysis mostly confirms intuitive findings (e.g., higher TIR means higher accuracy)\n\n- The paper’s main contribution (augmenting OSWorld with MCP tool invocation) is meaningful, but I’m concerned that OSWorld’s task design predates MCP and may not naturally align with tool-based workflows. The authors partly mitigate this by revalidating 361 tasks, identifying 250 “tool-beneficial” cases, and manually curating 158 tools, which shows care in adaptation. However, the benchmark still fundamentally inherits OSWorld’s task definitions, rather than introducing new ones designed from the ground up for MCP-enabled agents. This limits its novelty and may constrain how representative it is of modern tool-use scenarios\n\n- Some results (Tables 1–2) lack significance testing or variance measures; performance improvements could benefit from confidence intervals.\n\n- While clear overall, the draft is verbose in places and could improve figure readability and section organization. E.g. Figure 1 contained red wavy dash, need fix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S8nshRHXm1", "reviewer": "Reviewer_DEYi", "summary": "This paper presents OSWorld-MCP, a novel and comprehensive benchmark framework to evaluate computer-use agents within the OSWorld benchmark. This framework incorporates tool use (via the Model Context Protocol, or MCP) into the standard GUI benchmark. The authors show that by leveraging the MCP tools, agents achieve improved task success in OSWorld tasks.\nThis paper also introduces an end-to-end pipeline to curate MCP tools, leveraging both LLM tool generation and human verification. The paper presents 158 high-quality, manually-validated MCP tools, allowing agents to autonomously choose between GUI operations and tool calls.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a solid, well-written paper. The experimental design is very detailed, and the authors do a great job discussing the results, which are supported by clear visualizations. I especially appreciated the ablation studies on the number of tools, the RAG module, and the order of tools in the prompt—these were very interesting. The detailed case studies in the appendix are also a helpful addition.\n\nThe core idea of combining MCP tools with a GUI agent is novel. This is a strong piece of work that should be a valuable contribution to research on both MCP and GUI agents.", "weaknesses": "I would like to see more detail on the RAG module's design for the MCP tool selection. The paper doesn't specify what retrieval method was used or its retrieval accuracy, which makes it hard to fully understand that component of the experiment. Additionally, the tools were generated by \"producing code-based solutions for every task in OSWorld\", which may cause an unfair comparison. Since the tools were created using the same tasks they were evaluated on, there's a risk of task information leakage. It would be better to generate the tools using one set of tasks and then evaluate their performance on a separate, held-out validation set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)