Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

Authors: Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Keunho Jang, Zheng Hui
Primary area: datasets and benchmarks
Keywords: multi-modal agent

Abstract:
Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

Decision: reject
Reviewer scores: 6, 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730366620537

thread_type:
official_review

<thread_object>
creator: reviewer_wyzz

<content>
summary:
This paper introduces WINDOWSAGENTARENA, a benchmark designed to evaluate multi-modal agents interacting with a real Windows OS environment. The primary goal is to address the limitations of existing agent evaluation frameworks (which are often constrained to specific domains or modalities) by providing a scalable and reproducible environment that allows agents to perform a wide range of tasks within the Windows ecosystem. The benchmark includes 150+ tasks across various domains such as document editing, browsing, coding, and system management. 

Additionally, the authors propose a new multi-modal agent, Navi, and report its performance on WINDOWSAGENTARENA, achieving a 19.5% success rate. The paper also compares Navi’s performance to human benchmarks and other web-based agent benchmarks like Mind2Web.

strengths:
- The WINDOWSAGENTARENA augments the OSWorld benchmark and extends the evaluation environment to Windows, which is a good complement to the existing benchmarks.
- The authors accelerate the evaluation by enabling parallelized task execution in Azure. This is an important contribution for researchers working with time-consuming multi-step tasks, as it reduces evaluation times from days to minutes.
- The benchmark includes a relatively diverse set of tasks (154 in total), covering different types of user workflows.

weaknesses:
- It looks to me that this benchmark is merely an extension of OSWorld to the Windows environment, with similar tasks, evaluation procedures, and metrics, which limits the contribution of this work. In fact, the benchmark could be further enhanced in the following ways: 
    - **Multi-lingual tasks.** The WINDOWSAGENTARENA only covers the tasks and applications in English, but web tasks in different languages are also crucial for comprehensively evaluating the performance of the web agents.
    - **Diverse evaluation metrics.** Currently, the authors only use the success rate as the evaluation metric in OSWorld. Other metrics like the number of steps and abilities like self-reflection could be added.
    - **Cross-application tasks.** All 154 tasks are single-application tasks, yet real-world tasks often include multiple applications.
- The paper compares Navi's performance primarily with human benchmarks. More agents like MM-Navigator or WebAgent are expected to be included.
- The task curation process is not fully transparent. While the authors mention adapting tasks from OSWorld, it is unclear how these tasks were modified for Windows or how the difficulty levels were assigned.

questions:
- [Q1] Please clarify the contribution of this work compared to OSWorld---as the task definition, evaluation procedures, and metrics all follow OSWorld.
- [Q2] Could you provide more detailed qualitative analyses of why Navi fails in certain tasks? For instance, are the failures due to perception errors, action space limitations, or planning issues? This would offer valuable insights for both researchers using the benchmark and those developing future agents.
- [Q3] Why didn’t the paper include a direct comparison with agents from OSWorld (which focuses on Linux)? Even though these benchmarks focus on different OS environments, comparing Navi’s performance on similar tasks (e.g., browsing, coding) could provide more context for the challenges posed by WINDOWSAGENTARENA.
- [Q4] For human performance, an accuracy of 74.5% looks a bit low to me. Could you also provide more detailed qualitative analyses of the failure reasons for human participants? Would it be possible that the given task is infeasible given the current context?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619790987

thread_type:
official_review

<thread_object>
creator: reviewer_jfiz

<content>
summary:
This paper introduced a OS Agent benchmark on Windows OS, and proposed a new MLLM agent called Navi. The goal of this benchmark is to solve the challenges of  restricted modalities and slow process in agent evaluations. The benchmark defined 150+ create 150+ diverse Windows tasks across different domains. Navi is designed with chain-of-thought prompting and tested on this benchmark with different SoMs and MIND2WEB Dataset.

strengths:
The writing of this paper is well. The problem definition is very detailed and easy for the reader to understand. The benchmark is a good complementary to the existing work and the claimed evaluation system is more efficient to the current benchmark system.

weaknesses:
The experimental results lack the comparison of well-known agents, but just use different settings to test on multiple base models. The design of Navi agent is relatively simple and motivation is not clear. The additional experiments on MIND2WEB is not enough to support the agent performance.

questions:
Is Navi just designed as a baseline to this benchmark or an innovative agent method? 
Is there a plan to test on more benchmarks for evaluate the performance of Navi in the additional results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730808267988

thread_type:
official_review

<thread_object>
creator: reviewer_qmw7

<content>
summary:
This paper presents WindowsAgentArena, a benchmarking suite designed to evaluate multi-modal agents' abilities within a Windows OS environment. Building on previous work like OSWorld, WindowsAgentArena focuses exclusively on the Windows operating system, making it possible for agents to perform diverse tasks representative of real-world computer usage, such as document editing, web browsing, and multimedia manipulation. The study introduces a novel agent, Navi, which demonstrates the framework’s capabilities, achieving notable results in task performance compared to humans. The benchmark suite also emphasizes scalable and fast evaluation through parallelization on Azure, which reduces evaluation time significantly.

strengths:
1. WindowsAgentArena fills a gap in agent benchmarking by focusing on the Windows OS, which is widely used but less explored in agentic evaluations.
2. The ability to parallelize the evaluation on Azure for faster task completion enhances the framework's practicality and adaptability for large-scale studies.
3. By including tasks across multiple categories (e.g., document editing, web browsing, system management), WindowsAgentArena effectively simulates a real-world computer environment.
4. Detailed performance metrics, examples of successful and failed cases, and analyses of task complexity offer clear insights into the capabilities and limitations of agents.

weaknesses:
1. Windows OS is proprietary, which may limit access for researchers who rely on open-source environments or who cannot accommodate the licensing and resource costs associated with a Windows-based setup.
2. Running Windows virtual machines (VMs) with multiple agents and parallelized tasks can be resource-intensive. This may necessitate high-performance computing setups and large storage capacities, particularly for cloud-based evaluations. Configuring and maintaining the Windows environment, including installing software, handling updates, and ensuring compatibility with the benchmark suite, can also be complex and time-consuming.
3. Agents trained and evaluated solely in a Windows environment may develop strategies tailored to Windows-specific UI elements and workflows, potentially limiting their ability to generalize to other OS environments, computing platforms, or even to updated versions of Windows.
4. WindowsAgentArena builds upon OSWorld’s approach but is specific to the Windows environment. This focus on a different OS may not represent a fundamentally novel leap, as it largely adapts pre-existing methodologies rather than introducing groundbreaking new concepts.
5. Many tasks in WindowsAgentArena replicate basic or common actions found in Windows OS, such as file management or web browsing. The tasks may lack complexity or novelty that could reveal new challenges for agentic behavior, making the approach feel less groundbreaking.
6. The Navi framework leverages established multi-modal approaches, such as UIA tree parsing, DOM tree extraction, and OCR for screen parsing. While effective, these methods are widely used in similar environments, potentially limiting the originality of the agent interaction techniques.
7. The reward structure is based primarily on task completion, similar to other agent benchmarks. WindowsAgentArena does not seem to innovate with intermediate or adaptive reward mechanisms that might encourage more nuanced agent learning behaviors.
8. The framework relies on traditional Set-of-Marks (SoM) and bounding boxes for visual grounding, which are widely used in agent evaluation. Introducing innovative ways to handle visual-language misalignment might have added novelty to the approach.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730858297836

thread_type:
official_review

<thread_object>
creator: reviewer_zpbq

<content>
summary:
This paper introduces WindowsAgentArena, a benchmark designed to evaluate LLM agents in performing multimodal tasks within the Windows OS environment, including 154 tasks covering office software, web browsing, coding, etc. The benchmark assesses the ability of LLM agents to interact with a complex operating system through various modalities. The paper also provides a baseline agent Navi and test its performance on the benchmark. Codes and deployment guidance are open-source.

strengths:
（1）The paper proposes a new benchmark specifically targeting LLM agents’ performance in complex, real-world tasks on Windows OS. The benchmark is well-designed, with a diverse set of tasks that test multiple aspects of agent capabilities. 
（2）The paper provides adequate evaluation on the proposed benchmark with a proposed baseline agent and validates its competitive ability.

weaknesses:
（1）Although the benchmark is claimed to be extensive, it is specifically focused on Windows OS. I’m not sure whether this benchmark could be adapted or extended to other operating systems or environments, as this could broaden its applicability.
（2）Apart from the difference in operating environments, it is unclear how the tasks in this benchmark uniquely assess agent capabilities compared to OSWorld. It seems that the tasks have much overlapping with those in OSWorld. More discussion about key differences is needed to distinguish this work from existing works.
（3）I’m concerned about the usage cost of the benchmark. An API cost estimation for running the tasks in the benchmark is needed.

questions:
（1）What’s the uniqueness of the tasks in the benchmark compared to OSWorld?
（2）Line 263 mentioned that the task procedure is shown in Appendix A.3, but the content in A.3 is about reward evaluation.
（3）Is it possible to extend the benchmark to other operating systems？If the OS difference matters to the agent evaluation, maybe this is necessary
</content>
</thread_object>

</thread>
<thread>
creation date:
1730929727726

thread_type:
official_review

<thread_object>
creator: reviewer_sbpm

<content>
summary:
The paper introduces WindowsAgentArena, a comprehensive benchmarking environment tailored for evaluating multi-modal agents in Windows OS settings. This platform builds upon the OSWorld framework, specifically adapted to provide 154 diverse, complex tasks representative of typical Windows applications and workflows, such as document editing, web browsing, coding, and system customization. The benchmark environment is designed for scalability, allowing for parallelized evaluation on Azure, significantly reducing the time required for agent testing. As a demonstration, the authors develop Navi, a multi-modal agent achieving a 19.5% success rate on WindowsAgentArena tasks, highlighting the challenges in reaching human-level performance (74.5%).

strengths:
1. A new benchmark focuses on multi-modal tasks in the Windows environment.

2. Easy to deploy environment.

3. Authors design an extra multi-modal agent to validate benchmark’s effectiveness

weaknesses:
1. This benchmark does not show clear differences toward OSWorld

2. Agent design can be clearer

3. The number of tasks is limited

questions:
1. For all the designed tasks (Office, Web Browsing, etc), which is specific to the Windows environment? To distinguish WindowsAgentArena from OSWorld, authors need to discuss the unique tasks for Windows.

2. As stated within the paper, two thirds of the tasks are re-implemented from OS World and one third tasks are newly designed. It would be great if authors can show the newly designed tasks to help the audience clearly understand the contribution.

3. What is the backend model for the Agent? Have you tested on different backend models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731539409282

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Common Questions from Reviewers

comment:
We thank the reviewers for their time and greatly appreciate their insightful comments. We are particularly encouraged that all reviewers recognize the impact that a Windows-focused agent benchmark can have. We provide detailed responses to each reviewer’s questions below their individual reviews but also summarize our answers to the most common questions below:

======
- **[N1] Why only Windows?**
  - OSWorld is an agent benchmark primarily focused on agent tasks for non-Windows OS (~90% of OSWorld's tasks are Ubuntu/Linux tasks); Linux has ~4% market share among desktop operational systems. We focus this work exclusively on the Windows OS because, with 73% market share, it is the most widespread OS among PC users.
  - Despite its prevalence, Windows is still relatively under-represented in environments/benchmarks for agentic capabilities. Therefore, OS agents need to include Windows, and make use of its distinct UI, system-specific functions and command line, system-specific apps, and idiosyncrasies.
  - Given its uniqueness, proprietary nature, and under-representation as an OS, it is less clear how agents fare with even common actions/tasks in Windows. Differences between OS software ecosystems can significantly impact an agent’s observation/action spaces and performance, as noted by OSWorld itself (their agent performs ~2x better on Linux than Windows).
    - In our experiments, we see that common actions performed by humans in Windows can be challenging for state-of-the-art models that we use for our agents, suggesting that these tasks, the OS/environment, and the workflow aspects they embody still present specific challenges for agents.

======
- **[N2] What are our contributions / How is this different from OSWorld?**
  - In sum, the key differences and contributions are: **(1) OS focus (Linux vs Windows)** and **(2) parallelization of evaluation (time reduction from days to ~20-30 min, ~100x faster)**. 
    - **(1) OS focus**: We mention in our paper that ~2/3 of our tasks are adapted/inspired from OSWorld, and 1/3 are new. There is a non-trivial amount of engineering effort in porting tasks: for each individual task, we rewrite from scratch all configuration functions (which set the initial conditions for the task) and evaluation functions (which verify completeness) so they are compatible not only within a Windows OS but also within our custom parallelizable infrastructure. 
      - Section 3.2 understates this effort, which consumed ~800 person-hours for coding, testing, verification, etc. The remaining 1/3 new tasks consumed ~1600 person-hours. This excludes agent testing, etc.
      - **Only ~10% of OSWorld's tasks are specifically for Windows (43/412), We have >3x as many Windows tasks (>150) and focus exclusively on Windows.**
      - Also, despite not being an open-source OS, we made significant efforts to have Windows accessible via our benchmark as a self-contained docker image. We provide users a way to use a free evaluation copy which can be used for benchmark deployment and then easily/continuously renewed thereafter. 
      - As a result, our benchmark is one of the few (if not the only) that provides users open/free access to the Windows OS and computing environment for agent research and development. Our benchmark also allows users to install their own programs/applications, add and configure new tasks for their own needs, and more.

    - **(2) Quick/flexible evaluation via benchmark parallelization:** Agent evaluations based on task completion is inherently slow because of the sequential nature of tasks---especially as a benchmark grows in size and complexity. 
      - To our knowledge, we are the first to open-source infrastructure which allows researchers to fully parallelize evaluation on our benchmark in under 20-30 min for an arbitrary number of tasks, as opposed to tens of hours or even days or weeks of wait. This enables not only faster evaluation but also faster reiteration in research and development. 
      - With our benchmark, there is also *no* need for any high-performance computing setups or large storage capacities. Our setup relies on widely available and relatively low-spec inexpensive CPU cores and does not rely on any GPU resources (Table 10 in Appendix A.7 for details). Our setup also allows users to customize/configure the RAM allocated to the VM---allocated memory can be reduced down to as *small* as **2 GB** and still be able to run the environment. 
      - Users can install their own programs onto the image or VM and create their own tasks following details in our paper, README, etc.
      - We also note that our incurred costs are primarily from closed-source model API calls---which are similar to costs incurred from testing closed-source models on other agentic benchmarks as well. 

We hope our responses have addressed the common questions that reviewers have. Please let us know if there are any other questions. We thank the reviewers for their time!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732831500167

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Manuscript Revisions

comment:
# Manuscript Revisions

We thank the reviewers for their feedback and have recently uploaded a revised manuscript reflecting reviewers' suggestions, including (but not limited to):

---
## Clarifying Problem-Framing & Contributions
* **Revised introduction section and contribution portion (pages 1-2)** that better frames the problem we're tackling and its context/importance. We also spell out our specific contributions more clearly, such as:
  * ***Windows OS focus & significantly larger number of tasks:*** Our exclusive focus on Windows OS, with a new benchmark containing more than 3 times the number of Windows tasks than OSWorld (>150 vs. ~50), allowing us/researchers to conduct more in-depth analysis of agent performance on Windows---especially since Windows is widely-used but less well-represented in agent benchmark environments. 
   * ***Fast + scalable benchmark:***  our benchmark is efficient, scalable, and cheap---and does not require large amounts of storage or GPU resources, etc---but still allows for a full evaluation in less than 0.5 hour with what would normally otherwise take multiple hours or over a day using inexpensive CPU cores, *making agent research/development faster and more accessible.* We are the first to make this sort of benchmark infrastructure readily available.
     * Agentic benchmarks are typically large/complex which can slow down evaluation, reiteration, and research. Our benchmark does not suffer from that bottleneck.
  * ***More/new experiments with agent models:*** extensive experiments and ablations with variants of Navi that include both *popular and less-represented models by similar works* as well as different input processing (Section 4):
    * *Popular models*: large closed-source multi-modal models such as GPT4-V and GPT4-o.
    * *Newer models*: faster models such as GPT4-o mini and slower but more reasoning-capable text-only models like OpenAI's o1 
    * *Smaller models*: open-source models like phi-3/phi-3.5 vision.
---
## Additional Results
* **New discussion and results.** We added a new section **Appendix A.8 (page 25)** that includes discussion/details to clarify differences between Windows Agent Arena and OSWorld, including constraints re: comparability between OSWorld's agent and ours as well as results from additional experiments (**Table 12**) that attempt to make things more comparable. Specifically:
  * ***Additional experiments:*** we include additional results with gpt-4v and gpt4-o experiments on our benchmark that use only screenshot or screenshot + UIA tree (Windows accessibility tree analogous to a11y tree) to try and make comparisons possible with some of OSWorld's agent configuration.
    * Combined with our previous results in Table 4, Section 4.2 using Set-of-Marks with UIA trees, these experiments come as close as possible in terms of comparability with OSWorld’s agent due to comparability constraints (see * ***Potential problems in comparisons with OSWorld agents/experiments*** point below).
  * ***Navi***: more discussion/detail on our intended contributions and motivation behind our agent Navi.
  * ***Potential problems in comparisons with OSWorld agents/experiments:*** we discuss issues that unfortunately make direct comparisons misleading and, in some cases, not possible. Examples include but aren't limited to:
    * ***OS-specific differences that affect agent inputs/performance:*** e.g., a11y trees (Linux) and UIA trees (native to Windows) differ in implementation/usage across Windows and Linux due to platform-specific frameworks and conventions. In fact, this is why we believe it is important to have a benchmark that focuses on Windows' features.
    * ***Changes in closed-source models:*** this has resulted in issues with reproducibility and infeasible comparisons (e.g., the version of gpt-4v, gpt4-vision-0125, used by OSWorld is depreciated and no longer available).
    * ***Differences in programs/domains:*** We create many tasks for Windows for programs/domains that do not exist in OSWorld, resulting in both a different number of tasks, domains, and difficulty distributions. As a result, how success rates are scored/averaged/grouped together by domain will differ significantly, as well as overall success rates, and will not be a one-to-one or apples-to-apples comparison.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a Windows-centric extension to OSWorld, with 150 Windows-specific tasks.  On top of that, it provides architecture to run these tasks in parallel, and also on relatively cheap CPU-only hardware + (what I think is Azure-specific?) cloud calls to e.g. OpenAI models.  To validate their benchmark+parallelization approach, they also provide an agent, Navi, that navigates these tasks, with some insights that pop out (e.g., while Navi does pretty well compared to other AI agents, its performance far underperforms humans, and this is likely due to throwing a VLM or VLM+other tricks at a screen to ask for full understanding of state just not quite being there yet).  This is a tricky paper to judge, and that is reflected in the reviewers neutral to neutral-weak responses — on the one hand, this is a comprehensive and useful tool for Windows-oriented developers.  On the other hand, as brought up by nearly every reviewer, this does feel like a clear extension of OSWorld to the Windows ecosystem (+ some tasks), with a straightforward Microsoft-centric parallelized backend for deployment.  That is valuable, but from a novelty point of view reviewers were/are on the fence.