Title: macOSWorld: A Multilingual Interactive Benchmark for GUI Agents
Venue: NeurIPS 2025
Accepted: yes
Keywords: Benchmark

GUI agent

interactive

macOS

multilingual

safety
Abstract: Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks and facilitating accessibility, but existing interactive benchmarks are mostly English-only, covering web-use or Windows, Linux, and Android environments, but not macOS. macOS is a major OS with distinctive GUI patterns and exclusive applications. To bridge the gaps, we present macOSWorld, the first comprehensive benchmark for evaluating GUI agents on macOS. macOSWorld features 202 multilingual interactive tasks across 30 applications (28 macOS-exclusive), with task instructions and OS interfaces offered in 5 languages (English, Chinese, Arabic, Japanese, and Russian). As GUI agents are shown to be vulnerable to deception attacks, macOSWorld also includes a dedicated safety benchmarking subset. Our evaluation on six GUI agents reveals a dramatic gap: proprietary computer-use agents lead at above 30\% success rate, while open-source lightweight research models lag at below 5\%, highlighting the need for macOS domain adaptation. Multilingual benchmarks also expose common weaknesses, especially in Arabic, with a 28.8\% average degradation compared to English. Results from safety benchmarking also highlight that deception attacks are more general and demand immediate attention. Project page: https://macos-world.github.io.


Peer reviews:
comments: SUMMARY:
This paper presents macOSWorld, the first live evaluation benchmark for macOS GUI agents. It supports five languages and includes a safety evaluation subset.

STRENGTHS AND WEAKNESSES:
Strengths:

The paper's strength lies in introducing macOSWorld, the first live evaluation benchmark for macOS GUI agents. It fills a significant gap by offering a comprehensive set of 202 real-world tasks across 30 applications (28 exclusive to macOS). The benchmark is multilingual, supporting five languages, and includes a crucial safety subset for deception attacks. A thorough evaluation of six diverse GUI agents further strengthens its contribution to the community.

Weaknesses:

Similar benchmark designs and implementations exist for other operating systems. The paper has already cited some of them, such as OSWorld, AndroidWorld, MiniWoB, and WebArena. In other words, the paper lacks innovation except for adapting existing approaches to a new scenario. This could be a common weakness of most, if not all, benchmark papers.

Although I agree that this paper has a similar quality to previous benchmark papers accepted by NeurIPS, I am looking for new techniques that have not been reported in existing work in order to rate it higher. For example, all existing work (as well as this paper) either utilizes rules or an autorater (or both) in assigning environment rewards. An innovation in this direction could be automatic rule generation and implementation, a more accurate autorater, or a completely new way of finding rewards.

QUESTIONS:
No question

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces macOSWorld, a multilingual benchmark for evaluating GUI agents on macOS environments. It contains 202 multilingual interactive tasks across 30 applications on MacOS, with task instructions provided in 5 languages (English, Chinese, Arabic, Japanese, Russian). Evaluation of six GUI agents reveals that most of the existing agents/models perform less than 40% (except Claude CUA). The benchmark also exposes multilingual performance disparities and shows the performance degration of GUI agents in non-english scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**. \n- Fill the gap of GUI agent evaluation in macOS's environments, which is beneficial to GUI agent community. \n- Multi-linguistic support is good.\n- Evaluation is extensive. \n\n**Weaknesses**  \n- Evaluation metric only considers binary success/failure metrics, which could neglect intermediate progress of agents. \n- More agent frameworks, e.g, [1,2] in addition to native agent models can be considered. \n- Limit the steps in 15 screenshot/30 dialog turns could limit agent performance as complex tasks require explorations that could have 50 or 100 steps. \n\n[1] Agashe et al. Agent S: An Open Agentic Framework that Uses Computers Like a Human. ICLR 2025\n\n[2] Yu et al. ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning. ICLR 2025\n\nQUESTIONS:\n- How is the action space defined in detail? Actions like click, drag are categorized into the same type, does it mean that they will be mapped into a single function with different parameters? \n- Can this work be extended to support accessibility-tree included methods? If it can support, how could the accessibilty tree be obtained in the MacOS environment? \n- Does this work consider some infeasible tasks?\n- What is the rationale behind choice of 15 screenshot/30 dialog limit? The results in previous work [1] show that CUAs often require more steps, e.g., 50 and even 100 during the exploration. And what percentage of failures are due to step exhaustion instead of actual capability limitations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper proposes a new benchmark for GUI Agents. It is targeted for MacOS applications, comes with tasks and environments in 5 languages, and includes a safety eval subset following a prior work.

The benchmark includes ~200 tasks. Each task comes with an instruction, a Preparation Script to set up the initial state of the OS, and an evaluation script to programmatically test correctness. The tasks were originally prepared in English. To convert instructions to other languages, they use LLM. The OS applications natively come with an option to change the language, which is what they directly leverage for the environment.

STRENGTHS AND WEAKNESSES:
## Strengths

- Both the focus on MacOS and multilingualism are novel and needed 
- The benchmark has significantly more apps than current ones.
- The environment setup, programmatic task initial state setup, and evaluation are provided for reproducible evaluation.
- Experiments using the best open and proprietary GUI agents show that the benchmark is challenging.

## Weaknesses

Overall, this is a good environment and benchmark, inheriting design decisions from some of the previous benchmarks, and extending it to macOS and multilingual setups.

The paper, though, describes very little about the process of the task generation itself.

- How was the task quality or "naturalness" ensured?
- How was task diversity ensured?
- How were the annotators selected? What were they instructed to do? What there any quality-based filtering?

Also, the paper does not describe how the preparation and evaluations were conducted. Were they written by the annotators as well? It's also unclear what the human performance is on this benchmark.

QUESTIONS:
---

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes macOSWorld, the first comprehensive benchmark for evaluating GUI agents in macOS environments. The benchmark provides solutions to three critical limitations of existing interactive, OS-level benchmarks: lack of macOS applications, no multilingual support, and narrowly defined safety. There are a total of 202 interactive tasks split across 30 applications (28 macOS exclusive) in the benchmark. The structured tasks and associated OS interfaces are provided for copied & pasted in 5 languages (English, Chinese, Arabic, Japanese, and Russian). The benchmark utilizes AWS EC2 Mac instances cloud resources to host virtualized macOS environments. The benchmark allows reproducible evaluations that are saved by snapshot recovery, as well as programmatic agent assessments. In addition, the benchmark included a specific safety subset that includes realistic fake deceptive pop-up windows to assess agent demonstration of vulnerability to context deception attacks. The authors employed the benchmark to evaluate six representative examples of GUI agents, demonstrating remarkable performance gaps: proprietary computer-use agents performed >30% of the tasks successfully while open-source models completed <2% of the tasks successfully. The findings also revealed significant performance discrepancies based on language and concerning safety vulnerabilities.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nComprehensive Infrastructure: The benchmarking reference point has a technologically solid test bed featuring virtualized macOS environments, reproducible snapshots, and programmatic test scripts that support equitable comparison among agents.\n\nSignificant Coverage Gap: Fills a critical gap in current benchmarks for pioneering inclusion of macOS, one of the largest operating systems with unique GUI trends and 28 proprietary applications.\n\nMultilingual Innovation: Innovative integration of multilingual support at environment and task instruction levels, providing informative information on agent capabilities in various languages and writing systems.\n\nSafety Integration: Comprises realistic (not fabricated) false pop-up tests in interactive environments, generating useful information about agent weaknesses useful for immediate safe deployment.\n\n**Weaknesses**\n\nLimited Evaluation Granularity: Binary success/failure evaluation may miss out on fine-grained variation in performance and doesn't track partial completion of tasks or measures beyond minimal step counts.\n\nAccessibility Concerns: Mac hardware overdependence on AWS can limit research replicability and availability to researchers without sufficient cloud computing capabilities.\n\nLimited Safety Scope: English-only safety subset is moderately small (29 tasks) yet reveals interesting vulnerabilities (≈70% deception rate for CUAs) that ought to be explored further.\n\nQUESTIONS:\n1. Benchmark Accessibility: With the dependence on Mac computers hosted by AWS, are there plans to make the benchmark more accessible to researchers with fewer cloud-computing resources? Can we have cost estimates or alternatives for testing?\n\n2. Human Baseline Validation: Besides step counts, how do you ensure tasks are representative of typical human workflows and that the measurement metrics correctly capture task completion quality?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-10"}