ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data
Authors: Zhaoyang Liu
JingJing Xie
Zichen Ding
Zehao Li
Bowen Yang
Zhenyu Wu
Xuehui Wang
Qiushi Sun
Shi Liu
Weiyun Wang
Shenglong Ye
Qingyun Li
Zeyue Tian
Gen Luo
Xiangyu Yue
Biqing Qi
Kai Chen
Bowen Zhou
Yu Qiao
Qifeng Chen
Wenhai Wang
Venue: ICLR 2026 Oral (2026)
Keywords: GUI Agent
GUI Data Pipeline
Computer Use
Open Source
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Vision-Language Models (VLMs) have enabled computer use agents (CUAs) that operate GUIs autonomously, showing great potential, yet progress is limited by the lack of large-scale, open-source computer use data and foundation models. In this work, we introduce ScaleCUA, a step toward scaling open-source CUAs. It offers a large-scale dataset spanning 6 operating systems and 3 task domains, built via a closed-loop pipeline uniting automated agents with human experts. Trained on this scaled-up data, ScaleCUA can operate seamlessly across platforms.  Specifically, it delivers strong gains over baselines (+26.6 on WebArena-Lite-v2, +10.7 on ScreenSpot-Pro) and sets new state-of-the-art results (94.4% on MMBench-GUI L1-Hard, 60.6% on OSWorld-G, 47.4% on WebArena-Lite-v2). These findings underscore the power of data-driven scaling for general-purpose computer use agents. We will release data, models, and code to advance future research.

Published as a conference paper at ICLR 2026
SCALECUA: SCALING OPEN-SOURCE COMPUTER USE AGENTS WITH CROSS-PLATFORM DATA
Zhaoyang Liu
Jingjing X
Zichen Ding 9 Zehao Li Bowen Yang*
Zhenyu Wu
Xuehui Wang
Qiushi Sun
Shi Liu Weiyun Wang Shenglong Ye
Qingyun Li
Zeyue Tian Gen Luo
Xiangyu Yue
Biging Qi Kai Chen
Bowen Zhou o
Yu Qiao Qifeng Chen Wenhai Wang
HKUST
Shanghai AI Laboratory
ABSTRACT
Vision-Language Models (VLMs) have enabled computer use agents (CUAs) that operate GUIs autonomously with great potential. However, developing robust CUAs requires extensive in-domain knowledge about software interfaces and operations. Unlike image-text pairs that are widely available on the Internet, computer-use data, particularly operation trajectories, are rare, costly collect. Consequently, advancement in this field remains constrained by both data scale and the limited transferability of existing VLMs. In this work, we introduce ScaleCUA, a step toward scaling open-source CUAs. It offers a large-scale dataset spanning six operating systems and task domains, via closed-loop pipeline uniting automated agents with human experts. Trained on this scaled-up data, ScaleCUA can operate seamlessly across platforms. Specifically, it delivers substantial gains over basclines (+26.6 on WebArena-Lite-.. art results (94.4% on MMBench- --rdd 60.6% on OSWorld-G, 47.4% on WebArena-Lite-v2) These findings underscore the power of data-driven scaling for
+10.7 on ScreenSpot-Pry and sets new tfat--ffthhe-
cross cross-platform CUAs. We release data, models, and code advance future research: https: thub. com http:/hh...cccooce/ec//aaaaeeeee htt.://laaaaaalllauaaaca
INTRODUCTION
Humans are able to interact with digital environments through graphical user interfaces (GUIs) to acquire information and accomplish tasks efficiently. The recent advances in Vision-Language Models (VLMs), which exhibit powerful capabilities in visual perception and task planning, have made increasingly feasible to automate such interactions. Consequently, recent research has increasingly focused on computer use agents (CUAs), also referred to as GUI agents, aiming to autonomously operate desktop, mobile, and web platforms by relying exclusively visual observations.
Some works (Qin et al. 2025 Anthropic 2025 Hong et al., 2025 OpenAl 2025 Guo et al. 2025
Hong al. 2025) demonstrate strong performance computer use, while they are typically built on closed-source models or inaccessible proprietary datasets. More fundamentally, effective computer requires rich in-domain knowledge of software and operational procedures, which remains substantial gap for current foundation models. Unlike image-text pairs that are abundantly available on the Internet, computer-use data, particularly fine-grained action trajectories, are scarce, costly to collect, and not naturally archived online. Furthermore, as software, web pages and operating systems evolve rapidly, existing trajectories face the risk of obsolescence, further limiting their utility. These challenges result in significant bottleneck for scaling computer use agents in both data scale and model generalizability. To tackle these limitations, we make significant efforts on two aspects: (a) constructing a large-scale, cross plasforaaoorm and GUI-centric training corpus, and (b) developing a family of scalable, versatile foundation models for general-purpose computer use.
We first present Cross-Platform Interactive Data Pipeline composed of two synergistic loops. The Agent- Environment Interaction Loop enables automated agents to interact with diverse GUI environments, while the Agent-Human Hybrid Data Acquisition Loop integrates expert-collected
"Co-first authors." Corresponding authors (eqf@ust.hk & wangwennha3662@gmai....

---

Published as a conference paper at ICLR 2026
Aguxls-728
GU--wt-328
UHTARS-1
Cloude-33 3.7
ScaleCUA-70
Geen2 5-71-228
U-TAR5-728--000
Seedd 5-V1
ScaleCLU--11
Scw
90.3 24.2
66.0
42.3
203
76.1
SereenSguttPro
AndonssAAeenAArvm
09 (-5.2)
(+.744
(-13.3)
(+.007
(2.4
(+.2(+
25.7
(-23.)
43-
00.3
C8
12.4
42.6
175-
7.5
15.0
7.8
Muttemh-Gi TTlbed W11wwb-.112
Scwwetppe-Prn
Orilard
Understanding
Grounding
(Desktop
Figure I: Performance comparison. The top row showcases performance overview on GUI-centric benchmarks. The bottom row demonstrates the consistent improvements from our collected data.
trajectories ensure coverage and quality. The pipeline spans six major platforms, including Windows, macOS, Linux, Android, iOS, and Web, which facilitates the collection of rich screen-state observations, metadata (e.g., Ally Trees, XLM, DOM structures, etc.), and raw trajectories. this pipeline, we design a unified action space, allowing for more consistent and efficient interaction with diverse real-world environments. Leveraging this infrastructure, we curate and annotate comprehensive training dataset with advanced VLMs such as Claude-3.7 for an open computer use dataset, covering three major task families: (a) GUI Understanding with 471K examples covering regional captioning, OCR, and layout comprehension, etc.; (b) GUI Grounding with 17.1M training samples supporting more accurate UI element localization; and (c) Task Completion with over 15K weak-semantic trajectories and 4K high-level goal-diie direedeee trajectories.
Building upon this corpus, we train a series of base agent models termed as ScaleCUA with Qwen2.5- VL (Bai et al. 2025). It supports three inference paradigms to offer enhanced flexibility and compatibility with agent frameworks: (a) a Grounding Mode, which focuses on locating UI elements based textual descriptions, allowing for integration with more powerful planners, (b) Direct Action Mode, which enables efficient task completion by directly generating executable actions without additional intermediate reasoning and (c) Reasoned Action Mode, which enhances task planning with Chain-of-Thought process before generating the following action. We conduct extensive quantitative studies investigate how different data sources, diverse training tasks, agent designs, etc., influence agent performance. Our findings highlight the benefits of data augmentation, weak semantic trajectories, and general reasoning data for enhancing planning capabilities. As previous studies (Xu) al. 2024 Qin et al., 2025 Anthropic 2025) also probe into the important research questions with limited oourcsourced training data or under closed conditions with proprietary data, our investigations aim to provide foundational and unified insights for advancing vision-bar based computer automation.
Our contributions are summarized as follows:
1) We curate cross ploss-rl computer use dataset, collected via interactive data pipeline that integrates automated agents with human experts. It covers six major platforms (Windows, macOS, Linux, Android, iOS, and Web) and three GUI-centric task domains (i.e., understanding, grounding, and task completion). which provide a robust foundation for studying and training universal CUAs.
2) We develop ScaleCUA, family of robust base agent models that unify perception, reasoning, and action into a single model. It supports flexible inference paradigms, including grounding, direct action, and reasoned action, along with unified action space for seamless cross- platffrr interaction.
2

---

Published as a conference paper at ICLR 2026
3) We conduct comprehensive evaluation spanning understanding, grounding, and end-to-end task completion across several platforms. The results not only demonstrate that our agents can achieve competitive performance but also provide fundamental insights for developing more powerful CUAs.
RELATED WORK
Vision-Language Models (VLM). Recent years have witnessed rapid progress in VLMs spanning
proprietary APIs (Team al. 2023
OpenAl 2023 Hurst et al. 2024) and open-source source models (Wang et al. [2024] Bai et a 2025] Chen et al. 2024b Zhu et al.
2024 Anthropic 2024a
XAI 2025
2025 Xiaomi 2025 Team et al.) 2025a MetaAl 2025), greatly expanding task coverage. Some
VLMs (Team et al. 2025c Guo et al., 2025 Bai et al. 2025; Xiaomi 2025; Wang et al. 2025a)
integrate GUI knowled wedge during pre-training or SFT, thereby gaining explicit computer-use abilities.
Yet, despite strong generalization and planning capabilities, they still rely on proprietary GUI corpora.
Computer Use Agents (CUAs) / GUI Agents. Advances in general-purpose VLMs (e.g.. GPT-4o)
have enabled modular CUAs that decompose decision-making into planner- grounder roles (Cheng
al. [024) Hong et al.] [024 Lu et al. 2024b Yu al.. 2025) Wu al.) [025] Gou et al. 2024
Zhang et al. 2025b; Wu et al.. 2024b Zhou et al. 2025). A VLM-based planner predicts high-level
operations, while specialized grounder localizes targets. Enhancements such as incorporating action
histories Yang et al. 2024) improve contextual grounding. and multi-agent agentic workflows (Wu
ct al. 2023b Li et al. 2023 Hong et al. 2023 Wu et al. 2024at Liu et al. 2025a Zhao et al.
2025
Agashe et al. 2025: Chen al. 2025b) coordinate planning, reflection, and memory. Despite
strong performance, such workflows incur high computational latency and token cost, remaining
bounded by underlying VLM capacity. In contrast, native agents (Xu et al. 2024 Wu et al.) 2024b]
Sun et al. [2244b] Qin et al. 20225] Luo et al., 2025) Liu et al. [025b] Sun et al. 2025) integrate
planning and grounding end-to-end, directly predicting low-level executable actions from raw visual
inputs. Systems such as AGUVIS (Xu et al. (024) and UI-TARS (Qin et al.) (025) trained on
extensive trajectories show strong reasoning and adaptability. Native agents thus achieve tighter perception-action alignment while also benefiting modular setups. Our work extends this direction by training cross-platte plooss- base models and open-sourcing all data.
GUI Datasets. Open-source datasets have accelerated CUA's development by capturing diverse interactions and instruction-follooinn behaviors. For mobile, RICO (Deka et al.) (017)) contains +0k+ Android screens, A/TW (Rawles al.) 2023) offers 715k demonstrations with 30k commands, and
AitZ (Zhang et al. (204)) provides 18,643 screen-action pairs with action-thought annotations. In
the web domain, Mini WoB (Shi al. 2017) simulates diverse tasks, WebShop ( (aao et al. 2022
collects language-
e-commerce trajectories, and Mind2Wel (Deng et al. 2023) scales to 137
websites and 2,350 open-ended tasks. For desktop, Xie et al. (2024) synthesizes 4.M examples boost
grounding, and He et al. 2025) adds 312 human-annotaten data generation includes OS-Genesis (Sun et al. (2024b) for mobile/web exploration and AGUVIS (Xu)
trajectory-booseee samples. Scalable
et al. 2024) for multimodal grounding-reasoning corpora. Tutorial-style datasets mitigate scarcity:
META-GU7 et al.) 2022) introduces dialogue-based annotations: Tong UlngUl(Znn (Zhangg et al.
2025a)
offers ~ 143k trajectories linking instructions to screenshots; and GUI- World (Chen et al. 2025a)
records 12k GUI videos for temporal understanding. Nevertheless, coverage and diversity remain limited, especially for desktop, posing challenges for UI element grounding and multi-step planning.
INTERACTIVE DATA PIPELINE
Collecting computer use trajectories is exceptionally costly and inefficient, primarily due to the dynamic nature of environments and their frequent dependency on task- resources. In this section, we elaborate on the pipeline of data collection and annotation.
3.1 DATA ACQUISITION
Existing computer-use use datasets generally rely on either manual trajectory collection or automated search-basee based exploration. While manual collection (Zhang et al. 2024 Rawles et al. 2023 Deng jet al. 2023; Lu et al., (024a) yields high-quality trajectories, it is costly and difficult to scale. Automated exploration (Sun et al. 2024b) is more scalable but typically noisy. Neither approach
3

---

Published as a conference paper at ICLR 2026
Multi-Platform Environments
Androidl
iPudO6
(Ubunta)
macOS
Windows
Acrion
Obervasion
Gallasin
Thining
Raw Trajectories
Training Corpora
Motadam (Ally Thee DOM XME
Seten
GET Understanling
Angment
Annouu
GUI Grounding
Agent
Task Completion
Human Export
Figure 2: Cross-platform interactive data pipeline. Our pipeline consists of two synergistic loops: (1) the Agent-Environmen Interaction Loop, where agents interact with multi-platform GUI environments via observation and actions; and (2) the Agent-Human Hybrid Data Acquisition trajectories
Loop, where both autonomous agents and human experts contribute to collecting with sereenshots and structural metadata. The resulting trajectories are then annotated and processed into several GUT-centric tasks such understanding, grounding, and task completion.
alone achieves the required balance of quality and diversity for training versatile GUI agents. To this end, we propose a Cross Pross-Platffrm Interactive Data Pipeline integrating automated agents with human experts. operates in synergistic loops shown in Fig. 2 The Enver-Environmenn Interaction Loop enables agents or humans to interact with multi-platform env environments, while the Agent- Agent-Hlmman Hybrid Data Acquisition Loop merges trajectories from autonomous agents and experts.
Specifically, in the agent- -eviromevit interaction loop. we standardize observation acquisition and action exccution across Windows, Ubuntu, macOS, Web browsers, Android, and iOS. This unified abstraction supports closed-loog data collection and diverse agent architectures. Platform-specific metadata is extracted from Ally Trees (Desktop). DOM (Web), and XML layout files (Android). When metadata is incomplete or restricted, as in iOS/iPadOS, OmniParser Yu et al. 2025) estimates UI bounding boxes. In the agent-human hybrid data acquisition loop, human experts and automated agents both share the same interfaces to collect diverse trajectories. For automated agents, evaluate two exploration strategies: VLM-driven agents (c.g.. GPT-4o, Claude-3.7, etc.) and rule- driven random-wall agents. The former relied on proprictary VLMs, which often led to significant bias and hallucinations, especially for computer use, and thus was not used as the primary strategy for data collection. The latter performs depth-first exploration, randomly selecting actions from the available action space at each step. Heuristic pruning removes redundant or uninformative branches, broadening GUI coverage. Although these trajectories often lack clear high-level goals, their sub- sequences still yield valuable supervision for the agent. As both system-derived derived metadata and vision-based bounding boxes can be noisy, we complement with expert-curated trajectories. Human experts first create task list and then collect trajectory data in the environment. In addition, human experts are required to randomly sample and review 20% of the agent-collected trajectories after both collection and annotation to ensure quality. This is what we refer to as hybrid data acquisition. This unified pipeline decouples front-end interfaces from back-end environments, allowing collectors to efficiently switch between platforms and complete domain speciifc tasks. These screenshots and metadata are then annotated into GUI-centric tasks such as understanding, grounding, and task completion, forming robust foundation for training generalizable agents.
3.2 DATA ANNOTATION AND STATISTICS
This dual-loop framework collects extensive screenshots, structural metadata, and raw trajectories across Windows, macOS, Linux, Android, iOS, and Web platforms. Advanced VL.Ms (e.g., GPT-4o
4

---

Published as a conference paper at ICLR 2026
Table 1: Datasets comparisons on computer-use datasets in terms of platform coverage, data types (Understanding, Grounding and Trajectories), and collection methods.
Platform
Understanding
Grounding
Trajectories
Collection Method
Data
Desklop
Mobile
Web
#Samples
#Samples
Avg. Stops
SeeClick X02
X
763K
Auto
GUTEav
10.7M
Widget CaRcaas
RicoSC A 2020h
x
X
178K
Asto
RICO
20
x
V
K
72K
Hybrid
Omnin
GULAct 20244a
X
/
67K
5.7K
6.7
Auto
AitZ. 20:0
X
V
X
2.5K
6.0
Human
AndroniCantr
2024g
GUI Odvssev (20245)
X
X
7.7K
15.3
Human
AMEX 2024
X
V
X
3,0
11.9
Human
AitW 20225
OS-Allas 2024b
X
13.6M
Auto
OS Geneen 2024b
X
/
/
2.5K
6,4
Asta
JEDI 2025
877K
Agcali33 N3
20256
X
22K
Human
Ours
V
471K 17.1M
19.0K
9.0
Hybrid
and Claude-3.7) are then used to annotate the corpus into three major task families: GUI Understand- ing, GUI Grounding, and Task Completion. At the element and screenshot levels, understanding tasks cover visual description, OCR, layout reasoning. interface captioning, and state transition analysis, while grounding tasks provide point, bounding-box, and action-level supervision to align natural-language instructions with UI regions. Task Completion is composed of a) weak-semantic trajectories derived from rule-driven exploration that supply low-cost navigation patterns, and b) expert-curated demonstrations with realistic, goal-directed signals for reasoning and plamning. Aug- mentation techniques such element cropping, synthetic resolution scaling, and reasoning-pree prommpp enrichment further diversify the training data. The final corpus spans 471K GUI-understanding exam- ples, over 1M grounding annotations, and 19K trajectories averaging steps each. As summarized in Table 1 we believe this dataset enables balanced evaluation of understanding, grounding, and task completion across all platforms. More statistics are shown the Appendix.
Discussions. By leveraging dual-loop pipeline, we ensure coverage of low-level element recognition, mid-level grounding. and high-level task planning. Compared with current works (Sun et al. 2024b) Wu et al. 2024b) Zhang et al. [224]] Rawles et al., 2023), we explore more diverse data colle lellintion strategies (human experts and automated agents) and cover broader range of platforms (desktop, mobile, and web). Specifically. for the random-war walk agent, we designed more efficient algorithm through extensive experimentation and iterative improvements, significantly enhancing both data collection efficiency and GUI coverage. With this pipeline, we have collected over 2M raw screenshots across multiple platforms. We acknowledge that this pipeline is conceptually straightforward, but executing it across heterogeneous operating systems and software ecosystems entails substantial non- trivial engineering. Our contributions in the data pipeline are threefold: 1) We propose robust and scalable data acquisition pipeline that balances automation and expert supervision, along with set of effective heuristics improving data diversity and quality. 2) We summarize a comprehensive guideline covering platform-specific issues and their resolutions in the Appendix, which significantly improves the purity and efficiency of data collection. 3) We commit to releasing all data, ensuring transparency and reusability for future research. Generally, we emphasize that our work delivers a practically validated, cross-platform solution addressing real-world bottlenecks in scaling computer-up use agents. Despite involving many engineering-orieee efforts, we still aim to share these experiences and provide valuable guidance for future developments in this field.
4
THE DESIGN OF COMPUTER USE AGENTS
4.1
TASK DEFINITION
VLMS allow agents achieve pixel-level perception and interaction on graphical user interfaces. We formulate the interaction between the agent and environment at one time step as follows:
(lask, ot, het); oc+1 E(ac),
(1)
5

---

Published as a conference paper at ICLR 2026
Grounding Mode
Task: Click on the notifications icon
Action: <action>> click(1868, 166) <lactioe>> Bhox: <bors [1826, 1873, 190] -obaar-
GATONNES
Point: spolat [1848, 166] <jpoter>
TRAILER
Task: Find the button named "Shorts"
Action: <action? click (655, 411) <iacrion Bbox; <bor- (20, 368, 110, <abar>
Point: Spoint [655, 4115,411] <ipuint>
Reasoned Action Mode
Task: want to find some Python video tutoriak, you help me?
<chink> Based the observation, I that find Python video tatarials YouTube, I would use the bar at the top of the page. The scurch bar currently empty with the placeholder text "Scarch" visible, and it's the primary way find specific cunteni Tobe Clicking on ihe soarch bur and lyping "Pythun Tuterial" would allow you relevant video ... <Ahinkk
Direct Action Mode
and write "Python Tutorial" find programming tutorials, <iperation's <ardine>> elick(945,, Iokyiownta""ptan Python Tutorial") </action>
Koperation Click
Three Inference Paradigms in ScaleCUA: (1) Grounding Mode, which focuses on localizing target UI elements; (2) Direct Action Mode, where the agent solely generates executable actions based on current observations and instructions; and (3) Reasoned Action Mode, where the agent first generates chain-of-thour rationale before producing structured actions. These modes enable varying levels of functionality for computer use agents to complete tasks.
Figure 3:
where π denotes the agent model parameterized by 0, and & represents the environment, such as virtual machines Docker containers. The lask denotes the task instruction. The observation encompasses elements such as raw screen pixels, accessibility trees, or DOM data. The history her {(ao, oo)..... (0.-1.0r11 }}} provides context for agent's decision-making process. Similar to some works (Sun et al. 2024b Xu et al. 2024), we choose to generate natural language descriptions for (a;,0) as history, as it can save a large amount of inference cost budget. Each action specifies an operation with corresponding arguments, detailed in Table 14, which is then executed in the environment. this work, we adopt screenshots as the observation space. This paradigm aligns with human behavior and effectively avoids interference from noisy accessibility Tree and DOM data.
4.2
AGENT MODELS
We build our ScaleCUA family upon Qwen2.5- VL for its strong multimodal understanding and scalability across diverse GUI platforms. As shown in Fig. Bl it supports three inference paradigms. In Grounding Mode, the model localizes UT elements via points, boxes, or coordinate- bbaed actions from screenshots and instructions, making it suitable as a modular" "grounder" for external planners. In Direct Action Mode, the model directly emits low-level instructions and executable actions, enclosed in <operaticn inn> and <act ion> tags. Given the current screen and interaction history, it enables fast perception-action loops without explicit reasoning. In Reasoned Action Mode, it first generates a rationale inside <th <think> tags before producing the action, improving reliability and interpretability on ambiguous or long- long-horizon tasks with extra latency. This design allows flexible integration with different agentic workflows while maintaining consistent control semantics across platforms.
Action Space. We design a unified action space for data collection and environment interaction. Table 14 summarizes our cross-platform action space spanning desktop, mobile, and web. It combines universal operations (e.g., click, write, etc.) with platform-spect actions (e.g., long-press and open_app for mobile), ensuring consistent behavior modeling and simplifying downstream policy learning. More details appear in Sec. A.4
Training Recipes. We train three model scales under hardware-awart configurations: ScaleCUA-3B (mini-batcl batch 4, grad-accum 1 on 128 A100 GPUs), ScaleCUA-7B (mini-batch 2, grad-accum on 128 A100 GPUs), and ScaleCUA-32B (mini- batch 2, grad-accum 2 on 128 H200 GPUs). All use a learning rate of 1 10-5 and maximum token length of 40,960. To balance general multimodal

---

Published as a conference paper at ICLR 2026
knowledge with GUI-specific skills, we vary the ratio of general purepose data to GUI data: 25% for 3B, 50% for 7B, and 75% for 32B. Empirically, this scaling yields substantial gains on GUI understanding, grounding, and task completion benchmarks, confirming that larger models can absorb higher proportions general data without diluting GUT competence.
5 EXPERIMENTS
Evaluation Setup. We comprchensively evaluate our ScaleCUA across three dimensions: GUI understanding, GUI grounding, and end-to-end task completion. All evaluations are performed under pure visual observation align with real-world usage. For GUI understanding. we MMBench-GG L1 (Wang et al. (202cc). which tests fine-gree grained perception and reasoning about interface content. For GUI grounding, we conduct structured evaluations on ScreenSpot-v2 (Wul et al. 2024b). ScreenSpot-Pry et et al. 2025), and OSWorld-G (Xie et al. (2025). covering cross- platform localization and domain-specifie scenarios. By default, ScreenSpot- resolution and other benchmarks at 1080p. For end-to-end task completion, we test our models on AndroidControl, WoWdd (Xie al., 2024), Window AgentArrgnnn (WAA) (Bonatti al. 2024). macOSArems (MA) (Wang et al. 2025c). AndroidWorld (AW) (Rawles et al., 2024). and WebArena- Lite-v2 (WAL-v2). These benchmarks span desktop, mobile, and web settings, with 50-step budget applied when not specified, enabling realistic assessment of platform-specific performance. We further validate general vision-language capabilities on several well-known benchmarks (Yue et al.)
evaluated 2K
2024 Lu et al.) 2023 Liu et al. 2024b) xAI (024).. In addition, deploy Qwen2.5VL models with vLLM (Kwon et al. 2023) to ensure scalable and consistent online evaluation.
5.1
COMPREHENSIVE AGENT EVALUATION
GUI Understanding. MMBench-GUI LI (GUI Content Understanding) assesses fine- grained perception and reasoning across six platforms following MMBBncc-GUI protocols. In Table 2 our ScaleCUA consistently de- livers competitive or superior results. Even the lightweight ScaleCUA-3B attains 89.9%, surpassing Qwen2.5-VL-72B by +25.3 points. ScaleCUA-7B further improves to 92.3%, while ScaleCUA-32B reaches 94.4%. These results highlight the efficacy of scaling with cross- platform GUI-specific data, confirming that di- verse training corpora substantially enhance vi- sual comprehension across heterogeneous envi- ronments.
Table 2: Results on MMBench-GUI LI (GUI Con- tent Understanding) (Wang al. 2025c
Model
Easy
Medium
Hard
GPT-4o 2024
60.2
57.2
53.5
Claude-3.7 2025
39.1
38.4
35.7
Qwen2.5 V-LL-2 2025
67.0
67.5
64.6
UL-TARS-72B-DPO (2025
40.2
41,8
35.8
InternVV.3-72B (2025
79.2
77.9
75.7
GUI-OwI-7B 2025
84.5
86.9
90.9
GUI-Ow1-32B [2025]
92.8
91.7
94.2
ScaleCUA-3B
83.6
85.6
89.9
ScaleCUA-7B
88.4
90.1
92.3
ScalcCUA-32B
92.5
92.5
94.4
GUI Grounding. We then evaluate models on GUI grounding, which measures the ability to localize and associate visual elements with textual or functional references across desktop, mobile, and web. As shown in Fig. 4 our ScaleCUA consistently achieves state-of-the-an performance across different benchmarks. On the challenging ScreenSpot-Pree ScalcCUA-32B again dominates, achieving 59. 39.2% overall and delivering strong accuracy across diverse domains such as Creative software, CAD, and office applications. More detailed comparisons are presented in A.2 Overall, these results demonstrate that scaling with GUI-specific data yields substantial benefits for grounding. The consistent improvements across GUI grounding benchmarks confirm the effectiveness of our dual-loop data pipeline in learning robust UI element localization.
Task Completion. We evaluate end-to-ent task completion on Mobile (AndroidWorld), Ubuntu (OSWorld), Windows WWindowsAgent/Arenal macOS (MacOSArena), and Web WebArena-LLte-ve considering both native agents and planner-grounder workflows. The results shown in Table 3 First, our native ScaleCUA-32B achieves the strongest Web performance: 44.2% (15 steps budget) and 47.4% (50 steps), outperforming the best native baseline (UI-TARS- 7-B--72--00--- DPO) by +20.8 and +26.0 points, respectively, and substantially surpassing Qwen2.5-VL-72B. Then, the workflow setting with GPT-4o as planner and ScaleCUA-7B as the grounder yields 48.3% on AndroidWorld and 28.1 1 % on OSWorld (50 steps), outperforming other strong grounders such JEDI-7B. Beyond these
7

---

Published as a conference paper at ICLR 2026
Table 3: Online evaluation across different platforms. AndroidWorld has its own predefined step budget. denotes the unkown step budget and indicates more than 50 steps is used.
Mobile (AnheedWWeek))
Ubuntu
Windows
MucOS
Web MithAromn--ee122
Method
(08Wee))
(MacCS Arcral
Predefined Steps
15 Steps 50 Steps
15 Steps
50 Steps
15 Steps
50 Steps
15 Staps 50 Steps
Native Agenr
Kimi V38
20A56
8:24
10.4
GLM-4.1V
2025
41.7
14.9
GIM--.5V.Thea Tinkkiinn
$7.0
35.8
COMPUTERKL
47.34
Agont_F..22
GPT-4o 2023
21.6
6.8
10.1
3.6
3.5
0.0
1.4
2.0
Clande-377 2025
11:2
7.4
10.3
7.1
644
5.7
7.1
2.0
Been2.5-V2-722 2012
27.6
9.8
10.6
118
9.7
1,4
5.7
15.6
14.4
70255
29.7
11.1
11.6
15.2
18.0
2.9
5.7
11.7
11.7
Aguvis-720 72B
2024
3.8
42
4,1
3.5
0.0
0.0
5.8
UI-TARS-7E-SFT
UI-TARS-728-DDE
179
OpenCUA-12 2023
24.3
28.1
OpenCUA-3225 20236
29.7
34.1
ScaleCUA-33
23.7
9.6
12.4
13.1
15.2
0.0
1.4
31.8
33.1
ScateULAA778
27.2
14.3
15.0
18.0
2012
4,3
4.3
32.7
32.2
ScaleCUA 32B
30.6
16.5
17.7
21.4
24
7.1
44.2
Agenti Wortflow
Planser
Grounder
Aria-III
2076
44.8
15.24
20144
14.6
UGround-
13.1
UI-TARS-1
16.5
19.1
14.5
26.3
14
0.0
28.6
JEDI 2025
22.4
29.1
JEDI-78 201
22.7
25.0
30.2
32.
SesteCUAA-7R
48.3
22.9
28.1
31.7
36.6
5.7
8.6
28.6
35.1
highlights, several trends emerge. (i) Scaling from 3B-+7B-322 produces monotonic gains on different platforms, indicating that cross-platform data and unified action space translate into stronger computer use agents as capacity grows. (ii) The effect of the step budget is consistent: a majority the agents, including ScaleCUA, achieve substantial performance improvements under 50-step limit. (iii) Even employing our proposed data, the planning ability of our model still lags substantially behind GPT-4o in agentic workflows, and models trained with existing and proprietary datasets continue to exhibit a considerable performance gap. We must acknowledge that there remains significant room for improvement and further development.
5.2 DIAGNOSTIC ANALYSIS ON COMPUTER USE AGENTS
To elucidate the main factors that affect agent performance, conduct detailed ablations, which reveal key trade-offs between accuracy, efficiency, and generalization:
Input Resolution: WoWordd uses strictly standardized 1080p frames. When the input resolution is set at or above 1080p, the performance saturates because the inputs still match the maximum training resolution, i.e., 1080p. In ScreenSpot-v2, the majority of screenshots are at or below 1080p, yet this results in negative impacts when the resolution increased further. By contrast, ScreenSpot-PP Pro contains a large proportion of native 4K screenshots. The performance it benefits from higher
ScalaCUAA-308
Gill-wwi328
U6TARS-1.5-7
ScalaCUA--7
Qwen2: SVL--228
ScaleCL14-38
JDD778 UK-TAR--728
94.7
92.7
91.7
913
903
85.2
59.2
18.0
579
541
Aecurry (5
47.9
46.5
43
38.5
38.1
30-
ScreenSpot-v2
ScreanSpot-Pro
Figure Results on GUI grounding datasets.
8

---

Published as a conference paper at ICLR 2026
200
50
23
Bendinants Eagg Duselrn day /Sment 5v.300
32
60
Supurs Rtpu
42
Acttnn 2000
11000
1888p
PM
Golond
WPA
AW
Wiol
33
100
50
75
100
Ratto
(3) Test-tmee resolution GLJ govanding
(b) DiDeett Action v. Reasened Action (c) Performatee curve when scaling our Cara
(c) Geneval dara used training
Figure 5: Evaluations across diverse conditions. (a) Accuracy of GUI grounding under different screenshot resolutions. (b) Success rates of Direct Action vs. Reasoned Action Modes, where reasoning consistently improves performance. (c) Training data scaling. (d) Effect of using general data, showing distinct trends between GUI and multimodal benchmarks.
resolutions up 2K but drops at 4K. Overall, we observe that the impact of resolution on grounding performance depends largely on the benchmark's 's data distribution.
Inference Modes: Fig. (b) compares the two inference modes for computer use agents. Across all benchmarks, reasoned action mode (RAM) yields higher success rates than direct action mode (DAM), with absolute gains ranging from +1.4% to + 8.2%. However, this mode also incurs longer inference time and greater token cost. DAM, in contrast, produces actions directly from the disual-textual context, yielding faster responses but being more prone to cumulative drift in long-horizon tasks. In the Table 3 when ScaleCUA-7B as a grounding model is integrated with GPT-4o under agentic workflow, shows higher success on task completion benchmarks than the reasoned action mode (e.g.. 28.1% % vs. 15.0% on OSWorld, 36.6% vs. 20.7% on WindowsAgentAreen etc.). The agentic workflow allows GPT-4o to handle long-context planning while leveraging grounding mode in ScaleCUA, demonstrating complementarity between ScaleCUA and general VLMMs. Nevertheless, this paradigm cannot generate actions in an end-to-end manner and brings higher costs even than RAM.
Data Scaling: In Fig. (c), success rates generally improve with more training data. Specifically, WebArena-Lite--2 shows nearly linear gains, whereas ScreenSpot-Pa reaches strong accuracy with about half the data. For WindowsAgentAreent the observed gains appear smaller primarily because tasks in the online benchmark are more difficult with relatively low baseline scores, where even small improvements are challenging to achieve. These results intuitively reflect the task's difficulty, and also imply larger data volume required to achieve the desired performance.
Fig. (d) analyzes the effect of employing ceneral-purposs multimodal data in training. We find clear divergence: GUI benchmarks suffer gradual decline in performance as the ratio of general data increases, while general benchmarks improve steadily, peaking around 75%. As the multimodal corpus expands, the model's general capabilities improve, but GUI-specific knowledge may diluted. The results indicate that data- balancea training strategy is crucial for preserving GUI specialization without compromising general reasoning abilities. Since the larger models are able to memorize more knowledge, Since the larger VLMs can memorize more knowledge, it is reasonable to increase the ratio to 50% for the 7B model and further 75% for the 32B model.
General Multimodal Data:
Multi-platform Ablation: Furthermore, the Fig. 6 shows that models trained exclusively single domain slightly outperform the cross-domain model on desktop and web benchmarks, whereas the cross-domain model performs better on the mobile benchmark. One plausible reason lies in the inherent differences in aspect ratio and UI layout across platforms. Mobile interfaces typically feature more vertically constrained layouts and standardized compo- nents with larger, touch-friendly elements, whereas desk- top and web pages provide horizontally richer screens with denser and more variable UI structures. Since web/desktop data can enrich the feature space without fundamentally altering the underlying interaction patterns, the model trained on cross- ploss--aaofomm data can thus generalize more effectively the mobile domain with simpler visual hierarchies. Conversely, models trained
Trined only this domsin
Trained on domains
24.7
Sucress Ralte (%6)
17.7
14.5
13.3
11.1
AndrostiWWorr Wenteenne Lifere2
octierte Wisttenkgentreen (Daskcap)
The effects of training on domain-specific data.
Figure

---

Published as a conference paper at ICLR 2026
desktop or web data exposed to information-dense layouts where UI elements may be small, overlapping, or nested within complex DOM structures. Introducing mobile data during multi-domain training can dilute the model's specialized representations for these fine- ane-grained desktop layouts, leading to small performance drops in desktop and web benchmarks.
Generally, high-resolution inputs and reasoning- bassoning-based inference enhance grounding and task comple- tion but incur extra cost. Data scaling remains crucial but benchmark-sensitive and heterogeneous data mixtures improve general reasoning the expense of GUI capabilities. These insights motivate scalable, cross-platform training pipelines with deliberate data composition to build robust agents.
5.3 ABLATION ON DATA Table 4: Ablation studies on data. The maximum steps used in online benchmarks are set to 50.
(a) The ablation on data augmentation. We only use GUI-related data in training.
(b) The ablation on weak semantic trajectories. The public
datasets used are shown in Table
Model
Training
Aug.
Model
Training Data
Ws
OSWorld
WAL-v2
Qwen2.5VL-3B
ours-only
41.3
37,8
Owen2.5VL-3B
public-only
7.6
8.4
(c) The ablation on coordinate types.
(d)
The ablation on the maximum resolution during training.
Model
Type
ScreenSpot-Pro
Model
Res.
SS-Pro
OSWOrid-G
OSWorld
AW
Owen2.5VL-3B
Norm. 37.9
Raw
42.3
5VL-5
T080P
42.3
45.5
54.3
$2.5
11.2
12.4
23,3
134
In this section, we aim to ablate our data. As shown in Fig. T training with our curated training corpus yields consistent improvements over the baseline trained on public data. In Table we further highlight the effects of augmentation, weak semantic trajectories, coordinate formats, and resolution.
First, the results verify that data augmentation can improve performance by 3.5% on ScreenSpot-Pro. This confirms that augmentation enhances generalization and robustness by exposing the model to a wider range of visual conditions. Second, we investigate weak semantic trajectories derived from rule-based random exploration. Despite lacking explicit high-level goals, these trajectories provide low-cost supervision of interface navigation. Third, we study the impact of coordinate representations in grounding. Models trained with raw coordinates outperform those with normalized coordinates. This indicates that GUT grounding should follow the absolute position used Qwen2.5VL. Finally, we ablate the training resolution. Higher resolutions yield trade-offs across benchmarks: while 2K improves grounding on ScreenSpot-PP (45.5% vs. 42.3%) and preserves OSWorld-G accuracy (52.5% 54.3%). it slightly reduces agent success rates OSWorld and AndroidWorld. This suggests that fine-grained grounding benefits from high-resolution supervision, whereas agentic benchmarks may suffer from overfitting to pixel-level details.
6 CONCLUSION
In this work, we curate a large-scale multi multi-platform dataset with our dual-loop data pipeline that integrates automated agents and human experts into data construction. The training corpus spans understanding, element grounding, and task completion. With this dataset, we develop a new family of CUAs, i.e., ScaleCUA, which support flexible inference paradigms for scalable integration with agent frameworks. Extensive experiments demonstrated the efficacy of our proposed method. Together, these contributions advance the frontier of computer use agents by bridging vision- language modeling with practical GUI interaction. We hope that ScaleCUA and its released resources will serve as solid foundation for future research in building capable, trustworthy, and deployable CUAs.
Limitations. Although our framework study multi-platform agents with a scalable data pipeline, several challenges remain. First, integrating automatic data collection with iterative refinement into a self-improving loop is still insufficiently explored. Second, we have not employed advanced agentic techniques such as reflection or reinforcement learning, which are likely to improve long-horizon control. Third, the current history design is flat and cannot fully capture long- term dependencies. Despite not exploring these aspects in this work, we believe that releasing the full data, models, and training configurations lays solid foundation for future progress in computer-use agents.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The work in this paper was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project Reference Number: A0E/E-601/24-N)
ETHICS STATEMENT
Our work complies with the ICLR Code of Ethics. The proposed dataset and models are constructed without collecting any personally identifiable information or sensitive data. All screenshots, metadata, and trajectories are obtained from synthetic or publicly accessible software environments and do not involve real users' private data. When automated agents interact with platforms, they operate within controlled virtualized settings to avoid unintended data capture. Human experts are limited to interface-level information (e.g., UI element labels, bounding boxes, or action descriptions) without exposure to personal content. The released resources (dataset, models, and code) are intended solely for research purposes to advance open and reproducible study of cross pesssfoo computer use agents. We explicitly discourage any misuse of these models in ways that could compromise privacy, security, or fairness. No confiicts of interest or sponsorship bias exist in this work, and all authors adhere to research integrity practices, including transparent documentation of data sources, collection procedures, and evaluation protocols.
REPRODUCIBILITY STATEMENT
We have taken extensive measures to ensure reproducibility our results. This work elaborate on the data acquisition pipeline (Sec. 3). dataset composition and statistics T Fig. [1], unified action space (Table 14), training recipes for different model scales (Sec. 4.2). and comprehensive eval- uation protocols (Sec. 5p. Additional implementation details and ablation studies provided in the Appendix to guide replication of our experiments. We will release the dataset, model checkpoints, and source code to facilitate verification and reproducibility. Together, these resources allow researchers to reproduce our key findings and build upon our work with minimal additional assumptions.
REFERENCES
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-sseecialss framework for computer agents, 2025. URL https wttpxii..rg//bb//5555.ooo
Anthropic. Claude 3.5. Anthropic AI Assistant, September 2024a.
URL https: www
anthropic. com cttlcc.coocclaaaca claude Accessed: 2025-06-23.
Sonnet Anthropic. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet, 2024b.
URL sttp:///aapieeeenncccccoooooooo org/ :ttp:/ooppusDD::33333355
Sonnet Anthropic. Claude 3.7 sonnet system card, 2025. URL https: http::/wwwwwatttc.. www.anthrppic..
haa//new//oee nntes/cllaude-e --tee-37--soon sonnet
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. Uibert: Learning generic multimodal representations for UI understanding. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 2021.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- vision-language model with versatile abilities. arXiv preprint arXiv:2308. 229. 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijic Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2xXv:25502.1323 2025.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents scale. arXiv preprint arXiv:2409.08220 08264, 2024.
11

---

Published as a conference paper at ICLR 2026
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotaaion expo dataset for mobile gui agents. ArXiv preprint, 2024. URL htttt:://axxx>>
abs wwtpab//2400.1702220
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Huichi Zhou, Qihui Zhang, Zhigang He, Yilin Bai, Chujie Gao, Liuyi Chen, al. Gui-world: video benchmark and dataset for multimodal gui-oriented understanding. In ICLR, 2025a.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406. 1134406.1111,,, 2024a.
Xuetian Chen, Yinghao Chen, Xinfeng Yuan, Zhuo Peng, Lu Chen, Yuekeng Li, Zhoujia Zhang, Yingqian Huang, Leyan Huang, Jiaqing Liang, et al. Os-map: How far can computer-using agents go in breadth and depth? arXiv preprint arXiv:2507. 19132, 2025b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 24185-24198, 2024b.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10933 2024.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Google tic
Deepmind. Introducing gemini 2.0: our new ai model for the agen- era.
https://blos google-gemin ai poclate-december 2024/ 2024.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium user interface software and technology, pp. 845-854, 2017.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-222 14, 2023.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.052433 2024.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seedl. 5-vl technical report. arXiv preprint Sid:::205507062 2025.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:201.113919 2024.
Yanheng He, Jiahe Jin, and Pengfei Liu. Efficient agent training for computer use, 2025. URL https://erxiie wwtp://..g////oo//////oo 220..2555..0 13909
Sirui Hong, Xiawu Zheng. Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint 2308.0230..00352. 3(4):6, 2023.
12

---

Published as a conference paper at ICLR 2026
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14281-14294 2024.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng. Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006 2025.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AIShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pp. 161-178. Springer, 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing. Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerrl: Scaling end-to-end online reinforcement learning for computer use agents. arXiv preprint arXiv:2508, 14040, 2025.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 06651191--2000 2023.
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat- Tat-Seng Chua. Screenspot-pro. Gui grounding for professional high-resolution computer use. arXiv preprini arXiv:2504.07955 2025.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents, 2024a. URL https
// //arxiv..c org. abs 2406 03679
Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Camphell-Ajala, Divya Tyama- gundlu, and Oriana Riva. On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems, 77792130-92155 2024b.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020a.
URL https: hww_ttpss//cclaathoooyyy...
/awlaanthloogy..c.c/... wwwp://....ggg///oo/../o acl main. 729
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Asso- ciation for Computational Linguistics, 2020b. URL https::/ https:://cclammhoooog..... orw.h///olooo...222000000
emn p-maaaa-- ma nn. 443
Kevin Qinghong Lin. Linjie Li. Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One lisioo-language-actin model for gui visual agent, 2024. URL https:::///.......... http://www.ooooo////////oo// 17465
13

---

Published as a conference paper at ICLR 2026
Haowei Liu, Xi Zhang, Haiyang Xu. Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan. Changsheng Xu, Weiming Hu, and Fei Huang. Pe-agent: A hierarchical multi-agent collaboration framework for complex task automation on pe. arXiv preprint 14244 2025a.
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327, 2024a.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang. and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504. 12.5504.42399 /4239, 2025b.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng- Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12)2220102, 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310. 02.02555, 2023.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation mobile devices. arXiv preprint arXiv:2406.00 08451, 2024a.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024b. [tttttts::// //apxii/////xvv org awwp/a///24.. 00203
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing efficient action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.2162. 2025.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style vision-language action model for gui agents. arXiv preprint arXiv:2504. 10458, 2025.
MetaAl.
Ilama4, 2025.
https://at www...eeaaa.mm wwt.mmmttaaaaaa...mm ww..coomloloooo Accessed: 2025-06-23.
URL.
1lama-
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
URL /ttps://api
semanticschi irrraantccccolarr.cc wttp:la...ro/ooooooeoro ID:
OpenAI. Computer-using agent: Introducing universal interface for ai to interact with the digital world, 2025. URL httpsstts:///peele
com h
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406. 12373, 2024.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint urXiv:2501.12326 2025.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708-59722 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei L., Folawiyo Camphell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405. 14573, 2024.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web- based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3135- 31 144. PMLR, 2017. 7. URL htt.::tttpp:poceeeee ngs. ml wwwpass/v0//.. shi 7a htm
14

---

Published as a conference paper at ICLR 2026
Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards multi-modal conversational agents on mobile gui. arXiv preprint arXiv:2205. 2022.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403. 14734, 2024a.
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412. 19723, 2024b.
Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint arXiv:2505. 090222555.9997, 19897, 2025.
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. Gui-g" 2: Gaussian reward modeling for gui grounding. arXiv preprini wrXiv:2507.15844 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv: 19786, 2025a.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding. Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao. Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang. Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong. Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao. Tao Yu, Tianhui Song. Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang. Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Zijia Zhao, and Ziwei Chen. Kimi-VL technical report, 2025b. URL https: /ttss////xxvv../ /rg/ abs 2504. 07491
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint ar arX4:22504.07491 2025c.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409) /2/9/, 2024.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Lingli Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, arXiv preprint arXiv:2508. 18265, 2025a.
Xinyuan Wang, Bowen Wang. Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong
15

---

Published as a conference paper at ICLR 2026
Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang. Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foundations for computer-use agents, 2025b. URLattps:/arxiiy
www.abssss555 09123
Xuchui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, et al. Mmbench-gui: Hierarchical multi-platform evaluation framework for gui agents. arXiv preprint arXiv:2507.19475, 2025c.
Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey Bigham. Webui: A dataset for enhancing visual ui understanding with web semantics. ACM Conference on Human Factors in Computing Systems (CHI), 2023a.
Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang. Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, al. Gui-actor: Coordinate-free visual grounding for gui agents, arXiv preprint arXiv:2506.03/ 03143333.60,,,, 2025.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint 05X1v:2308.008155, 2023b.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with seff-improvement. arXiv preprint 4rX.42240..07456, 2024a.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang. Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint 2rXiv:2410.232111 2024b.
XAI. Grok-1.5 vision preview, 2024. URL
xAI. Grok-3, 2025. URL
LLM-Core-Tean Xiaomi. Mimo-vl technical report, 2025. URL 2506. 03569
Tianbao Xic, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances Neural Information Processing Systems, 37:52040-520999 2024.
Tianbao Xie, Jiaqi Deng. Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang. Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis. 2025. BtLLttps:: arw.v..iiii../..i/ www.wiwwww...i.aaaa.ooooo.o..ooe abs 122
Yiheng Xu, Zekun Wang. Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu,, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint ar 44X1::2412.04454, 2024.
Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01382 2025.
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al. Gtal: Gui test-time scaling agent. arXiv preprint arXiv:2507.05791, 2025.
Yuhao Yang, Yue Wang. Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412. 16256, 2024.
16

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scal- able real-world web interaction with grounded language agents. In Sanmi Koyejo. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022. URL aagaaleeeo11sSeeeeoooo111...111. Abstactt -bstracttooofeeeeeccce html
nips. ccppp.cpppppr.r..ccpccpppppppp.p
/
Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang. Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Foundamental agents for gui automation. arXiv preprint arXiv:2508. 15144, 2025.
Wenwen Yu, Zhibo Yang, Jianqiang Wan, Sibo Song, Jun Tang, Wenqing Cheng, Yuliang Liu, and Xiang Bai. Omniparser v2: Structured-poinet points-of-thougg for unified visual text parsing and its generality to multimodal large language models. arXiv preprint arXiv: 16161, 2025.
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng- Tao Jiang, et al. Enhancing visual grounding for gui agents via oolftvvolutionary reinforcement learning. arXiv preprint arXiv:2.005. 12370, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disciplie multimodal under- standing and reasoning benehmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9561 2024.
Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song- Chun Zhu, and Qing Li. Tongui: Building generalized gui agents by learning from multimodal web tutorials. arXiv preprint arXiv:2504. 12679, 2025a.
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu wwt--ooff-tttttttouuooe for GUI agents. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16. 2024, pp. 12016-12031. Association for Computational Linguisties, 2024. doi: 10. 110.3///112222FIINDINGGG WWW.ALILNNNINGNS-........... EMNLP.702. URL
Tang. Android in the zoo:
findi emalp 7022
Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, and Junxian He. Breaking the data bariier-buildig gui agents through task generalization. arXiv preprint arXiv:2504.10127, 2025b.
Di Zhao, Longhui Ma, Siwei Wang, Miao Wang, and Zhao Lv. Cola: A scalable multi-agent framework for windows task automation. arXiv preprint urXiv:2503.00 09263.. 2025.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprini arXiv: 13854, 2023.
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. Gui-gl: Understanding rl-zero-like training for visual grounding in gui agents. arXiv preprint arXiv:2505.158/0 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
click
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
This section provides supplementary materials that complement the main paper.
Large Language Model Usage: We make clarification on large language model usage.
More Results: We report extended evaluations across multiple benchmarks (MMBench-GUI
OSWorld-G, AndroidControl, ScienceBoard, and general multimodal benchmarks), highlighting the scalability and cross-platform generalization of our models.
2.
3 Public Data Used Training: We list the public datasets incorporated into ScaleCUA training,
specifying the portion of each source utilized.
A.4
Action Space: We describe the unified action space that abstracts caatform-speciffe operations into a concise yet expressive set of commands, enabling consistent control across desktop. mobile, and web environments.
A.5
Error Case Analysis: We provide representative failure cases on desktop, Android, and web platforms analyze limitations such as incomplete procedural understanding, insufficient state tracking, and positional reasoning errors.
A.6 Details Data Curation: We detail the multi-platform GUI data collection process underlying ScaleCUA, including sources, application coverage, and platform diversity, which jointly ensure comprehensive domain knowledge and improved generalization.
A.7
Data Visualization: Consolidates illustrative figures for GUI Understanding, GUI Ground- ing, Weak-Semantic and Human-Curated Trajectories, and trajectory annotation to aid qualitative inspection.
A.8
Lessons from Data Acquisition: We summarizes common pitfalls and platform-specifit notes (Windows, Ubuntu, macOS, Mobile, Web), distilling practical guidance for future collection runs.
The Details of WebArrna-LLte--e We clarifies details the benchmark construction and evaluation protocols (e.g., step budgets, metries) to ensure fair comparisons
A.9
A.10
Prompt Engineering: We release prompt templates for both agent inference and annotation workflows to facilitate reproducibility and adaptation.
A.1
LARGE LANGUAGE MODEL USAGE
In this submission, utilize LLMs (GPT-5, Gemini, etc.) help us polish paper writing and summarize related works.
A.2
MORE RESULTS
To fully demonstrate the potential of ScaleCUA, we provide additional results on serveral benchmarks.
On MMBench-GUT I.2 Wang al. 2025c), which incorporates stratified grounding difficulty across major operating systems, CcaleCUA.co AA A-328 demonstrates performance comparable to state-of-the-aa methods shown in Table g achieves leading scores in the basic difficulty setting across several platforms including Android (96.4), Web (93.9), Linux (81.2) 2), and macOS (88.1) 188,1), while maintaining competitive results in the advanced difficulty setting (e.g., Web 76.3, Android 81.7). Furthermore, ScaleCUA-7B and ScaleCUA-3B achieve average scores of 78.2 and 73.7, respectively. They demonstrate particularly robust performance in the basic difficulty setting, especially on Windows, where both models score 78.6, and on iOS, with respective scores of 96.1 and 93.0.
OSWorld-G (Xie al., 2025 for Ubuntu grounding, ScaleCUA-32B demonstrates impressive results with an overall performance of 60.6 shown in Table which includes strong marks in layout understanding (70.0), element recognition (66.7), and fine-grained manipulation 1...)) All of our models underperform
the Refusal subtask because we deliberately excluded the Refusal-specific training data provided by JEDI (Xie et al. 2025). Incorporating these examples may pose risk of biasing the model toward emitting await/refusal state in complex grounding scenarios. Such bias diminishes the agent's 'ss propensity for active exploration within the environment, thereby degenerating its success rate in task completion.
18

---

Published as conference paper at ICLR 2026
Table Results on MMBBench-GUI LI (GUI Content Understanding)
Wang et al.
2025c
Model
Windows
MacO$
Linux
iOS
Android
Web
Overall
Easy Level
GPT-40
2024
62.4
58.5
56.4
58.5
Claude-3.5
2024a
41.3
50.0
41.6
42.0
39.0
41.8
41.5
Cluude-37
202.5
34.7
49.1
39.4
42.8
37.5
40.8
39.1
owen-Mux-VL.
[2023
69.1
72.5
69.9
70,8
63.1
69.5
68.2
Qwen2.5-V1.77
[2125
67.2
72.1
UT-TARS-7BB-DPOr
2025
41.6
28.5
35.2
31.1
52.3
35.3
40.2
IntemVL.3-22B
(2025
74.7
78.7
79.2
83.6
80.1
81.2
79.2
GUI-Ow1-7B (2025
83.0
84.5
85.6
82.6
83.3
88.1
84.5
GUI-Owl-321B 22025
93.7
89.3
93.3
95.7
90.5
94.1
92.8
ScaleCUA--18
86.4
83.5
74.9
85.4
80.3
87.4
83.6
ScaleCUA-7B
89.5
86,9
89.1
86.2
87.0
90.1
88.4
ScaleCUA-32B
93.4
91.7
94.3
93.1
90.5
92.3
92.5
Medium Level
GPT-4o 2024]
56.3
63.1
59.7
54.1
57.7
$5.0
57.2
Claude-3.3
2024g
39.3
47.6
46.0
44.6
42.0
34.3
41.3
Claudo-3.7
2025
39.3
39.2
42.3
39.5
36.1
36.2
38.4
Qwen-Max-
2023
73.9
66.9
68.0
63.7
64.6
65,4
Qwen2.5-VL-7215
20255
66.3
72.7
72.6
59.3
66.2
68.2
67.5
1--7ARS--22--DPOTT
(025
38.8
41.6
37.1
41.7
54.7
31.6
41.8
IntemV1L3-72B
(2025
71.5
78.6
79.9
78.4
81.4
78.7
77.9
GUI-Owl-7B 20255
91.2
GUI-Ow1-32B
20225
2025
94.1
84.5
95.9
87,8
92.8
88.6
91.7
ScaleCUA-3B
91.8
78.5
88.7
74.8
88.6
79.5
85,6
ScaleCUIA-7B
93.6
91.7
93.4
84.3
89.6
85.8
90.1
ScaleCUA-32B
95.1
89.4
96.3
92.2
92.6
87.2
92.5
GPT-4o 2024
60.7
60.4
52.4
45.3
50.9
50.8
53.5
Clande
3.3
4024a
37.4
42.7
34.1
40.9
37.0
38.1
37.6
Claude-3.7
A07h
33.0
34.5
32.0
39.2
37.0
38.9
35.7
Qwen-Max-
2023
67.6
65.8
60.2
58.8
65.3
63.7
Dxen2.5-VL-72B5-22
2025
70.7
68.9
71.0
57.6
53.9
68.1
64.6
UT-TARS-72B-DPOT
2025
31.5
35.9
24.2
36.3
58.1
19.9
35.8
IntemVLL3-72B
(2025
75.1
77.4
76.2
70.4
75.7
78.1
75.7
GUI-Owl-7B 2025
87.8
96.4
94.3
87.8
88.9
94.1
90.9
GUI-Owl-3215 20225
93.3
95.2
95.9
92.2
95.4
92.7
94.2
ScaleCUA-3B
92.3
89.4
93.8
83.3
88.3
58.6
89.9
ScaleCUA-7B
91.9
91.9
94.9
89.6
92.9
91.4
92.3
ScaleCUA-32B
93.0
96.5
96.4
93.1
94.5
94.0
94.4
AndroidControl Li et al. 2024b) which is an offline planning benchmark developed for the Android, all ScaleCUA variants exhibit consistently strong performance demonstrated in Table TO On the AndroidConttrl-Loww, ScaleCUA-78 attains the highest task completion rate, whereas ScaleCUA- 32B achieves the most reliable grounding, indicating that the compact model favors execution efficiency while the larger capacity maximizes perceptual fidelity. As for AndroidCControl-HHigh ScaleCUA-32B demonstrates the highest success rate while showing the smallest degradation from Low to High. ScaleCUA-3B and ScaleCUA-7B achieve favorable trade-off, sustaining solid performance across both low and high settings. The relatively small variance in type prediction across sizes suggests that residual failures arise more from long-horizon interaction and error accumulation than from intent misclassification
localization.
On Board (Sun et al. 2025). computer use benchmark designed for scientific professionals, our models show modest yet meaningful capability as shown in [T The ScaleCUA-32B outperforms strong VL.Ms such as GPT-4o (1 (1.6) while remaining below Qwen2.5-VI.-721 (12.9) and Claude-3.7 3.7- Sonnet (10.5). Our model excels in domains demanding factual and visual-text reasoning over those requiring specialized symbolic workflows.
To evaluate the transfer learning capabilities of ScalcCUA-32B. we augment our training with diverse set of multimodal data focusing coding, math and reasoning. These data, sourced from the post-training corpus of Intern (Zhu et al. 2025), encompass range tasks, including OCR, mathematics, coding, reasoning QA, and general multimodal understanding. We then assess perfor- mance on four standard General Multimodal Benchmarks shown in Table These benchmarks jointly evaluate skills such as mathematical and commonsense reasoning, text comprehension, and
19

---

Published as a conference paper at ICLR 2026
Table 6: Results on ScreenSpot-v2
Wu al. 2024b).
Mobile
Desktop
Method
Avg
Text
leon/Widget
Text
Teon/Widget
Text
leon/Widget
Proprictary Models
Operator 2025
47.3
41.5
90.2
80.3
92.8
84.3
70.5
Clande-3 Sounet
87.6
Sead-1.5 VI. 2028
95.2
General Open source Models
Kimi VL-AABB
Thinking
2506
2025b
91.4
VI-78-
2025
90.5
InternV13.55244 213,5.2441---- 11441--A2IU
A2ST
2025a
97.9
91.5
97.4
82.9
94.0
89.2
92.9
Qwen2.5-VL-3B
Qwen2.5.VV 5-V-225-VL-78
20233
97.6
87.2
90.2
74.2
93.2
81.3
88.8
Quen2.5-VI.32H
97.9
88.2
98.5
79.3
91.2
86.2
91.3
GUT Specialist
OS-Atlas-Base-28 (20246
95.2
75.8
65.6
90.6
UI-TARS-2R 2025
95.2
79.
90.7
63.6
87.2
78.3
84.7
UI-TARS-70
4025
96.9
95.4
85.0
93.6
85.2
91.6
UI-TARS-72B
2025
94.8
86.3
91.2
87.9
91.5
87.7
90.3
UI-TARS-1.5
2025
94.2
99.0
96.9
85.0
93.6
92.8
GUI-Ow1-3218
2025
98.6
90.0
97.9
87.8
94.4
86.7
93.2
Grounding
Models
SeaClick
2024
78.4
50.7
70.1
29.3
55.2
32.5
55.1
JEDI-3B 2025
96.6
81.5
96.9
78.6
88.5
83.7
88.6
JEDI-7B
GUI-Actor-75 2025
97.6
88.2
96.9
85.7
93.2
86.7
92.1
GUI-G"-7B
2025
InfiUUI-GI--
2025
99.3
88.2
94,8
82.9
94.9
80.3
91.1
InfGUUL-1-D
99.0
82.1
97.9
93.5
GTA1-7B 2025
99.0
88.6
94.9
89.3
92.3
86.7
92.4
GTA1-32B
98.6
89.1
96.4
86.4
95.7
88.7
93.2
Ours
ScaleCUA-3B
94.1
86.3
79.3
89.7
85.7
89.2
ScalcCUA-7B
97.3
90.5
95.4
87.9
94.0
88.7
92.7
ScaleCUA-3228
98.6
91.9
99.0
90.0
94.4
91.6
94.7
open-domain visual question answering, which are also fundamental for computer-use agents. The "ScaleCUA-3B (25%)" specifies the proportion of this general-purpose purpose data relative to the core GUI data used in training.
Based on Table several consistent trends emerge regarding the interaction between the proportion of general-purpose data and agent performance general VLM benchmarks. First, incorporating moderate amounts of general-purpose data (e.g., 25-50% relative to GUI-specific data) yields notable gains exposing the agent to broader multimodal reasoning tasks improves its mathematical and cross- domain inference ability. For instance, ScaleCUA-3B rises from 52.8 58.7 Vista and from 48.8 to 52.4 on MMMU when increasing general data to 50%, while maintaining stable performance on RealWorldQA. Second, the results indicate a saturation effect: pushing the general data ratio to 75% 100% offers only marginal or inconsistent benefits. Third, scaling model capacity amplifies the positive effect of general data. However, our and 32B models still exhibit substantial performance gap compared to the baseline general benchmarks, indicating that the proportion of general-purpose data could be further increased. Such adjustments must also consider their potential impact the computer-use capability of agent models.
the 0% setting, particularly Math Vista and MMMU Valid suggesting that
A.3 PUBLIC DATA USED TRAINING
Table 13 summarizes the public datasets used for training ScaleCUA. Please note that the reported statistics refer to the portion each dataset actually utilized our experiments, rather than the original sizes of the source datasets.
A.4 ACTION SPACE
To enable robust cross plaatfom control, define unified action space that abstracts low-level GUI actions into a concise yet expressive set of semantic commands. As shown in Table [4] this action space is designed be platform-aware yet semantically consistent, allowing agents to
20

---

Published as conference paper at ICLR 2026
Table Results on ScreenSpot-Prg (Li et al. 2025).
Agent Model
Development
Creative
CAD
Scientific
Office
08
Text
loon
Text
leon
Text
Icon
Text
Icun
Arg
Text
Icon
Avg
Text
Ioen
Clande 10246
34
3.7
15.8
4.5
35.1
Upen
Qwmn2-V1 V1-77
2.6
0.0
0.9
GupAgen-118
0.0
100
Qwen2 VL
38.3
3.4
21.4
60.9
4.9
25.8
22.3
6.3
18.4
44.5
10.0
29.5
48.0
12.0
40.9
33.6
4.3
20.1
25.9
Owen2.5-5 7B
3305
51.9
4.8
29.1
36.9
8.4
24.9
17.8
1.6
13.8
48.5
8.2
31.1
53.7
18.9
45.7
34.6
7.9
22.4
27.6
Qwen2.-VV.
522
74.0
21.4
48.5
61.1
13.3
41.1
38.1
15.6
32.6
78.5
29.1
57.1
76.3
37.7
67.4
55.1
27.0
42.3
47.6
CLUI Specialist
SB
2024
16.9
1.4
9.4
9.1
QU0
5.3
2.5
0.0
1.9
13.2
7.3
10.5
15.3
7.5
13.5
10.3
2.2
6.6
7.2
05-Aila-78
33.1
1.4
12.7
28.8
2.8
17.9
12.2
4.7
10.3
37.5
7.5
24.44
33.9
5.7
27.4
27:1
4.5
16.8
18.9
UI-TARS-28
47.4
4.1
26.4
42.9
6.3
27.6
17.8
4.7
14.6
56.9
17.3
39.5
511.3
17.0
42.6
21.5
5.6
14.3
27.7
UP-TARS-26
2053
58.4
12.4
50.0
9.1
33.8
20.8
9.4
63.9
31.8
50,0
63.3
20.5
53.5
30.8
24.5
35.7
UL-TAR--22 73
40.8
18.8
20.9
30.1
TARS
UI-TARS-LL5
GUI-Ow)
79.1
87.9
GUL-Ow1-3215
62.5
62.4
39.1
39.6
71.8
54.6
58.0
Cironding Midals
SocClick
0.6
0.0
0.0
2.0
1.1
2.8
Adaa
16.2
0.0
23.7
2.1
14.7
7.6
1.6
6.1
27.1
6.4
18.1
20.3
1.9
16.1
4.7
0.0
11.3
UGroune
M04
35.5
21.8
13.5
38.8
18.8
25.1
31.1
LKiound-VV1-2
V1-726
2024
$1.1
35.8
13.8%
50.0
$1.3
25.5
54.5
JEDI-3B
305
61.0
13.8
58.1
53.5
8:4
34.6
27.4
9.4
23.0
54.2
18.2
78.6
64.4
32.1
$7.0
38.3
9.0
25.0
56.
JEDI 7B A3143
42.9
11.0
27.4
50.0
11.9
34.0
39.0
14.1
32.2
72.9
25.5
52.4
75.1
47.2
68.7
33.6
16.9
26.0
39.5
UI
2024
22.7
4.1
23.3
3,5
11.2
6.3
42.4
11.8
32.2
11.3
13.1
4.5
17.8
InSKUU-RR
1- 20246
51.3
12.4
44.9
7.0
33.0
14.1
59.3
20.0
65.5
28.3
43.9
12.4
35.7
InAGGU-RI--D
57.4
25.4
74.2
24.1
64.6
15.4
50.5
31.8
75.7
39.5
57.0
29.2
$1.9
GUI--1-98
2024
50.7
10.3
31.1
56.6
11.9
26.6
39.5
9.4
32.2
61.8
30.0
48.0
67.3
32.1
59.1
23.5
106
16.1
37.1
GUI-G2778
Scal:CUA-BB
53.9
Scal:CLAA-7B
2017
54.8
77.1
24.5
45.3
67.4
ScslsCUUA-328
35.
55.8
73.2
$5.4
60.4
76.4
46.4
63.4
$1.4
49.1
73.9
53.6
59.2
operate seamlessly across Desktop (Windows, macOS, Ubuntu), Mobile (Android, iOS), and Web platforms. The action set includes universally supported operations such as click, write, wa wait, and termi nate, which are shared across all platforms. It also accommodates platform-specific interactions, including swipe and longspress for mobile devices, and fine-grained mouse or keyboard controls such as dcubleClick, rightClick dragTo, and hotkey for desktop and web interfaces. To handle modern interactive elements, s swipe operation has also been implemented for Web. By standardizing the operation interface through a shared action space, we simplify training and inference while supporting both generalization and specialization. Each action is defined with explicit arguments (e.g., coordinates, keypresses), enabling precise control and compatibility with structured outputs in grounding, direct-action, and reasoned-action inference modes. This design facilitates modular training, policy transfer, and scalable data annotation, forming a critical foundation for developing universal GUI agents.
Table 8: Performance on the MMBench-GGUI (GUI Element Grounding) Wang al. 2025c).
Windows
MacOS
Linux
iOS
Android
Web
Model
Avg
Basic
Adv.
Basic
Basic
Adv.
Basic
Adv.
Basic
Adv.
Basic
Adv.
GPT-4
2014
8.7
4.3
1.1
1.0
5.1
3.3
2.5
1.4
3.2
2.9
Claude-3.7
12.5
7.5
1.1
0.0
13.7
10.6
1.4
1.4
3.2
2.3
4.7
Qwen Max
212.5
43.9
36.8
58.8
56.1
53.9
30.1
77.4
59.1
79.5
70.1
74.8
58.8
58.0
Aguvis-7B.7200
2112
37.3
21.7
48.1
33.3
33.5
25.0
67.5
65.2
61.0
$1.0
61.6
45.5
45.7
ShowU1-21
2024
9.2
4,4
24.1
10.4
25.1
11.7
29.0
19.7
17.4
8.7
22.9
12.7
16.0
OS-Atlas-Base-7
7B
2024h
36.9
18.8
44.4
21.7
31.4
13.3
74.8
48.8
69.6
46.8
61.3
35.4
41.4
UGround-V1-78
2074
66.8
39,0
56.5
31.1
92.7
70.9
71.0
88.7
64.6
65.7
InternV1.3-222
20225
70.1
42.6
75.7
52.3
59.2
41.3
93.6
80.6
92.7
78.6
90.7
65.9
72.2
Owen2.5-VL-721e
2025
55.7
33.8
30.1
40.3
20.9
56.1
28.2
55.6
25.4
68
45.8
41.8
Qwen2.5.V1-72
20723
31.4
16.5
31.3
22.0
21.5
12.2
66.6
55.2
35.1
35.2
40.3
32.5
33.9
UI-TARS-I.5-7B
68.3
39.0
64.4
37.8
88.5
69.4
81.0
56.5
UL-TARS-72B-DVU
2025
78.6
51.8
S0.3
62.7
68.6
$1.5
90.8
81.2
93.0
80.0
88.1
68.5
74.3
GUI-Ow1-7B
2025
86.3
61.8
74.4
61.7
94.9
83.0
83.7
93.2
72.7
80.5
GUI-Owl-328 22025
85.6
65.1
84.9
67.1
77.0
63.3
95.2
83.5
96.1
87.0
95.5
80.8
83.0
InfiGUl-Gi-3B
2025b
74.2
47.1
78.8
65.4
41.8
95.2
78.8
92.1
78.0
89.7
64.3
73.4
IntiGU--G1-7
20236
82.7
61.8
83.8
63.9
72.3
$2.0
94,9
89.4
95.2
83.6
93.5
76.3
80.8
ScaleCUA-3B
78.6
46.0
79.4
75.3
49.0
93.0
73.3
94,1
74.4
92.6
63.6
73.7
ScaleCUiA-7B
78.6
54.0
81.3
58.7
74.4
$6.6
94.3
81.8
96.1
81.1
92.6
73.1
78.2
ScaleCUA-32B
83.0
62.9
64.2
81.2
65.8
95.9
84.9
96.4
81.7
93.9
76.3
82.0
21

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison on OSWorld-G (Xie et al. 2025
Text Matching
Element Recognition
Layout Understanding
Fine-grained Manipulation
Agent Model
Refusal
Overall
Gemini-2.5-Pro
2025
59.8
45.5
49.0
33.6
38,9
45.2
Operator 2025
51.3
42.4
46.6
31.5
0.0
40.6
Seed1.5- V1.
2025
73.9
66.7
69.6
47.0
18.5
62.9
OS-Atlas-7B
2024b
44.1
29.4
35,2
16.8
7.4
27.7
UGround-VV 1-78--788 2024
51.3
40.3
43.5
24.8
0.0
36.4
Aguvis-7B: (2024
55.9
41.2
43,9
28.2
0.0
38.7
UI-TARS-7iss (0125
60.2
35.6
47.5
UI-TARS-1.5-755 22025
2025
70.1
57.9
59.7
51.7
0.0
$6.0
UI-TARS-72B (2025
69.4
60.6
62.9
45.6
0.0
57.1
Qwen2.5-VL-33k
2025
41.4
28.8
34.8
13.4
0.0
27.3
Qwen2,5-VL-7B
2025
45.6
32.7
41.9
18.1
0.0
314
Qwen2,5-VL-32H
[2025
63.2
47.3
36,9
0.0
46.5
L
2025a
64.4
58.8
55.3
43.0
0.0
53.2
JEDI-3B 2025
67.4
53.0
53.8
44.3
7.4
50.9
JEDI-7B k1%
65.9
55.5
57.7
46.9
7.4
54.1
ScaleCUA-3B
64.8
61.8
64.0
43.6
0.0
55.7
ScaleCUA-7B
67.8
61.8
64.8
49.7
0.0
57.8
ScaleCUAA33BB
69.0
66.7
70.0
51.0
0.0
60.6
Table 10: Performance comparison on AndroidControl (Li et al. 2024b)
AndroidConttoo-Lec
AndroidControl High
Agent Model
Type
Grounding
SR
Type
Grounding
SR
Claude 2024b
74.3
19.4
63.7
0.0
12.5
GPT-4o
74.3
19.4
66.3
0.0
20.8
SeeClick 2024
93.0
73.4
75.0
82.9
62.9
59.1
InternVL. LB-2-BBB 2024b
90.9
84.1
84.1
72.7
66.7
Owen2-VL--27
2024
91.9
77.7
69.7
Aria-UI 22024
43.2
10.2
OS-Atlas-1B3
(2024b
91.9
84.7
73.8
67.5
OS-Atlus-BB (2024b
93.6
78.5
71.2
Aguvis-7B (2024)
61.5
Aguvis-72B (2024
84.4
66.4
OS-Cenesiis-TH (2024b)
91.3
74.2
66.2
44.5
U1-TARS-28 (2025
98.1
87.3
89.3
81.2
78.4
68.9
UI-TARS-7B 22025
98.0
89.3
90.8
83.7
80.5
72.5
UI-TARS (2025
98.1
89.9
91.3
85.2
81.5
74.7
Qwen2.5.VL 2205
90.8
63.7
Qwen2.5. 2025
91.4
60.1
Qwen2.5.VL- 2025
93.3
69.6
Qwen2.5-VL-72B 2025
93.7
67.4
LntemWLL3.52211B-3222
2025a
88.1
93.4
82.1
81.0
81.5
68.2
ScaleCUA-3B
91.4
93.7
84.1
81.4
83.9
70.3
ScaleCUA-7B
93.3
93.1
86.0
86.3
84.3
74.8
ScaleCUA-32B
91.9
94.7
85.7
85.7
87.3
75.9
A.5 ERROR CASE ANALYSIS
We here provide several етror cases across different platforms to analyze the limitations of our ScaleCUA.
On desktop platforms, ScaleCUA frequently violates procedural prerequisites shown in Fig 7 and F such as attempting to compress files without selecting them or changing font styles without highlighting the target text. These issues stem from an incomplete understanding of interface states and sub-task dependencies. Moreover, a significant limitation of ScaleCUA emerges when actions result in silent failures, characterized by lack of discernible state transition. In such instances, the model tends to perseverate on the unsuccessful operation, revealing the absence of robust error-recovery mechanism. This issue underscores the critical requirement for fine-grained perception
22

---

Published as a conference paper at ICLR 2026
Table 11: Performance comparison on ScienceBoard (un et al.
2025).
Model
Algebra
Biochem
GIS
ATP
Astron
Doc
Overall
GPT-4o
2024
3.2
0.0
0.0
1.6
Claude-3.7 3.7-Sonnet
2025
9.7
37.9
2.9
0.0
6.1
6.3
10.5
2.0mii--.0--lash
2024
6.5
3.5
2.9
0.0
0.0
6.1
3.2
Owen2.5-VL-722
2025
22.6
27.6
5,9
0.0
9.1
12.5
12,9
InternVL3--8BB
2025
6,5
3.5
0.0
0.0
0.0
6.3
2.7
UI-TARS-1.5-7B (2025
12.9
0.0
5.9
ScaleCUA-3B
6.5
13.8
0.0
3.6
ScaleCUA-7B
3.2
3.4
0.0
0.0
1.8
0.0
1.8
ScaleCUA-32B
9.7
10.3
0.0
0.0
12.1
0.0
5.9
Table 12: Performance on General VLM Benchmarks. ScaleCUA-3B (25%) denotes that, during training, the number of general-purpose data samples was set to 25% the GUI data samples (e.g.. Understanding, Grounding, and Planning).
Model
Math VistaMINI
2023
OCRBench 2024b
MMMU Snain
2024
RealWorldQA 2024)
Qwen2.5-VL-3B (2025
797 (79.7)
65.4
ScaleCUA-3B (0%)
52.8
819 (81.9)
48.8
65.2
ScaleCUA-3B (25%)
58.6
823 (82.3)
50.6
65.4
ScaleCUA-3B (50%)
824 (82.4)
ScaleCUA-3B (75%)
59.3
818 (81.8)
55.6
65.2
ScaleCUA-3B (100%)
60.6
806 (80.6)
53.4
63.5
Qwen2.5VL-7B (2025
864 (86.4)
ScaleCUA-7B (50%)
65.4
852 (85.2)
54.7
69.8
Qwen2.5-VL-32B 22025
74.7
854 (85.4)
70.0
72.2
ScaleCUA-32B (75%)
827 (82.7)
72.3
and robust understanding of element state interact with context-dependent UI elements, such as focus and selection.
For the Android platform, there exist precision and positional challenges demonstrated in Fig 9 In the first case, the instruction explicitly requires appending text to the top of file within note-taking application (Markor). However, the agent fails recognize this positional constraint, instead inserting
Task: Create zip archive named "DoonnloabBBsckup.ipp from all files the Downloads folder.
compression.
Help me change \"H2OY" to subscript.
repeats clicking wrong spui
Figure 7: Error cases on the Windows platform. The first case shows ScaleCUA creating archive without having selected any files, revealing that sometimes fails to follow the full instruction and only completes sub-step. The second case shows ScaleCUA persistently repeating the same action until the step limit, when it misses the correct element and the screen remains unchanged.
23

---

Published as a conference paper at ICLR 2026
Table 13: Public data sources used for training our models. The "*" indicates that we count the number of samples we use rather than the full size of the original dataset because we have processed these datasets and filtered some noisy examples. Some statisties are calculated from Aguvis 2024))
GUI Task
Data source
Platform
#Elements / #Steps
SceClick
2004
Website
271K
GUIEnv 2024a
Wehsite
328K
GUIAct
2024a
Website
67K
WebUT
2023a
Website
57K
Widget
Captioning
2020h
Mobile
10IK
Grounding
RicoSCA
2020a
Mobile
173K
UI RefExp
2021
Mobile
16K
RICO Icon
2017
Mobile
16K
OmniACT
2024
Desktop
Wehsite
7K
UGround
2024
Website
1404K
OS-Atlas
2024b
Desktop
799K
JEDI
2025
Desktop
550K
Total
3789K
MM-Mind2Wch
2023
Website
7.8K
GUIAct 2024a
Wehsite
16.6K
MiniWoB+ 2017
Website
9.9K
AitZ 2024
Mobile
11.9K
Planning
AndroilControl 2024a
Mobile
74.8K
GUI Odyssey 2024a
Mobile
118.3K
AMEX 2024
Mobile
35.6K
AitW 2023
Mobile
19.0K
PC Agent-F 2025
Desktop
27.8K
Total
321.7K
Table 14: Actions space.
Action
Platforms
Description
cxick(x, y, clicks, button)
Perform mouse click at coordinates (x, y) using the specified button and number of clicks.
write (measage)
Input the given message. Pause execution the specified number of seconds.
(secondd)
responae (anawer)
Submit response the environment task prompt.
terminate (statuu))
Temninate the current task with given completion Performs scroll of the Perform double click coordinates y) with the specified button.
scroll (olicks,
Desktop
scroll wheel position y).
doubleclick(s. (x, Vr butten)
Desktop Web
eightClick(s (x, y, button)
Desktop & Web
Perform right click at coordinates (x. y) with the specified button.
hotkey (*args)
Desktop
Desktop Trigger keyboard
composed
keys.
movelo (x, y)
Desktop Drag the mouse pointer specified (x. y) position.
dragTo (x, y. buston)
y) while holding specified
preas (keys, presses)
Desktop
specified key(s) a given number
keyDown (key]
Desktop
hold key without releasing
kkyt)p(ke))
Desktop Swipe start end point speciffidd direction
previoudly held key.
swipe (from, to, direction, amount
Mobile
given intensity or distance.
navrigste_home ()
Mohile
Return the mobile home screen.
navigate_back
Mobile
Novigate back to the previous screen on mobile.
long.presa:!] ()
Mobile
Perform long-press gesture on the current focus or location.
open.app (app_name)
Mobile
Launch mobile application its name.
content at the current cursor location without adjusting it. This suggests that ScaleCUA lacks fine-grained understanding of positional semantics in natural language instructions, as well as the ability to reason about UI state changes like cursor positioning. In the second case, the agent is instructed "Take one photo." Despite correctly launching the camera and triggering the shutter once, the agent erroncously repeats the same actions multiple times. This behavior stems from failure detect visual feedback confirm state transitions (e.g., captured photo thumbnail),
24

---

Published as a conference paper at ICLR 2026
Task: am currently using an Ubuntt system, and have wrongly deleted a poster of party night. Could help me recover it from the Trash?
Atternpt image being
Task: In the first slide, insert the title "Happy Family" and make the font style "Microsoft Hein
Forgol select
Figure 8: Error cases on the Ubuntu platform. ScaleCUA repeatedly fails tasks because it does not comprehend procedural prerequisites. The agent attempts to execute final command without first performing the necessary intermediate step of selecting the target object. For instance, it tries to restore file without selecting it from the trash or alter a font without highlighting the text. Critically, this operational flaw generates no explicit error message, trapping the agent in repetitive loop of ineffective actions.
Task: Edit note SiFbwtxt Markor. Add top Hello, World!
Ignore key information instructions.
Task: Take photo.
Repeat
Figure 9: Error cases the Android platform. The first shows instruction requiring content to be inserted at the top of document; however, ScaleCUA opens the file and inserts directly at the current cursor location, ignoring the positional prerequisite. The second case shows that when the UI exhibits no obvious state change after an operation, ScaleCUA repeats the same action multiple times, causing tasks such as taking photo fail.
leading to unnecessary repetition. These failure modes indicate two key limitations: (1) insufficient grounding of spatial and contextual cues embedded in task descriptions, and (2) inadequate visual state tracking, particularly under conditions where UI feedback is subtle. Addressing these issues may require enhanced visual reasoning modules, memory-based state modeling, or task -uisedd grounding refinements.
25

---

Published as a conference paper at ICLR 2026
Task: What do customers say about brush from sephora?
X
Agent subjectively chooses another item with reviews rather than the brush task needs.
Task:
Edit my post on Star Trek Starfleet Academy series by adding a line to the body that says "Every watch makes me feel like a kid again".
selects
selects post for editing that not related to the "Star Trek Starfleet Academy" topic.
Figure 10: Error cases on the Web platform. The first case shows ScaleCUA made subjective analytical assumptions, presuming the product necessarily contained reviews, while disregarding the explicitly specified product category in the task instructions. The second case shows ScaleCUA struggles with complex tasks in complex initial environments (where numerous posts already exist on the starting interface). When faced with multifaceted requirements (needing to identify both "my" posts and posts a specified topic), it neglected the explicitly stated topic in the instructions, instead selecting only posts visible in the current observation space that belonged to me.
Empirical analysis of trajectories from web platform reveals that ScaleCUA may struggle with seman- tic disambiguation. ScaleCUA often selects visually salient but instructiion-ncconisseentt elements (e.g., wrong product category or unrelated post) presented in Fig TO revealing a bias toward superficial cues over explicit constraints like ownership ("my post") or topical relevance ("Starfleet Academy")..
To mitigate these issues, three avenues may show promise: (1) Reflection and State Verification. Integrating lightweight screen-change detectors and visual precondition checkers can allow agents to validate action effects and avoid ineffective loops. (2) Reinforcement Learning with Recovery Signals. Reward structures should penalize redundant, non- proogreeeee behaviors and incentivize predicate satisfaction (e.g.. "text selected"' "correct tab active") before proceeding. (3) Memory- Augmented Planning. By introducing episodic memory to recall past interactions (e.g., whether menu opened successfully). the agent can reason across time and avoid retrying failed subgoals.
A.6
THE DETAILS OF DATA CURATION
A. DATA SOURCES
We systematically collect GUI data across diverse platforms to construct ScaleCUA-Data, including desktop, mobile, and web environments. As shown in Table. [5] ScaleCUA-Data spans 7 major operation systems: Windows, Ubuntu, macOS, iOS, iPadOS, Android, and Web. Each platform features a broad spectrum frequently used applications designed for productivity, communication, entertainment, browsing, and utilities.
On desktop platforms, Windows includes both native and third-party applications such as Microsoft Office Suite, Adobe Creative Cloud, Visual Studio. and system utilities, offering comprehensive view of traditional GUI layouts. Ubuntu and macOS incorporate open-source and system software, including LibreOffice, GIMP, Terminal, Finder, and Safari.
Mobile data is collected from the iOS and Android platforms. The data from the iOS platform includes system applications such as Settings, Safari, Calendar, and Health, as well as third-panc hird-party applications including Weibo, Notability, and Spotify. The Android platform. by virtue of its open ecosystem, serves as the greatest diversity of data sources, encompassing both system applications
26

---

Published as a conference paper at ICLR 2026
Table 15:
The main sources of GUI corpora across different platforms.
Platform
Application
Windows
File Explorer, OS, Chrome, Microsoft Edge. Word, Excel, PowerPoint. LibroOftie Cale, LibreOffice Impress, LibreOffice Writer, Maps, Camera. Calculator, Microsoft Store. Clock, Photos, Outlook, Media Player, VLC Media Player, Calendar. Paini, Paint 3D, QQ Music, KuGou Music, Spotify. Tencent QQ. Visual Suudio Code, Dev Cey,Crt, Microsofi Solitaire & Casaal Game, Pycharm, Android Stadio, Vmware Workstation Pro, Vanaae Fusion, Adobe Photoshop, Adohe Premiere Pro, Adohe Illustrator, Blender, FL Smidi, Unreal Engine. DaVinci Resolve, AutoCAD, SolidWorks, loventor, Vivado, MATLAB, Origin.
Stata, Eviews
Uhuntu
Files, OS, Firefox, Chrome, LibreOffice Cale, I LibreOffice Impress, I LibreOffice Writer, OneNote, GIMP, Slack, Thunderbird, Visual Studio Code. Zotero
MacOS
Finder, OS, Safari, Chrome, Pagos, Numbass Keynote, Calculator, Maps, Notes, Calendar, Contacts, Reminders, Apple Music, Podcasts, Weatber, Stocks, Freeform, Terminal, Clock, Pycharm, Android Studio, App Store, Mail, Visual Studio Code
iOS
Wenther, Maps, Find My, Settings, Stocks, Safari, Mail, Calendar, App Store, Home, Camera, Files, Wallet, Contacts, Shortcuts, Clock, Twitter, Woibo, Outlook, Reddit, Instagram, Notes, Keynote. Reminders, Notability, GoodNotes, Rednote, Translate, Calculator, Voice Memos, Shadowrocket, Music, Podcasts, Spotily, iTunes Store, Apple TV. Books, Zhibu, Health
iPad08
Weather, Settings, Safari, Camera, Goodnates, Translate, Notes, Freeform Chrome
Android
Settings, Clock, Desktop Clock, Calendar, Contacts, Files, Camera, LinkedIn, Weibo, Twitter, Ticha, Reddit, Zoom, Gmail, Duolingo, Xusersi. Wikipedia, XuctangX, edX, Coursera, Skillshare, ZLibrary, To Do. Word. Excel. PowerPoint. On:Note. Taskdde Notion TicKTick. Google Maps, AMap, Tencent Map. Qunar, Trip.com, Cirip. Qunar, LY.com, Flggyy, Zhixing Train Tickets, Map.me. Booking. Amazon, eBay, Tachao, Alipay, Poizon. VIPShop. 58.com, Beike, Anjuke, Zhuanzhuan, Douyin Mall, Shihoo, Nike, Bilibii,, Bilibili CN. QQ Music, himalaya, Classical Music, News, Toutian, Sohu News, NetFase News, Hupu, Huya, Sohu Vidzo. Pi Music Player, NetEase Cloud Music, Kuaisbou, Kugou, WeSing, Douban, Xisshongshu. Zhihu, Qidian. Xisobeihe. Prime Video, CNN. Quora, Cantock, Spotify, Apple Music, YouTube, Fitness, Health, Health. Translate, Maji Weather, App Store, Google Chrome, BlueCuins, VPN, Shudowrocket, Suriboard, Speedlest, Meitu, Jianying. Canva, Procreate, Pinterest, GitHuh, DeepSeek, Grok
Web
Si5j(sh.5i5 com). AccoWatho(acuueeahee.
adidas China(adidas.com com.cn), en), Adobecadobe com), Amazcn(amazzo.comm
Baidu(baidu.c com). Baidu Baike baidu.com) Baidu fiebattieb..aidu.cco Beihang www.eersttybuaaaduuu.. Blibdiiiiiiiillcco
American Kennel Club(akc org). Appletapplc.com) arXiv(arxiv.org) BabyContorbabyyentee..
(eraeeGGGeekoorgggmggeeee cambridge.org org), Cars cancccom((ass.mmm)
BuurdMis(hoardmiix.cn Booking comm com). Budget(budgel.com comm)
Camhridge Dietionary(dictionn
(BBlogs(ebllogs.cmm) CNN((enn.comm com),
Courseratcoursern.org... (SDNicsdn.net)) Crip(etrip.com com)) Damai damai en), Drugs.
Dianping com). Dior(dier.oom. Doubanfdouban.com
eBByyeeha.soom), Britannicu(biilane brilaannnaaanni..mmm))) ePaylepay.comm) ggicurrioo/eppicriiaa comm), mougs.cmmlce dewgs...omm
Douyinídouyin.com
Fustly(fasill.com FedFEx(fedex, avm). Flggy(flggy.comm Food (frwaer(f cro)) Gaode Maps<gande com), Gmail(gmail.com com), GitHabgiithub.oo com). Google Finance( google.cooooooooo. com), Google Maps(map, google.com) com), Google Scholartscholar wow.ooa.goooo...oo.oooooooooo coogle.com), GOV.UKggov.uk Hcalthlinz/bsatthnnn com). Hugging Facollungingffccc.o.o co). Huputhupu.com com). KiNigg.ccom com). IMDotindh.comm) Indeed UKiuk.ineed..cmm :Qiyitiqiyi.comm) JD.comjjd.comm com)), jeBrainsjettrrinn cotm)ins.com). KAYAK(kayak. com). Kohl's s/kokks..coom com). Last fimflat fim). LeetCodeeleetccneeeco en), LinkedIn(linkedd) com). riott(mariiot.cco com). Microsoft Azure(azure.mienn www.auureemmrrsoot.ooo com), Microsoft Officetoffice.com com). BodeeSccop<mmoesccop en), MSN(msn.com.) com), NBA(aba com)), National Relostionnaaionnieecaaaooo. commm NotEase Cloud Music(music. 163 com). Neweggtnewegg.co com). DpenSSreetMagooopesteeenng QQ com). QQ Musicty-qg com). QS Chinalqschina..cm Reddiit(eddii.com. Redlinfredfin.com),, REllrei.comi) Rotten Tomatoes comateesmiteentomaaees..mm com). Ryanain(yanair.com) Samsung(samsung com)) Shimo(shimo.ing Sina News(news com.......n Skype(skype.comm, SpotHaroospathero.com) poothero.o.o) Stack Overfoosstackoverloo crw.o..ooo.o..oomm.... Steam
PsyPallpaypallccmm PJLab GiiLabgiilab, pjlabbor.......g.... en).
Storet store. www.mpwweeee.ccmm
Student.coml student com). TensorFlow tessoooloowtessrroooo.r...
Tencent Decsidocs.gg dow.s.gg..om.......oooooooooomo.
Tencent Videoí comcom) The Weather Chammel( weatterr.com) com)) The Weather Nelwork((heweauber wwweeeabbeeewwwrrccom. Thumb-
tack(thumhtcck.o com), Ticket Centertiicketcenter.com Trip.com (.((us..ri..com) TripAdvisortripddv China(uniqlo.cm en), United Airlinestunited.omm com), University of Cambridgercam.ae.n
UNIQLO
University of Michigan(umich.cuu
mnall(ymall..com. Virginia DMV virginia. gov). WebArena Foruntwa_forum) WebArena GitLab wiggitab....
WehArena Saapping(w..shoppgg
WehArena 2MS(wa.shopping admii.) WebMD(wwhand.comm Weiho(weibb..oml)
X(x.com). Sinnhongeho@siicchnnn.... com). Yahoo Finance(fiaanee yahoo.ceyyhhooooom. yahoo.comi, Yahoo Sports(sports, yahoo.aooooooooo com). Yelp(yelp.com) com) You YouTubeyyomuheecomm com), Zhihu[zhihu.com
Wikipedin(wikipp oogg.
Zhaopin() aanopio(.zhaopin.oo com))...com.. Zhaopin Landing Pago(landing, zhaopin.com). com). Zhipin( (itppin(hhpiiccomm and ~ 0.2M URLs selected from
TOP- IM BRLLshapss/traaco-lii
27

---

Published as a conference paper at ICLR 2026
and broad array of commercial software from domains such productivity, e-commerce, social media, and multimedia (e.g., WeChat, Taobao, TikTok, and Google Suite).
For tablet interfaces, our data collection primarily focused on iPadOS, encompassing selection of its most frequently utilized system applications.
As for Web, collected pages from over 200 frequently accessed websites spanning e-commerce, social media, education, government services, travel, and developer tools. These sources encompass major websites such as Amazon, YouTube, Reddit, Wikipedia, Coursera, and GitHub, with data captured through both static DOM snapshots and dynamic interaction traces.
The collected dataset constitutes high-coverage, pross-platform corpus of real-world graphical interfaces which endows the model with comprehensive domain knowledge and leads to significantly improved generalization.
A.6.2 GUT UNDERSTANDING
To support the development of general-purpose computer use agents, we construct large-scale corpus for GUI understanding that encompasses both element-level and screenshot-level semantics. This corpus is designed to facilitate fine-grained perception and reasoning over static and dynamic user interfaces.
For element-leve. understanding, we define five task formulations targeting visual appearance, spatial layout, textual grounding, and semantic functionality. First, introduce the Element Appearance Captioning task, which requires the model to describe visual features (e.g., shape, color, borders) of a given GUT component. These attributes often signal affordances and can help distinguish between interactive and static elements. Second, we incorporate Referring OCR, referring task where the model extracts the textual content within specified bounding box, enabling alignment between visual context and embedded text. Third, to capture spatial organization, the Element Layout Understanding task asks the model to predict both absolute screen coordinates and relative positions with respect to nearby components. Fourth, to understand the operational roles of components, we define the Element Functionality Captioning task, where the model infers the intended function a labeled element within its surrounding interface. Finally, we propose User Intention Prediction task, where the model is asked to infer the user's likely goal based on contextual clues and ongoing interactions.
For screenshot-level understanding, we formulate two tasks that promote global comprehension. The Interface Captioning task prompts the model to generate a high-level textual description summarizing the overall structure, visual hierarchy, and content of the interface. This encourages holistic reasoning and layout recognition. Complementarily, the Screen Transition Captioning task focuses on temporal changes by asking the model to describe the differences between two consecutive screenshots. This enables the model to understand GUI dynamics, such as state updates, navigation events, or content refreshes.
Together, these tasks define a comprehensive benchmark for GUT understanding. We leverage vision- language models automatically generate annotations for both element-level and screenshot-level tasks, using visual context, structural metadata, and interaction histories. This corpus provides the foundation for training agents capable of fine-gra me--grained perception, robust grounding. and high-level reasoning in complex GUI environments.
A.6.3 METADATA EXTRACTION
Windows Platform. To facilitate the automated analysis and interaction with graphical user interfaces (GUIs), we design and implement a framework for extracting UI metadata on the Windows operating system. The core of this framework leverages the UI Automation (UIA) technology to perform depth-first traversal of an application's 's Ally Trees, initiated from the foreground window identified via native Win32 T API calls. Subsequently, the collected raw data undergoes multi- filtering and refinement pipeline to ensure its relevance and actionability. This pipeline first performs geometric validity check to filter out improperly sized or off-screen controls, followed by a visibility and occlusion analysis to retain only the topmost, unobscured elements. Furthermore, a semantic pruning module uses a predefined keyword list (e.g., "close", "save") to remove controls that
28

---

Published as a conference paper at ICLR 2026
might cause task interruption, while system component exclusion module discards elements within standard OS regions like the taskbar based on their absolute coordinates. Each element that successfully passes through this pipeline is then abstracted into structured JSON object. This object encapsulates its multi-dimensional attributes, including identity properties (contrrol type, name), state information (1 s_enabl ed), spatial coordinates (bbox), and descriptive text (descript ion, t ccoltip). The aggregation of these objects yields a comprehensive metadata representation of the UI, establishing the foundation for subsequent automated tasks.
Ubuntu Platform. To extract Ubuntu metadata, we process an XML string representation of the Ally Trees, leveraging Python's 's built-in xml library for p The process commences by parsing the raw XML data into a tree structure. Following this, we linearize these nodes into structural elements. Specifically, for each node this set, we programmatically extract key attributes, including its tag (representing the element's role), name, class, and description. To capture the semantic content robustly, the element's text is derived either directly from its text content or inferred from its value attribute, particularly for input fields. Positional and dimensional data are extracted from screencoord and size attributes, which together define the element's 's bounding box. The final output is structured, tab-separated string where each line represents single UI element. This entry is composed of seven fields: (1) tag indicating the UI type. (2) name for the element's s given name, (3) text capturing its content or value, (4) class specifying its component class, (5) descript on for accessibility-relatee details, (6) pos t on as a top-left (x, y) coordinate, and (7) size as a width and height pair. In essence, this process distills raw, platform-specific Al Trees into flattened, understanding, grounding tasks.
aaatiicalll-annottte dataset, providing a crucial foundation for downstream
MacOS Platform. We extract UI metadata from macOS applications by leveraging the ma- cOS Accessibility API, primarily via the applicationServii
frameworks. allows structured traversal of the Ally Trees by programmatically accessing on-screen UI windows and querying attributes such as AXPos iiition, Axşize, AXRole, AxTitle, AXVa I ue, and AXDescripti on. To initiate the process, we identify top-level windows from the system window filter for visible application windows, and create AX references using AXUIELementtCreatee.ppppp ciatino....i..i..iiimiimmmmmiiii A recursive collection strategy is then applied, traversing each window's ' Ally Trees up to a bounded depth while filtering out off- screen or irrelevant elements. To ensure semantic clarity, enrich metadata by inferring contextual labels for interactive elements (e.g., AXBut ton, text, spatial layout, and role. Further, we apply spatial deduplication heuristics eliminate overlap- ping or redundant elements, and merge content-bearing AXStat cTeet regions with their parent interactive widgets when appropriate. The final output is flattened list of UI elements, each anno- tated with role, text content, description, and bounding box information. Structurally, each metadata entry consists of: (1) role indicating UI type (e.g., AxButton), (2) text and descr i pt on capturing semantic content, (3) bbox dictionary with x, y. width, and hei and (4) op- tionally a list of childree ldren for nested components. This pipeline enables robust and interpretable extraction of macOS GUI structures, supporting downstream tasks such as screen annotation, interac- tion modeling, and agent behavior learning. Additionally, due to the limited accessibility information exposed by some system-level macOS applications or the difficulty in filtering non-visible elements, we incorporate omniparser-v2 as a complementary mechanism to refine and validate extracted elements based screenshot alignment and bounding box overlap.
list using
based on their surrounding static
Mobile Platform. For Android, we begin by using U to dump the current app's accessibility hierarchy XML and parse into in-memory 1xml tree. In depth-first walk, we record each node's class, resourceID, text and content description and parse its bounds string (e.g. " [x1, yy] [x2, y2] ") into integer coordinates to build robust locators and raw geometry. During this pass we filter out any control that is off-screen, too small (for example, width px or height < 15 px), devoid of both text/description and interaction flags (clickable, focusable le. scrollable or only truly visible, actionable elements. For each remaining node, we generate concise label by combining up to the first ten words of its text or description with its UI role (e.g. "Button" or
or fully occluded by its parent-lcaving
http:///dwebooosppp....oooooooom hhtl https://haatmmmmmm/rraaaaddcdcec
29

---

Published as a conference paper at ICLR 2026
"EditText") and infer possible actions (click, swipe, long press, write). In second sweep, detect exactly which elements support taps, focus moves, scrolling, or long presses, then wrap each into structured record containing its unique identifier, bounding- box coordinates, a summary of core attributes (ID, text, type, state flags like enable led and visible to user), and the full raw attribute map ppackage, index, checkab checkable/checceeet e/cchccchhhkkeee password, etc.). Finally, we serialize this collection as a flat JSON array or tab-separated lines, producing a complete, coordinate-aware metadata set that underpins precise mobile UI analysis and automated testing. For iOS, we feed the screenshot directly into OmniParser V2 (Yu et al. 2025), which parses the page elements-exttracingg their type, bounding box, interactivity, content, and so on and uses this information as metadata.
Web Platform. Our web metadata extraction pipeline employs Selenium WebDriver with ChromeDrivee to automate web interaction trajectory acquisition using a random walk algo- rithm. At each step, it leverages browser-native rendering to ensure visual fidelity while cap- turing the current page's element metadata, including coordinates, descriptions, types, and spe- cial attribute information. The pipeline executes JavaScript parsing pipeline via Chrome DevTools Protocol (CDP) that implements a comprehensive element classification and filter methodology. Clickable elements identified through a multi-criteria approach combining semantic HTML tags (<a>, <button>, <input>, <select>, <textarea>, coption>, <video>) CSS properties (cursor: :orontrr,..ter.. since CSS properties cascade child elements, we only treat an element as clickable if its parent lacks cursor:pointer, ensuring accurate detec- tion of standalone clickable elements), JavaScript click event listeners, and element attributes (onclick, ondblc ck, roles contain button, option, tab); Non-interactive elements are systematically classified as text objects, media objects, or structural panels through DOM hier- archy analysis. All elements undergo rigorous validation including geometric verification using getBounddngllientRRee () to filter occluded components, visibility validation through CSS property checks (display: visibilitt via document... e el romPoi nt () center-point sampling to confirm visual prominence and top-layer activity. Finally, we perform set difference operation with the elements from the last step to filter out the set of new elements for the random walk. Text description metadata aggregation incorporates content from over 12 attributes including textContente i Innereett, va value, a tt, title, and aria-label work implements multiple integrity safeguards including dynamic language detection via langdetec sensitive lexicon pattern matching, and visual anomaly detection with adaptive boundary refinement. Cross-resolution robustness is achieved through randomized viewport initialization spanning device pixel ratios (1.4-2.1) and common resolutions (720p, 1080p, 2K. 4K, 2560 1600), stabilized via CSS viewport normalization techniques. This comprehensive web trajectory metadata extraction pipeline ensures exceptional data integrity, security, diversity, granularity, and accuracy, thereby establishing robust foundation for instruction construction and model training.
hi dden, opacity: 0), and active validation
normalized through whitespace compression algorithms. The frame-
A.6.4 GUT GROUNDING
GUI grounding is fundamental capability for computer agents, enabling them to associate the natural language instruction with a corresponding region of interest. Effective grounding determines whether the agent can interact with the correct interface components, directly impacting its ability to complete downstream tasks. In fact, a grounding-ond agent can be paired with a general-purpose planner (e.g., GPT-4o (Hurst al. 2024g) complete tasks via a modular style.
To support various grounding demands, we construct a multi-format GUI grounding corpus with three distinct supervision targets: point grounding, bounding box grounding, and action grounding. Point grounding requires the model to identify single pixel-level location, typically the center of a button, icon, or control, that corresponds capability by predicting rectangular regions that encapsulate target elements, which is particularly useful for operations involving region selection, such as dragging or editing. Action grounding combines spatial localization with operational semantics by producing an executable command, such as cl icck (x=105, y=23), that aligns with the intended interaction. As for the annotation, we reuse structured annotations generated during the GUI understanding stage. Specifically, appearance,
user instruction. Bounding box grounding extends this
$ "htts:/wwww.selenimm.. dev/
https:/wwww.googlcccom www.c/oocogcccccccoomeee http://gggaaua.omo/miimmonno/lln
30

---

Published as a conference paper at ICLR 2026
spatial, and functional descriptions of each UI element provide rich supervision signals. The center point and bounding box coordinates are extracted directly from UI layout metadata or visual parsing modules. Action-level grounding pairs these spatial targets with predefined atomic operations based on the element's s inferred function. In addition, we explore data augmentation strategies to expand the grounding corpus. Specifically, we filter out previously annotated elements from the metadata and use prompt templates combined with GPT-4o to generate larger set of grounding annotations. This augmented data designed to improve the model's generalization ability across diverse GUI layouts and interaction patterns. This annotated corpus serves as a foundation for learning robust visual-Iin linguussi alignment and facilitates both direct interaction and integration with high-level task planners.
A.6.5 WEAK-SEMANTIC TRAJECTORY
While the trajectories collected by rule-based agents do not correspond to explicit task objectives, incorporate heuristics into the exploration process encourage transitions into deeper and less frequently visited interface states. This results more diverse and representative interaction sequences, which are critical for training agents to generalize across complex GUI structures.
To further exploit the potential of these unsupervised trajectories, we segment long interaction sequences into shorter, weakly semantic trajttraeccssiss. The segmentation is based screenshot similarity: when a current screen is visually similar to a previous one, it often indicates that the agent has reached terminal redundant interface state with minimal novelty in further interactions. These limilarity-basee boundaries serve as natural points for restarting exploration, thereby improving coverage and trajectory diversity.
We refer to the resulting sequences as weak-semantic trajectories, as they preserve partial continuity and structural coherence without being aligned manually defined tasks. Despite their lack of strong supervision, such trajectories often reflect meaningful UI flows, especially when the agent is biased toward newly rendered elements.
We hypothesize that exposure to veak-semantic trajectories can help the agent internalize common patterns of GUT interaction and enhance its planning ability. If validated, this approach may offer a cost-effective alternative to large-scale manual annotation, accelerating the evolution of more capable computer use agents through low-cost, high-coverage exploration.
A.6.6 HUMAN-CURATED TRAJECTORY
In addition to rule-driven exploration, we incorporate human-curated trajectories to address the limitations of automatically collected data. While rule-driven agents enable scalable collection, they inherently exhibit stochasticity and often fail to uncover certain goal-directed operations, especially for tasks requiring deep or context-specific interactions. Moreover, although weak-semantic trajectories segmented from raw explorations provide partial structure, their action sequences aligned with human reasoning. As result, they may contain fragmented or noisy behaviors that limit their utility for downstream training.
always
To overcome these limitations, we design a saman--n-the-loog protocol for collecting high-qualin task trajectories. We begin by constructing a seed task set, categorizing applications into common use domains such as daily utilities, entertainment, and productivity. For each domain, we identify representative applications and select frequently used functions based on user documentation and empirical analysis. Annotators are then instructed to convert these functions into clear, goal-oriented task descriptions, ensuring linguistic clarity and operational feasibility. Using our unified cross- platform recording system, human experts remotely interact with each application environment encapsulated within a Docker container. This design provides process isolation, avoids side effects such misoperation. Annotators are able to finish tasks in natural and fluent manner, producing coherent action trajectories that reflect realistic usage patterns across platforms.
These curated trajectories serve high-quality supervision for training agents with accurate planning and execution capabilities. They complement the broader, noisier dataset collected via automation, and provide reference paths that guide model alignment with human intent and behavior.
31

---

Published as a conference paper at ICLR 2026
A.6.7 ANNOTATION SCHEMES
In our data acquisition, collect screenshots along with their metadata, which includes all potentially interactive elements on the page. Since different exploration paths can lead to the same state and common states like the homepage are visited frequently, we employed image feature similarity to deduplicate these screenshots. This yields a unique set of interface screenshots paired with their corresponding metadata. To reduce redundancy and mitigate noise within the metadata, we randomly sample 25 to 40 elements per screenshot. These elements are then semantically filtered using GPT-4o ensure both efficacy and diversity.
For each retained element, we mark its position on the image using a red box with an arrow. By combining with associated metadata, we prompt GPT-4o to generate appearance and position descrip- tions, and -.nne-3.7-Sonnet [Anthropic, 2025) to generate functional descriptions. These serve as ground truth annotations for our Element Appearance Captioning. Element Layout Understanding, and Element Functionality Captioning, respectively. These descriptions are further used to construct grounding tasks, where the appearance and position descriptions are used for non-action grounding and the function description is used for action-based grounding. To simulate all possible positions of elements and accommodate a wider range of usage scenarios, we perform data augmentation. This includes simulating higher resolutions by stitching two images together. well as cropping elements and pasting them onto solid-color backgrounds or real- world backgrounds from images captured by the author's s own device.
For each unique interface screenshot, GPT-4o is also used to generate an overall caption. If the image was not the final step of trajectory, additionally provided GPT-4o with the subsequent screenshot along the same exploration path to summarize the UI changes and infer the intention. These are used for Screen Transition Captioning and User Intention Prediction tasks.
For all trajectories, we provide Claude-3.7-Some with the current and next screenshots, as well as a cropped image of the interacted element, to infer both the step-level instruction and the reasoning process. For weakly semantic trajectories that primarily involve navigation across pages, we generate high-level task objectives. To do this, we provide Claude-3.7-Some of the trajectory to synthesize navigation-related task goal. Considering that different annotators have varying styles of writing instructions and different operational habits, we implement two types of augmentations for trajectories to improve model generalization. The first is instruction augmentation, where we prompt the model to generate task instructions diverse styles, aiming to cover all possible user scenarios. The second is trajectory augmentation, for which prompt the model to generate several step-level instructions and the reasoning process based on the trajectory. This can help mitigate the noise introduced by model labeling. All prompts used for annotation provided in the Appendix A.. 10.2
with the first and last screenshots
A.6.8 MORE DETAILS OF DATA DISTRIBUTION
In Fig. we visualize the data distribution for each task domain. Fig. [lb provides hierarchical view of the trajectory composition across platforms and types. By integrating agent-generated and expert-curated signals, we ensure both data diversity and quality. Our ScaleCUA-Data delivers the largest GUI grounding dataset to date, coupled with substantial understanding and planning examples. Its platform coverage and hierarchical task composition form a comprehensive foundation for training robust, cross-platform GUI agents. The performance of ScaleCUA validates the quality of ScaleCUA-Des -Dtaa and highlights future directions in data-centric agent training.
A.7 DATA VISUALIZATION
A.7.1 GUI UNDERSTANDING
To qualitatively demonstrate our data in GUI understanding tasks, we provide examples that cover both element-level and screenshot-level understanding. At the element level, we have designed five distinct tasks regarding individual GUI elements. Table 17 showcases specific examples of these tasks. At the screenshot level, we focus on the ability to comprehend the entire GUI interface globally and its dynamic changes. Table 18 provides examples for these two tasks.
32

---

Published as a conference paper at ICLR 2026
Table 16: Distribution of examples in our training corpus.
Task Domain
Tasks
#lmages
#Examples
Element Appearance Captioning, Referring OCR,
Flement Layout Understand,
Understanding
355.5K
471.4K
Flement Functionality Captioning. User Intention Prediction, Interface Captioning,
Sereen Transition Captioning
Bounding Box
Grounding
1.6M
17.1M
Point, Action
Weak Semantics Trajectories
5.5K
15.0K
Task Planning
Human-Curated Trajectorics
29.3K
4.0K
Enhanced Trajectorics
29.3K
48.2K
100. 280
Www (.0 PRe
INCY INCYO Reggemso
decunsing 1St to0
Nomue 17.20
Saeces
Coulr Hel
Planiing 20-
Web (34 nd
dlsan to seruan
Linux (35, 49%)
Eneanced (82-) 1N
Syt thi
ISLL postesus
(SLer (ae
Hur lamaany
(23 (8%)
two ag soan
the Begronsn
ART, (da
(a) Task domain distribution of dataset.
(b) The composition of Trajectories.
Figure 11: Data distribution of our dataset.
A.7.2 GUI GROUNDING
To enable precise interaction, a model must be able to map natural language instructions to specific elements interface. As mentioned Section 4.2 support three distinct GUI grounding modes to accommodate a variety of scenarios and Table 19 provides visual illustration of these three modes.
A.7.3 WEAK-SEMANTEE SEMANTIC TRAJECTORY
As mentioned in Sec. A.6.5] the weak- semantic trajectory processing pipeline begins with rule-driven agents performing depth-first exploration across GUI environments. These agents generate extensive interaction sequences by randomly selecting actions from available options at each step, while heuristic pruning eliminates redundant branches to improve efficiency. Although these trajectories lack explicit high-level goals, they provide broad coverage of interface states. To enhance the utility of these unsupervised trajectories, segment them into coherent sub-sequences based on visual similarity between screenshots.
Fig. 12 demonstrates the exploration pattern in desktop environments, while Fig. 1 and 14 show
the equivalent processes for mobile and web platforms, respectively. In each case, the random-walk agent generates comprehensive navigation patterns through systematic exploration of the interface.
33

---

Published as a conference paper at ICLR 2026
Table 17: Task visualization for element-level understanding.
Examples of Five Types of Tasks.
[Element Appearance Captioning]
Q: Please describe the appearance of the element marked in the image.
A: A rectangular search bar with white/light gray background and placeholder text 'Search with Google or enter address' in light gray font. The search bar has smooth, rounded corners and a subtle border. It appears be empty with no entered text.
[Referring OCR]
Q: Please output the text content correctly responding to this term: <box>>[[1, 563, 435, 651]]<</ox>> A: A rectangular search bar with white/light gray background and placeholder text 'Search with Google enter address' in light gray font. The search bar has smooth, rounded corners and a subtle border. appears be empty with entered text.
Buy Apple Watch Ultra 2
Faish
[Element Layout Understanding]
Q: Describe the position of the highlighted elements in the image, such their location relative other objects, alignment, and any spatial relationships. A: Located the top toolbar the application win- dow, positioned between the interface navigation con- trols on the left and the viewport controls on the right. It sits approximately in the upper third of the sereen, within the main toolbar area.
[Element Functionality Captioning]
Describe the position of the highlighted elements in the image, such as their location relative to other objects, aligmment, and any spatial relationships. A: Use the size grip at the corner of the PowerPoint Options window to change its size
Hootth
H
[User Intention Prediction]
Q: Analyze the current image and the provided action "click(x=540, y=1686)', then predict the user's intent based these inputs
A: The user likely intends to view detailed informa- tion complete list of their past workout activi- ties, specifically the "Laaees:: Karate Power Fat Burn" session recorded on 8/25/2024 at 20:27, the label "Show all" suggests an expansion or summary of his- torical data.
These weak-semantic trajectories valuable training signals despite their lack of strong task aligument. They expose models to common GUI interaction patterns and state transitions, complementing the more structured but limited human-curated data.
34

---

Published as a conference paper at ICLR 2026
Table 18: Task visualization for screenshot-level understanding.
Examples of Two Types of Tasks.
[Interface Captioning]
Q: Provide detailed description in the current image.
A:The image shows a weather application interface displaying the forecast for Shanghai. The current temperature is 6 (about 199C) with sunny conditions. The interface shows a detailed forecast for Friday, April 2025, with high 6 and a low of 48 F. The hourly forecast indicates temperatures will range from F 50'F throughout the day, with sunset occurring at 6: 14 PM. Wind conditions show a speed of 3 MPH with gusts up 11 MPH. The interface also displays additional weather information including visibility (14 mi), UV index (7 High), and five-day forecast showing temper- atures for upcoming days. The air quality map shows green areas indicating good air quality around the user's 's location.
[Screen Transition Captioning]
Q: Analyze the differences between two consecutive GUT screenshots. Describe the initial state, highlight the changes in the second screenshot.
A:In the first screenshot, we see the Trip.com mobile app homepage with various booking options including Hotels, Flights, Flight + Hotel, Trains, and other services. After tapping on Flight + Hotel option, the second screenshot shows the Flight + Hotel booking page where users can input their travel details. The page displays one-way/return toggle (with 'Return' selected), origin and destination fields set to Hong Kong and Dubai respectively, travel dates (Wed, Apr 23 Sat, Apr 26 for 3 nights), traveler count settings, and a search button at the bottom.
Summarization: aim to set language to English (USA) then return to my.
Figure 12: An example of a weak semantic trajectory on the Ubuntu platform.
Summarization: Forward address from the File Transfer chat in WeChat John.
90
62
Figure 13: An example of weak semantic trajectory on the Android platform.
35

---

Published as a conference paper at ICLR 2026
Table 19: Task visualization for GUI grounding
Examples of Three Modes.
[Point Grounding]
Q: Return the point within this UI element: <tef>Preformatted Text button in the EDITOR tab formatting toolbar that allows users insert pre-formatted text tags in MATLABB's editor.</reht
A: Preformmmte Text button in the EDITOR tab's formatting toolbar that allows users to insert pre-formatted text tags in MATLAB's cdior:<<het><<pont>(2222
4/Jon/point>>
[Bbox Grounding]
Q: Indicate the location with a bounding box to this element: <TreD>A white-faced analog clock with black nu- merals (1-12) and three hands, placed in the upper left corner. <ref>
A: <ref:> white-faced analog clock with black numer- als (1-12) and three hands, placed in the upper left cor- ner. ner.<ref>> [[97, 69, 218, 227]]<<bbox>>>
[Action Grounding] Q: Click the 'Open' option to open the selected file <action>>click(x00. y=0.4967) </action>>
BSCKRDOR 20R
Summarization: Try reading the Advanced volume of On Java Chinese version.
Figure 14: An example of a weak semantic trajectory on the web platform.
A.7.4 HUMAN-CURATED TRAJECTORY
Fig. 15 19
illustrate human-curated trajectories across five platforms: Windows, Ubuntu, macOS, Android and Web. Each trajectory demonstrates precise
interactions, rendered as mouse / gesture traces over consecutive screenshots, forming high-quality demonstrations for data collection. These trajectories span diverse applications such as Excel, SolidWorks, Gmail, Numbers, Amap, Twitter/X, and GitHub, showcasing real-world complexity in cross-platfonn environments. The visualizations highlight platform-specine GUI logic (e.g., desktop file operations vs. mobile touch navigation), as well long-horizon reasoning steps (e.g., multi-page exploration, search-before-ee workflows).
A.7.5 TRAJECTORY ANNOTATION
Building upon the annotation schemes detailed in Sec. A.6.7 we systematically process tra- jectory data to generate high-quality training corpora. Our trajectory annotation focuses two key aspects: (1) low-level operational instructions generated for each interaction step, (2) thain-of-thought rationales explaining the decision process. As demonstrated in Ta-
36

---

Published as a conference paper at ICLR 2026
Task: Visualize the 2024 business performance metrics using clustered column chart in Excel.
Task: Create 3D cylinder on the top reference plane in SolidWorks.
Figure 15: Examples of human-curated trajectories the Windows platform.
Task: Make copy of sheell and name it as "sheetl(copy)", positioning it after all existing sheets.
Task: Write email guiagent@gmail.com
The subject is OpenCUA for GUTAgent.
Figure 16: Examples of human-curated trajectories the Ubuntu platform.
Task: Use conditional formatting highlight values greater than 20 in Numbers.
B30 Ro BD BO
0F
88
B30 KOD
B0 DE
0 80
120kagDrBa BD 12
Task: Add high-priority reminder "Paper Reading" due tomorrow PM.
Ba0 -o- BOncears DF BB
BEOK BO
CR 80
130m Tea: 40.008 cm
Figure 17: Examples of numan-curatee trajectories on the macOS platform.
ble 20 these annotations are formally represented using XML tags to distinguish between operational instructions (coperations watprr...///aapii..e.e..e </pperation>) and their cognitive justification
37

---

Published as a conference paper at ICLR 2026
Task: Search "ChatGPT" in X and like the first tweet in the results.
Task: In Amap, navigate the route from Shanghai Jiao Tong University Xuhui Campus to the Bund.
0
Toor
Figure 18: Examples of trajectory data collection on the Android platform.
Task: Check the public list of members of the "google" organization.
which games top the popular games list.
Figure 19: Examples of trajectory data collection on the web platform.
(<think>.. </think>), which support both direct-action execution and reasoned-action paradigms.
A.8 LESSONS FROM DATA ACQUISITION
A.8.1 COMMON PROBLEMS
Adaptive Bounding Box Tightening(ABT). The heterogeneity across platforms results in significant discrepancies in UI interaction logic and responsive regions. Therefore, we aim to ensure that the bounding boxes of UI elements are as close as possible to their central regions, in order to minimize inconsistencies in actionable areas across different platforms. As illustrated at the top of Fig. 20, these parsed bounding boxes are aligned with the boundaries of the UI elements; however, their centers, which serve the ground truth for click prediction, often reside in empty regions rather than in areas with meaningful content. This misalignment introduces a detrimental click-w whitespaee bias during model training. To mitigate this issue, we propose the Adaptive Bounding Box Tightening (ABT)
38

---

Published as a conference paper at ICLR 2026
Table 20: Visualization of trajectory annotation
Examples of Two Modes.
[Reasoned-Action Mode]
Q: Please generate the next move according to the UI screenshot, task and previous operations.
Task:
Search openai and like the first video, sorted by view count and video type.
Previous operations:
Step I: Click the search box at the top of the YouTube page and type "openai"
Step 2: Search for OpenAl on YouTube and press Enter
to see search results and the OpenAl channel page A: <think>
The YouTube interface shows a search for "openai" with filters button visible in the top navigation
Clicking on the filters option would allow sorting videos by criteria such as view count, which is needed to complete the task of finding the most viewed OpenAl videos. This filtering functionality is essential to organize search results way that aligns with the requirement to sort by view count and video type before liking the first video.
</think>
<operation>>
Click the "Filters" button at the top right of the YouTube search results to access advanced filtering op- tions for your search.
</operation>>
<action>>
click(x=0.9043, y=0.0788)
</action>
[Direct-Action Mode]
Q: Please generate the next move according to the UI screenshot, task and previous operations.
Task:
In the Calendar app. show only events in the 'School' calendar.
Previous operations:
None
A: <operation>>>>
Click the blue checkbox next "Family" in the calen- dar sidebar under "On My Mac" section.
B0 DB ACCE 0I
</operation>>
<action>>
</action>>
algorithm. ABT dynamically refines bounding boxes through iterative, proportional contraction of their lateral boundaries. This contraction is guided by the detection of contiguous uniform-color regions, representing target whitespace, within defined tolerance thresholds. The process terminates when no significant contractible uniform regions remain, yielding substantially tighter bounding boxes where centers align with actual content, as shown the bottom Fig. 20 While ABT's effectiveness is inherently limited by complex backgrounds and gradients, empirical validation confirms its significant value improving ground truth alignment for interfaces featuring simple solid-color backgrounds. This paradigm remains dominant in modern systems and web design.
39

---

Published as a conference paper at ICLR 2026
Deep Exploration. Modeling GUI platform state transition graphs presents inherent complexity. Random walks, a common approach, suffer from limitations: unpredictable transitions induce pervasive back edges, causing frequent state revisitation or trapping in local loops due to insufficient backtracking mechanisms. To address these issues and enable automated deep exploration for acquiring meaningful weakly semantic trajectories, we propose a filtering algorithm. Specifically, we use a queue to maintain all interactive elements appearing in the last screenshot. At each exploration step during random walk, some of elements are filtered out when their Intersection over Union (IoU) exceeds predefined threshold and their textual content exactly matches any element in the queue. This guarantees exclusive interaction with elements absent in the preceding state, thereby actively steering exploration toward novel pages. This mechanism proves particularly effective for interfaces with persistent components (e.g., navigation bars, sidebars) or dense icon arrays, as evidenced in Fig. [T]]eee it achieves significantly broader page coverage and yields non-redundant, semantically valuable trajectories compared to conventional random walks.
history--faane element
TANITER
Plearr Evss
Plcce
plud
20001
niHoto
Fatirrstim
MaRo muRe WT
Prezani 2n1 Rcoastt FUBT: marraasssomae FR
M-RIANEALL
Figure 20: Examples of adaptive bounding box tightening(ABT) algorithm.
A.8.2 WINDOWS
Fross-Framework UI Parsing Challenges and Denoising Strategies. When processing Jave- software like PyCharm and Android Studio, the standard Win32 API exposes significant limitations. As illustrated in Fig. [22] the Win32 API fails effectively parse their UI structure, resulting in an incomplete A lyy Tree. Consequenty, we must switch to using the specialized Java Access Bridge (JAE
RM API. The JAB successfully retrieves the complete Ally Tree (as shown in Fig. 23 thus resolving the issue. This requirement to adapt different APIs for various application frameworks significantly increases the complexity of our data collection efforts. Moreover, the raw Ally lly Trees
40

---

Published as a conference paper at ICLR 2026
Macllook
iPad
iPhone
MacBoook Air
Flight speed
iPad
Figure 21: Examples of deep exploration algorithm.
present challenges: they are typically deeply nested, noisy, and the density of functionally relevant Ul elements low. To address these issues and improve data quality, we apply of heuristic filters to prune and refine the tree. 24 visualizes this transformation, showing comparison of the A Ally Tree before and after processing- Our filtering strategies exclude elements from background applications and select elements by their screen-to-ares ratio, roles (e.g., button, text, hyperlink).
Data Deduplication and Geometric Refinement. Data acquisition in Windows faces several significant data quality challenges. First, minimal UI changes following user interactions lead to high redundancy of UI elements due to nearly identical screen captures. Second, lack of layer information in the Ally Trees results in erroneous inclusion of occluded elements (e.g., droopdown).T overcome these challenges, we implemented multi-stage refinement pipeline. We first mitigate redundancy with similarity algorithm that filters images based on the Euclidean distance of their feature vectors. post-processing filter then identifies occluded elements by detecting solid-color regions within their bounding boxes.
Prioritized Random Walk for Automated UI Exploration. The random walk algorithm is central to our automated data acquisition on the Windows platform. To minimize redundant interactions and enhance element diversity, we have augmented the standard Random Walk with principles from Depth-Firs' Search (DFS). As mentioned in the above common problems, modified algorithm prioritizes interaction with newly appeared UI elements while concurrently reducing the selection priority of elements that have already been interacted with. If no elements detected, or if their count falls below predefined threshold, the algorithm defaults to interacting with any remaining, previously unvisited elements within the current view's A Ally Trees. Furthermore, we account for scenarios where interactions navigate away from the primary application, such as launching web browser to view a user manual. In such cases, our algorithm allows for limited interaction within the external application (e.g., the browser) before automatically shifting focus back to continue navigating the initial application.
A.8.3 UBUNTU
This section details the challenges encountered and solutions developed for autonomous agent interaction with the Ubuntu environment. The primary challenges originate from the inherent structure of the accessibility with the application. Our solutions focus on refining the accessibility tree data and optimizing the agent's interaction strategy to ensure reliable and efficient operation. The successful resolution of these issues is paramount, as clean, accurate, and efficiently navigable UI representation is the foundation for any effective automated UT-based task.
(A lly tree), which serves the main interface for observing and interacting
Denoising in the Accessibility Tree. The raw data provided by the accessibility tree on Ubuntu often noisy, containing redundant information and occasional inaccuracies that can mislead an autonomous agent. We identified and implemented solutions for three primary issues. First, the A1 Itree's 's hierarchical structure often includes redundant parent elements that do not correspond to distinct interactive components. This is particularly prevalent in applications built with Web, such as Chrome. To address this, we apply two-stage filtering process. We begin by pruning elements whose roles typically non-interactive structural based type, such 'heading',
41

---

Published as a conference paper at ICLR 2026
Figure 22: An example of Win32 API failing to parse Ally Trees in PyCharm.
CERIENN
fettome codeeet Action
Logellt sifers
a IB
Figure 23: An example of JAB API successfully parsing Ally Trees in PyCharm.
'paragraph', and 'section'. This denoising critical because exposes the true, underlying interactive elements, preventing the agent from attempting interact with large, non-interactive container widgets. Subsequently, we analyze the geometric relationships of the remaining elements' bounding boxes. If an element's bounding box is significantly occluded by smaller (i.e., the smaller box's area occupies a large percentage of the larger box's area), we infer container-child relationship and discard the larger, containing element. Second, the standard accessibility tree does not inherently account for the visual occlusion and invalidity of elements. An element may be present in the tree but be completely obscured by another element on the screen as shown in Fig. 25 We tackle this with L.Mss. Third, we observed that for certain applications, the accessibility tree reports incorrect coordinates for
elements immediately after the application is launched, as shown in Fig. 26 The entire tree appears to have coordinate offset. Through empirical testing, we discovered practical solution: initiating short sequence of random interactions within the application window causes the accessibility tree's coordinate system to recalibrate, restoring correct positional data. Ensuring coordinate accuracy is fundamental; without it, any attempt by the agent to click or type at a specific location would fail, rendering automation impossible.
42

---

Published as a conference paper at ICLR 2026
Neural Network
Neural networks are set of algorithms that mimic the function human brain. They are designed to recognize patterns, classify data, and make predictions.
Neural Network
Neural networks set of algorithms that mimic the function of a human brain. They are designed to recognize patterns, classify data, and make predictions.
Figure 24: An example of denoising on Windows, transforming raw Ally Tree (top) into clear structure (down).
GUI Exploration Optimization via an Improved Random Walk. A pure random walk over all available UI elements is highly inefficient. To improve the agent's ability to explore application's 's state space, we developed a more intelligent interaction strategy. This strategy is based on filtering the action space and prioritizing the exploration of novel UI states. To reduce the number of futile actions, the agent's 's action space is constrained to only include elements that are designated as interactive. We maintain a whitelist of interactive type, including 'button', 'box', 'menu', 'entry', 'link', 'bar', and 'item'. Conversely, elements with non-interactive roles are excluded from the potential action set. These non-interactive roles include 'heading'. 'static", 'document', 'label', 'cell', 'text', 'icon', 'paragraph', and 'section'. To prevent the agent from becoming trapped in interaction loops within a static UI state, we implemented a state-aware exploration logic. After the agent acts, we only visit newly appeared UI elements. These novel elements given interaction priority, they are most likely to lead to a new application state. the action does not yield any new elements, the agent then selects action randomly from the set of previously known elements that it has not yet interacted with in the current state. This process continues until all interactive elements have been exhausted. This exploration strategy vital for efficiency, it directs the agent towards discovering
43

---

Published as a conference paper at ICLR 2026
new functionalities and application states, thereby maximizing the coverage of the application features limited time and avoiding redundant, non produuudeeeive interactions.
Erlanl
Figure 25: Examples of visual occlusion and invalidity of elements.
softrok
8011
312 barnd
Srea entig Cach
Figure 26: Examples of coordinate offset.
A.8.4 MACOS
Robust Ally Tree Extraction and Denoising, The macOS pipeline first locates the active top-level window, then exhaustively traverses its accessibility hierarchy. Every bounding box is mapped from logical coordinates device pixels by multiplying by the screen-scale factor. After flattening the tree, only nodes whose roles are interactive (e.g. AXButton, AXPopUpRutton, AXText i eld) are retained. Moreover, we would discard boxes with width or height 2px less and remove nodes whose text, description and value are all empty punctuation. A role-aware merging process replaces overlapping AXStaticText siblings and their interactive parent with minimal bounding box. The resulting contains clean, tightly localised interactive elements. (see Fig.
Hybrid Ally Tree & Omniparser combination for System Panels. Several built-in utilities, most notably System Settings, draw controls in private layers that have corresponding accessibility tree Yu et al. (2025), as shown in Fig. 28 To recover these missing widgets, each screenshot is processed by Omniparser, yielding set of vision-detected bounding boxes. An element would be
44

---

Published as a conference paper at ICLR 2026
retained when its IoU with any Omniparser box exceeds 0.15 when it is selected during exploration. This combination renders previously invisible elements in the Ally tree, thereby yielding a more comprchensive understanding of macOS applications.
2la Lusedaa head comprtatte
Reuters
One Essential Read
mattrearddd Sdien Cales Nowaleter tokor matom
Mark Carney talks tough on Trump ahead Canada election
Figure 27: Refined AXTree overlay on the Mail application: all interactive elements are tightly bound after heuristic pruning.
Wi-Fi
VFU
Frevdl
Obter Services
Glets-20 Ann
Sisting
<
Figure 28:
The failure case in System Settings: the AXTree omits eight-pane controls, illustrating the necessity of Omniparser fusion.
A.8.5 MOBILE
Occlusion and Invisibility Correction. Mobile interfaces frequently employ high-level components such as dialogs, side drawers, and floating menus. These are rendered at the topmost Z-order, so underlying nodes remain in the XML yet Fig(29a). To improve visibility and hierarchy accuracy at the source, we replace the traditional adb shell 1. uiautcmator dump with ui aut.mmtor2.22.. wwwpppiweprrrhhhheceeooeee (). The latter
longer be clicked, producing "ghost" targets(see
45

---

Published as a conference paper at ICLR 2026
prunes recognisably occluded nodes while generating the XML and, for pages that adb fails to parse, still returns a complete hierarchy-SSigniffantt increasing data coverage. Coupled with the random-wall heuristic that "prioritises newly appeared elements." this greatly reduces mis-clicks a caused by occlusion. In addition, AlAuomator2 markedly lowers the probability of XMI. retrieval failures, accelerating exploration efficiency.
Attribute Completion and Correction. Many commercial apps do not fully propagate accessibility traits in their custom views; typical pattern is a parent node with cl ckacle-truue while all its children are cl i ckablellffff false, leading to the issue shown in Fig 29b Genuine clickable regions are thus ignored. We employ an "inherit- tethherrtthen-uuppressstt every descendant is marked non-clickable, the clickable flag is inherited downward; if any descendant already declared clickable, inheritance stops to avoid creating false hotspots. Experiments show that this method restores the vast majority of missing attributes while maintaining low false-positive rate,
strategy: when a parent is clickable and
Semantic and Functional Ambiguities. Semantic ambiguity arises when an XML bounding box is too large and covers multiple sub-controls (for example, the playback button, author area and more-options button), making a single node unable to convey precise meaning. In Fig 29c the green box shows one clickable bounding-bon -ox region in the XML, but taps in different parts of that region may produce different results, creating semantic ambiguity. To address this, we prioritise Icaf nodes and tighten their bounding boxes; we only retain a parent node when its centre lies outside every child's bounds, thus preserving the overall intent of the composite control. Functional ambiguity occurs when the same layout triggers different actions in different software or operating systems. In Fig and Fig [9e] for example, both the text and the icon of switch are tappable in stock Android settings, whereas in iOS only the icon responds to taps and the rest of the region is inert. To reduce such mispredictions, whenever we detect an "icon + text" sibling pattern we give the icon higher click priority. This approach produces more consistent cross-device behaviour during training and testing. By systematically handling overlay occlusion, attribute omissions and both semantic and functional ambiguities, we significantly improve the reliability of mobile-side data collection and increase the success rate of downstream automation tasks.
Soar
Centing
MLAN
Cot
Uncharted
Internet
MUNA
Offer
Networ canfurer
Sanadrattviio
usage
Are youaure you to deabe your wetzatss?
Augo Us WLAN Coluurr
moi dndp ablua Diice
spoitl mie and con Wor
Auk Jen Mileedyy
A Caltals
(a) Mobile element occlusion
(b) Mobile element attribute loss
(c) Semantic ambi- guity
(d) Effective bound-
(e) Effective bound- ing box for setting WiFi in iOS
ing box for setting
WiFi in Android
Figure 29: Examples of potential challenges in mobile data acquisition:(a) The problem of occluded elements being indistinguishable during XML extraction. (b) The potential inaccuracy of extracted bounding boxes due loss of element attributes.(c) The problem of semantic ambiguity caused by insufficiently detailed XML. extraction.(d. e) Differences in the functionality of similar regions across different systems apps.
A.8.6 WEB
Addressing Limitations in Automation Tools. Automation tools like Automation tools like Selenium and Playwright suffer from critical limitation where their page. screenshot function fails
46

---

Published as a conference paper at ICLR 2026
Playwright Origin Screenshot
Playwright Modified Screenshot
Operating System Screenshot
Figure 30: Examples of native browser UI limitations in automation tools.
to capture native browser UI components rendered outside the DOM. This omission disrupts essential visual feedback for sequential decision-making in web agents. We categorize these problematic elements into two classes: predictable UI triggered by deliberate actions (e.g., context menus, tab navigation, forward/back buttons), and unpredictable UI emerging during tasks (browser dialogs and native select dropdowns). The inherent invisibility of predictable UI components prevents agents from developing interaction intentions for these features; while our methodological constraint limiting interactions to left-clicks effectively eliminates potential negative impacts from this omission. However, to compensate for the unavoidably reducing behavioral diversity in captured data and ensure comprehensive functional coverage, we conducted extensive web data collection in native desktop environments, enriching our training corpus with full-spectrum browser interaction examples. The unpredictable UI category proves more severe, evidenced in Fig. 30 (Playwright Origin vs. OS Screenshot), where missing elements prevent task completion and impact evaluation integrity. Our behavioral simulation solution addresses this: for select elements, JavaScript modifies the size attribute to visually expand options within the DOM, with event listeners reverting the state; for dialogs, interceptor captures properties, dismisses the native instance, and injects visually identical DOM-based replica with non-functional buttons. The efficacy of this approach is demonstrated in Fig. 30 30 (Modified Screenshot), which illustrates the successful visual simulation of both UI components. While other potential related issues may exist beyond our current observations, they have not manifested in our evaluation scenarios and thus remain outside the scope of our present investigation.
Metadata Advantages and Parsing Challenges. Web page content, structured through HTML and DOM trees, inherently provides rich metadata advantages over alternative platforms. JavaScript enables precise element positioning and hierarchical analysis that significantly exceeds capabilities in other contexts, enhancing metadata extraction efficiency as illustrated in Fig. 31d However, the heterogeneity of the web Srosystem-diverrs frontend frameworks, inconsistent development practices, and variable standards prevents comprehensive coverage by data collection algorithms. Two representative challenges emerge: First, as shown in Fig. 3la developers misapply attributes such rol role=button to non-interactive images, introducing semantic inconsistencies that cause parsing anomalies. Second, current algorithms exhibit deficiencies in hierarchical analysis and visibility detection, resulting in inadequate filtration of underlying or invisible elements as demonstrated in Fig. 31b Considering the substantial volume of extractable elements in web environments, we propose that maximizing the recall of valid interactive elements should be the primary objective across platforms. This position advocates for aggressive filtering strategies rather than conservative approaches that might inadvertently retain invalid elements. While this methodology may occasionally exclude some valid elements, the benefits of reducing noise in the dataset significantly outweigh the potential costs of missing a limited number of interactive elements.
It is particularly noteworthy that the technical limitations have not been explicitly addressed in the extant literature on WebAgent papers, despite their profound implications for agent functionality and evaluation methodology. We therefore advocate for increased attention to these considerations in future WebAgent research. Additionally, our analysis reveals that web environments lacking browser UI elements significantly constrain an agent's 's exploration capabilities in the absence of compensatory action mechanisms (e.g., returning to a previous page-a trivial operation when using browser's back button-may require complex navigation sequences or prove entirely infeasible
47

---

Published as a conference paper at ICLR 2026
within the constrained visual context available). Fortumately, the refined WebArena-Lite benchmark evaluation has been specifically designed to eliminate such problematic scenarios, thereby ensuring methodological integrity and evaluation reliability. Nevertheless, based on our findings, strongly recommend that future research prioritize the execution of web-based tasks within native desktop environments, which may necessitate the development of new benchmarks and the migration of existing benchmarks.
Temporal Synchronization in Dynamic Page States. The web platform exhibits substantial dynamism, frequently causing temporal discrepancies between page states during element parsing and screenshot capture. The non inn-inttantteouus nature of parsing further compounds this issue by permitting mid-process element state changes. A characteristic scenario involves the auto- auto-hiding behavior of video player control bars, illustrated in Fig. 31c Current mitigation strategies employ dual measures: Initially awaiting complete page stabilization, followed by proactively triggering state persistence for specific elements- suchh as maintaining video control visibility through cursor hovering. Nevertheless, managing dynamic content remains a core challenge in web data acquisition.
Leveraging Multi-Source Textual Semantics. Web elements contain rich semantic description layers extending far beyond basic textContent compared to other platforms. Functional icons often convey operational semantics through alt and title attributes, while accessibility-compphe sites provide enhanced descriptions via properties like ari aeiia-is paria-labell label. Systematically aggregating these multi-source textual features establishes strong semantic associations, furnishing comprehensive contextual grounding for model annotations and effectively suppressing annotation hallucinations.
A.9 THE DETAILS OF WEBARENA-LITE-VZ
Current web platform evaluation benchmarks can be categorized into two main types based on the website environment. The first type utilizes real websites for online evaluation, primarily derived from the offline evaluation work Mind2Web (Deng et al. (023). Examples include Mind2Web Live (Pan et al. 2024), with UI-TARS (Qin et al. (025) employing Web Voyager and Online-Mind2Web for web domain evaluation. The second type conducts evaluations on locally deployed websites, pioneered by WebArena (Zhou al. 2023), which leverages open-source website code and databases (Sun et al.) 2024a) to provide highly simulated and interactive local Docker deployment environments for five functionally diverse websites, including GitLab, map services, forums. online shopping, and content management platforms (CMS). WebArena has constructed over 800 web tasks, inspiring derivative evaluation frameworks such as VisualAgentBench (WehArena- Visual VisualWebArre
(Xue et al. 2025), and Web Voyager (He et al.) 2024).
(Liu et al. (024a) and
WebArena (Koh et al.] 2024). Furthermore, the evaluation protocols can be classified into two
categories: rule-based evaluation exemplified by WebArena (Zhou et al. 2023) and VLM-as-a-Judde evaluation, such as Online-Mind2 (Xue et al. 2025).
Rationale for Selecting Local Website Environments. We deliberately abandoned evaluation benchmarks based real websites for several compelling reasons. The primary concern is the temporal instability of online environments tasks that are currently feasible may become impossible due to website updates, domain changes, or site closures. Despite efforts by frameworks like Mind2Web-LLiv to maintain and update tasks periodically, such updates inevitably compromise
evaluation fairness. Additionally, as noted in (Xu et al. 22024). automated tools frequently encounter anti-automation barriers such as reCAPTCHA verification. Moreover, since most target websites are hosted in the United States, researchers in non-US regions (particularly China) face persistent connectivity issues and access restrictions even with VPN services ddif-renfteetttee VPN providers often yield inconsistent access results. These factors significantly undermine fair model comparison and hinder the extraction of valuable insights from evaluation results.
Deehrrea-Litee-2. Consequently, we focused on the WebArena series, whose locally deployed website environments offer substantial stability and internal accessibility, enabling flexible task construction and evaluation design. Considering that WebArena often includes three or more itera- tions of the same task template, resulting in repetitive and time-consuming evaluations, we selected the WebArena-Lii subset, which provides 165 high-quality refined tasks. However, our empiri- cal evaluation and manual inspection revealed persistent issues. Therefore, we further refined the benchmark to create WebArema-Lite-v comprising 154 tasks optimized for both headed browser environments and headless automation tool environments. Recent developments, such OpenAl's
48

---

Published as a conference paper at ICLR 2026
Operator, demonstrate transition from headless environments provided by automation tools toward headed desktop browser environments for web agent evaluation. As detailed in [A.8.6] both environ- ments present distinct advantages and limitations. To facilitate comprehensive ablation studies on these different operational modes, WehArrea-LLie--22 ensures that all tasks can be solved through at least one viable path using desktop action spaces (without specialized web actions like forward, go_ bockaaa open_url, or tab uu_sswitc) in both headed and headless environments. Furthermore, all tasks are designed to provide sufficient visual information guidance, eliminating the necessity for DOM information and thus making the benchmark suitable for pure vision-based evaluation (while remaining compatible with SoM or DOM-enhanced assessment). Finally, we implemented comprehensive yet flexible evaluation criteria- conprrieccm in accommodating multiple possible solutions through the [OR| operator where satisfying any one solution sufficient and flexible in employing LLM-based fuzzy_m macycymtchhhh for semantic similarity assessment in tasks involving question answering or content completion.
Discussions between WebArena-Lite and WebArena-Lite-v2. Our refinements encompass both environmental and task improvements. For the evaluation environment, we implemented two significant enhancements. First, we addressed the OpenStreetMap website's limitations, where the official Docker environment lacked local database storage for node information, rendering tasks like "What is the phone number of Western Pennsylvania Hospital" impossible complete. We resolved this by importing Pennsylvania state PBF data, enabling the completion of such tasks. Second, we developed consistent solutions for headless automation environments to overcome the observation challenges with select option dropdowns and dialog windows, as illustrated in [A.8.6] with Fig. 30 Regarding task refinement, we eliminated Il tasks requiring multi-tab interactions, resulting in a curated set of 154 tasks. We conducted a comprehensive revision of instructions and evaluation functions for all remaining tasks. The instruction refinements encompassed semantic clarification, typographical correction, and minimal reconstruction of impracticable directives (e.g.. the instruction "Re-post the image of the costume contest in this page to the funny subreddit and note "from /f/pies"" proved infcasible since headless environments lack image URL extraction capabilitics). Our evaluation function enhancements incorporated supplementary valid solutions (e.g., for the query "What is the zip code of Chatham University?", augmented the answer from exclusively "15232" to "15232 |OR| 15208" after identifying multiple Chatham University locations through OpenStreetMap queries) and accommodated semantically equivalent solution expressions (e.g.. for "Show me products under $100 in 'Men Shoes' category", we recognized shoes- both -SHOPPINGG/clotiiinii hewe htttee/ssseeeerrrrree///oooooss http:////shheeee...eeeeeeeeee htt.:htpiieee--1111000000 and valid pathways to identical content pages). This methodological approach ensures comprehensive answer validation. Additionally, acknowledging language models' inherent variability in textual response generation, we systematically replaced all exact_match evaluation criteria within the string -mattmmaatoo ch classification with more nuanced must.include, must_exclude, and fuzzy_mat ch parameters, thereby significantly enhancing evaluation robustness and interpretative flexibility. However, WebArenaLite--V still employs static evaluation methodologies for certain tasks (such as when identifying user's most recent order, where the Ground Truth is predetermined as specific order number webpage). Although executing evaluations within a local environment has mitigated the impact of this limitation, a critical future direction involves developing evaluation protocols that both dynamic and precise. This advancement necessitates addressing the challenge of extracting Ground Truth information from web pages that may not have been accessed by the agent during its navigation trajectory. This capability is essential for comprehensive evaluation of agent performance across diverse web interaction scenarios.
as
A. PROMPT ENGINEERING
To facilitate reproducibility and offer practical guidance for future research, we include all prompt templates utilized throughout our work in this section. These prompts cover a wide range of use cases, including data filtering, annotation, and the prompts used in our ScaleCUA. Specifically, we detail the instructions employed for GUI understanding, grounding supervision, and trajectory annotation, well those used to elicit reasoning traces and alternative actions. Each prompt is carefully crafted to align with the capabilities of large vision-langueee models such as GPT-4o and Claude-3.7, ensuring high-quality outputs for downstream training. By releasing these prompts, we
49

---

Published as a conference paper at ICLR 2026
aim to enhance transparency and support the development of more robust and interpretable computer
use agents.
A.10.1 PROMPTS FOR OUR AGENT
To ensure generalizable and controllable agent behavior, we design structured system prompt template for ScaleCUA that explicitly encodes the available action space, This template serves as the foundational context for all three inference paradigms- Grounding Mode, Direct-Action Mode, and Reasoned-Action Mode Mode-guiding -guiding the model to produce spatially grounded and semantically aligned outputs. The system prompt defines the operational semantics of each action type, including spatial commands such as cl iick (x, y), dragTo (x, y) and wri te (text). as well as higher-level control tokens like terminate and wait.
We envision the system prompt as modular and extensible interface. In future iterations, we aim decouple the action space definition from the core prompt logic, allowing for plug-and-play architecture that can dynamically adapt to the interaction paradigms of diverse computing platforms. This modularity would enable seamless integration of sevice-specific actions, such as swips pe for mobile interfaces or hotkey for desktop environments. while preserving consistency in agent behavior. Our design lays the foundation for building unified prompting framework that can scale to arbitrary GUI-based control systems.
System Prompt Template For Action Grounding Mode
are astonomous GUT agent capable operating daskt.ops, devi ces, web browsers, Your primary function is analyze sereen captures and perform appropriate UI actiens to complete assigned tasks.
44 Action Space click(
x: fleat None Nome, y: flest None None, clicks: int =,,
button: str "left",
*frclick on the screea at the specified coordinszes. The 'x" and "y" parameter specify where the mouse is uoed. The "clicks" paramater apecifies how many times to click, and the "buston' parameter specifies which mouse button to use ('lett', 'right', or 'middle").
occurs. provided, current mouse position
def daubleClick(
x: fleat None None, flost None Mone,
button: str "left" None:
un*gerfomm a double click. This is a wrapper function for click(x, Y, 2, "laft")."** "*n
pass
def rightClick (x: float None None, y: flost None None) None: ""*gerforms right button slick. This is wrapper function for click (x, Y, 1, 'right').
def moveTo float, y1 float) -> Nones "ove the mouse the specified coordinates,""
dof dragTo
x: float Mone None, yt float None None, button: str "left"
None:
unfgerforms a drag-to action with optional "x" and "y" coordinates and butten,
def swipet (
[zom_coord: suplelfloat, float] None None,
50

---

Published as a conference paper at ICLR 2026
to_coerd; tuplo[float, flost]
direction: str = "up",
amount: float 0.5,
Noze:
""*Forforms awipe action on the screen, -fro_coord,
"to_coord" spect ty the starting and ending coordInates of the swipe. If "Lo_coord" ie not provided, the "direction" and "mmount" paremeters aze used to deternine the swipe direction and distance. the "cirection"
be "up', 'dewn', 'left', 'zight', and the "amount" specifies how far to swipe relative to the acreen aize (C to 1).**" pass
def pree_press float, float, duzation: int
None:
"Long press on the screen at the spenified coordinates. rhe "duration" specifies how long hold the seconds. "*n paas
Input Specification Screenshos of the
task description
# Output Format <action>
action
</action>
+ Note
Avoid (s) thot would load to invalid states.
within the defined fon apace. The genezaled action (s) ahould be enclosed within factions//aclaae Lags.
generated actioon
System Prompt Template For Direct Action Mode
You are autonomous GUI agent operating the ** (PLATFORM) platform (s) Your pr primary function is to ana lyze screen captures and parform appropriace act to complete assigned Lasks.
Action Space def click
x:
None, float None None,
clicks: int 1,
button: str "left",
None:
specified coordina.es. specify where the mouse event occurs. If net provided, the curreat mouse position used. The "clicks" paramater specifies many times to click, "buston' parameter speci fies which mouse button to use I'left', 'right', or 'middlo"). pass
"rClicck
"y" parametes
def doublečlicki
x: fleat None None,
y: fleat
None,
button: str leet", None:
""*gerforms deuble click. This is wrapper function for click(x, ye 2, 'laft').
def rightClick (x: Float Rone None, Flost None None) -> Nones un*gerforms a right mouse button slick. This is wrapper function for click Yr 1, 'right'). *nn paas
seroll (elicka: ins, x: float Mone Nore, y: float Mone Nonel -> Nones on onrerroomm scroll of the mouse scroll wheel at the specified coordinates. The "licks" specifies hew many clicks to scroll. The cirection of the scroll (vertical horizontal) dapenés the underlying oparating aystem. Normally, positive va valuea scroll vp, and negat ive values scroll down." pase
51

---

Published as a conference paper at ICLR 2026
def movelo (x: Elost, y: flest)
"Hova the mousa to the specified coordinatas. paas
def dragTol
x: flest None None, y: Eloat Noue None, button: str "left"
"**Fer forms a drag-to action with apt lonal "x" and "y" coordi nates and button, pass
def press (keys: str [sttl presses: int >> None:
serfform a keyboard key press down, followed by release. The function supports pressing single key list keys, multipla pressen, and customizable intervals between pressea.**"
pass
def hotkey (vargs: -> None:
sorfooomms key down presses on the argumenta passed in order, then performs key relessea
order. used to simulate keyboard shortouts (e.g.) 10 Stiihhtfftt... pass
def keyDown (key: str)
Forfoorsm a keyboord key press without the relaase. This will put that key in held down *en
paas
keyUp (key: SCH) None:
"* "**eeroomm keyboard key relense (without the press down beforehand), *** paas
write (message: str) None: en erritt the specified text.""" paas
(all_ Nones "sccall the user. pass
def wait Iseconds: int 3) None: snrWait for the change to happen. pass
def reaponse (answer: stx) -> None: Ansswee a question oz provide a response to en user query. "En
def terminate istatus: str "success", info: stx Nane = None) -> None: inForrmiatt current task with a status. "stacus" specifies terminacion statua ('success", "failure"), and the "info" can provide additional information about the terminssion.
# Input Specification Screenshot the current screen task description your past interaction hiatory with UI finish assigned Lesks.
#t Output Format
<operstion>
[Next intended speration description]
</operation>
sactions [A of executable action commends] </action>
# Nete
(s) would states.
52

---

Published as a conference paper at ICLR 2026
The generated action within the defined action space. The generated operotion and action should he enalosed within ppprratooo<soppraation <action></action> tags, respectively.
System Prompt Template For Reasoned-Action Mode
aze astenomous agent operating (PLATEE0RM) platform. Your primazy function: is to analyze screen captures and perform appropriate ur actiors to completa assigned tasks.
# Act-ion Space click(
x: flcst None None,
y: fleat
None, clicks: int =
button: str "left",
""rclioks on the acreen at the specified coardinaces, The "x" and "y" parameter specily where the mouse occure. provided, curreat is used. The "clicks" parameter specifies how many times to click, and the 'buzton' mouse button to ('left', 'right', , 'middle'). **.
position
parameter specifies
def doubleClick( None, fleat None None,
x:
button: str *laft", None:
"""Berforms a double click. This is wrapper function for click yr 2, "left"),"*" paas
rightClick float None None, y: float None None) -> Nones """FerIorms right mouse button elick. This is wrapper sunction for click(x, 1, 'right'). ()
def saroll (clicks: ins, x: float None Nore, float Nona None] -> None: ""rForormm a scroll the mouse scroll wheel "elicks" specifies hew many clicks scroll. The direction of the scroll (vertical or horizontal) depends on the underlying opereting sysrem. Normelly, positive values scroll up, negalive values scroll down. na* pass
apecitied coordinates. The
def tx: float, y: float) >> Noné: the specified coordinates. "**
""*Move the
pass
dragTo
float
None Mone, float None None, button: str "lett"
None:
""*eerforms drag-to action with optional "x" and "y" coordinates and butten. n*n pass
press (keya: str Hist [str], presses: 1nt ->
onrerIoomm keyboard key press down, followed by release. The function supports pressing a single key or list of keys, multiple presses, and customizable intervals between presses. ens
pass
hot key (vargs: str! -> None:
onrger Torms key down presses the arguments passed in ordez, then pezforms key releases reverse order. This is used to simulaze keyboard shortcuts (e.g., 'cer--shift-'') **n
paas
def keyDown (key: stx) None:
53

---

Published as a conference paper at ICLR 2026
serfform keyboard key press without the release. This will put that key held down state. paas
def keyUp (key: Sie) None:
""Fperfomms a keyboard key release (Mithout the press down beferehand),
def write (message: stx) -> None: "nrwrite the specified text."..
def ca [usert) -> None:
"n "*rall the pass
def wast Issconds: int = 3) -> None:
en "*tt for the change happen. "** paas
respense (answer: str) -> None: "* "**aneer pass
provide response
query. nE8
(staaat str "success", "n"rerminata the current task with a status. The "atatus" specifies the termination status ('susceas', 'failure"), and the can prov'de additioral information about the termination. enx pass
None Nioe)
input Specification Screenshot of the current acreen task deseription your post interaction history with tirish aasigned taaks.
#4 Output Format
<think>
[Your reasoning process here;
</thinio>
coperation>
[Next intended speration description]
</operation>>
[A set of executable action command] </action>
+ Note
Avoid actions that would lead to iavalid atales. generated (s)
within the defined acsion space.
The reasoning process, operation and actior (s) in your response should be enclosed <thinio<</hhink>, soperationx</oppreee respectively
<action></actoon taga,
User Prompt Template For Direct-Action Mode and Reasoned-Aetir Mode
Please generate the next move according to the uI screenshot, the tosk and previows operations.
Task: [instruction]
Previous opecations:
(history)
54

---

Published as a conference paper at ICLR 2026
A.10.2 PROMPTS FOR ANNOTATIONS
To support reproducibility and transparency. release all annotation-related prompts used in our data processing pipeline. These prompts cover wide range of tasks, including trajectory filtering. GUI understanding, grounding supervision and chain-of-thought generation for goal-directed demonstrations. Each prompt is carefully designed to elicit accurate and semantically consistent annotations from large vision- vision-language models such as GPT-4o and Claude-3.7.
Empirically, our prompts have demonstrated strong effectiveness in producing high-quality labels, which in turn significantly benefit the training of general-pupost computer agents. By sharing these templates, we aim to standardize annotation practices this emerging domain and foster broader progress in building scalable and open computer systems. We hope this contributes lowering the barrier for future research and accelerating the development of robust, multimodal GUT agents.
Prompt For Element Appearance, Layout and Functionality
You aze a 3U1 analysis agent, and are currently working with (os_name) device. follewing resources:
previded 1. The first Image is original acreenshot from an (applicstion)-
2. The second image is narked to highlight the selected element.
actributes of seleated element: [elemen:_al1<<=e))
Your tack generate detailed descriptions and pasition. each description mast uniquely identify the element and achere to the fallowing structure:
marked element appearance
"apgearance"; decalled color, size, text content (if any), and any distinguishing features.", element's location the screea, including its relative position to nearby clements (e.g., 'balow the search bar', 'to the right the logo*), bar'i, and its general area (e.g., left cozner of the window"). Avold using direct coordinates oz the red indicator.",
description
including anape,
"positicn": clear description
sequence (e.g.r "third butson
top navigation
Guidelines Generating Lescriptions: appearancee+:
Focus on viaual characteriatice that uniquely identify the element.
Include details such as color, shape, size, text content (if applicable), icons, borders, shadews, or patterns.
element contains Lext, describe the font style, Flease avoid using (marker) part of your description. Because we drax (narker) reference they not exist the original screenshet.
and content briefly.
sPPoiiioow::
Describe the element's lecation seletive to other prominent elements in the UI that uniquely idenzify the element.
(e.g., 'top-right cornar', 'centar of screen') and its order in group (e.g., 'secoad icen in the tsclbar').
speci its general
Fleasa avoid using [marker) part your deseription. Because for reference
[marker)
they does not exist in the original sereenahet. Avoid vague terms like "near" or 'close te', Ins:ead, use precise laaguage such ddirecty belox", 'sligned with', 'to the left af'.
# Example Output:
"apgearance"; circular lcon with white background and magnilying glass aymbol black, surrounded by thin gray border.", "positien": "Located in the top-right corner of the application windew, directly to the right of the profile avata" 1con.",
44 Ingortant Notes:
not copy paraphrase the content of the AllTzee altributes directly. #lease aveid using (markez) as part of your descripsion. Because we drew (marker) for raference and they does Enaure each deacriptton is deta' led erough to uniquely denttff fy the element without. anbiguity.
in the original screenshot.
RETURN THE DICTIONARY STRICT JSON FORMAT:
55

---

Published as a conference paper at ICLR 2026
Prompt For Screen Transition Captioning and User Intention Prediction
You GU1 agent cusrently operating (es_name) device. You will be provided with:
first image screenshot an lapplicationl which are (narker) to high-ight the selested element.
second image la "esulta the operation (action] executed the selected eLement.
3. The third image a sub-image, which is cropped from the screenshot around the selected element and marked with (marker).
4. The AllTree actributes of the selected element: [alemant_alltrreh
Your tack is to analyze these two consacutive screenshots and complete the follawing tasks:
1. vxState Trensition Explanationva: Describe the sta.e change caused the operation.
This should include a detailed description of the first screeashot, the actien performed the element, the differences abserved in the second screenshot compared the rirst, and an explanation of the most likely waer oon that occurred betweer the Czames,
2. ++Jser Intention Inference*+: Rased the action performed and the di fferences between the two acreenshota, inter the user's intent. Explain what the user likely almed to achieve and how the action led to the observed changes in the GUI.
Your response should be formatted follews:
"state transition"; "...",
"user-intentionn: "...",
# Example Dutput:
"sLate-transit transitionn: "In the first sczeenshot, the main dashboard of the Bluecoins app displayed with calendaz showing February 2025, After tapping the "3', the second screenshot navigatea the to detailed calendar view for February 2025, showing tabe like 'CATEGORIES," "ACCOUNTS, ""BANSACTIONS, ! and "REXINDERS," with transactions listed.", "user=intantion" "The usar likely wanted to view detailed transactions and account categories the selected date.",
is highlighted.
44 Inportant Notes:
Avoid directly copying the AllTree attributes of the element when writing instructiens.
Ensure the instructions are clear, unambiguous, and concise, preferably described in eingle
Do not reference the distinctive red indicator when describing UI elemeats.
RETURN THE DICTIONARY STRICT JSON FORMAT:
Prompt For Interface Captioning
You are a GITT analysta agent cur ly working with les_nare) device. You will objective comprehensive descriptions of the screenshat's 's conten:s and functicnality. These deseriptions should thoroughly explain each attriautes, spatial arrangement, and purpose within the inserface.
zeceive full sczeenahot of (application].
produce
by covering its
#t remente for bascriptions:
Contextual Details: Explain the interface's overall structure and the spatial ionshiss belween elemente.
Visual Characteristics: colers, shapes, icons, text labels, and other distinguishing proparties.
User Interaction: Specify how users can interset with each element and the expected results of those intersctions.
Punstionnll Purpose:
clarify the screarshot's within the broader acatioatton
workflow.
* important Notes:
syntheaiza tha attribute information create natural, user-friendly descriptions. Maintain conclaenesa while enauring the descriptions are sufficiently det to sonvey the GuI's structure and operatior.
PLEASE GENERATE CAPTION:
56

---

Published as a conference paper at ICLR 2026
Prompt For LLM-assist Filter
You GU1 analysis
tasked with evalusting interface (cs_namm)
davice. You will be provided with the following rescurces: first image
screenshot of an (applisation), is highlighted with (marker].
ares second image 13 sub-image, ch cropped from the screenshot around the selected element is marked with (market),
Your objective is determine whether the marked resides in the topmost layout and can be direatly clickad. Your response muat be raturnad in JSCN format, adharing to structure below:
"Jjson
("answer*: *No")
The value of ""answer" can only be one of the following: ""Yas"" Indicates that the marked tokable or val id element that la the focus elament of the current interface. "NNo"; Indicates or othervise non-clictable due to errors, loading issues, or the absence of valid interactive element, oz the marked area is not the focus element of the current interface.
the topmost view and contains
the
interrttte,,, obstrucced, intercepled,
conditions that The marked area resides in the background and is not the focua element of the current interIace.
non-alickable:
The image diaplays an error fails load conten; properly.
The mazked azea corresponds to an empty or blank regicn with ao visible or interactive elements.
The marked area containg anemalies such as ovarlapping alements, misplaced componenta, other irregulariztes
hinder propar interaction. The marked area located in background and not the Tocus element of the current interface.
RETURN THE DICTIONARY STRICT JSON FORMAT:
Prompt For High-Level Objective in Wcak-Semantic Trajectorics
are expert designing anslyzing navigation tasks. specializing in evaluating usarfa interaction trajectory within on lapplication) on (os_rama) device to deduce their overarching navigation goal.
You will be given the following information: sxInitial
Imagexs: visual representation the starting point the intersction shown in the first image.
**Final State Image**: visual representation
andpoint of interaction
shown in the second image.
3. sxInteraction Jectorrvy A detailed log of each step takea by the user, including the intent behind each action: (history)
Your task is to craft a concise summary (1-2 sentences) that dascribes the navigation journey by focusing on geal outcome.
1. v*Tdentiries the user's core objectivers:
Emphasize the transition from the initial state to the final state (implicitly or explicitly).
Focus on the usar's overall intent as inferred from the interaction history and final state, avolcing overly detaf descript describs the cask as "updating preferenzes* rather than "toggle the switch*).
operational steps (e.g.,
**Highlights the functionality
state=e:
Briefly describe the primary function of the final state, rocusing what the user can accomplish or access as a result of completing the asvigation task.
For exampla:
phone is displaying Amap's app info page. goal
the Guide*
saction Amop's hemepaga from here. view Amap's not intatiiaioon pernisator, I went to move from Amap'a homepage to system settings page for Amap.
Starting from Amap's battery usage settings, need to resch the "offline Maps" saction
app"
interface.
with the aim of aaving post in Tnatagram, please advance from the home screen to Post The acreanshot shows the Chrome rrfo page. want from here to the "Hiatozy" section in Chrome's is main menu.
InaLagiam's homepage.
57

---

generate Published as a conference paper at ICLR 2026
New, based on the provided input, assuming you are the usar, please generate an instruction of the operational navigation goal by using the rirat-person present or ingerative seateace:
Prompt For Low-Level Instructions in All Trajectories
You are a GUT agent ousrently operating on (os_name) device, Your taak concise and clear operational instruction for interacting with the selected UI element. These instructions should be relevant to the operation and include operated details such relevant content visible in the screenshots. Instruction can lnvelve the appearance, must
appearance,
position, order, file names, other
poeition, functional description the generated instructton yn quely corresponda to the selected element.
selected element,
You will be previded with:
1. The first image is original screenshot from an (applicationl, which are manuelly marked highlight the salected element.
2. The second image la the :esults of the operations ""(aaiion]"" executed or the element.
image not exist. 3. The third image is sub-image cropped from the original screenshot, forusing on the selected element, which is highlighted with zed bounding bex and arzow for better visibility.
'cerminste', then
4. The AllIree astributes of the selested element: [alemen:_alltzac).
REMBBER:
directly eepying Alliree attributes of selected
instructiens.
Do NOT reference the distinctive red indicater when deseribing VI elemeats.
re generate the operational instraction which can uniquely correapond to the selected element contain operations. Avoid "highlighted", "red box", *red cirele" and "zed poin=" in your output:
Prompt For Rationales All Trajectories
are a sur agent operating (os_name) device, Your to analyze
potential reasen behind operations,
will ba provided with:
1. The first Image is a original screenshot from an (applicationl, which are marked to highlight the selected element.
2. The second image is the results of the operations ("(ctiion""" executed en the the sction 'corminate', then sub-image crepped from the original screenshot, focusing the selected clement for bettar visibility.
selected element.
image does not exist.
3. The third image
A11Tree astributes of selected element: [ellmen:_altt>=e))
5. The Lask jectttive is (Lask_ob'eettvve" and history t:ace is "(history)".
Guidelines:
Examire the salected element relevart contexcual features that support task completion, considering both the objestive and interaction histery. [markez) higlighted in image is monually added to assist in identifying alements and reshould not** been mentioned.
Brovide your reasening in three seatences, ensuring al alignment with the goal and labeled action, but
artual action bounding as justification, these reflact hindaight rother than pradictiva insight. Restrict your analysta detal from the firat image only, and avoid referencing image order.
example:
The screenshot shows file dislog with active selection on formet dropdown. Chançing format completes the file configuration sub-task. Next, click sa lon.
confirm the
Tocua only on the thoughts leading up to the event, not what happens after, Do not zefer
highlights, boxes, or circles in your description think aloud as you work on this task:
58

---

Published as a conference paper at ICLR 2026
Prompt For Instruction Boost
helpful assistant refine the instructions should be claar, polite, and structured as a direct request or question, including:
instructions. the refined
A specific action or configuration change.
Optioral context "eason'ng (0.g., wart ensure my browsing private"), å conversational yes concise tone
Somee axamples for reference:*:
"Cenfigura the system to show seconda in the taskbar clack."
to automatically check for updates startup?" "Could you essist me in cleaning ny computer by removing tracking data that Chrome might have stored?*
you configure
"I want to hear semething seft and beautifil music when Windows stazte Can you set that NP3 file like as my startup sound?"
don't
these
the home page of Microsoft sdge. Remova them
in Page settings."
+*Output Formst:**
You should provide various styles the output should structured as tollows:
Can you ...7
...7 I don't want to ...j ....
<vinuut instructionn:: jbjectobeettive
Rewrite the provided imput instructions, ensuring they are actionable, polite, and include necessary detai "," separate different output:
59

---

Published as a conference paper at ICLR 2026
Gartner
PORRESTIN
BDC
(a) Failure Case 1: The red box pointed to by the red arrow is originally unclickable image element, but it is set role=button in the HTML.
(b) Failure Case As indicated by red arrow, some non-top-level elements invisible list elements are not filtered out by the rules.
Elasansca
(c) Success Case 1: Reduce web page dynamics.
Rener
(d) Success Case Correctly handle element hierarchy relationships.
Figure 31: Examples of visualizations in web data acquisition. (a) shows website developer element identity attributes incorrectly, (b) illustrates complexity or particularity of the web leads to problems with hierarchy and visibility analysis, (c) demonstrates web pages when playing videos, and (d) presents an example of correctly analyzing each element in page. The red box represents clickable elements, the green box represents non-clickable elements, and the blue box represents illegal elements that have been filtered out.
alleviate the dynamic problem of
60


Reviews:
[{"review_id": "OTPYNdG5Q4", "reviewer": "Reviewer_AxPt", "summary": "The authors build a large-scale training dataset for GUI agents, including 6 operating systems and 3 task domains, and develop the ScaleCUA series of models. The authors propose an automatic method to collect trajectories and grounding annotations and build a model that can perform well on GUI understanding, grounding, and planning. The authors benchmark ScaleCUA on a number of tasks to show its efficacy.", "questions": "1. On online evaluation tasks, especially OSWorld, ScaleCUA underperforms compared to OpenCUA and a few other models despite being trained on large quantities of data. Could the authors comment on this? This is especially relevant because it raises questions about the effectiveness of the data collection method.\n\nPlease see the weakness for more questions and suggestions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors curate a large-scale dataset for GUI agents. This includes understanding, grounding, and planning data. The data consists of several software applications across diverse platforms. The authors claim to release the data openly, which will be very valuable.\n2. The authors conduct a comprehensive set of experiments. They evaluate on several benchmarks across different tasks. They also provide several analyses on data scaling, the effect of screenshot resolutions, and several other design decisions. These provide good insights for future model development. \n3. The authors provide practical guidance for collecting data from various platforms in the Appendix, which is very insightful.", "weaknesses": "1. The data collection introduced by the authors consists of an automatic method where rule-based agents and heuristics are used, and another component that comprises human-created tasks and trajectories. The presentation of this section gives a misleading impression that the trajectories are created by both humans and agents collaboratively, but in actuality it is some trajectories created by agents and others created entirely by humans. I fail to see any new method or innovation here (besides the heuristics that the authors develop for rule-based agents). Could the authors please provide clarification for this?\n2. The authors do extensive evaluation, but compare with a limited set of models. Several recent models, like [1], [2], [3] use RL to train their models and show that they can achieve impressive performance using public data. I understand that the ScaleCUA models are only SFTed and not RL finetuned, but since models are also highlighted as one of the contributions of this work, comparing them more extensively would further show the effectiveness of their models.\n3. The authors should conduct a more controlled comparison between their data and open source datasets like Jedi. The authors currently compare the benefits of using their data over public data, but it is not clear if the quantity of data used for this comparison is the same or not. The authors could take a fixed random sample of data from different datasets like Jedi, OsAtlas etc., and fine tune the same model to show whether the data collected by the authors provides more gains than already available datasets.\n\n[1] Yang et al. Gta1: Gui test-time scaling agent\n\n[2] Liu et al. Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization\n\n[3] Tang et al. Gui-g2: Gaussian reward modeling for gui grounding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7WIh1XBp2u", "reviewer": "Reviewer_ojRP", "summary": "The paper tackles the challenge of building computer use agents (CUAs)—AI systems that can autonomously operate GUIs across desktop, mobile, and web platforms. Current progress is limited by the scarcity of high-quality, open computer-use data and the reliance on closed-source models.\n\nTo address this, the authors introduce ScaleCUA, a framework that combines a large-scale cross-platform dataset with a family of open-source agent models. Their approach integrates automated exploration with human expert trajectories, creating a diverse and scalable training corpus. The resulting models unify perception, reasoning, and action, and support multiple inference modes for flexible deployment.", "questions": "While ScaleCUA spans six platforms, it is unclear how well it generalizes to unseen applications or domains. For example, can a model trained on productivity and web apps adapt to enterprise software or creative tools?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces a large-scale, open dataset spanning six platforms (Windows, macOS, Linux, Android, iOS, Web) and three task domains (GUI understanding, grounding, and task completion), collected through a hybrid pipeline of automated agents and human experts.\n\n2. Develops a suite of base agent models that unify perception, reasoning, and action, supporting three inference modes (Grounding, Direct Action, Reasoned Action) within a unified action space for seamless cross-platform interaction.\n\n3. Demonstrates strong performance across multiple benchmarks, achieving new state-of-the-art results and showing that scaling diverse GUI-specific data significantly improves general-purpose computer use agents.", "weaknesses": "1. The dual-loop data collection pipeline (agent + human) is presented as a key innovation, but similar hybrid strategies have been explored in prior works such as OS-Genesis (Sun et al., 2024b) and AGUVIS (Xu et al., 2024). The paper risks being perceived as a scale-up rather than a conceptual breakthrough.\n\n2. A large fraction of trajectories come from random-walk or weakly semantic exploration, which may not reflect realistic user goals. This could bias the model toward superficial navigation patterns rather than meaningful task completion.\n\n3. The paper reports strong results but lacks fine-grained ablations. For instance, it is unclear how much each component (weak semantic trajectories, reasoning data, augmentation strategies) contributes to final performance. Similarly, the trade-offs between the three inference modes (Grounding, Direct, Reasoned) are not deeply analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "orfBxffpNR", "reviewer": "Reviewer_ZN1J", "summary": "This paper introduces ScaleCUA, a large-scale dataset with multiple OS and tasks supported and associated agents trained on the dataset. It begins by incorporating a dual-loop data collection pipeline consisting of agent-environment interaction loop and agent-human hybrid data acquisition loop. The authors then train Qwen2.5-VL on the collected dataset with three modes: grounding mode, direct action mode and reasoned action mode. Experiments are conducted on multiple datasets, including screenspot-pro, webarena-lite-v2, mmbench-gui, os-world-g etc. These evaluations cover multiple tasks and demonstrate a comprehensive & promising results for ScaleCUA. ScaleCUA shows the potential to scale up CUA performance with cross-platform data collected from automated agent data collection and human-in-the-loop data collection together.", "questions": "1. for Figure 5(a), I observed that the grounding accuracy keeps almost unchanged or dropping when the resolution is increased from 720p to 4k in screenspot-v2, also the grounding accuracy saturates at 1080p in osworld-g. Do these results demonstrate that \"Grounding accuracy rises steadily from 720p to 2K, with diminishing returns at 4K.\" is not a universal finding, but only occuring in screenspot-pro benchmark?\n2. It can be observed that scaling data has marginal effects on improving agent performance on WindowsAgentArena benchmark,  does this implicate that increasing train data amount cannot benefit navigation tasks? Can you also share the performance on OSWorld with different training data ratio?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. ScaleCUA contributes a large-scale dataset to the GUI Agent community, with 15k weak-semantic trajectories from automated agent data collection and 4k expert-curated trajectories from human-in-the-loop data collection. Notably, the collected data cover 6 OS, including desktop and mobile ones. \n2. ScaleCUA contributes several interesting key insights to the community backed with supporting experiment results, including the finding that models trained with raw coordinates perform better than those trained with normalized coordinates, especially in cross-platform scenarios; reasoned action mode is  always better than direct action mode, the multi-format GUI grounding corpus, etc.\n3. This paper is well written, clearly structured, and provides comprehensive details and ablation studies in the appendix for future research.", "weaknesses": "1. For the dual loop, currently it includes two components: agent-environment interaction loop and agent-human hybrid data acquisition loop, and these two components both contribute to the final data collection of ScaleCUA, with different properties, i.e., data collected from agent automated pipeline are of large scale but weak semantics, data collected from human-in-the-loop pipeline are of high quality but high cost. However, ScaleCUA did not comprehensively integrate these two components so that they can benefit each other. This is not a major drawback, but a point that the authors can improve further. The proposed dual loop is promising, but the current implementation of simply merging the two data sources is naive, which has been discussed in prior work。\n2. The dataset distribution is extreme, e.g., MacOS data takes only 0.6% of the total data, and grounding takes 97.6% of the total Windows data. As a cross-platform dataset with multi-task support, this distribution is not reasonable enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TquCl48p6b", "reviewer": "Reviewer_TkSU", "summary": "This paper introduces a cross-platform computer use dataset, collected via an interactive data pipeline that\nintegrates automated agents with human experts. They also developed ScaleCUA, a family of base agent models achieving reported SOTA on several GUI-oriented benchmarks.", "questions": "How does the model perform when faced with minor software changes (e.g., UI updates, version differences)? Is there a mechanism to adapt to these types of alterations without requiring retraining?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The model achieves impressive results on several challenging benchmarks, including SOTA performance on MMBench-GUI L1-Hard (94.4%).\n- The paper presents a comprehensive suite of ablation studies exploring design choices, including data augmentation, trajectory weighting, coordinate formats, resolution impact, inference methods, and data scaling.\n- The dataset spans six OSs and multiple GUI domains, broader than previous open datasets (e.g., OSWorld, JEDI, WebArena). Openly releasing all data, models, and code is a significant contribution, particularly within this research area, where datasets are often proprietary.", "weaknesses": "No major weaknesses, some suggestions:\n- The paper contains several minor typos and grammatical errors (e.g., \"expensively to annotate,\" capitalization issues) and inconsistencies in terminology. \n- They can add an analysis of the data quality, comparing the current approach and manual collection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3XTyUGoC3L", "reviewer": "Reviewer_56qp", "summary": "This paper presents a dataset and agent models for computer agents. Provides a rich source of open source computer use data that is curated across various platform to fine tune the models. They show that fine tuning with this data curated across various platforms creates a strong model that performs well across various computer use benchmarks.", "questions": "Is the fin-tuning used SFT in this case?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The comprehensive data set and model that the paper proposes have the potential to further the research in this field.\n2. Experimental results showing the overall approach and how well it generalizes across various tasks. \n3. Method presented for data curation, which can be further extended to create more data assets", "weaknesses": "no comment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)