OpenCUA: Open Foundations for Computer-Use Agents
Authors: Xinyuan Wang
Bowen Wang
Dunjie Lu
Junlin Yang
Tianbao Xie
Junli Wang
Jiaqi Deng
Xiaole Guo
Yiheng Xu
Chen Henry Wu
Zhennan Shen
Zhuokai Li
Ryan Li
Xiaochuan Li
Junda Chen
Zheng Boyuan
LI PEIHANG
Fangyu Lei
Ruisheng Cao
Yeqiao Fu
Dongchan Shin
Martin Shin
Hu Jiarui
Yuyan Wang
Jixuan Chen
Yuxiao Ye
Danyang Zhang
Yipu Wang
Heng Wang
Diyi Yang
Victor Zhong
Y.Charles
Zhilin Yang
Tao Yu
Venue: NeurIPS 2025 spotlight ()
Keywords: Computer-Use Agent
Visual Language Model
Planning
Reasoning
Scaling
Dataset
Evaluation
Primary area: applications

Abstract: Vision-language models have demonstrated impressive capabilities as computer-use agents (CUAs) capable of automating diverse computer tasks. As their commercial potential grows, critical details of the most capable CUA systems remain closed. As these agents will increasingly mediate digital interactions and execute consequential decisions on our behalf, the research community needs access to open CUA frameworks to study their capabilities, limitations, and risks. To bridge this gap, we propose OpenCUA, a comprehensive open-source framework for scaling CUA data and foundation models. Our framework consists of: (1) an annotation infrastructure that seamlessly captures human computer-use demonstrations; (2) AgentNet, the first large-scale computer-use task dataset spanning 3 operating systems and 200+ applications and websites; (3) a scalable pipeline that transforms demonstrations into state–action pairs with reflective long Chain-of-Thought reasoning that sustain robust performance gains as data scales. Our end-to-end agent models demonstrate strong performance across CUA benchmarks. In particular, OpenCUA-72B achieves an average success rate of 45.0% on OSWorld‑Verified, establishing a new state-of-the-art (SOTA) among open-source models. Further analysis confirms that our approach generalizes well across domains and benefits significantly from increased test-time computation. We release our annotation tool, datasets, code, and models to build open foundations for further CUA research.

OPENCUA: Open Foundations for Computer-Use Agents
Xinyuan Wang" " Bowen Wang" " Dunjic Lu L Junlin Tianbao Xie " Junli Wang" "
Jiaqi Deng' Xiaole Guo" Yiheng ChCn Henry Wu Zhennan Shen Zhuokai Li Ryan Li Xiaochun Li*
Junda Chen't Boyuan Zheng Peihang Li Linganyu Lei' Ruisheng Cao" Yeqiao Fu' Dongehan Shin Martin Shin'
Jiarui Hu" Yuyan Wang" Jixuan Chen Yuxiao Ye Danyang Zhang" Yipu Wang" Heng Wang"
Diyi Yang' Victor Zhong" Y. Charles" Zhilin Yang" Tao Yu"
XLANG Lab, The University of Hong Kong Moonshot Al Stanford University University of Waterloo *Carnegie Mellon University
Project Page: httpss//opennuuaxlanggal
Abstract
Vision-language models have demonstrated impressive capabilities computer-use agents (CUAs) capable of automating diverse computer tasks. As their commercial potential grows, critical details of the most capable CUA systems remain closed. As these agents will increasingly mediate digital interactions and execute conse- quential decisions on our behalf, the research community needs access to open CUA frameworks to study their capabilities, limitations, and risks. To bridge this gap, we propose OPENCUA, a comprehensive open-sourg framework for scaling CUA data and foundation models. Our framework consists of: (1) an annotation infrastructure that seamlessly captures human computer-use demonstrations; (2) AGENTNET, the first large-scale computer-use use task dataset spanning 3 operating systems and 200+ applications and websites; (3) scalable pipeline that transforms demonstrations into state-action pairs with reflective long Chain-of-Thooee soning that sustain robust performance gains as data scales. Our end-to-end agent models demonstrate strong performance across CUA benchmarks. In particular, OPENCUA-72B achieves an average success rate of 45.0% % on DSWorld-Veriffedd establishing new state-of-the-an art (SOTA) among open-source models. Further analysis confirms that our approach generalizes well across domains and benefits significantly from increased test-time computation. We release annotation tool, datasets, code, and models to build open foundations for further CUA research:
rea-
https: /ate://oppnnt /ppencua. xlang.l............
61.4
728
60
A
40
45
43.9
OSWorl SR
38.1
7B
31.4
A
Our Recipe
Recipe
10
42
OpenCUA-73B CUA CUA
Oweni-VV
Clande Somner
Claude Sconet 4.5
Data Scale (k)
Figure 1: DSWorid-verified Performance
Left: OpenCUA recipe helps performance scaling
with data size and model size. Right: OpenCUA-72B outperforms current open- source models and comparable with Claude 4 Sonnet 22]
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Trajctory
ScreenVideo
Rule-based method
Model--bal baaed method
Video
+
MAction Reduction
State-action matching
AgNet
Reflectrr
Vetadata
Generator
Ia
Ayenther Tool
Agentiol
Memed
Computer Use Agent Modeling
Monologue Modeling
Modias
OpencUA
OpanCUA
Agentier
DaretBunch
Sturr
Trajectory
60 srcc paper
Observation
Situction Reflection
low level Instruction
Selting
History Modeling
AgNet Dataset
Instruction
!
Realistic
History
Agent
Environment
AgNet
Sup clck
AgNet CUA
Banch
Step 2: with
Figure 2: Overview of the OPENCUA framework. Top left: AGENTNET TOOL captures user interac- tions across OSes with screen videos and action flows. Top right: Raw demos are processed into state-action trajectories with reasoning and history. Bottom right: AGENTNET Dataset and Bench provide diverse tasks and offline evaluation with stan-standard actions. Bottom left: OPENCUA models are trained and able to execute realistic environments.
Introduction
Computer-use agents (CUAs), powered by vision- Ission-language models (VLMs), aim to autonomously complete computer tasks and have great potential in facilitating daily and professional workflows. Despite their growing role in high-stakes decision-making, critical details including training data, architectures, and development processes about how state-of-the-arr CUA systems are built remain closed and proprietary and raises safety concerns [32
[I]
4
16
29
As the lack of transparency limits technical advancements 10), the research community needs truly open CUA frameworks
3
40
to study their capabilities, limitations, and risks.
However, current open-source attempts in CUA face significant challenges that impede progress. Firstly, there is no open-source scalable infrastructure for collecting diverse large-scale computer- use data a complex requirement that involves the real-time capture of user interactions and state information, followed by transformation into agent- executabee trajectories. Secondly, existing open- source graphical user interface (GUT) datasets remain limited in scope and scale due to the complexity and high cost of data collection; they either focus specific domains (grounding [9] 43 77 46),
mobile [3 23), or web 13 IT) or lack sufficient diversity for general computer-use applications.
Furthermore, many CUA works provide insufficient details about their modeling strategies and
training recipes, making replication difficult even with access to their collected data. These limitations collectively hinder advances in ceneral-purposs CUAs and restrict meaningful exploration of their scalability, generalizability. and potential learning approaches.
To address these challenges, we introduce OPENCUA, fully open-source framework to scale the CUA data and the foundation models (Figure 2). To address infrastructure challenges, we first develop user-friendly, cross-OS computer task annotation application AGENTNET TOOL that can be installed on personal computers to seamlessly record natural human demonstrations and corresponding computer states, without disrupting the user's workflow (Figure toop left). We then collect the AGENTNET dataset, including 22.6K open-domain computer task trajectories spanning over 100 applications and 200 websites across Windows, macOS, and Ubuntu (Figure 2 top right). This dataset authentically captures the complexity of human behaviors and environmental dynamics from users' personal computing environments. Furthermore, given that online CUA benchmarks such as OSWorld 44 require substantial environment setup effort and runtime resources, we curated AGENTNETBENCE based
collected human demonstrations (Figure 2 bottom right). This offline benchmark provides multiple gold-stan gold-standard actions per step, efficiently approximating online metrics to dramatically accelerate agent evaluation and development.
Critical to our OPENÇUA framework our (1) data processing pipeline and (2) novel modeling and training recipe for constructing CUA training data from human demonstrations, We first introduce an

---

Table 1: Comparison between AGENTNET and Other GUI Datasets
Avg. Step
Env.
Personalized Env.
Human Traj.
Dom/ AxTree
Inner
Dataset
Tasks
Type
Video
Monologue
AndroidContrll 20
15283
5.5
Mobile
Short
AMEX 7
2991
11.9
Mobile
X
AitW 33
2346
8.1
Mobile
X
X
x
AiiZ 54
1987
6.0
Mobile
X
Short
GUI Odyssey
23
7735
15.3
Mobile
X
/
OS-Genesis 39
2451
6.4
Mobile&Web
X
X
X
Short
WonderBread
39
598
8.4
Web
AgentTrek 48
10398
12.1
Web
X
Short
Mind2Web 1
2350
7.3
Web
X
X
X
GUIAct
2482
6.7
AgentNet
22625
18.6
Desktop
Long
action discretization pipeline that converts raw human demonstrations, which typically consist of videos and high-frequency, redundant keyboard/mouse actions, into state-action pairs feasible for vision language model training. Despite this, we observe that training on acatin pairs alone yields limited performance gains even as the dataset size scales (see Figure Left). Our first key insight is that scaling agent capabilities requires augmenting these trajectories with reflective long Chain-of-Thought (CoT) reasoning. We propose a reflective CoT synthesis method that explicitly injects planning, memory, and reflection into the per-step reasoning process through natural language "inner monologue" (Section 3.T). Different from previous work, our reasoning traces are notably more detailed and contain refletion thoughts that help the agent detect and recover from errors. Moreover, we identify key modeling details that improve agent performance (Section such as multi -aalt--image history. Finally, we show that carefully designing training data mixtures including diverse reasoning and general text-is beneficial for computer-use agent training (Section 3.3
Built upon our methodology, we developed strong computer-use agent models through supervised fine- tuning (SFT) (Figure@boottom bottom left). Our results show that our approach enables robust performance scaling with increased data size (Section 4.2). Our model, OPENCUA-72B achieves a success rate of
45.0% (100 step) on OSWorld-Vere Verified [44 47), establishing a new state-of-the-art among the open- source models, even outperforming the proprietary OpenAl CUA (GPT-4o based) [29 and Claude 4 Sonnet (2). It also has strong GUI grounding ability: 37.3% (SOTA) on UI- Vision (25] and 60.8 % on ScreenSpot-PPe
[19]. We did extensive experiments and analysis various model structures and data scales in Section S Because of the diversity and coverage of our training data, our models show strong cross-domain generalization. Our agent models also show promising scalability with increased test-time compute, such as increased number of steps and larger n in Pass n evaluation. We also did additional experiments, including grounding and robustness analysis. Finally, we also provide detailed ablations to justify the important design choices in our method and training recipe (Section 5). We open-source the complete suite of our OPENCUA framework, including the annotation tool, collected datasets, code, benchmark and models, providing open foundations for further CUA research.
2 AGENTNET Collection
OPENCUA aims to scale desktop computer-use data across diverse computer environments and user scenarios. We prioritize collecting demonstrations that follow natural user behavior, imposing the least additional constraints on how users interact with computers to improve the scalability of data collection. To this end, we developed AGENTNET TooL and collected AGENTNET dataset, the first large-scale desktop agent task dataset. The introduction AGENTNET Tool. and the data annotation strategies are demonstrated in the Appendix F.T
2.1 Task Definition
We model the agent's decision decision-making process iterative observation of the computer state followed by action prediction Given a task language instruction and initial state su, the agent sequentially predicts a action 0; until goal state s, and performs the termination action ar: P(a, 8o, Ap. .., 8:).
state-action transition trajectory: (T, (8o, an), ($1, az)...., (sr, ar)).
AA total of 41,428 trajectories were used to train OPENCUA- 7B and OPENCUA-3 32B, and 27,804 trajectories were used for training OPENCUA QWEN2-7B and OPENCUA-A A3B. Moonshot Al annotated the Ubuntu subset and generously agreed to release 5K of the annotated trajectories the public.
3

---

Tusk Instruction
Reflection
Instructiin
Hiatory
fr changed by action?
The actiio effective? acttion is neceosary?
reflect
CaT
Largen-
Action
arga
generate
Figure 4: Reflective long CoT synthesis pipeline: generator and reflector iteratively generate and verify the reasoning components between the observation and ground-truth actions.
An important design choice in building computer-use agent is to convert compute state s; into model observation. In this work, we follow the recent trend of building pure vision-based computer agents
50 4 and use the screenshot of the computer the observation for the agent. We use human computer-use use actions, including keyboard and mouse movements, as the action space. To ensure the action space is applicable across various operating systems, select subset of PutooGUI actions and augment them with several necessary agent actions including the 'success' and 'fail' termination actions. The complete action space and its parameters listed in Table g
30
2.2 AGENTNET Statisties
Our dataset consists of 22,625 human-annotatee computer-use tasks, including 12K from Windows, 5K from macOS and 5K from Ubuntu, with screen resolutions ranging from 720p to 4K. Each trajectory averages 18.6 steps, reflecting the complexity the task. As shown in Figure the data spans over 140 applications and 190 websites, often involving multi-app workflows, professional tools, and un- common features. Compared to previous GUI datasets (Table DT. AGENTNET is the first desktop trajectory-level dataset that is real- istic, complex. diverse, and multimodal. The statistics provided in the Appendix F.2
Exoonerc 13.0%
Danly SO SELER
Apvostivr
Thl
Virk 19.1%
Offic Norte
Cl 3.%%
3 Training Computer-Use Agent Model
Figure Domain distribution of tasks in AgentNet dataset
Our AGENTNET consist of task instructions and state-action (si,a) ai) trajectories. However, find that directly finetuning vision-language action (VLA) models on our 27K trajectories leads to poor performance (4.4% success rate on OSWorld [44), shown in Figure Left "Base Recipe"). This section presents modeling and training recipe enable scalable training of computer-use agent models, including novel reasoning augmentation, context encoding, and data mixtures techniques.
3.1 Synthesizing Reflective Long CoT Reasoning
we find natural language reasoning crucial for generalizable computer-use foundation models, helping CUAs internalize cognitive capabilities. We propose a multi-stage CoT framework synthesizing structured reasoning per state-action pair (s,, a:). Inspired by Aguvis (50), our structured CoT includes three reasoning levels. The hierarchy begins with L3, contextual observation capturing salient visual and textual elements. Next, L2 provides reflective rea- soning analyzing state transitions, recalling previous steps, correcting errors, and planning subsequent actions. Finally, the model predicts L1, concise executable action grounded in prior perception decision flow, equipping the
Consistent with prior works 53
50 30
and thought. This L3 L3-+L2-10 --2- L1 structure mirrors model with coherent, interpretable reasoning trajectories that enhance generalization and robustness.
Reflection augmentation for L2 reasoning Incorrect or redundant annotations in human demon- strations are not all bad, as long as we can identify and use them to teach the identification and correction of model errors. Therefore, we designed reflector to identify errors and generate reflection reasoning for each step. Our CoT synthesis framework extends the pipeline of Aguvis (50) and ActRe (52) by equipping the "Thought" with more comprehensive agent components, especially transtt perception and reflection, and minimizing hallucination. As shown in Figure our CoT synthesis pipeline consists of three components: reflector, generator, and summarizer.

---

The reflector inspects each step for correctness and redundancy by comparing screenshots before and after the action, examining the correctness of the action code itself and the generated CoT, expecially whether the "Action" aligns with the screenshot and code. When the step is incorrect or redundant, the reflector will elaborate reason and this step will be ignored during training. the step is correct, the reflector will explain the differences the actions brings to the before and after state. The generator conditions on the full agent context- reflections, action history, task goal, screenshots, and action code-to generate structured COT. To help the model ground coordinate-related actions more accurately, we incorporate visual cues: a red marker on the mouse ac- tion coordinate and zoomed-in in image patch (inspired by V [41).. Finally, the summar iierr refines vague user-written goals into more precise and aligned task objectives, and scores each trajectory for alignment, efficiency, and difficulty. Our method produces rich and meaningful CoTss that significantly improve model reasoning and planning. We use claude -----3onne----- -22250229 as the base model for synthesizing the three components. The reflection helps agent model identify former errors and adjust future plan to make the task back to the right track. An example of error identification and correction in evaluation can be seen in Section[[] Ablations in Section (decooonseemmonsraccc that this module is a important driver of performance gains.
3.2 Context Encoding and Test-Time Reasoning
For end-to-end agent models, history encoding plays a critical role in reasoning and planning.
Context encoding: (I) Textual history: We propose structured inner monologue framework for textual history representation. Specifically, we adopt dialogue-style history format representing the model's responses and corresponding computer screenshots. Following Aguvis [50]. we use L1 CoT (Action) to represent previous steps because it is more token-efficient and allows longer history windows without truncation. Morcover, our inner monologue includes memory components, further compensating for the absence of richer CoT in carlier steps. See Section S for ablations on history representation. (2) Visual history: Multi-image screenshot history is essential for agent model performance because screenshots are lossless visual representation of history, providing more grounded context than textual summaries. However, including more images also increases input length and reduces training efficiency. By default, we use three screenshots as visual representation, as our experiments show that this achieves a balance between performance and efficiency (see Figure(D)
Test-time reasoning format:
While the model is trained with a mixture of CoT levels, we adopt the L2 CoT format at inference time due to its richer reasoning content, which enhances the model's ability to reflect, plan, and reason. As shown in our ablation studies (Section 5). L2 CoT significantly improves test-time performance scalability- Pass @n success rates OSWorld increase markedly over Pass@ 1. In contrast, models lacking this reasoning augmentation exhibit limited scalability, highlighting the importance of strong reasoning signals inference time.
3.3 Training Strategy
CoT data mixture: As we mentioned in the Section 3.1 our structured inner monologue contains three levels of CoT: LI (Action), L.2 (..2(Toure (Thought + Action), and L3 (Observation + Thought + Action), each encoding complementary information for agent decision-making but has different conceptual information. LI CoT has direct connection to the actual action; while there is helpful screenshot perception information in the L3 COT, some irrelevant elements may also be described; L2 CoT contains planning and prediction that directly affect the predicted action in LI. Therefore, we propose to train the model with mixture of all three levels of CoT reinforce this different levels of connection. Data example of L1, L.2, and 1.3 can be seen in Appendix H We verify this design choice with ablations in Section 5 In our newest recipe of OPENCUA-72B we also mix rollout trajectories in real environment in Appendix C
Mixture of grounding, planning, and general SFT data: A general-purpose computer-use agent foundation model should be capable of both solving complex computer-use use tasks and performing general reasoning grounded in world knowledge. To achieve this, we train a mixture of data types that span both uomputer-use and general lasion-language domains.
For grounding, we initialize the model using existing datasets such as ShowUl 21 UGround [1] and 189K bounding-box samples parsed from collected AXTree structures. For planning and reasoning, we include diverse mix of Ubuntu and Windows/macOS demonstrations well as
5

---

task innstruction-augnenn samples (Section 3.1 To enhance generalization and reasoning ability, we additionally incorporate general supervised fine-tuning (SFT) data from the Kimi Team [35] The general text data covering domains such as instruction following, mathematical reasoning. and long-context understanding. The general vision data includes domains such as OCR and vision QA data. This mixture ensures both GUI grounding and high-level reasoning capabilities across domains. Although these general data are not directly related to GUI environments, we find that mixing them improves the model's overall agentic performance. We present ablation results on this effect in Section [
CUA training strategies: Different CUA training strategies are introduced in Appendix C
4 Experiments
4.1
Experimental Setup
Models Our experiments are conducted on multiple open-sourced vision-language models: Kimi-VL
A3B (35], Dwwen2-VL-7B-Instruu (38), Qwen2.5-VL-77B-Inntne
(5), Qwen2.5-VV-32D-Inmmmu [5)
and Dwen2.5-vL-722B-nntti
[5 Kimi-VL-A3B adopts a Mixtureoof-Experts (MoE) architecture
with 16B total parameters and 3B active during training and inference. It demonstrates moderate
capability as a computer-use agent, including grounding and planning. Qwen2- L- and Qwen2.5-V2 are seneral-purpose
Iision-anguage models, with Qwen2.5-VL exhibiting enhanced digital agent capabilities and expertise in high-resolution understanding. We conduct supervised fine-tuning (SFT) on these models and obtain our OpenCUA model variants: OPENCUA- OPENCUA-QWENT 7B, OPENCUA-7B, OPENCUA-32RD and OPENCUA-72B.
Evaluation We evaluated our models on online evaluation benchmarks, offline agent evaluation benchmark, and GUI grounding benchmarks.
1. Online agent evaluation: (I) OSWorld-Veriffed. OSWorld 44 originally curated 369 buman-crafiee tasks covering wide range of applications, each accompanied by its own en- vironment setup and evaluation script. The OSWorld team has now verified every task, fixing those that were infeasible to test because of outdated dependencies, evaluation errors, or am- biguous instructions, and has released the improved benchmark as OS Verified [47 Our results are obtained through the public evaluation by the OSWorld Team on AWS infrastruc- ture. The results are reported in Table 2 Windows-centie tasks, spanning native Windows applications as well as several open-source programs also featured OSWorld. It can reflect the agent's 's online performance on Windows system.
(WAA) contains 154
Evaluation setting: The resolution of the systems is 1920 1080. 4 clock tasks in WAA dropped due to API and system image limitation. We adopt the 1.2 COT format (Thought + Action) for all models, following results of our ablation in Section [ Temperature is set to 0 for deterministic decoding during evaluation. The results in OS WoWlldd Verified are the average of runs.
2. Offline agent evaluation: AGENTNET BENCH includes 100 representative held-out tasks cover- ing diverse domains on Windows and macOS. The introduction and details of AGENTNET are in Appendix B We also validated its correlation with online benchmark results in Figure g
3. GUI grounding evaluation: We evaluate our model's GUI grounding ability, the ability to map natural language instructions to specific actions within graphical user interfaces on 5 benchmarks: OSWorld-G [46), Screenspot-V2 [42), Screenspot-Pro [19, and UI-Vision [25]. OSWorld-G has 564 samples that systematically cover text matching, element recognition, layout understanding and fine-grained manipulation, with annotations for the element types required to solve each task. Screenspot-V2 includes screenshots from three platforms: mobile, desktop, and web. Screenspot- Pro focuses on high-resolution desktop environments, especially in professional settings. UI- Vision includes fine-to-course grained tasks to evaluate model's 's performance on understanding professional software, spatial reasoning, and complex actions in desktop environments.
"To align with the training infrastructure of the Kimi Team, adopt the same chat template and tokenizer as Kimi- VL-A3B. M-RoPE Qwen models is not implemented; we use ID RoPE (33) instead.
'OSWorld Verified Leaderboard: Inttps://os world, github. 10/ OSWorld Verified blog: https://
xlang. htt/blog/ssworllrr /owoordwwwldoooo verified

---

4.2 Main Results
Table 2:
DSWorld-Verified results. OPENCUA-72B achieves the best performance among all open-source models with average success rate of 45.0% %. outperforming prior baselines by large margins. also closes the gap to proprietary agents. This demonstrates the scalability and strength of OpenCUA training pipeline. The details of each run and the Pass @3 evaluation results shown in Appendix Table 7
Model
15 Steps
50 Steps
100 Steps
Proprictary
OpenAI CUA 29
26.0
31,3
31 31.4
Reed1.5-VI. I6
27.9
34.1
Claude 4 Sonnet
31.2
43.9
41.5
Claude Sonnet 4.5
61.4
Open-Source
3.0
3.9
Swen2.5-VLL-72B-
5.0
BL- [35]
9.7
10.3
UI-TARS-72B-DE 30
24.0
25.8
27.1
Qwen3 VL 5
38.1
OpenCUA-7BB 7B (Ours)
24.3+1.
28. 28. +0.7
26.6
OpenCUA-322 (Ours)
29.7
34.1 0.6
34.8 10
OpenCUA-775 (Ours)
39.0
44.9
45.0
Online agent evaluation. Table 2 lists success rates of end-to-end agents for 15-, 50- and 100-step budgets on Verorid-Veriffed. The details of each run, Pass evaluation success rate, and the results of OPENCUA-A3E and OPENCUA-7B are demonstrated in Appendix Table 7 Besides quantitative metrics, we also provide qualitative case study in Appendix G which shows that our agent can recognize and correct earlier mistakes during long-horizon execution, ultimately enabling more reliable task completion.
1. Proprietary models still lead, but the gap is closing, Claude Sonnet 4.5 achieves 61.4 % at 100 steps, followed by Claude Sonnet at 43.9 %. Our OPENCUA- reaches 45.0 in 100 steps, establishing new state-of-the-ee among open-source systems, closing the gap from open-source models Claude models.
2. OpenCUA method applies to models with different architectures and sizes. We apply our method on 5 model architectures, including Kimi-VL-A33 355 Qwen2-VL-7B (38), Qwen2.5- VL-7B [5, Sive2.55-LL3221 [5] and Swen2.5-VL-72.
[S covering MoE and dense structures and activation parameter number from 3B 72B. As the results Appendix 7 all the models' performance improves by a large margin upon the base models. In confirming that our data pipeline and training recipe scale effectively with model capacity. Furthermore, their performance scales with the model size- OPENCUA-72B consistently outperforms OPENCUA-7B under every step budget.
3. Effect of step limit. Most agents profit from a larger step budget from 15 to 50 steps, yet the gain from 50 100 steps is often smaller than that from 15 50 steps.
Claude 4 Sonnet: +12.7 % from 15 + 50 steps, then 2.4 % from 50 + 100.
OPENCUA-322
+4.4 (29.7 + 34.1) from 15 50 steps, and +0.7 (34.1 - 34.8)
from 50 + 100 steps.
Base models including Qwen2.5-Vi and Kimi-Vis VL: marginal gains (< %), reflecting limited step scale capability of base models.
Even top-tier agents (e.g.. Claude Sonnet) still struggle benefit significantly on larger step budgets for truly long mor-horizon tasks. The modest gain when moving from 50-step to a 100-step limit stems from two main factors: (i) most tasks need more than 15 but fewer than GUT actions, so the extra head-room often goes unused; (ii) current models are still not good at recognising their own mistakes, recovering from errors, and deciding when to stop- sallhaalucinatioos and repetitive loops frequently waste the additional steps. We did detailed error study in Apendix D.4
7

---

Table Computer-use agent performance on AGENTNETBENCH. Coord actions: click, rightClick,
doubleClick, moveTo, dragTo, To, scroll; Content actions: write, press, hotkey; Function action: terminate.
Model
Coord. Content SR
Func. SR
Avg. SR
Owen2.5-VL-7B [
50.7
40.8
3.1
48.0
Aguvis-7B 50
56.7
43.3
52.4
Qwen2,5-V1-322B 5
66.6
47.2
41.5
64.8
Qwen2.5-VL-72B
67.2
52.6
50.5
67.0
OpenAl CUA [2
71.7
57.3
80.0
73.1
OpenCUA-77B (Ours)
79.0
62.0
44.3
75.2
OpenCUA-32B (Ours)
81.9
66.1
55.7
79.1
4. OPenCUA models achieve markedly higher Pass@n scores. As reported in Ap- pendix 7 the Pass @3 success rate of OPENCUA-32B
ASWORLD-Verifiee jumps from 34. 34.2%(Passee 1) to 45.6%. The performance of OPENCUA-728 increases from 45.0%(Passse 1) to 53.2% (Pass @ This large margin suggests ample headroom for future post-training, rerank- ing or multi-agent methods. Additional analyses of test-time upper bounds and robustness are provided in Section 5
These observations validate the effectiveness of our pipeline and highlight the remaining headroom for open-source agents to close the gap with larger proprietary models.
Offline benchmark evaluation. AGENTNET BENCII is constructed from representative tasks in the OPENCUA dataset. It is held-out testset including 100 task trajectories on Windows and macOS. To account for domain similarity, we group models into Zero-shot models- those not trained on AGENTNET D00en2.5-VL--7B22B/222 Aguvis-7B, and OpenAI CUA (GPT-4o)) and Fine-tumed models, namely our OPENCUA-78 and OPENCUA-32B. As summarized in Table
1. Scaling with model size. In the zero-shot group, performance scales with model size; the specialist Aguvis-7B surpasses the general-purpose Qwen2.5-VL-7B.
2. OpenAI CUA generalize well unseen tasks. OpenAl CUA outperforms all open-source zero-shot models and approaches fine-tuned agents, particularly excelling in terminate-state detection and content-based actions.
3. Offline benchmark has correlation with the ranking on online benchmark. The overall ordering generally matches the online leaderboard in Table OPENCUA-32B OpenAI CUA > Qwen2.5-VIL models. OPENCUA-32D and OPENCUA-7B are relatively higher due to the alignment of the domain and action space.
4. Coordinate-action performance reflects model grounding performance. OPENCUA-32B exceeds OPENCUA-7B coordinate-bast actions, aligned the grounding results in Table
Performance scaling with data scaling, We explore the effect of data scale on Qwen2-VI VL from three perspectives: cross- domain data, in-domain data, and out-of-domain data. We first investigate cross-domain data in Figure [ Specifically, we compare three training settings: (1) 7K Ubuntu data, (2) 7K Ubuntu 14 14K Win&Mac data, and (3) 10K Ubuntu + 17K Win&Mac data. On OSWorld, performance improves significantly from 9.8 % to 18.5%, despite the added Win& data coming from different platform. This indicates that even out-of-domain data can substantially enhance generalization and reasoning ability, rather than causing negative transfer. To further study the impact of in-domain and out-of-domain data scale, we randomly sampled 3K, 7K, 10K trajectories from Ubuntu data and 3K, 7K 14K from Win&MMcc.
As shown in Figure 5 performance scales consistently across all benchmarks with both in-domain and out-of-domain data. When increasing the Ubuntu data from 3K to 10K, the average performance improves by 72%. Scaling the Win/Mac data from 3K to 14K yields 125% improvement average. These results demonstrate a strong positive correlation between data quantity and agent performance, highlighting the importance of large-scale, diverse CUA data for model generalization.
Performance on grounding benchmarks. The evaluation results on grounding benchmarks are demonstrated in the Appendix D.2

---

OSWorld
WindowsAgentAAren
AgentNetBench
15
Success Rate (%)
15
Uhuntu Min&MMcc
70
9.7
10
8.5
10
8.1
59.23
60
5.4
1.6
55.37
53.5
5
50.9
48
3K
7K
10K
14K
3K
7K
10K
14K
3K
7K
10K
14K
Figure 5: Scaling curves on three benchmarks as data volume from various OS domains increases.
Analysis
Model performance upperbound analysis by scaling test-time compute We further explore our model's performance upperbound by doing Pass @n evaluation on OSWorld. We set the temperature to 1.1 and evaluate OPENCUA- -WEE-2---WN22 7B for 16 times on the budget of 15, 30 and 50 steps and calculated the pass @ 1,4.8,16 success rate. In Table we find: (I) There a significant performance gap of our model between Pass@) I and Pass @ 16. On 15 step. the success rate increases from 16.9
34.6 (+104%), while on 50 step, the increacement is even large from 18.4 39.2 (+113%). (2) With larger n, the performance gains from increasing the step budget become more significant. (3) Online benchmarks have large variance. To study model robustness, we did Pass @n evaluation when temperature is 0 in Figure D.3land and find higher temperature leads to higher Pass @n performance but lower Pass@) @ performance.
We then investigated in the results and find variance comes from these factors:
1. The agent chooses different solutions in different runs. For example, in the task "Re-open the last closed browser tab," the agent sometimes uses Ctrl Shift + T (only one step) and other times navigates through the history menu (needs many steps). Agents may fail on harder solutions.
2. Minor omissions or extras. In Chrome or VSCode settings, forgetting to click "Save" (or performing an additional stray click) converts correct solution into failure.
3. Environment dynamics: Occasional CAPTCHA dialogs, machine variability, and network latency can change the interaction sequence and lead to inconsistent outcomes.
Cross-platform training improves generalization, even with domain differences. As shown in Figure there a consistent performance gap between models trained on different domains. Models trained Ubuntu data perform better on OSWorld, while those trained on Win- dows/macOS data perform better WindowsAgentAreen and AGENTNET BENCH. This domain gap reflects the underlying differences in GUI layouts, system styles, and application behavior across platforms. OSWorld primarily focuses applications and websites aligned with Ubuntu contains sev-
40
39.20%
Pes@ 16
OSWozld SR (%)
36,44%
Passgi4
Pask@11
20
15.90%
17.30%
18.00%
10
Steps
environments, whereas WindowsAgentAAng eral OSWorld Windows-specifh applications. Interest- ingly, the performance gap between training on Win&Mac data versus Ubuntu data is narrower WAA than on OS-
Figure 6: OSWorld @ass@N perfor-
mance curves of OPENCUA-QWENZ. 7B, temperature=0.1
World, suggesting that application-level knowledge can partially transfer across operating systems, even if interface styles differ.
Other analysis and ablation studies. More analysis on model robustness, error analysis and the ablation studies of reasoning format, history image number, CoT mixture, general data, and reflective long CoT are claborated in the Appendix D.3

---

Related Work
CUA benchmarks and datasets Autonomous computer-use agents now judged primarily through execution-level benchmarks that embed the agent inside genuine software environments rather than synthetic simulators. On the desktop side, suites such as 45 551 orchestrate multi-step workflows that span office productivity, source-code editing, file management, and operating -ystem configuration across Linux, Windows, and macOS. For the web domain, campaigns including [56] deploy agents on self-hosted or live sites with dynamic content, long navigation chains, and non-trivial information-retrievall subtasks. To support training the necessary scale, several high-volume data pipelines have appeared: tutorial-to- -tajeeccory conversion for browser tasks [48] 22]. cross-device grounding and action logs [17, [5], plus our own collection of 27 K desktop demonstrations that pair screenshots, low-level mouse/keyboard events, and reflective chain-of-thought annotations.
18 11 51 12
CUA frameworks and models Approaches building computer-use agents be grouped into three broad categories. First, text-based language models operate on structured GUI meta- data-such DOM trees accessibility labels- and issue symbolic commands; representative work ranges from early page-centric agents [24] to more recent language-only planners that still eschew pixels [4 Second, vision-centric agents integrate screen imagery. Some focus grounding learning to associate natural-language references with bounding boxes or coordinate
clicks [5 while others pursue end-to-end policies that translate full screenshots directly into action sequences [50 30 29 []. Third, modular agent frameworks wrap large language models with additional sproonenttsssecciiiseeeee vision encoders, hierarchical search-based planners, episodic memory, and tool AP tackle long-horizon tasks requiring perception, reasoning, and control [57]
Conclusion
We presented OPENCUA, comprehensive open-source framework addressing critical gaps in computer-use agent development. By offering annotation infrastructure, data processing pipelines, diverse datasets, effective training recipes, and efficient evaluation benchmarks, we establish essential foundations for CUA rescarch. Our models demonstrate strong performance across benchmarks while exhibiting clear data scaling laws and cross-domain generalization capabilities. By releasing all components- ttools, datasets, code, and models-we aim to accelerate transparent CUA research, enabling the community systematically investigate these agents' capabilities, limitations, and risks as they increasingly mediate our digital interactions and execute consequential decisions on our behalf.
8 Aknowledgement
The authors of this paper were supported by the ECS (27212023) and Areas of Excellence Scheme (AoE/E-601/24-N) from RGC of Hong Kong. We thank Yu Su, Caiming Xiong, and the anonymous reviewers for their insightful discussions and valuable feedback. We are grateful to Moonshot AI for providing training infrastructure and annotated data. We also sincerely appreciate Jin Zhang, Hao Yang, Zhengtao Wang. and Yanxu Chen from the Kimi Team for their strong infrastructure support and helpful guidance. The development of our tool is based on the open-source projects- DuckTrack [36] and OpenAdapt (27). We are very grateful to their commitment to the open source community. Finally, we extend our deepest thanks to all annotators for their tremendous effort and contributions to this project.
10

---

References
[1] Anthropic.
Claude
computer use.
5-5-models- and cudd-omppter--see 2024. Accessed: 2025-05-03.
[2] Anthropic. Introducing claude 4, 5 2025. URL
claude-4 4
Accessed: 2025-07-28.
[3] Anthropic. Introducing claude sonnet 4.5. Technical report, Anthropic, Sep 2025. URL
https://www http://wwwww.atrop..ccco http:/ppw...coo/////ccccccc/cc https//llaleeeeee-------1 Online; accessed 2025-10-04.
[4] Anthropic. Claude's extended thinking. https: //wpw. anthroppic.commreeeeee
visible- extended-thinkingg 2025. Accessed: 2025-05-03.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502.13923. 2025.
[6] Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal agents at scale, 2024. URL https://arxin org/abs/2409 08264
[7] Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents, arXiv preprint arXiv:2407. 2024.
[8] Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406. 11317, 2024.
[9] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401. 10935, 2024.
[10] Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner Marc Fischer, and Florian Tramèr. Agentdojo: dynamic environment to evaluate prompt injection attacks and defenses for Ilm agents, 2024. URL https://////r org/ abs/ 2406 13352
[11) Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances Neural Information Processing Systems, 36:28091-281 14, 2023.
[12] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vázquez. Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents solving common knowledge work tasks? In Forty-first International Conference on Machine Learning, 2024. URL URR.itttps://pperrviiee
net/ wetp:/orum.dd-BBfg
[13] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, arXiv preprint arXiv:2410.05243 2024.
[14] Boyu Gou, Ruohan Wang. Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. CoRR, abs/2410.05243.. 2024. URL httts://ddoii.... orgooogg111 10. 48550/arXiv. 2410. 06243
[15] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference Learning Representations, 2025.
[16] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu liang, Jiawei Wang, et al. Seedl. 5-vl technical report. arXiv preprint arXiv:2505.0766. 2025.

---

[17] Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference Computer Vision, pages 161-178. Springer, 2024.
[18] Jing Yu Koh, Robert Lo. Lawrence Jang. Vikram Duvvur, Ming Chong Lim, Po-Yo Huang, Gra- ham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
[19] Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo. Yuchen Tian, Jing Ma. Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07030 0 2025.
[20] Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala Divya Tyamagundlu. and Oriana Riva. On the effects of data scale on computer control agents, 2024. URL
https://arxiv. org. abs 2206. 03679
[21] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-actinn model for gui visual agent. arXiv preprint arXiv:2411.17444 2024.
[22] Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang. Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, Yuchen Mao, Jingren Zhou, Junyang Lin, Binyuan Hui, and Tao Yu. Videoagenttrek: Computer use pretraining from unlabeled videos, 2025.
[23] Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08444 2024.
[24] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-ansseriing with human feedback. ArXiv preprint, 2021. URL
29322
[25] Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, et al. Ui- vision: A desktop-centric gui benchmark for visual perception and interaction. arXiv preprint arXiv:2503. 15661, 2025.
[26] OBS Project Obs studio. 2025-08-04.
Accessed
[27] OpenAdaptAL. Openadapt: Ai-first process automation with large multimodal models (Imms).
URL http://ghhhhcccccmopppaaaaaaaaap
Accessed 2025-08-04.
[28] OpenAI. Introducing openai o3 and 04-mini. Technical report, OpenAl, Apr 2025. URL http:////paaa..oooomeeeeeeeiiiie 09-25.
and-o4-mini Online; accessed 2025-
[29] OpenAl. Operator, 2025. URL htRllltttppp/opppaaaccc//reeeere
[30] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL httpss////aa org/ aba/ab////500 12326
[31] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708-599 59728, 2023.
12

---

[32] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of Im agents with an Im-emulated sandhox. ArXiv, (bss/2309.18817, 2023. URI.htps::/ / api semanticscholar
org/Corpus (CCCoppusI::222444000 262944419
[33] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neuroconputing. 2024.
[34] Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprini arXiv:2412.19723, 2024.
[35] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07499 07491, 2025.
[36] TheDuckAL. Ducktrack. URL hctpss//ggtaabb..cm///heebukkk/p 2025-08-04.
Accessed
[37] Bowen Wang. Xinyuan Wang, Jiaqi Deng, Tianbao Xie, Ryan Li, Yanzhe Zhang, Gavin Li, Toh Jing Hua, Ion Stoica, Wei Chiang, Diyi Yang. Yu Su, Yi Zhang, Zhiguo Wang, Victor Zhong, and Tao Yu. Computer agent arena: Compare and test computer use agents crowdsourced real-world tasks, 2025.
[38] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge. Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception the world at any resolution. ArXiv preprint, 2024. URL
https://arxiv. org/ abs/2409. 12191
[39] Michael Wornow, Avanika Narayan, Ben Viggiano, Ishan S. Khare, Tathagat Verma, Tibor Thompson, Miguel Angel Fuentes Hernandez, Sudharsan Sundar, Chloe Trujillo, Krrish Chawla, Rongfei Lu, Justin Shen, Divya Nagaraj. Joshua Martinez, Vardhan Agrawal, Althea Hudson, Nigam H. Shah, and Christopher Re. Wonderbread: A benchmark for evaluating multimodal foundation models on business process management tasks, 2024. UtLtttps:: //atxi/ org/ abs/2406. 1322244666.244.
[40] Chen Henry Wu, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, and Aditi Raghunathan. Dissecting adversarial robustness of multimodal Im agents. In International Conference on Learning Representations, 2024. URL httph://appl www.stiiscconsie.ee... .org/ Lhag/CCrpusDD 270562791
[41] Penghao Wu and Saining Xie. V*: Guided visual search as core mechanism in multimodal Ilms, 2023. URL /https://arrt http://axxvv..o// www.orgg............ /tabs//312... 14135
[42] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: A foundation action model for generalist GUI agents. CoRR, 0bs/22410.22211 2024. doi: 10.48500ARRXI.241022222 URL https://doi. / ar arviv. 2410. 23218
[43] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL battps::/arxiie org/ abs/2410. 23218
[44] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese. Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
[45] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. ArXiv preprint, 2024.
URL UtLhttts: //aaxi/...ii//a/a org/ ab0//40/..2.////////////// 07972
13

---

[46] Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen. Wenjing Hu, Xinyuan Wang, Yuhui Xu. Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025. /RL.itttps://axxxiooor//aaa
13227
[47] Tianbao Xie, Mengqi Yuan, Danyang Zhang. Xinzhuang Xiong, Zhennan Shen, Zilong Zhou, Xinyuan Wang, Yanxu Chen, Jiaqi Deng, Junda Chen, Bowen Wang, Haoyuan Wu, Jixuan Chen, Junli Wang, Dunjic Lu, Hao Hu, and Tao Yu. Introducing asworld-verified. alang.ai, July 2025.
URL https://xlangg aa/ballll//lll blog/ osworld -eriiiiieeeed
[48] Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. arXiv preprint arXiv:2412.000 2200:222..00005 2024.
[49] Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng. Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu. Lemur: Harmonizing natural language and code for language agents. In International Conference on Learning Representations, 2024.
[50] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint
2024.
[51] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents, 2025. URL https://arxin: http://axxiv..g/abaaaa/ htwpg//bb/2500.00000 01382
[52] Zonghan Yang, Peng Li, Ming Yan, Zhang, Fei Huang, and Yang Liu. React meets actre: When language agents enjoy training data autonomy. arXiv preprint arXiv:2403. (4t.:2443..458,, 2024.
[53] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR), 2023.
[54] Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei. and Duyu Tang. Android in the zoo: Chain-of-action thouggh for gui agents. ArXiv preprins, 2024.
URL WRRptttss://xxii org/ abs /tabs/2444
[55] Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng Yan. Agentstudio: A toolkit for building general virtual agents, 2025. URL httttt:///axxi/ org/
abs/2403. 17918
[56] Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024.
[57] Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong WU, Tiannan Wang, Shi Qiu, Jintian Zhang. Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang. Ningyu Zhang, Huajun Chen. Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents, 2023. URL //arxivv .orr/v 2t////3300000
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions: (1) the development of AGE TOoL, a cross-system annotation software for capturing human demonstrations with computer system states; (2) a data processing and augmentation pipeline that converts raw demonstrations into high-quality agent trajectories; (3) the creation of AGENTNET, a diverse dataset of 27K computer-us, tasks; (4) training recipe for effective computer-use use agent (CUA) learning: and (5) the development AGENTNETBENCHA comprehensive offline benchmark for CUA evaluation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix A we outline two key constraints: AGENTNET 's scalability is bound by manual annotation efforts, and its generality is curtailed by our ethical mandate for explicit user consent.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
15

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper primarily focuses on empirical contributions, including the develop- ment of datasets, training methodologies, and benchmarks for computer-use agents. It does not consist any theoretical results, theorems, or proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The detailed experiment settings required to reproduce our results are com- prehensively described in Section 4.1 This includes all necessary information about model configurations, data composition, training detail, and evaluation settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
16

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open-source all of our models and datasets, along with detailed instructions and scripts to reproduce all experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
popllccgguide.. http///ooobosisoonnno.oo oeriionooolyl for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips wttp://pppppp...dd...oosppps htapl/ic/guides//oodddiee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setup and methodology are detailed in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
17

---

Justification: The main evaluation of OpenCUA models on OSWorld- Verified contains the results of runs including the average and variance. While the paper does not include error bars or confidence intervals for the other experimental results, this omission is primarily due to the significant computational cost of training large vision-language models and conducting online evaluations for computer-use agents. We rely on the consistent perfor- mance observed across different models (AgentNet-A.) AgentNet-7B) and benchmarks (OS World WindowsAgentArennd to validate our conclusions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed computational resource usage is provided Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps:/neuurps..... httprrppp.../uubbtt///hhcccciiid
Answer: [Yes]
Justification:
The paper strictly follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
discuss the broader impacts of the paper. OPENCUA introduces efficient data collection tools, curated training recipe, and open-sourced datasets and models, providing foundational platform for future research and development in the field of computer-use agents (CUA).
Justification: In Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We have discussed the privacy concerns related human annotation data in Section F.2
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: We appropriately cite the sources all models and datasets utilized in paper. Qwen2-VL-7B and OSWorld are released under the Apache 2.0 license, while Kimi-VL VL-33B and WindowsAgentArent are distributed under the MIT license. The Aguvis dataset is derived from publicly available existing work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All models and datasets introduced in the paper will be open-sourced, accom- panied by comprehensive documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: During the crowdsourcing process of data annotation, we provided annotators with comprehensive documentation of AGENTNET TOoL, and we will open-source this full documentation, including the detailed instructions given to the annotators. We informed all participants of the wage, and paid them at least the minimum wage in the country of the data collector.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper describes potential privacy risks to participants, and these risks were disclosed to the participants as part of the study process. For each user, presented and obtained agreement on a consent form detailing data access, collection, release, and potential risks. We considers this type of purely digital study exempt from IRB review and therefore did not require formal IRB approval.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We explicitly describe the use of LLMs as a critical component of data augmentation pipeline, detailed in Section 3.1 Specifically, employ
-laude-3-7-sonne--20555 as the foundational model for generating CoT synthesis.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httppp////errreeccccc///eeereres for what should or should not be described.
21

---

Table of Contents in Appendix
Limitations
23
B BEENT
23
CUA Training Strategies
25
D Experiment Setup, Extra Experiments and Analysis
25
D.I Experiment Setup and Training settings
25
D.2
GUI grounding evaluation.
26
Ablation Studies
26
D.4 Error Study
................................
27
D.5 Extra analysis
29
E
OeWorld-Verilted Results Detail
30
Dataset Statistics, Annotation Details, and TgentNet Details
AGENTNET TooL
31
F.111
Tool Features
................................
32
R12
Privacy Protection
................................
32
F2
AGENTNET Statistics and Analysis
33
................................
F2.1
Diversity
33
................................
F2.2
Complexity
34
F2.3
Action distribution
34
Annotation Details: Annotation Strategy, Annotator Source and Cost
35
OSWorld Case Example
37
AgentNet Training Data Example
41
H.1 LI Data Example
H.2 L2 Data Example
43
H.3 L3 Data Example
47
22

---

A Limitations
The scalability and quality of AGENTNET dataset is inherently limited by human annotation efforts. Although AGENTNET TOOL streamlines the data collection process, expanding the dataset beyond its current size would require additional human resources. Also, annotators may have the expertise to complete computer tasks in the most effective approach (i.e. using shortcuts, even write coding scripes). Exploring alternative data sources or remi-automated annotation methods could help address this limitation. Additionally, though OPENCUA strives to collect authentic computer-use data from personal devices, our ethical requirement for explicit informed consent regarding data practices inevitably introduces selection bias. While our dataset maintains high diversity and authenticity, necessarily excludes data from users who, upon understanding the potential risks, opt not to participate. This is a limitation we accept to uphold responsible data collection.
B AGENTNETBENCH
AgentNet Benchmark
Sleph Find song 'Only God Was Above Us"
Instruction History
CD 3
Open Sydney'
Action:
Option
Optice
D1 A
Evaluation Pipeline
Diversed Actions
Multiple Options
Figure 7: Illustration of the AgentNet Benchmark evaluation pipeline
There are several online benchmarks [4 66] that evaluate agent performance in desktop environments. However, these online benchmarks typically require substantial computational resources for environ- ment setup, making evaluations expensive, slow, and difficult to reproduce consistently through time due to their reliance on dynamic environments. Meanwhile, they only provide sparse, high-variance signals (i.e., trajectory-lever accuracy). Another line of works, such ComputerAgentAree evaluate agent's s performance of open-ended tasks on live environments through human preference. To address the limitations of online evaluation benchmarks, benchmark, AGENTNET BENCH, comprising 100 representative tasks selected from the AGENTNET dataset. Tasks were strategically chosen from the center of sub-domain clusters (as detailed in Sec- tion F.2 , ensuring diversity and representativeness across applications and websites on Windows and macOS platforms. Each task was manually reviewed to refine goals and remove redundant actions. Notably, we manually provide multiple valid action options at each step because of the inherent multiplicity of valid actions in computer-use tasks.
(37),
introduce offline CUA evaluation
Benchmark statistics and evaluation dimensions The AGENTNET maintains balanced domain distribution consisting of 38 Work tasks, 29 Daily tasks, 24 Professional tasks, and 9 System & Web Setup tasks. The tasks are split between two operating systems, with 61 tasks from Windows and 39 tasks from macOS. Screen resolutions are categorized into three levels (high, medium, and low) as detailed in Table (note that, for practical purposes, all images in the benchmark are resized from their original resolutions). The distribution of actions within these tasks and additional benchmark statistics are also presented comprehensively in Table
Multiple action choices for enhanced accuracy Previous offline benchmarks [3 20 typically define a single ground-truth truth action at each step. This practice can negatively impact accuracy by disregarding alternative valid choices that agent may reasonably make in real-world interactions. In contrast, in CCEENTNETBENCE we annotate multiple plausible action choices for each step to better reflect real-world decision-make making variability.
Step success rate calculation and action matching criteria To calculate the Step Success Rate (Step SR), we evaluate the correctness of agent actions each individual step using precise matching
23

---

Table 4: Comprehensive Statistics of AgentNetBench
Domain Distribution
Operating System Distribution
Work
38
Windows
61
Daily
29
macOS (Darwin)
Professional
24
System & Web Setup
9
Resolution Distribution
Overall Statistics
High
20
Total Tasks
100
Medium
33
Avg. Steps/Task
17.63
Low
47
Total Actions
2143
Action Distribution
click
850
(67.0%)
doubleClick
19
(1.5%)
rightClick
(1.3%)
press
28
(2.2%)
drag
27
(2.1%)
write
137
(10.8%)
moveToo
45
(3,5%)
hotkey
30
(2.3%)
scroll
(1.4%)
terminate
100
(7.6%)
criteria tailored to different action types. For coordinate-basse actions (e.g., click, doubleClick. moveTo, dragTo, rightClick and hscroll), we define bounding boxes around each action's 's target location; the agent earns the step success point if its predicted coordinates fall within these bounding boxes. For content- or keyboard-basee based actions, such as write, we measure correctness by computing the edit distance between the predicted and ground-truth text; actions like hotkey and press require perfect matches the specified key combinations. For the scroll action, correctness depends on two key criteria: the agent's ss output coordinates must be within the designated bounding box, and the scrolling direction must exactly match the ground truth. Finally, the correctness of the terminate action depends on the agent appropriately terminating at precisely the correct step neither prematurely nor delayed. Considering the distribution of actions (see Table 4. these fine- grained evaluation rules ensure accurate and fair evaluation of agent capabilities in diverse interaction scenarios.
AGENTNETBENCH strongly correlates with online benchmark performance The offline benchmark pri- marily assesses an agent's decision-making capability by evaluating its first-choice accuracy at each task step. While agents can leverage self-reflection to recover from errors made in earlier steps, offline and online SRs should cor-
Fit: 16.813c4.913
8
Data Points
Online Task SR
6
relate under low step budget. Figure [ and Table 3 indeed demonstrate clear positive correlation, specifi- cally following a power-law relation between the online task success rate (under 15 step budget) and the offline step success rate. Therefore, metrics obtained from our of- fline benchmark provide reliable indicator of agent's foundational proficiency and its adaptability to realistic, ceource-constrained online tasks.
4
0.4
0.5
0.6
0.7
Offline Step SR
Figure 8: Offline Online evaluation.
24

---

C CUA Training Strategies
Depending on computing budget, dataset availability, and the target model-e.g. building a spe- cialized computer-use agent general-purpose VLM with agentic dapabllitie--diffeeen training strategies may be adopted. Following Aguvis (50), which proposes a two-stage curriculum training (Stage for grounding and Stage for planning and reasoning), we further explore three strategies:
1. Stage only:
When training resources are limited and the focus is on computer-ust agent data, we aim to adapt general open-source VLM into specialized CUA. To preserve general instruction- following ability, we use training mix of 70% CUA data (with a planning-to-grounding ratio of 4:1) and 30% general SFT data. We fine-tune Qwen2-VL with 30B tokens and Kimi-VL-A3B with 20B tokens. Both models exhibit strong improvements on CUA tasks (see Table 7
2. Stage Stage 2: With more resources and diverse data, staged approach yields better performance. In Stage I, we enhance grounding and understanding using grounding trajectories, tutorial-style demonstrations, state-transition caption data, general vision language tasks, and general text SFT data. We trained Swen2.55VL--3 on this mixture for 40B tokens. In Stage 2, we shift focus CUA planning, using 45% planning, 20% grounding, and the rest general data. This results in OPENCUA-32D which achieves substantial gains in both grounding and planning benchmarks (Table Table 5). Using the same strategy, we trained VLL but with more data to build our OPENCUA- 8k trajectories rolled out in an Ubuntu environment using o3 []8]+ [46]. We designed the CoT to emphasize the most effective information cipeccatoo-esspecilllly reflection and placed this data in Stage transform the model's CoT into a more efficient, information-dense format. this way, the model first learns rich behaviors (reflection, planning.... in Stage and then leverages them more effectively in Stage 2.
In addition to more annotated trajectories, we also used
3. Joint training: To build general- purpose VLM with strong CUA abilities, we adopt joint training across domains with balanced data mixing. Given the complexity of multi-image trajectory data, we train these samples for 3 epochs to ensure sufficient learning. Based on Sive2.55VLL77H we train model on 200B tokens budget, maintaining a data ratio of 20% planning, 20% grounding, and 60% general. The resulting model, OPENCUA-7B. achieves state-of-the-art performance among 7B-scale open-source CUAs, achieving 27.3% success rate on OSWorld Online Evaluation Platform.
D Experiment Setup, Extra Experiments and Analysis
D.1 Experiment Setup and Training settings
All models are trained on the Kimi Team's infrastructure with the Megatron framework and Deep- Speed (ZeRO-3). We employ three training strategies:
1. Stage-2 only. OPENCUA-QWEN2-1
and OPENCUA-A3B share a configuration of sequence length 32,768. learning-rate 2 x 10-5, weight-decay 0.1, and global batch size 384 (512 in ablations) on 96 x A100GPUs. They are trained on 18k Win&macOS 10k Ubuntu trajectories. SHENCUA-QWENT 7B runs for 3,400 steps (about 45 h) after a 400-step grounding warm-up; OPENCUA-A3B runs for 2,000 steps (about 10 h).
2. Stage-1 + Stage-2. OPENCUA-32B is first pretrained on 35B tokens of general text, vision, and grounding data (batch 3,584, LR 3 x 10-5 224 A100). We take the step- 1200 checkpoint. Stage-2 then continues for 60B tokens on trajectory + general + grounding data (batch 512, LR 2.5 X 10-5, 128 x 100) using 18k Win&macOS + 20k Ubuntu trajectories. The final model corresponds to step 4,700. For OPENCUA-72B, we trained Qwen2.5-VL--77
using more data. Stage contains 250B token data (batch size 600, LR 2.5 x 10-5 decay to 1.5 x 10-5, 5 480 x A A100). Stage 2 contains 16B token (batch size, LR 1.5 x 10 decay to 2 x 10-6, 480 x A100).
3. Joint training. OPENCUA-7B is trained end-to-end on the full data mixture for 200B tokens (18k Win&macOS + 20k Ubuntu trajectories) with batch 512, peak LR 2.5 x 10-5 (min LR x 10-6), decay tokens 200B, on 128 x A 4100 for eight days. The best checkpoint is at step 14,600.
25

---

Table 5: GUI grounding results.
Model
OSWorid-G
Sereenspot-Pre
Screenspot-V2
UI-Vision
Qwen2.5-VI.
31.4
27.6
88.8
0.85
Qwen2255V1L-322
46.5
39.4
87.0
UI-TARS-72B
57.1
38.1
90.3
25.5
OpenCUA-A3B (Ours)
48.6
28.5
91.4
OpenCUA-Qwen2-7BB (Ours)
45.7
23.7
88.5
OpenCUA-7B (Ours)
55.3
50.0
92.3
29.7
OpenCUA-32B (Ours)
59.6
55.3
93.4
33.3
OpenCUA-72B (Ours)
59.2
60.8
92.9
37.3
D.2 GUI grounding evaluation.
Table 3 and Table reveal several key findings:
1. OPENCUA-72B and OPENCUA-32B rank the first. OPENCUA-72B and OPENCUA- 32B are the best-performing agentic model all mainstream GUI-grounding benchmarks. OPENCUA-72D achieves 60.8% on ScreenSpot-Pre and 37.3% (SOTA) on UI-Vision. Their advantage stems from (i) the substantially larger grounding corpus used in Stage-1 training and (ii) its larger 32B parameter scale.
2. Joint-training boosts OPENCUA-7B, OPENCUA-77 also scores competitively because large-scale grounding data are injected during joint training.
3. Pixel-budget advantage of Qwen2.5-VL. Qwen2.5-VV based models uses a higher max- max-iixels limit (12.845,056 829,440 for Qwen2-VL and Kimi-VL-A3B), yielding stronger results on high-resolution ScreenSpot-Pro, OPENCUA-322 achieves 55.3% on Screenspot-Pro and OPENCUA-7B achieves 50.0%.
4. Grounding alone BHENCUA-QWEN2... and OPENCUA-A A3B on OSWorld-G and ScreenSpot-V2, OpenCUA models achieve far higher success rates on the full OSWorld benchmark (OPENCUA-A3B 19.9% and OPENCUA-QWERT
enough. Although Qwen2.5-VL-322 matches or even surpasses
23.0% shown in Appendix 7 This shows that solid grounding is necessary yet insufficient for realistic agentic tasks in the environment; high-level planning and reflective reasoning ultimately drive reliable task completion.
5. Domain correlation. Because OSWorld-G [46] is collected in an Ubuntu environment, its scores correlate more closely with the online OSWorld evaluation than with ScreenSpot- and Screenspot- V2.
D.3 Ablation Studies
L2 reasoning format achieves the best inference performance. Note that we trained the models with mixed reasoning format (L1, L2, L3, see Section 3.I. D. In this part, we explore which type of thinking format is the best at inference. We test OPENCUA -AAA-2-RN2-7 and OPENCUA-A3H on OSWorld 15 steps. As Table using the I.2 format, the performance higher than L1 and L3. This result is actually different from the conclusion from previous work (50 30 that L1 outperforms L2. We think this because our L.2 CoT has higher quality (e.g.. planning and reflection), which can help the model make better decisions. On the other hand, L3 underperforms L2. By case study, find that when model describes the information the screenshot, there tend be many elements irrelevant to the task or the next action, which may mislead the model. In summary, our results show that the right choice of high-quality, non-redundant reasoning can leverage VLM's language reasoning capability improve the agent performance.
Using a moderate number of visual history images and concise textual history yields the best trade-off between performance and efficiency. We ablate history representation from both visual and textual perspectives. For vision, we vary the number of history screenshots (1, 3, 5) and fine- tune Qwen2-VL--B on trajectories. As the World results shown in Figure using multiple screenshots substantially improves performance single-image inputs, GUI agents rely entirely
26

---

Agent data
Agent data wio
Table 6: Ablation results on OSWorld for different Chain-of-Thoughh (CoT) settings.
World SR (%)
20.1
Ablation
CoT Variant
SR (%)
20
L2
13.1
CoT Mixture
Mixture-CoT
18.5
18
17.6
Short-CoT
11.5
Reflective Long CoT
Advanced-Coo
15.3
LI
16.9
16
15
50
Test-time Reasoning Format
I.2
L3
17.6
Max Steps Allowed
Figure 10: General text data ablation.
on vision for observing state changes. However, increasing from 3 to images yields marginal gains while incurring 3K more context tokens and delayed convergence, suggesting diminishing returns.
On the textual side, we compare LI and L2 history under the same 3-image setting. In Figure L2 history offers no benefit and may introduce hallucinations that distract attention, while also reducing training efficiency. Hence, we adopt L1 CoT images as the default setting.
Training with a mixture of CoT formats outperforms using only L, reasoning. Since our best performance is from L2 CoT inference, and L3 and L1 is lower than L2, we did an ablation only training the L.2 data instead of
10
OSWorld SR (%)
the mixture of L1, L2, and L3. We use the same recipe our OPENCUA- SEEUUA-QQE2---E CoT data with L2 data. As the OSWorld result in Table 6 the model trained on L2 data using the same steps as
7.5
but only replace
mixture
6.5
OPENCUA- SAAACUA-QWEN2-77 BB,, but the performance drops to
L.2 (Thought)
13.1, which is aligned with the conclusion of Aguvis [50].
4
Ll (Action)
[mmg
3 Img
Img
General-domain text data provides a positive effect to agent performance. As we mentioned in Section 3.3 used 35% general text data in main experiment, we also use the same agent data without the text data to fine-tune Owen2--V-7BB with grounding warm-up stage for 2400 steps (approximately the same amount of agent data tokens) ablate its influence. According Figure TO] the general text data slightly improves model's ' agentic performance. Therefore, adding text data from totally different general domains doesn't t impair the agent model's performance, on the contrary, helps improve the performance. We think the reason that general text data may help agent model's generalization and instruction understanding.
Screenshots
Figure 9: Effect of history representa- tion: LI (Action) benefits from more screenshots, while L2 (Thought) at 3 im- ages lags behind.
Reflective long CoT significantly boosts performance by improving error correction. To understand the effect of reflective long CoT (Secion 3.I.. we do an ablation study Qwen2-VL-7B with 14K Win&Mac and 3K Ubuntu trajectories. Without reflective long CoT, the CoT reduces to that reflective long CoT improves the performance from 11.5 to 15.3. Since the reflective reasoning focuses on error correction, we conjecture that the improvement comes from improved seff-correction capability.
that used by Aguvis [50]. In Figure I
D.4 Error Study
After inspecting our online evaluation results, we group the failures into the following categories:
1. Insufficient task knowledge Foundation models may lack domain-specifer GUI knowledge the procedural know-how to finish task. Fach application has its own UI conventions and operation logic, and some tasks require specialized skills (c.g., spreadsheet formulas).
27

---

Task: the "Feanures" slide, insert table with and columns
Step Clikk on Thlamber rowe" Inpue feed selected editiig
Kérrn
Step 9 Long CoT
Mode:
misalonly dick
Reflection
the wronu place
changee tee
can see that we've made progress setting up the tabre for the Festuree slide The number of oolumns conceclly sut to whsch al aligms with our goal. However, the number of rows is currently set to 1, but we
Lineradd of5
from to
need 5 rows according to our gool.
Memory
Looking previour us actions, ready:
Step 9: Click and drig aoress the "I'i
Navigated
the stnntt
the
NNmber rows fold select is for replacement.
selected "Table..." 5. Set the number of rows (though need change this
Opened Insert
See the number cotirme
Reasoning
The curment state show that welre in the Inourt Table dieloo with the number rovs field acivve and showing "t" Bince we need rows insouad of we need change this value. There are a fow ways to do this:
We could click the incresse sutton mull times inoemeet from
We cicade delete the curment valua and type '5" directly
could sclect current
replace
Scep 10: type '5" replace selected the Number of rawa Sold
Correctl
resolved formani
Looking selecting
the fiekd,
mistake
movemennt crgg openitin would highlight the the feid making ready replacement
Prediction
Aftor selecting the *1" the next logiiaal stop would be to type "5" replace it, and then click the OK button create the table with our desired dimensions (5 rows and columne).
Criel
Figure 11: Reflective Long CoT Example: Before predicting the actual action, the model first reason according to the history and current action with reflection, memory, rean&nngg and prediction in the COT. The model identifies the former mistake and correct it in the later steps.
Example 1: "I have lookup table for the officers of each branch. Please fill the second
table using VLOOKUP." The agent does not know the VLOOKUP function and therefore fails.
Example 2: "Fill all the blank cells with the value in the cell above." 1 The agent does not know the bulk-fill feature and instead edits cells one by one.
High-precision grounding errors Tasks that demand pixel-accurate actions frequently fail.
Example: "Change the 2 in 'H20' subscript." The agent must precisely drag-select only the "2," but often selects extra characters. Such fine-grained, letter-level grounding data are also hard to synthesize.
3. Action repetition When an incorrect action has no observable effect, the agent may keep predicting the same incorrect step. Occasionally it recovers, but often it loops indefinitely.
Ternination misjudgment Sometimes the agent fails to notice that the task is already complete and continues acting, causing failure due to excessive extra actions. In other cases, it incorrectly assumes success and terminates prematurely.
Pass@16
World SR (%)
40
35.57%
38.60%
Pass@0
32.96%
Pass:44
5. Long-horizon task failures OSWorld includes tasks requiring >30-50 gold actions. Maintaining coherent context over many steps remains chal- lenging. For example: "Organize my desktop by placing academic papers in 'Paper_ saaperrraddigg,.. coding projects in 'Projects', , and everything else in "Miscel- lancous'. For files without clear names, determine the category by content."
Pass@1
30-
20
18.50%
19.00%
20.10%
15
50
Steps
6. Insufficient error perception and recovery Al- though the agent can detect some mistakes and reflect, the agent is still not good at perceive error as human does. For example, high-precision edits still pose problems. It may insert text one character off yet judge the action correct, and it often lacks reliable strategy to undo and retry after mistakes.
Figure 12: OSWorld @ass@N performance
curves of OPENCUA-QWEN2--7H tempera- ture=0
28

---

D.5
Extra analysis
Agent model is not robust: small variance in the environment affects the task result. As illustrated in Figure 12 OPENNCA--WENN-2 's OSWorld performance (Pass@N) under tempera- ture=0 exhibits sigmificatt outcome divergence despite nearly identical initial states-with only minor variations (e.g., system date). The curves for Pass@ 16 (38.60% SR at 50 steps) and Pass @ (20. 30%10% SR) demonstrate a >18% absolute gap, highlighting how minimal initial perturbations propagate into starkly different trajectories. This underscores the model's ' sensitivity to initial conditions even in deterministic (temp=0) settings, suggesting that seemingly trivial factors (e.g., temporal context) may critically influence multi-step reasoning.
29

---

E
OSWorld-Verified Results Detail
Table 7: Evaluation results of OPENCUA MODELS on OSWorld-Verineee OPENCUA-32B and
OPENCUA-7B have been for 3 turns, Pass success rate is also calculated.
Setting
#Step
Suce. Rate
Cale
Impr
Writer
Chrome
VLC
Multi
Total
Turn
39.03%
7.95
12.96
13
19
13
25
12
13
141
Turn
44.89%
12.86
27.55
15
21
14
27
7
11
21
161
OPENCUA- PRR-UUA--B-PREVIEE
46.08%
16.30
17
25
26
14
19
166.
Turn 2
100
43 35.85%
17.70
41.25
18
20
13
23
5
14
19
16
22
158
Torn
45 35.05%
17.38
42.25
15
23
16
162
Avg.
100
44.99%
17.13
40.92
16.67
22.67
13.00
24.00
6,33
8.67
14.67
19.33
16.00
20.67
162,00
G
100
53.02
19.01
39.02
20
29
155
27
19
28
192
Turn
7
14
17
10
10
Turn 2
30.56%
7.34
12.85
5
15
8
21
13
9
110
OPENCUA-32B
Turn
30.28%
7.31
12.67
7
14
18
13
8
109
Avg.
29.71%
7.33
12.77
6.33
14.33
9.00
18.67
4.33
6.67
12,00
14.33
12,00
9.00
106.67
@
37.34%
7.83
12.85
9
19
25
13
12
135
Turn
33 83.8%%
10.40
18
16
122
Turn 2
33.43%
10.53
24.62
7
16
10
19
12
12
120
OPENCUA- 32B
Turn 3
35.28%
11.29
24.42
20
21
14
14
127
Avg.
24.20%
10.74
24.28
6.33
18.00
9.33
20.00
4.33
8.00
12.33
18.00
12.67
14,00
123.00
@ass@3
45.58
122.33
24.34
11
22
12
29
15
165
33
12.86
15
16
12
17
122
Turn 2
100
35.00%
10.86
29.90
8
18
8
19
5
11
17
126
OPENCUA- 32B
Turn 3
100
35.75%
11.13
29.47
7
20
20
15
12
128
Avg.
100
34.88%
11.62
30.63
8.67
17.67
8.33
18.33
5.67
7.00
12.67
17.33
14.33
15.33
125,33
Pass
45.10
13.02
12
23
15
163
26.
7.70
12.73
5
17
17
Turn 2
23 89%
6,97
12.77
4
14
7
13
10
7
86
OPENCUA-7B
Turn 3
23.06%
6.94
12.92
4
12
5
15
8
9
83
Avg.
15
24.38%
7.20
12.81
4.33
14.33
6.00
15.00
4,00
6.67
8.33
11.67
10.00
7.33
87.67
Pass
30.74%
19
Turn
28.81%
10.06
24.14
7
13
18
11
13
104
Turn 2
27.73%
8.93
24.77
5
17
18
10
9
99
OPENCUA- 7B
Turn
27.93%
8.12
24.34
7
16
17
9
100
Avg.
50
28. 16%
9.04
24.42
6.33
15.33
7.67
17.67
4.67
6,33
10,00
11.33
10.67
11.00
101.00
Pass
35.75
20
129
Turn
100
27.30%
9.79
28.71
4
16
18
10
9
98
Turn 2
100
26.04%
9.68
27.55
5
13
18
10
10
94
OPENCUA-7B
Turn 3
100
26.67%
9.61
27.20
6
16
14
10
96
Avg.
100
26.67%
9.69
27.82
5.00
15.00
6.33
16.67
5.00
6,00
10,00
12,33
10,33
9.33
96.00
Pas
36.48%
11.73
7
22
10
13
1.32
Turn
16.90%
7.66
12.36
1
9
12
4
5
OPENCUA-AAR
Tur
19. 99.94%
9.10
21.13
1
11
10
9.
6
72
Turn
100
17.73%
10.77
26.60
3
12
12
2
3
2
64
Tum 15
19.94%
7.25
12.74
4
11
8
9
5
4
10
12
3
72
OPENCUA QWEN2-7U
Tum
20.61% 16
8.54
21.28
3
12
10
15
5
2
4
74
Tur
100
23.06%
9.73
26.19
5
14
10
12
6
6
83
30

---

F Dataset Statistics, Annotation Details, and AgentNet Tool Details
E1 AGENTNET TOOL
Efficient and accurate annotation is essen- tial for collecting high-quality computer- use agent data, yet existing tools sup- port natural, cross-platform task recording by non-technical users. To address this, we developed user-friendly annotation tool that streamlines the collection and verifica- tion of computer-use demonstrations (Fig- ure D3. runs on annotators' personal com- puters and records demonstrations in the background, capturing: (1) screen videos, (2) mouse and keyboard signals, and (3) Figure 13: AGENTNET TOoL annotation and verifica- accessibility trees (Axtree). These data are tion.
Tnjeperr
Vandi
HTHL
then processed into state-action trajectories (see details below in Section F. allowing annotators to review, edit, and submit demonstrations along with task instructions describing the overall goal. Former works require the annotators to demonstrate "gold" trajectories with all-correct steps, but this actually limits model's capability to detect and recover from errors. We believe that annotation error is not all bad, as long requirement of all correct actions. Our implementation leverages several established tools: mouse and keyboard input tracking is based on DuckTrack (36) and OpenAdapt [27]: screen recording utilizes OBS Studio [26]; and accessibility tree (Axtree) parsing follows the OSWorld framework Additional implementation details can be found in Appendix H.I.I
can identify and utilize them (see Section 3.1 so we relax the
Annotation pipeline We designed our data collection with two key goals: diversity and complexity. Annotators were provided a curated list of around 200 applications and websites spanning various domains and were encouraged to demonstrate complex workflows involving professional features or multi-app interactions. Tasks were required to have more than 15 steps; those with <5 steps were rejected. To ensure wide coverage and real-world authenticity, crowd-sourcing platforms and annotation companies. All annotators signed consent forms, and we use multi-layer privacy protection mechanism to safeguard user data (Appendix (...ndiiiF........ To study model generalization, we split data into Windows/macOSS and Ubuntu, ensuring no overlap with OSWorld tasks prevent data leakage. All tasks were manually verified and labeled as rejected, ok, good, or excellent based on goal clarity, diversity, and complexity. Other annotation details are provided in Appendix F3
recruited annotators from both
Constructing compact state-action trajectories Raw demonstrations consist of high tigh-freeuency screen recordings and fine-grained interaction signals (mouse movements, clicks, scrolls, key presses). A typical task can produce thousands of low-level actions that are too dense and inefficient for training. To address this challenge, we developed techniques including action reduction and state- action matching to construct compact state-action pairs (st, ag). (1) Action reduction: We developed a arulebbase method to compress and reduce these dense action signals into a smaller set of meaningful actions while preserving essential action information. We first compress atomic signals into higher- level operations. Mouse move events are treated as preconditions for clicks or drags, and only their start and end positions retained. Scrolls accumulated wheel counts. Consecutive key presses are merged into text input strings, while modifier combinations (c.g.. CTRL+C) are abstracted into hotkey actions. We also combine common multistep gestures such as drags or double-clicks. This process yields a streamlined action sequence aligned with the pyautogui action space, as shown in Table (2) State-action matching: To pair each action with a representative state s,, we extract keyframes from the screen recording that capture the system state immediately before the action occurs. However, naively aligning keyframes to action timestamps of mouse clicks risks leaking future information; e.g., the mouse may already be positioned over a button, making the prediction trivial. To address this challenge, for mouse clicks, we backtrack to the beginning of the mouse's pre-movement phase and search backward to find the last visually distinct After the final action, we append a terminal frame along with a corresponding termination action.
merged into single-directionll actions with
31

---

F1111 Tool Features
Action Reduction: We use tools like pynput to capture users' atomic actions. These atomic actions are then reduced to semantically meaningful actions, such as 'click', kee_presss, kke_releasse "type', 'drag'. 'move', and 'scroll'. This reduction enables models to more effectively learn from human demonstrations and allows annotators and verifiers to understand trajectories more easily.
2. A Tree and HTML Processing: To obtain textual representations of observations, implement efficient fetching and processing mechanisms for accessibility (al trees and HTML. For the al Iy tree, we apply pruning rules select only essential element attributes, ensuring the fetching process for each tree takes minimal time. For HTML, we develop a browser plugin that automatically captures the HTML structure of websites.
Element Localization: To help users verify the correctness of their actions, we extract text associated with click locations. Using the ally tree HTMI,, we fetch the bounding box most likely clicked and extract textual information from it. If the extracted text insufficient, we leverage GPT predict the semantic information of the clicked element.
Trajectory Visualization: We design a user-friendly interface to ensure seamless annotation experience. For each action, we display its description, corresponding video clip, and the al ly tree. Additionally, we provide the full video the entire trajectory for better context.
5. Verification and Administration Systems: To ensure the quality of the collected data, we develop verification and administration systems that streamline the process of validating annotations and maintaining dataset integrity.
F.1.2 Privacy Protection
We implemented multi-layer privacy protection framework in our data collection process. First, annotators must agree to a consent form that clearly states the scope of data collection, including screen recordings, actions, and system information. The form explicitly prohibits recording private or sensitive information. The tool is designed with privacy-first principles: no data is transmitted to servers without manual upload by annotators, and annotators videos, actions, and accessibility tree structures) before submission. We further ensure privacy through two-stage verification process: manual review by internal team members during task verification, and automated examination of the task trajectory using GPT-4o during post-processing. Tasks containing private information rejected immediately.
review all collected data (including
1. GPT-Based Privacy Analysis
The system loads task descriptions and step-by-step user actions (Observations, Thoughts, Action Descriptions, etc.) from JSON. These records provide details of users' intent, the interface elements users interacted with, and any textual or visual cues relevant to the task.
Data Ingestion:
GPT Inference: The script calls OpenAl's API with carefully structured prompt, requesting GPT to produce a privacy classification in one of four levels: None, Low, Medium, or Hlgh. By passing the user's detailed action steps and observations GPT, the system gathers a structured output that includes an explicit privacy_sensital tissittivttyy label.
GPT accelerates the classification process, effectively scanning large volumes of user trajectories with consistent logic. It can identify and label potentially sensitive fields or behaviors with minimal human oversight. However, GPT's inherent reliance keywords sometimes causes it to mark scenario as High, even if the user simply viewed (but did not fill) a password field. Such errors underscore the necessity of post-processing or additional contextual checks distinguish between potential versus actual private data entry.
Human Verification Human reviewers examine the same recordings or textual records to determine whether genuinely confidential information address mentioned, human reviewers might label as Low or None. direct password or banking details appear, human reviewers assign High.
was entered. For example: If only an email
32

---

Business Analytics
Cloud Infrastructure
Creative Design Multimedia
Development & Engineering
E-commerce & Travel
Education & Research
Entertainment Lifestyle
Office Tools
Operating Systems Utilities
Others
Media
Task Management Collaboration
Web Tools Internet Utilities
Figure 14: Clustering and t-SNE visualization of all task domains.
In "false alarm" cases (for example, password field is displayed but left blank), humans typically label them as None. This nuanced reasoning often yields higher precision comes at the cost of increased labor and time. Furthermore, variability in human judgments may occur if guidelines are not strictly enforced.
F.2 AGENTNET Statistics and Analysis
F2.1
Diversity
Task Domains We categorize the collected tasks into 4 main domains and 11 subdomains based on their topics, involved applications and actions in the tasks (Figure 3). Table g lists representative applications for each domain. To label each task trajectory, classification by representing each task using the task instruction and L1-level COT. We manually examined 200 tasks randomly and the classification accuracy is over 96%. We then embedded the task trajectories using OpenAl's sext--embedddnng---aee
leveraged GPT-4o complete the
model and visualize them t-SNE visualization in Figure 14 Interestingly, the layout mirrors typical computer-usage patterns: for instance, Office Tools cluster near Business & Anal-Ana, while E-commerce & Travel sit close Social-Media Connect on the opposite side of the map. Finally, we chose 100 representative tasks around the cluster centroids to form our offline benchmark, AGENTNETBENCH
Domain
App/Web
E-commerce Travel
Amazon.com com, Booking.com
News, Entertainment & Lifestyle
Spotify, Netflix
Social Media & Communication
WhatsApp, Instagram
Office Tools
Microsoft Office, Google Docs
Task Management Collaboration
Zoom, Gmail, Slack
Creative Design & Multimedia
Photoshop VSCode, PyCharm, Git
Development & Engineering
Knowledge Discovery & Research
Google Scholar, ResearchGate
Data Analysis, Business & Cloud
Tableau, Power BI, AWS
Web Tools & Internet Utilities
Chrome Extensions Finder, Activity Monitor
Operating Systems & Utilities
Table 8: Example App/Web by Doman
33

---

Applications and Websites Applications and websites are tracked using the AgentNet Tool. Specifi- cally, application names are captured by recording the process name when a user opens an application. while website URLs are recorded through our browser plugin. Discrepancies in process names across different operating systems and different versions are resolved using GPT, achieving an accuracy of up to 83% with this combined method. Based on these results, we analyzed the distribution of the dataset across applications and websites. Web browsers account for a significant portion of the dataset, nearly half of the data involves web applications. The results indicate that the dataset encompasses over 140 applications and 190 mainstream websites.
F.2.2 Complexity
Our collected tasks average 18.6 steps per task. We evaluate task complexity across five dimensions:
muldi-applicaioon/eesste usage, professional knowledge requirement, uncommon feature usage, repetitive simple subtasks, and logically coherent long sequences. Based GPT-4o analysis, 30.6% tasks require multiple applications/websiicee
12.9% involve professional knowledge, and 12.9% use 1-10 scale, where represents basic operations like file opening, and 10 indicates complex tasks requiring multiple steps, domain knowledge, sophisticated reasoning. The complexity distribution is shown in Figure [5 It can seen that most the tasks have a medium or high level of complexity.
uncommon features in Figure T6 Then we asked GPT to rate the complexity of tasks
6,599
6,000
Count
4,000
2,643
2,000
1,358
2
224
3
4
7
Figure 15: Complexity Score
F22.3 Action distribution
Human Action
Action Description
Action
Click
Click at specific position
click(x, y. button)
Middle
Middle click specific position
middleClick(x, y)
Double Click
Double click specific position
doubleClickóx, y. button) tripleClick(s, button)
Triple Click
Triple click specific position
Mouse Move
Move mouse specific position
moveTo(x, y)
Drag
Drag mouse from one position to another
dragTo(x.
Scrall
Scroll vertically or horizonially
scrolldd,, dy) hscroll dy)
Type
Type string of text
write(text)
Press
Press specific key
prese(key)
Hotkey
Perform a combination of keys
hotkey(key key2)
Wait
Wait for few seconds
wait() terminate 'success' 'failure')
Terminate
Bad the task with success or failure
Table 9: Overview of Human Actions and Corresponding Agent Action Functions
Table 10 summarizes the action frequencies across 3 systems. We counted the action distribution of 41,428 computer-use tasks, including Ubuntu 23,797 tasks, Windows 12,431, and macOS 5,200. Click the dominant action-more than 60% on three systems. Hardware and usage patterns drive the secondary behaviors: macOS trackpads lead to heavier vertical/horizontal scrolling and more hotkey use; Windows mouse workflows show higher proportions of right-click and middle-click; and Ubuntu's 's keyboard-centrie, terminal-oriented culture results in the greatest shares of text input.
34

---

Table 10: Action type distribution (%) per operating system.
System
click
doubleClick
dragTo
hotkey
hscroll
middleClick
move'o
Windows
60.03
1.99
4.35
1.52
0.00
0.01
8.95
macOS
57.22
2.20
3.48
2.48
0.46
0.00
9.02
Ubuntu
63.62
3.19
2.30
2.21
0.00
0.00
4.53
System
press
rightClick
scroll
terminate
tripleClick
write
Windows 5.27
1.10
4.51
4.42
0.00
7.85
macOS 6.39
1.01
5.52
4.40
0.08
7.73
Ubuntu
3.78
2.40
2.06
5.89
0.22
9.81
Multi Application Website
Professional Knowledge
Uncommon Features
Thue
True
Thue
12.9%
12.9%
30.6%
False
87.1%
1.5
False
False
Operating System Distribution
Alignment Level Low
Aligned 0.6%
2.0%
Medium
Linux
35.4%
Windows 45.4%
77.2%
Darwin
Figure 16: Distributions of data characteristics: presence multi-application websites, inclusion of professional knowledge, presence of uncommon features, source operating systems and alignment levels.
F.3
Annotation Details: Annotation Strategy, Annotator Source and Cost
Annotation Strategy We first surveyed the most popular websites and applications across wide range of coomains-enteraiinnent office tools, and more. We selected 200 + most widely used ones. Tasks were not pre-assigned, because annotators' familiarity varies (especially with professional tools). We set a limit number for the apps and allowed annotators to choose. Annotators could also brainstorm tasks with provided relevant YouTube tutorials they could explore and create additional tasks.
Annotator Source We recruited annotators from four sources: internal students, external univer- sity students, annotation companies, and crowd-sourcing platform Prolific. Table [ shows the distribution of annotators and tasks. While annotators from Prolific and Company were native English speakers, others were native Chinese speakers. All annotators were required to document task goals in English and try to use English system settings, applications and websites ensure broader applicability. (The numbers in the table are annotated tasks before verification)
Annotation and CoT synthesis cost Annotating 22K tasks takes 6 months. All annotators are part-time. The total annotation cost was about USD 20,000. Annotation speed is roughly ten tasks
35

---

Annotator Source
Accepted Uploads
Annotator Count
Internal Students
4943
38
External Students
5168
135
Prolific
1218
294
Company
2235
72
Company2
3556
51
Company3
1975
14
Company4
5000
30
Total
24095
634
Table 11: Annotation Source Statistics (before verification)
hour. The cost of synthesizing CoT costs USD 0.6 per task average. The total cost of building this dataset is about USD 32,000.
36

---

G OSWorld Case Example
The trajectory outlines the process of installing a manually developed Chrome extension located in the Desktop directory into the Google Chrome browser for testing or usage purposes. Notably, the step labeled in red highlights the agent's capacity for reflection and error correction it initially diverges into an incorrect path, subsequently recognizes the mistake, and successfully reorients itself to proceed with the correct course of action.
Thought: I need to install a Chrome extension that's locsed in the Desktop directory. Looking the state. I'm
Google homepage Chrome. To install extension,
Google
access
Store use
soe that there's ' "Web Store" shoricut on the Google homepage which could be useful. However, also nosice the Chrome menu button (three vertical dois) the top right corner of the browser. This extensions.
would give
Chrome setiings and features, inchuding
Since need to install a local extension file. the most direct approuch would to: Click the Chrome menu (thres duis) 2. Navignte to "More tools" > "Estensions' or direclly "Chrome Store" 3. Upload extension from button
logical action because provides access to all Chrome seitings and features, including extensions management. Clicking this button will open drupdown menu with various optians, including those related extensions. unticipate that after clicking the button. I'll opoions that will allow
top right corner
Step
extensions page Chrome Web Store, where can upload the extension file Desstop.
Action: Click on the thrre dot menu button the right comer the Chrome bruwser window open Chrome
Code: pyautoqui.clkkkt clcc.e--ccckc-eee.e y=0.081)
After clicking on the three-dot memm the previous step. the Chrome mena his socoessfully opened, displaying various options. This exactly what expeeted hagpen, we're making good progress toward pool Looking at the currest state, sce that Chrome menu is now open with options. instal Chrome extension. I need to access the Estensions page or use the Web Sture Based on the pool of installnng extension from the Desktop directory. I should look for the Extensions option this can that there an "Extensions" option the menu, which exactly what noed. Clicking on this option will take me to Chrome's 's Extensions page, where can thes lond an ungacked extension from the Desktop directocy.
Thought:
Google
Chroese extension
the Thsskrop
would click the "Extensions option the Chrome govigate to Extensions management page where umpucked extension. From thene. I'll be able browse to the Deskiop dircciory and select the estension file for installation.
logical
This option
Step 2
Extensions management page list currently installed extensions and options to munage them, including an option to "I and unpacked which I'll nead to use to install the extension from the Deskiop. Action: Click the "Extensions" option the Chrome menu to access the extensions page
clicking
option. expect be
where I'll
Code: peautogui click(y=0).86 867, y6 4006
After clicking on the "Extensioms" option in the Chrome menu, I can see that the has expanded show options related peevious action.
Thought:
expected of
Google
Looking at current siate, can see that there are two main options related extensions: I. "Manage Extensions" This would take me the where can manage currently installed extensions "Visit Chrome Web Store" This would downlead extensions Since our goal is to install Chrome extension that located on the Desktop directory, I need to access the extensions management page firtt From there, I would be able to access the 'Load aupoekedd bunoa which allows installing exteasions
the Chrome Web Store where
rather than from Chrome
Store.
Step 3
The logical next action would be to click on 'Mangge Extensions" since is the opion that will take me the extessions manggemeat where can install the extension from the Desktop. direct
This
achieving goil instilling locally developed By clicking on 'Manage Extensions. espect
taken page shows currently installed extensions and provides options to add now extansions, including the "Load unpacked" option that I'll need use install the extession from the Deskoop disectory.
Action: Click the 'Manage Extensions" opion the Chrome management page.
the extensions
Code: pyuntogui clickrx=0,755 (00.403)
37

---

Thought:
Afier clicking "Manage Extencions" in the previous step, I successfully navigneed the Chroese extensions mazagement page. exactly where need to be to cominue with the installation of the new Chrome extension. Lookingg at the current state. cas sze that I'm on the chrome//lxtension page which shows the list currenily installed cuensions. However, don't any extensions listed which suggesis either there no extensions installed or page
that I've
to be refreshed to show them. To install new estension manmally from a local file on the Desktop, I need to: Access the Chrome Web Store simular interface where can upload or load an estension from tile 2. Navigate
extension file locasion (Desktup) Install the extension
notice there are options available me: are any instuiled extensions I could click on "Keyboard sherccuss to confiquue shoctcuts 1 could click on the "Chrome Wab Store" link to browse the store- could click on the "Chrome" link in and themes the Chrome Store" locally developed exsension,
click "My exsensions"
there
Step 4
"Dsscovrr
Since mty goal is to me to load an extension from a file. The "Chrome Web Store" lisk secus to be the most logical next it would likely provide access to extension management functions including loading local extensions.
option
"Disewer more extensions themes oa Chrome Web Stoce" would take me to the Chrome Wes Store, where might find optioes to access developer tools or lond extensions from local files. This scems like the most logical nest action progress installing Action: Click on "Chrome"
Clicking on the "Choome"
from Desklap directory. the text "Discover more extensions themes
Chrome Web Store'
Code: pynntogui click(x-01155. y-0.284)
Afler clicking the "Chrome" link the Extensions management page. I've boon good stort, but notice that I'm curiently Estensions categeey of the Chrome Web Store, which is for browsing and downlouding estensions from the store, not for installing local estencions. To install Chrome extension the poal), need to access the Chrome Web Store's is developer page or use a difTerent apprunch. Looking at the current state, need to navigote to different section that allows for installing umpacked extensions. can
Thought:
taken to Chrome Webs Store.
local (ocased in Desktop directory as mentioned
the lop-right corser of page Signing in Chrome often necesarr extension developemnnt installation. Additionally, to access develuper options or special section for installing unpucked extensions after signing in. Given the goal of installing locally developed Chnome extension from the Desktop directory. next logical step would be to sign to Chrome. would likely give me access more advanced options for extension installation, including the ability to load an unpucked extension from local directory.
"Sign
might be way
Step 5
After signing in. expect either find a direet option load unpacked extension need to uplond the extemsion from Destiop Signing in necessary step in process of munullly installing a Chrome extension from a local directory. Action: Click on the "Sign In' button in the top right comeer of the Chrome Web Store page.
praulogui click(x=0 y=0.131)
After clicking the "Sign button the Web Store. been redirected Google sign-in page. This mikes sense as many Chrome extessions a signal-in poofile to be installed propeity. The sign-in
Thought:
is asking for Google account credentiass to proceed.
the goal of installing Chrome extension manually from Desitop dircetory. I need to: Acsess the Chooae Weh Store (completed) 2. Sign to Chrouse (in progress) 3. Navigate to the extension file on the Desktop 4. Install the extension manually I'm currently stop 2, where need to sign in to Google account to priceed. However. I nocice might downloaded extension, might local file without fully signing in.
Looking
peogress
complete the
sign-in prncess. Since goal install locally to go back and find an option to load an extession fross
Looking at the browser interface, I see the back button in the top-left corner of the browser. Clicking back butten would
Step 6
the Chrome Web Store page where might unpockod"
alternative options for installing an extension from local file, such as
buntoa that's 's ty typically available when you're in the extessions management page.
The logical next action would be to click the back button return the Chrome Web Store. This would allow
to patentially find
dircet way install the estension
Desittop
siga-in process. After clicking back. expest return the Cheome Web Store page where can look for options related loading local extensions.
without completing
Action: Click the back hutton in the top-left corner of the Choome browser window return to the previous page.
Code: peautogui click(y= D48, (:06666)
Afier clicking the back button. I've successfuulyy returned to the Chrome Web Store puge from the Google sign-in puge. This was the right since signing Now I need to find a way to instll the extemsoon from the Deskiop directory. Looking at the current state of the Chrome Web Stone pape. I need to find an opion that would allow me to load extension from local flc. This functionality typically available through "More" similar opion the Chrome Web Store puge.
Thought:
't necessary
Tean see that there's a three-áxt monu button in the top ri-hight comer the Chrome browser window. This licely where I'll find additional options for managing exsensions, incloding the ability to lod exsension from local file. Clicking this button should reveal dropdown various options. posemially including "More Extensions" would allow me to navigate the Deskiop directory and select the exsension file.
with 'Loud inpacked estension' which
Step 7
The logical nest action to click on this three dot menu bution in the top right corner of the browser. This should open
with additiona optoon relited extensions, which will help estension Action: Click on the three dol menu button in the top-right comeer the Chrome bruwsgr window to access additional browser optices.
me progress toward manually installing
Desktop directury.
Coderpyauogui www.uuiiiiiiiieeeee.e. chicktss 988, y=0.083)
38

---

Thought: After clicking the three- dot in the top-right of the Chrome browser,, dropdown menu his appeared vanous browser options. is exactly what expecting seo, as this mena provides access to Chrome's 's core functionaliy including extensions management. Looking at progress toward the goal of installing Chrome extension manually from the Desktop directory. I've successfully: Opened Chrome
Chrome
Accessed the Extensions deternined wasn't
option Navigated
Sture Attempted to sign
accessary) 5. Opened the Chrome ueem again to access more options New need to find the best way to install the extension from the Desktop, Since I already know the exension file
Desktop, I chould look for option that would allow load aspacked extension sccess Deskiap files.
Looking at the dropdown mesu, can see an "Extensions" option with a submenu indicatoo (>)), This is exectly what I need click next because: It will take me to the Extessions mansgemeent page or provide extension relused options From there. should able to find option load an unpacked extension kading them from local files
Step 8
standard way in Chrome to manage extensions, including
Clicking on the "Excensions" menu stem will likely show me additional options such "Lood unpacked" which would allow nivigute
Desktop select the extension folder. This the most dircct path manually asstalling the extension from the Desktop required. Action: Click on the "Extensions" option in the Chrome mena to access extension management funcrions.
Code:
paantogui click((--0866.. y-0.406)
Thought:
After clicking on the "Extensions" option the Chrome menu, sulmenu has appeared with two options. "Manage Extensions" and "Visit Chrome Web Store". This is ecsefly what need to continue with the installation process. Looking
installing the Chrome extensioon: I. opennd Chrome accessed Estensions went to Chrome Stoee's Extensions page intemppedd to sign but decided to go back 4. upened the Chrome menu again und clicked on Essensions 5. Now I'm socing submenu with extendion-relseed options
prgress
To install
estension the Desttop directury, need to access extension management interfire. The 'Manage Extensions" option exaetly what I need. as will allow me to access the extension instalation functionality. This is where I'll he abll to load an inpacked extencion from local directory: The "Manage Extensions" option the first item in the submenu that has opcion take
Step 9
extencions management page where an impacked extension From there direclory and select the excension file to complete the installation. The logicall nest action is to click on "Manage Extensions" the submene. This will take me the extensions managementt page where proceed with installing the extension from the Deskuup directory. Action: Click on the "Mange Extensions" option in the Chrome menu subenenu.
appeared, Clicking
the option
able to navigate to Desktop
Cude: pynutogai clicktee0 738, y=0.401)
After clacking "Manage Extensiens" the Chrome menu, I've been the Chrcase extensions manggement page. This is exactly where I need to continue with the installation of the new Chrome extension. Looking at pougress toward the gual of installing Chrome extension manually from the Deskiop dicectory: extensions management page 3. Low need to find a way load excension from Desklop The extensices page shows sevena options, including "My extensions" and "Keyhoard sharteuss. but don't see direct option to upload or add extension from a file. Howeves, notice the need.
Thought:
successfully accessed Chrome browser I've navigated
"Developer mode" toggle Developer made typically provides additional options for develogers, including ability to load unpacked exteusions. To install a manual extession, I need to enable Developer mode first. Once Developer mode is enabled, should get additional options that will allow me load an aspocked extension from
top-rghtt
the page. This exactly
Step 10
action click the "Develoger mode" 'oogle switch the top-tight comer of page. will enable developer options and should reveal additonal costrols that will allow me to load the extansion from the Desktop directory. After clicking this toggle, I expeer to new buttons appear that will ler me file.
directory like the Desktop logical
the file system select
Action: Click on the "Developer made" toggle suitch the top-right conner the page entile developer options.
Code: pyantogui clicktx-0.988 0.888 3-0.131)
Thought: elicking the "Developer mode' togple switch, has enccesafully crabled as shown by the blue loggle in the top right corner the page. This has revealed additional funcsionality that wren't visible before, including the "Load unjucked" button which is exacily what need install extension manually Now that Developer mode enabled,
nather than from Chrome Web Store This is perfect for our goal since we need install extension that's 's's locsted in the Desktup directory, Looking al the available options. I can see: 1. 'Load unpacked" button- This what need install local extemsion 2. 'Pack exuension" builin- Used for creating packaged version
"Lood unpacked butlon
install extensions
"Update" button- sidebar navigation showing "My extensions" and "Keyboard shortcuts To install the extension from the Desttop directory. the logical noxt action is to click on the 'Loud unpucked" buiton. browser dislog that will allow select the folder. Chrume files, Clicking the "Load inpucked bution the most appropriate action bocause: 1. directly addresses our goal of instaling locally developed extension 2. It's the standard way install extensions from local files Chrome The Developer mode alteady enabled, which feature
updating existing extensions
Step 11
This will
navigate the Desktop and select extension from unpacked
extension
perrquisstee using
After clicking this button. I expect file beowser dialog to appear, allowing us to navigute to the Destoo and select the exsension folder,
Action: Click on the "Lood unpicked" button open file browser dialog for selecting the extension folder from Desksop.
Code: pyuutogai click0x=00077 0.02. 3=0.179)
39

---

Thought: clicking the "Load unpacked" button, file browser dialug has opened expected. This dialng allows to navigate our goul should be the Desktop directory.
location where extension stored, which according to
Looking at the current stase, can see that the file browser dialng is showing the contents of what appears the home directory, with folders like tmp, Downloads, and visable. However, our goal instal Chrome extension that's located to navigate to the Desktop folder.
Desktup directury. Therefore, need
notice that the Desktup folder conveniently listed in the left navigation panel of the file browser. Since need
the Deskiup find extension files, clicking the Deskoop folder left navigation panel the logical next step. the Desktop directory, where we should be able to find the exacusion files.
change the view show contents
Step 12
Once click Desking, expeect the main panel to update and show all files and foldes: located on Desktop. From there, I'll instailation process.
locate the
folder
complete
The logical next action is to click on the "Desitop" option the left navi gationn panel of the file browser dialog navigate the Desktop dusectary where our exsension located Action: Click on the "Deskäop" option left navigation punel
browser dialog.
Code: prantogui click(x-0.353 0333. y-0.77))
After clicking on the "Desktop" opton in the left novigation panel, the file browser dialog has now displayed the contents of the Desktop disestory. This etucily whit we noaded to pruceed inctaling Chnome
Though:
Looking at current state. can see flat Desktup directory contains two iteme a folder namad "helloFstension" and another folder named" MAAOSX". Since we' te looking to install Chrome extension, the "helloEssemion" fobder most likely the extensioo need install, Chrome extensions typically follow spacific fokler mot, "telloFxtensioo" appears he properly namod extension falder. New that located what appears to be the estension folder, the neu. logical stop would he to select this folder to proceed with the incallation. According the we seed to install a Chrome extension locsted found Tave several uprions this point: 1. Click on the "helkeExiession" folder to select it Click "Select button (which would likely select the corrently highlighted item) Navigate into the folder to confirm contemis
"manifest joon"
Desktup directory, and we's
be extension.
Step 13
install, browses, clicking on folder select most logical next action. This will highlight the folder and prepare it for selection, alter which I can click the "Selec" button to complete the insiollation process.
the "helliEEsiesioon folder the likely candidate for extension already visible
become highlighted. indicating it's selected. Then I can proceed to click the "Seleci" bution load this unpacked exsension into Chrome.
expect after clicking on "hellolEseension folder,
Action: Click the 'belloExvensiun" folder in the file browser dialog select for instaallation.
Code: pysutogui click(s=0) 154, y-0.456
Thought: After clicking on the "hello@ixteasioe" folder in the tile browser dialog, I can see that the folder is now selected. This is etuctly what we needed to do to idemitt the extension install. "bello@itensioo" folder contains newly developed Chrome exteasion.
shows
Desitop directory,
Now that we' ve selected the "helleExtession" folder, the nest logical step is to contimm this selection by clicking the "Select" button in the file browser dialog. This button is prominently top right
displayed in orange
dialog. Clicing butten finalize selection preezed with installation process.
The installation workflow so far has been: 1. We enabled Developer mode in Chrome 2. We clicked 'Load unpacked" open the tile browser 3. We navigated to the Desitop directory 4. We selected the "Nellodxtesssoo" folder
Step 14
Chrome to lood the inpacked extension from the solosted folder After clicking this button, Chrome should anstall the extension and possibly show confirmation message automatically add the browser. expect that after clicking "Setect" bution, the file browser dialog close, and we'll rerum to Extensions page in Chrome where we might see the newly installed extension appear in the list instailed extemsions, or possibly see a notifiction about the successful installation.
final step to complete the installation
the "Salect" bunoo, which
Action: Click "Select" button selection the belloExteasion folder and complete installation process.
top-iight
the file browser dinlog confinu
Code: pyantogui clickt(x- 7188, y-0.358)
Thought: Afier clicking the "Select" bution the file bruwser dialog. the extension installation been returned
prucess hus completed successfully. The dialog closed "Hello Extensions" extension his Exiensions munagement page where we can now see been installed, The extension appears with its icon. version number (1 1.0). and description (' ('Base Level Estension") Looking
asabled Developer mode prugress loward the gual incialing the Chrome exiensiom: page 22. dicked
and ungocked which ohichopened the file browser dhilog v3, navigated to the Destoo dinectory /44 We selected the "bellsfEstemiim" folder v We clicked the 'Solett' button confirm the installation successuulyy installed The has completed successullly!
The extension has installed and appears in Extensions management page. enabled (the toggle switch is turned on) If we wanted configure any specilic settings for this extension, we could click on the "Details" button which would likely show more options for the extensiun Since poul
extensoon
Step 15
instal the extension manually from the Desktop directory, and accomplished, Ohere's further action needed for the installation task. extension now iv walaable for use Chrome.
Action: Temminate the prucess with success status as the extension installation is complete. Code: computerterminen iermintttottattee sucess") )
40

---

H
AgentNet Training Data Example
H.1
L1 Data Example
LI data includes CoT-LI(Action) in its inner monologue. The assistant's final output will be included the loss calculation (green color content).
LI Data Example
"role": "system",
content": "You GUI agent. You given task and a acreenshot
the screen. You need to perform series of pyautogui actions to complete the For each step, provide your response in this format: Action: Provide clear, concise, and actionable the action involves interacting with specific target: explicitly without using coordinates element names when possible (use original language if non-English)
instructions:
- Describe target Specify
Describe features (shape, color, position) name unavailable buttons, identify correctly (minimize, maximize, close) if the action involves keyboard actions like 'press', 'srite', 'hotkey'; repetitive keypresses with expected text outcome for typing actioneFinally. output the action as PyAutoGUI code or the folloving functions:- {"name": "computer triple_click", description": "Triple click the screen",
For window control
Consolldatte Speeciyy
parameters": {"type": "object", "properties"; {"x": "type": "number", "description'. "The x coordinate of the triple click"), "y": ("type": "number", description": "The coordinate of the triple click" },, "required": ["x", "y"]3}- ("name": "computer. terminate", "description": "Terminate the current task and report its completion status", "parametera": ("type": "object", "properties"; ("status"; {"type": "string", "enum": ["success", "failure"], description"; "The status the task"}}, "required": ["status"]}""
"role": "assistant",
"content": "# Step
## Action: Click on the "Destination Columns document thumbnail to open the document.
Step
12
## Action: Select the word "Travelers" at the beginning of the second paragraph by clicking at the start of the word and dragging the end of it."
14
15
"role": "user", "image": ;;
41

---

},
"role": "assistant", " "content"; "# Step 3: ## Action:Preat Ctrl+C copy selected word Travelers"*
" "role": "user", "image:
10
},
"role": "assistant", "content": "# Step 4: ## IicionnClica on the Google search tab near the top of the browser window."
}.
rroee:: "user", "image":
42

---

}
footnote to that word in "role": "user"
"content": "# Task Instruction:
Could you open the document 'Destination Columns AA,, find the first word of the second paragraph, look up what that word means on Wikipedia, and then add that Wikipedia description the document?
Please generate the next move according to screenshot, task instruction and previous steps (if provided)."
},
10
"role": "assistant", "content"; "s Step 5:
11
## Action: Click on the Google search box where "gpt" is currently displayed to prepare to enter new search query.
13
14
## Code: "2 ppython Apyautogui. click (x=0. 157, y=0. 1229) "}
16
H.2 1.2 Data Example
1.2 data includes CoT-LL2(houuht, Action) in its inner monologue. The assistant's final output will be included in the loss caleulation.
L.2 Data Example
{
"role": "system",
"cntennt"; "You are GUI agent. You are given task and a screenshot of the screen. You need to perform a series of pyautogui actions to complete the task. each step, provide your response in this format:
Thought:
Step by Step Progress Assessment: Analyze completed task parts and their contribution to the overall goal Reflect on potential errors, unexpected results, obstacles
If previous
43

---

## Action: Select the word "Travelers" at the beginning of the second paragraph by clicking at the start of action incorrect, predict logical recovery step Next Action Analysis: actions based on current state considering current state and previous actions Propose most logical next action
List possible next
Evaluate options
Anticipate consequences of the proposed action For Text Input Actions:
Note current cursor position Consolidate repetitive actions (specify count for multiple keypresses)
Describe expected final text outcome Use first-person perspective in reasoningAction: Provide clear, concise, and actionable instructions:
If the action involves Describe
interacting with specific target: target explicitly without using coordinates Specify element language if non-English)
when possible (use original
Describe features
shape, color, position) if name unavailable window control buttons, identify correctly (minimize maximize, close) if the action involves keyboard actions like 'press', 'urite', 'hotkey'; Consolidate repetitive keypresses with count Specify expected text outcome for typing actionsFinally, output the action PyAutoGUI code the following functions:- {"name": "computer. triple,click" "description": "Triple click on the screen", "parameters": ("type"; "object", "properties ": ("x": ("type": "number", "description", "The x coordinate of the triple click"), "y"; ("type": number", "description": "The y coordinate of the triple click"}}, "required": ["x", "y"]}}- ("name": computer tentterttemmmnate... "description". "Terminate the current task and report its completion status", parameters": ["type": "object", "properties"; {" status": {"type": "string" "enum": ["success", failure"], "description"; "The status of the t)sk"}}, "required": ["status"]]]"
For
"role": "assistant", "content": "# Step 1:
## Action:Click on the "Destination Columns-A... document thumbnail to open the document.
11
12
Step
13
the word and dragging the end of it."
14
},
15
{
"role": "user", "image":
44

---

},
"role": "assistant", " "content"; "# Step 3: ## Action:Preat Ctrl+C copy selected word Travelers"*
" "role": "user", "image"
10
},
"role": "assistant", "content": "# Step 4: ## Oetion:Clion on the Google search tab near the top of the browser window."
}
rrole:: "user", "image":
45

---

}
footnote to that word in "role": "user"
"content": "# Task Instruction:
Could you open the document 'Destination Columns AA, find the first word of the second paragraph, look up what that word means on Wikipedia, and then add that Wikipedia description the document?
Please generate the next move according to screenshot, task instruction and previous steps (if provided)."
},
10
"role": "assistant", "content"; "s Step 5:
11
## Thought: After clicking the Google search tab, I can see that the browser has successfully switched
step Google search page. This forward in my task, as need to search for the word "Travelers" on Wikipedia to complete the footnote insertion. Currently, I see that the search box contains "gpt" rather than the word "Travelers" that copied from the document. need to:l. Clear the current search term "gpt" from the search box2. Enter "Travelers Wikipedia" or navigate directly to Wikipedia and search there3. Find relevant information about "Travelers" Wikipedia4. Return to the Google Docs document and insert this information as a footnotelly progress on the overall task so far:1. I've opened the Destination Columns-A" document v2. I've identified and selected the first word "Travelers") v3. I've copied this word to the clipboard v4. I've switched to a browser tab for searching v5. Now I need to clear the current search and look up "Travelers" insert the Wikipedia description as a footnoteLooking at the current state, I need to click directly the search box where "gpt" is currently displayed. By clicking able select with "Travelers Wikipedia" to search for information about Travelers on Wikipedia. This is the most efficient next action it allovs me directly modify the search query without any additional steps. After clicking on the search box,
proceed with my task,
the second paragraph
Wikipedia6. Then need
the search box, I'll be the text, delete it, and replace
46

---

expect the text box, allowing me to the current search term. can then clear "gpt" and type "Travelers Wikipedia" to continue with my search. This will help me find the Wikipedia page for "Travelers" so I can obtain the description needed for the footnote.
be positioned within the
13
14
## Action: Click on the Google search box where "gpt" is currently displayed to prepare to enter new search query.
15
16
## Code: ppython www.oaaruooou..ccccc.e.......... "}
y-0. 1229)
18
H.3
L3 Data Example
L3 data includes COb-e3nO Thought, Action) in its inner monologue. The assistant's final output will be included in the loss calculation.
L3 Data Example
"role": "system",
"cntentt; "You are GUI agent. You are given a task and screenshot of the screen. You need to perform series of pyautogui actions each step, provide your response Observation:
complete the For this format:
Describe the current computer state based on the full screenshot in detail.
Application Context:
The active application
The active window or page
Overall layout and
Menu items visible interface Buttons and controls and toolbars
Key Elements:
Text Dialog boxes or popups
fields and content Error messages notifications
Loading states Other key elements Describe any content, elements, options, information or clues that are possibly relevant to achieving the task goal, shape (if possible)
including their name, content, Thought: Step by Step Progress Assessment: Analyze completed task parts and their contribution to the overall goal
Reflect on potential errors, unexpected results, or obstacles
If previous
action was incorrect, predict logical recovery step
List possible next actions based on current state Evaluate options considering current state and previous actions Propose most logical next action
Next Action Analysis:
Anticipate consequences of the proposed action For Text Input Actions:
Note current cursor position Consolidate repetitive actions (specify count for multiple keypresses)
Describe expected final text outcome Use first-person perspective in reasoningAction: Provide clear, concise, and the action involves interacting with specific target: Describe target explicitly without using coordinates
actionable instructions:
47

---

## Action: Select the word "Travelers" at the beginning of the second paragraph by clicking at the start of Specify element when possible (use original language if non-English)
Describe features For
shape, color, position) if name unaviilable window control buttons, identify correctly (minimize, maximize, close) if the action involves keyboard actions like 'press', 'urite', 'hotkey'; Consolidate repetitive keypresses with count Specify expected text outcome for typing actionsFinally, output the action PyAutoGUI code the following functions:- ("name": "computer. triple_click", "description": "Triple click on the screen", "parameters": ("type": "object", "properties ("x": {"type": "number", "description". "The x coordinate of the triple click"), "y": {"type": number", "description": "The y coordinate of the triple click"}}, "required": ["x", "y"] ("name": computer.teraat terminate", "description". "Terminate the current task and report its completion status", parameters"; {"type": "object", "properties": {" status": {"type": "string", "enum": ["success", failure"), "description"; "The status the task"}}, "required"; ["status"]]]-
}.
"role": "assistant", "content"; "s Step 1:
## Action: Click on the "Destination Columns-A.... document thumbnail to open the document.
11
Step
12
the word and dragging the end of it."
13
},
14
15
"role": "user", "image":
}
{
"role": "assistant",
"content": "# Step ## Action: Press Ctrl+C to copy the selected word "ravelers""
48

---

footnote that word in "role": "user",
10
"image":
}
"role": "assistant", "content": "# Step 4: ## oetion:Clich on the Google search tab near the top of the browser window."
"role": "user", "image":
},
{
"role": "user" "content": "# Task Instruction:
Could you open the document 'Destination Columns AA, find the first word of the second paragraph, look what that word means Wikipedia, and then add that Wikipedia description the document?
Please generate the next move according to the screenshot, task instruction and previous steps (if provided).
10
"role"; "assistant",
49

---

the current state, I need currently "content"; "s Step 5:
12
## Observation:l'a looking at a Google search page where "gpt" has been entered in the search box. The interface shows Google search results for "gpt" with various tabs available at the top (A11, Images, Videos, Shopping, News, More). The search results show information about GPT Multiple search results are visible, including: Information about GPT ChatGPT (from https:// chatgpt .oppt.coom
related technologies.
-omr) com) Multiple websites with relatel content The browser including a tab for "Destination Columns-Al
multiple tabs open at the top
document, which appears to be a Google Docs document that previously being worked search bar the top with the query "gpt" entered. Below the search resulte are some related questions in a FAQ section, such as "What does the GPT stand for?" and "Is ChatGPT free?" among others The Google search interface includes various filters and options to refine the search, including some buttons like Gpt-4 (GPT-4 relevance) and several other filter options.
There's also
## Thought: After clicking the Google search tab, I can now see that the browser has successfully switched to a Google search page. This is a step forward in task, as need to search for the word "Travelers"
Wikipedia to complete the footnote insertion. Currently,
that the search box contains "gpt" rather than the word "Travelers" that copied from the document... To proceed with task, need to:1. Clear the current search term "gpt" from the search box2. Enter "Travelers Wikipedia" navigate directly to Wikipedia and search there3. Find relevant information about "Travelers" on Wikipedia4. Return to the Google Docs document and insert this information the overall task far:1. I've opened the " Destination Columns document v2. I've identified and selected the first word of the second paragraph "Travelers") v3. I've copied this word to the browser tab for
footnoteMy progress
clipboard v4. I've switched searching v5. Now need to clear the current search and look up "Travelers" on Wikipediaf. Then need to insert the Wikipedia description as a footnoteLooking
click
the search box where "gpt" directly
displayed. By clicking on the search box, I'1l be able to select all the text, delete it, and replace it with "Travelers Wikipedia" to search for information about Travelers most efficient next action as it allows me directly modify the search query vithout any additional steps. After clicking on the search box, expect the text cursor to be positioned within the box, allowing can then clear "gpt" to continue with my search. This will help me find the Wikipedia page for "Travelers" so I can obtain the description needed for the footnote.
Wikipedia. This is the
edit the type "Travelers Wikipedia"
search
15
50

---

## Action:Clica the Google search box where "gpt" is currently displayed to prepare to enter new search query.
17
## Code: ". python npyautogui. click 157, y=0. 1229)
19
20
51


Reviews:
[{"review_id": "ABsd5CskAM", "reviewer": "Reviewer_6mQE", "summary": "The paper proposes a framework for creating datasets and training models for computer-use agent (CUA). The framework consists of 1) an annotation tool that records video, actions and accessibility tree when human performs computer tasks, 2) a dataset of 27k computer use examples across different OS and applications (a training set and a held-out set), 3) a pipeline that converts the dataset into state-action examples with chain-of-thoughts reasoning, and 4) a model training recipe for CUA. The authors train two models (3B and 7B) on the collected dataset combined with public datasets, and demonstrated good performance on online benchmarks OSWorld and WindowsAgentArena.", "questions": "Please see the Weaknesses section above.", "limitations": "I did not find the discussion of limitations and potential negative societal impact of the work in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The collected dataset covers a wide range of operating systems, websites and applications, with diverse screen sizes and fairly complex tasks (17.2 steps on average). As the first dataset of desktop trajectories, it would facilitate the research in this domain along with the annotation / data conversion tool, which can be used to collect new data.\n2. The experiments and analysis show that the dataset is useful for training models that obtain good performance on live CUA benchmarks.\n\nWeaknesses:\n1. Other than the new dataset, the novelty of other components is limited as no significantly new methods are proposed. For example, the state-action pair data, the CoT reasoning, and error reflection were already used in previous works like UI-Tars and AGUVIS.\n2. Some important details of the dataset are missing. 1) How are the tasks selected? Are they pre-compiled or created by the annotators? 2) How much time and cost needed for creating the dataset? 3) What are the action types and distributions? Any difference across operating systems and/or applications?\n3. Including errors and recovery in the trajectory makes sense, but no details are given how they are detected and used. The paper points to Appendix E, but I don't see how error steps are being detected and used in Figure 13. Figure 15 gave an example of reflective CoT, but how this is constructed is not clear (e.g., how can we detect the errors in the trajectories?)\n4. There seems no error analysis of the results on the live benchmark. For example, what are the errors for the tasks on which the agent sometimes failed (pass@k < k) or always failed (pass@k = 0)?", "was_revised": true, "final_justification": "I find the data statistics and error analysis provide a lot of useful insights into the dataset and the agent. The SOTA results further prove the usefulness of the dataset, which did take a lot of efforts to create based on the cost details provided by the authors. I think the dataset and the extensive experiments are the main contribution, although authors provided some more argument to differentiate their agent design from previous works like AGUVIS and UI-Tars, but I found those to be minor points.", "rebuttal": "We sincerely thank Reviewer 6mQE for recognizing both the diversity of our computer-use task dataset and its contribution to the research community, as well as our model’s strong performance on it. We also appreciate the reviewer’s valuable questions and discussion points; we address them below and will add these discussions into the paper.\n## Response to Weakness 1:\nAlthough Aguvis[1], UI-TARS[2], and our work all employ structured CoT, our design is more fine-grained and exhibits higher reasoning and reflection quality. Specifically, we introduce a new pipeline that iteratively synthesizes and refines reflective long-form CoT for human demonstrations. Our CoT explicitly covers reflection, planning, memory, and prediction—all important for agent design.\n\n(1) Ours provides richer information than Augivs. In Aguvis Appendix C.1, Fig. 5, the 'reasoning' part is less informative, and in the left example, the ‘thought’ and ‘low-level instruction’ largely duplicate each other with no reflection or high quality reasoning. By contrast, Appendix F of our paper shows our CoT that includes a detailed task plan, state-change understanding, error identification, and correction.\n\n(2) UI-Tars states it adds reflection in Section 4.4, but it does not describe how these reflections are constructed, nor provide data-quality or scale details. We think reflection and state-transition understanding are essential for agents, so we attach them to every step of action.\n\n(3) Evidence suggests our CoT is more helpful for task completion:\n- Aguvis evaluates OSWorld with L1 CoT and observes that L2 CoT actually lowers success (10.3 → 8.1).\n- UI-TARS (Section 5.5, Fig. 8) shows that adding thought reduces Pass@1 performance; only Pass@n improves.\n- In our ablation study (Section 5 Table 5), L2 (thought + action) outperforms L1 on Pass@1 for OSWorld, demonstrating the effectiveness of our richer CoT.\n## Response to Weakness2:\nWe sincerely thank the reviewer for asking these useful details.\n\n(1) Task selection: We first surveyed the most popular websites and applications across a wide range of domains—entertainment, office tools, and more. We selected the 200 + most widely used ones. Tasks were not pre-assigned, because annotators’ familiarity varies (especially with professional tools). We set a limit number for the apps and allowed annotators to choose. Annotators could also brainstorm new tasks with provided relevant YouTube tutorials so they could explore and create additional tasks.\n\n(2) Annotation schedule and cost: Annotating 22K tasks takes 6 months. All annotators are part-time. The total annotation cost was about USD 20,000. Annotation speed is roughly ten tasks per hour. The cost of synthesizing CoT costs USD 0.6 per task on average. The total cost of building this dataset is about USD 32,000.\n\n(3) Action distribution: The following tables summarize action frequencies across systems. Up to now, Ubuntu has 23,797 tasks, Windows 12,431, and macOS 5,200. Click is the dominant action—more than 60% on three systems. Hardware and usage patterns drive the secondary behaviors: macOS trackpads lead to heavier vertical/horizontal scrolling and more hotkey use; Windows mouse workflows show higher proportions of right-click and middle-click; and Ubuntu’s keyboard-centric, terminal-oriented culture results in the greatest shares of text input.\n\n| System | total_tasks | total_steps | total_actions |\n| --- | --- | --- | --- |\n| Windows | 12,431 | 239,767 | 280,174 |\n| macOS | 5,200 | 99,337 | 118,018 |\n| Ubuntu | 23,797 | 386,328 | 401,477 |\n\n| System | click | doubleClick | dragTo | hotkey | hscroll | middleClick | moveTo | press | rightClick | scroll | terminate | tripleClick | write |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Windows | 60.03 | 1.99 | 4.35 | 1.52 | 0.00 | 0.01 | 8.95 | 5.27 | 1.10 | 4.51 | 4.42 | 0.00 | 7.85 |\n| macOS | 57.22 | 2.20 | 3.48 | 2.48 | 0.46 | 0.00 | 9.02 | 6.39 | 1.01 | 5.52 | 4.40 | 0.08 | 7.73 |\n| Ubuntu | 63.62 | 3.19 | 2.30 | 2.21 | 0.00 | 0.00 | 4.53 | 3.78 | 2.40 | 2.06 | 5.89 | 0.22 | 9.81 |\n## Response to Weakness 3: \nWe sincerely apologize that, due to time and space constraints of the submission deadline, we omitted details of the reflective long-form CoT synthesis pipeline from Appendix Section 5. \n\nOur CoT synthesis framework extends the pipelines of Aguvis [1]. We enrich the 'Thought' with more comprehensive agent components—especially state-transition perception and reflection. As illustrated in Fig. 13, the pipeline comprises three modules: reflector, generator, and summarizer.\n- Reflector. For each step it (i) compares pre- and post-action screenshots, (ii) checks whether the action code matches the UI, and (iii) verifies that the generated CoT is consistent. If a step is incorrect or redundant, the reflector explains why and the step is excluded from training. Otherwise, it describes the state changes the action produces.\n- Generator. Conditioned on the full agent context—prior reflections, action history, task goal, screenshots, and action code—it produces structured CoT. To improve grounding of coordinate-based actions, we add visual cues: a red marker at the click coordinate and a zoomed-in image patch.\n- Summarizer. It rewrites vague user-supplied goals into precise, aligned objectives and scores each trajectory for alignment, efficiency, and difficulty.\n\nAll three modules are instantiated with claude-3-7-sonnet-20250219. Reflection enables the agent to spot earlier errors, adjust future plans, and return the task to the correct track; Appendix F shows an example of error identification and correction. Ablations in Section 5, Table 4, show that this module is an important driver of the performance gains.\n## Response to Weakness 4:\nWe thank the reviewer for suggesting an error analysis of the live OSWorld benchmark. After inspecting our online evaluation logs, we group the failures into the following categories:\n\n1. Insufficient task knowledge: Foundation models still lack domain-specific GUI knowledge. Each application has unique UI conventions and operation logic; without explicit exposure, both agents and even humans can struggle.  \n   Example: In the task “I have a lookup table for the officers of each branch. Please fill the second table using VLOOKUP.” the agent does not know the VLOOKUP function and therefore fails.\n\n2. High-precision grounding errors: Tasks that require pixel-accurate actions frequently fail.  \n   Example: In “Change the 2 in ‘H₂O’ to a subscript,” the agent must precisely drag-select only the “2,” but it often selects more characters.\n\n3. Action repetition and failure to terminate: When the agent misjudges a state after an action, it may keep executing (and reflecting on) the same incorrect action. Occasionally it realizes and does a different action to break the loop, but sometimes it loops indefinitely. In some cases, the agent fails to notice that the task is already complete and keeps acting instead of terminating.\n\n4. Long horizon tasks: OSWorld contains tasks requiring > 30-50 gold actions. Maintaining context over so many steps remains challenging.  \n   Example: “Organize my desktop by placing academic papers in ‘Paper_reading’, coding projects in ‘Projects’, and everything else in ‘Miscellaneous’. For files without clear names, determine the category by content.”\n\n5. Error perception: Although the agent model has a certain level of ability to identify errors and reflect, in some cases that requires high accuracy, it is still challenging. In fine-grained text-editing tasks, the agent may insert a word one character off, yet judge the action correct and proceed.\n\nWe compared the tasks in the Pass@n experiment and find that the main difference between different runs comes from these factors:\n\n1. The agent chooses different solutions in different runs. For example, in the task “Re-open the last closed browser tab,” the agent sometimes uses Ctrl + Shift + T (only one step) and other times navigates through the history menu (needs many steps). Agents may fail on harder solutions.\n\n2. Minor omissions or extras. In Chrome or VS Code settings, forgetting to click “Save” (or performing an additional stray click) converts a correct solution into failure.\n\n3. Environment dynamics: Occasional CAPTCHA dialogs, machine variability, and network latency can change the interaction sequence and lead to inconsistent outcomes.\n\n4. Tasks that consistently fail. Most persistent failures stem from either insufficient task knowledge or long-horizon complexity. In some cases, the agent has no idea how to begin the task at all; in others, the task requires numerous delicate, precision actions, and the agent always breaks down in the middle of the task.\n## Response to Limitation: \nWe apologize that due to the limit of the paper length, we have written our Limitations in the Appendix Section A in our submission. \n## Latest scaling results:\nOver the past two months we further expanded both data and models: the dataset now contains 40 K trajectories with additional grounding data, and we fine-tuned Qwen 2.5-VL-7B and Qwen 2.5-VL-32B on our dataset. The 7B model reaches a 27.9 % success rate, and the 32 B model averages **34.8 %**, which is the state-of-the-art for open-source agent models and surpasses GPT-4o-based OpenAI CUA.\n\n| Model | OSWorld |\n| --- | --- |\n| UI-TARS-72B-DPO| 27.1 %  |\n| OpenAI CUA (GPT-4o) | 31.4 % |\n| AgentNet-Qwen2.5-VL-7B | 27.9 %  |\n| AgentNet-Qwen2.5-VL-32B | **34.8 %**  |\n\n[1] Xu, Yiheng, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. \"Aguvis: Unified pure vision agents for autonomous gui interaction.\" arXiv preprint arXiv:2412.04454 (2024).\n\n[2] Qin, Yujia, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang et al. \"Ui-tars: Pioneering automated gui interaction with native agents.\" arXiv preprint arXiv:2501.12326 (2025)."}, {"review_id": "Yte7l7X8Jr", "reviewer": "Reviewer_jJf2", "summary": "This paper presents an open-source data collection system for CUAs called AgentNet Tool, and releases a dataset collected using AgentNet that is large enough to train CUAs called AgentNet Dataset. Two CUAs are trained on AgentNet Dataset with different levels of reasoning trajectories, as well as grounding and general supervised fine-tuning data. A new offline benchmark, AgentNet Bench, is constructed from the collected data and used for evaluation. Extensive evaluations covering various aspects of the models, including online and offline performances as well as reasoning capabilities, are performed with detailed analyses, and show that the models trained on the proposed dataset outperform the baselines.", "questions": "- (typo) On  line 196 “A” should not be capitalized?\n- Were there any architectural considerations you had during your research?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths\n- It presents an open-source data collection system for CUAs, which would contribute greatly to the research community.\n- It presents a dataset large enough to be used to train CUAs, another huge win for the wider research community.\n- The training recipe for the proposed model covers many different considerations and is backed by reasonable justifications, especially the CoT reasoning dataset.\n- The designs of the evaluation experiments are sound and extensive.\n- Analyses of the evaluation results are insightful.\n\nWeaknesses\n- (minor) The paper’s contribution on model architecture is light.", "was_revised": true, "final_justification": "I believe this paper lays an important foundation for research in CUAs, which currently lacks a scalable open-source data collection system as well as a large training dataset. Furthermore, the evaluation experiments conducted in this paper as well as the analyses will be a good reference for other researchers to compare different CUAs in a rigorous and fair manner. My comment about the relatively small architectural contributions was a minor one, and the SOTA results the authors provided to me during the discussion period partially addressed it. Taken together, I believe my original score of 5 (Accept) is appropriate.", "rebuttal": "We sincerely thank Reviewer jJf2 for recognizing AgentNet’s contributions. Our goal is to narrow the gap between proprietary and open-source CUA models by delivering the necessary infrastructure, data, models, and benchmarks to the community.\nOver the past two months we have scaled both data and models further. The dataset now contains 40 K trajectories with additional grounding data, and we fine-tuned Qwen 2.5-VL-7B [3] and Qwen 2.5-VL-32B [3] on these datasets. The 7B model reaches a 27.9 % success rate on OSWorld, while the 32B model achieves an average success rate of **34.8 %**—state of the art among open-source agents and surpasses open-sourced UI-TARS-72B-DPO [4]  and proprietary GPT-4o-based OpenAI CUA [5]. Both models also excel at grounding: the 32 B version scores **55.3%** on ScreenSpot-Pro, surpassing the previous SOTA UI-TARS-72B (39.4%).\n| Model | Size | OSWorld | ScreenSpot-Pro |\n| --- | ---  | ---     | ---            |\n| UI-TARS-72B-DPO | 72 B | 27.1 %  | 39.4 %         |\n| OpenAI CUA (GPT-4o) | —  | 31.4 %  | —              |\n| AgentNet-Qwen2.5-VL-7B  | 7 B  | 27.9 %  | 50.0 %         |\n| AgentNet-Qwen2.5-VL-32B | 32 B | **34.8 %**  | **55.3 %**         |\n\n## Response to Weakness and Question2:\nWe thank the reviewer for asking the model architecture consideration. For open-source consideration, we apply our method on the currently popular open-source models and didn’t make novel improvement on the model structures. But we investigated the effect of model structures and sizes by training on 4 different models including Kimi-VL-A3B[1], Qwen2-VL-7B[2], Qwen2.5-VL-7B[3] and Qwen2.5-VL-32B[3]. Kimi-VL-A3B employs a MoE design, whereas the others are dense models. On the same data scale, the MoE model, despite having more total parameters, underperforms the dense models because fewer parameters are activated. We also observe that Qwen 2.5-VL benefits from a much higher max-pixels budget (12,845,056 versus 829,440 for Qwen2-VL and Kimi-VL-A3B), leading to superior results on high-resolution ScreenSpot-Pro. We appreciate the reviewer’s suggestion and plan to explore more efficient and effective architectures for digital agents in future work.\n\n[1] Team, Kimi, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen et al. \"Kimi-vl technical report.\" arXiv preprint arXiv:2504.07491 (2025).\n\n[2] Wang, Peng, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen et al. \"Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution.\" arXiv preprint arXiv:2409.12191 (2024).\n\n[3] Bai, Shuai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[4] Qin, Yujia, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang et al. \"Ui-tars: Pioneering automated gui interaction with native agents.\" arXiv preprint arXiv:2501.12326 (2025).\n\n[5] OpenAI. Operator, 2025. \n\n## Response to Question1:\nWe thank the reviewer for the careful reading. Because of length constraints we had to merge several paragraphs in the current draft. In the next version, we will promote “Mixture of grounding 196 and general SFT data” to an independent subsection heading; the sentence beginning “A general-purpose computer-use agent …” will then appear as a full sentence."}, {"review_id": "KsqUPwuLm4", "reviewer": "Reviewer_q328", "summary": "This paper introduces AGENTNET, an open-source framework for training and evaluating computer-use agents (CUAs). It consists of several key components, including a cross-platform annotation tool, a large-scale dataset of 27.8K human-annotated desktop task trajectories, reasoning-augmented training with multi-level Chain-of-Thought, and an offline benchmark (AGENTNETBENCH) to enable scalable evaluation. Two fine-tuned vision-language models (Qwen2-VL-7B, Kimi-VL-A3B) are presented and achieve superior performance among open-source 7B models on OSWorld and WindowsAgentArena. All components of the proposed framework including tool, data, models, benchmark will be released, which offers an open foundation for future CUA research.", "questions": "- How well does the dataset transfer to novel domains or unseen applications?\n- Are there any plans to adapt AGENTNET for online interaction settings?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed dataset, tools, and benchmarks fill a major transparency gap in current CUA research and enable reproducibility.\n- The proposed dataset covers real-world workflows across Windows, macOS, and Ubuntu, with realistic errors and long-horizon tasks.\n- The multi-level reflective CoT framework is thoughtful and empirically validated through ablation studies.\n\nWeaknesses:\n- Despite the scale and quality of the dataset, AgentNet-7B shows only marginal improvement over prior 7B models (e.g., UI-TARS-7B-SFT) on OSWorld, raising questions about generalization benefits.\n- The proposed dataset heavily features Windows/macOS environments, while OSWorld focuses on Ubuntu tasks, which may limit out-of-domain performance.\n- The proposed benchmark is offline only, which may not fully reflect performance in realistic CUA deployment settings.", "was_revised": true, "final_justification": "The author's rebuttal addressed the minor concerns I had. I decide to maintain my original positive ratings.", "rebuttal": "We sincerely thank Reviewer q328 for recognizing AgentNet’s contributions—particularly our realistic, diverse dataset and reflective CoT framework. Over the past two months we have further expanded both data and models. We now have 40 K trajectories with additional grounding data, and we have fine-tuned Qwen 2.5-VL-7B [1] and Qwen 2.5-VL-32B [1] on this corpus. The 7B model achieves a 27.9 % success rate on OSWorld, while the 32 B model reaches **34.8 %**—state-of-the-art among open-source systems and surpassing GPT-4o based OpenAI CUA [4]. Both models also excel at grounding: the 32 B version scores **55.3** on ScreenSpot-Pro[2], surpassing the previous SOTA UI-TARS-72B (39.4) [3]. We will incorporate these new results and model checkpoints in the next revision of the paper.\n\n| Model | Size | OSWorld | ScreenSpot-Pro |\n| --- | ---  | ---     | ---            |\n| UI-TARS-72B-DPO | 72 B | 27.1 %  | 39.4 %         |\n| OpenAI CUA (GPT-4o) | —  | 31.4 %  | —              |\n| AgentNet-Qwen2.5-VL 7B  | 7 B  | 27.9 %  | 50.0 %         |\n| AgentNet-Qwen2.5-VL 32B | 32 B | 34.8 %  | 55.3 %         |\n\n## Response to Weakness 1:\nThrough continued data scaling, our model now achieves an average success rate of **34.8 %** on OSWorld, with a highest score of **35.7 %**. This surpasses all open-source baselines by a wide margin and proprietary model OpenAI CUA (GPT-4o).\n\n## Response to Weakness 2:\nThank you for the question about dataset domain effects and model generalization. We have also collected tasks on Ubuntu, and our tool supports that platform.\n\nIn Figure 5 we analyze system-specific training. We fine-tune Qwen 2-VL-7B on (i) Windows & macOS data and (ii) Ubuntu data, then evaluate on three benchmarks: OSWorld (Ubuntu), WindowsAgentArena (Windows), and our offline benchmark (Windows & macOS). Training on a single operating system yields the highest performance on that same system, yet still transfers some gains to the others.\n\nWe additionally evaluate our 32 B model on WindowsAgentArena; it attains an average score of 28.0 (max 30.0). (Manual inspection shows some false negatives caused by environment errors, so the true score is likely even higher.)\n\n## Response to Weakness 3:\nWe fully acknowledge the reviewer’s concern: an offline benchmark cannot mirror every aspect of real-world deployment. We nonetheless chose to build an offline suite—rather than another online benchmark like OSWorld—for three practical reasons:\n\n1. Live environments are highly dynamic, which makes experiment signals unstable. During extended experiments we observed that a single Chrome update could deprecate UI elements and invalidate previous evaluation metrics. Additional variables—login prompts, CAPTCHA checks, fluctuating network latency—may render results collected just two weeks apart incomparable. An offline setting avoids these shifting sands.\n\n2. To mitigate the usual limitations of offline testing, we carefully selected representative tasks from our dataset and manually annotated each step. We drew bounding boxes for every click action, and found multiple feasible actions in each state, to capture the full space of valid solutions. This manual curation increases robustness and reduces the chance of false negatives when agents choose alternative but correct strategies.\n\n3. Because no live environment needs to be spun up, evaluation is fast and reproducible. We can score every training checkpoint, identify the best-performing models, and then proceed to costlier online testing with higher confidence.\n\n## Response to Question 1: \nThank you for highlighting the domain-transfer issue. As we discussed in the response to Weakness 2, in Section 5 Figure 5, we show the effect of domain shift. Our conclusion is that:\n1. There are cross-domain gains. Many low-level GUI patterns—window management, menu traversal, standard shortcuts, drag-and-drop, etc.—are shared across operating systems, applications, and even websites. Training on a mix of domains therefore improves overall robustness.\n\n2. Tasks requiring specific domain knowledge are hard to be completed only by domain transferring. When a task hinges on domain-specific knowledge—e.g., triggering an uncommon Excel formula or operating a professional CAD tool—generic GUI skills alone are not enough. The agent needs additional task-specific demonstrations or fine-tuning.\n\n## Response to Question 2: \nFor the reasons outlined in our reply to Question 1, we initially designed AgentNetBench as an offline benchmark. Nevertheless, we fully agree that online evaluation is essential, and we have begun experimenting in that direction. The difficulty of building an online evaluation benchmark lies in the evaluation method. OSWorld’s current protocol relies on hand-written evaluation scripts for open-source applications. Extending this approach to a broader range of real-world, closed-source, or rapidly evolving applications is often infeasible, because reliable scripts either do not exist or would require prohibitive maintenance. \nCurrently we did some attempts to use a general foundation model to evaluate the agent’s result. We replaced OSWorld’s official scripts with two large language models—GPT-4o and OpenAI o3-mini—and measured their agreement with the ground-truth scripts. .\n\n| Evaluator | Accuracy | F1 score |\n| ---       | ---      | --- |\n| GPT-4o    | 78 %     | 0.71 |\n| o3-mini   | 85 %     | 0.78 |\n\nWe are investigating hybrid schemes that combine lightweight domain-specific checks with VLM-based reasoning and try to build an online extension of AgentNetBench supporting diverse, real-life tasks. (We have interpreted “online interaction settings” as referring to an online evaluation benchmark. If that is not what you meant, please let us know so we can address your concern directly.)\n\n[1] Bai, Shuai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[2] Li, Kaixin, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. \"Screenspot-pro: Gui grounding for professional high-resolution computer use.\" arXiv preprint arXiv:2504.07981 (2025).\n\n[3] Qin, Yujia, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang et al. \"Ui-tars: Pioneering automated gui interaction with native agents.\" arXiv preprint arXiv:2501.12326 (2025).\n\n[4] OpenAI. Operator, 2025."}, {"review_id": "ASOyPEX8zg", "reviewer": "Reviewer_8D4Y", "summary": "This study introduces AgentNet, an open-source framework designed to capture human-computer user interactions, curate a rich multi-modal dataset, and train agents to automate user tasks through a long Chain-of-Thought (CoT) reasoning. The framework fine-tunes off-the-shelf vision-language models using supervised learning on (state, action) pairs derived from user interaction data. Given the growing demand for automation and intelligent agents, this work fills a critical gap by providing a free, open-source, and non-proprietary solution for developing and evaluating Computer Use Agents (CUAs).\n\nI find this line of research innovative and clear. However, I have some concerns with the reasoning method used here and the results shown in Tables 2 and 3. I will express those concerns later in this review.", "questions": "1. Did you observe any conditions under which the agent would get stuck in a loop or fail to terminate properly? If so, it would be valuable to document these cases. Since this is an open-source framework, highlighting such limitations would benefit the research community by guiding future efforts to develop more robust and advanced methods.\n2. Recent researches on various in-context learning methods and prompting have discovered that CoT often leads to hallucinations in LLMs. To overcome this, the reflector module is added; however, there are some advanced methods, such as REACT: Synergizing Reasoning and Acting in Language Models, Toolformer (connecting APIs to LLMs), LLM planning with Monte-Carlo Tree search, etc. These methods have shown to reduce or eliminate hallucinations and perform better than the CoT method. I advise the authors to try at least one other reasoning method apart from CoT and check for gains.", "limitations": "The authors have clearly stated all the limitations of this study in the Appendix.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The main objective of this study is to provide a free, open-source framework to advance research in agentic development.\n2. The framework supports multi-modality, i.e., language and vision.\n3. The authors have meticulously designed an end-to-end pipeline, also shown in the great illustration in Figure 1.\n4. The dataset collected using this pipeline is rich, and I appreciate that the authors have carefully compared their dataset with other existing datasets in Table 1 and Figure 3.\n5. The proposed framework not only surpasses 'some' state-of-the-art models in performance but also offers an open-source tool for the scientific community. I use the word 'some' deliberately, as I have concerns about the results shown in Tables 2 and 3, which I will address below in the 'Weakness' section.\n6. The authors have designed a reflector module, which I find an extremely important addition to this research.  \n\nWeakness:\n\n1. [MINOR] Missing capitalization after period in Line 201. Figure 8 comes after Figure 7. Typo in Line 134. Please fix these very minor issues.\n2. I recommend that authors check and consider citing the paper 'Camel: Communicative Agents for \"Mind\" Exploration of Large Language Model Society. The paper is very similar to this work and also offers an open-source agentic framework to the research community.\n3. I have deducted one point due to concerns regarding the results presented in Table 2. The authors claim that fine-tuning alone leads to poor performance and that incorporating long Chain-of-Thought (CoT) reasoning significantly improves outcomes. However, the reported results for OSWorld (15 steps) appear unconvincing when compared to other models. While the proposed framework outperforms several baselines, it does not surpass comparable architectures such as OpenAI CUA and UI-TARS variants. If the authors believe this performance gap is due to factors such as model architecture size, training data scale, or a combination of both, they should clearly articulate this to support their claim.\nFurthermore, in the case of OSWorld (50 steps), the proposed framework still underperforms relative to existing methods. The comparison for the WAA task is also insufficient to determine whether the framework offers a significant improvement over current approaches. Therefore, I encourage the authors to provide additional analysis explaining the performance limitations of their framework on OSWorld, and to clarify the conditions under which their method is most effective. Do you think that the model under performs in case of 50 steps due to hallucination which is common in the CoT reasoning method? \n\nIf the main focus of the paper is on the dataset pipeline, then the authors should clearly mention that their objective is not to compete with state-of-the-art models. In that case, the weak results in table 2 would be sufficient for this study.\n\nFinally, Table 3 results are stronger than Table 2 results and significantly outperform existing models.", "was_revised": true, "final_justification": "I am convinced that the authors' proposed method is a valuable contribution to the field of Agentic-AI. Their well-curated multi-stage pipeline will offer many research directions in the future for building autonomous agents in different work domains.\n\nI found this rebuttal phase very useful in helping the authors identify the limitations of their work, which I strongly suggest that they should add in the final version. I was also able to understand their long-term vision with this research line, and therefore I after being fully convinced of their methods, I have decided to increase my score.", "rebuttal": "We sincerely thank reviewer 8D4Y for the insightful questions and suggestions. We are grateful that you find our work “innovative and clear,” and we appreciate the recognition of the contributions our framework, dataset, and models make to the agent-research community.\n## Response to Weakness 1:\nWe apologize for the typos and thank the reviewer for the careful reading. All errors will be fixed in the next revision.\n\n## Response to Weakness 2:\nWe thank the reviewer for pointing us to “Camel: Communicative Agents for ‘Mind’ Exploration of Large Language Model Society.” This open-source framework, together with its accompanying library and datasets, has substantially advanced multi-agent research. We will cite this related work and expand our discussion of multi-agent approaches in our next revision.\n\n## Response to Weakness 3 and Question 1:\nWe thank the reviewer for the thoughtful feedback on our model’s performance and limitations.\n\n(1) Over the last two months, we have scaled our data to 40K trajectories and added additional grounding data and scale our model from 7B to 32B size. We fine-tuned Qwen2.5-VL-7B [1] and Qwen2.5-VL-32B[1] on these datasets. As shown in the following table, our 32B model achieves **34.8%** (maximum score **35.7%**), surpassing UI-TARS-72B-DPO= and even GPT-4o based OpenAI CUA, although they have much more parameters. Our model is now a new state-of-the-art among the open-source models. We will include these new results in the next revision and release both models\n\n| Model | Size | OSWorld | ScreenSpot-Pro |\n| --- | ---  | ---     | ---            |\n| UI-TARS-72B-DPO | 72 B | 27.1 %  | 39.4 %         |\n| OpenAI CUA (GPT-4o) | —  | 31.4 %  | —              |\n| AgentNet-Qwen2.5-VL-7B  | 7 B  | 27.9 %  | 50.0 %         |\n| AgentNet-Qwen2.5-VL-32B | 32 B | 34.8 %  | 55.3 %         |\n\n(2) We thank the reviewer for the suggestion for adding more limitations and error analysis. We have inspected the evaluation results on OSWorld and summarize the errors into the following categories:\n\n1. Insufficient task knowledge: Foundation models still lack domain-specific GUI knowledge. Each application has unique UI conventions and operation logic; without explicit exposure, both agents and even humans can struggle.  \n   Example: In the task “I have a lookup table for the officers of each branch. Please fill the second table using VLOOKUP.” the agent does not know the VLOOKUP function and therefore fails.\n\n2. High-precision grounding errors: Tasks that require pixel-accurate actions frequently fail.  \n   Example: In “Change the 2 in ‘H₂O’ to a subscript,” the agent must precisely drag-select only the “2,” but it often selects more characters.\n\n3. Action repetition and failure to terminate: When the agent misjudges a state after an action, it may keep executing (and reflecting on) the same incorrect action. Occasionally it realizes and does a different action to break the loop, but sometimes it loops indefinitely. In some cases, the agent fails to notice that the task is already complete and keeps acting instead of terminating.\n\n4. Long horizon tasks: OSWorld contains tasks requiring > 30-50 gold actions. Maintaining context over so many steps remains challenging.  \n   Example: “Organize my desktop by placing academic papers in ‘Paper_reading’, coding projects in ‘Projects’, and everything else in ‘Miscellaneous’. For files without clear names, determine the category by content.” It requires take actions on many files and folders.\n\n5. Error perception: Although the agent model has a certain level of ability to identify errors and reflect, in some cases that requires high accuracy, it is still challenging. In fine-grained text-editing tasks, the agent may insert a word one character off, yet judge the action correct and proceed.\n\n(3) During the error analysis, we notice that a small portion of steps contains hallucinations in their CoT. The hallucination mainly includes planning and predicted action misalignment and misinterpreting the success of the last action. To study the source of this hallucination in the evaluation, we inspected the CoT of our training data. We use GPT-4 to check each step’s CoT for these kinds of errors in our data. We find that around 3% of trajectories contain one or two such errors. We think that this hallucination indeed has a negative impact on the performance, it may cause incorrect action prediction and partially cause the action repetition. Although hallucination exists in our training data, our experiments show that as long as the hallucination rate is kept within an acceptable low percent, scaling the dataset continues to yield performance gains. (After identifying the errors in the CoT, we are now re-synthesize the CoTs of these actions. We appreciate the reviewer’s insightful question.)  \n## Response to Question 2:\nWe thank the reviewers suggestion on investigating in-context learning and prompting-based methods. \n\nTo better illustrate the following new experiment results and discussion, we would like first to elaborate more about our method. The motivation of our work is to facilitate the research of agentic foundation models, which have the ability to act as an agent themselves and automatically perceive environment, plan, reflect, reason and take actions iteratively. Claude 3.7 Sonnet [2], Claude 4 Sonnet [3], and Kimi K2 [4] fall in this line of work. They also have the ability to be part of the multi-agent systems through prompting thanks to their generalization capability and the agentic data in the pre-training. To integrate the required agentic abilities, including planning, reflection and memory, into the foundation model, we created a pipeline to first build a trajectory dataset. Each step of the trajectory has observation, meticulously designed reflective long CoT and grounded action. Base on ActRe [5], we designed a pipeline including a reflector and a generator to synthesize these CoTs for training. After training, the model's computer environment understanding and agentic capablities are improved, so the foundation model can reason first and output the action in the same prediction. During executing, the model follows the ReAct [6] style: observation-> (thought, action)->observation… In the ‘thought’, the model may make plans and reflect the former errors.\n\nThere are several in-context learning methods in the CUA field. For example, Agent S2 [7] carefully designed an agent framework including manager worker and mixture of grounding modules based on Claude 3.7 Sonnet. Based on the framework they improved the performance on OSWorld from 26.0% to 34.5%.  In another work, “Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis” [8], they trained a specialized grounding model called Jedi. The grounding model itself can not complete tasks in the OSWorld, while GPT-4o has general reasoning and GUI knowledge, but without grounding capability, it only has a success rate of 5% on OSWorld. But when using GPT-4o as planner and Jedi as grounder, the OSWorld success rate boosts to 27.0%. \n\nDue to the limit of rebuttal time, we implemented Jedi’s pipeline in our setting and use our own model as planner and grounder. In each step, we first prompt the agent to reason and plan, then we parse the low-level-instruction and give it to the grounder to predict the action that will be executed. We did an experiment on our 32B model and the result is 23.5% in 15 steps and 24.2% in 100 steps. We think the reason Jedi+GPT-4o improves performance, while ours not is because GPT-4o is a good planner but bottlenecked by the grounding capability, but our model has integrated both ability into our model and it naturally doing a ReAct style prediction, splitting it into two stages may affect its internal logic. However, we think a multi-agent framework is still promising by adding another advanced model as a reflector/judger after our model makes predictions to inspect the errors/hallucinations/incorrect screenshot understanding, and we believe this can mitigate the negative effect of repetition and terminate failures. \n\nWe also discussed building an MCTS based agent framework. We think that the most challenging part in the system is exploring the environment and assessing the reward, because the computer environment can not be backtraced in many cases, for example, many changes in the live environment can not be reverted. Building such a system would require a reward model in the CUA domain and a digital world model or a strong engineering improvement to overcome this barrier. We will explore this direction in our future works.\n\nThanks again to the reviewer for the insightful questions and suggestions!\n\n[1] Bai, Shuai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[2] Anthropic. Claude’s extended thinking. \n\n[3] Anthropic. Introducing claude 4, 5 2025.\n\n[4] Team, Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen et al. \"Kimi K2: Open Agentic Intelligence.\" arXiv preprint arXiv:2507.20534 (2025).\n\n[5] Yang, Zonghan, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. \"React meets actre: When language agents enjoy training data autonomy.\" arXiv preprint arXiv:2403.14589 (2024).\n\n[6] Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. \"React: Synergizing reasoning and acting in language models.\" In International Conference on Learning Representations (ICLR). 2023.\n\n[7] Agashe, Saaket, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. \"Agent s2: A compositional generalist-specialist framework for computer use agents.\" arXiv preprint arXiv:2504.00906 (2025).\n\n[8] Xie, Tianbao, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu et al. \"Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis.\" arXiv preprint arXiv:2505.13227 (2025)."}]

Author rebuttal:

Meta-review: This paper introduces AgentNet, a comprehensive open-source framework for developing, training, and evaluating computer-use agents (CUAs). The work aims to bridge the gap between proprietary and open-source CUA models by providing essential infrastructure, data, and benchmarks to the research community. The reviewers appreciate the design and effectiveness (as shown from the experiments) of the proposed pipeline as well as the thorough evaluation and analysis.

The reviewers identified several concerns, such as limited novelty, performance (the proposed architectures while outperforming some baselines, did not outperform larger models, limited error analysis, and others. The authors, for the most part, addressed the concerns satisfactorily which the reviewers appreciated. While only one reviewer increased their score, all reviewers are very positive about this work.
Decision: Accept (spotlight)