Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Authors: Yiheng Xu
Zekun Wang
Junli Wang
Dunjie Lu
Tianbao Xie
Amrita Saha
Doyen Sahoo
Tao Yu
Caiming Xiong
Venue: ICML 2025 poster (2025)
Keywords: GUI Agent
Visual Language Model
Large Language Model
Grounding
Reasoning
Planning
Computer Use Agent
Vision Language Action Model
Primary area: applications->everything_else

Abstract: Automating GUI tasks remains challenging due to reliance on textual representations, platform-specific action spaces, and limited reasoning capabilities. We introduce Aguvis, a unified vision-based framework for autonomous GUI agents that directly operates on screen images, standardizes cross-platform interactions and incorporates structured reasoning via inner monologue. To enable this, we construct Aguvis data collection, a large-scale dataset with multimodal grounding and reasoning annotations, and develop a two-stage training pipeline that separates GUI grounding from planning and reasoning. Experiments show that Aguvis achieves state-of-the-art performance across offline and real-world online benchmarks, marking the first fully autonomous vision-based GUI agent that operates without closed-source models. We open-source all datasets, models, and training recipes at https://aguvis-project.github.io to advance future research.

AgUVIS: Unified Pure Vision Agents for Autonomous GUI Interaction
Yiheng Xu
Zekun Wang" 1 Junli Wang
Dunjie Lu Caiming Xiong?
Tianbao Xie
Amrita Saha? Doyen Sahoo
Tao Yu
Abstract
reasoning synthesize observations into effective actions. While recent advances in vision-language models (VLMs) have significantly enhanced visual interface interpretation, developing truly autonomous GUI agents remains challeng- ing due to fundamental limitations in current approaches.
Automating GUI tasks remains challenging due to reliance textual representations, platform- specific action spaces, and limited reasoning capabilities. We introduce AGUVIS, unified vision-based framework for autonomous GUI agents that directly operates on screen images, standardizes cross-platform interactions and in- corporates structured reasoning via inner mono- logue. To enable this, construct AGUVIS DATA COLLECTION, large-scale dataset with multimodal grounding and reasoning annota- tions, and develop a two-stage training pipeline that separates GUI grounding from planning and reasoning. Experiments show that AGUVIS achieves state-of-the-art performance across of- fline and real-world online benchmarks, mark- ing the first fully autonomous vision-based GUI agent that operates without closed-source models. We open-source all datasets, models, and train- -wwaaiiisssoeeeecceoe
Although recent advances in large vision- -aiiune mod- els (LVLMss) (OpenAl 2024 Team et al. 2024 Li et al.
2024a Wang et al. 2024b; Deitke et al.) 2024 Chen et al. 2024b have significantly enhanced the ability to interpret complex visual interfaces, we identify several critical barri- ers to advancing GUI automation. First, existing approaches predominantly rely on textual representations (e.g.. HTML or accessibility trees) rather than visual ones (Gur al. 2024 Kim al. 2023 Deng et al. 2023 Zhou et al. 2024 Xie al. 2024). whose input observation is lengthy (more than 4K), and the length increases as the complexity of the GUI grows (Xie et al., (024), limiting generalization and increasing computational overhead compared to more natural image-based representations. Second, the hetero- geneous action spaces across different platforms prevent effective cross-environment learning, constraining the avail- able training data for each enviromment and impeding further scalability. Third, current methods either lack reliable vi- sual grounding (Zheng et al. 2024a) or depend heavily on closed-source language models for reasoning (Gou et al.,
ing recipes at github. io to advance future research.
1. Introduction
2025 Lu et al. 2024), creating a fundamental bottleneck in advancing model capabilities through training. Fourth, existing methods typically train agents generate "reac- tive" low-level actions directly (Hong al. 2024 Cheng ct al. 2024), failing to leverage the sophisticated reason- ing capabilities inherent in vision-language models. This reactive approach struggles with complex scenarios in the real world that require careful planning and broad general- ization. These limitations have prevented the development of scalable, generalizable GUI agents that can operate au- tonomously across diverse digital environments.
Graphical User Interfaces (GUIs) represent the primary medium of human-computer interaction in digital envi- ronments, from websites to desktop and mobile applica- tions (Deng et al. 2023; Zhou et al., 2024; Xie et al. 2024 Rawles et al. 2024b). Creating autonomous agents that can effectively navigate these interfaces could revolutionize hu- man productivity by enabling automated task execution us- ing existing human-centric tools. Such automation requires mastery of three core competencies: visual understanding to comprehend complex interfaces, grounding to map natural language instructions to visual elements, and planning &
To address these challenges, we introduce AGUVIS (as shown in Figure unified vision-based framework that harmonizes visual observation and consistent action spaces across diverse GUI environments. Our approach eliminates dependence on platform-specific textual representations by operating directly on screen images, enabling more natu- ral and generalizable interface understanding. We develop
Equal contribution "Salesforce Research. Correspondence to: Yiheng Xu Syhxu@cs.hku.hte Tao Yu <tyu@es.hku..te Caiming Xiong
University of Hong Kong
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Instruction
Pure-vision Observation
Inner Monologue
User: Plan from Boston Station.
Thought: have net my starting paint Boston Legan Al Airport. To preceed, I nece cllox on the "frl imput field and
Unifled Action Space
Schaddiles
Fanker
AarTs
Low-level Instruction: Inpur and tope destination.
pysutoqua. clsck
pyeut ogu perrieeeeerrgpp
Pysutogul.. press! Tenter")
GUI Action
Rippssble Actions
Cortects
Action:
:05
back()
www.toguiddcss--eeee 88755 y-2.41
:
torn (sttttsss
pyoutogui wwtteetttt North Station]
Figure I: Overview of AGUVIS unified GUI interaction framework for autonomous GUI agents.
standardized action space through plugin system that maintains consistent interaction patterns across platforms while accommodating eiviroiment-specific requirements. Crucially, we incorporate explicit inner monologue during training, allowing the model to develop sophisticated reason- ing patterns that emulate human problem-sol -olving processes. This inner monologue enables the model to break down complex tasks into manageable steps, consider alternative approaches, and adapt to novel siuutions-capabiliiie capabilities that go beyond simple reactive behaviors.
: x A x O - [0, 1] specifies observation probabilities given states and actions.
At each time step i, the agent receives an image observation o, from the GUI environment and generates an action a through a structured reasoning process. This process in- volves inner monologue (Huang et al. 2022), which helps the agent interpret observations and determine appropriate actions. The agent then exccutes a,, receives new obser- vation 01+1, and continues until achieving the goal G or reaching terminal state.
To enable this unified framework, we make several technical contributions. First, we construct AGUVIS DATA COLLEC- TION, a large-scale cross-pland pooss-pltfoom dataset of GUI agent trajectories that features comprehensive multimodal ground- ing and reasoning annotations, including explicit reason- ing paths captured through inner monologue. Second, we develop novel two-stage training pipeline that separates GUI grounding from planning and reasoning, incorporating structured thought processes to enhance autonomous navi- gation capabilities. Finally, we demonstrate that AGUVIS achieves state-of-the-ae tftt--rrttt performance in both offline evalua- tion and weal-world online scenarios, marking the first fully autonomous vision-based GUI agent that operates without relying on closed-soure models. By open-sourcing our datasets, models, and training recipes, we provide a founda- tion for future research in autonomous GUI interaction.
2.2. Unified GUI Interaction Framework
Contemporary GUI agents predominantly rely on platform- specific representations like HTML or accessibility trees for interface interpretation, leading to fragmented approaches across different environments. We propose a unified frame- work that operates purely through visual observations and standardized interactions, addressing key limitations of ex- isting methods while improving computational efficiency.
Our framework unifies both observation and action spaces across platforms while incorporating structured reasoning processes. For observations, we leverage direct visual in- put instead of parsing platform-specific interface code, en- abling the model to process GUIs as humans do through visual perception. The vision-centric approach not only enhances generalization across platforms but also signifi- cantly reduces computational overhead. While traditional textual methods typically require processing 4k-6k tokens per interaction as shown in Figure 3 for HTML and re- ported in Xie et al. (2024) for accessibility tree-our visual approach maintains a constant token cost of 1,196 tokens for 720p images, independent of interface complexity.
2. AGUVIS
2.1. Problem Formulation
GUI interaction presents unique challenges due par- tial observability and sequential decision-making, natu- rally lending itself to modeling as Partially Observable Markov Decision Process (POMDP). We formalize this as a tuple AS O,T, T, T,O). O). where S represents possible environment states, A denotes available actions, and O refers possible observations. The state transition func- tion T S A x S + [0, 1] defines state transition probabilities given actions, while the observation function
At each interaction step, the agent employs two- component inner monologue to bridge visual perception with action execution. The first component performs ex- plicit reasoning (h₂) about the current state relative to the task goal and previous thoughts h4-1- enabling adaptive planning. Finally, the agent generates precise action instruc-
2

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
tions (afest) that translate high-level intentions into concrete interface interactions. This structured thought process
Template-augmented Grounding Data. To create com- prehensive grounding data, we employ dual-source ap- proach. First, we unify existing GUI datasets across plat- forms by converting their instruction-action annotations into our standardized pyaut format. Second, we leverage the rich metadata available in broader UI datasets without action annotations, including all element positions and attributes, to generate synthetic instruction-actiin pairs through carefully designed templates. This approach not only expands our training data but also ensures coverage of diverse interface patterns and interaction types.
ables reliable handling of complex multi-step tasks.
For action execution, we adopt pyaut as our univer- sal interaction interface, supplemented by a flexible plugin system. The pyautogui library provides a comprehen- sive set of programmatic commands that mirror human in- put behaviors, allowing us to represent GUI interactions consistently across platforms. As shown in Table [ this standardized action space enables the model to translate its inner monologue into concrete actions without requiring environment--pecific design.
UI Elements
Trajectories
G0U Screenshor
Our plugin system extends the base pyaut ogui action space to handle platform-specific requirements while main- taining the natural flow of thought-to-action conversion. It incorporates specialized interactions like mobile gestures, platform-specinf shortcuts, and meta-actions such as pro- viding responses or signaling task completion. The system aligns new actions with existing commands where possible, using explicit descriptions only when necessary, ensuring that the agent's inner reasoning remains coherent across different interaction modes. Details of these pluggable func- tions, particularly for mobile environments, are provided in Appendix A.2 where we describe specific mobile interac- tion functions and their corresponding prompts.
Visual Observation
Element
Coordinates
CLSENO EMEE)
Instruction aboat
Ground Truth Action [
Moide Library
Facebook
Masfarr
237281
Instruction-Action Augmentation
SPT structured Irmee Maralogue
GPT-4o
Augmented Inst. and Action Pairs
Augmented Planning Data
Action
Thought Idenahynng Wi- sdocrption
By integrating visual perception, structured reasoning through inner monologue, and unified action representa- tion, our framework enables training a single model capable of operating across diverse GUT environments. This inte- grated approach not only simplifies the training process but also promotes human-like interaction patterns and better generalization to novel interfaces. The reduced computa- tional overhead of visual processing, coupled with the power of structured reasoning, makes this framework particularly effective for real-world applications.
Ling to Pi Encticer
Ireer
Low-lovel Instruction on "oa
Figure 2: AGUVIS DATA COLLECTION augmentation pipeline for two-stage training data.
VLMMauumeeted Planning & Reasoning Trajectories. While existing GUI agent datasets provide high-level goals and action sequences (Deng et al., 2023 Rawles al.) 2024b Li ct al. 2024c), they often lack the intermediate reasoning steps crucial for advanced agent behavior. We address this limitation through a novel VLM-based trajec- tory augmentation process. For each trajectory step, we construct a rich training example by first highlighting rele- vant UI elements in the observation og to guide the VLM's attention. Given the high-level goal G, the current image observation ot, and the grounded action ae, we prompt GPT- 4o to generate the inner monologue components: thoughts hy, and low-level action instruction aimar. To maintain tem- poral coherence, we include previous action instructions af ,..., instr in the context. The complete prompting tem- plate and example are detailed in Appendix(B.2lamm Figure Our carefully crafted approach ensures the generated inner monologues are predictive rather than post-hoc hoc explanations, enabling the agent to develop planning capabilities. Through extensive human evaluation with quantitative analysis de- tailed in Appendix B.3 we validate the quality of these aug- mented trajectories, confirming that they effectively capture not just the actions to take, but also the complete reasoning
2.3. AGUVIS DATA COLLECTION
The effectiveness of GUI agents critically depends on high- quality training data that captures both grounding accuracy and complex reasoning patterns. However, collecting such data presents unique challenges due to the diverse nature of GUI environments and the need for detailed reasoning anno- -ro-pronged tations. We address these challenges through data collection strategy that leverages existing resources and automated augmentation techniques shown in Figure Our dataset consists of two splits: a grounding split focusing on element localization and interaction (Table [O), and plan- ning& & reasoning split capturing multi-step task completion (Table II). This division aligns with our framework's 's dual emphasis on visual understanding and structured reasoning.
3

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Comparison of various planners and grounding methods on ScreenSpot across various device and input modalities. The top part of table shows the results on original instructions evaluation setting while the bottom part shows results on self-plan evaluation setting. Best results are bold.
Table I:
Mobile
Desktop
Web
Planner
Grounder
Avg
Text
Icon/ Ccon/Widget
Text
Icon/Widget
Text
Icon/Widget
GPT-4
22.6
24.5
20.2
11.8
9.2
8.8
16.2
GPT-4o
20.2
24.9
21.1
23.6
12.2
7.8
18.3
CogAgent
67.0
24.0
74.2
20.0
70.4
28.6
47.4
SceClick
78.0
52.0
72.2
30.0
55.7
32.5
53.4
Qwen2-N VL
75.5
60.7
76.3
54.3
35.2
25.7
55.3
UGround
82.8
60.3
82.5
63.6
80.4
70.4
73.3
AGUVIS-G-7B
88.3
78.2
88.1
70.7
85.7
74.8
81.8
SeeClick
76.6
55.5
68.0
28.6
40.9
23.3
48.8
GPT-4
OmniParser
93.9
57.0
91.3
63.6
81.3
51.0
73.0
UGround
90.1
70.3
87.1
55.7
85.7
64.6
75.6
SeeClick
81.0
59.8
69.6
33.6
43.9
26.2
52.3
GPT-4o
UGround
93.4
76.9
92.8
67.9
88.7
68.9
81.4
AGUVIS-7B
95.6
77.7
67.1
88.3
75.2
84.4
AGUVIS-72B
94.5
85.2
95.4
77.9
91.3
85.9
89.2
process leading to those actions. Our analysis reveals that 86.7% of the augmented data successfully demonstrates in- termediate reasoning that aligns with both the ground truth actions and the overall goal intention, with detailed failure case analysis provided in Appendix B.3.2
Each stage leverages distinct split from AGUVIS DATA COLLECTION to progressively build the agentic abilities. The complete training example templates and prompt for- mats for both stages are detailed in Appendix C.I
Stage 1: Grounding Training The first stage focuses on developing fundamental GUI interaction capabilities through efficient processing of single-screenshe environ- ments. To address the challenge of multiple interactable objects within each screenshot generating redundant train- ing data, implement grounding packing strategy. This approach bundles multiple instruction-action pairs into sin- gle image, creating single-image saggll--maa-mmliplee--neee format. By processing several grounding examples simultancously from each screenshot, we significantly reduce training overhead while maintaining performance (shown Appendix [.2). The result of this stage is AGUvIS-G, model equipped with advanced GUI understanding and interaction capabilities.
2.4. Model Architecture
Vision-based GUI agents require direct mapping between visual observations and actions, necessitating an architec- ture optimized for high-resolution image processing while preserving spatial relationships. We selected Qwen2-Vi as our foundation, leveraging its VaViT-style image encoder's native support for dynamic resolution processing-4 criti- cal feature for handling diverse interface layouts. Another key strength of the architecture lies in its position embed- ding mechanism. By replacing traditional absolute position embeddings with 2D-RoPE, the model maintains precise spatial awareness across varying screen dimensions while efficiently converting interface screenshots into visual to- kens. These improvements significantly reduce computa- tional overhead compared to conventional methods.
Stage 2: Planning & Reasoning Training Building AGUVIS-G's foundation, the second stage develops advanced decision-making and reasoning abilities neces- sary for complex, multi-step tasks. We leverage our de- tailed inner-monologue trajectory data (as described in Sec- tion 2.3) to implement reasoning mixture approach, ex- posing the model to varying levels of cognitive complexity. This ranges from basic action instructions comprehensive inner monologues encompassing thoughts, and detailed ac- tion plans. The dynamic adjustment of trajectory complexity ensures the model develops adaptable reasoning patterns and sophisticated decision-man makinn capabilities. The final result is AGUVIS, a fully-trained model capable of handling both offline and online GUT tasks across diverse environ-
To validate our framework's 's flexibility, we also implemented using VA--eeO Vision, which similarly supports high- resolution image processing with variable aspect ratios, al- beit with higher token costs. This implementation confirms our framework's model-agnostic nature, with detailed com- parisons presented in Section 4.1
2.5. Training Paradigm
The training process of AGUVIS is divided into two stages: Grounding Training and Planning & Reasoning Training.

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table 2: Performance comparison on Multimodal Mind2Web across different settings. We report element accuracy (Ele.Acc), Operation FI (O,.F1), and step success rate (Step SR). Best results are in bold. "T" means the textual HTML code as inputs. "I" means the GUT images as inputs. More explanation about result source in Appendix D..
Cross-Tass
Pooss-Website
Cross-Domain
Obs.
Planner
Grounder
Ele.Acc
Op.F1
Step SR
Ele.Acc
Op.F1
Step SR
Ele.Ace
Op.F1
Step SR
GPT-3.5 GPT-4
Choice Choice
19.4
59.2
16.8
14.9
56.5
14.1
25.2
57.9
24.1
40.8
63.1
32.3
30.2
61.0
27.0
35.4
61.9
29.7
GPT-4
Choice
46.4
73.4
40.2
38.0
67.8
32.4
42.4
69.3
36.8
T+1
GPT-4
SoM
29.6
20.3
20.1
13.9
27.0
23.7
GPT-4o
SeeClick
32.1
33.1
33.5
GPT-4V
OmniParser
42.4
87.6
39.4
41.0
84.8
36.5
45.5
85.7
42.0
GPT-4o
UGround
47.7
46.0
SceClick-9.68
28.3
87.0
25.5
21.4
80.6
16.4
23.2
84.8
20.8
I
AGUVIS-7B
64.2
89.8
60.4
60.7
88.1
54.6
89.2
56.6
AGUVIS-72B
69.5
90.8
64.0
62.6
88.6
56.5
63.5
88.5
58.2
ments with nuanced understanding and precision. Complete training implementation details, including hardware con- figurations, training durations, and hyperparameters, are provided in Appendix C.2
et al. 2023 or Set-of-Marks (Zheng et al. 2024a). AGUVIS operates solely on GUI screenshots. The results in Table show that AGUVIS achieves superior performance across all metrics, with a particularly notable improvement in Step Success Rate (+51.9% average), highlighting enhanced planning capabilities.
3. Experiments
3.1. GUI Grounding Evaluation
Table 3: Step Accuracy of out-of-domain data on Android- Control under high-level tasks and low-level tasks. Best performance is in bold. TreeTree" means the textual acces- sibility tree.
ScreenSpot. We first evaluated AGUVIS'S fundamental GUI grounding capabilities using ScreenSpot (Cheng et al. 2024), a benchmark that spans mobile, desktop, and website platforms. Following established protocols from previous work (Cheng et al., 20244 Gou et al. 2025). we tested under two conditions: direct execution from original instructions and self-planned exccution requiring natural language plan- ning before action.
Step Acc.
Obs.
Planner
Grounder
High
Low
GPT-4-Turbo
Choice
42.1
55.0
Acc. Tree
PaLM 2S*
Choice
58.5
77.5
GPT-4-Turbo
SeeClick
39.4
47.2
The results in Table demonstrate AGUVIS's exceptional grounding capabilities across platforms. Our AGUVIS-G- 7B, trained with the proposed grounding stage, significantly outperforms existing models on original instructions. The full model AGUVIS-7B shows even stronger performance after planning trajectory training, surpassing previous ap- proaches that rely on closed-source LLMs like GPT-4o. The scaled version, AGUVIS-72B, achieves state-of-the-ard per- formance with an average score 89.2.
GPT-4-Turbo
UGround
46.2
58.0
Image
GPT-4o
SeeClick
41.8
52.8
GPTT-4o
UGround
48.4
62.4
AGUVIS-7B
61.5
80.5
Image
AGUVIS-72B
66.4
84.4
AndroidControl. For mobile interface interaction, we
evaluated AGUVIS on AndroidControl using a subset of 500 randomly sampled step-actions following the setting et al. (2024c). We tested both high-level planning scenarios and low-level instruction execution. comparing against mod- els using various input modalities. Table 3 demonstrates AGUVIS'S superior performance in both settings, confirming its effectiveness across different interaction paradigms.
3.2. Offline GUI Agent Evaluation
We assessed AGUVIS'S planning capabilities through two major offline benchmarks: Multimodal-MMnd2Wead (Zheng
et al. 2024a) for website interaction and AndroidControl (Li
al. 2024c for mobile device operation.
3.3. Online GUI Agent Evaluation
To validate real-world applicability, we evaluated AGU- VIS across three comprehensive benchmarks: Mind2 Web. Live (Pan et al. 2024). AndroidWorld (Rawles et al. 2024a). MobileMiniWan (Rawles es et al. 2024b).
Multinodal-Mind2Webh. Multimodal Maltimodal-Mind2Wadl evalu- ations focused on website navigation and interaction tasks. Unlike previous approaches that utilize textual inputs (Deng
5

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Mind2Web-Live provides dynamic web-based environ-
4. Analysis
ment derived from Mind2Web, evaluating task completion through step-by-step success rates. AndroidWorld operates in virtual Android environment, using Pixel 6 phone sim- ulator for mobile agent assessment. MobileMini Wob adapts 92 tasks from MiniWob+++ to AndroidWorld environment, providing standardized GUI interaction scenarios. More details are Appendix[D33 We test two distinct configura- tions: one pairs GPT-4o planner with AGUVIS-7B grounder, and the other employs AGUVIS-72B in both roles.
4.1. Impact of Training Stages
We first assess the impact of cach stage in our training pipeline by evaluating several variants of AGUVIS, As shown in Table 6 we examine the performance of: (a) model trained without the second stage (planning & rea- soning), referred to AGUVIS-G, and (b) Qwen2-VL, the base model without both stages of specialized training. The results demonstrate clear performance degradation when either training stage is omitted. In particular, removing Stage (planning & reasoning) leads to significant drops in performance across all metrics.
Table 4: Task Success Rate (SR) and efficiency costs Mind2Web-Live. Cost is calculated by dividing the model's total inference cost USD by the number successful steps.
To verify that these improvements stem from our methodol- ogy rather than the inherent capabilities of Qwen2-VL, we conducted parallel experiments using LLaVA as an alterna- tive backbone. The results in Table [ a weaker foundation model, the AGUVIS training pipeline yields substantial improvements. For instance, LA'ss performance on ScreenSpot improves from 3.8% 81.2% after applying complete training process, validating the effectiveness of our approach across different architectures.
Inputs
Planner
Grounder
SR
Cost
GPr-o-Turbo
Choice
21.1
that even with
GPT--4o
Choice
22.1
0.142
HTML
Llama-3.1-405B
Choice
24.0
0.174
Llama-3.1-70B
Choice
20.2
0.031
GPT-3.5-turbo
Choice
17.3
0.092
GPT-4-Turho
UGround
23.1
Image
GPT-4o
UGround
19.2
GPT-4o
AGUVIS-7B
24.0
0.106
4.2. Role of Inner Monologue
Image
AGUVIS-72B
27.1
0.012
Inner monologue plays erucial role in enhancing both plan- ning and grounding capabilities. As demonstrated in Table 6 removing inner monologue from training data results in significant performance drops across all benchmarks. The impact is particularly noticeable in low-level tasks. For in- stance, ScreenSpot performance falls from 84.4% to 79.3%, and low-level AndroidControl drops from 80.5% to 69.1%. This suggests that inner monologue aids only high-level planning but also precise low-level execution. More detailed results are in Appendix 1.1.2
Results Tables and 5 present our comprehensive find- ings. When using GPT-4o as the planner, AGUVIS-7B demonstrates superior performance across benchmarks com- pared to existing methods. The unified AGUVIS-728 ap- proach achieves best-in-class performance on Mind2Web- Live and Mobilc.MiiiWob. These results, combined with our model's significant efficiency advantages, demonstrate the strong potential of pure uision-based agents for real-world GUT automation tasks.
4.3. Cross-Platfore Benefits
Our unified training approach enables effective knowledge transfer across different platforms, allowing the model to develop generalizable interaction capabilities. As shown in Table [7] training on both web and mobile data leads to significantly better performance compared to platform- specific training. On web-specific tasks in Multimodal- Mind2Web, models trained with both web and mobile data achieve superior results compared to those trained solely web data or Mind2 data alone.
Table 5: Task Success Rates (SR) on AndroidWorld (AW) and MobileMiniWob(MMM)) Best results are in bold.
Input
Planner
Grounder
AWSR SR
MMW
GPT-4-Turbo Gemini 1.5 Pro
Choice
30.6
59.7
AXTree
Choice
19.4
57.4
Image
GPT-4-Turbo
SoM
25.4
67.7
+ AXTree
Gemini 1.5 Pro
SoM
22.8
40.3
These improvements highlight the ability of our framework to leverage commonalities across different GUI environ- ments, fostering generalization beyond individual datasets. The combination of a pure vision approach and standardized pyautogui actions establishes shared representation space, enabling effective cross- plos--plaffomm learning.
GPT-4-Turbo
UGround
31.0
Image
GPT-4o
UGround
32.8
GPT--4o
AGUVIS-7B
37.1
55.0
Image
AGUVIS-72B
26.1
66.0
6

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table 6: Ablation on AGUVIS-7B on MM-Mind2Web and AndroidControl benchmarks. We report the step success rate. We
provide a more comprehensive ablation in Appendix E.
Multimodaa-Mind22Wel
AndroidControl
Settings
ScreenSpot
Cross-Task
Cross-Websie Website
Cross-
High-Level
Low-Level
AGUVIS-7B
84.4
60.4
54.6
56.6
61.5
80.5
(a) w/o Stage 2
81.8
50.9
45.2
45.3
58.0
75.6
(b) w/o Stage
77.4
59.7
55.3
56.8
58.8
79.8
(c) w/o Stage 1 &
55.3
50.9
44.9
47.7
59.1
59.2
(d) w/o Inner Monologue
79.3
55.4
53.7
54.9
60.3
69.1
Table 7:
Ablation study on Multimodal-Mind2W4 analyz- ing the impact of training data from different device domains within a unified action space.
Table 8: Success rate on the OSWorld benchmark in the screenshot-only setting.
Planner
Grounding
Task SR
Data
#Traj.
Task
Website
Domain
GPT-4o
SoM
4.59
Web + Mobile
35k
58.5
55.4
54.8
GPT-4o
AGUVIS-7B
14.79
Web Only
6k
53.1
50.3
52.2
GPT-4o
AGUVIS-72B
17.04
Mind2Web Only
50.9
44.9
47.7
GPT-4o
5.03
GPT-4V
5.26
Gemini-Pro-
5,40
Claude Computer-
14.9
To further evaluate the generalization capabilities of our model, we tested on OSWorld, a unified computer envi- ronment designed for multimodal agents. OSWorld presents complex workflows, encompassing 369 real-world com- puter tasks that span web applications, desktop software, and OS-level operations. The results are shown in Table
OpenAl Operator
19.7
AGUVIS-72B
10.26
in Mind2Web-Live, detailed in Figure 3 and Table
Remarkably, despite being trained exclusively web and mobile trajectory data, our model demonstrates strong gen- eralization to desktop GUT tasks. On OSWorl,, when paired with GPT-4o for planning, our model achieves 17.04% task success rate, significantly outperforming SoM-based approaches (4.59%) and even surpassing Claude Computer- Use (14.9%). Furthermore, AGUVIS-72B, when deployed as an independent model, achieves 10.26%, demonstrat- ing that our approach is competitive even without external planning support. This result underscores that our approach does not overfit to specific environments but instead captures fundamental GUT interaction principles, enabling effective transfer novel computing scenarios.
Figure 3: Comparison of Input Tokens per Step and USD Effi- ciency in GUT Interaction. The bar chart shows the input tokens required step during GUI interactions, while the line graph illustrates USD Efficiency for all models.
Input Tokens Per Step
HTML
Image
USD Efficiency
4,000
3,000
2,000
1,000
GPT:3.5 AGEVIS
4.4. Efficiency Benifits from Pure Vision Perception
4.5. Error Analysis and Future Work
The pure vision approach significantly reduces computa- tional overhead compared to traditional textual methods. As illustrated in Figure while HTML-based approaches typi- cally require processing about 4,000 tokens per interaction, our vision-based method maintains constant token cost of 1.196 tokens for 720p images, independent of interface com- plexity. This efficiency translates to substantial practical benefits in deployment, with our method reducing costs by 93% and input tokens per step by 70% compared to GPT-4o
To understand failure modes and potential improvements. we conducted a detailed error analysis on 50 samples from the ScreenSpot dataset under the self-plan setting, Our anal- ysis reveals two primary categories of errors, shown in Figure 40% stem from ambiguous instructions that could refer to multiple grounding targets, while the remaining 60% are grounding errors. critical finding is that the model currently lacks the ability to indicate uncertainty or refuse actions when faced with ambiguous instructions an essen-

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
tial capability for real-world deployment where incorrect
2024 Koh et al.
2024a Lu al. 2024 Drouin et al. 2024
actions could have significant consequences.
Pan et al.
2024). desktop (Xie al.
2024 Bonatti et al.
2024), and mobile environments (Rawles et al.
2024a b 2024
Figure 4: Error analysis on Screenspot under the self-plan setting.
Zhang et al.
2024b Chai et al. 2024 Lu et al.
et al. 2024c Furthermore, cross pross-platffom datasets such as
Ambiguous Error Grounding Error
ScreenSpot (Cheng et 2024), OmniACT (Kapoor et al.
2024). GUICourse (Chen et al. 2024a), and CRAB (Xu et al. 2024a) provide comprehensive evaluation frameworks spanning multiple devices and interfaces. Evaluations on specialized applications have also emerged, such Spider- 2V (Cao et al., 2024) targeting data science and engineering workflows. To thoroughly evaluate our proposed model's grounding and planning capabilities, we conduct extensive experiments on relevant benchmarks under both online and offline settings.
1
Planning Bonus
0.8
0.6
0.4
0.2
0
Self Plan
Enforced Plan
5.2. GUT Agent Models
When enforce planning by prompting the agent model to generate inner monologue before execution as detailed in Appendix E.2.1 it resolves 20% of these grounding errors, suggesting that explicit reasoning helps the model leverage its knowledge more effectively. However, our analysis re- veals a significant challenge: while many queries appear simple syntactically, they actually require deeper semantic understanding and domain knowledge. In these cases, the model struggles to recognize the need for planning and de- faults to direct grounding instead of explicit reasoning. We provide illustrative examples of these semantically challeng- ing cases in Appendix E.2.2
Significant progress has been made developing more capable autonomous GUI agents. For web navigation, mod- els such as WebGPT (Nakano et al.] 2021), Lemur (Xu)
et al. 2024b), Agent-Lumos Yin et al. 2024). CogA- gent (Hong et al. 2024). AutoWebGI.M (Lai et al. 2024
and xLAM (Zhang et al. 2024a) have demonstrated en- hanced performance. Auto-GUI (Zhang & Zhang 2024)
AppAgent (Zhang et al. 2025), and ScreenAgent (Niu et al. 2024) propose novel approaches for direct GUI interaction without relying on application-speeifee APIs. More recently, research has targeted core capabilities of GUI agents like grounding and planning & reasoning. [Gou et al. (2025) and Wu et al. (2055) propose scaling grounding data to improve grounding ability. As for enhancing the planning and rea- soning ability of GUI agents, some GUI agents for mobile environments (Li et al.) 2023 2024b] Wang et al.) 2024a) explicitly incorporate planning trajectories for training. Koh et al. (2024)) introduces an inference-time search algorithm in interactive web environments. These advancements col- lectively enable more sophisticated and capable GUI agents for automated task completion across digital platforms.
These insights highlight several promising directions for fu- ture work, along with concrete solutions. To develop more reliable GUI agents for real-world settings, we propose in- corporating adversarial training examples where the correct action is to refuse execution or raise safety concerns, helping models learn to handle ambiguous or potentially harmful situations appropriately. To enhance the model's s ability to identify semantically complex tasks requiring planning, we suggest augmenting training data with explicit annotations of task complexity and required reasoning depth, potentially combined with dynamic threshold system during inference to balance planning overhead against accuracy gains. By pursuing these directions, we can work toward GUI agents that not only perform tasks accurately but also do so with ap- propriate caution and self-awareness a crucial requirement for real-world deployment.
6. Conclusion
We introduced AGUVIS, unified pure vision-based frame- work for autonomous GUI agents that operate across diverse platforms. By leveraging vision-only observations and a standardized action space, AGUVIS eliminates reliance on platform-specific representations and closed-source mod- els. Our structured reasoning approach, combined with a large-scale dataset and two-stage training pipeline, enables superior grounding, planning, and reasoning. Extensive ex- periments demonstrate -fate-of-the-art performance in both offline and online GUT tasks. We open-source all datasets, models, and training recipes to accelerate future research in this domain.
5. Related Work
5.1. GUI Agent Benchmarks
Recent advancements in autonomous GUI agents have spurred the development of numerous benchmarks assess- ing agent capabilities across diverse platforms, including
those focused on the Web (Deng et al. 2023 Zhou et al.
8

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Acknowledgment
Chai, Y., Huang, S., Niu, Y., Xiao, H., Liu, L., Zhang, D., Gao, P., Ren, S., and Li, H. Amex: Android multi- annotation expo dataset for mobile gui agents. ArXiv preprint. 2024. URL 2407. 17490
This project is partially funded by the ECS (27212023) from the Hong Kong RGC.
Impact Statement
Chen, W., Cui, J., Hu, J.., Qin, Y., Fang, J., Zhao, Y., Wang, C., Liu, J., Chen, G., Huo, Y., al. Guicourse: From general vision language models to versatile gui agents. ArXiv preprint, 2024a. URL https://arxivv.
Creating autonomous agents capable of navigating Graphi- cal User Interfaces (GUIs) has the potential to revolution- ize human productivity by automating tasks using existing human-centric tools. In this work, we focus on democratiz- ing GUI agents by leveraging open-source models, rather than relying on proprietary LLMss By training open-source models to possess native planning and reasoning capabili- ties, we enable, for the first time, end-to-end autonomous GUI interactions based on pure vision and open-source mod- els. Security remains paramount in real-world applications of GUI agents. Ensuring that agents do not execute harmful actions is a critical area that warrants further investigation. To foster the advancement of future research on GUI agents and computer-use tasks, we have open-sourced all relevant datasets, models, and training recipes. This initiative aims to provide foundational platform for researchers and develop- ers to build upon, encouraging innovation and collaboration in the field.
abs Abs/2406.1135
Chen, Z., Wang. W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J. Ye, S., Tian, H., Liu, Z., Gu, L., Wang, X., Li, Q., Ren, Y., Chen, Z., Luo, J., Wang. J., Jiang, T., Wang. B., He, C., Shi, B., Zhang, X., Lv, H., Wang, Y., Shao, W., Chu,
P., Tu, Z., He, T., Wu, Z., Deng, H., Ge, J., Chen, K., Dou, M., Lu, L.,, Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., and Wang, W. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. CoRR, abs/2412. 22.02771, 2024b. doi: 10.45550AARXX22222..507177 URL https: //dp: prg//o..4550//axxxxy wtt...tww.....11........... 05271
Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 11-16 2024, 2024.
References
Bai, C., Zang, X., Xu, Y., Sunkara, S., Rastogi, A., Chen, J.,
and y Arcas, B. A. Uibert: Learning generic multimodal representations for UI understanding. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 2021.
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., Lu, J., Anderson, T., Bransom, E., Ehsani, K., Ngo, H., Chen, Y., Patel, A., Yatskar, M., Callison- Burch,, C., Head, A., Hendrix, R., Bastani, F., VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J., Skjonsberg, S., Schmitz, M., Sarnat, A., Bischoff, B., Walsh, P., Newell, C., Wolters, P., Gupta, T., Zeng, K.-H., Borchardt, J., Groeneveld, D., Dumas, J., Nam, C., Lebrecht, S., Witlli,, C., Schoenick, C., Michel, O., Krishna, R., Weihs, L., Smith, N. A., Hajishirzi, H., Girshick, R., Farhadi, A., and Kembhavi, A. Molmo and pixmo: Open weights and open data for state of art multimodal models. arXiv preprint
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z.., Wan, J., Wang, P., Ding, W., Fu, Z Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. Qwen2.5-vl technical report. CoRR, (bs/2500.119223 2025.
Bonatti, R., Zhao, D., Bonacci, F., Dupont, D., Abdali, S., Li, Y., Lu, Y., Wagle, J., Koishida, K., Bucker, A. F. C., Jang, L., and Hui, Z. Windows agent arena: Evalu- ating multi-modal os agents at scale. ArXiv preprint, 2024. URL https: /ttp::///ppp..ee semant cscholar ort//corpus1D::222604. ID:
2024.
Deka, B., Huang, 7.,, Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., and Kumar, R. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Anmual ACM Symposium on User Interface Software and Technology, 2017.
Cao, R., Lei, F., Wu, H. Chen, J.. Fu, Y., Gao, H., Xiong, X., Zhang, H., Hu, W. Mao, Y. Xie, T., Xu, H., Zhang, D., Wang, S. I., Sun, R., Yin, P., Xiong, C., Ni, A., Liu, Q., Zhong, V., Chen, L., Yu, K., and Yu, T. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? In Advances in Neural Information Processing Systems 38: Anmual Conference on Neural Information Processing Systems 2024, 2024.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards generalist agent for the web. In Advances in Neural Information Processing Systems, 2023.

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Verme, M. D., Marty, T., Vázquez, D., Chapados, N., and La- coste, A. Workarena: How capable are web agents at solving common knowledge work tasks? In Forty- first International Conference on Machine Learning, 2024. URL
Conference on Knowledge Discovery and Data Mining, pp. 5295-229-202 -5306 2024.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H.,, Zhang, K., Li, Y., Liu, Z.,, and Li, C. Llava-onevision: Easy visual task transfer. ArXiv preprint, 2024a. URL attps://arxiv. wwtp:/wwwrg//bb////0000 03330
wwwp:rrviiieeeeee/ooeeeeeeee norum?
id-BRfqYrikdo
Li, T., Li, G., Deng, Z., Wang, B., and Li, Y. A zero-shot language agent for computer control with structured re- flection. In Findings of the Association for Computational Linguisties: EMNLP 2023, 2023.
Gou, B., Wang, R., Zheng, B., Xie, Y., Chang, C., Shu, Y., Sun, H., and Su, Y. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thireentth International Conference on Learning Representations, 2025.
Li, T., Li, G., Zheng, J., Wang, P., and Li, Y. MUG: interac- tive multimodal grounding on user interfaces. In Findings of the Association for Computational Linguistics: EACL 2024, 2024b.
Gur, I., Furuta, H., Huang, A. V., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. A real-world webagent with plan- ning, long context understanding, and program synthesis. In International Conference on Learning Represenations, 2024.
Li, W. Bishop, W. E.,, Li, A., Rawles, C.,, Camphell-Ajala F., Tyamagundlu, D., and Riva, O. On the effects of data scale on UI control agents. In Advances in Neural Infor- mation Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, 2024c.
Hong, W., Wang, W., Lv, Q. Xu, J., Yu, W., Ji, J., Wang, Y., Wang. Z., Dong, Y., Ding, M., et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Li, Y., He, J., Zhou, X., Zhang, Y., and Baldridge, J. Map- ping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Associa- tion for Computational Linguistics, 2020a.
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P. Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Jackson, T., Brown, N., Luu, L.., Levine, S., Hausman, K., and lchter, B. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robol Learning, CoRL 2022, 2022.
Li, Y., Li, G., He, L., Zheng, J., Li. H., and Guan, Z. Widget captioning: Generating natural language descrip- tion for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP). Association for Computational Linguistics, 2020b. URL httpss:/
Kapoor, R., Butala, Y. P., Russak, M., Koh, J. Y., Kamble, K., AlShikh, W., and Salakhutdinov, R. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In Computer Vision ECCV 2024- 18th European Conference, 2024.
aclantho ogy. wwwoo///00022 emmnp-mminn 443
Loshehilov, and Hutter, Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019. New Orleans, LA, USA. May 6-9, 2019 OpenReview.net, 2019. URL https //openreview.net/Eeeeee ftaeee//foooooo http:////f..ooom.oddccd..s/o.
Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. In Advances Neural Information Processing Systems, 2023.
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.. Neubig. G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualweharena: Evaluating multimodal agents realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024a.
Lu, Q. Shao, W., Liu, Z., Meng, F,, Li, B., Chen, B., Huang, S., Zhang, K., Qiao, Y., and Luo, P. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. ArXiv preprint, 2024. URL https
//arxiv.org/abs////
08451
Koh, J. Y., McAleer, S., Fried, D., and Salakhutdinov, R. Tree search for language model agents. ArXiv preprint, 2024b.
Lù, X. H., Kasner, 7.,, and Reddy, S. Weblinx: Real-world website navigation with multi-turn dialogue. In Forty-first International Conference on Machine Learning, ICML 2024, 2024.
Lai, H., Liu, X., Iong, I. L.. Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y., and Tang, J. Autowebglm: A large language model-based web navi- gating agent. In Proceedings of the 30th ACM SIGKDD
Lu, Y., Yang, J., Shen, Y., and Awadallah, A. Omniparser for pure vision based gui agent, 2024. URL https 7/axxiv. www./org/////eeeooooo..o... /200. 2408000203
10

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Microsoft. Playwright for python documentation. https: ttt.
Wang, H., Li, T., Deng. Z., Roth, D., and Li, Y. Devil's advocate: Anticipatory reflection for LLM agents. In Findings of the Association for Computational Linguis- tics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, 2024a.
2024.
Nakano, R., Hilton, J., Balaji, S., Wu, J.. Ouyang. L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. ArXiv preprint, 2021. URL www.aggg////ooooo..ot..o/t..aer/ org/ abs 2112.11122. 09332
Wang, P., Bai, S., Tan, S.,, Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's percep- tion of the world at any resolution. ArXiv preprint, 2024b. URL hRLpttps: /wwpv/rrxvvvv//aa////a//l.. org http:/aabb/2999.222999
bttpps//aaae
Niu, R., J., Wang, S., Fu, Y., Hu, X., Leng, X., Kong. H., Chang, Y., and Wang, Q. Screenagent: A vision language model-driven computer control agent. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, 2024.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface 's transformers: State-of-the-art natural language processing. ArXiv preprint, 2019. URL https: www.aawwwv.rggroooaaaa
OpenAl. Hello gpt-4o, 2024. URL https://oppe
/he https/heeelllptp-ee gpt-
Pan, Y., Kong. D., Zhou, S., Cui, C., Leng, Y., Jiang, B., Liu, H., Shang, Y., Zhou, S., Wu, T., et al. Webcanvas: Benchmarking web agents in online environments, ArXiv preprint, 2024. URL https:://axxii www.vvoorrgabbb /bs/
9100 03771
Wu, J., Wang. S., Shen, S., Peng, Y.-H., Nichols, J., and Bigham, Webui: A dataset for enhancing visual ui understanding with web semantics. ACM Conference on Human Factors in Computing Systems (CHI), 2023.
2406. 12373
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., De Vito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high- -esh-performnce deep learning library. In Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurlPS 2019, 2019.
Wu, Z., Wu, Z., Xu, F., Wang, Y., Sun, Q. Jia, C.,, Cheng, K., Ding, Z., Chen, L., Liang, P. P., and Qiao, Y. OS- ATLAS: A foundation action model for generalist GUI agents. In The Thirteenth International Conference on Learning Representations, 2025.
Xie, T., Zhang, D., Chen, J., Li. X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y.. Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. Osworld: Benchmarking multimodal agents for open- ended tasks in real computer environments. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, 2024.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, 2020.
Xu, T., Chen, L., Wu, D.-J., Chen, Y., Zhang, Z., Yao, X., Xie, Z., Chen, Y., Liu, S., Qian, B., Torr, P. H. S., Ghanem, B., and Li, G. Crab: Cross-environment agent benchmark for multimodal language model agents. ArXiv preprint. 2024a. URL
Rawles, C., Clinckemailie,, S.. Chang, Y., Waltz, J., Lau, G., Fair, M., Li, A., Bishop, W., Li, W., Campbell-Ajala, F, Toyama, D., Berry, R., Tyamagundlu, D., Lillicrap, T., and Riva, O. Androidworld: A dynamic benchmark- ing environment for autonomous agents, 2024a. URL http::://axxiv.orr//bbbs//
14573
Xu, Y., Su, H., Xing, C., Mi, B., Liu, Q., Shi, W., Hui, B., Zhou, F., Liu, Y., Xie, T., Cheng, Z., Zhao, S., Kong, L., Wang. B., Xiong, C., and Yu, T. Lemur: Harmoniz- ing natural language and code for language agents. In International Conference on Learning Representations, 2024b.
Rawles, C., Li, A., Rodriguez, D., Riva, O., and Lillicrap, T. Androidinthewild: A large-scale dataset for android de- vice control. Advances in Neural Information Processing Systems, 2024b.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang. S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint. abs/2403.05530, 2024. httttp:://axii..rrt 0ass/203.05555
Yin, D., Brahman, F., Ravichander, A., Chandu, K. R., Chang, K., Choi, Y., and Lin, B. Y. Agent lumos: Unified and modular training for open-source language agents. In Proceedings of the 62nd Annual Meeting of the Associa- tion tionfor Computational Linguistics, 2024.

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z.., Fu, B., and Yu, G. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Sys- tems, CHI 2025, 2025.
Zhang, J., Lan, T., Zhu, M., Liu, Z.,, Hoang, T., Kokane, S., Yao, W. Tan, J., Prabhakar, A., Chen, H., Liu, Z., Feng, Y., Awalgaonkar, T., Murthy, R., Hu, E., Chen, Z., Xu, R., Niebles, J. C., Heinecke, S., Wang, H., Savarese, S., and Xiong, C. xlam: A family of large action models to empower agent systems, ArXiv preprint, 2024a.
Zhang, J., Wu, J., Teng, Y., Liao, M., Xu, N., Xiao, X., Wei, Z., and Tang, D. Android in the zoo: Chain-of-action- thought for GUI agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024b.
Zhang, 7. and Zhang, A. You only look at screens: Multi- modal chain-of-action agents. In Findings of the Associa- tion for Computational Linguistics, 2024.
Zheng, B., Gou, B., Kil, Sun, H., and Su, Y. Gpt--v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, ICML 2024, 2024a.
Zheng, L., Wang, R., Wang, X., and An, B. Synapse: Trajectory-as-exempla prompting with memory for com- puter control. In The Twelfth International Conference on Learning Representations, ICLR 2024. OpenReview.net, 2024b.
Zhou, S., Xu, F. F,, Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. Webarena: realistic web environment for building autonomous agents. In International Conference on Learning Representations, 2024.
12

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table of Contents in Appendix
AGUVIS Unified Design
14
A.1 Details of Action Space in AGUVIS
14
A.2 Pluggable Functions: Mobile Environments as Example
14
Data Curation of AGUVIS DATA COLLECTION
15
Detailed Source Dataset Statisties
15
B.2
Prompt for Augmenting Planning & Reasoning Trajectories
16
B.3
Human Study on Augmented Data
17
B.3.1
Qualitative Human Study.
17
B.3.2
Failure Cases Under Noisy Training Data
17
AGUVIS Training
17
Training Example Schema
17
Training Details
19
D Evaluation Benchmarks
21
D.I GUT Grounding Evaluation
21
D.2 Offline GUT Agent Evaluation
D.3 Online GUI Agent Evaluation
21
D.3.1 Prompts for using GPT-4o as Planning Model
22
Analysis
25
E.1
More Training Ablation
25
E.1.1
Training Strategy Ablation
25
E.1.2
1.2
Data Strategy Ablation
26
E.2
Planning Analysis
27
E.2.1
Prompts for self self- plannillnnngg and enforced planning mode.
................................
27
E.2.2
Cases of Inner Monologue Bonus
29
AGUVIS Trajectories Examples on Online Evaluation
30
Mind2Web-Live Case: AGUVIS-72B as Planner and Grounder
30
E
E.3.2
Mind2Web-Live Case: GPT-4o Planner and AGUVIS-7B Grounder
".o.............................
31
33
Worlo Case: AGUVIS-72B as Planner and Grounder
32
3,4
AndroidWorld Case: GPT-4o as Planner and AGUVIS-7B as Grounder
33
Case of AGUVIS Generalization in Real World Scenarios
34
13

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
A. AGUVIS Unified Design
A.1. Details Action Space AGUVIS
In this section, we introduce our unified action space our pure vision agent framework AGUVIS, As shown in Table 9 we use default standard pyautogui actions with pluggable actions the action space of AGUVIS, which ensures the agent model's universality across environments well its flexibility in the specific environment.
Table 9: Default standard pyautogui actions A with pluggable actions.
Category
Action Space
pyautogui coaitooguimmoveeoooooooo
moveTo (x, y)
gyautogui.click (x, y)
wraitogui.writte(( ('text'))
Basic Actions
Pyautogui.press ('enter')
Pyautogui.hotkey ('trrl', )e()
pyautogui.scrool (200)
Pyautogui.drragoo (x, y)
browser. (x, y, value)
mobile.swipe (from, to)
mobile.home ()
Pluggable Actions
mobile.back ()
mobile.open.app (name)
terminate (status)
answer (text)
A.2. Pluggable Functions: Mobile Environments as An Example
We provide the following pluggable functions for Aguvis in the mobile environment. along with their corresponding descriptions.
14

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Pluggable Functions for AGUVIS
You are a GUT agent. You are given a task and screenshot of the screen. You need to perform series of pyautogui actions to complete the task.
You have access to the following functions:
| "naaee: "mobile moobi, home", "descriptica": "Press the home button")
"naam": backk" "description"; "Press the back butten")
"name": "mobile. long_p,
"description"; "Long press on the screen",
"parameters":
"type": "object",
"properties": ("x": ("type": "number", "description": "The coordinate of the long press"), "y": ("type": "number", "description"; "The y coordinate of the long press")), "required"; 1"x", "y"]
"name": "mobile. open app",
"description": "Open an app the device",
"parameters";
"type": "object", "properties"; ("app_name*: ( ("type": "string", "description"; "The name of the app to open")), "required"; ["app_name"]
"name": "terminate",
"descriptica": "Terminate the current task and report its completion status", "pa "parameters";
"type" "object",
"properties; ("status": ("type": "string", *enum": ["success"), "description"; "The status of the task")), "required"; ["status")
"name"; "answer" "description"; "Ansver question", "parameters": "type": "object", "properties"; ("answer": ("type": "string", "description": "The answer to the questicn")), "required"; ["answer"]
B. Data Curation of AGUVIS DATA COLLECTION
B.I. Detailed Source Dataset Statistics
We present the detailed statistical information of all training datasets utilized both the grounding and planning & reasoning stages. The statistics are shown in Table 10 and Table [1 respectively.
15

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table 10: The grounding split of AGUVIS DATA COLLECTION. Each example in this split consists of a single-step trajectory.
Data source
Platform
Instruction
#Trajectory
SeeClick Cheng et al. 2024)
Wehsite
Augmented
271K
GUIEnv Chen et al
2024a
Website
Augmented
328K
GUIAct (
hen et al.
2024a
Website
Original
67K
WebUI Wu et
2023)
Website
Augmented
57K
Widget Captioning
2020b
Mobile
Original
101
RicoSCA (Li al.
2020a)
Mobile
Original
173K
UI RefExp Bar et al. 2021
Mobile
Original
16K
RICO Icon (Deka al.
2017
Mobile
Augmented
16K
OmniACT Kapoor
2024
Desktop & Website
Original
7K
Total
1.036M
B.2. Prompt for Augmenting Planning & Reasoning Trajectories
Prompt for GPT-4o generating planning & reasoning data
Goal: {goal}
Previous Actions: (previous_act ions)
Given the current screenshot and the next ground truth action labeled "[current acttonnt_action.oo ""json
the action cemmands is:
(action_commands]
This element is highlighted in red bounding
the image.
Describe the situation in detail, fecusing on the goal and current observation. Ensure your reascning aligns with the geal and the labeled action, but avoid using the labeled acticn the highlighted bounding box reasening support, as they represent hinesight rather than predictive insight. Conclude with a clear, actionable instruction in one sentence. Aim to reason through the task as if solving it, rather than simply reflecting on the labeled outcome. Use the first-person perspective to represent the annotater's thought process.
We GPT-4o the foundational model to augment our integrated agent trajectory. In this stage, the Goal represents the target of the trajectory, Previous Actions is a stack of all previous low-level instructions, current.aco inttnt.action www.aacciinnnttteeeeeee and action commands is the representation of the current action in the form of pyautogul code within the dataset. We show the augmented examples generated by GPT-4o Figure 5 This augmentation data serves enrich reasoning trajectories and can be generated using open-source VLMs (Bai al. 2025 we leave exploring that approach for future work.
refers to the low-level instruction corresponding to the current action in the dataset,
Table 11: The planning & reasoning split of AGUVIS DATA COLLECTION.
Data source
Platform
Inner Monologue
Avg. Steps
#Trajectory
Mind Zheng et al.
2024a)
Website
Generated
7.7
1,009
GUIAct (Chen et al. 2024a)
Website
Generated
6.7
2,482
MiniWoB++ (Zheng ct al
2024b)
Website
Generated
3.6
2,762
AitZ (Zhang et al. 2024b
Mobile
Original
6.0
1,987
AndroxdControl [Li
2024c
Mobile
Original
5.5
13,594
GUI Odyssey [Lu et
Mobile
Generated
15.3
7,735
AMEX (Chai et 2024
Mobile
Generated
11.9
2,991
AitW (Rawles et al
2024b
Mobile
Generated
8.1
2,346
Total
35K
16

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
B.3. Human Study Augmented Data
B.3.1. QUALITATIVE HUMAN STUDY
Based our findings that our Augmented Planning and Reasoning Data improves the performance Aguvis, we conducted qualitative study augmented data. From the VLMMaugmented data, we selected 90 samples for human study and evaluated them according to specific criteria.
We determined that for augmented data to be considered successful, must:
Match the action type and action target elements of the ground truth,
Correctly describe the step's intention,
Establish clear connection between the step's intention and the overall goal,
Assist the agent in successfully completing the task.
Among the sampled data, we found that 86.7% demonstrated intermediate reasoning that aligned with the ground truth actions and the overall goal's ' action intention. The remaining 7.8% cases influenced by dataset noise (irrelevant unnecessary actions within the task), and 5.5% cases were due to misinterpretations of the action intention under clean data.
B.3.2. FAILURE CASES UNDER NOISY TRAINING DATA
We analyzed error cases in the generated data and identified several issues. Specifically, we found that unnecessary actions the training data can lead to the VLM failing to establish a connection between these extra actions and the overall goal, ultimately resulting incorrect reasoning and planning.
While these redundant actions do not compromise the trajectory's overall completeness correctness, they do introduce challenges for the VLM in generating accurate planning.
C. AGUVIS Training
C.1. Training Example Schema
Training Data Schema of Stage Grounding
Prompt
<| <|im_start >>system
You are a GUI agent. You are given a task and sereenshot the sereen. You need to perform series of pyautogui actions to complete the task. bask.</im_end) [immsstar|>Susee
|
Please generate the next according to the ui screenshot, instruction and previous actions.
Instruction: coverall_goad Previous actions: oprevious_actions) 

Generation
<lim_ siinsssar//// bassistant< rectti//eeeiienttt >os Action: (pyautogui function) <ldiff_marker|>
17

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
United Wi-Fi
Sag Sood mounly
What bring with you
Using
yorr
Prichng
Goal: Show me the page with information about Wi- Fi subscriptions.
Goal: Find the documents required to sell a car in New Jersey for which payments are being paid and it has single person on the title who can come to the store
Previous actions:
Previous actions: Step 1: Click on the 'Sell/Trade' link in the navigation bar to proceed.
Step 1: Click on the TRAVEL INFO to explore available options for Wi-Fi subscription information.
Step 2: Click on the Pre-paid Wi-Fi' link to access Wi-Fi subscription information.
Step 2: Select 'New Jersey' from the dropdown menu labeled 'In what state are you selling the car?'.
Action: wwwp:ppooouuucccccc/-0000011111 y-0.6833)
Action: pyautogui.click(-005993,, y-0.8633)
Generated Planning
Generated Planning
Thought: Identifying the 'Wi-Fi subscriptions' link on the United Wi-Fi page aligns perfectly with the goal of finding subscription information. available options and see dedicated link for subscriptions.
Thought: To determine the document requirements for selling New Jersey, need select options that accurately reflect the current status of the car's payments and ownership. Since payments are being made, will choose 'Yes' for that question to proceed and assess any further requirements or conditions this might bring into play.
considering
Low-level Instruction: Click on the WWi-Fi subscriptions' link to access information about Wi-Fi subscriptions.
Low-level Instruction: Click on the Yes' radio button to 'ree you making payments the car?'
Figure 5: Examples of augmented planning and reasoning data generated by GPT-4o. The position of the mouse in the image represents the ground truth click position in the training data.
18

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Training Data Schema Stage 2 Planning
Prompt
<lim_start Pssysten
You are a GUI agent. You are given a task and a screenshot of the screen. You need to perform series of pyautogui actions to complete the task. bask<</im_end]>> <lin_start
[Ivision_start><<] p
Please gencrate the next move according to the ui screenshot, instruction and previous ous actions.
Instruction: {overall_goal)
Previous actions: (previous_act
<lim_endl] end|>
Generation
<|im_stare sttss/asaat((
|>all
Thoughh: (Planning)
Low-level Instruction: [Iow-level Instruction]
<lim_ end
<lim start /astis/sssittttt/// / recipient |>os
Action: (pyautogui function)
<ldiff_marker/>
AGUVIS introduces novel explicit planning and reasoning training framework that differs from existing approaches. We illustrate these differences with visual examples in Figure 6 While existing training datasets utilize trajectory data to fine-tune agents, these approaches often involve agents directly outputting action commands (e.g., via pyautogui), bypassing the generation of observations, thoughts, and low-level instructions in natural language that correspond to actions. To elicit the reasoning and planning capabilities of vision- vision-language models and provide the model with richer context for action generation, scale up training datasets that explicitly require the model output reasoning and planning steps. Moreover, this approach enhances the interpretability of computer-use agents' behavior, laying solid foundation for future research.
C.2. Training Details
For AGUVIS based the Qwen2-VL backbone, set the maximum pixels for each image 1280 720 achieve better trade-off between performance and efficiency]] Following the SFT strategy in Wang al. (2024b), freeze the VIT backbone, we adopt the anyres strategy, which splits high-resolution images into multiple patches following (Li et al. 2024a). The maximum sequence length of tokens is set to 8192 for all models. We use Adam optimizer (Loshchilov & Hutter 2019) for both grounding and planning & reasoning training stages and employ a cosine learning rate scheduler with a warm-up ratio 3% steps. In the grounding stage, we introduce grounding packing strategy to enhance training efficiency. We conduct an ablation study using the grounding data of website platform to investigate the strategy effectiveness. We observe that it reduces overall GPU hours from 6 hours to I hour. Moreover, this strategy even marginally improve the performance of ScreenSpot website split from 73.3 to 76.8.
parameters during training. For AGUVIS based on the ViLaVA-OneViii
We train AGUVIS with a batch size of 128 for epoch each stage. The peak learning rate is set to le-5 for AGUVIS-7B and 5e-6 for AGUVIS-72B. Our codebase is based on Pytorch (Paszke et al, 2019) and Huggingface Transformers (Wolf et al. 2019). During training, we utilize the strategies DeepSpeed optimization (Rajbhandari et al. 2020), 1F format and gradient checkpointing save GPU memory. We train AGUVIS on a cluster H100-80G GPUs: AGUVIS-7E uses nodes and completes the grounding training within 5 hours and planning & reasoning training within 1 hour. AGUVIS-72B uses 16 nodes and completes the grounding training within 30 hours and planning & reasoning training within 6 hours.
'During preliminary experiments, observe that increasing the maximum pixels 1920 1080 does not yield significant improvements on ScreenSpot performance.
19

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Image Input
Contact
Abost
foathforrod
23 Schedules
2 Planner
@
Alerts
Biuue dopant, Hottrrsh:
eice alsebe
Find Location
Treert Year
Contact Us
Existing GUI Agent Data
Aguvis Collection Data
Prompt
Prompt
Please generate the next move according to the UI screenshot, instruction and previous actions.
Please generate the next move according to the UI screenshot, instruction and previous actions.
Instruction: trip from Boston Logan Airport North Station.
Instruction: Plan a trip from Boston Logan Airport North Station.
Previous actions:
Previous actions:
Step 1: praatopuuliiddk--000444440000000
Step 1: Click on the Trrip Planner' tab to begin
Step 2: prattoppiciiccxx0.22255.....11 pyautogulwwiteetxxx---Boooonoonn Airport') Step 3: yyautoopi.ciicck-00..222........ y0..7777766
planning the trip. Step 2: Click on the 'rom input field and type
"Boston Logan Airport'.
Step 3: Click on 'ooston Logan Int'l Airport,
Harborside Dr, East Boston, MA 02128, United
States set it my starting location.
Generation
Generation
have set my starting point as Boston Logan Airport. To proceed, need to set the destination to North Station, allowing the trip planner suggest routes.
Thought:
Low-Ieve Instruction: Click on the To input field and type 'North Station' the destination.
Action: (pautogui.clickkx-067766, y=0.4)
Action:
syautogui.clickkxo....f y=0.4)
wyautogui.writtttet--NNCC Station')
pyautoguiwrrteetexttNNttth Station')
Figure 6: Compared the schema of exisiting gui agent data (left), the schema of AGUVIS planning & reasoning data (right) includes explicit reasoning process with informative natural language previous action context.
20

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
D. Evaluation Benchmarks
In this section, we introduce more details of evaluation benchmarks used in our work.
D.1. GUI Grounding Evaluation
ScreenSpot. ScreenSpot (Cheng et al. 2024 is a typical benchmark designed specifically for GUI visual grounding. consisting of 1.2K single-step instructions and coordinates of the target elements. This dataset encompasses a variety of grounding instructions tailored for mobile, desktop, and website platforms, and categorizes element types into text and icons/widgets. The benchmark assessed under two distinct settings: (1) Original Instructions: models perform grounding actions directly following the original instructions; and (2) Self-plan: models required generate plans in natural language based on the original instructions before executing grounding actions.
D.2. Offline GUI Agent Evaluation
Multimodal-Mnn22Weh We utilize Multimodal-Mind2Wer (Zheng al. 2024a for evaluating the offline planning capabilities of GUI agents websites, which builds the original Mind2Web (Deng al. 2023). We report element accuracy (Ele.Acc), Operation F1 (Op.F1), and step success rate (Step SR).
In Table 2 for Multimodal Mind2 (Zheng et al., 2024a), we only report element accuracy for SeeClick (Cheng et al. 2024) and CogAgent (Hong et al. 2024). This is because the original SeeClick and CogAgent models were evaluated on Mind2Web (Deng al. (023). not Multimodal Mind2Web, making the examples misaligned and incomparable. Therefore, referenced the results from UGround (Gou et al. 2025), where they report the element accuracy of the SeeClick and CogAgent models on Multimodal Mind2Web, striving comprehensively present all previously representative methods.
Following the setting in Li et al. (2024c), we randomly sample 500 step-actions from AndroidControl full test set to create subset, and we report the step accuracy on out-of-domain (OOD) data within both high-level and low-level tasks. The high-level task setting necessitates that the model plans and executes actions, whereas the low-level task setting requires the model to simply adhere to human-labeled instructions for executing the next-step action.
AndroidControl.
D.3. Online GUT Agent Evaluation
Mind2Web-Live. We adopt Mind2Web-Live (Pan et al. 2024) to evaluate GUI agents' online planning, a derived dynamic data set from Mind2Web, comprising 104 real-time interactive web tasks. It evaluates whether each required step within a task has been successfully completed and uses the task success rate (Task SR) as the reported metric. The original Mind2Web-Lin is built with WebCanvas (Pan et al. 2024 which is a text-based agent framework. To better accommodate the unified observation and action space of pure vision models, we utilize BrowserGym (Drouin et al. 2024) as the evaluation environment for online web tasks which provide support for pure vision-based agent models. BrowserGym is browser testing environment built on the Playwright (Microsoft, (024) engine. We incorporate all Mind2Web-Lan tasks and evaluation into BrowserGym, involving registering all Mind2Web-LLive tasks, setting up the entry points for these tasks, and porting the Mind2Web-Live evaluation functions BrowserGym.
As Mind2Web-Live is a text-based benchmark, we have to adapt its evaluation function to suit our pure vision-based model. To achieve this, we introduce the two modifications following:
For the Mind2Web-Lin benchmark's 's click verification, we adapt our coordinate- booddinate-based approach by comparing the ground truth CSS selector's bounding box (when available) with our click coordinates, as we cannot directly identify HTML elements.
Similarly, for input validation, we retrieve and compare the value of the ground truth input element (if present) with the expected value, circumventing the need for precise HTML element identification based on CSS selectors.
The Mind2Web-Live environment relies on real-world websites, many of which implement detection systems for automated browser testing and reCAPTCHA challenges. These factors created difficulties during evluation on the Mind2Web-Live dataset, resulting in a lower task success rate (Task SR). Specifically, we observed the following websites to have significant issues with automation detection:
21

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
kohls. Model using the search functionality on the Kohls website through Playwright directly results in 502 Bad
Gateway error.
target. We are unable to open target's job website using Playwright due to network connection error.
united. We are unable to open united website using Playwright due to network connection error.
In addition to the websites that were consistenly prone to failure, several other sites intermittently blocked our Playwright access during testing. In total, we encountered 18 network errors and reCAPTCHA tasks that the model was unable to complete, preventing our model from scoring on these 24 tasks.
AndroidWorld. AndroidWorld (Rawles et al. 2024b) is a benchmark operating on an Android virtual environment, capable of dynamically instantiating with randomly generated parameters to generate unique tasks for automatic evaluation. It spans 20 real- world applications, encompassing 116 diverse tasks. To assess the pure vision agent models, we follow the instructions in Rawles et al. (2024b), installing Pixel 6 phone simulator environment. The benchmark incorporates fully automated task-level evaluation system that automatically assesses whether a state has successfully completed a designated task. The AndroidWorld environment supports optional inputs such as Sct-of-Mark (SoM) and textual AXTree information, which most multimodal models currently rely on to complete tasks. However, we solely use raw screenshots as the observation input and restrict the model coordinate-level actions and basic mobile functions.
computers to serve the experimental
MobileMfniWob. MobileMiniWob (Rawles al. 2024b) is the instantiation of 92 tasks from MiniWob++ (Zengg et al. 2024b) in the AndroidWorld environment. Thus, adopt the same observation and action space used in AndroidWorld and use real-time evaluation function to determine task success.
D.3.1. PROMPTS FOR USING GPT-40 AS PLANNING MODEL
In all online experiments, we employed two settings: GPT-4o as the planner, AGUVIS-7B as the grounder, and AGUVIS-728 as both the planner and grounder. For experiments where AGUVIS-72B served as both the planner and grounder, the prompt was straightforward: we only needed to provide AGUVIS-72B with single prompt at each step, and it could independently handle reasoning, planning, and grounding. We use prompt for forcing plan to improve AGUVIS-72B's performance on the online experiments, as illustrated in Appendix E.221
In the GPT-4o + AGUVIS-7B setting, the situation was more complex. Two key challenges needed to be addressed: making GPT-4o's planning usable by AGUVIS-7B and determining which actions required AGUVIS-7B for ground- ing. To address these challenges, we modified GPT-4o's prompts based on Mind2Web- (BrowserGym) and An- droidWorld to enable to delegate grounding actions to AGUVIS-7B when necessary and to share its planning outputs with AGUVIS-7B. Specifically, we append im.start >assi stant< recipient <>all \nThhught: {GPT-40 Thought A on: (GPT-4o 4o Low-level Instruction) the end of the prompt and therefore let AGUVIS-7B generate grounding actions based on GPT-4o's response.
Table 12: Prompt used for the planning model in Mind2Web-Live, modified from the prompt in Drouin et 2024
Instructions
Review the current state of the page and all other information to find the best possible next action to accomplish your goal. Your answer will be interpreted and executed by a program, make sure to follow the formatting instructions.
Goal: {Goal}
Observation of current step
Current URL:: {URL.
History of interaction with the task: (History)
Action Space
8 different types of actions available.
Continued on the next page
22

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table 12 - Continued from the previous page
Instructions
Review the current state of the page and all other information to find the best possible next action to accomplish your goal. Your answer will be interpreted and executed by a program, make sure to follow the formatting instructions.
noop(wait_ms: float = 1000)
Description: Do nothing, and optionally wait for the given time (in milliscconds).
send_msg_to_uue toxustt(t:xt: str)
Description: Sends message to the user.
scroll(delta x: float, delta float, relative: bool = False)
Description: Scroll horizontally and vertically. Amounts in pixels, positive for right or down scrolling, negative for left or up scrolling. Dispatches a wheel event.
fill(element: str, value: str)
Description: Fill out form field. It focuses the element and triggers input event with the entered text. It works for <input>>, <textarea>> and [contenteditable] elements. The 'element' parameter represents the semantic information of the element you want to fill.
click(element: str, button: Literal['eeft', 'middle', 'right'] = 'left')
Description: Click an element. The 'element' parameter represents the semantic information of the element you want click.
cblelick str, button: Literal['lett 'middle', 'right'] 'left')
Description: Double click an element. The "element' parameter represents the semantic information of the element you want to double click.
hover(clement: str)
Description: Hover over an element. The 'element' parameter represents the semantic information of the element you want to hover over.
keyboard.press(kee str)
Description: Press a combination of keys. Accepts the logical key names that are emitted in the keyboardE. vent.key property of the keyboard events: Backquote, Minus, Equal, Backslash, Backspace, Tab, Delete, Escape, ArrowDown, End, Enter, Home, Insert, PageDown, PageUp, ArrowRight, ArrowUp, FI F12, DigitO Digit9, KeyA KeyZ, etc. You can alternatively specify single character you'd 'd like to produce such as "a" or "#" Following modification shorteuts are also supported: Shift, Control, Alt, Meta.
Only single action can be provided once. Example: fill(' "comment text area', 'This is an example') Note: you are on mac so you should use Meta instead of Control for Control+C etc.
Table 13: Prompts used for the planning model in Android modified from the prompt in (Rawles et al. 2024a
Instruction
You are agent who operate Android phone behalf of user. Based user's goal/request, you may Answer back if the request/goal is question (or chat message), like user asks "What my schedule for today?".
Continued on the next page
23

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table 13 - Continued from the previous page
Complete some tasks described in the requests/goals by performing actions (step by step) on the phone.
When given user request, you will try to complete it step by step. At each step, you will be given the current screenshot and history of what you have done (in text). Based on these pieces of information and the goal, you must choose to perform one of the action in the following list (action description followed by the JSON format) by outputing the action in the correct JSON format.
If you think the task has been completed, finish the task by using the status action with complete as goal status: (*action type": "status", "goal status": "complete")
If you think the task is not feasible (including cases like you don't have enough information or can not perform some necessary actions), finish by using the 'status" action with infeasible goal status: {"actionntyyl type": "status", "goal_status" "infeasible")
Answer user's 's question: ('action type": "answer", "text": "answer text"}
Click/tap on an element on the screen. Please describe the element you want to click using natural language.
("action.type .tppe::tteye"eeee "click", "target":
gegget.eleeent.deee descrrption
LLong press on an element on the screen, similar with the click action above, use the semantic description to indicate the element you want long press: ("action type": "long press", "target": target element description): Type text into text field (this action contains clicking the text field, typing in the text and pressing the enter, so no need to click on the target field to start), use the semantic description indicate the target text field: ["action type": "input text", "text": text input, "target": target element description)
- Press the Enter key: pncctionttpe""" "eeyboardeenter""
Navigate to the home screen: ("action type": "navigate }
Navigate back: ""action_type". "navigate_hack")
Scroll the screen or scrollable UI element in one of the four directions, use the same semantic description as above if you want scroll a specific UI element, leave empty when scroll the whole screen: ("action.type" htw.aaa.ttggee.llmee.eeeeeeee -Open an app (nothing will happen if the app is not installed): {"action_type"" "open_app". "app_name". name) Wait for the screen to update: ("action_type" :tpee: "wait")
"scroll", "direction": up, down, left, right, "clement":
Guidelines
Here are some useful guidelines you need to follow:
General:
- Usually there will be multiple ways to complete a task, pick the easiest one, Also when something does not work as expected (due to various reasons), sometimes simple retry can solve the problem, but if it doesn't t (you can see that from the history), SWITCH other solutions.
Sometimes you may need to navigate the phone to gather information needed to complete the task, for example if user asks "what is my schedule tomorrow", then you may want to open the calendar app (using the 'open app' action), look up information there, answer user's question (using the 'answer' action) and finish (using the "status' action with complete as goal status).
For requests that are questions (or chat messages), remember to use the 'answer' action to reply to user explicitly before finish! Merely displaying the answer on the screen is NOT sufficient (unless the goal is something like "show me ..."..
If the desired state is already achieved (e.g.. enabling Wi-Fi when it's already on), you can just complete the task.
Action Related:
Use the 'open app' action whenever you want to open an app (nothing will happen if the app is not installed),
do not use the app drawer to open an app unless all other ways have failed.
Use the 'input_text" action whenever you want to type something (including password) instead of clicking characters on the keyboard one by one. Sometimes there is some default text in the text field you want to type in, remember delete them before typing. For 'click', 'long-press' and 'int"' the target.clememnt parameter you choose must based on a VISIBLE element in the screenshot.
Continued on the next page
24

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table 13 - -Continued from the previous page
Consider exploring the screen by using the 'scroll' action with different directions to reveal additional content. -The direction parameter for the 'scroll' action can be confusing sometimes as it's opposite to swipe, for example, to view content at the bottom, the 'scroll" direction should be set to "down". It has been observed that you have difficulties in choosing the correct direction, so if one does not work, try the opposite as well.
Text Related Operations:
- Normally to select certain text on the screen: (i) Enter text selection mode by long pressing the area where the text is, then some of the words near the long press point will be selected (highlighted with two pointers indicating the range) and usually text selection bar will also appear with options like 'copy', 'paste", 'select all', etc. (ii) Select the exact text you need. Usually the text selected from the previous step is NOT the one you want, you need to adjust the range by dragging the two pointers. If you want to select all text in the text field, simply click the 'select all' button in the bar.
At this point, you don't 't have the ability to drag something around the screen, so in general you can not select arbitrary text.
To delete some text: the most traditional way is to place the cursor at the right place and use the backspace button the keyboard to delete the characters one by one (can long press the backspace to accelerate there are many delete). Another approach is first select the text you want to delete, then click the backspace button in the keyboard.
To copy some text: first select the exact text you want to copy. which usually also brings up the text selection bar, then click the 'copy' button in bar.
To paste text into text box, first long press the text box, then usually the text selection bar will appear with a 'paste' button in it.
When typing into a text field, sometimes an auto-complete dropdown list will appear. This usually indicating this is enum field and you should try select the best match by clicking the corresponding one in the list.
E. Analysis
E.1. More Training Ablation
E.1.1. TRAINING STRATEGY ABLATION
To further demonstrate the contribution of Stage 1 (GUI Grounding), Stage 2 (GUI Planning & Reasoning), and their combination to model training, we conducted an ablation study. Specifically, we designed five experimental settings on AGUVISQQENZ-vL and AGUVISLLAVA.OV:
Stage 1 + Stage 2 corresponds to the staged configuration AGUVIS used in our paper, where Stage is followed by Stage 2 sequentially.
Stage 1 + Stage 2 represents joint training setup, where two stages are combined into a training process.
w/o Stage x indicates the absence of the respective stage in the setting.
Note that for the setting of removing Stage 2 (w/o Stage 2 or w/o Stage 1 & 2), the models are fine-tuned on the corresponding task-specific dataset for planning tasks.
From the first two rows in Table 14 it can be observed that the differences between models trained with Staged Training and Joint Training setups are relatively minor. However, a clear trend emerges: models trained using the Joint Training setup perform better on GUI grounding tasks but exhibit inferior performance on datasets requires planning ability such as MM-Mind2Web and AndroidControl High-level. This trend implies grounding data in Stage is more abundant, dominating the optimization process and biasing the model toward grounding tasks. In contrast, the data in Stage 2, which combines planning and grounding, is of higher quality and better aligned with the agent's deployment scenarios. This rationale underpins our decision to position Stage 2 later in the training sequence.
Moreover, it is observed that compared to AGUVIS trained through both Stage and Stage 2, the model trained with only Stage 2 data maintains similar performance on MM-Mind2Weh and AndroidControl but exhibits a notable decline
25

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
in GUI grounding performance SoreenSpot. This suggests that the stability Mind2Web and AndroidControl be attributed Qwen2VL''s pre-training on natural image grounding. However, the diverse image and domain requirements of the SpotenSpot GUI grounding test set highlight the necessity of extensive and varied grounding training from Stage 1. This training is essential for improving the grounding performance required for cross- cross-platform GUI agent model.
To verify this analysis, we conduct the same ablation study on the LLaVA model, as shown in Table 1. From the results, we can see that the original LL did not undergo extensive natural image grounding training during the training process, making it insufficient for LLaVA to excel when only Stage or Stage is conducted. When both Stage I and Stage 2 performed, LLaVA can be significantly improved. even surpassing previous SOTA results. This validates the above analysis and further demonstrates that method is model-agnostic and universally applicable to popular VLMs like Qwen2- VL and LA.
Table 14: Ablation study of AGUVISQweN2-VL on training strategy.
Multimodal-Minn22Weh
AndroidControl
Settings
ScreenSpot
Cross-Taske
Webss-
Cross-Domain
High-Level
Low-Level
Stage + 2
84.4
58.5
55.4
54.8
61.5
80.5
Stage 1
85.0
56.1
53.1
55.6
59.2
80.9
w/o Stage
81.8
50.9
45.2
45.3
58.0
75.6
w/o Stage
77.4
59.7
55.3
55.8
58.8
79.8
w/o Stage
55.3
50.9
44.9
47.7
59.1
59.2
Table 15: Ablation study of AGUVISLLAVA-OV training strategy.
Multimodal-Mind2Webb
AndroidControl
Settings
ScreenSpot
Cross-Task
Cross-Websire Website
Cross-Domain
High-Level
Low-Level
Stage 1 2
81.2
55.3
50.0
50.8
60.7
82.4
w/o Stage 2
70.0
43.4
39.0
40.7
54.9
65.6
w/o Stage
71.3
42.5
40.3
42.8
61.4
80.5
w/o Stage &
3.8
33.8
30.5
32.4
50.4
50.0
E. DATA STRATEGY ABLATION
To investigate the impact of different device domain datasets within unified action space, designed three settings on the MM-Mind2Web dataset: (1) training with the complete dataset comprising both Web and Mobile data, (2) training using only the Web data, and (3) fine-tuning exclusively on the MM-Mind2Web dataset. All three experiments include fine-tuning on the MM-Mind2Web dataset.
Table 16: Ablation Study of The Impact of Mobile Data on MM-Mind2Wehh
MM-Mind2Web
Model
Training Data
Cross-Tase Task
Cross- Website
Cross- Domain
Web + Mobile (Stage 2 Equivalent)
58.5
55.4
54.8
AGUVISQWENZ-VL
Web Only Mind2Weh Only
53.1
50.3
52.2
50.9
44.9
47.7
Web Mobile (Stage 2 Equivalent)
55.3
50.0
50.8
AGUVISILLVA.OV
Web Only Mind2Web Only
44.9
43.5
42.1
43,4
39.0
40.7
26

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Table 17: Ablation Stuyy of the Impact of Inner Monologue.
Multimodal-Mind22all
AndroidControl
AGUVIS
ScreenSpot
Cross-Tase
Cross- Website
Cross-Doman
High-Level
Low-Level
AGUVIS
84.4
58.5
55.4
54.8
61.5
80.5
AGUVIS w/o IM
79.3
53.4
53.7
54.9
60.3
69.1
The experimental results, presented in the Table demonstrate that training AGUVIS with both Web and Mobile data consistently outperforms the setting trained exclusively on MM-Mind2Web. This performance gain underscores the contribution of Mobile data to enhancing cross-device domain generalization in the Web domain, validating the effectiveness of our cross-platform data.
In addition, conducted ablation study the role of incorporating inner monologue (IM) in training. The result shown in Table [7 demonstrated clear performance gain from inner monologue. This gain can be attributed to two key factors: the use of inner monologue enables the model to elicit reasoning about the current step while also serving as context to facilitate more effective planning for subsequent steps. Additionally, incorporating low-level instructions from the training data improves the accuracy of the model's action execution, as demonstrated both the Screenspot and AndroidControl low-level tasks. This mainly because the low-level instructions of inner monologue act as atomic instruction and grounding action pairs, also enhancing the grounding ability of GUI agents.
E.2. Planning Analysis
E. PROMPTS FOR SELF-PLANNING AND ENFORCED PLANNING MODE.
In Appendix C.I we present the training data schema for Stage and Stage 2. We use the special token recipient) |> along with os or all to control whether the message content is an inner monologue or pyautogui action command. Thanks to this design, can rrecppa pient during the inference phase to control the content generated by the agent model.
In the Enforced Plan Setting, we employ the < < rerriipient |>all] Inhoughhh prompt to compel the model to generate a planning phase following this. This setting explicitly requires the model to utilize inner monologue for high-level reasoning before taking actions. While in the Self-plan setting, we do not add any word after < rreipppent >>, so the model can choose to generate os to directly produce a pyautogui command, or generate all to first create natural language reasoning and then generate a yyautogu. command. Thus, the model can autonomously determine whether generate planning thoughts based on the complexity of tasks.
As noted in Section 4.5 the enforced planning resolves approximately 20% of grounding enrors by encouraging the model to carefully consider the task context, potential ambiguities, and available UI elements before committing to action.
Prompt Template For Self-plan Setting
<lim_start >system
You are GUI agent. You given task and screenshot of the screen. You need to perform scries of pyautogui actions complete the task. i stan_ />user / | >llaae generate the next move according <Ivision_start|><<]
to the vi screenshot, instruction and previ ous actions.
Instruction: (geal)
Previous actions: Iprevious_action

 aaraaara/DDssssstttt/eeeeeeeaaar
27

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
Prompt Template For Enforced Plan Setting
<lim_start |>system
You are GUT agent. You given task and screenshot of the screen. You need to perform series of pyautogui actions complete the task. i clim_start/Puse htapl/aanaaaa////sssaaaaaa paiiss end|>Please Lesee generate the next move according <lvision strrssttrr///// image
to the ui screenshot, instruction and previous actions.
Instruction: (overall_goal)
Previous actions: ppreviou_aactions
<lim_end]]>
<|im_ | >>all
Thought:
28

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
E.2.2. CASES OF INNER MONOLOGUE BONUS
Task Instruction: add a new file
Task Instruction: send current page
pysutogu.click y4)
ppuutnguicclcckgeee y+0.9066)
Thought: To stare the currett pagt, need to find and cluck the
Thought: add new file, I need select the 'New option likely open dailgg grompe to create a new file, allgzing with the goal of adding a new tile to the direetor..
share whidh typically
netwerk
symiall This is saually lecated in lncener luoliar minu.
Actions Click the ahure live the browner chane the curment page.
Action: Click
option sysutogui.clckks y.6336 0.2555) e
prostogul clickge- 9792, snay @
Task Instruction: go to personal homepage
Task Instruction: rate this product with four stars
pyuuingu..lcckq....oo www.kp-..ckk0.0.. 00000000000000333303 y.0.333) X
prautogui 2271, 2-0.2926) X
Thought: reach personal homepage, interxtt with the user profle element which is rypically represented by an avatar usemame This elenzent usially cliclatle and lead user's profile page.
Thought:
goal eating product with
star, I need to clics the fourth ste the rating section.
This will ensure the product roted with four stars, allgning with the task requieemnnt
Action: the use prodil kosse usemaate at the top right conee to go to the perional hamepagee
Action: Click on fourth star in satling seetion to pooduct with four stars.
pysuiogui dlickspet 0615, yeou0120) E
Task Instruction: switch to song lyric
Task Instruction: change dark mode options
prautogui.chcch 0.S.y 0.9252) X
pysutogul.clccog 0.7386, 0,3375) X
Disolay
Light moce
Dartmoce
Schecola Dark
Mou Dark rodiypotion
trightness ivel
Thoughe: To switcd to song tottss I Ined to fled and clickon the Jyfis buttos, wöich typceill supeesented loon. This usually located ut the bottom the screen among ooler playlack coninia.
Thought: To rhunge the derk mode eptionss I need sellct the oprions sprion. This additionss settings relaned dark mode.
Action: Click on the lyrics izon the bottom of the screen swittch lyties.
Action: Click on More Derk mode options explore further settings
pyoutogul. clidax- p00622 B
pssutreul.d 2555, 9-6712
Figure 7: Benefits of inner monologue in autonomous GUT interactions across Desktop, Website, and Mobile environments. Gray indicates output without inner monologue, yellow with inner monologue.
29

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
E.3. AGUVIS Trajectories Examples on Online Evaluation
E. MIND2WEB-LIvE CASE: AGUVIS-72B AS PLANNER AND GROUNDER
Task
Show me the list of Men's Blazers, Black, Size M uniglo.
Planner
Grounder
Aguvis72B
Step 1
Step
Step 3
Thought:
Thought: To lid, Menis
Thought: find man's blazers, to use the search bar to input the keyword blazers This will help locate the spec fie categerr clothing I'm interested in.
Blazers, 1 noed bo narigane to the MEN exction firtt The '4EN is visible in logical next stop access mon's clsching categories.
Blasers, need to navigate to the relevust section. The sourch available, which can be used dinectly search for Naaeer'
novigation
Low level instruction: Click on the MEV the nasigation
Low-- nevel instraction: Click en the search bar to start typing Nazcess
Low- lovel instruction: Type filecers' into the search bar find men's ' blozers
Action:
proutogui clickie 0,203, y 0.538)
Action: pyw.troeel www.:ppsssslleeeeeeooooo
pouttppl pwertppiiiettepmeeaaae "staneer)
Step 4
Step
Stop
Thought: aasd to adlet the Blacers' option under MIN/ Duterwer to procoed floding man's blaces black, sior M.
Thoughh: To find the specific black aioee
Thoughh: To Gltte the list of menin biunern to shew only black opeions the BLACK color opoion from the coloe filbe munu.
M hisner,, I need apply the appeoprtaze filters.
Color crusial for narrowing down the uptions to black blazers.
instruction: Sizeers opcioo under MEN Outerwear the list men's blazens.
the
instruction:
the
Low- level instructions Click on the "Goloe to select black.
"BLACK color option to filter the blusers by black color.
Action: pww.ttopii.lllllpd..... yo00364))
Action: prwatoouiccccheeeeooooooo (.. 1-0.251)
Action: proaztngois 11.00--0,300, y-0.334)
Thoughtt To achieve the goal
Thought: need the stze tiltr narrow down resuls The mest lngiral to interset with the sine tiler opcion.
fittring for mons a boak hisomm size
34, I need to select the b4 size sprioc.
frum the lixi. This will update the weathh
reeults show only the blazers
DONEE
available in
Luwelleeel instruction: Click on the 'Stee' select size
Low Instroction:
chackhox neat size treolts wize blooees.
the
Action: telko 0440, (.353)
Action: pyudtogii.diddo y
Figure 8: Example of AGUVIS-728 as planner and grounder executing on Mind2Web-Live task. Due to space limitations, we present here the trajectory generated guided by Thought.
30

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
E.3.2. MIND2WEB-LIVE CASE: GPT-40 AS PLANNER AND AGUVIS-7B GROUNDER
Task
Find walkthrough guide for Assassin's Creed Valhalla on ign
Planner
GPT-4o
Grounder
Aguvis7B
Step
Step 2
Stgp 3
Thought: To find willchruugh guide Creed Valhalls shouldd use the sourch functionality. The wearch loon visible on the left sidetur. Clicking should allow the search querry.
Thought: The wanch har now Viaible. To find walkchrough guide for Assassin's Valhaila, the scarch query in the search bar.
Thought: sourch
for
AAsssssi''s Creed Valhalla walkthhoughh displaayed. The first resalt 'Assassin's Creed Valhalls, to contain the wa walltthouuhh guide we are looking should to proceed.
likely
Low instraction:
bearch input field), Assassin's Creed Valhalia walkthrough)
Low-leew -evel instruction: <leck(sseach icon')
Low-level instruction: click(Assusinn'' Creed Valhullle)
Action:
pysutogul clickge-a, (0.3117 pyzutogul werttetest Asssssinns Creed Vollallla wallshhoughh)
Action: pyontogul y00. y0040000
Action: prautogul (19.)g-...9).......
Step
Step
Step
Assossioo
Thought: find ough guide, should look for relevunt sections or links like Overview, Playllitt, and Reviews. The walkthrough guide likelly to be under thear tabte, possibly the Overview or Plzylists tab.
Thought: To find the walkthruugh guide, I should look section or link specifically mentions guides walkchroughs. Since 1 don't see any relevant sections immetiately. should screll further down the page to see content revealed.
Thoough:: currently the Assassin's s Creed Valhalla page IGN, have clicked on
this puge. page
To find the walkihrough guille I should look for a section link that specifically mentions walkchrough" "uide."
Low-len Low-level instruction: click Overview
Low-level instruction: scrollgo, S000
Low-lewe instruction: scrolk00 500)
Action: pysutogul pysutogul.ickko00311333 2115 7-..7200)
Action:
Action:
Step?
Step 8
Wellhroagh
Thought: There also "Walkthrough" link under the "Top Grande Sections" the right side. Clicking the "Walkhhreugh" link should
*DONN***
directly to the walkchrough guide.
Low-lewe level instruction: 3ickhwoikthhggh))
Action: pynutogul clickg-- 005520 (.711)
Figure 9: Example of GPT-4o as planner and AGUVIS-7B grounder executing on Mind2Web-Live task.
31

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
E.3.3. ANDROIDWORLD CASE: AGUVIS-72B AS PLANNER AND GROUNDER
Delete all but one of any expenses in pro expense that are exact duplicates, ensuring at least one instance of each unique expense remains.
Task
Planner
Grounder
Aguvis72B
Step
Step
Step
Thought: schioee the gool deketing duplicate expentees, I Trnedd to aaxesa the all expemss, The curmet seneen shose recett expenom, to navigate menu lcon
Thought: lo achiove the geal of deleting deplicate expastess need trspenne increnore and icentiif deplicates The Espense Logs option te vitthle in the
Thought: To schivee the geal of mueraging irsprreo the ltapanns, need to open the app first: The of apps includes tho Expense, which ts pelevant task hand.
left to blaely the gateway to accessing more options, including the full list copersso.
Low level instruction: Open the Pro Expenso usiing resiikc en
Low level instruction: Click on Excerro Loggi to view the list of incparno.
Instruction:
left
optroma.
Action:
mebiie apoleer yuumee-
Action: pywatnger cloppeeccccoeeeeeo.oo o 500,
Action: po aateega www.ttgu..chriieeaaa... ciriigeet.27,
Sactvi
Thought: luced deicce one of the duplicate Public Trunsit estr is to gis gual of nemosing deplicate rspenses.
Thought: need delete depicaae entras, erouting less: one unique entry remaiss. The curnett focus is on the Public Trmitt enny, which soons to be duplianee Ishold deiee tiis entr meet the gool.
Thought: I need to confirm the deletion the duplicnte Public 'haraal'
only unique entry
Low level lnstruction: Click the "CONFIRM button delete the solested duplicate entry.
Low-level Instraction: the "Pulli Transtt entries to delete it.
Low level Instruction: Click on delmm lern nest in the "hiblic Traner entry semoor
Action: proutopul didoy 0,545, (-1.614)
Action: pww.togu.ccccceee MM9 y+0.893)
Action: pouutogui dlicag- 5S3, y-1:233)
*DONE**
Figure 10: Example of AGUVIS-72B as planner and grounder executing on AndroidWorld task. Due to space limitations, we present here the trajectory generated guided by Thought.
32

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
E.3.4. ANDROIDWORLD CASE: GPT-40 AS PLANNER AND AGUVIS-7B AS GROUNDER
Send a text message using Simple SMS Messenger to +16597910719 with message: Beauty is in the eye of the beholder.
Task
Planner
GPT-4o
Grounder
Aguvis7B
Step
Step
14
Thought Simple Moosegger app mansage.
Thoughh Inpur phone number
Thought: conversation
Tudl Number.."
send the text message.
Instriction: type':
Low- -evall Instruction: ('aclion type:: spp', aap. "Stmple Messenger")
-evvll instruction: ('acioo, "dick', "uaget": 'Start conversation")
"Input "ses""
"sangei'
Actionn:
meonioggs clccko- yea 1204
Action::
presntiogas weileetreeeeeeee ssage 1899999928999
Step
Step
Strp
Thonght:: Impat the meoage "Roeaty is in the eye of the behatdler? isto the mrasagee fold.
Thought: Tap checkmaris contirm phone pocceed icessage Inpur screen.
Thought: Send the message by tupping the
Low-level Instraction: ('action, type": 'Input pext, bext" "Beuity in the eye of the behoider"; "torget": : "Type mensaae....
Low- Instraction: ("action, hype "cuck" 'send button with toon')
level instruction: ('action, type' "slckk; "Larpet 'chedomark lcon')
Action:
panstogua. dwwpppedddeeee.t....o pebua075) psautoput wette Bearry the epo awteldenc)
Action: www.nogsscchhlppe...oo 9407, 1403)4
Actions powslogui a llligei ymtl055
Step
*DNNE*
Figure 11: Example of GPT-4o as planner and AGUVIS-7B as grounder executing on Android World task.
33

---

AGUVIS: Unified Pure Vision Agents for Autonomous GUT Interaction
E.4. Case of AGUVIS Generalization in Real-World Scenarios
YmU
Ymu
loek
ralU pou piarey
G MOMENTS
GOLDEN MOMENTS S
Thought: To proceed with finding a flight, I need to dismiss the privacy pop-up to access the search functionality.
Low-level instruction: Click NN, thanks' to dismiss the privacy pop-up.
Action:
pyautogui. (atp:pgiiicicckk-04448...,. y=0.660 y=0.033
HERo
HERO
HERO
Reserve Parking Now Save
Thought: To proceed with contacting support, need to close this overlay to access the main page where the 'Contact Us' or 'Support' link is likely located.
Low-level instruction: Click the X' button to close the cookie consent overlay.
Action: pyautogui.clickxx007704. y=0000)
Figure 12: Case of AGUVIS generalization in real-world scenarios: closing cookie pop-ups, which an out-of- dommnnnn situation in our training data.
34


Reviews:
[{"review_id": "gcUB73Pjv4", "reviewer": "Reviewer_cBz9", "summary": "This paper introduces AGUVIS, a vision-based UI agent designed to operate across diverse digital platforms. The authors collect data from existing resources and do some essential augmentation. They then leverage a vision-language model to train AGUVIS in two stages, grounding and planning, to improve interaction capabilities. The framework is evaluated on multiple datasets, including grounding, offline agent and online agent benchmarks, demonstrating strong performance across various GUI benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and positive assessment of our work. We're particularly encouraged by your recognition of our comprehensive evaluation benchmarks, the engineering effort involved in dataset aggregation and standardization, and the value of our open-sourced models and datasets to the research community.\n\nRegarding your questions and comments:\n\n> **Q1: On Inner Monologue's Contribution to Grounding Performance**\n\nThank you for this insightful question. You correctly observed that while Inner Monologue is introduced in Stage 2, it also enhances grounding performance, which might seem counterintuitive. Let us clarify:\nIn standard trajectories without inner monologue, the data structure is:\n> (High-level goal G, observation $o_1$, action $a_1$, observation $o_2$, action $a_2$, ...)\n\nWhen we augment with inner monologue, we introduce low-level instructions, transforming it to:\n\n> (High-level goal G, observation $o_1$, low-level instruction $a_1^{inst}$, action $a_1$, observation $o_2$, low-level instruction $a_2^{inst}$, action $a_2$, ...)\n\n\nThis transformation creates high-quality explicit instruction-action pairs ($o_i$, $a_i^{inst}$, $a_i$) within each step, essentially embedding \"grounding examples\" throughout the trajectory. The model learns to:\n1. Interpret high-level goals into precise low-level instructions\n2. Ground these instructions to specific UI elements\n3. Generate appropriate actions\n\nAs shown in Table 6, removing inner monologue reduces performance on ScreenSpot from 84.4% to 79.3% and also has a strong impact on AndroidControl Low-Level tasks (80.5% → 69.1%). This suggests that the ability to decompose tasks into explicit low-level instructions significantly improves grounding precision.\n\n> **Q2: On Self-Plan vs. Enforced Plan Settings**\n\nThank you for pointing out this confusion. As shown in Appendix E.2.1 and Figure, the difference between these settings lies in how we prompt the model in response:\n\n**Self-Plan Setting:**\n    \n```    \n
assistant
\n[The model decides whether to plan first with all or directly execute actions with os]\n``` \n\n\nIn this setting, the model autonomously determines whether to generate planning thoughts based on task complexity. For simple tasks like \"Click the 'Buy' button,\" it might directly output:\n\n```\n
user\nClick the 'Buy' button.\n
\n
assistant
os\npyautogui.click(0.34, 0.45)\n
\n``` \n\nWhile for some implicit tasks, it might choose to plan first:\n\n```\n
user\nSend current webpage.\n
\n
assistant
all\nThought: To share the current page, I need to find and click the share icon, which is typically represented by a network or link symbol. This icon is usually located in the browser's toolbar or menu.\\nAction: Click the share icon in the browser to share the current page.\n
\n
assistant
os\npyautogui.click(0.34, 0.45)\n
\n``` \n\n**Enforced Plan Setting:**\n    \n``` \n
assistant
all\nThought: [The model is forced to generate planning thoughts before actions]\n```\n\n\nThe enforced plan setting explicitly requires the model to engage in high-level reasoning before taking actions. As noted in our error analysis (Section 4.5), this enforced planning resolves approximately 20% of grounding errors by encouraging the model to carefully consider the task context, potential ambiguities, and available UI elements before committing to action.\n\nWe will further clarify this part in our revised manuscript. Thanks for helping improve our work!\n\n\n> **Q3: On AGUVIS Model's Generation Approach**\n\nYes, AGUVIS generates the next step iteratively rather than first generating an overall plan and then executing steps. At each time step, given the current observation and task history, the model:\n\n1. Generates thoughts about the current state in relation to the goal\n2. Determines the appropriate next action\n3. Executes the action and receives a new observation\n4. Repeats the process until task completion\n\nThis iterative approach allows AGUVIS to adapt to changing UI states and unexpected outcomes during task execution, rather than rigidly following a predetermined plan.", "claims_and_evidence": "The claims are adequately supported with sufficient experimental results.", "methods_and_evaluation_criteria": "The method is simple and straightforward. The authors use GPT-4o to \"translate\" GUI actions into natural language, which helps agent understanding during fine-tuning. They then apply standard fine-tuning on the aggregated grounding data and augmented planning trajectories. Essentially, they do not create a new dataset or propose a novel framework, so the methodological novelty may seem limited. However, significant engineering effort is also involved, such as aggregating datasets from various sources, standardizing formats, validating through human studies, etc. Also, the authors open-source their trained models and datasets, which could benefit the academic community. These altogether could build up sufficient contribution for the paper overall.\n\nThe evaluation benchmarks are comprehensive, including grounding, offline GUI agent and online GUI agent evaluation.", "theoretical_claims": "N/A. The paper does not include much theoretical claim. (Not a weakness)", "experimental_designs_or_analyses": "In Section 4.2 and Table 6, the authors state that Inner Monologue benefits both grounding and planning. However, only the planning stage (Stage 2) incorporates Inner Monologue, while the grounding stage (Stage 1) only relies on existing datasets without such augmentation. So I wonder how Inner Monologue contributes to performance gains in grounding? I notice that the authors provide some explanation in Appendix E.1.2 that \"This is mainly because the low-level instructions of inner monologue act as atomic instruction and grounding action pairs, also enhancing the grounding ability of our GUI agents.\" However, it's kind of hard to understand this explanation. It would be helpful if the authors could elaborate further on this point.", "relation_to_broader_scientific_literature": "The open-sourced dataset and model checkpoints are helpful for future academic research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could the authors further clarify the difference between the self-plan and enforced plan settings? I reviewed the prompt templates on pages 27 and 28 but couldn’t identify any differences between them. Could the authors provide an explanation?\n\n2. Regarding the proposed Aguvis model, does it generate the next step iteratively, or does it first generate an overall plan and then generate each step iteratively? I assume it's the first one, but would like to confirm.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bRcGR8EybB", "reviewer": "Reviewer_VD9H", "summary": "This paper introduces Aguvis, a vision-based framework that operates directly on screen images, providing a standardized cross-platform interaction method enhanced by structured reasoning through inner monologue. The researchers developed a comprehensive dataset with multimodal annotations and implemented a two-stage training approach that separately handles GUI grounding and planning. Experimental results demonstrate that Aguvis achieves leading performance on both offline and real-world benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our AGUVIS paper. We appreciate your recognition of our comprehensive roadmap for developing a pure-vision GUI agent, particularly our data curation approach and training strategies. Your positive assessment of our experimental validations across multiple benchmarks is encouraging.\n\n\n> Q1. Recently, many co-current works have proposed similar unified model architectures, such as UI-TARS, OS-ATLAS, ShowUI, and CogAgent-9B. Could the authors compare their approach with these works, particularly in terms of data? The proposed method in this paper appears to be one of the best in balancing data utilization efficiency and model performance.\n\nWe're pleased to see the growing interest in GUI agent research through concurrent works like UI-TARS, OS-ATLAS, ShowUI, and CogAgent-9B[4]. In comparison with these impressive efforts, we believe that AGUVIS makes a unique and complementary data contribution to the GUI agent community in several important ways:\n\n- **Reasoning via Inner Monologue:** A defining feature of AGUVIS is the use of reasoning via inner monologue, which is not present in many concurrent works such as OS-ATLAS, ShowUI, and CogAgent-9B. This inner monologue allows AGUVIS to perform reasoning during interaction, which is crucial for the effective handling of complex tasks across multiple platforms (mobile, desktop, web). As shown in Section 4.3, AGUVIS achieves strong generalization by using a unified action space after inner monologue reasoning, enabling knowledge transfer across diverse environments. Additionally, the inner monologue surprisingly enhances GUI grounding performance, as detailed in Section 4.2 and Appendix E.1.2, as well as in our response to [Reviewer cBz9 Q1](https://openreview.net/forum?id=PlihOwfx4r&noteId=AzjUJ6TwN5). We believe that these contributions jointly underpin our balance between data utilization efficiency and model performance.\n- **Open-Source Data Collection and Pipeline:** While recent concurrent works, such as OS-ATLAS and ShowUI, focus on grounding-centric training data, and more recent UI-TARS has made significant strides in leveraging in-house human-annotated trajectories with reasoning, AGUVIS offers a unique advantage with its large-scale, open-source data collection. Our dataset collection not only includes both unified grounding and trajectory annotations but also integrates reasoning into the data pipeline. This transparency and open-source nature of our data collection make AGUVIS a valuable resource for the community to build upon and extend our work more easily.\n\nWe greatly appreciate your recognition of AGUVIS as one of the leading approaches in balancing data efficiency and model performance. We believe that our open-source data and the novel inner monologue reasoning offer complementary contributions that will drive continued progress in the development of autonomous GUI agents.", "claims_and_evidence": "The claim is clear, focusing primarily on the data construction methodology with inner monologue in GUI data. The experiments also confirm its effectiveness.", "methods_and_evaluation_criteria": "The mode architecture's innovation is relatively weak, as it appears quite similar to the baseline Qwen2-VL. However, the paper mainly proposes a new pipeline for GUI data construction. And it shows promising results on a wide range of benchmarks.", "theoretical_claims": "This is not a theoretical paper.", "experimental_designs_or_analyses": "The authors employ a comprehensive evaluation across both offline benchmarks (ScreenSpot, Multimodal-Mind2Web, AndroidControl) and online benchmarks (Mind2Web-Live, AndroidWorld, MobileMiniWob).\n\nThe ablation studies in Section 4 are well-structured, particularly those examining the impact of training stages, inner monologue, and cross-platform benefits. The error analysis in Section 4.5 provides a balanced view of the model's limitations.", "relation_to_broader_scientific_literature": "Unified Vision and Action Model: AGUVIS extends research on GUI understanding through vision models, moving beyond traditional approaches that rely on accessibility trees or HTML (like WebGPT, Mind2Web). \nCoT on GUI Domain: It also incorporates inner monologue techniques to enhance reasoning capabilities in multimodal contexts.", "essential_references_not_discussed": "The paper has a comprehensive literature review covering most relevant work.", "other_strengths_and_weaknesses": "Strengths:\n- The performance is outstanding across multiple benchmarks.\n\nWeaknesses:\n- Recently, many co-current works have proposed similar unified model architectures, such as UI-TARS, OS-ATLAS, ShowUI, and CogAgent-9B. Could the authors compare their approach with these works, particularly in terms of data? The proposed method in this paper appears to be one of the best in balancing data utilization efficiency and model performance.", "other_comments_or_suggestions": "The paper is well-written and structured logically.", "questions_for_authors": "The paper is well-prepared, and I have no further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JqFy7RsLgu", "reviewer": "Reviewer_X3pb", "summary": "The paper introduces AGUVIS, a unified vision-based framework for autonomous GUI agents designed to overcome limitations of existing approaches, which rely on textual representations, platform-specific actions, and closed-source models for reasoning. AGUVIS enables direct operation on screen images, standardizes cross-platform interactions via a plugin system, and incorporates inner monologue-structured reasoning through explicit thought processes—to handle complex tasks requiring planning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful review and the opportunity to further clarify our contributions.\n\n> **W1: Technical Innovation in Training Method**\n\nThank you for your insightful comments regarding the training methodology! We recognize that recent approaches share similar high-level components, such as grounding pre-training and decision training. However, AGUVIS's primary innovation lies in its unified framework that incorporates inner monologue reasoning. This enables the agent to operate effectively in new, previously unseen platforms without additional retraining. This integration, along with our training methods, distinctly sets AGUVIS apart from recent methods [1,3,4].\n\nThese innovations are empirically validated:\n\n- Section 4.1 shows that our two-stage training strategy outperforms joint training and ablations.\n- Section 4.2 highlights how inner monologue enables explicit planning and task decomposition, surpassing reactive action decision methods.\n- Section 4.3 demonstrates cross-platform generalization, where AGUVIS trained on web/mobile tasks transfers well to desktop GUIs in OSWorld.\n\nWe would also greatly appreciate it if you could review these contributions highlighted by other reviewers as well. We strongly believe that the framework design, analyses, open-source model, and data contributions could significantly benefit and advance the GUI agent community.\n\n> **W2: Claim of “first fully autonomous vision-based GUI agent that operates without relying on closed-source models”**\n\nThank you for highlighting this important aspect of our work. Our claim specifically emphasizes that AGUVIS, once trained, operates fully autonomously during task execution without dependence on closed-source models. Crucially, the model itself—including all training data, architectures, and procedures—is completely open-sourced, enabling full transparency and reproducibility.\n\nWhile our data pipeline leveraged GPT-4o to generate inner monologue reasoning from human-annotated actions, this step was employed solely to enrich reasoning data rather than define action policies. And we believe such annotations could equivalently be generated by open-source VLMs, and exploring this capability with open-source models is definitely an important part of our future work.\n\nMoreover, while recent research ([1, 2, 3]) has similarly aimed at competitive performance using open-source models, AGUVIS uniquely provides a unified vision-based  framework capable of seamlessly operating across multiple diverse GUI environments. We will further clarify this comparative context in our revised manuscript to better highlight AGUVIS’s contributions and clearly differentiate it from concurrent work.\n\n> **W3. Testing settings with multi-image trajectories vs. text history + single image.**\n\nWe appreciate the suggestion. AGUVIS currently uses text history + single image, balancing context richness and computational feasibility. Incorporating multiple images poses significant token cost challenges (~1200 tokens/image), especially when paired with inner monologue reasoning and 72B model size.\n\nNonetheless, we recognize the potential of multi-image context and plan to explore it in future iterations. Our open-source format supports multi-image inputs, enabling future work to build on this. As models like Qwen2.5-VL with advanced video ability, we anticipate AGUVIS will scale to multi-frame settings more efficiently.\n\n> **W4. Evaluation on other datasets, such as AITZ.**\n\nThank you for this suggestion. We evaluated AGUVIS on the AITZ benchmark. Results are summarized below and demonstrate AGUVIS’s advanced performance:\n\n\n|Model|Total Match|\n|----|----|\n|CogAgent(Zero-shot)|44.5|\n|CogAgent(CoAT-finetuned) | 53.3 |\n|AUTO-UI(Zero-shot)|34.5|\n|AUTO-UI (CoAT-finetuned)|47.7|\n|OS-Atlas-Pro-7B (CoAT-finetuned)|58.3|\n|AGUVIS-7B|63.3|\n|AGUVIS-72B|66.1| \n\nThese results will also be included in our revision to strengthen the paper’s empirical validation.\n\n> **Q1. Discrepancy between results on Mind2Web-Live and AndroidWorld.**\n\nWe appreciate your attention to the detailed results. We think the discrepancy between Mind2Web-Live (Table 4) and AndroidWorld (Table 5) may stem from the differences in how GPT-4o understands and interacts with web interfaces versus mobile interfaces. We observed that GPT-4o tends to be distracted by extraneous details in high-resolution, information-rich web interfaces, which can lead to failures in planning. We will clarify these environmental factors and their impact on agent performance in our revision.", "claims_and_evidence": "They are clear.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Proofs for theoretical claims are correct", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Related to the GUI understanding and Agent Reasoning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. It achieves good results across offline and real-world online benchmarks.\n2. It provides an effective framework to collect the GUI grounding and planning data.\n\nWeaknesses:\n1. The training method proposed in this paper does not have technical innovation. Most of the existing end-to-end GUI Agent training methods include Grounding pre-training and decision training[1,3,4], which is a general idea.\n2. I do not agree about what authors claim in.the paper “first fully autonomous vision-based GUI agent that operates without relying on closed-source models”. Firstly, it is disputable that data collection pipline in this paper also need to leverage closed-source model (GPT). This is essentially distilling the common sense capability in from closed source models. Additionally, several studies have attempted to create algorithms that can enable open-source models to achieve the same or even better performance compared to the current open-source models [1,2,3]. This statement exaggerates the contribution of the article.\n3. During the testing, is the model tested based on the setting of multi-image trajectories, or on the setting of text history + single image? Suffice it to say, is a setting based on multi-graph trajectories more appropriate.\n4. I suggest author to consider to evaluate the model on other frequently-used datasets (e.g., AITZ[4]). \n\n[1]. OS-ATLAS: A Foundation Action Model for Generalist GUI Agents\n[2]. InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection\n[3]. GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices\n[4]. MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding\n[5]. Android in the Zoo: Chain-of-Action-Thought for GUI Agents", "other_comments_or_suggestions": "N/A", "questions_for_authors": "When comparing GPT-4o+AGUVIS-7B and AGUVIS-72B, I found that the conclusion on Mind2Web-Live （Table 4) is not consistent with the conclusion on AndroidWorld (Table 5). Can you explain the reason?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bMs8EM7S0i", "reviewer": "Reviewer_zR1B", "summary": "This paper focuses on the autonomous GUI interaction task from the pure vision agent perspective. A large-scale cross-platform dataset of GUI agent trajectories is constructed. A two-stage training pipeline is proposed to separate GUI grounding from planning and reasoning. The experiments demonstrate the effectiveness of the proposed method in both offline and real-world online benchmarks.\n\n## update after rebuttal\n\nI appreciate the authors' clarifications. Most of my concerns have been addressed by the rebuttal. I would lean to accept the paper by involving the additional discussions in the revised version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work and providing constructive feedback! We greatly appreciate your recognition of our contribution in the realm of pure-vision based autonomous GUI agents, including our grounding-then-planning training pipeline and open-sourcing the large-scale, curated data. We are also delighted that you recognized the promising performance of our approach in both offline and real-world online benchmarks, which highlights its generalizability across multiple digital platforms. We have fully open-sourced our roadmap towards building pure-vision based autonomous agents, and are committed to support further research in this field.\n\nWe also noticed you have some constructive questions about our work, and we're happy to elaborate further below!\n\n> C1: The first paragraph in Section 3.3 claims that “we evaluated AGUVIS across four comprehensive benchmarks”. But only three benchmarks are mentioned here.\n\nThank you for pointing out this inconsistency! We will correct it in the revised version of the manuscript.\n\n> Q1: Which metric is used for the result comparison in Table 1?\n\nFor Table 1 (results on Screenspot), we report Click Accuracy as the evaluation metric, following the definitions provided in the original benchmark paper [1]. Specifically, Click Accuracy measures the proportion of test samples where the predicted location falls within the ground truth element's bounding box.\n\n[1] SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. Cheng et al., 2024.\n\n> Q2: Why are the evaluation metrics used on different benchmark datasets different? For example, the computational cost analysis is only conducted on Mind2Web-Live, and the element accuracy and Operation F1 are only conducted on Multimodal Mind2Web.\n\n\nThank you for your thoughtful feedback! The differences in evaluation metrics arise because each benchmark is designed to assess different aspects of the agent's performance, and we adhere to the metric definitions specific to each benchmark. This approach ensures that our results remain comparable to previous methods and provides a clear understanding of the agent's performance under distinct conditions.\n\n- **Offline vs. Online Evaluations**: Some metrics, such as element accuracy and Operation F1, are more suitable for offline evaluations (e.g., Multimodal Mind2Web), where the goal is to evaluate step-level accuracy and action predictions without environments. In contrast, online evaluations like those on Mind2Web-Live focus on final trajectory-level execution results, which offer a more holistic view of the agent's ability to complete tasks in dynamic environments.\n\n- **Benchmark-Specific Metrics**: Different benchmarks introduce their own unique metrics to capture specific aspects of performance. For instance, Mind2Web-Live introduces the USD Efficiency Score, which evaluates the efficiency of resource utilization during task execution. This metric provides insights into the agent’s performance in real-world settings, where efficiency is a key factor.\n\nWhile we acknowledge the importance of following standardized evaluation metrics, we also agree that it is valuable to align the evaluation criteria with the specific needs and goals of each benchmark. Developing unified metrics across benchmarks requires significant effort but remains an important direction for our future work. \n\n> Q3: Based on the results in Table 8, it seems that the proposed method paired with GPT-4o for planning, the result is better than that of the AGUVIS-72B. I was wondering about the results of the same configuration on other benchmarks.\n\nWe appreciate your interest in this observation! In addition to the AGUVIS-7B paired with GPT-4o for planning shown in Table 8 of OSWorld, we have also applied this configuration to other online GUI agent benchmarks, such as Mind2Web-Live, AndroidWorld, and MobileMiniWob, as detailed in Tables 4 and 5.\n\n\n> Q4: For the VLM-based trajectory augmentation process, the current inner monologue components are generated by GPT-4o. Can other latest VLMs be used and compared?\n\nOur data processing pipeline is model-agnostic and can be extended to more recent VLMs. Our data pipeline utilizes a VLM annotator to generate inner monologue reasoning from human-annotated actions. This is a natural fit for human behavior, where humans easily generate actions but find it costly to record their inner monologue. Conversely, current VLM struggle to make accurate action decisions but can more easily infer inner monologues given action decisions. This novel approach provides a promising pipeline for building agent trajectories with reasoning. We believe such annotations could equivalently be generated by open-source VLMs. Although time and cost constraints during rebuttal have prevented us from fully re-collecting data and retraining with alternative models, we believe that exploring other open-source VLMs for this process is an exciting avenue for future work!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The current experiments are not thorough enough. Specifically, the evaluation metrics on different benchmarks are not the same. For example, the computational cost analysis is only conducted on Mind2Web-Live, and the element accuracy and Operation F1 are only conducted on Multimodal Mind2Web. To make the results more convincing, it would be better to add additional evaluations and discussions.", "relation_to_broader_scientific_literature": "The proposed datasets and models could benefit future research in autonomous vision-based GUI agents.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The proposed two-stage training pipeline could incorporate structured thought processes to enhance the performance of autonomous vision-based GUI agents. The collected dataset could also be useful for future research in the community.", "other_comments_or_suggestions": "The first paragraph in Section 3.3 claims that “we evaluated AGUVIS across four comprehensive benchmarks”. But only three benchmarks are mentioned here.", "questions_for_authors": "1. Which metric is used for the result comparison in Table 1?\n\n2. Why are the evaluation metrics used on different benchmark datasets different? For example, the computational cost analysis is only conducted on Mind2Web-Live, and the element accuracy and Operation F1 are only conducted on Multimodal Mind2Web.\n\n3. Based on the results in Table 8, it seems that the proposed method paired with GPT-4o for planning, the result is better than that of the AGUVIS-72B. I was wondering about the results of the same configuration on other benchmarks.\n\n4. For the VLM-based trajectory augmentation process, the current inner monologue components are generated by GPT-4o. Can other latest VLMs be used and compared?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Aguvis, a novel, purely vision-based framework for autonomous GUI interaction across different platforms. Key contributions include a unified agent architecture operating directly on screen images, the integration of structured reasoning via "inner monologue," a large-scale open-source dataset (Aguvis Data Collection) with multimodal annotations, and a two-stage training pipeline separating grounding and planning. The authors demonstrate state-of-the-art performance on various offline and online benchmarks without reliance on closed-source models during execution.

All reviewers recognize the paper's strengths, particularly the impressive performance achieved across a comprehensive set of benchmarks (cBz9, VD9H, X3pb, zR1B). The open-sourcing of the dataset, model, and training pipeline is consistently highlighted as a valuable contribution to the community (cBz9, VD9H, zR1B). The unified vision-based approach and the incorporation of inner monologue for reasoning are also seen as positive aspects (cBz9, VD9H). The engineering effort in data collection and standardization was acknowledged (cBz9).

Initial concerns included the perceived novelty of the training pipeline compared to concurrent work (VD9H, X3pb), clarification needed on the "fully autonomous without closed-source models" claim given the use of GPT-4o in data annotation (X3pb), understanding the mechanism by which inner monologue improved grounding (cBz9), and requests for additional comparisons or evaluations (VD9H, X3pb, zR1B).

The authors provided thorough rebuttals addressing these points. They clarified that the core innovation lies in the integration of inner monologue within their unified framework enabling cross-platform generalization, explained the grounding benefits, detailed the autonomy claim (runtime independence from closed-source models), provided comparisons with concurrent work focusing on data contributions, and presented strong additional results on the AITZ benchmark as requested (X3pb). Reviewers acknowledged the rebuttals, with Reviewer X3pb raising their score and Reviewer zR1B confirming their leaning towards acceptance post-rebuttal.

In conclusion, the paper presents a nice contribution towards general-purpose, vision-based GUI agents. It demonstrates good empirical results on diverse benchmarks and provides valuable open-source resources. The authors effectively addressed reviewers' concerns in their rebuttal, strengthening the paper and leading to a clearer consensus towards acceptance among the reviewers. The work's performance, open contribution, and novel integration of reasoning make it suitable for publication.

I suggest the authors to discuss these works in the revision as they resemble some of the earlier works for incorporating planning trajectories (like inner monologue) in UI grounding:

Devil's Advocate: Anticipatory Reflection for LLM Agents, Wang et al., EMNLP Findings 2024

MUG: Interactive Multimodal Grounding on User Interfaces, Li et al., EACL 2024

A Zero-Shot Language Agent for Computer Control with Structured Reflection, Li, et al., EMNLP 2023
Decision: Accept (poster)