Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Authors: Boyu Gou
Ruohan Wang
Boyuan Zheng
Yanan Xie
Cheng Chang
Yiheng Shu
Huan Sun
Yu Su
Venue: ICLR 2025 Oral (2025)
Keywords: GUI Agents
Visual Grounding
Multimodal Large Language Models
GUI Grounding
Large Language Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multimodal large language models (MLLMs) are transforming the capabilities of graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across various platforms. However, the effectiveness of these agents hinges on the robustness of their grounding capability. Current GUI agents predominantly utilize text-based representations such as HTML or accessibility trees, which, despite their utility, often introduce noise, incompleteness, and increased computational overhead. In this paper, we advocate a human-like embodiment for GUI agents that perceive the environment entirely visually and directly perform pixel-level operations on the GUI. The key is visual grounding models that can accurately map diverse referring expressions of GUI elements to their coordinates on the GUI across different platforms. We show that a simple recipe, which includes web-based synthetic data and slight adaptation of the LLaVA architecture, is surprisingly effective for training such visual grounding models. We collect the largest dataset for GUI visual grounding so far, containing 10M GUI elements and their referring expressions over 1.3M screenshots, and use it to train UGround, a strong universal visual grounding model for GUI agents. Empirical results on six benchmarks spanning three categories (grounding, offline agent, and online agent) show that 1) UGround substantially outperforms existing visual grounding models for GUI agents, by up to 20\% absolute, and 2) agents with UGround outperform state-of-the-art agents, despite the fact that existing agents use additional text-based input while ours only uses visual perception. These results provide strong support for the feasibility and promises of GUI agents that navigate the digital world as humans do.

Published as a conference paper at ICLR 2025
Navigating the Digital World as Humans Do:
UNIVERSAL VISUAL GROUNDING FOR GUI AGENTS
Boyu Gou'
Ruohan Wang's
Boyuan Zheng'
Yanan Xie
Cheng Chang? Yiheng Shu
Huan Sun' Yu Su
'The Ohio State University Orby AI
{gou. 43, 397, su. 809 @osu. edu, yanan@orby. ai
https://osu--nlpgroos thub io/UGround/
ABSTRACT
Multimodal large language models (MLLMs) are transforming the capabilities graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across various platforms. However, the effectiveness of these agents hinges on the robustness of their grounding capability. Current GUI agents predominantly utilize text-based representations such as HTML or accessibility trees, which, despite their utility, often introduce noise, incompleteness, and increased computational overhead. In this paper, we advocate a human-like embodiment for GUI agents that perceive the environment entirely visually and directly perform pixcl-level operations on the GUI. The key is visual grounding models that can accurately map diverse referring expressions of GUI elements to their coordinates the GUI across different platforms. We show that a simple recipe, which includes web-based synthetic data and slight adaptation of the LLaVA architecture, is surprisingly effective for training such visual grounding models. We collect the largest dataset for GUI visual grounding far, containing 10M GUI elements and their referring expressions over 1.3M screenshots, and use it to train UGround, a strong universal visual grounding model for GUI agents. Empirical results on six benchmarks spanning three categories (grounding, offline agent, and online agent) show that 1) UGround substantially outperforms existing visual grounding models for GUI agents, by up 20% absolute, and 2) agents with UGround outperform state-of-the-an agents, despite the fact that existing agents use additional text-based input while ours only uses visual perception. These results provide strong support for the feasibility and promise GUI agents that navigate the digital world as humans do.
UGround
Web
Mobile
Desktop
Screen Desitop
Prior SOTA
Find the trade-in value for PS4
Turn on Wi-Fi
Install the Township apolication
Seceeertpntt--W Web
Accrend Mobile
31
Weino 2s.
AndroddWarts
0.1
50.
innrreeeContro Low
22.5
*0HRACT Contor High
Figure 1: Examples of agent tasks across platforms and performance GUI grounding ( ScreenSpot), offline agent (4: Multimodal-Mind2WWd Contol,, and OmniACT), and online agent benchmarks (V: Mind2Web-Live and AndroidWorld) when using GPT-4 as the planner.
INTRODUCTION
GUI (graphical user interface) agents, which are autonomous agents acting in the digital world via operating GUIs, have been rapidly co-evolving with large language models (LLMMs). On the

---

Published as a conference paper at ICLR 2025
one hand, the general multimedia understanding and generation capabilities of (multimodal) LLMs
empower GUI agents to generalize beyond simple simulated settings (Shi et al. 2017 Humphreys
et al. 2022) to diverse and complex real-world environments, including the web (Deng al. [023
Zhoou et al. 2024 Yao et al. 2022), desktop (Xie et al., 2024 Wu et al. 2024) and mobile operating
systems (Rawles et al.) 2023] Yan et al. 2023 Rawles al. 2024). On the other hand, GUT agents
have become an important testbed for LLMs, providing both the necessary breadth and depth for
driving continued development as well as a pathway to many commercially viable automation
applications.
Most humans perceive the digital world visually and act via keyboards, mice, or touchscreens, In principle, the embodiment of a GUI agent should already be complete if it can 1) visually perceive the GUT renderings, and 2) have effectors equivalent to a keyboard for typing and equivalent to a mouse or touchscreen for pixel-level operations like clicking and hovering [ However, current GUI agents assume more than that. For perception, most current agents rely on reading the underlying text-based representations such as HTML or accessibility (al Iy) trees (Deng et al. 2023 Gur et al. 2024. Zhou et al. 2024) P Only with the recent advances in multimodal LLMs (MLLMs) does visual perception become broadly viable, but text-based representations are still used jointly (Zheng et al., 2024 Koh et al. 2024 Zhang et al.) (024a). For effectors. most current agents act via selecting from list of options, e.g.. HTML elements (Deng al., 2023 Zheng et al., 2024) labeled bounding boxes (He et al. 2024 Zhang et al. (024a), instead of pixel-level operations directly on the GUT.
Obtaining those options in turn often requires access text-based representations and/or separate models for detecting objects and text (Wang et al. 2024a Kapoor et al. 2024).
However, there is no free lunch, and those additional requirements come with their limitations. On the one hand, text-based representations are noisy and incomplete. Full HTML documents contain a considerable amount of irrelevant information. Ally trees are more compact and mainly contain semantic information, but similar to other semantic annotations that rely on voluntary participation, they widely suffer from incomplete and incorrect annotations 3 contrast, visual renderings, by design, are pnteraation-oomplee and only contain information relevant to users. On the other hand, the additional input increases latency and inference costs. Zheng et al. (2024) found that HTML can consume up to 10 times more tokens to encode than the corresponding visual. Mcanwhile, obtaining an al ly tree can be time-consuming in itself, especially in desktop or mobile environments. The added latency and cost at every step are further compounded in the long-horizon agent tasks, compromising user experience and practicality.
In this work, we are interested in how far GUI agents with human-like embodiment, i.e., only visual observation of environments and pixel-level operations, can go. There have been a few attempts (Sha et al. 2023] Hong al.. 2024 Cheng al.) 2024), but they are rarely adopted in state-of-the-ar solutions. We find that a major bottleneck is grounding, i.c., mapping textual plans generated by an (M)LLM to the precise locations on the GUI. There are three desiderata for GUI agent grounding model: 1) High accuracy. A single grounding error can get an agent stuck and fail the whole task. 2) Strong generalization. It should work on different GUIs: desktop (Windows, Linux, macOS), mobile (Android, iOS), different websites, etc. 3) Flexibility. It should plug and play in different MLLMS instead of being tightly coupled with a certain model. Existing visual grounding methods for GUI agents (Shaw et al. 2023) Hong et al.] [024] Cheng et al., (024) fail to meet these desiderata, hindering the advances towards GUI agents with human-like embodiment.
The main contributions of this work are three-fold:
1. We make careful arguments and a strong case for GUI agents with human-like embodiment that perceive the digital world entirely visually and take pixel-level operations on GUIs, and propose a generie framework, SeeAct-V, V, for building such agents by adapting from the popular SeeAct framework (Zheng al. 2024).
Except for auditory perception, which is beyond the scope of this study. "The ally tree a compact yet informative representation intended for assistive technologies facilitate people with disahilities, e.g., visual impairment.
A 2024 survey over the top one million websites found that 95.9% of the home pages had accessibility conformance errors such as missing alternative text for images missing form input labels, with an average of 56.8 errors per page (WehAIM 2
2

---

Published as a conference paper at ICLR 2025
2. We show that simple recipe, which includes web-based synthetic data and slight adaptation of the LLaVA architecture (Liu et al. 2024c). is surprisingly effective for GUI visual grounding. Using this recipe, we construct and release the largest GUI visual grounding dataset to date, covering 10M GUI elements and their referring expressions over 1.3M GUI screenshots. We also train and release a universal visual grounding model, UGround, the dataset.
3. We conduct the most comprehensive evaluation for GUI agents to date, covering six benchmarks spanning three categories (Figure I): grounding (desktop, mobile, and web), offline agent evaluation (desktop, mobile, and web), and online agent evaluation (mobile and web). The results demonstrate: 1) UGround substantially outperforms existing visual grounding models for GUI agents across the board, by up to 20% absolute. 2) SeeAct-V agents with UGround can achieve at least comparable and often much better performance than state-of-the-art agents that use additional text-based input. These results provide strong support for the feasibility and promises of GUI agents that navigate the digital world as humans do.
METHOD
Vision-Only Observation
Planning
Human-Like Operation
TASK: Find the cheapest dk monitor
User: Dacide the next action for the task. Elemant Description: The search bar at the top of the page.
Click (556, 26) Type ("4k monitor"))
Fast chopping
Action: Type
Value: 4k monitor
Grounding
User: What pixel coordinatos the element corresponding "The bar the top page" (896,
Figure 2: SeeAct-V, which uses screenshots as the only environmental observation (task instructions are input as text), without relying on HTML or al ly trees. It includes an MLLM that generates textual plans and a visual grounding model to map textual plans into coordinates on the screenshot. Note: "Click" is always automatically inserted before "Type."
2.1 OVERVIEW
We adapt the popular SeeAct framework (Zheng et al. 2024) to one in which agents only take visual observation of the environment and directly conduct pixel-level operations, denoted as SeeAct-V V (Figure 2}. The original SeeAct has two stages: planning and grounding, both handled by an MLLM. At each step. the MLLM first generates textual plan, then selects grounding candidates from short list. The grounding candidates are either filtered HTML elements or labels of Set-of-Mark (SoM; Yang et al. (2023) annotations the screenshot, both of which require HTMLL or ly trees as additional input. In contrast, SeeAct-V only uses screenshots for environmental observation. For grounding, SeeAct-V uses a separate model specialized for visual grounding that directly produces the coordinates on the current screen where the agent should act. We provide our philosophy behind the modular design of SeeAct-V in Appendix B
A strong visual grounding model therefore becomes the key for making SeeAct-V a compelling framework. Ideally, it should generalize across platforms (e.g., web, desktop. and mobile) and handle diverse ways of referring to GUI elements. Considering the rapid evolution of MLLMs, this grounding model should be easily pluggable into different MILMS to help ground their plans into different GUI environments. Finally, GUI screenshots can vary drastically in resolution and orientation, therefore the grounding model should handle a wide range of input resolutions. The main technical contribution of this work is a surprisingly simple recipe (incl. data and modeling) for training such universal visual grounding models. We introduce our simple data synthesis strategy in $22 followed by modeling considerations in 22. With this simple recipe, we construct the largest training data for GUI grounding to date and trai UGround, a strong universal visual grounding model for GUI agents.
3

---

Published as a conference paper at ICLR 2025
2.2 DATA CONSTRUCTION
We synthesize a large, high-quality and diverse set of (screenshot, referring expression, coordinates) triplets as training data for visual grounding, where we use the center point coordinates of an element as the expected output. Our data synthesis is fully based on webpages. Wehpages are ideal for grounding data synthesis because of their dual representation- we can easily get the full HTML, the visual rendering, and fine-grained correspondences between the two (e.g.. HTML elements to precise bounding boxes). HTML elements also contain rich metadata such as CSS or accessibility attributes, opening numerous opportunities for synthesizing diverse referring expressions (REs). Finally, since GUI designs share many similarities across platforms, hypothesize that visual grounding models trained only on web data will generalize to other platforms like desktop and mobile Uls.
Common RE Types for GUTs. People use diverse ways to refer to GUI elements (Figure 3). Previous visual ground-
1. Red icon labeled "UNIQLO"
2. Button at the top left corer
ing works (Hong al., 2024 Cheng et al., 2024) have
3. Navigate back to the homepage
not sufficiently considered this dimension of diversity. We categorize common REs for GUI elements into three types: 1) Visual REs, i.e., salient visual features like text or im- age content, element types (e.g., buttons or input fields), shapes, colors, etc. 2) Positional REs, including both absolute (e.g.. "at the top left of the page") and relative po- sitions (e.g.. "to the right of element X"") to other elements, Besides straightforward positional information, contex- tual references (e.g.. "for Item A," "under the section X"") are more challenging for grounding because they require understanding both positional relationships and semantic relationships between elements (e.g., a like button is asso- ciated with product). 3) Functional REs, i.e., referring to elements by their main functions (c.g., "Navigate to Home," " "Go to My Carr"). Composite types that combine two or more of these types are also common, especially when stronger disambiguation is needed, e.g.. "click the heart button under the Pokémon shirt to add to favorite."
BLS
Tops
T-Shirts
UT: Graphic Tome
Sweetshhits Hood
In-stock: online only
Calsgory
Options
KIDS, 5- 5-O11150-- 12V1100 Prlvemon:: [Shott Steatt- Graphic
HDDS, 2-1111 1211100 LEGde 1-Shit)
$14.90
$14.90
* sm
* s
1. Hollow heart button
2. Button below the Pokémon shirt
3. Favor the Pokémon shirt
Examples of visual, positional, and functional REs.
Figure 3:
Hybrid RE Synthesis from Web. We propose a novel hybrid synthesis pipeline, orchestrating both carefully curated rules as well as LLMs to generate diverse REs for HTMI elements: 1) Primary Descriptors: We extract abundant visual and functional information that are embedded in the attributes of HTML elements. For example, HTML attributes like inner-text and alt provide visual clues (including text content), while accessibility attributes like aria-label reveal more functional aspects of HTML element. However, HTML attributes are often incomplete. To harvest visual and functional signals beyond HTML attributes, we use an open MLLM, ALLVVA-NeT--133 element along with its available attributes to the MLLM and prompt it to generate diverse REs. This process often yields composite REs that combine some HTML attributes with visual features (e.g.. "hollow heart") or new knowledge from the MLLM (c.g.. a blue bird icon represents Twitter). Similar to Lai et al. (2023), we also employ an LLM (Llama- ---na-3-BB-- SB-Instruct; Metal (024) to make these generated REs more concise. We randomly select an HTML attribute (that may contain functional or visual information) or the synthesized description by LLMs as the primary descriptor of an element.
(Liu al. 2024b). We input the visual rendering of an HTML
2) Positional Expressions: We curate rules to generate positional REs according to the absolute position of an element in the screenshot as well as its spatial relationship to neighboring elements (e.g., "at the top of the page," "benween element A and B"). We also create multiple rules generate contextual references. For example, we identify elements of certain types in the screenshot (c.g., radio buttons, checkboxes. input fields), and generate REs for them based on their spatial and structural relationship (e.g., hierarchical structure of the DOM tree) to others (e.g., "the input field labeled Birthday")..
We collect screenshots (mix of portrait and landscape views in various resolutions) and metadata of web elements (salient HTML attributes, bounding box coordinates) from Common Crawl g and then
hnups//commmnncrallorg
4

---

Published as a conference paper at ICLR 2025
Table Overview of training datasets used for UGround.
Dataset
Annotation
# of Elements
# of Screenshots
Platform
Web-Hybrid (Ours)
Rule LLM
9M
773K
Web
Web-Direct (Ours)
GPT
408K
408K
Web
GUIAct (Chen al. 2024
GPT Human
140K
13K
Web
AndroidComtnn conrol (La ald
2024b
Human
47K
47K
Andmid
Widget Caption (Li et 2020b
Human
41K
15K
Android
UlBert (Bai 2021
Human
16K
5K
Android
AIT7 Zhang e1 al 2024c
GPT Human
8K
8K
Android
Total
10M
1.3M
Web Android
apply our data synthesis pipeline to get our main training dataset (Web-Hybrid), We leave more details to Appendix E.1
Supplementary Data. There have been multiple prior efforts constructing grounding data for Android, so we incorporate the existing datasets as well. We also use GPT-4o to directly synthesize small set of REs for web elements, with focus type) and functional REs (Web-Direct). These additions help provide more diverse RFs and cover elements in Android, especially those not commonly found on the web (e.g., toggle buttons).
open-ended REs (no constraints the
In total, we compile a dataset totaling 10M UI elements, with the majority (90%) from our hybrid synthesis pipeline (Table I). Elements on the same screenshot are batched to accelerate training.
2.3 MODEL DESIGN
We adopt a widely used open-source model architecture, 7B LLaVA-NeXT (Liu et al. 2024b). as our backbone model for visual grounding. We make a few adaptations to tailor for GUI grounding.
Input-Output Formulation. We always instruct the model to answer "In the screenshot, what are the pixel element coordinates corresponding to {Description] ?" Following recent work in visual grounding (Cheng et al.) 2024). we represent the answer in natural language so we can directly use autoregressive decoding. Specifically, we opt for coordinates in the numerical form (e.g.. "(1344, 1344)") to precisely point to an element without any normalization.
Image Resolution. GUT screenshots are much larger than typical natural images, often requiring resolution above 1,000px for legibility. LL (L.iu et al., (024cc a) was initially built for 336px images, and was later scaled up to at most 772px via the AnyRes technique (Cheng et al., 2023 Gao et al. 2024 Liu et al.) 2024bb Guo et al. 2024: Dong et al.) 2024). It resizes and splits large image into small slices, encodes each slice independently with the vision encoder, and adds a special token at the end of each row to help the language model keep track of the image shape. AnyRes allows easy scaling up of input resolution. However, it is always a trade-off between the diversity of supported resolutions and the speed of training and inference. To strike balance and avoid meaningless excessive resolutions, we enlarge the allowed input sizes to 36 ViT (Dosovitskiy et al., 202I) slices, and use CLIP@ 224px (Radford et al., 202I) as the image encoder for more flexible splitting, pushing the maximum supported resolution to 1,344 x 1,344 (landscape) and 896 x 2,016 (portrait). Additionally, we use Vicuna- 1.5-aa--.-------- 16k (Zheng al. 2023) with 16K context length to handle long visual contexts. Finally, there is a low-resolution image fusion module commonly used in AnyRes. However, we find it ineffective for GUI grounding, as 224px is too small provide informative global context, we leave it out from our model. More details are in Appendix
EXPERIMENTS
Most existing studies on GUI agents typically evaluate on one or two benchmarks. In contrast, we conduct a much more comprehensive evaluation on GUI agents to show the universality of our method. Our evaluation employs six benchmarks that span all three major platforms (i.e.. web, desktop, and mobile) and cover three settings: visual grounding ($3.1). offfine agent evaluation on cached environment states ($3.2). and online agent evaluation in live environments ($3.3). The visual grounding setting focuses on the grounding performance of UGround, while the agent settings test the end-to-end effectiveness of the SeeAct-V framework with UGround integrated. On the agent
5

---

Published as a conference paper at ICLR 2025
Table 2: Grounding acouracy on ScreenSpot (Standard Setting). Results for GPT-4, CggAgent, and SeeClick are from Cheng et al. (2024)
Mobile
Desktop
Grounding Model
Text
lcon/Widget
Text
Icon/ Coon/Widget
Text
Icon/Widget
Average
GPT-4
22. 6
24.5
20.2
11.8
9.2
8.8
16.2
GPT-40
23.6
12.2
18.3
CogAgent (Hong et al.
2024
67.0
24.0
74.2
20.0
70.4
28.6
47.4
SeeClick Cheng
78.0
52.0
72.2
30.0
55.7
32.5
53.4
UGround (Ours)
82.8
60.3
82.5
63.6
80.4
70.4
73.3
Table 3: Grounding accuracy on ScreenSpot (Agent Setting) with planner-generated REs.
Mobile
Desktop
Web
Planner
Grounding
Avg.
Text
Kcon/Widget
Text
Icon/ Moon/iidget
Text
Icon/ Ceon/Widget
SceClick UGround
76.6
55.5
68.0
28.6
40.9
23.3
48.8
GPT-4
90.1
70.3
87.1
55.7
85.7
64.6
75.6
SeeClick
81.0
59.8
69.6
33.6
43.9
26.2
52.3
GPT-4o
UGround
93.4
76.9
92.8
67.9
88.7
68.9
81.4
benchmarks, we compare the vision-only SeeAct-V framework with prior SOTA methods that usually require additional text-based representations (HTML or ally tree) input. Within SeeAct-V, also compare UGround with existing visual grounding models whenever possible.
3.1 GUI VISUAL GROUNDING
We first evaluate UGround on the ScreenSpot benchmark (Cheng et al. 2024), which is specifically designed for visual grounding on GUIs. The benchmark consists of 1, 1,272 single-step instructions and the corresponding bounding boxes of the target elements across mobile (e.g., iOS and Android), desktop (e.g., macOS and Windows), and web environments. These elements vary between text-based elements, icons (e.g., the trash can icon) and widgets (e.g., to-do lists), representing diverse GUI element types.
We evaluate under two settings: 1) Standard Setting. In the standard setting of SereenSpot, the instructions are written by human annotators with a primary focus on functional description of the target elements, e.g., simply "close" to refer to the 'X' button that closes a window or "set alarm for 7:40" when the input image shows the iPhone clock app with list of inactive alarms. 2) Agent Setting. For GUI agents, grounding model needs to work with planning model (e.g.. MLLM) and ground the REs it generates, which includes not only functional REs but also visual and positional comprehensive evaluation
REs (see 2.2).. To provide we input each ScreenSpot example to an MILM,, which acts as a planning model, and asks it to generate diverse REs for the target element. This setting therefore more representative of the grounding challenges in GUI agents. We mainly compare UGround with SeeClick (Cheng et al.) 2024), the state-of-the-art visual grounding model on ScreenSpot, and another visual grounding model CogAgent (Hong et al. 22024). To show the challenge of visual grounding for general-purposs. models, we also compare with GPT-4 and GPT-4o.
visual grounding for GUI agents,
Results. As shown in Table 2 and Table 3, UGround outperforms all existing models across all the settings and platforms by substantial margin, about an absolute improvement of 20% % on average under the standard setting and 29% under the agent setting. Interestingly, UGround performs remarkably well on desktop UIs, despite the fact that it is never trained on desktop screenshots (Table 1). Compared with existing models, UGround performs especially well on icons and widgets, which are generally more challenging for grounding because that requires deeper understanding of the contextual (e.g., positional) and semantic (e.g., functional) information. Overall, the strong results on ScreenSpot clearly demonstrates UGround's universal grounding capability across platforms and planners well the remarkable effectiveness of simple data synthesis and modeling recipe.

---

Published as a conference paper at ICLR 2025
Table Element accuracy on Multimodal-Midd2Wed. Results by Choice and SoM are from Zheng et al.
2024). The SoM results are on subsets of 30 tasks for each split.
Input
Planner
Grounding
Cross-Task
Bebss-Website
Cross-Domain
Avg.
Choice
46.4
38.0
42.4
42.3
Image
Text
GPT-4
SoM
29.6
20.1
27.0
25.6
SocClick
29.7
28.5
30.7
29.6
GPT-
Image (SceAct-V)
UGround
45.1
44.7
44.6
44.8
SeeClick
32.1
33.1
33.5
32.9
GPT-4o
UGround
47.7
46.0
46.6
46.8
3.2 OFFLINE AGENT EVALUATION
We discuss the experimental setup for three offline agent evaluation benchmarks followed by result discussion. Concrete examples from each benchmark are given in in/pppddix D
Web: Multimodal-Mind2Wel. We use Multimodal-Mind2We extension of Mind2Web (Deng et al.] 2023), for our evaluation on realistic web tasks. The test split consists of 1,013 tasks spanning over 100 different websites. Each task contains high-level task instruction and sequence of actions, with screenshot of the webpage before each action, as the golden trajectory. All the wehpages along the golden trajectory are cached to support offline evaluation. The tasks crowdsourced with a focus on ensuring real-world meaningfulness (i.e., what real users would need on those websites).
(heng et al. 2024), the multimodal
Zheng et al. (0024) have clearly demonstrated the necessity of visual perception for web agents, So we mainly compare with zero-shot methods that use MILMs as planners and omit text-only LLMs. Zheng et al. (2024) have also identified grounding the main challenge and proposed several grounding strategies, including 1) Choice, where the planner is asked to choose from a short list of filtered HTML elements, and 2) SoM, where the input screenshot is superposed with Set-of- Mark Yang et al.] 2023) labels and the planner is asked to select from the labels. Both strategies require additional text-based representations (i.c., HTML) to obtain the candidates and/or locate the elements in the screenshot to label. We report element accuracy, i.c., accuracy of selecting the correct element, and omit operation scores because they are orthogonal to grounding comparisons.
Mobile: AndroidControl. We use AndroidControl eLi al. 2024b). large-scale Android dataset comprising 15K unique tasks over 833 Apps. Screenshots, action sequences, and al ly trees are cached from human demonstrations as golden trajectories for training and evaluation purposes. Fach action is also labeled by a corresponding low-level instruction (e-g.. "set the hours 6*). Following Li et al. (2024b). we use 500 random steps from the test set. We compare with the SOTA zero-shot method, the text-only version of M3A (Rawles et al.. 2024), which instructs GPT-4 to generate textual actions as well select elements from the a ally tree (Choice). We adopt the two task settings in Li et al. (024bb): high-level tasks, where only the high-level intent is provided, and low-level tasks, where both the high-level intent and the corresponding low-level instruction for each step available. We use the standard metric, step-wise accuracy, where a step is considered successful only if all the predicted actions, elements, and arguments (if applicable) are correct.
Desktop: OmniACT. We use OmniACT (Kapoor et al. 2024) to evaluate the accuracy of UGround on desktop tasks. The dataset consists of 9,002 tasks covering 38 desktop applications and 27 websites across different desktop platforms (macOS, Windows, and Linux). Each task requires the generation of PyAutoGUI script, which sequence of actions complete the task on a single screenshot. The SOTA method, DetACT (Kapoor et al. 2024). extracts UI elements and their coordinates through a combination of OCR (optical character recognition), icon matching, and color detection modules. These elements are filtered by task relevance and then passed to LLMS or MILMs to generate the PyAutoGUI script with the appropriate coordinates for interaction.
For SeeAct-V, we replace the input of the DetACT pipeline with only screenshots and instruct MLLMs to generate element descriptions rather than directly generate coordinates. We then employ UGround to obtain the coordinates of the elements, which are subsequently integrated into the PyAutoGUI scripts. To ensure a fair comparison, we strictly follow the approach in Kapoor et al. (2024), including the same prompt and retrieval strategy that selects five in-contest examples from the training set based on task similarity. We report the action score, which measures the accuracy of the action sequences while penalizing errors in generated arguments.
7

---

Published as a conference paper at ICLR 2025
Table 5: Step accuracy on AndroidControl over 500 random actions from the test split. Baseline results are from et (2024b).
Table 6: Action scores (AS) on OmniACT
Baseline results are from Kapoor al. (024)
Inputs
Planner
Grounding
AS
Step Accuracy
Input
Planner
Grounding
High
Low
Image Text
Text
GPT-4
DetACT
DetACT
17.0
11.6
Text
GPT-4
Choice
42.1
55.0
GPT-4
SeeClick
39.4
47.2
Image (SecAct-V)
GPT-4
UGround
SeeClick
28.9
Image (SeeAct-V V)
UGround
46.2
58.0
SeeClick
29.6
GPT-4o
UGround
SeeClick
41.8
52.8
GPT-4o
UGround
Table 7: Completion rate (CR) and task success rate (SR) on Mind2 Liv-Lie. Baseline results are from Pan al. 2024).
Table Task success rate (SR) on Android- World. Baseline results from Rawles et al. (2024).
Inputs
Planner
Grounding
SR
Input
Planner
Grounding
SR
GPT-4
44.3
21.1
Text Image
Choice SoM
30.6
Text
Choice
GPT-4
GPT-40
47.6
22.1
Text
25.4
Image
GPT-4
60.7
23.1
Image (SecAct-V)
GPT-4
31.0
UGround
UGround
(SeeAct-V)
GPT-4o
50.8
19.2
GPT-4o
32.8
and Table 6 SeeAct-V with UGround outperforms all the baselines across the board, despite only using raw screenshots as input while baselines use additional input. UGround also consistently outperforms a strong GUI grounding model, SeeClick. These results provide solid support for human- like vision-only embodiment for GUI agents, position this work aims to make a case for. The results also further validate UGround's efficacy as a universal grounding model for GUI agents.
Results. shown Table 4 Table 5
3.3 ONLINE AGENT EVALUATION
We further evaluate our approach in end-to-end manner on two online agent benchmarks that closely resemble the offline web and Android benchmarks in $3.2 but involve interactions with live websites and mobile applications. Due to the high cost of online evaluation, we only use UGround for grounding.
Web: Mind2Web-Live. We use the test set from Mind2Web-Live (Pan et al. 2024). The benchmark built on Mind2Web (Deng et al. 2023) by adding functional evaluation to the tasks that makes automated evaluation possible live websites. Specifically, it defines and annotates key nodes for each task, which are critical steps that must be completed for a task to be considered successful, regardless of which trajectory an agent takes. The baseline agent from Pan al. (2024) is text-only, perceives and interacts with webpages by hundreds of HTMM. elements at a time. For SeeAct-V V, we change the observation to be screenshots only, and make necessary changes the original action space to fully eliminate the dependency on HTML during planning, grounding, and execution (details in Appendix G.5). We use standard metrics: micro completion rate, which measures the proportion of completed key nodes across all the tasks, and task success rate, which measures the proportion of fully completed tasks.
Mobile: Wnrldd We use AndroidWorld (Rawles et al. 2024), an online mobile agent benchmark running Android emulators. It includes 116 tasks across 20 Apps, with evaluation based the final states of the device. We compare with the SOTA agent M3A and its text-only variant from Rawles et al. (2024). They receives both raw and SoM images, together with textual UI elements, only the textual UI elements as the observation respectively. Both variants employ ReAct-style reasoning process (ao et al. 2023) to select the next target element from a list of UI elements. Additionally, they integrate reff-reflection (Shinn et al.) 2024) for the agent to summarize its current action and improve decision makinn in subsequent steps. We report task success rate, which measures the percentage of fully completed tasks.
8

---

Published as a conference paper at ICLR 2025
Performance
70
Mobile
UlGoound
Web
Desitop
Muttioddi mdzWio
6.7
60
Average
27.7
SneClick (Arg)
50
AndroaaConbe Low
50
100
200
400
773
SpreenSpot Desktop
46.5
Web Synthetic Training Data (K) (# Screenshots)
d
20
Porcomage (5)
Figure 5: Scaling curve of UGround ScreenSpot w.r.t. Web-Hybrid data size.
Figure 4: Error distribution from manual analysis.
Results. SeeAct-V with UGround gets comparable or higher performance online agent evaluation, shown in Table and Table 8 Particularly, it achieves much higher success rate compared with the SoM variant of M3A, even though Android environments have less dense UI layouts and generally
suitable for SoM (i.e., less obstruction by the SoM labels). These results again provide solid support for the feasibility and promises of human-like vision-only embodiment for GUI agents and the effectiveness of UGround.
3.4 ERROR ANALYSIS
We conduct a manual error analysis of the best performing method, SeeAct-V with UGround, to understand the bottleneck for further improvement. We randomly sample 60 failure cases from each split of ScreenSpot (agent setting with GPT-4o), AndroidControl, and Mullimodal-Min22V Except for data annotation errors, errors from the models can be categorized into planning errors, i.e. generating plans with incorrect element descriptions, and grounding errors, i.e., predicting incorrect coordinates for correct element description from the planner.
As shown in Figure 4, planning errors are the dominant cause of failures across all benchmarks, further confirming the strong grounding capability of UGround. The most frequent error is that the planner generates (otherwise correct) description of an incorrect element the screen, indicating lack of correct understanding of either the task and/or the elements. Other common planning errors include hallucinating non-existent elements producing overly generic descriptions that are too vague to uniquely locate the target element, even for human evaluators.
On the other hand, ScreenSpot-Mobile and ScreenSpot- Dessttp. considerable portion of the failures do stem from grounding error.. Both desktop and mobile UIs feature a pervasive use of icons with idiosyneratic meaning. For example, stylized dollar sign represents the Zelle App. with two cartoon people represents one's contact list in Microsoft Outlook. We find that pretrained MLLMs and our web-centric grounding training are effective in capturing the semantics of popular icons (e.g., icons representing Google) or commonsense meaning (e.g., clock icons usually represent time-related functions like alarms). However, it is challenging capture the idiosyneratic semantics of icons in the long tail, which arguably requires either additional documentation or more targeted exploration learn. This is a major cause of the grounding errors. Interestingly, when tested on more realistic agent tasks, e.g., in AndroidControl, AndroidWorld, and OmniACT, UGround still proves to be relatively robust. This is because most of the agent tasks concern things in the head of the distribution; things in the long tail are naturally rare (though still important). This explains the strong performance of UGround on mobile and desktop agent benchmarks. Nonetheless, how to capture idiosyncratic semantics in the long tail is still an open challenge for grounding.
icon
3.5 TRAINING DATA ANALYSIS: SCALING AND ABLATIONS
We conduct scaling analysis and ablation studies on our training data to better understand the contribution of different data for UGround's strong performance, and the agent setting of SereenSpot for the evaluation (with GPT-4o as the planner). Further ablations around data, model design, and RE types provided in Appendix C

---

Published as a conference paper at ICLR 2025
Table 9: Training data ablations for UGround on ScreenSpot (Agent Setting).
Mobile
Desktop
Web
Training Data
Text
Icon/Widget
Text
Icon/Widget
Text
Icon/Widget
Average
Web-Hybrid
89.0
73.4
88.1
61.4
84.8
64.6
76.9
Others
92.3
71.2
84.5
46.4
87.0
59.2
73.4
All
93.4
76.9
92.8
67.9
88.7
68.9
81.4
Scaling Curve on Web-Hybrid. We investigate the scaling of our primary synthetic dataset, Web- Hybrid, which consists of 9M data instances over 773K web screenshots in total. The scaling results in Figure 5] show that the average performance consistently improves the data scales up, though the return starts diminishing after 100K screenshots. Notably, with just 50K screenshots (about 600K elements) as training data, UGround surpasses SceClick by more than 10%, which is trained on about 3M web and Android elements from about 400K screenshots. The results clearly show the high data quality and the effectiveness for grounding training of our data synthesis pipeline. Upon manual inspection, we observe that additional data after 100K screenshots primarily enhances understanding of less frequent elements such radio buttons, checkboxes, or very small text elements. As data increases, the model can point to the center of element bounding boxes more accurately and better handle tiny hyperlinks.
Training Data Ablations. To further investigate the impact of training data sources, we compare the performance of UGround trained on only Web-Hybrid, only the supplementary data, or both (see TTalle Tj. Results in Table 9 further validate the necessity of Web-Hybrid. Training other data without Web-Hybrid often underperforms training on Web-Hybrid alone. This most evident on icons and widgets, which require understanding more diverse aspects, such visual features and functions, than text-based elements. Finally, these two data sources are complementary and their combination yield the best performance across the board.
CONCLUSIONS AND LIMITATIONS
We introduce UGround, universal GUT visual grounding model developed with large-scale web- based synthetic data. UGround shows strong cross-platform generalization and substantially out- performs the prior models. We propose vision-only framework SeeAct-V that allows pixel-level interactions based solely on visual input. Comprehensive evaluation on both offline and online agent benchmarks demonstrates that SeeAct-V agents with UGround can achieve comparable and often better performance than prior SOTA agents that rely on additional textual inputs like HTML or ally trees for observation or grounding.
Nevertheless, there are still some limitations that could be addressed in future work to advance visual grounding in GUT applications and visually grounded GUT agents. First, UGround trained on very large-scale synthetic data. Considering the similarity and repetition of elements between web pages, there is room to improve on data efficiency during training, for example by better data grouping and deduplication. On the other hand, despite the cross-platform generalization shown in our experiment results, the issue of long-tail elements remains under-addressed in this work. Mobile UIs and desktop UIs often feature specific icons with idiosyncratic semantics, and it can be impractical to account for every long-tail element in a training set. Additionally, no desktop UI data is incorporated in the training of this work, which limits the performance on desktop Uls. Given the scarcity of training datasets for desktop UIs. we anticipate the development of more comprehensive datasets in this domain. Lastly, UGround depends on an external planner; it is not meant function independently as GUI agent. Nonetheless, we hope that our datasets, model, and framework can contribute to future studies of vision-only agents, as well as contribute advancing the grounding capabilities of end-to-end models, as strong grounding data has been shown to improve end-to-end models (Cheng
et al. 2024 Hong al. 2024 Chen et al. 2024).
ETHICS STATEMENT
This work employs web-based data synthesis to develop visual grounding models for GUIs. The synthesis pipeline and data collection presented in this paper are intended solely for research purposes
10

---

Published as a conference paper at ICLR 2025
related GUI grounding and GUI agents, in line with prior works in the field (Hong et al. 2024 Cheng et al. 2024).
The webpages utilized in our work are sourced from the Common Crawl datase which is publicly available Internet archive for research and non non-commercial use. We use only a small subset of it and strictly adhere to Common Crawl's is terms of use throughout our study.
Our use and dissemination of the data are exclusively for academic research and fully comply with Section 107 of the U.S. Copyright Law regarding Fair Use. Prior to release, the data undergoes rigorous content moderation. We acknowledge full responsibility for any legal issues arising from our data collection and accept all associated risks. Furthermore, the distribution of the data is managed in strict accordance with applicable regulations and guidelines to ensure compliance with Al ethics standards and non-commercial usage.
ACKNOWLEDGMENTS
We are grateful for the collaboration with the Orby Al team (particularly Sanjari Srivastava, Peng Qi, Gang Li, and Will Lu) for their contribution on data collection and analysis, well for providing computing resources. We would also like to extend our appreciation to colleagues from the OSU NLP group and Kanzhi Cheng, Yulu Guo, Lizi Yang for their insightful comments. Special thanks to Yichen Pan, Christopher Rawles, Dehan Kong, Alice Li, and Raghav Kapoor for their assistance with evaluation. This work supported in part by Orby AI, ARL W91 1911NN220014 and NSF CAREER #1942980. The views and conclusions contained herein are those of the authors and should not be interpreted representing the official policies, either expressed implied, of the U.S. government. The U.S. government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notice herein.
REFERENCES
AI@ Meta. Llama 3 model card, 2024. URL httts:///gitabb..cm/mmaaaaalll wtwpchhh...oomtttaaeeeeeaaaaaaae
main/MODEL__AAR CARD._ md
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi. Jindong Chen, and Blaise Agüera y Arcas, UlBert: Learning generic multimodal representations for ui understanding. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1705-1712, 2021.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL:: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint ar0iv:2308. /2966, 2023.
Pratyay Banerjee, Shweti Mahajan, Kushal Arora, Chitta Baral, and Oriana Riva. Lexi: Self- supervised learning of the ui language. In Findings of the Association for Computational Linguis- tics: EMNLP 2022, 2022.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, et al. Spider2-V V: How far are multimodal agents from automating data science and engineering workflows? In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmurks Track, 2024.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: large language model as a unified interface for vision- lision-language multi-task learning. arXiv preprint arXiv:2310.09478. 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's 's referential dialogue magic. arXiv preprint arXiv:2306.1 2519. 2023b.
hhtpss//commoncrawl.orgg https//commooncralloyyyeemmmmee

---

Published as a conference paper at ICLR 2025
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. GUICourse: From general vision language models to versatile gui agents. arXiv preprint ar arXiv:2406. 11317, 2024.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2024.
Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang. Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13877-13888, 2023.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings the 30th annual ACM symposium on user interface software and technology, pp. 845-854, 2017.
the
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems, volume 36, pp. 28091 14, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, xinyue zhang, Wei Li, Li Jingwen, Wenhai Wang. Kai Chen, Conghui He, Xingcheng ZHANG, Jifeng Dai, Yu Qiao, A pioneering large vision-language model handling resolutions from 336 pixels to 4K HD. In Advances in Neural Information Processing Systems, volume 37. pp. 42566 42592, 2024.
Dahua Lin, and Jiaqi Wang.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. SPHINX-X: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.055: 059345 2024.
Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaVA-UHD: An LMM perceiving any aspect ratio and high-resolution images. In Computer Vision ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXIII, volume 15141, pp. 390-406, 2024.
Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. CogAgent: A visual language model for GUI agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14281-14290, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, pp. 9466-9482. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. OmniACT: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.17553 2024.
Andrej Karpathy, Armand Joulin, and Li F Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in Neural Information Processing Systems, volume 27, 2014.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models solve computer tasks. In Advances in Neural Information Processing Systems, volume 36, pp. 0111333649999 -96777, 2023.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena Evaluating multimodal agents realistic visual web tasks. In Proceedings the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 881-905, 2024.
Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Alcksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, et al. From scarcity to efficiency: Improving clip training via visual-enriched captions. arXiv preprint arXiv:23/0.07699 2023.
Bo Li, Hao Zhang. Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. LLaVA-NeXT: What else influences visual instruction tuning beyond data?, May 2024a. http:::aa///llaaaaaaa vl. ww..gll.g..... gitb 1otpll/g///oloo////42// 05- 25- -ee-lllaa-eeee- nextaanneetaaae -ww---aaa-eeet--laatiiiooo ablations/
Gang Li and Yang Li. Spotlight: Mobile ui understanding using vision-language models with focus. In The Eleventh International Conference on Learning Representations, 2022.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.03677 2024b.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions mobile ui action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198-8210, 2020a.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp- 5495-5510, 2020b.
Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. Ferret-UI 2: Mastering universal user interface understanding across platforms. arXiv preprini arXiv:2410. 18967, 2024c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, January 2024b. URL https
//1lava-vl.github. it/ii/////// blog// 2024-01 30- 1lxta-next/
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Chuofan Ma. Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual
tokenization for grounding multimodal large language models. arXiv preprint 240. 2024.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu. Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IÉEE conference on computer vision and pattern recognition, pp. 11-20, 2016.
13

---

Published as a conference paper at ICLR 2025
Runliang Niu, Jindong Li, Shiqi Wang. Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. SereenAgent: A vision language model-driven computer control agent. In Proceedings of the Thirty-Third International Joint Conference Artificial Intelligence, IJCAI-24, 2024.
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. WebCanvas: Benchmarking web agents in online environments. arXiv preprint SKKv:2406.12377 2024.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Yijun Qian, Yujie Lu, Alexander G Hauptmann, and Oriana Riva. Visual grounding for user interfaces. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pp.
97-107, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PMLR, 2021.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. In Advances in Neural Information Processing Systems, volume 36, pp. 59708-59728, 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. AndroidWorld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405. 14573, 2024.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. From pixels to UI actions: Learning follow instructions via graphical user interfaces. In Advances in Neural Information Processing Systems, volume 36, pp. 34354-34370, 2023.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning. pp. 3135-3144, PMLR, 2017.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024a.
Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all all-seeing project: Towards panoptic visual recognition and understanding of the open world. In The Twelfth International Conference Learning Representations, 2024b.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. VisionLI.M: Large language model is also open-ended decoder for vision-centric tasks. In Advances in Neural Information Processing Systems, volume 36, pp. 61501-61513, 2023.
WebAIM. The WebAIM Million. httpss//webaan.oor///oogtttst//p 2024-08-04.
2024. Accessed:
Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao. Tao Yu, and Lingpeng Kong. OS-Copilot: Towards generalist computer agents with self-imprrvemmnn, In ICLR 2024 Workshop on Large Language Model (I.LM) Agents, 2024.
14

---

Published as a conference paper at ICLR 2025
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujum Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks real computer environments. In Advances in Neural Information Processing Systems, volume 37, pp. 12040-52094 2024.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. GPT-4V in wonderland: Large multimodal models for shot smartphone gui navigation. arXiv preprint arXiv:231 1.07562, 2023.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-Mark prompting unleashes extraordinary visual grounding in GPT-4v. arXiv preprint arXiv:2310.11441, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 0020004442075777 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting language models. International Conference on Learning Representations (ICLR), 2023.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang. and Zhe Gan. Ferret-Ul: Grounded mobile ui understanding with multimodal llms. ArXiv, abs/2404.057199 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In Computer The Netherlands, October 11-14, 2016, Proceedings, Part 1I 14, pp. 69- 85. Springer, 2016.
2016: 14th European Conference, Amsterdam,
Zhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309. 114366 2023.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. UFO: A Ul-focused agent for windows os interaction. arXiv preprint arXiv:2402.07993 07939999799, 2024a.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang. Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. MM1.5: Methods, analysis & insights from multimodal Ilm fine-tuning. arXiv preprint arXiv:2409. 20566, 2024b.
Jiwen Zhang, Jihao WU, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action -ww.oo---tioooooouhhh for GUI agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024c.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, if grounded. Forty-first International Conference on Machine Learning. 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pp. 46595-46623, 2023.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. WebArena: realistic web environment for building autonomous agents. In The Twelfth International Conference Learning Representations, 2024.
15

---

Published as a conference paper at ICLR 2025
Table of Contents in Appendix
Related Work
17
Philosophy Behind SeeAct-V and UGround
18
Further Ablation Studies
18
Controlled Comparison to Baseline Models
18
Model Design
19
RE Types
19
D Examples
20
D.1 Multimodal-Mind2Wall
20
D.2 AndroidControl
21
OmniACT
21
D.4 Training Data
22
Data Construction
23
Web-Hybrid
23
Web-Direct
24
E.3
Open-Source Data
25
Model and Training Details
25
Overview
25
F2
AnyRes
26
F3
Training
26
Evaluation Details
26
Model Endpoints
26
G.2
Multimodal-MindZNee
26
AndroidControl
26
G 4
OmniACT
27
Mind2Web-Live
27
6.6
AndroidWorld
27
Prompts
28
16

---

Published as a conference paper at ICLR 2025
A RELATED WORK
GUI Agents. LLMs and MLLMs have demonstrated great capabilitties and potentials in GUI automation, working as digital agents in various GUI environments (Yan et al. 2023 Kim al.) 2023] Wang et al. 2024a Zheng et al. 2024, Xie et al. 22024) Despite the growing number of studies focused on building multimodal agents (Koh al. [024]] Zhou et al. 2024 Cao et 2024). most work still relies on HTML or al ly trees for grounding, even when they are not used for observation. In this work, we advance an alternative line of research: pixel-level visually grounded GUI agents (Shaw et al., 2023; Zhhan & Zhang 2023 Hong et al., 2024: Cheng et al.) 2024 Niu et al. 2024). Unlike nearly all previous work of this line, we propose generic two-stage approach that separates planning and visual grounding to build vision-only GUI agents, which perform remarkably well on realistic agent benchmarks with vision-only input, and offers the ficxibility to the choices of planning and grounding models.
Visual Grounding. Visual grounding has been long studied on natural images (Karpathy et al. 2014: Mao et al.) 2016 Yu et al. 2016). More recently, with the advancements of MLLMs. their
visual grounding capabilities on natural images have attracted significant attention (Bai et al. 2023
Chen et al., 2023a Peng al. 2023 Wang al. 2024bb 2023 Ma al.. 2024). However, due to
significant gaps in image resolution and GUI understanding, these models trained on natural contexts work poorly on GUI visual grounding (Cheng et al. 2024). One of the most popular approaches,
SoM (Yang et al. (023), proposes a visual prompting method that adds marks such as boxes and numbers to images and instructs MLLMs identify the referred objects by the labels. It is widely adopted in GUI scenarios Yan et al. 2023 He et al.) (2024] Koh et al. (024). but still suffers from problems including reliance on complete object information or object segmentation. Only few studies have been conducted for visual grounding on GUI screenshots. Based on Rico (Deka et al. 2017),
Bai et al. (202T) annotates referring expressions by humans; RicoSCA (Li ct al. 2020a) generates a
larger synthetic referring expression dataset; and Li ct al. (2020b) collect numan-labbled captions of UI clements. They have been primary resources for GUI grounding for a long time (Li & Li| 2022]
Banerjee et al. 2022). Later on, Qian et al. (024) synthesize referring expressions from Rico by heuristic rules and train a vision language model by a new layout-aware contrastive learning technique. CogAgent (Hong et al., 2024) compiles HTML documents and screenshots from real websites to GUI grounding data for the pretraining stage, and finetunes on open-source and in-house human-labeled data, build 18B MILM with strong pixel-level GUI grounding capabilities. Ferret-UI (You et al.) 2024) develops a UI generalist MLLM trained on series of UI-related tasks including grounding The most similar effort ours SeeClick (Cheng et al., 2024), which enhances Qwen- VI. (Bai et al. 2023) by finetuning on GUI grounding data, including simplistic synthetic data compiled from real websites. It still falls short of the small image resolution of Qwen-VLz as well as the simplistic nature of the training data. Cheng et al. (2024) also create new grounding benchmark for GUIs, which benefits our evaluation and analysis.
17

---

Published as a conference paper at ICLR 2025
B PHILOSOPHY BEHIND SEEACT-V V AND UGROUND
When it comes to agent designs, the current wisdom, by and large, is to train a monolithic LLM (e.g.. CogAgent (Hong et al., 2024), SeeClick (Cheng et al., 2024), along with several recent supervised fine-tuning endeavors aimed enhancing "agentic behaviors") "). At a philosophical level, part of the goal of SceAct-V V is to challenge that status quo and advocate a modular design for language agents instead.
fundamental challenge of language agents arises from the complexity, dynamism, and inherent idiosyncrasies of the environments in which they operate. For instance, consider web agents: the internet comprises over one billion websites, each of which can exhibit an extremely large and dynamic number of states, and each can be constantly changing (for example, due frequent updates in backend databases). Furthermore, there a considerable amount of highly idiosyncratic semantics in each environment, e-g., uncommon icons, jargon, and counter-intuitive designs.
As a result, although we are still at the early stage of agent research, we posit that a monolithic model, regardless of its future scale and capabilities, is unlikely to fully encapsulate the diverse complexities and idiosyncrasies across all environments. Therefore, developing a generalist agent that reliably generalizes across various contexts necessitates modular system design. This involves synergistically orchestrating foundation model (e.g., GPT-4o) with multiple specialized modules, each tailored to specific functionalities.
Grounding, in particular, is capability for which dedicated module is highly advantageous. Fundamentally, grounding involves interpreting domain-specifee semantics and creating map between that and natural language representations understood by generic LLM. A specialized grounding module simplifies the capture of idiosyncratic semantics and facilitates easier adaptation across different domains (for example, by fine-tuning the grounding model rather than the entire foundation model). Consequently, the grounding module provides domain-spectie semantic input to the foundation model. This constitutes central motivation for the design of SeeAct-V V and the work presented herein.
Our design also offers several practical advantages:
Modularity: It permits the independent study and enhancement of UGround as a standalone ground- ing model, decoupled from specific planning modules.
Flexibility: It is compatible with diverse multimodal LLMs and grounding models without requiring specialized fine-tuning on downstream benchmarks.
Comparative Consistency: By standardizing the planning stage, the design minimizes confounding variables, thereby facilitating clearer assessment of how various grounding models and methods influence agent performance.
Empirical results demonstrate that SeeAct-V, when integrated with UGround, outperforms end-to-end MLLMs (whether employing textual or SoM grounding). This is particularly noteworthy considering that training end-to-end models demands extensive high-quality data on agent trajectories (which combine both planning and grounding), which is both challenging and costly.
C FURTHER ABLATION STUDIES
In addition to the studies in $3. we present further ablation experiments to investigate both model design choices and the effectiveness of web-based synthetic dataset. We report grounding accuracy on ScreenSpot (Agent Setting), with GPT-4o as the planner.
C.I CONTROLLED COMPARISON TO BASELINE MODELS
Both model design and training data contribute critically to the strong performance of UGround. To isolate their individual contributions, introduce new variant, (Ground--ween which is fine-tuned
18

---

Published as a conference paper at ICLR 2025
Table C.I: Ablations of data and base models for UGround on ScreenSpot (Agent Setting).
Model
Model Design
Continual SFT Data
Mobile
Desktop
Web
Avg
Text
Icon/ Widget
Text
lcon/Widget
Text
leun/Widget
Qwen-VI VI,-Chat
Owen-V VL.
None
21.3
21.4
18.6
10.7
9.1
5.8
14.5
SeeClick
Qwen.
SeeClick
69.6
33.6
"Ground-Qwer
Qwen-VV VL
Web-Hyyrid Hyhrid
80.2
57:2
76.3
39.3
74.4
47.1
62.4
UGround
73.4
88.1
61.4
Table C.2: Ablations of image resolution for UGround on ScreenSpot (Agent Setting).
Mobile
Desktop
Web
Continual
Image Resolution
Text
Icon/ Widget
Text
Icon/Widget
Text
Icon/Widget
Avg.
Fixed 448
89.4
65.1
83.5
56.4
77.0
61.7
72.2
Web-Hybrid
Fixed 896 896
86.8
69.0
85.1
62.9
81.4
57.8
73.8
Fixed 1,341/ x 1,344
79.9
68.6
86,
62.1
79.1
63.6
73.2
Dynamic (Ours)
73.4
61.4
64.6
76.9
from Qwen- VL-Chat (the same backbone used in SeeClick), using only our main web-based synthetic dataset, Web-Hybrid The results are presented in Table C.I
Training Data: When using the same backbone DOwen-VL-Chat) UGround-Wwen trained solely on Web-Hybrid achieves average absolute improvement of 10.1% over SeeClick, even though SeeClick incorporates additional open-soure mobile UI data. This result underscores both the high quality of our synthetic web data and its capability generalize across platforms.
Model Design: UGround demonstrates 14.3% absolute improvement over "Ground-Qwen thereby highlighting the effectiveness of our model design.
We omit comparisons with CogAgent due to its inferior performance relative to SeeClick, despite its substantially larger model size (18B parameters) and dataset (140M grounding samples).
C.2 MODEL DESIGN
We analyze the effect of image resolution performance, focusing on key aspects: (1) the impact of increasing image resolution using scaled-up up AnyRes grid settings, and (2) the benefits of dynamic resolution and aspect ratio adjustments compared to fixed square configurations.
Scaling of Image Resolution. We scale up image resolution with fixed square sizes for convenience (448% x 448- >> 896 896-+ 1, 1,344 x 1,344).
As shown in Table C.2 larly on web and desktop UIs that often contain small links and icons. However, mobile UIs, as being less dense, do not benefit as significantly from increased resolution.
larger image resolution generally improves the model performance, particu-
Dynamic Image Resolution and Aspect Ratio. As shown in Table C.2 UGround benefits from dynamic image resolution supported by AnyRes, effectively adapting to varied resolutions and aspect ratios (for example, to mobile UIs or desktop UIs). This flexibility results in improved performance across platforms. For example, desktop and web UIs, UGround achieves comparable superior results using approximately 2/3 of the tokens required by the fixed 1,344 x 1,344 model 16:9 scenarios.
Similar findings around these two aspects are also discussed in general domains (Li et al, 2024a Zhang al. 2024b), well some concurrent GUI works (Chen al. 2024
Li et al. 2024c).
C.3 RE TYPES
The taxonomy for REs introduced in this work represents novel contribution and has not been addressed in prior studies (Li et al. 2020bb; Hong et al.) 2024 Cheng et al. (024).. In this section, we present ablation studies focused on the role of positional REs. We omit detailed studies on
Thee data is converted to the format used in SeeClick. Given the maximum sequence length used in the training of Qwen-VI VI. and SeeClick, reduce the elements maximum of 30 for each page.
19

---

Published as a conference paper at ICLR 2025
Table C.3: RE ablations for UGround on ScreenSpot (Agent Setting).
Mobile
Desklup
Web
Training Data
Text
Icon/ Widget
Text
lcon/Widget
Text
Icon/ WWddet
Average
Web- Hyhrid RFs)
87.1 88.1
65.55 64.6
76.0
Web-Hybrid
89.0
73.4
61.4
84.8
76.9
Table C.4: RE ablations for UGround on ScreenSpot (Standard Setting).
Mobile
Desktop
Web
Training
Text
Icon/ Widget
Text
Icon/ Widget
Text
Icom/ Widget
Average
Web-Hybrid (wlo Pos REs)
72.2
32.0
72.7
55.0
76.5
61.2 68.0
64.9
Web Hybrid
75.5
54,2
79.9
58.6
77.0
68.8
visual and functional REs because (I) they are interleaved in HTML DOMs and are challenging to fully disentangle, and (2) they have been extensively studied in prior work. For example, an HTML attribute (e.g., aria aria-labell may convey both visual and functional cues, and the MLLM can exploit different aspects of the input.
We train new checkpoint with Web-Hybrid, omitting all positional REs while maintaining the overall number of web elements. As shown in Table C.3 and Table C.4 the inclusion of positional REs generally enhances model performance.
We hypothesize that the integration of positional and contextual data enables the model to more effectively capture and attend to the spatial relationships among UI elements. This enhanced contex- tual understanding is crucial for grounding tasks that cannot rely solely on visual or functional cues, especially in challenging cases where those cues alone insufficient.
D EXAMPLES
D.1 MULTIMODAL-MIND2W
Task: Find the page with instructions how to return orders online.
Dividing into blocks
Planning
Grounding
Block
User: screenshat, coordinates (s. element comesponding "Link Iibeled 'Returns Exchanges In the footor of the webpage" ?
pivol
ACTION: SUROLL DOWN
ELEMENT: Nove
VALUE: None
Block 2
GPT-4:
ACTION CLICK
UGround: (325, 604)
ELEMENT: Link laboled
wabpaage
VALUE: Nono
Next Action: CLICK (326, 604)
Figure D.1: Example of the Multimodal-MMnd2WWW evaluation pipeline.
20

---

Published as a conference paper at ICLR 2025
D.2 ANDROIDCONTROL
Task: am feeling hungry and want to try something new. Search for margherita pizza recipe in the SideChef app.
Planning
Grounding
High-Level
Highh-Lvvl
High-Level Goal:
Previous Actions: C'Cpnn
coordinates (s.
sioeChel opo" "Emer the marghorta
comespanding to "the first aearch reault labeled 'margherita pizza" ??
aearch bar]]
GPT-4o: ("action type" "cick"
UGround: (540, 398
Telement: "the first search result Iabeted tmarghents pizza")
Low-Level
Low-Level
conrcdinatio
User: High-Level Goal: (Task Above)
comsspanding 'marghertta pizzaa"
Low-Lavel leatr Inttrction: Click on the
UGround: (540, 389)
GPT-40: ( Faction bypee "clck". search
Next Action (High Low) ("actionn_ bype"" "click", "x" 540, "y:: 399)
mmarheenta
Figure D.2: Example the AndroidControl evaluation pipeline.
D.3 OMNIACT
Task: Fill "Singapore" as the travel destination on the search bar.
Planning
Grounding
Rased acreanshot, generati PyAutaGUI scrpt for the task.
alament conrasanndigg to "Input field labeled 'Flying to"
UGround: (1460,
pyautogul caliiclkkkk""iii field laboled 'Flying to"
Final Script:
mrautogui.clicki 1440, 306)
pyautogull Siipppprrr"""
pyautogui "Singe""iigapore"
pyautogui.pres "resst"enterr)
prrautoguupresssennee.
Figure D.3: Example of the OmniACT evaluation pipeline.
21

---

Published as a conference paper at ICLR 2025
D.4 TRAINING DATA
Mobile
AndroidContral
WidgelCaption
Gmail
38
HEW
Wetch Liv
Trafle
Cid ov
Old ov St vocas"" antiocanni sown.
odd ti arew 1L/mtye
apai
Web
Iramecto: Nodgalle
festrvetoon Aoc dicuratte adaiis intto lea
Instrvation 8 goup accurt buics
Polshad Ponat Tatak
Glocoo boibe bookic "Worann" wh top sowerss.
@rodown rorra
GUIAct
Web-Hybrid
Worry free PostgreSOL hosting
Lanest News
invang paing
Whar's Haypwinggg
Instuctio::: nooe anout Parkpedid hosking
Enstraation: Agran to GOtt
the coulle mull calon
Clack Fart arfire
Stilee
Web-Direct
LACANONLINE.COM
Ieprestion
SMEEZION
St liper Sct
natu betwoes HCME and RLANN
Dear intt Jeane" adica page
Figure D.4: Examples of training data from different sources.
22

---

Published as a conference paper at ICLR 2025
E DATA CONSTRUCTION
We describe the details of our data construction this section. Illustrative examples of all our training data are provided in Figure D.4
E.1 WEB-HYBRID
Following prior work Hong et al. 2024 Cheng et al. 2024), we download and randomly sample from the latest Common Craw We apply several filtering methods exclude non-webpage files based on URL patterns and to remove non-English pages as indicated by the language labels provided by Common Crawl. We employ Playwright to load and render webpages, capture screenshots, and collect metadata for web elements. To ensure a diverse set of data, we simulate vertical scrolling capture screenshots and elements various positions on each webpage. The metadata for each element includes bounding box coordinates and relevant HTML attributes, such as the element's tag, inner text (inner text), and alternative text (e.g., alt).
During rendering, we randomly select image sizes to cover a diverse range of resolutions and aspect ratios. Approximately one-third of the data is rendered in mobile-friendly aspect ratios, thereby triggering the mobile version of certain websites and enhancing the coverage of mobile UI environments. For each long webpage, up to three blocks content within a viewport-sized area are randomly sampled to ensure content diversity, In total, the dataset comprises approximately 773K screenshots from around 700K URLs.
As detailed in 2.2, we employ a hybrid strategy to generate REs for webpage elements. Below, first describe how we leverage MLLMs (LLaVA-NeXT-133) and LLMs (Llama-3-8B) to generate concise, element-level descriptions without positional contextual information.
We extract the bounding box regions from the webpage screenshots corresponding to the elements and pass these smaller cropped element images along with their salient HT attributes LLaVA. Using the prompts outlined below, we prompt LLaVA to generate an element description hased on its internal knowledge, the element's ' image, and relevant HTMM. attributes:
Based on the attached image of web element, please provide short description of the web element displayed. The goal capture the intuitive and visual appearance HTML information as context but focus more on describing what is visually observable. Avoid directly referencing HTML attributes; instead, interpret their possible visual implications if they can be inferred from the image. Be cautious of potential inaccuracies in the HTMI attributes and use them to enhance understanding only when they align reasonahly with what can be inferred visually.
element. Use accompanying
HTML: (A list of salient HTML attributes}
We observe that since the input to LLaVA is small cropped image, the model tends have less hallucinations compared directly caption an element language capabilities of the 13B LLaVA model, the generated interpretations tend to be lengthy. To address this, the lengthy output is subsequently processed by Llama-3-8B with the prompt below that instructs it to condense the description into brief referring expression:
bounding box overlaid in the image. However, due the limited
Here is a description of element in wehpage. Using the detailed description provided, create concise phrase that captures the essential visual and functional characteristics of the web element. The rephrased description should be straightforward, simple and precise enough to allow humans quickly spot this element in a webpage screenshot. Focus the most prominent visual features and any critical function indicated by the text.
Deseription: {}
Leave only your final description the answer, without any explanation.
Next, the generation process for each crawled element is follows.
We begin by categorizing the webpage elements based on their tags into two groups: interactive elements (c.g.. a, input, select, etc.) and pure text elements (c.g., p. h1, h2, etc.). Referring expressions are generated only
CCC-MAIN-2023-5
23

---

Published as a conference paper at ICLR 2025
Table E.1: Statistics of element types (by HTML tags) in Web-Hybrid (%).
img
button
input
svg
select
textarea
video
68.99
15.41
6.81
5.32
2.25
0.99
0.18
0.04
Table E.2: Statistics of element HTML attributes and MLLM-based synthetic REs used in Web- Hybrid (%). Calculated as the number of elements using an attribute/RE divided by the total number of elements.
MLLM-based RE
inner-text
title
alt
aria-label
aria-describedby
placeholder
value
11.19
43.58
20.01
12.25
11.32
0.21
0.06
0.02
for interactive elements, as these constitute the primary targets in GUI grounding tasks. In addition, pure text elements are utilized as potential sources for referring expression generation.
For each interactive element, we first apply an OCR model (FasyOCR*) extract text from the element's ' hounding box. If the similarity between the OCR-extracted text and the element's inner nnxer_text exceeds threshold of 0.7, the element is considered textual, and the MI..M-basedd synthesis pipeline is bypassed. This procedure prevents the generation of trivial data (e.g.. "Gray links labeled by link text"). Moreover, for textual elements, those sharing identical text with other elements on the same page are filtered out to avoid grounding ambiguities.
Based on manually crafted rules, we label each element's neighboring elements various directions (multiple neighbors are allowed), mark the nearest upper h1, h2, or h3 elements (titles), and determine its absolute position (e.g., center of the screenshot, top, top-left corner) to generate position-based referring expressions. We randomly select up to neighboring elements in different directions and randomly pick elements whose distance from the larget within 500 pixels (empirically, always selecting the closest element does not yield the best performance). These are used to generate relative position descriptions. Some of the relative descriptions are further randomly modified to common terms such as "next to" or "between". For contextual references, if an element is identified as checkbox or radio button based on its HTML properties, it is assumed to have an associated label (c.g.. "radio button for Yes"). If such labels are provided in the HTML attributes, they are used directly; otherwise, the nearest element the same row (or column, if necessary) is selected as the lahel. Similar procedures are followed for input fields and select boxes. Additional expressions such as "under," "in," or "under section A" are generated based on the hierarchical structure of titles (primarily h1, h2, and h3). Attributes like title, alt, or ar aria-label are always considered as potential descriptors, typically contributing functional information.
Finally, for each element, descriptors from accessibility labels, the element's 's own text, MLLM-based descriptions randomly combined with absolute positional information (included on random basis) and supplemented by between zero and two relative or contextual descriptions. For interactive elements such as radio buttons, the label is always included. In each webpage, up to 100 elements are selected, prioritizing those with accessibility labels MLLM annotations. The number of pure text elements is limited to no more than three times the sum of elements with accessibility labels and those annotated via MLLMs (with minimum of 10, the total available elements, whichever is lower) reduce the number of pure text elements. Additionally, unique accessibility labels and their frequencies are counted; labels occurring more than 1,000 times are downsampied to a maximum of 1,000 occurrences. For example, the label "Next" appears 13K times, and is downsampled to 1K occurrences in our training data.
To illustrate the primary data distribution, provide statistics about HTML element types, as well as attributes and positional RE types used the final REs within Web-Hybrid. The statistics are shown in Table E. Table E.2) and Table E.3] We omit exact percentages of visual and functional REs because they are often interleaved in HTML DOMs and MLLM based synthetic REs, and generally are hard to distinguish.
E.2 WEB-DIRECT
For the Web-Direct dataset, we directly employ GPT-4o to generate referring expressions. We observed that, due to its limited grounded understanding capabilities, simply enclosing an element in the image with a bounding box often leads to notable hallucinations, particularly when it provides descriptions of nearby elements. To mitigate these hallucinations without incurring the high cost of manual post-verification we find that annotating an element with both a red bounding box and a red arrow pointing substantially reduces hallucinations.
24

---

Published as a conference paper at ICLR 2025
Table E.3: Statistics of relative positional REs, absolute Positional REs, and contextual REs used in Web-Hybrid (%) Contextual References are also counted relative positional REs. Calculated as the number of elements using an RE divided by the total number of elements.
Relative Positional RE
Contextual RE
Absolute Positional RE
23.49
8.43
3.05
In addition, we explicitly query GPT-4o regarding the identification of the element, which further minimizes potential hallucinations and filters out small number of crawling errors or occluded elements.
Two separate prompts used in Web-Direct: one to generate free-form referring expressions and another to generate functionally oriented referring expressions:
Here is supposed to be an interactive element (button, link, dropdown, text box. etc.) in the red box pointed by arrow in the screenshot. Can you find it? Is it visible from the screenshot? Can you write concise description that is sufficient for humans to locate it from the screenshot? Your response should be ISON. For example, "visible": true, "description": "your description here".
Here is supposed to be an interactive element (button, link, dropdown, text box, etc.) in the red box pointed by arrow in the screenshot. Can you find it? Is it visible from the screenshot? What unique function does this element enable? Your response should be ISON. For example, "visible": true, "action": "subscribe the latest updates".
E.3 OPEN-SOURCE DATA
We leverage several high-quality open-source referring expression datasets in Android, well the GUTAct dataset, as supplementary sources web data, Specifically:
We use the annotated data from GUIAct (web-single), Steps that do not involve coordinates or that are marked as multi-step operations (for example, "click then type") are filtered out. We use both the Instruction and Action annotations for grounding (i.e., each element is seen in training twice with different expressions).
1. GUTAct:
Contoid Similarly, we use annotatd actions from the training set. filter any actions that do not have associated coordinate data, ensuring that only steps with specific visual grounding targets are included in the dataset.
3. Widget Caption: For each element in the training set, multiple functional captions are provided. To enhance diversity, captions per element randomly selected from the available set of functional captions during data construction.
UlBert: use training set elements from UlBert without any additional special processing, directly utilizing the referring expressions provided by this dataset.
5. AITZ:
We incorporate the annotated actions (Thought) from AITZ, using each step's action annotation for grounding in the dataset. These annotations contribute to diverse set of referring expressions, particularly for action-oriented grounding tasks.
F MODEL AND TRAINING DETAILS
F. OVERVIEW
flexible investigation of model architecture, build the architecture based LLaVA-NeXT (Liu 2024b). and train from scratch using open-source source data from Liu et al. (2024a). We use CLIP ViT-L-14 (224px) as our base image encoder for more flexible splitting of AnyRes, and keep it frozen during training. We use Vicuna- 1.5-7a-1.5-7h--1 1.5-7b-16k (7heng et al. 2023) as the language backbone as long-context LM backbone for handling long visual contexts.
25

---

Published as a conference paper at ICLR 2025
F.2 ANYRES
As described in 2..3 AnyRes allows convenient scaling up of image resolutions, although it's not always beneficial to enlarge image resolutions (L1 at. 2024a). We keep the main pipeline of AnyRes, splitting images into 22 grids. However, to keep the original image aspect ratios, we resize only by width and pad to the bottoms if needed, and use pixel-level coordinates in numbers that are compatible with this design. We allow most 36 grids, for maximum resolution of 1,344 1,344 and 896 2,016. We empirically find AnyRes does not generalize unseen image resolutions for visual grounding. Therefore, we resize images by width to keep them within the training resolution ranges when needed. We remove the low-resolution image for providing global context, because intuitively does not provide informative contexts when images are larger than 1,000px. and we empirically find slightly hurt the performance.
F.3 TRAINING
Our training primarily consists of two stages:
VA-15 Pretraining and Finetuning: We follow the exact pretraining in Liu al. (2024a). Then, in the instruction finetuning stage, we change the grounding data from normalized coordinates to absolute coordinates as we wish, and start to use our modified AnyRes setting.
2. GUI Visual Grounding: Then we train UGround our training datasets.
Due the huge computation cost of handling high-resolution images, we use LoRA (Hu al. 2022 for instruction finetuning in the stages, with device batch size of 4.
The first stage takes about hours on a single NVIDIA A100 machine (global batch size 128 with gradient accumulation). For the large-scale GUT data training, we use 112 NVIDIA H100 GPUs and finish the training in about hours (global batch size 448).
G EVALUATION DETAILS
G.1 MODEL. ENDPOINTS
As studied Paan al. 2024), different GPT endpoints could lead to slight differences the performance of GUI tasks. Hence, we provide the specific endpoint names we use in our evaluation, as well as those of the basclines we use (if available).
Ours (across every benchmark);
and
Bittimodal-Mind2Weal got-4-1106-vissionppreeeww
OmniACT: gpt-4-061- and got-4-1106-visioo-rreveww
AndroidWorld: 2t 200--022-2-----222-00--111111111
Mind2Web-Live gpt-4-0125-previeve
G.2 MU..IMMDAA-MINDDNW
Many screenshots Mulliodal--Mndd2WW have giant vertical heights (e.g.. 1,280 x 10,000 pixels). Similar to Zheng al, (2024). to avoid overly long screenshots, we divide whole webpage screenshots into viewport-sized blocks, and simulate scrolling down to the next block whenever agents determine that no valid action be taken explicitly choose to scroll. Specifically, we divide each full-page screenshot into 1,280 1,000 pixel blocks, except for the final block, which may be shorter depending on the page's 's total height. Most of the target elements are within the first block (about 80%). See Ligur D.I for an illustrative example of the pipeline.
We report element accuracy the benchmark, and the grounding is considered to be correct if the output coordinates fall in the box coordinates of the ground truth element.
G.3 ANDROIDCONTROL
We adopt the M3A (Multimodal Autonomous Agent for Android) prompt (Rawles et al. 2024). the state-of- the-art zero-shot method Li et al. (202b)). We only make minor modifications integrate UGround into M3A.
We follow the standard data processing steps outlined in Li el al. (024b). During evaluation, coordinates generated by grounding models are translated to the smallest visible element that includes the coordinates.
26

---

Published as a conference paper at ICLR 2025
G.4 OMNIACT
We follow the method in Kapoor al. (2024) for prompt design and the selection of five in-context examples. prompt slightly modified generate element descriptions as function parameters PyAutoGUI scripts, instead of directly outputting coordinates. After generating the PyAutoGUI script with element deseriptions, we use grounding models to predict the corresponding coordinates and substitute them back into the original script. See Fegure D.3 for an illustrative example of the pipeline.
We compare our method with DetACT KKaoor al. 2024), state-of offthhe-rtt method in Kapoor et al. (2024), which extracts UT elements and their coordinates through combination of OCR, icon matching. and color detection. These elements are filtered by task relevance and passed to L.MMs or MILMS to generate the PyAutoGUI script. In contrast, our method does not use pre-generated elements list. The planner model focuses generating precise element descriptions based solely on the screenshot. Additionally, we corrected basic errors the public evaluation scripts (for example, wrong file paths wrong calculation of distances).
G.5 MIND2WEB-LIVE
baseline agent in Pan et al. 20244 text-only, perceives and interacts with webpages by hundreds of textual HTML elements at time. To study vision-only agents, we change the observation to pure screenshots. We also make necessary changes to the standard action space to entirely isolate HTML from the planning, grounding, and execution: We add Scroll ___ and Scroll Down to the action space to better support vision-only agents with vicwport-sized observation. 2) We remove Fill_Form and Fill Fill_Search from the action space, which use an additional judgment model to determine whether to press enter after typing through HTML information. Instcad, we use Type and Press Enter to let the agent make its own decisions autonomously. 3) We disable API-based Select, and force agents select options merely through clicking and make the action more challenging. We admit some select buttons cannot easily operated with only Click. We compromise this point fulfill the motivation of this vision-only study.
G.6 ANDROIDWORLD
We build SeeAct-V agents based on M3A agent in Rawles (2024), which receives raw and SoM images, and reason about the next action in a ReAct style Yao et al. 2023 and choose the next target element from the element list. It also adopts self-reflection (Shinn et al. (024 in the agent pipeline to instruct agents to summarize the current move and facilitate the following steps.
We mainly remove SoM images and textual list of elements from the al tree in the observation (in both planning and reflection phases), and change element-based actions pixel-level actions.
27

---

Published as a conference paper at ICLR 2025
H PROMPTS
Table H.1: Prompt used for the planning model in Multimooda--Mind2Weeb, modified from the prompt in (heng et 2024)
System Role
You are imitating humans doing web navigation for task step step.
each stage, you can see the webpage like humans by screenshot and know the previous actions before the current step through recorded history.
You need to decide on the first following action to take. You click element with the mouse, select option, type text with the keyboard, scroll down.
Task Description
You are asked to complete the following task: [ask description} Previous Actions: {List of previous actions, if any)
The screenshot below shows the webpage you see.
Useful Guidelines
First, observe the current webpage and think through your next step based on the task and previous actions.
be successful, it important follow following rules:
1. Make sure you understand the task goal to avoid wrong actions.
2. Ensure you carefully examine the current screenshot and issue valid action based on the observation.
3. You should only issuc action at time.
4. The element you want to operate with must be fully visible the screenshot. If is only partially visible, you need to SCROLL DOWN to see the entire element.
5. The necessary element to achieve the task goal may be located further down the page. If you don't interact with any elements, simply select SCROLL DOWN move the section below.
Reasoning
Explain the action you want to perform and the element you want to operate with (if applicable). Describe your thought process and reason in 3 sentences.
Output Format
Finally, conclude your answer using the format below.
Ensure your answer strictly follows the format and requirements provided below, and clear and precise.
The action, element, and value should each be three separate lines.
Choose an action from CLICK, TYPE, SELECT, SCROLL DOWN. You must choose one of these four, instead of choosing None.
ACTION:
ELEMENT: Provide description of the element you want operate. (If ACTION SCROLL DOWN, this field should be none.) It should include the element's identity, type (button, input field, dropdown menu, tab, etc.), and text on it (if applicable).
Ensure your description is both concise and complete, covering all the necessary information and less than 30 words.
you find identical elements, specify location and details to differentiate from others.
VALUE: Provide additional input based on ACTION. The VALUE means:
If ACTION TYPE, specify the text typed. If ACTION = SELECT, specify the option to be chosen. Otherwise, write 'None'.
28

---

Published as a conference paper at ICLR 2025
Table H.2: Prompts used for the planning model in AndroidControl, modified from the prompt in Li et al. 2024b) and (Rawles et al. 2024)
General Instruction
You are an agent who can operate an Android phone on behalf of a user. Based on user's goal/request, you may complete some tasks described in the requests/goals by performing actions (step by step) on the phone.
When given a user request, you will try to complete step by step. At each step, you will be given the current screenshot and a history of what you have done (in text). Based on these pieces of information and the goal, you must choose to perform one of the action in the following list (action description followed by the JSON format) by outputting the action in the correct JSON format.
If you think task has been completed, finish the task by using the status action with complete as stalus ("action_type "status" "goal_ssatus" "successful""" If you think the task is not feasihle (including cases like you don't 't have enough information or cannot perform some necessary actions), finish by using the status'action with infeasible goal status: { ("action type": "status", "goal status": "infeasible")
Click/tap on an element on the screen, describe the element you want to operate with: ("action type":
"click", "element": wtapeeeeeeemettdssriiepee Long press on element on the screen, similar with the click action above: ("action.type" "long-press" "description":
Type text into a text field:
puaction.type.. "type_text", "text": (text, "element":
(target.clement desceritioo))) }
Scrool the screen in one of the four directions: ("action_type": "scroll", "direction": (up, down, left, right))
Navigate the home screem: ("action type": "navigate home")
Navigate back: ("action type": "navigate back")
Open an app (nothing will happen if the app is not installed): ("action.type": "open_app".
nape_ (name)}
Wait for the screen to update: {"action_type": "wait")
Useful Guidelines
Here are some useful guidelines you need to follow:
General:
Usually there will be multiple ways complete task, pick the easiest one. Also when something does not work as expected (due to various reasons), sometimes simple retry solve the problem, but if doesn't it (you can see that from the history), SWITCH to other solutions. If the desired state is already achieved (e.g., enabling Wi-Fi when it's already on), you can just complete the task.
Action Related:
Use the 'open_app' action whenever you want to open an app (nothing will happen if the app is not installed), do not use the app drawer to open an app unless all other ways have failed. Use the 'type. text' action whenever you want to type something (including password) instead of clicking characters
the keyboard one by one. Sometimes there is some default text in the text field you want to type in, remember to delete them before typing.
For 'click', 'long-press' press' and 'expe,text,, the element you pick must be VISIBLE in the screenshot to interact with it.
The 'element' field requires concise yet comprehensive description of the target element in single sentence, not exceeding words. Include all essential information to uniquely identify the element. If you find identical elements, specify their location and details to differentiate them from others.
Consider exploring the screen by using the 'scroll' action with different directions to reveal additional content.
The direction parameter for the 'scroll' action specifies the direction in which the content moves and opposites to swipe: for example, to view content at the bottom, the 'scroll' direction should be set to 'down'.
Text Related Operations:
Continued the next page
29

---

Published as a conference paper at ICLR 2025
Table H.2 Continued from the previous page
Normally to select certain text on the screen: (i) Enter text selection mode by long pressing the area where the text is, then two pointers indicating the range) and usually text selection bar also appear with options like 'copy'. paste', 'select all', etc. (it) Select the exact text you need. Usually the text selected from the previous step is NOT the one you want, you need to adjust the range by dragging the two pointers. If you want select all text in the text field, simply click the 'select all' button in the bar.
of the words near the long press point will be selected (highlighted with
At this point, you don't 't. have the ahility drag something around the screen, in general you cannot select arbitrary text.
To delete some text: the most traditional way is to place the cursor at the right place and use the backspace button in the keyboard to delete the characters one by one (can long press the backspace to accelerate if there many delete). Another approach is to first select the text you want to delete, then click backspace button in keyboard.
To copy some text: first select the exact text you want to copy, which usually also brings up the text selection bar, then click the 'copy' button in bar.
To paste text into text box, first long press the text box, then usually the text selection bar will appear with 'paste' button in it.
When typing into a text field, sometimes an auto-complete dropdown list will appear. This usually indicates this is enum field and you should try to select the best match by clicking the corresponding one in the list.
High-Level Prompt
[General Instruction)
The current user goal/request is: (High-level goal)
Here is history of what you have done so far: (History)
The current raw screenshot is given to you. {Useful Guidelines)
Now output action from the above list in the correct JSON format, following the reason why you do that. Your answer should look like:
Reason: Action: t"action.type"....
Your Answer:
Low-Level Prompt
[General Instruction)
The user's high-level goal/request is: [High-level goal] The current next step's s low-level goal is: (Low-level goal)
The current raw screenshot is given to you.
{Useful Guidelines)
Now output action from the above list in the correct JSON format, following the reason why you do that. Your answer should look like:
Reason:
Action: ("action type":
Your Answer:
30

---

Published as a conference paper at ICLR 2025
Table H.3: Prompt used for the planning model in OmniACT, modified from the prompt in Kapoor et
2024)
General Instruction
You are an excellent robotic process automation agent who needs to generate a PyAutoGUI script for the tasks given to you. You will receive some examples to help with the format of the script that needs to be generated.
There are some actions that require you to provide an element description for the elements you want to operate on. For the description, follow the requirements below:
Element Description Requirements:
Provide concise description of the element you want operate.
should include the element's identity, type (button, input field, dropdown menu, tab, etc.), and text on it (if have).
If you find identical elements, specify their location and details to differentiate them from others. Ensure your description is both concise and complete, covering all the necessary information and less than 30 words, and organize it into one sentence.
[IMPORTANT!] Stick to the format of the output scripts in the example. [MMPORTANT!] Use only the functions from the API docs. [MPORTANT!]] Follow the output format strictly. Only write the script and nothing elsc.
API Reference
Here is the API reference for generating the script: def clickclemeu--deccipiionn)
"MMoes the mouse to the element corresponding to the description and performs left click.
Example:
High Level Goal: Click at rectangular button labeled "Next".
Python script:
import pyautogui
pyautogui.clickt cititoguu.cicc(RReettnnulla RRectangular red button labeled "Next"
pass
def highhCChikkeeeeettzeeeerrpppiiii
"Moves the mouse the element corresponding to the description and performs right click.
Example:
High Level Goal: Right-elick at link labeled "vacation rendels"under the "housing" section..
Python script:
import pyautogui
pyautogui. rrautoguirigghiClckk...... labeled "vacation rentals"under the "housing" "scction""
pass
def wuaubleCCiccceeeee--deeepppp
"Moves mouse the element corresponding to description and performs double click. Example:
High Level Goal: Double-click at folder named "courses".
Python seript:
import pyautogui
named "courses" ")
def scroll .o-scoll):
"'Scrolss the window that has the mouse pointer by float value to.srll
Example:
High Level Goal: Scroll sereen by 30.
Python script:
import pyautogui
pyautogui 330))(3)
pass
Continued on the next page
31

---

Published as a conference paper at ICLR 2025
Table H.3 Continued from the previous page
def hscrollcchcks--ammun... to.scollll
"Scrolls the window the mouse pointer horizontally by float value (amount to scroll).
Example:
High Level Goal: Scroll screen horizontally by 30.
Python script:
import pyautogui
pyautogui. pyautogui.hscrol((30 (3crol((3))))
pass
def To cllement-descriptionn button=boldButton));
"Drags the mouse to the element corresponding to the description with (holdButton) pressed. hold- Button can be 'left', 'middle', or 'right..
Example:
High Level Goal: Drag the screen from the current position recycle bin with the left click of the mouse.
Python script:
import pyautogui
wrapttoggi.dragTooRRReeeooo bin with trash shape", "left")
pass
def move move To(element description):
"Takess the mouse pointer the element corresponding the description. Example:
High Level Goal: Hover the mouse pointer to search button.
Python script:
import pyautogui
pyautogui.move moveeoo""eqqusee"e appointment button")
pass
def wrrtenstr=stringTypee interval=secs hetween keys): "Writes the string wherever keyboard cursor at the function calling time with sees.beeween.key)) seconds between characters.
Example:
High Level Goal: Write "Hello world"with 0.1 seconds rate.
Python script:
import pyautogui
pyautogui. write("Hello world", 0.1)
pass
def poassssttstring.tootypo
"Simulates pressing a key down and then releasing it Sample keys include 'enter', 'shift', arrow keys, fl'.
Example:
High Level Goal: Press enter key now.
Python script:
import pyautogui
pyautogui.
pass
def hotkey(Pargs list_o hotkey): "Keyboard hotkeys like Ctrl-S Cirl-Shift-1 be done passing list of key hotkey(). Multiple keys can pressed together with hotkey.
Example:
High Level Goal: Use Ctrl and V to paste from clipboard.
Python script:
import pyautogui
Continued on the next page
32

---

Published as a conference paper at ICLR 2025
Table H.3 Continued from the previous page
syautooguihotkky(ctttt "v")
pass
Examples
Here are some examples similar to the tasks you need to complete. However, these examples coordinate format for actions like click, rightClick,, doubleClick, moveTo, dragTo, instead element description. You should only refer the actions these examples, and for the output format, stick the content in the API reference.
For example, do not output wttpttoggii.licc111...... menu button with downward arrow in the top right corner") Omit "import pyautogui", do not include any comments or thoughts. Your output should only contain the script itself.
instead output "pyautogui.clikk" clickeGGiiillc Tools
{Example list}
Task Description
Based on the screenshot, generate PyAutoGUI script the following task: {Task description) You should list all the necessary steps to finish the task, which could involve multiple steps. Also, ensure simplifying your steps as much as possible, avoid dividing a single task into multiple steps if it can be completed in one.
Table H.4: Prompt used for the planning model in ScreenSpot (Agent Setting).
Task Deseription
You are an excellent agent for mobile, web, and desktop navigation tasks. Describe target element for task based the provided screenshot: Task: (Task description)
Element Description Requirements
Provide concise description of the element you want operate. Ensure your description is both concise and complete, covering all the necessary information less than 30 words, and organized into one sentence. If you find identical elements, specify their location and details to differentiate them from others.
Output Format
Your output should only include element description itself and follow the requirements. Do not start with "the target element" or "the element".
33


Reviews:
[{"review_id": "wr0LkoMj69", "reviewer": "Reviewer_pZoa", "summary": "This paper introduces a vision-only approach to GUI agents that mimics how humans interact with interfaces through visual perception and pixel-level operations. Moving away from conventional reliance on HTML and accessibility trees, the authors develop SeeAct-V, a framework for human-like GUI interaction, and UGround, a visual grounding model trained on the largest GUI grounding dataset to date (19M elements). Through comprehensive evaluation across six benchmarks, they demonstrate that UGround outperforms existing grounding models by up to 20%, while SeeAct-V agents match or exceed state-of-the-art agents that use additional text-based input.", "questions": "- What factors contribute to the higher Completion Rate (CR) of the image-based model (GPT-4o with UGround) compared to the text-based model on Mind2Web-Live (Table 7), even though the Success Rate (SR) is comparable or lower? Could the grounding method (UGround vs. Choice) impact CR and SR differently across input modalities?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper contributes to reframes GUI interaction as a pure visual grounding problem, challenging the conventional wisdom that additional textual representations are necessary.\n- The authors develop a novel way to generate diverse referring expressions (REs) by categorizing them into visual, positional, and functional types. And they introduce an innovative hybrid data synthesis pipeline that combines rule-based and LLM-based approaches\n- This paper includes a comprehensive agent evaluation covering Three platforms (web, desktop, mobile), Three evaluation settings, (grounding, offline agent, online agent) six different benchmarks.", "weaknesses": "- UGround relies on an external LLM planner and cannot operate independently as a GUI agent without training on downstream tasks. When combined with the Scaling Curve in Figure 5 on Web-Hybrid, it becomes challenging to enhance agent performance by merely increasing grounding data. Instead, improvements depend on the external LLM planner, which may limit the potential of the SeeAct-V framework.\n\n- In the current model architecture, the authors have increased the input image size to 36 grids of CLIP@224. This results in a large number of image tokens ((224/14)^2 * 36 = 9,216), leading to inefficiency. The authors could conduct an ablation study to determine if this resolution is necessary, considering the cost of image tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cEzojQ8uCw", "reviewer": "Reviewer_QsLt", "summary": "This paper introduces UGround, a universal GUI visual grounding model, and SeeAct-V, a vision-only framework for GUI agents. The key contributions include: (1) A novel hybrid data synthesis pipeline for creating large-scale GUI grounding training data from web sources, (2) A universal visual grounding model that achieves strong cross-platform generalization, and (3) A vision-only framework that achieves comparable or better performance than methods requiring additional textual inputs. The authors conduct comprehensive evaluations across six benchmarks spanning web, desktop, and mobile platforms, demonstrating the effectiveness of their approach in both offline and online settings.", "questions": "Thank you for this interesting paper on vision-based web UI understanding. I have several questions and suggestions that I believe could help strengthen the work:\n\n1. **Data Collection and Processing Details:**\n   - Could you provide specific details about your webpage rendering and screenshot capture process? In particular:\n     - What techniques do you use to capture content beyond the initial viewport?\n     - How do you handle dynamic content that requires scrolling?\n     - What is your approach for capturing interactive elements (dropdowns, expanded menus) that only appear after user interaction?\n   - Documenting these technical details would help others reproduce your data collection pipeline.\n\n2. **Desktop Application Extension:**\n   - Your paper mentions potential extensions to desktop applications. Could you elaborate on:\n     - What specific approaches have you considered for adapting your data collection pipeline to desktop environments?\n     - How would you address the challenge of capturing UI states in desktop apps with complex interaction patterns?\n     - What modifications to your current methodology would be needed to handle the diverse widget types and layouts found in desktop applications?\n\n3. **Performance Benchmarks:**\n   - Have you conducted timing experiments comparing your vision-only approach to methods that use accessibility trees, particularly for the online agent evaluation tasks?\n   - Could you provide detailed end-to-end latency comparisons between your approach and traditional methods across different scenarios?\n   - What are the specific performance implications of eliminating accessibility tree extraction in real-world applications?\n\n4. **Benchmark Selection:**\n   - I noticed that the evaluation doesn't include the widely-used WebArena/WorkArena/OSWorld/WindowsAgentArena benchmark. Could you explain:\n     - What specific technical or methodological challenges prevented its inclusion?\n     - Whether your approach is fundamentally incompatible with the these testing environments?\n     - If there are plans to evaluate on this benchmark in future work?\n   \nThis additional context would help readers better understand the scope and limitations of your approach, particularly regarding its applicability to different UI environments and its performance characteristics.\n\nThe current results are promising, but addressing these points would significantly strengthen the paper's contribution to the field.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Technical Innovation:\n- Novel hybrid synthesis pipeline combining rule-based and LLM-based approaches\n- Successful demonstration of cross-platform generalization without platform-specific training\n- Effective vision-only framework that eliminates dependency on HTML/accessibility trees\n\n2. Experimental Rigor:\n- Comprehensive evaluation across multiple platforms and settings\n- Strong performance improvements (up to 20% absolute improvement in standard setting)\n- Thorough ablation studies on training data sources\n\n3. Practical Impact:\n- Reduces dependency on noisy and incomplete text-based representations\n- Potentially more efficient due to reduced input processing requirements\n- More closely mimics human interaction patterns with GUIs\n\n4. Reproducibility:\n- Clear methodology description\n- Release of the largest GUI visual grounding dataset to date\n- Public release of the UGround model\n\n5. Thorough Motivation and Analysis:\n- Excellent analysis of the necessity for vision-only approaches\n- Clear quantification of overhead costs associated with a11y tree extraction\n- Convincing demonstration of additional computational burden from processing textual information\n- Well-reasoned arguments for moving away from text-based representations", "weaknesses": "1. Data Efficiency:\n- Heavy reliance on large-scale synthetic data\n- Potential redundancy in web-based training data\n- Room for improvement in data deduplication and grouping\n\n2. Limited Coverage:\n- Lack of desktop UI data in training\n- Incomplete handling of long-tail elements\n- Platform-specific icons and elements not fully addressed\n\n3. Dependencies:\n- Reliance on external planner\n- No end-to-end training with downstream tasks\n- Limited standalone capability as a GUI agent", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QB36SPxjNE", "reviewer": "Reviewer_bKGa", "summary": "This paper proposed a visual-based GUI Agent framework (SeeAct-V) with a GPT-aid planner and a visual grounding model (UGround). The planner first generates a textual plan, and the visual grounding model produces the coordinate of the target element. The paper constructs a training dataset (Web-Hybird) to train the grounding model. SeeAct-V is evaluated on 3 settings (visual grounding, offline and online agent evaluation).  The results validate UGround’s efficacy as a universal grounding model for GUI agents.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper propose a visual-based GUI Agent to avoid the limitations of language-based approaches. A large-scale dataset is collected through a carefully designed data collection method and used to train the visual grounding model. The paper is well-structured, clearly articulated, and demonstrates solid effectiveness in the proposed method.", "weaknesses": "Limitations in Completeness\n\n1. This paper compares *UGround* with other models in Table 2 and shows UGround’s universal grounding capability. However, these methods differ in both their model settings and the training data used. An ablation study is missing to clarify the contributions of the model design (specifically the image resolution setting) and the training data to UGround's performance.\n\n2. Same issue as in 1. This paper proposes 3 types of REs for GUI elements. An ablation study is missing to clarify the contribution of  each type of RE.\n\n3 Related work on current GUI agents is missing from the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aGzIwp6oK6", "reviewer": "Reviewer_Xm5m", "summary": "The paper presents a novel approach to enhancing GUI agents by adopting a human-like embodiment that solely relies on visual perception and various operations. It addresses the limitations of current GUI agents that depend on text-based representations (HTML, accessibility trees, etc.), which often introduce noise and computational overhead. The authors propose UGround, a visual grounding model trained on a large dataset of GUI elements (19M GUI elements across 1.3M screenshots). Their framework, SeeAct-V, allows GUI agents to perceive and interact with their environment visually, improving grounding accuracy and agent performance. The empirical evaluation across six benchmarks shows that UGround outperforms state-of-the-art models, offering a promising solution for GUI agents to function more like humans in digital environments.", "questions": "See Weakness above. \n\nI think the authors can include a section to discuss copyright problems.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think this paper addresses well on the bottleneck of MLLM-based GUI Agents, UI Grounding, with a newly constructed synthetic dataset, which is timely and important.\n\n- **Human-Like Embodiment for GUI Agents**: The paper makes a compelling argument for GUI agents that perceive their environment visually, which aligns better with how humans interact with GUIs.\n- **Extensive Grounding Dataset**: UGround is trained on a comprehensive dataset of 19 million GUI elements, making it the largest dataset for GUI visual grounding.\n- **Significant Performance Gains**: The empirical results demonstrate that UGround improves grounding accuracy by up to 20%, and agents using it outperform models that rely on both visual and text-based inputs.\n- **Cross-Platform Generalization**: UGround shows strong performance across different platforms (desktop, web, mobile), which demonstrates its potential as a universal solution.\n- Comprehensive Evaluation: The authors evaluate UGround on six benchmarks, which span grounding tasks and online/offline agent evaluations, providing a thorough analysis of the model's capabilities.", "weaknesses": "- **LLM Usage in Synthesizing Data**: While the dataset is large, much of it is synthetically generated using LLMs. This raises concerns about potential hallucinations during data synthesis. The authors should consider sampling a subset of the data for human evaluation to verify the accuracy and goal alignment of the generated content, especially when leveraging models like LLaVA-Next-13B and GPT-4o for generating and refining referring expressions (REs), as well as LLaMA-3-8B in polishing. Here is a potential experiment setup suggested by GPT's feedback: `Randomly sample 1000 generated referring expressions and have human annotators rate their accuracy and relevance.` I think this advice is acceptable.\n- **Dataset Analysis**: A deeper analysis of the dataset's diversity, as mentioned in line 169, would strengthen the work. Techniques such as PCA could provide insights into the data distribution. Additionally, more information is needed on how well the instructions used for planning in GUI agents align with the dataset used for training the grounding model. A breakdown of the types of GUI elements represented in the dataset, or t-SNE plots to visualize the distribution of referring expressions is suggested.\n- **Typos**: There is a citation typo in line 307 related to CogAgent, which should be corrected.\n- **Copyright Concerns**: The paper uses webpage data crawled from Common Crawl. It would be helpful for the authors to address any potential copyright issues associated with using this data. The authors are suggested to specifically discuss their data usage policy and any steps they've taken to ensure compliance with copyright laws when using Common Crawl data.\n- **Environment Limitation**: The grounding dataset was collected entirely in a web environment, which represents only a subset of GUIs. The authors have discussed this limitation and I think it is not a big problem given that GUI elements can transfer smoothly. However, it's still worth emphasizing that this could impact the model’s performance in other GUI environments. Include a small-scale experiment or case study demonstrating the model's performance on a non-web GUI task is suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)