Grounding Multimodal Large Language Model in GUI World
Authors: Weixian Lei
Difei Gao
Mike Zheng Shou
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
GUI Element Grounding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have accelerated the development of Graphical User Interface (GUI) agents capable of automating complex tasks across digital platforms. However, precise GUI element grounding remains a key challenge for accurate interaction and generalization. In this work, we present an effective GUI grounding framework, which includes an automated data collection engine that gathers extensive GUI screenshots and annotations to ensure broad generalization. We also propose a lightweight and flexible GUI grounding module designed to efficiently localize UI elements by pre-training on the collected data, and introduce a novel method to integrate this module with MLLMs for the effective execution of GUI tasks. Our approach demonstrates superior performance in task accuracy and adaptability, as validated by benchmarks such as ScreenSpot, MiniWob, AITW, and Mind2Web.

Published as a conference paper at ICLR 2025
GROUNDING MULTIMODAL LARGE LANGUAGE MODEL IN GUI WORLD
Weixian Lei, Difei Gao, Mike Zheng Shou Show Lab, National University of Singapore
ABSTRACT
Recent advancements in Multimodal Large Language Models (MLLMs) have accelerated the development of Graphical User Interface (GUT) agents capable of automating complex tasks across digital platforms. However, precise GUI element grounding remains a key challenge for accurate interaction and generalization. this work, we present an effective GUI grounding framework, which includes automated data collection engine that gathers extensive GUT screenshots and annotations to ensure broad generalization. We also propose a lightweight and flexible GUI grounding module designed to efficiently localize UI elements by pre- training on the collected data, and introduce a novel method to integrate this module with MLLMs for the effective execution of GUI tasks. Our approach demonstrates superior performance in task accuracy and adaptability, as validated by benchmarks such ScreenSpot, MiniWob, AITW, and Mind2Web. Our code and data are released at https: : /ttpttttttttt///////////tt.thttt com/ wwwwshhwwwhooo ab Assi ott.e/////ssssttttGrooootoo GUIGGnud
INTRODUCTION
A) Automated GUT Grounding Data C Collection
Data Collection Engine
Technology
Nows. & journalism
Eduration
Sports
Frnd Panking
lissh & Wellnrss
118 D Domains, 0.5M Screenshols, 35M UT Element Annolations
B) GUI Grounding Model
C) Integrated with MLLM Agent
Ansotalion:
Query:
Screenshat:
Ull Element Nome, Bounding Box, Query
Locate the figght option
MLLM
GUX Grounding Mode!
Row action: Click 4 -|PO5
User: flight
(0.06,0.15.0122.2200)
Action: (0.05.0.175)
Figure I: Overview our pipeline for building a vision-based GUI agent. (A) data collection engine gathers diverse set of website screenshots spanning broad range of topics. (B) A GUI grounding model trained on the collected data with dense elements and annotations. (C) The grounding model is integrated with MLLMs to form vision-based agent.
Recent advancements Large Language Models (LLMs) (Achiam et al., 2023; Touvron et al., 2023a;b; Jiang al., 2023; Chiang al., 2023) and Multimodal Large Language Models (MLLMs) (Li et al., 2023; Zhu et al., 2023; Liu et al., 2024; Bavishi et al., 2023; Hong et al., 2023) have significantly improved multimedia comprehension, logical reasoning, and decision- making capabilities, driving the evolution of Graphical User Interface (GUI) agents. These agents are designed to automate complex tasks on digital platforms such as PCs and mobile devices, reducing human effort in performing monotonous duties (Cheng et al., 2024; Gao et al., 2024; Shi et al., 2017; Zhou al., 2023; Koh al., 2024; Deng et al., 2023; Chen al., 2024a;b; Lu al., 2024a; Yan et al., 2023; Wang et al., 2024b; Yang et al., 2023b; Shaw et al., 2023; Lin et al., 2024).
Corresponding Author.

---

Published as a conference paper at ICLR 2025
Some pioneering works (Deng et al., 2023; Gur et al., 2023; Zheng et al., 2023; Kim et al., 2024) have designed text-only agents that feed structured plain text representations of sereen content, such as HTML and DOM, to LLMs for reasoning, planning, and executing specific GUI actions. However, plain text representations of GUIs have inherent limitations. First, structured text is not always readily available, restricting its use. Second, raw text metadata-ofter extensive and noisrerequires substantial preprocessing and filtering, which increases computational latency and inference costs and may compromise model accuracy.
Humans interact with digital interfaces by visually perceiving content, making decisions, and ex- ecuting actions accordingly. Drawing inspiration from this natural interaction with GUIs, another approach adopted by GUI agents is to use visual renderings as inputs. This approach circumvents the limitations associated with text metadata processing and is better suited for generalization across various platforms. Several pretrained MILMs (Bavishi et al., 2023; Hong et al., 2023) included UI data during pre-training and have demonstrated their ability to understand basic elements within UI screenshots. A recent work, SeeClick (Cheng et al., 2024), trained visual GUI agent using screenshots from web and mobile interfaces, enhancing vision-based capabilities in GUI tasks. De- spite these advancements, their performance in GUI automation tasks still falls short compared to text-based models.
We identify primary bottleneck is GUI element grounding, which involves aligning the LLM- generated textual plan with precise screen locations. Robust GUI agents require strong grounding capabilities, characterized by: 1) Precision in Targeting and Interaction: The agent must accurately identify and interact with the necessary elements at each step of a task, ensuring the reliable execution of complex tasks. 2) Generalization Across GUI Scenarios: The agent should effectively generalize across diverse real-world scenarios, such as online shopping, entertainment, and software- based office tasks. 3) Flexible Grounding Adaptation: The grounding model should seamlessly integrate with any LLM planner, enabling easy invocation and adaptation for planning and decision -aasin...
The main challenge of building such a system is developing this capability the absence of large-scale annotations available on the internet, similar to relevant topic, segmentation (Kirillov et al., 2023a). Therefore, carefully tailored approach to data curation and methodical model training is essential. From the data curation, the grounding model should be trained on comprehensive and diverse set of GUI screenshots and annotations to ensure strong generalization to various GUI applications. Currently, public datasets lack dense, diverse, and richly annotated GUI grounding data. To resolve this, we have created an automated GUI data collection engine to gather relevant screenshots and grounding annotations, as shown in Figure. (A). We also perform data cleaning to maintain high quality and augment query expression for different GUT elements to enhance generalizability.
From model perspective, our goal is to develop lightweight yet powerful grounding model and seamlessly integrate it into an MLLM to perform complex GUI tasks, as shown in Figure. 1(B) & (C). Therefore, we first design a GUI Grounding model with a robust image encoder for generating embeddings and a language encoder for processing queries, combined in a lightweight decoder to predict bounding boxes. This well-trained model is then integrated with the MLLM for downstream tasks like navigation. We add < POSI > token to the MLLM's vocabulary, which triggers grounding when needed. Its hidden embedding passed the GUI grounding model to locate the corresponding position for operation. This design separates reasoning and grounding into modular components, utilizing the MLLM for reasoning and the grounding model for precise localization.
We evaluated our approach on the ScreenSpot (Cheng et al., 2024) benchmark, demonstrating the efficacy of our GUI grounding pre-training methods. Additionally, we integrated our model comprehensive GUI agent. This agent has been adapted for various
with MLLMs to function mobile and web-based tasks, including MiniWot (Shi et al., 2017), AITW (Rawles et al., 2024), and Mind2Web (Deng et al., 2023). Our evaluations show significant improvements in task accuracy and adaptability compared to previous methodologies.
RELATED WORK
Datasets for GUI Understanding. Several noteworthy benchmarks have been developed as testbeds for GUI agents, each incorporating test data or GUI environments for evaluating the per- formance of models. Examples include MiniWob (Shi et al., 2017), MiniWob++ (Liu et al., 2018),
2

---

Published as a conference paper at ICLR 2025
Mind2Web (Deng al., 2023), and VisualWebArema (Koh et al., 2024), which provide environments for web-based navigation tasks. Additionally, OS- World (Xie et al., 2024) introduces a simulated environment for system operations, while AssistGUI (Gao et al., 2024) focuses on PC software navigation tasks.
In addition to these benchmarks, various publicly available datasets have contributed to training models for UI understanding. For instance, Widget Captioning (Li et al., 2020) offers a dataset that includes mobile UI elements across 88 tasks. RICO (Deka et al., 2017; Sunkara et al., 2022) provides approximately 66k mobile screenshots with view hierarchies, alongside around 500k human annotations that identify various icons by their shapes and semantics. Moreover, datasets such as Pix2Struct (Lee et al., 2023) and Pix2Act (Shaw et al., 2023) have been employed to train models with screen parsing capabilities, leveraging in-house data. Recently, SeeClick (Cheng et al., 2024) has gathered a set of pretraining data to develop an MLLM-baseed GUT agent.
Nevertheless, these existing datasets are still insufficient in terms of diversity and scale to support the development of robust GUI grounding model (Lu et al., 2024b). To address this limitation, we have designed an automated data collection engine to assemble a dataset for GUI element grounding, characterized by its high diversity and dense annotations.
Multimodal LLM for GUT Understanding. Multimodal Large Language Models (MLLMs) (Liu al., 2024; Zhu al., 2023; OpenAI, 2023b; Abdin et al., 2024; Bavishi et al., 2023) have recently made significant advancements, integrating visual perception and language generation to perform tasks such image captioning (Li et al., 2023; Wang et al., 2020), visual question answering (Hudson & Manning, 2019), and particularly in OCR-free image text comprehension (Luo et al., 2024). The Qwen-VL series (Bai al., 2023b;a) adds grounding capabilities, allowing models to localize image regions based on language input. LISA (Lai et al., 2024) adds segmentation capabilities to MLLMS by introducing an additional segmentation modules MLLMs.
These newly added capabilities enable models to better observe the details of images, which supports tasks in GUI environments. As result, some models (Hong et al., 2023; You et al., 2024; Shaw et al., 2024; Cheng et al., 2024) have been proposed to generate actions in these environments. GUl-based tasks, often requiring the understanding of high-resolution images, are addressed by models like (Cheng et al., 2024; Hong et al., 2023). enabling interaction with GUIs through visual grounding, thus improving accessibility and widget localization.
While existing models exhibit strong object grounding capabilities, accurately localizing GUT ele- ments in screenshots remains challenging due to their typically small size and similar visual features, such as icons. To overcome this, we designed specialized decoder, inspired by LISA (Lai et al., 2024), dedicated to grounding GUI elements. Unlike previous approaches that rely solely on pure Transformer architectures to predict bounding boxes for GUI elements (Cheng et al., 2024; Hong et al., 2023; Bavishi et al., 2023), our method addresses the unique challenges posed by GUI screenshots.
DATA COLLECTION ENGINE
A robust GUI grounding model means strong performance across diverse GUI applications. To achieve this, it is essential collect large, high-quality, and varied dataset. The most effective way to gather such a comprehensive dataset is through web screenshots, given their vast quantity and diversity in layouts and elements, which can be generalized to various application scenarios.
Therefore, we propose a data collection engine that automates the browsing of websites and collects high-quality annotations of GUT elements. The collected data includes screenshots of graphical user interfaces, the corresponding GUI elements with basic details (including names and bounding boxes), and the hierarchical information of these elements. The data we aim collect through this engine is characterized by:
Screens should have comprehensive annotations of elements, including but not limited to types, displayed content, and positional information.
Dense Annotations:
Rich Elements: This includes actionable elements, such as various buttons or controls, as well as static elements like embedded images and static text.
Diverse Domains: The data should cover a broad range of styles and layouts, spanning multiple domains, to ensure versatility and the ability generalize various application scenarios.
3

---

Published as a conference paper at ICLR 2025
Datasct
Task
Platform
Vis, Input
# IInst. Anno.
Anno. type
MiniWob (Shi al., 2017)
Navigation
Web
Image
2.8K
9.8K
traj.. pos.
Mind2Web (Deng al., 2024)
Navigation
Web
(multi.) Image
2.3K
17K
traj.. pos.
AITW (Rawles et al., 2023)
Navigation
Mobile
(multi.) Image
4.6K
23.6K
traj. pos.
AITZ (Zhang et al., 2024)
Navigation
Mobile
(multi.) Image
2.5K
18.6K
Iraj. pos.
Odyssey (Lu et al., 2024s)
Navigation
Mobile
(multi.) Image
7.7K
119K
traj. pos.
GUIWord (Chen et al., 2024a)
Understanding
Diverse
Video
12.4K
98K
cap., QA type, cont., pos.
ScreenSpot (Cheng et al., 2024)
Grounding
Web, Desk, Mobile
Image
1.3K
1.3K
GUIAct (Chen et al., 2024b)
Navigation
Web, Mobile
(multi.) Image
79K
191K
type, cont., pos. pop.type.cot.cclas.pee chat. pos.
worden2words (Wang et al,, 2021)
Understanding Mobile
Image
79K
79K
GUIChat (Chen et al. 2024b)
Understunding Web, Mobile
Imuge
50K
50K
RICO (Deka et al., 2017)
Grounding
Mobile
Image
243K
763K
type, cont. pos. type, cont, pos. type, cont., pos. type, cont, pos.
GUIEnv (Chen et al., 2024b)
Grounding
Web, Mobile
Image
70K
589K
AMEX (Chai et al., 2024)
Grounding
Mobile
Image
97K
8KSK
SceClick (Cheng et al,, 2024)
Grounding
Web
Image
0.3M
3.0M
Ours
Grounding
Web
Image
0.5M
35M
type.cont.c www.conn.clls..s.hhiee.. pos.....
Table I: Statisties for GUI related datasets. The upper upper half datasets of smaller size are used for downstream evaluation. The lower half datasets of larger size be used for pretraining. Understanding type includes chat and/or captioning data. "traj.": trajactory, "pos.": position, "'app" captioning, "QA": quesntion answer, "cont.": displayed content, "class": class type defined by system, "hier": hierarchy info. Among these datasets, our collected data provides the most screenshots and element annotations.
To achieve these goals, our data engine is structured around the following stages:
Topic Selection. We follow previous work (Cheng et al., 2024; Hong et al., 2023), sourcing URLs from the Common Crawl. However, many web pages in Common Crawl only contain plain texts with simple layouts, such pages consisting of only a few lines of text, few links, or simple images, thus offering few interactive elements. This simplicity creates a significant gap when compared to the complex GUI structures in common GUT tasks. To collect large number of interactive elements, we selected 11 topics and identified 1 118 domains related to those topics. These domains were then used to filter URLs from the Common Crawl dataset. The selection of topics and website domains was guided by GPT-4 (Achiam et al., 2023), which proposed diverse set of topics relevant to worldw applications. This domain list then extracts relevant web page instances from the original Common Crawl dataset. In the Appendix, we display the selected topics and their associated domains.
GUI Data Collection. Given the URLs for the selected websites, we built a system that automatically loads these web pages in web browser, such as Chrome. Since web pages are often lengthy, our system simulates user interaction by pressing the "Page Down" key to scroll through the entire content until the bottom of the page is reached. During this process, we capture screenshots of the web pages along with corresponding element data. The element data, which includes attributes, class names, display contents, and bounding boxes, is obtained using UI Automation (UIA). This approach allows us to comprehensively capture both the visual and structural aspects of the user interface, ensuring that we accurately document and analyze the interactive elements present on each webpage.
Data Cleaning. The goal of the data cleaning stage is to ensure the quality and usability of the collected data. Following the gathering of screenshots and element data from various websites, we implement several critical cleaning steps to filter out unusable or irrelevant data, resulting in high-quality dataset. First, we eliminate websites that are either empty or inaccessible as many URLS may lead to pages with no meaningful content, broken links, or require authentication that cannot be handled automatically. This step is crucial for maintaining the reliability of our dataset. Next, we clean the element data by removing entries with erroneous or missing element names. We also address cases where the element hierarchy is ambiguous or misrepresented by restructuring it to accurately reflect the parent-child relationships between GUI elements. By performing these steps, we refine the dataset to ensure it contains accurate, meaningful, and well-structured data, which is essential for further processing and the effective training of robust GUI grounding models.
Augmenting GUI Grounding Queries. To ensure the model's generalization capabilities, it must adapt various query types and handle range of tasks such element identification, grounding, and referring expression. However, the metadata collected primarily provides basic information like class names and display contents, which are insufficient for generating diverse queries.
We address this by generating instructions through a set of defined templates that guide the model to focus on specific aspects of the GUI data. For example, a button labeled "go back" could be
4

---

Published as a conference paper at ICLR 2025
augmented to query like "Where is the button to go back to the previous page?" These templates. created using GPT-4 after categorizing the collected elements, enable us to generate a diverse set of queries that encompass wide range of interactions and contexts. Some examples of constructed templates are shown in Appendix. This augmentation process ensures that the model can effectively manage various tasks, improving its ability generalize across different application scenarios and enhancing its practical usability and robustness.
Data Overview. In total, we collected 0.5M screenshots with 35M UI element annotations. Our collected data covers a broad range of topics and features rich elements with dense annotations. We provide a comparison of our collected dataset with previous GUI data in Table. 1.
MODEL ARCHITECTURE
Within our scope, the GUT grounding model should exhibit generalized capabilities and easily adaptable for seamless integration with current MILMs, requiring minimal modifications and ensuring minimal performance loss.
We propose AssistGUGrrrrder (Agg), a GUI grounding model designed with these objectives in mind. For the GUI grounding model itself, takes a GUI image and textual query about the target element as inputs and outputs the position of the target element corresponding to the query. In terms of integrating the GUI grounding model with an MLLM, it should excel in vision-based GUI agent tasks. The inputs include GUI image representing the current status, a text description of the user goal, and optionally, a history of past actions. The MILM,, acting as a "planner", generates the representation of the action and target element for the next step. Meanwhile, the GUI grounding model extracts visual features from the input image and combines them with the decision information from the MILM, thus outputting the position of the target element when required.
In the following sections, we will outline the model architecture of the GUI grounding model and discuss how it can be effectively integrated with MILMs.
4.1 GUI GROUNDING MODEL
GUI Screenshot
Image Embedding
Decoder
Updatal Emhed.
Score)
Ing-to-Quaery
Image Encoder
Query: wmg.://maaaooom
1x2
Self-Attn
+
Queey Tokeas
Language Prompt Encoder
Grounding Query:
Locute "Search destinations" item
Figure 2: Overview of GUT Grounding model. The model consists of three main components: an image encoder, language prompt encoder, and lightweight decoder. The image encoder extracts visual features from the input GUI image, while the language prompt encoder transforms the textual query into query embeddings. The lightweight decoder then combines these visual features and query embeddings to predict the position of the target GUI element.
Our GUI grounding model AGG comprises an image encoder I, language prompt encoder L, and lightweight decoder D, as illustrated in Figure. 2. As overview, the image encoder extracts the visual features of a GUT image Ximg- while the language prompt encoder learns to represent the textual query X xuuery- The lightweight decoder then fuses these visual features and the textual query representation, learning to predict the position of the target GUI element, denoted as b. This can be formulated as: D (I(gg))). L L(Xquer)). Below is a detailed description these components:
Image Encoder. We leverage the scalable pretraining benefits by employing the SAM (Kirillov et al., 2023b) pretrained Vision Transformer (ViT) (Dosovitskiy et al., 2021). This choice is made due to its demonstrated effectiveness in promptable point-to-mask segmentation tasks, showcasing its strong
5

---

Published as a conference paper at ICLR 2025
visual perception capabilities. For handling high-resolution GUI inputs, utilize windowed attention along with four equally spaced global attention blocks, following the methodology in (Kirillov et al., 2023b; Li et al., 2022).
Language Prompt Encoder. The language encoder is applied to maps text queries into query embeddings. Specifically, we employ language transformer model (Devlin et al., 2019) as the prompt encoder. We prepend <CLS> to each text query and use its output hidden states from the final transformer layer as the query embedding.
Lightweight decoder. The decoder's role is to process the embeddings from both the image encoder and the language prompt encoder, mapping them the output bounding boxes which indicate the precise locations of GUI elements. Our decoder architecture is inspired by the design principles outlined in SAM (Kirillov al., 2023b). Before inputting the embeddings into the decoder, we introduce a set of learnable embeddings and concatenate them with the query embeddings from the prompt encoder. These combined embeddings are collectively referred to as "query tokens".
The decoder comprises two layers, each performing a series of operations: (1) Bi-directional self- attention on the query tokens, allowing the model to convey information within the tokens; (2) Cross-attention from query tokens (as query) to the image embedding: (3) An MLP updates the query tokens; (4) Cross-attention from the image embedding (as query) to the query tokens, which updates the image embeddings with contextual information derived from the queries. The next decoder layer then takes the updated query tokens and the updated image embeddings from the previous layer. Finally, the output of the query tokens from the last decoder layer is used for grounding prediction.
Training for Grounding. For the GUI image input, we follow (Kirillov et al., 2023b) to rescale the input image to 1024 x 1024, by adjusting the long side while maintaining the aspect ra- tio and padding the short side. For the textual query input, to enhance the generalizability of the grounding model, we augment the text queries by sampling templates from predefined set for different elements. For instance, the metadata from the data collection engine might only provide structured information such as { ("class__ _amee""a"""""" ; "button" "content": "go back", "level- -inee- : ("navigat on bar", "go back"), of merely concatenating this information into "button: go back", we generate more diverse queries like "Seek the go back button and click", or "Help me find the go back button in the navigation bar".
} for a button. Instead
Following object detection methods (Carion al., 2020), we add prediction heads to the decoder outputs for bounding box prediction. Each output from the learnable embeddings predicts both score and bounding box coordinates. The predicted score is optimized to match the IoU of the predicted and ground-truth boxes using mean-square-eris linear combination of 4 loss Li, and the generalized IoU loss LGIU (Rezatofighi et al., 2019). The overall training loss for GUI grounding is given by: L = XuseeLuse Xx, Lt, + λutau Lawau, where Amse, >o, and ActoU are hyper-parameters, parameters. Unlike (Kirillov et al., 2023b), which backpropagates only from the lowest loss, we found that backpropagating from all predictions improves performance in GUI grounding. During inference, bounding boxes are ranked by score, and the top-1 box is selected as the final prediction.
loss Lmse. For bounding box regression, we use
4.2 INTEGRATING GROUNDING MODEL WITH MLLM AS AGENT
For vision-based agents, completing a complex GUI task typically involves several steps, each requiring the model to: (1) Perceive and understand: gather and interpret information from GUI images, user goals, and action history to understand the current state. (2) Reason and plan: analyze the current state to decide on the next action. (3) Execute: perform the desired action on the target GUI element. In this work, we propose the simple yet effective embeddings-as-ponn aspossiioo paradigm to integrate the GUT grounding model's capabilities into an MILMM, thereby creating a vision-based agent tailored for these tasks. We detail the design below.
Architecture for the integrated agent. The proposed agent consists of an MLLM M and a well- trained GUI grounding model, as is shown in Figure. 3. Inspired by (Lai et al., 2024), this approach involves several key steps. Initially, we add a new token < PPOS| >> to the original MLLM vocabulary, indicating request for GUI grounding output. Given the textual input Xust- which includes the user goal and optionally, the action history, along with the visual input of the GUI image Ximg- these inputs are fed into the MILM,, which generates text-form response â This process is formulated

---

Published as a conference paper at ICLR 2025
User Goal: want to buy power axapter for
Viszalization of Granding Resullse
Gripet:
iPhone. Can help
History: Step Click (797.0, 54.0).
GUT Scrvershot: <image of S,27
Multimodal
>P008>
LLM
So
Tape
No groundig is reqindd byjng oction
S. Ouuppe
"'eess "oles "outsis
PGS
Image
Encoder
Decoder
The proposed agent integrates an MILM with trained GUI grounding model. MILM processes textual inputs (user goals, action history) and visual inputs (GUI screenshot) to generate a response. A special token triggers GUI grounding. The image encoder extracts visual features, then the lightweight decoder predicts the target GUI element's position. The figure illustrates the process of task completion and highlights the step 2.
Figure 3: Integrate AGG with MILM as a GUT agent.
as ousr = M (img: Xusr). When the MLLM aims to generate a bounding box, the text-form output includes < IPOS| token. We then extract the last layer embedding corresponding to the < PPSS >> token and apply a linear projection layer to it. We denote this output as h Meanwhile, the image encoder from the grounding model extracts visual features from the input GUI image. Finally, these visual features and the projected embedding are fed into the lightweight decoder of the grounding model predict the position P of the desired GUI element, formulated P = D (Aposs I ((Xing)).
Training for agent tasks. The model is trained using the text token prediction loss Liex and the grounding loss Lgrd. The overall objective for training the vision-based agent is: L >eee AgrdCrrd. where Atext and Lgrd are hype-parameters. Especially, when the form of the target position a bounding box, Lyd is the same as that used in GUI grounding training. When the form of the target position is point, we slightly modify the prediction head upon the decoder output to predict the normalized distance the ground truth point as score indicator, and predict the 2D point coordinates as the position output.
5 EXPERIMENT
5.1 EXPERIMENTAL SETUP
Training data for GUT Grounding. We train our GUI grounding model on two datasets: web UT data collected by engine and mobile UI data reorganized from public sources. For web UI. applied a data smoothing strategy (Hudson & Manning, 2019), resulting in 0.36M images with 22M element annotations. For mobile UI, following (Cheng et al., 2024), we used 27K screenshots and 275K query-bbox annotations from the WidgetCaption (Li et al., 2020) and RICO (Deka et al., 2017).
Training for Agent-based Tasks. We integrate AGG with MLLM using the embeddings -as- position approach. We verify its efficacy on agent-based tasks, including MiniWob (Shi et al., 2017) 7), ARTW (Rawles et al., 2023), and Mind2Web (Deng et al., 2024). Following (Cheng et al., 2024), pretrain Qwen-VI. VL. (Bai et al., 2023b) using our dataset combined with a mixed public dataset to build an MLLM for GUI tasks. We then combine it with our trained grounding model to finetune on each agent-based task. Instructions and previous action memory are provided during both training and inference, as described in (Cheng al., 2024).
5.2 EXPERIMENTS FOR GROUNDING AND AGENT TASKS
ScreenSpot. ScreenSpot is benchmark for GUI grounding. It comprises over 600 interface screenshots and more than 1200 annotations for query-element pairs. It spans mobile platforms (IOS, Android), desktop platforms (macOS, Windows), and web platforms.
For this benchmark, we compare our GUI grounding model with generalist MLLMs capable of recognition and reasoning (Chen et al., 2023; Bai et al., 2023b; Yan et al., 2023; Abdin et al., 2024), as well as recent MLLMs that have incorporated GUI-related tasks during pretraining (Bavishi et al., 2023; Hong et al., 2023; Cheng et al., 2024).
7

---

Published as a conference paper at ICLR 2025
Results. The results for grounding, shown in Table. 2, highlight that while generalist MLLMs possess extensive knowledge about natural images, their GUI grounding performance on ScreenSpot is subpar due the significant differences between GUIs and natural images. Even GPT-4V struggles to accurately locate screen elements. GUI-specific MILMM demonstrate better performance. In particular, Our AGG achieves the highest performance across different platforms and for both text and icon elements, even with fewer model parameters. This underscores the advantages of our collected data for GUI grounding training, allowing AGG to effectively handle diverse queries and rich elements encountered during training. With rich elements and diverse query seen during training, AGG possess the capability for GUI grounding, even for the very less scenarios like iOS. Compared to text element grounding, the model's performance is less strong on icon-based tasks, indicating that grounding non-text elements in GUIs remains challenging.
Mobile Icon
Desktop Icon
Weh
Method
GULS
Model size
Avg.
Text
Text
Text
Icon
MiniGPTv2 (Chen al., 2023)
X
7B
8,4
6.6
6.2
2,9
6.5
5.7
Qwen-VL VL (Bai et al., 2023b)
X
9.6B
9.5
4.8
5.7
5.0
3.5
2.4
5.2
Phi-3,5-V (Ahdin et al., 2024)
X
4.2B
1.7
1.7
4.1
0.7
4.7
2.0
2.5
GPT-4V OpenAl (2023a)
X
22.6
24.5
20.2
11.8
9.2
16.2
Fuyu (Bavishi et al., 2023)
8B
41.0
1.3
33.0
3.6
33.9
4.4
19.5
CogAgent (Hong et al., 2023)
18B
67.0
24.0
74.2
20.0
70.4
28.6
47.4
SeeClick (Cheng al., 2024)
9.6B
78.0
$2.0
72.2
30.0
55.7
32.5
53.4
AGG (Ours)
0.4B
86.1
62.8
81.8
46.2
74.2
48.4
66.6
Table 2: Performance of various models on SereenSpot. "GULS" indicates "GUT Specific", denoting whether the model was trained using domain-specific data.
MiniWob. MiniWob (Shi et al., 2017) features 2000 open-ended tasks from 137 real web environments, each with high-level instructions and action trajectories. These tasks require the agent to navigate simplified web environment and execute numan-given instructions.
Following (Cheng al., 2024), we conduct 2.8K episode rollouts for training. We compared our method with models using image-only inputs, text-only inputs, and combination of both. Note that with text imput from the environment, the model selects the HTML element from candidates as the action target. We follow (Cheng et al., 2024) and report performance on two groups of data splits. We compute the success rate over 50 random seeds for each task and then calculate the mean success rate across all tasks the final score.
Method
Modality
Datasel Score
Method
Modality Dataset Score
(C-Net(SL) (Humplreys al., 2022)
DOM+Image
2.4M
36.5
CC-Net(SL.) (Humphreys al., 2022)
Image
2.4M
23.4
T5 (Gur al., 2022)
HTMI.
12K
55.2
Pix2Act (Shaw et 2024)
Image
12K
55.2
WebGUM (Puruta et al., 2024)
WebGUM (Furuta al., 2024)
HIMLe+mage
HTML+Image
347K
288K
65.5
86.1
Qwen-VL. (Bai al., 2023b)
Image
288K
48.4
SeeClick (Cheng et al., 2024)
Image
2.8K
73.6
SeeClick (Cheng al., 2024)
Image
2.8K
67.0
MLLM AGG (Ours)
Image
2.8K
75.8
MLLM o AGG (Ours)
Image
2.8K
69.3
Table 3: Results on MiniWob 45-tasks split.
Table 4: Results on MiniWob 35 tasks split.
Results. We present results in Table. 3 and Table. 4. We denote our method as "MLLM AGG". The results show that our episodes, it outperforms WebGUM, which uses both HTML and GUI images as input and exceeds the vision-based Pix2Act, which used less than 0.3% as much training data. Furthermore, our method outperforms MILMS like Qwen- VI. and SeeClick, demonstrating the efficacy of our framework.
agent surpasses previous methods. Notably, with 2.8K training
AITW. Android In The Wild (AITW) (Rawles et al., 2023) dataset is a mobile navigation benchmark including 30K instructions and 715K corresponding operation trajectories. Following the approach detailed in (Cheng et al., 2024), we apply the same train/test split based on instructions, retaining single trajectory per instruction and ensuring no overlap between the training and test sets.
We compare our method to API-based LLMs and previous MLLM-based methods. We use the screen-wind wrreen-wise action matching score as the main metric, as outlined in (Rawles et al., 2023), and follow (Cheng et al., 2024) to compute click accuracy to evaluate grounding capabilities.
Results. As illustrated in Table. 5, our method achieved the best performance across all tasks, surpassing both the API-based LLMs and the previous GUI data pretrained MLLMs. Notably, our method exhibits higher click accuracy than other methods, demonstrating stronger GUI grounding capabilities.
8

---

Published as a conference paper at ICLR 2025
Method
Modality General Install GoogleApps Single WchShopping Overall ClickAcc
CharGPT-CoT (Zhang Zhang, 2024)
Text
5.9
4.4
10.5
9.4
8.4
PaLM2-CoT (Rawles al., 2023)
Text
39.6
GPT-4V (Yan al., 2023)
Image
41.7
42.6
49.8
72.8
45.7
50.5
Qwen- VL et (2023b)
Image
49.5
59.9
46.9
64.7
50.7
57.3
SeeClick Cheng et (2024)
Image
54.0
66.4
54.9
63.5
57.6
59.3
66.4
MLLM AGG (Ours)
Image
56.4
67.8
56.6
67.2
63.1
62.2
68.1
Table 5: Performance of different models on AITW. The upper half corresponds to the performance of API-based LLMs.
Cross- Task
Cross- Website
Method
PT
Fle.Ace
Op.FI
Step.SR Fle.Acc
Op.FI
Step.SR Fle.Ace
Op.FI Step.SR
MindAct (Gen) (Deng et al,, 2024)
20.2
52.0
17.5
13.9
44.7
11.0
14.2
44.7
11.9
MindAct (Deng et al., 2024)
55.1
75.7
52.0
42.0
65.2
38.9
42.1
66.5
39.6
GPT-3.5-Turbo
20.3
56.6
17.4
19.3
48.8
16.2
21.6
52.8
18.6
GPT-4 (OpenAI, 2023a)
41.6
60.6
36.2
35.8
51.1
30.1
37.1
46.5
26.4
SoM= (Yang et al., 2023a)
29.4
67.5
22.6
19.4
54.2
20.7
23.8
60.3
24.2
SccAct(Anno)* (Zheng et al., 2024)
20.3
13.9
23.7
SeeAct(Choice)) (Zheng et al., 2024)
x
46.4
73.4
40.2
38.0
67.8
32.4
42.4
69.3
36.8
Qwen-VI. (Bai (Bai et
al., 2023a)
15.9
86.7
13.3
13.2
83.5
9.2
14.1
84.3
12.0
SeeClick (Cheng et al., 2024)
28.3
87.0
25.5
21.4
80.6
16.4
23.2
84.8
20.8
MLLM AGG (Ours)
87.4
36.8
84.8
33.8
85.2
34.2
Table 6: Performance of different models on Mind2Web. "Ele.Ace" denotes element accuracy, "Op.F1" denotes operation Fl score and "Step.SR" denotes step success rate. "PT" indicates if
the MILM is pretrained on GUI data. "SoM and SeeAct(Anno)) are tested on partial test set. (SeeAct(Choice) uses an image and text candidates from a ranking LM as inputs.
Mind2Web. Mind2Web (Deng et al., 2024) is benchmark that evaluates an agent's capabilities in web navigation tasks. It encompasses over 2000 open-ended tasks sourced from 137 real website URLs, each accompanied by high-level instructions and corresponding human action trajectories.
In this work, we follow (Cheng et al., 2024) to use parsed screenshots and target element bounding boxes from the raw HTML data Mind2Web. We compare our approach with both text-based web agents (Deng et al., 2024; Op 2023a) and vision- based agents (Bai et al., 2023b; Cheng et al., 2024; Yang et al., 2023a; Zheng et al., 2024). Mind2Act empoyAttmploys two-step process: a small language model first generates candidate elements from raw HTML, followed by large language model that selects the target via multiple-choice QA. Mind2Act (gen) directly generates the target element. GPT-3.5 and GPT-4 use the same multiple-choice QA formulation and include three demonstrations for in-context learning. SoM overlays label indices for element proposals on the input image and prompts GPT-4V to select the matching label. SeeAct (Anno) overlays labels at ground truth positions of candidate elements predicted by ranking language model. For SeeAct et (Choice), GPT-4V selects answer from candidate text choices predicted by a ranking language model. Qwen- VI., SeeClick and our model are fine-tuned using trajectory annotations.
Following the evaluation metrics from (Cheng et al., 2024; Deng et al., 2024), we measure element accuracy, operation F1, and step success rate (StepSR). For vision-based models, a prediction is considered correct the predicted point (or the center of the predicted bounding box) falls within the target element's 's ground-trun -ruth bounding box.
Results. In Table. 6, the results on Mind2Web demonstrate that pretraining on GUI grounding data significantly enhances agent performance. Our model outperforms SoM and SeeAct (Anno). both of which struggle with accurately proposing element regions or associating GPT-4V's action output with the correct label (Zheng et al., 2024). Our method also surpasses the previous hate-oofthhe-a vision-based agent, SeeClick, by a large margin in terms element accuracy, underscoring the robust grounding capabilities of our model. However, our model still falls short of the nop-performing involved SeeAct (Choice), as grounding
text-based GUI agents, and the fext-candidate-llmeee elements in images is inherently more complex than selecting element from multiple choices. This highlights the challenges associated with GUI element grounding and suggests that there remains substantial room for improvement in deploying vision-based GUT agents for real-world applications.

---

Published as a conference paper at ICLR 2025
5.3 ABLATION STUDY
In this section, we conduct ablation studies to analyze the effectiveness of various design elements in the grounding model and vision-based GUI agent.
Model Var
avg. cle.acc
Training config
avg.
prompt emb.
avg.
Owen-V VL
14.4
#1. Random init, Full
41.1
1
46.2
Qwen-VL VL o AGG
27.4
#2. SAM init, Lora
46.8
2
48.4
Qwen- Owen-VL(SC data)
24.3
#3. prop min
47.3
3 49.2
Qwen- VL(SC data) AGG
35.2
#4. prop
48.6
4 49.0
Owen-VL(our data)
30.7
#5. w/o aug
44.5
5
49.8
Qwen- (o(uur data) AGG
39.8
#6. defaud
49.2
Table 9: Number of
Table 7: Effective factors for agent tasks: different data to pretrain the MLLM and whether incorporate AGG. The default set- ting is marked with a color box
Table 8: Training configura- tion for AGG training. De- fault setting is marked with color box
prompt embedding for decoder. Default set- ting is marked with color box
Effective factors for improving agent tasks. We assess the impact of our collected data and AGG. Specifically, we evaluate the Qwen-VI VL (Bai et al., 2023b) MLLM without GUI data pretraining, trained with SeeClick's pretraining data, and trained with our pretraining data. We then integrate AGG to these model variants. The average element accuracy Mind2Web (Deng et al., 2024) is reported. Results in Table. 7 indicate that using our collected data during pretraining results in an MLLM with superior GUI grounding capabilities. Additionally, integrating AGG leads to further improvements, demonstrating the effectiveness of our proposed paradigm.
Next, investigate the key factors influencing the effectiveness of GUI grounding training. We use the ViT-SAM -Base configuration for the grounding model and fixed seed to randomly sample 25% of the collected GUI images and their corresponding annotations for training. We report the average grounding accuracy on ScreenSpot.
Training Configurations for the Image Encoder. We examine the impact of using SAM model weights for initialization and different tuning strategies for the image encoder during training. The results in Table. 8 (#1.2.6) demonstrate that initializing the image encoder with SAM model weights and fully tuning it during training yields the best performance.
Grounding Loss Propagation During Training. We examine the impact of loss propagation configurations for GUI grounding training. The results shown in Table. 8 (#3,4,6) indicate that propagating the loss from all prompt embeddings yields the best performance for GUI grounding.
Effectiveness of augmenting prompt queries. We investigate the effect of augmenting the query from metadata during GUI grounding training. Results in Table. (#5,6) show that augmenting query significantly enhance the GUI grounding capability, which is important to build a generalized basedn GUI agent.
Number of Learnable Prompt Embeddings for Decoder. We investigate the impact of the number of learnable prompt embeddings used the decoder. Table. shows that increasing the number of prompts beyond? 3 yields no significant improvement in grounding performance.
6 CONCLUSION
In this work, we investigated the critical challenge of precise GUI element grounding, which is essen- tial for accurate interaction and generalization GUI agents. We developed simple yet effective GUI grounding framework that includes an automated data collection engine to gather extensive GUI screenshots and annotations, ensuring broad generalization. We introduced lightweight GUI grounding module designed for efficient UI element localization by pre-training the collected data. By integrating this module with MLLMs, we demonstrated superior performance in task accuracy and adaptability across multiple benchmarks, including ScreenSpot, MiniWob, AITW, and Mind2Web. Our results highlight the effectiveness of our approach in enhancing the capabilities of GUI agents for complex tasks digital platforms.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This research involves the use of a training dataset collected from publicly available web data for the purpose of training GUI-based AI models. The dataset used in this paper was sourced from Common Crawl, which provides data under specific term of use. According to these terms, users are permitted copy, modify, distribute, and use the data. Our use of the data transformative in nature, focusing on creating novel Al models for GUI understanding.
We have ensured that the data utilization complies with all applicable laws, ordinances, rules, and regulations. We have processed and filtered the collected data to ensure it contains no sensitive or personally identifiable information. By adhering to Common Crawl's terms of use, we confirm that our data usage is legally permissible and meets the ethical standards set forth by both Common Crawl and the research community.
Additionally, we acknowledge that we are solely responsible for any legal violations concerning the collected data and accept all associated risks. We have implemented measures to ensure that the data collection process respects privacy, copyright, and other legal constraints.
Furthermore, any distribution of the data will adhere to relevant regulations and guidelines, ensuring the protection of privacy and security, and compliance with Al ethics standards.
ACKNOWLEEDGEMENTS
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG3-RP-2022--300) The authors would like to thank Kevin Qinghong Lin for his valuable discussions and insightful feedback on this project, and Peiran Li, Xiangwu Guo, and Qinchen Wu for their assistance in data collection.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint 2rXiv:2404.14211 2024. 3, 8
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.0877: 2023. 1,4
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a. 3, 9
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023b. 3, 7, 8, 10
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL. httpps: //www//wwww/./////////////.// wwwpt//wwww......./e//////// a blog/ Bllgg/fyy-B88 fuyu-8b. 8b. 1, 3. 8
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android preprint arXiv:2407. 24t...4777..99 2024. 4
expo dataset for mobile gui agents. arXiv
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. Gui-world: dataset for gui-oriented multimodal -lm-based agents. arXiv preprint arXiv:2406. 10819, 2024a. 1,4

---

Published as a conference paper at ICLR 2025
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint 1rXiv:2310009478, 2023. 7,8
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406. 11317, 2024b. 1,4
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li Tan Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1313-9332 Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: acl www.cannnhologg org/ 2024 acl- acl-long 505. 1, 2, 3, 7, 7, 8, 9,
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//lmsys. org /tlgg///////////oo///gg///oooo// 2023-03--2--30 30 30-vicuna/.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845-8544 New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781449949819 doi: (0.11445/12259933126666 URL hattps://on //tp///ooiiooo /tt//oioogg....////// 20tg//...11// 3126594. 3126651. 3, 4,
Xiang Deng. Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL https: /taps:///pppeeeeeeee net/ forum?
id=kiigbbOwqw. 1,
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024. 4, 7, 9, 10, 20, 21
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-41888 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/w1/N19- 1423. URL / haa/aaaaaanttoooooooeoe wwwp:ooogg//g11/ 6, 20
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL https: /openreiie.........e net / /tr//oorrmmmii/e ciF.TTyyyTT 5
Hiroki Furuta, Kuang- Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instructionnfiietunee foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL https: // /poenreeview. .nt/ www.reeiew.nnt/ormmee..
8
Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang. Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, and Mike Zheng Shou. Assistgui: Task-oriented pc graphical user interface automation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13289-13298, June 2024. 1,3
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models. arXiv preprint 20Xiv:22 2022.
12

---

Published as a conference paper at ICLR 2025
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint axXiv:2307.12866, 2023. 2
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding. et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914, 2023. 1, 2, 3, 4, 8
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
1xXiv:2106.0968... 2021. 20
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019. 3,7
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. data-driven approach for learning to control computers. In International Conference on Machine Learning, pp.
9466-9482. PMLR, 2022. 8
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.1155.. 2024. 21
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023a. 2
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVV International Conference on Computer Vision, pp. 4015-4026, 2023b. 5, 6, 20
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.
2024. 1,
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589, 2024. 6,6
Kenton Lee, Mandar Joshi, Iulia Raluca Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893-18912. PMLR, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-197422 PMLR, 2023. 3,3
Yang Li. Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In Bonnie Webber, Trevor Cohn, Yulan He. and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495-5510, Online, November 2020. Association for Computational Linguistics. 3,7
13

---

Published as a conference paper at ICLR 2025
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pp. 280-296. Springer, 2022. 6
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Zechen Bai, Weixian Lei. Lijuan Wang. and Mike Zheng Shou. Training ssiionlanguage-action model as generalist GUI agent. In NeurlPS 2024 Workshop Open-Wortee Agents, 2024. URL https: wwoooppeeevieeeeee eew.ww.w.... net/ forum? 1d=0xxxxkkttxx 16
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint I 2018. 2
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1,
I Loshchilov. Decoupled weight decay regularization, arXiv preprint arXiv: 2017. 20
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406. 08451, 2024a. 1,
Yadong Lu, Jianwei Yang. Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.002000 2024b. 3
Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layoutllm: Layout instruction tuning with large language models for document understanding. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15630-15640, 2024.
OpenAI. Gpt-4 technical report, 2023a. 8.9
OpenAL. Gpt-4v(ision) system card, 2023b.
URL https://openaii: /tnee///inn///////
pypt-v--at w------ysstem-aaaaee card/.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088, 2023, 4, 8,
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024. 2
Hamid Rezatofighi, Nathan Tsoi, Jun Yon'o Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lec, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. Advances in Neural Information Processing Systems, 36: 14354-34370, 2023. 1,3
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. Advances in Neural Information Processing Systems, 36, 2024. 8 8
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3135-3144. PMLR, 06-11 Aug 2017. 1, 2, 7, 8
14

---

Published as a conference paper at ICLR 2025
Srinivas Sunkara, Maria Wang. Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Abhanshu Sharma, James Stout, et al. Towards better semantic understanding of mobile interfaces. arXiv preprint TrXxv:2210.02663, 2022. 3
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Au- tomatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 498-510, 2021. 4
Jiacong Wang, Bohong Wu, Haiyong Jiang, Zhou Xun, Xin Xiao, Haoyuan Guo, and Jun Xiao. World to code: Multi-modal data generation via self-instructed compositional captioning and filtering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.
4608-4623, 2024a.
16
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang. Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXivv2401.16158 2024b. I
Li Wang, Zechen Bai. Yonghua Zhang, and Hongtao Lu. Show, recall, and tell: Image captioning with recall mechanism. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 12176-12183, 2020. 3
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint wwXiv:2404.0797 2024. 3
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang. Yiwu Zhong. Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562 2023. 1,7,
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.11441. 2023a. 9, 21
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint 13Xiv:2312.13771, 2023b.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. arXiv preprint arXiv:2404.05719 2024. 3
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: ofain-of-action thooght for gui agents. arXiv preprint XXiv:24433.02713 2024.4 4
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 3132-3149, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10.18653/v 12n.55//1/22....nnnnaaaa....... URL wwwps:tpp//.cm /acllallc aaattnooooooooo ogy. .rgg 2024. f inniinggn--nn aacgg-acl.... .11.186. 9
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded. In Forsy-first International Conference on Machine Learning, 2024. URL https: httss://pppeeviieeee net forum? f 9,20
15

---

Published as a conference paper at ICLR 2025
Longtao Zheng, Rundong Wang, and Bo An. Synapse: Leveraging few-shot exemplars for human- level computer control. arXiv preprint arXiv:2306.07863, 2023. 2
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprini urXiv:2307.18852 2023. 1
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 1059,,, 2023. 1.3
APPENDIX
A MORE INFORMATION FOR THE COLLECTED DATA
In this section, we provide additional details about collected data.
Topics and Domains. We list the topics and some sampled domains in our collected data, as is shown in Table 10.
Topic
Related Domains
News and Journalism Technology and Science
www. bbc com, Nww cnn. com, www. wired. com, www. www.scieecedally com, com
cnn.cnn.
Sports
www.espp.coo com, www www. www.ccbsspots.. comm www. www.immbb... www.immb.com com, www.niifii....mmmm com, ... www.wellandgood. con, www.gocp.com ...
Entertainment
Lifestyle and Wellness
Shopping and E-commerce Travel
www.amazn.ccomm com, xwx.ebay. www.airbme com, book woookiig...... com, tasty.co co, www foodnetwork com, www. www.etwwrk.......m com, www. hww.z.ouz.comm com, www. www.sera.....co org, www.ted.com com, www.githab. com, www.capeut.com.
Food and Cooking
DIY and Home Improvement
Educational and Learning
Others
Table 10: Examples topics and some corresponding domains in collected data.
Data Example Visualization. In Figure. 4, we present several samples from our collected dataset. These visualizations include bounding boxes highlighting the corresponding elements.
Examples of Augmented Query Templates. In Table. 11, we showcase portion of the augmented query templates used to generate queries for GUT grounding training. These templates include both common templates and element-speeiffc templates. During training, the query is generated using cither the common templates or the clement-specific templates.
Statistics of Web GUI Data. Figure. 5 presents the distribution of the number of instances (elements) in our collected Web GUI dataset. It shows that most web screenshots in our dataset contain 20-100 annotated elements, highlighting the "dense annotation" characteristic of our dataset.
Figure. illustrates the distribution of element types in our collected dataset. We combined the same element types from the navigation bar and the main content areas of the websites. The "Static" element, which primarily includes text elements, ranks first in the dataset, likely enhancing grounding performance for text elements. The "Button" element ranks second, which may also contribute to improved grounding performance for this type. Our collected data encompass a broad range of elements, showcasing the "diverse and rich annotation" characteristic of our dataset.
Looking towards future directions, data construction for GUI Agents could focus on building high- quality SFT data across different platforms, applications, and scenarios (Wang et al., 2024a; Lin et al., 2024).
16

---

Published as a conference paper at ICLR 2025
Trumea Mochis Migrthhssethhh Butiness Aralytic
Sperting
Vlsuggo
Figure 4: Data examples from our collected data.
17

---

Published as a conference paper at ICLR 2025
Augmented Query Examples.
Examples for common queries
"Look for [content]".
"Identify the element (content)",
"Locate the {class name) called {content]",
"Scroll the (class name] with [content]",
"Highlight fclass name) that contains [content]".
"Focus on the [class name] that displays [content]".
"Locate the [class name) that is part of [parent info] with the name [content]""..
Examples for specific element (button)
o "Click the (content) button".
"Press the button labelled [content)",
AAtivatee the [content] button".
"Find and click the [content] on the page".
"Where the [content] button located?".
"Can you locate the button that says [content]?",
"Click the [content) button in the [parent info!"...
Examples for specific element (Dataltem)
"Tick the [content] checkbox",
"Check the [content] option".
"Mark the checkbox for [content]".
"Select the (content) checkbox from the list",
"Ensure the (content) checkbox checked",
"Activate the checkbox labeled (content)".
o "Verify that the (content) checkbox is selected"..
Examples for specific element (checkbox) o "Identify data item laheled [content]",
"Locate the [content) data field on the page",
"Find the [content) item in the dataset",
"Access the information for [content]".
"Highlight the data item (content)". o "Explore the {content] item for more info"
Examples for specific element (hyperlink)
"Use the [content] link access the desired page", "Identify link called [content) and click it",
"Proceed by using the hyperlink (content)".
"Check the [content] link for more details".
"Open the web page linked by (content)".
"Access resources through the [content] hyperlink",
"Click (content) URL to navigate",
"Follow the online pointer marked as (content)""
Examples for specific element (static text)
"Highlight statement [content]".
OOberrve the static text [content] on this page".
'Search for the label [content) in the content",
"Inspect the static field titled (content)",
"See displayed static text (content)".
"Look the static description (content]",
"Spot the phrase (content) in the view".
"Find the static wording (content)"
Table 11: Augmented query examples during GUI grounding training. Shown are examples of some of the augmented templates.
18

---

Published as a conference paper at ICLR 2025
Percentage of Number of Instances in Web GUI Data
25
20
Percentage (%)
15
10
100
150
200
250
300
350
400
Number Instances
Figure 5: Distribution of "Numer of instances (elements) per GUI image" of our collected Web GUI dataset.
Percentage of Gill Eements Web GUI Data
20
Dateheen
Mershhon
ensige
Commasttion
Oocckstoce
Madiduutton
Teeaturr
Roolop
Loodoo
500d
Proposss
Cuobom
Upttom
Figure Distribution of "element type" of collected Web GUI dataset.
19

---

Published as a conference paper at ICLR 2025
B ETHICAL CONSIDERATIONS OF THE COLLECTED DATASET
Discussion on NSFW Content Filtering, To ensure that our dataset does not contain NSFW content, we perform several steps. Before collection, we carefully screen candidate websites to filter out those that may contain NSFW content. During the data collection process, we also apply manual reviews for verification to ensure the dataset is suitable for academic research and deployment without the risk of including inappropriate material.
Discussion on Copyright. Our Web GUI data is sourced from the internet, and address potential copyright issues, we ensure compliance with copyright laws and regulations by:
We primarily collect data from publicly accessible websites explicitly available for educational and research purposes.
Sourcing Data from Public Domains:
Respecting Website Policies: We adhere to the terms of service and data policies of the websites we crawl.
Anonymizing and Aggregating Data: We anonymize and aggregate the collected data, focusing on general patterns and characteristics minimize potential copyright infringements.
Statement of Responsibility and Risk Acknowledgment: The authors explicitly acknowledge and accept full responsibility for any legal implications arising from the collection and usage of the benchmark datasets in this work. We hereby declare that we understand and accept all associated risks regarding data usage, copyright compliance, and legal responsibilities.
C ADDITIONAL EXPERIMENTS
In this section, we present further details, experiments, discussions, and analyses to demonstrate the effectiveness of our model and explore future directions.
Implementation details. We configure the image encoder and lightweight decoder of the GUI grounding model using the ViT-SAM-Large (Kirillov et al., 2023b) and the language prompt encoder with BERT-based (Devlin et al., 2019) by default. Training runs on 32 V100 GPUs with a global batch size of 128 for 150K steps. The AdamW (Loshchilov, 2017) optimizer is used with 61 = 0.9, 32 0.98, and a weight decay of le-4. Cosine Annealing scheduler manages the learning rate, starting with a warm-up over 200 steps. The max learning rate is 1e-3, dropping to 5e-5. Hyperparameters for the training objective are set as Amse 10, L1, = 5, and AGIOU = 2. For vision-based agent tasks, (rank 8, alpha 16) for the language model. The decoder parameters are unfrozen, while the image encoder remains frozen. We AdamW with 31 0.9, A2 0.98, and no weight decay. The training objective hyperparameters are Auaa = and Lyd = 1. For vision-based agent tasks, we fine-tune with 2 A100 GPUs, applying LoRA (Hu al., 2021) tuning (rank 8, alpha 16) for the language model. The decoder parameters are unfrozen, while the image encoder remains frozen. We use AdamW with 31 0.9, B2 = 0.98, and weight decay. The training objective hyperparameters are Amex= = 1 and Cgd 1.
fine-tune with 2 A100 GPUs, applying LoRA (Hu et al., 2021) tuning
Scaling Pretraining Data for GUI Grounding, We explored the effect of scaling our collected web GUI data on model performance. Using the base-size model AET--MM-Basse we conducted experiments by sampling 25% 50%, 75%, and 100% of the training data from websites within the same domain. We then evaluated the model's grounding performance on ScreenSpot
Results in Table. 12 indicate that increasing the amount of training data leads better grounding performance. However, we observed a slowdown in performance improvement when scaling from 75% to I 100% compared to the gains from 25% to 50% and 50% to 75%. Despite this, the data scaling paradigm still shows potential for further improvement. Future efforts focus collecting more diverse and high-quality GUI data to enhance performance even further.
Comparison with Text-involved Method. Both SeeAct (Choice) (Zheng et al., 2024) and Min- dAct (Deng et al., 2024) adopt text-form candidates as input and select an answer from them. Although this approach is casier than visually grounding the target element using only image, these methods are still state-of-the-as for web agent tasks. However, they face challenges with similar or identical elements, which are common webpages. The model often selects the first text choice that
20

---

Published as a conference paper at ICLR 2025
Mobile
Desktop
Web
Data scale
Avg.
Text
Icon
Text
Icon
Text
Icon
25%
72.4
50.8
59,4
27.3
57.5
28.0
49.2
50%
78.2
53.5
68.4
31.9
62.8
33.5
54.7
75%
79.6
54.8
74.5
35.2
66.2
36.8
57.9
100%
80.2
56.1
75.9
39.1
68.2
41.6
60.2
Table 12: Results of scaling pretraining data for ScreenSpot grounding.
appears correspond to its intent. This issue is inevitable since many webpage elements may have identical HTML attributes.
Our model addresses this problem effectively, In Figure. 7, (A) shows a task for "schedule repair service" and (B) depicts a task for "rent truck". With identical element names provided as input, it is impossible for an LLM to definitively identify the correct one. As result, the text-involved methods failed to select the correct button, as shown in the red boxes. In contrast, our model successfully grounds the correct position to click, as indicated by the green dots Figure. This success is due to the direct integration of AGG, which allows the context from the MLLM to be effectively passed to the grounding module for context-awe aware grounding, thereby avoiding the information loss seen in SeeAct (Choice) and MindAct.
Evaluation on Desktop Benchmark. To test the our model's abilities on software or OS level, we use the Desktop split of the OmniAct benchmark (Kapoor et al., 2024). This benchmark covers 38 desktop applications across different desktop platforms, and requires the model to generate a piece of PyAutoGUI script at each step complete the task. We compare our method with SeeClick (Cheng et al., 2024) and SoM (Yang et al., 2023a). We finetuned models on the OmniAct training set and followed (Kapoor et al., 2024) to measure the Action Score (AS) and include the grounding successful rate for steps requiring GUI grounding.
Results in Table. 13 demonstrate that on OmmiAct-Desktop, our model outperforms SeeClick in grounding-related operations, resulting in higher overall Action Score. Additionally, the label- marking model of SoM struggles accurately identify interactive elements from desktop screen images, leading to poor performance on this task, consistent with findings in (Kapoor et al., 2024).
Method
Grd.Acc
Action Score
SoM (Kapoor et al., 2024)
12.3
17.6
SeeClick (Cheng et al., 2024)
55.2
23.8
MLLM a AGG (Ours)
62.4
27.2
Table 13: Results on OmniActt-Dssttop split.
Error Analysis. We conduct error analysis for our proposed model from grounding task and agent-based task.
Errors in Grounding Task. We manually inspected test cases from ScreenSpot and found that a significant portion of failures stem from grounding errors involving unseen buttons or icons on mobile or desktop platforms. Despite being pretrained on extensive web GUI data, our model struggles to localize icons with specific meanings and shapes unique to particular platforms. For example, in Figure. 8 (A-C), the lef- simple-designed icon denotes "save the file". "AC" in a mobile calculator app means "All clear calculator'. and the bug-shaped vector icon the IDE means "debug". In these instances, our model incorrectly predicts the "File" item in the PPT navigation bar and nonsensical positions in cases (B) and (C). Additionally, the model struggles with ambiguous queries, as seen in Figure. 8 (D). From this error analysis, conclude that capturing specific semantics in long-tail categories, such as platform platform-specific icons and buttons, remains an open challenge for grounding.
Errors in Agent-based Tasks. By inspecting test cases from the Mind2Web (Deng al., 2024) and OmniAct (Kapoor et al., 2024) benchmarks, we have identified several error patterns in the current model. First, when MLLM or planner generates incorrect actions or context, it can adversely affect the grounding process. For example, if an MLLM incorrectly outputs the action "type" and subsequently the grounding token, the decoded position is likely be search bar text box,
21

---

Published as a conference paper at ICLR 2025
correlating with the erroneous action. Thus, incorrect context from the MLLM negatively impacts subsequent grounding. Second, similar to errors observed in grounding tasks, element grounding enrors in agent-based tasks are more likely when the target element is small and less common in the training set. This analysis underscores the importance of enhancing the model's grounding capabilities, particularly for long-tail categories.
22

---

Published as a conference paper at ICLR 2025
Phat
bolnn
Q
Nai of Oeer
Sorl by
Name
Chagg
Tunile Mocizz
Jees Cora
Chorg
Tothars D00
Shoce Mlve Dours
(A) Schedule repair service for my gaming console closest the zip code 10001 on March 24, anytime after 6 pm.
Budget
Teak soild
29.99
Salicc TrAck
Mockin TrAi
19.99
Woniin Truck
Budget
Movinn la hand, bnjal
ENGGAENNERAA ConmeIcal Traces
Suppr
Campary
(B) Rent a truck with the lowest rent with two dozen furniture pads for 100 miles at zip 08817 on April 12 at 2:30 pm rented truck is to be returned to the exact location and date, and the pickup and drop off will be at the nearest location.
Figure 7: Examples for comparing method with text-involved methods.
23

---

Published as a conference paper at ICLR 2025
AC
%
Famder
7
9
Cipcarrd
1
3
Click
0
(A) Save the file
(B) All clear calculator
GNL
(C) Debug
(D) Sign
Figure 8: Visualization of grounding error ScreenSpot. The green box indicates the ground truth area. The purple dot the predicted position of our model. We show four examples with corresponding query.
24


Reviews:
[{"review_id": "XZBzaLMiEz", "reviewer": "Reviewer_9768", "summary": "In this work the authors propose a framework for improving the GUI grounding of multimodal LLMs, i.e., the localization of relevant elements in a UI from a textual description of the elements. \n\nThe framework presented includes an automated way of collecting large scale data across a variety of websites along with automatic annotation of the locations, properties and descriptions of the various UI elements. The authors then clean this data programmatically to remove data with low-quality annotations (mislabeling, missing annotations) to arrive at a clean dataset. \n\nThis data is used to train a novel lightweight grounding model to identify the location of a UI element given a textual query. This grounding model achieves SoTA (or close to SoTA) performance on grounding tasks like Screenspot. The authors then present how to integrate their grounding model with a multimodal LLM for completing automation tasks on datasets like MiniWob, Mind2Web and Android in the Wild. They fine-tune an MLLM to work with their grounding model and demonstrate strong performance compared to other image based approaches. Finally, the authors perform ablations to measure the impact of their dataset and model on the performance of their overall framework.", "questions": "* \"Next, we clean the element data by removing entries with erroneous or missing element names.\" [203-205] \nHow are erroneous element names identified? What is the groundtruth to which they are compared to identify a name as erroneous?\n\n* \"We also address cases where the element hierarchy is ambiguous or misrepresented by restructuring it to accurately reflect the parent-child relationships between GUI elements.\" [206-208] \nCan you explain how you identify ambiguous or misrepresented hierarchies? Also how do you identify the parent-child relationships between GUI elements?\n\n* \"First, we eliminate websites that are either empty or inaccessible as many URLs may lead to pages with no meaningful content, broken links, or require authentication that cannot be handled automatically.\" [201-203]\nCan you specify how many of unique websites are present in the filtered dataset? Also, did this result in any of the topics listed in Table 10 being completely removed?\n\n* Did you conduct any ablations for the filtering and cleaning of the pre-training data on Screenspot or other evals? Any numbers here will be useful for differentiating the pre-training dataset from other existing datasets.\n\n* For the agent/UI automation evals, are the models separately fine-tuned for each of Miniwob, AiTW and Mind2Web? Do you have a comparison to one model trained on all 3 datasets? It will be useful to assess the robustness of the grounding model across different UI types.\n\n* For mind2web, did you run any evals that can be directly compared to a text-based multiple choice or Set of Marks setting (in terms of using the same LLM output for planning)? I think it will be useful to see what is the gap between different grounding methods since you have SoTA image based grounding.\n\n* Can you specify what is the metric/dataset used for Tables 8 and 9? \n\n* For mind2web, can you verify if some of the test-set domains are present in your grounding training set? I don't think it is a big problem but it will be useful to know for the test-website and test-domain splits.\n\n* Do you have any experiments on the scaling of grounding performance with more automatically generated training data? This could indicate if there is any headroom with scaling of pre-training data.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. SoTA image only performance: This work achieves SoTA performance on image based grounding across Screenspot and Automation tasks like MiniWob, Android In the Wild, Mind2Web. Given the recent interest in developing MLLM based GUI agents, and also given that grounding of actions in UIs is identified as a bottleneck for current approaches, this work can be very impactful for the research community.\n\n2. Light-weight model for grounding: Propose a light-weight model for grounding compared to other approaches like Fuyu, SeeClick although this is a single-task model specifically fine-tune for grounding. The authors also presented detailed ablations for the modeling choices which is very informative.\n\n3. Extensible dataset collection approach: The authors present an approach for large scale data collection and annotation which will be of interest to the research community. The dataset seems to result in SoTA performance on grounding tasks.", "weaknesses": "1. The paper achieves state of the art results for image based models on grounding/automation tasks. However, I have some questions about the novelty of this work. The dataset collected has many similarities to the dataset from SeeClick with the main difference being the source of the annotations: UI Automation framework vs attributes from the DOM. The model architecture is quite similar to LISA and SAM with detection heads (and loss) replacing segmentation ones.\n\n2. The process for cleaning the annotations is missing some details. In particular, to identify elements with erroneous element names, restructuring the hierarchy of UI elements it is not clear the source of the “correct information” to identify errors and to do restructuring.\n\n3. Error analysis: In lines 484-487 the authors note that image only approaches fall short of text-based approaches. It will be very useful if the authors can analyze the errors compared to a text-based approach and try to identify patterns (if any) which can point to future directions of improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7iRjyO7cQ", "reviewer": "Reviewer_3qqW", "summary": "This paper seeks to enhance the grounding capabilities of MLLM-based GUI agents by introducing a new grounding dataset. The authors propose a lightweight grounding plugin trained on this dataset, demonstrating its effectiveness. Additionally, they fine-tune an MLLM-based GUI agent using Qwen-VL, showing promising results on various GUI grounding and navigation benchmarks.", "questions": "- Conduct experiments on additional benchmarks to solidify claims about the model's general GUI grounding abilities.\n- Include a more thorough analysis of the dataset and address potential concerns regarding NSFW content and copyright.\n- Include failure case analysis.\n- Compare with stronger baselines, such as SeeClick-V and Set-of-Marks, to underscore the model's performance relative to current approaches.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Comprehensive GUI Grounding Dataset: The paper presents an automated data collection engine that generates large-scale, diverse, and annotated GUI screenshots. This helps mitigate the shortage of publicly available GUI datasets, significantly contributing to the field.\n- Improved Grounding Capabilities: The proposed GUI grounding module demonstrates enhanced task accuracy and adaptability, yielding impressive performance across various GUI benchmarks.\n- Robust Technical Contribution: The paper includes extensive experiments that effectively demonstrate the efficacy of their model for GUI grounding and agent tasks.", "weaknesses": "- Over-claimed Contributions and Misused Terminology: While the ablation studies are appreciated, the authors overstate their contributions. The claim of strong performance in general GUI grounding is unsupported, as they evaluate their model only in web-based tasks (Android-AITW and Mind2Web). Additional experiments on software- or OS-level benchmarks, such as OS-World [1], WindowsAgentArena [2], or VisualAgentBench [3], are necessary to substantiate their claims. Moreover, the term \"general GUI grounding\" is misleading, as their dataset is collected from webpage. It would be more accurate to call it a \"web grounding dataset\", although I admit that GUI elements may transfer across different platforms.\n\n- Writing Suggestions: The Experiment section lacks clarity, particularly in its structure. It is recommended to introduce details about benchmarks and evaluation metrics in the Experiment Setup section, rather than in the Results section. For instance, information about the sample size and metrics for AITW is provided in line 432 of the Results section but should be included in the setup. A clearer organization would be to separate experiments by benchmarks, as each uses different baselines and metrics.\n\n- Failure Cases: Including failure cases of the proposed grounding model and agent would offer a more comprehensive evaluation of the approach and provide insights into its limitations.\n\n- Dataset Analysis: While the paper discusses the dataset size, a more detailed analysis is essential. Providing statistics on different levels of data and characteristics of the dataset would offer more insight. Additionally, the authors should address whether the dataset contains NSFW content, which would be crucial for future research and deployment. Moreover, given that your dataset is crawl from the internet, copyright problems should be discussed in this paper.\n\n- Comparison with Stronger Baselines: The paper lacks comparison with recently released baselines, such as SeeClick-V. Including such comparisons, especially highlighting cases where their model succeeds while others fail, would further emphasize the model's strengths. Additionally, comparing against Set-of-Marks, a common baseline for non-GUI-specific models, would provide a broader perspective on the effectiveness of their GUI-specific model.\n\n[1] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments\n\n[2] WindowsAgentArena: Evaluating Multi-Modal OS Agents at Scale\n\n[3] VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIAlxCl44w", "reviewer": "Reviewer_ivbR", "summary": "This presents an automated data synthesis for web screenshots and a lightweight model for precise UI element localization, which can be further integrated with MLLMs. The approach enhances MLLM accuracy and adaptability across GUI grounding and agent benchmarks.", "questions": "Overlap Issue in Mind2Web Evaluation: Is the model's high performance on Mind2Web genuinely indicative of its capabilities, or is it influenced by an overlap with similar website domains (e.g., shopping, travel) in the training data?\n\nFor the e2e agent setting, despite the ablation study, finally we probably can solve the problem with only an end2end MLLM, without AGG (for example, using Qwen 2 VL, which is known for being capable of grounding). Can you discuss more about the advantage of this work compared to a better finetuned MLLM?\n\nA minor issue: In Fig. 3, does recording \"Click at (coordinates)\" as part of action history make sense for web agents? (When there is no screenshot recorded for understanding clicking at a point)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a lightweight model architecture based on SAM and BERT that achieves strong grounding performance (on ScreenSpot), demonstrating impressive results despite being much smaller (0.4B) than contemporary 7B models.\n\nThe writing is clear and cohesive, and the experimental results, including ablation studies, reasonably support both the model design and the data’s effectiveness.", "weaknesses": "The web-based data collection pipeline is not novel or technical.\n\nAs shown in Table 2, the model struggles with grounding icon elements, which are both essential and challenging in GUI tasks. In contrast, text elements are relatively easier to ground through OCR. Maybe the authors should add more discussions about the discrepancy and the ways to address the challenges.\n\nThe model's results do not significantly outperform SeeClick on benchmarks like ScreenSpot, AITW, and MiniWob, despite using a larger and higher-quality training dataset. The significant improvement on Mind2Web is likely due to overlap witt websites and domains (e.g., shopping, travel) rather than a generalized performance boost, whereas SeeClick’s data is more randomly sampled from Common Crawl.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3p1Jpf0jLJ", "reviewer": "Reviewer_UKHa", "summary": "This paper seeks to enhance the grounding capabilities of MLLM-based GUI agents by introducing a new grounding dataset. The authors propose a lightweight grounding plugin trained on this dataset, demonstrating its effectiveness. Additionally, they fine-tune an MLLM-based GUI agent using Qwen-VL, showing promising results on various GUI grounding and navigation benchmarks.", "questions": "- Improve the paper's presentation by addressing formatting issues and clarifying the organization of sections.\n- Conduct experiments on additional benchmarks to solidify claims about the model's general GUI grounding abilities.\n- Include a more thorough analysis of the dataset and address potential concerns regarding NSFW content and copyright.\n- Include failure case analysis.\n- Compare with stronger baselines, such as SeeClick-V and Set-of-Marks, to underscore the model's performance relative to current approaches.\n\n**I will consider raising the overall score if author address my concerns, especially the writing and presentation.**", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Comprehensive GUI Grounding Dataset:** The paper presents an automated data collection engine that generates large-scale, diverse, and annotated GUI screenshots. This helps mitigate the shortage of publicly available GUI datasets, significantly contributing to the field.\n- **Improved Grounding Capabilities:** The proposed GUI grounding module demonstrates enhanced task accuracy and adaptability, yielding impressive performance across various GUI benchmarks.\n- **Robust Technical Contribution:** The paper includes extensive experiments that effectively demonstrate the efficacy of their model for GUI grounding and agent tasks.", "weaknesses": "- **Presentation Issues:** The paper appears rushed, with room for improvement in formatting and clarity. For instance, the overview figure on the first page should be placed above the Abstract section instead of between the section title and content of the Introduction. Additionally, there are several missing spaces before `\\cite`, and incorrect usage of `\\citep` and `\\citet`.\n- **Overclaimed Contributions and Misused Terminology:** While the ablation studies are appreciated, the authors overstate their contributions. The claim of strong performance in general GUI grounding is unsupported, as they evaluate their model only in web-based tasks (Android-AITW and Mind2Web). Additional experiments on software- or OS-level benchmarks, such as OS-World [1], WindowsAgentArena [2], or VisualAgentBench [3], are necessary to substantiate their claims. Moreover, the term \"general GUI grounding\" is misleading, as their dataset is collected from webpage. It would be more accurate to call it a \"web grounding dataset,\" although I admit that GUI elements may transfer across different platforms.\n- **Writing Suggestions:** The Experiment section lacks clarity, particularly in its structure. It is recommended to introduce details about benchmarks and evaluation metrics in the Experiment Setup section, rather than in the Results section. For instance, information about the sample size and metrics for AITW is provided in line 432 of the Results section but should be included in the setup. A clearer organization would be to separate experiments by benchmarks, as each uses different baselines and metrics.\n- **Failure Cases:** Including failure cases of the proposed grounding model and agent would offer a more comprehensive evaluation of the approach and provide insights into its limitations.\n- **Dataset Analysis:** While the paper discusses the dataset size, a more detailed analysis is essential. Providing statistics on different levels of data and characteristics of the dataset would offer more insight. Additionally, the authors should address whether the dataset contains NSFW content, which would be crucial for future research and deployment. Moreover, given that your dataset is crawl from the internet, copyright problems should be discussed in this paper.\n- **Comparison with Stronger Baselines:** The paper lacks comparison with recently released baselines, such as SeeClick-V. Including such comparisons, especially highlighting cases where their model succeeds while others fail, would further emphasize the model's strengths. Additionally, comparing against Set-of-Marks, a common baseline for non-GUI-specific models, would provide a broader perspective on the effectiveness of their GUI-specific model.\n\n[1] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments\n\n[2] WindowsAgentArena: Evaluating Multi-Modal OS Agents at Scale\n\n[3] VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)