AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs
Authors: Hongxin Li, Jingran Su, Jingfan CHEN, Yuntao Chen, Li Qing, Zhaoxiang Zhang
Accepted at: acl25

AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs

Hongxin Li*1,2,3, Jingfan Chen*5, Jingran Su*5, Yuntao Chen†4
Qing Li5, Zhaoxiang Zhang†1,2,3,6
1University of Chinese Academy of Sciences (UCAS)
2New Laboratory of Pattern Recognition (NLPR), CASIA
3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA
4Hong Kong Institute of Science & Innovation, CASIA
5The Hong Kong Polytechnic University 6Shanghai Artificial Intelligence Laboratory

Abstract

User interface understanding with vision-language models (VLMs) has received much attention due to its potential for enhancing software automation. However, existing datasets used to build UI-VLMs either only contain large-scale context-free element annotations or contextualized functional descriptions for elements at a small scale. In this work, we propose the AutoGUI pipeline for automatically annotating UI elements with detailed functionality descriptions at scale. Specifically, we leverage large language models (LLMs) to infer element functionality by comparing UI state changes before and after simulated interactions. To improve annotation quality, we propose LLM-aided rejection and verification, eliminating invalid annotations without human labor. We construct a high-quality AutoGUI-704K dataset using the proposed pipeline, featuring diverse and detailed functionality annotations that are hardly provided by previous datasets. Human evaluation shows that we achieve annotation correctness comparable to a trained human annotator. Extensive experiments show that our dataset remarkably enhances VLM’s UI grounding capabilities and exhibits significant scaling effects. We also show the interesting potential use of our dataset in UI agent tasks. Please view our project at https://autogui-project.github.io/.

1 Introduction

User interface understanding with visual language models(VLMs) (Hong et al., 2024; You et al., 2024; Wu et al., 2024) has received wide attention due to its potential in fundamentally transforming how we interact with software (Xie et al., 2024).

While recent work has made progress by employing structural mapping between UI code and visual layout, such as UI REG/REC(Hong et al., 2024; Li et al., 2020a) and layout-to-code conversion (Xia et al., 2024; Liu et al., 2023a; Baechler et al., 2024), a more critical challenge remains: understanding the semantic purpose and interactive affordance of individual UI elements, known as functionality understanding.

Accurate functionality understanding requires VLMs to possess strong element grounding capabilities - the ability to connect fine-grained visual elements with their referring expressions. To enhance this capability, large-scale training data is indispensable. However, the scale of open-source datasets with detailed element annotations (Li et al., 2020a,b; Kapoor et al., 2024; Gou et al., 2025) is unsatisfactory, significantly smaller than natural im-

![Our annotations are rich in functional semantics (bottom) compared with existing UI datasets.](page_1012_384_388_627.png)

Figure 1: Our annotations are rich in functional semantics (bottom) compared with existing UI datasets.
age datasets such as LAION-5B (Schuhmann et al., 2022). Additionally, traditional annotation methods (Deka et al., 2017a; Li et al., 2020a) are labor-intensive, leading to prohibitive costs that hinder scalability. Moreover, existing datasets typically focus on describing either element alt-texts (Cheng et al., 2024), or brief intents weakly related to UI context (Bai et al., 2021) shown in Fig. 1. These datasets lack contextual functional descriptions of UI elements, which poses a challenge for VLMs in comprehending the roles these elements serve within specific UI contexts, such as distinguishing between two visually similar magnifier icons that may represent distinct functionalities like searching and zooming.

To address the challenge, we propose AutoGUI, an automatic annotating pipeline that provides unlimited element functionality annotations. Our pipeline collects UI interaction trajectories and leverages large language models (LLMs) to infer element functionalities based on UI state changes, eliminating the need for manual annotation by human experts. Initially, the proposed pipeline crawls a multitude of interaction trajectories on either a web browser or an Android emulator. Subsequently, we use open-source LLMs (AI@Meta, 2024) to annotate the functionalities of elements on collected GUIs based on changes to UI contents when interacting with these elements. To ensure data quality, LLM-aided rejection is utilized to eliminate invalid samples, such as incompletely rendered UIs. Additionally, inspired by LLM verification (Weng et al., 2022; Lightman et al., 2023), multiple LLMs are prompted as verifiers to identify false functionality descriptions. With both the rejection and verification processes, our pipeline removes unclear and invalid samples.

We curate the AutoGUI-704k dataset with the proposed pipeline, providing high-quality functionality grounding and referring tasks used to finetune and evaluate open-source VLMs. Pioneer experiments find that our pipeline achieves annotation accuracy of \(96.7\%\) comparable to a trained human annotator.

Based on the collected AutoGUI-704k dataset, we finetune open-source VLMs and demonstrate that our data significantly enhances the VLMs’ UI grounding accuracy and exhibits remarkable scaling effects. The results also show that our functionality annotation type is superior to the data type directly derived from web HTML code and metadata (Hong et al., 2024; Cheng et al., 2024), serving as a promising data source for building VLMs capable of UI grounding. Moreover, VLMs trained with our data can assist in GUI agent tasks by refining element grounding, which shows more potential use of our dataset.

2 Related Works

2.1 Recent Advancement of VLMs

Recent research has enhanced LLMs with the capability of processing both visual and textual information (Alayrac et al., 2022; Liu et al., 2023b; Lin et al., 2023; Chen et al., 2024b; Lu et al., 2024; Wang et al., 2024a,b; Li et al., 2024b; Zhang et al., 2024a; You et al., 2024; Laurençon et al., 2024; Peng et al., 2024; Driess et al., 2023), opening the new field of VLM. Pioneering efforts Flamingo (Alayrac et al., 2022) uses interleaved visual and language inputs as prompts and shows few-shot visual question-answering capability. LLaVA (Liu et al., 2023b) and LLaMA-Adapter (Zhang et al., 2024a) have attempted to align vision encoders (Dosovitskiy et al., 2021) with LLMs to enable visual instruction following. Advanced models such as InternVL (Chen et al., 2024b) and Qwen2-VL series (Wang et al., 2024a) are further equipped with impressive high-resolution and multi-lingual understanding abilities. Additionally, VLM are applied to scenarios rich in textual imagery (Ye et al., 2023b,a; Liu et al., 2024b) and embodied interactions (Driess et al., 2023; Kim et al.). Despite these advancements, VLMs lag in UI understanding probably due to data scarcity. This paper contributes an automatic UI annotation pipeline to tackle this challenge, aiming to expand the data available for training VLMs in this crucial area.

2.2 Existing UI Datasets

Unlike natural image datasets (Russakovsky et al., 2014; Schuhmann et al., 2022), UI understanding datasets are much smaller. Early-stage datasets (Wang et al., 2021; Li et al., 2020a,b; Bai et al., 2021; Burns et al., 2022) primarily annotate the RICO screenshot collection (Deka et al., 2017b), which includes 72K screenshots from Android apps. Examples include Widget Captioning (Li et al., 2020a), which analyzes captions and linguistic features of UI elements, and RICOSCA (Li et al., 2020b), which maps single-step instructions to UI locations. Recently, AITW (Rawles et al., 2023) and AndroidCon-
Table 1: Comparing our AutoGUI dataset with existing UI grounding datasets. Multi-Res means the samples are collected on devices with various resolutions. Auto Anno. means the samples are collected autonomously. #Anno. means the number of annotated samples provided by the datasets. (Methods combining open-source individual datasets are not compared.)

<table>
  <tr>
    <th>Dataset</th>
    <th>UI Type</th>
    <th>Multi Res.</th>
    <th>Auto Anno.</th>
    <th>Functionality Annotation Type</th>
    <th>#Anno.</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>S2W (Wang et al., 2021)</td>
    <td>Mobile</td>
    <td>x</td>
    <td>x</td>
    <td>N/A</td>
    <td>112k</td>
    <td>Screen Summarization</td>
  </tr>
  <tr>
    <td>Wid. Cap. (Li et al., 2024a)</td>
    <td>Mobile</td>
    <td>x</td>
    <td>x</td>
    <td>N/A</td>
    <td>163k</td>
    <td>Element Captioning</td>
  </tr>
  <tr>
    <td>RICOSCA (Li et al., 2020b)</td>
    <td>Mobile</td>
    <td>x</td>
    <td>x</td>
    <td>N/A</td>
    <td>295k</td>
    <td>Action Grounding</td>
  </tr>
  <tr>
    <td>MoTIF (Burns et al., 2022)</td>
    <td>Mobile</td>
    <td>x</td>
    <td>x</td>
    <td>N/A</td>
    <td>6k</td>
    <td>Mobile Navigation</td>
  </tr>
  <tr>
    <td>RefExp (Bai et al., 2021)</td>
    <td>Mobile</td>
    <td>x</td>
    <td>x</td>
    <td>N/A</td>
    <td>20.8k</td>
    <td>Element Grounding</td>
  </tr>
  <tr>
    <td>SeeClick Web (Cheng et al., 2024)</td>
    <td>Web</td>
    <td>x</td>
    <td>x</td>
    <td>N/A</td>
    <td>271k</td>
    <td>Element Grounding</td>
  </tr>
  <tr>
    <td>MultiUI (Hong et al., 2024)</td>
    <td>Web, Mobile</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>3M</td>
    <td>Act. & Elem. Ground</td>
  </tr>
  <tr>
    <td>UGround-Web (Gou et al., 2025)</td>
    <td>Web</td>
    <td>✓</td>
    <td>✓</td>
    <td>Brief</td>
    <td>1.3M</td>
    <td>Element Grounding</td>
  </tr>
  <tr>
    <td>UI RECREG (Hong et al., 2024)</td>
    <td>Web</td>
    <td>✓</td>
    <td>✓</td>
    <td>N/A</td>
    <td>400k</td>
    <td>Box2DOM, DOM2Box</td>
  </tr>
  <tr>
    <td>Ferrer-UI (You et al., 2025)</td>
    <td>Mobile</td>
    <td>✓</td>
    <td>✓</td>
    <td>Brief</td>
    <td>250k</td>
    <td>Elem. Ground & Ref.</td>
  </tr>
  <tr>
    <td><b>AutoGUI (ours)</b></td>
    <td>Web, Mobile</td>
    <td>✓</td>
    <td>✓</td>
    <td><b>Contextual</b></td>
    <td><b>704k</b></td>
    <td><b>Functionality Ground & Ref.</b></td>
  </tr>
</table>

trol (Li et al., 2024a) have been proposed to focus on interpreting high-level instructions in Android environments. To increase data scale, SeeClick (Cheng et al., 2024), CogAgent (Hong et al., 2024), and OS-ATLAS (Wu et al., 2024) have utilized the UI metadata from Common Crawl webpages to produce massive element referring expressions. Several works (Gou et al., 2025; Lin et al., 2024; Xu et al., 2024) also filter and combine existing datasets to produce all-in-one collections that incorporate diverse training tasks. In contrast, our AutoGUI-704k dataset contributes large-scale element functionality annotations, which convey contextual functionality semantics that are hardly provided in previous datasets. The advantages of our dataset are summarized in Tab. 1.

3 AutoGUI: Automatic Functionality Annotation Pipeline

This section introduces AutoGUI, an annotation pipeline (Fig. 2) that automatically produces contextual element functionality annotations used to enhance VLMs’ GUI grounding capabilities.

3.1 Collecting UI Interaction Trajectories

Our pipeline initiates by collecting interaction trajectories, which are sequences of UI contents captured by interacting with UI elements. Each step captures all interactable elements and the accessibility tree (AXTree) that briefly outlines the UI structure, which will be used to annotate functionality. To amass these trajectories, we utilize the latest Common Crawl repository as the data source for web UIs and Android Emulator for mobile UIs. The open-source trajectories from Android-Control (Li et al., 2024a) and MobileViews (Gao et al., 2024) are also included to enhance diversity. Note that illegal UIs are manually excluded from the sources. Please refer to Sec. A for collecting details and data license.

3.2 Automatic Functionality Annotation

The pipeline generates functionality annotations for elements in the collected trajectories. Interacting with an element e, by clicking or hovering over it, triggers UI content changes. In turn, these changes can be used to predict the functionality f of the interacted element. For instance, if clicking an element causes new buttons to appear in a column, the element likely functions as a dropdown menu activator (an example in Fig. D). With this observation, we utilize a capable LLM (i.e., Llama-3-70B (AI@Meta, 2024)) as a surrogate for humans to summarize an element’s functionality based on the UI changes resulting from interaction. Concretely, we generate compact content differences for AXTrees before (\( s_t \)) and after (\( s_{t+1} \)) the interaction using a file-comparing library\(^1\). Then, we prompt the LLM to analyze the UI content changes (addition, deletion, and unchanged lines), present a detailed Chain-of-Thoughts (Wei et al., 2022) reasoning process explaining how the element affects the UI, and finally summarize the element’s functionality.

In cases where element interactions significantly transform the UI and cause lengthy differences—such as navigating to a new screen—we adjust our approach by using UI description changes instead of the AXTree differences. This annotation process is formulated as: \( f = \text{LLM}(p_{anno}, s_t, s_{t+1}) \) where \( f \) is the predicted functionality, \( p_{anno} \) is the annotation prompt (Tab. B and Tab. C). Examples

\(^1\)https://docs.python.org/3/library/difflib.html
Figure 2: The proposed pipeline for automatic UI functionality annotation. An LLM is utilized to predict element functionality based on the UI content changes observed during the interaction. LLM-aided rejection and verification are introduced to improve data quality. Finally, the high-quality functionality annotations will be converted to instruction-following data by applying task templates.

are depicted in Fig. 3 and more annotation details are explained in Sec. A.5.

3.3 Removing Invalid Samples via LLM-Aided Rejection

The collected trajectories may contain invalid samples due to broken UIs, such as incomplete UI loading, which can mislead the models trained with them. To filter out these invalid samples, we introduce an LLM-aided rejection approach. Initially, hand-written rules (detailed in Sec. A.6) are used to detect obvious bad cases, such as blank UIs, UIs containing elements indicating content loading, and interaction targets outside of UIs. However, a few types are difficult to detect with the rules. For instance, interacting with a “view more” button might unexpectedly redirect the user to a login page instead of the desired information page due to website login restrictions. To identify these challenging samples, we prompt the annotating LLM to also act as a rejector. Specifically, the LLM takes the UI content changes as input, provides detailed reasoning through whether the changes are meaningful for predicting the element’s functionality, and finally outputs predictability scores ranging from 0 to 3 (3 is empirically chosen for a balance between annotation efficiency and quality.). This process is formulated as follows: \( score = \text{LLM}(p_{\text{reject}}, e, s_t, s_{t+1}) \) where \( p_{\text{reject}} \) is the rejection prompt (Tab. D).

This approach ensures that predictable samples receive higher scores, while unpredictable ones receive lower scores. For instance, if a button labeled "Show More", upon interaction, clearly adds new content, this sample will be considered to provide sufficient changes that can anticipate the content expansion functionality and will get a score of 3.

We deploy this rejector to discard the bottom 30% of samples based on score ranking to strike a balance between the elimination of invalid samples and the preservation of valid ones (Details in Sec. A.7). The samples that pass the rejection procedure are submitted for functionality annotation. Please see examples in Fig. G.
This element provides access to a specific category or topic within the website, allowing users to explore related content.

This element provides access to a user’s account, enabling personalized experiences and features on the platform.

Web

This element enables users to create an account, allowing them to access exclusive content or engage with the community.

This element allows users to save content for later reference or use.

This element enables users to share content with others.

This element triggers a dropdown menu for users to access additional interaction options, probably including reporting, copying, or editing the content.

This element enables users to subscribe to a content creator’s updates or profile.

Q: From this webpage image, I will describe a specific element. Please predict their exact locations (with points). This element enables users to share content with others.
A: (660, 350)

Q: Describe the function of the element at (660, 350) on the screen.
A: This element enables users to share content with others.

Task Template

Functionality grounding

Functionality captioning

This element initiates the process of joining the community or forum, allowing users to participate in discussions or solve challenges related to working with archival materials.

This element allows users to see the number of awards a particular comment has received.

This element triggers a dropdown menu for sorting comments, with the option to select “Best”, which likely orders the comments based on their popularity or votes.

This element triggers a vote for the comment, indicating the user’s appreciation from the user interacting with it.

Mobile

Figure 3: Element functionality annotations generated by the AutoGUI pipeline for both web and mobile domains.

3.4 Improving Annotation Quality via LLM-Based Verification

The functionality annotations produced by the LLM probably contain incorrect and hallucinated samples. To improve dataset quality, we prompt LLMs to verify the annotations, inspired by works that justify the feasibility of LLM-based verification (Zheng et al., 2023; Chen et al., 2024a; Lee et al., 2023). This process presents the LLMs with the interacted element, its UI context, the UI changes induced by the interaction, and the functionality annotation generated in the previous stage. Then, the LLMs analyze the UI content changes and predict whether the interacted element aligns with the given functionality. If the LLMs determine that the interacted element fulfills the functionality given its UI context, the LLMs will grant a full score (An example in Fig. H). If not, this functionality will be seen as incorrect as this mismatch indicates that it may not accurately reflect the element’s role within the UI context.

To mitigate the potential biases in LLMs (Pannickssery et al., 2024; Bai et al., 2024), two different LLMs (i.e., Llama-3-70B and Mistral-7B-Instruct-v0.2) are employed as verifiers and prompted to output 0-3 scores (This scoring range is chosen as it empirically achieves a high verification accuracy). The scoring process is formulated as follows:
\[
score = \text{LLM}(p_{verify}, e, f, s_t, s_{t+1})
\]
where \( p_{verify} \) denotes the verification prompt (Tab. E). Only if the two scores are both 3s do we consider the functionality annotation correct (Details in Sec. A.8). While this approach seems stringent, we can make up the number of annotations through scaling.

3.5 Task Generation

After rejecting, annotating, and verifying, we obtain a high-quality UI functionality dataset containing triplets of \(\{UI\ screenshot,\ Interacted\ element,\ Functionality\}\). To convert this dataset into an instruction-following dataset for training and evaluation, we generate functionality grounding and referring tasks using diverse prompt templates. The coordinates of element bounding boxes are normalized within the range [0, 999] (see Fig. 3).

We finally annotate 2k grounding samples as a test set and 702k as a training set (The details of ensuring no overlap between the two sets can be seen in Sec. A.1). The statistics of our dataset in Tab. 2 and Sec. A.1 show that our dataset covers diverse UIs and exhibits variety in lengths and functional semantics of the annotations.

3.6 Analysis of Data Quality

Comparison with Human Annotation \( N = 145 \) samples (99 valid and 46 invalid) are randomly selected as a testbed for comparing the annotation correctness of a trained human annotator and our pipeline. Here, correctness is defined as \( Correctness = C/(N - R) \), where \( C \) and \( R \) denote the numbers of correctly annotated and rejected samples, respectively. The denominator subtracts the number of rejected samples as we are more interested in the percentage of correct samples after rejecting invalid samples. The authors rigorously evaluate the annotation results based on
Table 2: The statistics of the AutoGUI datasets. The Anno. Tokens and Avg. Words columns show the total number of tokens and the average number of words for the functionality annotations regardless of task templates. The Domains/Apps column shows the number of unique web domains/mobile Apps involved in each split.

<table>
  <tr>
    <th>Split</th>
    <th>#Tasks</th>
    <th>Anno. Tokens</th>
    <th>Avg. Words</th>
    <th>Domains/Apps</th>
    <th>Device Ratio</th>
  </tr>
  <tr>
    <td>Train</td>
    <td>702k</td>
    <td>17.9M</td>
    <td>23.1</td>
    <td>916</td>
    <td>Web: 54.6%, Mobile: 45.4%</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>2k</td>
    <td>53.4k</td>
    <td>22.5</td>
    <td>299</td>
    <td>Web: 50%, Mobile: 50%</td>
  </tr>
</table>

![Word cloud and histogram plots showing diversity of AutoGUI dataset](page_180_370_1087_246.png)

Figure 4: Diversity of the AutoGUI dataset. Left: The word cloud illustrates the ratios of the verbs representing the main intents in the functionality annotations. Right: Comparing the distributions of the annotation token numbers for our AutoGUI training split, SeeClick Web training data (Cheng et al., 2024), and Widget Captioning (Li et al., 2020a). The comparison demonstrates that our dataset covers significantly more diverse task lengths.

three criteria outlined in Fig. 1. Details can be found in Sec. B.1.

After experimenting with three runs, Tab. 3 shows that the AutoGUI pipeline achieves high correctness comparable to the trained human annotator (r6 vs. r1). Without rejection and verification (r2), AutoGUI is inferior as it cannot recognize invalid samples. Notably, simply using the rules written by the authors can improve the correctness, which is further enhanced with the LLM-aided rejector (r4 vs. r3). Moreover, utilizing the annotating LLM itself to self-verify its annotations helps AutoGUI surpass the trained annotator (r5 vs. r1). Introducing another LLM verifier (i.e., Mistral-7B-Instruct-v0.2) brings a slight increase which results from Mistral recognizing Llama-3-70B’s incorrect descriptions of how dropdown menu options work. Overall, these results justify the efficacy of the AutoGUI annotation pipeline.

Qualitative comparison (Fig. P) shows that our pipeline generates more detailed annotations which would take more time for the human annotator.

Impact of LLM Output Uncertainty Despite LLM output uncertainty, our pipeline achieves a high annotation consistency of 94.5%. LLM uncertainty affects rejection but has a minimal overall impact due to the low prevalence of invalid samples. More experimental details in the Appendix.

Table 3: Comparing AutoGUI and human annotator. AutoGUI with the proposed rejection and verification achieves correctness comparable to the trained human annotator. One LLM means Llama-3-70B and Two LLMs include Mistral-7B-Instruct-v0.2.

<table>
  <tr>
    <th>No.</th>
    <th>Annotator</th>
    <th>Rejector</th>
    <th>Verifier</th>
    <th>Correctness</th>
  </tr>
  <tr>
    <td>r1</td>
    <td>Human</td>
    <td>-</td>
    <td>-</td>
    <td>95.5%</td>
  </tr>
  <tr>
    <td>r2</td>
    <td>Llama-3-70B</td>
    <td>-</td>
    <td>-</td>
    <td>64.5%</td>
  </tr>
  <tr>
    <td>r3</td>
    <td>Llama-3-70B</td>
    <td>Rules</td>
    <td>-</td>
    <td>83.1%</td>
  </tr>
  <tr>
    <td>r4</td>
    <td>Llama-3-70B</td>
    <td>Rules+LLM</td>
    <td>-</td>
    <td>94.4%</td>
  </tr>
  <tr>
    <td>r5</td>
    <td>Llama-3-70B</td>
    <td>Rules+LLM</td>
    <td>One LLM</td>
    <td>96.0%</td>
  </tr>
  <tr>
    <td>r6</td>
    <td>Llama-3-70B</td>
    <td>Rules+LLM</td>
    <td>Two LLMs</td>
    <td><b>96.7%</b></td>
  </tr>
</table>

4 Fine-Tuning Experiments
This section validates that our dataset effectively enhances the GUI grounding capabilities of VLMs.

4.1 Experimental Settings
Evaluation Benchmarks We base our evaluation on the UI grounding benchmarks for various scenarios: FuncPred is the test split from our collected functionality dataset. This benchmark requires a model to locate the element specified by its functionality description. ScreenSpot (Cheng et al., 2024) and ScreenSpot-v2 (Wu et al., 2024) require a model to locate elements based on short instructions on mobile, desktop, and web platforms. VisualWebBench (VWB) (Liu et al., 2024a) is a comprehensive multi-modal benchmark assessing the understanding capabilities of VLMs in web sce-
Table 4: Element grounding accuracy across benchmarks. We compare the base models fine-tuned with our AutoGUI data and representative open-source VLMs. General-purpose models (Qwen-VL, SliME-8B, and Qwen2-VL) show significant performance improvements after fine-tuning with the AutoGUI data. The UI-specialized model (SeeClick) also improves when AutoGUI data is added to their fine-tuning datasets. Green text indicates gains over the base models. † denotes metrics quoted from the original benchmark paper.

<table>
  <tr>
    <th rowspan="2">Type</th>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="6">FuncPred</th>
  </tr>
  <tr>
    <th>ScreenSpot</th>
    <th>ScreenSpot-v2</th>
    <th>MOTIF</th>
    <th>VWB EG</th>
    <th>VWB AG</th>
  </tr>
  <tr>
    <td rowspan="4">General</td>
    <td>GPT-4o</td>
    <td>N/A</td>
    <td>9.8</td>
    <td>17.8</td>
    <td>20.4</td>
    <td>30.5</td>
    <td>5.6</td>
    <td>6.8</td>
  </tr>
  <tr>
    <td>Llama-3-2-Vision-Instruct</td>
    <td>11B</td>
    <td>4.9</td>
    <td>11.7</td>
    <td>11.6</td>
    <td>19.7</td>
    <td>7.0</td>
    <td>3.9</td>
  </tr>
  <tr>
    <td>SliME (Zhang et al., 2024b)</td>
    <td>8B</td>
    <td>3.2</td>
    <td>13.0</td>
    <td>13.4</td>
    <td>7.0</td>
    <td>6.1</td>
    <td>4.9</td>
  </tr>
  <tr>
    <td>Qwen-VL (Bai et al., 2023)</td>
    <td>10B</td>
    <td>3.0</td>
    <td>5.2†</td>
    <td>5.6</td>
    <td>7.8</td>
    <td>1.7</td>
    <td>3.9</td>
  </tr>
  <tr>
    <td>Qwen2-VL (Bai et al., 2023)</td>
    <td>7B</td>
    <td>38.7</td>
    <td>66.4</td>
    <td>66.9</td>
    <td>71.1</td>
    <td>55.9</td>
    <td>62.1</td>
  </tr>
  <tr>
    <td rowspan="4">UI Experts</td>
    <td>CogAgent (Hong et al., 2024)</td>
    <td>18B</td>
    <td>29.3</td>
    <td>47.4†</td>
    <td>52.1</td>
    <td>45.1</td>
    <td>55.7</td>
    <td>59.2</td>
  </tr>
  <tr>
    <td>SeeClick (Cheng et al., 2024)</td>
    <td>10B</td>
    <td>19.8</td>
    <td>53.4†</td>
    <td>54.0</td>
    <td>66.5</td>
    <td>39.2</td>
    <td>27.2</td>
  </tr>
  <tr>
    <td>Ucrobot (Wang et al., 2025)</td>
    <td>7B</td>
    <td>55.8</td>
    <td>85.9</td>
    <td>87.4</td>
    <td>92.7</td>
    <td>69.9</td>
    <td></td>
  </tr>
  <tr>
    <td>OS-ATLAS (Wu et al., 2024)</td>
    <td>7B</td>
    <td>52.1</td>
    <td>82.5</td>
    <td>84.1</td>
    <td>78.8</td>
    <td>82.6</td>
    <td>71.8</td>
  </tr>
  <tr>
    <td rowspan="4">Finetuned</td>
    <td>Qwen-VL-AutoGUI702k</td>
    <td>10B</td>
    <td>48.7 (+45.7)</td>
    <td>41.2 (+36.0)</td>
    <td>40.2 (+34.6)</td>
    <td>44.0 (+36.2)</td>
    <td>42.1 (+40.4)</td>
    <td>35.9 (+32.0)</td>
  </tr>
  <tr>
    <td>SliME-AutoGUI702k</td>
    <td>8B</td>
    <td>62.6 (+59.4)</td>
    <td>44.0 (+31.0)</td>
    <td>42.5 (+29.1)</td>
    <td>44.9 (+37.9)</td>
    <td>25.4 (+19.3)</td>
    <td>13.6 (+8.7)</td>
  </tr>
  <tr>
    <td>Qwen2-VL-AutoGUI702k</td>
    <td>7B</td>
    <td>65.0 (+62.3)</td>
    <td>80.0 (+13.6)</td>
    <td>83.2 (+16.3)</td>
    <td>72.3 (+1.2)</td>
    <td>90.3 (+34.4)</td>
    <td>70.9 (+8.8)</td>
  </tr>
  <tr>
    <td>SeeClick w/ AutoGUI702k</td>
    <td>10B</td>
    <td>50.0 (+30.2)</td>
    <td>54.2 (+0.8)</td>
    <td>54.7 (+0.7)</td>
    <td>67.0 (+0.5)</td>
    <td>56.2 (+17.0)</td>
    <td>45.6 (+18.4)</td>
  </tr>
</table>

![Line graph showing the scaling effect of the AutoGUI data. Three lines represent Qwen-VL, SLiME, and Qwen2-VL, with increasing accuracy as the amount of used AutoGUI data increases.](page_120_670_627_312.png)

Figure 5: Scaling effect of the AutoGUI data. The three general-purpose VLMs are fine-tuned with three scales of AutoGUI data. Using more data consistently enhances the grounding accuracy of the three models. Note that the grounding accuracy (Y-axis) is averaged over all the element grounding benchmarks.

Table 5: Comparing the AutoGUI functionality annotation type with existing types. Qwen-VL is separately fine-tuned with the four annotation types. Our functionality annotation leads to superior grounding accuracy.

<table>
  <tr>
    <th>Data Size</th>
    <th>Variant</th>
    <th>FuncPred</th>
    <th>MOTIF</th>
    <th>ScreenSpot</th>
  </tr>
  <tr>
    <td rowspan="3">25k</td>
    <td>w/ Elem-HTML</td>
    <td>5.3</td>
    <td>11.7</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td>w/ Condensed Func.</td>
    <td>3.8</td>
    <td>19.8</td>
    <td>4.8</td>
  </tr>
  <tr>
    <td>w/ Func. (Ours full)</td>
    <td>21.1</td>
    <td>22.5</td>
    <td>16.4</td>
  </tr>
  <tr>
    <td rowspan="3">125k</td>
    <td>w/ Elem-HTML</td>
    <td>15.5</td>
    <td>15.8</td>
    <td>17.0</td>
  </tr>
  <tr>
    <td>w/ Condensed Func.</td>
    <td>14.1</td>
    <td>23.7</td>
    <td>23.8</td>
  </tr>
  <tr>
    <td>w/ Func. (Ours full)</td>
    <td>24.6</td>
    <td>28.7</td>
    <td>27.0</td>
  </tr>
</table>

narios. We select the element and action grounding tasks from this benchmark. MOTIF (Burns et al., 2022) requires an agent to complete a natural language command in mobile Apps. Samples of the benchmarks are visualized in Fig. J. We report the grounding accuracy (%): \( \text{Acc} = \sum_{i=1}^{N} 1 (\text{pred}_i \text{ inside GT bbox}_i) / N \times 100 \) where 1 is an indicator function and \( N \) is the number of test samples. This formula denotes the percentage of samples with the predicted points lying within the bounding boxes of the target elements.

Training Details We select SliME-8B (Zhang et al., 2024b), Qwen-VL (Bai et al., 2023), and Qwen2-VL-7B (Wang et al., 2024a) as the base models and fine-tune them on 25k, 125k, and 702k samples of the AutoGUI training data to investigate how the AutoGUI data enhances their UI grounding capabilities. We fine-tune Qwen-VL and Qwen2-VL with LoRA (Hu et al., 2022) and fine-tune SliME (Zhang et al., 2024b) with only the visual encoder frozen. We also test the benefits of our dataset for a UI expert VLM, i.e., SeeClick (Cheng et al., 2024), by adding our data to its fine-tuning data. All models are fine-tuned on 8 A100 GPUs for one epoch. (More details and hyper-parameters in Sec. B.2)

Compared VLMs We compare with both general-purpose VLMs and UI expert VLMs. During the evaluation, we manually craft grounding prompts suitable for these VLMs.

4.2 Experimental Results and Analysis

A) AutoGUI functionality annotations effectively enhance VLMs’ UI grounding capabilities and achieve scaling effects. We endeavor to show that the element functionality data collected by AutoGUI contributes to high grounding accuracy. The results in Tab. 4 demonstrate that the base models embrace notable performance gains on all the benchmarks. The two general-purpose VLMs (Qwen-VL and SLiME), which perform poorly, witness huge performance increases after fine-tuning with AutoGUI data. Qwen2-VL fine-tuned with
Thought: The screen shows the home screen of a smartphone with various application icons. The search bar is at the top. The user wants to know the news in Indonesia. I can use the search bar to find the information. I need to click on the search bar first.

Expected Functionality: This element allows users to enter text to search for content on their phone.

Action: {'action_type': 'click', 'target': (600, 600)} # Wrong localization

Locate the element according to its detailed functionality description. This element allows users to enter text to search for content on their phone.

Figure 6: An example of the 2-stage planning setting used in Sec. 5 to demonstrate the potential use of AutoGUI data. The planner (a proprietary VLM, e.g., Gemini) is bad at outputting numeric coordinates when locating elements. The grounding model, finetuned with AutoGUI functionality grounding tasks, can correctly locate the task-related target element according to the expected functionality description output by the planner.

Table 6: Applying our AutoGUI dataset to 2-stage GUI agent task planning on AITW benchmark. The results show that Qwen2-VL trained with AutoGUI functionality grounding tasks can overtake the element grounding process of the proprietary models to achieve significantly higher step accuracy. Step Acc. means the percentage of correctly planned actions while Click acc. means the percentage of correctly planned click actions.

<table>
  <tr>
    <th rowspan="2">Planner</th>
    <th rowspan="2">Grounding Model</th>
    <th colspan="2">General</th>
    <th colspan="2">Install</th>
    <th colspan="2">Google Apps</th>
    <th colspan="2">Single</th>
    <th colspan="2">Webshopping</th>
    <th rowspan="2">Avg Step acc.</th>
  </tr>
  <tr>
    <th>Step acc. / Click acc.</th>
    <th>Install</th>
    <th>Step acc. / Click acc.</th>
    <th>Google Apps</th>
    <th>Step acc. / Click acc.</th>
    <th>Single</th>
    <th>Step acc. / Click acc.</th>
    <th>Webshopping</th>
    <th>Step acc. / Click acc.</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>Qwen2-VL-7B SFT w/ AutoGUI</td>
    <td>14.85 / 9.58</td>
    <td>11.17 / 5.76</td>
    <td>12.08 / 6.85</td>
    <td>21.09 / 11.24</td>
    <td>10.89 / 11.22</td>
    <td>14.01</td>
  </tr>
  <tr>
    <td>Gemini-2.0-flash-exp</td>
    <td>Qwen2-VL-7B SFT w/ AutoGUI</td>
    <td>26.37 / 18.16</td>
    <td>28.49 / 26.91</td>
    <td>30.30 / 22.88</td>
    <td>41.84 / 21.66</td>
    <td>20.23 / 22.65</td>
    <td>29.59</td>
  </tr>
  <tr>
    <td>Gemini-2.0-flash-exp</td>
    <td>Qwen2-VL-7B SFT w/ AutoGUI</td>
    <td>36.34 / 36.54</td>
    <td>50.95 / 48.95</td>
    <td>40.99 / 40.52</td>
    <td>50.95 / 48.95</td>
    <td>32.83 / 43.52</td>
    <td>39.23 (±0.73)</td>
  </tr>
</table>

our data achieves high accuracy comparable to the expert models, i.e., UGround and OS-ATLAS. Interestingly, the UI expert VLM (i.e., SeeClick) also benefits from our data, with remarkable performance gains on FuncPred and VWB.

Fig. 5 shows that the three general-purpose VLMs obtain progressively rising grounding accuracy as the AutoGUI data size scales from 25k to 702k, indicating that increasing AutoGUI data amount leads to better localization performance.

In summary, our functionality data enhances VLMs UI element grounding ability and exhibits clear scaling effects as the data size increases.

B) Our functionality annotations are effective for enhancing UI grounding capabilities. To assess the effectiveness of functionality annotations, we compare this annotation type with three types: 1) Naive element-HTML pairs, which are directly obtained from the UI source code (Hong et al., 2024) and associate HTML code with elements in specified areas of a screenshot. Examples are shown in Fig. 1. To create these pairs, we replace the functionality annotations with the corresponding HTML code snippets recorded during trajectory collection. 2) Brief functionality descriptions that are generated by prompting GPT-4o-mini2 to condense the AutoGUI functionality annotations. For example, a full description such as ‘This element provides access to a documentation category, allowing users to explore relevant information and guides’ is shortened to ‘Documentation category access’.

After experimenting with Qwen-VL (Bai et al., 2023) at the 25k and 125k scales, the results in Tab. 5 show that fine-tuning with the complete functionality annotations is superior to the other three

2https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/
types. Notably, our functionality annotation type yields the largest gain on the challenging FuncPred benchmark that emphasizes contextual functionality grounding. In contrast, the Elem-HTML type performs poorly due to the noise inherent in HTML code (e.g., numerous redundant tags), which reduces fine-tuning efficiency. The condensed functionality annotations are also inferior, as the consensus loses details necessary for fine-grained element grounding. In summary, the AutoGUI functionality annotations provide a clear advantage in enhancing UI grounding capabilities.

4.3 Grounding Failure Case Analysis

After analyzing the grounding failure cases, we identified several failure patterns in the fine-tuned models: a) difficulty in accurately locating small elements; b) challenges in distinguishing between similar but incorrect elements; and c) issues with recognizing icons that have uncommon shapes. Please refer to Sec. D.2 for details.

5 Potential Use of AutoGUI Data

We apply our dataset to a downstream GUI agent task to demonstrate how our dataset can benefit GUI agents based on proprietary VLMs. The used benchmark is ATIW (Rawles et al., 2023) which requires an agent to complete high-level user instructions on mobile apps. The metric is step accuracy, where a planned step is considered correct only if the action type and arguments match ground truths.

2-stage planning Following UGround (Gou et al., 2025), a planner model initially performs reasoning through task progress and UI content and then plans the next action. We prompt the planner to also describe the expected functionality of the target element for click actions. Next, a grounding model (Qwen2-VL-7B) trained with our functionality grounding tasks is used to locate the target according to the functionality description. See an example in Fig. 6 and additional details in Sec. C).

The results in Tab. 6 show that even the strong proprietary VLMs (e.g. Gemini) possess weak UI element grounding capability. Qwen2-VL trained with AutoGUI functionality grounding tasks can overtake the element grounding process of the proprietary models and help the planners achieve significantly higher step accuracy by correcting the target locations of the click actions.

Although this experiment is not designed to surpass expert models tailored for agent tasks, we hope it can facilitate further research of GUI agents with strong element grounding ability.

6 Conclusion

We propose AutoGUI, a scalable and automatic annotation pipeline aimed to produce massive UI element functionality annotations used to enhance UI grounding capabilities of VLMs. The pipeline prompts an open-source LLM to generate element functionalities based on the UI content changes induced by interacting with the elements. LLM-aided rejection and verification are used to guarantee high quality. Fine-tuned with the data collected by AutoGUI, the base models obtain stronger UI grounding ability and exhibit data scaling effects. We hope that AutoGUI will open up possibilities for advancing the field of UI agents.

Limitations

AutoGUI is dedicated to providing an autonomous way to collect scalable UI grounding/captioning data for training capable UI-VLMs. However, AutoGUI still encounters several limitations:

Lack of Diverse Mobile App Data. As many Apps implement anti-emulator code, it is extremely difficult to navigate through popular Apps, such as TikTok and WeChat, on Android emulators. To circumvent this issue, AutoGUI renders webpages at various resolutions, including smartphone resolution, to mimic diverse device types. Although mainstream websites, such as YouTube and Reddit, provide delicately designed webpage responsiveness for various resolutions, a number of less common websites do not possess such flexible responsiveness and distort severely when rendered at smartphone resolutions. Therefore, collecting UI data at a smartphone resolution probably leads to domain gaps between the collected data and real smartphone Apps that are not rendered with HTML.

AutoGUI is Not Indented to Record Task-Oriented Interaction Trajectories. AutoGUI randomly interacts with UIs to record transition trajectories and utilize the UI content changes to predict the functionalities of the interacted elements. Hence, the collected trajectories do not provide high-level task semantics. In other words, the AutoGUI dataset does not contain tasks that combine multiple low-level steps, such as selecting a check-in date and then a check-out date. These long-horizon tasks are usually generated by human an-
notators in the existing works (Deng et al., 2024; Rawles et al., 2023). In future work, we can also utilize capable LLMs to generate high-level tasks and then prompt the LLMs to interact with UIs according to the tasks.

AutoGUI Cannot Annotate UI Elements That Modify Content on The Internet. To avoid causing potential contamination on the Internet and bearing unexpected responsibilities, we try our best to eliminate interaction samples that manipulate sensitive elements that probably modify contents on the Internet. For example, elements used to post comments, make purchases, and enter account information are discarded. Consequently, the AutoGUI pipeline mainly annotates elements that only support read-only functionalities.

Acknowledgments

This work was supported in part by the National Key R&D Program of China (No. 2022ZD0160102), the National Natural Science Foundation of China (No. U21B2042, No. 62320106010), and in part by the 2035 Innovation Program of CAS.

References

AI@Meta. 2024. Llama 3 model card.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736.

Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cárbone, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024. Screemai: A vision-language model for ui and infographics understanding. arXiv preprint arXiv:2402.04615.

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. 2021. Uibert: Learning generic multimodal representations for ui understanding. In International Joint Conference on Artificial Intelligence.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vi: A versatile vision-language model for understanding, localization, text reading, and beyond.

Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2024. Benchmarking foundation models with language-model-as-an-examiner. Advances in Neural Information Processing Systems, 36.

Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022. A dataset for interactive vision-language navigation with unknown command feasibility. In European Conference on Computer Vision.

Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. 2024a. Is a large language model a good annotator for event extraction? Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17772–17780.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24185–24198.

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332, Bangkok, Thailand. Association for Computational Linguistics.

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hirschman, Daniel Afergan, Y. Li, Jeffrey Nichols, and Ranjitha Kumar. 2017a. Rico: A mobile app dataset for building data-driven design applications. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology.

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hirschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017b. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pages 845–854.

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichtler, Ayaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023. PaLM-e: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8469–8488. PMLR.

Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. 2024. Mobileviews: A large-scale mobile gui dataset. Preprint, arXiv:2409.14337.

Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. Co-gagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14281–14290.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2024. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pages 161–178. Springer.

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Raffailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. In 8th Annual Conference on Robot Learning.

Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Preprint, arXiv:2405.02246.

Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryan White, and Sujay Jauhar. 2023. Making large language models better data creators. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15349–15360, Singapore. Association for Computational Linguistics.

Wei Li, William Bishop, Alice Li, Chris Rawles, Fola-wiyi Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024a. On the effects of data scale on ui control agents. In Advances in Neural Information Processing Systems, volume 37, pages 92130–92154. Curran Associates, Inc.

Y. Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020a. Widget captioning: Generating natural language description for mobile user interface elements. In Conference on Empirical Methods in Natural Language Processing.

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020b. Mapping natural language instructions to mobile ui action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8198–8210.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024b. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. ArXiv, abs/2305.20050.

Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. Showui: One vision-language-action model for gui visual agent. Preprint, arXiv:2411.17465.

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Qiao. 2023. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. Preprint, arXiv:2311.07575.

Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023a. Deplot: One-shot visual language reasoning by plot-to-table translation. In The 61st Annual Meeting Of The Association For Computational Linguistics.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc.

Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. 2024a. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? In First Conference on Language Modeling.
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024b. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhoushu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024. Deepseek-vl: Towards real-world vision-language understanding. Preprint, arXiv:2403.05525.

Arjun Panickserry, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems, volume 37, pages 68772–68802. Curran Associates, Inc.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. 2024. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations.

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. Androidinthewild: A large-scale dataset for android device control. In Advances in Neural Information Processing Systems, volume 36, pages 59708–59728. Curran Associates, Inc.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. 2014. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211 – 252.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Itsev. 2022. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2words: Automatic mobile ui summarization with multimodal learning. The 34th Annual ACM Symposium on User Interface Software and Technology.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. Preprint, arXiv:2409.12191.

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2024b. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.

Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2022. Large language models are better reasoners with self-verification. In Conference on Empirical Methods in Natural Language Processing.

Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. 2024. Os-atlas: A foundation action model for generalist gui agents. Preprint, arXiv:2410.23218.

Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. 2024. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185.

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheh Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, volume 37, pages 52040–52094. Curran Associates, Inc.

Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024. Aguvis: Unified pure vision agents for autonomous gui interaction. Preprint, arXiv:2412.04454.

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023a. mplug-docowl: Modularized multimodal large language model for document understanding. Preprint, arXiv:2307.02499.

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, and Fei Huang. 2023b. UReader: Universal OCR-free visually-situated language understanding with multimodal large language model. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2841–2858, Singapore. Association for Computational Linguistics.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2024. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations.

Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2025. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In Computer Vision – ECCV 2024, pages 240–255. Cham. Springer Nature Switzerland.

Renrui Zhang, Jianming Han, Chris Liu, Aojuan Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. 2024a. LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations.

Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. 2024b. Beyond llava-hd: Diving into high-resolution large multimodal models. Preprint, arXiv:2406.08487.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyun Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.

A Appendix

<table>
  <tr>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>A</td>
    <td>Details of the AutoGUI Pipeline</td>
    <td>14</td>
  </tr>
  <tr>
    <td></td>
    <td>A.1 Extra Statistics of the AutoGUI Dataset</td>
    <td>14</td>
  </tr>
  <tr>
    <td></td>
    <td>A.2 License of The Used Artifacts</td>
    <td>14</td>
  </tr>
  <tr>
    <td></td>
    <td>A.3 Recording Interaction Trajectories on Web</td>
    <td>14</td>
  </tr>
  <tr>
    <td></td>
    <td>A.4 Recording Interaction Trajectories on Android Devices</td>
    <td>14</td>
  </tr>
  <tr>
    <td></td>
    <td>A.5 Functionality Annotation Details</td>
    <td>15</td>
  </tr>
  <tr>
    <td></td>
    <td>A.6 Details of Rejecting Invalid Samples via Hand-Written Rules</td>
    <td>17</td>
  </tr>
  <tr>
    <td></td>
    <td>A.7 Details of Rejecting Invalid Samples via LLMs</td>
    <td>17</td>
  </tr>
  <tr>
    <td></td>
    <td>A.8 Details of LLM-Based Verification</td>
    <td>19</td>
  </tr>
  <tr>
    <td></td>
    <td>A.9 Details of Grounding/Captioning Task Generation</td>
    <td>19</td>
  </tr>
  <tr>
    <td>B</td>
    <td>Implementation Details</td>
    <td>19</td>
  </tr>
  <tr>
    <td></td>
    <td>B.1 Human Evaluation Details</td>
    <td>19</td>
  </tr>
  <tr>
    <td></td>
    <td>B.2 Fine-Tuning Details</td>
    <td>19</td>
  </tr>
  <tr>
    <td></td>
    <td>B.3 Samples of Benchmarks</td>
    <td>22</td>
  </tr>
  <tr>
    <td>C</td>
    <td>Details for Potential Use of AutoGUI Dataset</td>
    <td>22</td>
  </tr>
  <tr>
    <td>D</td>
    <td>Additional Experimental Analysis</td>
    <td>22</td>
  </tr>
  <tr>
    <td></td>
    <td>D.1 Growing Grounding Performance Brought by Scaling Data Size</td>
    <td>22</td>
  </tr>
  <tr>
    <td></td>
    <td>D.2 Case Analysis on FuncPred Test Split</td>
    <td>23</td>
  </tr>
  <tr>
    <td></td>
    <td>D.3 Case Analysis on MoTIF Test Split</td>
    <td>23</td>
  </tr>
  <tr>
    <td>E</td>
    <td>Potential Societal Impact</td>
    <td>23</td>
  </tr>
</table>
The appendix comprises the following sections:
Section A: Details for implementation details for the autonomous annotation pipeline, including dataset statistics, visualized annotation pipeline, and LLM prompts.
Section B: Details for model implementation and training.
Section C: Details of potential use of the AutoGUI dataset.
Section D: Additional experimental analysis including analysis of successful and failure cases on two benchmarks.
Section E: Limitations and Potential Societal Impact.

A  Details of the AutoGUI Pipeline

A.1 Extra Statistics of the AutoGUI Dataset
Fig. A visualizes the verb-noun statistics of the AutoGUI dataset, highlighting its extensive coverage of diverse UI functionalities. Fig. B lists the top 50 most frequent top-level domains in the AutoGUI dataset, showing that the AutoGUI dataset involves a broad spectrum of real-world scenarios, including technology (e.g., apple.com), entertainment (e.g., tiktok.com), office (e.g., outlook.com), news (e.g., medium.org), and finance (e.g., paypal.com).

The approach to avoiding overlap between train and test data: Since our focus is on annotating contextual functionality for GUI elements, we define two elements as distinct if they serve different functions within their respective contexts. For example, two "magnifier" buttons on the same GUI might have different roles—one for zooming in and the other for searching. To ensure no contamination, we investigated whether the test elements appeared in the training set by checking if bounding box overlapping occurred on the same GUIs. After this analysis, we found no such overlap.

A.2 License of The Used Artifacts
The licenses of the data sources on which the AutoGUI dataset is built are listed in Tab. A. These sources are all allowed to be used for academic research.

A.3 Recording Interaction Trajectories on Web
Interactive Crawler for Common Crawl We design an in-house web crawler that interacts with most elements rendered on the web page. In contrast with existing methods which contain information for elements on the initial static web page for a given URL, our crawler randomly interacts with a rendered web page for multiple steps within a given action horizon \( T_{act} \) to collect UI data with abundant functional semantics. Fig. C compares the proposed AutoGUI and the existing annotation methods. We empirically set \( T_{act} = 10 \) in all our recordings. Therefore, our interactive crawler could collect functionality of elements that are not visible to static pages, including nested drop-down menus, date and location selectors, and secondary menus.

Data Source and Data Format To incorporate a wide basis of web pages, we first obtain a list of the top-200 most visited domains \( ^3 \) and manually remove content delivery network (CDN) and not safe for work (NSFW) sites. We use URLs in this curated list as seeds to query the Common Crawl index \( ^4 \) to find additional URLs with maximum sub-domain and path diversity. Querying URLs from the Common Crawl index ensures that our crawler respects each site’s robots.txt file, making the dataset collection process legally safe. By obeying the directives in robots.txt, we avoid potential legal issues associated with unauthorized web scraping. For each web page, we collect the following data:

• Screenshot image of the rendered page
• Accessible Tree (AXTree) text representing the page’s accessibility structure
• HTML source code of the page
• Accessible Node (AXNode) text for the specific element our crawler interacted with at each step

A.4 Recording Interaction Trajectories on Android Devices
We also implement an in-house crawler that interacts with multiple emulated Google Pixel phones. The phones are reset to different starting UIs before a script randomly interacts with these phones to record trajectories. To improve data diversity, the starting UIs include the home page, drop-down panel, settings page, and Apps drawer.

Similar to webpage HTML, mobile phone UIs are rendered with XML code, which is cleaned and converted to AXTree-like content before being used to annotate functionalities.

\( ^3 \)https://tranco-list.eu/
\( ^4 \)https://index.commoncrawl.org/
Figure A: Diversity of the verb-noun phrases of the AutoGUI dataset. The top 10 verbs and their top 5 following nouns are displayed. This diagram shows that our dataset contains diverse tasks that involve various UI functions.

Table A: License or terms for use and/or distribution of the used artifacts in this work.

<table>
  <tr>
    <th>Artifacts</th>
    <th>License</th>
    <th>URLs containing Term-of-Use or other license information</th>
  </tr>
  <tr>
    <td>Common Crawl</td>
    <td>CC BY</td>
    <td>https://commoncrawl.org/terms-of-use</td>
  </tr>
  <tr>
    <td>AndroidControl Trajectories</td>
    <td>Apache 2.0</td>
    <td>https://github.com/google-research/google-research/tree/41e4f1cbe1db648cb518a60501f638d9c8b25f2/android_control</td>
  </tr>
  <tr>
    <td>Mobile Views Trajectories</td>
    <td>MIT</td>
    <td>https://huggingface.co/datasets/nlinTeam/MobileViews</td>
  </tr>
</table>

A.5 Functionality Annotation Details

The AutoGUI pipeline utilizes UI content changes to predict the functionalities of the interacted elements. For interactions that manipulate the existing UI, the pipeline analyzes differences in the AXTrees to annotate functionalities. Conversely, when interactions result in navigation to a new UI, the pipeline examines changes in UI descriptions to guide the annotation process. Details on these methodologies are outlined below:

UI manipulation case We use a file-comparison library, DiffLib, to generate line-by-line differences of the AXtrees before and after interactions. To balance efficiency with annotation integrity, we limit the differences to 250 lines. In addition to the standard markings by DiffLib—addition, deletion, and unchanged status—we incorporate two additional change markers: ‘Repositioning’ and ‘Attribute Update’. These markers provide detailed information about UI content changes, essential for representing realistic structural variations. For example, without the attribute update marker, a clicked menu icon would erroneously appear as both deleted and added in the difference output, despite the menu element remaining in place. An example of this case is shown in Fig. D. The used prompt is shown in Tab. B.

UI navigation case When an interacted element causes navigation to a new UI, the resultant changes are often extensive, potentially exceeding the con-
Table B: The functionality annotation prompt used in the AutoGUI pipeline in UI manipulation cases.

(Requirements for annotation)
Objective: As an Internet expert, your task is to describe the usage and functionality of a webpage element based on the changes observed in the webpage contents before and after interacting with the element.
Instructions:
1. You will be shown line-by-line differences between the webpage content before and after interacting with the element. Here’s what each prefix indicates:
   Unchanged: Lines that are identical before and after the interaction.
   Added: New lines that appear after the interaction.
   Deleted: Lines that were present before the interaction but removed afterward.
   Renaming: Lines indicating elements that were renamed due to the interaction.
   Attribute Update: Lines showing elements whose attributes were updated during the interaction.
   Response: Elements that respond to user actions.
2. You MUST thoroughly analyze the changes in webpage content (Added, Deleted, Unchanged lines) caused by interacting with the element, present a detailed reasoning process elucidating how the element affects the webpage, and finally summarize the element’s overall purpose based on your analysis
3. Avoid detailing very specific functionality of the webpage element. Instead, focus on describing its broader impact on the webpage experience. For example, if interacting with a “Photo” button reveals a photo gallery menu, do not catalog the subsequent webpage changes in exhaustive detail.
4. Your output MUST follow this format:
   Reasoning: (Examine the webpage variation carefully to figure out how the interacted element changes the webpage)
   Summary: This element ... (Provide a concise high-level description of the element’s function. This description should contain the meaningful feature of the element in its context.)
5. Avoid mentioning specific elements from the webpage before interaction in the Summary. Instead, focus directly on the outcome of the interaction. For instance, rather than detailing a transition from one element to another, like “This element switches the focus from the ‘Destination’ input field to the ‘Check-in Date’ field, providing a date picker,” simplify it to “This element triggers a date picker for users to select their check-in date.” This approach keeps the description concise and focused on functionality, eliminating unnecessary details.
6. For simplicity, you MUST not mention the text of the element in the summary.
Now it’s your turn.

Element interaction: After clicking a <link> element named “Community submenu” (an example of the interaction target), the webpage exhibits the following variations:

(UI content changes)
Unchanged button 'Extend submenu' expanded: False
Unchanged button 'Extend'
Unchanged button 'Learn submenu' expanded: False
Unchanged StaticText 'Learn'
Before Attribute Update button 'Community submenu' expanded: False
After Attribute Update button 'Community submenu' focused: True expanded: True
Unchanged StaticText 'Community'
Added link 'Make WordPress'
Added link 'Photo Directory'
Added link 'Events in the Future'
Added link 'Job Board'
Unchanged button 'About submenu' expanded: False
Unchanged button 'About'
Unchanged button 'Open Search' hasPopup: dialog
Unchanged link 'Get WordPress'

Now please think step-by-step, examine the content variation in detail, and finally summarize the high-level usage of the interacted element. Do follow the output format
“Reasoning: ... Summary: This element ...”.

(LLM response)
Reasoning: After interacting with the “Community submenu” button, the webpage content changes significantly. The button’s “expanded” attribute is updated to “True”, indicating that the button is now expanded. Additionally, the button gains focus, as indicated by the “focused: True” attribute. The most notable change, however, is the appearance of new. These new elements suggest that the button has triggered the display of a submenu or dropdown list related to the “Community” theme.
Summary: This element reveals a submenu of community-related links and resources.
text limit of an LLM and complicating the analysis of these changes. To handle this situation, UI description changes are used to predict functionalities. Concretely, the LLM is initially prompted to describe the UIs before and after interaction given UI AXTrees as input. Subsequently, the LLM then uses these descriptions to analyze content changes and predict the functionality of the interacted element. The description length of the AXTree is limited to 150 lines. An illustration of this process is shown in Fig. E. The corresponding prompt is detailed in Tab. C.

A.6 Details of Rejecting Invalid Samples via Hand-Written Rules

To clarify the hand-written rules used in the process of removing invalid samples: (1) Removing blank GUIs. We remove blank GUIs by verifying whether the accessibility tree contains more than one node. If no nodes are present, the GUI is considered blank. (2) Removing UIs containing elements indicating content loading. GUIs containing elements indicative of content loading (e.g., keywords such as "loading", "please wait", or "refreshing") are excluded. These keywords typically suggest that the content has not fully loaded and may affect the validity of the sample. (3) Removing interaction targets outside of screens. Occasionally, part of the UI, including the interacted element, may fail to be captured. We filter out GUIs where interaction targets appear outside of the visible screen area. This is determined by checking whether the interacted element exists within the bounds of the recorded accessibility tree. Note that these rules are designed mainly for the domains from which we collected GUI metadata. Nevertheless, one can extend the rules flexibly according to the noise characteristics of new domains.

A.7 Details of Rejecting Invalid Samples via LLMs

Rejection process To eliminate invalid samples before functionality annotation, the AutoGUI pipeline prompts the annotating LLM to also determine the validity of samples by analyzing the predictability of the UI content changes. The LLM evaluates each sample against three criteria: 1) Explicitness of Changes: This measures how clearly the changes indicate the element’s functionality. Changes that directly suggest functionality receive higher scores, while vague or irrelevant changes are not scored. 2) Relevance of Changes: This criterion assesses the significance of the modifications in relation to the element’s intended function. Highly related modifications obtain a high score. No scores for irrelevant or unrelated content changes. 3) Predictability of Outcome: This involves determining how anticipated the interaction outcome is based on the changes, considering common web conventions and user experience principles. Highly predictable changes obtain a high score, whereas moderate, unexpected, or counter-intuitive outcomes receive no
Table C: The functionality annotation prompt used in the AutoGUI pipeline in UI navigation cases. This example shows how the LLM

Requirements for annotation
Objective: Your mission, as a digital navigation specialist, is to deduce and articulate the function and usage of a specific webpage element. This deduction should be based on your analysis of the differences in webpage content before and after interacting with said element.
Instructions:
1. You will be given descriptions of a webpage before and after interaction with an element. Your primary task is to meticulously analyze the differences in content resulting from the interaction to understand what the functionality of the element is in the webpage context.
2. You must present a detailed reasoning process before finally summarizing the element’s overall purpose based on your analysis.
3. Prioritize examining changes in the webpage’s regional context over individual element variations. This approach will provide a more holistic view of the element’s impact on the webpage.
4. You should emphasize on the main content changes and pay less attention to less meaningful regions, such as headers, navigation bars, and footers.
5. Your response MUST follow this format:
   Reasoning: (Examine the webpage variation carefully to figure out how the interacted element changes the webpage)
   Summary: This element ... (Provide a high-level description of the element’s functionality. This description should contain the meaningful feature of the element in its context.)
6. Avoid mentioning specific elements from the webpage prior to interaction in the Summary. Instead, focus directly on the outcome of the interaction. For instance, rather than detailing element changes, like “This element triggers the disappearance of the header and language selector elements and the emergence of a login form”, simplify it to “This element triggers the display of a login page.” This approach keeps the summary concise and focused on functionality, eliminating unnecessary details.
7. The final summary must emphasize the distinctive functionality of the element and not be too vague. For example, “This element redirects the user to a specific website, allowing them to access its content and services” is too broad and lacks meaningful detail. Instead, it should precisely describe how the element provides users with unique value.
Now it’s your turn.
The description of the initial webpage:

An example description of the UI before the interaction)
Region 1 (Header): Navigation and login functionality
Region 2 (Main Content): Announcing updates to Instagram Direct Message, providing a blog post with a title, date, and content
Region 3 (Footer): Additional links and buttons (less meaningful region, summarized in a single sentence)
Overall Functionality: A blog post webpage announcing updates to Instagram Direct Message, providing a title, date, and content, with navigation and login functionality at the top.
After clicking a <link> element named “<an example of the interaction target>”, we jump to a new webpage.
The description of the new webpage is:

An example description of the UI after the interaction)
Region 1 (Header): Displays the webpage title and heading.
Region 2 (Login Form): Allows users to log in to their Facebook account by providing their email address or phone number and password.
Region 3 (Forgot Password): Provides a link to recover a forgotten account password.
Region 4 (Create New Account): Offers an option to create a new Facebook account.
Overall Functionality: Enables users to access their existing Facebook account or create a new one, providing a gateway to the social media platform.

Now please think step-by-step, examine the content variation in detail, and finally summarize the high-level usage of the interacted element. Do follow the output format “Reasoning: ... Summary: This element ...”.

LLM responded
Reasoning: Upon examining the webpage variations, I notice that the initial webpage is a blog post announcing updates to Instagram Direct Message, with navigation and login functionality at the top. After clicking the “<link>” element named “<an example of the interaction target>”, the webpage transitions to a new page with a Facebook title and heading. The main content of the new webpage is a login form, allowing users to log in to their Facebook account. Additionally, there are options to recover a forgotten password and create a new Facebook account.
Summary: This element redirects the user to a Facebook login page, allowing them to access their existing account or create a new one.
A.8 Details of LLM-Based Verification

Verification process To improve the quality of functionality annotations, the AutoGUI pipeline prompts two LLMs (i.e.g., Llama-3-70B and Mistral-7B-Instruct-v0.2) as verifiers to assign scores to samples based on how well the target elements adhere to their functionality annotations. The LLMs receive as the input a) the target element along with its surrounding UI content (up to 20 lines), b) the functionality annotation of this element, and c) the outcome of interacting with the element, either being the UI line-by-line differences (at most 250 lines) in manipulation cases or the UI description after the interaction in navigation cases. Given these inputs, the two LLMs generate two responses containing a score. Samples that do not achieve two full scores are discarded for higher quality of the AutoGUI dataset. The used prompt is shown in Tab. E and an example is illustrated in Fig. H.

A.9 Details of Grounding/Captioning Task Generation

After collecting the element-functionality pairs, the AutoGUI pipeline converts these pairs into functionality grounding and captioning tasks. A functionality grounding task requires a VLM to output point coordinates of the element fulfilling the given functionality, while a captioning task demands that the VLM articulate a functionality description for an element, given its coordinates. It is important to note that each element-functionality pair is utilized to generate both a grounding task and a captioning task.

B Implementation Details

B.1 Human Evaluation Details

To justify the efficacy of the AutoGUI pipeline, we conducted a comparative evaluation of annotation correctness between a trained human annotator and the AutoGUI system. The human annotator was a graduate student proficient in using digital devices, ensuring familiarity with diverse user interfaces.

We selected a set of 30 invalid samples, each showcasing a variety of element functionalities, to prepare the annotator for the annotation process. These functionalities included drop-down menu expansions, menu item selections, date-pickers, filtering options, pop-up modals, webpage navigation, and zooming in/out buttons. The purpose of this selection was to expose the annotator to a broad spectrum of potential UI interactions, enhancing their ability to accurately assess element functionality based on UI content changes.

During the training phase, we provided the annotator with detailed guidelines, including three specific criteria outlined in Fig I, to ensure the clarity and correctness of their annotations. Additionally, we incorporated 15 invalid samples to instruct the human annotator on how to identify and exclude these cases during the evaluation process. These invalid samples encompassed scenarios such as incompletely loaded UIs, network failure incidents, login restrictions, and UIs displaying inappropriate content.

Following the training stage, the human annotator evaluated a total of 146 samples. Remarkably, the annotator successfully identified all invalid samples, achieving an overall annotation correctness rate of 95.5%. The few incorrect annotations were categorized as such due to vagueness or instances of hallucination, where the descriptions did not accurately reflect the UI elements.

B.2 Fine-Tuning Details

Qwen-VL-Chat (Bai et al., 2023), SliME (Zhang et al., 2024b), and Qwen2-VL-7B (Wang et al., 2024a) are selected as the base models in the experiments. To investigate the scaling effects of our dataset, 25k, 125k, and the entirety of the 702k samples in the training split are used as training data in the three scaling experiments. For the first two smaller-scale experiments, a subset of the 702k data is randomly sampled.

Pilot experiments find that the non-UI training data (i.e., LLaVA-instruct-150k and the Cauldron) significantly outnumber the 25k and 125k UI training data, resulting in data imbalance that biases the trained UI-VLM towards the general Q&A tasks in the non-UI data and leads to inferior UI grounding performance. To tackle this issue, the 25k/125k samples are resampled to the same number of the non-UI training data to enable the UI-VLM to acquire more supervising signals from the UI data. This resampling approach is not employed in the 702k experiment as this experiment does not encounter the imbalance issue.

We train our models based on the HuggingFace Transformers\footnote{https://huggingface.co/docs/transformers/index} and the PEFT library\footnote{https://huggingface.co/docs/peft/index}. The training
Table D: The rejection prompt used in the AutoGUI pipeline in UI manipulation cases. This example shows how the LLM assigns a low score to a sample that exhibits meaningless and unpredictable UI content changes.

(Requirements for rejection)
Your primary objective is to determine whether the changes in the webpage’s content are sufficient for predicting the functionality of the webpage element causing these changes after being interacted with.
Instructions:
1. You will be shown the outcome (webpage changes) resulting from interacting with the element. The outcome can take one of two forms: changes to the webpage description, or line-by-line differences. For the latter form, here’s what each prefix indicates:
   Unchanged: Lines that are identical before and after the interaction.
   Added: New lines that appear after the interaction.
   Deleted: Lines that existed before the interaction but removed afterward.
   Renaming: Lines indicating elements that were renamed due to the interaction.
   Attribute Update: Lines showing elements whose attributes were updated during the interaction.
   Repositioned: Elements that were moved to a different part of the webpage.
2. Analyze the provided outcome and provide detailed reasoning for whether this outcome helps to predict the element’s functionality, considering the following stringent criteria:
   1) Explicitness of Changes: Rate how directly the changes suggest the element’s functionality. Score 1-3 for clear, unambiguous changes. Clearer changes obtain a higher score. No scores for vague, meaningless, or non-specific changes.
      Positive Example: A button labeled “Show More” that, upon interaction, clearly adds new content below it. The direct addition of content clearly indicates a content expansion functionality. Score: 3
      Negative Example: After clicking a “Details” button, the page layout changes subtly without adding relevant information or altering content in a meaningful way. The changes do not clearly relate to the button’s presumed functionality. Score: 0
   2) Relevance of Changes: Evaluate significance of changes in relation to the button to the element’s intended function. Score 1-3 for changes that enhance understanding of the element’s role. Highly related modifications obtain a high score. No scores for irrelevant or unrelated content changes.
      Positive Example: Clicking on a “Contact Us” button opens a form to fill out, which is highly relevant to the button’s intended functionality. Score: 3
      Negative Example: Clicking on a “View Profile” link leads to a page refresh without displaying the profile or any related information, making the change irrelevant to the link’s intended function. Score: 0
   3) Predictability of Outcome: Assess how anticipated the interaction outcome is based on the changes, considering common web conventions and user experience principles. Score 1-3 for highly predictable outcomes. Highly predictable changes obtain a high score. No scores for outcomes that are moderate, unexpected, or counterintuitive.
      Positive Example: Clicking or hovering over a “Help” icon reveals a tooltip with information, an outcome that users can easily predict given the icon’s universal symbol for help. Score: 3
      Negative Example: Hovering over a “Settings” icon changes its color but does not display any settings options, tooltips, or lead to a settings page, making the outcome unpredictable and the icon’s purpose unclear. Score: 0
   Provide a detailed analysis based on these criteria, ending with an overall score between 0-9.
4. Your output MUST follow this format:
Reasoning:
- Explicitness of Changes: [Detailed reasoning and score]
- Relevance of Changes: [Detailed reasoning and score]
- Predictability of Outcome: [Detailed reasoning and score]
Overall score: <score>[criterium 1 score] + [criterium 2 score] + [criterium 3 score] = [total score]</score> (output an additive equation enclosed with a pair of <score></score>)
Now it’s your turn.
The interacted element is a <button> element named “Cancel” (interaction target)
Outcome:
(An example of interaction outcome)
After clicking a <button> element named “Cancel”, the webpage exhibits the following variations:
Unchanged StaticText: ‘Email Address’
Unchanged textbox ‘Email Address’ required: True
Unchanged button ‘Submit’
Unchanged StaticText ‘Submit’
Before Action: StaticText button ‘Cancel’
After Attribute Update button ‘Cancel’ focused: True
Unchanged StaticText ‘Cancel’
Unchanged contentinfo
Unchanged StaticText ‘Your contact information is used to deliver requested updates or to access your subscriber preferences.’
Now please think step-by-step, examine the content variation in detail, and finally calculate scores according to the criteria. Do follow the output format “Reasoning: ... Overall score: <score>...<score>”.

(LLM response)
Reasoning:
- Explicitness of Changes: The changes resulting from interacting with the “Cancel” button are mostly unchanged elements, except for the “Cancel” button itself, which now has a “focused: True” attribute. This change is not directly related to the button’s functionality, as it does not provide clear insight into what the button does when clicked. Score: 1
- Relevance of Changes: The changes do not provide any significant information about the button’s intended function. The button’s name “Cancel” suggests that it might cancel some action or process, but the changes do not reveal what exactly is being canceled or how the button’s functionality relates to the surrounding elements. Score: 1
- Predictability of Outcome: Given the name “Cancel” and the context of a form with an “Email Address” field and a “Submit” button, a user might expect the “Cancel” button to reset the form, clear the input field, or abort a submission process. However, the outcome does not provide any clear indication of what happens when the button is clicked. The changes are not predictable based on common web conventions and user experience principles. Score: 1
Overall score: <score>1 + 1 + 1 = 3</score>
Table E: The self-verification prompt used in the AutoGUI pipeline in UI manipulation cases. This example shows how the LLM assigns a low score to the incorrect functionality.

(Requirements for self-verification)
Given the following inputs:
1) Webpage content: This input represents the hierarchical structure of a webpage’s elements, emphasizing semantic information and relationships. Each node in the tree includes details such as the element’s role (e.g., button, link, heading), relevant attributes (e.g., expanded), and hierarchical relationships with other elements.
2) Task Description: This describes the action a user intends to perform (such as submitting a form, navigating to a particular section, or adjusting settings) or the information they seek (such as a specific content piece or form field). It also introduces a candidate element for evaluation and then presents the webpage changes caused by interacting with this element. Your task is to assess whether this element effectively facilitates the specified user action.
Your job is to:
1) Analyze the provided webpage content to understand the structure and semantics of the webpage’s elements.
2) Evaluate the Candidate Element: Determine the suitability of the specified candidate element for the described action. Consider the element’s role, attributes, and position within the hierarchy. Your evaluation should be grounded in how well these aspects align with the required functionality for the user’s intended action.
3) Score the Element: Assign a score ranging from 0 to 3, enclosed within <score></score> tags. This score should reflect the degree to which the candidate element meets the action’s requirements.
0: The element does not support the action in any capacity.
1: The element provides minimal support for the action.
2: The element supports the action but with limitations.
3: The element fully supports the action without significant limitations.
4) Provide Reasoning: Before proceeding to scoring, offer a detailed explanation of your reasoning. This should cover your analysis of the webpage content, the relationship between the candidate element and the specified action, and how these factors informed your scoring decision.
5) Format for Your Answer:
Reasoning: Provide a comprehensive analysis covering the webpage’s insights, the relationship between the specified action and the candidate element, and the rationale behind your scoring decision.
Score: <score>[Your Score Here]</score>
(Ain-context exemplar)
Example:
Webpage content:
[0] RootWebArea ‘Rental Cars at Low, Affordable Rates’
[1] dialog ‘Vehicle Class’ modal: True
[2] radiogroup ‘Vans’
[3] radio ‘Minivans’ checked: false
[4] radio ‘Passenger Vans’ checked: false
[5] radio ‘Cargo Vans’ checked: false
[6] button ‘Cancel’
[7] button ‘Remove Filter’
Task Description: Please identify the target element. The element helps users narrow down their vehicle choices to minivans specifically
Candidate element: [3] radio ‘Minivans’
After interacting with the candidate element, the webpage exhibits these changes:
Upon clicking the ‘Minivans’ <input> element, a new “Remove Filter” button is added to the dialog modal. The radio buttons for different vehicle classes remain unchanged, but the “Minivans” radio button is checked after the interaction.
Reference response:
Reasoning: The provided webpage content outlines a clear hierarchical structure for selecting vehicle types on a car rental webpage, categorized into sections like Cars, Vans, and Trucks, each with its own set of options represented as radio buttons.
The task involves narrowing vehicle choices to minivans. The candidate element is part of the ‘Vans’ radiogroup on a car rental webpage. This directly supports the user’s action of narrowing choices to minivans. The element’s interaction leads to its checking and introduces a “Remove Filter” button. Its role, position, and functionality support the user’s intent.
Score: <score>3</score>
(The interaction task)
Now it’s your turn.
Current webpage content: (A UI content example)
[0] RootWebArea ‘X for Business | X tips, tools, and best practices’ focused: True
[1] link ‘Get started in your ads account’
[2] link ‘Campaigns overview’
[3] link ‘Editing and optimization’
[4] link ‘Measurements’
[5] heading ‘What’s happening’
[10] StaticText ‘Spread your message, grow your audience, and get more traffic with Twitter Ads.’
=> [11] link ‘Learn more’ (The interaction target)
[13] heading ‘Twitter is where people come to discover what’s happening.’
[14] heading ‘Discover what’s happening on Twitter being sent each day…’
[15] StaticText ‘Whether you’re a CEO or someone building their personal brand…’
[16] link ‘Get started with Twitter for Business’
[17] button ‘Get started with Twitter for Business’
[18] heading ‘What’s new on Twitter Business Outlook’
[20] link ‘Go to Twitter’
Task Description: Please identify the target element. This element redirects users to a non-existent page, providing a custom error message and alternative content instead of the expected information. (The functionality annotation of the element)
Candidate element: [11] link ‘Learn more’ (The interaction target)
(The interaction outcome)
After interacting with the candidate element, we navigate to a new webpage that contains these contents:
Region 1 (Header): Contains the website’s logo and navigation links.
Region 2 (Error Message): Displays an error message indicating that the page does not exist, accompanied by a humorous image and a link to potentially related content.
Region 3 (Footer): Provides links to various policy and information pages, as well as copyright information.
Overall Functionality: This webpage serves as a custom 404 error page, providing a lighthearted and engaging way to notify users that the requested page is not available.
Now please do your job. Your answer:

(LLM response)
Reasoning: The candidate element, a link labeled ‘Learn more’, is part of the webpage’s main content, surrounded by descriptions and headings related to Twitter for Business. However, upon interacting with this element, the webpage exhibits significant changes, redirecting users to a custom 404 error page with a humorous image and a link to potentially related content. In fact, the element’s interaction leads to a completely different webpage with an error message and alternative content, which is unrelated to the initial webpage’s purpose. The element’s role, position, and functionality do not align with the user’s intended action.
Score: <score>0</score>
Table F: The training hyper-parameters used for fine-tuning Qwen-VL in the experiments.

<table>
  <tr>
    <th>Hyper-Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Global batch size</td>
    <td>128</td>
  </tr>
  <tr>
    <td>#GPUs</td>
    <td>8</td>
  </tr>
  <tr>
    <td>LoRA Rank</td>
    <td>64</td>
  </tr>
  <tr>
    <td>LoRA Alpha</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>3e-5</td>
  </tr>
  <tr>
    <td>weight decay</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>ADAM Beta2</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>Warm-up ratio</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>LR scheduler</td>
    <td>Cosine</td>
  </tr>
  <tr>
    <td>Model max length</td>
    <td>768</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>ViT + LLM</td>
  </tr>
  <tr>
    <td>DeepSpeed</td>
    <td>ZeRO-2</td>
  </tr>
  <tr>
    <td colspan="2">Trainable params: 234,500,864</td>
  </tr>
  <tr>
    <td>#Parameters</td>
    <td>All params: 9,891,436,032<br>Trainable%: 2.3707</td>
  </tr>
  <tr>
    <td>Data type</td>
    <td>BFloat16</td>
  </tr>
</table>

Table G: The training hyper-parameters used for fine-tuning SliME in the experiments.

<table>
  <tr>
    <th>Hyper-Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Global batch size</td>
    <td>128</td>
  </tr>
  <tr>
    <td>#GPUs</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>3e-5</td>
  </tr>
  <tr>
    <td>weight decay</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>ADAM Beta2</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>Warm-up ratio</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>LR scheduler</td>
    <td>Cosine</td>
  </tr>
  <tr>
    <td>Model max length</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>Frozen module</td>
    <td>ViT</td>
  </tr>
  <tr>
    <td>DeepSpeed</td>
    <td>ZeRO-2</td>
  </tr>
  <tr>
    <td colspan="2">Trainable params: 7535796224</td>
  </tr>
  <tr>
    <td>#Parameters</td>
    <td>All params: 8364644352<br>Trainable%: 90.09</td>
  </tr>
  <tr>
    <td>Data type</td>
    <td>BFloat16</td>
  </tr>
</table>

Table H: The training hyper-parameters used for fine-tuning Qwen2-VL in the experiments.

<table>
  <tr>
    <th>Hyper-Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Global batch size</td>
    <td>128</td>
  </tr>
  <tr>
    <td>#GPUs</td>
    <td>8</td>
  </tr>
  <tr>
    <td>LoRA Rank</td>
    <td>128</td>
  </tr>
  <tr>
    <td>LoRA Alpha</td>
    <td>256</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>3e-5</td>
  </tr>
  <tr>
    <td>weight decay</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>ADAM Beta2</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>Warm-up ratio</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>LR scheduler</td>
    <td>Cosine</td>
  </tr>
  <tr>
    <td>Model max length</td>
    <td>2048</td>
  </tr>
  <tr>
    <td>Frozen module</td>
    <td>ViT</td>
  </tr>
  <tr>
    <td>DeepSpeed</td>
    <td>ZeRO-0</td>
  </tr>
  <tr>
    <td colspan="2">Trainable params: 322,961,408</td>
  </tr>
  <tr>
    <td>#Parameters</td>
    <td>All params: 8,614,337,024<br>Trainable%: 3.75</td>
  </tr>
  <tr>
    <td>Data type</td>
    <td>BFloat16</td>
  </tr>
</table>

configurations are shown in Tab. F, Tab. G, and Tab. H.

Fine-tuning Qwen-VL-AutoGUI1702k, SLiME-AutoGUI1702k, Qwen2-VL-7B-AutoGUI1702k, SeeClick w/ AutoGUI1702k, UGround w/ AutoGUI1702k consumed approximately 25 hours, 36 hours, 20 hours, 25 hours, 46 hours, respectively.

The framework used to fine-tune Qwen-VL and SeeClick w/ AutoGUI1702k is the SeeClick codebase (Cheng et al., 2024); The framework used to fine-tune SLiME-AutoGUI1702k is the SLiME codebase (Zhang et al., 2024b); The framework used to fine-tune Qwen2-VL-7B-AutoGUI1702k and UGround w/ AutoGUI1702k is LLaMA-Factory (Zheng et al., 2024).

B.3 Samples of Benchmarks

For clarity, the benchmarks’ samples are visualized in Fig. J.

C Details for Potential Use of AutoGUI Dataset

We mainly conduct 2-stage planning on the AITW (Rawles et al., 2023) benchmark to assess the benefits of our AutoGUI data on downstream agent tasks.

As illustrated in Fig. 6, a planner is utilized to conduct reasoning and step prediction while a grounding model locates target elements for the actions that require targets (click, long-press, and hover). For other actions like swipe, back, home, and input-text, the grounding model is not involved. As this experiment requires the planner to describe the expected functionality of target elements, we use strong proprietary VLMs, such as GPT-4o-mini and Gemini-2.0 as the planners. Expert models, such as OS-ATLAS (Wu et al., 2024), are not used as they typically have lost general instruction following capability after large-scale fine-tuning.

The results in Tab. 6 show that Qwen2-VL-7B trained with our functionality grounding tasks can help the planners to more accurately locate target elements.

D Additional Experimental Analysis

D.1 Growing Grounding Performance Brought by Scaling Data Size

To further investigate the benefit of scaling the AutoGUI functionality data, the histogram of dis-
tance from a predicted point to the ground truth box center is plotted for the 25k, 125k, and 702k experiments. The results in Fig. K demonstrate that the distance distributions become denser at lower ranges, suggesting that increasing the AutoGUI training data leads to consistently improved grounding performances.

D.2 Case Analysis on FuncPred Test Split
Successful cases Fig. L demonstrates several examples of the grounding results from Qwen-VL trained with the 25k, 125k, and 702k AutoGUI data. The model trained with the 702k data (ours-702k) exhibits more accurate functionality grounding performance. For instance, Fig. L (a) shows that ours-702k predicts the point right on the target (The ‘Get an account’ button) while the other two models slightly miss the target. Case (c) shows that ours-702k correctly understands the functional intent to locate the WordPress logo, in contrast to the other models, which incorrectly focus on the text ‘Get WordPress’. Additionally, case (f) illustrates that ours-702k successfully locates the three-dot menu icon, aligning with the intent to expand a dropdown menu. These results suggest that increasing the AutoGUI training data enhances the model’s ability to understand complex functional intents and to recognize diverse iconic elements accurately.

Failure cases To explore the limitations of our model, we analyze several failure cases across the scaling experiments, as shown in Fig. M. The primary failure cases comprise (1) Difficulty in accurately locating very small target elements, as illustrated by the tiny ‘Policy’ button in case (a); (2) Misunderstanding functional intents, as shown in case (b) where the three models fail to locate the element for account creation and case (g) where ours-702k mistakenly focuses on navigating to previous content instead of subsequent content; (3) Challenges in recognizing abstract iconic elements, as seen with the map style icon in case (d) and the compass icon in case (f).

Despite these challenges, the enhanced performance observed with ours-702k supports the potential of the AutoGUI pipeline to further improve functionality grounding. The successful cases underscore that increasing the size of the training dataset not only boosts the model’s ability to interpret functional intents but also its capability to process a variety of textual and iconic elements effectively.

D.3 Case Analysis on MoTIF Test Split
We evaluate the instruction following ability on MoTIF dataset. Our analysis focuses on two aspects: (1) what improvements our model can achieve with the scaling of our functionality dataset (Fig. N); and (2) in which scenarios our model still fails to achieve correct grounding (Fig. O).

Fig. N shows that the model can more accurately understand the action instruction and make meaningful localization as scaling improves from 125k to 702k. For instance, when the objective is to click sleep noise recording and click enable, the model can comprehend the semantics of this global objective and identify turn on. Additionally, the model can mitigate localization errors, such as the 702k being more accurately positioned on the target element (e.g., the icon of reservation) than the 125k. However, MoTIF still struggles with certain tasks. For example, as shown Fig. O, it has difficulty with localization in fine-grained steps for the instruction search for Kingston Drive and show me the route to it. It can be seen that the model does not effectively understand situations involving widget pop-ups (e.g., protocol and advertisement). This may be attributed to the weak semantic connection between pop-ups and the instruction. Furthermore, the model still falls short in precise localization. Enriching the dataset further could alleviate this issue.

E Potential Societal Impact

The potential societal impacts of the proposed AutoGUI can be considered across various dimensions:

Accessibility Enhancements VLMs trained with the AutoGUI data obtain stronger UI grounding capabilities, thereby possessing the potential to act as UI agents. By enabling context-aware understanding of UI functionalities, the VLMs can help users locate elements on complex UIs, significantly improving accessibility features in software. This could lead to the development of applications that are more intuitive for users with disabilities, such as those requiring screen readers or other assistive technologies.

Research Impact: By reducing the labor and time required for annotating UI data via the AutoGUI, the industry and academia could lower costs to easily build UI agents. This could also shift labor demands towards more creative and strategic roles rather than repetitive annotation tasks.
Privacy and Security Concerns: Although we employ precautions of eliminating samples related to sensitive UI elements (e.g., avoid interacting with elements modifying the Internet and use only popular public websites without exposing privacy), corner cases still exist on the vast Internet. UI data involving either content modification or personal information are hard to discern as UI designs are distinct and no universal detection rules exist. Therefore, it is essential for cyber-security research to consider the potential leakage of personal information in the collected data and devise preemptive protective approaches.

Potential for Bias and Fairness: The bias of the LLMs used in the AutoGUI annotation pipeline is probably reflected in the collected data, leading to a trained UI-VLM that inherits the bias. Therefore, mitigating bias in the LLM’s annotations will be important for developing fair VLM agents that align with the values of users from diverse cultures.
Figure C: Comparing the proposed AutoGUI annotation pipeline with existing methods. AutoGUI is able to manipulate real UIs and interact with elements hidden beneath deeper levels (e.g., the buttons hidden in collapsed dropdown menus), thereby collecting considerably rich element-functionality annotations from the immense UI resources on the Internet. In contrast, SeeClick (Cheng et al., 2024) only uses static webpages and collects static element-text pairs. Likewise, CogAgent collects static element-HTML pairs while OmniAct generates Python scripts only for visible elements. These three existing methods can only annotate visible static UI elements and ignore the rich UI functional semantics entailed in interaction trajectories which are provided by our AutoGUI pipeline in abundance.
Get WordPress
Everything you need to set up your site just it

AXTree
RootWebArea 'Download - WordPress.org'
link 'Skip to content'
figure *
link 'WordPress.org'
image 'WordPress.org'
link 'News'
link 'Showcase'
list 'Home'
button 'Extend submenu' expanded: False
StaticText 'Extend'
button 'Learn submenu' expanded: False
StaticText 'Learn'
> button 'Community submenu' focused: True expanded: False
button 'About' hasPopup: dialog
StaticText 'About'
button 'Open Search' hasPopup: dialog
link 'Get WordPress'
main ''
heading 'Get WordPress'
StaticText 'Download WordPress for you...'
StaticText 'For anyone...'
link 'Download WordPress...'
link 'Installation guide'
heading 'Set up with a hosting provider'

AXTree
RootWebArea 'Download - WordPress.org'
link 'Skip to content'
figure *
link 'WordPress.org'
image 'WordPress.org'
link 'News'
link 'Showcase'
list 'Home'
button 'Extend submenu' expanded: False
StaticText 'Extend'
button 'Learn submenu' expanded: False
StaticText 'Learn'
> button 'Community submenu' focused: True expanded: True
> StaticText 'Community'
> link 'Make WordPress'
> link 'Photo Directory'
> link 'Five for the Future'
> link 'Events'
> link 'Job Board'
button 'About submenu' expanded: False
StaticText 'About'
button 'Learn submenu' hasPopup: dialog
link 'Get WordPress'
main ''
heading 'Get WordPress'
StaticText 'Everything you...'
StaticText 'For anyone...'
link 'Download WordPress...'
link 'Installation guide'
heading 'Set up with a hosting provider'

UI Differences
<table>
  <tr>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>Unchanged</td>
    <td>Button 'Extend submenu' expanded: False</td>
    <td></td>
  </tr>
  <tr>
    <td>Unchanged</td>
    <td>Button 'Learn submenu' expanded: False</td>
    <td></td>
  </tr>
  <tr>
    <td>Unchanged</td>
    <td>StaticText 'Learn'</td>
    <td></td>
  </tr>
  <tr>
    <td>Before Attribute Update</td>
    <td>Button 'Community submenu' expanded: False</td>
    <td></td>
  </tr>
  <tr>
    <td>After Attribute Update</td>
    <td>Button 'Community submenu' focused: True expanded: True</td>
    <td></td>
  </tr>
  <tr>
    <td>Added</td>
    <td>link 'Make WordPress'</td>
    <td></td>
  </tr>
  <tr>
    <td>Added</td>
    <td>link 'Photo Directory'</td>
    <td></td>
  </tr>
  <tr>
    <td>Added</td>
    <td>link 'Five for the Future'</td>
    <td></td>
  </tr>
  <tr>
    <td>Added</td>
    <td>link 'Events'</td>
    <td></td>
  </tr>
  <tr>
    <td>Added</td>
    <td>link 'Job Board'</td>
    <td></td>
  </tr>
  <tr>
    <td>Unchanged</td>
    <td>Button 'About submenu' expanded: False</td>
    <td></td>
  </tr>
  <tr>
    <td>Unchanged</td>
    <td>StaticText 'About'</td>
    <td></td>
  </tr>
  <tr>
    <td>Unchanged</td>
    <td>Button 'Learn submenu' hasPopup: dialog</td>
    <td></td>
  </tr>
  <tr>
    <td>Unchanged</td>
    <td>link 'Get WordPress'</td>
    <td></td>
  </tr>
</table>

Prompt: Now please think step-by-step, examine the content variation in detail, and finally summarize the high-level usage of the interacted element.

Reasoning: After interacting with the "Community submenu" button, the webpage content changes significantly. The button's "expanded" attribute is updated to "True", indicating that the button is now expanded. Additionally, the button gains focus, as indicated by the "focused: True" attribute. The most notable change, however, is the appearance of new. These new elements suggest that the button has triggered the display of a submenu or dropdown list related to the "Community" theme.

Summary: This element reveals a submenu of community-related links and resources.

Figure D: An example of the AutoGUI functionality annotation using UI AXTree differences. AutoGUI records the AXTrees before and after interaction and then generates line-by-line differences with our custom change markers. Subsequently, the LLM takes the differences as input to predict the element functionality.
New: Updates to Instagram Direct Message

RootWebArea 'New: Updates to Instagram Direct Message | Instagram Blog' focused: True
navigation 'Main'
button 'Navigation menu' expanded: False
controls: menu-list
    StaticText 'Site'
    StaticText 'Navigation menu'
    link ''
    link 'Log in'
    button 'Navigate to the home page'
    StaticText 'Navigate to the home page'
    heading 'New: Updates to Instagram Direct Message'
    StaticText 'April 11, 2017'
    > link ''
    link ''
    link ''
    link ''
    link ''

AXTree

RootWebArea 'Facebook' focused: True
heading 'Facebook'
labelText 'Log in to your Facebook account to share.'
StaticText 'Email address or phone number:'
textBox ''
StaticText 'Password:'
textBox ''
button 'Log in'
link 'Forgotten account?'
button 'Create New Account'
    StaticText 'Create New Account'

Description

Here is the description of the webpage regions and overall functionality:

Region 1 (Header): Navigation and login functionality
Region 2 (Main Content): Announcing updates to Instagram Direct Message, providing a blog post with a title, date, and content
Region 3 (Footer): Additional links and buttons (less meaningful region, summarized in a single sentence)

Overall Functionality: A blog post webpage announcing updates to Instagram Direct Message, providing a title, date, and content, with navigation and login functionality at the top.

Prompt: Now please think step-by-step, examine the content variation in detail, and finally summarize the high-level usage of the interacted element.

Description

Here is the description of the webpage regions and overall functionality:

Region 1 (Header): Displays the webpage title and heading.

Region 2 (Login Form): Allows users to log in to their Facebook account by providing their email address or phone number and password.

Region 3 (Forgot Password): Provides a link to recover a forgotten account password.

Region 4 (Create New Account): Offers an option to create a new Facebook account.

Overall Functionality: Enables users to access their existing Facebook account or create a new one, providing a gateway to the social media platform.

Reasoning: Upon examining the webpage variations, I notice that the initial webpage is a blog post announcing updates to Instagram Direct Message, with navigation and login functionality at the top. After clicking the <link> element named "", the webpage transitions to a new page with a Facebook title and main content. The second page is then a login form, allowing users to log in to their Facebook account. Additionally, there are options to recover a forgotten password and create a new Facebook account.

Summary: This element redirects the user to a Facebook login page, allowing them to access their existing account or create a new one.

Figure E: An example of the AutoGUI functionality annotation using UI descriptions. AutoGUI records the AXTrees before and after interaction and then prompts the LLM to describe the AXTrees in detail. Subsequently, the LLM takes the two descriptions as input to predict the element functionality.
Description
Region 1 (Header): Navigation and branding region with links to various Yahoo services and a search button.

Region 2 (Main Content): News article listing region displaying a collection of news articles with images, headlines, and timestamps.

Overall Functionality: A news aggregator webpage providing access to various news articles and categories, allowing users to browse and read news content.

Description
Region 1 (Header): Branding and navigation (contains the Yahoo logo and link)

Region 2 (Login Form): Allows users to sign in to their Yahoo account by entering their username, email, or mobile number and proceeding to the next step

Region 3 (Forgot Username Link): Provides an option to recover forgotten usernames

Region 4 (Create Account Button): Enables users to create a new Yahoo account

Overall Functionality: A login page that facilitates access to Yahoo Mail accounts, offering options for forgotten usernames and new account creation.

Prompt: Before the interaction, the current webpage description is {before_description}
After clicking a <link> element named "Inbox", we navigate to a new webpage whose description is: {after_description}

Now please think step-by-step, examine the content variation in detail, and finally calculate scores according to the criteria. Do follow the output format "Reasoning: ... Overall score: <score>... </score>"

Reasoning:
- Explicitness of Changes: The changes are quite explicit, as clicking on the "Inbox" link navigates to a new webpage with a distinct layout and functionality. The new page is clearly a login page, with regions for signing in, recovering forgotten usernames, and creating new accounts. The changes directly suggest that the "Inbox" link is related to accessing a Yahoo Mail account. Score: 3
- Relevance of Changes: The modifications are highly relevant to the element's intended function. The "Inbox" link is expected to lead to a page related to accessing mail accounts, and the new page meets this expectation by providing a login form and options for account management. Score: 3
- Predictability of Outcome: Based on common web conventions and user experience principles, the outcome is highly predictable. Users would typically expect an "Inbox" link to lead to a page related to accessing their email account, and the new page's functionality aligns with this expectation. Score: 3

Overall score: <score>3 + 3 + 3 = 9</score>

Figure F: An example of AutoGUI prompting the LLM as a rejector to determine whether a sample shows meaningful UI content changes sufficient for predicting the functionality of the interacted element. The sample shown is a navigation case in which AutoGUI uses UI descriptions, instead of line-by-line differences, to make decisions.
Figure G: Examples of samples rejected by the AutoGUI pipeline. The first sample encounters incompletely loaded content that interferes LLM annotation. The second encounters a no-response issue where the pop-up window fails to appear. The third shows a case where an unexpected log-in page pops up to interrupt the functionality of the “Add a Comment” element.
Figure H: An example of AutoGUI prompting the LLM as a self-verifier to determine whether an element supports its functionality annotation. The sample shown is a manipulation case in which AutoGUI uses UI line-by-line differences to make decisions about whether a button fulfills the intent of hiding promotional content.
• Criterium 1: Context-specificity
  This element opens a date-picker to select a check-in date.
• Criterium 2: Appropriate details
  This element triggers a date picker for users to select their check-in date
• Criterium 3: No hallucination
  This element checks an item shown in a filter dialog.

Context-related description
This element reveals a calendar.
Unnecessary details
This element switches the focus from the 'Destination' input field to the 'Check-in Date' field, providing a date picker.
Hallucinated functionality
This element immediately filters and reorganizes the content of the UI.

Figure I: The checking criteria used for comparing AutoGUI pipeline and the human annotator.

ScreenSpot (v2)
Anno. Type: Brief descriptions
Query: Windows settings
Target: (X=30,Y=800)

VisualWebBench Action Gnd.
Anno. Type: Action intent
Query: View the content in Chinese
Target: (X=560,Y=640)

MOTIF
Anno. Type: Displayed Text
Query: chose "how to cut pineapple"
Target: (X=500,Y=410)

FuncPred
Anno. Type: Elem. Functionality
Query: This element adjusts the size of the content
Target: (X=440,Y=170)

Figure J: Samples of the UI grounding benchmarks used in the experiments
(a) FuncPred Test

(b) ScreenSpot

Figure K: Histograms of distances from predicted points to ground truth box centers. The distance from the normalized coordinate of a predicted point to its corresponding GT box center is calculated for all samples. Then, the histograms of these distances are illustrated to demonstrate the growing grounding performances brought by scaling the AutoGUI data size. The averaged distance for each experiment is displayed on the subplot title.
(a) Functionality: This element navigates to a page for creating or obtaining an account.

(b) Functionality: This element allows users to reorder the topic list by view count, making it easier to find popular or frequently viewed topics.

(c) Functionality: This element represents the primary brand or logo of the webpage, providing users with a direct access point to the homepage of the 'WordPress.org' website.

(d) Functionality: This element enables users to share content on Twitter.

(e) Functionality: This element triggers the expansion of the search functionality on the webpage, allowing users to access more extensive search options.

(f) Functionality: This element triggers additional functionality or navigation within the webpage, such as revealing a dropdown menu.

(g) Functionality: This element serves as a login gateway for the Pinterest app, allowing users to authenticate their accounts using Facebook.

(h) Functionality: This element is a password input field, allowing users to securely enter their account password for authentication during the login process on the CNN website.

Figure L: Visualization of the successful functionality grounding examples for ours-625k. The ground truth bounding boxes, ours-625k predictions, ours-125k predictions, and ours-25k predictions are drawn in green, pink, blue, and orange, respectively.
(a) Functionality: This element provides access to the privacy policy of GitHub, giving users important information about how their data is managed and handled.

(b) Functionality: This element initiates the account creation process for new users.

(c) Functionality: This element provides access to basic information and resources about the Commons system.

(d) Functionality: This element allows users to customize the map's visual style.

(e) Functionality: This element allows users to navigate back to the previous menu or page within the SONOACE R3 | Samsung Support Bangladesh webpage.

(f) Functionality: This element allows users to discover and explore the platform's trending and popular content, providing a gateway to various sections and categories of the video-sharing platform.

(g) Functionality: This element advances the user to the subsequent slide within the slideshow of featured products and announcements, providing a means for users to browse through the displayed content.

(h) Functionality: This element is an upvote button for users to express their approval of an article.

Figure M: Visualization of failure examples in the scaling experiments. The ground truth bounding boxes, ours-625k predictions, ours-125k predictions, and ours-25k predictions are drawn in green, pink, blue, and orange, respectively.
Click Sleep noise recording and click enable
Open settings and open push notifications and turn off deals and announcements
Set an alarm for a nap that is 30 minutes long
Click Reservations Tab

Figure N: Evaluation results of the model trained on \(625k\) (blue dot) and \(125k\) (red dot).

Search for Kingston Drive and show me the route to it

Figure O: Bad cases on MoTIF.
Interaction: clicking a <link> element named "C#"

Human: This element navigates users to a forum related to C# questions.
AutoGUI: This element filters the webpage content to display questions related to a specific programming language, in this case, C#.

Interaction: clicking a <link> element named "Creative"

Human: This element redirects users to a page displaying creative content and providing search and filter functions.
AutoGUI: This element filters the showcase of WordPress-built websites by a specific category, allowing users to focus on a particular type of website.

Interaction: clicking a <SvgRoot> element

Human: This element triggers the expansion of the current user's profile menu.
AutoGUI: This element triggers a dropdown menu for account management, providing access to personal and business account-related features.

Figure P: Comparing the annotations generated by a trained human annotator and the proposed AutoGUI pipeline. We can see that AutoGUI annotations are more detailed and clear than those by the human annotator.

Reviewer paper summaries: First of all, I really like this paper for its novel method, broad contribution, and comprehensive experiments. This paper introduces AutoGUI, an automatic annotation pipeline that creates high-quality functionality descriptions for UI elements by leveraging multiple pretrained models as sub-module to infer element functionality through analyzing UI state changes before and after simulated interactions. The authors construct a dataset of 704k annotations and demonstrate that models fine-tuned on this data exhibit significantly improved UI grounding capabilities with clear scaling effects. The pipeline includes LLM-aided rejection and verification components to ensure annotation quality without human labor. Experiments show that the functionality annotations produced by AutoGUI are superior to alternative approaches like HTML code extraction or alt-text for enhancing models' abilities to locate and understand UI elements.

This paper proposes AutoGUI, a pipeline for automatically annotating UI elements with detailed functionality descriptions at scale to enhance the GUI grounding capabilities of Vision-Language Models (VLMs). By leveraging Large Language Models (LLMs) to infer element functionalities through UI state changes during interactions, AutoGUI constructs the AutoGUI-704k dataset, which covers diverse UI contexts from web and mobile platforms. The pipeline incorporates LLM-aided rejection and verification to ensure annotation quality, achieving 96.7% human-comparable accuracy. Experiments demonstrate that VLMs trained with this dataset exhibit improved UI grounding performance.

This paper focuses on addressing the challenges in training vision-language models (VLMs) for user interface (UI) understanding. VLMs have the potential to transform software interaction, but accurately understanding the functionality of individual UI elements is difficult. Existing datasets for training UI-VLMs are limited in scale and lack comprehensive functionality annotations, while traditional annotation methods are costly.

The authors propose the AutoGUI pipeline to generate large-scale, detailed functionality annotations for UI elements. It starts by collecting interaction trajectories from web browsers and Android emulators, using sources like Common Crawl and incorporating open - source trajectories. Then, it uses large language models (LLMs) to infer element functionality based on UI state changes. For example, it analyzes AXTree differences for UI manipulation or UI description changes for navigation. To ensure data quality, it uses LLM - aided rejection to filter out invalid samples and LLM - based verification to correct inaccurate annotations. The pipeline then converts the data into functionality grounding and referring tasks, creating the AutoGUI - 704k dataset with 702k training and 2k test samples.

Experimental results show that the AutoGUI - 704k dataset significantly improves VLMs' UI grounding accuracy. When fine - tuned on this dataset, various base models perform better on multiple UI grounding benchmarks, and the performance scales with the increase in data size. The functionality annotation of AutoGUI outperforms other annotation types such as those directly derived from HTML code. However, the fine - tuned models still face challenges like accurately locating small elements and differentiating similar ones.
The dataset also shows potential in GUI agent tasks. In a two - stage planning task on the AITW benchmark, a model trained with AutoGUI functionality grounding tasks can help a planner more accurately locate target elements.

Despite its achievements, the AutoGUI pipeline has limitations. It struggles to collect diverse mobile app data due to anti - emulator code in apps. The collected interaction trajectories lack high - level task semantics, and the pipeline cannot annotate UI elements that modify internet content to avoid potential risks. Overall, the paper presents a novel approach to improving UI - VLM training, with implications for software interaction and the development of more intelligent UI agents, while also highlighting areas for future research.
Summary of strengths: - Novel approach to UI annotation: The automated pipeline addresses a significant bottleneck in creating large-scale UI datasets by eliminating manual annotation, and the step-by-step functionality based address well on current missing dataset for temporal information in current GUI dataset. The multi-stage verification and rejection process ensures high-quality annotations without human intervention. It also shows clear scaling effects, suggesting further improvements with additional data.
- Comprehensive evaluation: The paper presents extensive experiments across multiple benchmarks including grounding, GUI understanding and agent tasks, demonstrating significant improvements in grounding performance. The authors demonstrate real-world utility by showing how their dataset can enhance GUI agent tasks.

- The basic idea of AutoGUI, which eliminates manual labeling by using LLMs to infer functionalities from UI interaction trajectories—enabling large-scale data collection (704k samples) at a low cost—is interesting and reasonable.

- The data is both large-scale and richly annotated.

- The paper is well-written and easy to understand.

By leveraging large language models (LLMs) to automate the annotation of UI elements with functionality descriptions, the AutoGUI pipeline overcomes the limitations of traditional, labor-intensive annotation methods. This not only reduces the cost and time associated with data annotation but also enables the creation of a much larger and more comprehensive dataset. For example, the process of collecting interaction trajectories from various sources and using LLMs to infer functionality based on UI state changes is a unique approach. It can scale to handle a vast number of UI elements, providing a scalable solution for training VLMs.

The inclusion of LLM - aided rejection and verification steps in the pipeline is a novel way to ensure data quality. Instead of relying on human intervention for error - checking, the pipeline uses LLMs themselves to identify and discard invalid samples and verify the accuracy of annotations. This not only streamlines the annotation process but also sets a new standard for automated data curation in the context of UI-VLM training.

The comparison of different annotation types shows that the functionality annotations provided by AutoGUI are superior to other common types such as those derived from web HTML code and metadata. This finding is crucial as it directly impacts the design of future datasets for training VLMs. It provides clear evidence that functionality - focused annotations are more beneficial for enhancing UI understanding in VLMs, guiding researchers towards more effective data collection and annotation strategies.

The comparison of the AutoGUI dataset with existing UI - grounding datasets in a tabular format is a clear and effective way to present the differences and advantages of the proposed dataset. By highlighting aspects such as UI type, multi - resolution support, auto - annotation capabilities, presence of contextual functionality semantics, number of annotations, and the associated tasks, readers can easily see how AutoGUI stands out from the rest. This organized presentation of related work not only serves as a reference for future research but also positions the paper's contributions within the broader context of the field.

Relevance to ACL Community：As ACL focuses on the intersection of language and various other fields, this paper's work on training VLMs for UI understanding is highly relevant. It combines natural language processing techniques (through the use of LLMs for annotation) with computer vision concepts (understanding visual UI elements). This multi - disciplinary approach is in line with the evolving interests of the ACL community, which is increasingly exploring the integration of language with other modalities.
Summary of weaknesses: I don't see obvious weakness in this paper. The only problem I would like to raise is its insufficient analysis of why the approach works. The paper demonstrates impressive results but lacks deeper theoretical analysis of why their method is effective for UI understanding and agent tasks. Functionality annotation *may* be one of the reason, but I think this paper goes beyond it. One potential perspective from my side is that this paper seems to solve important challenges in GUI interaction that require dynamic understanding and sequential reasoning (as highlighted in [1] and [2]), and this method synthesize data from paired data cleverly solve this problem. The model trained with these kind of data may equipped with better *temporal* reasoning capability, and I wish the author can somehow establish experiments to prove this hypothesis. I want to see how the author think of the potential reason for how their method specifically addresses step-by-step understanding and reasoning are much better than attributing it with providing more or higher-quality data. This is the only missing piece for this excellent paper, and it would significantly strengthen the paper's contribution to the field.

Look forward to your rebuttal.

[1] GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding
[2] VideoGUI: A Benchmark for GUI Automation from Instructional Videos

- The abstract mentions that one of the advantages of this paper is the inclusion of functional descriptions. However, existing datasets, such as Uground and Ferret-UI, also contain functional descriptions. What is the fundamental difference between the functional descriptions proposed in this paper and those in previous works? 

- Moreover, what actual differences does the so-called "contextual functionality" produce in the results? Specifically, does generating functional descriptions from interaction trajectories offer a significant advantage over generating them from a single image? Both qualitative and quantitative experiments are needed to demonstrate it.

- The example in Figure 1 is quite confusing. What is the motivation behind this paper? Most GUI data collection relies on the accessibility tree, including this work (as indicated in Line 183). Therefore, "similar look but different functions" should not affect annotation at all since the annotation process does not require looking at the image. The accessibility tree already records whether the element is named "search" or "zoom in."

- The improvement of the proposed data over Uground and SeeClick is relatively limited. Does this suggest that the training data is quite similar to theirs and does not provide a significant advantage?

1. Data Collection Limitations

（1）Mobile App Data Diversity: The paper mentions that the collection of mobile app data is hampered by anti - emulator code in many apps. This limitation restricts the diversity of mobile app - based UI data in the AutoGUI - 704k dataset. As a result, the generalizability of the trained VLMs to real - world mobile apps may be compromised. Since a significant portion of human - computer interaction occurs through mobile apps, this lack of diversity could limit the overall impact of the proposed methods in the broader context of UI understanding.

（2）Absence of High - Level Task Semantics: The collected interaction trajectories lack high - level task semantics. Understanding the high - level tasks that users are trying to accomplish is crucial for developing truly intelligent UI - interacting VLMs. Without this information, the models may struggle to handle complex user intents, which restricts the potential of the dataset and the trained models in real - world applications.

 2. Model Performance and Generalization

(1) Grounding Failure Cases: The analysis of grounding failure cases reveals that fine - tuned models still face challenges in accurately locating small elements, differentiating between similar elements, and recognizing icons with uncommon shapes. These issues suggest that there is room for improvement in the model's performance, especially in handling complex and diverse UI elements. This may limit the practical applicability of the proposed approach, as accurate element grounding is essential for effective UI interaction.

(2) Limited Scope of Benchmarking: Although the paper evaluates the performance of the fine - tuned VLMs on multiple benchmarks, it is possible that the selected benchmarks may not fully represent the complexity and diversity of real - world UI scenarios. As a result, the reported performance improvements may not accurately reflect the model's performance in all relevant use cases, casting doubt on the generalizability of the results.

3. Exposition and Clarity

(1) Technical Jargon and Complexity: At times, the paper uses a significant amount of technical jargon and complex sentences, which may make it difficult for readers who are not familiar with the field to fully understand the concepts and methods. This could limit the accessibility of the paper and reduce its potential impact, as a wider audience may not be able to fully appreciate the contributions.

(2) Lack of Visual Aids for Complex Concepts: Some of the concepts, such as the operation of the AutoGUI pipeline and the interaction between different components, could benefit from more visual aids. Visual representations would make it easier for readers to grasp the complex processes and relationships, enhancing the overall clarity of the paper.
Comments, suggestions and typos: See weakness above.

- The comparison in Table 1 seems unfair. "Contextual functionality semantics" appears to be a forced distinction to differentiate this work from others, and the term itself is vague. Why not just call it "functionality description"? Why do Uground and Ferret-UI not have it?

1. Introduction and Motivation Section

Clarify the Problem Urgency: While the motivation for improving VLMs' UI understanding is clear, the authors could further emphasize the immediate and practical consequences of the current limitations. For example, they could provide more real-world examples of where inaccurate UI understanding by VLMs leads to poor user experiences or inefficiencies in software applications.

Highlight the Gap More Clearly: In addition to listing the limitations of existing datasets, it might be beneficial to create a more visual or structured representation of the research gap. A diagram or a table comparing the ideal characteristics of a dataset for UI-VLM training with what currently exists could help readers quickly grasp the significance of the AutoGUI pipeline and dataset.

2. Methodology Section

Provide More Details on LLM Selection: The paper mentions using open-source LLMs for annotation, rejection, and verification. However, it would be useful to explain the criteria for choosing specific LLMs. Discussing factors such as their performance in similar natural language processing tasks related to UI analysis could add depth to the methodology.

Illustrate the Interaction Trajectory Collection: Since the collection of interaction trajectories is a crucial part of the pipeline, some visual examples or a more detailed flowchart showing how the trajectories are captured and what information is included at each step would be helpful. This could make it easier for readers to understand the process and potentially replicate it.

Explain the Role of File-Comparing Library: When describing how the pipeline analyzes the differences in AXTrees for UI manipulation cases, a brief explanation of the file-comparing library used and how it functions in this context would be beneficial. This would help readers understand the technical details of the analysis process.
Overall assessment scores: 4.5

3

3.5