Aria-UI: Visual Grounding for GUI Instructions
Authors: Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, Junnan Li
Accepted at: acl25

Aria-UI: Visual Grounding for GUI Instructions

Yuhao Yang1, Yue Wang3, Dongxu Li4, Ziyang Luo2, Bei Chen5, Chao Huang1†, Junnan Li1,2

1The University of Hong Kong, 2Salesforce AI Research, 3Alibaba Group, 4Australian National University, 5Independent Researcher

Abstract
Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant challenge due to reliance on HTML or AXTree inputs. In this paper, we introduce Aria-UI, a large multimodal model specifically designed for GUI grounding. Aria-UI adopts a pure-vision approach, eschewing reliance on auxiliary inputs. To adapt to heterogeneous planning instructions, we propose a scalable data pipeline that synthesizes diverse and high-quality instruction samples for grounding. To handle dynamic contexts in task performing, Aria-UI incorporates textual and text-image interleaved action histories, enabling robust context-aware reasoning for grounding. Aria-UI sets new state-of-the-art results across offline and online agent benchmarks, outperforming both vision-only and AXTree-reliant baselines. We release all training data and model checkpoints at https://ariaui.github.io to foster further research.

1 Introduction
The rapid expansion of graphical user interfaces (GUIs) across web, desktop and mobile platforms has made them indispensable for digital interactions. From completing daily tasks like shopping or booking tickets to complex professional workflows, GUI agents play a critical role in automating these processes. As illustrated in Figure 1, a typical GUI agent operates in two stages: planning and grounding. In the planning stage, the agent generates action decisions to accomplish the user’s task based on the current screen state as its observation. In the grounding stage, the agent is tasked with locating and interacting with the target element as referred in the instructions provided by planning, thus make actions truly happen in the environment-

While efforts have been put to improve the planning of large multimodal models (LMMs) with CoT (Yao et al., 2022b; Wei et al., 2022), and inference-time scaling (Saha et al., 2024), effectively grounding GUI elements from language remains a significant challenge. The problem is compounded by the diverse visual layouts across diverse devices, wide variability in planned instructions, and the dynamic nature of task execution in real-world environments, all of which demand robust, adaptable, and efficient solutions.

The basic grounding method involves leveraging HTML or accessibility trees (AXTrees, or Ally) to identify the target element. However, feeding long textual contexts of the tree often leads to inefficiencies, hallucination, and biases due to missing information in the tree. The absence of visual input further limits the method’s ability to address instructions requiring visual or positional cues. Set-of-Mark (Yang et al., 2023) combines visual and tree tag information. However, its reliance on HTML or AXTrees limits flexibility in diverse environments, as platform standards are inconsistent and, particularly on mobile and desktop, the quality of AXTrees depend largely on app developers’ implementation. Additionally, LMMs struggle to accurately select from numerous tags in images, constraining grounding performance (Xie et al., 2024). To this end, building a pure-vision solution for GUI agent grounding is crucial.

Training an LMM for GUI instruction grounding is non-trivial. Existing LMMs are: 1) heavily skewed towards natural images due to data biases. 2) rarely trained for grounding. While some models are trained with datasets like RefCOCO (Kazemzadeh et al., 2014), these datasets are not aligned with GUI scenarios and are sparsely populated. Recently, some studies (Cheng et al., 2024; Gou et al., 2024) have leveraged LMMs’ powerful vision and language capabilities, using public mobile- or web-sourced data as (GUI image,
Figure 1: The two-stage task performing process for general GUI agents. Aria-UI serves as a robust grounding model to make the planned actions truly happen.

<table>
  <tr>
    <th>Collection</th>
    <th>#Web Img.</th>
    <th>#Mobile Img.</th>
    <th>#Desktop Img.</th>
    <th>Input Text</th>
    <th>Supervision</th>
    <th>Open Source</th>
    <th>Action History</th>
    <th>#Elements</th>
    <th>#Samples</th>
  </tr>
  <tr>
    <td>Etay-UAAAP</td>
    <td>54K</td>
    <td>/</td>
    <td>/</td>
    <td>HTML</td>
    <td>Point Coordinates</td>
    <td>X</td>
    <td>X</td>
    <td>70M</td>
    <td>176K</td>
  </tr>
  <tr>
    <td>CogAgenda-CCS400K</td>
    <td>400K</td>
    <td>/</td>
    <td>/</td>
    <td>HTML+Text</td>
    <td>Point Coordinates</td>
    <td>X</td>
    <td>X</td>
    <td>70M</td>
    <td>176K</td>
  </tr>
  <tr>
    <td>UGround-Web-Hybrid</td>
    <td>773K</td>
    <td>/</td>
    <td>/</td>
    <td>HTML Attr + Refer Caption</td>
    <td>Point Coordinates</td>
    <td>X</td>
    <td>X</td>
    <td>18.1M</td>
    <td>9M</td>
  </tr>
  <tr>
    <td>UGround-Web-Direct</td>
    <td>408K</td>
    <td>/</td>
    <td>/</td>
    <td>Refer Caption</td>
    <td>Point Coordinates</td>
    <td>X</td>
    <td>X</td>
    <td>408K</td>
    <td>408K</td>
  </tr>
  <tr>
    <td>SeeClick</td>
    <td>270K</td>
    <td>/</td>
    <td>/</td>
    <td>HTML Text</td>
    <td>Point Coordinates</td>
    <td>X</td>
    <td>X</td>
    <td>3.3M</td>
    <td>3.3M</td>
  </tr>
  <tr>
    <td>GUIEe-local</td>
    <td>73K</td>
    <td>9K</td>
    <td>/</td>
    <td>HTML Text</td>
    <td>Point Coordinates</td>
    <td>X</td>
    <td>X</td>
    <td>700K</td>
    <td>700K</td>
  </tr>
  <tr>
    <th>Aria-UI Collection</th>
    <th>173K</th>
    <th>164K</th>
    <th>7.8K</th>
    <th>Diversified Instr.</th>
    <th>Refer. Caption + Point Coordinates</th>
    <th>✓</th>
    <th>✓</th>
    <th>3.9M</th>
    <th>11.5M</th>
  </tr>
</table>

Table 1: Grounding data of Aria-UI compared to existing collections.

instruction, coordinates) tuples to train LMMs as grounding models. Despite their effectiveness, we identify two key limitations in these approaches: (1) They overly depend on rigid instruction sources and formats, mainly HTML or AXTree-based textual elements. This lack of diversity hinders their robustness in adapting to the flexible and heterogeneous instructions generated by task planners. (2) They overlook the dynamic contextual information during task performing, such as the action history, which can provide valuable references for more accurate element grounding.

In this paper, we introduce Aria-UI, a robust LMM designed specifically for GUI grounding. Aria-UI is built upon Aria (Li et al., 2024a), the state-of-the-art multimodal MoE model with 3.9B activated parameters. Aria-UI adopts a pure-vision approach, avoiding reliance on AXTree-like inputs while achieving superior grounding accuracy across diverse tasks and platforms.

By addressing the core limitations of existing methods, we propose two key contributions in Aria-UI. For the challenge of rigid instructions, we design a large-scale, diverse data synthesis pipeline from our Common Crawl collection and public available data. This pipeline first leverages strong LMMs to generate detailed and accurate element captions and then utilizes an LLM to create diverse, human-like instructions that align with potential interactions based on these captions. We further incorporate the high-quality captions as additional supervision during training, enabling the model to better associate diverse instructions with their corresponding elements. For the challenge of ignoring dynamic contexts, we further leverage textual or text-image interleaved action history from trajectory data for training. This equips Aria-UI with robust grounding capabilities, enabling it to perform effectively in dynamic, multi-step real-world task scenarios.

To summarize, our contributions are:
• We propose a novel approach to address the challenge of rigid instructions with a scalable, data-centric pipeline. It generates high-quality and diverse (element caption, instruction) samples from Common Crawl and publicly available data, enabling Aria-UI to generalize effectively across diverse instructions in different environments.
• Aria-UI introduces innovative designs for incorporating dynamic action history in textual or interleaved text-image formats. The improvements allow Aria-UI to ground elements more effectively in dynamic, multi-step task scenarios, especially under zero-shot settings.
• We conduct comprehensive evaluations on extensive benchmarks including both offline and online agent tasks, showcasing Aria-UI’s state-of-the-art performance. Notably, Aria-UI achieves higher grounding accuracy and task success rates compared to both vision-only and AXTree-reliant baselines.

2 Method

Aria-UI is designed to seamlessly integrate into the latest general-purpose multimodal GUI agent framework (Zheng et al., 2024; Xie et al., 2024; Koh et al., 2024; Rawles et al., 2024a), serving as a robust grounding model. We outline a solution to the challenges from a scalable, data-centric approach, as shown in Figure 2. In Section 2.1.1, we
detail the synthesizing of diverse grounding data. Section 2.1.2 discusses building grounding samples with task context for dynamic scenarios, and Section 2.2 explains Aria-UI’s training details.

2.1 Large-scale Diverse GUI Data Synthesizing

As summarized in Table 1, several existing methods have collected diverse corpus for GUI grounding. However, these corpora fail to effectively address GUI grounding for LMMs. They are either not open-source, too small, or lack coverage of all the major platforms. Moreover, they rely on rigid instruction sources and formats, from HTML extraction or specifically formatted referring caption. Additionally, they overlook the importance of the contextual information for grounding during dynamic task performing. We present how to solve these challenges by a data-centric approach with diverse data scaling from multiple platforms and context-aware data extension with trajectories.

2.1.1 Diverse Data Scaling from Multiple Platforms

We propose a two-stage pipeline to transform raw samples into high-quality and diverse element instructions for grounding training. At the first stage, we utilize a strong LMM (GPT-4o or Qwen2-VL-72B (Wang et al., 2024a)) that takes element screenshots and text extracted from HTML as input for accurate and detailed element descriptions. To enhance accuracy and reduce hallucination, the model perceives two screenshots: (1) an isolated image of the element and (2) a zoomed-in view, where the element is highlighted with a red bounding box. Additionally, the HTML text and the screen position of the element are provided for reference. The model is then prompted to generate a detailed caption of the element, including its visual properties, functionality, positional relationships, and any other distinctive attributes. In the second stage, we utilize an LLM to generate natural language instructions that correspond to potential interactions with the elements, based on their detailed captions. For instance, for the caption "The “subscribe” button, colored in bright red with white text and a bell icon, is positioned in the upper-right section of Chef-Maria’s cooking channel header, showing “2.3M” subscribers” underneath," the synthesized instruction could be "subscribe to ChefMaria’s channel." To ensure diversity and expand the data volume, we produce three instructions for each element.

We apply our pipeline to three key GUI environments: web, desktop, and mobile, each with distinct challenges and characteristics.

Web. Web data, with its diversity and dynamic rendering, is ideal for expanding GUI grounding datasets with varied element samples in size, type, and resolution. We leverage the latest collection of Common Crawl for data collection. We build a rigorous data curation and filtering pipeline to produce high-quality samples. We first filter out harmful webpages using fastText (Bojanowski et al., 2017). Subsequently, we identify and select interactive elements by checking the HTML attributes. Considering that LMMs have acquired fundamental OCR skills during pretraining, we prioritize graphical elements over text-based elements. To reflect real-world grounding tasks in complex, element-rich environments, we heuristically retain webpages containing more than 20 valid elements. We use Playwright to render these webpages at 1920×1080 and 2440×1600 resolutions to accommodate common resolution requirements. We gather a diverse set of 173K webpages containing 2M elements through the procedure. With the data pipeline, we build detailed caption and instructions for elements, result in 6M high-quality and diverse instruction samples.

Desktop. Since desktop environment is less scal-
able and human annotation costs high, desktop data has remained scarce. OmniACT (Kapoor et al., 2024) manually annotated 7.3K instruction-grounding pairs. However, creating an automated data scaling pipeline for desktop remains a challenge. To mitigate the research gap, we develop a traverse agent powered by an LMM to explore an OS environment for data collecting. We build the agent on an Ubuntu Desktop with Gemini 1.5 Flash. Leveraging the accessibility tree, the agent selects the next element to click in each screen state, aiming to reach previously unexplored screens. The implementation details can be found in Appendix A.2. We collect all screenshots and the corresponding A11y to parse all elements. Using this automated pipeline, we collected 88K unique elements across 7.8K screenshots tailored for desktop environment. We then utilize the data pipeline to extend the samples to 264K by generating diverse instructions.

Mobile. Since automated GUI agents for mobile environments were explored earlier, a substantial amount of open-source data has been accumulated for mobile environment. Currently, the largest-scale grounding dataset for mobile is AMEX (Chai et al., 2024), which provides 104K screenshots and 1.6M elements. While AMEX provides a large-scale dataset, it has only 712K elements with basic textual descriptions extracted from accessibility tags, and merely 3K elements are paired with human-like instructions. To address this gap, we regenerate high-quality caption and instruction samples with the data pipeline for AMEX, improving the training effectiveness.

To further expand our grounding corpus and introduce more diverse sources for GUI images and instructions, we incorporate the following public datasets: 3M Web and 273K mobile elements from SeeClick training data (Cheng et al., 2024; Li et al., 2020b,a), 15K mobile elements from (Bai et al., 2021), 748k Web elements from GUICourse (Chen et al., 2024), 131K desktop elements from OmniAct (Kapoor et al., 2024), and 693K Web and mobile elements from AutoGUI (Li et al., 2025). We summarize the details of the datasets in Table 8.

2.1.2 Context-aware Data Extension from Trajectories
Accurately and efficiently performing grounding tasks within the dynamic context of real-world environments is a crucial capability for GUI agents. Despite its importance, existing approaches largely focus on grounding tasks under a single-step setting, where LMMs are trained to infer grounding results based only on the current state and instruction. Such approaches overlook the dynamic nature of GUI grounding and the critical role of context in real-world scenarios. For example, after executing a TYPE action, the next grounding step is likely associated with an ENTER or SUBMIT button. Similarly, in multi-step tasks that involve navigating through a multi-layered menu to locate a target entry, there is a strong contextual relationship between consecutive grounding actions. Leveraging such contextual information enriches the grounding context and aids the model in avoiding bias, thereby enhancing grounding performance.

We utilize publicly available agent trajectories to simulate grounding tasks with contexts. We focus on constructing two types of contextual setups: (1) textual action history and (2) text-image-interleaved history. The text-based setup incorporates the ultimate task along with prior action histories, and the text-image-interleaved setup extends this by including N historical screen images, providing richer contextual cues and training the model to understand multimodal interaction history. Notably, most trajectory data only includes basic sequential information, such as the click coordinates, thus lacks comprehensive stepwise instruction semantics. To address this, we augment all grounding steps within the trajectory data using the proposed data pipeline to generate detailed stepwise instructions. For non-grounding actions, we encode them (e.g., SWIPE and TYPE) using rules for natural language formats. For the interleaved setting, we collect data with image number \( N = [1, 2, 3] \), and for the text-based setting, we input all historical actions in text. Finally we collect 992K samples with the trajectories from GUI-Odyssey (Lu et al., 2024), Android in the Zoo (Zhang et al., 2024d), Android Control (Li et al., 2024b), Android in the Wild (Rawles et al., 2024b) and AMEX (Chai et al., 2024). The details are presented in Table 9.

2.2 Model Architecture
We build Aria-UI with the state-of-the-art multimodal MoE model, Aria (Li et al., 2024a). We leverage two strengths from Aria for GUI agents: 1) Aria is multimodal-native, built for better understanding of complex and interleaved contexts; 2) with only 3.9B activated parameters, Aria shows even faster inference speed than 7B dense models.

2.2.1 Ultra Resolution Support
With the shift from 1080p to 2K resolutions on computers and mobile devices, training grounding LMMs at high resolutions has become essential.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Mobile</th>
    <th colspan="2">Desktop</th>
    <th colspan="2">Web</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Text</th>
    <th>Icon/Widget</th>
    <th>Text</th>
    <th>Icon/Widget</th>
    <th>Text</th>
    <th>Icon/Widget</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>22.6</td>
    <td>24.5</td>
    <td>20.2</td>
    <td>11.8</td>
    <td>9.2</td>
    <td>8.8</td>
    <td>16.7</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>20.2</td>
    <td>24.9</td>
    <td>21.1</td>
    <td>23.6</td>
    <td>12.2</td>
    <td>7.8</td>
    <td>18.1</td>
  </tr>
  <tr>
    <td>CogAgent</td>
    <td>67.0</td>
    <td>24.0</td>
    <td>74.2</td>
    <td>20.0</td>
    <td>70.4</td>
    <td>28.6</td>
    <td>49.6</td>
  </tr>
  <tr>
    <td>SeeClick</td>
    <td>78.0</td>
    <td>52.0</td>
    <td>72.2</td>
    <td>30.0</td>
    <td>55.7</td>
    <td>32.5</td>
    <td>55.8</td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>75.5</td>
    <td>60.7</td>
    <td>76.3</td>
    <td>54.3</td>
    <td>35.2</td>
    <td>25.7</td>
    <td>55.3</td>
  </tr>
  <tr>
    <td>UGround</td>
    <td>82.8</td>
    <td>60.3</td>
    <td>82.5</td>
    <td>63.6</td>
    <td>80.4</td>
    <td>70.4</td>
    <td>74.1</td>
  </tr>
  <tr>
    <th>Aria-UI</th>
    <th><b>92.3</b></th>
    <th><b>73.8</b></th>
    <th><b>93.3</b></th>
    <th><b>64.3</b></th>
    <th><b>86.5</b></th>
    <th><b>76.2</b></th>
    <th><b>82.4</b></th>
  </tr>
</table>

Table 2: Results on ScreenSpot. We report element accuracy and the micro average results.

Aria originally supports high-resolution images up to 980×980, which we extend to a maximum of 3920×2940 on Aria-UI by splitting the image into smaller blocks, significantly increasing the range of image sizes to handle. To maintain positional accuracy, we take inspiration from NaViT (Dehghani et al., 2024) to place padding before resizing for keeping the original screenshot ratio.

2.3 Training and Inference Paradigm
We train Aria-UI following a two-phase procedure. We first leverage all the single-step grounding data to train the foundation GUI grounding capability of Aria-UI. Specifically, Aria-UI is tasked with generating grounding answers given the prompt "Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction or description: [...]". We follow (Gou et al., 2024) to group all the samples for the same GUI image into a multi-turn conversation format. Then, context-aware data with both text-based and text-and-image-interleaved history settings are fed into the model to further enhance the grounding capability under the dynamic setting. For this phase, we add extra 20% samples from the single-step data to keep the generic grounding capability and avoid over-fitting. We place more training details in Appendix B.

During inference, Aria-UI outputs the grounded pixels coordinates normalized to [0, 1000]. Since Aria-UI is also trained with context-aware trajectories, it can take historical agent actions and grounding actions as chat history, formulating a stronger grounding system in dynamic environments.

3 Experiments

We testify the performances of Aria-UI via extensive experiments including single-step grounding, grounding under offline agent trajectories and grounding in dynamic online agent environments.

3.1 GUI Grounding Evaluation
We first examine Aria-UI’s foundational GUI grounding capabilities on ScreenSpot (Cheng et al., 2024). The benchmark compasses six subsets spanning over two types of elements and three major platforms. Each test entry provides a unique GUI image and a human-annotated instruction for locating a specific element. The typical resolution for mobile and web subsets is 2k, and for desktop samples it is 540p. We include the state-of-the-art UGround (Gou et al., 2024), with previous grounding models SeeClick (Cheng et al., 2024) and CogAgent (Hong et al., 2024) as baselines. We also include generic LMMs – GPT-4, GPT-4o and Qwen2-VL (Wang et al., 2024a).

From the results in Table 2, Aria-UI achieves the highest average accuracy (82.4%) across all subsets, demonstrating its superior grounding performance. Aria-UI achieves a significant margin over the state-of-the-art UGround, particularly excelling in tasks for textual elements. The results showcase Aria-UI’s robustness and generalizability across diverse platforms and element types.

3.2 Offline Agent Evaluation
Mobile Agents. We further testify how Aria-UI performs under an offline dynamic setting, where the model is required to provide grounding coordinates in agent task trajectories. We employ AndroidControl-Low (Li et al., 2024b), GUI-Odyssey (Lu et al., 2024) and AndroidControl-High, the first two has human-annotated or generated stepwise instruction, while the last one only provides the user task, and needs an additional planner for stepwise instructions. We follow (Li et al., 2024b; Gou et al., 2024) to utilize GPT-4o as the planner. We report element accuracy and the task success rate in Table 3. Specifically, we evaluate Aria-UI and the baselines on both zero-shot and training split-included settings. As we
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">AndroidControl-Low</th>
    <th colspan="2">AndroidControl-High</th>
    <th colspan="2">GUI-Odyssey</th>
  </tr>
  <tr>
    <th>Grounding</th>
    <th>Task SR</th>
    <th>Grounding</th>
    <th>Task SR</th>
    <th>Grounding</th>
    <th>Task SR</th>
  </tr>
  <tr>
    <td colspan="7"><i>Zero-shot</i></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>16.36</td>
    <td>5.12</td>
    <td>10.36</td>
    <td>2.84</td>
    <td>19.66</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Qwen2-VL</td>
    <td>64.24</td>
    <td>32.53</td>
    <td>30.32</td>
    <td>4.08</td>
    <td>49.56</td>
    <td>2.00</td>
  </tr>
  <tr>
    <td>SeeClick</td>
    <td>45.55</td>
    <td>17.72</td>
    <td>20.17</td>
    <td>4.29</td>
    <td>45.19</td>
    <td>1.45</td>
  </tr>
  <tr>
    <td>UGround</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>50.25</td>
    <td>2.02</td>
  </tr>
  <tr>
    <td><b>Aria-UI</b></td>
    <td><b>79.70</b></td>
    <td><b>54.39</b></td>
    <td><b>35.12</b></td>
    <td><b>5.95</b></td>
    <td><b>64.81</b></td>
    <td><b>5.28</b></td>
  </tr>
  <tr>
    <td colspan="7"><i>W. Training Set</i></td>
  </tr>
  <tr>
    <td>UGround</td>
    <td>74.28</td>
    <td>46.85</td>
    <td>37.98</td>
    <td>9.15</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Aria-UI</b></td>
    <td>85.71</td>
    <td>66.30</td>
    <td>41.78</td>
    <td>9.97</td>
    <td>84.57</td>
    <td>31.87</td>
  </tr>
  <tr>
    <td><b>Aria-UI<sub>TH</sub></b></td>
    <td><b>87.69</b></td>
    <td><b>67.33</b></td>
    <td><b>43.16</b></td>
    <td><b>10.17</b></td>
    <td>86.75</td>
    <td>36.47</td>
  </tr>
  <tr>
    <td><b>Aria-UI<sub>IH</sub></b></td>
    <td><b>87.20</b></td>
    <td><b>67.26</b></td>
    <td><b>42.97</b></td>
    <td><b>10.10</b></td>
    <td><b>87.02</b></td>
    <td><b>37.30</b></td>
  </tr>
</table>

Table 3: Results for offline mobile agent evaluation. We report element accuracy for grounding and the task success rate. For AndroidControl-High, GPT-4o serves as the planner to generate stepwise instructions for all methods.

<table>
  <tr>
    <th rowspan="2">Input</th>
    <th rowspan="2">Planner</th>
    <th colspan="2">Grounding</th>
    <th colspan="2">Cross-Task</th>
    <th colspan="2">Cross-Website</th>
    <th colspan="2">Cross-Domain</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>GPT-4</th>
    <th>Choice</th>
    <th>GPT-4</th>
    <th>SoM</th>
    <th>GPT-4</th>
    <th>SeeClick</th>
    <th>GPT-4</th>
    <th>UGround</th>
    <th>GPT-4</th>
    <th>OmniParser</th>
  </tr>
  <tr>
    <td rowspan="6">Image</td>
    <td>GPT-4</td>
    <td>46.4</td>
    <td>38.0</td>
    <td>42.4</td>
    <td>42.3</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>29.6</td>
    <td>20.1</td>
    <td>27.0</td>
    <td>25.6</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>29.6</td>
    <td>28.5</td>
    <td>30.7</td>
    <td>29.6</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>45.1</td>
    <td>44.7</td>
    <td>44.6</td>
    <td>44.8</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>42.4</td>
    <td>41.0</td>
    <td>45.4</td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>32.1</td>
    <td>33.1</td>
    <td>33.5</td>
    <td>32.9</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>47.7</td>
    <td>46.0</td>
    <td>46.6</td>
    <td>46.8</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td><b>56.1</b></td>
    <td><b>57.0</b></td>
    <td><b>59.5</b></td>
    <td><b>57.5</b></td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td><b>57.6</b></td>
    <td><b>58.0</b></td>
    <td><b>61.2</b></td>
    <td><b>58.9</b></td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td><b>57.6</b></td>
    <td><b>57.7</b></td>
    <td><b>61.4</b></td>
    <td><b>58.9</b></td>
  </tr>
</table>

Table 4: Results on Multimodal-Mind2Web, with grounding element accuracy reported. None of the methods adopted the training split, therefore we exhibit a fully zero-shot out-of-distribution evaluation.

evaluate Aria-UI with agent trajectories, we extend the model with two variants: Aria-UI<sub>TH</sub> and Aria-UI<sub>IH</sub>, for textual action history input and text-image interleaved history input, separately. We choose \( N = 1 \) for Aria-UI<sub>IH</sub> to include additional one GUI image from history during inference. For Aria-UI<sub>TH</sub>, we always input the full action history.

The results demonstrate the superior performance of Aria-UI across different evaluation settings and metrics. Specifically, Aria-UI and its variants consistently outperform existing baselines, with Aria-UI<sub>TH</sub> achieving peak performance of grounding accuracy and task success rate on AndroidControl, and Aria-UI<sub>IH</sub> achieving the best performances on GUI-Odyssey. Empirically, we found that the incorporation of historical actions, whether in text-only (\( TH \)) or text-image interleaved (\( IH \)) format, provides crucial context for accurate element grounding and task completion. In particular, we observe that the textual action history (Aria-UI<sub>TH</sub>) strikes an effective balance between efficiency and performance compared to both the base model and Aria-UI<sub>IH</sub>.

In summary, the significant performance gap between Aria-UI and existing approaches like SeeClick and UGround underscores the effectiveness of our proposed model in understanding and executing mobile interface interactions.

Web Agents. We evaluate how Aria-UI and its variants perform on multimodal Web agent tasks with the Multimodal-Mind2Web (Deng et al., 2024) benchmark. The original training split is not included by Aria-UI and the baselines during the training stage, thus we form a fully zero-shot out-of-distribution scenario. Three subsets, cross-task, cross-website and cross-domain are employed for a comprehensive evaluation.

Shown in Table 4, Aria-UI and its variants significantly outperform all baselines across the three subsets, achieving an average accuracy of 57.5% for the base model and 58.9% for Aria-UI<sub>TH</sub> and Aria-UI<sub>IH</sub>. Notably, Aria-UI<sub>IH</sub> demonstrates the strongest performance in the cross-website and cross-domain subsets, showcasing its robust ability to leverage historical multimodal context. The improvements over previous models, including UGround and SeeClick, underscore Aria-UI's ef-
<table>
  <tr>
    <th>Input</th>
    <th>Planner</th>
    <th>Grounding</th>
    <th>AndroidWorld</th>
    <th>MobileMiniWob++</th>
  </tr>
  <tr>
    <td>AXTree</td>
    <td>GPT-4-Turbo<br>Gemini 1.5 Pro</td>
    <td>Choice<br>Choice</td>
    <td>30.6<br>19.4</td>
    <td>59.7<br>57.4</td>
  </tr>
  <tr>
    <td>Image + AXTree</td>
    <td>GPT-4-Turbo<br>Gemini 1.5 Pro</td>
    <td>SoM<br>SoM</td>
    <td>25.4<br>22.8</td>
    <td>67.7<br>40.3</td>
  </tr>
  <tr>
    <td rowspan="4">Image</td>
    <td>GPT-4-Turbo</td>
    <td>UGround</td>
    <td>31.0</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>UGround</td>
    <td>32.8</td>
    <td>48.4</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>Aria-UI</b></td>
    <td><b>39.7</b></td>
    <td><b>60.4</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>Aria-UI<sub>TH</sub></b></td>
    <td><b>44.8</b></td>
    <td>-</td>
  </tr>
</table>

Table 5: Task success rate results for online mobile and Web agents on AndroidWorld and MobileMiniWob++.

<table>
  <tr>
    <th>Models</th>
    <th>OS</th>
    <th>Calc</th>
    <th>Impress</th>
    <th>Writer</th>
    <th>VLC</th>
    <th>Thunderbird</th>
    <th>Chrome</th>
    <th>VSC</th>
    <th>GIMP</th>
    <th>Multi</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>GPT-4o + SoM</td>
    <td>20.83</td>
    <td>0.00</td>
    <td>6.77</td>
    <td>4.35</td>
    <td>6.53</td>
    <td>0.00</td>
    <td>4.35</td>
    <td>4.35</td>
    <td>0.00</td>
    <td>3.60</td>
    <td>4.59</td>
  </tr>
  <tr>
    <td>CogAgent + SoM</td>
    <td>4.17</td>
    <td>2.17</td>
    <td>0.00</td>
    <td>4.34</td>
    <td>6.53</td>
    <td>0.00</td>
    <td>2.17</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td>GPT-4o + A11y</td>
    <td>41.67</td>
    <td>4.26</td>
    <td>6.81</td>
    <td>8.70</td>
    <td>9.50</td>
    <td>6.67</td>
    <td>15.22</td>
    <td>30.43</td>
    <td>0.00</td>
    <td>7.46</td>
    <td>11.21</td>
  </tr>
  <tr>
    <td>CogAgent</td>
    <td>4.17</td>
    <td>2.17</td>
    <td>0.00</td>
    <td>4.35</td>
    <td>6.53</td>
    <td>0.00</td>
    <td>2.17</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.10</td>
    <td>1.11</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>8.33</td>
    <td>0.00</td>
    <td>6.77</td>
    <td>4.35</td>
    <td>16.10</td>
    <td>0.00</td>
    <td>4.35</td>
    <td>4.35</td>
    <td>3.85</td>
    <td>5.58</td>
    <td>5.03</td>
  </tr>
  <tr>
    <td>GPT-4o + Aria-UI<sub>TH</sub></td>
    <td><b>25.00</b></td>
    <td><b>4.26</b></td>
    <td><b>15.32</b></td>
    <td><b>8.70</b></td>
    <td><b>30.06</b></td>
    <td><b>26.67</b></td>
    <td><b>23.80</b></td>
    <td><b>21.74</b></td>
    <td><b>19.23</b></td>
    <td><b>8.55</b></td>
    <td><b>15.15</b></td>
  </tr>
</table>

Table 6: OSWorld results. The top part denotes methods with both accessibility tree (A11y) and screenshot input, while the bottom part is for pure-vision methods that rely only on screenshots.

fectiveness in handling zero-shot grounding tasks on diverse and unseen web interfaces.

3.3 Online Agent Evaluation

Mobile and Web. We use AndroidWorld (Rawles et al., 2024a) for online mobile agent evaluation in an Android emulator environment. The evaluation is fully based on success of the task by checking the system state of the virtual device. We also include the MobileMiniWob++ task collection provided by AndroidWorld, which adapts the Web agent environment MiniWob++ (Liu et al., 2018) to AndroidEnv (Toyama et al., 2021), the same environment as AndroidWorld. We evaluate Aria-UI with the strongest baseline, UGround under the same M3A agent framework, compared with SoM and Choice methods that require AXTree input. We report task success rate, the most important metric for real agents in Table 5. Our observations are:
• In AndroidWorld, our approach achieves the best performance to date, with a task success rate of 44.8%, achieved by Aria-UI<sub>TH</sub>. This surpasses the previous state-of-the-art method, UGround, as well as non-pure vision methods such as SoM and Choice, which rely heavily on AXTree input. The results highlight Aria-UI’s superior ability to handle diverse element instructions in real-world settings, demonstrating its robustness and adaptability for pure-vision GUI agents.
• On MobileMiniWob++, Aria-UI outperforms UGround, and choice-based methods. Due to the simplicity of MiniWob++ layouts, GPT-4-Turbo with SoM achieves the highest performance. However, Aria-UI still demonstrates the highest scores with pure-vision input.

OSWorld. We further evaluate Aria-UI on the most up-to-date and complex computer use simulator benchmark, OSWorld (Xie et al., 2024). Following the pure-vision agent framework in OSWorld, we place Aria-UI as the grounding model to work collaboratively with GPT-4o on the 369 real tasks provided. We compared Aria-UI with previous SOTA methods and summarize the task success rate in Table 6. With GPT-4o as planner and Aria-UI<sub>TH</sub> as the grounding model, we achieve the highest average task success rate of 15.15%, outperforming previous methods across all computer-use scenarios in OSWorld. Notably, it excels in tasks like VLC (30.06%), Chrome (23.80%), and Impress (15.32%), highlighting Aria-UI’s strong performance in diverse, complex GUI tasks.

3.4 Ablation Study
Model Components.
• (-) Ultra Resolution. We remove the ultra resolution support for Aria-UI.
• (+) Visual CoT Prompting. We use CoT prompting for Aria-UI inference, as in Figure C.2.

Training Data Ablation.
• (-) Aria-UI Pipeline Data. We remove the data from our pipeline during training.
• (-) Diversified Instruction. We directly use refer. caption as input and coordinates as output for training, removing the diversified instructions.
• (-) Refer. as Supervision. We use only coordinates for supervision for our pipeline data.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Mobile</th>
    <th colspan="2">Desktop</th>
    <th colspan="2">Web</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Text</th>
    <th>Icon/Widget</th>
    <th>Text</th>
    <th>Icon/Widget</th>
    <th>Text</th>
    <th>Icon/Widget</th>
  </tr>
  <tr>
    <td>Aria-UI</td>
    <td>92.3</td>
    <td>73.8</td>
    <td>93.3</td>
    <td>64.3</td>
    <td>86.5</td>
    <td>76.2</td>
    <td>82.4</td>
  </tr>
  <tr>
    <td>(-) Ultra Resolution</td>
    <td>87.5</td>
    <td>61.1</td>
    <td>70.6</td>
    <td>40.0</td>
    <td>53.5</td>
    <td>40.3</td>
    <td>61.1</td>
  </tr>
  <tr>
    <td>(+) Visual CoT Prompting</td>
    <td>93.8</td>
    <td>59.8</td>
    <td>80.4</td>
    <td>51.4</td>
    <td>73.0</td>
    <td>57.8</td>
    <td>71.4</td>
  </tr>
  <tr>
    <td>(-) Aria-UI Data</td>
    <td>89.0</td>
    <td>60.7</td>
    <td>78.3</td>
    <td>34.3</td>
    <td>79.6</td>
    <td>52.9</td>
    <td>68.7</td>
  </tr>
  <tr>
    <td>(-) Diversified Instruction</td>
    <td>88.3</td>
    <td>67.2</td>
    <td>83.0</td>
    <td>57.1</td>
    <td>82.2</td>
    <td>63.1</td>
    <td>74.9</td>
  </tr>
  <tr>
    <td>(-) Refer. as Supervision</td>
    <td>92.7</td>
    <td>69.0</td>
    <td>81.4</td>
    <td>54.3</td>
    <td>85.2</td>
    <td>70.0</td>
    <td>77.5</td>
  </tr>
</table>

Table 7: Ablation study results on ScreenSpot.

We summarize the ablation results in Table 7. The results highlight the critical role of ultra resolution (Avg. 61.1) and Aria-UI data, particularly for Icon/Widget grounding. Removing diversified instruction or refer. as supervision degrades performance across platforms, due to weak alignment between instruction, refer. caption and grounding coordinates. We also found that adding CoT improves text-based tasks on mobile but struggles with others, caused by noise in visual reasoning.

**Context-aware Grounding Effect.** We propose two variants—text-only (\( TH \)) and text-image interleaved (\( IH \)) for Aria-UI —to evaluate the context-aware grounding design. We present the ablation results comparing the two variants and the base Aria-UI model in Tables 3, 4, and 5, across both offline and online agent settings. Notably, in the dynamic agent environment AndroidWorld, the context-aware model outperforms the base model by a significant margin of 12.8%. The results demonstrate that incorporating dynamic context substantially enhances the performance of Aria-UI.

4 Related Work

**Vision-language Grounding with Large Multimodal Models.** Foundational approaches for vision-language grounding, such as (Zou et al., 2023; Liu et al., 2023; Li et al., 2023), integrate CLIP with specialized vision models to tackle language-guided grounding tasks. To address the limitations in complex reasoning scenarios, researchers have begun leveraging LMMs (Liu et al., 2024; Dai et al., 2023; Shao et al., 2024) as a promising direction. Notable works (Peng et al., 2023; Pi et al., 2023; Wang et al., 2024b) train LMMs to respond to fine-grained language instructions by grounding them in specific visual regions, while general-purpose models (Bai et al., 2023; Li et al., 2024a) incorporate grounding as a core function during training. Additionally, significant advances in spatial information processing (Zhang et al., 2023b; Chen et al., 2023; Zhang et al., 2023c; You et al., 2023; Zhang et al., 2024b) have enhanced regional visual comprehension capabilities. However, these methods, while effective for natural images, face challenges when applied to GUI screenshots due to insufficient adaptation.

**General GUI Agents.** Automating GUI operations with capable agents has become a trending research area that leverages LMMs. Existing efforts have been put to design autonomous agents for complex task completion on mobile (Rawles et al., 2024a; Bai et al., 2024; Li et al., 2024c; Zhang et al., 2023a; Wen et al., 2024; Nong et al., 2024; You et al., 2024; Li et al., 2024d), Web (Koh et al., 2024; Yao et al., 2022a; Zhou et al., 2023; Lai et al., 2024; He et al., 2024; Abuelsaad et al., 2024; Ma et al., 2023; Zhang et al., 2024c) and desktop (Xie et al., 2024; Wu et al., 2024; Gao et al., 2023; Zheng et al., 2023; Zhang et al., 2024a; Niu et al., 2024) environments. These methods initially relied on HTML or AXTrees for element grounding to perform actions. Recently, several notable studies (Cheng et al., 2024; Gou et al., 2024) have proposed developing pure vision-based GUI grounding models with LMMs. However, due to their lack of instruction diversity and insufficient consideration of dynamic context, these approaches have delivered sub-optimal performances.

5 Conclusion

In this paper, we introduced Aria-UI, a robust LMM for GUI grounding across diverse environments. We designed a two-stage data pipeline for high-quality and diverse GUI grounding data from multiple platforms. We further incorporated dynamic action history as effective cues for stronger grounding capabilities in real-world environments. As a scalable and data-centric method, Aria-UI outperforms existing methods on all evaluated benchmarks, with both offline and online agent tasks. The model demonstrates strong zero-shot generalization across platforms, establishing Aria-UI as a powerful solution for universal GUI grounding.
6 Limitations

While Aria-UI demonstrates strong performance in grounding target elements based on instructions provided by a planner, it currently lacks the ability to autonomously perform both planning and grounding for a given task. This reliance on the planner model introduces a dependency on the quality and effectiveness of the planner, which can affect overall performance for complex tasks. Additionally, Aria-UI ’s training does not yet incorporate error correction for planner-generated instructions, limiting its ability to correct mistakes made by the planner during dynamic tasks. Future work will focus on enabling Aria-UI to perform integrated planning and grounding, as well as enhancing its ability to handle and correct planner errors in real-time.

References

Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. 2024. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032.

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. 2021. Uibert: Learning generic multimodal representations for ui understanding. arXiv preprint arXiv:2107.13731.

Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. 2024. Di-girl: Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406.11896.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135–146.

Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. 2024. Amex: Android multi-annotation expo dataset for mobile gui agents. arXiv preprint arXiv:2407.17490.

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195.

Wentong Chen, Junbo Cui, Jinyi Hu, Yuqia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. 2024. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.11317.

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seecllick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructclip: Towards general-purpose vision-language models with instruction tuning. Preprint, arXiv:2305.06500.

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdelmohsin, et al. 2024. Patch n’ pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36.

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.

Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. 2023. Assistgui: Task-oriented desktop graphical user interface automation. arXiv preprint arXiv:2312.13108.

Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2024. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243.

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290.

Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. 2024. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv e-prints, pages arXiv–2402.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to
objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798.

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649.

Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. 2024. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent. arXiv preprint arXiv:2404.03648.

Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. 2024a. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993.

Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. 2023. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767.

Hongxin Li, Jingfan Chen, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. 2025. Autogui: Scaling gui grounding with automatic functionality annotations from llms. arXiv preprint arXiv:2502.01977.

Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024b. On the effects of data scale on ui control agents. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. 2024c. Appagent v2: Advanced agent for flexible mobile interactions. arXiv preprint arXiv:2408.11824.

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020a. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776.

Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020b. Widget captioning: Generating natural language description for mobile user interface elements. arXiv preprint arXiv:2010.04295.

Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. 2024d. Ferret-ui 2: Mastering universal user interface understanding across platforms. arXiv preprint arXiv:2410.18967.

Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.

Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451.

Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. 2023. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172.

Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. 2024. Screenagent: A vision language model-driven computer control agent. arXiv preprint arXiv:2402.07945.

Songqin Nong, Jiali Zhu, Rui Wu, Jiangchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. 2024. Mobileflow: A multimodal llm for mobile gui agent. arXiv preprint arXiv:2407.04346.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824.

Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Linpeng Kong, et al. 2023. Detgpt: Detect what you need via reasoning. arXiv preprint arXiv:2305.14167.

Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. 2024a. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573.

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2024b. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36.

Swarnadeep Saha, Archiki Prasad, Justin Chih-Yao Chen, Peter Hase, Elias Stengel-Eskin, and Mohit Bansal. 2024. System-1.x: Learning to balance fast and slow planning with language models. arXiv preprint arXiv:2407.14414.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403.16999.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053.

Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glasea, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. 2021. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2105.13231.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2024b. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543–557.

Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. 2024. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456.

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972.

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022a. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. 2023. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704.

Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Sweargin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-ui: Grounded mobile ui understanding with multimodal llms. arXiv e-prints, pages arXiv–2404.

Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. 2024a. Ufo: A u-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939.

Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023a. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771.

Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. 2023b. Llava-grounding: Grounded visual chat with large multimodal models. arXiv e-prints, pages arXiv–2312.

Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. 2024b. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973.

Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. 2024c. xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215.

Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024d. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713.

Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. 2023c. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601.

Boyan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614.

Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. 2023. Synapse: Trajectory-as-exemplar prompting with memory for computer control. In The Twelfth International Conference on Learning Representations.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abhishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. 2023. Segment everything everywhere all at once. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 19769–19782.
A  Details on Aria-UI Datasets

A.1  Comprehensive Dataset Statistics

We summarize the statistical details of the single-step and context-aware grounding datasets in Table 8 and Table 9. The key points are:
1. Within each platform, the collection of Aria-UI dataset possesses the largest sample size, highlighting the scale-up capacity of our automated data pipeline.
2. Our collection adopts high-quality diversified instructions for the input text, while other publicly available datasets use plain tags from the tree information, or small-scale human annotations.
3. We use diverse trajectory data for context-aware training, with the average steps spanning from 5.5 to 15.4. For the text-based action history setting, we treat all previous actions for a specific grounding step as the context. For the text-image-interleaved setting, we adopt a window size of \( N = [1, 2, 3] \). We then scale up the available training samples to nearly 1M.

A.2  Pseudo-code for Desktop Grounding Data Scaling Agent Implementation

We present the pseudo-code for OS traverse agent with LMM (Gemini 1.5) guidance in Figure A.1. To summarize, the key ideas for developing the agent are:
1. The system is designed as a heuristic depth-first search over the OS environment. A large multi-modal model (Gemini 1.5) is employed to prioritize informative UI elements that are more likely to lead to novel system states, while avoiding interactions with exit elements until other options are explored.
2. We represent the system state by hashing all intractable elements on the current screen, using these hashes as unique state IDs. Empirically, this approach effectively identifies identical system states and discriminates between similar yet distinct states.
3. Operating system hotkeys, such as ESC and SPACE, are used to force a transition from the current state to a previous one for backtracking purposes.

A.3  Discussion on the Overlap between OSWorld Tasks and the Desktop Data

As discussion in Section 2.1.1, we traverse and scale-up desktop GUI data from general OS and software functionality interfaces on Ubuntu. For clarification, in the process, we do not target any specific downstream tasks in OSWorld, and we strictly do not load any task-specific file or pre-defined system states from OSWorld configurations, to prevent data leakage. For example, in Chrome, we traverse settings and general functionality but avoid browsing any specific webpages or configured set of pages that are part of the test examples. Similarly, in Impress and VLC, we collect general interface functionalities without accessing any specific slides or videos used in the test. Further, we only collect general single-app grounding screenshots. But the test tasks in OSWorld are more complex – 1/3 of them involves multiple opened windows on the screen with specific contents. Finally, our data collection also extends to other parts of the OS (e.g., Ubuntu’s system settings and app store), which are not within the scope of the OS-World benchmark tasks.

B  Aria-UI Training Details

We train Aria-UI with 64 NVIDIA H800 GPUs using the Megatron-LM (Shoeybi et al., 2019) framework. For the phase-1 single-step grounding training, we use a context length of 4096, which is kept the same as the original Aria-base model. The phase-1 training takes 18 hours with 10K steps. The phase-2 training aims at enabling Aria-UI’s context-aware grounding ability with trajectory-based samples. In this stage, we extend the model’s context length to 8192 to accommodate long trajectory data. It takes 6 hours for the phase-2 training with 2K steps. During training, expert parallelism (EP) was enabled with a factor of 8 to balance workload distribution across GPUs. Global batch size is set to 256, and we use a learning rate of 2e-5 for phase-1 training and 1e-5 for phase-2 training, separately. The minimum learning rate is set to 1e-8. We enable ViT training and set the learning rate of ViT parameters to 1/10 of the LLM’s learning rate for better performance in multimodal multi-task learning (Li et al., 2024a).

C  Prompts for Data Augmentation and Model Inference

In the following boxes, we present comprehensive and detailed prompts we use for Aria-UI’s data augmentation, training and inference.
<table>
  <tr>
    <th>Data Collection</th>
    <th>Platform</th>
    <th>Input Text</th>
    <th>#Screenshots</th>
    <th>#Samples</th>
  </tr>
  <tr>
    <td>SeeClick (Cheng et al., 2024)</td>
    <td>Web</td>
    <td>HTML Text</td>
    <td>270K</td>
    <td><span style="color:blue">3.3M</span></td>
  </tr>
  <tr>
    <td>Widget Captioning (Li et al., 2020b)</td>
    <td>Mobile</td>
    <td>Instruction</td>
    <td>14.4K</td>
    <td>101K</td>
  </tr>
  <tr>
    <td>RicoSCA (Li et al., 2020a)</td>
    <td>Mobile</td>
    <td>Instruction</td>
    <td>18.1K</td>
    <td><span style="color:red">173K</span></td>
  </tr>
  <tr>
    <td>UIBert (Bai et al., 2021)</td>
    <td>Mobile</td>
    <td>Refer. Caption</td>
    <td>16.9K</td>
    <td>16.9K</td>
  </tr>
  <tr>
    <td>GUIEnv (Chen et al., 2024)</td>
    <td>Web</td>
    <td>HTML Text</td>
    <td>50K</td>
    <td>700K</td>
  </tr>
  <tr>
    <td>GUIAct (Chen et al., 2024)</td>
    <td>Web</td>
    <td>Instruction</td>
    <td>13K</td>
    <td>67K</td>
  </tr>
  <tr>
    <td>OmniACT (Kapoor et al., 2024)</td>
    <td>Desktop</td>
    <td>A11y Text</td>
    <td>7.3K</td>
    <td><span style="color:olive">131K</span></td>
  </tr>
  <tr>
    <td>AutoGUI (Li et al., 2025)</td>
    <td>Web & Mobile</td>
    <td>Instruction</td>
    <td>693K</td>
    <td>693K</td>
  </tr>
  <tr>
    <td>Aria-UI Web</td>
    <td>Web</td>
    <td>Diversified Instr.</td>
    <td>173K</td>
    <td><span style="color:blue">6.4M</span></td>
  </tr>
  <tr>
    <td>Aria-UI Mobile (from AMEX (Chai et al., 2024))</td>
    <td>Mobile</td>
    <td>Diversified Instr.</td>
    <td>104K</td>
    <td><span style="color:red">4.8M</span></td>
  </tr>
  <tr>
    <td>Aria-UI Desktop</td>
    <td>Desktop</td>
    <td>Diversified Instr.</td>
    <td>7.8K</td>
    <td><span style="color:olive">264K</span></td>
  </tr>
  <tr>
    <th>Total</th>
    <th></th>
    <th></th>
    <th>1.37M</th>
    <th>16.6M</th>
  </tr>
</table>

Table 8: Statistics information for Aria-UI single-step grounding datasets. The collections with largest size of samples for each platform are highlighted in blue, red and olive, separately.

<table>
  <tr>
    <th>Data Collection</th>
    <th>#Avg. Steps</th>
    <th>#Trajectories</th>
    <th>#Samples</th>
  </tr>
  <tr>
    <td>AitW (Rawles et al., 2024b)</td>
    <td>9.67</td>
    <td>24.5K</td>
    <td>473K</td>
  </tr>
  <tr>
    <td>AitZ (Zhang et al., 2024d)</td>
    <td>7.5</td>
    <td>2.0K</td>
    <td>26K</td>
  </tr>
  <tr>
    <td>AMEX (Chai et al., 2024)</td>
    <td>12.8</td>
    <td>3.0K</td>
    <td>68K</td>
  </tr>
  <tr>
    <td>AndroidControl (Li et al., 2024b)</td>
    <td>5.5</td>
    <td>13.6K</td>
    <td>156K</td>
  </tr>
  <tr>
    <td>GUI Odyssey (Lu et al., 2024)</td>
    <td>15.4</td>
    <td>7.7K</td>
    <td>269K</td>
  </tr>
  <tr>
    <th>Total</th>
    <th></th>
    <th>50.8K</th>
    <th>992K</th>
  </tr>
</table>

Table 9: Statistics information for Aria-UI context-aware grounding datasets.

OS Traverse Agent with LMM Guidance

1. Initialize:
   - stack ← [(start_state, 0)] # Stack holds (state, depth)
   - visited ← {} # Track visited states using hashed entrances
   - memory ← {} # Store state transitions

2. While stack is not empty:
   - (current_state, depth) ← stack.pop()
   - current_hash ← hash(extract_entrances(current_state)) # Hash entrances to represent state
   - If current_hash in visited or depth > max_depth: continue
   - visited.add(current_hash)

2.1. Extract entrances:
   - entrances ← extract_entrances(current_state) # From accessibility tree

2.2. Rank entrances using LMM:
   - selected_entrances ← LMM.rank_and_select(entrances, memory)

2.3. For each entrance in selected_entrances:
   - next_state ← simulate_interaction(current_state, entrance)
   - next_hash ← hash(extract_entrances(next_state))
   - If next_hash not in visited:
      - stack.push((next_state, depth + 1))

2.4. Update memory:
   - memory[current_state] ← selected_entrances

2.5. Log transitions:
   - Record (next_state ← current_state) for debugging or visualization.
System Prompt for Element Captioning

Given two related images:
A context image: One of a 1x3 grid layout from a full UI screenshot, where a specific element is highlighted in a red box
A detail image: The isolated element itself

Task: Provide a comprehensive sentence describing the button by combining:
Visual properties (text, shape, color, icons)
Functionality (what the button does)
Position (both within the screen layout and in relation to nearby elements)
Any distinctive visual characteristics

The description must include ALL these elements, structured naturally in concise and accurate sentences.

Position descriptions should reference both:
Screen quadrant location (e.g., top-left, bottom-right)
Relative position to surrounding elements

Guidelines:
Do not mention the red highlighting box
Keep the description concise but complete
Include all specified properties
Reference surrounding elements for context
Incorporate the provided screen relative position: {relative_position}
Do not mention the position to the cropped image, only to the full screen layout

Satisfy these requirements to receive a reward. Failure to do so will result in a penalty.

Starting with "The [short text/shape/visual feature] [button/icon/menu/image/bar /...]" If the element has no text, use the most prominent icon or shape.

System Prompt for Aria-UI Single-step Grounding Training

<Input>
<img>Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction: {instruction}

<Output>
...
{coordinates}
...

<CoT Input>
<img>Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction: {instruction}
Think step-by-step, provide referring for the element first and then the grounded point coordinates.

<CoT Output>
```referring
{elem_caption}
```
```grounding
{coordinates}
```
System Prompt for Aria-UI Context-aware Grounding Training

The agent is performing the ultimate task: {ultimate_task}.

History of the agent's steps:\n{history_list}.

<Text-Image Interleaved Context>
|<img|>Step {step_idx}. Instruction: {prev_instruction}

|<img|>Step {step_idx}. Given a GUI image, what are the relative (0-1000) pixel point coordinates for the element corresponding to the following instruction or description: {instruction}

System Prompt for Instruction Diversification

Attention! You know a lot about GUIs on mobile, desktop and web. Given a detailed description of a GUI element, your task is to generate several user-oriented instructions that would require interacting with the GUI element.
For example:
Input: The 'Search Jobs' button, located at the center-right part of the image and just below the search bar, features a magnifying glass icon on a blue background, indicating its function to initiate a job search.
Output: The 'Search Jobs' button is key to starting or updating a job search after users enter or change their criteria. Instructions should focus on interacting with this button directly after inputting search terms or making adjustments.
...
"search for jobs."
"initiate job search."
"retry the job search."
"begin a new search"
```

Include the important identifications of the specific object to interact with.
    Examples:
Input: The "subscribe" button, colored in bright red with white text and a bell icon, is positioned in the upper-right section of ChefMaria's cooking channel header, showing "2.3M subscribers" underneath.
Output: (your reflection here)
```
"subscribe to ChefMaria's channel"
"click subscribe on ChefMaria's cooking channel"
```

First do a short reflection and give your answers that involve several possible user instructions. Wrap your answers in ``` as in the example. Use \n to separate multiple possible instructions.

{elem_caption}

Reviewer paper summaries: This paper proposes a pure-vision approach for GUI grounding to address the reliance on structured text like HTML. It introduces a scalable data synthesis pipeline to adapt diverse instruction and dynamic contexts during task execution.

This paper proposes a new pipeline to synthesize training data for pure-vision GUI grounding, including Web, desktop, and mobile environments.
Compared with existing methods for data synthesis in pure-vision GUI grounding, their pipeline features two core designs: 1) instead of directly using element caption as the input for their training data, this paper proposes to further generate diverse instructions for different captions and use instructions as the input for grounding; 2) in addition to single step training data (i.e., instruction-coordinates pairs), this paper also incorporates dynamic context as part of their input for grounding, motivated by the rationale that sometimes there could be contextual relationship between consecutive grounding actions such as expanding a multi-layered menu.
Based on their pipeline, this paper in total curated 11.5M training samples (combined with public data).
With the collected data, this paper fine-tunes an advanced OSS multimodal LLM, Aria, and achieves promising results on multiple benchmarks across different environments.
Summary of strengths: - The paper introduces a novel approach for GUI grounding. By avoiding the reliance on auxiliary inputs such as HTML, the model simplifies the pipeline and broadens its applicability to a wider range of GUI systems.
- This paper proposes scalable data synthesis pipelines with the potential to migrate to more GUI scenarios.
- The experiments cover different GUI domains and benchmarks, and the experimental results demonstrate state-of-the-art performance.

1. The dataset contributed in this paper can be valuable to the GUI agent community.
2. The evaluations in this paper are very comprehensive, spanning multiple benchmarks across the web, desktop, and mobile environments. The results also look promising.
Summary of weaknesses: - The Impact of History on Grounding Process: History is typically considered to affect the process of planning. However, this paper utilizes history to enhance the performance of grounding. This warrants deeper investigation. The role of contextual information in the grounding process requires additional experiments and analysis.
- Lack of Training Details: The paper lacks details regarding the training process, such as the training time, the number of GPUs used, or other computational resources. Providing this information would enhance reproducibility and allow a deeper understanding of scalability and practical feasibility of the model.

1. One major claim made in this paper is that dynamic context is important for grounding, and this motivates the core design of their training pipeline. However, this is not very convincing to me; the action history/contextual information is important for planning, but not as important for grounding. I can imagine in most cases, we can just rely on the description of the current step to ground an action. Also, no ablation on this design is shown in the experiments.
2. The comparison with baseline models may not be entirely fair due to the different choices of backbone models. For example, Uground, one major baseline in this paper, is based on llava. However, when Uground uses a more advanced backbone model like Qwen2. Its performance can be much better than Arial-UI. See https://gui-agent.github.io/grounding-leaderboard/
Comments, suggestions and typos: Refer to weakness

one clarification question:
When using Aria-UI with an off-the-shelf planner (e.g., GPT-4o) for agent tasks, do you always input the full context of the action history to Aria-UI, given that it is trained to handle dynamic context?
Overall assessment scores: 4

2.5
Metareview score: 3
Metareview: The paper proposes a novel pipeline for data curation for MLLM-based GUI grounding across web, desktop, and mobile environments. The work is solid with extensive experiments. 

Authors try their best during the rebuttal to convince reviewers on paper novelty, such as dynamic contexts, and data pipelines. However, two reviewers have negative feedback on this paper, and AC chose to side with them: the paper puts much efforts on data pipeline and combines various public datasets (which is really important to make a successful products). Though it shows limited contributions in architecture/algorithms and limited insights.
Summary of reasons to publish: The paper proposes a novel pipeline for data curation for MLLM-based GUI grounding across web, desktop, and mobile environments. This pipeline includes various types of instructions and incorporates dynamic context. The resulting open-source multimodal LLM, Aria, achieving good results on benchmarks. The paper is generally well-written and easy to follow.