OS-ATLAS: Foundation Action Model for Generalist GUI Agents
Authors: Zhiyong Wu
Zhenyu Wu
Fangzhi Xu
Yian Wang
Qiushi Sun
Chengyou Jia
Kanzhi Cheng
Zichen Ding
Liheng Chen
Paul Pu Liang
Yu Qiao
Venue: ICLR 2025 Spotlight (2025)
Keywords: GUI agent
language agent
GUI grounding
executable language grounding
Primary area: foundation or frontier models, including LLMs
TL;DR: Through innovations in both data and modeling, we are releasing the first foundational action model specifically designed for GUI agents.

Abstract: Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared to their closed-source counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD) scenarios. To facilitate future research in this area, we developed OS-Atlas—a foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling.
We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements. This dataset, combined with innovations in model training, provides a solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces.
Through extensive evaluation across six benchmarks spanning three different platforms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.

Published as a conference paper at ICLR 2025
OS-ATLAS: A FOUNDATION ACTION MODEL FOR GENERALIST GUI AGENTS
Zhiyong Wu'; Zhenyu Wu Fangzhi Xu ", Yian Wang²* Qiushi Sun Chengyou Jia', Kanzhi Cheng', Zichen Ding', Liheng Chen', Paul Pu Liang", Yu Qiao' Shanggaai Al Laboratory SShnnhai Jiaotong University The University of Hong Kong *MIT
wuzhiyong@pjlab org. cn
https://osan http://ssaaaass.ttttt......aa
github. io/
ABSTRACT
Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such GPT-4o and GeminiPro- Vision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared their closed-source counterparts, partie- ularly in GUI grounding and future research in this area, we developed OS-Atlas foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling. We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platfoma GUI grounding corpus to date, which contains over million GUI elements. This dataset, combined with innovations in model training, provides solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces.
(OOD) scenarios. To facilitate
Through extensive evaluation across six benchmarks spanning three different plat- forms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.
INTRODUCTION
With the recent adoption of large language models (LLMss, the fantasy of building digital agents (Wu al., 2024)- to JARVIS in The Iron Man-to automate daily tasks is evolving from science fiction into a tangible reality. Many current agents make decisions based on textual descriptions of the environments, such as HTML and accessibility trees, which is often lengthy (Zheng et al., 2024a), noisy (Cheng et al., 2024; WebAIM, 2024), and hard to acquire in practice. More recent studies (Cheng al., 2024; Hong et al., 2024b; Li et al., 2024) have explored the use of large vision- language models (VLMs) to develop graphical user interfaces (GUT) agents capable of performing complex tasks simply by analyzing the screen information-complete medium for agent's decision- making, allowing for greater flexibility. At the core of GUI agent lies an action model that enables GUI grounding- the process of transforming natural language instructions into executable actions within the operating system (e.g., clicking somewhere on the screen).
Despite their advancements, existing spen-source VLM-based GUI action models have been criti- cized for their poor performance in GUI grounding and generalizing to Out- Of-Dstrriiiiinnnnnn (OOD) scenarios (Lu al., 2024b; Chai et al., 2024), significantly restricting their applicability in real- world situations. The ineffectiveness of current models can be attributed to two primary factors.
First, most existing VLMM are rarely pretrained on GUI screenshot images. While some early efforts have focused on gathering screenshots corpus for websites (Lee et al., 2022; Chen et al., 2024b)
Equal Contribution.

---

Published as a conference paper at ICLR 2025
Grounding Mode
Action Mode
Agent Mode
(Web)
Action (Desktop)
Plugein
Planner
Deedopp Mubile
Ma
Grounding 5
Sprgk Agnntic Trits
Action (Mobilel
Crounding
Ginmnttzatio
SD
Agent (Web)
Agent (Mubille)
OS-Atlas
Agunt (Desktop)
Figure 1: (Left) The OS-Atlas model operates in three distinct modes to cater to various research needs. In Grounding mode, OS-Atlas predicts element coordinates based on user instructions and can be integrated with a planner module to create complete agent. In Action mode, OS- OS-Atlas functions independently to solve step-level agent tasks universally across different platforms and applications, in zero-shot OOD scenarios. In Agent mode, OS-Atlas undergoes further supervised fine-tuning to address specific agent tasks. (Right) Overall performance comparisons between OS-Atlas and other state-of-the-art models.
and mobile applications (He et al., 2020; Wang et al., 2021), there remains significant lack of a large-scale, open-source corpus of screenshots that encompasses multiple platforms (Windows, MacOS, Linux, iOS, Android), variety of applications, and different resolution sizes. Given that all GUIs operate under similar design principles, we believe that pre-training on such comprehensive corpus would enable GUI agents to achieve better GUI grounding, especially in OOD generalization.
Second, the heterogeneity content and format existing datasets (Zhang et al., 2024c; Chen et al., 2024c), along with the issue of action naming confficts, further undermines generalization. In current datasets, the same action is often labeled with different names across platforms. For instance, the "tap" action mobile devices and the "click" action desktop platforms are logically equivalent yet labeled differently. This inconsistency can create confusion during model training and ultimately result in decreased performance.
In this work, we are motivated to build a strong foundation action model to facilitate the development of future generalist GUT agents. Toward this goal, make the following contributions:
We have developed and released the first multi-platform GUI grounding data synthesis toolkit. This toolkit enables the automatic synthesis of GUI grounding data across various platforms, including Windows, macOS, Linux, Android, and the Web. By doing so, it significantly reduces the engineering efforts required for data curation future research.
2. Leveraging this data toolkit, we curated and open-sourced the largest multi multi--lattfom GUI grounding corpus to date, which comprises over 2.3 million distinct screenshots and more than 13 million GUI elements. Notably, our corpus includes desktop grounding data that has not been present in previous works. To facilitate evaluation of GUI grounding, we identify and re-annotate 11.32% incorrect samples in the popular benchmark ScreenSpot (Cheng et al., 2024) and release ScreenSpot-V2
Through the above data innovation and an approach resolving action naming conflicts during training, we developed OS-Atlas, a highly accurate foundation action model that operates universally across all GUIs. OS-Atlas can function in three different modes when developing GUI agents as depicted in Figure 1.
We present the most comprehensive evaluation of GUI agents to date, covering six benchmarks across three different platforms: desktop, mobile, and web. As shown in Figure 1, OS-Atlas demon- strates a superior performance improvement over previous SOTA models. This strong performance indicates that OS-Atlas can serve open-source alternative powerful commercial VLMs, such as GPT-4o, for developing future GUI agents.
2 RELATED WORK
GUI Agents and Large Action Models. Autonomous agents powered by LLMs, known as lan- guage agents (Weng, 2023; Sumers al., 2023), have recently gamered significant attention due
2

---

Published as a conference paper at ICLR 2025
to their interactive capabilities (Wang al., 2023; Sun al., 2023; Hong et al., 2024a; Durante et al., 2024). Recent efforts have begun to enable agents to interact with operating systems via programs (Sun al., 2024) API calls (Wu et al., 2024; Zhang et al., 2024a). However, the closed-source nature most commercial software imposed significant limitations, as agents don't t have access their internal APIs codes. Consequently, research shifts toward GUI-based agents that interact with digital devices through human-like mouse and keyboard actions (Cheng et al., 2024; Hong al., 2024b; Zheng al., 2024a). To facilitate effective agent interactions, Large Action Mod- els (LAMs)) have been developed to address general agentic tasks by interpreting human intentions and predicting actions in the form of lunction-calling (Zhang al., 2024c;b; Zeng et al., 2023; Yin et al., 2023). Nevertheless, progress is hindered by the limited quantity and vast diversity of available agent data (Li et al., 2024; Xu al., 2024). Specifically, LAMs focusing on GUI interactions remain underexplored, with only few attempts made to train GUT grounding models or agents (Cheng et al., 2024; Hong et al., 2024b; Gou et al., 2024).
To the best of our knowledge, OS-Atlas is the first LAM specifically designed for GUI agents.
GUI Executable Language Grounding. The core functionality of an LAM is to convert natural lan- guage (NL) instructions into actions and associated parameters (e.g., element coordinates), commonly known as GUI Executable Language Grounding, or simply GUI grounding. Existing GUI grounding training data be divided into two types: referring expression grounding (REG) (Liu al., 2023) and instruction grounding (IG) (Li et al., 2020). REG focuses on locating specific elements on the screen based on explicit references in the language instructions, such as "click the Open button." " Collecting REG data from webpages is straightforward through crawling and parsing (Cheng et al., 2024; Chen al., 2024b). However, when it comes to other platforms (e.g., desktop and mobile), presents significant challenges and often requires substantial human effort.
Compared REG, IG data more crucial for real-world applications. IG can be considered superset of REG, as it also includes actions that do not require specific coordinates, such as "Type". Moreover, the instructions in IG data are often nuanced and lack explicit element identification. For instance, instruction like "delete the last file" requires reasoning to identify the targeted action type and element. IG data is often limited in size and diversity (Zhang et al., 2024b; Zheng et al., 2024b), due to the need for human annotation during collection (Li et al., 2024).
OS-Atlas tackles these data-related challenges by developing a multi-platform infrastructure for collecting GUI grounding data. A concurrent study by Gou et al. (2024) also addresses these challenges; however, their focus is limited to scaling web data.
3 OS- AS-ATIAS
To establish robust foundation action model for GUI agents, we propose enhancements from both data ($ 3.2) and methodological ($ 3.3) perspectives. Leveraging these innovations, we trained OS-Atlas, the first foundation action model specifically designed for GUI agents.
3.1 TASK FORMULATION AND TRAINING
Our training process consists of two consecutive phases: (1) GUI Grounding Pre-training, which equips VLMs with the knowledge to understand GUT screenshots and identify elements on the screen, and on top of (2) Action Fine-tuning, which transforms instructions into executable GUI actions. The framework overview can be found in Figure 2.
GUI Grounding Pre-training. This phase requires a large, high-quality, and diverse set of <screenshot, element referring expression or instruction, element coordinate>> triplets, where the coordinates are represented as either points or bounding boxes. Models use the screenshot and the referring expression or instruction predict the corresponding element coordinates. To facilitate large-scale pre- pre-training, we have collected the largest muuti--lattorm GUI reference corpus to date and synthesized set of instruction grounding data using VLMs. detailed in $ 3.2. As shown in Table 1, our pre- pre-training corpus covers distinct platforms and includes over 2.3 million unique screenshots containing more than 13 million elements. We denote the pre-trained model as OS-Atlax Base.
3

---

Published as a conference paper at ICLR 2025
GUI Grounding Pretraining
Action Fine-tuning
Web
Desktop
Mobile
Thunight
Tass
"North"
Value 20.3
Search
Action
(819,830)
(407 (81)
(152.733
Unifled Space
Aclion
CLICK [ix
LONG, PRESS [[x
ScreenShots
>13M Elements
Diverse Platforms
SCROLL [[tc]
-..
TYP [Ted]
SELECT [Ek:)
VLM
OS-Atlas-Base
OS-Atlas
Figure 2: Overall training pipeline of OS-Atlas. We first perform large-scale pre-training using 13 million GUI grounding data collected to build OS-Atlas-Bar Basee. Next, we conduct multitask fine-tuning on agent data, resulting in OS-Atlas.
Action Fine-tuning. To enable OS-Atlas to solve OS tasks effectively, compile existing agent datasets for multi-task imitation learning. Specifically, we use <screenshot, task instruction, action history> triplets model input and train the model predict the corresponding action. Each action can be further represented as <thoughts, action type, action parameters(e.gg coordinates)>> triplets. In our preliminary investigation, we discovered that fine-tuning with multiple diverse datasets can introduce conflicts between actions, which degrade performance (see $ 5.3). To address this issue, we propose the use of unified action space during training (see $ 3.3).
3.2 GROUNDING DATA COLLECTION
As shown in Table 1, existing GUI ground- ing corpora predominantly focus web- page screenshots, as these can be easily ob- tained using web crawlers (Cheng et al., 2024; Hong et al., 2024b; Chen et al., 2024b) or on mobile screenshots (You al., 2024; Zhang et al., 2024d), leaving a sig- nificant gap for desktop screenshots. Fur- thermore, many of these corpora are either not open-sourced or are available only relatively small scales. To lay solid foun- dation for GUI agents, we have developed and open-sourced the first cross- cross-platfon platform GUI grounding data collection platform, along with dataset comprising 13 million GUI grounding instances that cover Windows, macOS, Linux, Android, and the Web. However, due to significant discrepancies between these platforms, we were required create distinct infrastructures for each one, which presents unique challenges in ensuring consistent data quality and compatibility across different environments.
#Sereenshots Mobile Desktop
Open Source
Dataset
#Elements
Web
SeeClick Ferret-Ul
270K
94K
3.3M
124K
x
<IM
GUICourse
73K
/
10.7M
CogAgent
400K
x
70M
OS-Atlas
1.9M
285K
54K
V
13.58M
Table 1: Statistics of the grounding data we collected compared to existing efforts. (For open-soure datasets, only count the amount of data made publicly avail- able.)
Web. We crawled about million web pages from the latest URLs obtained from Feb (Penedo et al., 2024), a cleaned and deduplicated English dataset derived from CommonCrawl. For each webpage, we extracted all visible clickable elements from the HTMI code - including buttons, scroll bars, search bars, hyperlinks, and SVG images with titles along with their referring expressions and coordinates derived from the associated HTML attributes. Unlike previous methods (Cheng et al., 2024) that primarily focused processing only the upper portions of websites, render entire websites and then segment them into 1920x 1080 resolution screenshots. This approach enhances the diversity of our web data by capturing more comprehensive view of each webpage.
By excluding all error pages (e.g., 404 errors), we initially curated 3.7 million webpage screenshots and 37 million elements. However, upon human examination, identified numerous low-quality
4

---

mobile device; Published as a conference paper at ICLR 2025
samples within this dataset. To address this issue, we implemented rule-based data filtering to exclude webpages that were either incompletely rendered or contained poorly distributed elements (e.g., all elements clustered at the bottom of the screen). Additionally, elements per webpage to 10 to encourage diversity. As a result of these stringent filtering criteria, we obtained a cleaned corpus consisting of 1.6 million screenshots and 7.7 million elements.
restricted the maximum number of
Desktop & Mobile. Capturing desktop and mobile screenshots is significantly more complex than collecting web screenshots. Previous methods primarily relied on manual collection, which resulted in relatively small dataset. However, large-scale automated data collection presents the following challenges: (1) the substantial engineering efforts required to set up a simulation environment for data collection within real operating system, and (2) the necessity of designing program to mimic human interactions with the operating system, thereby changing system states to obtain new screenshots.
For Android, we utilize AndroidEnv (Toyama et al., 2021) to create a simulation environment, and for Linux, we employ OSWorld (Xie al., 2024). Given the difficulties associated with virtualizing Windows and MacOS, we deploy the data synthesis platform on physical machines to collect data from these two operating systems. On these platforms, we leverage A Ally tree collect grounding data. Due to the differences in Al tree APIs and tools supported by each operating system, we utilize pyatspi to access the A ly tree on Ubuntu, pywinauto on Windows, and ApplicationServics. on macOS. We then simulate human-compune
interactions by sampling actions from the obtained Ally tree. In our simulation environment, we employ two different exploration methods: Depth-First Search (DFS) and Random Walk. We apply a similar data filtering pipeline to the grounding data obtained as we did for the webpages.
Instruction Grounding Data Collection. In addition the large-scale automated collection of referring expression data, we also annotated existing trajectory datasets using GPT-4o to obtain instruction grounding data. Given high-level task instruction along with the and-aer interface screenshots of an action, we instruct GPT-4o to carefully analyze the changes in the interface to derive sub-instruction for the current action. Specifically, we employ Set-of-Mark prompting (Yang et al., 2023) to indicate the locations of the operated elements, which helps GPT-4o better comprehend the screenshots. We annotated the training sets four trajectory datasets collected from both web and mobile platforms, namely Mind2Web (Deng et al., 2023b), AMEX (Chai et al., 2024), and AITZ (Zhang et al., 2024d). We also utilize instruction grounding data from two publicly available datasets: AndroidControl (Li et al., 2024) and Wave-UI
3.3 UNIFIED ACTION SPACE
Our preliminary investigation found that blindly mixing data from different sources for multitask fine-tuning can significantly harm performance due to action space conflicts. For instance, the action "click" in desktop environment is logically equivalent to the "tap" operation training with such conflicts can confuse the model. To address this issue, we propose a unified action space that standardizes the format of all existing datasets. Our unified action space comprises both Basic Actions and Custom Actions. The prompt can be found in Table 6.
Basic Actions. These are standardized and available across all platforms. They provide essential functionality and are defined with specifie format, ensuring consistency and reliability. In the current design, we have three basic actions: click, type, and scroll. This design significantly reduces the size of action space when fine-tuning, and facilitates knowledge sharing across platforms and apps.
Custom Actions. These are unique to each user's platform and device. They enable the model to support new and unseen actions defined by users. The design of custom actions is crucial to OS-Atlas's good out-of-distribution performance, as they allow for on-demand extensions to support previously unseen tasks and actions. Typical custom actions include open app (to open the specified application) and drag (to move an object to another location).
hhtps:/huuggingfacecco http:/gggggngggccoo////sstt//e/e co//aatsett/ggentte///aaaa We remove entries from ScreenSpot (Cheng et al., 2024). Mind2Weh, and Omniact avoid data contamination in downstream evaluation.
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS. GROUNDING TASKS
4.1 EVALUATION DETAILS
Benchmarks. We begin by conducting a comprehensive evaluation of the GUI grounding perfor- mance of OS-Atlas-Base Our evaluation utilizes ScreenSpot (Cheng et al., 2024), which assesses single-step GUI grounding capabilities across multiple platforms. During the evaluation, we identified approximately 11.32% percent annotation errors in the ScreenSpot dataset. To enhance the accuracy of our grounding evaluation, we corrected these errors and re-annotated certain examples, ensuring that the total number of test samples remains unchanged. In recognition of ScreenSpot's contributions, we have named the revised grounding dataset ScreenSpot-V2
Settings. Following Gou et al. (2024), we evaluate under two settings: 1) the Grounding Mode Setring, which utilizes planner model (e.g., gpt-4o) before grounding. The instructions from SereenSpot are treated as subtask instructions and input into the planner to generate more detailed instructions for the grounding models. 2) the Standard Setting without planner, which directly uses the original instructions from ScreenSpot.
Models. We consider two distinct backbone models: Qwen2-Ve VL (Wang et al., 2024), which is trained explicitly with GUI data, and Intern (Chen et al., 2024d) which is trained without GUI data. These models also differ in their handling of image resolutions. AnyRes (Liu et al., 2024; You et al., 2024) resize images and segment larger images into smaller patches, which are then encoded independently using vision encoders. In contrast, Qwen2-VL-7B supports arbitrary image resolutions by directly mapping an image into dynamic number of visual tokens. We denote our model as A---All--Bas-4/11 based on the backbones being used. Further details regarding the training setups can be found in Appendix E.
VL-22-4B employs
Baselines. We focus VLMs that are explicitly trained with GUI data, including Fuyu (Bavishi et al., 2023), CogAgent (Hong et al., 2024b), Qwen2-VL (Wang et al., 2024), SeeClick (Cheng et al., 2024), and even concurrent work UGround (Gou al., 2024). We omit general VLMs such as GPT-4V, as they are well-studies and perform poorly on ScreenSpot (Cheng et al., 2024).
Metrics. We follow previous practices by using grounding accuracy on ScreenSpot, where a prediction is considered correct if the predicted location falls within the ground truth element's bounding box. However, this metric does not capture more fine-grained grounding errors. Therefore, we also use Intersection over Union (IoU), widely used metric for measuring localization accuracy in object detection. IoU quantifies the overlap between the predicted bounding box and the ground truth bounding box.
4.2 RESULTS AND ANALYSIS
As shown in Table 2. under both settings, OS-Atlas-Bar significantly outperforms previous grounding models on ScreenSpot across mobile, desktop, and web platforms, achieving state-of ofee----the-arr the-art results. similar trend is observed in ScreenSpot- V2 (see Appendix B). Notably, even for VLMs like Qwen2- VL, which have been pre-trained on GUI screenshots, incorporating GUI grounding pre-training can further enhance grounding capabilities. To gain deeper insights into the reasons behind this strong performance, we conducted a series of analyses under the standard setting (without a planner), including those in $ 5.3, using Intern VL-2-4B due to GPU constraints.
The Effect of Grounding Data Scaling. We plot the changes in grounding accuracy and IoU of OS-Atlas Batlas--BaeeBBBB on ScreenSpot throughout the training process. As illustrated in Figure 3, grounding accuracy and IoU exhibit a clear positive correlation with the scaling of data, particularly in the case of loU and the web domain, where we have nearly 10 million elements. The correlation is relatively weak grounding accuracy because it cannot capture finer-grained errors. On one hand, this suggests the significant potential of continuously scaling the grounding data to further enhance performance. On the other hand, it underscores the need for more challenging benchmarks and improved metrics to effectively track performance improvements.

---

Published as a conference paper at ICLR 2025
(a) Web
(b) Desktop
Mobile
100
Text
lcon/Widget
82.61
85.71
84,98
72.16
61.17
70.00
62.89
60
45.71
60-
54.59
40
32.98
28.57
40
17.58
20
20
10.04
OS-Atlas-Base w/o IG w/o IG, Mobile, -48
OS-Atlas-Base w/o IG w/o Mo,boblle, -48
OS-Atias-asse w/o IG w/o iGbMoblle, -48
Desktop
Desktop
Desktop
Figure 4: Ablation studies and performance on ScreenSpot. IG/Mobile/Desktop refers to instruction grounding, mobile, and desktop grounding data, respectively.
Mobile
Desktop
Web
Planner Grounding Models
Avg.
Text Icon/ Soo/Wt
Text Icon/Widges Text Icon/Widge Widget
Fuyu
41.00
1.30
33.00
3.60
33.90
4.40
21.31
CogAgent
67.00
24.00
74.20
20.00
70.40
28.60
49.58
SeeClick
78.00
52.00
72.20
30.00
55.70
32.50
55.75
Intern VL-2-44
9.16
4.80
4.64
4.29
0.87
0.10
4.32
Aive22VVL-771
61.34
39.29
52.01
44.98
33.04
21.84
42.89
UGround-7B
82.80
60.30
82.50
63.60
80.40
70.40
74.15
OS-Atlas-Base-4BB
85.71
58.52
72.16
45.71
82.61
63.11
70.13
OS-Atlas-Base-7B
93.04
72.93
91.75
62.86
90.87
74.27
82.47
SeeClick
83.52
59.39
82.47
35.00
66.96
35.44
62.89
UGround-7B
93.40
76.90
92.80
67.90
88.70
68.90
82.71
GPT-4o
94.14
73.80
77.84
47.14
86.52
65.53
76.81
OS-
93.77
79.91
90.21
66.43
92.61
79.13
85.14
Table 2: Grounding accuracy on ScreenSpot. The best results are in bold.
Ablation. We first remove the instruction grounding (IG) data from the pre-training phase to conduct more controlled ablation analysis. Next, we further exclude mobile and desktop data to investigate whether pre- training solely on web data generalize to other platforms. The results presented in Figure reveal the following insights: (1) Referring expression data is nearly sufficient for training a strong grounding model and can be easily scaled compared to instruc- tion grounding data. (2) Despite the sim- ilarities between different GUI platforms, pre-training solely on web data struggles to generalize to other platforms. This empha- sizes the importance our data infrastruc- ture in facilitating the scaling of desktop and mobile referring expression data.
50
45
IoU (96)
25
25
Web
Desktop
Mobile
Action Acc. (96)
75
65
55
50
600
1200
1800
3000 Training Steps
2400
3600
Final
Figure 3: The effect of grounding data scaling two metrics. The performances on three different domains are reported.
4.3 APPLICATION: GROUNDING MODE
We evaluate how OS-Atlas Base work under the grounding mode Figure 1: can serve as a replacement for the grounding module of existing GUI agent, thereby enhancing overall perfor- mance, In this study, we benchmark our approach on the challenging OS agent testbed, OSWorld (Xie al., 2024). OSWorld is interactive environment just like computers, where the agent must
7

---

Published as a conference paper at ICLR 2025
interact with the operating system at each step and wait for response before proceeding to the next step. Refer to Figure 8 for a concrete example from the benchmark. Following their best practices, constructed screenshot-
GUI agent using GPT-4o. Given specific task, the agent generates a detailed, step-hy-step plan to accomplish it. It then executes this plan by generating actions and coordinates each step. We substitute these coordinates with those generated by an external grounding model, either OS-Atlas- -aaas- or SeeClick.
As shown in Table 3, although GPT-4o with OS-Atlas-Bas -aae the grounding module still lags behind human performance, it significantly outperforms other grounding methods such as SeeClick and Set-of-Mart (SoM). This demonstrates the potential of OS-Atlas-Bei as standalone grounding module for developing future GUI agents.
Successful Rate
Models
Avg.
OS
Cale
Impress
Writer
VLC
TB
Chrome
VSC
GIMP
WF
GPT-4o SoM
20.83
0.00
6.77
4.35
6.53
0.00
4.35
4.35
0.00
3.60
4.59
GPT-4o
8.33
0.00
6.77
4.35
16.10
0.00
4.35
4.35
3.85
5.58
5.03
SeoClick
16.67
0.00
12.76
4.35
23.52
6.67
10.86
8.70
11.54
7.92
9.21
OS-Atlas-Base-4BB
20.83
2.23
14.89
8.70
23.52
13.33
15.22
13.04
15.38
7.92
11.65
-a-AAtaa-BBae---
25.00
4.26
17.02
8.70
29.41
26.67
19.57
17.39
19.23
8.91
14.63
Human
75.00
61.70
80.85
73.91
70.59
46.67
78.26
73.91
73.08
73.27
72.36
Table Successful rate on OS World benchmark, divided by apps (domains). Workflow (WF) is special domain that requires navigation across multiple apps.
5
EXPERIMENTS: AGENT TASKS
5.1 EXPERIMENT SETUPS
Training details. Given that there are currently relatively few agent benchmarks, especially in the desktop domain, we have only utilized three datasets AMEX (Chai et al., 2024) (mobile), AITZ (Zhang et al., 2024d) (mobile), and Mind2Web (Deng et al., 2023a) (web) - to train our model, leaving a significant number of available benchmarks for OOD testing. For the sake of simplicity in notation, we denote our model as OS- OS-Atlas-4/7B, which reflects the different backbone models utilized: Intern VL-2-4B and VB-77.
Evaluation Benchmarks. We examine five distinct agent benchmarks across three different plat- forms: AndroidControl (Li et al., 2024) and GUI-Odyssey (Lu et al., 2024a) for mobile agents; GUI-Act- Web (Chen al., 2024a) and OmniAct-Webb (Kapoor al., 2024) for web agents; and OmniAct-Desktop for Windows environments. We only use the test split from these benchmarks for evaluation. Detailed statistics for these benchmarks can be found in Appendix C. Following common practices (Cheng et al., 2024; Deng et al., 2023a; Zhang et al., 2024d), we evaluate all benchmarks at the subtask granularity, as described each step based on the task instruction, the associated screenshot, and action history (if available).
1. This involves allowing the model to predict actions for
Settings and Baselines. We evaluate under two different settings to demonstrate two different practical applications of foundation action models like OS-Atlas: (1) zero-shot OOD setting (the Action Mode in Figure 1). In this setting, action models are benchmarked on unseen tasks, domains, applications in a zero-shot manner, mimicking real-world usage scenarios for GUT agents.; (2) supervised fine-tuning setting (the Agent Mode): In this setting, researchers fine-tune models on downstream tasks create agents specifically tailored for their intended applications.
In the zero-shot OOD setting, we use GPT-4o as the baseline, as existing VLMs perform poorly under this setting. For the supervised fine-tuning setting, we select LL-2, Qwen2-VL, and the grounding model, SeeClick, as our backbone for training.
Metrics.
We evaluate our models using three commonly used metrics for GUI agents that assess the accuracy of action type prediction, coordinate prediction, and step success rate, denoted as Type, Grounding, and SR, respectively. Type measures the exact match score between the predicted action types (e.g., CLICK, SCROLL.) and the ground truth, often referred as Type EM in the literature.
8

---

Published as a conference paper at ICLR 2025
GUI-Act-Web Grounding
OmniAct-Wel Grounding
OmniAct-Desktop Grounding
Models
Type
SR
Type
SR
Type
SR
7cro-shot OOD Setting
GPT-4o
77.09
45.02
41.84
79.33
42.79
34.06
79.97
63.25
50.67
OS-Atlas-4B
79.22
58.57
42.62
46.74
49.24
22.99
63.30
42.55
26.94
OS-Atlas-BB
86,95
75.61
57.02
86.12
69.35
59.99
90.24
62.87
56.73
Supervised Fine- tuning Setting
InterVVL-2-45
81 ,42
47.03
36.17
47.51
51.34
24.39
67.00
44.47
29.80
Qwen2-VL-72
89.36
90.66
82.27
89.22
85.94
78.58
96.27
94.52
91.77
SeeClick
88.79
78.59
72.34
86.98
75.48
68.59
96.79
70.22
72.69
OS-Atlas-4B
89.36
89.16
81.06
88.56
82.00
73.91
96.51
85.53
84.78
OS-Atlas-7B
89.08
91.60
82.70
97.15
95.41
93.56
97.15
95.85
94.05
Table Results web and desktop tasks. Intern I VL and OS-Atlas-47BB differ that the former utilizes the original checkpoints, while the latter is fine-tuned on OS-Atlas-Bar Base.
AndroidControl-Low Grounding
AndroidConntol--Hgg Grounding
GUI-Odyssey Grounding
Models
Type
SR
Type
SR
Type
SR
Zero-shur OOD Setting
GPT-40
74.33
38.67
28.39
63.06
30.90
21.17
37.50
14.17
5.36
OS-Atlas-4B
64.58
71.19
40.62
49.01
49.51
22.77
49.63
34.63
20.25
OS-Atlas-7B
73.00
73.37
50.94
57.44
54.90
29.83
60.42
39.74
26.96
Supervised Fine tune-tuning Setting
InterVV1.-2442
90.94
84.05
80.10
84.09
72.73
66.72
82.13
55.53
51.45
Qwen2-V1.-7B
91.94
86.50
82.56
83.83
77.68
69.72
83.54
65.89
60.23
SeeClick 93.00
73.42
75.00
82.94
62.87
59.
70.99
52.44
53.92
OS-Atlas-4B
91.92
83.76
80.64
84.69
73.79
67.54
83.47
61.37
56.39
OS-Atlas-7B
93.61
87.97
85.22
85.22
78.48
71.17
84.47
67.80
61.98
Table 5: Results on mobile tasks. IiternWL-2/Qwwe2-V1 and OS-Atlas-4/7B differ in that the former utilizes the original checkpoints, while the latter fine-tuned on OS-Atlas- Base. AndroidControl- Low refers to the scenario where both low-level and high-level instructions provided as inputs, while AndroidControl High indicates that only high-level instructions are given.
Grounding evaluates the performance of GUI grounding in downstream tasks. SR represents the step-wise success rate, where step deemed successful only both the predicted action and its associated arguments (e.g., coordinates for click action) correct. Appendix D provides detailed information on how these metrics are calculated.
5.2 RESULTS
The performances are presented Table 5. OS-Atla achieved SOTA performance across three different platforms, six distinct datasets, and two evaluation settings. In comparison with GPT-4o, our model demonstrated superior capabilities in addressing unseen tasks across all six OOD evaluation datasets, even the desktop domain models haven't seen during action fine-tuning. This suggests that in the realm of GUI agents, OS-Atlas has the potential to be a robust open-source alternative to leading commercial VLMs. Additionally, the results of the SFT setting further confirm that OS-Atlas can serve as robust foundation for researchers to train their custom GUI agents.
5.3 ANALYSIS
In this paper, we present two key research contributions: the development of a data infrastructure for grounding data synthesis and the proposal of unified action space. We conduct experiments to analyze the significance of these factors in enhancing the zero-shot OOD performance of a foundational action model.
First, we investigate the effect of grounding pre-training by training OS-Atlas directly from the original VLMs, which refer to as w/o pre-training. As illustrated in Figure 5, omitting the

---

Published as a conference paper at ICLR 2025
(a) Step-wise Success Rate
(b) Grounding Acc.
40
05-NUas-4B
60
Performances (%)
Pre-training
Performances (%)
Unity Action
50
30
40
20
30
20
Web
Desktop
Mobile
Web
Desktop
Mobile
Figure 5: Ablation studies on the zero-shot OOD setting. The results are reported respectively across three platforms.
pre-training stage significantly degrades performance, particularly the desktop and web platforms, where we have very limited data available for fine-tuning (7k samples for web and none for desktop). These results highlight the critical importance of the data infrastructure for grounding data synthesis; with this infrastructure in place, we can easily improve OOD downstream performance simply by scaling the pre-training corpus.
Next, we investigate the impact of removing the unified action space during fine- tuning, denoted as w/o unified action. For each fine-tuning dataset, adhere the optimal action space design proposed in SOTA models. As illustrated in Figure 5, we again observe a noticeable drop in performance. This validates our hypothesis that the conflicting action spaces indeed degrade model performance. Quantitatively, we find that employing
unified action space reduces the number of unique action types from to 10, effectively resolving several naming conflicts, such as between "tap" and "click", "press home" and "home", as well as "type" and "input".
5.4 OS-
To ensure that most datasets remain available for OOD evaluation, OS-Atlas initially trained using limited selection of 3 agent datasets. To fully leverage its po- tential for broader applications, we use all previously mentioned agent datasets for multitask fine-tuning. We report the average Success Rate (SR) across three do- mains: Web (GUI-Act-Web and OmniAct-Wert Mo- bile AAndroidControl-Low/Hiiii
0S-Atlas-4B
OS-Atlas-Pro Pro--B
Avg. Performance (%))
90
80-
77.49
79.33
70.93
70
and GUI-Odyssey), and Desktop (OmniAct-Deec Destktttp.. As illustrated in Figure 6, large-scale multitask fine-tuning significantly enhances model performance, thereby ensuring bet- ter user experience when deployed in real-world appli- cations.
60
Web
Desktop
Mobile
DS5-AtlasProo-P7
Avg. Performance(%)
100
57.04
94.05
90
88 11
$9.80
CONCLUSION
72.79
70
In this paper, we present OS-Adlas foundation action model for GUI agents. OS-Atlas demonstrates excep- tional performance in tackling open-environment GUI tasks across six complex benchmarks. This strong performance highlights the potential of OS-Atlas as an open-source alternative to powerful commercial VLMs, such as GPT-4o, for the development of future GUI agents.
Desktop
Mobile
Figure 6: OS-Atlas-Pro evaluation results.
10

---

Published as a conference paper at ICLR 2025
ETHICS AND REPRODUCIBILITY STATEMENT
This research focuses on constructing a foundation action model for generalist GUI agents. The data used are obtained either from synthesizing or reprocessing from previously released datasets, with all datasets or benchmarks properly cited. There are no discrimination, bias, or faireess issues that need be addressed in this paper. Further, our models are not expected to generate potentially harmful content. To ensure reproducibility, we provide all experimental details in Section 5 and their corresponding appendices. We will release all data, source code, and model checkpoints to support reproducibility.
ACKNOWLEDGMENT
This work is supported by the Shanghai Artificial Intelligence Laboratory, with sincere appreciation for the support.
REFERENCES
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL https:://w//tt adept ai/ blog/ Borgg/Euuu-Bbb
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents. arXiv preprint arXiv:2407.1744 2024.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang. Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile GUI agents. CoRR, abs/2406. 11317, 2024a. doi: 10.48550/ARXIV.246..111 1311. URL https://ddot /ttpp:://oo..oog/1..... 485.0/45550//.00 2406. 11317.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint ar arXiv:2406.11 11317, 2024b.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Desigming data and methods of effective agent tuning for large language models. arXiv preprint 12X.::2403.1288/ 2024c.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi gpt-4v? closing the gap commercial
Hu, Jiapeng Luo, Zheng Ma, al. How far multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024d.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935. 2024.
Xiang Deng, Yu Gu, Boyuan Zheng. Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.066 2023a.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL ptttps://openreee (d=kixgbO3wqM.
n forum?
Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori. Yusuke Noda. Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi. Hoi Vo. Li Fei-Fei, and Jianfeng Gao. Agent ai: Surveying the horizons of multimodal interaction, 2024. URL wtg./oo///////oooooo 03568.
11

---

Published as a conference paper at ICLR 2025
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024.
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby B. Lee, and Jindong Chen. Actionbert: Leveraging user actions for semantic understanding of user interfaces. In AAAI Conference Artificial Intelligence, 2020. URL https: http:////appp.eeaaimniiccoccoooe
semant icscholar. org/CorpusID :22::32226677
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representutions, 2024a. URL https::////..mc http:::://ppereee opwnrevieeeeeeeeeeeeeeeeeee net/ www.eeiwww..etttuuureereeeeeea
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pp. 14281-14290, 2024b.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 0rXiv:2001.08360 2020.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. CoRR, abs/2402. 2024. doi: 10.48550/ARXIV. 2402.17553. URL hwwps:ttt//........m //oii. org 10. 48550 arxiv. 2 12.4..2..555
Kenton Lee, Mandar Joshi, Iulia Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wa Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. ArXiv, abs/2210.03347,, 2022. URL
petps://apt) semant icshho ar. org/ :D: 42394.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents, arXiv preprint 1xXiv24406.03679 2024.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198-8210, Online, July 2020. Association for Computational Linguistics. doi: 10. /..18853/1//2220.aal main. 729. URL https: http///laaathhooocoooo 729.
ort/ acl acl-main.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge, 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0 305499999 2023.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. GUI odyssey: comprehensive dataset for cross-app GUI navigation on mobile devices. CORR, abs/2406 08451, 2024a. doi: 10. 00-4850/AAXXI..246..022 URL https::///// / doii org/10.../...
arwiv. 2406. 08451.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408. 00203, 2024b.
Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data scale, 2024. URL https: /wwxiiivvvooooooo org/ / 7557.
12

---

Published as a conference paper at ICLR 2025
Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive architec- tures for language agents, arXiv preprint arXiv:2309.02427, 2023.
Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. arXiv preprint 1.Xiv:2310000280) 2023.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint 2rXhv 2024.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. AndroidEnv: A reinforcement learning platform for android. abs/2105. 13231. 2021. URL http: /ttp://aaavvvvviiii wwtp:/s..rgga/ba..aeaa//o/bbbs / 2005 13231.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. The 34th Annual ACM Symposium on User Interface Software and Technology, 2021. URL https::////. appi. semant icschola arg/Corpus 236957064.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-R Ji-Rong Wen. A survey on large language model based autonomous agents. http://arrivv.rr//bb/22333 1432, 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 2024.
WebAIM. Webaim: The webaim million - the 2024 report on the accessibility of the top 1,000,000 home pages, 2024. URL https: // //webaim..memm//.//// http/project//miliono Accessed on September 30, 2024.
Lilian Weng. Lim-powered autonomous agents. illianweng.gihubbe Jun 2023. URL https:
//lilianweng git github..t....... io/ http//sss/////// 22222002206-2- 23 23-agent/.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao. Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:240.007972. 2024.
Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, and Zhiyong Wu. Interactive evo- lution: A neural-symbolic self-training framework for large language models. arXiv preprint arXiv:2406. 1136, 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:23 /1441, 2023.
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai- Wei Chang, Yejin Choi, and Bill Yuchen Lin. Lumos: Learning agents with unified data, modular design, and open-source llms. arXiv preprint www.x::3311006666 2023.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and The Gan. Ferret-ui: Grounded mobile ui understanding with multimodal lms.. arXiv preprint arXiv:2404.00 05719, 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310. /2823, 2023.
13

---

Published as a conference paper at ICLR 2025
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang. Qingwei Lin, Saravan Rajmohan, et al. Ufo: ui-focused agent for windows os interaction. arXiv preprins arXiv:2402.07939, 2024a.
Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint wrXi::2402115066 2024b.
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, al. xlam: A family of large action models empower ai agent systems. arXiv preprint arXiv:2409.053: 03, 2024c.
Jiwen Zhang. Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu for gui agents, 2024d.
Tang. Android in the zoo: Chain-of-action-thonn
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-Av(ision) is generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024a. URL https::/openrewe
net. / forum? h ddppiecKJ2D1B.
Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, An, and Shuicheng Yan. Agentstu- dio: A toolkit for building general virtual agents, arXiv preprint arXiv:2403.1790 2024b.
14

---

Published as a conference paper at ICLR 2025
Unitied Action Space Prompt
You foundational action model capable of automating tasks across various digital environ- ments, including desktop systems like Windows, macOS, and Linux, as well as mobile platforms such Android and iOS. You also excel in web browser environments. You will interact with digital devices in a human-like manner: by reading screenshots, analyzing them, and taking appropriate actions.
Your expertise covers two types of digital tasks:
Grounding: Given screenshot and description, you assist users locating elements mentioned. Sometimes, you must infer which elements best fit the description when they aren't explicitly stated.
Executable Language Grounding: With screenshot and task instruction, your goal is to determine the executable actions needed to complete the task. You should only respond with the Python code in the format described below:
You are now operating in Executable Language Grounding mode. Your goal is to help users accomplish tasks by suggesting executable actions that best fit their needs. Your skill set includes both basic and custom actions:
1. Basic Actions
Basic actions are standardized and available across all platforms. They provide essential function- ality and are defined with specific format, ensuring consistency and reliability.
Basic Action I: CLICK
purpose: Click at the specified position.
format: CLICK <point>> [[xaaxss, y-axis]]</point>
example usage: CLICK <point>[[101, 8/pon=/pponn>>
Basic Action 2: TYPE
purpose: Enter specified text at the designated location.
format: TYPE [input text]
example usage: TYPE [Shanghai shopping mall]
Basic Action 3: SCROLL
purpose: SCROLL in the specified direction.
format: SCROLL [direction (UP/DOWN/LEFTRRIGII
example usage: SCROLL [UP]
2. C.Custom Actions
Custom actions are unique to each user's platform and environment. They allow for flexibility and adaptability, enabling the model support new and unseen actions defined by users. These actions extend the functionality of the basic set, making the model more versatile and capable of handling specific tasks.
Your customized actions varied by datasets.
Table 6: The prompt for the action fine-tuning with unified action space.
A DATA STATISTICS
We detailed the statistics of the pre-training corpus collected in Table 7.
B
SCREENSPOT-V2
During our error analysis of Screenspot, we identified that several errors stem from incorrect or ambiguous annotations in the benchmark. Specifically, we observed the following issues:
1. Some instructions contain spelling mistakes or reference elements that are not present in the sereenshots.
2. Certain questions are ambiguous, allowing for multiple valid answers, while the ground truth includes only one of these options.
3. Several questions exhibit high degree of similarity to another.
15

---

Published as a conference paper at ICLR 2025
Training dataset
Type
Platform
Source
#Elements
#Screenshots
FineWeb-filtered
REG
Web
synthetic
7,779,922
1,617,179
Windows-desktop
REG
Windows
synthetic
1,079,707
51,726
Linux-desktop
REG
Linux
synthetic
41,540
1,186
MacOS-
REG
MacOS
synthetic
13,326
1,339
Pixel6-mobile
REG
Mobile
synthetic
104,598
21,745
SeeClick
REG
Web & Mobile
public
3,303,479
364,760
AMEX
REG
Mobile
public
1,097,691
99,939
Ulbert
REG
Mobile
public
16660
5682
Mind2Web-amnottted
IG
Web
GPT-4o
5,943
5,943
AITZ-annotated
IG
Mobile
GPT-4o
10,463
10,463
AMEX-annotated
Mobile
GPT-4o
5,745
5,745
AndroidControl
IG
Mobile
public
47,658
47,658
Wave-UT
IG
All platforms
public
65,478
7,357
Total
13,582,210
2,240,717
Table Grounding training datasets statistics overview.
Mobile
Desktop
Web
Planner
Models
Avg.
Text Icon/Widget
Text Icon/Widget Text Icon/Widget
SeeClick
78.39
50.66
70.10
29.29
55.22
32.52
55,09
OS-Atlas-Base-44
87.24
59.72
72.68
46.43
85.90
63.05
71.86
OS-Atlas-Basee 7B
95.17
75.83
90.72
63.57
90.60
77.34
84.12
SeeClick 85.17
58.77
79.90
37.14
72.65
30.05
63.60
GPT-4o
DS-Atlas-Base-4e
95.52
75.83
79.38
49.29
90.17
66.50
79.09
Ai-Atlas-Bas---
96.21
83.41
89.69
69.29
94.02
79.80
87.11
Table &: Grounding accuracy on ScreenSpot-v2. The best results are in bold.
4. Some ground truth bounding boxes are incorrectly labeled.
Given that the aforementioned factors could lead to biased evaluation results, we revised and edited the questions in the Screenspot benchmark. We ensured that the total number of questions remained the same in the release of Screenspot-v2. Our specific approach is outlined as follows:
We removed the problematic questions and replaced them with new ones.
2. We revised the instructions that were in the REG form and rewrote them as natural language instructions.
3. We corrected mislabeled ground truth bounding boxes.
We modified a total 63 out of 436 (s=14 14.4%) questions in the web domain, 28 out of 334 (3%.4%) in the desktop domain, and 53 out of 502 (== x0.6%.%) in the mobile domain. The evaluation results the new benchmark can be found in Table 8.
C DETAILS OF EVALUATION BENCHMARKS
We display the statistical details of the evaluation benchmarks in Table 9. Notably, AndroidControl- Low denotes that both low-level and high-level instructions are provided as the inputs, while AndroidControl- High denotes that only the high-level instruction the input. Although screenshots from the training set of AndroidControl are used during the pretraining phase, still classify it an OOD dataset because it contains diverse OOD splits that differ from the training set. GUI-Odyssey- Random/Task// Dasic/pv datasets four different test splits based the categories. We report the
16

---

Published as a conference paper at ICLR 2025
maero-average performance across these splits. For OmniAct, the original dataset only provides the initial screenshot and does not have the dynamic environment, thus we evaluate the first action of each example under the OOD setting (Action mode). While under the supervised fine-tuning setting (Agent mode), we evaluate all actions in the trajectories.
Benchmarks
Platforms
#Test Samples
History?
# Unified Actions
GUI-Act-Web
Web
1,410
342
Web
1,427
3+11
Omniact
Desktop
594
3+11
AndroidControl- AndroidControl High
Mobile
7.708
3+5
Mobile
7,708
3+5
QUT-Oddssey-Raadde
Mobile
29,414
346
GUI-Odyssey-Taan
Mobile
17,920
3+6
GUI-Odi Sill-Odyssey-DDee Device
Mobile
18,969
3+6
-dlls-yyyssyAApp
Mobile
17.455
3+6
Table 9: Details of the agentic benchmarks. History represents whether the history information of the previous actions is provided the input. #Unified Actions denotes the number of actions (basic actions + custom actions) for each task.
D DETAILS OF EVALUATION METRICS
To ensure fair comparisons across all baseline methods, standardize the evaluation metries for each action.
For click-based actions (e.g., CLICK, LONG_PRESS), the action models must generate both the action type and the position coordinates (x.y). Since the ground-trum bounding box is not always available the test data, we measure the performance by calculating the distance between the predicted coordinates and the ground-truth coordinates. Following Lu et al. (2024a), we consider the coordinates correct if they fall within a distance of 14% screen width from the ground truth.
type-based actions (e.g., TYPE, OPEN OPEN_APP) are considered correct if and only if both action type and action content are correct. We calculate the Fl score between the predicted text and the ground truth. The text is considered correct if F1 0...
For scroll action, the direction argument (i.e., UP, DOWN, LEFT, and RIGHT) must precisely match the ground truth.
For other actions PRESS PRESS PRESS_BACK), they must exactly match the ground truth to be considered correct.
E TRAINING DETAILS
OS-Atlas-Base and OS-Atlas (4B) Intern employs Dynamic Aspect Ratio Matching to process dynamic high-resolution input. We set the max dynamic e patch parameter to 6 to ensure the model captures sufficient pixel information.
a result, the input image, after resizing, is divided into maximum of 6 tiles of 448x448 pixels, along with thumbnail of the entire image to capture global context. In terms of grounding data format, to maintain consistency with the original InternVL training process, x2, /b]]</box>, where (x1, yl) and (x2, y2) are the normalized relative coordinates within the range [0,1000] Similarly, point data is converted into <point> > yl]< /////tint>>> format. <box>, </box/>, <point>, and </point> are treated as special tokens.
convert all box format data into the form <box>[[xl] yl,
OS-Atlas-Base and OS-Atlas (7B) Qwen2-VL can handle arbitrary image resolutions by map- ping them into dynamic number of visual tokens, offering more human-like visual process- ing experience. Through our experiments, we discover that setting the max pixel of image input 1024x 1024 during both training and inference yields excellent results for GUT grounding
17

---

Published as a conference paper at ICLR 2025
tasks, while also optimizing the model's training and inference cost. Similarly, maintain consis- tency with Qwen2-VL's VL's original grounding training format, we convert box data into the format (lox start >x1.y (xx2222))))) box end where < boxxx start and < |oxx end are treated as special tokens.
We follow SeeClick to preprocess grounding pre-training data, formatting each REG data sample into three types: point grounding, box grounding, and OCR. Each type of data wrapped up using 30 distinct GPT-generated prompts. To accelerate the training process, we group samples into single conversation, using 100 predefined prefix prompts.
F SCALING LAW OF GUI GROUNDING PRETRAINING
We also find that downstream task performance is not an ideal metric to measure scaling law. This is because downstream datasets often cannot accurately reflect the data distribution, and evaluation metrics are too coarse coarse-grained - for instance, correctly clicking an element does not necessarily mean the predicted coordinates exactly match the ground truth.
To study the scaling effect more rigorously, we plot the loss curve and, following Kaplan et al. (2020), fit power law-based scaling curve, as shown Figure 7. The horizontal axis represents the number of model training steps. Each step encompasses 1,024 samples, with each sample containing up to 15 grounding elements. The dark blue curve depicts the smoothed loss trajectory. From the figure, we observe a compelling trend suggesting significant potential for continually scaling pretraining data. Through our scaling law analysis, we estimate that increasing training data by times could lead to 40% relative reduction in loss. Moreover, scaling data by 64 times might potentially yield 57% relative decrease in loss.
0.7
Smoothed Loss Curve Predicted Curve
0.6
0.5
Loss
0.4
0.3
0.2
Data
64x Data
0.1
10 11
12
13
14
15
16
17
18
19 20
Number of Pretraining Steps (2^N)
Figure 7: The curve scaling law for the GUI grounding pretraining stage.
18

---

Published as a conference paper at ICLR 2025
G
CASE STUDY: OS- WO-LO
Please help me modify VS Code setting to hide all pycache_ folders in the explorer view.
Task
GPT-40
g
Grounding Model
Step 1
Step 2
Step 3
Click on the gear icon Settings
S
Click on \SSettings\" in the dropdown menu
S
Click on the search settings input box
[[364 1036]]
[[443, 844]]
[1948, 398]]
Step
Step
Step 6
Type \fiiles exclude (" in the search settings input box
Click on \'Add Pattern\" button under Files: Exclude
Sp
Type the pattern to exclude pycache folders
S
TYPE ('files: excluuee]
[1925, 76711
TYPE 1Ty Ppycchhe___ '
Step?
Step
Step 9
Click on the \'OKY" button to confirm the exclusion pattern
Press Enter to confirm the exclusion pattern
S
Close the settings tab using the keyboard shortcut
[[1221,759]]
ENTER
HOTKEY [CTRL, "W"]
Figure& 8: A case study from OS-World. OS-Atlas- Base works in the grounding mode, integrating GPT-40 as a task planner to create an agent. For each Click step, OtsAtlas-Base outputs the coordinates based on the provided step-level instructions.
19


Reviews:
[{"review_id": "mernQYEMX4", "reviewer": "Reviewer_eJ95", "summary": "The work introduces OS-ATLAS, a foundational GUI action model for GUI grounding and Out-Of-Distribution (OOD) agentic tasks. The authors synthesize large-scale multi-platform GUI grounding data. The OS-ATLAS model trains on this data and further fine-tunes with action datasets to address specific agent tasks. The model demonstrates significant performance improvements over previous models and shows the effectiveness of the synthetic GUI grounding dataset. The author also promises to open-source the data and related models, which will significantly impact the development of the GUI agent.", "questions": "See the Weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The work proposes a pretraining-then-fine-tuning framework for GUI grounding and agentic tasks, which is natural and unifies the training and evaluation process of these tasks.\n2. The work synthesizes a large-scale multi-platform GUI grounding data for pertaining and promises to open-source the data.\n3. The experimental results showed significant improvement, demonstrating their effectiveness.", "weaknesses": "1. The authors propose synthesizing large-scale GUI grounding datasets across platforms. However, the number of instances in Table 1 is not similar for each platform, especially for desktop (54K).\n2. There are some missing details: how to ensure the correctness of the synthetic data, the specifics of action tuning and the format of the inputs and outputs, and the precise meaning of \"thought.\"\n\nNote that, the score given is based on all concerns addressed in the discussion stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8KtJzagzy2", "reviewer": "Reviewer_Gb96", "summary": "This paper presents a project that aims to replace commercial VLMs such as GPT-4o in serving as the base model for GUI agents, with a special focus on GUI grounding and atomic action execution. The project involves a large scale grounding and action dataset, and a trained model named OS-Atlas. Benchmarking on various grounding and atomic action datasets show SOTA performance.", "questions": "Please see weaknesses for questions, mostly on the analyses part. The released data and model would be a good contribution to the community, looking forward to seeing it being released.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main claim of the paper is to serve as an open-source alternative to commercial VLMs like GPT-4o in building GUI agents. The claim is well supported, with the condition that all the data, code, and model are open-sourced as promised.\n\nSpecifically, the contribution mainly involves (1) a dataset with 2.3 million screenshots on various platforms and operating systems; (2) OS-Atlas (4B/8B) model that achieves SOTA performance on GUI grounding and atomic action execution.\n\nHaving a good base model and data would be beneficial for the community on future GUI research.", "weaknesses": "1. The contribution and novelty limited to the main claim of serving as an open-source alternative to commercial VLMs. Such VLMs like GPT-4o are not designed specifically for GUI scenarios, and fail to address GUI specific challenges. So does this work.\n\n1.1. The study does not touch any GUI specific challenges, such as specific visual and text distribution, cross-system differences, long action planning, and so on.\n\n1.2. It also does not present new evaluations that are more suitable for GUI. For example, setting up an emulator-style benchmark with the data synthesis toolkit.\n\nNonetheless, those aspects are beyond the claim of the paper, and future research on related topics could benefit from the data and model released by this study.\n\n2. Extra analyses would be beneficial:\n\n2.1. There could be more analyses on the data and model scaling, to help future study understand the amount of extra data and compute needed if the application needs further model enhancement or finetuning. The existing scaling analyses in Figure 3 and Sec. 4.2. are somewhat noisy and fail to give clear directions.\n\n2.2. To better support the claim on being effective “universally across all GUIs,” there could be more discussion on the per-task and environment performance. It remains unclear why certain tasks perform better or worse on web/desktop/mobile and different operating systems. Do such differences attribute to model, data, or task setting? And what are the corresponding data and compute needed to improve on a specific setting of interest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zZOQsOxrNv", "reviewer": "Reviewer_byA6", "summary": "The paper aims to enhance foundation model for GUI agent research. The authors introduce a GUI grounding synthesis tool and collect a large-scale dataset using the toolkit. Using the datasets, OS-Atlas is developed, which is a cross-platform foundation action model for GUI. Detailed results and evaluations are also discussed.", "questions": "- Could the authors comment on the usability of the toolkit and how other researchers could potentially customize it for their own use cases.\n- Any discussions on choosing a pertaining + fine-tuning over mixing grounding and action data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Open-source toolkit and data are available for the research community, which is currently lacking in GUI agent research area\n- Data collection and modeling processes are extensive and well-planned and motivated.", "weaknesses": "- I would appreciate more details about the toolkit, particularly on its usability and how other researchers could potentially customize it for their own use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RiJqQmYnIy", "reviewer": "Reviewer_giX8", "summary": "This paper proposes a high-performance GUI Agent, OS-ATLAS, trained on a synthetic grounding dataset and operating within a unified action space. Experiments on grounding tasks and agent tasks demonstrate that OS-ATLAS significantly outperforms previous state-of-the-art models.", "questions": "Which benchmark is used in Figure 5?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper produces a large-scale, open-source corpus of screenshots covering multiple platforms and proposes a high-performance GUI Agent that significantly outperforms previous VLM-based GUI action models, making a substantial contribution to the field of vision-based GUI Agents. Additionally, the paper is logically coherent, relatively complete in content, and well-organized.", "weaknesses": "Experimental Concerns\n\n1. Table 4 and Table 5 lack the results of state-of-the-art methods. It would be more appropriate to include these SOTA results for comparison with OS-ATLAS to demonstrate that it outperforms previous methods. Specifically, OdysseyAgent achieves 65.22% AMS (or SR, as referred to in this paper) on the GUI-Odyssey benchmark, which outperforms OS-ATLAS (4B/8B).\n\n2. OS-ATLAS (8B) performs worse than OS-ATLAS (4B) on the AndroidControl-Low/High and GUI-Odyssey benchmarks. Moreover, OS-ATLAS (8B) exhibits poorer performance on the AndroidControl-High and GUI-Odyssey benchmarks compared to SeeClick, which uses Qwen-VL as the base model and a lower fixed input resolution (448x448). It would be beneficial to provide an explanation for this performance gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)