Large Language Model-Brained GUI Agents: A Survey
Authors: Chaoyun Zhang
Shilin He
Jiaxu Qian
Bowen Li
Liqun Li
Si Qin
Yu Kang
Minghua Ma
Guyue Liu
Qingwei Lin
Saravan Rajmohan
Dongmei Zhang
Qi Zhang
Venue: Accepted by TMLR ()

Abstract: Graphical User Interfaces (GUIs) have long been central to human-computer interaction, providing an intuitive and visually-driven way to access and interact with digital systems. Traditionally, automating GUI interactions relied on script-based or rule-based approaches, which, while effective for fixed workflows, lacked the flexibility and adaptability required for dynamic, real-world applications. The advent of Large Language Models (LLMs), particularly multimodal models, has ushered in a new era of GUI automation. They have demonstrated exceptional capabilities in natural language understanding, code generation, task generalization, and visual processing. This has paved the way for a new generation of ''LLM-brained'' GUI agents capable of interpreting complex GUI elements and autonomously executing actions based on natural language instructions. These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands. Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software. This emerging field is rapidly advancing, with significant progress in both research and industry.
    
To provide a structured understanding of this trend, this paper presents a comprehensive survey of LLM-brained GUI agents, exploring their historical evolution, core components, and advanced techniques. We address critical research questions such as existing GUI agent frameworks, the collection and utilization of data for training specialized GUI agents, the development of large action models tailored for GUI tasks, and the evaluation metrics and benchmarks necessary to assess their effectiveness. Additionally, we examine emerging applications powered by these agents. Through a detailed analysis, this survey identifies key research gaps and outlines a roadmap for future advancements in the field. By consolidating foundational knowledge and state-of-the-art developments, this work aims to guide both researchers and practitioners in overcoming challenges and unlocking the full potential of LLM-brained GUI agents. We anticipate that this survey will serve both as a practical cookbook for constructing LLM-powered GUI agents, and as a definitive reference for advancing research in this rapidly evolving domain.

Published in Transactions on Machine Learning Research (06/2025)
Large Language Model-Brained GUI Agents: A Survey
Chaoyun Zhang Shilin He Jiaxu Qian', Bowen Li?, Liqun Li Si Qin Yu Kang", Minghua Ma', Guyue Liu Qingwei Lin', Saravan Rajmohan', Dongmei Zhang", Qi Zhang' "iicrsooft Shhnnhaai Artificial Intelligence Laboratory Peking University
Reviewed on OpenReview: https: openreview. net/ forun? id= d==ChwYYjuXTT
Abstract
Graphical User Interfaces (GUIs) have long been central to human-computer interaction, providing an intuitive and visually-driven way to access and interact with digital systems. Traditionally, automating GUI interactions relied on script-based or rule-based approaches, which, while effective for fixed workflows, lacked the flexibility and adaptability required for dynamic, real-world applications. The advent of Large Language Models (LLMs), particularly multimodal models, has ushered in a new cra of GUI automation. They have demonstrated exceptional capabilities in natural language understanding, code generation, task generalization, and visual processing. This has paved the way for new generation of "LLM-brained" GUI agents capable of interpreting complex GUI elements and autonomously executing actions based on natural language instructions. These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversat ional commands. Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software. This emerging field is rapidly advancing, with significant progress in both rescarch and industry.
To provide a structured understanding of this trend, this paper presents comprehensive survey of LLM-powered GUI agents, exploring their historical evolution, core components, and advanced techniques. We address critical research questions such existing GUI agent frameworks, the collection and utilization of data for training specialized GUI agents, the development of fine-tuned models tailored for GUI lasks, and the evaluation metrics and benchmarks necessary to assess their effectiveness. Additionally, we examine emerging applications powered by these agents. Through a detailed analysis, this survey identifies key rescarch gaps and outlines a roadmap for future advancements the field. By consolidating foundational knowledge and state-of-the-arl developments, this work aims to guide both researchers and practitioners in overcoming challenges and unlocking the full potential of LLM-powered GUI agents. We anticipate that this survey will serve both as a practical cookbook for constructing LLM-powered GUI agents, and as definitive reference for advancing research in this rapidly evolving domain.
The collection of papers reviewed in this survey will be hosted and regularly updated on the GitHub repository: httts:://gi Additionally, a searchable webpage is available at https: //aaaa aka. ms/ /ui agent for easier access and exploration.
http://aaaaaaaaaaaaaaaaidd------ -rn------nnadddd-------- Survey
1 Introduction
Graphical User Interfaces (GUTs) have been a cornerstone human-computer interaction, fundamentally transforming how users navigate and operate within digital systems Jansen 1998). Designed to make
Chhaoyun Zhang and Shilin He are corresponding authors: ahayyn..hhanghnscrooofff. shilin. crooofttooo

---

Published in Transactions on Machine Learning Research (06/2025)
User Request
GUI Agent
Web
Word
Photos
des Purtuggl
Extract
Summnarize
Observe
Creation
Sand
Teams
Click
PowerPoint
Adobe Acrobat
Illustration of the high-level concept of an LLMM-ooered GUI agent. The agent receives a user's natural language request and orchestrates actions scamlessly across multiple applications. extracts information from Word documents, observes content. in Photos, summarizes web pages in the browser, reads PDFs in Adobe Acrobat, and creates slides in PowerPoint before sending them through Teams.
Figure I:
computing more intuitive and accessible, GUTs replaced command-line interfaces (CIJs) Sampath et al. 2021 with visually driven, user-friendly environments. Through the use of icons, buttons, windows, and menus, GUIs empowered broader range of users to interact with computers using simple actions such as clicks, typing, and gestures. This shift democratized access to computing, allowing even non-technical users to effectively engage with complex systems. However, GUIs often sacrifice efficiency for usability, particularly in workflows requiring repetitive multi-step interactions, where CLIs can remain more streamlined Michalski
et al. 2006).
While GUIs revolutionized usability, their design, primarily tailored for human visual interaction, poses siguificant challenges for automation. The diversity, dynamism, and platform-specífic nature of GUI layouts make it difficult develop flexible and intelligent automation tools capable of adapting to various environments. Early efforts automate GUI interactions predominantly relied on script-based or rule-based methods
Hellmann & Maurer [2011]: Steven et al. (000).. Although effective for predefined workflows, these methods were inherently narrow in scope, focusing primarily tasks such soft testing and robotic process automation (RPA) Ivančić et al. (2019). Their rigidity required frequent manual updates to accommodate new tasks, changes in GUI layouts, or evolving workflows, limiting their scalability and versatility. Morcover, these approaches lacked the sophistication needed to support dynamic, human-like interactions, thereby constraining their applicability complex or unpredictable scenarios.
The rise of Large Language Models (LLMs} Zhao et al. (023b) Naveed et. al. 20023 especially those
augmented with multimodal capabilities Yin et al. (2023), has emerged as game changer for GUI automation, redefining the the way agents interact with graphical user interfaces. Beginning with models like ChatGPT et al. 2023c), LLMs have demonstrated extraordinary proficiency in natural language understanding, code generation, and generalization across diverse tasks Liu et al. 2024g Shen (2024): Feng et al. Zhao et al.
By LLMs, refer the general concept of foundation models capable of accepting various input modalities (c.g., visual language models (VLMss, multimodal LLMs (MLLMs)) while producing output exclusively textual sequences contributors 2024].
2

---

Published in Transactions on Machine Learning Research (06/2025)
2023b). The integration of visual language models (VLMs) has further extended these capabilities, enabling these models to process visual data, such as the intricate layouts of GUTs Hong et al. (2023) This evolution bridges the gap between linguistic and visual comprehension, empowering intelligent agents to interact with GUIs in a more human-like and adaptive manner. By leveraging these advancements, LLMs and VLMs offer transformative potential, enabling agents to navigale complex digital environments, execute tasks dynamically, and revolutionize the field of GUI automation.
1.1 Motivation for LLM-Powered GUI agents
With a LLM serving as its cognitive core, LLM-powered GUT automation introduces a new class of intelligent agents capable of interpreting natural language instructions, analyzing GUI elements, and autonomously executing corresponding actions. These L.M-powered GUI agents operate without reliance on complex platform-specific scripts or hard-coded workflows, enabling more flexible and generalizable approach to automation. We define them as:
Intelligent agents that operate within GUI environments, leveraging LLMs as their core inference and cognitive engine to generale, plan, and execule actions in flerible and adaptive manner.
This paradigm fosters dynamic, human-like interactions and real-time decision-making across diverse platforms, as illustrated in Figure To summarize, the motivation behind LLM-powered GUI agents can be distilled into the following three key aspects:
Overcoming the Limitations of Traditional Automation. Traditional GUI automation tools are typically constrained by rigid, rule-based logic and narrow task scopes. They struggle to generalize across diverse applications or adapt to dynamic environments. In contrast, LLM-powered GUT agents integrate natural language understanding, visual perception, and reasoning into a unified framework. This enables them to handle a broader range of tasks and react intelligently to context changes. Furthermore, unlike API-based agents agents-which depend on exposed or accessible interfaces GUI agents interact with applications through their graphical frontends. This GUT-centric approach allows non-intrusive automation even in closed-source or legacy applications, significantly broadening the applicability of these agents across platforms and ecosystems.
Democratizing Access Through Natural Interaction. By enabling users to control complex software systems using natural language, LLM-powered GUI agents lower the barrier to automation for non-technical users. They climinate the need for programming skills or manual scripting, making multi-stop operations more intuitive and accessible. Examples such as SeeAct [Zheng el al. (2024a) for web navigation, AppAgent /Zhang et al. (2023a) for mobile apps, and UFO Zhhang et al. (2024a) for Windows automation demonstrate the growing breadth of use cases. Multi-platform agents like CogAgent CogAgent Team (2024) further extend these capabilities across heterogeneous environments. These systems are increasingly resembling intelligent virtual assistants Guan et al. (2033 akin to J.A.R. from Tron fan-that understand user intent and perform tasks fluidly across applications.
Emerging Real-Worke Adoption. LLM-powered GUI agents are rapidly transitioning from research to deployment. For instance, Microsoft Power Automate integrates L.Mss to assist. users building automation
workflows across applications using natural language 2 Similarly, Al copilots in productivity software like
Microsoft Copilof are bridging the gap between intent and application behavior. These agents also hold
https://copilot. microsoft.com
3

---

Published in Transactions on Machine Learning Research (06/2025)
promise for improving accessibility for example, enabling visually impaired users to control GUI interfaces using voice text input Aljedaani et al. (2024).
The convergence of LLMs and GUI automation has sparked growing research interest, spanning topics such as application frameworks Zhang et al. (2024a],, data collection and GUT grounding Cheng et al. (0024a), model optimization Hong et al. (2023), and evaluation methodologies Zhuge al. (2024). This momentum reflects both the transformative potential and the technical complexity of the field. Despite recent advances, fundamental challenges remain- suchhsschhh robust GUI grounding, error recovery, and generalization across unseen interfaces. However, comprehensive survey of this rapidly evolving domain is still lacking, creating an urgent need for systematic synthesis and analysis. Our work aims to address this gap.
1.2 Scope of the Survey
To address this gap, this paper provides pioncering, comprehensive survey of LLM-powered GUI agents. We cover the historical evolution of GUI agents, provide step-by-step guide to building these agenls, summarize essential and advanced techniques, review notable tools and research related to frameworks, data and models, showcase representative applications, and outline future directions. Specifically, this survey aims to answer the following research questions (RQs):
1. RQI: What are the key background, enabling factors, and unique characteristics of LLM-powered GUI agents? (Section 3
RQ2: What are the essential components and advanced technologies that form the foundation of LLM-powered GUI agents? (Section
3. RQ3: What are the principal frameworks for LLM GUI agents across web, mobile, computer and multi-platform and what are their defining characteristics? (Section 5
4. RQ4: What are the existing datasets, and how can comprchensive datasets be collected to train optimized LLMs for GUI agents? (Section 6
RQ5: How can the collected data be used to train purpose-built Large Action Models (LAMs) 2 for GUT agents, and what are the current leading models in the field? (Section
6. RQ6: What metries and benchmarks are used to evaluate the capability and performance of GUI agents? (Section S
RQ7: What are the most significant real-world applications of LLM-powered GUI agents, and how have they been adapted for practical use? (Section g
8. RQ8: What are the major challenges, limitations, and future research directions for developing robust and intelligent GUI agents? (Section 10
Through these questions, this survey aims provide comprehensive overview of the current state of the field, offer a guide for building LLM-powered GUI agents, identify key research gaps, and propose directions for future work. This survey is one of the pioneers to systematically examine the domain of LLM-powered GUT agents, integrating perspectives from LLM advancements, GUI automation, and human-computer interaction. To start, we present a structural overview in Figure 2 For ease of reference, a list. of abbreviations used throughout the paper is provided in Table !
2 Related Work
The integration of LLMs with GUI agents an emerging and rapidly growing field of rescarch. Several related surveys and tutorials provide foundational insights and guidance. We provide a brief review of existing
IIn our context, LAM denotes fine-tuned large language or Assionllnguuge model that specifically tailored for executing GUT agent tasks.
4

---

Published in Transactions on Machine Learning Research (06/2025)
Moticstion MiPooweed Agente
Introduction
Ecope Strvey
Buveys on ELM Agrits
Related Work
22
Hewy on GLL Artonsstoo
R1
arg Langange Modcle Founltikss and Capulditics
42
JM Agude: From Langrage Artivt
Backgrooud
Autonaaioos: Tods, Todnigaos Chullsnngs
3ET Agee
API Bood Agra)
eevinien to H.M-Poooee (1I1 Agrata
Ds.nteggnted Ageric Oppertecils Chalesgs
Aecaltecture and Weeilor in Nitdcel
Operating Lnviinntzen
Porcrztion
Seciossec Reclock
Prospe Enginerria
Peasitg
Mode Inference
12 Aeice lakerano
Smmpkeaararr Outpets
LLM- MMMPood Agence Foundations Design
Opencken
46 KrEios Exrention
aaat-lun Menuer
Menoryy
Lig Ter Mirrey
Miesoed Falenements
Seggtter Votze-Beed GL Grensdns
Puuz Foundtiiser
Malkkger Prizenock
atiin &Radhma
GUI Agena]
U.M-Pooeeed GUI Agent Pramowork
Agrits
biifomeent Loursiny
Tempoter GUI Ageste
Pmnney Takmerns
Iow Paafor Agrezs
Large Language Modne-
Nui Culladion
GUI Agents
Wilelio Pipllie)
Ves Agrez Dsta
c Data for Optimising
Lowe-ooered GUI Aquata
6.5
Boidk Agcai Duis
8. Campottr Aguat Duis
Plaffra Agent
Red Seee-e Medkle
Modtk
Oper-Soem Korkle
ange Action Middl
Alte for Web GUI Aguets
Modele for Opeimlsing
LcM-Powered CUI Ageats
AS6 Fa Molik: GUI Aguata
AS6 ke Cospaler GUI Aguata
1
Yoee Platforn Large Action Modiik
Surrying Trrah Anni Rapal Inprovencnts LAM Capablitice
Sallution Nervics)
Platiome
Ievaluttinn for Powered GUL Agouts
Agrst Bardhuzkk
Moble Agest Berehmark
Sappltr Ageal Beadhusrk
Ioce Patforn Beschancck
Pag Hepky
p Applications of LLM- Powered GUT Ageuts
2D Tweing
Netionste
Josoliala
Princy Corcrme
Jjr Saroo Pat
aterry, Pefermoton
Productios
Seky un Reholdlls
Limifaations Challenges
Emaa-AApe Interaction
Randampp
10.3
Caromietiin and Ponsaaniiatiin ktn
Conciusion
Raical and ltaysiooeey Challengu
10.7
Relliiiiky und Conerlieation
10.8
Suumuny
Figure 2: The structure of the survey on LLM-powered GUI agents.
overview articles on GUI automation and LLM agents, as these lopics closely relate to and inform our research focus. To begin, we provide an overview of representative surveys and books on GUI automation, LLM agents, and their integration, as summarized in Table [ These works either directly tackle one or two core areas in GUI automation and LLM-driven agents, or provide valuable insights that, while not directly addressing the topic, contribute indirectly to advancing the field.
5

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: List of abbreviations in alphabetical order.
Acronym
Explanation
AT
Artificial Intelligence
AITW
Android in the Wild
AITZ
Android in The Zoo
API
Application Programming Interface
CLI
Commandd Line Interface
CLIP
Contrastive Saaguage--magee Pre- Training
CoT
Chain-of-Thought
CSS
Cascading Style Sheets
CuP
Completion under Policy
CV
Computer Vision
DOM
Document Object Model
DPO
Direct Preference Optimization
GCC
General Computer Control
GPT
Generative Pre-trained Transformers
GUI
Graphical User Interface
HCI
Human- Compan-C Interaction
HTML
Hypertext Markup Language
ICL
In-Context Learning
loU
Intersection over Union
LAM
Large Action Model
LLM
Large Language Model
LSTM
Long Short- Memory
LTM
Long-Term Memory
MCTS
Monte Carlo Tree Search
MoE
Mixture of Experts
MDP
Markov Decision Process
MLLM
Multimodal Large Language Model
OCR
Optical Character Recognition
OS
Operation System
RAG
Generation
ReAct
Reasouing and Acting
RL
Reinforcement Learning
RLHF
Reinforcement Learning from Human Feedback
RNN
Recurrent Neural Network
RPA
Robotic Process Automation
UI
User Interface
UX
User Experience
VAB
VisuaJAgentBenchh
VLM
Visual Language Models
ViT
Vision Transformer
VQA
Visual Question Answering
SAM
Segment Anything Model
SoM
Sct-of-Mark
STM
Short- Term Memory
6

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Summary of representative surveys and books on GUI automation and LLM agents. A Vsymbol indicates that a publication explicitly addresses a given domain, while an O symbol signifies that the publication does not focus on the area but offers relevant insights. Publications covering both GUI automation and LLM agents are highlighted for emphasis.
Scope
Survey
One Sentence Summary
GUI Automation
LLM
LLM Agent +
Agent
GUI Automation
Li & Wi 2006
A book on how to develop an antomated GUI testing tool.
Rodriguez- Valdés et al.
A survey on automated GUI testing in 30 years.
22021
Arnatovich & Wang 2018
A survey on automated tochniques for mobile functional GUT lesting.
el al. Ivancié el sl.
2019
A literature review on RPA.
Said et a. 2020
An overview on mobile GUI testing.
2023
An survyy on Android GUT testing.
Oesanen 2023
GUT testing on Windows OS.
Deshmukh ct m
2023
A survey on GUT testing for improving user experience.
Bajommaa of a|
2020
A survey on the use of computer vision for sottware engineering.
Yu ct 0l.
2028
A survey on using computer for mobile app GUI testing,
5yed ct al. 2020
A review of coutemporary themes and challenges in RPA.
/
Claakraborl et a.
2020
A review of emerging trends of intelligent process automation.
Euriquez et al
2020
A scientific and industrial systematic mapping study of RPA.
Riberro ei g.
2121
A review of combining Al and RPA in industry 4.0.
Nass el al
21
Discuss the chanllenges of GUI testing.
Ägostinelli ii et al.
2019
Discuss the research challenger of intelligent RPA.
Wall rt m.
X223
A review on task antomation with intelligent agents.
Zhao et ml.
2023b
A comprehensive survey of LLMs,
Zhao ml.
2023B
A survey of LLM-based agents.
Cheng et
2021b
An overview of LLM-based Al agent.
Li ct al.
202 11
A survey personal LLM agents their capability, efficiency and security.
Xi ml.
2023
A comprehensive survey of LLM- agents.
Wang et a..
2024g
A survey L-Mebased autonomous agents.
Guo sl
2021b
A survey of mult-agent LLM frameworks.
Han et
2124
A survey LLM multi-agent systems, with their challenges and
problems.
Sun et al.
2024g
A survey L.MM-b multi-agent reinforcement loarning.
[luang et al
2024b
A survey planning in LLM agents.
4
Agghzal ct al
21/23
A survey sutomated planning in LLMs.
Zheug
202bc
Discuss the roadmaa of lifelong learning in LLM agents.
/
Zhang
2/224q
A survey the memory of LLM-basod agents.
Sten [2124
A survey of the tool usage in LLM agents.
Chang el
2024
A survey evaluation LLMss
Li cl al.
202-
A survey benchmarks mulsimodal opplications.
T cl al.
2025
A survey benchmarking evaluations, applications, and challenges of vieual L.Mss.
Huang & Zhang
2024
A survey eval raaluation of multimodal LLMss
Xie
(2024a
A survey LLM based multimodal agent,
Durante al.
2024
A survey of multimodal interaction with Al agents.
Wi al. 2024a
A survey of foundations and trend mulsimodal mobile agents.
Wang et
2024k
A survey the integration of foundation models with GUT agents.
Gao
X024F
A survey autonomous agents across digital platforms.
Nguyen
2024
A survey GUI agents.
Lau
2025b
A survey GUI agent on phone automation.
Huu
202:b
A survey MLLM based agents for OS.
A survey building trustworthy GUI agents.
Ning
A survey of agents for Web automation.
Tang
A survey GUI agenls powered by (nultimodal) LLMs.
Sageer
A review agent computer use.
work
A comprehensive survey I.LM-brained GUI agents, on their foundations, technologies, frameworks, data, models, applications, challenges and future roadmap.
2.1 Survey on GUI Automation
GUI automation has long history and wide applications in industry, especially in GUI testing Li & Wu
(2006); Rodríguez- Valdés et. al. (2021); Arnatovich & Wang (2018) and RPA Ivančić et al. (2019) for task
automation Wali et all (2023
7

---

Published in Transactions on Machine Learning Research (06/2025)
Said et al., Said et al. (2020) provide an overview of GUI testing for mobile applications, covering objectives, approaches, and challenges within this domain. Focusing on Android applications, Li T.i (2023) narrows
the scope further, while Oksanen et al., Oksanen (2023) explore automatic testing techniques for Windows
GUI applications, a key platform for agent operations. Similarly, Moura et al., Moura et al. (023) review GUT lesting for web applications, which involves diverse lools, inputs, and methodologies. Deshmukh el al., Deshmukh et al. (2023) discuss automated GUI testing for enhancing user experience, an area where LLMs also bring new capabilities. A cornerstone of modern GUI testing is computer vision (CV), which is used to interpret UI elements and identify actionable controls Bajammal et al. (2020). Yu et al., Yu et al. (2023) survey the application of CV in mobile GUT testing, highlighting both its significance and associated challenges. In LLM-powered GUI agents, application UI screenshots are equally essential, serving as key inputs for reliable lask comprehension and execution.
On the other hand, RPA, which focuses on automating repetitive human tasks, also relies heavily on GUT automation for relovant processes. Syed et al., Syed al. (2020) review this field and highlight contemporary RPA themes, identifying key challenges for future research. Chakraborti el al., Chakraborti et al. (2020) emphasize the importance of shifting from traditional, script-based RPA toward more intelligent, adaptive paradigms, offering a systematic overview of advancements in this direction. Given RPA's extensive industrial applications, Enriquez et al., Enríquez et al. (2020) and Ribeiro et al., Ribeiro et al. (0021) survey the field from an industrial perspective, underscoring its significance and providing a comprehensive overview of RPA methods, development trends, and practical challenges.
Both GUI testing Nass et al. (2021) | and RPA Agostinelli et al. 2019) continue to face significant challenges in achieving greater intelligence and robustness. L.W--po GUT agents are poised to play transformative
role in these fields, providing enhanced capabilities and adding substantial value to address these persistent
issues.
2.2 Surveys on LLM Agents
The advent of LLMs has significantly enhanced the capabilities of intelligent agents Zhao et al. (023a), )
enabling them to tackle complex tasks previously out of reach, particularly those involving natural language understanding and code generation Cheng et al. (2024b). This advancement has spurred substantial research
into ALMM-aseed agents designed for a wide array of applications Li et al. (2024i).
Both Xie et al., Xi et al. (023) and Wang et al. Wang et al. (024gg offer comprehensive surveys L.L-- powered agents, covering essential background information, detailed component breakdowns, taxonomies, and various applications. These surveys serve as valuable references for foundational understanding of LLM-driven agents, laying the groundwork for further exploration into LLM- based GUI agents. Xie et al., Xie et al.] (2024a) provide an extensive overview of multimodal agents, which can process images, videos, and audio in addition to text. This multimodal capability significantly broadens the scope beyond traditional text-based agents Durante et al. (0024). Notably, most GUT agents fall under this category, as they rely on image inputs, such as screenshots, to interpret and interact with graphical interfaces effectively. Multi-agent frameworks are frequently employed in the design of GUI agents to enhance their capabilities and scalability.
Surveys by Guo el. al., Guo et al. (2024b) and Han el al., Han et al. (2024) provide comprehensive overviews of the current landscape, challenges, and future directions in this area. Sun et al., Sun et al. (2024a) provide an overview of recent methods that leverage reinforcement learning to strengthen multi-agent LLM systems, opening new pathways for enhancing their capabilities and adaptability. These surveys offer valuable insights and guidance for designing effective multi-agent. systems within GUT agent frameworks.
In the realm of digital environments, Wu el al., Wu et al. (2024a) presents survey on LLM agents operating in mobile environments, covering key aspects of mobile GUI agents. In boarder scope, Wang et al., Wang ct al. (2024k) present survey on the integration of foundation models with GUI agents. Another survey by Gao el al., provides an overview of autonomous agents operating across various digital platforms Gao et al.) (2024T), highlighting their capabilities, challenges, and applications. All these surveys highlighting emerging trends in this area.
Regarding individual components within LLM agents, several surveys provide detailed insights that are especially relevant. for GUT agents. Huang et al., Huang et al. (0024b) examine planning mechanisms in
8

---

Published in Transactions on Machine Learning Research (06/2025)
LLM agents, which are essential for executing long-term tasks- frequent requirement in GUI automation. Zhang et al., Zhang et al. (2024q) explore memory mechanisms, which allow agents to store critical historical information, aiding in knowledge retention and decision decision-making, Additionally, Shen Shen (2024) surveys the use tools by LLMs (such as APIs and code) to interact effectively with their environments, grounding actions in ways that produce langible impacts. Further, Chang el al., Chang el al. (2024) provide a comprehensive survey on evaluation methods for LLMs, which crucial for ensuring the robustness and safety of GUI agents. Two additional surveys, [i et al. (2024d) and Huang & Zhang) (2024), provide comprehensive overviews of benchmarks and evaluation methods specifically tailored to multimodal LLMs. The evaluation also facilitates a feedback loop, allowing agents to improve iteratively based on assessment results. Together, these surveys serve as valuable resources, offering guidance on essential components of LLM agents and forming a foundational basis for LLM-based GUT agents.
Compared to existing surveys, our work offers a significantly more comprehensive and up-to-date overview of the LLM-powerod GUI agent landscape. We curate and synthesize over 500 references, covering a wide range of topies including loundation models, data sources, system frameworks, benchmarks, evaluation methodologies, and practical deployments. While prior surveys often concentrate narrower aspects on selected platform (e.g., web, mobile), our survey takes holistic perspective that spans the full development and deployment lifecycle. Beyond narrative summaries, we also provide consolidated reference tables for each subdomain, enabling readers to quickly categorize and locate relevant works across platforms and research themes serving as a practical handbook for both researchers and practitioners. Furthermore, we incorporate foundational background material and propose evaluation taxonomies that make the survey accessible newcomers, addressing gaps in prior work that often assume high degree of prior familiarity.
3 Background
The development of LLM--poered GUI agents is grounded in three major advancements: (i) large language models (LLMs) Zhao al. (2023b), which bring advanced capabilities in natural language understanding and code generation, forming the core intelligence of these agents; (ti) accompanying agent architectures and tools Wang et al. (2024g) that extend LLM capabilities, bridging the gap between language models and physical environments to enable tangible impacts; and (iii) GUI automation Yeh et al. (2009), which has cultivated a robust set of lools, models, and methodologies essential for GUT agent functionality. Each of these components has played a critical role in the emergence of LLM-powered GUI agents. In the following subsections, we provide brief overview of these areas to sct the stage for our discussion.
3.1 Large Language Models: Foundations and Capabilities
The study of language models has a long and rich history Shannon (951)) boginning with early statistical language models Cavnar et al. (1994) and smaller neural network architect Chung et al. (2014). Building on these foundational concepts, recent advancements have focused on transformer- based basedd LLMs, such as the Generative Pro-trained Transformers (GPTs) Mann al. (022)). These models are pretrained text corpora and feature significantly larger model sizes, validating scaling laws and demonstrating exceptional capabilities across a wide range of natural language tasks. Beyond their sheer size, these LLMss exhibit enhanced language understanding and generation abilities, as well as emergent properties that are absent in smaller-scale language models Wei et al. (2021).
extensive
Early neural language models, based on architectures like recurrent neural networks (RNNs) Medsker et al. (2001) and long short-term memory networks (LSTMs) Hochreiter (197)), were limited in both performance and generalization. The introduction of the Transformer model, built on the attention mechanism Vaswani (2017). marked transformative milestone, establishing the foundational architecture now prevalent across almost all subsoquent LLMs. This development led to variations in model structures, including encoder-only models (e.g., BERT Devlin (2018), RoBERTa Liu (201)). ALBERT Lan (2019)), ), decoder-only models (e.g., GPT-1 Radford 20188, GPT-2 Radford et al. (2019) ), and senooder--ecoder models (e.g., T5 Raffel et al.
(2020) BART Lewis (2019)). In 2022, ChatGPT Wu et al. (2023c) based on GPT-3.5 Ouyang et al. (0022) launched as a groundbrekking LLM, fundamentally shifting perceptions of what language models can achieve.
Since then, numerous advanced L.LMs have emerged, including GPT-4 Achiam et al. (2023), L.LMM-3 Dubey

---

Published in Transactions on Machine Learning Research (06/2025)
et al. 2024 and Gemini Team et al. 2023
propelling the field into rapid growth. Today's LLMs highly versatile, with many of them are capable of processing multimodal data and performing range of tasks, from question answering to code generation, making them indispensable tools in various applications Hurst ct al.
b024;
Jiang et al. 2024]: Zhang et al. (2024b): Liu et al. (2024d)
The emergence of LLMs has also introduced significant advanced properties that invigorate their applications, making previously challenging tasks, such as natural language-driven GUI agents feasible. These advancements include:
1. Few-Shot Learning Mann et al. (2020): Also referred to as in-context learning Dong et al. (2022) LLMs can acquire new tasks from a small set demonstrated examples presented in the prompt during inference, climinating the nced for retraining. This capability is crucial for enabling GUI agents to generalize across different environments with minimal effort.
2. Instruction Following Zhang et al. (0023c): After undergoing instruction Luning, LLMs exhibit a remarkable ability to follow instructions for novel tasks, demonstrating strong generalization skills Ouyang et al. 2022). This allows LLMs to effectively comprehend user requests directed at GUI agents and to follow predefined objectives accurately.
Long- Term Reasoning Huang Chang (2022): LLMs possess the ability to plan and solve complex tasks by breaking them down into manageable steps, often employing techniques like chain- of-thought (CoT) reasoning Wei et al. (2022); ; Ding et al. (2023). This capability is essential for GUI agents, as many tasks require multiple steps and a robust planning framework.
Code Generation and Tool Utilization Chen et al. (2021a): LLMs excel in generating code and utilizing various tools, such as APIs [Shen (2024). This expertise is vital, as code and tools form the essential toolkit for GUI agents interact with their environments.
5. Multimodal Comprehension Yin et al. (2023): Advanced LLMs can integrate additional data modalities, such as images, into their training processes, evolving into multinodal models. This ability is particularly important for GUI agents, which must interpret GUI screenshots presented as images in order to function effectively White et al. (2019).
To further enhance the specialization of LLMs for GUI agents, rescarchers often finc-tune these models with domain-specific data, such as user requests, GUI screenshots, and action sequences, thereby increasing their customization and effectiveness. In Section f we delve into these advanced, tailored models for GUT agents, discussing their unique adaptations and improved capabilities for interacting with graphical interfaces.
3.2
LLM Agents: From Language to Action
Traditional Al agents have often focused enhancing specific capabilities, such symbolic reasoning excelling in particular tasks like Go Chess. In contrast, the emergence of LLMs has transformed AI agents by providing them with a natural language interface, enabling human-like decision-making capabilities, and equipping them to perform a wide variety of tasks and take tangible actions in diverse environments Wang let al. (024gg;; Kim et al.] (2023); ; Liu et al. (2024g): Qiao et al. (2023). In LLM agents, if LLMs form the "brain" of a GUT agent, then its accompanying components serve as its "eyes and hands", enabling the LLM to perceive the environment's 's status and translate its textual output into actionable steps that generate tangible effects Xi et al. (2023). These components transform LLMs from passive information sources into interactive agents that execute tasks on behalf of users, which redefine the role of LLMs from purely text-generative models to systems capable of driving actions and achieving specific goals.
In the context of GUI agents, the agent typically perceives the GUT status through screenshots and widget trees Boshart & Kosa (2003), then performs actions to mimic user operations (e.g., mouse clicks, keyboard inputs, touch gestures on phones) within the environment. Since tasks can be long-term, effective planning and task decomposition are often required, posing unique challenges. Consequently, an LLM-powered GUI agent usually possess multimodal capabilities Xie et al. (2024a), a robust planning system Huang al.
10

---

Published in Transactions on Machine Learning Research (06/2025)
2024b), memory mechanism to analyze historical interactions Zhang et al. (2024g), and specialized toolkit to interact with its environment Li & Wu (2006). We will discuss these tailored designs for GUT agents in detail in Section !
3.3 GUI Automation: Tools, Techniques, and Challenges
GUI automation has been a critical arca of research and application since the carly days of GUTs in computing. Initially developed to improve software testing efficiency, GUI automation focused on simulating user actions, such as clicks, text imput, and navigation, across graphical applications to validate functionality Said et al. (2020). Early GUI automation tools were designed to execute repetitive test cases on static interfaces Rodriguez- Valdés et al. (002]). These approaches streamlined quality assurance processes, ensuring consistency and reducing manual lesting time. As the demand for digital solutions has grown, GUT automation has expanded beyond testing to other applications, including RPA Ivančić et al. (2019) and Human- Computer Interaction (HCT) Li & Hilliges 20021) RPA leverages GUI automation to replicate human actions in business workflows, automating routine tasks to improve operational efficiency. Similarly, HCI research employs GUI automation to simulate user behaviors, enabling usability assessments and interaction studies. In both cases, automation has siguificantly enhanced productivity and user experience by minimizing repetitive tasks and enabling greater system adaptability Abuaddous et al. 2022 Gao et al. (2024b).
Traditional GUI automation methods have primarily depended on scripting and rule-based frameworks Hellmann & Maurer 22011;; Qian et al. (2020). Scripting- based automation utilizes languages such as Python, Java, and JavaScript. to control GUT elements programmatically. These scripts simulate a user's actions on the interface, often using tools like Selenium Bruns et al. (2009) for web-based automation or Autolt Rupp ct al. (2022) and SikuliX Granda ct al.] (2021) for desktop applications. Rule-based approaches, meanwhile, operate based predefined heuristies, using rules to detect and interact with specific GUT elements based on properties such as location, color, and text labels Hellmann & Maurer (2011). While effective for predictable, static workflows Xu et al. 2024c),, these methods struggle to adapt to the variability of modern GUIs, where dynamic content, responsive layouts, and user-driven changes make it challenging to maintain rigid, rule-based automation Gove & Faytong (2012).
CV has become essential for interpreting the visual aspects of GUIs Yu et al. 2023 Li et al. 2021 Chang lt al. (2010), enabling automation tools to recognize and interact with on-screen elements even as layouts and designs change. CV techniques allow GUT automation systems to detect and classify on-screen elements, such buttons, icons, and text fields, by analyzing screenshots and identifying regions of interest [Zou et al. (2023); ;; Ye et al. (2021); ; Chen et al. (2020). Optical Character Recognition (OCR) further enhances this capability by extracting text content from images, making it possible for automation systems to interpret labels, error messages, and form instructions accurately Qian et al.] (2022). Object detection models add robust allowing automation agents to locate GUI elements even when the visual layout shifts [White] et al. 2019). By incorporating CV, GUT automation systems achieve greater resilience and adaptability dynamic environments.
Despite advances, traditional GUI automation methods fall short in handling the complexity and variability of contemporary interfaces. Today's 's applications often feature dynamic, adaptive elements that cannot be reliably automated through rigid scripting or rule-based methods alone Gambino et al. (2018); He et al. 20088. Modern interfaces increasingly require contextual awareness Stefanidi et al. (2022), such as processing on-screen text, interpreting user intent, and recognizing visual cues. These demands reveal the limitations of cxisting automation frameworks and the need for more flexible solutions capable of real-time adaptation and context-sensstive responses.
LLMs offer promising solution to these challenges. With their capacity to comprehend natural language, interpret context, and generate adaptive scripts, LLMs can enable more intelligent, versatile GUI automation Liu et al. (2024k). Their ability lo process complex instructions and learn from context allows them to bridge the gap between static, based methods and the dynamic needs of contemporary GUIs Brie et al. (002)). By integrating L.Mss with GUI agents, these systems gain the ability to generate scripts on-the-fly based on the current state of the interface, providing a level of adaptability and sophistication that traditional methods cannot achieve. The combination of LLMSs and GUT agents paves the way for an advanced, user-centered
11

---

Published in Transactions on Machine Learning Research (06/2025)
2025+
WebVoyager
Mobile-Agent
Mobile
SeeAct
Sep-Dec
Computer
CaCo-Agent
NaviQAte
Cross
DUAL-VCR
Steward
Hybrid Agent
CogAgent
OS-Copilot
2024
WMA
Auto-GUI
Cradle
apeain
AgentOccam
WebGUM
AutoWebGLM
May-Aug
NNetnav
MMAC-Copilot
MobA
Tancont
Zero-shot Agent
SeeClick
Agent S
OpenAgents
AutoGLM
MM-Navigator
WGE
WebArena
Tencent AppAgent
Jan-Apr
Lenowa Mobile-Experts
D GUI Narrator
TinyClick
DOM-Q-NET
WebAgent
orargince Agent-E
Claude Computer Use
Use
FLIN
AutoDroid
Sep-Dec
Search-Agent
WebGPT WebShop
DroidBot-GPT
Openwebagent
May-Aug
WebPilot
2023
2023
Jan-Apr
Figure 3: An overview of GUI agents evolution by year and platforms.
automation paradigm, capable of responding flexibly user requests and interacting seamlessly with complex,
evolving interfaces.
3.4 Evolution to LLM-Powered GUI Agents
As illustrated in Figure 3 prior to 2023 and the emergence LLMs, work on GUI agents was limited in both scope and capability. Since then, the proliferation of LLM-based approaches has fostered numerous notable developments across platforms including web, mobile, and desktop environments. This surge is ongoing and continues to drive innovation in the field. The key takeaway is that GUI agents not only growing in number but also rapidly advancing in terms of data, models, frameworks, and applications. This section takes on journey tracing the evolution of GUI agents, emphasizing key milestones that have brought the field to its present state.
3.4.1 GUI Agent vs. APl-Based Agent
In the field of LLM-powered agents operating within digital environments, the action space can be broadly categorized into two types:
1. GUI Agents, which primarily rely on GUI operations (e.g., clicks, keystrokes) to complete tasks.
A API-Based Agents, which utilize system or application-native APIs to fulfill objectives.
We show the principle of both agent types in Figure Each type has distinct advantages, and a deeper understanding of these approaches critical for designing effective agents.
GUI operations provide a universal control interface that can operate across diverse applications using the same action primitives. This makes GUI agents highly generalizable, they can interact with wide range of software environments without requiring application-speeife adaptations. However, GUI-based interactions are inherently more complex; even simple tasks may require multiple sequential steps, which can increase both the deeision-making cost. for the agent and the computational resources required for long-term, multi-step workflows. Another key aspect the transparency of actions in GUI agents. Since GUI agents interact with applications in the same way a human would, by clicking, typing, and navigating through the interface, their actions are inherently more observable and interpretable users. This transparency fosters better trust and comprehension in agent-computer interactions.
12

---

Published in Transactions on Machine Learning Research (06/2025)
API Information
User Query
API 1
Description:
Args:
GUI Observation
Return:
Examples:
API 2
Description:
Args:
API Agent
GUI Agent
Return:
Examples:
API n
API 6
Description:
CLICK (X=.., Ye....
argl=.... arg2=....
Args:
CLICK (X=.", Y=....
Return:
CLICK (X=",, YE...)
Examples:
Figure The comparison between API agent vs. GUI agent.
In contrast, API-based agents offer a more efficient and direct approach to task completion. By leveraging native APIs, tasks can often be fulfilled with a single, precise call, significantly reducing execution time and complexity. However, these native APIs are often private or restricted specific applications, limiting accessibility and generalizability. This makes API-based agents less versatile in scenarios where API access is unavailable insufficient. In addition, API-basod agents operate behind the scenes, executing tasks through direct system calls, which, while often more efficient and reliable, can make their operations less visible and harder to debug for end users.
The most effective digital agents are likely to operate in a hybrid manner, combining the strengths of both approaches. Such agents can utilize GUI operations to achieve broad compatibility across software while exploiting native APIs where available maximize efficiency and effectiveness. These hybrid agents strike balance between generalization and task optimization, making them a critical focus area in this survey. For more comprehensive comparison between GUI agents and API agents, please refer to Zhang et al. 2025a).
3.4.2 OS-Integrated Agents: Opportunities and Challenges
As agent capabilities continue to advance, we are witnessing the emergence of OS-integrated assistants such Apple Intelligence (built into iOS (S/macoS) and Microsoft Copilot (integrated across Windows and Microsoft 365). These systems exemplify a growing trend in which foundational models are embedded directly into the operating system, offering native capabilities for task execution, summarization, and user assistance.
While these assistants represent significant progress bringing foundation models closer to users, their functionality largely shaped by APT-based control paradigms. They are optimized for environments where fino-grained APIs are available and sanctioned, enabling streamlined and sccure task exccution. However, discussed in Section 3.4.1 this model imposes inherent limitations: many third-party or legacy applications lack sufficient API exposure, and system-level privileges often restricted, especially in closed ecosystems like iOS. This restricts the agent's 's ability to generalize across diverse software landscapes.
GUI-based agents offer a compelling complement in this context. By simulating human interaction with visual interfaces, GUI agents can extend automation capabilities to software that is otherwise inaccessible to OS-level API agents. This makes them especially valuable in scenarios where application internals are opaque, APIs are limited or unstable, or extensibility across software is desired. In essence, GUI agents serve as a universal fallback enabiiin broader coverage and versatility without requiring privileged access
13

---

Published in Transactions on Machine Learning Research (06/2025)
deep integration. The growing presence of OS-level agents amplifies the importance of developing hybrid agent architectures that combine the precision and efficiency of APIs with the generality and accessibility of GUI-based control. For example, an assistant like Mierosoft Copilot could selectively use GUI-based interactions for third-party or legacy apps while relying on APIs for native tools like Word or Excel. Such hybridization can enhance lask coverage, robustness, and user experience across heterogeneous environments.
Looking forward, we envision increasing pressure for these integrated systems to support modular and extensible agent interfaces, allowing third-party developers and rescarchers to build on top of foundation agents with GUI-based capabilities. This direction not only broadens the deployment landscape for LLM- powered agents but also fosters a richer ecosystem of intelligent assistants that are adaptable to real-world variability in software availability, permissions, and user needs.
4 LLM-Brained GUI Agents: Foundations and Design
In essence, LLM-powered GUI agents are designed to process user instructions or requests given in natural language, interpret the current state of the GUI through screenshots or GUI element trees, and execute actions that simulate human interaction across various software interfaces nnterfacesZZhng et al.] [2024a]. These agents harness the sophisticated natural language undorstanding, reasoning, and generative capabilitics of LLMs to accurately comprehend user intent, assess the GUI context, and autonomousyy engage with applications across diverse environments, thereby enabling the completion of complex, multi-step tasks. This integration allows them to scamlessly interpret and respond to user requests, bringing adaptability and intelligence to GUI automation.
As a specialized type of LLM agent, most current GUI agents adopt a similar foundational framework, integrating core components such as planning. memory, tool usage, and advanced enhancements like multi- agent corattona,io,, among others Wang et al. (2024g). However, each component must be tailored to meet the specific objectives of GUT agents to ensure adaptability and functionality across various application environments.
In the following sections, we provide an in-depth overview of each component, offering a practical guide and tutorial on building an L.M--powred GUI agent from the ground up. This comprehensive breakdown serves as cookbook for creating effective and intelligent GUI automation systems that leverage the capabilities of LLMs.
4.1 Architecture and Workflow In Nutshell
In Figure we present the architecture of LLM-brained GUI agent, showcasing the sequence of operations from user input to task completion. The archilecture comprises several inlegrated components, each contributing to the agent's 's ability to interpret and execute tasks based on user-provided natural language instructions. Upon receiving user request, the agent follows a systematic workflow that includes environment perception, prompt engineering, model inference, action execution, and contimuous memory utilization until the task fully completed. It is important to recognize that user-agent interaction can be bi-directional, as opposed a passive, one-shot task execution. GUI agents may initiate communication with users request clarification, recover from errors, resume control when task completion reliability and user experience.
that significantly improve
In general, it consists of the following components:
1. Operating Environment:
The environment defines the operational context for the agent, encom- passing platforms such as mobile devices, web browsers, and desklop operating systems like Windows. To interact meaningfully, the agent perceives the environment's 's current state through screenshots, widget trees, or other methods of capturing UI structure Memon et al. (2003b). continuously monitors feedback on each action's 's impact, adjusting its strategy in real time to ensure effective task progression.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Agent
Memory
Prompt Engineering
Input
Error Recovery,
Clarification,
Communication
Instruction
Example
Step t-1
Step t-2
Step E-n
Resumption
Request
Input
Example
Action Execution
Input
Input
Example
Model Inference
Action
Input
Edkk "ake""
Perception
"Dupp" "pokko"" 262262211 Comtehe
Screenshots
Widget Tree
Ul Element Properties
Environment State
Operating Enviromment
Figure 5: An overview of the architecture and workflow of basic LLM-powered GUI agent.
Prompt Engincering: Following environment perception, the agent constructs a detailed prompt guide the I.M's inference Wang et al. (2023b). This prompt incorporates user instructions, processed visual data (e.g., screenshots), UI element layouts, properties, and any additional context relevant to the task. This structured input maximizes the LLM's ability to generate coherent, context-aware responses aligned with the current GUI state.
Model Inference: The constructed prompt is passed to a LLM, the agent's inference core, which produces sequence of plans, actions and insights required to fulfill the user's request. This model may be general-purpose LLM specialized model fine-tuned with GUL-specific data, enabling more nuanced understanding of GUI interactions, user flows, and task requirements.
Actions Execution: Based on the model's inference results, the agent identifies specific actions (such as mouse clicks, keyboard inputs, touchscreen gestures, or API calls) required for task execution Shen (2024). An executor within the agent translates these high-level instructions into actionable commands that impact the GUI directly, effectively simulating human-like interactions across diverse applications and devices.
Memory: For multi-step tasks, the agent maintains an internal memory to track prior actions, task progress, and environment states Zhang et al. (2024q). This memory ensures coherence throughout complex workflows, as the agent can reference previous steps and adapt its actions accordingly. An external memory module may also be incorporated to enable continuous learning, access external knowledge, and enhance adaptation to new environments requirements.
By itcratively traversing these stages and assembling the foundational components, the LLM-powered GUI agent operates intelligently, seamlessly adapting across various software interfaces and bridging the gap between language-bar based instruction and concrete action. Each component is critical to the agent's robustness, responsiveness, and capability handle complex tasks in dynamic environments. In the following subsoctions, we detail the design and core techniques underlying each of these components, providing comprehensive guide for constructing L.M-powered GUT agents from the ground up.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Summary platform-specific challenges, action spaces, and typical tasks for Web, Mobile, and Computer GUI environments.
Platform
Typical GUI Challenges
Action Space
Representative Tasks
Mobile
Constrained screen real es-
Tap, swipe, pinch, and other touch gestures
App-based login and form filling
tate
Heavy reliance on touch and
Virtual keyboard input
Messaging, social media posting
gesture recognition Mitra
In-app navigation (menus, tabs)
Acharya (2077)
Location- based services and map interactions
App architectures (native
Accessing hardware features
hybrid)
(camera, GPS)
Handling push notífications and permission dialogs
Accessibility frameworks
(e.g.; Android's Accessibility
API, iOS VoiceOver)
Platform-specifi
straints
(permissions,
security, privacy)
Web
Dynamic and responsive lay-
Click, hover, scroll
Form completion (registra-
outs
DOM-based form filling
tions, checkouts)
Asynchronous
updates
Link uavigation and element
Data extraction /web /weet scrap- ing
(AJAX, fetch APIs)
inspection
HTML/DDM-based struc-
JavaScript event triggering
Searching and filtering (e.g., e-commerce)
tures
browser inconsisten-
Multi-step web navigation (redirects, pop-ups)
cies
Computer
Full-fledged OS-level inter-
Mouse click, drag-and-drop
File management and sys-
faces
Keyboard shortcuts and text
tem settings
Multi-windoo operations
input
Productivity software usage (office suites, IDEs)
system-level shortouts
Menu navigation, toolbars
Automation APIs (e.g.. Win-
Access to multiple applica- tion windows
haaalliinguunnntllllnnggng appli- cations
dows UT Automation Oksa
nem 2023
Coordinsting
multi-
Frequent software updates
application workflows
requiring adaptation
Complex, multi-layered soft-
ware suites
5
(a) Web GUI
(b) Mobile GUI
(c) Computer GUI
Figure 6: Examples of GUIs from web, mobile and computer platforms
4.2 Operating Environment
The operating environment for LLM-powered GUI agents encompasses various platforms, such as mobile, web, and desktop operating systems, where these agents interact with graphical interfaces. Each platform
16

---

Published in Transactions on Machine Learning Research (06/2025)
DR
(a) clean screenshot.
(b) A GUI screenshot with widgets highlighted by SoM.
(c) GUI screenshot with widgets highlighted by bounding boxes.
Figure 7: Examples of different variants of VS Code GUI screenshots.
has distinct characteristics that impact the way GUI agents perceive, interpret, and act within Examples of GUIs from each platform are shown in Figure D This section details the nuances of each platform, the ways agents gather environmental information, and the challenges they face in adapting to diverse operating environments.
4.2.1 Platform
GUI agents can interact with wide range of platforms, including mobile devices, web applications, and computer operating systems like Windows. Fach platform offers unique capabilities and constraints for GUT automation, requiring agents to adapt their perception and interaction strategies accordingly. We provide comparative overview of the platforms, including their challenges, action spaces, and reprosentative tasks, in Table
1. Mobile Platforms: Mobile devices operate within constrained screen real estate, rely heavily on touch interactions Hardy & Rukzio (2008), and offer varied app architectures (e.g., native vs. hybrid apps). Mobile platforms often use accessibility frameworks, such as Android's Accessibility APp al. (2022) and iOS's VoiceOver Accessibility Inspoctor expose structured information about GUI
elements. However, GUI agents must handle additional complexities in mobile environments, such
gosture recognition Mitra & Acharya (2007), , app navigation Jokinen 20088), and platform-specific
constraints (e.g., security and privacy permissions) Enck et al. [2011); Egele et al. (2011).
Web Platforms: Web applications provide relatively standardized interface, typically accessible through Hypertext Markup Language (HTML) and Document Object Model (DOM) structures Sierkowski (2002); Fernandes et al. (2011). GUI agents leverage HTML attributes, such element ID, class, and tag, to identify interactive components. Web environments also present. dynamic content, responsive layouts, and asynchronous updates (e.g., AJAX requests) Garrett et al. (2005), requiring agents to contimuously assess the DOM and adapt their actions to changing interface elements.
3. Computer Platforms: Computer OS platforms, such as Windows, offer full control over GUI interactions. Agents can utilize system-level automation APIs, such as Windows UT Automation Oksanen (2023), to obtain comprehensive GUI element data, including type, label, position, and bounding box. These platforms often support a broader set of interaction types, mouse, keyboard,
https::/deweeooper.app apple. http:///ww...toooooom/ttssltttts
https://learnniccce http:://aaaaaacccoooocceeeereees http:///s/.aatttcc.////////////s
17

---

Published in Transactions on Machine Learning Research (06/2025)
GUI
Comprss
Irarspareccy
95 Change
Ravet Fkture
Betton Remove Background
Menultam
Corrections
Menultam
Croupilose Adjust
Menultem Transparency
Menuitemm Compress Fictures...
Menultern Change Picture
Button Reset Picture
SpleButton Reset Picture
Menultem More
Widget Tree
Figure 8: An example of GUT and its widget tree.
and complex multi-window operations. These enable GUI agents to execute intricate workflows. However, these systems also require sophisticated adaptation for diverse applications, ranging from simple GUT to complex, multi-layered software suites.
In summary, the diversity of platforms, spanning mobile, web, and desktop enviromments, enable GUI agents to deliver broad automation capabilities, making them a generalized solution adaptable across a unified framework. However, each platform presents unique characteristics and constraints at both the system and application levels, necessitating tailored approach for effective integration. By considering these ppatform-specific features, GUI agents can be optimized to address the distinctive requirements of each environment, thus enhancing their adaptability and reliability in varied automation scenarios.
4.2.2 Environment State Perception
Accurately perceiving the current state of the environment is essential for LLM-powered GUI agents, it directly informs their decision- decision-making and action- plabning processes. This perception is enabled by gathering a combination of structured data, such as widget trees, and unstructured data, like screenshots, to capture a complete representation of the interface and its components. In Table 4 we outline key toolkits available for collecting GUT environment data across various platforms, and below we discuss their roles detail:
GUI Screenshots: Screenshots provide a visual snapshot of the application, capturing the entire state of the GUI at a given moment. They offer agents a reference for layout, design, and visual content, which crucial when structural details about GUI elements are either limited or unavailable. Visual elements like icons, images, and other graphical cues that may hold important context can be analyzed directly from screenshots. Many platforms have built-in tools to capture screenshots (e.g.,
18

---

Published in Transactions on Machine Learning Research (06/2025)
Table 4: Key toolkits for collecting GUI environment data.
Tool
Platform
Environment
Accessible Information
Highlight
Link
Selemmua
Web
Browser
[Cross-
DOM elements, ITTML struc-
Extensive browser support
platform)
ture, CSS propertks
and antomation espabilities Headless browser automa-
actga://wow.co sww.ntun.dd//
Puppeteur
Web
Browser
(Chrome
DOM elements HT ML,CSS.
Firefox)
network requsts
tion with rich API
hetga://pptzz dev/
Flaywright
Web
Browser
[Cross.
DOM clements, HTML.
Multi browser support with susomation and testing capa- blittes
actpa: /ww.
platform)
network interactions
CaseCate
Web
Browser
[Cross
DUM eloments, ML. strue-
Easy setup with JavuSeript/-
htttaa///eatcaff..te
platform]
turo, CSS propertie
TypeScript support
BesutituSoup
Web
HTML Parsing
HTML contenl, DOM ele- ments
Python library for parsing
https://www. httpe://ew.ccruayy..u cow////coo///oooo softeere/
ITML and XML documents Designed for Angular appli- cations, Integrates with Sele- nhum
DeastifulBoup/
Protractor
Web
Browser (Angular)
DOM clemeuts, Angular-
hctga: http:///w..cotraattttaaattoo
spacific attributes
WebDriverrO
Web
Browwer
(Cross-
DUM elements, MLCCSS.
Highly extonsible
hnttpa://wnbrrivrr..///
platformi
welwork interactions
plugin ecssysten
Glost Iuspec-
Web
Browser
(Cross-
DOM elementss screenshots, test scripts
Cloud based automated
actpe: httpp:///bottteeeetttoooo com/
tor
platform)
browser testing and monitor-
ing
Cypress
Web
Browser
Crrss-
DOM eleneels,, HTSL,/CSS,
Real-time relossta and inler- active dehngging
hstpp: Ptttpe///ww...o cyprese....
platform)
network requssts
UlAutomator
Mobile
Android
GUI hieerrrly, widget prop- erties, screca content
Native Android GUI testing framework
brtge: //te:////eoe.......... android.
training/ /tastttg/ttttt -ustttonnaoo
Espresso
Mobili
Andlvvid
GLT componennts, View hier-
Google's native Android CUI
ht hetpa 1t.ppa///i //davel oper android. con/
archy, widget propertias
bestnn fromework
braining/teet wtw.ggggttee////eeeeeeeeeee /eepresso
Android View
Mobile
Android
GUI hierarcly, widget prop-
View Lierarchy accessible via developer tooks
https: /tpp:://dovee /dwee/oorrro.r..o.oo android. com/
Hheraroky
erties, layout information
atudie/ wwwp://aaaaooauaa -asput-n
TOS Aevesshbil
Mobilo
108
Accrseibility tree, CUT clo-
Tool for insporting 10S app
actpa://dere oper.
con/
iyy Imspector
meats, properties
GUI elemenne
ibcumestatiio///cceesib biltty
accessibl inspactor
XCUITest
Mobile
1O8
GUI elements, accesability properties, view Lierardyy
Apple's 1OS GU1 testing frumework
attpa:// applo.. accusstaticc///eeee//uuu intorface tasta
Plutter Driver
Mobile
Plutter apps
Widget tree, propertics, leractions
Automation for Flutter ap- plications
Androcdds Me-
Mobile
Audroid
Screesstoss, screen record- ing
Capsuring device screen cod- lent proganmmtieally
hetpe: httpe://deeeloprrs android. com/
diaerojection
reference/ infferecee/ndrrrdd/eeee http:////////d///////////pp/e/ /o
API
NedisFrojoctioon
W Windows GUT
Compater
Windows
Control properties, widget trees, accueibility tree
Native Windows support with 08 integration
Automation
hetpa://bol //docs wwcrosoft.. con//indow////
unttr--uuuuoo uiauto- vin32
Sikuli
Computer
Windows,
macOS.
Screetshots (imuaee recogni-
Image based automition us- ing compater vision
hctp://simollr, coo/
Linux
tica). GUI elements
Autolt
Computer
Windows
Window titles, control prop-
Scripting language for Win- dows GUI ausomation
httpa://wou wtwp:/arottarrpt.coooooooo autoit/
erties, coordinates
Inspect.exe
Computer
Windowe
GUI elementss, conirol prop-
Tool for inspecting Windows
https: microeoft cwt.wiioooe/////ooocooooooo
erties, accesiiilitty tree
GUI elements
vin32/vinauto/ www.a/tttonnnneetoojoeette
macOS Acces.
Computer
macOS
Accessibiity tree, CUI elc-
macOS support for accessi-
httpa: //dp:////ee.....eee apple.coo/
Sbility API
control properties
bility and GUI automation
accessibilitty
Pywinaulo
Computer
Windowe
Control properties, GUT hi-
bython-boued Windows GUI automation
hstpe: / hwtpiaattt.reeeteeeeoooooooo rewdthedoce. ie/
esarhhy, window information
Electron spector
In-
Computer
Electron apps
DOM elenents, HT HTML/CSS
Tool for Electron applics- tions
httpe://www.. electronje org/doce/
JavaScript state Screenshots
Iatest/cutoraa//uut wtt.ooorrraluuttutteetee tosting
Windows Snipping Thol
Computer
Windows
Tool for capturing screen-
hetpa: http://wwwnnncooccoo wicrosoft con/ an-us/
shots In Windows
vindows/ pa/am pping-too
Computer
mac08
Scrweshols, recurd- ing
Tool for capluring scrven- shots and recording screen
hstpe: httppp//euppors..o apple. guide/
Screenshot Ulility
nac hwc-belp/taeeeeee take Ecreeurhoo ork
2Decrenn "Decreen-eecoddnng recording 25022//2
Acccsskit
Cross-
Various 08
Accessibility tree, control properties ralos
Standardized APIs platforms Mubile automation Trame- woek
actpa://githon cool /Acceaaii// accesskit
Platform
Appiuna
Crose-
Audroid, 1OS, Win-
GUT elemenls accesibility properties, gestures
Platform
dows, macO8 Web, Mobile, Desic
hetpe: hetpp://aopiun.io)
Robot Frame-
Cross-
elements, DOM, screen- shots
Extensible various bearles
httpa: actga://robotrraneerzz
work
Platform
top
Cueumber
Crowe
Web, Mobile, Desk-
Slep dedinitions, GLI inter- actions
BDD fromework supporting automation tools
hetpe://eacunber.. io/
Platform
top
Cempleeeeee
Cross
Web, Mobile, Desic top
GUI elements, DOM, contral properties
Tool with exteusive feature
coo. cwtpp/oddctt//c///cccccooccoccct
Platform
teetccoppooee/overvse
Katalon
Stu-
Crose-
Web, Mobile, Desk- top
GUI oloments, DOM, screen- shots
All-in-one automation solu-
hctpe: / //www kesal con/
dio
Platform
tion
Ramorex
Cross-
Web, Mobile, Desk
GUI elements, DOM, control properties
Tool with strong reporting festures Al-powenod visual testing
hctpa: h/wp:///www../o www.w/wwwwr..ccoooo com/
Platform
top
Applitook
Cross-
Web, Mobile, Deske top
Scromshoos, visual check- points, DOM elements
hetpa: http:///gppppptlloooo stools.cor/
Platform
19

---

Published in Transactions on Machine Learning Research (06/2025)
Widget
Widget Name
Position
Attributes
title= Remove Background;;
Button - 'Remove Background
L-3810, T128, R-3708, B243
ie_jidd Picture BackgroundReeovval"
Remooe Barkgrmund
control control_type= wettyyppe-BBtttt
title= Cole==CCorectioss;
Menultem "Corrections"
L-3689, T128, R-3592, B243
auto iddFPitureCorrcciionMenni
Corret ons
control_ iaptroottye=Meenuteen
title=Coolor;
Menultem 'Color
L-3589, T128, R-3527, 8243
auto juto_id=PPicturCColee iddieiicccurrColoMMennj
Color
control_type= Myarroo_yyee/Meuuteem
title= Effects';
Menultem 'Artistic Effects'
L-3524, T128, R-3448, B243
auto jdd=Pcccurerrsstcccffettalerr
Arteetc Effects w
control iyptrol_type-MMMuuteen
Tellsprranparrncc
Menultem Transparency
L-3445, T128, R-3336, 8243
auto auto_id= ji_ppiddpctueeeee Tronpuurrraansreennyylllllee
Pransareecy
control Be__ttypee-Meniteee
title= Compress Pictures...'; auto auto_id="Picteee iidPPiicuueeoommeeesss
Button 'Compress
Compress fit.urs
Pictures...!
L-3333, T128, R-3138, B164
title='Change Picture'; auto iidiiiPPcuurrrCaaneeeeem control control_type= ttt_ttyypeeeeuutmmmm
Menultem 'Change Picture'
Charge
L-3333, T167, R-3149, B203
SplitButton 'Reset
title= Reset Picture'; control control_type= typp=:Splltttuttoon
Rest Ficuze
Picture'
L-3333, T206, R-3160, B242
Figure 9: Examples of UI element properties in the PowerPoint application for GUT Agent interaction.
Windows Snipping Too macOS Screenshot Utility 9 and Android's MediaProjection AP and screenshots can be enhanced with additional annotations, such as Set-of-Mark (SoM) highlights Yang et al. 2023 bounding boxes Wu et al. (023d) around key GUT components, to streamline agent decisions. Figure illustrates various screenshots of the VS Code GUI, including clcan version, well as ones with SoM and bounding boxes that highlight actionable components, helping the agent focus on the most critical areas of the interface.
2. Widget Trees:
Widget trees present hierarchical view of interface elements, providing structured data about the layout and relationships between components Gamma (1995). We show example of a GUI and its widget tree in Figure X By accessing the widget tree, agents can identify attributes such as element lype, label, role, and relationships within the interface, all of which are essential for contextual understanding. Tools like Windows GUI Automation and macOS's Accessibility AP provide structured views for desktop applications, while Android's Accessibility API and HTML DOM structures serve mobile and web platforms, respectively. This hicrarchical data indispensable for agents map out logical interactions and make informed choices based the GUT structure.
GUI Element Properties: Each GUI element in the interface contains specific properties, such as control type, label text, position, and bounding box dimensions, that help agents target the appropriate components. These properties are instrumental for agents to make decisions about spatial relationships (e.g., adjacent elements) and functional purposes (e.g., distinguishing between buttons and text. fields). For instance, web applications reveal properties like DOM attributes (id,
-rc. 2D6SS
DDSSs//anr220////77999666...
httpp://dooooooerraddrroo /t http://oo..aaaoooooocooaaaaaaaaa
http:://dvveooer.appll
20

---

Published in Transactions on Machine Learning Research (06/2025)
Widgets detected by APls
CV-Detected Widgets Information
MB
Id: 1, type: icon, label: Page thumbnail
Single Agent vs. MultiAgent
Id: 2, type: icon, label: Page thumbnail
Pesturesc Specss cen Oulibosrion
Id: 3, type: icon, label: Page 6 thumbnail
Id: 4, type: icon, label: Page 7 thumbnail
the
Id: 5, type: editbox, label: Canvas
Widgets detected by CV models
Figure 10: An example illustrating the use of CV approach to parse a PowerPoint GUI and detect non- standard widgets, inferring their types and labels.
class, name) and CSS styles that provide context and control information. These attributes assist agents in pinpointing precise elements for interaction, enhancing their ability to navigate and operate within diverse GUI environments. Figure | illussrates examples of selected GUI element properties extracted by the Windows UI Automation API, which support GUI agents in decision-making.
Complementary CV Approaches: When structured information is incomplete or unavailable, computer vision techniques can provide additional insights Wang et al. (2024a). For instance, OCR allows agents to extract text content directly from screenshots, facilitating the reading of labels, error messages, and instructions Qian et al. (2022). Furthermoro, advanced object detection Chen et al. (2200) models like SAM (Segment Anything Model) Kirillov et al. (2023). DINO Liu et al. 2023a and OmniParser Lu et al. (2024d) can identify and classify GUT components in various layouts, supporting the agent in dynamic environments where GUI elements may frequently change. These basedn-based methods ensure robustness, enabling agents to function effectively even in settings where standard GUI APIs are insufficient. We illustrate an example of this complementary information in Figure 10 and further detail these advancod computer vision approaches Soction 4.7.1
Together, these elements create comprehensive, multimodal representation of the GUI environment's 's current state, delivering both structured and visual data. By incorporating this information into prompt construction, agents are empowered to make well-informed, contextually aware decisions without missing critical environmental cues.
4.2.3 Environment Feedback
Effective feedback mechanisms are essential for GUI agents assess the success of each action and make informed decisions for subsequent steps. Feedback can take several forms, depending the platform and interaction type. Figure [1 presents examples of various types of feedback obtained from the environment.
By comparing before-and-after screenshots, agents can identify visual differ- ences that signify state changes in the application. Sercenshot analysis can reveal subtle variations the interface, such as the appearance of a notification, visual cues, or confirmation messages, that may not be captured by structured data Moran et al. 2018).
Screenshot Update:
21

---

Published in Transactions on Machine Learning Research (06/2025)
After
Before
Click the "Video"
Vide
Audoo
Inoort Vikoo from
Record
"dick success" " True)
Dewinn.
Wab
Audia
Scren Bacadaa
Function Return Values and Exceptions
Screenshot Update
Menultem This Device
Menultem
Menultem Video
Menultem
Menultem
Video
...
Menultem Online Videos
Groupbox
Menultem
Grouphox Media
Audio
Difference
Button Screen Recording
Screen Recording
Before
After
Ul Structure Change
Figure 11: Examples of various types of feedback obtained from PowerPoint application environment.
2. GUI Structure Change:
After executing action, agents detect modifications in the widget Iree structure, such as the appearance disappearance of elements, updates to element properties, or hierarchical shifts Ricós et al. (2023). These changes indicate successful interactions (e.g., opening a dropdown or clicking a button) and help the agent determine the next steps based on the updated environment state.
3. Function Return Values and Exceptions: Certain platforms offer direct feedback on action outcomes through function return values or system-generated exceptions Du et al. (2024). For example, API responses or JavaScript return values can confirm action success on web platforms, while exceptions or error codes can signal failed interactions, guiding the agent retry or select an alternative approach.
These feedback provided by the enviromment is crucial for GUI agents to assess the outcomes of their previous actions. This real-time information enables agents evaluate the effectiveness of their interventions and determine whether to adhere to their initial plans or pivol towards alternative strategies. Through this process of self-reflection, agents adapt their decision decision-making,, optimizing task execution and enhancing overall performance in dynamic and varied application environments.
4.3 Prompt Engineering
In the operation of LLM-powered GUI agents, effective prompt construction is crucial step that encapsulates all nccessary information for the agent to generate appropriate responses and execute tasks succossfully Wang el al. (023b). After gathering the relevant data from the enviromment, the agent formulates a comprehensive prompt that combines various components essential for inference by the LLM. Each component serves a specific purpose, and together they enable the agent to exccute the user's request efficiently. Figure [2 illustrates basic example of prompt construction in an LLM-powered GUI agent. The key elements of the prompt summarized as follows:
22

---

Published in Transactions on Machine Learning Research (06/2025)
Prompt Engineering
User Request:
Action Information:
Create PowerPoint slide based on
surmary: 1-
"click" 1s click the contrel Itee with nouse,
|-
Agent Instruction:
call: <lick(buttons atr, double: bool)
[2] Anga:
You are GUI agent, your task is to
bui Lun: The but lun clisk. Dow of "left""
Please follow the guidelines below...
I'"nghe"' (Defauler Milet"l)
doucle: whethen to perfore double click or net (Derauit: False)
[3] Return: bool
+
Environment State:
Demonstrated Examples:
Thoughts
Useel "aene"" 12ILAR
zm now on the chat vindow of Ton on Toses, reed Input wessagt
Screenshots
Widget Tree
Ul Element Properties
Control: I-
in the control namvil "Type now missage" for the current action.
Type message
Function: |-
Complementary Information:
("tex""
It's you Jein the weeting now?")
Statua:
CONTIMIE
Memory
External Knowledge
Plant
(4) Click the Send button to send the seseape-
Figure 12: A basic example prompt construction in a LLM-pooered GUI agent.
1. User Request: This is the original task description provided by the user, outlining the objective and desired outcome. It serves as the foundation for the agent's actions and is critical for ensuring that the LLM understands the context and scope the task.
2. Agent Instruction:
This section provides guidance for the agent's operation, detailing its role, rules to follow, and specific objectives. Instructions clarify what inputs the agent will receive and outline the expected outputs from the LLM, establishing a framework for the inference process. The core agent instructions are usually embedded within the base system prompt of the LLM, with supplementary instructions dynamically injected updated based environmental feedback and contextual adaptation.
3. Environment States: The agent includes perceived GUI screenshots and GUT information, introduced in Section 4.2.2 This multimodal data may consist of various versions of screenshots (e.g., a clean version and SoM annotated version) to ensure clarity and mitigate the risk of GUT controls being obscured by annotations. This comprehensive representation of the environment is vital for accurate decision-making.
4. Action Documents:
This component outlines the available actions the agent can take, detailing relevant documentation, function or tools names, schemata, arguments, return values, and any other necessary parameters. Providing this information equips the LLM with the context needed to select and generate appropriate actions for the task hand.
Demonstrated Examples: Including example input/output pairs essential to activate the in-context learning Dong al. (2022) capability of the LLM. These examples help the model comprehend and generalize the task requirements, enhancing its performance executing the GUI agent's responsibilities.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Output
Action:
Click(button= "left" double-False,
object= "Remove Background"
Inference
Plan:
1. After clicking the "Remove Background "Transparency" menu.
will click the
LLM
2. will click the "Transparency 50%. adjust the
transparency of the picture.
Complementary Outputs:
Thought:
First need remove background by
Status: Continue
clicking the "Remove Background"
Figure 13: An example of the LLM's inference output in GUI agent.
6. Complementary Information: Additional context that aids in planning and inference may also be included. This can consist of historical data retrieved from the agent's memory (as detailed Section 4.6) and external knowledge sources, such as documents obtained through retrieval-augmented gencration (RAG) methods Lewis et al. (2020): Gao al. (2023). This supplemental information provide valuable insights that further refine the agent's decision-making processes.
The construction of an effective prompt foundational for the performance of LLM-powered GUI agents. By systematically incorporating aforementioned information, the agent ensures that the LLM is equipped with the necessary context and guidance to execute tasks accurately and efficiently.
4.4 Model Inference
The constructed prompt is submitted to the LLM for inference, where the LLM is tasked with generating both a plan and the specific actions required to execute the user's request. This inference process is critical as il dictates how effectively the GUT agent will perform dynamic environments. It typically involves main components: planning and action prediction, well as the generation of complementary outputs. Figure 13 shows an example of the LLM's inference output.
4.4.1 Planning
Successful execution of GUI tasks often necessitates a series of sequential actions, requiring the agent engage in effective planning Zhang et al. [024jj A nalogous to human cognitive processes, thoughtful planning is essential to organize tasks, schedule actions, and ensure successful completion Huang et al. (2024b); Cho et al.
(2024). The LLM must initially conceptualize a long-term goal while simultaneously focusing on short-term actions to initiate progress toward that goal Dagan et al. (2023)
To effectively navigate the complexity multi-step tasks, the agent should decompose the overarching task into manageable subtasks and establish timeline for their execution Khot et al.) (2022). Techniques such be employed, enabling the LLM to develop structured plan that guides
CoT reasoning Wei et al. (2022) the execution of actions. This plan, which can be stored for reference during future inference steps, enhances the organization and focus of the agent's activities.
The granularity of planning may vary based on the nature of the task and the role of the agent Huang et al. (2024b). For complex tasks, a hicrarchical approach that combines global planning (identifying broad subgoals) with local planning (defining detailed steps for those subgoals) can significantly improve the agent's ability to manage long-term objectives effectively Chen al. (2024]
24

---

Published in Transactions on Machine Learning Research (06/2025)
Table 5: Overview of actions for GUI agents.
Action
Category
Original Executor
Examples
Platform
Environment
Toolkit
Mouse actions
GUI Operations
Mouse
Click, scmil, hover, drag
Computer
W Windows
GUI
An-
tomation
Pywinauto
Sweigaatt
2024
Mouse actions
GUI Operations
Mouse
Click, scroll, hover, drag
Computer
macOS
AppleScript
10.
Automa-
11
actions
GUI Operations
Mouse
Click, scroll, hover, drag
Web
Selenium
Puppeteer
Keyboard actions
GUI Operations
Keyboard
key presses, short-
Typing.
Computer
Windows
GUI
tomation
cuts
Pywinanto
Sweigart
2024
Keyboard actions
GUI Operations
Keyboard
Typing, key presses, short- euts
Computer
macOS
ApploSeripe
Autona- sor 11
10.
Keyboard
GUI Operations
Keyboard
key presses, short-
Typing,
Weh
Hrowser
Selenium
Puppeteex
Touch actions
GUI Operations
Touchscreen
Tap, swipe,
Mobile
Audruid
Appium
pinch, 2o0m
UlAutoma-
Got
Touch actious
GUI Operatious
Touchscreen
Tap. swipe, pinch, 200m
Mobile
1O8
Appium
XCU1
Grsture actions
GUI Operations
User
Rotate, multi.
Mobile
Andrid, 10S
Appium
hand
fiuger gestures
Gesture-
Tools 12
Voice commands
GUI Operations
Speech impul,
Mobile
Audruid
Spees Recoognseee
voice
voice commands
Voice commands
GUI Operations
User
Spench imput,
Mobile
108
Sirikit
vuice
vuice commnads
Clipboard operations
GUI Operations
System cliphoard
Copy, paste
Cruss platform
Cross OS
Pyperclip 15 Clipboard.js 16
Screen interactious
GUI Operations
User
Screen rolation, shake
Mobile
Audruid, 1O8
Device sen-
sons APIs 17
Shell Commands
Native API Calls
Command Line In-
File manipu- lation, system operations,
Computer
Unix Limux, macO8
Bash, Termi- nal
terface
script execulion
Application APIs
Native API Calls
Application APIs
Send email, ere- ate document, fetch data
Computer
Windows
Microsoft
Office COM
APIs 18
Application APIs
Native API Calls
Application
Access calendar,
Mobile
Android
Android
APIs
mesages Access calendar,
SDK API 19
Application APIs
Native API Calls
Application
Mobile
105
IO
SDK
APIs
send messages
APIs 20
System APIs
Native API Calls
System
File operations,
Computer
Windows
Wind2 API
APIs
network requests
System APIs
Native API Calls
System
File operations,
Computer
macOS
Cocoa APIs
APIs
network requests
22
Web APIs
Native API Calls
Web Ser-
Fetch data, sub-
Web
Browser
Fetch API
vices
mit forms
Axios 24
AI Models
Tools
Mod-
Screen
Cross-platform
Crose-OS
DALL
els
derstanding, summarization, image
Ramess et al.
2021 Ope- aAI APIs 26
genera-
tion
"(tttps://devolpprr
Introduction/ASLRR intro. html
25

---

Published in Transactions on Machine Learning Research (06/2025)
4.4.2 Action Prediction
Action prediction is the core objective of the inference stage, as it translates the planning into executable tasks, The inferred actions are typically expressed as function call strings, encompassing the function name and relevant parameters. These strings can be readily converted into real-world interactions with the environment, such as clicks, keyboard inputs, mobile gestures, or API calls. A detailed discussion of these action types is presented in Section 4.5
The input prompt must include predefined set of actions available for the agent to select from. The agent can choose action from this set or, if allowed, generale custom code or API calls to interact with the environment Tan et al. (2024)). This flexibility can enhance the agent's ' adaptability to unforeseen circumstancos; however, may introduce reliability concerns, as the generated code may be prone to errors.
4.4.3 Complementary Outputs
In addition to planning and action prediction, the LLM can also generate complementary outputs that enhance the agent's capabilities. These outputs may include reasoning processes that clarify the agent's 's decision-making (e-g-, CoT reasoning), messages for user interaction, or communication with other agents or systems, or the status of the task (e.g., continue or finished). The design of these functionalities can be tailored to meet specific needs, thereby enriching the overall performance of the GUI agent. We note that that such complementary outputs typically capture immediate reasoning, intermediate reflections (e.g., <think> in DeepSeck-rl Guo al. (025)), or communications directed at users or system components, rather than representing structured sequence of future actions as in explicit planning.
By effectively balancing planning and action prediction while incorporating complementary outputs, agents can navigate complex tasks with a higher degree of organization and adaptability.
4.5 Actions Execution
Following the inference process, a crucial next step is for the GUI agent to execute the actions derived from the inforred commands within the GUI environment and subsequently gather foodback. Although the term "GUI agent" might suggest a focus solely on user interface actions, the action space can be greatly expanded by incorporating various toolbaxes that enhance the agent's versatility. Broadly, the actions available GUI agents fall into three main categories: (2016), and (iii Al tools Masterman et al. (024g). Each category offers unique advantages and challenges, enabling the agent to tackle a diverse range of tasks more effectively. We summarize the various actions commonly used in GUI agents, categorized into distinct types, in Table 5 and provide detailed explanations of cach category below.
(i) GUI operations Li et al. (2020a), (ti) native API calls Gu al.
4.5.1 Ul Operations
UT operations encompass the fundamental interactions that users typically perform with GUIs in soft applications. These operations include various forms of input, such as mouse actions (clicks, drags, hovers), koyboard actions (key presses, combinations), touch actions (taps, swipes), and gestures (pinching, rotating).
18 httpes///ppppooro//////////ppppp
bttpe:/cllppoordjj.cco
overview
18
18
21 https: :ttp:://leam.accrrooff.... com/
com/1os/
2:
What VbatIeCocoa.htll
http:://developerrmozill www.wooollll.oo.. www.:pppw...iilllla.ooeaaeaaaaaa
25
https://platfores openai...pppe.n....mm..mm www.aai..coo/ooooae
26

---

Published in Transactions on Machine Learning Research (06/2025)
The specifies of these actions may differ across platforms and applications, necessitating tailored approach for each environment.
While GUI operations form the foundation of agent interactions with the GUI, they can be relatively slow due to the sequential nature of these tasks. Each operation must be executed step by step, which can lead to increased latency, especially for complex workflows that involve numerous interactions. Despite this drawback, GUI operations are crucial for maintaining broad compatibility across various applications, they leverage standard user interface elements and interactions.
4.5.2 Native API Calls
In contrast. to GUT operations, some applications provide native APIs that allow GUT agents to perform actions more efficiently. These APIs offer direct access to specific functionalities within the application, enabling the agent to exccute complex tasks with a single command Lu et al. (2024a). For instance, calling the Outlook API allows an agent to send an email in one operation, whereas using GUI operations would require a series of steps, such as navigating through menus and filling out forms Song et al. (2024b).
While native APIs can significantly enhance the speed and reliability of action execution, their availability is limited. Not all applications or platforms expose APIs for external use, and developing these interfaces require substantial effort and expertise. Consequently, while native APIs present powerful means for efficient task completion, they may not be generalized across different applications GUI operations.
4.5.3 Al Tools
The integration of AI tools into GUI agents represents transformative advancement in their capabilities. These tools can assist with a wide range of tasks, including content summarization from screenshots text, document enhancement, image or video generation (e.g., calling ChatGPT [w et al. (2023c), DALL E Ramesh et al. (2021)). ) and even invoking other agents or Copilot tools for collaborative assistance. The rapid development of generative AI technologies enables GUI agents to tackle complex challenges that were previously beyond their capabilities.
By incorporating AI tools, agents can extend their functionality and enhance their performance in diverse contexts. For example, GUI agent could use Al summarization tool to quickly extract key information from a lengthy document or leverage an image generation tool to create custom visuals for user presentations. This integration not only streamlines workflows but also empowers agents to deliver higl-quality outcomes in a fraction of the time traditionally required.
4.5.4 Summary
An advanced GUT agent should adeptly leverage all three categories of actions: GUT operations for broad compatibility, native APIs for efficient execution, and AI tools for enhanced capabilities. This multifaceted approach enables the agent to operate reliably across various applications while maximizing efficiency and effectiveness. By skillfully navigating these action types, GUI agents can fulfill user requests more proficiently, ultimately leading to more scamless and productive user experience.
4.6 Memory
For a GUI agent to achieve robust performance in complex, multi-step tasks, it must retain memory, enabling it to manage states in otherwise statcless environments. Memory allows the agent to track its prior actions, their outcomes, and the task's 's overall status, all of which are crucial for informed decision-making in subsequent steps Lee et al. (2023). By establishing continuity, memory transforms the agent from a reactive system into proactive, stateful one, capable of self-adjustment basod on accumulated knowledge. The agent's memory is generally divided into two main types: Short-Tenn Memory Lu et al. (2023) and Long-Tere Term Memory Wang et al. (022n). We show overview of different. types of memory in GUT agents in Table
27

---

Published in Transactions on Machine Learning Research (06/2025)
Task Play Recap of Arcane Scason
Task B: Download the game related to the video just played.
Step!
Plan
()) use google search
the previous task the video "Arcane" which is related to the "League of Legend which should search for. (2) Enter the official site the game.
" Recap of Arcane Season
(5) top
and dlick.
(3) Play the video.
Stóp2
(3) Click Download.
Stápa
Plan
()) need to click the "Arcane Seacon
(1) need to click the first result of "League of Legend the website.
Recap |Netfix"
(3) Play the video just entered.
Steps
Click Download.
Step3
GUI Agent
GUI Agent
Plan
Plan
Click "Play" button play the video.
(1) Clck Download
the game.
Short-term Memory:
Short-term Memory:
Imput(
Inpurt Seacont Recap
Seasont Recap"
Clicid "AArcane Season Recap] Netflix
Click( "Play"
Click( "League Legend"
Click, " "Downlead Windows"
"Search"
Ciick(
"Search"
Plan
Plan a
Plan 3 Step 3
Plan
Plan 3
Step
Step a
Step
Step 2
Step
Long- term Memory: Task just used Google search for the "Recap of Arcane Season 1" and play the top searched results on YouTube.
Figure 14: Illustration short-term memory and long-term memory in an LLM-brained GUT agent.
Table 6: Summary of memory in GUI agents.
Memory Element
Memory Type
Description
Storage Medium/Method Meehhod
Action
Short-term
Historical actions trajectory taken in the environment
In-memory, Context window
Plan
Short-term
Plan passed from previous step
I n-memoryy Context window
Execution Results
Shorl-lerm
Return values, erzor uraces, and other environmental leedback
In n-memory, Context window
Enviroament State
Shori term
Important environment state data, e.g. UI elements
In memory, Context window
Self experience
Long term
lask completion trajectories from historical tasks
Database, Disk
Self- guidance
Long term
Guidance and rules summarized from historical trajectories
Database, Disk
External Knowledge
Long-term
Other external knowledge sources aiding task completion
External Knowledge Base
Task Suocess Metries
Long-term
Metries from task snccess or failure rases across sessions
Database, Disk
4.6.1 Short- Term Memory
Short-Term Memory (STM) provides the primary, ephemeral context used by the LLM during runtime Tack et al. (2024). STM stores information pertinent to the current task, such recent plans, actions, results, and environmental states, and continuously updatos to reflect the task's ongoing status. This memory is particularly valuable in multi-step lasks, where each decision builds on the previous one, requiring the agent to maintain a clear understanding of the task's 's trajectory. As illustrated in Figure 14 during the completion of independent tasks, the task trajectory, comprising actions and plans is stored in the STM. This allows the agent track task progress effectively and make more informed decisions.
However, STM is constrained by the LLM's context window, limiting the amount of information it carry forward. To manage this limitation, agents employ selective memory management strategies, such as selectively discarding or summarizing less relevant details to prioritize the most impactful information. Despite its limited size, STM essential for ensuring coherent, contextually aware interactions and supporting the agent's capacity to execute complex workflows with immodiate, relevant foodback.
4.6.2 Long- Term Memory
Long- Term Memory (LTM) serves as an external storage repository for contextual information that extends beyond the immediate runtime Zhu et al. [2023a] Unlike STM, which is transient, LTM retains historical
28

---

Published in Transactions on Machine Learning Research (06/2025)
task data, including previously completed tasks, successful action sequences, contextual tips, and learned insights. L.T can be stored on disk or in a database, enabling it to retain larger volumes of information than what is feasible within the LLM's immediate context window. In the example shown in Figure [4 when the second task requests downloading a game related to the previons task, the agent retricves relevant information from its LTM. This enables the agent to accurately identify the correct game, facilitating efficient. task completion.
LTM contributes to the agent's self-impro self-improvement over time by preserving examples of successful task trajectories, operational guidelines, and common interaction patterns. When approaching a new task, the agent can leverage RAG techniques to retrieve relevant historical data, which enhances its ability to adapt strategies based on prior success. This is similar to the lifelong learning Zheng et al. (2025c), which makes LTM instrumental in fostering an agent's capacity to "learn" from experience, enabling to perform tasks with greater accuracy and efficiency as it accumulates insights across sessions. For instance, [Zheng et al.] (2024d) provides an illustrative example of using past task trajectories stored in memory to guide and enhance future decision-makknn, a technique that is highly adaptable for GUT agents. It also enables better personalization by retaining information about previous tasks.
4.7 Advanced Enhancements
While most LLM-ppoered GUT agents incorporate fundamental components such as perception, planning, action execution, and memory, several advanced techniques have been developed to significantly improve the reasoning and overall capabilities of these agents. Here, we outline shared advancements widely adopted in research to guide the development of more specialized and capable L.M--powreed GUT agents.
4.7.1 Computer Vision-Basse GUI Grounding
Although various tools (Section 4 enable GUI agents to access information like widget location, captions, and properties, certain non-standard GUIs or widgets may not adhere to these tools' protocols Zhan et al. (2021). rendering their information inaccessible. Additionally, due to permission management, these tools are not always usable. Such incomplete information can present significant challenges for GUI agents, as the LLM may need independently locate and interact with required widgets by estimating their coordinates perform actions like clicking task that is inherently difficult without precise GUI data.
CV models offer a non-intrusive solution for GUI grounding directly from screenshots, enabling the detection, localization, segmentation, and even functional estimation of widgels Li et al. (2020b); White al. (2019); Wang et al. (2021); Bai et al. (2021). This approach allows agents to interpret the visual structure and clements the GUI without relying on system-level tools or internal metadata, which may be unavailable or incomplete. CV-based GUI parsing provides agents with valuable insights into interactive components, screen layout, and widget functionalities based solely on visual cues, enhancing their ability to recognize and act upon elements on the screen. Figure I0 provides an illustrative example of how a CV-based GUL parser works. While standard API-based detection captures predefined widgets, the CV model can identify addit ional elements, such as thumbnails and canvases, which may not have explicit APT represent intvessntations ations in the PowerPoint interface. This enhances widget recognition, allowing the agent to detect components beyond the scope of API detection. We show an overview of related GUI grounding models and benchmarks in Table 7
9 and 10
A notable example OmniParsor Lu et al. (2024d), which implements a multi-stage parsing technique involving fine-tuned model for detecting interactable icons, an OCR module for extracting text, and an icon description model that generates localized semantic descriptions for each UI element. By integrating these components, OmniParser constructs a structured representation of the GUT, enhancing an agent's 's understanding of interactive regions and functional elements. This comprehensive parsing strategy has shown to significantly improve GPT-4V's screen comprehension and interaction accuracy.
Such CV-based GUI grounding layers provide critical grounding information that significantly enhances an agent's 's ability to interact accurately and intuitively with diverse GUIs. This is particularly beneficial for handling custom or non-standard elements that deviate from typical accessibility protocols. Additionally,
29

---

Published in Transactions on Machine Learning Research (06/2025)
Table 7: A summary of GUI grounding models (Part 1).
Model
Platform
FoundationSize
Dataset
Input
Output
Highlight
Link
Model
OmniParset
Mobile,
BLIP-2
67,000 UT
UT
Ms. bound- boxes,
Introduces purely vision-based
https:// github. com/
Lu
al
Desktop.
Li
screenshots with shots bounding box annotations and 7,185 icon- description pairs generated using GPT-4
2024d
and Web
2023e
and descrip- tions of for general UI interactable understanding elements without external information, signif- icantly improving action prediction accuracy for LLM- driven agents
parsing framework
microsoft/
YOLOv8
CmniParser
Reis
2023
Iterative Nar-
Mobile, Web, and Desktop
Qwen2- VL OS-Atlas- Base
ScreenSpot
GUI
(x,)
C0-
Progressively ctops regions the GUI to refine predictions, euhancing precision for GUI grounding
httpa:
Cheng et al. screenshot 2024a
ordinates
//github.
rowing
and a natu- ral language query
representing ing
com/ant-8l
Nguyen
target
GUI- -rl-Gronding--- -ia-iin----ivvvi Iterative-lk
20/24
location
GUI
Iris
Ge
Mobile (iOS,
Qwen-
9.6B
850K
GUI- High-
Referring: Generates detailed de- scriptions of shots with
Information- Sensitive Crop- ping for efficient handling high- resolution GUI images, and Self- Refining Dual elements Learning to iter- atively enhance GUI grounding and referring tasks without additional annotations
Bai
specific annota- tions and 150K vision-language instructions
resolution
2024
Android), Desktop (Win- dows,
2023b
GUI screen-
natural language
UI elements. Grounding: Locates UI
macOS), and Web
instructions
the screen.
Attention- driven
Mobile, Web, and Desktop
MiniCPM- Llama3- V 2.5
8.5B
Wen Deng
GUT scrven-
Element localization via bounding boxes, text- to-image mapping for ground- ing. actionable descriptions of GUI
Utilies attention mechanisms in pre-trained MLLMs without fine-tuning
https://
et al.
2023
shots and
github. com/
Ground- ing I
ScreenSpor pot
textual user queries
HeiningX/ TAG
Xm
Cheng
ct al 20124 Visual
2024b
WebBench Lin
et al.
20240
ponents GUI screen Pixel
Aria UI
Aria
3.9B
million ele- and 11.5 shots, ordinates instructions, for
purely
ht
Yang
Desktop, and Mo-
ct
based approach avoiding reliance AXTree-like inputs
/ariaui.
ef
2024b
million samples and action histories
GUI elements and correspond ing actions
titb..
20244b
bile
prompting methods like iterative narrowing have shown promise in improving the widget grounding capabilities of VL.Ms Nguyen| (2024). Together, these approaches pave the way for more adaptable and resilient GUT agents, capable of operating effectively across broader range screen environments and application contexts.
Several works have introduced benchmarks to evaluate the GUI grounding capabilities of models and agents. For instance, ScreenSpot Cheng et al. (2024a) serves pioneering benchmark designed to assess the GUI grounding performance of LLM-powered agents across diverse platforms, including iOS, Android, macOS, Windows, and web environments. It. features a dataset with over 600 screenshots and 1,200 instructions, focusing on complex GUI components such as widgets and icons. This benchmark emphasizes the importance of GUI grounding in enhancing downstream tasks like web automation and mobile UT interaction. Building upon this, ScreenSpot-Pro Li et al. (2025b) extends the scope to more professional, high-resolution environments. This evolved version includes 1,581 tasks with high-quality annotations, encompassing domains such
30

---

Published in Transactions on Machine Learning Research (06/2025)
Table summary of of GUI grounding models (Part 11).
Model
Platform
FoundationSize
Dataset
Input
Output
Highlight
Link
Model
UGround
Web,
LLaVA-
7B
Weh- and
GUI
scrren-
Pixel enordi- of GUT elements
universal GUI
httpe://
Gon al
Desktop
NeXT-7B
other existing datasets
shots, queries
grounding model oau-nip- p-goopoppppp
2024
(Win-
Liu et al
that relies solely on vision, eliminat-- ing the need for text-based represen- tations
github. 110
dows,
2024b
Urronnd/
MacOS,
Limux),
Mobile
(Android,
10S]
GUI-Bee Fan et al.
Web
SeeClick Cheng ct
7B-13B
po
GUI screen- shots, user queries, BC- cessibility tree
GUI element grounding locations, actions and function calls, navi- gation steps, predicted GUT changes after interac- tion
Autonomously ex-- plores GUI envi- romments, with Q- ICRI optimizing ex- ploration efficiency and euhancing data diversity.
httpa:
/gui- bee.
2025
github.
2024a
Qwen-
Ist
(20224)
and
LTX-
Lu
ct
2024e
RWKV-
Web
1.613
SIGLIP Zhai of al.
Websight
Lau-
High-
Element grounding, Action pre- diction, CoT reasoning
Introducrs bigh-resolution three encoder archi- tecture with visual prompt enginecring and CoT reasoning,
UI
Yang
rençon
resolution webpage
Hoi
2023
2024b
WebUT
2022
DINOv2
kkbal
Wn
images
Oquab
et
al.
2023b
Web2Code
Code
Ym
21223
et al.
2124
SAM Kir
Flvv et al
2023
TRISHUL
Web, Desk- top, and Mobile platforms
(Triiing
GUI Screen- shots, user instruction- functionality descriptions
Action grounding,
Utilizes hierarchical screen parsing and spatially enhanced element descrip- tions euhance LVLMs without additional training.
Singh
(Triniing- Free)
(Training- Free)
Free)
2025
s/queries, hierarchi- of GUI ele- screen parsing out- referring, SoMs
ments,
puts, OCR- extracted text descrip-
software development, creative tools, CAD, scientific applications, and office productivity. Key features of ScreenSpot- Pro include authentic high-rosolution screenshots and moticulous annotations provided by domain experts.
These benchmarks provide critical evaluation criteria for assessing GUI grounding capabilities, thereby advancing the development of GUI agents for improved GUI understanding and interaction.
4.7.2 Multi-Agent Framework
The adage "two heads are better than one" holds particular relevance for GUI automation tasks, where single agent, though capable, can be significantly enhanced within a multi-agent framework Li et al. (2023a) Chen el al. (2024h). Multi-agent systems leverage the collective intelligence, specialized skills, and complementary strengths of multiple agents to tackle complex tasks more effectively than any individual agent could alone. the context of GUI agents, multi-agent systems offer advanced capabilities through two primary mechanisms: (i) specialization and (ti) inter-agent collaboration. Figure illustrates an example of how an LLM-pooerod multi-agent collaborates to create desk.
31

---

Published in Transactions on Machine Learning Research (06/2025)
Table 9: A summary of GUI grounding models (Part 111).
Model
Platform
FoundationSize
Dataset
Input
Output
Highlight
Link
Model
AutoGUI
Web, Mo-
Qwen-
10B 8B
AutoGUI-704k
GUI
scrren-
Element
Automatically
https://
Gi
al
bile
VI-10B
shots, User
functionali- Lies, Element
labels UT ele- ments based interaction-incueed
autogui Paoogu-projeen
20256
Bai et al.
queries
github.io/
2023b
locations changes, making it scalable and high-quality.
SiMME-BB
Zhang
et
al.
2024o
Query Inference
Mobile
wwen2
7B
UIBERT
Wn
GUI screem-
Action- oriented queries, ordinates
Improves reasoning
ht
Android
V VL-7B-
et al. 20241
shots
without requiring
/github.
Wu et al.
Instruct
Co-
large-scale training data.
com/ 2rw0o/
2025d
Wang
CUIPivot
d
al.
2024j
WinClick
W Windows
P Phn3-
4.2B
WinSpot Bench-
GUI screen- shots, Natu- ral language instructions
Element loca- tions
The first GUI grounding model specifically tailored
httpa://
Hui et al.
OS
Vision
mark
github. c
20225
Abdin
zackbui113
ot
for Windows.
WinSpot
20241
FOCUS
Qwen2-
2B
GUICourse
GUI Scroen- Nortallizdd coordinates (x, y)
dual-spstem httpa://
Tang et 2023a
mobile
VL-2B-
Chen
shot task instruction
grounding archi- github. com/
applica- and
Instruct
20241
Aguvis-
inspired human cognition, which dyuamically switches between fast. (intuitive) and slow (analytical) grounding modes based on task com- plexity
sugarandgug u/
Wang
stagol
Focua
desktop
et
2024k
20244j
Wave-
Zheng
et
al.
2024c
Drsktopp
CT Lin
et al.
2024c
UI-E2I- Synch
Web, Windows,
InternVL2- 4B and
4B
and
16M screen-
GUI screen- shot
Element ordinates
Introduces a three- stage synthetic data pipeline for GUI grounding with both explicit and implicit. instruction synthesis
https:
7B
shots,
9.9M
//colmnd6.
Liu et al.
and Au- droid
Qwen2- VL-7B
Instructions
github. 10/
2025d
12e bench leaderboard/
Region Web-
UL-TARS
72B
Noue (test-time only)
GUI screen- shots with a point of inter- est
Coordinate-
Introduecs a visual test: time scaling framework that tiangeluo/ zooms into salient regions integrates an image- as-map mechanism track history and avoid repeated mistakes boosting grouding accuracy without model retraining
httpa://
Luo et al.
based and
and Qwen2.5-
based actions
github. comm
2025b
Deslctop
interfaces
VL
RegionFocus
Specialization of Agents: In multi-agent framework, each agent is designed to specialize in specific role function, leveraging its unique capabilities to contribute to the overall task. As illustrated in the Figure 15 specialization enables distinct agents to focus different aspects of the task pipcline. For instance, the "Document Extractor" specializes in extracting relevant content from local documents, such as PDFs, while the "Web Retriever" focuses on gathering additional information from online sources. Similarly, the "Designer" transforms the retrieved information into visually appcaling slides, and the 'Evaluator' provides foodback to refine and improve the output. This functional separation ensures that each agent becomes highly adept its designated task, leading to improved efficiency and quality of results Song et al. (2024d).
32

---

Bounding boxes for lo- cating target elements Published in Transactions on Machine Learning Research (06/2025)
Table 10: summary of of GUI grounding benchmarks.
Benchmark
Platform
Dataset
Input
Output Bounding boxes or
Highlight
Link httpe://gichuu
ScreenSpot Cheng et al. 2024a
iOS, An- droid, ma- cOS, and Windows
GUI screenshots ac- shots and 1,200 companied by user instructions
Over 600 screen-
realistic
verse GUI ground- ing benchmark cov- ering multiple plat- forms and variety of elements
com/ jom/ajucckevin/
structions
coordinates of action-
Seeclick
able GUI elements
ScreenSpot- Pro L et al.
Windows, macOS, and Linux
1,581 instruction- screenshot pairs covering appli- cations industries and
High-resolution
Introduces
httpa://
GUI screenshots
risl-resolution benchmark for professional envi- romments
github.com
2025b
paired with natural
Likaixin2000/
language instrue- tions
ScreeuSpot-Poo-UU Grounding
operating systems 100,000 webpages
PixelWeb
Web
Rendered webpage screenshots and DOM information
BBex, mask, contour
The first GUI httpa://
Yang ct al.
dataset to provide pixel-level annota- tions including mask and con- tour-for web UIs, enabling high-precision GUI grounding and detection tasks
huggingface.
2025c
co/ co/datasets/
cyberalchenist/
PixelWeb
Task: Create desk for LLM-based multi-agent system.
Local Documents
Document
Created Desk
Extractor
Designer
Creation
5D'
Communication
Feedback
Evaluation
Web Retriever
Evaluator
Webpages
Figure 15: An example of multi-agent system collaboration in creating desk.
Collaborative Inter-Agent Dynamics: The multi-agent system shown in the Figure 15 exemplifies how agents collaborate dynamically handle complex tasks. The process begins with the " "Document. Extractor" and "Web Retriever", which work in parallel to collect information from local and online sources. The retrieved data is communicated to the "Designer", who synthesizes it into cohesive set of slides. Once the slides are created, the "Evaluator" reviews the output, providing feedback for refinement. These agents share information, exchange context, and operate in a coordinated manner, reflecting a human-like leamwork dynamic. For example, as depicted, the agents' roles are lightly integrated each output feeds into the next stage, creating a streamlined workflow that mirrors real-world collaborative environments Zhang al. (2024a).
In such system, agents can collcctively engage tasks requiring planning, discussion, and docision-making, Through these interactions, the system taps into each agent's domain expertise and latent potential for specialization, maximizing overall performance across diverse, multi-step processes.
33

---

Published in Transactions on Machine Learning Research (06/2025)
Task: Make Line Drawing effect to the figure in the page.
Step Click(
"Design"
2: Click * Picture Format"
Step ClicK( "Artistic Effects"
Step Click( "Line Drawing"
accessed the "Design" but
After some reflection experimentation, seems to select the figure, then navigate to the "Picture Format" tab. There, I can the "Artistic Effects" menu, where the Drawing" option hidden. Task complete!
GUI Agent
couldn't find an option to create a screen.
drawing effect
Self-reflection
Perhaps there's another approach
should consider...
Figure 16: An example of self-reflectiog in task completion of an LLM-powered GUI agent.
4.7.3 Self-Reflection
"A fault confessed is half redressed In the context of GUI multi-agent systems, self-reflection refers to the agents' capacity to introspectively assess their reasoning, actions, and decisions throughout the task execution process Renze Guven (2024). This capability allows agents to detect potential mistakes, adjust strategies, and refine actions, thereby improving the quality and robustness of thoir docisions, especially in complex or unfamiliar GUT environments. By periodically evaluating their own performance, self-reflective agents adapt dynamically to produce more accurate and effective results Pan et al. (024a)
Self-reflection is particularly critical for GUI agents due to the variable nature of user interfaces and the potential for errors, even in human-operated systems. GUT agents frequently encounter siluations that deviate from expectations, such as clicking the wrong button, encountering unexpected advertisements, navigating unfamiliar interfaces, receiving error messages from API calls, or even responding to user feedback on task outcomes. To ensure task success, a GUI agent must quickly reflect on its actions, assess these feedback signals, and adjust its plans to better align with the desired objectives.
As illustrated in Figure 16 when the agent initially fails to locate the "Line Drawing" option in the Design menu, self-reftection enables it to reconsider and identify its correct location under Artistic Effects" in the "Picture Format" " menu, thereby successfully completing the task.
In practice, self-reflection techniques for GUI agents typically involve two main approaches: et al. (022bb and (ti) Reflexion Shhinn al. (2024).
(i) ReAct Yao
1. ReAct (Reasoning and Acting): ReAct integrates self-reflcction into the agent's action chain by having the agent. evaluate each action's 's outcome and reason about the next best step. Tn this framework, the agent doesn't 't simply follow a linear sequence of actions; instead, it adapts dynamically, continuously reassessing its strategy in response to feedback from each action. For example, if a GUI agent attempting to fill a form realizes it has clicked the wrong field, it can adjust by backtracking and selecting the correct element. Through ReAct, the agent achieves higher consistency and accuracy, as learns refine its behavior with each completed step.
Reflexion emphasizes language-based feedback, where agents receive and process feedback from the environment linguistic input, referred to as self-reflective feedback. This feedback is contextualized and used input in subsequent interactions, helping the agent to learn rapidly from prior mistakes. For instance, if a GUT agent receives error message from an application, Reflexion enables the agent to process this message, update its understanding of the interface, and avoid similar mistakes in future interactions. Reflexion's iterative feedback loop promotes continuous improvement and is particularly valuable for GUI agents navigating complex, multi-step tasks.
Reflexion:
Overall, self-reflection serves as an essential enhancement in GUI multi-agent systems, enabling agents better navigate the variability and unpredictability of GUT environments. This introspective capability not.
34

---

Published in Transactions on Machine Learning Research (06/2025)
Task Summarize the image content
New Toolkits: Image summarization
Skill Set
New Task: Learn to make GUI
Task Read and summarize the paper
Tool
agent from a GitHub repository
New Tool
Tool
Guidance and Rules: How make CUI agent?
Evolve
Completion
Experience Pool
Task 3: Read and extract Information from a webpage
Trajectory
Guidance
Task Trajectory:
Webpages extraction
Figure 17: An example self-evolution in a LLM-powerod GUI agent with task completion.
only boosts individual agent performance but also promoles resilience, adaptability, and long- term learning in collaborative setting.
4.7.4 Self-Evolution
Self-evolution Tao al. (024)) a crucial attribute that GUI agents should possess, enabling them enhance their performance progressively through accumulated experience. In the context of GUI multi-agent systems, self-cvolution allows not only individual agents improve but also facilitates collective learning and adaptation by sharing knowledge and strategies among agents. During task execution, GUI agents generate detailed action trajectories accompanied by complementary information such as environment states, internal reasoning processes (the agent's 's thought processes), and evaluation results. This rich data serves as a valuable knowledge base from which GUI agents can learn and evolve. The knowledge extracted from this experience can be calegorized into three main areas:
Task Trajectories: The sequences of actions executed by agents, along with the corresponding environment states, are instrumental for learning Zhao et al. (0024a). These successful trajectories can be leveraged in two significant ways. First, they can be used to finc-tune the core LLMs that underpin the agent models with the state-action pairs Wang al. (2024h). Fine-tuning with such domain-specific and task-relevant data enhances the model's ability to generalize and improves performance on similar tasks in the future. Second, after summarized by LLMs Tao et al. (024)), these trajectories can be utilized as demonstration examples to activate the in-context learning capabilities of L.MSs during prompt engineering. By including examples of successful task executions in the prompts, agents can better understand and replicate the desired behaviors without additional model training.
For instance, suppose an agent successfully completes a complex task that involves automating data entry across multiple applications. The recorded action trajectory trajectoyy-comprisnng comprising the steps taken, decisions made, and contextual cues can be shared with other agents. These agents can then use this trajectory as guide when faced with similar tasks, reducing the learning curve and improving efficiency.
Guidance and Rules:
From the accumulated experiences, agents can extract high-level rules guidelines that encapsulate best practices, successful strategies, and lessons learned from past.
35

---

Published in Transactions on Machine Learning Research (06/2025)
Task: Make Line Drawing effect to the figure in the page.
Click( "Picture Format"
Action 2: Click( "Artistic Effects )
Action 3: Click( Drawing" )
State
State
State
Evaluator
GUI Agent
Reward
Action 1:: Click( "Design"
Action Click "Dsigner"
Action Click( "Format 3
x
State
State
State 3'
Figure 18: An example of MDP modeling for task completion in a GUI agent.
mistakes Zhu et al. 2003b) : Zhang et al. (202pp). Such guidance can be acquired by the LLM itself through trajectory summarization Zhhu et al. (2023b), or even via search-based algorithms, such as Monte Carlo Tree Search (MCTS) Zhang et al. (2024p). This knowledge can be formalized into policies or heuristics that agents consult during decision-making processes, thereby enhancing their reasoning capabilities.
For example, if agents repeatedly encounter errors when attempting to perform certain actions without proper prerequisiles (e.g., trying to save a file before specifying a file path), they can formulate rule to check for these prerequisites before executing the action. This proactive approach reduces the likelihood of errors and improves task success rates.
3. New Toolkits: Throughout their interactions, GUT agents may discover or develop more efficient. methods, tools, or sequences of actions that streamline task execution Tan et al.) (2024a). These may include optimized API calls, macros, or combinations of GUI operations that accomplish tasks more effectively than previous approaches. LLMs can be leveraged to automatically analyze execution trajectories in order to summarize, discover, and generate high-level shorteuts frequently used fast APls, which can then be reused for future executions Jiang et al. (2025). By incorporating these new tools into their repertoire, agents expand their capabilities and enhance overall efficiency.
As example, an agent might find that using batch processing API automate repetitive tasks more efficiently than performing individual GUI operations in loop. This new approach can be shared among agents within the multi-agent system, allowing all agents to benefit from the improved method and apply it to relevant tasks.
Figure 17 illustrates how a GUI agent evolves through task completion. During its operations, the agent adds new capabilities to its skill set, such as an image summarization toolkit, gains insights from reading paper creating GUI agents, and stores task trajectories like webpage extraction in its experience pool. When assigned a new task, such as "Learn to make GUI agent from GitHub repository", the agent draws on its acquired skills and past experiences to adapt and perform effectively.
This dynamic evolution highlights the agent's ability to continually learn, grow, and refine its capabilities. By leveraging past experiences, incorporating new knowledge, and expanding its toolsct, GUI agents can adapt to diverse challenges, improve task execution, and significantly enhance the overall performance of the system, fostering collaborative and ever-improving environment.
36

---

Published in Transactions on Machine Learning Research (06/2025)
4.7.5 Reinforcement Learning
Reinforcement Learning (RL) Kaelbling ct al. (1996) has witnessed significant advancements in aligning LLMs with desired behaviors Wang et al. (2023c). and has recently been employed in the development of LLM agents Sun et al. (2024a); Zhai et al. (2024). In the context of GUI multi-agent systems, RL offers substantial potential to enhance the performance, adaptability, and collaboration of GUI agents. GUI automation tasks naturally align with the structure of Markov Decision Process (MDP) Putermani (1990). making them particularly well-suited for solutions based on RL. In this context, the state corresponds to the environment perception (such as GUI screenshots, GUI element properties, and layout configurations), while actions map directly to GUI operations, inchuding mouse clicks, keyboard inputs, and API calls. Rewards can be explicitly defined based on various performance metrics, such as task completion, efficiency, and accuracy, allowing the agent to optimize its actions for maximal effectiveness. Figure 18] illustrates an example of MDP modeling for task completion in a GUI agent, where state, action and reward are clearly defined.
By formulating GUT agent interactions as an MDP, we can leverage RL. techniques to train agents that learn optimal policies for task exccution through trial and error Toyama et al. (2021a). This approach enables agents to make decisions that maximize cumulative rewards over time, leading to more efficient and effective task completion. For example, an agent learning to automate form filling in a web application can use RL to discover the most efficient sequence of actions to input data and submit the form successfully, minimizing errors and redundant steps. This process helps aligu the agents more closely with desired behaviors in GUT automation tasks, especially in complex or ambiguous situations where predefined action sequences are insufficient.
As a representative approach, Bai et al., introduce DigiRL Bai et al. (0044), a two-phase RL framework for training GUT agents in dynamic environments. DigiRT. begins with an offline RT. phase that uses offline data to initialize the agent model, followed by online fine-tuning, where the model interacts directly with an environment refine its strategies through live data within an Android learning environment using an LLM evaluator that provides reliable reward signals. This adaptive setting enables the agent to learn and respond effectively to the complexities of dynamic GUTs. Wang et al., propose DistRL Wang et al. (2024l), RL fine-tuning pipeline specifically designed for on-device mobile control agents operaling within Android. DistRL employs an asynchronous architecture, deploying RL fine-tuned agents across heterogeneous worker devices and environments for decentralized data collection. By leveraging off-policy RL techniques, DistRL enables centralized training with data gathered remotely from diverse environments, significantly enhancing the scalability and robustness of the model. These representative methods illustrate the potential of RT to improve GUI agents, demonstrating how both centralized and distributed RL frameworks can enable more responsive, adaptable, and effective GUI automation models in real-world applications.
4.7.6 Summary & Takeaways
In conclusion, the advanced techniques significantly enhance the capabilities of LLM--bained GUI agents, making them more versatile, efficient, and adaptive within multi-agent frameworks. Importantly, these techniques are not mutually teehusive-mary can be integrated to create more powerful agents. For instance, incorporating self-reflection within multi-agent framework allows agents to collaboratively improve task strategies and recover from errors. By leveraging these advancements, developers can design LLM--bained GUI agents that are not only adept at automating complex, multi-step tasks but also capable of contimuously improving through self-evolution, adaptability to dynamic environments, and effective inter-agent collaboration. Future rescarch is expected to yicld even more sophisticated techniques, further extending the scope and robustness of GUT automation.
4.8 From Foundations to Innovations: A Roadmap
Building robust, adaptable, and effective LLM-powered GUI agents is a multifaceted process that requires careful integration of several core components. With a solid foundation in architecture, design, environment interaction, and memory, as outlined in Scction + we now shift our focus the critical elements required for deploying these agents in practical scenarios. This exploration begins with an in-depth review of state-of fate-of-thheart L.M--poeere GUT agent frameworks in Section 5 highlighting their ad advancements and unique contributions
37

---

Published in Transactions on Machine Learning Research (06/2025)
Table 11: Taxonomy of LLM-pooered GUI agent frameworks by target platform.
Platform
References
Web
Zheng et al. (0024a); Song et al. (2024b); Chae et al.
20244]; Gur al.
(2024);
Ma et al [0244b] He et al. (2024b]; Lai et al. [2024]
Xie et al. (2023
Kil et al.
(2024]
A buelsaad et al. (2024); Koh et al. (2024b); Zhang et al. (2024n) 0 Yang
ct al.
(2024a):
; Murty et al. (2024); Shahbandch ct al. (2024); long ct al. 2024);
Tang & Shin (2024); Putta et al. 2024):
: Gu et al. (2024); Verma et al.
(2024))
Kim et al. (2024b); Shen et al. (2024b) Zhou et al. (202 Liu et al.
[2024c];
Huang al. (2025b):
Zhang ct al. (2025el: Pahuja ct al. (2025): Wornow
2024
Zhang et al.
2025c
Dammu [2025]
; Erdogan al. (2025 Zheng
2025a
Wang al.
(2025g Zhang et al.
(025m)
Mobile
Zhang ct al. (2023a)
Wang et al.
(024c));
Wen et al. (2024a]; Zhang et al.
(2024g Song et al.
(2024c); Wen et al. (2024c); Ma et al. (2024d); Zhhag
Zhang 2024); Yan al. (2023a); Li (2024g); Wen et al. (2024bb); Wang
et al. 20224 ; Zhang et al. (2024e) Christianos al. (2024): 0 Zhu et al. (024b));
Lee ct al. (2024c): Wang et al. (2025e): Hoscilowicz et al.
(0024)
Wu et al.
(2025c 5 Wang et al. (2025c); |uann el al. Wang al. 2025b
Liu el al.
2025g
: Jiang et al. (2025 ; Zhou et al. (2025) 0 Cheng et al.
(2025a):
Dai et al.
2025
;
Liu et al. (2025a); Lai ct al. (025 Wang et al.
2023a
Kahlon et al.
2025
Bishop et al.
2024
Computer
Zhang et al.
(024aa)
;
Tan et al.
(0224a): Wu et al. (2024e) Li et al. (024g): ;
Agashe ct al.
[0244]
Wu et al.
(2024b)
Li et al. (2023d)
He et al. (2024d)
Liu
ee al.
(2025c]:
Aggarwal & Welleck (2025]
Zhao el. al. (2025a | Lu el al. 2025
Zhang et al.
2025b
Yin et al. (2025)
Cross-
Liu et al.
20244h
Xu et al.
(202kk)
Pawlowski et al. (2024); Song al. 2024d))
Platform
Su et al.
2025
He et al.
2025):
Wang & Liu (2024)) b;
Jia et al. (2024]
Wang
et al.
2024o
Liu et al.
(025e)
Agashe et al. (022 )
Hu et al. (0025)
:
Huang
ct al.
2025a
the field. Building on this, we delve into the methodologies for optimizing LLMs for GUI agents, starting with data collection and processing strategies in Section and progressing to model optimization techniques in Section 7 To cnsure robust development and validation, then examine evaluation methodologics and benchmarks in Section which are essential for assessing agent performance and reliability. Finally, we explore a diverse range of practical applications in Section demonstrating the transformative impact of these agents across varions domains.
Note that we intentionally limit in-depth discussions the main text (Sections 9h to a carefully curated set of representative works. The selection criteria include demonstrated impact (e.g., citation count, GitHub adoption), methodological novelty, and alignment with our core taxonomy. Works that less mature narrower in scope- scope-regardless of publication status-an are instead cataloged in summary tables (Table 16 69 which have been relocated to the appendix to improve readability and streamline the main narrative. Together, these sections provide a comprehensive roadmap for advancing LLM-powered GUI agents from foundational concepts to real-world implementation and innovation. This roadmap, spanning from foundational components to real- world deployment, encapsulates the essential pipeline required to bring an LLM- LLM--owered GUI agent concept from idcation to implementation.
5 LLM-Powered GUI Agent Framework
The integration of LLMs has unlocked new possibilities for constructing GUI agents, enabling them to interpret requests, analyze GUT components, and autonomously perform actions across diverse environments. By
38

---

Published in Transactions on Machine Learning Research (06/2025)
equipping these models with essential components and functionalities, outlined in Section rescarchers have created sophisticated frameworks tailored to various platforms and applications. These frameworks represent a rapidly evolving area of research, with each introducing innovative techniques and specialized capabilitics that push the boundarics of what GUI agents can achieve.
The landscape of GUT agent frameworks has seen notable advancements, particularly in terms of multi-agent. architectures, multimodal inputs, and enhanced action sets. These developments are laying the groundwork for more versatile and powerful agents capable of handling complex, dynamic environments. Key takeaways from recent advancements include:
Multi-Agent Synergy: Multi-agent systems, such as those in UFO Zhang et al. (2024a) and MMAC-Copilos Song et al.] (2024d), represent significant trend in GUI agent development. By assigning specialized roles to different agents within a framework, multi-agent systems can enhance task efficiency, adaptability, and overall performance. As agents take on more complex tasks across diverse flatforms, the coordinated use of multiple agents is proving to be a powerful approach, enabling agents to handle intricate workflows with greater precision and speed.
2. Multimodal Input Benefits: While some agents still rely solely on text-based inputs (e.g., DOM structures or HTML), incorporating visual inpuls, such as screenshots, has shown clear performance advantages. Agents like Web He et al. (2024b) and SeeAct Zheng et al. (2024a) highlight how visual data, combined with textual inputs, provides a richer representation of the environment state, helping agents make better-informed decisions. This integration of multimodal inputs is essential for accurate interpretation in visually complex dynamic environments where text alone may not. capture all necessary context.
3. Expanding Action Sets Beyond UI Operations: Recent agents have expanded their action sets beyond standard UT operations to include API calls and AT-driven actions, seen in Hybrid Agent (Son et al. (2024b) and WetoWe Lai et al. (2024). Incorporating diverse actions allows agents to achieve higher levels of interaction and task completion, particularly in environments where data can be directly retrieved or manipulated through API calls. This flexibility enhances agent capabilitios, making them more efficient and adaptable across a wider range of applications.
4. Emerging Techniques for Improved Decision-Making Novel approaches such world models in WMA Chae et al. (2024) and search-based strategies in Search- Agent Koh et al. (2024b) represent promising directions for more advanced decision-making. World models allow agents to simulate action outcomes, reducing unnecessary interactions and improving efficiency, especially in long-horizon tasks. Similarly, search-based algorithms like best-first and MCTS help agents explore action pathways more effectively, enhancing their adaptability in complex, real-time environments.
5. Toward Cross-Platform Generalization: Cross-platfore frameworks, such as AutoGLM Liu et al. (202hh) and OSCAR Wang & Liu (2024). underscore the value of generalizability in GUT agent design. These agents are pioneering efforts to create solutions that work seamlessly across mobile, desktop, and web platforms, moving closer to the goal of a one-stop GUI agent that can operate across multiple ecosystems. Cross-platform flexibility will be crucial for agents that aim to assist users consistently across their digital interactions.
6. Pure Vision-Based Agent:
To enable universal GUI control, pure vision-based frameworks have emerged as a prominent solution. These agents rely solely screenshots for decision- making, eliminating the need for access to metadata such as widget trees or element properties. Notable work like AGUVIS Xu et al. (2024k) exemplifies this approach. While pure vision-basod methods ofler greater generalizability and bypass system API limitations, they require strong "grounding" capabilities to accurately locate and interact with UI elements an ability often lacking in many foundational models. Fine-tuning models specifically for visual grounding and GUI understanding, or integrating GUI parsing techniques like OmniParser Lu et al. (2024d), can address this challenge and enhance the agent's 's ability to perform precise interactions.
39

---

Published in Transactions on Machine Learning Research (06/2025)
(a) Goal-Oriented Selection
(b) Reflection-Enaaneed Node Expansion
(c) Dynamic Evaluation and Simulation
(d) Maximall Value Backpropagation
Start
Start
Start
Explorer
Explorer
Explorer
Search Result
Search Resultt
Search Result
Search Result Qm6 9.5
Qu4
Qu4
Q=4
Explorer
Explorer
Explorer
Netflix Articl
Netfix Article
YouTube Q-466
Netfix Article Qu
YouTube Q
Netflix Artide O
You Tube Q ..6
0
Explorer
Play
Appralser
Simulation
Figure 19: An illustration of the local optimization stage in WebPilot Zhang et al. (024n using MCTS. Figure adapted from the original paper.
Task Completed
Step
Step
Task: Play the video of Arcane Season
METSLE
0-4
LLM
03
Execution
LLM
Plan:
(1) Click "YouTube Arcane Season
V- 0.9
Recap Netflix
(2) Click the "Play" Button
Figure 20: An example illustrating how WebDreamer Gu et al. (2024) uses an LLM to simulate the outcome of each action. Figure adapted from the original paper.
We offer detailed discussion of each framework, examining their foundational desigu principles, technical advancements, and the specific challenges they address in the realm of GUI automation. By delving into these aspects, we aim to provide deeper insights into how these agents are shaping the future of human-computer interaction and task automation, and the critical role they play in advancing this transformative field. Before delving into the details, we first present a laxonomy of L.M.-poereed GUT agent frameworks categorized by target platform, as shown in Table II
5.1 Web GUI Agents
Advancements in web GUI agents have led to significant strides in automating complex tasks within diverse and dynamic web environments. Recent frameworks have introduced innovative approaches that leverage multimodal inputs, predictive modeling, and task-specific optimizations to enhance performance, adaptability, and efficiency. In this subsection, we first summarize key web GUT agent frameworks in Tables 16
17 18
40

---

Published in Transactions on Machine Learning Research (06/2025)
Workflow construction and demonstration
3. Evaluation
RPA
Hard
Deterministic
Manual
Human Design
Hardcode
Workflow
Validation
Step1
Step 1
Step
Migal Loco
logss
Apent-based
Easy
Flaxible
Automatic
Human Demonstration
Valldatiion
Video Recording
Documentation
Testual
Agent
Execution logs
LLM
Figure 21: Comparison of RPA and agent based automation. Figure adapted from Wornow et al. (2024
20 and then delve into representative frameworks, highlighting their unique contributions and how they collectively push the boundaries of web-based GUI automation.
One prominent trend is the integration of multimodal capabilities to improve interaction with dynamic web content. For instance, SecAct Zheng et al. (2024a) harnosses GPT-4V's multimodal capacitios to ground actions on live websites effectively. By leveraging both visual data and HTML structure, SeeAct integrates grounding techniques using image annotations, HTML attributes, and textual choices, optimizing interactions with real-time web content. This approach allows SeeAct to achieve task success rate of 51.1% on real-time web tasks, highlighting the importance of dynamic evaluation in developing robust web agents.
Building upon the advantages of multimodal inputs, Web He et al. (024b) advances autonomous web navigation by supporting end-to-end task completion across real-world web environments. Utilizing GPT-4V for both visual (screenshots) and textual (HTML clements) inputs, WebVoyager effectively interacts with dynamic web interfaces, including those with dynamically rendered content and intricale interactive elements. This multimodal capability allows WeyVoya to manage complex interfaces with a success rate notably surpassing traditional text-only methods, setting a new benchmark web-based task automation.
In addition to multimodal integration, some frameworks focus on parsing intricate web structures and generating executable code to navigate complex websites. WebAgent Gur et al. (2024) employs two- tiered model approach by combining HTML-T5 for parsing long, complex HTML documents with Flan- U-PaLM Chung et al. 2024) for program synthesis. This modular design enables WebAgent to translate user instructions into executable Python code, autonomously handling complex, real-world websites through task-specific sub-instructions. WebAgent demonstrates a 50% improvement in success rates on real websites compared to traditional single-agent models, showcasing the advantages of integrating HTMI-specific parsing with code generation for diverse and dynamic web environments.
To enhance decision-making in web navigation, several frameworks introduce state-space exploration and search algorithms. LASER Ma et al. (2024b) models web navigation as state-space exploration, allowing flexible backtracking and efficient decision- making without requiring extensive in-context examples. By associating actions with specific states and leveraging GPT-4's function-calling feature for state-based action selection, LASER minimizes errors and improves task success, particularly in e-commerce navigation tasks such as WebShop and Amazon. This state-based approach provides a scalable and efficient solution, advancing the efficiency of LLM agents in GUI navigation.
Similarly, Search- Koh et al. (2024b) innovatively introduces best-first search algorithm to hance multi-step reasoning in interactive web environments. By exploring multiple action paths, this approach improves decision-making, achieving up to a 39% increase in success rates across benchmarks like WebArena Zhou et al. Search-Agent's compatibility with existing multimodal LLMs demonstrates the effectiveness of search-basee algorithms for complex, interactive web tasks.
Expanding on cearch-based strategies, WebPilorZhang combining global and local Monte Carlo Tree Search (MCTS) Browne et al. (2012) to improve adaptability
(0024n) employs a dual optimization strategy
41

---

Published in Transactions on Machine Learning Research (06/2025)
in complex and dynamic environments. As illustrated in Figure 19 WebPilot decomposes overarching tasks into manageable sub-tasks, with each undergoing localized optimization. This approach allows WebPilot to contimuuusy adjust its strategies in response to real-time observations, mimicking human-like decision- and flexibility. Extensive testing on benchmarks like WebArena Zhoou et al. and MiniWoB++ Liu et al. (2018) demonstrates WebPilot's 's state-of-the-arl performance, showcasing exceptional adaplability compared to existing methods.
Furthering the concept of predictive modeling, the WMA Chac et al. 2024 introduces world model to simulate and predict the outcomes of UI interactions. By focusing transition- based observations, WMA allows agents to simulate action results before committing, reducing unnecessary actions and increasing task efficiency. This predictive capability is particularly effective in long-horias tasks that require high accuracy, with WMA demonstrating strong performance on benchmarks such as WebArena Zhou et al. and Mind2Web Deng et al. (023)
Along similar lines, WebDreamer(Gu et al.
2024g introduces an innovative use of LLMs for model-based
planning in web navigation, as depicted in Figure 20
WebDreamer simulates and evaluates potential actions and their multi-step outcomes using LLMs before execution Yao et al. (2024), akin to a "dreamer" that envisions various scenarios. By preemptively assessing the potential value of different plans, WebDreamer selects and executes the plan with the highest expected value. This approach addresses critical challenges in web automation, such as safety concerns and the need for robust decision-makee in complex and dynamic environments, demoustrating superiority over reactive agents in benchmarks like VisualWebArres Koh et al. (024a) and Mind2Web-live Pan al. (022bb).
Beyond predictive modcling, integrating API interactions into web navigation offers enhanced fiexibility and efficiency. The Hybrid Agent (Song et al. (2024b) combines web browsing and API interactions, dynamically switching between methods based on task requirements. By utilizing API calls for structured data interaction, the Hybrid Agent reduces the time and complexity involved in traditional web navigation, achieving higher accuracy and efficiency in task performance. This hybrid architecture underscores the benefits of integrating both structured APT data and human-like browsing capabilities in AT agent systems.
Addressing the challenges of complex web structures and cross-domain interactions, Auto WettWe Lai et al. 2024) offers an efficient solution by simplifying HTML to focus key webpage components, thereby improving task accuracy. Using reinforcement learning and rejection sampling for fine-tuning, AutoWebGT.M excels in complex navigation tasks on both English and Chinese sites. Its bilingual dataset and structured action-perception modules make it practical for cross-domain web interactions, emphasizing the importance of efficient. handling in diverse web lasks.
ECLAIR Wornow et al. (2024) represents a pioneering application that replaces traditional RPA with a foundation model-powered GUI agent for enterprise automation. Unlike conventional RPA, which relies manually programmed rules and rigid scripts, ECLAIR dynamically learns workflows from video demonstra- tions and textual SOPs (Standard Operating Procedures), significantly reducing setup time and improving adaptability. It operates on enterprise web applications, leveraging GPT-4V and CogAgent CogAgent Team (2024) to perceive GUI elements, plan actions, and execute workflows, and validate automatically. By eliminating the high maintenance costs and execution brittleness of RPA, ECLAIR introduces a more flexible and scalable approach to GUI automation. We show a comparison of such agent- based- vs. RPA automation in Figure 21 This work establishes an important foundation for L.M.-poeree GUT automation, demonstrating how multimodal foundation models can bridge the gap between process mining, RPA, and fully autonomous enterprisc workflows.
Summary Recent advances in web GUI agents have significantly broadened the design space of intelligent web automation systems. Across the surveyed frameworks, we observe three dominant trends: (i) Multimodal grounding, exemplified by SeeAct and Web WoyVggg,,, boosts success rates on dynamic websites by integrating screenshots, HTML, and textual cues; (ii) Predictive planning and simulation, as employed by WMA and WebDreamer, reduces action errors and improves long-horizer reasoning by modcling environment dynamics before execution; and (iii) Search-based and hybrid control, as seen in LASER, Search-Agent, and WebPilot,
42

---

Published in Transactions on Machine Learning Research (06/2025)
enables robust decision-making via MCTS state-based planning, achieving up to 39% improvement in complex web benchmarks like WebArena and Amazon.
Overall, these frameworks collectively demonstrate how combining multimodal perception, planning, and hybrid action strategies leads to marked gains in success rates (up to 51.1%), adaptability, and robustness in real-world web environments. We envision future research further exploring generalization across unseen domains, long-horizon task stability, and the integration of memory and self-correction mechanisms.
5.2 Mobile GUI Agents
The evolution of mobile GUI agents has been marked by significant advancements, leveraging multimodal models, complex architectures, and adaptive planning to address the unique challenges of mobile environments. These agents have progressed from basic interaction capabilities to sophisticated systems capable of dynamic, context-aware operations across diverse mobile applications. We first provide an overview of mobile GUL
agent frameworks in Tables 22
23
24
25
and 26
Wang et al., Wang et al. (2033 pioneer the use of L.MSs to enable conversational interaction with mobile Uls, establishing onc of the carliest foundations for mobile GUT agents. Their approach involves directly prompting foundation models such as PaLM using structured representations of Android view hierarchies, which are transformed into HTML-like text to better align with the LLM's training distribution. The authors define and evaluate four core tasks, including Screen Summarization, Screen QA, Screen Question Generation, and instruction-to-U Mapping demonstrating that strong performance can be achieved with as few as two prompt examples per lask. Emphasizing practicality and accessibility, the work enables rapid protolyping without model fine-tuning, and stands out as a seminal effort in prompt-based evaluation of -LWe-eeeered GUI agents for mobile applications.
Early efforts focused enabling human-like GUT interactions without requiring backend system access. One such pioneering framework is AppAgent Zhang et al. (2023a), which utilizes GPT-4V's multimodal capabilities to comprehend and respond to both visual and textual information. By performing actions like tapping and swiping using real-time screenshots and structured XML data, AppAgent can interact. directly with the GUT across a variety of applications, from social media to complex image editing. Its unique approach of learning through autonomous exploration and observing human demonstrations allows for rapid adaptability to new apps, highlighting the effectiveness of multimodal capabilities in mobile agents.
Building upon this foundation, AppAgent-VZ V2 Li et al. 2024gh advances the framework by enhancing visual recognition and incorporating structured data parsing. This enables precise, context-aware interactions and the ability to perform complex, multi-step operations across different applications. AppAgent- V2 also introduces safety checks to handle sensitive data and supports cross-app tasks by tracking and adapting to real-time interactions. This progression underscores the importance of advanced visual recognition and structured data processing in improving lask precision and safety in real-time mobile environments.
Parallel to these developments, vision-centric approaches emerged to further enhance mobile task automation without relying app-specific data. For instance, Mobile-Agent Wang et al. (002cc) leverages OCR, CLIP Radford et al. 2021h, ) and Grounding DINO Liu et al. 2023a) for visual perception. By using screenshots and visual tools, Mobile- Agent performs operations ranging from app navigation to complex multitasking, following instructions iteratively and adjusting for errors through a self-reflcctive mechanism. This vision-basee based method
positions Ageit as a versatile and adaptable assistant for mobile tasks.
To address challenges in long-soquence navigation and complex, multi-app scenarios, Mobilo- Dobile-Agent-v2 Wang et al. (2024d) introduces a multi-agent architecture that separates planning, decision-making and reflection. By distributing responsibilities among three agents, this framework optimizes task progress tracking, retains memory of task-relevant information, and performs corrective actions when errors occur. Integrated with advanced visual perception tools like Grounding DINO Liu et al. 2023a)) and Qwen-VL-IntA4 VL-Int4 [ai et al. (023b)) Mobile-A Agget--22 showcases significant improvements in task completion rates on both Android and Harmony OS, highlighting the potential of multi-agent systems for handling complex mobile tasks.
In addition to vision-centric methods, some frameworks focus on translating GUI states into language enable L.LM-based action planning. VisionTasker (Song et al. 2024ch combines vision-based UT interpretation
43

---

Published in Transactions on Machine Learning Research (06/2025)
with sequential LLM task planning by processing mobile UI screenshots into structured natural language. Supported by YOT.O-v8 Reis et al. (2023) and PaddleOCR 29 for widget detection, Vision Tasker allows the agent to automate complex tasks across unfamiliar apps, demonstrating higher accuracy than human operators on certain tasks. This two-stage design illustrates a versatile and adaptable framework, setting a strong precedent mobile automation.
Similarly, Dro Wen al. (222cc) showcases an innovative approach by converting GUI states into natural language prompts, enabling LLMs to autonomously decide on action sequences. By interpreting the GUI structure and translating it into language that GPT models can understand, DroidBot-GPT gencralizes across various apps without requiring app-specific modifications. This adaptability underscores the transformative role of LLMs in handling complex, multi-step tasks with minimal custom data.
To enhance action prediction and context awareness, advanced frameworks integrate perception and action systems within a multimodal LLM. CoCo-Agent Ma et al. (2024d) exemplifics this by processing GUI elements like icons and layouts through its Comprehensive Event Perception and Comprehensive Action Planning modules. By decomposing actions into manageable steps and leveraging high-quality data from benchmarks like Android in the Wild (AITW) Rawles et al. (002)) and META-GUI [unn et al. (2022), CoCo-Agent demonstrates its ability to automate mobile tasks reliably across varied smartphone applications.
Further advancing this integration, CoAT Zhang et al. (202gg introduces a chain-dfaaction-thuugh process to enhance action prediction and context awareness. Utilizing sophisticated models such as GPT-4V and set-of-mark tagging, CoAT addresses the limitations of traditional coordinate-basee action recognition. By leveraging the Indroid-nn-The-Zonn (ATT7) dataset it builds, CoAT provides deep context awareness and improves both action prediction accuracy and task complction rates, highlighting its potential for accessibility and user convenience on Android platforms.
Addressing the need for efficient handling of multi-step tasks with lower computational costs, AutoDroid Wen ct al. (024a) combines LLM-based comprehension with app-specific knowledge. Using an HTML-style GUI representation and a memory-based approach, AutoDroid reduces dependency on extensive LLM queries. Its hybrid architecture of cloud and on-device models enhances responsiveness and accessibility, making AutoDroid a practical solution for diverse mobile tasks. AutoDroid-V22 Wen et al. (2024b) enhances its predecessor AutoDroid, by utilizing on-device language models to generate and execute multi-step scripts for user task automation. By transforming dynamic and complex GUT elements of mobile apps into structured app documents, it achieves efficient and accurate automation without depending on cloud-based resources. The acript-based approach reduces computational overhead by minimizing qucry frequency, thereby improving task efficiency and addressing the limitations of stepwise agents. This ad advancement enables privacy-preservigg and scalable task automation on mobile platforms.
MobileGPT Lee et al. (2024c) automates tasks on Android devices using a human-like app memory system that emulates the cognitive process of task decomposition Explore, Select, Derive, and Recall. This approach results in highly efficient and accurate task automation. Its hicrarchical memory structure supports modular, reusable, and adaptable tasks and sub-tasks across diverse contexts. MobileGPT demonstrates superior performance over state-of-the-art systems in task success rates, cost efficiency, and adaptability, highlighting its potential for advancing mobile task automation.
In a more advanced distributed setting, FedMobile Wang et al. (2025c) employs a federated learning framework to train mobile automation agents using self-sourced data from users' phone interactions. It addresses the high cost and privacy concerns associated with traditional human-annotated datasets by introducing Auto-Annotation which leverages vision-language models (VLMs) to infer user intentions from screenshots and actions. The system enables decentralized training through federated learning while preserving user privacy, and its adaptive aggregation method enhances model performance under non-llD data conditions. Experimental results on several mobile benchmarks demonstrate that FedMobileAgent achieves performance comparable human-annotate models at fraction of the cost.
Summary In summary, mobile GUI agents have evolved significantly, progressing from single-agent systems to complex, multi-agent frameworks capable of dynamic, context-aware operations. These innovations
28 httttp://gghubb.cc//aaeeeeeeeeee hwtpa//cc.ccccccceaadaaaaaaaaaaa
44

---

Published in Transactions on Machine Learning Research (06/2025)
HostAgent
User Request
Sub-tasks
Assignment
AppAgent1
AppAgentz
AppAgent
AppAgent
P
X
W
PowerPoint
Edge Browser
Excel
Word
Figure 22: The multi-agent architecture employed in UFO Zhang et al. (024a) Figure adapted from the original paper.
demonstrate that sophisticated architectures, multimodal processing, and advanced planning strategies are essential in handling the diverse challenges of mobile environments. Notably, frameworks like AppAgent- V2 and Mobile-Agent-v2 emphasize strong real-time perception and multi-agent collaboration enable robust. cross-app operations, achioving notable improvements task completion accuracy and stability. Systems such TasionTaskkr and DroidBot-GPT further highlight the elfectiveness of translating GUI states into structured language to facilitate generalizable and instruction-follwwee following behavior without app-specific tuning. Mcanwhile, solutions like CoCo-Agent and CoAT integrate high-quality benchmark-driven training and modular perception-action pipelines, leading to significant gains in action prediction accuracy and reliability across unseen mobile apps. On the efficiency front, AutoDroid-V2 and MobileGPT push the boundary by incorporating on-device models and reusable memory structures, demonstrating improvements not only in success rates but also in latency and query affeiency-critiaal for practical deployment. Collectively, these frameworks mark substantial progress in the field, revealing how combining vision-language reasoning, hierarchical planning, and efficient execution mechanisms leads to higher task generalization, reduced computational overhead, and improved real-world applicability of mobile GUI agents.
5.3 Computer GUI Agents
Computer GUT agents have evolved to offer complex automation capabilities across diverse operating systems, addressing challenges such as cross-application interaction, task generalization, and high-level task planning, They have led to the development of sophisticated frameworks capable of handling complex tasks across desktop environments. These agents have evolved from simple automation tools intelligent systems that leverage multimodal inputs, advanced architectures, and adaptive learning to perform multi-application tasks with high efficiency and adaptability, We provide an overview of computer GUI agent frameworks in Table 28 and
30
One significant development in this area is the introduction of multi-agent architectures that enhance task management and execution. For instance, the Ul-Focused Agent, UFO Zhang et al. (2024a) represents a pioneering framework specifically designed for the Windows operating system. UFO redefines Ul-focused automation through its advanced dual-agent architecture, leveraging GPT- Vision to interpret GUI elements and exccute actions autonomously across multiple applications. The framework comprises a Host.Agett, responsible for global planning, lask decomposition, and application selection, and an AppAgent, tasked with executing assigned subtasks within individual applications, as illustrated in Figure 22 This centralized structure enables UFO to manage complex, multi-application workflows such as aggregating information and generating reports. Similar architectural approach has also been adopted by other GUI agent frameworks AgentSeaff AT (022)); Zhu et al. (2024b); Zhang et al. (2024e). By incorporating safeguards and customizable
45

---

Published in Transactions on Machine Learning Research (06/2025)
Traditional
(b) AgentOS
Deep OS Integration
Shalloo OS
Screenshot
Integration
GUI
Screenshot
GUI actions
Atr vigg
Knowledge
Piiturre in Picture UX
Locking
Figure 23: The comparison of traditional CUAs and the Desktop AgentOS UFO2. Figure adapted from the original paper.
actions, UFO ensures efficiency and security when handling intricate commands, positioning itself cutting-edge assistant for Windows OS. Its architecture, exemplifies dynamic adaptability and robust task- solving capabilities across diverse applications, demonstrating the potential of multi-agent systems in desktop automation.
UFO2 Zhang al. (2025b), the successor UFO, elevates GUI automation from a vision-only prototype deeply integrated, Windows-native AgentOS (Figure 23 It coordinates tasks through a centralized HostAgent, which delegates subtasks to application-specillizee AppAgents. hybrid perception pipeline that fuses Windows UI Automation (UIA) metadata with OmniParser-v22 visual grounding delivers robust control identification even for custom widgets. Via unified GUI-API action layer, AppAgents preferentially invoke high-level application APIs and fall back lo pixel-level clicks only when necessary, cutting both latency and brittleness. picture-in-picture virtual desktop cleanly isolates agent execution from the user's main session, enabling non-intrusive multitasking. Runtime performance is further boosted by retrieval-augnnentel help documents and execution logs, coupled with speculative multi-action planning that executes several steps per single LLM invocation. Tested 20- real Windows applications, UFQ2 exceeds Operator OpenAT (2025b) and other CUAs by more than 10 percentage points in succoss rate while halving LLM calls. Because the framework is model-agnostic, swapping GPT-4o for stronger LLM such as o1 yields additional gains without code changes.
Building upon the theme of adaptability and generalist capabilities, Cradle Tan et al. 2024a pushes the boundaries of general computer control by utilizing VLMs for interacting with various soflware, ranging from games to professional applications, without the need for AP1 access. Cradle employs GPT-4o to interpret screen inputs and perform low-level actions, making it versatile across different types of software environments. Its six-module structure, covering functions such information gathering and self-reftection, enables the agent to execute tasks, reason about actions, and utilize past interactions to inform future decisions. Cradle's capacity to function in dynamic environments, including complex software, marks it as significant step toward creating generalist agents with broad applicability across desktop environments.
Extending the capabilities of computer GUT agents to multiple operating systems, OS-Copilot [w al. (2024e) introduces general-purpose framework designed operate across Linux and macOS systems. Its notable feature, FRIDAY, showcases the potential of self-directed learning by adapting to various applications and performing tasks without explicit training for each app. Unlike application-spedi integrates APIs, keyboard and mouse controls, and command-line operations, creating flexible platform that can aulonomously generale and refine tools as it interacts with new applications. OS-Copilot's 's ability to generalize across unseen applications, validated by its performance on the GALA benchmark, provides a foundational model for OS-level agents capable of evolving complex environments. This demonstrates promising directions for creating adaptable digital assistants that can handle diverse desktop environments and complex task requirements.
agents, FRIDAY
46

---

Published in Transactions on Machine Learning Research (06/2025)
In the emerging field of LLM-powered GUI agents for desktop environments, Programming with Pixels (PwP) Aggarwal & Welleck (2025) introduces a compelling alternative to traditional tool-based software engineering agents, as illustrated in Figure X. Rather than relying on predefined API calls, PwP enables agents to interact directly with an IDE using visual perception, keyboard inputs, and mouse clicks, mimicking the way human developers operate within an IDE. This approach allows for generalization beyond predefined APIs, providing highly expressive environment where agents can exccute a wide range of software engincering tasks, including debugging, UI generation, and code editing. Evaluations conducted on PwP-Benchh demonstrate that computer-use agents, despite lacking direct access to structured APIs, can match even surpass traditional tool-based approaches in certain scenarios.
Summary In summary, computer GUI agents have evolved significantly, progressing from single-task automation tools to advanced multi-agent systems capable of performing complex, multi-application tasks and learning from interactions. Frameworks such as UFO, Cradle, and OS-Copilot exemplify different design philosophies, each addressing core challenges like cross-application control, adaptability to unseen environmenss, and efficient long-horizon plamning. UFO showcases a highly structured dual-agent architecture that achieves robust task decomposition and accurate subtask execution within the Windows ecosystem, demonstrating superior performance in cross-app workflows like document synthesis and report generation. In contrast, Cradle focuses on generality and reasoning by leveraging a six-module pipeline with self-reflection and memory to handle diverse software types, including games and creative tools highhigghhhhhiiggggg strong adaptability without relying on application-speeific APIs. Mcanwhile, OS-Copilot pushes the boundarics of platform generalization, achieving cross-OS compatibility (Linux/maCOS) via a hybrid interaction approach combining CLI, GUT, and API-level control, with demonstrated success the GATA benchmark for unseen applications. herether, these systems illustrate clear trend: the most effective computer GUI agents integrate multimodal perception, modular reasoning, and dynamic learning mechanisms to balance generality and performance, setting the foundation for future AgentOS frameworks that are scalable, OS-agnostic, and capable of long-term autonomous operation in real-world desktop environments.
5.4 Cross-Platform GUI Agents
Cross-platform GUI agents have emerged as versatile solutions capable of interacting with various environments, from desktop and mobile platforms to more complex systems. These frameworks prioritize adaptability and efficiency, leveraging both lightweight models and multi-agent architectures to enhance cross- plossfrla operability. In this subsection, we first We overview cross-platform GUT agent frameworks in Table 31 and
32 then explore key frameworks that exemplify the advancements in cross-platform GUI automation.
A significant stride in this domain is represented by AutoGLM Liu et al. 2022hh) ; which bridges the gap between web browsing and Android control by integrating large multimodal models for scamless GUI interactions across platforms. AutoGLM introduces an Intermediate Interface Design that separates planning and grounding tasks, improving dynamic decision-making and adaptability. By employing self-evolving online curriculum with reinforcement learning, the agent learns incrementally from real-world feedback and can recover from errors. This adaptability and robustness make AutoGLM ideal for real-world deployment in diverse user applications, setting a new standard in cross-platform automation and offering promising directions for future research in foundation agents.
While some frameworks focus on integrating advanced models for cross-platform interactions, others emphasize efficiency and accessibility. Tiny Pawlowski et al. (2024) addresses the need for light solutions by focusing on single-turn interactions within GUIs. Utilizing the Plorenco---Bass Vision-Language Model, TinyClick executes lasks based on user commands and screenshots with only 0.27 billion parameters. Despite its compact size, it achieves high accuracy 1 on Screenspot Cheng et al. (2024a) and 58.3% on OmniAct Kapoor et al. (2024) -utperforming larger multimodal models like GPT-4V while maintaining efficiency. Its multi-task training and MLLM-based data augmentation enable precise UI element localization, making it suitable for low-resource environments and addressing lateney and resource constraints in UT grounding and action execution.
In addition to lightweight models, multi-agent architectures play a crucial role in enhancing cross- pross-platfomm GUT interactions. OSCAR Wang & Liu (2024) exemplifies this approach by introducing generalist GUT
47

---

Published in Transactions on Machine Learning Research (06/2025)
agent capable of antonomously navigating and controlling both desktop and mobile applications. By utilizing state machine architecture, OSCAR dynamically handles errors and adjusts its actions based on real-time feedback, making it suitable for automating complex workflows guided by natural language. The integration of standardized OS controls, such as keyboard and mouse inputs, allows OSCAR to interact with applications in a generalized manner, improving productivily across diverse GUT envirouments. Its open-source design promotes broad adoption and seamless integration, offering a versatile tool for cross-platform task automation and productivity enhancement.
Expanding on the concept of multi-agent systems, AgentStore Jia et al. 2024) introduces flexible and scalable framework for integrating heterogeneous agents to automate tasks across operating systems. The key feature of AgentStore is the MetaAgent, which uses the innovative ToketToken strategy to dynamically manage a growing number of specialized agents. By enabling dynamic agent enrollment, the framework fosters adaptability and scalability, allowing both specialized and generalist capabilities to coexist. This multi-agent architecture supports diverse platforms, including desktop and mobile environments, leveraging multimodal perceptions such as GUT structures and system states. AgentStore's contributions highlight the importance of combining specialization with generalist capabilities to overcome the limitations of previous systems.
Further advancing cross-plat form GUT interaction, MMAC-Copilot Song et al. 2024d employs a multi- agent, multimodal approach to handle tasks across 3D gamiing office, and mobile applications without relying on APIs. By utilizing specialized agents like Planner, Viewer, and Programmerr MMAC-Copilot collaborates to adapt to the complexities of visually rich environments. Using GPT-4V for visual recognition and OCR for text analysis, it achieves high task completion rates in visually complex environments. The framework's ' integration with VIBench,, a benchmark for non-APl applications, underscores its real-world relevance and adaptability. CMMA--opillt's robust foundation for dynamic interaction across platforms extends applications to industries like gaming, healthcare, and productivity.
AGUVIS Xu et al. (022k) leverages a pure vision approach to automate GUI interactions, overcoming limitations of lext-based systems like HTML or accessibility trees. Its platform-aguostic design supports web, desktop, and mobile applications while reducing inference costs. AGUVIS employs a two-stage training process: the first focuses on GUI grounding, and the second integrates planning and rcasoning within a unified model. This approach delivers stale-of-the-arl performance in both offline and online scenarios, streamlining decision- decision-making and execution.
Agashe et al. (2025) builds upon its predecessor, Agent S Agashe et al. (2024), by introducing a hierarchical and compositional framework for GUT agents that integrates generalist models with specialized grounding modules. Departing from monolithic architectures, it employs Mixture of Grounding (MoG) strategy delegate fine-grained grounding tasks to expert modules, and adopts Proactive Hierarchical Planning (PHP) to dynamically revise action plans based on evolving observations. Relying solely on GUI screenshots, Agent S2 generalizes effectively across Ubuntu, Windows, and Android platforms. It demonstrates strong scalability and consistently outperforms larger monolithic models by strategically distributing cognitive responsibilities. The design of Agent S2 underscores the ad advantages of modular architectures for handling long- horizon, high-fidelity GUI interactions.
Agent S2
Summary In summary, cross-platform GUI agents have demonstrated significant progress in bridging heterogeneous environments through varied design choices- choices-ranging from lightweight efficiency to multi- agent scalability and robust multimodal perception. Frameworks such as AutoGLM and MMAC- Copiiot highlight the power of large multimodal models and currieulum-based reinforcement learning to achieve high task success in diverse, visually rich environments without reliance on APIs These systems exhibit strong adaptability, error recovery, and high task completion rates across domains such mobile control, 3D gaming, and office automation. In contrast, models like TinyClick illustrate the fcasibility of achieving competitive performance- surpassing even GPT-4V in some benchmarks using highly compact architectures, thus addressing latency and resource constraints critical for real-time or on-device applications. Meanwhile, OSCAR and AgontStore exemplify the growing role of modular multi-agent systems, combining state tracking, error correction, and flexible enrollment of specialized agents to achieve scalability and platform coverage. These architectures enable dynamic agent composition, facilitating generalist. behavior without sacrificing
48

---

Published in Transactions on Machine Learning Research (06/2025)
domain-specific precision. Notably, AGUVIS introduces a vision-only paradigm, unifying GUI grounding, planning, and reasoning into a platform-agnostie model that generalizes across web, mobile, and desktop systems. Collectively, these frameworks reflect a clear trend toward combining scalability, multimodal grounding, and architectural efficiency to meet the demands of cross-platform GUI automation increasingly complex, real-world settings.
6 Data for Optimizing LLM-Pooerrd GUI Agents
In the previous section, we explored general frameworks for I.M.-powered GUT agents, most of which rely on foundational LLMs such as GPT-4V and GPT--o. However, to elevate these agents' performance and efficiency, optimizing their "brain", the underlying model is crucial. Achieving this often involves fine-tuning foundational models using large-scale, diverse, and high-quality contextual GUI datasets Li et al. (022ee), which are specifically curated to enable these models to excel in GUI-specific tasks. Collecting such datasets, particularly those rich in GUI screenshots, metadata, and interactions, necessitates an elaborate process of data acquisition, filtering, and preprocessing, each requiring substantial effort and resources Chen et al. (2024b).
As GUI agents continue to gain traction, researchers have focused on assembling datasets that represent. broad spectrum of platforms and capture the diverse intricacies of GUI environments. These datasets are pivotal in training models that can generalize effectively, thanks to their coverage of varied interfaces, workflows, and user interactions. To ensure comprchensive representation, innovative methodologies have been employed to collect and structure these data assets. In the sections that follow, we detail an end-to-end pipeline for data collection and processing tailored to training GUL-specific LLMs. We also examine significant datasets from various platforms, providing insights into their unique features, the methodologies used in their creation, and their potential applications in advancing the field of LLM-pooered GUI agents.
6.1 Data Collection
Data is pivotal in training a purpose-built GUT agent, yet gathering it requires substantial time and effort. due the task's complexity and the varied environments involved.
6.1.1
Data Composition and Sources
The essential data components for GUI agent training closely align with the agent's perception and inference
requirements discussed in Sections 4.2.2 and 4.4 At a high level, this data comprises:
User Instructions: These provide the lask's overarching goal, purpose, and specific details, typically in natural language, offering clear target for the agent to accomplish, e.g., "change the font size of all text to 12"
Environment Perception: This typically includes GUI screenshots, often with various visual augmentations, as well as optional supplementary data like widget trees and UI element properties to enrich the context. This should encompass both the static assessment of environment states (Sec- tion 4.2.2 and the dynamic environment feedback that captures post-action changes (Section 4.2.3 thereby providing sufficient contextual information.
3. Task Trajectory: This contains the critical action sequence required to accomplish the task, along with supplementary information, such as the agent's 's plan. A trajectory usually involves multiple steps and actions to navigate through the task.
While user instructions and environmental perception serve as the model's input, the expected model output is the task trajectory. This trajectory's 's action sequence is then grounded within the environment. to complete the task.
For user instructions, it is crucial to ensure that they are realistic and reflective of actual user scenarios. Instructions be sourced in several ways: (i) directly from human designers, who can provide insights
49

---

Published in Transactions on Machine Learning Research (06/2025)
Existing
Public
State-Action Data Collection
Dataset
Resources
LLM
Human
Instantiaion Task: Chonge the font size the third Word decumeett draft.dock 20.
thidd pangrach
paragrppht
for edting.
dropptooon the Word toolhrr Actions (oaooToaaoooon Siee")
select "ap* as forr Action (l fotttot
font she Action Ver Fertyonttseeeeed 20)
Actione SwwecTethhaaaappiiaa
Betten Inset...
[Bostor [nsen...] .
(Botton Insert, ..] (Betton Chart, oo UI Properties
Stecton- inaet.... (Bocton: Chart, UI Properties
Hotton Charl, of]
(Better -hart..)
Widger UI Proportes
UI Properties
widget Tree
Widget Tree
Yas
Prototypical Instructions
Task Instantiation Data filtering Deta Augmentation
Screenshot
Screenshot
Screenshot
Screenshot
Task Execution
(Evaluation)
GUI Environments
Human
GUI Agant
Human
Figure 24: A complete pipcline for data collection for training GUI agent model.
based on real-world applications; (ii) extracted from existing, relevant datasets if suitable data is available; (iii) sourcing from public materials, such as websites, application help documentation, and other publicly available resources; and (iv) generated by LLMs, which can simulate a broad range of user requests across different contexts. Additionally, LLMs can be employed for data augmentation Ding et al.) (2024a). increasing both the quality and diversity of instructions derived from the original data.
For gathering environment perception data, various toolkits such as those discussed in Section 1.2.2 can be used to capture the required GUI data, This can be done within an environment emulator (e.g., Android Studio Famulaton 30 Selenium WebDriven 31 Windows Sandbox [2) or by directly interfacing with a real environment to capture the state of GUI elements, including screenshots, widget trees, and other motadata essential for the agent's 's operation.
Collecting task trajectories, which represent the agent's action sequence to complete a task, is often the most challenging aspect. Task trajectories need to be accurate, exccutable, and well-validated. Collection methods include (i) using programmalically generated scripts, which define action sequences for predefined tasks, providing a highly controlled data source; (ii) employing human annotators, who complete tasks in a crowdsourced manner with each step recorded, allowing for rich, authentic action data; and (tit) leveraging model or agent bootstrapping Tan et al. (2024b), where an existing LLM or GUI agent attempts to complete the task and logs its actions, though this method may require additional validation due to potential inaccuracies. All these methods demand considerable effort, reflecting the complexities of gathering reliable, task-accurate data for training GUI agents.
6.1.2 Collection Pipeline
Figure 24 presents a complete pipeline for data collection aimed at training GUI agent model. The process begins with gathering initial user instructions, which may come from various aforement ioned sources. These instructions are typically prototypical, not yet tailored or grounded to a specific environment Liu et al. (2024i). For instance, an instruction like "how to change the font size?" from a general website lacks specificity and doesn't 't align with the concrete requests a user might make within a particular application. To address this, instantiation step is required Liu et al.) (024i)), where instructions are contextualized within specific environment, making them more actionable. For example, the instruction might be refined to "Change the font frcx size of the third paragraph in a Word document of draft. docx to 20.", giving it clear, environment-speciff goal. This instantiation process can be conducted either manually by humans or programmatically with an LLM.
htttps://ww..eeeeniuu
32 vindous haap:a/////iibb---aaaabaaeeeeerr
50

---

Published in Transactions on Machine Learning Research (06/2025)
Following instantiation, instructions may undergo filtering step to remove low-quality data, ensuring only relevant and actionable instructions remain. Additionally, data augmentation techniques can be applied to expand and diversify the dataset, improving robustness. Both of these processes can involve human validation or leverage LLMs for efficiency.
Once instruction refinement is complete, task trajectories and environment perceptions are collected simulta- neously. As actions are performed within the environment, each step is logged, providing a record of the environment's 's state and the specific actions taken. After full task trajectory is recorded, phase is necessary to identify and remove any failed inaccurate sequences, preserving the quality of the dataset. By iterating this pipeline, a high-quality datasct of GUI agent data can be compiled, which is crucial for training optimized models.
evaluation
Data collection and curation for LLM-powered GUI agents is an intensive process, often requiring substantial human involvement, particularly for gencrating accurate action sequenees and annotations. While carly datasets were limited in scale and task diversity, recent advancements have led to large-scale, multi-platffrm datasets that support more complex and realistic GUI interactions. Key insights from these developments include:
1. Scale and Diversity:
High-quality, large-scale data is essential for training robust GUL agents capable of handling diverse UI states and tasks. Datascts like MobileViews Gao et al. (2024c) and ScreenAl Baechler et al. (2024) illustrate the importance of vast and varied data to accommodate the dynamic nature of mobile and desktop applications, enhancing the agent's resilience across different. environments.
2. Cross-Platfore Flexibility: Cross-platform datasets such as VisualAgeetBBnnch Liu et al. (2024) and GUI-World Chen et al. (2024c) underscore the value of generalizability, enabling agents to perform consistently across mobile, web, and desktop environments. This cross-platform adaptability crucial step towards creating one-stop solutions where a single GUI agent can operate seamlessly across multiple platforms.
Automated Data Collection: Al-driven data collection tools, as exemplified by OmniParser Lu ct al. (2024d) and MobileViews Gao et al. (2024e), showcase the potential to significantly reduce manual elforts and accelerate scalable dalaset creation. By automating the annotation process, these tools pave the way for more efficient data pipelines, moving towards a future where Al supports AI by expediting data gathering and labeling for complex GUI interactions.
4. Unified Data Formats and Protocols: xLAM's unified data format, is an essential innovation that improves compatibility across diverse platforms Zhang et al. (2024d), addressing a significant bottleneck in cross-platform GUI agent development. Establishing standardized protocols or action spaces for data collection, particularly given the varied data formats, action spaces, and environment. representations across platforms, will be vital in furthering agent generalization and consistency.
In the following sections, we review existing GUT agent datasets across various platforms, offering insights into current practices and potential areas for improvement. Before delving into the details, we first present a taxonomy of LLM-powered GUI agent datasets, categorized by target platform, shown in Table 12
6.2 Web Agent Data
Web-based GUI agents demand datasets that capture the intricate complexity and diversity of real-world web interactions. These datasets often encompass varied website structures, including DOM trees and HTML content, as well as multi-step task annotations that reflect realistic user navigation and interaction patterns. Developing agents that can generalize across different websites and perform complex tasks requires comprehensive datasets that provide rich contextual information. We provide an overview of wel-based GUT agents dataset in Table 33 and 34
Building upon this need several significant datasets have been developed to advance web-based GUI agents. Unlike traditional datasets focusing on narrow, predefined tasks, Mind2Web Deng et al. (203)) represents
51

---

Published in Transactions on Machine Learning Research (06/2025)
Table 12: Taxonomy of LLweeeee GUI agent datasets by target platform.
Platform
References
Web
Deng et al.
(2023
g
Chen et al. (2024e);
Lu et al.
2024cc
Pan et al.
(024b);
Xu et al.
(2024j);
Trabucco et al
(2025
Mobile
Li et al.
)
Zhang et al.
[0224g]
You et al.
(2025): Wu et al.
(024c):
Meng et al. 2024
: Deka et al.
(2017):
Burns et al
(0022)
; Sun et al.
(2022) )
Rawles al.
(2023
Lu al.
(2024b
Chai et al.
(024);
Chen et al.
(2024
Wang et al.
2024f);
Xu et al.
202410;
Gao et al.
(2024e] Sun et al.
2025b)
Computer
Niu ct al.
2024h;
Wang et al.
(2024h) ; Xu ct al.
2025)
Cross-Platform
Lu et al.
(0224d)
Chen et al.
(2024i); Gon et al.
(2024); Sun et al
(024b); :);
Chawla la et al.
(2024); Chen et al. (2024c];
Zhang el al.
(2024d
Shen el al.
[2024a]: Liu et al. (2024) Baechler et al. (2024): Chaimalas ct al. 2025
Cheng
et al.
(2025b]
significant step forward by emphasizing open-ended task descriptions, pushing agents to interpret high-level goals independently. It offers over 2,350 human-annotated tasks across 137 diverse websites, capturing complex interaction patterns and sequences typical in web navigation. This setup aids in evaluating agents' generalization across unseen domains and serves as a benchmark for language grounding in web-based GUIs, enhancing adaptability for real-world applications.
Similarly, Web Chen et al. (2024e) expands on web GUT tasks by combining navigation with question- answering. It provides agents with text-based queries that guide them to locate relevant web pages and extract information. By leveraging both HTML and visual content from websites, WebVLN aligns with real-world challenges of web browsing. This dataset is particularly valuable for researchers aiming to develop agents capable of complex, human-like interactions in GUT-driven web spaces.
Morcover, WebLINX Lu et al. (2024c) focuses on conversational GUI agents, particularly emphasizing real-world web navigation through multi-turn dialogue. Featuring over 2,300 expert demonstrations across 155 real-world websites, WebLINX creates rich environment with DOM trees and screenshots for training and evaluating agents capable of dynamic, user-guided navigation tasks. This dataset promotes agent generalization across new sites and tasks, with comprehensive action and dialogue data that provide insights into enhancing agent responsiveness in realistic web-based scenarios.
MultiUI Liu et al. (2024e) is a large-scale dataset designed to enhance GUI agents' text-rich visual understanding. It. comprises 7.3 million multimodal instruction samples collected from 1 million websites, covering key web UI tasks such as element grounding, action prediction, and interaction modeling. Unlike traditional datascts that rely raw HTML, MultiUI utilizes structured accessibility trees to generate high-quality multimodal instructions. Models trained on MultiUT demonstrate substantial performance improvements, achieving a 48% gain on VisualWebBennc Liu et al. (2024f) and a 19.1% increase in element accuracy on Mind2Web Deng et al. (2023).
InSTA Trabucco et al. (2025) is an Internet-scale dataset for training GUT-based web agents, generated entirely through an automated LLM pipeline without human annotations. covers 150k diverse websites sourced from Common Crawl and includes rich web navigation tasks, trajectories in Playwright API calls, and evaluations using LLM-based judges. The dataset highlights strong generalization capabilities and data efficiency, significantly outperforming human-collected datascts like Mind2Web Deng et al. (2023) and WebLINX Lu et al. (2024c) in zero-shot and low-resource settings. InSTA represents a key advancement in scalable data curation for L.L.-powered GUT agents, offering unprecedented coverage across real-world web interfaces.
Collectively, these datasets represent essential resources that enable advancements in web agent capabilities, supporting the development of adaptable and intelligent agents for diverse web applications.
52

---

Published in Transactions on Machine Learning Research (06/2025)
6.3 Mobile Agent Data
Mobile platforms are critical for GUI agents due to the diverse range of apps and unique user interactions they involve. To develop agents that can effectively navigate and interact with mobile interfaces, datasets must offer a mix of single and multi-step tasks, focusing on natural language instructions, UI layouts, and user interactions. We first overview mobile GUI agents dataset in Tables 35
36 and 37
An early and foundational contribution in this domain is the Rico datasct Deka et al. 2017 which provides over 72,000 unique UI screens and 10,811 user interaction traces from more than 9,700 Android apps. Rico has been instrumental for tasks such as UT layout similarity, interaction modeling, and perceptual modeling, laying the groundwork for mobile interface understanding and GUI agent development.
Building upon the need for grounding natural language instructions to mobile UI actions, PIXELHELP et al. (2020a) introduces a dataset specifically designed for this purpose. Tt includes multi-step instructions, screenshots, and structured UI element data, enabling detailed analysis of how verbal instructions can be converted into mobile actions. This dataset has significant applications in accessibility and task automation, supporting agents that autonomously execute lasks based on verbal cues.
Further expanding the scope, the Android in the Wild (AITW) dataset Rawles et al. 20233 offers one of the most extensive collections of natural device interactions. Covering broad spectrum of Android applications and diverse UI states, AITW captures multi-step tasks emulating real-world device usage. Collected through interactions with Android emulators, it includes both scrcenshots and action sequences, making it ideal for developing GUI agents that navigate app interfaces without relying on app-specific APIs. Due to its scale and diversity, AITW has become a widely used standard in the field.
In addition, META-GUI Sun et al. (022) provides a unique dataset for mobile task-oriented dialogue systems by enabling direct interaction with mobile GUIs, bypassing the need for API-based controls. This approach allows agents to interact across various mobile applications using multi-turn dialogues and GUT traces, broadening their capabilities in real-world applications without custom API dependencies. The dataset's support for complex interactions and multi-turn dialogue scenarios makes it valuable for building robust conversational agents.
Recently, MobileViews Gao et al. 2024e) emerged as the largest mobile screen dataset to date, offering over 600,000 screenshot-vier hicrarchy pairs from 20,000 Android apps. Collected with an LLM-enhanced app traversal tool, it provides a high-fidelity resource for mobile GUT agents in tasks such as screen summarization, tappability prodiction, and UI component identification. Its scale and comprchensive coverage of screen states make MobileViews a key resource for advancing mobile GUI agent capabilities.
Collectively, mobile platforms currently boast the richest set of datasets due to their versatile tools, emulator support, and diverse use cases, reflecting the demand for high-quality, adaptive GUI agents in mobile applications.
6.4 Computer Agent Data
In contrast to mobile and web platforms, the desktop domain for GUI agents has relatively fewer dedicated datasets, despite its critical importance for applications like productivity tools and enterprise software. However, notable efforts have been made to support the development and evaluation of agents designed for complex, multi-step desktop tasks. We show related dataset for computer GUI agents in Table 38
A significant contribution in this area is ScreenAgent Niu et al. 2024), dedicated dataset and model designed to facilitate GUI control in Limux and Windows desktop environments. ScreenAgent provides a comprehensive pipeline that enables agents to perform multi-step task execution autonomously, encompassing planning, action, and reflection phases. By leveraging annotated screenshots and detailed action sequences, it allows for high precision UI element positioning and task completion, surpassing previous models accuracy. This dataset is invaluable for researchers aiming to advance GUI agent capabilities in the desktop domain, enhancing agents' decision-making accuracy and user interface interactions.
53

---

Published in Transactions on Machine Learning Research (06/2025)
The LAM Wang et al. 2024hh is specifically designed to train and evaluate Large Action Models (LAMs) for GUT environments, bridging natural language task understanding and action execution. It. comprises two core components: Task-Plan data, detailing user tasks with step- hep-by-step plans, and Task-Action data, translating these plans into exccutable GUI actions. Sourced from application documentation, WikiHow articles, and Bing search queries, the dataset is enriched and structured using GPT-4. Targeting the Windows OS, with a focus on automating tasks Microsoft Word, it includes 76,672 task-plan pairs and 2,688 task-action trajectories, making it one of the largest collections for GUI-based action learning. Data quality is ensured through a robust validation pipeline that combines LLM-based instantiation, GUI interaction testing, and manual review. Each entry is complemented with GUT screenshots and metadata, enabling models to learn both high-level task planning and low-level execution. The dataset's modular design supports fine-tuning for specific GUT lasks and serves as a replicable framework for building datasets in other environments, marking a significant contribution to advancing GUl-based automation.
Although the desktop domain has fewer datasets compared to mobile and web, efforts like ScreenAgent and LAMs highlight the growing interest and potential for developing sophisticated GUT agents for computer systems.
6.5
Agent Data
Cross-platform datasets play a pivotal role in developing versatile GUI agents that can operate seamlessly across mobile, computer, and web environments. Such datasets support generalizability and adaptability, enabling agents to handle varied interfaces and tasks in real-world applications. We provide an overview of related dataset for cross-platform GUI agents in Table 39 and Table
40
One significant contribution is ACreenAI Baechler et al. 20044 which extends the scope of data collection to include both mobile and desktop interfaces. Covering tasks such screen annotation, question-answerii and navigation, ScreenAI offers hundreds of millions of annotated samples. Its comprchensive scale and mixed-plat.orm coverage make it a robust. foundation for GUT agents that, need to manage complex layouts and interactions across diverse interfaces. By emphasizing element recognition and screen summarization, ScreenAl advances the development of multi-platform GUI agents capable of handling varied visual structures.
Building upon the need for evaluating visual foundation models across environments, VisualAgent Liu et al. (2024i) is groundbreaking cross-platform benchmark designed to assess GUT agents in both mobile and web settings. It emphasizes interaction-focusee tasks, using enviromments like Android Virtual Device and WebArena-Lite /hhou et al.] to evaluate and improve agent responses to GUI layouts and user interface actions. The dataset's 's innovative collection method, which combines program- bograa--based solvers and large multimodal model bootstrapping, facilitates robust training trajectorics that enhance adaptability and crror recovery in GUT agent lasks.
Furthermore, Gor- Chen et al. (2024c) spans multiple platforms, including desktop, mobile, and XR environments, with over 12,000 annotated videos. Designed to address the challenges of dynamic and sequential GUT tasks, GUT-World allows researchers to benchmark GUT agent capabilities across diverse interfaces. By providing detailed action sequences and QA pairs, it sets a high standard for evaluating agents in complex, real-world scenarios.
Additionally, XLAMZhang et al. (2024d) contributes significantly to actionable agent development by providing unified datasct format designed to support multi-turn interactions, reasoning, and function-calling tasks. Sourced from datasets like WebShor Yao et al. (2023), ToolBench Guo et al. (2024c), and AgentBoard Ma et al. (024a)), xLAM standardizes data. formats across diverse environments, addressing the common issue of inconsistent data structures that hinder agent training and cross-environment compatibility. By offering a consistent structure, xLAM enhances the adaptability and error detection capabilities of GUI agents, allowing for more seamless integration and performance across different applications.
OS- OS-Genesis Sun et al. 2224bb adopts reverse task synthesis approach for the Android and web platforms. It leverages GPT-40 to interactively explore the environment and generate instructions in a reverse manner. This process construets high-quality, diverse GUT trajectories without relying on human annotations
54

---

Published in Transactions on Machine Learning Research (06/2025)
predefined tasks. By eliminating these dependencies, OS-Geneshr achieves scalable and efficient training for GUT agents while significantly enhancing the diversity and quality of the generated data.
Collectively, these pross-platform datasets contribute to building multi-platform GUI agents, paving the way for agents that can scamlessly navigate and perform tasks across different interfaces, fostering more generalized and adaptable systems.
7 Models for Optimizing LLM-Powered GUI Agents
LLMs act as the "brain" of GUI agents, empowering them to interpret user intents, comprehend GUI screens, and exccute actions that directly impact their environments. While several existing foundation models are robust enough to serve this core, they can be further fine-tuned and optimized to evolve into Large Action Models (LAMs) specialized models tailored to improve the performance and efficiency of GUI agents. These LAMs bridge the gap between general-purpose capabilities and the specific demands of GUI-based interactions.
The exploration of LAMs for GUT agents has revealed several key insights that are shaping the future of intelligent interaction with graphical user interfaces:
1. Smaller Models for On-Device Inference: Many of the optimized LAMs are built from smaller inndatiional models, often ranging from 1 billion to 7 billion parameters. This reduction in model size enhances computational efficiency, making it feasible to deploy these models on resource-constraines devices such as mobile phones and edge devices. The ability to perform on-devico inference without relying on cloud services addresses privacy concerns and reduces latency, leading to more responsive user experience.
Enhanced GUI Comprehension Reduces Reliance on Structured Data: Models like VGA Meng al. 20244) and OmniParscr Lu et al. (2024d) emphasize the importance of visual grounding and image-centric fine-tuning to reduce dependency on structured UI metadata. By improving GUI comprehension directly from visual imputs, agents become more adaptable to different software environments, including those where structured data may be inaccessible or inconsistent.
3. Reinforcement Learning Bridges Static and Dynamic Environments:
The application of reinforcement learning in models like DigiRL [ai et al. (2024) demonstrates the effectiveness of bridging static training data with dynamic real-world environments. This approach allows agents to learn from interactions, recover from errors, and adapt to changes, enhancing their robustness and reliability in practical applications.
4. Unified Function-Calling Enhances Interoperability: Efforts to standardize data formats and funetion-calling mechanisms, seen in models like xLAM Zhang et al. (2024d), facilitate multi-turn interactions and reasoning across different platforms. This unification addresses compatibility issues and enhances the agent's ability to perform complex tasks involving multiple APIs and services.
Inference-Timee Computing and Reasoning Models: Recent work highlights the importance of inference-time computing, where models plan, reason, and decompose tasks the fly without architectural changes. Techniques such extended context windows and thain-hf-thought prompting (e.g., "ol-style" reasoning) enable more robust, Long-horizon preiiiinnmakingg UI-R1 Lu et al. (025b),, GUI-R1 Xia & Luo (2025) and InfiGUI-R1 Liu et al. (2025f) are pioneering efforts in this direction. There is also growing interest in rule-based rewards and cost functions that guide inference- time behavior, integrating explicit. heuristics to improve the stability, interpretability, and generalization of GUI agents.
In this section, we first introduce the foundation models that currently form the backbone of GUI agents, highlighting their strengths and limitations. We then delve into the concept of LAMs, discussing how these models are fine-tuned with GUI-specific datasets to enhance their adaptability, accuracy, and action-orientation in GUT environments. Through this exploration, we illustrate the progression from general-purpose L.LSs
55

---

Published in Transactions on Machine Learning Research (06/2025)
Table 13: Taxonomy of LLM-powered GUT agent models by target platform.
Platform
References
Foundation
Wang et al. 2024]) Abdin et al. (2024); OpenAl
(2025a);
Li et al.
(2023c)
;
Models
Bai et al.
(2023b); Team et al. (2023):
; Hurst et al.
[2024]
Anthropic
(2024
OpenAl
2023
Chen et al. (20241 11 Wang et al. 20224
You et al.
2023
Liu ct al.
2024b a Huang ct al.
(0025c)
Web
Furuta al
(2023): : Putta al.
(2024);
Fereidouni & Siddique (2024): Thil
et al.
(0024);
He al. (2024cb;
al.
(202))
Zhang al. [2025d]
Mobile
Chen ct al.
2024ff) Bai ct al. (2024); You et al.
(2025):
Wu ct al. (2024c): Meng
et al. [2024]
| Qian et al. p0 Chen et al.
(2024g)
; Chen & Li (2024a b
Nong et (024)) L. al. (2024l) Wu et al.
2025bb;
Papoudakis al.
2025
Bai et al. 22025a) Zheng al. (2025b);
Wang al.
2025t); Lu et al.
2025b
Zhang al.
(002dd Luo et al.
2025a
Computer
Niu et al.
[2024]: Yang et al.
2025bb
Wang et al. (024hh
Jin et al.
2025
Hong
et
al.
(2023);
Cheng et al.
(2024a)) Shen et al.
(2024a):
Zhang al.
[2023e]
Lu ct al.
(2024d]: Bacchler et al.
(2024] Zhang ct al.
(2024d]; Li et al.
2024j
Lin et al.
(2024c) : Wu et al. (2024f): Qin et al. (2025
Rahman et al.
2024 ; Yang et al.
(2025a) Huang et al. (0025c Xia. & Luo
(025) Liu et al.
20025
to purpose-built LAMs, laying the foundation for advanced, intelligent GUT agents. Before delving into the details, we first present a taxonomy LLM-ppoered GUI agent models, categorized by target platform, as shown in Table B
7.1 Foundation Models
Foundation models serve as the core of LLM-ppwered GUI agents, providing the essential capabilities for understanding and interacting with graphical user interfaces. Recent advancements both closo-source and open-source MLLMs have significantly enhanced the potential of GUI agents, offering improvements in efficiency, scalability, and multimodal reasoning. This subsection explores these foundation models, highlighting their innovations, contributions, and suitability for GUI agent applications. For quick reference, Table and 42 present an overview of the key models and their characteristics.
7.1.1
Models
While proprictary models are not openly available for customization, they offer powerful capabilities that be directly utilized as the "brain" of GUT agents.
Among these, GPT-4V OpenAl (2023) and GPT-4o Hurst et al. 2024 are most commonly used in existing GUI agent frameworks due to their strong abilities, as discussed in Section b advancement in multimodal AI, combining text and image analysis expand the functionality of traditional LLMs. Its ability to understand and generate responses based on both textual and visual inputs makes it well-suited for GUI agent tasks that require deep multinodal reasoning. Although its deployment is limited due to safety and ethical considerations, GPT-4V underscores the potential of foundation models to revolutionize GUT agent development with enhanced efficiency and flexibility.
GPT-4V represents a significant
Similarly, GPT-4o offers a unified multimodal autoregressive archilecture capable of processing text, audio, images, and video. This model excels in generating diverse outputs efficiently, achieving faster response times at lower costs compared its prodecessors. Its rigorous safety and alignment practices make reliable for sensitive tasks, positioning it as robust tool for intelligent GUI agents that require comprehensive multimodal comprehension.
56

---

Published in Transactions on Machine Learning Research (06/2025)
The Gemini model family Team et al. (2023) advances multimodal AI modeling by offering versions tailored for high-complexity lasks, scalable performance, and on-device efficiency. Notably, the Nano models demonstrate significant capability in reasoning and coding tasks despite their small size, making them suitable for resource-constrainee devices. Gemini's versatility and efficiency make a compelling choice for powering GUT agents that require both performance and adaptability.
Emphasizing industry investment in GUI automation, Claude 3.5 Sonnet (Computer Use) introduces a pioncering approach by utilizing vision-only paradigm for desktop task automation Anthropic (024]] |; Hu
et al. 2024a). It leverages real-time screenshots to observe the GUI state and generate actions, eliminating the need for metadata or underlying GUI structure. This model effectively automates GUT tasks by interpreting the screen, moving the cursor, clicking buttons, and typing text. Its unique architecture integrates a ReAct- based Yao et al. (2022b) reasoning paradigm with selective observation, reducing computational overhead by observing the environment only when necessary. Additionally, Claude 3.5 maintains a history of GUT screenshots, enhancing task adaptability and enabling dynamic interaction with software environments in human-like manner. Despite challenges in handling dynamic interfaces and error recovery, this model represents significant step forward in creating general-purpose GUI agents. Its development highlights substantial industry investment in this area, indicating growing focus on leveraging LLMs for advanced GUI automation.
The Operator model OpenAl 2025a b): developed by OpenAl, represents a new frontier in Computer- Using Agents (CUA), akin to Claude 3.5 Sonnet (Computer Use). Designed to interact with GUI environments through LLM-powered reasoning and vision capabilities, Operator builds upon GPT-4o, integrating rein- forcement learning to navigate and execute tasks across digital interfaces such as browsers, forms, and applications. By perceiving screenshots, interpreting UI elements, and performing actions via virtual cursor and keyboard, Operator enables the automation of complex GUl-based workflows, including online purchases, email management, and document editing. Notably, Operator excels in understanding and manipulating digital environments, cstablishing itself as a powerful tool for human-computer interaction automation. Its exceptional performance on various benchmarks underscores its leading capabilities in GUI-based lask automation.
7.1.2 Open-Source Models
Open-source models provide flexibility for customization and optimization, allowing developers to tailor GUI agents with contextual data and deploy them devices with limited resources.
The Qwen- VL series Bai et al. (023bb) is notable for its fine-grained visual understanding and multimodal capabilitics. With a Vision Transformer-basse visual encoder and the Qwen-7B language model Bai et al. (2023a), achieves state-of-the-art results on vision-language benchmarks while supporting multilingual interactions. Its efficiency and open-source availability, along with quantized versions for resource efficiency, make it suitable for developing GUI agents that require precise visual comprchension.
Building upon this, Qwen2-VI. Wang et al. (2024j introduces innovations like Naive Dynamic Resolution and Multimodal Rotary Position Embedding, enabling efficient processing of diverse modalities including extendedl-engtth videos. The scalable versions of Qwen2-VL. VL balance computational efficiency and performance, making them adaptable for both on-device applications and complex multimodal tasks in GUI environments.
InternVL-22 Chen et al. [2024] k) combines a Vision Transformer with a Large Language Model to handle text, images, video, and medical data inputs. Its progressive aligument strategy and availability in various sizes allow for flexibility in deployment. By achieving state-of- waateof-thhe-art the-art performance in complex multimodal tasks, InternVI-2 demonstrates powerful capabilities that are valuable for GUI agents requiring comprehensive multimodal understanding.
Advancing efficient integration of visual and linguistic information, CLM Wang et al. 2024m excels in cross-modal tasks with a relatively small number of trainable parameters. Its ability to deeply integrate visual and language features while preserving the full capabilities of large language models makes it a cornerstone for GUT agent development, especially in applications where resource efficiency is critical.
57

---

Published in Transactions on Machine Learning Research (06/2025)
Enhancing spatial understanding and grounding, Ferret You et al. (2023) offers innovative approach tailored for GUT agents. By unifying referring and grounding tasks within single framework and employing hybrid region representation, it provides precise interaction with graphical interfaces. Its robustness against object hallucinations and efficient architecture make ideal for on-device deployment in real-time GUI applications.
The LLaVA model Liu et al. (2024b) integrates a visual encoder with a language decoder, facilitating efficient alignment between modalities. Its lightweight projection layer and modular design enable quick experimentation and adaptation, making it suitable for GUI agents that require fast development cycles and strong multimodal reasoning abilitios. Building on this, LLaVA-1.5 Liu et al. (2024a) introduces a novel MLP-based cross-modal connector and scales to high-resolution image inputs, achieving impressive performance with minimal training data. Its data efficiency and open-source availability pave the way for widespread use in GUT applications requiring detailed visual reasoning.
BLIP-2 Li et al. (2023c) employs compute--ffieient stralegy by leveraging frozen pre-trained models and introducing lightweight Querying Transformer. This design allows for tate-of-thheart performance vision-language tasks with fewer trainable parameters. BLIP-2's modularity and efficiency make it suitable for resource-constrained environments, highlighting its potential for on-device GUI agents.
Finally, Phi-3.5-Vision Abdin et al. (024)) achieves competitive performance in multimodal reasoning within a compact model size, Its innovative training methodology and efficient integration of image and text understanding make robust candidate for GUI agents that require multimodal reasoning and on-device inference without the computational overhead of larger models.
In summary, both close-source and open-source foundation models have significantly advanced the capabilities of LLM-powered GUI agents. While proprietary models offer powerful out-of-the-box performance, open- source models provide flexibility for customization and optimization, enabling tailored solutions for diverse GUI agent applications. The innovations in multimodal reasoning, efficiency, and scalability across these models highlight the evolving landscape of foundation models, paving the way for more intelligent and accessible GUI agents.
7.2 Large Action Models
While gencral-purpose foundation LLMs excel in capabilities like multimodal understanding, task planning, and tool utilization, they often lack the specialized optimizations required for GUI-oriented tasks. To address this, researchers have introduced Large Action Models (LAMs))ffundattee foundation LLMs fine-tuned with contextual, GUL-specific datasets (as outlined in Section 6 to enhance their action-driven capabilities. These models represent significant step forward in refining the "brain" of GUI agents for superior performance.
In the realm of GUT agents, LAMSs provide several transformative advantages:
1. Enhanced Action Orientation: By specializing in action-oriented tasks, LAMs enable accurate interpretation of user intentions and generation of precise action sequences. This fine-tuning ensures that LA can seamlessly align their outputs with GUT operations, delivering actionable steps tailored to user requests.
2. Specialized Planning for Long, Complex Tasks: LAMs excel in devising and executing intricate, multi-step workflows. Whether the tasks span multiple applications or involve interdependent operations, LAMs leverage their training extensive action sequence datasets to create coherent, long-term plans. This makes them ideal for productivity-focused tasks requiring sophisticated planning across various tools.
3. Improved GUI Comprehension and Visual Grounding: Training on datasets that incorporate GUI screenshots allows LAMs to advance their abilities detecting, localizing, and interpreting UI components such as buttons, menus, and forms. By utilizing visual cues instcad of relying solely on structured UI metadata, LAMs become highly adaptable, performing effectively across diverse software environments.
58

---

Published in Transactions on Machine Learning Research (06/2025)
Foundation LLM
Step u: Select the note on Slide
60%
Textual Reply
Step 2: Delete the selected note.
Step 3: Repeat for each slide until all notes delete.
Fine-tuning Dataset
Instruction
Widget Tree
UI Properties (Botton Insert...) (Botton Chart, ... (Menu Design. (Ustttem Art-..
Action Sequences
Screenshots
Fine-tuning
Task: Delete all notes on the slide deck
Action Execution
Optimized
GUI Agent
Figure 25: The evolution from foundation LLMs to GUT agent-optimized LAM with fine-tuning.
4. Efficiency through Model Size Reduction: Many LAMs are built on smaller foundational models- lypically around billion parameters that are oplimized for GUT-specific tasks. This compact, purpose-driven design reduces computational overhead, enabling efficient operation even in resource-constrained environments, such on-device inference.
As illustrated in Figure 25 the process of developing a purpose-built LAM for GUI agents begins with a robust, general-purpose foundation model, idcally with VLM capabilities. Fine-tuning these models on comprehensive, specialized GUT sequences, and annotated screenshots transforms them into optimized LAMs, effectively equipping them to serve the "brain" of GUI agents.
iincluding instructions, widget. trees, UT properties, action
This optimization bridges the gap between planning and execution. A general-purpose LLM might provide only textual plans or abstract instructions in response to user queries, which may lack precision. In contrast, a LAM-enpovered GUI agent moves beyond planning to actively and intelligently execute tasks on GUTs. By interacting directly with application interfaces, these agents perform tasks with remarkable precision and adaptability. This paradigm shift marks the evolution of GUI agents from passive task planners to active, intelligent executors.
7.3 LAMs for Web GUI Agents
In the domain of web-based GUI agents, researchers have developed specialized LAMs that enhance interaction and navigation within web environments. These models are tailored to understand the complexities of web GUTs, including dynamic content and diverse interaction patterns. We present an analysis of LAMs tailored
for web GUI agents in Table 43
Building upon the need for multimodal understanding, WebGUM Furuta et al. (0023 integrates HTML understanding with visual perception through temporal and local tokens. leverages Flan-T5 Ching et al. (2024) for instruction fine-tuning and ViT Dosovitskiy et al. (2021) for visual inputs, enabling it process both textual and visual information efficiently. This multimodal grounding allows WebGUM to generalize
59

---

Published in Transactions on Machine Learning Research (06/2025)
tasks effectively, significantly outperforming prior models on benchmarks like MiniWoB++ Liu et al (0018) and WebShop Yao et al. (2023). With its data-efficient design and capacity for multi-step reasoning, WebGUM underscores the importance of combining multimodal inputs in enhancing GUI agent performance.
Addressing the challenge of multi-step reasoning and planning in GUI environments, researchers have introduced frameworks that incorporate advanced search and learning mechanisms. For instance, Agent Q Putta al. (2024) employs MCTS combined with self-critique mechanisms and Direct Preference Optimization (DPO) Rafailov et al. (2024) to improve success rates in complex tasks such as product search and reservation booking. By fine-tuning the LLaMA-3 70B model Dubey et al. (2024) to process HTML DOM reprosentations and generate structured action plans, thoughts, and environment-specifc commands, this framework showcases the power of integrating reasoning, search, and iterative fine-tuning for autonomous agent development.
Leveraging smaller models for efficient web interaction, GLAINTEL Fercidouni & Siddique (2024) demon- strates that high performance can be achieved without large computational resources. Utilizing the Flan- T5 Chung et al. (2024) model with 780M parameters, it focuses dynamic web environments like simulated e-commerce platforms. The model incorporates RL to optimize actions such query formulation and navigation, effectively integrating human demonstrations and unsupervised learning. Achieving results comparable to GPT-4-based methods at a fraction of the computational cost, GLAINTEI, underscores the potential of reinforcement learning in enhancing web-based GUI agents for task-specific optimization.
To enable contimuous improvement and generalization across varied web domains, OpenWeb Voyager He et al. (2024c) combines imitation learning with an iterative ppplooattoo-feedlee---tttiiaaaii large multimodal models like Idefics2-8B Laurengon al. 2024a it performs autonomous web navigation lasks. By Iraining diverse datasets and fine-tuning using trajectories validated by GPT-4 feedback, the agent addresses real-world complexities without relying on synthetic environments. This approach significantly advances GUI agent frameworks by demonstrating the capability to generalize across varied web domains and tasks.
cycle. Leveraging
Moreover, tackling challenges such as sparse training data and policy distribution drift, WebRL Qi al, (2024) introduces a self-evolving curriculum and robust reward mechanisms for training LLMs proficient web agents. By dynamically generating tasks based on the agent's performance, WebRL fine-tunes models like Llama-3.1 Dubey et al. (2024) and GLMM4 GLM et al. (2024), achieving significant success rates in web-based tasks within the WebArena environment. This framework outperforms both proprietary APIs and other open-source models, highlighting the effectiveness of adaptive task generation and sustained learning improvements in developing advanced GUT agents.
These advancements in LAMs for web GUI agents illustrate the importance of integrating multimodal inputs, efficient model designs, and innovative training frameworks to enhance agent capabilities in complex web environments.
7.4 LAMs for Mobile GUI Agents
Mobile platforms present unique challenges for GUI agents, including diverse screen sizes, touch interactions, and resource constraints. Researchers have developed specialized LAMs to address these challenges, enhancing interaction and navigation within mobile environments. We present an overview T.A LAMs tailored for mobile
GUI agents in Table 44 and 46
Focusing detailed UI understanding, MobileVLM Wu et al. (202cc) introduces advanced vision- language model designed specifically for mobile UI manipulation tasks. Built on Qwen-VL-Chat Bai et al. (2023b). it incorporates mobile-speeific pretraining tasks for intra- and inter-UI comprchension. By leveraging the Mobile3M dataset- a comprehensive corpus of 3 million UI pages and interaction traces organized inlo directed graphs the model excels in action prediction and navigation tasks. MobileVLM's novel two-stage pretraining framework significantly enhances its adaptability to mobile Uls, outperforming existing VLMs in benchmarks like ScreenQA Hsiao et al. (2024) and Auto-Ul Zhang & Zhang (2024). This work highlights the effectiveness of tailored pretraining in improving mobile GUI agent performance.
60

---

Published in Transactions on Machine Learning Research (06/2025)
Screenshot State Action
Policy Model
Shorsts
Exploration
CLICK(SO, 60)
CLICK(30, 120)
AI
VEM
State-Action Values
PPO
X
Policy Model
Figure 26: The PPO training process of VEM Zheng et al. (202bb.. Figure adapted from the original paper.
Addressing the need for robust interaction in dynamic environments, DigiRL Bai et al. (2024) presents reinforcement berning-based framework tailored for training GUI agents in Android environments. By leveraging offline-to-online RI., DigiRL adapts to real-world stochasticity, making it suitable for diverse, multi-stop tasks. Unlike prior models reliant imitation lcarning, DigiRL autonomously learns from interaction data, refining itself to recover from errors and adapl to new scenarios. The use of a pre-trained Vision- Assion-Languag Model with 1.3 billion parameters enables efficient processing of GUI screenshots and navigation commands. Its performance on the AITW dataset demonstrates a significant improvement over baseline methods, positioning DigiRL as a benchmark in the development intelligent agents optimized for complex GUT interactions.
Both Digi-Q Bai et al. (0255a) and VEM Zheng et al. (202bb) investigate the use of offline RL to enhance the performance of GUI agents without requiring direct interaction with the environment. Digi-Q employs temporal-diiference learning to train Q-function offline and derives policies through Best-of-N selection strategy based on the predicted Q-values. Similarly, VEM introduces an environment-free RL framework tailored for training LLW-re GUI agents using PPO. directly estimates state-action values from offline data by fine-tuning with annotated value data from GPT-4o, thereby enabling policy training without. real-time exccution a GUI environment. At inference time, only the policy model utilized. Figure 26 illustrates the overall architecture of VEM. The study further demonstrates that oflline RL with structured credit assignment achieve performance comparable to interactive RL models. Overall, VEM offers scalable and layout-agnostic approach for training GUT agents while minimizing interaction costs. Both works underscore the potential of offline RL for GUI agent training.
To enhance GUI comprehension and reduce reliance on textual data, VGA Meng et al. (0024) employs finc-tuned vision-language models that prioritize imago-basod cues such as shapes, colors, and positions. Utilizing the RICO Deka al. (2017) dataset for training, VGA tailored for Android GUIs and employs two-stage fine-tuning process to align responses with both visual data and human intent. The model
61

---

Published in Transactions on Machine Learning Research (06/2025)
excels in understanding GUI layouts, predicting design intents, and facilitating precise user interactions. By outperforming existing models like GPT-4V in GUT comprehension benchmarks, VGA sets a new standard for accuracy and efficiency in mobile GUI agents.
In the context of lightweight and efficient models, UINav Li et al. 2024f demonstrates a practical system for training neural agents to automate UT tasks on mobile devices. It. balances accuracy, generalizability, and computational efficiency through macro actions and an error-driven demonstration collection process. UTNav uses compact mecder-docoder architecture and SmallBERT Ture et al. (2019) for text and screen element encoding, making it suitable for on-device inference. A key innovation is its ability to generalize across diverse tasks and apps with minimal demonstrations, addressing key challenges in UI automation with a versatile framework.
UI-R1 Lu et al. 2025b) introduces a for multimodal large language models (MLLMs). The resulting model, UI-R1-3B, fino-tunes Qwen2.5- VL-3B using a novel rule-based reward function that jointly evaluates action type correct and click coordinate accuracy, while also enabling ol-style Jaech et al. (2024) chain-of-thought (CoT) reasoning through structured <think> tags. UI-RI relies on only 136 high-quality samples selected via a three-stage filtering strategy. Despite this limited supervision, UI-R1-3B achieves significant improvements on both in-domain and out-of- domain benchmarks. By leveraging Group Relative Policy Optimization (GRPO) Shao et al. (2024), the framework aligns policy optimization with the goals of GUI grounding and task execution. UI-R1 establishes a scalable and data-efficient approach for training GUI agents via RL and paves the way for light yet effective agent design. Its methodology has also been successfully extended to cross-platform agents Xia &
training paradigm aimed at enhancing GUI action prediction
Luo 2025
Liu et al. (2025 demonstrating strong generalization capabilities.
In addition to action models, ViMo Luo et al. (2025a) introduces novel generative visual world model for GUI agents, aimed at improving App agent decision-makin making by predicting the next GUI state as an image rather than a textual description. A key innovation of ViMo is the Symbolic Text Representation (STR), which replaces GUI text regions with structured placeholders to facilitate accurate and legible text synthesis. This decoupled design allows the system to handle GUT graphics generation using a fine-tuned diffusson model, and text generation through an LLM, thereby achieving high visual fidelity and semantic precision. ViMo significantly boosts both GUI prediction quality and downstream agent performance, with reported 29.14% relative improvement in GUI generation metrics and enhanced planning accuracy for long-horizon lasks. As a forward simulator, ViMo represents a crucial advancement toward reliable world models for mobile GUI agents, supporting more effective decision evaluation and trajectory planning in visual environments.
These models collectively advance the field of mobile GUT agents by addressing platform-specffic challenges through innovative training methods, efficient model architectures, and specialized datasets.
7.5 LAMs for Computer GUI Agents
For desktop and laptop environments, GUI agents must handle complex applications, multitasking, and varied interaction modalities. Specialized LAMs for computer GUI agents enhance capabilities in these sings,,,, enabling more sophisticated task execution. We overview of LAASs for computer GUT agents across
in Table 47
Integrating planning, acting, and reflecting phases, ScreenAgent Niu et al. (2024) is designed for autonomous interaction with computer screens. Based on CogAgent Hong et al. (2023). , it is fine-tuned using the ScreenAgent Datasot, providing comprehensive GUT interaction data across diverse tasks. With inputs as screenshots and outputs formatted in JSON for mouse and keyboard actions, Screen achieves precise UI element localization and handles continuous multi-step tasks. Its capability to process real-time GUI interactions using a foundation model sets new benchmark for L.M.-powered GUT agents, making it an ideal reference for future research in building more generalized intelligent agents.
Bridging high-level planning with real-world manipulation, Octopus Yang et al. (025bb) represents a pioneering step in embodied vasion-language programming. Leveraging the MPT-7B MosaicML (2023) and CLIP ViT-L/14 Radford et al. (2021), Octopus integrates egocentric and bird's-eye views for visual comprehension, generating executable action code. Trained using the Verse suite, its datasets encompass
62

---

Published in Transactions on Machine Learning Research (06/2025)
richly annotated environments like OmniGibson, Minecraft, and GTA-V, covering routine and reasoning- intensive tasks. Notably, Octopus innovates through Reinforcement Learning with Environmental Feedback, ensuring adaptive planning and execution. Its vision-dependent functionality offers seamless task generalization in unseen scenarios, underscoring its capability as a unified model for embodied agents operating in complex GUT environments.
Wang et al,, Wang et al. (2024h) present a comprehensive overview of LAMs, a new paradigm in AL designed to perform tangible actions in GUI environments, using UFO Zhang et al. (2024a) at Windows OS case study platform. Built on the Mistral-7B [Jiang et al. 2023 foundation, LAMs advance beyond traditional LLMs by integrating task planning with actionable outputs. Leveraging structured inputs from tools like the UI Automation (UIA) API, LAMs generate executable steps for dynamic planning and adaptive responses. A multi-phase training strategy--encompasee encompassing task-plan pretraining, imitation learning, self- boosting exploration, and reinforcement learning (eariing-ensuees ensures robustness and accuracy. Evaluations on real-world GUI tasks highlight LAMs' superior task success rates compared to standard models. This innovation establishes a foundation for intelligent GUT agents capable of transforming user requests into real-world actions, driving significant progress in productivity and automation.
These developments in computer GUI agents highlight the integration of advanced visual comprehension, planning, and action execution, paving the way for more sophisticated and capable desktop agents.
7.6 Cross-Platform Large Action Models
To achieve versatility across various platforms, cross- poss--latform LAMs have been developed, enabling GUI agents to operate scamlessly in multiple environments such as mobile devices, desktops, and web interfaces. We provide an analysis of LAMs tailored for cross-platform GUI agents in Tables 48 and 50
CogAgent Hong et al. 2023) stands out
advanced visual language model specializing in GUI understanding and navigation across PC, web, and Android platforms. Built on CogVLM Wang et al.] (2024m) it incorporates a novel high-resolution cross-module to process GUI screenshots efficientl,, enabling detailed comprehension of GUT elements and their spatial relationships. Excelling in tasks requiring OCR and GUI grounding, CogAgent achieves state-of-the-ars performance on benchmarks like Mind2 Web Deng et al. (2023) and AITW Rawles et al. 2023). Its ability to generate accurate action plans and interface with GUIs positions it as a pivotal step in developing intelligent agents optimized for GUI environments. CogAgent has further evolved into its beta version, GLM-PC CogAgent Team 22024 offering enhanced control capabilities.
Focusing on universal GUI understanding, UI 2 Li et al. (2024j) from Apple is stale-of-the-arl multimodal large language model designed to master UI comprehension across diverse platforms, including iPhones, Android devicos, iPads, web, and ApplcTV. By employing dynamic high-resolution image encoding, adaptive gridding, and high-quality multimodal training data generated through GPT-4, it outperforms its predecessor and other competing models in UT referring, grounding, and interaction tasks. Ferret-UT 2's advanced datasets and innovative training techniques ensure high accuracy in spatial understanding and user-centered interactions, setting new benchmark for eross-platform UI adaptability and performance.
Advancing GUT automation, Show Lin et al. [004c] introduces a pioneering Vision-Language ActtinnannAttinnn model that integrates high-resolution visual inputs with textual understanding to perform grounding, navigation, and task planning. Optimized for web, desktop, and mobile environments, Show leverages the Phi-3.5-vision- instruct backbone and comprehensive datasets to achieve robust results across benchmarks like ScreenSpot. Cheng et al. (2024a) and GUI-Odyssey Lu et al. (2024b). Its ability to process multi-frame and dynamic visual inputs alongside -SON-structured output actions highlights its versatility. With innovations in interleaved image-text processing and function-calling capabilities, ShowUl sets new standard for LLM--ooered GUI agents.
Action
Addressing the need for a unified action space, AS-ATLAS Wu et al. 2024f introduces a foundational action model specifically designed for GUI agents across platforms like Windows, macOS, Linux, Android, and the web. By leveraging a massive multi-platform datasct and implementing a unified action space, OS-ATLAS achieves ttate-of-the-art performance in GUI grounding and scalable configurations adapt to varying computational needs while maintaining versatility in handling natural
generalization tasks. Its
63

---

Published in Transactions on Machine Learning Research (06/2025)
language instructions and GUI elements. As powerful open-source alternative to commercial solutions, OS-ATLAS marks significant step toward democratizing access to advanced GUT agenis.
Magma Yang et al. (2025a) is foundation model for multimodal Al agents that integrates LLMs with vision and action understanding to complete UI navigation and robotic manipulation tasks. Unlike previons models optimized for either UT automation or robotics, Magma jointly trains on heterogeneous dataset (about 39M samples) spanning U1 screenshots, web navigation, robot trajectories, and instructional videos. It employs SoM and Trace-of-Mark techniques, which enhance action grounding and prediction by labeling actionable elements in GUI environments and tracking motion traces in robotic tasks.
UI-TARS Qin et al. (2025) is an advanced, vision-based Large Action Model (LAM) optimized for multi- form GUT agents. Unlike (raditional approaches, it relies solely on GUI screenshots for perceplion, eliminating the need for structured representations. By incorporating a unified action space, UI-TARS enables scamless exccution across Web, Windows, macOS, and Android environments. Built on Qwen-2-VL VI, is trained on million GUI tutorials, large-scale screenshot datasets, and multiple open-source benchmarks. A key innovation of UI-TARS is its System-2 reasoning capability, which allows it to generate explicit reasoning steps before executing actions, enhancing decision-making in dynamic environments. Additionally, it employs an iterative self-impprvement framework, refining its performance through reflection-based learning. Experimental results demonstrate that UI-TARS outperforms existing models, including GPT-4o and Claude, in task execution benchmarks.
These rross-platform LAMs demonstrate the potential of unified models that can adapt to diverse environments, enhancing the scalability and applicability of GUT agents in various contexts.
7.7 Emerging Trends Amid Rapid Improvements in LAM Capabilities
Recent, breakthroughs in LAM have driven considerable innovation in GUT agent research. While these large models excel at general reasoning, open-domain conversation, and multimodal interpretation, certain longstanding challenges in GUI-based automation remain only partially addressed. In this subsection, we outline three interrelated trends that are poised to remain central- even as baseline LAM capabilities continue to risc-and and discuss how these trends may benefit from, or be constrained by, ongoing advancements in foundation models.
Live Environments and Data Coverage. A recurring challenge in GUI automation is acquiring rich, diverse data that truly reflect the complexities of real user interfaces. Many LAM systems are trained on static snapshots or curated datasets, which may not capture the dynamic, event-driven nature of GUI environments. As these agents become more capable, the neod for live environment interaction grows in importance. Operating in a live environment allows agents to iteratively collect data and refine their understanding of various apps, platforms, or usage contexts- thereby improving coverage of niche scenarios and rare interface states. Nevertheless, live data collection can be both time-consuming and costly, raising questions about how to balance on-device exploration with robust offline datasets. Expanding open-source benchmarks and real-time simulators can help address coverage gaps without incurring prohibitive data-collection expenses, ensuring that agent training encompasses both realistic and representative GUI states.
Reinforcement Learning for GUI Agents. A second trend persists in using Reinforcement Learning (RL) to optimize agent behavior over extended interactions. With the advent of more powerful LAMs, RL can integrate dynamic feedback signals-suu or time-to-complltion. In online RL, agents interact with live systems to continually refine their policics, though the exploration cost can be substantial if an agent must repeatedly trial-and-error in production environments. Offline RL is a compelling alternative, enabling large-scale pretraining on recorded trajectories or user logs to reduce reliance on potentially expensive or risky online exploration. These methods can leverage improved baseline model capabilities to interpret and navigate complex UIs more effectively, while learning from failures suboptimal states without exposing end users to frequent disruptions or errors.
rewards constraints derived from user feedback, success rates,
64

---

Published in Transactions on Machine Learning Research (06/2025)
Task 1: Change font size to 20 in the PPT page
f
Evaluation Measurements
Actions
Task List
Screenshots
Trajectory
Ul structure
GUI Agent
Env State
Metric
Score
Task Success
0.65
Interaction
Step Success Rate
0.82
105
Evaluation Metrics
Platform
Figure 27: An illustrative example of evaluation of task completion by a GUI agent.
Inference-Tinee Computing and Reasoning Models. As LAM research accelerates, there growing emphasis on anference-time computing where models dynamically plan, reason, and decompose tasks without extensive architectural modifications. Novel approaches (e.g., "R1-like" extended context windows chain- of-thought prompting) can deliver longer-horizon reasoning, thereby improving the accuracy and stability of GUI interactions. This can be especially beneficial for long or multi-step tasks that require the agent to keep track of evolving interface states and dependencics. In tandem, we see increased interest in rulo-basod rewards or cost functions that guide inference- time decisions- for example, penalizing unnecessary clicks or prioritizing user-facing safety chocks. By integrating explicit heuristics with advanced model reasoning, GUI agents can achieve more stable and interpretable behavior, despite the inherent open-endedness of LLM-driven interactions.
8 Evaluation for LLM-Powered GUI Agents
In the domain of GUI agents, evaluation is crucial for enhancing both functionality and user experience Li
et al. (2024d); Huang Zhang (2024) and should be conducted across multiple aspects. By systematically
assessing these agents' effectiveness across various tasks, evaluation not only gauges their performance
different dimensions but also provides a framework for their continuous improvement Liu et al. (2023b)
Furthermore, it encourages innovation by identifying areas for potential development, ensuring that GUI agents evolve in tandem with advancements in LLMs and align with user expectations.
As illustrated in Figure 27 when a GUI agent completes a task, it produces an action sequence, captures screenshots, extracts UI structures, and logs the resulting environment states. These outputs serve as the foundation for evaluating the agent's performance through various metrics and measurements across diverse platforms. In the subsequent sections, we delve into these evaluation methodologies, discussing the metrics and measurements used to assess GUI agents comprehensively. We also provide an overview of existing benchmarks tailored for GUT agents across different platforms, highlighting their key features and the challenges they address.
8.1 Evaluation Metrics
Evaluating GUI agents requires robust. and multidinensional metrics to assess their performance across various dimensions, including accuracy, efficiency, and compliance (e.g., safety). In a typical benchmarking sotup, the GUT agent provided with a natural language instruction as input and expected to autonomously execute actions until the task is completed. During this process, various assets can be collected, such as the sequence of actions taken by the agent, step-wise observations (e.g., DOM or HTML structures), screenshots,
65

---

Published in Transactions on Machine Learning Research (06/2025)
runtime logs, final states, and execution time. These assets enable evaluators to determine whether the task has been completed successfully and to analyze the agent's 's performance. In this section, summarize the key evaluation metrics commonly used for benchmarking GUI agents. Note that different research works may use different names for these metrics, but with similar calculations. We align their names in this section.
1. Step Success Rate:
Completing task may require multiple steps. This metric measures the ratio of the number of steps that successful over the total steps within task. A high step success rate indicates precise and accurate execution of granular steps, which is essential for the reliable performance of tasks involving multiple steps Deng et al. (2023); Pan et al. (2024b); Rawles et al. (0033
Turn Success Rate: A turn indicates single interaction between the user and the agent. turn may consist of multiple steps, and completing a task may consist of multiple turns. This metric measures the ratio of turns that successfully address the request in that interaction over all turns. It focuses on the agent's ability to understand and fulfill user expectations during interactive dialog-based tasks, ensuring the agent's 's responsiveness and reliability across iterative interactions, particularly in lasks requiring dynamic user-agent communication Lu et al. (2024c): Deng et. al. 2024bb).
Task Success Rate:
Task success rate measures the successful task completion over all lasks set
the benchmark. It evaluates whether the final task completion state is achieved while ignoring the intermediate steps. This metric provides an overall measure of end-to-end task completion, reflecting the agent's 's ability to handle complex workflows holistically Yao et al. (2023) Zhang et al. (2024c); Xie et al. 2024b).
4. Efficiency Score: Efficiency score evaluates how effectively the agent completes lasks while
considering resource consumption, execution time, or total steps the agent might take, This metric can be broken down into the following sub-metrics:
Time Cost: Measures the time taken complete tasks.
Resource Cost: Mcasures the memory/CPU/ GPU usage to complete tasks.
LLM Cost:
Evaluates the computational or monetary cost of LLM calls used during task
execution.
Step Cost:
Mcasures the total stops required to complete tasks.
Depending on the specific metrics used, the efficiency score can be interpreted differently in different papers Chen al. (2024d): Deng et al. (2024a)
Completion under Policy: This metric measures the rate at which tasks completed succossfully while adhering to policy constraints. It ensures that the agent complies with user-defined organizational rules, such as security, ethical, safety, privacy, or business guidelines, during task execution. This metric is particularly relevant for applications where compliance is as critical as task success Levy et al. (2024).
Similar to the previous metric, the risk ratio evaluates the potential risk associated with the agent's actions during task execution. It identifies vulnerabilities, errors, or security concerns that could arise during task handling. A lower risk ratio indicates higher trustworthiness and reliability, while a higher ratio may suggest areas needing improvement to minimize risks and enhance robustness Levy et al. (024))
Risk Ratio:
The implementation of metrics in each GUT agent benchmark might vary depending on the platform and the task formulation. In all tables in this section, we mapped the original metrics used in the benchmarks, which may possess different names, to the categories that defined above.
66

---

Published in Transactions on Machine Learning Research (06/2025)
8.2 Evaluation Measurements
To effectively evaluate GUI agents, various measurement techniques are employed to assess their accuracy and aligmment with expected outputs. These measurements validate different aspects of agent performance, ranging from textual and visual correctncss to interaction accuracy and system state awareness, using codc, models, and even agents as evaluators Zhuge et al. (202)). Below, we summarize key measurement approaches used benchmarking GUT agents. Based on these measurements, the evaluation metrics defined beforehand can be calculated accordingly.
This measurement evaluates whether the text-based outputs of the agent match the expected results. For example, whether target product name is reached when the agent is browsing an e-commerce website. It can involve different levels of strictness, including:
Text Match:
Ensures the output perfectly matches the expected result. Partial or Fuzzy Match: Allows for approximate matches, which are useful for handling minor variations such typos synonyms.
Exact Match:
Semantic Similarity: Measures deeper alignment in semantic meaning using techniques like
cosine similarity of text embeddings or other semantic similarity measures.
Text Match is widely applicd in tasks involving textual selections, data entry, natural language responses.
Image Match focuses on validating whether the agent acts or stops on the expected page (e.g., webpage, app UT), selects the right image. It involves comparing screenshots, selected graphical elements, or visual outcomes against ground truth images using image similarity metrics or visual question answering (VQA) methods. This measurement is particularly crucial for tasks requiring precise visual identification.
Image Match:
3. Element Match:
This measurement checks whether specific widget elements (e.g., those in HTML, DOM, or application UT hierarchies) interacted with by the agent align with the expected elements. These may include:
HTML Tags and Attributes: Ensuring the agent identifies and interacts with the correct.
structural elements.
URLs and Links: Validating navigation-related elements.
DOM Hierarchies: Confirming alignment, with expected DOM structures in dynamic or complex web interfaces.
UI Controls and Widgets: Verifying interactions with platform-specifie controls such buttons, sliders, checkboxes, dropdown menus, or other GUI components in desktop and mobile applications.
Utilizing accossibility identifiers or resouree IDs in mobile platforms like Android and iOS to ensure correct element selection.
Accessibility Identifiers:
View Hicrarchics: Assessing alignment with expected view hicrarchics in mobile applications, similar to DOM hierarchies in web applications.
System Controls and APIs: Ensuring correct interaction with operating system controls or
APIs, such as file dialogs, system memus, or notifications in desktop environments.
Flement Match ensures robust interaction with user interface components across different platforms during task execution.
Action Match:
This measurement assesses the accuracy of the agent's 's actions, such as clicks, scrolls, or keystrokes, by comparing them against an expected sequence. It involves:
Action Accuracy: Validates that each action (including action type and its arguments) is performed correctly (e.g., clicking the correct button, typing the right input).
Action Sequence Alignment: Ensures actions occur in the correct order meet task requirements.
67

---

Published in Transactions on Machine Learning Research (06/2025)
Location Prediction: Checks that spatial actions, such as mouse clicks or touch gestures, target the intended regions of the interface.
Action Match is vital for evaluating step-wise correctness in task completion.
5. State Information: State Information captures runtime data related to the system's environment during task execution. It provides insights into contextual factors that may influence the agent's behavior, such as:
Application State: Information about the state the application being interacted with (e.g., open files, active windows, saved files in given locations).
System Logs: Detailed logs recording the agent's decisions and interactions.
Environment Variables: Contextual data about the operating system or runtime enviromment.
This measurement is valuable for debugging, performance analysis, and ensuring reliability under diverse conditions.
Each of these measurement techniques contributes to a comprehensive evaluation framework, ensuring that the agent not only completes tasks but does so with precision, efficiency, and adaptability. Together, they help build trust in the agent's 's ability to perform reliably in real-world scenarios while maintaining compliance with policy constraints.
8.3 Evaluation Platforms
Evaluating GUI agents requires diverse platforms to capture the varying environments which these agents operate. The platforms span web, mobile, and desktop environments, each with unique characteristics, challenges, and tools for evaluation. This section summarizes the key aspects of these platforms and their role in benchmarking GUT agents.
1. Web: Web platforms among the most common environments for GUI agents, reflecting their prevalence in everyday tasks such as browsing, form filling, and data scraping. Key characteristies of web platforms for evaluation include:
Dynamic Content:
Web applications often involve dynamic elements generated through JavaScript, AJAX, or similar technologies, requiring agents to handle asynchronous updates effectively.
The variety of web technologies (e.g., HTML, CSS, JavaScript frame- works) demands robust agents capable of interacting with a range of interface designs and structures.
Diverse Frameworks:
Tools and Librarics:
Evaluation often uses tools such as Sclenium, Puppeteer, or Playwright to emulate browser interactions, collect runtime information, and compare outcomes against expected results.
Accessibility Compliance: Metrics like WCAG (Web Content Accessibility Guidelines) adherence can also be evaluated to ensure inclusivity.
Mobile: Mobile platforms, particularly Android and iOS, pose unique challenges for GUI agents due to their constrained interfaces and touch-based interactions. Evaluating agents on mobile platforms involves:
Screen Size Constraints: Agents must adapt to limited screen real estate, cnsuring interactions remain accurate and efficient.
Touch Gestures: Evaluating the agent's ability to simulate gestures such as taps, swipes, and pinches is essential.
Platform Diversity: Android devices vary significantly in terms of screen sizes, resolutions, and system versions, while iOS offers more standardized conditions.
68

---

Published in Transactions on Machine Learning Research (06/2025)
Table 14: Taxonomy of LLM-powered GUI agent benchmarks by target platform.
Platform
References
Web
Liu et al.
2024f;
Shahbandeh et al.
30224; Lai et al.
(2024); ; Xu et al. (202)
;
Shi et al.
(017):
Liu et al. (2018); Deng al. (2023 Chen et al. (2024e); Lu
et al.
(2024c); [hou et al.]
Koh et al. (0224a): Deng et al. (2024c) Zhang et al.
[2024sb;
Pan et al.
(0024b)
; Levy et al. (2024); Furuta et al. (2024); ); Xu et al.
[2024e]
Xie et. al.
2024bb
: Drouin et al. (2024); Jang el al.
(202 Ma el al.
(2024c) Wornow et al. Zheng et al. (2024b); Yao et al. [2023 Chezelles et al.
(2024);
Wu ct al.
(2025a); Thomas et al. (2025): Tur al. 20025 Kara ct al.
Thomas
Xue al. (2025):
Zharmagambetor
al. (2025); Lu al.
(2025 0 Ye al.
2025
Garg et al.
2025 Song et al. 2025 Evtimov et al.
(0025)
Mobile
Rawles ct al.
(2023
;;
Xu ct al.
(2024)) Wen et al.
(2024a):
Li ct al.
(2020a):
Toyama et al.
(2021a); Wang et al.
(2024e); Zhang et al.
2024ch:
Lee et al.
(2024b
Rawles
(2024];
Xing al. (2024): Deng
(2024a):
Lee et al.
2024a
Chen al.
(2024d
0 Zhang al. (2024h) Wang et al.
(2024i) Zhao
ct al.
2024b);
Chai et al. (2025) Ran et al. 20025
Chen et al.
(2025c)
Sun
el. al.
2025a
Wang et al. 2025d
Computer
Gao
al.
(0024b);
Bonatti et al.
(2024]
Xie et al. (2024b);
Cao et al.
(2024)
Wang et al.
(2024p
b Zheng et al. (2024ch
Zhao et al. (2025ch; Nayak al.
(2025]:
Wang et al. (2025a)
Wu et al.
(2024b):
Liu et al. (2024i); Kapoor et al. (2024): Lin et al.
2024b
Xu et al.
2004g Cheng et al. (0024a) ; Fan al. (2024)
Tools like Appium and Espresso (for Android) or XCTest (for iOS) and emulators are commonly used for testing and evaluation.
Evaluation Tools:
3. Desktop:
Desktop platforms provide a richer and more complex environment for GUI agents, spanning multiple operating systems such as Windows, macOS, and Linux. Evaluations on desktop platforms often emphasize:
A Application Diversity: Agents must handle wide range of desklop applications, including productivity tools, web browsers, and custom enterprise software.
Interaction Complexity: Desktop interfaces often include advanced features such as keyboard shortcuts, drag-and-drop, and context menus, which agents must handle correctly.
Cross-Platform Compatibility: Evaluations may involve ensuring agents operate across multiple operating systems and versions.
Tools such as Windows UI Automation, macOS Accessibility APls, and Limux's AT-SPI are used to automate and monitor agent intoractions.
Automation Frameworks:
Resource Usage: Memory and CPU usage are significant metries, particularly for long-running tasks resource-intensive applications.
Each platform presents distinct challenges and opportunities for evaluating GUI agents. Web platforms emphasize scalability and dynamic interactions, mobile platforms focus touch interfaces and performance, and desktop platforms require handling complex workflows and cross-application tasks. Some benchmarks are cross-platform, requiring agents to be robust, adaptable, and capable of generalizing across different environments. All the metrics, measurements, and platforms discussed are essential for a comprehensive evaluation of GUI agents across multiple aspects. Most existing benchmarks rely on them for evaluation.
The evolution of GUI agent benchmarks reflects a broader shift towards more realistic, interactive, and comprehensive evaluation environments:
69

---

Published in Transactions on Machine Learning Research (06/2025)
1. Towards More Interactive and Realistic Environments: Recent advancements in GUI agent benchmarking emphasize the transition from synthetic scenarios to more interactive and realistic environments. This shift is evident in the use of simulators, Docker containers, and real- world applications to create 'live' environments that better mimic genuine user interactions. Such environments not. only provide a more accurate assessment. agent capabilities but also pose new challenges in terms of performance and robustness.
Cross-Platform Benchmarks: The emergence of cross-platform benchmarks that encompass mobile, web, and desktop environments represents significant step towards evaluating the generaliz- ability of GUT agents. However, these benchmarks introduce fundamental challenges unique to each platform. A unified interface for accessing platform-specifie information, such as HTML and DOM structures, could substantially streamline the benchmarking process and reduce implementation efforts. Future work should focus on standardizing these interfaces to facilitate seamless agent evaluation across diverse environments.
3. Increased Human Interaction and Realism: There is a growing trend towards incorporating more human-like interactions in benchmarks, as seen in multi-turn and conversational scenarios. These setups mirror real-world use cases more closely, thereby providing rigorous test of an agent's ability to handle dynamic, iterative interactions. As GUT agents become more sophisticated, benchmarks must continue to evolve to include these nuanced interaction patterns, ensuring agents can operate effectively in complex, human-centric environments.
Scalability and Automation Challenges: Scalability remains a significant concern bench- marking GUI agents. The crcation of realistic tasks and the development of evaluation methods for individual cases often require substantial human effort. Automation of these processes could alleviate some of the scalability issues, enabling more extensive and efficient benchmarking. Future research should explore automated task generation and evaluation techniques to enhance scalability.
5. Emphasis on Safety, Privacy, and Compliance: There is a notable trend towards evaluating GUL agents on safety, privacy, and compliance metrics. These considerations are increasingly important as agents are integrated into sensitive and regulated domains. Encouraging this trend will help ensure that agents not only perform tasks effectively but also adhere to necessary legal and ethical standards. Future benchmarks should continue expand on these dimensions, incorporating evaluations that reflect real-world compliance and data security requirements.
In what follows, we first present a taxonomy of L.M-ppoered GUT agent benchmarks, categorized by target. platform, as shown in Table I
and detail these benchmarks for GUI agents selectively.
8.4 Web Agent Benchmarks
Evaluating GUI agents in web environments necessitates benchmarks that capture the complexities and nuances of web-based tasks. Over the years, several benchmarks have been developed, each contributing unique perspectives and challenges to advance the field. We first provide an overview of these benchmarks in
Tables 51 52 53
54
55 and 56
One of the pioneering efforts this domain MiniWoB++ Shi al. (2017): Liu et al. (2018). focusing on assossing reinforcement learning agents on web-based GUI tasks. introduces realistic interaction scenarios, including clicking, lyping, and navigating web elements, and leverages workflow-guideed exploration (WGE) to improve efficiency in environments with sparse rewards. Agents are evaluated based success rates, determined by their ability to achieve final goal states, highlighting adaptability and robustness across various complexities.
Building upon the need for more realistic environments, Mind2 Wel et al. 2023) represents a significant advancement by enabling agents to handle real-world HTML environments rather than simplified simulations. Established after the advent of LLM&Yan al. (023b)) it offers a large datasct of over 2,000 tasks spanning multiple domains, presenting challenges from basic actions to complex multi-page workflows. The benchmark
70

---

Published in Transactions on Machine Learning Research (06/2025)
emphasizes end-to-end task performance through metries like Element Accuracy and Task Success Rate, encouraging rigorous evaluation of agents.
Extending Mind2Web's capabilities, MT-Mind2Web Deng et al. (2024c) introduces conversational web navigation, requiring sophisticated interactions that span multiple turns with both users and the environment. This advanced benchmark includes 720 web navigation conversation sessions with 3,525 instruction and action sequence pairs, averaging five user-agent interactions per session, thereby testing agents' conversational abilities and adaptability.
To further enhance realism, WebArena Zhou et al. sets new standard with its realistic web environment that mimics genuine human interactions. Featuring 812 tasks across multiple domains, it requires agents to perform complex, long-horizon interactions over multi-tab web interfaces. By focusing on functional correctness rather than surface-level matches, WebArena promotes thorough assessment of agents" practical abilitics.
Recognizing the importance of multimodal capabilities, VisualWeb an extension of WebAreng Zhou et al., was designed to assess agents on realistic visually grounded web tasks. Comprising 910 diverse tasks in domains like Classifieds, Shopping, and Reddit, it adds new visual functions for mcasuring open-ended tasks such as visual question answering and fuzzy image matching, thereby challenging agents in multimodal understanding.
Similarly, Video WebArens Jang et al. (2024) focuses on evaluating agents' abilities to comprehend and interact with video content on the web. It presents 74 videos across 2,021 tasks, challenging agents in video-based information retrieval, contextual reasoning, and skill application. This benchmark highlights critical deficiencies in current models, emphasizing the need for advancements in agentic reasoning and video comprehension.
Complementing this, VesnalWebBench Liu et al. (2024f) offers a multimodal benchmark that assesses understanding, OCR, grounding, and reasoning across website, element, and action levels. Spanning 1.5K samples from real-world websites, it identifies challenges such as poor grounding and subpar OCR with low-resolution inputs, providing a crucial evaluation perspective distinct from general multimodal benchmarks.
Beyond the challenges of multimodality, understanding agents' resilience to environmental distractions is crucial. Enviiraaattion Ma et al. (2024c) introduces a benchmark that evaluates the faithfulness of multimodal GUI agents under non non-malicious distractions, such as pop-ups and recommendations. The study demonstrates that even advanced agents are prone to such distractions, revealing vulnerabilitics that necessitate robust multimodal perception for reliable automation.
Focusing on safety and trustworthiness, Bentteh Levy et al. 2024) takes a unique approach by emphasizing the management of unsafe behaviors in enterprise settings. It features human-in-the-loop system and a policy-driven hierarchy, introducing the Completion under Policy (CuP) metric to evaluate agents' compliance with organizational, user, and task-specific policies. This benchmark operates in web environments using BrowserGym Chezelles et al. (2024) and includes 235 tasks with policies addressing various safety dimensions, providing a comprehensive framework for evaluating agents in enterprise scenarios.
Addressing the automation of enterprise software tasks, WorkArena Drouin al. (0244) offers a benchmark emphasizing tasks commonly performed within the ServiceNow platform. With 19,912 unique instances across 33 tasks, it highlights the significant, performance gap between current state-of-the-art agents and human capabilities in enterprise UI automation, setting a trajectory for future innovation.
BrowserGym Chezelles et al. (2024) builds ecosystem designed for web agent research. It unifies various benchmarks like MiniWoB(+ +++ Liu et al. (2018), WebArena Zhou et al., and WorkArena Drouin et al. (2024) under a single framework, addressing the issue of fragmentation in web agent evaluation. By leveraging standardized observation and action spaces, it enables consistent and reproducible experiments. BrowserGym'' extensible architecture make it a vital tool for developing and testing GUT-driven agents powered by LLMs, significantly accelerating innovation in web automation rescarch.
In the realm of interacting with live websites, WebOlympus Zheng et al. 2024bb introduces an open platform that enables web agents to interact with live websites through a Chrome extension- based interface.
71

---

Published in Transactions on Machine Learning Research (06/2025)
Supporting diverse tasks and integrating safety monitor to prevent harmful actions, it promotes safer automation of web-based tasks and provides critical tool for evaluating agent performance in realistic scenarios.
Collectively, these benchmarks have significantly contributed to advancing the evaluation of web-based GUI agents, each addressing different aspects such as realism, multimodality, safety, and enterprise applicability. Their developments reflect the evolving challenges and requirements in creating sophisticated agents capable of complex web interactions.
8.5 Mobile Agent Benchmarks
Evaluating GUI agents on mobile platforms presents unique challenges due to the diversity of interactions and the complexity of mobile applications. Several benchmarks have been developed to address these challenges, each contributing to the advancement of mobile agent evaluation. We first provide an analysis for these mobile benchmarks in Tables 57
58
59
and 60
An early effort in this domain is PIXELHELP Li et al. (2020a). which focuses on grounding natural language instructions to actions mobile user interfaces. Addressing the significant challenge of interpreting and executing complex, multi-step tasks, PIXELHELP provides a comprehensive dataset pairing English instructions with human-performed actions on a mobile UI emulator. It comprises 187 multi-step instructions across four task categories, offering a robust resource for evaluating models on task accuracy through metrics like Complete Match and Partial Match.
Building upon the need for systematic evaluation, ANDROIDLAB Xu et al. (2024i) establishes compre- hensive framework for Android-based autonomous agents. Tt introduces both an action space and operational modes that support consistent evaluations for text-only and multimodal models. By providing X.ML and SoM operation modes, ANDROIDLAB allows LLMs and LMMs to simulate real-world interactions in equiv- alent environments. The benchmark includes 138 tasks across nine apps, encompassing typical Android functionalities, and evaluates agents using metrics such as Success Rate and Reversed Redundancy.
To further challenge agents in handling both API and UI operations, Mobile-Bench Deng et al. (0244a) offers an innovative approach by combining these elements within realistic Android environment. Its multi-app sctup and three distinct task categories test agents' capabilitics in handling simple and complex mobile interactions, pushing beyond traditional single-app scenarios. The evaluation leverages CheckPoint metrics, assessing agents at each key action step, providing insights into planning and decision martiinnmaiingggg skills.
Emphasizing safety in mobile device control, MobileSafetyyBenchh Lee et al. (2024a) provides a structured evaluation framework that prioritizes both helpfulness and safety. It rigorously lests agents across common mobile tasks within an Android emulator, focusing on layered risk assessment, including legal compliance and privacy. A distinctive feature is its indirect prompt injection test to probe agent robustness. The evaluation ensures agents scored practical success while managing risks, advancing research in LLM reliability and secure autonomous device control.
Expanding the scope to multiple languages and application scenarios, SPA-BENCH Chen et al. (024dd) introduces an extensive benchmark for smartphone agents. It assesses both single-app and cross-app tasks in a plug-and-play framework that supports seamless agent integration. With a diverse task collection across Android apps, including system and third-party apps, SPA-BENCH offers a rcalistic testing environment measuring agent capabilities in understanding UIs and handling app navigation through metrics like success rate, efficiency, and resource usage.
Focusing on efficient and user-friendl evaluation, MobileAgent Wang et al. (0024 presents a benchmark tailored for agents on Android devices. It offers a fully aulonomous testing process leveraging final UI state matching and real-time app event tracking. With 100 tasks across 10 open-source Android applications categorized by difficulty, accommodates multiple paths to success, enhancing reliability and applicability. Comprehensive metrics, including task success rate, efficiency, latency, and token cost, provide insights into agent performance.
72

---

Published in Transactions on Machine Learning Research (06/2025)
Complementing these efforts, LlamaTouch Zhang et al. (2024h) introduces benchmark and testbed for mobile UT task automation in real-world Android environments. Emphasizing essential state annotation, it enables precise evaluation of tasks regardless of execution path variability or dynamic UI elements. With 496 tasks spanning 57 unique applications, LlamaTouch demonstrates scalability and fidelity through advanced matching techniques, integraling pixel-level screenshots and lextual screen hierarchies, reducing false negalives and supporting diverse task complexities.
Zhao et al., introduce GTArena Zhao et al. (024b), formalized framework and benchmark designed to advance autonomous GUI testing agents. GTAArena provides a standardized evaluation environment tailored for multimodal large language models. Central to its design is the novel Transition Tuple data structure, which systematically captures and analyzes GUI defects. The benchmark assesses three core tasks test intention generation, task execution, and defect detection using a diverse dataset comprising real-world, artificially injected, and synthetic defects, establishing G'Arrena as a pioneering benchmark for GUT testing agents.
Collectively, these benchmarks have significantly advanced the evaluation of based GUI agents, addressing challenges in task complexity, safety, efficiency, and scalability, Their contributions are instrumental in developing more capable and reliable agents for mobile platforms.
8.6 Computer Agent Benchmarks
Evaluating GUI agents on desktop computers involves diverse applications and complex workflows. Several benchmarks have been developed to assess agents' capabilities in these environments, each addressing specific challenges and advancing the field. We overview benchmarks for computer GUI agents in Tables 61 and 62
An early benchmark in this domain is Act2Cap Wu et al. (2024b), which focuses on capturing and narrating GUT actions in video formats using a cursor a pivotal visual guide. Act2Cap emphasizes the detailed nuances of GUI interactions, particularly mussr--based actions like clicks and drags, essential for advancing automation capabilities in GUI-intensive tasks. It includes a substantial datasct of 4,189 samples across various Windows GUT environments, employing metrics based on element-wise Intersection over Union to evaluate semantic accuracy and temporal and spatial precision.
To provide a scalable and genuine computer environment for multimodal agents, OSWorld Xie et al. (2024b) introduces a pioneering framework that supports task setup, execution- based evaluation, and interactive learning across multiple operating systems, including Ubuntu, Windows, and macOS. OSWorld serves as unified environment that mirrors the complexity and diversity of real-world computer use, accommodating arbitrary applications and onen-ended computer tasks. It includes comprehensive suite of 369 tasks on Ubuntu and 43 tasks on Windows, utilizing execution-based evaluation metrics like success rate for rigorous assessment.
Building on OSWorld, Windows Booatti et al. (2024) adapts the framework to create over 150 diverse tasks specifically for the Windows operating system. Focusing on multi-modal, multi-step tasks, it requires agents to demonstrate abilities in planning, screen understanding, and tool usage within real Windows environment. Addressing the challenge of slow evaluation times, WindowsArena enables parallelized deployment in the Azure cloud, drastically reducing evaluation time and allowing for comprehensive testing across various applications and web domains.
Focusing on office automation tasks, OFFICEBENCH Wang et al. (202pp) introduces a groundbreaking framework for benchmarking LLM agents realistic office workflows. Simulating intricate workflows across multiple office applications like Word, Excel, and Email within a Linux Docker environment, it evaluates agents' proficiency in cross-application automation. The benchmark challenges agents with complex tasks at varying difficulty levels, demanding adaptability to different complexities and use cases. Customized metrics assess operation accuracy and decision-making, providing critical insights into agents' capabilities in managing multi-application office scenarios.
Addressing the automation of data science and enginecring workflows, Spider2-V V Cao et al. (2024) offers a distinctive benchmark. It features 494 real-world tasks across 20 enterprise-level applications, spanning the entire data science workflow from data warehousing to visualization. Assessing agents' abilities to handle
73

---

Published in Transactions on Machine Learning Research (06/2025)
both code generation and complex GUI interactions within authentic enterprise software environments Ubuntu, it employs a multifaceted evaluation method that includes information-based validation, file-based comparison, and execution- based verification.
In the realm of productivity software, AssistGUI Gao et al. (2024b) provides a pioncering framework for evaluating agents' capabilities. It introduces an Actor-Critic Embodied Agent framework capable of complex hierarchical task planning, GUI parsing, and action generation. The dataset includes diverse tasks across design, office work, and system settings, supported by project files for reproducibility. By emphasizing outcome-driven evaluation with pixel-level precision and procedural adherence, AssistGUI highlights the potential and limitations of current LLM-based agents in managing intricate desktop software workflows.
WorldGUI Zhao et al. (2025c) is a benchmark designed to evaluate GUI agents under dynamic conditions on the Windows platform. Unlike previous static benchmarks, introduces varied initial states to simulate real-world interactions across both desktop and web applications. Rather than always starting from a fixed default state, agents must. adapl to changing UI layouts, user interactions, system settings, and pre-existing conditions, requiring robust adaptability to perform effectively. The benchmark comprises 315 tasks spanning 10 popular software applications and incorporates instructional videos, project files, and multiple pre-action scenarios, providing a comprehensive and realistic evaluation framework for assessing an agent's 's ability to handle complex task execution.
Computer Agent Arena Wang et al. (2025a) presents a new paradigm for benchmarking LLM-based GUI agents through live, user-configured desktop environments. Unlike traditional static datasets, provides interactive cloud-based infrastructure where agents are evaluated on tasks spanning web browsing, programming, and productivity using real applications like Google Docs, VSCode, and Slack. Its innovation lies in using head-to-head agent comparisons, human judgment, and Elo-based ranking to evaluate general- purpose digital agents in realistic settings. The benchmark supports Windows and Ubuntu, with MacOS support planned, and allows customized task scenarios with diverse software and website sctups. By enabling crowdsourced evaluations and planning open-source releases, it fosters community-driven improvements and robust comparisons.
Collectively, these benchmarks provide comprehensive evaluation frameworks for GUI agents on desktop platforms, addressing challenges in task complexity, pross-application automation, scalability, and fidelity. Their contributions are instrumental in advancing the development of sophisticated agents capable of complex interactions in desktop environments,
8.7
Agent Benchmarks
To develop GUI agents capable of operating across multiple platforms, cross-platform benchmarks are essential. These benchmarks challenge agents to adapt to different environments and interfaces, evaluating their versatility and robustness. We provide an overview of benchmarks for cross-platform GUI agents in Tables 63
Addressing this need, TisualAgent.Beccl (VAB) Liu et al. (0224) represents a pioneering benchmark for evaluating GUI and multimodal agents across a broad spectrum of realistic, interactive tasks. Encompassing platforms such as Web (WebArena-Litt Zhou et al.), Android Mobiee Xu et al.] (2024i)), ), and game environments, VAB focuses on vision-based interaction and high-level decision-making tasks. The benchmark employs a multi-level data collection strategy involving human demonstrations, program-bass based solvers, and model bootstrapping, Evaluation metries concentrale on success rates, ensuring comprehensive performance assessments in tasks like navigation and content modification, thereby filling significant gap in benchmarking standards for GUI-based LLM agents.
Complementing this, CRAB Xu el al. (202g) introduces an innovative benchmark by evaluating multimodal language model agents in cross-envine coos-environment interactions. It uniquely supports seamless multi-device task execution, evaluating agents in scenarios where tasks span both Ubuntu Limux and Android environments. By introducing a graph-based evaluation method that breaks down tasks into sub-goals and accommodates multiple correct paths to completion, CRAB provides nuanced assessment of planning, decision-making,
74

---

Published in Transactions on Machine Learning Research (06/2025)
and adaptability. Metrics such as Completion Ratio, Exccution Efficiency, Cost Efficiency, and Success Rate offer comprehensive insights into agent performance.
Focusing on GUI grounding for cross-llafform visual agents, ScreenSpot Cheng et al. (2024a) offers a comprehensive benchmark emphasizing tasks that rely on interpreting screenshots rather than structured data. SpotenSpot includes over 600 screenshots and 1, 1,200 diverse instructions spanning mobile (iOS, Android), desktop (macOS, Windows), and web platforms. evaluates click accuracy and localization precision by measuring how effectivcly agents can identify and interact with GUI elements through visual cues alone. By challenging models with a wide variety of UI elements, ScreenSpot addresses real-world complexities, making it an essential resource for evaluating visual GUI agents across varied environments.
These pross-platform benchmarks collectively advance the development of GUT agents capable of operal- ing seamlessly across multiple platforms. By providing comprehensive evaluation frameworks, they are instrumental in asscssing and enhancing the versatility and adaptability of agents in diverse environments.
9 Applications of LLM-Powered GUI Agents
As LLM-powered GUI agents continue to mature, a growing number of applications leverage this concept. to create more intelligent, user-friendly, and natural language-driven interfaces. These advancements are reflected in research papers, open-source projects, and industry solutions. Typical applications encompass (i) GUI testing, which has transitioned from traditional script-based approaches to more intuitive, natural language-based interactions, and (ii) virtual assistants, which automate users' daily tasks in a more adaptive and responsive manner through natural language interfaces.
The application of LLM-powered GUI agents has ushered in new capabilities and interfaces for tasks such as GUT testing and virtual assistance, introducing natural language interactions, enhanced automation, and improved accessibility across platforms. These agents are transforming the way users interact with software applications by simplifying complex tasks and making technology more accessible. However, despite these advancements, I.L.-powered GUT agents are still in their infancy, and several challenges need to be addressed for them reach maturity. Key insights from recent developments include:
Natural Driveen Intcractions: L.M-ppoored GUI agents have enabled users to interact with applications using natural language, significantly lowering the barrier to entry for non-expert users. In GUI testing, tools like GPTDroid Liu et al.] (2024k) and AUlTestAgent Hu et al.) (2024c) allow testers to specify test cases and requirements in plain language, automating the execution and verification processes. Similarly, virtual assistants like LLMPA Guan et al. (2024a) and ProAgent Yo et al. (023) interpret user commands to perform complex tasks, showcasing the potential of natural language interfaces in simplifying user interactions across platforms.
These agents have demonstrated the ability to automate multi-step and intricate workflows without the need for manual scripting. Projects like AutoTaak Pan et al. (2023b) and Voieoiicaasker Vu et al. (2024) autonomously explore and interact. with GUI environments, executing tasks based on high- goals or voice commands. In GUI testing, agents have improved coverage and efficiency by automating the generation of test inputs and reproducing bugs from textual descriptions, as seen in CrashTranslatte Huang et al. (024cc and
Enhanced Automation of Complex Tasks:
AdbGPT Feng & Chen (202))
3. Multimodal Perception and Interaction: Integrating visual and textual inputs has enhanced the agents' understanding of GUI contexts, leading to better decision-making and interaction accuracy. Agents like VizAbility Gorniak et al. (2024) and OpenAdapt OpenAdapt Al (0244) utilize screenshots, OpeenAdat/(penAAdapp
UT trees, and OCR to perceive the environment more comprehensively. This multimodal approach is crucial for applications that require precise identification and manipulation of GUI elements, especially in dynamic or visually complex interfaces.
4. Improved Accessibility and User Experience: Lowered GUI agents have contributed making technology more accessible to users with disabilities limited technical proficiency. Tools like
75

---

Published in Transactions on Machine Learning Research (06/2025)
Table 15: Taxonomy of application scenarios for GUI agents.
Application Area
References
GUI Testing
Zimmermann & Koziolek (2023b) ; Liu et al.
(0244)] 1; Yoon et al.
(2024 Hu
et al. (2024c); Liu et al.
(20241)
; Taeb et al.
(2024): Cui et al. (2024); : Liu
et al. (002cc):
Huang et al. [2024c Feng & Chen [2024); Ding et al. (024b);
Beyzaci ct al.
(2024): Lu ct al. (2025a): Ran et al. (2024 0: Li et al. (0244h)
Demissie el al.
(2025) Yapageı et al. (2025): Li et al. (2025c); Chevrol et al.
(002
Rosenbach et al. (2025): ; Kong et al. Feng et al. (2025)
Virtual
Assis-
Microsoft
Gorniak et al. (2024); Ye et al. (2023)
: Guan et al. 2024a); ;
tants
Vu et al.
2024); Pan et al. (2023b] ): Gao et al.
[024a]): Huang et al.
2024a
Gao et al.
(024gg); : OppenAappt Al (2024); AgentSeaf Al (2024); Interpreter
b: MultiOn AT (2024): : HONOR (2024); Srinivasan & Patapati (2025)
Step i: Click on the Accrowbilty" section In the left hand navigation menu. Actiono Clck(Menultem" Accessi Iky"))
Click TText from the lst opsiion. Action: Click((tenntteerr) fiest "ize"))
Move the "Text size" siidero the Settiier(SSdee sire"), 175)
Test Caser Adjust the font size to 1758 in the Windows Settings and verify that all displayed
across scale up -ppropriately.
Stop 4 the "Aopl"" button confirm in fors size edqustment. Action: Click(Button" ))
Step 5: Wall for the screen to complete and changes to apply. Action: War((s)
Step Vertfy sire has been successfully adjusted and applied. Action: Var ilyy Setting Text size", 175)
GUI Agennt Tester
Newvvt
Figure 28: An example of testing font size adjustment using an LLM-powered GUI agent.
VizAbility Gorniak et al. 20244] aid blind and low-vision users in understanding data visualizations, while Ask Gao et al. 2024g) assists older adults in navigating smartphone functions. By tailoring interactions to the needs of diverse user groups, these agents enhance inclusivity and user experience.
Next, we present a taxonomy of application scenarios for GUI agents, categorized by application area, shown in Table 15 followed by a detailed discussion of reprosentative works.
9.1
GUI Testing
GUI testing evaluates a software application's graphical user interface to ensure compliance with specified requirements, functionality, and user experience standards. It verifies interface elements like buttons, menus, and windows, as well as their responses to user interactions. Initially conducted manually, GUI testing evolved with the advent of automation tools such as Selenium and Appium, enabling testers to automate repetitive tasks, increase coverage, and reduce testing time Wang et al. (2024ch; Yu et al. (2023). However, L.L.-powered GUT agents have introduced a paradigm shift, allowing non-experts to test GUTs intuitively
76

---

Published in Transactions on Machine Learning Research (06/2025)
through natural language interfaces. These agents cover diverse scenarios, including general testing, input generation, and bug reproduction, without the need for traditional scripting Wang et al. (2024ch.
Figure 28 and illustrates the use of an LLM-powered GUI agent to test font size adjustment on Windows OS. With only a natural language test case description, the agent autonomously performs the testing by executing UT operations, navigating through the settings menu, and leveraging its screen understanding capabilities to verify the final outcome of font size adjustment. This approach dramatically reduces the effort required for human or script-based testing. Next, we detail the GUI testing works powered by GUI agents, and first provide an overview Tables 64
6a
66
and
9.1.1 General Testing
Early explorations demonstrated how LLMs like GPT-3 could automate GUT testing by interpreting natural language test. cases and programmatically executing them. For example, one approach integrates GUL states with GPT-3 prompts, leveraging tools like Selenium and OpenCV to reduce manual scripting and enable black box testing Zimmermann & Koziolek (2023a). Building this, subsequent study employed GPT-4 and Selenium WebDriver for web application testing, achicving superior branch coverage compared to traditional methods like monkey testing Zimmermann & Koziolek (2023b). These advances highlight how LLMs simplify GUI testing workflows while significantly enhancing coverage and efficiency.
Further pushing boundaries, GPTDroid reframed GUI testing as an interactive Q&A task. By extracting structured semantic information from GUI pages and leveraging memory mechanisms for long-term exploration, it increased activity coverage by 32%, uncovering critical bugs with remarkable precision Liu et al. (2024k). This approach underscores the potential of integrating conversational interfaces with memory for comprehensive app testing. For Android environments, DROIDAGENT introduced an intent-driven testing framework. It automates task generation and execution by perceiving GUT states in JSON format and using LLMs for realistic task planning. Its ability to set high-level goals and achieve superior feature coverage demonstrates how intent-based testing can transform functional verification in GUI applications Yoon et al. (2024).
Prophet.gent Kong al. introduces a novel approach to I.L--powrred GUT testing by automatically synthesizing Android application test scripts from natural language descriptions. Departing from previous methods that directly apply LLMs to GUI screenshots or app behaviors, ProphetAgent builds Clustered UI Transition Graph (CUTG) enriched with semantic annotations. This structured representation enables more accurate mapping between natural language test steps and GUI operations, lcading to significant improvements in completion rate (78.1%) and action accuracy (83.3%). The system employs a dual-ageet. architecture: SemanticAgent handles semantic annotation, while Generation generates executable scripts. ProphetAgent demonstrates strong scalability and real-world applicability reducing tester workload by over 70% at ByteDance. Its performance underscorss the effectiveness of combining LLMs with explicit semantic knowledge graphs in GUI-based environments.
AUTTestAgent extended the capabilities of LLM-powered GUI testing by bridging natural language-driven requirements and GUI functionality Hu et al. (2024c). Employing multi-modal analysis and dynamic agent organization, it efficiently executes both simple and complex testing instructions. This framework highlights the value of combining multi-source data extraction with robust language models to automate functional testing in commercial apps. Incorporating vision-based methods, VisionDroid redefined GUI testing by aligning screenshots with textual contexts to detect non-crash bugs Liu et al (20241). This innovation ensures application reliability by identifying logical inconsistencies and exploring app functionalities that conventional methods often overlook.
Accessibility testing has also benefited from LLM-powered agents. AXNav addresses challenges in iOS accessibility workflows, automating tosts for features like VoiceOver and Dynamic Type using natural language instructions and pixel- based models. Its ability to generate annotated videos for interactive review positions A AXNav a scalable and user-friendly solution for accessibility testing Taeb al. (2024).
77

---

Published in Transactions on Machine Learning Research (06/2025)
9.1.2 Text Input generation
In the realm of text input generation, Cui et al., demonstrated how GPT-3.5 and GPT-4 could enhance Audroid app lesting by generating context-aware text inpuls for UI fields Cui et al. (024). By systematically evaluating these inputs across multiple apps, they revealed the potential of LLMs in improving test coverage and detecting unique bugs with minimal manual intervention. Similarly, QTypist formulated text input generalion as fillin-the-blank lask, leveraging LLMs to improve activily and page coverage by up to 52% Liu et al. 2023c
9.1.3 Bug Replay
For bug reproduction, Tranharaslltoor automated the reproduction of crashes from stack traces by integrating reinforcement learning with LLMs. Its iterative navigation and crash prediction steps significantly reduced debugging time and outperformed state-of-the-art methods Huang et al. (2024c). Mcanwhile, AdbGPT demonstrated how few-shot learning and chain-of-thought reasoning could transform textual bug reports into actionable GUI operations. By dynamically inferring GUI actions, AdbGPT provided an efficient and lightweight solution for bug replay Feng Chen (2024).
BugCraft Yapageı el al. 2025 leverages LLM-powered GUI agents to automate bug reproduction in games, specifically targeting the open-ended and complex environment of Minecraft. Tt employs GPT-4o the inference engine, integrating textual bug reports, visual GUI understanding through OmniParser Lu et al.) (2024d), and external knowledge from the Minecraft Wiki to generate and execute structured reproduction steps. Actions are carried out via a custom Macro API, enabling robust interaction with both the game's GUI and environment. BugCraft's 's ability to translate unstructured bug descriptions into executable in- in-game behaviors highlights the strong potential of vision-enhanced LLM agents for advancing software testing and debugging.
9.1.4 Verification
Finally, as a novel application in testing, Magic showcased the potential of LLMs in automating "How-lo" verifications. By extracting, executing, and refining instructions from search engines, it addressed critical challenges in user-centric task automation, improving the reliability of GUI-driven workflows Ding et al. 2024b).
In summary, LLM-powered GUI agents have revolutionized GUT testing by introducing natural language- driven methods, vision-based alignment, and automated crash reproduction. These innovations have enhanced test coverage, efficiency, and accessibility, setting new benchmarks for intelligent GUI testing frameworks.
9.2 Virtual Assistants
Virtual assistants, such as Sin 33
are Al-driven applications that help users by performing tasks, answering questions, and executing commands across various platforms, including web browsers, mobile phones, and computers. Initially, these assistants were limited to handling simple commands via voice or text input, delivering rule-based responses or running fixed workflows similar to RPA. They focused on basic tasks, such as setting alarms or checking the weather.
With advancements in LLMS and agents, virtual assistants have evolved significantly. They now support more complex, context-aware interactions device GUIs through textual voice commands and provide personalized responses, catering to diverse applications and user needs on various platforms. This progression has transformed virtual assistants from basic utilities into intelligent, adaptive tools capable of managing intricate workflows and enhancing user productivity across platforms. Figure 29 presents a concoptual example of a GUT agent-powered virtual assistant on a smartphone 31 In this scenario, the agent enables users to interact through chat, handling tasks such as setting up a screenshot shorteut on their behalf. This
Pttttss://wret http::/wwwwwpppll.../////app www.appppp.s.iia...eae con/siri/
34The application and sccuario depicted in the figure conceptual and fabricated. They do not reflect the actual functionality of any specific smartphone. Readers should consult the phone manual or official guidance for accurate information on Al assistant. rapabilities.
78

---

Published in Transactions on Machine Learning Research (06/2025)
Use Back Top on your ihhone
User
Can you set a shortcut for taking screenshots on my phone?
Sure! You shortcut take screenshots double-tan double-tapping or triple tapping the back of your phone. Would you like set it to double-tap?
Turn onmennnBel Back Tap
Can be set to double-tap?
Absolutely! Let take care of that for you.
...tter settingg shartsut...
The shortcut has been successfully set! screenshot by double tapping the back of your phone.
take
GUI Agent Virtual Assistant
2010
Settinge
Figure 29: A conceptual example of a GUI peent-powered virtual assistant on smartphone.
feature is particularly beneficial for users unfamiliar with the phone's functionalities, simplifying complex tasks into conversational commands.
To explore more real-world applications of virtual assistants powered by GUI agents, provide overview of advancements across research, open-source initiatives, and production-level applications, as summarized in Table 68 69 and 70
9.2.1 Research
Recent research efforts have significantly advanced the capabilities of virtual assistants by integrating LLM-powered GUI agents, enabling more intelligent and adaptable interactions within various applications.
Firstly, the integration of LLMs into GUI-based automation has been explored to enhance business process automation. For instance, Ye et al. (2023) introduces Agentic Process Automation through the development. of ProAgent, which automates both the creation and execution of workflows GUT environments. By utilizing agents like ControlAgent and DataAgent, supports complex actions such as dynamic branching and report generation in applications like Slack and Google Sheets. This approach transcends traditional RPA by enabling flexible, intelligent workflows, significantly reducing the need for manual intervention and highlighting the transformative potential of LLM-powered agents in virtual assistants.
Building upon the idea of integrating LLMs with GUI environments, researchers have focused on mobile plat forms automate complex tasks. LLMPA Guan et al. (2024a) a pioneering framework that leverages LLMs to automate multi-stop tasks within mobile applications like Alipay. It interacts dircctly with app GUIs, mimicking human actions such clicks and typing, and employs UI tree parsing and object detection for precise environment understanding. A unique controllable calibration module ensures logical action execution, demonstrating the potential of L.L--pewered virtual assistants to handle intricate workflows and real-world impact in assisting users with diverse tasks.
Similarly, the automation of smartphone tasks through natural language prompts has been addressed by PromptRPA Huang et al. (2024a). Utilizing multi-agent framework, it automates tasks within smartphone GUI environments, tackling challenges like interface updates and user input variability. Advanced perception methods, including OCR and hierarchical GUT analysis, employed to understand and interact with
79

---

Published in Transactions on Machine Learning Research (06/2025)
mobile interfaces. By supporting real-time feedback and iterative improvements, PromptRPA underscores
the importance of user-centered design in LLM-driven virtual assistants.
In the realm of accessibility, LLM-powered GUI agents have been instrumental in enhancing user experience for individuals with disabilities. For example, VizAbility Gorniak et al. (2024) enhances the accessibility of data visualizations for blind and low-vision users. By combining structured chart navigation with LLM-based conversational interactions, users can ask natural language queries and receive insights on chart content and trends. Leveraging frameworks like O and chart specifications such as Vi VizAbility allows exploration of visual data without direct visual perception, addressing real-world accessibility challenges in GUTs.
Furthermore, addressing the needs of older adults, Easy Ask Gao et al. (2024g) serves as a context-aware in-app assistant that enhances usability for non-technical users. By integrating muuli-modaa inputs, combining natural voice queries and touch interactions with GUI clements, it generates accurate and contextual tutorial searches. AskyAsk demonstrates how GUI agents can enhance accessibility by integrating contextual information and interactive tutorials, empowering users to navigate smartphone functions effectively.
Voice interaction has also been a focus arca, with tools like GPTVoiceTasker Vu et al. 2024) facilitating hands-free interaction with Android GUIs through natural language commands. It bridges the gap between voice commands and GUI-based actions using real-time semantic extraction and a hierarchical representation of UI elements. By automating multi-step tasks and learning from user behavior, it enhances task efficiency and reduces cognitive load, highlighting the transformative potential of LLMs in improving accessibility and user experience in mobile environments.
Expanding voice-powered interactions, Auto Pan et al. 2023b) enables virtual assistants to execute multi-step tasks in GUI environments without predefined scripts. It autonomously explores and learns from mobile GUIs, effectively combining voice command interfaces with dynamic action engines to interact with GUI elements. Utilizing trial-and-crror and experionco-driven learning, AutoTask adapts to unknown tasks and environments, showcasing its potential in enhancing voice-driven virtual assistants for hands-free interactions.
Finally, in the domain of creative workflows, AssistEditor Gao et al. (0244)) exemplifies multi-agent framework for automating video editing tasks. By interacting with GUI environments, it autonomously performs complex workflows using dialogue systems and video understanding models to bridge user intent with professional editing tasks. The innovative use of specialized agents ensures efficient task distribution and execution, demonstrating the practical application LLM-powered GUI agents in real-world scenarios and expanding automation into creative domains.
These research endeavors collectively showcase significant advancements in LLM-powered GUI agents, highlighting their potential to transform virtual assistants into intelligent, adaptable tools capable of handling complex tasks across various platforms and user needs.
9.2.2 Open-Source Projects
In addition to research prototypes, open-source projects have contributed substantially to the development and accessibility of LLM--rained GUI agents, enabling wider adoption and customization.
One such project OpenAdapt OpenAdapt Al 2024). : an open-source framework that utilizes large multimodal models to automate tasks by observing and replicating user interactions within GUT environments. It captures screenshots and records user inputs, employing computer vision techniques to understand and execute standard UI operations. Designed streamline workflows across various industries, OpenAdap learns from user demonstrations, thereby reducing the need for manual scripting and showcasing adaptability in GUI-based task automation.
Similarly, AgentSea Agent Seal AT 2024) offers a comprehensive and modular toolkit for creating intelligent agents that can navigate and interact with various GUI environments across multiple platforms. Its flexibility
https://nitvis. hwtp:aittwwiiiiiiiiooeeeeeoreeee github.io/olll/ 1o/olli/ 36 haap:::://eaaggthu.. 10/
80

---

Published in Transactions on Machine Learning Research (06/2025)
is particularly beneficial for developing virtual assistants capable of automating complex tasks within applications, enhancing user productivity. By adhering to the UNIX philosophy, AgentSea ensures that each tool is specialized, promoting ease of use and extensibility. Its open-source nature fosters community collaboration and innovation in Al-driven GUI automation.
Open Interpreter Interpreter (2024) further exemplifies the potential of open-source contributions by leveraging large language models to execute code locally. Users can interact with their computer's GUL through natural language commands, supporting multiple programming languages and operating across various platforms. By facilitating tasks such data analysis, web automation, and system management, Open Interpreter provides unrestricted access to system resources and librarics, enhancing flexibility and control. Its customization capabilities make it a valuable asset for users aiming to streamline operations through AI Al-powered virtual assistance.
These open-souree projects not only advance the state of LLM-powerod GUI agents but also democratize access to intelligent virtual assistants, enabling developers and users to tailor solutions to specific needs and applications.
9.2.3 Production
The integration of LLM--brinnd GUI agents into production environments demonstrates their practical viability and impact on enhancing user experiences in commercial applications.
Power Automate Microsoft (2024) exemplifies an AT-powered GUT agent that enhances user interaction with desktop applications. By allowing users to describe tasks in natural language while recording actions, it translates these descriptions into automated workflows, effectively bridging the gap between user intent. and execution. Its ability to record and replicate user actions within the GUI streamlines the automation of repetitive tasks, making it valuable tool for incrcasing efficicncy and highlighting advancements in user-friendly automation solutions.
In the realm of web interactions, MultiOn MultiOn Al (024) serves as a personal Al agent that autonomously interacts with web-based GUIs to exccute usor-defined tasks. Leveraging large language models, interprets natural language commands and translates them into precise web actions, effectively automating complex repetitive tasks. MultiOn's approach to perceiving and manipulating web elements enables scamless functioning across various web platforms, enhancing user productivity and streamlining web interactions.
On mobile platforms, the YOYO Agent in MagicOS HONOR (2024) exemplifies an LLM-powered GUI agent operating within the MagicOS 9.0 interface. Utilizing Honor's 's MagicLM, it comprehends and executes user commands across various applications, learning from user behavior to offer personalized assistance. This integration demonstrates how large language models can enhance virtual assistants, enabling them to perform complex tasks within GUI environments and improving user experience and productivity on mobile devices.
Eko AI 2025) serves a prime example of a versatile and efficient tool for developing intelligent agents capable of interacting with GUTs across various platforms. Its integration with multiple LLMSs and the innovative Visual-Interactive Element Perception (VIEP) technology highlight its capability to perform complex tasks through natural language instructions. Eko's comprehensive tool support make it a valuable resource for developers aiming to create customizable and production-ready agset- workflows. By facilitating seamless interaction within GUI environments, Éko exemplifies the advancements in virtual assistants powered by LLMs.
These production--evvll inplementations highlight the practical applications and benefits of LLM-brained GUT agents in enhancing automation, productivity, and user engagement across different platforms and industries.
10 Limitations, Challenges and Future Roadmap
Despite significant advancements in the development of L.M--brained GUI agents, important to acknowl- odge that this field is still in its infancy. Several technical challenges and limitations hinder their widespread adoption in real-world applications. Addressing these issues is crucial to enhance the agents' effectiveness,
81

---

Published in Transactions on Machine Learning Research (06/2025)
safety, and user acceptance. In this section, we outline key limitations and propose future research directions to overcome these challenges, providing concrete examples to illustrate each point.
10.1 Privacy Concerns
Privacy is critical concern uniquely intensified in the context of LLM-powered GUI agents. These agents often require access to sensitive user data-suce such as screenshots, interaction histories, personal credentials, and confidential documents- to effectively perceive and interact with the GUI environment. In many cases, this data must be transmitted to remote servers for model inference, especially when relying on cloud-basod LLMs Liao et al. (2024): He et al. (2024a); Gan et al. 2024h. Such deployments raise significant privacy risks, including data breaches, unauthorized access, and misuse of personal information. These concerns are further amplified when sensitive inputs are routed through third-parly APIs or processed off-device, creating compliance and security vulnerabilities that can deter real-world adoption.
For instance, GUI agent tasked with managing a user's email inbox may need to read, classify, and respond to messages containing highly personal or confidential content. Offloading this processing to the cloud introduces risks of exposure, prompting hesitation among users and organizations due to potential privacy violations Zharmagambeto et al. ); Yang et al. (202cc); Zhang et al. (2024]), Compared to traditional LLM applications, GUI agents operate at a finer granularity of user activity and often require broader system
access, making privacy-preserving deployment strategies a critical and domain-specifie challenge.
To mitigate privacy concerns, future research should focus enabling on-device inference, where the language model operates directly on the user's device without uploading personal data Xu
Potential Solutions:
b0 Qu et al. 20244. Achieving this requires advancements in model compression techniques Lin et al.
ct al. (204a), on-device optimization Liu et al. (2024j), and efficient inference algorithms Zhou et al. 2022b to accommodate the computational limitations of user devices. In addition, frameworks must incorporate data redaction, secure communication channels, and explicit scoping of data usage within the agent's context. Furthermore, integration with system-level privacy controls and user consent mechanisms (e-g., runtime permission dialogs or sandboxed execution) is essential for deployment in regulated domains.
From the technical perspective, implementing privacy-preservine techniques like federated learning Kuang et al. (2024), differential privacy Mai et al. (2023). and homomorphic encryption de Castro et al. can enhance data security while allowing the model to learn from user data. Furthermore, developers of GUI agents should collaborate with privacy policymakers to ensure that user data and privacy are appropriately protected WWll let al. (2024). They should make the data handling processes transparent to users, clearly informing them about what data are being transmitted and how they are used, and obtain explicit user consent Zhang et al.
(022r)
10.2 Latency, Performance, and Resource Constraints
One challenge that is particularly salient for GUI agents srintts-distnnt from general LLM applications is the issue of latency in interactive, multi-step execution environments. Since GUI agents rely on large language models to plan and issue actions, their computational demands can lead to high latency and slow response times, which directly impact user experience Li et al. (2024a). This is especially critical in time-sensitive or interactive scenarios, where delays in action execution can cause user frustration or even trigger unintended system behavior. Unlike single-shot LLM tasks, GUT agents typically operate over extended sequences of steps, making latency cumulative and more disruptive over time. The problem is further amplified in on-device deployments, where computational resources are limited. For example, running an ALM--powered agent within mobile app may result in sluggish performance rapid battery depletion, significantly undermining usability on resource-constrained platforms Xu et al. (2024f); Chen et al. (002a); Krupp et al. (2025). These concerns are uniquely pronounced in GUI agents due to their need for real-time perception, decision-makkin and UT control in dynamic environments Chen et. al. (2024a).
Potential Solutions: Future work should aim to reduce inference latency by optimizing model architectures for speed and efficiency Wan et al. (2023). Techniques such as model distillation can create smaller, faster models without substantially compromising performance Xu et al. (2024h). Leveraging hardware accelerators
82

---

Published in Transactions on Machine Learning Research (06/2025)
like GPUs, TPUs, specialized AI chips, and exploring parallel processing methods can enhance computational efficiency Kachris (2024). Implementing incremental inference and caching mechanisms may also improve responsiveness by reusing computations where applicable Lee et al. (2024d). Additionally, research into model optimization and compression techniques, such as pruning Wang et al. (001)) and quantization Lin et al. (2024a) can produce light models suitable for deployment on resource-consl rained devices. Exploring edge computing Qu et al. (2024) and distributed inference Wu al. (2023a) can help distribute
the computational load effectively.
Moreover, GUI agents should collaborate with application developers to encourage them to expose high-level native APIs for different functionalities Song et al. (2024b); Lu et al. (2024a), which combine several UT operations into single API calls. By integrating these APIs into the GUI agent, lasks can be completed with fewer steps, making the process much faster and reducing cumulative latency.
10.3 Safety and Reliability
The real-world actuation capabilities of GUT agents introduce unique and significant safety and reliability risks beyond those faced by general-purpose LLMs. Because GUI agents can directly manipulate user aiterfaces-clickeee buttons, delcting files, submitting forms, or initiating system-level operations crrors in action generation can have irreversible consequences Anwar et al. (2024); Gan et al. (2024). These may include data corruption, accidental message dispatches, application crashes, unaut access to sensitive system components Zhong & Wang (2023); Yuan et al. (2024). Such risks are compounded by the inherent uncertainty and non-eeterminism in LLM outputs: agents may hallucinate actions, misinterpret UI contexts,
or behave inconsistently across sessions Zhang et al. (2024) Zhao et. al. (2025b); Zhang et al. (2023d); Chiang et al. ( ; Chen et al. (2025a). For example, an agent automating financial transactions could mistakenly
execute the wrong transfer, leading to material losses. Furthermore, GUI agents expose a. broader attack surface than traditional LLM applications -the are susceptible to black- box adversarial attacks that could manipulate their inputs exploit their decision policies Xu et al. (0024)).
Unlike passive language models, GUI agents operate within dynamic software ecosystems where incorrect actions can propagate across applications or escalate into wystem-wide disruptions. Integration challenges also arise, including compatibility with evolving UT frameworks, user permission boundaries, and software-specific safety constraints, and malicious attacks Yang et al. (2025d); Aichberger et al.) These concerns, coupled with the lack of interpretability and formal guarantees, contribute to skepticism and reluctance from users and developers alike. Addressing safety and reliability in GUI agents thus requires not only robust model behavior but also runtime safeguards Loe et al. (2025). rollback mechanisms, and interfaco-aware verification techniques (ailored specifically to this interaction paradigm.
Potential Solutions: Ensuring safety and reliability necessitates robust error detection and handling mechanisms Pan et al. (2023a). Future rescarch should focus on intograting validation stops that verify the correctness of inferred actions before execution Huang et al. (2023). Developing formal verification methods
Jha et al. 2023), implementing exception handling routines Zhang et al. (2023b), and establishing rollback procedures Koo & Toueg (1986) are essential for preventing and mitigating the impact of errors. Additionally,
incorporating permission management Luo et al. (001): Hao et al. (0013) Felt et al. (2011) Lutaaya (2018)
to limit the agent's 's access rights can prevent unauthorized or harmful operations.
Furthermore, creating standardized interaction protocols can facilitate smoother and safer integration with various applications and systems Xiang et al. 2024). Ensuring that agents comply with security best practices, such as secure authentication and authorization protocols Berkovits et al. [1998), essential.
10.4 Human-Agent Interaction
Human-agent interaction introduces distinct challenges in the context of GUI agents, where the agent and user operate within the same dynamic interface. Any user intervention such as moving the mouse, altering window states, or modifying imputs-cal can inadvertently interfere with the agent's ongoing execution, potentially causing conflicts, unintended actions, or breakdowns in task flow Gao et al. (0024)): Bradshaw et al. 2017
83

---

Published in Transactions on Machine Learning Research (06/2025)
(a) Ask for manual invervention
(b) Handle ambiguity
(c) Request for confirmation
Task: Draft an email to congratulate Tom on his paper about GUI agents being accepted by top conference.
tom@Babc com tomm xBlay..com
GUI Agentt
Great, found two potential matches for your recipient: "tom@abc.comm and "tom s@xxyzcomm' Could you confirm which one is correct, or let me know if it's neither?
Sure! To proceed, could you please log to your Gmail account?
have completed email draft, would you like me to send this email?
User
(...user Logs into their Gmail accounn mankally)
it "tom@abc.com".
Looks good, please send it.
Figure 30: An illustrative example of human-agent interaction for completing an email sending request.
Designing robust collaboration protocols that govern when the agent should yield control, pause execution, or defer to the user is a non-trivial problem specific to GUI-based automation.
Further complicating this interaction is the ambiguity of user instructions. Natural language commands may be vague, under-specifeed, or context-dependent,, leading to misinlerpretattios or incomplete task plans. GUT agents may also encounter runtime uncertainties such as unexpected popups, missing inputs, or conflicting UI states that require them to seek user clarification or feedback Zhang et al. (2024a) : Feng et al. 2024).
Determining when and how an agent should request user input-whether whether for disambiguation, permission, or
verification- is critical for ensuring both reliability and user trust Amayuelas et al. (0223 Shi et al. 2025
This challenge is exemplified in the fabricated scenario shown in Figure 30 where a GUI agent is instructed to send an email to "Tom." The agent must first prompt the user to log in securely, protecting credentials by avoiding automated input. It then encounters ambiguity when multiple contacts named "Tom" " are found, and resolves it by prompting the user to select the intended recipient. Finally, before dispatching the email, the agent requests explicit confirmation, recognizing that email-sending is a non-reversible action with privacy implications Zhang et al. (2024a). Although the task appears simple, it reflects the complexity of real-world human-GUl agent collaboration, involving privacy presorvation, ambiguity resolution, and intentionality confirmation Kim el al. (2024a). These are not generic LLM issues, but domain-specific challenges rooted in shared interaction with sore seerfaces-underscnnee
the need for new design paradigms around shared control, interruption handling, and proactive clarification in GUI agent systems.
Emphasizing user-centered design Lu et al. (022e)) principles can address user needs and concerns, providing options for customization and control over the agent's 's behavior Feng et al. (2024). Equipping agents with the ability to engage in clarification dialogues when user instructions are unclear can enhance task accuracy Wester et al. (2024). Natural language understanding components can detect ambiguity and prompt users for additional information. For instance, the agent could ask, "There are two contacts named John. Do you mcan John Smith or John Doc?" Incorporating human-in--te-lonn systems allows for human intervention during task execution, enabling users to guide or correct the agent's decisions when necessary Wang et al. (2024b). Developing adaptive interaction models that facilitate seamless collaboration between humans and agents is essential. Additionally, providing transparency and explainability in the agent's 's
Potential Solutions:
reasoning processes can build user trust and improve cooperation Cambria et al. (2024] Wu et al. 2024d]
Shi et al. 2025;; Chen et. al. (2025b).
Lastly, developing a virtual desktop environment for the agent to operate in-one one that connects to the user's main desktop session without disrupting their workflow, can significantly enhance the user experience (UX) in human-agent interaction. The picture-in-pictun picture mode implemented in UFO² Zhang et al. (202bb)
84

---

Published in Transactions on Machine Learning Research (06/2025)
MOA Oppi
Monkli
Calorggte www D01
Agandeate
Mondanees
121
attng Ditresaition sitter at (ptpt Astartud
133
rrtam relpart
Main Desktop
1.00 mage
--3 Seuntcins
171
230
orfure Aatatin STevingpony seater Coare
211
1-080 Veen
37
slaotTeee SEAP(()
340
434
Candetzril
0130
UCC debkog
9.35
130
230
keening
Virtual Desktop
Sedeet
Ropety
OcClerg
Oepul Gealle Peaid Soorid
Mirresh
Figure 31: The Picture-in-Picture interface in UFO2: a virtual desktop window enabling non-disruptive aulomation. Figure adapted from Zhang et. al. (2025b).
demonstrates this concept in practice, illustrated in Figure 31 By allowing the agent to within a resizable and movable virtualized desktop, users can casily minimize reposition the agent window as needed. This flexibility improves both the usability and the overall UX of interacting with GUI-based agents.
10.5 Customization and Personalization
Effective GUI agents must go beyond generic task completion and provide experiences that are personalized individual users, adapting to their unique workflows, preferences, and behavioral patterns Li al. (022i]) Cai
et al. (024a).. Unlike general LLM applications that operate in isolated prompts conversations. GUT agents work across software environments where user interaction styles can vary significantly. A one-size-fits--eee- agent may fail to align with how a particular user edits documents, navigates interfaces, or organizes tasks resulting in friction, inefficiency, or user frustration Li et al. (002cc..
For instance, a GUI agent assisting with document oditing must learn the user's preferred tone, formatting conventious, and vocabulary. Without this contextual understanding, the agent may offer irrelevant suggestions enforce formatting inconsistent with the user's intent. Personalization in GUI agents thus requires longitudinal learning, where the agent continually adapts based on prior interactions, fine-tunes its behavior to match user expectations, and preserves consistency across sessions Li et al. 2023b
However, this introduces new challenges. The high variability in user preferences especially free-form GUI environments- makes it difficult define universal personalization strategics. Morcover, collecting and leveraging user-specific data must be done responsibly, raising critical concerns around privacy, data retention, and on-device learning. Striking balance between effective customization and user trust is particularly
85

---

Published in Transactions on Machine Learning Research (06/2025)
important for GUI agents, which often operate over sensitive documents, personal applications, or system-level interfaces.
Potential Solutions: Future research should focus on developing mechanisms for user modeling TTan & Jiang (2023) and preference learning Gao et al. (2024c), enabling agents to tailor their actions to individual users. Techniques such as reinforcement learning from user feedback Kaufmann et al. (2023), collaborative filtering
Kim et al. (2024c), and context-aware computing Talukdar & Biswas (2024) can help agents learn user preferences over time. Ensuring that personalization is achieved without compromising privacy is essential Xiao & Tao (2006), potentially through on-device learning and anonymized data processing, In more futuristic, cyberpunk-inspieed sccnario, agents may inversely generate GUIs tailored to users' nceds, enabling greater customization and personalization Hojo et al. (2025).
10.6 Ethical and Regulatory Challenges
LLM-powered GUI agents raise distinct ethical and regulatory concerns due to their ability to perform real-world actions across software interfaces. Unlike traditional LLMs, these agents can autonomously trigger operations, manipulate data, and interact with sensitive applications amplifying risks around accountability,
fairness, and user consent Gan et al. (2024): Sarker (0024); Biswas & Talukdar (2023): Li et al. (002ee;;
Zhang et al. (2025g).
A key concern is bias inherited from training data, which can load to unfair behavior in sensitive workflows. For example, GUI agent assisting in hiring may unknowingly exhibit gender or racial bias Ferrara (0233 Yu et al. (2024). These risks are harder to audit at the GUT level due to limited traceability across multi-application actions. Regulatory compliance adds further complexity. GUI agents often operate across domains with strict data protection laws, but lack standardized mechanisms for logging actions or securing user consent. This makes it challenging to meet legal and ethical standards, especially when agents act in opaque or background contexts. Addressing these issues requires tailored solutions for GUI agents, including permission controls, runtime confirmations, and transparent activity logs ensuring safe, fair, and compliant deployment across diverse environments.
Potential Solutions: Addressing these concerns requires establishing clear ethical guidelines and regulatory frameworks for the development and use of GUI agents Piñeiro-Martin et al. (2023). ). Future work should focus on creating mechanisms for auditing and monitoring agent behavior Zheng et al. to ensure compliance with ethical standards and legal requirements Chan et al. (2024). Incorporating bias detection and mitigation strategies in language models can help prevent discriminatory or unfair actions Lin et al. (2024d). Providing users with control over data usage and clear information about the agent' 's capabilities can enhance transparency and trust.
10.7
Scalability and Generalization
GUI agents often struggle to scale beyond specific applications or environments, limiting their generalization. Each software interface features unique layouts, styles, and interaction patterns- even common UI elements like pop-up windows can vary widely Zhang et. al. (2024m). These variations make it difficult to design agents that operate robustly across platforms without retraining or fine-tuning.
A further challenge is the dynamic nature of real-world GUIs. Frequent changes due to software updates, A/B testing, or interface redesigns- such as repositioned buttons or modified widget hierarchies can easily break previously functional agents. For example, an agent trainod on one version of a word processor may fail when the layout changes, or when deployed on a different program with similar functionality but a different interface structure. Even when GUIs share visual similarities, agents often fail to generalize without additional exploration or adaptation Shekkizhar & Cosentino (2(25). This lack of robustness restricts deployment in practical settings and increases the cost of maintenance, requiring frequent updates or retraining to stay aligned with evolving environments Grosse el al. (022)) Zhang el al. (024k);; Li & Waldo (2024). Overcoming this challenge remains critical for developing truly scalable and adaptable GUI agents.
Potential Solutions: To enhance scalability and generalization, one solution from the dataset perspective is to create comprehensive GUT agent datasets that cover wide range of environments, user requests, GUT
86

---

Published in Transactions on Machine Learning Research (06/2025)
designs, platforms, and interaction patterns. By exposing the LLM to diverse data sources during training, the model can learn common patterns and develop a more generalized understanding, enabling it to adapt to infer the functionality of new interfaces based on learned similarities Song et al. (2024a).
To further enhance adaptability, research can focus on techniques such as transfer learning Weiss et al. 2016 and meta-learning Chen et al. (2021b). Transfer learning involves pre-training a model on a large, diverse dataset and then fine-tuning it on a smaller, task-specific dataset. In the context of GUI agents, this means training the LLM on wide array of GUI interactions before customizing it for particular application domain. leaen-learning, enables the model to rapidly adapt to new tasks with minimal data by identifying underlying structures and patterns across different tasks. These approaches enable agents to generalize from limited data and adapt to new environments with minimal retraining.
However, even with these measures, the agent may still encounter difficulties in unfamiliar environments. To address this, we advocate for developers to provide helpful knowledge bascs, such as guidance documents, application docunentation, searchable FAQs, and even human demonsttations on how to use the application Zhu et al. (2024a); Guan et al. (2024b); Hsieh et al (2023). Techniques like RAG Gao et al. (0223 can be employed, where the agent retrieves relevant information from knowledge base at runtime to inform its decisions Kagaya et al. (2024). For instance, if the agent encounters an unknown interface element, it can query the documentation to understand its purpose and how to interact with it. This approach enhances the agent's ' capabilities without requiring extensive retraining. Implementing these solutions requires collaborative efforts not only from agent developers but also from application or environment providers.
10.8 Summary
LLM-brained GUI agents hold significant promise for automating complex tasks and enhancing user produc- livity across various applications. However, realizing this potential requires addressing the outlined limitations through dedicated research and development efforts. By addressing these challenges, the community can develop more robust and widely adopted GUT agents.
Collaboration among researchers, industry practitioners, policymakers, and users is essential to navigate these challenges successfully. Establishing interdisciplinary teams can foster innovation and ensure that GUI agents are developed responsibly, with a clear understanding of technical, ethical, and socictal implications. As the field progresses, continuous evaluation and adaptation will be crucial to align tochnological advancements with user needs and expectations, ultimately leading to more intelligent, safe, and user-friendly GUT agents.
11 Conclusion
The combination of L.MS and GUT automation marks a transformative moment in human-computer interaction. LLMs provide the "brain" for natural language lanuuae processing, comprehension, and GUI understanding, while GUI automation tools serve as the "hands", translating the agent's cognitive abilities into actionable commands within software environments. Together, they form L.I.-powered GUT agents that introduce a new paradigm in user interaction, allowing users to control applications through straightforward natural language commands instead of complex, platform-specific UI operations. This synergy has shown remarkable potential, with applications flourishing in both research and industry.
In this survey, we provide a comprehensive, systematic, and timely overview of the field of LLM-powered GUI agents. Our work introduces the core components and advanced techniques that underpin these agents, while also examining critical elements such data collection, model development, frameworks, evaluation methodologies, and real-worid applications. Additionally, we explore the current limitations and challenges faced by these agents and outline a roadmap for future research directions. We hope this survey serves as a valuable handbook for those learning about L.LM-powered GUT agents and as a reference point for researchers aiming to stay at the forefront of developments in this field.
As we look to the future, the concept of LLM-brained GUI agents promises to become increasingly tangible, fundamentally enhancing productivity and accessibility in daily life. With ongoing research and development,
87

---

Published in Transactions on Machine Learning Research (06/2025)
this technology stands poised to reshape how interact with digital systems, transforming complex workflows into seamless, natural interactions.
A Evolution and Progression of LLM-Powered GUI Agents
"Rome wasn't built in day." The development of L..M-powered GUT agents has been gradual journey, grounded in decades of research and technical progress. Beginning with simple GUI testing scripts and rule- based automation frameworks, the field has evolved significantly through the integration of machine learning techniques, creating more intelligent and adaptive systems. The introduction of LLMs, especially multimodal models, has transformed GUT automation by enabling natural language interactions and fundamentally reshaping how users interact with software applications.
A.1 Early Automation Systems
In the initial stages of GUT automation, rescarchers relied on random-based, rule-based, and script-based strategies. While foundational, these methods had notable limitations in terms of ficxibility and adaptability.
A.1.1 Random-Basel Automation
Random-based automation uses random sequences of actions within the GUI without relying specific
algorithms or structured models using monkey test Wetzlmaier et al. (2016). This approach was widely used in GUT testing to uncover potential issues by exploring unpredictable input sequences Zeng et al. (2016). While effective at identifying edge cases and bugs, random- based methods were often inefficient due to a high number of redundant or irrelevant trials.
A.1.2 Based Automation
Rule-based automation applies predefined rules and logic to automate tasks. In 2001, Memon et al., Memon ct al. (2001) introduced a planning approach that generated GUI test cases by transforming initial states to goal states through a series of predefined operators. Hellmann el al., Hellmann Maurer (2011) (2011) demonstrated the potential of rule-based approaches exploratory testing, enhancing bug detection. In the RPA domain, SmartRPA Agostinelli et al. (2020) (2020) used rule-based processing automate routine tasks, illustrating the utility of rules for streamlining structured processes.
A.1.3 Script-Based Automation
Script- aulomation relies on detailed scripts to manage GUT interactions. Tools like jRapture Steven et al. (2000) (2000) record and replay Java-based GUT sequences using Java binaries and the JVM, enabling consistent execution by precisely reproducing input sequences. Similarly, DART Memon al. (2003a) (2003) automated the GUI testing lifecycle, from structural analysis test case generation and execution, offering comprehensive framework for regression testing.
A.1.4 Tools and Software
A range of software tools were developed for GUI testing and business process automation during this period.
Mierosoft Power Automate Microsoft) (2024) (2019) provides a low-code/ low-code/no-code no-code environment for creating
automated workflows within Microsoft applications. Selenium (slenniu | 2024] (2004) supports cross- cross-browser browser web testing, while Appium apppiu (2024) (2012) facilitates mobile UI automation. Commercial tools like
Test Test Complete seartbear (0024) (1999), Katalon Studio katalon 2024] (2015), and Ranorex ranorex 2024
(2007) allow users to create automated tests with cross-platform capabilities.
Although these carly systems were effective for automating specific, predefined workflows, they lacked flexibility and required manual scripting or mll-based logic. Nonetheless, they established the foundations of GUT automation, upon which more intelligent systems were built.
88

---

Published in Transactions on Machine Learning Research (06/2025)
A.2 The Shift Towards Intelligent Agents
The incorporation of machine learning marked major shift towards more adaptable and capable GUI agents. Early milestones in this phase included advancements in machine learning, natural language processing, computer vision, and reinforcement learning applied to GUI tasks.
A.2.1 Machine Learning and Computer Vision
RoScript Qian et al. (2020) (2020) was a pioneering system that introduced a non-intrusive robotic testing system for touchscreen applications, expanding GUI automation to diverse platforms. AppFlow Hu et al. (2018) (2018) used machine learning to recoguize common screens and UI components, enabling modular testing for broad categories of applications. Progress in computer vision also enabled significant advances in GUI testing, with frameworks [Chang et al. (2010) (2010) automating visual interaction tasks. Humanoid Li ct al. 2019 (2019) uses deep neural network model trained on human interaction traces within the Android system to learn how users select actions based on app's 's GUI. This model is then utilized to guide test. input generation, resulting in improved coverage and more human-like interaction patterns during testing. Similarly, Dcep GUI YazzamiBanafsheeDrrgg & Malek (2021) (2021) applics deep learning techniques to filter out irrelevant parts of the screen, thereby enhancing black-box testing effectiveness in GUI testing by focusing only on significant elements. These approaches demonstrate the potential of deep learning to make GUI testing more efficient and intuitive by aligning it closely with actual user behavior.
Widget detection, as demonstrated by White et al., White et al. 2019 (2019), leverages computer vision to accurately identify UI elements, serving as supporting technique that enables more intelligent and responsive UI automation. By detecting and categorizing interface components, this approach enhances the agent's 's ability to interact effectively with complex and dynamic GUIs Xie et al. 2020).
A.2.2 Natural Language Processing
Natural language processing capabilities introduced a new dimension GUI automation. Systems like RUSS Xu et al. (2021) (2021) and FLIN Mazunder & Riva (2020) (2020) allowed users to control GUIs through natural language commands, bridging human language and machine actions. Datasets, such those in Li et al. 20200a (2020), further advanced the field by mapping natural language instructions to mobile UT actions, opening broader applications in GUI control. However, these approaches are limited to handling simple natural commands and are not equipped to manage long-lerm tasks.
A.2.3 Reinforcement Learning
The development of environments like World of Bits (WoB) Shi et (2017) (2017) enabled the training of web- based agents using reinforcement learning (RL). Workffow-guider exploration Liu et al. (2018) (2018) improved RL efficiency and task performance. DQT Lan et al. (2024) (2024) applied deep reinforcement learning to automate Android GUT testing by preserving widget structures and semantics, while AndroidEnv Toyama
et al. (021bb (2021) offered realistic simulations for agent training on Android. WebShop Yao et al. 2202aa (2022) illustrated the potential for large-scale web interaction, underscoring the growing sophistication of RL-driven GUI automation.
While these machine learning-basee approaches were more adaptable than carlier rule-basod systems Zhang et al. (2019); Martins al. (2020), they still struggled to generalize across diverse, unforeseen lasks. Their dependence on predefined workflows and limited adaptability required retraining eustomization for new environments, and natural language control was still limited.
A.3 The Advent of LLM-Powered GUI Agents
The introduction of LLMs, particularly multimodal models like GPT-4o Hurst et al. (2024) (2023), has radically transformed GUI automation by allowing intuitive interactions through natural language. Unlike previous approaches that required integration of separate modules, LLMSs provide end-to-end solution for
89

---

Published in Transactions on Machine Learning Research (06/2025)
GUI automation, offering advanced capabilities in natural language understanding, visual recognition, and reasoning.
LLMs present several unique advantages for GUI agents, including natural language understanding, multimodal processing, planning, and generalization. These features make LLMs and GUI agents a powerful combination. While there were earlier explorations, 2023 marked pivotal year for I.L.-powered GUT agents, with significant developments across various platforms such web, mobile, and desktop applications.
A.3.1 Web Domain
The initial application LLMS in GUT automation was within the web domain, with early studies establishing benchmark datasets and environments Yao et al. (2022a) ; Shi et al. (2017). key milestone was WebAgent Gur
ct al. (023) (2023), which, alongside WcbGUM Furuta et al. (2023) (2023), pioncered weal-world web navigation
using LLMs. These advancements paved the way for further developments Ma et. al. (2023); Zheng al.
22024a
Deng al. 2024b), utilizing more specialized LLMs to enhance web-based interactions.
A.3.2
Mobile Devices
The integration of LLMs into mobile devices began with AutoDroid Wen et al. (2024a (2023), which combined LLMs with nomain-sspciffe knowledge for smartphone automation. Additional contributions like MM-Navigator Yan ct al. (023b) (2023), AppAgent Zhang al. (2023a) (2023), and Mobile-Agent Wang
et al. 2024e (2023) enabled refined control over smartphone applications. Research has continued to improve accuracy for mobile GUT automation through model fine-tuning Nong et al. (2024); Zhang et al. (2024))
(2024).
A.3.3 Computer Systems
For desktop applications, UFO Zhang et al. (2024a) (2024) was of the first systems to leverage GPT-4 with visual capabilities to fulfill user commands in Windows environments. Cradle Tan et al. 2024a (2024) extended these capabilities to software applications and games, while Wu et al., Wu et al. 2024c (2024) provided interaction across diverse desklop applications, including web browsers, code terminals, and multimedia tools.
A.3.4 Industry Models
In industry, the Claude 3.5 Sonnet model Anthropic (2024) (2024) introduced "computer use" feature capable of interacting with desktop environments through UI operations Hu et al. (2024a). This signifies the growing recognition of LLM-powered GUI agents as valuable application in industry, with stakcholders increasingly investing in this technology.
Undoubtedly, LLMs have introduced new paradigms and increased the intelligence of GUI agents in ways that were previously unattainable. As the field continues to evolve, we anticipate wave of commercialization, leading to transformative changes in user interaction with GUI applications.
B Additional Summary in Tabular Form
While the major works have been discussed in detail in Sections XX-XX, provide additional summary tables (Table 16 70 for quick reference and ease of lookup. This table is intended to help readers conveniently locate key references and insights associated with each section.
90

---

Published in Transactions on Machine Learning Research (06/2025)
Table 16: Overview of LLM-brained GUI agent frameworks web platforms (Part 1).
Agent
Platform
Perception Accessibility from
Action
Model
Architecture
Highlight
Link
WMA
Chac
Web
UI operations,
Llama-3.1
Single-agent
Uses a world model to predict state changes before committing ac- tions, improving task and mini- mixing unnecessary in- leractions with the vironment
httpe://
05 nl.
2024
tree
6.gn lype, hover
clock, and
88-Instrruct Dubey el al.
with simulation-
github. com/
DOM
based observa-
kyle8581/
2024
tion
WMA Agentaan
predicting observations and GPT-4
for policy
modeling HTML-T3
Agen
Web
HTML stree-
Ul interactions
Two--tage archi- tecture for plan- ning and pro- gram synt hesis
Leverages specialized LLMs to achieve task
Gur
et al
lure
for lask planning and sum- marization and Flan U-PaLM Chung al.
2024
HTML-bbsed planning and program- matie action execution
2024 for
code genera- tion
LASER Ma Web
GUI strue- ture of the web envi- romment, defined
Defined GPT-4 state, such as searching, selecting items, navigating pages, and finalizing purchase
Single-ngent
Uses state-spare ploration approach, al- lowing it to handle novel situations with flexible backtracking
https://
el al.
2024b
github. con/
Mayer123/ LASER
Voyaggr
Web
Scremsshots with mumeri- cal labels on interactive elements
Standard operations
UI
GPT-4V
Single-agent
Integrates visual and textual cues within real- world, rendered web pages, enhancing its ability to uavigate com- plex web structures Its HTML simplifica. tion method for webpage compre-
https://
He
al
github. con/
2021b
MinorJerry/
RebVoyager
AutoWeb-
Web
Simplified HTML and OCR for text recoguition
operations such as click- ing. typing, scrolling, and selecting, and advanced APIs like jumping to specifie URLs
ChatGLM3
Single-agent agent
https:
GLM
Lad
CLM
/github.
nl.
et
com/ BOm/THUDM/
hension and its bilin- gual benehnark
AutoWebGLM
91

---

Published in Transactions on Machine Learning Research (06/2025)
Table 17: Overview of LLM-brained GUI agent frameworks web platforms (Part IT).
Agent
Platform Web
Perception
Action
Model
Architecture
Highlight
Link
OpenAgents
DOM
ele-
Standard
UI
GPT-1
and
Multi-agent architecture, with distinct agents (Data Agent, Plugins Agent, and Web Agent)
Democrstizes access
httpa://
Xie
al.
ments
operations,
Clande
to language agents by providing an open- source, multi-agent framework optimized real-world tasks
github... com/
2023
browser-based based Authropic
xlang-ai
actions con- trolled, API calls for tool execution, and structured data manipula- tion Screesshut Staudard images and operations
2024
OpenAgenta
SeeAct Zheng et al.
Web
U1 GPT-4V
Siugle-ages
Its use GPT-4V's
httpp://
multimodal capabilities to integrate both vi. sual and HTML infor- mation, allowing accurate per- formance dynamic web content
github com/
2024a
HTML, strue-
OSD XLP Greup/
ture
SeeAct
DUAL-VCR
Web
HTML ele- ments and screenshots
Standard operations base Chung
U1
Flan-T's
Two-stage single- agent architec- ture
Dual-view contextual- ization
Kil
617
2024
et al.
2024
Agent-B E
Web
DOM strue- Standard operations
U1
GPT
Hierarehical multi-agent architecture, composed of planner agent and browser navigation agent Single-agent with search
Hierarchical archi- https://
Abuelsaatt saadd
ture and change obeer-
Turbo
tecture and adaptive DOM perception
github com/
65 al. [202]
Emergencekl Agent- E.
vation
Search-
Web
Screeushot Standard operations
UI
GPT-4
Novel inference-tiue soarch algorithm that euhances the agent's ability to perform multi-stop planning and decision-making Dynamically con- structs an internal web environment represen- tation more robust decision making. The integration replay buffer and analy- sis reduces uavigation errors and improves task completion rates.
https://
Agent
Koh
and text descriptions
jykoh. con/
eL al.
2724b
search agents
R2D2 Huang
Web
DOM
Staudard operations
U1
GPT-4u
Single Single-ageut
httpp: //gthubb
et al.
2025b
com/ AmenRa/ retriv
92

---

Published in Transactions on Machine Learning Research (06/2025)
Table 18: Overview of LLM--rainne GUT agent frameworks on web platforms (Part III..
Agent
Platform
Perception
Model Single-agent architecture
Architecture
Highlight
Link
Agent
Standard operations
Specialized fine Lunittnnng ap prosch using production- scale workflow dats. to outper- form general- purpose LLMs GPT-4 in web navigation
Drtps://gatinn
Shen
DOM
com/colonylabs/
2024b
Scribedgent
Zhon
Standard Operations
Clande
multi-agent architecture involving task propeser to suggest tasks, an agent policy to perform tasks, and an autonomous evalustor to sucress and provide feedback using RL. Multi-agent architecture, with Global Optimization and Loral Opti. mization
Autonomous skill covery real-world vironments using task proposers and reward- based evaluation
https:
2024a
Sounet
//yanqval.
Authropic
github.10/ 10/
2024
PAE/
Qwen2VI.
71
Wane 20/44
et a.. and
LaaVa
Liu ml.
2024b
WebPPiot Zhang et al.
Web
Accessibility trees and dynamic observations
Standard operations
U1
GPT-4
Dual optimization
https:
strategy
(Global and
//yao2220.
2024n
Local)
with Monte
github i0/
Carlo
Tree Search
NebPilot/
(MCTS) Browne et al.
2012 allowing dy-
namie adeptation to complex, real world
web euviromments
Hybrid
Web
Accessibility and screenshots
Staudard operations,
U1
GPT-4
Multi-agent sys- tem, combining both API and browsing capa- bilities Single-ageu
Hybrid Agent seam- lessly integrates web
https:
Agent Song
//github
et al.
B024b
API calls, and generating code Staudard operations
browsing and API calls
com/ yueqia/
APT Banad Agent
Agent
Web
HTML
U1
GPT-4
Simple design that timizes the observation and artion spares Trains web agents us- ing synthetic demon- strations, eliminating
Yang et al.
2024a
N Netnaw
Web
DOM
Standard operations
Ul
GPT-4
Single-agent
https://
Murty et al.
github con/
2024
MurtyShikhat
the need expensive human input
Nnetnav
93

---

Published in Transactions on Machine Learning Research (06/2025)
Table 19: Overview of L.M--briined GUT agent frameworks on web platforms (Part IV).
Agent
Platform
Perception
Action Staudard
Model GPT-4
Architecture
Highlight:
Link
NaviQAte
Web
Screenshots
Siugle-ageut sys-
Frames web naviga- tion as question-and-
Shabbandlch
operations
tem
2024
task
OpenAoll Yeb-
Web
HTML
operations, Web APIs, and self-generated code
GPT-4
Modular single- agent
Modular drsign that https:
Agent
long
screenshots
AutoWe- bGLM Lai
allows developers soamlessly integrate various models to automste web tasks Ability to automate web interactious using natural language in- structions
/github.
2024
com/THUDN/
et al.
2024
OpenWebAgent//
Steward
Web
HTML and screenshols
Standard operations
Ul
GPT-4
Single-ngent
Tang & Shin
2021
WebDreamer
Web
Standard combined operations and navigation DOM actious
Screenshots
Ul
GPT-4o
Model-based
Pioneers the use of https://
Gu al.
single-agent architecture
LLMs as world models planning in complex web euvironments
github. con/
2024
with SoM,
DSU-XLP-Gr Greppp
HTML
MebDreaner
Agent
Web
interactions, querying the
LLaMA-3
Single-agent with MCTS and
Combination of MCTS- guided search and self- critique mechanisms ables iterative improve- ment in reasoning and task execution
https://
Purta
tual imput, screenshots for help
70B
Dubey
github. con/
2024
et
2024
RI.
sentient-Bn eereeeetttrgttering//
for visual
for
policy
agent-q
feedback
learning and execution,
GPT-V for visual feod-
Auto-Intent
Web
ITML strue-
Standard Operations
GPT-3.5,
Single-agent with exploration
Introduces a unique self-exploration self.
Kim
GPT-4,
2024b
Llama-3
egy generate tically diverse intent hints
Dubey al. 2024 for
action in-
ference:
Mistral--2 7B
Jiang ct al.
2023 and
Flan- 1,
Chung
2024
for
intent predic-
Lion
94

---

Published in Transactions on Machine Learning Research (06/2025)
Table 20: Overview of LLM-brained GUI agent frameworks on web platforms (Part V).
Agent
Platform
Perception
Action Standard Operations and Playwright
Model
Architecture Single-ager
Highlight
Link
AdapttAgent
Web
GUI screen-
GPT-4o
Adapts with just multi- modal human demon- strations
Verma al.
shots with
CogAgent
2024
HTML/-
Hong al.
DOM strue-
scripls
2023
tures
WEPO
Lin
Web
Standard DOM Operations
HTML and
U1
Llama3
Single- aggen ar- chitecture.
Incorporates a distance- based sampling mech- anism tailored to the DOM tree structure, euhancing preference learning by distinguish- ing between salient
et al.
2024c
SB
Dubey
ct al.
20024
Mistrrll 7B
Jaang
al.
20223
Gemma
Team sl.
and non-salient web elements with DPO Rafailov al. [2124]
2024
Agent-
Web
Accessible tree struc- ture of web elements
Standard operations
U1
Large LLMs:
Multi-ageut it- erative architec. ture
Introduces at Itere- tive, symbiatic learning process between large and small LLMs web automation. hances both data syn- thesis task perfor- mance through spec- ulative data synthe- sis, multi-tas learning, and privacy-preerving by hybrid modes.
Symbiotic Zhann et al. 2025e
GPT
Claude-3.5.
Small LLMs:
LLaMA-
Dubey
et 2024
DeepSeek-2 R1
Gmo
2025
LiteWebAgeul Zhang et al.
Web
DOM,
Standard operations, Playwright script
U1
Any
LLM
Single agent
First open source, production-ready web agent integrating tree search for multi-step task execution.
https://
Screen- shots
and MLLM
github com/
2025e
PathOnAT/
LiteWebAgent
ECLAIR
Web
Screenshots
Standard operations
CPT-1V, GPT-lo, CogAgent
Siugle-agent ar- chitecture
high setup costs, brittle ecution, and burden- maintenance ciated with traditional RPA by learning from video and text ducu- mentation.
Eliminates
httpe://
Wornow
github com/
2024
Hazykesearch
eclair-agente
Danmn et Web
DOM
el- Standard operations
U1
Not specihed
Single-agent at- chitecture
User-aligned task exe- cution, where the agent adapts to individual preferences ethical manner.
at_ Dammm
ements, Webpage
2025
acceesiiiity
attributes
95

---

Published in Transactions on Machine Learning Research (06/2025)
Table 21: Overview of LLM-brained GUI agent frameworks on web platforms (Part VI).
Agent Plan Plan-and--
Platform Web
Perception HTML
Action
Model
Architecture Two stage modu- lar architecture:
Highlight
Link
Standard
UT
LLaMA 3.3
Decouples planning from execution in LLM- based GUI agents and introduces scalable synthetir data gen- eration pipeline to each compo-
Act Erdogan
operations
70B-Instruct
et al.
2025
Dubey et al.
PLANNER EXECUTOR
2024
Weaver
Web
GUT screen- shots and Accossibility Tree
Standard operations and high-level skill APIs
GPT-44
Single-ager
Introduces self. improvement frame- work for web agents that autonomously discover, synthesize, and refine reusable skill APIs through exploration Introduces program- matic skills that are verified through ex- zorazro/
https://
Zheng al.
github
2025a
OSU- MLPP-L-PPPopppp
Skil Weaver
ASI et al.
Wang
Web
Webpage Accessiiiitty Tree
Standard GUT actions
Claude-3.5- Somnet
Single-agent
https://
2026g
github. com/
agent- induction
quality and used as callable actions to improve efficiency
Rollback
Web
Accessibility trees
Standard GUI actions
Multi-agent architecture
Multi-module, ReAcctinspiree agent architec- ture
Introduces lar rollback mechanism
modu-
A gent
Zhang
et al.
2025|
that enables multi-step rollback to avoid dead- end states
96

---

Published in Transactions on Machine Learning Research (06/2025)
Table 22: Overview of LLM- LLM-brained GUI agent frameworks on mobile platforms (Part 1).
Agent Wang
Platform Android Mo-
Perception
Action
Model
Architecture
Highlight
Link
Android view
(1)
Screen
PaLM
Anii
Single-agent
The first paper study
https://
Wang al.
bile
hierarchy structure
Question Gon-
et al.
2023
Sereen Question Gener- ation and Screen QA using LLMs
github. con/
2023a
eration,
(2)
google research
Screen Sum-
google-research
macization,
tree/
(3) Screen
master/
Question An-
11minoblee
swering. and
(4) Mapping
Instruction
UI Action
Vision Tasker al.
Android bile devices
UI screen- shots with widget detec- tion and text extraction
UI operations
ERNIE
Bot Single-agent
Vision-based UI derstanding approach, with vision- based UI
https://
Song
such tap-
Baich
Re
github. con/
2024c
ping, swiping, and entering
search
2024
which allows it to in- terpret UI semantics di rectly from screenshots without view hierarchy dependencies
AkimotoAyako/
understanding
VisionTacker
text
and sequential task planning
DroidBot
Android mo- bile devices
Translates the GUI state in- formation of Android applications into natural
Cl operations, including ac- tions like click, scroll, check, and edit
GPT
Single agent
Automates Android ap- plications without mod- ifications app the model
https://
Wen
github. com/
HD 2024e
loLLM/
DroidBot
language
prompts
CoCo-Ageut
Android mo-- bile devices
GUI screen-
GUI actions, such as click-
CLIP
Radi
Single-agent
Its dual approach of Comprehensive Envi- romment Perception and Conditional Action Prediction
httpe:
Mn et al
shots, OCR
ford ct ml.
//github.
2024d
layouls, and historical ac- tions
ing, scrolling,
2021
com/ /xbx/////////////
and typing
Vision
60-
CoCo-Agen
ending
LLaMA-2 2-
chat 7B for
language
processing
Anto-GUI
Android mo- bile devices
GUI screen- shots
GUI operations
BLIP-2 vi- Singlo-agent
Its direct interaction
https:
Zhang
sion encoder
with chain-of-
with GUI elements. Its chain-of-action mech- anism enables it to leverage both past and planned actions
/github.
Zhang
2024
Li et
8l
action
com/ cooel//
20233 with
Auto-CUT
FLAN-
Alpaca
Wei
et
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2444.14219, 2024.
Hayfa Abuaddous, Ashraf Mousa Salch, Odai Enaizan, Fahad Ghabban, and Anas Bassam Al-Badarcen. Automated user experience (ux) testing for mobile application: Strengths and limitations. International Journal of Interactive Mobile Technologies, 16(4), 2022.
Tamer Abuclsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agentic systems, 2024. URL
https://arxiv.. org/
13032
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 08774 2023.
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human, 2024. URL
08164
97

---

Published in Transactions on Machine Learning Research (06/2025)
Table 23: Overview of LLM- brained GUI agent frameworks on mobile platforms (Part I1).
Agent
Platform Android bile devices
Perception
Action
Model
Architecture
Highlight:
Link
MobileGPT
Simplifidd
Standard Ul
GPT-4
Single-agent
Introduces human- like app memory that
https://
Lee
al
HTML repre-
operations and navigation actions
turbo for
chitecture aug- mented by hi- erarchical mem- structure
mobile-ga gp.
2024c
sentation
screen under-
allows for task decom- position into modular sub-tasks
g1 github.10
standing and rensoning,
GPT--35- turbo
slot-filling
sub-task
parameters
MM-
Mobile 1OS
Smartphone screenshots with a8- sociated
Clickable operations
GPT-4V
Single-agent
Using set-of-mark prompting with GPT.
httpa:
Navigator
and Android
//github.
Yau et al.
4V for precise GUI navigation on smart- phones
com/ z2xelp/
2023a
MM Ma-Navigatoo
set-of mark
tags
Prompt2Task Huang et al.
Android mo- bile devices
GUI struc- ture and layout hicrar- dhy, full-page textual de- scriptions, OCR-based text extrac- tion Real-time User-like ac- screenshots tions, like Top,
Staudard operations
U1 GPT-4
Multi-agent ar- chitecture
Euables UI automation through free form tex- tual prompts, eliminat- ing the need for users to script automation tasks.
https://
gi github. com/
PromptRPA/
Prompt2TankDatasee
Appngent Zhang et. al.
Android mn-- bile devices and XML. Long press, Swipe, Text input, elements and Exit
GPT-4V
Single-agent
Its ability to perform tasks on any smart- phone app using a humsa-like interaction method
https://
appagent- of ficial.. g github.io
2023a
detailing the interac-
App Agent-
Android bile devices
GUI screm- Standard
GPT-4
Multi-phase ar- chitecture with Exploration Phase and De- ployment Phase lunes
Enhances adaptability and precision mobile
V2
al.
shutt with annotated elements,
Operations: Tap,
2024g
in- put, long press, swipe, back, and stop
environments
bining structured dota parsing with visual fea-
OCR for delecting text and icons, Strue- tured XML metadata
98

---

Published in Transactions on Machine Learning Research (06/2025)
Table 24: Overview of LLM-brained GUI agent frameworks on mobile platforms (Part III..
Agent
Platform Android moo- bile devices
Perception
Action
Model
Architecture
Highlight
AutoDroid
Simplified HTML-style
Standard
GPT-3.5, GPT-4, and
Siugle-agent ar- chitecture
use app-specific
https://
Wen
al.
operations
knowledge and multi- gramularity query opti- mixation module duce the computstionsl cost
autodroid-gy aye.
2024a
representa- Lion
Vieuna-BB
githbb.1o
Chiang al. 2023
ArtoDrooid
Audroid Mo- bile Devices
Structured GUI Repre- sentations
Multi-step scripls
Llama-3 1-
Seript-besed at- chitecture.
Converts GUT task an- lomation inlo scripi generation problem, en- hancing efficiency and task success rates.
V2 Wen et al 2024b
of UI
8B Dubey
standard operations and API
2021 ct al.
Zhang 2024g
Android mo-- hile devices
Screenshot- hased con- and semantic information
Standard operations
GPT-4V
Single-agent 8t- chitecture
integration of chain-of-action. thought process, which explicitly maps each action screen de- scriptions, reasoning
httpa://
github.com con/
Zhangl
CoAT
steps, and anticipated outcomes Hierarchical Multi-Agent System
Mobile- Agent-E Wang et al. 2025e
Mobile An- droid
GUI screen- shots, OCR for detecting
Standard operations and APIs
UI GPT-lo, Claude-
Hierarchical multi- agent framework that separates planning from exeeution improved long reasoning and self- evolution, enabling the system learn reusable tips and shortruts Introduces privacy- preserving federated learning for mobile automation, euabling large-scale training withaut centralized
https:
//x plug. github. 1o/
3.5-Soumet, Gemini-1.5 Pro
leggnt
Fed Mobile
Audroid mo- bile devices
GUI Screen- shots
Staudard operations
U1 Qweu2-VL VL-
Multi-agent fed- erated learning
Agent
Wang 2025e
Instruct-7B Wang et al. 2024
eb al.
human sanotation.
99

---

Published in Transactions on Machine Learning Research (06/2025)
Table 25: Overview of LLM-brained GUI agent frameworks mobile platforms (Part IV).
Agent
Platform Android Mo-
Perception
Action
Model InternVi. 1-
Architecture
Highlight
Link
ChickAgent
Screensoots
Standard
U1
Single-agen
Combines MILLM rea- soning with a dedicated location model to enhance interaction accuracy
https://
Hoscilowie
bile Devices
operations
Chen
github. com/
et al.
2024
ct al. 20241
Samaung/
Tiny lick
Clickagent
Pawlowski
al. 2024
SeeClick
Cheng sl.
2024n
Mobile- Agent et al.
Mobile droid
Au-
Screesshuls with icon
Staudard operations
U1
GPT-IV
Siugle-agen
Vision-ceutric proach that eliminates dependency on system- specifie data
ht https:
Wang 2024e
with Ground-
github com/
detection
ing
DINO
X X-PLUG/ Mobiledgent
al.
20/23a
CLIP
Rad
ford
2021
for
detec-
tion
Mobile-
Mobile An-
Screenshnts with text, icon recog- nition, and description
Standard operatious on mobile phones
UT
GPTT-V
Multi-agent ar- chitecture with Planning Agent, Decision Ageut, and Reflection Agent
Multi-agent, architec- ture enhances task navigation for long- sequence operations
https://
Agent-v2 Wang et al.
deoid OS and
with Ground-
github com/
Harmony OS
ing
DINO
X-PLUG/
2024d
Liu
Mobil leAgent
2023a
and
Qwen-
Ini4 Bai
2023a
MobA el al.
Zhu
Mobile droid
An-
GUI Standard operations and API function calls
Ul
GPT-1
Two-level agent: Global Agent and Agent.
Two-level agent sys- tem that separates task planning and exeen- tion into two special. ized agents
https://
2024b
tures, screen- shots with annolation
github con/
Local
OpenDFH/
MobA
Mobile- Experts
Mobile droid
Au-
Interface memory and procedural memory
Staudard U1 operations and eode combined tool formula-
VLMs
Multi-agent framework with double planning
Code-combined tool formulation method and double-layer planning mechanism
Zhang et al.
2024e
collaborative task execution
LIMAC Christianos
Mobile droid
An- and sponding widget trees
Screeushots
Standard operations
UI
Lightweight transformer and fine- tuned VLMs
Single-agent
Balances computs-
tional efficiency and natural lauguage derstanding
al. 2024
100

---

Published in Transactions on Machine Learning Research (06/2025)
Table 26: Overview of LLM-brained GUI agent frameworks mobile platforms (Part V).
Agent OS: Kairos
Platform Mobile
Perception
Action
Model
Architecture Single-geett with critie-in-
Highlight
Link
An-
GUI screen-
Standard
UI
OS Atlas
Introduces an adaptive
https://
Cheng et al.
droid
shots
operations
Pro-7B and GPT-4o
interaction framework where each action paired fidence score, dynami- cally deciding between autonomous execution and human interven- tion Introduces a novel verifier-driven architec- ture where LLM does not generate directly instead scores and selects from finite of extracted actions,
github. con/
the-loop design
Wuzheng02/
con-
Kairos
V- D-Deoid Dai
Mobile droid
Au-
Android Accessiility Tree
Staudard operations
U1
LLaMA-3.1-
Verifirr Driven Single-Agent Architecture
e5 al.
2025
80-Instruct
Dubey et al. 2024
improving task success rafes and siguificantly reducing latency
ReachAgent
Android mo- bile devices
GUI Screen- shots, XML document
Standard operations
U1
MobileVLN LM
Single-agent, two-stage train- ing
Divides tasks into sub- tasks: "Page Reaching" (navigating to the cur- rect screen) and "Page Operation" (perform- ing actions on the screen), using RL with preference based train- to improve long- term task success.
Wn ct al.
Wn ot
ml
2025cc
2024c
Mobile- Agent-V Wang et al.
Mobile droid
An-
Video guid- ance, XML hierarchy
Standard operations
U1
GPT-4n 4o
Multi agent sys: tem
Introduces video guided learning, allowing the agent to acquire operational knowledge efficiently. Introduces all app- oriented multi-agent framework with self- evolution, overcoming the complexitt of cross- app interactions by dynamically recruiting specialized agents.
https://
gi tbub com/
X- P-PUUG/
2025b
Mobiled Mobiledgent
SobileSteward
Mobile droid
An-
XML layouts, Screenshots
Standard interactions, Code execution
GPT-IV, GPT-4o
App-oriented multi-agent
httpe:
Li
/github,
2025g
framework
com/ /
MobileSteward
AppAgeutX
Mobile droid
An-
Screenshots
Standard operations
U1
GPT-1o
Single-agent at- chitecture
Introduces an evolu- tionary mechanism that enables dynamic learning from past in- teractions and replaces inefficient low-level op- erations with high-level level actions.
https://
Jiang ct al.
appagentx.
2025
github. io/
101

---

Highlight: Published in Transactions on Machine Learning Research (06/2025)
Table 27: Overview of L.M--briied GUT agent frameworks on mobile platforms (Part VT).
Agent CHOP
Platform
Perception
Action
Model
Architecture Multi-agent ar- chitecture
Link
Zhom
Mobile
An-
Screemshots
Standard
Ul
GPT-4o
Introduces a basis
https://
eL sl.
2025
droid
operations
subtask framework, where subtasks are predefined based human task decomposi- tion patterns, ensuring better executability and efficiency.
github. con/
Yug1-2hov
CEOP
LearnAAt Act Liu el al.
Mobile droid
An- GUI screen-
Standard GUI actions
Gemini-
Multi-agent
Introduces strue- tured, demonstration
https:
2025a
shuls, UI trees, and demon- stration trajectories
1.5 1.5-Pro,
7/1yy0444.
UI-TARS-
based learning pipeline mobile GUI agents. addresses long-tail generalization via few- shot demonstrations, achieving substantial performance gains complex real-world mobile tasks
github. 10/
7B-SFT,
LearnAct:
Qwen2 VI-
AncrroidGen
Mobile An- droid
XML
Ul
Standard GUI
GLM-4-9B
Multi-module single-ageut
Innovatively addresses data scarcity for An- droid agents through self-improving ar- chitecture. zero
https:
Lai
al.
structure
actious
GLM et. al
/github.
2025
2024
com/TH TTUDT
MA-M
AndroidGen
Dubey
el
luman annolation training pipeline, and effective generalization from hard tasks Pioneers ageul- initiated interaction in mobile UI antomation
Agent- Initiated Interaction
Audroid Mo- bile
Accessibility tree and screenshots
Standard GUI operations
Gemini 1.5
Single-agent chitecture
https://
github con/
google-raseareh
Kahlon et al
research/ tree/
2022
master/
Latent State Estimattio
Audroid Mo-
Accessibility
Standard GUI operations
PaLM
Two module design with Reasoner and
First formalize
bile
estimation of latent UI
Bishop al.
states using LLMS to
2024
Grounder
support UI automation
102

---

Published in Transactions on Machine Learning Research (06/2025)
Table 28: Overview of LLM-brained GUT agent frameworks on computer platforms (Part T).
Agent
Platform
Perception
Action
Model
Architecture
Highlight:
Link
UFO
Zhang
Windows computer
Screenshots
Stundard operations with additional
GPT- Pis--ision
Dual-agent architecture,
dual-agent that seamlessly nav-
https://
al.
2021a
with notated controls, and widget properties
github com/
consisting of HostAgent (for application selec- tion and global planning) and an AppAgent specific task execution within applications)
igates and interacts
mi microsoft/
customized operations
with multiple applica-
UFU
tions to fulfill complex
user requests in natural
language on Windows
CFO2
Zhang 2025b
Windows desktops
GUI Unilied
GPT--o (and
Centralized
Transforms
httpe:/
et al.
shots and textual control prop- erties list
GUI API action layer
GPT-4V.
HostAAent with application-
tional CUA into OS native, pluggable AgentOS with deep Windows integration, hybrid GUI-API ac- tions, vision + UIA perception, speculative multi-action planning, retrieval-angmented
gi github com mi microsoft/
ol, Gemini
Flash):
specialized
UFD/
Vision
AppAgents
ground-
ing
via OmniParser-
knowledge, and
intrusive PIP virtual desktop
ScreenAgent
Linux and Windows desktop
Screenshots
U1 ScreenAgent operations model
Standard
Single agent
Integrated planning acting-reflecting
https://
Niu
e5 al.
github com/
2024
pipeline that simulates
riuzaishong
coutinuous thought process
Screendgeat
OS-Copdlot Wu
Linux MacOS
Unified in- Standard terface that includes Bash and Python and mands, as well API calls
GPT-4
Multi- component architecture involving planner, confign- racor, actor, and critic modules
Self-directed learning httpe://
operations,
capability, allowing copiop. it adapt to applications by github. tonomously generating and refining tools
2024e
keyboard control, API calls, and Bash or
Python inter-
preters
103

---

Published in Transactions on Machine Learning Research (06/2025)
Table 29: Overview of LLM-brainee GUI agent frameworks on computer platforms (Part I1).
Agent
Platform
Perception
Action
Model
Architecture
Highlight
Link
Cradle
Tan
Windows
Complete videos
board and
GPT-4
Modular single- agent architec- ture
generalizability
https://
2024a
computer
actions
varions digital enviromments, allowing it to operate without relying on internal APIs
baai-agents
with Ground-
github.io
ing
DINO
Cradle/
Liu et
a.
(2023n and
SAM Kiilloo
jtt al. 20023
for object de-
Lection and
localization
Agent Agushe al.
S
Ubuntu and Windows computer
Screenshots and accessi- bility Uree
Standard operations and system-Jevel controls
Ul
GPT-1 and
Multi-agent chitecture
Experience-augmeeneed hierarchical planning
https://
Claude-3.5
gi thub.. con/
2024
Somnet
prising Man- ager and Worker structure
simular ai/
Anthropic
Agent- 9
2024
GUT Narra- for Wi et al.
Windows computer
High- resolution screenshot
Standard operations
U1
GPT-4 and
Two-stage architecture,
Uses the cursor as focal point to improve understanding of high- resolution actions
httpe:
QwenVL-12 1-BB
//hhowlab.
2024b
Bai
al
detecting the cursor location selecting keyIrames, then generating
github.io/
2023a
GUI- QU1-Narra5or
tion captions
PC Agent He
Windows Computer
Screeushots and event- based track- ing
Standard Operations
U1
Qweu2-VE
planning agent for
Human cognition trans- fer framework, which transforms raw interac. tion data into cogui- tive trajectories to en- able complex computer tasks.
httpe:
et al.
2024
720-Instruet
//gair-nlp.
Wang et al.
decision making combined with
gi fithub.io
[024]
and
PC- Agent/
Molmo
grounding agent
Deilke el ml.
executing actions. Single-ngent
2024
Zern-shot
Computer
code and DOM operations
Standard
PaI.M
Zero-shot capability in performing computer control tasks
https://
Agent
et al.
2023
github.
2023d
google-resears
google- researchh
tree/
master/
zero shot
structured
reflection
TaskMind
Windows Computer
Standard GUI actions
GPT-3.5 GPT-4
Single-ugent architocture
Introduces task graph representation with cognitive dependen- cies, enabling LLMs to bet- ter generalize demonstrated GUI tasks
https://gither com/ Svennaire/TaakMin
Yin
2025
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialist framework for computer use agents. arXiv preprint S0f:::25040000000 2025.
AgentSeaf AT. Introduction to agentsea platform, 2024. URL https://ww.ageenteee.oou 2024-10-26.
Accessed:
Pranjal Aggarwal and Sean Welleck. Programming with pixels: Computer-use meets software engineering. arXiv preprint dxX:::2522.18555 2025.
Mohamed Aghzal, Erion Plaku, Gregory J Stein, and Ziyu Yao. A survey on large language models for automated planning. Xix preprint arXin:2502. 12435, 2025.
104

---

Published in Transactions on Machine Learning Research (06/2025)
Table 30: Overview of LLM-brained GUI agent frameworks on computer platforms (Part III).
Agent
Platform
Perception
Action
Model
Architecture
Highlight
Link
PC-Agent
Windows
UT
tree,
Standard
U1
GPT-4o
Hierarchical Multi-Agent
PC-Agent's hierarchi- cal multi-agent design
https://
Lin
al
computers
Screenshots
operations
gi ttub.. con/
2025c
enables efficient decom- position of complex PC tasks. Its Ac- tive Perception Module euhances fine grained
X- PLUG/
Mobi leAgent
tzee/ tzee/mein/
PC- Agent
GUI understanding by combining acressibility structures, OCR, and intention grounding.
PwP
Ag- &
VSCode- based IDE
Screenshots, File lem access, Terminal outputs
Standard interactions, Claude-3.5 File operations, Bash com- mands, Tools in VSCode
U1 GPT-lo,
Single-agent chitecture
Slifts software eugi- neering agents from API-based tool inter- actions to direct GUL based computer use, al- lowing agents to inter- act with an IDE as a hu- man developer would. A dynamic task scheduling mechanism with a plug- aud-play agent pool, enabling adaptive handling of GUI tasks
https://
garwal
progrann www.hiiiittt
Wellech
Computers
Somnet, Gemini 1.5
2025
COLA Zhao 2025a
Windows computers
GUI struc- ture, prop- and screenshots
Standard U1 operations and system APIs
GPT-4o
Hierarchical Multi-Agent
httpe:
/github.
con//lokia/
COLA-demo
STEVE
Windows Desktop
GUI screen- shuts and Ally Tree
Standard operations
U1
Qwen2-VI VL
Single-ngent
scal- able step verification pipeline using GPT-lo to generate binary labels for agent actions, and applies KTO optimization to incor- porate both positive and negalive actions into agent learning
Introduces
https://
e5 ml.
2025
Wang et al.
github. con/
[024 and
Fanbiniu/
GPT-40
STEVE
Simone Agostinelli, Andrea Marrella, and Massimo Mecella. Rescarch challenges for intelligent robotic procoss automation. In Business Process Management Workshops: BPM 2019 International Workshops, Vienna, Austria, Seplember 1 1-6, 2019, Revised Selected Papers 17, pp. 12 12-18. Springer, 2019.
Simone Agostinelli, Marco Lupia, Andrea Marrella, and Massimo Mecella. Automated generation of executable rpa scripts from user interface logs. In Business Process Management: Blockchain and Robotic Process Automation Forum: BPM 2020 Blockchain and RPA Forum, Seville, Spain, September 13-18, 2020, Proceedings 18, pp. 116 131. Springer, 2020.
Fellou AI. Eko build production-ready agentic workflow with natural language. https://eko.felllo 2025. Accessed: 2025-01-15.
Lukas Aichberger, Alasdair Paren, Yarin Gal, Philip Torr, and Adel Bibi. Attacking multimodal agents with malicious image patches. In ICLR 2025 Workshop on Foundation Models in the Wild.
Wajdi Aljedaani, Abdulrahman Habib, Ahmed Aljohani, Marcclo Modeiros Eler, and Yunhe Feng. Does chatgpt generate accessible code? investigating accessibility challenges in llm-generated source code. In International Cross-Disciplinary Conference Web Accessibility, 2024. URL https://app http:://ppppssmam..eeee.mmm www.ssttieccooolaaaa....om org/
CorpusIDD2733502277
Alfonso Amayuelas, Liangming Pan, Wenhu Chen, and William Wang. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. arXiv preprint arXic:2305. 13712, 2023.
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jcan Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Compulational Linguistics, 8:556-88 571, 2020.
105

---

Published in Transactions on Machine Learning Research (06/2025)
Table 31: Overview LLM- brained cross-platform GUI agent frameworks (Part 1).
Agent
Platform
Perception
Action
Model
Architecture
Highlight
Link
AutoGLM
Web and Mo- bile Android
Screeushots with SoM
Standard
UI
ChatGLM
Single-agent chitecture
Self-evolving online riculum RL framework, which enables contin- uous improvement by interacting with real- world environments
https:
Liu
al
operations,
GLM
nl.
//xiac9905.
2024h
aunotation and OCR
Native API
2024
github.io
interactions, and Al-driven
AutoGLM/
actions Standard operations, Native API
Clicklick
Web, bile,
Mo- and
GUI screen- shots
UI
Florence-2- Base VLM
Singlo-agent, with single turn tasks
Compact size (0.270 parameters) with high performance
https://
naw Jowski et al.
huggingface
2024
Windows platforms
Xlao es al
co/ co/Sansung/
interactions, and Al-driven actions
2023
TinyClick
OSCAR Wang & Liu
Drsktop Mobile
Screenhonts
Standard Uf operations
GPT-4
Singlo-agent chitecture
Ability to adapt to real- time leedback and dy- namiicllly adjust its ac- tions
2024
A Store
Desktop and mobile euvi- ronments acressibll- ity trees, screenshots and terminal output etc Screenshots
GUI strue- lures and properties,
Standard operations, API calls
Uf
GPT--4 and
Multi-agent chitecture
ar-
Dynamically ly intograte wile variety of het- erogeneous agents, en- abling both specialized and generalist capabili- ties
https://
la
el
al.
IntervL2. 8B et al.
chengyou-
2024
Chen
github 10/ AgentStore- Homee
20241
MMAC- Copilot (Song et al. 2024d
Windows OS Desktop, mobile appli- cations, and game euvi-
Standard Ul operations, Na- tive APIs, and Collaborative
GPT-4V, Multi-agent SeeClick architecture with Planner, Programmerr, Viewer, Mentor, Video Analyst, and Librarian
Collaborative multi- agent architecture where agents specialize
Cheng et sl. 2024a and
in specifie tasks
multi-agent
Geuin
ronments
actions
Vision for agents
different
AGUVIS
Web, desk- top, and mobile
Image-basod observations
Standard operations
Fine-tuned
Single-agent Bt- chitecture
vision- ap- proach for GUI interac- tion, bypassing textual UI representations and euabling robust cross- platform generalization Purely dision-based GUI agent that does not require non-visual inputs
https://
eb 20/44
Qwen2 VL Wang al.
aguvis- project.
github.
2024
Ponder
&
Web. An- droid, [OS Mobile, Win-
Purely visual
Standard operations
UI
GPT-1o and Claude 3.5
Divide-and- conquer archi- tecture
https://
Prese
Wang 2024g
imputs
inv invinciblewyg
eB al.
Sonnet for levellevel task decom- position,
github. 10/
dows. and
ponder press press-page/
macOS
line-tuned
Qwen2-VL-
Instruct
Wang et sl.
2024i for
GUT element
grounding
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellssern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquctte-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagiclski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei
106

---

Published in Transactions on Machine Learning Research (06/2025)
Table 32: Overview of LLM-brained cross-platform GUT agent frameworks (Part I1).
Agent
Platform Mobile, Web,
Perception
Action
Model
Architecture
Highlight
Link
IudiGULAgent
Raw screet- shots
Standard
Qwen2-VL-
Single-agent architecture eahanced bierarchical reasoning.
Introduces native reasoning skills, such by as hieracciica and expectation-refietion reasoning, euabling advanced and udaptive task handling,
https://
Liu
Desktop
operations.
2B
Wang
github. con/
2025e
et al.
2024j
Realln Labe
ImfiGUlAgent
Learn-by by-
code development,
Webb,
GUI screen- shots with SoM and accessiiitty
Standard UI in- teractions and code exocution
Claude
Multi agentt
Introduces a fully an tonomous data synthe- sis process, eliminating the for luman- labeled agentic data
Interact
3.5-Sonnet,
eE nl.
deskrops
Gemini-1.5
Pro
et al.
2023
CodeGemma-
7B,
CodeStral-
22B
CollabULAgen Mobile An-
Screesshos, UI trees
Standard operations
U1
Qwen2-7B
Multi-ggent sys-
A mullti-gennt rein- forcement learning framework that in
https://
[e
et
al
droid, Web
Wang et ah.
tem
github. com/
2025
2024]
THUNLP MT/
troduces a Credit Assignment (CR)
CollabUlAgents
strategy, using LLMs instoad of environment-
specific rewards to
enhance performance and generalization.
Agent
$2
Ubuntu, Windows, Android
GUT shot
screen-
Standard operations and system APIs
UT
Claude 3.7-Somnet, Claude- 3.5 3.5-Somne,, GPT-lo (for Manager and Worker roles), UL Mixture of
Compositional multi-agent architecture with a Manager planning, Worker grounding tive replauning long-
Features Mixture of Grounding technique and Proactive Hierar- chical Plaaning, en- abling more accurate adap-
https://
Agashe et al.
github. com/
2025
Aimularaa
Agent-S
execution, and
horizon tasks
TARS- Crounding ex- perts
T2B-DPO,
Tesseracl OCR, and UNO
(for grounding
experts)
NaidNav Hhl
Android and Weh
GUI screen- shots
Standard operations and system APIs
GPT-lo, Gemini
Single-ogent
Introduces novel pro- oes reward model that provides fine-grained, step-level leedback to enhance GUI task curacy and success
e5
Flush, Qwen- VL-Plus
Scale' Track Huang et al.
Web, An- deoid Mobile, and Deaktop Computers
GUI screen- shots
Standard GUI operations
Qwen2-Vi VI.
Single-ngent
First GUI agent frame- work to introduce back- tracking learning not only the next action but also historical ac- tion sequences
7B
2025a
Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Sacta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wicting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023. URL. httts://aars: wwwaaaii.rgg.aaa..eeaae httg/bba/2300 10403
107

---

Published in Transactions on Machine Learning Research (06/2025)
Table 33: Overview of datasets for optimizing LLMs tailored for web GUI agents (Part 1).
Dataset
Platform
Source
Content
Scale
Collection
Highlight
Link
Method
Wend
Web
Crowdssurreed
Task descrip- tious, action sequences, web- page snapshots
2,350
tasks
Human demon-
Develops gener- alist web agents with diverse user interactions on real- weal-world web- sites Emphasis on dynamic evalua-
httpe://
Deng et 2023
from 137
strations
osu-nip group.
websites
github. 10/
Mind2keb/
Wind2Web-
Web
Sampled and Textual task re-annotated descriptions, with internetiate detailed annotation steps.
542 tasks, 4,550
Annotated by human experts.
https://
Live ct al.
Pan
hugging angginaaae
2024b
from the Mind2Web
tion using "key
co/
nodes", which represeut criti- cal intermediate states in web tasks.
Ratasets/
ct al. 2023
states, action sequences, and metadata, GUT screenshots
iMeanAI/
Mind2web-Lud
WebVLN
Web
Hunan- designed,
Text instruc- tious, plans, GUI scroen-
8,990 naviga- paths, QA
WobVLN simulator, LLM- generated pairs
Visiou-and- https:
Chen ct
tion 14,825 pairs
language nav- //ithub. igation
2024c
LLM-
QA
com/WebVLN/
generated
shots, HTML
human-like web browsing
WebVLN
WebLINX
Web
From human experts
Conversational interactions, strations action se- quences, DOM 100,000 inter- and screen- shots
2,337 demon-
Annotated by human experts
first large- scale dataset de- signed to eval- uate agents in real-world con- versational web navigation
https://
Lu
65
al.
mogil1 Degill-nlp.
2024c
with over
github.
weblinx/
actions
Agent Trek
Web
Web als
futori-
Task metadata, step- step-by-stey instructions, action se- quences, visual observations, reproducible native traces
4,902 trajec- tories
VLM agent guided by tu- torials, with trajectory data
Synthesizes sugh-quality
Xm
et
al.
2024j
Playwright cap- by leveraging
turing the tracess
web tutorials
Anthropic. Introducing computer use, new claude 3.5 sonnet, and claude 3.5 haiku, 2024. URL https www.aaattrrrpicccoommmn//////// candeel--and-ooopteeeee usee Accessed: 2024-10-26.
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Palcka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jemner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404. (04,::2404009932, 2024.
appium. Appium: Cross platsor automation framework for all kinds of apps, 2024. URL https://appiunn io/doca/en/latest/ Accessed: 2024-11-05.
Yauhen Leanidavich Arnatovich and Tipo Wang. A systematic literature review of automated techniques for functional gui testing of mobile applications. arXiv preprint arXiv:1812.11477 2018.
Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cārbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understanding, 2024. URL https://arxxvvoorr/bbb/222
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generic multimodal representations for ui understanding. arXio preprint arXim:2107.13791 2021.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-contre agents with autonomous reinforcement learning, 2024. URL https
11896
108

---

Published in Transactions on Machine Learning Research (06/2025)
Table 34: Overview of datasets for optimizing LLMs tailored for web GUI agents (Part 11).
Dataset
Platform
Source
Content
Scale
Collection
Highlight
Link
Method
MultiUT
Liu
Web
Combination
Textual task descriptions, instruction samples from million
7.3 million
LLMs and Play- wright
Supports httpe:
et aL.
20/4
human- designed instruc- tions and automated extraction from web structures
broad range
//neulab.
plans, action sequences, GUT screenshots, sc- crssibility trevs, bounding box annotations
Ul-related tasks, including GUT under- standing, action prediction, and
github. /
MultiUI/
websites
element ground- ing.
Explorer Pahuja et sl.
Web
Popular Textual task descriptions, cessful web
94K suc-
Generated by multi-agent LLM pipeline
Largest-scale web trajectory dataset to date;
URLs with systematic Action quences, GUI screenshots, Accessibility trees, HTML content Web gation natural language, task plans ac- tion soquenoes, HTTML based observations converted to markdown, and evalua- tions from LLM-based judges
2025
trajectories, 49K unique URLs, 720K
web explo- ration by LLMs
dynamically explores web pages to create contextually
screenshots
relevant tasks
InSTA
Trae
Web
Automatically
150,000 tasks across 150,000 web- sites
Generated by LLMs using
Presents fully automated
https://
generated by
agtenso
2025
LLMs across
the Playwright API and filtered by LLM-basod
three-stage
github.1o
1M websites
data generation pipeline task generation, ac- tion execution, and evalua- tion Mion-using only language models without any human annotations
from
Com-
mon Crawl
judges
Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning q-value functions for training devico-control agents. arXiv preprint arXi::2502.1500 2025a.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wi, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023a. URL https::///// /ttpps://arxiv org/
abs/ /200 16609
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large laalunlaaggage model with versatile abilitics. arXiv preprint ar arxiv:230.. 12966, 2023b.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song. Kai Dang, Peng Wang, Shijic Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025b.
Baidu Research. ERNIE Bot: Baidu's Knowledge Large Language Model Built on Full Al Stack Technology, 2024. URL https://researrt baidu. 9-November-20244)
[Online; accessed
Mohammad Bajammal, Andrea Stocco, Davood Mazinanian, and Ali Mesbah. survey on the use of computer vision improve software engincering tasks. IEEE Transactions on Software Engineering. 48(5): 1722-177 1742, 2020.
109

---

Published in Transactions on Machine Learning Research (06/2025)
Table 35: Overview of datasets for optimizing LLMs tailored for mobile GUT agents (Part 1).
Dataset
Platform
Source
Content
Scale
Collection
Highlight
Link
Method Generated
VGA ct 8..
Meng
Android Mo- bile
Rico
Deka
GUI screen- shots, task stances,
63.8k
by
Prioritizes
vi-
httpe://
2024
et al.
2017
GPT-4 models
sual content to reduce inaccura- cles
github. com/
descriptions, 22.3k
Linziyang
action se- quennes, bounds, layout, and functions of GUI ele- ments Gathered Textual data, screenshots, ac- tion sequenoes, UI structure, 10,811 annotated UI representa- tions Natural guage instruc- tions, action sequeness, GUT scrcenshots, syuthetic
instruction-
Visionk
following data pains;
20GUIX
2Dassistant
41 conver-
sation pairs
data
Rico ct
Deka 2017
Android Mo- bile
72,219
Crowdsourcing, automsted ex- ploration
Comprehensive datasct for mobile U1 de- sign, interaction modeling, layout generation
http://ver
from real Au- droid apps Google Play Store
unique UI
interaction
scroens,
org/
interaction traces
Hixp
Android Mo- bile
Human, web "How-to", Rico UT cor- pus syuthetie
187 multi- step structions, 295,476
Human Pioncering https:// notation and synthetic gener- ation
2020s
method
github. com/
grounding nat- ural language instructions to exccutable mo- bile actions
google-renearch
google-resarcch
tree/
structured U1 data Natural lan- written guage instruc- actoss 125
singlo-step
master/
commands
seq2act
MoTIF Burns ct ml.
Android Mo- bile
Human-
6,100 tasks
Human annota- tion
Tasl feasibility prediction for in- teractive GUI in nobile apps
https://
2022
github. comm
com/
tions, action Android sequeness, GUI screenshots, apps
aburns4/
MoTIF
structured U1 data
META GUI
Android Mo-
SMCal low n- Dialogues, ac- tion sequences, sereenshots, Android view hierarchies
1,125 dia- logues and 4,684 turns
Human annota- tion
Task-oriented
https:
Smn
bile
dreas al 2020
dialogue system
7/x lance. github. 10/
et al.
2022
for mobile GUI without relying back-end
NETA-GU
APIs
ATTW Rawles 2023
Android Mo- bile
Human- generated instruc- tions, LLM-
Natural guage instruc- tions, UT and 30,378 screenshots, generated observation- action pairs
715,142 episodes
Human raters using Android emulators
Large-scale dataset device control research with extensive app and diversity
https://
github. com/
google research/
unique structions
google- researrr
prompts
master/
android
in the wild
Shimshon Berkovits, Joshua D. Guttman, and Vipin Swarup. Authentication for mobile agents. In Mobile Agents and Securily, 1998. URL https://api. semanticscholl.r org/Corpuslip : 13987376
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXio preprint 2r7.::240..07726, 2024.
Benyamin Beyzaci, Saghar Talebipour, Ghazal Rafici, Nenad Medvidovic, and Sam Malck. Automated test transfer across android apps using large language models. arXio preprint arXiv:2411.1795. 2024.
William E Bishop, Alice Li, Christopher Rawles, and Oriana Riva. Latent state estimation helps ui agents to reason. arXiv preprint orXix:2405. 111000 2024.
Anjanava Biswas and Wrick Talukdar. Guardrails for trust, safety, and ethical development and deployment of large language models (1m)). Journal of Science & Technology, 4(6):55 82, 2023.
110

---

Published in Transactions on Machine Learning Research (06/2025)
Table 36: Overview of datasets for optimizing LLMs tailored for mobile GUI agents (Part II).
Dataset
Platform
Source
Content
Scale
Collection Method
Highlight
Link
GUI
Android Mo-
Human designers, GPT-4
Textual tasks, plans, action sequenecs, GUT screenshots
7,735
Human demon- strations
Focuses
https://
Odyssey
Lu
bile
episodes
app navigation
github. com/
al.
2024b
across
201
tasks mobile devices
OpenGVLab/
apps
GU1-Odyasey
Chai
Android Mo- bile
Human- designed, ChatCPT-
Text tasks, ac- tion sequenoes,
104,000
Human
Multi-level,
https://
2024
screenshots, 1.6 million interactive scripts elements,
notations, autonomous
large-scule
yuxianghhai
high-res screen- shots with multi-level annotatious
annotations sup-
github. /0//
gencrated
porting complex mobile tasks
AMEX/
2,946 instruc-
tions
crret-
iOS, Audroid Mobile
Sigtlight datasct, GPT-4
Text tasks,, ac- tion plans, GUI
40,000 el ementary
GI Tgge-errtted
Benchmaakk for Ul-centric tasks with adjustable screen aspect ra- tios
https:
UI You et sl. 2025
//gthub.
clementt amno- tations, bound- ing boxes Screen-action
tasks, 10,000
com/apple/
advanced
m1 mi-ferret
tasks
AITZ
Zhang 2024g
Android Mo- bile
AITW Rawle
18,613
GPT-4V, detection
icon mod-
Structured
attpe://
et al.
et al. 2023
pairs, action
screen-
"Chain-
github. ..tlub.com/ com/
descriptions
action pairs
els
of of-Action-
INNearth/
acmess 70+
Thought
CoAT
apps, 2,504
hancing GUT
episodes
navigation
Octo- planner Chen
Android Mo- bile
GPT-4 gener- ated
Text tasks, decomposed
GPT-4 points ated
1,000 data
gener- Optimized task planning
https://
hugginglas
2024f
plans, action sequences
with GUI
tions
RexaAIDev/
octopus Eitopus-planning
Android tiny- apps
Human haviors scriptions, screenshots, ac- tion sequenoes, page element data
be- Task de- traces, tion
40,000+
Human annota-
First large-scale Chinese dataset
ANT
Wang 2024F
ct ml.
10,000 action inteuts
for GUI navign- tion with real hu- man interactions
111

---

Published in Transactions on Machine Learning Research (06/2025)
Table 37: Overview of datasets for optimizing LLMs tailored for mobile GUT agents (Part ITI..
Dataset
Platform
Source
Content
Scale
Collection
Highlight
Link
Method
Mobile3M
Android Mo- bile
Real-world
UI screenshots,
3,098,786
Simulation algo-
Largo-scale
https://
Wu
ct
interactions, simulatious
XML
docu-
UI pagos,
rithm
Chinese mobile
github. com/
2024c
action
20,138,332
GU1 dataset
Meituan out
sequences
actions
with unique
MobileVLN
igation graph
AndroidLab ct.
Android Mo- bile
Humau design, LLM
Text instruc- tions, action self- sequenees, XML data, semeeshots
traoes, 94.3k steps
Human aunota-
XML-bused interaction data
https:
tion, LLM self-
//gitubb
20246
exploration
with unified
com/ COm/THUDM/
exploration, academic
action space
Android-Lab
datasets
MobileViews iews Gao et al.
Android Mo- bile
LLM-
Screenshot- view hierarchy
600,000 screenshots, VH pairs from 20,000+ apps
Largest open- source mobile screen dataset
https://
enhanced app traversal
crawler
hugging aagaee acee co/
2024e
pairs
tool
datasets/
mi milmTeam/
MobileViezz
FenMABench Android Mo- bile
AndroidContsol Textual task
6 dataset
Inferred from isting Android datasets
The first dataset designed benchmark
https://
Wang al.
I
al.
descriptions, action
ries with 30 subsets
to federated mobile GUT agents
github. .omm/
2025d
2024e
wwh0411/
ATTW
quences, and
FedMABanch
Rawles et al.
GUT
2023
shots
GUI-Xplore
Mobile An-
Combination automated exploration and mamal design
Exploration vidcos, textual
apps, hours vidco, 32,569
Automated and human explo- ration
Introduces am exploration- based pretrain- ing paradigm that provides
https://
Sun et
github. com/ 921112343/
2025b
tasks, QA pairs, view hierarchies, GUI screen- shots, action sequences, and
QA pairs,
Cploxee
41,293 ac-
tions, about 200 pages
rich app-specific priors through video data
per app
GUT transition
graphs
112

---

Published in Transactions on Machine Learning Research (06/2025)
Table 38: Overview of datasets for optimizing LLMs tailored for computer GUI agents.
Dataset
Platform
Source
Content
Scale
Collection
Highlight
Link
Method
AcreenAgent
Limux, Win- dows OS
Human- designed
GUI scroen- shots, action sequences
task ses-
Human annota-
VLM-based
attpe://
Niu et
sions, 3,005
tion
agent multiple desktop environments
across
github. com/
2024
training screenshots,
niuzaisheng Screenkgent
898 test screenshots
LAM
Wang 2024h
Windows OS
Application
Task descrip- tions in natural
Instantiated us- ing GPT-4, with task-plan pairs, 2,192 actions tested
76,672
Provides a struc- tured pipeline for collecting,
https://
ct. al.
documenta-
github. com/
tion, Wiki- How articles, Bing search queries
language, step- by-stop plans, action sc- quenoes, GU1 screenshots
mi microsoft/
task-action trajectorics
and validated in the Windows euvironment us- ing UFO Zhang et al. 2024a
validatingg and sugmenting data, enabling high-quality training for
UF0/tree/
main/
dataflow
action-oriented AI models.
DoskVision Xu et
Windows, macOS, and Limux desk- tops
Internet
GUI 54,855
elements detected using OmniParser and PaddleOCR
The first large- scale, open- source dataset focusiig on real- world desktop GUI secnarios across operating systems
shots with screenshots annotated with 303.622 62..22
2025
bounding UI element annotations
boxes for UI clements and detailed region captions
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. URL https:// org/ wwtp///.......e..
Mark A Boshart and Martha Kosa. Growing gui from an xml tree. ACM SIGCSE Bulletin, 35(3):223-2) 223, 2003.
Jeffrey M Bradshaw, Paul J Feltovich, and Matthew Johnson. Human agent interaction. In The handbook of human-machine interaction, pp. 283-300. CRC Press, 2017.
Paul Brie, Nicolas Burny, Arthur Sluýters, and Jean Vanderdonckt. Evaluating large language model on searching for gui layouts. Proceedings of the ACM Human-Computes Interaction, 7(EICS):1-37, 2023.
Cameron Browne, Edward Powley, Daniel Whitchouse, Simon M Lucas, Peter Cowling, Philipp Rohlfsha- gen; Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in gomes, 4(1): 43, 2012.
Andreas Bruns, Andreas Kornstadt, and Dennis Wichmann. Web application tests with selenium. IEFE software, 26(5):88 91, 2009.
Andrea Burns, Deniz Arsan, Sanjua Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vision-language navigation with unknown command feasibility, 2022. URL https://arxiv. orgooo////ooo abs/ab/2..2222o2 02312
Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, and Tat-Seng Chua. Large
language models empowered personalized web agents. arXiv preprint ar Xrxi::2410 10.. 17236, 2024a.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey mixture of experts. arXiv preprint arXiv:2407 05.::2407,0244, 2024b.
Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Navid Nobani, and Andrea Seveso. XAI meets 1lms: A survey of the relation between explainable ai and large language models, 2024. URL https://arxiv.orr
aba/2407.C 15248
113

---

Published in Transactions on Machine Learning Research (06/2025)
Table 39: Overview datasets for optimizing LLMs tailored for cross-platform GUI agents (Part 1).
Dataset
Platform
Source
Content
Scale
Collection
Highlight
Link
Method
Visual-
Android Mo- bile, Web
VAB-Mobile:
TTask instruc-
VAB-Mobile:
Program- solvers, agent
Systematic eval- httpe:
AgentBench
Android
tious, action
1,213 trajec-
uation of VLM as a visual foun- dation agent across multiple scenarios
//gthhu.
Liu et
Virtual
sequences,
tories, 10,175
bootstrapping, human demon-
Com/THUDN/
2024:
Device, VAB-
screen observa-
steps: VAB-
VieualAgeat
WehArena-
tions
WehArena-
strations
Lite:
We-
Lite: 1,186 trajectories,
bArena Koh
et al.
2024a
9,522 steps
GUICourse
Android Mo- bile, Web
Web
scrap-
GUI
screen-
10 million
LLM- based
Datasct suite for cubancing VLM GUI navigation web and mo- bile platforms
https:
Chen
ct
ing.
simula-
shots, action
website page- annotation
auto- annotation,
//github.
20245
tion, mannal
sequences,
com/yiye3/
design
OCR tasks, QA pairs
pairs, 67,000 crowd-sourcing action in- structions
GUlCourse
GUI-
OS, Mobile, Web, XR
GUI videos, workers, human- annotated 83,176 instructional keyframes, videos captions, QA data, action sequenoes Crawling Screen an- Annotation: Model, human notation, hundreds webpages, sereen QA, millions;
Student
12,000 videos,
Human annota- tion
Designed for dy- namic, soquen- tial GUI tasks with video data
httpa://
World
Chen
gui-world.
2024cc
YouTube
github.10
frames
ScreenAT Baechler
Android, iOS, Desk- apps and top/Wer
Comprehensive pretraining and fine-tumin GUT tasks across platforms
https://
annotation
github. com/
d al. 2024
googleX
synthetic QA
navigation, sumumerization millions; Navigation:
QA: tens
2Dresearch2
20datasets/
screen
millions
annotation
Web Hybrid
Web,
An- Web-
Screenshots, text-based referring
10 mil-
Rale-based syn- thesis, LLMs for referring expres- sions
Largest dataset for GUT visual grounding
https://
Gon
al.
droid Mobile synthetic data
lion GUT
osu osu-nlpe
2024
uX- elements, pressions, 1.3 million screenshots
github. i0/
UGround/
coordinates on GUTs
GUIDE
Computer and Web
Direct sub- missions tions, GUI screenshots, tion sequences, survey CoT reasoning, sponses
Task descrip-
N/A
Collected through NEX- TAG, auto- mated annota- tion tool
Iutegratos https://
Chawla
ages, action sequenoes, task descriptions, and spatial grounding into unifted dataset
g github.com com/
2024
from busi- and
superagi/
GUIDE
spatial ground- ing
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, and Tao Yu. Spider2-v: How far are multimodal agents from automating data science and engineering workflows?, 2024. URL https: //apsii///rxvv...// arxii.iiiv...aa org/ hbsg/abs/200 2407 10956
William B Cavnar, John M Trenkle, al. N-gram- based text categorization. In Proceedings of SDAIR R 3rd annual symposium document analysis and information retrieval, volume 161175, pp. 14. Ann Arbor, Michigan, 1994.
Hyungjoo Chac, Namyoung Kim, Kai Tzu iunn Ong. Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment. dynamics in web navigation, 2024. URL htttp:://araat org.eiiivorrreeee wwwp/rrg/g/baaaeeeeeeee awwp/aa/....aeaaaaa 13232.
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shnai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents, 2024. URL 17490
https://arxiv..rrt
114

---

Published in Transactions on Machine Learning Research (06/2025)
Table 40: Overview of datasets for optimizing LLMs tailored for cross-platform GUI agents (Part II).
Dataset
Platform Source
Content
Scale
Collection
Highlight
Link
Method
Zhang 20/24d
and
Syuthesized data, existing dataset.
Textual tasks, action quences, function- calling data
60,000 data
Collected using models with
Provides anified format across diverse environments, enhancing gener- alizability and error detection for GUT agents Iustructiou-free paradigm and auto-
httpe://
ct al.
tools used
se-
points
github. com/
human verifica- tion steps
Selesfoorce]
XLAN
Insight-
iOS, An-
Common Crawl corpus
Textual tasks,
434,000
Automatic simulations performed by
Shen al.
droid, dows, Limux,
Win-
plans, action
episodes,
2024a
sequenecs, GUI
1, 1,456,000 000
cutirely
Web
scrcenshots images
browser API
generated
OS- Gemesis
Web and An-
Reverse task syuthesis, instructions, sized trajec- where the GUI cuvi-
High-level
1,000 synthe-
Model-based interaction-
Reverses the conventional task-driven data collection pro- enabling
https://
Sam
droid
giushisun.
2024b
low-level in- structions, roument action explored quences, and interactively enviromment without pre- defined tasles or human annotations.
tories.
driven approach with CPT-4o.
github.io
08 08-Genesis-B
exploration-fiss trajectory syn- thesis.
states.
Navi-plus Cheng et al.
Web and An-
AndroidContril Task de- scriptions, (204e and GUI action
L.M-- automated
Introduces Self- Correction with human validation
droid
Li
ml.
2025b
GUI Navigation task featuring the novel ASK action for recov- ering missing information
Wind2Web trajectorics, low-level step instructions, sereenshots, and follow-up ASK/SAY interaction pairs
Deng ct al.
2023
Explorer
Web and An- droid
Automated traversal of real websites and Audroid apps
UT screenshots, bounding boxes of in- teractable elemnnts, similar- ity labels, and user actions
KhanAcademy (Web): 2,841 interactables, 378
Automated tools, HTML parsing, Accessi- bility Tree
Platform- https://
Chaimalas
independent, github. com/
2025
supports varnelis/ Explorer
labeling, and cuables trace recording and voice-controlled
similarity
samples;
Spotify (An- droid): 1,207
GUT navigation
interactables,
451 screen similarity samples
Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, and Hongsheng Li. A3: Android agent arena for mobile gui agents, 2025. URL http::///aarii.oo//ab///////
Iason Chaimalas, Arnas VyL.Aniuuskss, and Gabriel Brostow. Explorer: Robust collection of interactable gui elements. arXiv preprint or orXin:2504.09952, 2025.
Tathagata Chakraborti, Vatche Isahagian, Rania Khalaf, Yasaman Khazaeni, Vinod Muthusamy, Yara Rizk, and Merve Unuvar. From robotic process automation intelligent process automation: -emerging trends-. In Business Process Management: Blockchain and Robotic Process Automation Forum: BPM 2020 Blockchain and RPA Forum, Seville, Spain, September 13 18, 2020, Proceedings 18, pp. 215 228. Springer, 2020.
Chi-Min Chan, Jianxuan Yu, Weize Chen, Chunyang Jiang, Xinyu Liu, Weijie Shi, Zhiyuan Liu, Wei Xue, and Yike Guo. Agentmonitor: A plug-and-play framework for predictive and secure multi-agent systems.
115

---

Published in Transactions on Machine Learning Research (06/2025)
Table 41: Overview of foundation models for LLM-brained GUI agents (Part 1).
Model
Modality
Model Sixe
Architecture
Training Methods
Highlights
Open-
Link
Source
GPT-4o Huret
Texl audio,
Multinodal autore- gressive ture
Pre-trained on a mix public data, further trained for alignment human preferences safety considerations
Unitied mullinodal archi-
No
a.
architec-
tecture that seamlessly processes
2014
image, and video
generates oulputs acress ted, dio, image, and video, of- fering faster more cost- effective operation prodermsorr
and
GPT-4V OpenAT
Text and Image
Pre-trained dataset text and im- age data, followed by fine- luning with reiuforcement learning from luman food- back (RLHF)
large
Notable for its multinedal capabllitiss, allowing it to analyze and understand imngee alongside text
No
2023
Gemini
Text, image, andio, and video
Nano ver- sions: L.8B/3.55B
Enhanced Trans- former decoders
Large-scalo le pre-traiiing on multimodal data, followed by supervised fine-tuning, reward modeling, and RLHF
Achieves state-of-the-art No performance across mud- timodal tasks, including groundbresking 90% on the MMLU benchmark, demonstrates capacity for ou-device deployment with small model sizes Pioncering role in GUT an- tomation as the first pub- beta model utilise
Team et sl 2023
Claude
Text
ReAcct-buer reason-
No
3.5 Sonnet
and
ing
(Computer
image
Use)
An-
vision-only paradigm desktop task automation
Ahropic
20/24
Hu
et sl.
20
Operator
Text
Buil on GPT-4o
Supervised learning and inforcemeeal learning
Trained computer No
OpenAl
and
like human, achieving markable performance beachunaries stages pre-tzainigg and final stage of instrue-
2025p
Image
Qwen-VV VL
9.6B
Viston Trans-
Achieves state-of-the-art performance on vision- language benchnaarks and supports fine-grained vi- sual understanding
httpes: //gitaub.
and
former (VIT)
2023b
image
Desovitskiy al.
tion fine-tuning
con/ /WennM/ Qwea- VL
2021 as the visual
encoder, with a
large
language
model bused the
Qwen-7B architec-
fure
Qwen2-VL
Text, image, and video
2B/7B/72B
VIT Dreovitekly
ViT is trained with image-text paits; all pa- rameters are unfromen for broader multimodal learn- ing with various datasets; fine-tuiing LLM struction datasets
Introduees Naive Dynamic Resolution for variable olution image processing and Multimodal Rotary Position Embedding for ea- hanced multinodal gration
httpes: //github. con/ QweniM/ Quen2-Vi VL
Wang
22221
20241
the vision encoder,
paired with the
Qwen2 series
language models
arXiv preprint arXiv:2408.14977 2024.
Tsung-Hsiang Chang, Tom Yeh, and Robert C Miller. Gui testing using computer vision. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp.
2010.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan WI, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
Rajat Chawla, Adarsh Jha, Muskaan Kumar, Mukunda NS, and Ishaan Bhola. Guide: Graphical user interface data for execution. arXiv preprint ar arx::24404.1000 16048, 2024.
Chaoran Chen, Zhiping Zhang, Bingcan Guo, Shang Ma, Ibrahim Khalilov, Simret Gebrocgziabher, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, al. The obvious invisible threat: Llm-powered gui agents' vulnerability to fine-print injections. arXiv preprint ar 1xXiv:2504.11281 2025a.
116

---

Published in Transactions on Machine Learning Research (06/2025)
Table 42: Overview of foundation models for LLM-brained GUI agents (Part II).
Model
Modality
Model Sixe
Architecture
Training Methods
Highlights
Open- Sourch
Link
InterrvV 2
Texl image, video, and mediral data
1B/28/48/ 8B/26B/ 10B
VEI us the vissou en- coder and a LLM as language compo- nent
Progressive aligunmnt strategy. starting with fine data
Demonstrates poweriul capabiities in handling complex mudtimoddl lasks
Yes
httpee:
Chen et sl.
//interavl
and moving
gitbub.
with various model sizes
ia/blog/
204-077-2 InternVl.22.
0/
CogVLM
17B
encoder,
Stage focuses on itt- age captioning: Stage combines image caption- ing and referring expres- slon compreheusion tasks
Achieves deep integration visual and language lea- tures while preserving the full capabilities of large language models
httpes:
Wang
and
two-layer MLP
//gttbub
Jmu image
adapter, pre-
con/ THUDN/
trained large
CogVLM
language model,
and visual expert module
Perret
You
Text and image
7B/113 Decoder-only archi- tecture based on the model, com- bined with visual encoder
A combination of super- vised training and addi- tional instruction tuning
Ability to handle fine- form region inputs via its hybrid region representation. en- abling versatile spatial derstanding and grounding Its weigt architecture enables quick experimenta- tion, demoustrating caga- bilities cluse to GPTT in multimodal reasoning Notable for its data effi- ciency and scaling to high- resolution image inpute
Yes
httpss:
ct al.
2023
/github.
con/ con/app apple/ al-ferret
VA Liu
Texl and image
7B/13B
A vision encoder (CLIP VIT-1,/14); language decoder (Vicuna)
Pre- training using tittend image-text pairs, fine- tuniug with mulinooda instruction following dataset A vision encoder (CLIP-ViT) and encoder-decoder
Yes
httpee:
e al.
2024b
//Llava-v1
github.io
LLL 1,5
Text and image
7B/13B
Pre-training on vision- language alignment image-text paizs; visual instruction tuning with specific tusk data
Yes
httpss:
L10 et
7/Llava-u vl.
github.io
LLM architecture
(e.g.. Vieuma or LLaMA]
BLIP-2 2023cc
3.4
frosen image en- coder, lightweight Querying Trans- former to bridge the modality gap, and frozen large language model
Vision- laiionlanguagg represen- tation learning: trains the Q Fororr with frozen image encoder; Vision-to- language geuerative leam- ing: connects the Q- Former to frowen LLM to enable image-to-text gener- ation
Achieves state-of-the-art performance various vision-language tasks with a compute-efficintt strat- egy leveraging frozen pre-tzained models
httpss://
and imago
github
sa lasforcee/
LAVIS/
tree/mand eree/niin///
projecta/ blap2
Phi-3.5- V Vision
Image encoder: CLIP VIT-L/14 to promess visual inputs. and trans- former decoder based the Phi- 3.5-mini model for textual outputs
Pre-training on a combina- tion interleaved image- text datasets,, synthetic. OCR data, chart//able comprehension data, and text-oaly data; supervised fine-tuning using large- scale multimodal and text datasets; Direct Preference Optinization (DPO) improve allgument, safety, and multimodal task per- formance
Excels reasoning over visual and textual puts, demonstrating com- petitive performance on single-image and multi- image tasks while being compact
httpss://
Abdin
and imago
github. con/ microsoft//
w ml.
2024
Phi -hoo3CooBBopk// tree/nain
Chaoran Chen, Zhiping Zhang, Ibrahim Khalilov, Bingcan Guo, Simret Gebreegziabler, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, and Toby Jia-Jun Li. Toward a human-centered evaluation framework for trustworthy llm-powered gui agents. arXiv preprint ar arXiv:2504. 1934,, 2025b.
Daihang Chen, Yonghui Liu, Mingyi Zhou, Yanjie Zhao, Haoyu Wang, Shuai Wang, Xiao Chen, Tegawendé F Bissyandé, Jacques Klein, and Li Li. Lim for mobile: An initial roadmap. arXiv preprint (07.072,07006579, 2024a.
Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Gc, Dawei Gao, Yuexiang Xic, Zhaoyang Liu, Jinyang Gao, et al. Data-juicer: A one-stop data processing system for large language models. In Companion of the 2024 International Conference on Management of Data, pp. 120- 120-134, 2024b.
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan,
117

---

Published in Transactions on Machine Learning Research (06/2025)
Table 43: An overview of GUL-optimized models on web platforms.
Model
Platform
Foundation
Size
Imput
Output
Dataset
Highlights
Link
Model
Ageett
Web
LLaMA-3
70B
HTML DOM
Plane, thoughts, actions, and action expla- nations
WelShop bench- mark OpenTable dataset.
Combines Monte Carlo Tree Search (MCTS) with self-aritique mechanisms, levrraging reinforeement
bttpe: github con/
Pusta
7033 Dubey
regresentations
and
2024
et al
2024
sentients
Danginaer
learning to achieve excep- tiomal performance
agent-
GLAINTEL
Plan-To Chung al
780M
Tser tions
Instrue- and
GUT actions
roal- products, crowd-
Efficient use smaller
Pereic
world 12,087
LLMe, and integration of RL and human demon-
A Siddique
2024
observations. webpage state
sourced strations for robust perfor- manee
language intents, 1,010 Iuman demonstratioos
Wo
Web
T5
Raffel
and with screensbots
MTVL. DOM
Hierarchical
MiniWoB 4+4, 13,000 hurau-- demon- strations
Combines supervisod learning and reinforee- ment learning address linitations of previons models in memorization and generalization
Tall et
ct ml.
2020
novigation
20124
plans GUI interac- tions
made
OpenWWeb-
Web
Idefice22
8B
GUI screenshots, accessibility
Actions on Mind2Web GUL, plan- and Web ning and ager datasets, thought, and generated answers queries queries reul-world web uavigation WebArema-Late Introduces self-evolving
Combining imitation learn- ing with feedback loop for continuous improve- ment
bttps://
Voyaueer
He
Sb-iustruct
github. coo/
2024c
Laurengon
Jerry/ Opankobloyager
ct nl. 2024g
WobKL
Llama-3.1
SB/9B/
Instruc- Actions, tioms, action clemens history, HTML content or notes
Task
bttpe:
2024
Dobey ml.
70B3
online currieulum rein-
7/ //tthub.
20:24
and
ideutißers, explanatioos
forcement learaing frame-
con/ THUDH/
work, which dynamically generates tasks based past failures and adapts to the agent's skill
Kebkl
et 2021
WebGUM
Web
Flan-15
3B
HT ML, screen- shote, history, instructions
Web naviga- tion actions and free-Eorm text
MiniWol3-- and WebShopp benchmarks
Integrates temporri and lo- cal muitim siltimldsl perception. combining HTML and susl tokens, and uses diemectionfiintunee lan- guage model for enhanced reasmingg and task gener- alization
bttps:
Purnta et a
Chung et al
interae-
7 /ccaeole.
20133
slon Trans- Former (VIT)
google. con/ storage/
itskly
broweer/
cT. ml. 2021
gresearch/
weblim
Pan Zhou, Jianfeng Gao, and Lichao Sun. Gui-world: A dataset for gui-oriented multimodal Ilm-based agents, 2024c. URL https://arxiv... org/ htt/a/bb//22.../// 10819
Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. Object detection for graphical user interface: Old fashioned or deep learning or a combination? In proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1202-1214, 2020.
Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. In NewrIPS 2024 Workshop on Open- Open-World Agents, 2024d.
Mark Chen, Jerry Tworck, Hcewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. XrXiv preprint ar orXii:2107.088 03374, 2021a.
Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Hsiang-Ting Chen, and Qi Wu. Webvin: Vision- and-language navigation websites. Proceedings of the AAAT Conference Artificial Intelligence, 38(2): 1165 1173, Mar. 2024e. doi: 10.1609/ v382...38...77877. URL https://ojs. aaai org//nder: http:/innxxppp//AAAT.. article/view//2e00
Wei Chen and Zhiyuan Li. Octopus v2: On-device language model for super agent, 2024a. URL https: sarxiv.org/abs//40000 01744
118

---

Published in Transactions on Machine Learning Research (06/2025)
Table 44: An overview of GUL-optimized models on mobile platforms (Part 1).
Model
Platform
Foundation
Size
Input
Output
Dataset
Highlights
Link
Model
Mobile VLM
Mobile An-- droid
Qwen-V
9.8B
Screenshots and structured XML documentt
Action pre- dictions.
MobileM. In- cludes 3 million
Mobilespecilie pretradning
bt
Wu
Chat
Be
tasks that enhance intra-
//github.
202te
et al.
2023b
nav aavigation stope, element loca- tions
UI pages, 20- actions, and XML data structured as directed graphs
and Inter-UI understand- ing.
con/ X iioof
uniquely large and graph-structured Chi- nese UI datusct (Mo-
Octo-
Mobile viees
de-
Phi-3
Mini
3.8B
User queties aud wvallable fune- descriptions
Execution
1,000 data sam-
Optimized for resource- construined devicrs to en- lateney, privacy: and offline functionality
beipe://
planner
Abdim ot a.
stops
ples generated
bugg Baggingfan ace
Chen
21/44
wsing GPTT4
2024F
Nexak Davv/
octopus planning
AntoUl Zhang
1.3B
Sercenshots
OHfine-to-anline
bttpa://
et
ml.
2024
droid
&
mont learning, bridging
g ththh. con/
Zhang
20124
gepe in static and dynamic euvironments
Dig Rl. agent/
digit
IVG
Quan
Mobile An- droid
Swin
Trans-
UT screenshots and free-form
Bounding box eoordi-
UlBert dataset syuthetic grounding tuslss through
Unifies detection and
ct
2024
former
Lam
and
language expeees stous
syouttguuiedd coutrastive learning
and
BERT
Devla
2018
Ferret-UP
Android and PPhne plat- forms
Ferres
You
713/ 131
Raw screen pix- els, sub-innges divided for Bluer resolution, bounding boxes and regional aumotations
Widget bounding boxes, from OCR
Generated from
Multi-platform support with higl-resolution adap-
https:
You et
ct al
2023
RICO (for An-
//github.
2025
droid) and AMP [for iPhone)
image encoding
apple/
m1 fesret/
tasks, seriptions of UI elements or func-
Lree/ /ee//naii
Farretui
tionality. UI interaction
actions
AppVLM Papoudakss
Android mo- bile devices
Paligemnn--
3B Anpotated
GUI actions
AndroidCouud
lightweight model that
Beves
screenshots
20
achieves near-G1" per- formance Android con-
sl. 2025
al
2024
bounding boxes
Audroid Fordd
and UT labels
Haxles
et al.
trol taskss while being 10x faster and more efficient.
2024
Digl-Q
Rai
Mobile droid
An-
L.VVA-15
7B
GUT servenshots
GUT actions, Q-values
Ait
Raw
Introduces V-bbob Q-function for GUI agent training, enabling rein- forcement learning without online interactions.
httpa://
20056
Li
2123
gitbub.co
2024n
Digt aggnn
digiq
VEM
Zhong 2025b
Mobile droid
An-
Qwen2VT.
7B
GUT scrvenshots
CUT actions,
W
Raw
Unlike traditional RL methods that require environment interactions.
bttpa://
c ml
Wang nl
Q-values
ct ml
2024
github.com coo/
2024
creaoftt
VEM enables training purely on offline data with Value Eaviromment Model.
QUI Agent-Rl
Wei Chen and Zhiyuan Li. Octopus v3: Technical report for on-device sub-billion multimodal ai agent, 2024b. URL
abs/2404 11459
Wei Chen and Zhiyuan Li. Octopus v4: Graph of language models, 2024c. URL https://arxii: org.abs/////aaaaooooooeboeeeeeae
Wei Chen, Zhiyuan Li, Zhen Guo, and Yikang Shen. Octo-planner On-device language model for planner- action agents, 2024f. URL
18082
Wei Chen, Zhiyuan Li, and Mingyuan Ma. Octopus: On-device language model for function calling of software apis, 2024g. URL htttpp::/aaxxvvorr//a///////////
Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Chong Yang, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Internet of agents: Weaving web of heterogeneous agents for collaborative intelligence, 2024h. URL https: https://arxiv. httpp::/aaxx...or//aa/////..... wttp://b////4..000055555
119

---

Published in Transactions on Machine Learning Research (06/2025)
Table 45: An overview of GUI-optimized models on mobile platforms (Part 11).
Model
Platform
Foundation
Size
Input
Output
Dataset
Highlights
Link
Model
Octopus
Mobile
CodeLlama-
7B,
API documenta--
Function
RapidAPI Hub
conditional masking enforce correct output formatting
Chen
vices
71
Rozière
2B
examples
names with
2024g
et x]
22221
arguments calls
Google
Gemma
2B
Tram et ml
21/24
Octopus v2
Edge devices
Gemmaa-28
2B
User and descriptions
queries
Function calls with precise pa- rameters
20 Android APIs, with up 1,000 points genorated for training
Fanctionsl Toketization stradogy, which assigns unique toleess to function calls, significantly reduc- ing the context length required for prodiction
hom
Ar
Tram et ml
21/24
available Functions
Octapus
v3
Edge devices
CLile-based model and a causal language
Lesss than billion pa-
Queries commands, images and func- tional tokens
and
Funccioonl tokeus actious
Leveraged from Octopus
Introduction functional tokens for multimodal ap- plications enables the rep- resentation any function token, enhaneing the moddl's flexiblitty
Chen
2124b
Chen &
Li
102
model
tets Vacies
Octopus v1
Serverless cloud-based
models
User queries
Domaiin- specifiie answens, actions
Syathetic datasets similar to Octopus v2
Grapar iranework in- tegrating multippe special- ised models for optimised performance
httpe:
hen
7/github.
2024e
platforns
con/ NoxaA!
odge
octopus-
devires
VGA
Meng
Mobile
An-
LLaVA-v1.G-
7B
GUI acreeushots with positional, visual, and hier- archical data
Actions and function calls, deserip- tious of GUI componeuts, navigation and task planning
63.8k 63.8k-image dataset structed the RICO
Minimizess bullucinations bttpe://
2024
mistesl-7B
con- from
GUT comprehen- by employying image-crutric line-tuiin approach, ensuring bul-
github. com/
Lin
mL
inziyangtSpp 1999/
2024bb
VGAX
anced acteation between
20GU1N
text and visual content
2Dase istant
MobileFlon Fow
Mobile phones
Qwen- VL-
21B
GUI screenshots
GUI actions
70k manually labeled business- specific data spanning 10 businesss sectors, and datasets like R-CCOCO. ScreenQA. Flickr30K
Hybrid visual encoder capable of varisble-
Nong
et al
Chat
Fai
with OCR tex- information
and question
auswering
resolution imput and
and bounding hoxes
Mixture of Experts (MoE) Cal et al. (222h)
enhanced performance and efficiency
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile gui agents, 2024i. URL https: wwtp://pww....////////////// .111//111..11177777
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Scbastian Goodman, Tbrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint orXiv:2310.09199 2023.
Yanan Chen, Ali Pesaranghader, Tanmana Sadhu, and Dong Hoon Yi. Can we rely on Ilm agents to draft long-horizon plans? let's take travelplanner an example. arXie preprint arXiv:2408.066 06318, 2024j.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. arXiv preprint arXiv:2110.072 07,,711,, 2021b.
Yurun Chen, Xucyu Hu, Keting Yin, Juncheng Li, and Shengyu Zhang. Acia-mn: Evaluating the robustness of multimodal llm-powered mobile agents against active environmental injection attacks. arXie preprinl ar 2rXiv:2502.1300 2025c.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arxiv preprint arXin:2404. 16821, 2024k.
120

---

Published in Transactions on Machine Learning Research (06/2025)
Table 46: An overview of GUL-optimized models on mobile platforms (Part I1I).
Model
Platform
Foundation
Size
Input
Output
Dataset
Highlights
Link
Model
UTNaY
Mobile droid
Au-
SmallBERT
Agent
UI elements, ut-
Predicted ac-
tacks across
Intzoduces a macco action framework and driven demonstration ool- procese, signif- eantly reducing training of- fort while enabling robust performance smalll, efficieut models sult- able for mobile devices
L1
Ture et
model:
screen repessentation
tious and el-
128 Android
2019
320k,
ement to act upon
apps and web-
sites, collecting 3,661 demon-
model:
strations
430k,
Small-
BERT
model:
17.6MB
D1-R1
La
Mobile
An-
Qwen2.5 5-L
3B
GUI acrecushots
Reasoning and actions
ScreenSpo and AndroidControl
Introduscss a rule bused re- inforoemeut learning ap- prosch using GRPO en- hance ressoning action prediction in GUT tasles with only 136 examples
bttps: 7/github
20225b
droid
con/ 1116/1
Ri
1M0
Ltto
Mobile droid
At--
Pre-Irsined Stable Diffu- sion model
Current GUT GUI text rep- Image, resentation tiom (in natural of the next languago),
Audroid Control and ATTW
First GUI world model that predlcts future visual GUI states
bttpe:
2025a
al-agenta-
github.
state constructed full GUI image (visunl prediction of the next screes)
ViNo/
2022
representa- tion
Table 47: An overview of GUI-optimized models on computer platforms.
Model
Platform
Foundation
Size
Output
Dataset
Highlights
Link
Model
Screem
Limix
and
CogAgent
18B
GUI scrveushots
Mouse
and
273 task sensions
Compreeeensive pipeline of planning, acting, and flecting to bandle real com- puter acreen operations tonounously
bttpe://
Agent
Windows
Hong
Kitbub.
2021
desttop
2123
actions
ciuzaishen
Screendgent
Octopus Yang
Dektop
MPT17B
7B
im-- Executabble ages, scene graphs action code taining objects relations,
Visual
OctoCibsoo: 476 tasks with strettree ini-
Incorporates reinfor cemenns learning with eavironmen-
bttpe:
31s
7choiazt.
2025h
2023
and
and plans
tal fredbark
github. 10/
CPPP
VIT-
tial and goal states;
Octopus/
I,/14
Ford
environment messages
toMC: 40 taskss
2021
across blomes;
OctoG'TA:
crafted
tasks
spanning
ferent
setsing:
LAM
Wane
Windows OS
Mistral-7B
7B Task: requests natural language, appli-
Plans, tions paits, 2,192 task- action trajecto-
Ar-
76,672 task--plan
The LAM model bridges planning
httpa://
2024bb
Jiang
the gap and action exccution in GUI environments. It introduers molti-phuse training pipeline combin- ing task planning, luitt- loarning, seif-hoosting exploration, and reward- optimisation bust acticn- oriented per- formancr.
gitbub.
2023
crosoftt
cation environ-
ries
UPO/trea/
mentall data
main/
dataflow
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 24198, 2024I.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Secclick: Harnessing gui grounding for advanced visual gui agents, 2024a. URL https: ://arxiv. waw.iivoorr www./org//a........ wwwg//og//////e//.... abs/24011
121

---

synthetic imagee, visual grounding datasets and GUI dataset snd Published in Transactions on Machine Learning Research (06/2025)
Table 48: An overview of GUI-optimized models on cross-platform agents (Part I).
Model
Platform
Foundation
Size
Iuput
Output
Dataset
Highlights
Link
Model
RUIG
Zhane
Mobile and desktop
Swin
Traus-
UI screen-
Bounding box
MoTIP dataset and
Innovatively uses policy gradients to improve the special decoding in phell-so-soqueenn
2l/23e
Former
Liu
coder
shots and
pre-
RicoSCA dataset
et al
2021
layers
text instrue- tioms
dictions linguistic form Task plans, action quences,
in
for mobile UI data
and
Common Crawd for desktop UI data
Lewis
2019
peradigm
CogAgent
PC, web, and Android plat- forms
Cog
183
GUI screen- shots combined with OCR- and textual derived text
CCSTOKK
text
Higl-resclution croes-
btupe:
Hong
Wang
recognition
module
Iulance
7/gthhb
2023
datasets:
80M
computational efficiency
con/ /OU////D CogViN
2024m
text
and high-msolttion inpot
descriptions
processing
Mind2Web ArTW
SeeClick 202ta
iOS, droid,
An-
Qwenn VL
9.6B
GUI screem- shota
GUT actions and element
300K webpages with text and icone, RICO, and data from LLaVA
Ability to perform GUI tasks purely from shots and its novel GUI grounding pre-training sp- proach
https://
Bai
github
macO8, Windows, and web Mobile, desls- top. and
2023b
textral in- structions
locatious for interaction
ajucckevin
SeeClick
StreenA1 Baechler 2024
PaLL-3 Chen
51
GUI screen- shota with OCR texs, image screen un-
262M mobile web screenshote and 5-M mobile app tions, notations screenshots other visual with bound- ing box and labels, navigation Instructions, summaries of screen
Tect-bused bused snswers for questions,
Text
Unified reprusentation of Uls and infographics com- bining visual and textual elements
https://
al 2023
github coo/
tablet UIs
kyegonee/ ScreanAl
elements
Ferret-UI OT
PPhone
Vicama-138 13B
2B/8B/ 130 screon- Descriptions shots, ITI- of UI notated ementts, hounding widget clas- boxes and sitication,
Chro-set, CromddU1-188. GUIDE, Spotlight
Multi-platform support with high-resolution adap-
m1
Android,
Chinng
201243
image encoding
AppleTV
G Cemmit
Teama
labells for UI wid- gots, OCR-
OCR, tape-
Lluras -B
bility, and
m1.
text wii//w
2024
detected text
interaction location,
and hound- ing
for text tions and elements, multi-round source interaction- based QA
HTML hier- arehy trees
for web
10935
Pengzhou Cheng, Zheng Wu, Zongru Wu, Aston Zhang, Zhuosheng Zhang, and Gongshen Liu. Os-kairos: Adaptive interaction for mllm- pollerd gui agents. arXiv preprint arXiv:2503.1646: 2025a.
Yuheng Chong, Coyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, et al. Exploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint vxXiv2200.03344 2024b.
Ziming Cheng, Zhiyuan Huang, Junting Pan, Zhaohui Hou, and Mingjic Zhan. Navi-plus: Managing ambiguous gui navigation tasks with follow-up. arXiv preprint ar 2rXiv:2503.224 24180, 2025b.
Antoine Chevrot, Alexandre Vernotte, Jean-Rémy Falleri, Xavier Blanc, and Bruno Legeard. Are autonomous web agents good testers? arxio preprint arXi::2504.012 2025.
122

---

Published in Transactions on Machine Learning Research (06/2025)
Table 49: An overview of GUL-optimized models on cross-platfone agents (Part II),
Model
Platform
Foundation
Size
Iuput
Output
Dataset
Highlights
Link
Model
V-Zen
Rala
Computers and Web
Vicuna-7B
7B
Text, Images
GUI
Action
Pre-
GUIDE
Chawla
Dusl-rsolution visual eu- coding for precise GUI grounding and task execu-
bttpe://
Chiong
diction, GU1
ct al.
2024
github con/
2024
et al (2023
Bounding
abdu:75648)
Box
V-Zen
et al. 2023a
EVA-2-CLL11
San
2023
Show
Lin
Wobsites, deskoops,
Phi--.5 3-
4.2B
GUI scroen-
GUT actions, muvigation, element loca- tioa
SereemSpot, RICO,
Interleawrd
Vision-
httpa://
20/24c
Abdiu
shots with
GULEav, GULAct, AITW, ATTZ, GUL- World
Lauguage- Action up- pronch, allowing soanlees navigation, grounding, and understanding of GUI envirommnnss
gitbub.com
and mobile phones
ct al.
2024
OCR for text based UI elements and visnal grounding
Showlab/
Showl]
for icons and widgets
OS-ATILS
Windows, macOS, Linnx, draid, the web
LatervV L-2
1B GUI screen- shots
GUI sctions
AndroidContzol, SreClick, and others anmotated with GPT-4, over 13 million GUI elements and 2.3 screenshots
The lirst foundistio action model resigned for general- ist GUI agents, supporting crosepplatform GUT tasks. and introduring unified action space
bttpe:
Wo 20244
hhe
al
7 /oaatlas.
An-
2024 and
github. 10/
Owen2-VV
Bei
ml
20b3b
xLAM Zhang ct aL
Diverse cnvi- touments
Mistrul-71
Range Unitied function- calling data formats
Function calls, thought
Synthetic and augmented data, over 60,000 qugliquality
Excels function-calling tasks by leveraging unilied
bttpa://
2122d
Jiang et al
from
github. com/
21/33
and
including
sualable data pipelimes
DeepSeek- Codee-7B
8x22B
processes
<LAV
samples generated using APIGen from 3,673 APIs across 21 categgrris
2024n
De Chezelles, Thibault Le Sellier, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, et al. The browsergym ecosystem for web agent research. Xixiv preprint ar Xrxiv:24 12. 05467, 2024.
Jeffrey Yang Fan Chiang, Seungjae Lee, Jia-Bin Huang, Furong Huang, and Yizheng Chen. Why are web ai agents more vulnerable than standalone 1lms? security analysis. arXiv preprint ar 2r0::2250..2 20333330200083, 2025.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, Ton Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: /log/ 2023-03-300 00-3000-viuaaa
Junhee Cho, Jihoon Kim, Daseul Bae, Jinho Choo, Youngjune Gwon, and Yeong- Kwon. Caap: Context-aware action planning prompting to solve computer tasks with front-end ui only. arXiv preprint urXiv:2406.069443 2024.
Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, and Kun Shao. Lightweight neural app control, 2024. URL
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruccionffeneeeeee language models. Journal of Machine Learning Research, 25(70):1-5) 2024.
Junyoung Chung, Caglar Gulcchre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 14 2014.
CogAgent Team. Cogagent: Cognitive ai agent platform. https://cogagentt.aam aminer en cn/home 2024.
Accessed: 2024-12-17.
Wikipedia contributors. Large language model wikipedia, the free encyclopedia, 2024. URL https
//t..//wi/ineeee wikipedia wttpa/d/aa..gg////l/aagegeee language 1aael Accessed: 2024-11-25.
123

---

Published in Transactions on Machine Learning Research (06/2025)
Table 50: An overview of GUI-optimized models on cross ccoss-platform agents (Part I1I).
Model
Platform
Foundation
Size
Iuput
Output
Dataset
Highlights
Link
Model
Falcon-
1OS, Au--
Qwen2-VLL
7B
Screenshots GUI actions coor-
Insight-UI dataset, Burther Ene-tuned on datasets such as ATTZ, droid Control, and Mind2Web
Decouples GUI context
U1 Shee et
droid,
Win-
7B
of GUI
and
comprehension from Instrnction-following tasks, levrraging instruction- free pretraining approsch.
202ta
dows, Linux,
with node dinates
Wob
information bounding and OCR boxes for
unnotations internetion for visible elements GUT scroen-
TT-TARS
Web,
Drak- (Win-
Qwmn--VV VT.
7B
GUT Actions
GUT screeshoot
Pure vision-lused porcep- with standardized
httpa://
top
and 721
shots
and collected from
metadata
gitbub.com
22025
dows, mo-
Wang
al.
GET actions arross plat-
bytedance/
cOS).
and
20/4g
websites, apps. and systems; action
forms (Web, Mobile, Desk- top).
1ARS
Mobile
operating
droid)
trace datasets
from varions GUI
agent benchunarke;
GUI cutori-
Bor reasoning
eubancement: mul- tiple open-souren datasets
Magmas
Yaos
Web, Mobile,
LLaMA-3-
8.6B
GUI screen- shots, Gual deseriptions
GUI actions,
UL, robotica data, instructional videos
Jointly trains on hesero- geneens datasets, enabling generalization across digi- tal and physical tasks
bttpe://
Robotics
Duber
robotic nipulation
microsoft.
2021
task
github. 10/
Cony Ne-
Magna/
Xxlarge
st al. 2022
SpiritSight
Web,
An-
Intern
2B,
GUT screen- shots
GUT actions
A Rawles et al.
Introduces a Universal Block Parsing (UBP) method to resolve github. 10/
bttpa:
droid,
Win-
8B.
Common-- Chomd websites, and customm annotations
2028
7bzbiyuan
20025e
dows
Deak-
20241
and
top
2613
sitional ambiguity in high-reeolution visual imputs.
SpiritSight Agont
GUL-RI Xia
Windows,
QwenVL12.5
3B
GUI screen- shots
Reasoming and actions
Mixture 3K high- quality samples
First framework to apply rule-based veiuforeemen learning (IIT)) to high- level tasks across plat- forms.
https://
Luo
Limus,
Bai
github.
MacOS, An- droid, and Web
2025h
7B
F1tzz-a1/
GUI-BE git
InfiCU-RRI et
Web, Desk: top, and An-
Qwen2.5-V 31-Instruct
3B
screen- shota, Acces- siblity Tree
GUT
Reasoning text and GU1 actions
Diverse daraset mix- ture
Two-stage training frame-
httpa://
worls RetoreR:ssue:: (1)
gitbub. coo/
20250
droid
Ressening Injection Spatial Reasoning Distil-
Bealln
InfiGUI-P R1
lation, and (2) Delibera- Eahanocnenn Re-
inforoement Learning with Sub-goul Guidunce vor Recovery Scenario Con- struction
Qwen2-VL. VL
7B
seroen- shots
Thonghts and grounded coordinate- based actions
datasnts with 56K GUI jectory sumplets
Introduors mid-rraining diverse aun-GUI reasoning tasks (partienlarly math code) to substautially enhance GUI agent plan- ning capabilities
httpa://
cralization
Android (Moblle)
7B-Instruct Wang of nl. 21/4
github.com con/
Zhang at mL.
bkust-nlp/
20125e
GUIMid
Chenhui Cui, Tao Li, Junjie Wang, Chunyang Chen, Dave Towey, and Rubing Huang. Large language models for mobile gui text input generation: An empirical study, arXiv preprint Xrx:::2404. 088448,, 2024.
Gautier Dagan, Frank Keller, and Alex Lascarides. Dynamic planning with 1mm. 0:X0022308.0639,, 2023.
arXin preprinl
Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. Advancing mobile gui agents: A verifier-driven approach to practical deployment. arXiv preprint arXiv:2509. 15937, 2025.
Prectam Prabhu Srikar Dammu. Towards ethical and personalized web navigation agents: framework for user-aligned task execution. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pp. 1074-1076 2025.
124

---

Published in Transactions on Machine Learning Research (06/2025)
Table 51: Overview of web GUI agent benchmarks (Part 1).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
MiniWoB++
2017
Yes
Evaluates agents on ba- sic web interactions like elicking, typing, and form navigation.
web
Task Success Rate
Element Match
https://
Shi et al.
2017
interac-
github. com/
Liu al.
2018
tion tasks
Farama?
2DFoundation
miniwob/
2pplusplus
RUSS
Xu of al.
2021
No Uses ThingTalk for mapping natural lan- guage to web actions, enabling precise web- based task execution in real HTML environ- ments.
741 instruc- tions
Task Success Rate
Text Match,
https:
2021
Element
tgi.
Match
com xnanyy/
russ
WebShoop
Yao
2022
Yes
Simulates e-commerrce navigation with real- world products, chal- lenging agents with in- struction comprehen- sion, multi-page page naviga- tion, and strategic ploration.
12,08 12,087 instruc- tions
Task Success Rate, Step Success Rate
Text Match
https://
al. (2023
webshop-pa pulp.
github.io/
Mind2Web Deng al. 2002
2023
No Tests adaptability real-world, dynamic websites across do- mains.
2,000 tasks
Step Success Rate, Task Success Rate
Element Match, Ac- tion Match
https://
github... com/
OSU Gropp
Xind2Web
Mind2Web-
2024
Yes Provides intermediate
542 tasks
Step Success Rate, Task Success Rate, Efficiency Score
Element Match, Text Match, trajectory length
https:/ ://
Live Pan et al.
action tracking for realistic task ment, along with an updated Mind2Web- Live dataset and tools for annotation.
huggingface
2024b
datasets
MMannII
Mind2Web- Live
Mind2Web- Live- Abstracted
2024
Yes Abstract the descrip- tions by omitting task-specific details and user input infor- mation in Mind2Web- Live, which are more streamlined and less lime-consuming to compose.
104 ples
Task Success Rate, Effi ciency Score
Text Match, Image Match,
https://
anonymous.
Shahbandch al. 2224
Element
4open.
Match, Path Length
acience/r/
navigate
WebArena Zhou
2023
Simulates realistic, multi-tab browsing Docker-hosted websites, focusing on complex, long-horizon tasks that mirror real online interactions.
812 long- horizon tasks
Step Success Rate
Text Match
https:
et al.
//vebarena
dev/
Leo de Castro, Antigoni Polychroniadou, and Daniel Escudero. Privacy-preserving large language model inference via gpu-accelerated fully homomorphic encryption. In Neurips Safe Generative AI Workshop 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jac Sung Park, Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Fhsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head,
125

---

Published in Transactions on Machine Learning Research (06/2025)
Table 52: Overview of web GUI agent benchmarks (Part 11).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
VisualWobArena
2024
Yes
Assesses multimodal agents on visually grounded tasks, quiring both visual and textual interaction capabilities in web environments..
tasks
Step Success Rate
Text Match, Image Match
https:
Koh
o8
aL
7/jykoh.
2024s
com/vwa
MT Webd
2024
No Introduces conversa- tional web navigation with multi-turn inter- actions, supported by specialized multi-turn web dataset.
720 ses-
Step Success Rate, Turn Success Rate
Element Match, Ac- tion Match
https://
Deng
al.
sions/ /3525 instruc- tions
sions/3525
github. com/
2002cc
magicgh/
self-map
MMIμ Zhang
2024
Yes
Tests multihop, multi- modal tasks on real- world websites, requir- ing agents handle page information extraction and reason- ing for complex tasks. Bilingual web brows- ing benchmark with 10,000 browsing traces, supporting evaluation language- specific environments.
1,050 tasks
Step Success Rate, Task Success Rate
Text Match,
https:
al. (2024s)
Element Match
/mmina.
cl Finnyy.
com/
Auto WebtoWh
2024
No
10,000 traces
Step Success Rate, ciency Score Time
Element Match, Ac-
https:
Lai et al. 2024
Eff-
//github.
tion Match,
com THUDM/
AutoWebGLM
WorkArena
2024
Yes
Focuses real-world enterprise software interactions, targeting tasks frequently per- stances formed by knowledge workers
19,912 unique task in-
Task Success Rate, Effi- Match, github. ciency Score, Completion under Policy,
Element https://
Drouin
2024
Text Match,
ServiceNos/
Execution- based Valida- Turn Success tion
WorkArena
Rate
2024
Focuses on long- context multimodal agents using video tutorials for task com- pletion
74 videos (approx.
Task Suc-
Element https:
Jang
et
al.
coss Rate, Match, State
7/github.
2024
4 hours),
Intermediate Information, Intent Suc- Exact and cess Rate, Fuzzy Text tasks
com/1 j
2,021
videowebarena
Elficiency
Matches
Scores
Eny Distracciti
2024
No Evaluates the "faith- fulness" multimodal GUI agents by assess ing their susceptibility to environmental dis- tractions like pop-ups, fake results, mislead- ing elements
1,198
Task Success
Text Match, Element Match, State Information
https:
Ma et al. 4c
tasks
Rate
//ithub.
com xbmxb/ EnvDistraction
Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz,, Aaron Sarnal, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schocnick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannanch Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models, 2024. URL https: wwtpesvvvorggg/eee absp//bbb/2449/// 17146
126

---

Published in Transactions on Machine Learning Research (06/2025)
Table 53: Overview of web GUI agent benchmarks (Part 111).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
WebVLN-v1
2024
No
Combines navigation and muestion-answering
8,990
Task Success
Element Match, Path Length, Trajectory
https:
Chen
et
aL
paths and
Rate, Effi-
7/github.
2024e
shopping sites, inte- grating visual and tex- tual content for unified web interaction evalua- tion.
14,825
ciency Score
com/ COm/WebVLN/
QA pairs
WebVLN
Length
WEBLINX
Lu
2024 No
Focuses on conversa- tional navigation, quiring agents to fol- low multi-tunn user instructions realis- tic, dislogue-based web tasks.
100k interac- tions
Turn Success Rate
Element Match, Text Match, Ac- tion Match
https://
al. (202cc
mogill-ni
github. 10/
weblinx/
ST- WebAgentBench Levy al.
2024
Yes
Evaluates policy-driven safety in web agents, using the Completion under Policy metric ensure compliance in enterprise-like environ- ments.
tasks
Task Success Rate, Com- pletion under
Element Match, Ac-
https:
//sites.
tion Match,
google.
2024
Policy (CuP),
Text Match
com/ /iew/
Risk Ratio
st we ee-wbbagentbench.
Comp WoB Fu-
2023
No
Tests agents quential, compositional tasks that require state management across multiple steps, simulating weal-world automation scenarios.
50 positional tasks
Task Success
Element Match
https:
ruta et 2024
Rate
github com
google reaearchh
google reaearchh
tree/
master/
compositional
rl/ 31/compwbb
TURKING BENCH
2024
Yes
Uses natural HTM. tasks from crowdsourc- ing to assess interaction skills with real-world layouts and ele- ments.
32.2K in- stances
Task Success Rate
Text Match, Element Match, Tm- age Match
https://
Xu
turkingbench.
et al. 2222ee
github.10
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium User Interface Software and Technology, UIST '17 pp. 845-854, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450349819, doi:
(0.1145/312654431222
URL httpp::/ /ttt::://ooi.or/11... 1145/3126594 3126651
Biniam Fisscha Demissic, Yan Naing Tun, Lwin Khin Shar, and Mariano Ceccato. Vlm-fuzz: Vision language model assisted recursive depth-first search exploration for effective ui testing of android apps. ar preprin! arXiv:2504,11675 2025.
Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, al. Mobile-bench: An evaluation benchmark for Ilmebased mobile agents. arXiv preprint arXiv:4477000008 2024a.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-2814 2023.
Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, Soo-Kiong Ng, and Tat- Chua. On the multi-turn instruction following for conversational web agents. arXio preprint ar 10.:2402.150555 2024b.
127

---

Published in Transactions on Machine Learning Research (06/2025)
Table 54: Overview of web GUT agent benchmarks (Part IV).
Benchmark
Year
Live
Highlight
Data Size
Metric
Measurement
Link
Beench
2024
No
Provides & fine-grained
1,534 instances from
Task Success
Text Match, Image Match,
https://
Liu et sl 20241
assossment of multimodal
Rate, Turn
visualyebbeach.
large language models
Success Rate,
Flement
github.io/
(MLLMs) on web-specific
139 real websites
Efficiency
Match, Ac-
lasles
Metries
tion Match
87 sub-
domains
WONDER-
2024
No Focuses on business pro- cess management (BPM)
2,928
Task Success
Text Match, Rate, Step Success Rate, Efficiency
httpe://
BREAD
Wornow
human demon- strations
Action Match, State Information
github com/ HazyResearch
al.
tasks like documentation, knowledge transfer, and process improvement
wonderbread
across 598
Score, Com-
distinct
pletion under
workflows
Policy
WebOlympus
2024
Yes
An open platform for web agents that simplifies run- ning demos, evaluations, and data collection for web agents on live websites Provides a unifiedl exten- sible, and open-source en- vironment for evaluating web agents with standard- ized APIs and observations
50 tasks
Task Success
Action
Zheng et al. 2024b
Rate, Step Match
Success Rate
BrowserGym
2024
Yes
Benchmarks include Mini-
Suc- Text-based Rate, matching Step Success and element
Task
https://
Chezellss et al.
cess
github. com/
2021
ServiceNoz/
WoB( I
Rate, Turn match Success Rate,
BrowserGym
with 125
tasks, We- bArena with 812 tasks, and WorkArena with to 341 tasks per level
Efficiency
Metrirs
WebWalkerQA Wu
2025
Benchmorks the capacity of L.Mss handle deep, structured, and realistic web bebebased navigation and reasoning tasks
high-
Task Success
Text Match,
https://
et al.
2025a
quality
Rate, Elti-
Action
github. com/
QA pairs
ciency Score
Match
Alibaba-ll NLP
Nebwalker
Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, and Tat-Seng Chua. On the multi-turn instruction following for conversational web agents, 2024c. URL htttss///axiu org/ aa////bb///22... 15057
Parth Deshmukh, Saroj S Date, Parikshit Mahalle, and Janki Barot. Automated gui testing for enhancing user experience (ux): survey of the state of the art. In International Conference on ICT for Sustainable Development, pp. 619 628. Springer, 2023.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprinl ar
2018.
Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Tuu Anh Tuan, and Shafiq Joty. Data augmentation using 1mss: Data perspectives, learning paradigms and challenges. In Findings of the Association for Computational Linguistics ACL 2024, pp. 1679 1705, 2024a.
Lei Ding, Jeshwanth Bheemanpally, and Yi Zhang. Improving technical" how-to' query accuracy with automated scarch results verification and reranking. arXiv preprint orXiv:2444008860 2024b.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint ar arxi:22311.0 004254,, 2023.
128

---

246 human- annotated Published in Transactions on Machine Learning Research (06/2025)
Table 55: Overview of web GUI agent benchmarks (Part V).
Benchmark
Year
Live
Highlight
Data Size
Metric
Measurement
Link
WebGames Thhoma
20/25
Yes
comprehensive bench- mark designed to evaluate the capabilities of general- purposse web-browsing Al agents through 50+ in- teractive challenges. It wiquelly provides her- metic testing environment with verifiable ground- truth solutions.
50+ chal-
Task Success
Action
https://
et 2025
lenges
Rate
Match
github. com/
convergence a1/
vebgames
AafeArena Tur el
2025
Yess
The first. benchmark specif- ically designed to evaluate the deliberate misuse of web agents by testing their ability to complete both safe and harmful tasks.
500 tasks
Task Sucoess Rate, Com- pletion under Policy, Risk
Text. Match, State Infor- mation
https://
2025
safearena
gathub.
Ratio
WABER Kara et al.
20/25
Yes
Introduces new eval- uation metrics Efficieucy and Reliabilit- go beyoud standard success rate measurements A real-world online evalu- ation benchmark designed to reflect actual inter- actions with live web inter- faves
tasks
Task Sucoess Rate; Effi- ciency Store
Action Match, State Information
https://
github com/
SumanKNath/ WABER
Online Mind2Web
2025
Yes
300 tasks from 136 websites
Task Success Image Match, Ac- tion Match, State Tn- formation,
httpe://
Xue et al. 2025
Rate, Effi- ciency Score
github, com/
0SU- 1U0-NLP-rroul
Maline-Kind2We
LLM-as- a-Judge Evaluaion
DAM
Zhar
2025
Yes
The first benchmark to evaluate privacy leakage risks in multimodal, real- istic web envirouments ing agentic models
Task Success
Action
https://
muggambetov al.
Rate, Risk Ratio
Match, Text Match
github. com/
2025
facebookrenparcc/
si-agent-o aggent-privacy
Rewar-
2025
No
The first benchmark to rigurously evaluate LLM- based judges against hu- man expert annotations across multiple web agent tasks
1,302 tra- jectories, 351 tasks
Task Success Rate, Com- plction under Policy
Image Match, Elemeut /S- tate Match
httpa://
Bench L <t all
agent-rewal www.rd-baab...eeee...e d-bncch.
2025
github.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. survey on in-context learning. anXie preprint (tt::2301.00233 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021. URL httts:///aret
org/abs/2010. 11929
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable web agents at solving common knowledge work tasks? arXiv preprint 0an24433.07116 2024.
Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls. arXiv preprint arXim:2402.04950 04255,, 2024.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXio preprint ar arXiv:2109. 10360, 2021.
129

---

Published in Transactions on Machine Learning Research (06/2025)
Table 56: Overview of web GUT agent benchmarks (Part VI).
Benchmark
Year
Live
Highlight
Data Size 1,885 instruc- tions
Metric
Measurement
Link
Ressstt
Ye
20/55
No
The first benchmark to evaluate long-horizon web assistance using real-world users' sequential instrue- tions expressed in natural and often ambiguous lan- guage
Task Success
Action
httpe:
et ml.
2025
Rate, Step
Match
//ccai cs.
Success Rate,
Juu jbu.edu/
Elficiency
projects/
Score
RealWebaasist//
REAL 2025
Garg
20/55
Yes
Fully deterministic, high-fideityy replicas of weal-world websites (e.g.. Airbub, Amazon, Gmail), enabling safe, repro- ducible, and configuralle testing for multi-turn GUI-based agents
Task Sucoess Rate
Text. Match, Action Match, State Information Match
https://
across deter- ministic websites
github com/ agi-inc/
agindk
BEARCUBS
Song 20/25 Yes
Emphasizes interaction with live weh pages and includes multimodal tasks (e-g., video, audio, 3D) that cannot be solved by text-only methods, addressing limitations of prior benchmarks relying on static simulated environments
ques- tions
Task Sucoess Rate,
Text Match, Action Match
https://
2025
Fffi- ciency Score
bear- cubr. github.1o
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprinl Xiviv:247. 21783, 2024.
Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. Agent ai: Surveying the horizons of multimodal interaction. arXiv preprint Xrxi::2401.033 0336668, 2024.
Manuel Egele, Christopher Krucgel, Engin Kirda, and Giovanni Vigna. Pios: Detecting privacy leaks in ios applications. In NDSS, volume 2011, pp. 18th, 2011.
William Enck, Damien Octeau, Patrick D McDaniel, and Swarat Chaudhuri. A study of android application security, In USENIX security symposium, volume 2, 2011.
José Gonzalez Enríquez, Andres Jiménez-Ramírez Francisco José Domínguez-Mayo, and Julián Alberto Garcia-García Robotic process automation: a scientific and industrial systematic mapping study. IEEE Access, 8:39113 0591133-9129,, 2020.
Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipall;, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks. arXiv preprint XrXi::2508.000 09577,, 2025.
Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, and Kamalika Chaudhuri. Wasp: Benchmarking web agent security against prompt injection attacks. arXiv preprint ar arXiv:2504.188 18575, 2025.
Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, and Xin Eric Wang. Read anywhere pointed: Layout-aware gui screen reading with tree-of-lens grounding, 2024. URL
bttps:://axxiv.or//bbbbb
19263
Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. Gui-boc: Align gui action grounding novel environments via autonomous exploration, 2025. URL https://erxiv.org// wttp://wwwrr/a/////////aaa 13896
130

---

Published in Transactions on Machine Learning Research (06/2025)
Table 57: Overview of mobile GUI agent benchmarks (Part 1).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
AndroidEnv
2021
Yes
Provides an open- source platform based the Android ecosys- with over 100 tasks across approximately 30 apps, focusing reinforcement learning for various Android interactions.
100+
NA
NA
https://
Toyama ot al. 2021ss
tasks
github. com/
google-deepsing
android
env
PIXELHELP Li
2020
No Includes a corpus of natural language in- structions paired with UI actions across four task categories, aiding in grounding language to UI interactions.
187 multi- step instruc- tions
Step Success Rate
Element Match, Ac- tion Match
https://
al. 2020a
github.com com/
google-researche
google google-reeearch
tree/
master/
Beq2act
MlobileEEuv
2024
Yes
Comprehensive toolkit for Android GUI bench- marks to enable con- trolled evaluations of real-world app interac- tious.
224 tasks
Task Success Rate, Step
Text Match,
httpa://
Zhang et al. 2024c
Element Match, Im- age Match, State Infor-
github. com/
Success Rate
L-LCECE
Mobile Env
mation
B-MOCA Lee 2024
Benchmarks mobile device control agents realistic tasks, incorporating layout and language randomization evaluate generalization capabilities.
tasks
Task Success Rate
Element Match, State
https:
et al. 2024b
/
//b-moca.
Information
github. io/
AudroidWorld
2024
Yes
Offers dynamic An- droid environment, al- lowing for diverse nat- language instruc- tion testing.
tasks
Task Success Rate
State Infor- mation
https://
Rawles et al.
github. com/
2024
google-research, android. world
Adrienne Porter Felt, Erika Chin, Steve Hanna, Dawn Song, and David Wagner. Android permissions demystified. In Proceedings of the 18th ACM conference on Computer and communications security, pp- 627-638, 2011.
Sidong Feng and Chunyang Chen. Prompting all you need: Automated android bug replay with large language modcls. In Proceedings of the 46th IEEE/ACA AACM International Conference Software Engineering, pp. 1-13, 2024.
Sidong Feng, Changhao Du, Huaxiao Liu, Qingnan Wang, Zhengwei Lv, Gang Huo, Xu Yang, and Chunyang Chen. Agent for user: Testing multi-user interactive features in tiktok. arXiv preprint ar arxiv:2504. 154744 2025.
Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. How far are we from agi: Are 1ims all we need? Transactions on Machine Learning Research.
Xucyang Feng, Zhi- Yuan Chon, Yujia Qin, Yankai Lin, Xu Chen, Zhiyuan Liu, and Ji-Rong Wen. Largo language model-based human-agent collaboration for complex task solving, arXio preprind ar arx:222402. 12914, 2024.
Moghis Fercidouni and A. B. Siddique. Search beyond queries: Training smaller language models for web interactions via reinforcement learning, 2024. URL https://aara wtg.ab//////..aobba...////
1088770447887777 10887
131

---

Published in Transactions on Machine Learning Research (06/2025)
Table 58: Overview of mobile GUI agent benchmarks (Part 11).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
Mobile-BB Eval
2024
Yes
Benchmark based on mainstream Android spps, and designed to common mobile interactions.
in-
Task Success
Text Match, Path Length
https://
Wang
et al.
struc-
Rate, Step Success Rate,
github.com
2024e
tions
X-PLUG/
Efficiency
MobileAgent
Score
Task Wen
2024
Yes
Android Task Automas tion benchmark sup- ports exploration and task recording in real apps with correspond- ing GUI action traces. large-scale dataset, which is partly inspired by PIXELIELL, ing diverse Android in- teractions.
158 tasks
Step Success Element Match, Ac- tion Match
httpa://
al. 2024a
Rate, Task Success Rate
github com/
MobileLLM
AutoDroid
ATTW
Rawles
2023
No
715,142 episodes
Task Success Rate, Step Success Rate
Action Match
https://
et al.
2023
github. com/
google-research
google
tree/
master/
android.
in the
wild https:// github. com/
AreroidArens
2024
Yes
Focuses on daily cross- app and constrained tasks within the An- droid ecosystem, pro- viding single-app and multi-app interaction scenarios.
221 tasks
Thak Success Rate, Step Success Rate, Efficiency
Action Match, Path Length
Xing
et al.
2004
AndroidArenaAgent
AndroidArena
Score
LearGGUl Lin
2025
Yes
The first bench- mark to systemati- cally study few-shot demonstration- based learning in mobile GUT agents, featuring both offline and online task environments
Offline: 2,252
Task Success Rate
Action Match
httpa:
et al.
2025a
//1gy0404.
tasks
github.10/
(k-shot
LearnAct
variants)
across
apps;
Online:
101 in-
teractive
tasks
across 20
apps
Nádia Fernandes, Rui Lopes, and Luis Carriço. On web accessibility evaluation environments. In Proceedings of the International Cross- Diss-Discyilinary Conference on Web Accessibility, pp. 1 10, 2011.
Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint #xXiv:2304.03788 2023.
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instructionfinetumee foundation models. arXie preprint 0rXiv:2855.18554 2023.
Hiroki Furuta, Yutaka Matsuo, Aleksandra Faust, and Izzeddin Gur. Exposing limitations of language model agents sequential-tast compositions on the web. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
Orazio Gambino, Leonardo Rundo, Vincenzo Cannella, Salvatore Vitabile, and Roberto Pirrone. framework for data-driven adaptive gui generation based dicom. Journal of biomedical informatics, 88:37-57 52, 2018.
132

---

Published in Transactions on Machine Learning Research (06/2025)
Table 59: Overview of mobile GUI agent benchmarks (Part I1I).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
ANDROIDLAB
2024
Yes
Provides a structured evaluation framework with 138 tasks across nine apps, supporting both text-only and mul- timodal agent evalua- tious on Android.
tasks
Task Success
Element Match, Im-
https:
Xu et al. 2024i
Rate, Step
7/github.
Success Rate, Efficiency Score
age Match
com THUDM/
Android-Lat
GTArena Zhao
2024 No
Introduces a Transition Tuple for GUI defects, enabling large-scale de- feet dataset crestion and reproducible, end- to-end autornaee test- ing.
10,000+ GUI display and GUI interac- tions
Task Success Rate, Step Success Rate
Text Match, Element Match, Ac- tion Match
https://
al. 2024b
github
2JU-ACES ISE/
ChatUlTest
A3 Chai et
2025
Yes Introduces a novel business-level LLM- across 21 Rate
tasks
Task Success
Element Match, Ac- tion Match
https://
2025
yuxiangchai
based evalustion process, significantly reducing human labor and coding expertise requirements.
widely
github. Atthub.i//
used apps
Android- Satroid-Agent-- Agent-Arens
Llama
2024
Yes Enables faithful and scalable evaluations mobile UI task au- tomation by matching task execution traces against annotated es- sential states
496 tasks covering 57 unique Android applica- tions
Task Success Rate, Step Success Rate, Efficiency Score
Text Match, Action Match, State Information
https://
Zhang ot al.
github.. . com/
2024h
Ll llamaTouch
Ll lamaTouch
Match
Mobile-
2024
Yes Provides a fully au- tonomous evaluation process on real An- droid devices and flexibility in judging success conditions across multiple paths completion
100 tasks across 10 open- source Android applica- tions
Task Success Rate, Eff ciency Score, Latency, Token Cost
State Infor- mation (UI State Match- ing)
https:/ ://
Agent Bench
mobil beblleggenteech..
Wang
github. 1o/
20244
Erich Gamma. Design patterns: elements of reusable object-oriented software. Person Education Inc, 1995.
Yuyou Gan, Yong Yang, Zhe Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, Yunjun Gao, Yingcai Wu, and Shouling Ji. Navigating the risks: A survey of security, privacy, and ethics threats in llm-based agents, 2024. URL https: // //tpx//axvvvvvvv org/ org/abs/ waws///1l.... 2et.//111..92
Difei Gao, Siyuan Hu, Zechen Bai, Qinghong Lin, and Mike Zheng Shou. Assisteditor: Multi-agent collaboration for gui workflow automation in video creation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11255-11257, 2024a.
Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, ct al. Assistgui: Task-oriented pc graphical user interface automation. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recoguition, pp. 13289 13298, 2024b.
Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, and Dipendra Misra. Aligning Ilm agents by learning latent preference from user edits. arXiv preprint arXiv:2404. 15269, 2024c.
Jie Gao, Simret Araya Gebroegziabher, Kenny Tsu Wei Choo, Toby Jia-Jun Li, Simon Tangi Perrault, and Thomas W Malone. A taxonomy for human-llm interaction modes: An initial exploration. In Extended Abstracts of the CHT Conference Human Factors in Computing Systems, pp. 1-11, 2024d.
133

---

Published in Transactions on Machine Learning Research (06/2025)
Table 60: Overview of mobile GUI agent benchmarks (Part IV).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
Mobile- Bencch
2024
Yes
Supports both UI API-based actions in multi-app scenarios, testing agents on single and multi-task structures with checkpoint- evalu- ation approach.
en-
Task Success
Action Match, Path Length
https:
Dong 2024s
et
al.
tries
Rate, Step
7/github.
(200+
Success Rate,
com XiaoMi MobileBench
tasks)
Efficiency
Score
Mobile Safety Bench Lee et al.
2024
Yes
Prioritizes safety eval- uation mobile con- trol tasks, with distinct tasks focused help- fulness, privacy, and le- gal compliance.
100 tasks
Task Success Rate, Risk
Action Match with
httpa://
mobilesad et beach.
2024s
Mitigation Success
Safety Con- sidered,
github. 10/
Element
Match, State Information
SPA-BENCH
2024
Yes
Extensive evaluation framework supporting single-app and cross- upp tasks in English and Chinese, providing a plug-and-play struc- ture for diverse task scenarios.
340 tasks
Task Success
Action Match, State
https://
Chen
al.
Rate, Step Success Rate,
spa-bench.
2024d
Information,
g1 th
Efficiency
Time Spent,
Score
API Cost.
SPHINX
Ran 2025
Yes
Provides fully au- tomated benchmarking suite and introduces a multi-dimensional eval- uation framework.
284 com- tasks
Task Success Rate, Effi- ciency Score, Completion
Text Match,
al. 2025
Image Match, Element Match, Ac- tion Match
under Policy, Turn Success
Rate
AEIA-MA
MNN Chen
Yes Introduces the Active Environment Injec- tion Attack (AEIA) framework that manip- ulates environmental elements (e.g., noti- fications) to mislead LLM-powered agents.
Text Match, State Info, World) Ratio, Elfi- Action
tasks Task Success
et al. 2025c
(Android-
Rate, Risk
tasks
ciency Score
Match
(AppA- gent)
AvtoEval
Sun
Yes Fully autonomous evaluation framework for mobile agents, removing manual reward-signal defini- tion and evaluation coding.
tasks
Task Success Rate
Action Match, State Info
et al. 2025a
Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. Mobileviews: A largo-scale mobile gui dataset. arXio preprint (t.2449.114997 2024c.
Minghe Gao, Wendong Bu, Bingchen Miao, Yang Wu, Yunfei Li, Juncheng Li, Siliang Tang, Qi Wu, Yueting Zhuang, and Meng Wang- Generalist virtual agents: survey on autonomous agents across digital platforms. arXiv preprint 1r11922411.10943, 2024f.
Weiwei Gao, Kexin Du, Yujia Luo, Weinan Shi, Chun Yu, and Yuanchun Shi. Easyask: An in-app contextual tutorial search assistant for older adults with voice and touch inputs. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(3):1 27, 2024g.
134

---

Published in Transactions on Machine Learning Research (06/2025)
Table 61: Overview of computer GUT agent benchmarks (Part 1).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
OSWorld
Xic
2024
Yes
Scalable, real com- puter environment for multimodal agents, supporting task setup, execution-basee evalu- ation, and interactive learning across Ubuntu, Windows, and macOS.
Task Success
Execution- based State
https:
et al.
2024b
Ubuntu
Rate
//oswwolld
tasks, 43
Informa-
github.io/
Windows
tion (such
tasks
as internal file inter-
pretation,
permission
manage-
ment)
Windows Agent
2024
Yes
Adaptation of OS- World focusing exclu- sively on the Windows OS with diverse multi- step tasks, enabling agents to use a wide range of applications and tools.
154 tasks
Task Success
Same
https://
Arena et al. 2024
Bonarti
Rate
OSWorld,
microsoft.
scalable with cloud paral- lelization
github. i0/
WindowsAgentArena
OmniACT
2024
No Assesses agents" capa- bility generate ecutable programs for computer tasks across desktop and web appli- cations in various OS environments, prioritiz- ing multimodal chal- lenges.
9,802 data points
Task Success Rate, Step Success Rate
Action Match
https://
Kapoor
al.
huggingface
2002
co/
datasets/
Writer/
omniact
VideoGUI
Lin 2024 No
Focuses visual- centric tasks from instructional videos, emphasizing planning and action precision in applications like Adobe Photoshop and Premiere Pro.
178 tasks,
Task Success
State Infor- mation, Ac- tion Match
https:
al. 2224b
463 sub-
Rate
//showlab
tasks
github.io
videogui
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrievallangmented generation for large language models: A survey. arXie preprint arXiv:2312.1090 2023.
Divyansh Garg, Shaun VanWeelden, Diego Caples, Andis Draguns, Nikil Ravi, Pranav Putta, Naman Garg, Tomas Abraham, Michael Lara, Federico Lopez, et al. Real: Benchmarking autonomous agents deterministic sinulations of real websites. arXiv preprind orx:222504. 115433, 2025.
Jesse James Garrett et al. Ajax: A new approach to web applications. 2005.
Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wengiao Zhang, Siliang Tang, and Yueting Zhuang. Iris: Breaking gui complexity with adaptive focus and self-refining, 2024. URL
https://arxias wwtreivv.oog/aee ht//abbb/2422
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: family large language modols from glm-130b to glm-4 all tools.
Xixiv preprinl 2rXi::2406.122 12793, 2024.
Joshua Gorniak, Yoon Kim, Donglai Wei, and Nam Wook Kim. Vizability: Enhancing chart accossibility with Ilm-based conversational interaction. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pp. 1-19, 2024.
135

---

Published in Transactions on Machine Learning Research (06/2025)
Table 62: Overview of computer GUI agent benchmarks (Part 11).
Benchmark
Year
Live
Highlight
Data
Metric
Measurement Link
Size
Spider2-V V al. 2024
Cao
2024
Yes
Benchmarks agents
tasks
Task Success Rate
Action Match, State Information
https://
across data science and engineering workflows in authentic enterprise software environments, covering tasks from data ingestion visualization.
spider2-v.
github.io
Act2Cap
Wu
2024
Yes
Emphasizes GUI action narration using cursor- based prompts in video format, covering vari- ety of GUI interactions like clicks, typing, and dragging.
4,189 samples
Step Success Rate
Element Match
httpa:
ct al. (2024b
//showlab.
github.io
GUI Narraarrtoor
OFFICEBENCH
2024
Yes
Tests cross-application automatioo in office workflows with com- plex multi-step tasks across applications like Word and Excel, assess- ing operational integra- tion in realistic scenar- ios.
300 tasks
Task Success
Action Match, Text Match, State Information
https://
Wang 2202pp
al.
Rate
github. zlwang-cn ca/ OfficeBench
AssistGUI Gao
2024
Yes
The first beuchmark fo- cused on task-oriented
100 tasks from popular applica- tions
Task Success Rate, ciency Score
Element Match, tion Match
https:
al. 2024b
Eff-
Ac- //showlab.
desktop GUI automa- tion
github.1o. 10/
assistgui/
GUT Zhao
2025
Yes
First GUT benchmark designed to evaluate dy- namic GUT interactions incorporating vari- ous initial states.
315 total tasks
Task Success
Image Match, Element Match, Ac- tion Match
al.
2025c
Rate, Effi- ciency Score
from 10
Windows applica-
tions
UI-Vision Nayak et al.
2025
No
The first large-scale benchmark specifically designed for desktop GUI agents
8,227 Task Success
Action Match, Text Match
httpa:
2025
query-labo Rate pairs in total
//uivision.
githbb.10
Computer Agent
2025
Yes
The first large-scale, open-ended evaluation platform for multi- modal LLM-basod agents in real desktop computing environ- ments
User- proposed
Task Success tasks Rate
Human eval- uators
https:
Arena
Wang
7/ /arena.
al. 2025a
xlang.ai/
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su.
Navigating the digital world as humans do: Universal visual grounding for gui agents, 2024. URL https://arxiv.orr 2atpaa//41000550..
Robert Gove and Jorge Faytong. Machine learning and event-based software testing: classifiers for identifying infeasible gui event sequences. In Advances in computers, volume 86, pp. 109 135. Elsevier, 2012.
Maria Fernanda Granda, Otto Parra, and Bryan Alba-Sarango. Towards model-driven testing framework for gui test cases generation from user stories. In ENASF, pp. 453 460, 2021.
136

---

Published in Transactions on Machine Learning Research (06/2025)
Table 63: Overview of cross-platform GUI agent benchmarks.
Benchmark
Platform
Year 2024
Live Yes
Highlight
Data Size
Metric
Measurement
Link
Visual Agent
Web,
First benchmark designed for visual foundation agents CUI and multi- modal tasks, focusing vision-centrie in- teractions in Android, web, and game environ- ments.
4,482 jectories
Task Sucress Rate
Text. Match
https:
Bench
Liu
droid, Game,
//github
al.
2024i
Virtual Em-
com/THUDM/
bodied.
ViaualAgentBench/
SPR mark
Bench-
Mobile, Web, and Operat-
2024
Yes
Evaluates GUT screen readers' ability to de- scribe both content and layout informstion
Includes 650
Task Sucress Rate, Elfi- ciency Score
Text Match, Element
Fan
et al.
2024
ing Systems
screen- shots
Match
amno-
tated
with
1,500 tar-
get points
and
re-
gions
StentStudlo
Windows, Linux, ma- cOS
2024
Yes
Open toolkit for creat- ing and beuchmarking general purpose virtual agents, supporting com- plex interactions across diverse software appli- cations.
NA
Step Success
Action Match, State Information and Image Match
attps://
Zheng ct al. 2024c
Rate
computer-agenta
github.io/
agent-studin
CRAB Xn
Linux, An- 2024 droid
Yes Cross environment benehmark evaluating agents across mobile and desktop devices, using graph- based evaluation method handle multiple correct paths and task flexibility.
120 tasks
Step cess Rate, Elfiency Score
Suc-
Action Match
https://
et al. 2024g
gi github.commm com/
crab beachmark
SereenSpot Cheng al. droid, 2024a
2024
No
Vision based benchmark with pre- trained GUI grounding, ussossigg agents' ability interact with GUI elements across mobile, desktop, and web platforms using only screeushots.
GUI
1,200 instruc- tions
Step Suocess Rate
httpa://
Match
github. com/ njucckevin/ See0lick
macOS,
Windows, Web.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. Xiv preprint ar "xXx:2308.00920 03296, 2023.
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. Deep api learning, In Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pp. 631-642, 2016.
Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, and Yu Su. Is your Ilm secretly a world model the internet? model-basen planning for web agents. arxiv preprint 2rXie:241100655 2024.
Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song, Longfei Li, Jinjic Gu, and Chenyi Zhuang. Intelligent virtual assistants with Ilm-based process automation, ArXiv, abs/2312.06677 2023. URL. stttp::/apii.smmatticccc. htt.://ccoeoaaaroooooooraoaaaaaa
137

---

Published in Transactions on Machine Learning Research (06/2025)
Table 64: Overview of GUL-testing with LLM-powered GUT agents (Part 1).
Project
Category
Platform
Model
Perception
Action Standard
Scenario
Highlight Applics GPT-3's guage understanding ca-
Link
Daniel
and
General
General-
GPT-
GUI
Automates the soltware testing process
https:
Anne
Zine
testing
purpose
and
UI opera- tions
//github.
plat-
state
pabilities to GUl-based software testing,
com/
&c
Koziolek
forms
using natural
neuroevolution!
2023s
language test cases
abling natural interac-
2Da1/
tion through text-based
Sof tuat ceftwareTennn BestsorreTsstttnggggggaallliii
test case descriptions.
Daniel
and
General testing
Web plat- lorms
GPT-
ITTL DOM structure
Standard UI opera- tions
Automated GUT testing
Performs end-to-end
https: 7/github.
Anne
Zim
4
GUI testing using GPT- 4's natural language understanding and
to enhance branch cov- erage and
com/
& Koziolek
Sof BottwareTst
20233
reasoning capabilities.
kebtestingWich..N
efficiency
GP'drroid
General testing
Mobile An- droid
GPT- 3.5
view erarchy liles
Standard Automates GUI testing
Formulates GUT testing as a Q& A task, utiliz- ing LLM capabilities to provide human like in- teraction.
https:
Liu
et
aL.
opera-
//github.
2024k
tions and
to improve testing Dow- erage and detect bugs efficiently Standard Semantic, intent-driven
com/
compound actions
frankl inbill/
CFTDroid
DROID- AGENT
General testing
Mobile An- droid
GPT-
JSON rep-
Autonomously gen-
https:
3.5,
resentation of the GUI
UT op- erations, leveer-level APIs, and custom actions
erates and executes high-level, realistic automation of GUI testing
/github.
Yoon al.
GPT- 4
com/
2024
state
tasks for Android GUI testing based on app- specific functionalities.
coinse/
droidagent
Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song, and Chenyi Zhuang. Intelligent agents with llm-based process automation. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Dala Mining, pp.
2024a.
Yanchu Guan, Dong Wang, Yan Wang, Haiqing Wang, Renen Sun, Chenyi Zhuang, Jinjie Gu, and Zhixuan Chu. Explainable behavior cloning: Teaching large language model agents through learning by demonstration. 24::2410: 10. 22916, 2024b.
Xiviv preprinl ar 24
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming the rise of code intelligence, arXiv preprint ar 2f11:2401.14116, 2024a.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, el al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint Xix::2400.00 01680, 2024b.
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking tool learning of large language models, 2024c.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. real-world webagent with planning, long context understanding, and program synt hesis. arXiv preprint ar xrXii:2307.122 12856, 2023.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Alcksandra Faust. real-world webagent with planning, long context understanding, and program synthesis, 2024. URL http:::/aax...oo///a////////////
138

---

Published in Transactions on Machine Learning Research (06/2025)
Table 65: Overview of GUI-testing with LLM-pwwered GUI agents (Part 11).
Project
Category
Platform
Model
Perception
Action
Scenario
Highlight
Link
AUTTest-
General
Mobile
GPT-
GUI screen-
Standard
Automated functional testing
Features dynamic agent organization for step- oriented testing and multi source data traction strategy for cise fumetion verifica- tion.
https:
Agent
Hu
testing
An-
shots, UI hierar- tions chy files, and CV- enbanced techniques
UI
opera-
//github.
2024c
droid
of
com/
GUIs
bz lab/
AUlTestAgent
like Vision GPT-
UI
Droid
General testing
Mobile
GUI screen-
Standard UI opera- tions
Identifies non- crash bugs
Integrates vision-driven prompts and GUI text alignment with vision- language models to en- hance understanding and app
https:
Li
er
An-
4
shots with annolated
//github
20241
droid
com/
bounding
testtestA6
boxes, View hierarchy
VieionDroid
GUI logic.
files GUI Gestures, Automates capturing accessibil-
AXNav Taeb 2024
Accrssibility iOS mo- testing
GPT-
Adapts natural
bile de- vices
shots, UI element screeushots, ity testing and high- lighting
guage instructious and generates tated videos visually and interactively review accessibility test results.
detection model, and OCR
workflows, including test- potential accessibil- ity issues
ing features like VoiceOver,
Dynamic
Type, Bold Text, and But- ton Shapes
LLMigrate Beyzaci et sl.
General testing
Mobile An- droid
GPT- 4o
DOM and screenshots
Standard UI opera- tions
Automates the transfer of usage based tests be- tween Android apps Generating and validating inputs for Android applications
Leverages multimodal
LLMs to perform UI
2024
test transfers without re-
quiring source code ac- cess
Cui
Test im-
Mobile An-
GPT-
GUI turrs
Entering inputs
Demonstrates
Cui
put gem-
3.5,
effertiveness various LLMs in generating context-aware text inputs, improwing UI test coverage, and identifying previously unreported bugs
2024
eration
drold
GPT-
contextual
information
Tyysit Liu ef al.
Test in- put gen-
Mobile An- droid
GPT-
hierar- chy files
UI
Generates semantic text inputs
Automates mobile GUI testing by generating appropriate text inputs
Formulates text imput generation as cloze- style All-in-the-blank language task.
2023c
3
ecation
Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. Lim multi-agent systems: Challenges and open problems. arXiv preprint or 035:5402.03578. 2024.
Hao Hao, Vicky Singh, and Wenliang Du. On the effectiveness of api-level access control using bytecode rewriting in android. In Proceedings of the 8th ACM SIGSAC symposium on Information, computer and communications security, pp. 25 36, 2013.
Robert Hardy and Enrico Rukzio. Touch & interact: touch-based interaction of mobile phones with displays. In Proceedings of the 10th international conference on Human computer interaction with mobile devices and services, pp. 245-254 2008.
Feng He, Tianqing Zhn, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S Yu. The emerged security and privacy of Imm agent: A survey with case studies. Xiv preprint Xrv::2247. 19354, 2024a.
139

---

Published in Transactions on Machine Learning Research (06/2025)
Table 66: Overview of GUL-testing with LLM-powered GUI agents (Part III).
Project
Category
Platform
Model
Perception
Action
Scenario
Highlight
Link
Crash-
Bug
Mobile
GPT-
Crash- related
Standard
Automates reproduction of
Leverages LLMs for it- erative GUI navigation and crash reproduction from stack traces, inte grating reinforcement learning- hegg---aedd scoring system to optimize ex- ploration steps.
https:
Translator
play
Au- droid
UI opera-
//github.
Uuang al.
stack trace
tions
mobile
com/
2024c
information and GUI structure
application
kuchiuvong
creshes
CrashTrans
AdlGPP
Bug re-
Mobile An- droid
GPT- 3.5
GUI struc- ture and hi- erurchy
Standard UI opera- tions
Automates bug repro- duction by extracting $2R (Steps to Reproduce) entities Standard opera-
Combines prompt engi- necring with few-shot learning and chain-of- thought reasoning to sidongfeng leverage LLMs for GUI- based tasks.
htsps:
Feng & Chen
play
7/ //github
2024
com/
AdbaPT
MagicWand
Verrificution Mobile An- droid
GPT- screen-
Features threo-stage verification proress: extracting instructions, executing them simulated cnvironment,
Automates
Ding
shots hierarchical control tree
2024b
tions of "How-to" instructions
from search engine
and reranking search results based execution outcomes.
UXAeent La et sl.
Usability testing for web design
Web
Self- designed
Simplilied ITML representa- tions
Standard UI opera- tions
Automated us- ability testing of web applica- tions
Enables LLM-ppowred automsted usability trsting by simulating thousands of user interactions, collecting both qualitative and quantitative data, and providing researchers early foedback before real studies. Improves L.LM-driven testing by offloading planning tasks ternal runtime system.
https: //xageent hailab. 10
2025a
Guardian 2024
Mobile An- droid
GPT-
GUI turn, Prop- erties
Standard opera- tions
Autonomously explores mobile ap- plications, interacting with the UI to validate core functionali-
Testing
ties.
Agent
GUT
Andmidd, jOS, Har- mony
Not Men- tioned
GUT screen- shots, UI structure information
Standard UI opera- tions
Cross- platform mobile testing
Fliminates the need for pre- written test scripts by leveraging LLMs and multimodal perception to generate exenute test cases automatically
Li
et
al.
Testing
2024h
OS
VLM-PE LLM-Fuzz
GUI
Android (Mo- bile)
GPT-
GUI screen- shots and information
Standard erations, system- detection of level crashes and bugs
Automated Audroid app testing
vision- Integrates langnage reasoning
Demossie
Testing
op-
et al.
2025
with heuristic-based
depth-first search (DFS) systematically ex- plore complex Audroid Uls, achieving signif- irantly higher code coverage
tions
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building end-to-end web agent with large multimodal models, 2024b. URL https://arxiv/ http://aaxivv.orga/////10/000 13919
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, and Dong Yu. Openwebvoyager: Building multimodal web agents via iterative real-world exploration, feedback and optimization, 2024c. URI. htttss://aaaat wwtp:/iwvwrraa///a/aeaaaaaaee/ http//abb2211..1110
140

---

opera- Published in Transactions on Machine Learning Research (06/2025)
Table 67: Overview of GUL-testing with LLM-powered GUI agents (Part IV).
Project BugCraft
Category
Platform
Model
Perception
Action
Scenario
Highlight
Link
Bug
Windows
BugCraft GUI screen-
Standard
Automatically
First eud-to-end frame-
https://
pager
al.
Repro-
Com-
based
shots
UI opera-
reproduces crash bugs in Minerraft by reading user- submitted bug reports, generating structured steps, and executing them cause crash Migrates GUI
work that automates
bugcreft2025.
2025
duction
puter
tions
crash bug reproduction
g1 g1thub.
GPT-
in complex open world game (Minecraft) us-
i0/
4o
ing LLM-driven agents,
vision- busod UI parsing, and structured action execution
ReuseDroid
Mobile drold
ReuseDrolCUI screen-
Standard
Leverages visual con- texts and dynamic feod- back merhanisms sig- nificantly boost migra- success com- pared to prior mapping- and LLM- based meth- ods
Testing
based
shots widget
opera-
test cases
2025c
tions
Android
GPT-
properties
spps that
share similar functionality
but differ in operational logic
SenAct-ATA
GUT Testing
Web
SeeAct Zheng
GUT
strue- and
Standard UI opera-
Automatess mannal end-to- end (E2E) web application testing
First open-sonrce at- tempt to adapt LLM- powered Autonomous Web Agents into Au- tonomons Test Agents (ATA) for web testing
and PinATA Chevrot et al.
ture
et ml
DOM
tions
2025
402
GERALLT Resenbach
GUI Testing
Desktop
GPT-
GUI screen-
Standard UT opera- tions
Finds unint-- itive behavior; inconsisten- cies, and functional errors GUIs without pre- defined tost scripts
P Pioneers LLM-driven treting on real- desktop GUI appli- or DLR-SC/
https:
(Win- (ows/Linux)
shots and structure information
7/github.
et al. 2025
com/
cations (not mobile), combining
structured GUT parsing
with LLM-based control and evaluation
AgenhetAgent Kong al,
GUT Testing
Android Mobile
GPT- 4o
XML trees
UT
Executable
Automates
Innovatively combines https:
UI test GUI test case scripts
LLM reasoning with semantically enriched GUI graph (CUTG),
7/github
generation from natural language for regression and compatibility testing in
con/
prophetagent/
siguificantly improving
Home
GUI test synthesis per- formance and efficiency over state-of-tho-urt
tools mobile apps Automated testing multi-user where each agent
Agent:
GUI Testing
Android Mobile
GPT-
XML view hierarchy
Standard
Introduces a multi-
User
Feng
agent LLM framework
al.
2025
tions
interactive
user on a device
features
Jiang He, I-Ling Yen, Tu Peng, Jing Dong, and Farokh Bastani. adaptive user interface generation framework for web services. In 2008 IEEE Congress on Services Part 1l (services-2 2008), pp. 175 182. IEEE, 2008.
Yanheng He, Jiahe Jin, Shijie Xia, Jiadi Su, Runze Fan, Haoyang Zou, Xiangkun Hu, and Pengfei Liu. Pe agent: While you sleep, ai works cognitive journey into digital world, 2024d. URL https:
//arxiv. Sttprvvv.org/bb///22222 17589
Zhitao Hc, Zijun Liu, Peng Li, May Fung, Ming Yan, Zhang, Fei Huang, and Yang Liu. Enhancing language multi-agent learning with multi-agent credit re-assignment for interactive environment generalization. arXiv preprinl ar Xiv:2502. 14496, 2025.
141

---

Published in Transactions on Machine Learning Research (06/2025)
Table 68: Overview of virtual assistants with LLM-powered GUI agents (Part 1).
Project ProAgent Ye
Type Research
Platform Web and Desk- top
Model GPT4
Scenario Automates business pro- such data analysis, report gen- eration, and notifications via GUI-based tools AntLLM UT tree Standard Automates
Perception
Action Standard
Highlight Introduces dynamic workflows where agents interpret and execute tasks flexibly, surpuss- ing traditional RPA systems
Link
Task
https:
2023
scriptions UI opera- and struc- tions and
/github.
tured dynamic application branching data
OpenBMB/
Prokgent
LLMPA
Research
Mohile (An- droid)
Integrates LLM reason- ing capabilities with modular design that supports task decompo- sition, object detection, and robust action pre- diction in GUT environ- ments
Guan ec al. 2024a
10b structures, UI opera- interac- visual tions modeling, and text extraction modules
tions within mobile apps,
such ticket
booking
VizAbblitt Gormiak et al.
Research
Desktop
GPT- 4V
Keyboard- navigable tree views
Assists bliud and liw-vnsion chart strue- turrs and generates answers
Navigates
Integrates structured chart navigation with
https:
7/dur
2024
users in explor- ing and under- standing data
L.M-powered conver- sational capabilities, enabling visually im-
bc. edu/
vizability/
visualizations
paired users to query in natural language
GPiVe Tasker Vn
Research
Mobile (An- droid)
GPT-4 Android Ac-
Standard UI opera- tions
Automates uset inter- actions on mobile devices through voice commands
Integrates LLMs for nat- urul command inter- pretation and real-time GUT interactions, us- ing graph-hased local database to record and replicate interactions
httpe:
cossibility Tree
//github
ct al
2024
con/
vuninhduc796/
@PTVoiceTanker
Theodore Hellmann and Frank Maurer. Rule-based exploratory testing of graphical user interfaces. Tn 2011 Agile Conference, pp. 107 116. IEEE, 2011.
Hochreiter. Long short-term memory. Neural Compulation MIT-Press, 1997.
Nobukatsu Hojo, Kazutoshi Shinoda, Yoshihiro Yamazaki, Keita Suzuki, Hiroaki Sugiyama, Kyosuke Nishida, and Kuniko Saito. Generativegui: Dynamic gui generation leveraging Ilms for enhanced user interaction on chat interfaces. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Compuling Systems, pp. 1-9, 2025.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023. URL /ttth:tt///aaaa
org/
HONOR. Honor introduces magicos 9.0, 2024. URL http::/ww..oomareraaco///// htt.://aaarraaa.cooooooo////////
Accessed: 2024-11-16.
Jakub Hoscilowicz, Bartosz Maj, Bartosz Kozakiewicz, Oleksii Tymoshchuk, and Artur Janicki. Clickagent: Enhancing ui location capabilities of autonomous agents. arXiv preprint arXiv:2410.11872, 2024.
Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Victor Carbune, Jason Lin, Maria Wang, Srinivas Sunkara, Yun Zhu, and Jindong Chen. Screenqa: Large-scale question-answer pairs over mobile app screenshots, 2024. URL
wtwp:/iwww..r//b/////000000 08199
Cheng- Yu Hsich, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yi Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arxiv preprint ar Xxx:2230080000 2023.
142

---

Published in Transactions on Machine Learning Research (06/2025)
Table Overview of virtual assistants with LLM- MaMM--eoweedd GUT agents (Part IT).
Project
Type
Platform
Model
Perception
Action Standard
Scenario
Highlight
Link
AutoTTak
Research
Mobile
GPT-4
Android Ac-
Automates
Operates without prede- fined scripts config- urations, autonomously
httpe:
Pan
(An-
cossibility
UI opera-
multi-step
/github.
2023b
droid)
Tree
tious
tasks
mobile devices
exploring GUT environ- ments
BezenBryanWang/
AutoTask
Editor
Assist Editor
Reseurch
Windows
UTVTG GUI ele- Standard
Automates video editing workflows
Employs multi-agent collaboration frame- work where agents specialize in roles to integrate user require- ments into video editing worktlows
Geo
Lin
ments, UI opera- require-
2024a
al.
tions
2023
and
video data
PromptRPA Huang al.
Research
Mobile (An- droid)
GPT-4 and GPT- 3.5 Turbo
Layout hier- archy and screenshots with OCR
Automates smartphone tions and tasks and creates in- level fune- teractive tinnlliies tutorials
Standard UI opera-
Integrates user feedback loops for continuous im- provement, addressing interface evolution and task variability
2024a
application-
Easy Ask Gao
Research
Mobile (An- droid)
GPT-4 Android Ac-
Highlights Assists older specific UI adults in elements learning and navigating smartphone
Combines voice and touch inputs, supple- menting incomplete or ambiguons queries with in-app contextual infor- mation
ct al.
2024g
cessibility Tree
for user interaction
functions
through
in-app interac- tive tutorials
WebNav Srinivasan
Reseurch
Web
Gemini 2.0
Standard
GUI screct-
Assistive tech- nology for visually im time DOM labeling, and voien-driven interaction to support intelligent web navigation for visu- ally impaired users
Combines ReAct-style reasoning loop, real-
UI opera-
shots
and
&c Patapali
Flash
tions
DOM
2025
Think-
paired users, enabling voice- based navigation
ing
complex websites
143

---

Published in Transactions on Machine Learning Research (06/2025)
Table 70: Overview of virtual assistants with LLM-powent GUI agents (Part III).
Project
Type
Platform Desktop
Model
Perception Screenshots with
Action
Scenario
Highlight
Link
OpenAdaptOpert
LLM,
Standard
Automates repetitive tasks across Industries
Learns task automation by observing user in- teractions, climinsting mannal scripting
https:
Ope-
source
VLM
CV UI opera-
//iihuub
nAdapt
[e.g.
tools for tions
com/
AT
GPT-4,
GUT pars
OpenAdapeA
(0024
ACT-1)
ing
OpenAdap:
AgentSea
Open- source
Desktop and
LLM,
Screenshots with CV UI opera- tions
Standard
Automates tasks within
Offers modular toolkit adhering the UNIX philosophy, allowing de- velopers to tom Al ageuts for di- verse GUT environments Executes code locally, providing access to //github
https:
AgentSed
VLM
//vwv.
A0
Web
tools
for pars-
GUI environ- ments
agentsea.
2024
GUI
ing
Open
Open- source
Desktop, Web, Mobile (Au-
LLM System per- ception via commaand-
Shell com- mands, code, and line native APIs
Automates tasks, con- ducts data analysis, man-- ages files, and controls web browsers for research Automates web-based tasks
httpe:
Inter-
preter
system resourcts and li- braries, overeoming liun- itatious of cloud-basod services
com/
Lnter-
OpenInterpreteer/
preter
droid)
open- ween-interpreter
2024
MultiOn
Production Web
LLM
Standard UI opera- tions
Performs web actions natural language commands
https:
Mul-
//wxv.
fiOn
multion.
2024
YOYO
Production Mobile
MagicLM text
cou-
Executes in-app and cross-app uperations
Automates daily tasks, enhancing productivity
Leverages MagicLM to understand and execute complex tasks across ap- plications, leuruing user habits to provide per- sonalized assistance
Agent
(Magi- cOS
Magi-
cOS
9.0)
HONOR
2024
P ower
Production Windows
LLM, VLM
Records user interac- tions with the GUI
Standard U1 opera- tions
Automates repetitive tasks and streamlines workflows
Translates natural lan- guage deseriptions of de- sired automations into executable workflows
https:
Auto-
//learn.
mate
microsoft.
M
com/
crusott
2024
power nate/
desktop
2Dflow-use ing
2Da1 rei--recoderr
Eko
A1
Production Web ChatCPT
Visual- Standard Internctive Element Perception (VIEP)
Automates tasks by han- dling diverse workflows.
Decomposes uatural lan-
https:
2025
browsers and com- 3.5 puter
and Claude
UI opera- tlons.
guage task descriptions
/ //eko.
into executable work-
fellou.
flows, enabling seamless integration of natural
ai/
environ-
technol- ogy for Interacting with GUL elements.
language and program- ming logic in agent de- sign.
ments
144

---

Published in Transactions on Machine Learning Research (06/2025)
Gang Hu, Linjie Zhu, and Junfeng Yang. Appflow: using machine learning to synthesize robust, reusable ui tests. In Proceedings of the 2018 26th ACM Joint Meeling on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2018, pp. 269 282, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355735. doi: 10.1145 0211445//226002433330555 URL https:///diiii org/
10. 1145/3236024 3236055
Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A preliminary case study with claude 3.5 computer use, 2024a. URL httts://arxiv.co httg/bbs/4415 10323
Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for general computing devices use. 2024b.
Yongxiang Hu, Xuan Wang, Yingchuan Wang, Yu Zhang, Shiyu Guo, Chaoyi Chen, Xin Wang, and Yangfan Zhou. Auitestagent: Automatic requirements oriented gui function testing, 2024c. URL https:
09018
Zhiyuan Hu, Shiyun Xiong, Yifan Zhang, See-Kiong Ng. Anh Tuan Luu, Bo An, Shnicheng Yan, and Bryan Hooi. Guiding vlm agents with process rewards at inference time for gui navigation, 2025. URL
https://arxiv http:/arvvv....raai// htggaab//5504.111 16073
Jiaxing Huang and Jingyi Zhang. A survey on evaluation of multimodal large language models. arXic preprint 200::2408.522 2024.
Jie Huang and Kevin Then-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprinl Xiv:2221 10403, 2022.
Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, and Lin Ma. Scaletrack: Scaling and back- trccktracking automated gui agents, 2025a. URL pectps://arxii org/abs/
2505.00416
Tenghao Huang, Kinjal Basu, Ibrahim Abdelaziz, Pavan Kapanipathi, Jonathan May, and Muhao Chen. R2d2: Remembering, reflecting and dynamic decision making for web agents. arXie preprint arXiv:2501.12485 2025b.
Tian Huang, Chun Yu, Weinan Shi, Zijian Peng, David Yang, Weiqi Sun, and Yuanchun Shi. Promp12task: Automating ui tasks on smartphones from textual prompts. ACM Transactions on Computer-Hune Humon Interaction.
Tian Huang, Chun Yu, Weinan Shi, Zijian Peng, David Yang, Weiqi Sun, and Yuanchun Shi. Promptrpa: Gen- erating robotic process automation on smartphones from textual prompts. arXio preprint arXiv:2/04.02/7 2024a.
Xiaowei Huang, Wenjie Ruan, Wei Huang, Gao Jin, Yizhen Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, Kaiwen Cai, Yanghao Zhang, Sihao Wu, Peipei Xu, Dengyu Wu,, André Freitas, and Mustafa A. Mustafa. A survey of safety and trustworthiness of large language models through the lens of verification and validation. Artif. Intell. Rev., 57:175, 2023. URL https: //api. seapiisematticsccoller... wtt.://d.r...og/oorrppD-22333330 258823083
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey, anXiv preprint arxi::2422.0200 02711, 2024b.
Yuchao Huang, Junjie Wang, Zhe Liu, Yawen Wang, Song Wang, Chunyang Chen, Yuanzhe Hu, and Qing Wang, Crashtranslator: Automatically reproducing mobile application crashes directly from stack trace. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-13, 2024c.
Zhiyuan Huang, Ziming Cheng, Junting Pan, Zhaohui Hou, and Mingjie Zhan. Spiritsight agent: Advanced gui agent with one look. arXiv preprint Xio:2503.003166 2025c.
145

---

Published in Transactions on Machine Learning Research (06/2025)
Zheng Hui, Yinheng Li, Dan zhao, Tianyi Chen, Colby Banbury, and Kazuhito Koishida. Winelick: Gui grounding with multimodal large language models, 2025. URT http///wwww...r///////...s..o/
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint orXiv:2410.21276, 2024.
Open Interpreter. Open interpreter: A natural language interface for computers. GitHub repository, 2024. URL
Accessed: 2024-10-27.
lat Long long, Xiao Liu, Yuxuan Chen, Hanyu Lai, Shuntian Yao, Pengbo Shen, Hao Yu, Yuxiao Dong, and Jie Tang. Openwebagent: An open toolkit to enable web agents on large language modcls. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),
pp. 72-8 12-81,22 2024.
Lucija Ivančié, Dalia Suša Vugec, and Vesna Bosilj Vukšié. Robotic process automation: systematic literature review. In Business Process Management: Blockchain and Central and Eastern Europe Forum: BPM 2019 Blockchain and CEE Forum, Vienna, Austria, September 1 1-6, 2019, Proceedings 17, pp. 280-295. Springer, 2019.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. Xrxiv preprinl ar 22 167200 2024.
Lawrence Jang, Yinheng Li, Charles Ding, Justin Lin, Paul Pu Liang, Dan Zhao, Rogerio Bonatti, and Kazuhito Koishida. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks. arxiv preprint arXiv:2/10.19100, 2024.
Bernard Jim Jansen. The graphical user interface. ACM SIGCHI Bull, 30:22 26, 1998. URL https
//api. hapii.semattcsscoolar org/ htt://oopuIII::111000 18416305
Susmit Jha, Sumit Kumar Jha, Patrick Lincoln, Nathanicl D. Bastian, Alvaro Velasquez, and Sandeep Neema. Dehallucinating large language models using formal methods guided iterative prompting. 2023
IEFE International Conference on Assured Aulonomy (ICAA), pp. 149-152, 2023. URL. https::////
Senaatt:cccoollrrrrr///orpaaaa:p 260810111
Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, and Zhiyong Wu. Agentstore: Scalable integration of heterogencous agents as specialized generalist computer assistant.
arXiv preprinl arXiv:2410. 18603, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310. 06825, 2023.
Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, and Chi Zhang, Appagentx: Evolving gui agents proficient smartphone users. arXiv preprint orXin:2503.022 02266, 2025.
Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, et al. Xpert: Empowering incident management with query recommen- dations via large language models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1 1-13, 2024.
Yiqiao Jin, Stefano Petrangeli, Yu Shen, and Gang Wu. Screenlim: Stateful screen schema for efficient action understanding and prediction, 2025.
Kristinna Jokinen. User interaction in mobile navigation applications. In Map-based Mobile Services: Design, Interaction and Usability, pp. 168-197. Springer, 2008.
Christoforos Kachris. A survey on hardware accelerators for large language models. arXiv preprint 20110:2401.09890, 2024.
146

---

Published in Transactions on Machine Learning Research (06/2025)
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmente planning with contextual memory for multimodal Ilm agents. arXiv preprint or Xix::2200 03610, 2024.
Noam Kahlon, Guy Rom, Anatoly Efros, Filippo Galgani, Omri Berkovitch, Sapir Caduri, William E Bishop; Oriana Riva, and Ido Dagan. Agent-initiated interaction in phone ui automation. arXiv preprint arXiv:2503. 19537, 2025.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024. URL
17553
Su Kara, Fazle Faisal, and Suman Nath. Waber: Web agent benchmarking for elficiency and reliability. In ICLR 2025 Workshop Foundation Models in the Wild
katalon. Katalon studio: Easy test automation for web, api, mobile, and desktop, 2024. URL https Accessed: 2024-11-05.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Htillermeier A survey of reinforcement learning from human feedback. arXiv preprint ar Xim:23312. 14925, 2023.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint 0ao:22100024000 2022.
Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei Chao. Dual-view visual contextualization for web navigation, 2024. URL https://arxiv... hwtpg//bb/244222 24t1/24222004776
Callie Y Kim, Christine P Lee, and Bilge Mutlu. Understanding large-language model (llm)-oowered human- robot interaction. In Proceedings of the 2024 ACM/IEEE International Conference on Haman-Robbo Interaction, pp. 371-380, 2024a.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks, 2023. URL sttps://arxiv.org//ae
17491
Jackyeom Kim, Dong-Kt Ki Kim, Lajanugen Logeswaran, Sungryull Sohn, and Honglak Lee. Auto-intent: Automated intent discovery and self-exploration for large language model web agents. arXiv preprint 3rXiv:2410.225:0 2024b.
Sein Kim, Hongseok Kang, Seungyoon Choi, Donghyun Kim, Minchul Yang, and Chanyoung Park. Large language models meet collaborative filtering: An efficient all-round Ilm-based recommender system. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1395-1406, 2024c.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4015 4026, 2023.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024a. URL https: / //tp/a/avvvv//////a//eeeeeaaalv/ attg//bb/2401.... 13649
Jing Yu Koh, Stephen McAlcer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. Xiu preprint ar 0771:22407.01476, 2024b.
147

---

Published in Transactions on Machine Learning Research (06/2025)
Qichao Kong, Zhengwei Lv, Yiheng Xiong, Jingling Sun, Ting Su, Dingehun Wang, Letao Li, Xu Yang, and Gang Huo. Prophetagent: Automatically synthesizing gui tests from test cases in natural language for mobile apps.
Richard Koo and Sam Toueg. Checkpointing and rollback-recovery for distributed systems. IEEE Transac- tions Software Engineering, SE- 31, 1986. URL
wrg. org/CorpusIDD
206777989
Lars Krupp, Daniel Geißler, Paul Lukowicz, and Jakob Karolus. Towards sustainable web agents: A plea for transparency and dedicated metrics for energy consumption. arXiv preprint arXiv:2502.1190: 2025.
Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federaatedscope-lii A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5260 5271, 2024.
Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Aulowebglm: Bootstrap and reinforce a large language model-based web navigating agent, 2024. URL ptttps://arxiv.. wwwp:/iv.rr//bb/////000/ 0ttp//24440046.4
Hanyu Lai, Junjie Gao, Xiao Liu, Yifan Xu, Shudan Zhang, Yuxiao Dong, and Jie Tang. Androidgen: Building an android language agent under data scarcity. ar preprint arXiv:2504. 19298, 2025.
Yuanhong Lan, Yifei Lu, Zhong Li, Minxue Pan, Wenhua Yang, Tian Zhang, and Xuandong Li. Deeply reinforcing android gui testing with deep reinforcement learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1 13, 2024.
Z Lan. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint ar arXiv:1909. wtt.ii9999.9999 11942, 2019.
Hugo Laurengon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vasion-la models? ar preprint urx:2:2405. 0224000 2024a.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.00 00000999, 2024b.
Hansoo Lee, Joonyoung Park, and Uichin Lee. systematic survey on android api usage for data-driven analytics with smartphones. ACM Computing Surveys, 55(5):1-33 2022.
Jungiae Lee, Dongjae Lee, Chihun Choi, Youngmin Im, Jaeyoung Wi, Kihong Heo, Sangeun Oh, Sunjae Lee, and Insik Shin. Safeguarding mobile gui agent via based action verification. ar preprint arXiv:2503.18492, 2025.
Juyong Lee, Dongyoon Hahm, June Suk Choi, W Bradley Knox, and Kimin Lee. Mobilesafetybenchr Evaluating safety of autonomous agents in mobile device control. ar preprint ::Xii:2410.175200 2024a.
Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking mobile device htss/abs/244..1555
control agents across diverse configurations, 2024b. URL https://arxiv.
Sunjae Lee, Junyoung Choi, Jungiae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y Ko, Sangeun Oh, and Insik Shin. Explore, select, derive, and recall: Augmenting Ilm with human-like memory for mobile task 2023.
automation. ar preprint arXiv:2312.0300
Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, and Insik Shin. Mobilegpt: Augmenting Ilm with human- like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pp. 1119-1133, 2024c.
Wonbcom Lee, Jungi Lee, Junghwan Sco, and Jacwoong Sim. (InfiniGen]: Efficient generative inference of large language models with dynamic (KV) cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 155 172, 2024d.
148

---

Published in Transactions on Machine Learning Research (06/2025)
Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yacli, and Segev Shlomov. St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint ar 10.0241000703, 2024.
M Lewis. Bart: Denoising requence-to-sequeee pre-training for natural language generation, translation, and comprehension. Xix preprint orXii:1910.1344 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmmenee generation for knowledge- intensive nlp tasks. Advances in Newral Information Processing Systems, 33:9459-9477 2020.
Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Llm inference serving: Survey of recent. advances and opportunities. arXiv preprint arXiv:2407.1233 2024a.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixturo-of-cxperts model. arXiv preprint arXiv:2410.0590 2024b.
Eric Li and Jim Waldo. Websuite: Systematically evaluating why web agents fail. arXiv preprint arXiv:240600162) 2024.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for 'mind' exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. Hello again! I1m-powered personalized agent for long-term dialogue. arXiv preprint arXiv:2406. (xX0::2066.05925, 2024c.
Haoyuan Li, Hao Jiang, Tianke Zhang, Zhelun Yu, Aoxiong Yin, Hao Cheng, Siming Fu, Yuhao Zhang, and Wanggui He. Traineragent: Customizable and efficient model training through Ilm-powered multi-agent. system. arXiv preprint drXiv:231100622,, 2023b.
Hongxin Li, Jingfan Chen, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. Autogui: Scaling gui grounding with automatic functionality annotations from Ilms. arXic preprint Xrx:::2502.011 01977, 2025a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19743 PMLR, 2023c.
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use, 2025b.
Kanglin Li and Mengqi Wu. Effective GUI testing automation: Developing on automated GUI testing tool.
John Wiley & Sons, 2006.
Lin Li, Guikun Chen, Hanrong Shi, Jun Xiao, and Long Chen. A survey on multimodal benchmarks: In the era of large ai models. arXiv preprint arXiv:2409.1811 2024d.
Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. A zero-shot language agent for computer control with structured reflection. arXiv preprint 2xXf=:2310008740, 2023d.
Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. Sereen2vec: Semantic embedding of gui screens and gui components. In Proceedings of the 2021 CHT Conference on Human Factors in Computing Systems, pp. 1 15, 2021.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.00 0367999......... 2024c.
Wei Li, Fu-Lin Hsu, William Bishop, Folawiyo Campbell-Ajala, Max Lin, and Oriana Riva. Uinav: A practical approach to train on-device automation agents. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pp. 36 51, 2024f.
149

---

Published in Transactions on Machine Learning Research (06/2025)
Xiaolei Li, Jialun Cao, Yepang Liu, Shing- Chi Cheung, and Hailong Wang. Reusedroid: A vim-empowered android ui test migrator boosted by active feedback. or Xiv preprint or Nxc:2504.02233 2025c.
Xuan Li. Gui testing for android applications: survey. In 2023 7th International Conference on Computer, Software and Modeling (ICCSM), pp. 6-10. TEEE, 2023.
Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent 11824
v2: Advanced agent for flexible mobile interactions, 2024g. URL httpp:://arxivvorr///ttt
Yang Li and Otmar Hilliges. Artificial intelligence for human computer interaction: a modern approach. Springer, 2021.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequenees. In Proceedings of the 58th Anaual Meeting of the Association for Computational Linguistics, pp. 8198 8210, 2020a.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. arXiv preprinl ar 0an:2000004295 2020b.
Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint ar Xim:2980 10149, 2023e.
Youwei Li, Yangyang Li, and Yangzhao Yang. Test-agent: multimodal app automation lesting framework based on the large language model. In 2024 IEFE 4th International Conference on Digital Twins and Porallel Intelligence (DTPI), pp. 609 614. IEEE, 2024h.
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. Humanoid: A deep bearning-based approach to automated black-box android app testing. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1070-1073. IEEE, 2019.
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal Ilm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXim:2411.0 054559 2024i.
Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfci Yang, and Zhe Gan. Ferrot-ui 2: Mastcring universal user interface undorstanding across platforms. arxiv preprint or arxiv:2400.1 18967, 2024j.
Zongxia Li, Xiyang WU, Hongyang Du, Huy Nghicm, and Guangyao Shi. Benchmark evaluations, applications, and challenges of large vision language models: A survey. arXiv preprint ar Xrx::22501 002189, 2025d.
Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. Fia: Environmental injection attack on generalist web agents for privacy leakage. ar Xiv preprinl arXiv:2409.112 11295, 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024a.
Kevin Qinghong Lin, Pengehuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2794-2800 2023.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen WU, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. Videogui: A benchmark for gui automation from instructional videos, 2024b. URL
https://arxiv.ot org/abs//////////////////.b httgg//bs/2406.... 2406.10227
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent, 2024c. URL
https://arxiv. https://arxiv.oor/abb// www/aba/////eeeo.oooo..o.ee /2aps//2411111 17465
150

---

Published in Transactions on Machine Learning Research (06/2025)
Luyang Lin, Lingzhi Wang, Jinsong Guo, and Kam-Fai Fai Wong. Investigating bias in llm-based bias detection: Disparities between llss and human perception. arXiv preprint or arXin:2403.1 1489,, 2024d.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupal, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration, 2018. URL httpp///aaaxx...r/////////22
Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Shuai Ren, Hao Wang, Shibo He, and Wenchao Meng. Learnact: Few-shot mobile gui agent with unified demonstration benchmark, 2025a. URL
13805
Guangyi Liu, Pongxiang Zhao, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, Hao Wang, et al. poo-powered gui agents in phone automation: Surveying progress and prospects. arXiv preprint ar Xix:222004. 198338, 2025b.
Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp.
2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, et al. Pe-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc. arXiv preprint arXiv:2502. 1/282, 2025c.
Jiarun Liu, Jia Hao, Chunhong Zhang, and Zheng Hu. Wepo: Web element preference optimization for Ilm-based web navigation, 2024c. URL http::////wariw..//aaa/////s////
Jun Liu, Chaoyun Zhang, Jiaxu Qian, Minghua Ma, Si Qin, Chetan Bansal, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Large language models can deliver accurate and interpretable time series anomaly detection. Xiv preprint arXiv:2405.1537 2024d.
Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, and Xiang Yue. Harnessing webpage uis for text-rich visual understanding. arXiv preprint 11.,124100.122.. 2024e.
J Junpeng Liu, Yilan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visual- webbench: How far have multimodal lims evolved in web page understanding and grounding?, 2024f. URL https://arxiv. org/ abs/ 2404.044000 05955
Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language masel-based agents for software engineering: A survey, arXiv preprint or Xiv:2409. 02977, 2024g.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05400 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating 1lss as agents. arXiv preprint ar Xiv:2300.. 03688, 2023b.
Xíao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Tat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.0088 2024h.
Xiao Liu, Tianjie Zhang, Yu Gu, lal Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong. Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang. Visualagentbench: Towards large multimodal models visual foundation agents, 2024i. URL https://arxiv. wwwp://ppwwab...ooo///////
151

---

Published in Transactions on Machine Learning Research (06/2025)
Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, and Yan Lu. Ui-eli-syuth: Advancing gui grounding with large-scale instruction synthesis. or Xio preprint 11257 2025d.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprinl urXio:1907.11699 364, 2019.
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection, 2025e. URL
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners, 2025f. URL
14239
Yuxuan Liu, Hongda Sun, Wei Liu, Jian Luan, Bo Du, and Rui Yan. Mobilesteward: Integrating multiple app-oriented agents with self-evolution to automate cross-app instructions. arxiv preprinl arXiv:2502.1677 16796, 2025g.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012 10022, 2021.
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402. 14905, 2024j.
Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEFE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1355 1367. IEEE, 2023c.
Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. Make 11m a testing expert: Bringing human-like interaction to mobile gui testing via funetionality-awarr decisions. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp.
1-13, 2024k.
Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Boyu Wu, Yawen Wang, Jun Hu, and Qing Wang. Vision-driven automated mobile gui testing via multimodal large language model, 20241. URL https:
Zhuang Liu, Hanzi Mao, Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet. for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11976-11986, 2022.
Fanbin Lu, Zhisheng Zhong, Ziqin Wei, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Steve: Astep verification pipeline for computer-use use agent training. arXiv preprint arXiv:2503. 12532, 2025.
Junru Lu, Siyu An, Mingbao Lin, Gabriele Pergola, Yulan He, Di Yin, Xing Sun, and Yunsheng Wu. Memochat: Tuning 1mss to use memos for consistent long-range open-domain conversation. arXiv preprinl arXiv:2308.00 08239,, 2023.
Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Turn every application into agent: Towards efficient human-agent- computer interaction with api-first llm-based agents. arxiv preprint arxiv:2409. 1e24499..11111 17140, 2024a.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices, 2024b. URL htttp:://arriv.org//b//////////
Xing Han Lu, Zdenök Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. In International Conference on Machine Learning, pp.
PMI.R, 2024c.
152

---

Published in Transactions on Machine Learning Research (06/2025)
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina. Stanczak, Peter Shaw, Christopher J Pal, and Siva. Reddy. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. arXiv preprint arXiv:2504.0894 2025.
Yadong Lu, Jianwei Yang, Yclong Shen, and Ahmed Awadallah. Omniparsor for pure vision based gui agent. arxiv preprinl ar orXie:2408.00222 2024d.
Yuwen Lu, Yuewen Yang, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. Ai assistance for ux: A literature review through human-centered ai. arXiv preprint arXim:202200088) 2024c.
Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Laurence Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. Uxagent: An Ilm agent-based usability testing framework for web design. arXiv preprinl Xiv:25502. 12561, 2025a.
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.1630) 2025b.
Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye Hao, Jun Wang, and Kun Shao. Vimo: A generative visual gui world model for app agent, 2025a. URL https:
13936
Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. Visual test-time scaling for gui agent grounding, 2025b. URL https://arxiv..orgg
00684
Yang Luo, Qixun Zhang, Qingni Shen, Hongzhi Liu, and Zhonghai Wu. Android multi-level system permission management approach. ArXie, abs/1772.00221,, 2017. URL
semaaiiscchhir
htg//oorpusID:20099995
Michael Lutaaya. Rethinking app permissions on ios In Extended Abstracts of the 2018 CHI Conference Human Factors in Compuling Systems, pp. 1-6, 2018.
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn 11m agents. arviv preprint or arXiv:2401.13178, 2024a.
Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Lim agent with state-space exploration for web navigation. arXiv preprinl or arXi::2309.088 081702 2023.
Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Lim agent with state-space exploration for web navigation, 2024b. URL htttss://arx//// www.aas./////eoooooo//....ee. 23090081722222
Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. Caution for the environment: Multimodal agents are susceptible to environmental distractions, 2024c. URL
/ttpps//arriv.orr/bbb////// wtt.:oog///aaa///oeeaeeee 2404.02548448448
Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. Coco-agent: A comprehensive cognitive mllm agent for wrtp/bs/////...22// 11941
smartphone gui automation, 2024d. URL
Peihua Mai, Ran Yan, Zhe Huang, Youjia Yang, and Yan Pang. Split-and-denoise: Protect large language model inference with local differential privacy. Xi preprint urXim:2310.09133 2023.
Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. arXio preprint 141620055.4166, 1, 2020.
Podro Martins, Filipe Sá, Francisco Morgado, and Carlos Cunha. Using machine learning for cognitive robotic process automation (rpa). In 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1-6. IEFF, 2020.
153

---

Published in Transactions on Machine Learning Research (06/2025)
Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. arxi preprint orXiv:2404. 1 11584, 2024.
Sahisnu Mazumder and Oriana Riva. Flin: A flexible natural language interface for web navigation. arXic preprint Xrx::20000. 12844, 2020.
Larry R Medsker, Lakhmi Jain, al. Recurrent neural networks. Design and Applications, 5(64-67):2, 2001.
A. Memon, 1. Banerjee, N. Hashmi, and A. Nagarajan. Dart: framework for regression testing "nightly/dan /dily builds' of gui applications. In International Conference on Software Maintenance, 2003. ICSM 2003. 419, 2003a. doi: 1AAL099/ICCMM.20.223344..
Proceedings. pp. 410-419,
Atif M Memon, Martha E Pollack, and Mary Lou Soffa. Hierarchical gui test case generation using automated planning, IEEE transactions software engineering, 27(2):144 155, 2001.
Atif M Memon, Ishan Banerjee, and Adithya Nagarajan. Gui ripping: reverse engineering of graphical user interfaces for testing. In WCRE, volume 3, pp. 260, 2003b.
Ziyang Meng, Yu Dai, Zezheng Gong, Shaoxiong Guo, Minglong Tang, and Tongquan Wei. Vga: Vision gui assistant minimizing hallucinations through image-centric fine-tunig, 2024. URL https://arxiv.org abs/2406. 14056
Rafal Michalski, Jerzy Grobelny, and Waldemar Karwowski. The effects of graphical interface design characteristics on human-computer interaction task efficiency. Arxiv, abs/1211.6712, 2006. URL https:
stapp...mantiisccolaa..ooo :tt.://ooolaaatoooooooooopsi::sd 14:909550 14695409
Microsoft. Create desktop flows using record with copilot (preview), 2024. URL https://leart: microsoft
recoooo Accessed: 2024-11-16.
Sushmita Mitra and Tinku Acharya. Gesture recognition: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Revieus), 3113):311-324, 2007.
Kevin Moran, Cody Watson, John Hoskins, George Purnell, and Denys Poshyvanyk. Detecting and summa- rizing gui changes in evolving mobile apps. In Proceedings of the 33rd ACM//BER international conference on automated software engineering, pp. 443,553, 2018.
MosaicML. Mosaicml: Mpt-7b, 2023. URL http::/wwwwwoooiimmmcccoomo http://www...ll..ooooooooooaot// Accessed: 2024-11- 19.
Thiago Santos de Moura, Everton LG Alves, Hugo Feitosa de Figueirêdo, and Cláudio de Souza. Baptista. Cytestion: Automated gui testing for web applications. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering, pp. 388-397,11 2023.
MultiOn AI. Multion ai: Ai agents that act on your behalf, 2024. URL htttps::/wwwmmlttttttt ai/ 2024-10-26.
Accessed:
Shikhar Murty, Dzmitry Bahdanau, and Christopher D Manning. Nnetscape navigator: Complex demonstra- tions for web agents without a demonstrator. arXiv preprint 1XXix:2100002000 2024.
Michel Nass, Emil Alégroth, and Robert Feldt. Why many challenges with gui test automation (will) remain. Information and Software Technology, 138:106625, 2021.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. ar preprint arXiv:2307.06435, 2023.
Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, al. Vi-vision: desktop-centric gui benchmark for visual perception and interaction. Xiv preprint arxio:2509. 10008.15500 15661, 2025.
154

---

Published in Transactions on Machine Learning Research (06/2025)
Anthony Nguyen. Improved gui grounding via iterative narrowing, 2024. URL https://arxiv.org abs/
13591
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjic Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, and Franck Dernoncourt Gui agents: A survey, 2024. URL pattps:://aatt wwweriii.ogg/ooeeo absaaas//aa /242..2212..1 13501
Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip Yu, al. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. arXiv preprint arXiv:2503. 23350, 2025.
Runliang NIu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent, 2024. URL https: //arxiv. wwwp://r/abb//2..009000/
Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. Mobileflow: A mullimodal Ilm for mobile gui agent, 2024. URL https://arxiv. org/ wwtgg/bb//aaa //t.s///40..... 0ttp:/2277004444 04346
Juho-Jaakko Oksanen. Test automation for windows gui application. 2023.
OpenAdapt AI. OpenAdapt: Open Source Generative Process Automation, 2024. URL https://github com//ppendaattA//ppnnaaapp Accessed: 2024-10-26.
OpenAL. Gpt-4v(ision) system card. Technical report, OpenAI, September 2023. URL https://cdn.opent
con/paapers/GPV__Syseee. http:///eePYY_Syyttaa_rrrrrrr.
OpenAI, Computer-using agent: Introducing a universal interface for ai to interact with the digital world.
2025a. URL httpss//ppeeaal.cm//nnn/////// agina--anntt
OpenAl. Operator system card, January 2025b. Released on January 23, 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. Xr preprint arXiv:2304.07198 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human fcodback. Advances in neural information processing systems, 35:27730-2788 -77744,7774, 2022.
Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Awadallah. Explorer: Scaling explor web trajectory synthesis for multimodal web agents. Xin preprint Xrx::22502. 1 1357, 2025.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. In First Conference Language Modeling, 2024a.
Liangming Pan, Michael Stephen Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-corraction strategics. ArXio, abs/230 abss/2308.03111 03188, 2023a. URL htttts:://apppp wew.nticccoooaaaaaao..eoaaac org/CorpusID: set.anttccchoolrr..oooooooooooop :20.1226666666955
Lihang Pan, Bowen Wang, Chun Yu, Yuxuan Chen, Xiangyu Zhang, and Yuanchun Shi. Autotask: Executing arbitrary voice commands by exploring and learning from mobile gui. arXiv preprint TT:2232216006 2023b.
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. Webeanvas: Benchmarking web agents in online environments, 2024b. URL https://arxiv org/ haabs/2406..m 12373
155

---

Published in Transactions on Machine Learning Research (06/2025)
Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, and Kun Shao. Appvlm: A lightweight vision language model for online app control. aixiv preprint or Nii:2502.06395 2025.
Pawel Pawlowski, Krystian Zawistowski, Wojeiech Lapacz, Marcin Skorupa, Adam Wiacek, Sebastien Postansque, and Jakub Hoscilowiez. Tinyclick: Single-turn agent for empowering gui automation. arXiv preprint arXiv:2410. 11871, 2024.
Andrés Piñieiro-Martín, Carmen Garcia-Mateo, Laura Docio-Fernánder and Maria Del Carmen Lopez-Perez. Ethical challenges in the development of virtual assistants powered by large language models. Electronics, 12(14):3170, 2023.
Martin Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331 -3331-434, 1990.
434,
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199, 2024.
Zehan Qi Xiao Liu, lat Long Tong, Hanyu Lai, Xueqiao Sun, Xinyue Yang, Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training Ilm web agents via self-evolving online curriculun reinforcement learning, 2024. URL https: httpp:///arrii.oog////////1/// wwtp:os///aab./1..// 24111/23773
Ju Qian, Zhengyu Shang, Shuoyan Yan, Yan Wang, and Lin Chen. Roscript: A visual script driven truly non-intrusive robotic testing system for touch screen applications. In 2020 IEEE/ACM 42nd International
Conference on Software Engineering (ICSE), pp. 297-308, 2020.
Ju Qian, Yingwei Ma, Chenghao Lin, and Lin Chen. Accelerating oer-basod widget localization for test automation of gui applications. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1-13, 2022.
Yijun Qian, Yujic Lu, Alexander Hauptmann, and Oriana Riva. Visual grounding for user interfaces. In Yi Yang, Aida Davani, Avi Sil, and Anoop Kumar (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pp. 97 107, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/vv 100553/vv// 1122022..222mma www.024.aacclliimmmm.mmmm haaccl-nndsssrry...........n.... URL
hrrpo/looy.ogg////0/ org/2024 naacl - iind-strddut.
Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, et al. Taskweaver: A code-first agent framework. arXiv preprint 2rXi::2311.174/1 2023.
Yujia Qin, Yining Yo, Junjic Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Lí, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioncering automated gui interaction with native agents, 2025. URL https: http://xxxvvvog///aa// wwgg/ba///ae aaaaaaaaa2250 2501. 12326
Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. Mobile edge intelligence for large language models: A contemporary survey. arXiv preprint arXiw:2407.1802) 2024.
Alec Radford. Improving language understanding by generative pre-training. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748 8763. PMLR, 2021.
156

---

Published in Transactions on Machine Learning Research (06/2025)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Newral Information Processing Systems, 36, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 67, 2020.
Abdur Rahman, Rajat Chawla, Muskaan Kumar, Arkajit Datta, Adarsh Jha, Mukunda NS, and Ishaan Bhola. V-zen: Efficient gui understanding and precise grounding with a novel multimodal Ilm. arXiv preprinl ar Xiv:2405. 15341, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. Zero-shot text-to-image generation. In International conference machine learning, pp- 8821 8831. Pmlr, 2021.
Dezhi Ran, Hao Wang, Zihe Song, Mengzhou Wu, Yuan Cao, Ying Zhang, Wei Yang, and Tao Xie. Guardian: A runtime framework for l1m- lmmeb ui exploration. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 958 958-970, 2024.
Dezhi Ran, Mengzhou Wu, Hao Yu, Yuetong Li, Jun Ren, Yuan Cao, Xia Zeng, Haochuan Lu, Zexin Xu, Mengqian Xu, et al. Beyond pass or fail: A culti-dimensional benchmark for mobile ui navigation. arXiv preprint arXiv:2501.028 02866,, 2025.
ranorex. Ranorex studio: Test automation for gui testing, 2024. URL https://www.ranores com/ Accessod: 2024-11-05.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36: 59708 59728, 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Coupbell-Ajaala Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents, 2024. URL https:///xxvvvoo///aa/e
14573
Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8. Xiv preprint arXiv:2305.00972 2023.
Matthew Renze and Erhan Guven. Self-refloction in Ilm agents: Effects on problem-solving performance. arXiv preprint Xix::244 066822 2024.
Jorge Ribeiro, Rui Lima, Tiago Eckhardt, and Sara Paiva. Robotic process automation and artificial intelligence in industry 4.0-a literature review. Procedia Computer Science, 181:51-58, 2021.
Fernando Pastor Ricós, Rick Neeft, Beatriz Marín, Tanja EJ Vos, and Pekka Aho. Using gui change detection for delta testing, In International Conference on Research Challenges in Information Science, pp. 509 517. Springer, 2023.
Olivia Rodríguez-Valdo Valdés, Tanja EJ Vos, Pekka Aho, and Beatriz Marín. 30 years of automated gui testing: a bibliometric analysis. In Quality of Information and Communications Technology: 14th International Conference, QUATIC 2021, Algarve, Portugal, September 8-11, 2021, Proceedings 14, pp. 473 488.. Springer, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE conference on computer vision and pattern vecognition, pp. 10684-10695, 2022.
157

---

Published in Transactions on Machine Learning Research (06/2025)
Tim Rosenbach, David Heidrich, and Alexander Weinert. Automated testing of the gui of a real-life engineering software using large language models. In 2025 IEFE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 103 110. IEEE, 2025.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2024. URL http:pps//aaxi..ogggbb//////////
12950
Nicole Rupp, Katrin Peschke, Michael Köppl, David Drissner, and Thole Zuchner. Establishment of low-cost. laboratory automation processes using autoit and 4-axis robots. SLAS technology. 27(5):312 318, 2022.
Pascal J Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F Grewe, and Thilo Stadelmann. Ai agents for computer use: A review of ructoonnoodd computer control, gui automation, and operator assistants. arXie preprinl arXiv:2501.16130 2025.
Kabir S Said, Liming Nie, Adekunle A Ajibode, and Xueyi Zhou. Gui testing for mobile applications: objectives, approaches and challenges. In Proceedings of the 12th Asia-Pacific Symposium Internetware, pp. 51-60, 2020.
Harini Sampath, Alice Merrick, and Andrew Peter Maevean. Accessibility of command line interfaces. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021. URL https stapiissematiicscollaa http://ccccccoooooooooooosososos :ttg//ooppsII:::3877733
Iqbal Sarker. Lim potentiality and awareness: a position paper from the perspective of trustworthy and responsible ai modeling. Discover Artificial Intelligence, 4(1):40, 2024.
selenium. Selenium: Browser automation, 2024. URL
selenium. deviiiuudeee////e Accessed: 2024-11-05,
Mobina Shahbandeh, Parsa Alian, Noor Nashid, and Ali Mesbah. Naviqate: Functionality-guidee web application navigation. arXiv preprint arXiv:2409.1070 10741, 2024.
Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1):50-64 1951.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXix:242..0 08300, 2024.
Sarath Shekkizhar and Romain Cosentino. Agi is coming... right after learns to play wordle, 2025. URL
https://arxiv.org/ atee//4b/2500.... 15434
Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, and Xiangyang Ji. Falcon-ui: Understanding gui before following user instructions. arXiv preprint arXiv:2412. 09362, 2024a.
Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Monad Hadji, Aaron Podolny, and Ameet Talwalkar. Scribeagent: Towards specialized web agents using production-scale workflow data, 2024b. URL atttp:://axiia http://arvvvv.../a/a/./ httgg/ab//2411.... 2abbb///11.1000
Zhuocheng Shen. Lim with tools: A survey, arXiv preprint arXit:2409.18800 2024.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open- domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3135 3144. PMLR, 06 11 Aug 2017. URL httpps://proceeddnn// www.eiinnnnni.iimmemememmmeeeeee pr...pprsss/v7//hhii7a html
Yucheng Shi, Wenhao Yu, Wenlin Yao, Wenhu Chen, and Ninghao Liu. Towards trustworthy gui agents: A survey. ar Xiv preprint or orXiv:2003.221 23434, 2025.
158

---

Published in Transactions on Machine Learning Research (06/2025)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyn Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Newral Information Processing Systems, 36, 2024.
Brian Sierkowski. Achieving web accessibility. In Proceedings of the 30th annual ACM SIGUCCS conference on User services, pp. 291, 2002.
Kunal Singh, Shreyas Singh, and Mukund Khanna. Trishul: Towards region identification and screen hierarchy understanding for large vlm based gui agents, 2025. URL
08226
smarthear. Testcomplete: Automated ui testing tool, 2024. URL
testcomplete/ Accessod: 2024-11-05.
Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Agentbank: Towards generalized Ilm agents via fine-tuning on 50000+ interaction trajectories. arXiv preprinl ar Xix:24 0.07706, 2024a.
Yixiao Song, Katherine Thai, Chau Minh Pham, Yapei Chang, Mazin Nadaf, and Mohit Tyyer. Bearcubs: A benchmark for computer-using web agents. arXiv preprint arXiv:2503.074 079119, 2025.
Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-bases web agents. arXii preprint 1rX1::2410.166 2024b.
Yumpeng Song, Yiheng Bian, Yongtao Tang, Guiyu Ma, and Zhongmin Cai. Visiontasker: Mobile task
automation using vision based ui understanding and Ilm task planning. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST '24, pp. 1-17. ACM, October 2024c. doi: 10.1145/3654777 01114///66477733666666 URL.
doi 1rw///.....1//55777....7 3666666
Zirui Song, Yaohang Li, Meng Fang, Zhenhao Chen, Zecheng Shi, Yuan Huang, and Ling Chen. Mmac-copilot: Multi-modal agent collaboration operating system copilot. arXiv preprint arXiv:2404.1800 18074, 2024d.
Trisanth Srinivasan and Santosh Patapati. Webnav: An intelligent agent for voice-controlled web navigation.
arXiv preprint 13Xie25503.138/3, 2025.
Zinovia Stefanidi, George Margetis, Stavroula Ntoa, and George Papagiannakis. Real-time adaptation of context-aware intelligent user interfaces, for enhanced situational awareness. IEEE Access, 10:23367-23388 2022.
John Steven, Pravir Chandra, Bob Fleck, and Andy Podgurski. jrapture: A capture/replay tool for observation- based testing. SIGSOFT Softw. Eng. Notes, 25(5):158- August 2000. ISSN 0163-5948. doi: 00.1145/347636.34440 URL https:://di....o org/10..g1..... /145/117766633488888
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan O. Ank. Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments, 2025. URL https://arxiv.org:
Abb/22501.08893
Chuanneng Sun, Songiun Huang, and Dario Pompili. Llm-based multi-agent reinforcement learning: Current and future directions. arXiv preprint arXiv:2405. 11106, 2024a.
Jiahui Sun, Zhichao Hua, and Yubin Xia. Autoeval: A practical framework for autonomous evaluation of mobile agents. arXi preprint ar orXin2203. (xXi:2503300203,, 2025a.
Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards multi-modal conversational agents on mobile gui, 2022. URL httptttttaaaai...otot. www.aawwww...rga/a/////..... 11029
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu,, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412. 11..2/22.191110 19723, 2024b.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. Xiv preprint Xrv: 15389, 2023.
159

---

Published in Transactions on Machine Learning Research (06/2025)
Yuchen Sun, Shanhui Zhao, Tho Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. Gui-xplore: Empowering generalizable gui agents with one exploration. or Xie preprint arXin:203. 17709, 2025b.
Al Sweigart. Pyautogui: A cross-platform gui automation python module. GitHub repository, 2024. URL https://github. cww. ccwm///swweigaatt/yyttgoggg Accessed: 2024-10-27.
Rehan Syed, Suriadi Suriadi, Michael Adams, Wasana Bandara, Sander JJ Leemans, Chun Ouyang, Arthur HM Ter Hofstede, Iuge Van De Weerd, Moe Thandar Wynn, and Hajo Reijers. Robotic process automation: contemporary themes and challenges. Computers in Industry, 115:103162, 2020.
Jihoon Tack, Jachyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, and Jonathan Richard Schwarz. Online adaptation of language models with memory of amortized contexts. arXiv preprint #xXx:2403.04317, 2024.
Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. Axnav: Replaying accessibility tests from natural language. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-16, 2024.
Wrick Talukdar and Anjanava Biswas. Improving large language model (Ilm) fidelity through context- grounding: A systemalic approach to reliability and veracity. arXiv preprinl arXiv:2408.. 04023, 2024.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Jumpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifci Bi, Tianbao Xic, Pengjic Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojic Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering foundation agents towards general computer control, 2024a. URL httts:://arriiii org/ abs//403.3333
Zhaoxuan Tan and Meng Jiang. User modeling in the era of large language models: Current rescarch and future directions. arXiv preprint xrxi::2312. 11518, 2023.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrila Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey. Xixiv preprint arXiv:2402.130 13446, 2024b.
Brian Tang and Kang G Shin. Steward: Natural language web automation. orXio preprint (XXix:2409.15444 2024.
Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song. Weiming Lu, and Yucting Zhuang. Think twico, click once: Enhancing gui grounding via fast and slow systems. arXiv preprint arXiv:2503. 06470, 2025a.
Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, and Yueting Zhuang. A survey (m)llm-based gui agents, 2025b. URL
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models. arXie preprint arXiv:2404.14395 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint ar arXiv:2312. 11805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, al. Gemma: Open models based on gemini research and technology. Xiv preprint ar arXiv:2403.08.0 08295, 2024.
160

---

Published in Transactions on Machine Learning Research (06/2025)
Andae-Andrei Thil, Mirela Popa, and Gerasimos Spanakis. Navigating webai: Training agents to complete web tasks with large language models and reinforcement learning. In Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, volume 30 of SAC' '24, pp. 866 874. ACM, April 2024. doi: 10.1145/ 3605098.3635903 URL http:://x.ddo/ doi.. oww..o.gg1.... 001111114///000 330500660000... 3635903
George Thomas, Alex Chan, Jikun Kang, Wenqi Wu, Filippos Christianos, Fraser Greenlee, Andy Toulis, and Marvin Purtorab. Webgames: Challenging general-purpose web- ceb-browsing ai agents. orXiv preprint or arXiv:200..1 18356, 2025.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android, 2021a. CRL http::///aarxv.orrrrs///11.5. 13231
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint ar Xiv:21105. 13231, 2021b.
Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. Towards internet- scale training for agents. arXiv preprint arXie:2502.0677 2025.
Ada Defne Tur, Nicholas Meade, Xing Han Lù,, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Stańczak, and Siva Reddy. Safearena: Evaluating the safety of autonomous web agents, 2025. URL
04957
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint Xrt::1908. 0888,,2,,, 2019.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, and Manuela Veloso. Adaptagent: Adapting multimodal web agents with few-shot learning from human demonstrations. arXiv preprint 11X1v.2411.13451, 2024.
Minh Duc Vu, Han Wang, Zhuang Li, Jieshan Chen, Shengdong Zhao, Zhenchang Xing, and Chunyang Chen. Gptvoocetasser:: Lim-powered virtual assistant for smartphone. arXiv preprint Xrx::24401. 14268, 2024.
Abdul Wali, Saipunidzam Mahamad, and Suziah Sulaiman. Task automation intelligent agents: A review. Future Internet, 15(6):196, 2023.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, al. Efficient large language models: A survey. ar preprint orXin:2312.03880 2023.
Bowen Wang, Xinyuan Wang, Jiaqi Deng, Tianbao Xie, Ryan Li, Yanzhe Zhang, Gavin Li, Toh Jing Hua, lon Stoica, Wei-Lin Chiang, Diyi Yang, Yu Su, Yi Zhang, Zhiguo Wang, Victor Zhong, and Tao Yu. Computer agent arena: Compare & test computer use agents on crowdsourced real-world tasks, 2025a.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2creen2words: Automatic mobile summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 498 510, 2021.
Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Humnn Factors in Compuling Systems, pp. 1 17, 2023a.
Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjic Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llss, and trustworthiness, 2024a. URL http://awwww..rr///////1......0
161

---

Published in Transactions on Machine Learning Research (06/2025)
Jiaqi Wang, Zhengliang Liu, Lin Zhao, Zihao Wu, Chong Ma, Sigang Yu, Haixing Dai, Qiushi Yang, Yiheng Liu, Songyao Zhang, et al. Review of large vision models and visual prompt engineering. Mela-Radiokenn Radiology. pp. 100047, 2023b.
Jiayin Wang, Weizhi Ma, Peijie Sun, Min Zhang, and Jian- Yun Nie. Understanding user experience in large language model interactions. arXw preprint arXiv:2401.0832) 2024b.
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 2024c.
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. aobite-agent--2: Mobile device operation assistant with effective navigation via multi-agent collaboration, 2024d. URL htttps://axxi...t./ wtw.:orgg/bb..oo././// 01014
Junyang Wang. Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception, 2024e. URL https
16158
Junyang Wang, Haiyang Xu, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-
v: Learning mobile device operation through vidco-guidet multi-agent collaboration. arXiv preprint arXiv:2502.11110, 2025b.
Ke Wang, Tianyu Xia, Zhangxuan Gu, Yi Zhao, Shuheng Shen, Changhua Meng, Weiqiang Wang, and Ke Xu. E-ant: A largo-scale dataset for efficient automatic gui navigation, 2024f. URL attps://arxils org/ /bs/
2406.14250 14250
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024g.
Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, Qisheng Su, Jiayi Ye, Yudi Zhang, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Large action models: From inception to implementation, 2024h. URL
https://arxizs http://aaii...rrr///////2111/ 2042
Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. Mobilcagentbench: An efficient and user-friendly benchmark for mobile Ilm agents, 2024i. URL
https://arxiv.orge
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing lasion-l model's perception of the world at any resolution, 2024j. URL htttpe://arrdv..rr//aa////// 12191
Shuai Wang, Weiwen Liu, Jingxuan Chen, Weinan Gan, Xingshan Zeng, Shuai Yu, Xinlong Hao, Kun Shao, Yasheng Wang, and Ruiming Tang. Gui agents with foundation models: A comprehensive survey, 2024k. URL https: www.ariivv...oraeaeooeoe abs//////////// 2211/ 04890
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents. ar preprint ar 2rXiv:2410.1453 14803, 20241.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2024m. URL https: //ts:////xvvvi wwrxiv.or//ao htrg/abs/231111 03079.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. Advances in Neural Information Processing Systems, 36, 2024n.
162

---

Published in Transactions on Machine Learning Research (06/2025)
Wenhao Wang, Zijie Yu, William Liu, Rui Ye, Tian Jin, Stheng Chen, and Yanfeng Wang. Fedmo- bileagent: Training mobile agents using decentralized self-soureed data from diverse users. arXiv preprinl arXiv:2502.02998 2025c.
Wenhao Wang, Zijie Yu, Rui Ye, Jianqing Zhang, Siheng Chen, and Yanfeng Wang. Fedmabench: Bench- marking mobile agents decentralized heterogeneous user data, arXiv preprint ar arx::25033055 0514433 2025d.
Xiaoqiang Wang and Bang Liu. Oscar: Operating system control via state-aware reasoning and re-planning. arXiv preprint 2r1::22410.1888 2024.
Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yansong Tang. Ponder & press: Advancing visual gui agent 01268
towards general computer control, 2024o. URL
Yufci Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint orXiv:2307. 12077.1200 12966, 2023c.
Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks, 2025e. URL attts://arxivv. hww...org/bbss..... abs/ 2501. 11733
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models, arXiv preprint arXiv:1910.04733 782. 2019.
Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang, Officebench: Benchmarking language agents across multiple applications for office automation, 2024p. URL https
Ziwei Wang, Weizhi Chen, Leyang Yang, Sheng Zhou, Shengchu Zhao, Hanbei Zhan, Jiongehao Jin, Liangcheng Li, Zirui Shao, and Jiajun Bu. Mp-gui: Modality perception with milms for gui understanding. arXiv preprint ar Xiv:22503 14021, 2025f.
Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, and Daniel Fried. Inducing programmatic skills for agentic tasks. arXiv preprint ar arXiv:2504.0682) 2025g.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprinl ar Ni::2199.01652 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, el al. of prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3:1 40, 2016.
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pp.
2024a.
Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, and Yuanchun Li. Autodroid-v2 Boosting sim- based gui agents via code generation, 2024b. URL htttps://arx//
hwwgg/bb/22412.. 18116
Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. Droidbot-gpt: Gpt-powered ui automation for android, 2024c. URL https:://axxiv. orwpsiiiiiorrreeeeee abs /a////abb//230400666
Joel Wester, Tim Schrills, Henning Pohl, and Niels van Berkel. "as an ai language model, cannot": Investigating IIm denials of user requests. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2024.
163

---

Published in Transactions on Machine Learning Research (06/2025)
Thomas Wetzlmaier, Rudolf Ramler, and Werner Putschogl. A framework for monkey gui testing. In 2016 IEEE international conference on software tesling, verification and validation (ICST), pp. 416-4 423. IEFE, 2016.
Thomas D White, Gordon Fraser, and Guy Brown. Improving random gui testing with image-based widget detection. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, pp. 307-317, 2019.
Josephine Wolff, William Lehr, and Christopher Yoo. Lessons from gdpr for ai policymaking. Virginia Journal of Low & Technology, 27(:). 2024.
Michael Wornow, Avanika Narayan, Benjamin Viggiano, Ishan Khare, Tathagat Verma, Tibor Thompson, Miguel Angel Fuentes Hernandez, Sudharsan Sundar, Chloe Trujillo, Krrish Chawla, al. Wonderbread: A benchmark for evaluating multimodal foundation models on business process management tasks. In The Thirly- Conference on Neural Information Processing Systems Dalasels and Benchmarks Track.
Michael Wornow, Avanika Narayan, Krista Opsahl-Ong, Quinn McIntyre, Nigam Shah, and Christopher Re. Automating the enterprise with foundation models. Proceedings of the VLDB Endowment, 17(11): 1305-28122 2024.
Biao Wu, Yanda Li, Meng Fang, Zirui Song, Zhiwei Zhang, Yunchao Wei, and Ling Chen. Foundations and recent trends in multimodal mobile agents: A survey, arXiv preprint arXiv:2411.000 2024a.
Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXie preprint XXx:23055005222 2023a.
Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey Bigham. Webui: A dataset for enhancing visual ui understanding with web semantics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2023b.
Jialong WI, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, and Fei Huang. Webwalker: Benchmarking Ihms in web traversal, 2025a. URL htttps://are //wrxvvarvvvvv//////// org/abs/ 2501.07572
Qinchen Wu, Difci Gao, Kevin Qinghong Lin, Zhuoyu Wu, Xiangwu Guo, Peiran Li, Weichen Zhang, Hengxu Wang, and Mike Zheng Shou. Gui action narrator: Where and when did that action take place?, 2024b. URL https: http:///aaxiv.oggbb////////
13719
Qingyuan Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Vse-rl: Advancing autonomous vision-language agents with variational subgoal-conditioned reinforcement learning. arXiv preprint arXiv:2502.07944, 2025b.
Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. Mobilevlm: A vision-language model for better intra- and inter-ui understanding. 2024c. URL http://axxxvv.or/bbaaa/
14818
Qiuzhuo Wu, Wei Liu, Jian Luan, and Bin Wang. Reachagent: Euhancing mobile agent via page reaching and operation. arXiv preprint arXiv:2502 02955, 2025c.
Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1122 1136, 2023c.
Xiongfei Wu, Jiaming Ye, Ke Chen, Xiaofei Xie, Yujing Hu, Ruochen Huang, Lei Ma, and Jianjun Zhao, Widget detection-basod testing for industrial mobile games. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 173-184 IEEE, 2023d.
164

---

Published in Transactions on Machine Learning Research (06/2025)
Xuansheng Wu, Haiyan Zhao, Yaochen Zhu, Yucheng Shi, Fan Yang, Tianming Liu, Xiaoming Zhai, Wenlin Yao, Jundong Li, Mengnan Du, et al. Usable xai: 10 strategies towards exploiting explainability in the llm era. arXio preprint arXiv:2403.0880
2024d.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self- iefroveprv,nn 2024c. URL
https://arxivi www/wws.gbbb/... 2402.07456
Zhiyong Wi,, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint (tfe:2440.222111 2024f.
Zongru Wu, Pengzhou Cheng, Zheng Wu, Tianjic Ju, Zhuosheng Zhang, and Gongshen Liu. Smoothing grounding and reasoning for mllm-powered gui agents with query-oriented pivol. tasks. Xiv preprinl arXiv:2503.00400 2025d.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey, arXiv preprint (rX0: 07864, 2023.
Xiaobo Xia and Run Luo. Gui-rl: A generalist r1-style vision- languaee action model for gui agents. arXiv preprint ar Xiv:22504. 10458, 2025.
Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, et al. Guardagent: Safeguard Ilm agents by a guard agent via knowledge-enabled reasoning. arXiv preprint arXi::2406.090 09187, 2024.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for variety of vision tasks, 2023. URL
https://arxin httg//////311...../
Xisokui Xiao and Yufei Tao. Personalized privacy preservation. In Proceedings of the 2006 ACM SIGMOD international conference on Monagement of data, pp. 229-240, 2006.
Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large multimodal agents: A survey. arXiv preprint arXim:2402.5111 151166 2024a.
Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. Uied: a hybrid tool for gui element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Sofbware Engineering, pp.
2020.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. Openagents: An open platform for language agents in the wild, 2023. URL https://aast org.obgg/
2310. 10634
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks real computer environments, 2024b. URL pestps///arxiss org/
07972
Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the weakness of large language model agents within a complex android environment. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6061 6072, 2024.
Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. Advweb: Controllable black-bor box attacks on vlm-powered web agents. arxiv preprint arXiv:2410.17400 2024a.
165

---

Published in Transactions on Machine Learning Research (06/2025)
Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning, 2024b. URL htttp:://arxivvorr//bbb//// 2
Jia Xu, Weilin Du, Xiao Liu, and Xuejun Li. Llmdworkflow: An Ilm-based automated workflow model generation tool. Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024c. URL
Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. On-device language models: A comprehensive review. arXiv preprint arXiv:2409.00066 2024d.
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Ado Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, and Daniel Khashabi. Tur [k] ingbench: A challenge benchmark for web agents. arXiv preprint arXiv:2403. 1 11905, 2024e.
Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource-efficient Ilm and multimodal foundation models. arXiv preprint arX:::2401.066 08009, 2024f.
Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica S Lam. Grounding open-domain instructions to automate web support tasks, 2021. URL https://arxiv.ot org/ abs/2103.16057
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Philip Torr, Bernard Ghanem, and Guohao Li. Crab: Cross-environment agent benchmark for multimodal language model agents, 2024g. URL http:: wwwp:/wwww.rg/bb//o/.000.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models, arXiv preprint 131166,402.13116, 2024h.
Yibin Xu, Liang Yang, Hao Chen, Hua Wang, Zhi Chen, and Yaohua Tang. Deskvision: Large scale desktop region captioning for advanced gui agents. arXiv preprint arXiv:2503. 11170, 2025.
Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents, 2024i.
URL https://arxiv org org/ /a////b///211 24024
Yiheng Xu, Dunjic Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchon Mao, Caiming Xiong, and
Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials, 2024j. URL
httpa:://axxvv.or/abbbss org/abs////eeeeeoooeoo.goo..eere attpaa///aaa////////////aaeeaa/a 021//212.00999 09605
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2024k. URL
https://arxiv.ort http///b////2......// 2212.222220 04454
Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.018 013882 2025.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation, 2023a. URL https: org/ abs/ 23511
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint ar Xiv 1..22662,,77,, 2023b.
Jianwei Yang, Hao Zhang, Feng Li, Xucyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 11441, 2023.
166

---

Published in Transactions on Machine Learning Research (06/2025)
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. or Xiv preprinl arXiu:2502.13130, 2025a.
Jiaxi Yang and Haowen Hou. Rwkv-ui: Ui understanding with enhanced perception and reasoning, arXiv preprinl or Xic:2500. 03971, 2025.
Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-langnage programmer from environmental leedback. In European Conference on Computer Vision, pp. 20 38. Springer, 2025b.
Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzela Rangwala. Agentoccam: A simple yet strong baseline for Ilm-based web agents, 2024a. URL https
13825
Qi Yang, Weichen Bi, Haiyang Shen, Yaoqi Guo, and Yun Ma. Pixelweb: The first web gui dataset with pixel-wise labels, 2025c. URL attpss//arzztii org/ habbs//550...444
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions, 2024b. URL http:://arrxw..orrgabb///// 16256
Yulong Yang, Xinshan Yang, Shuaidong Li, Chenhao Lin, Zhengyu Zhao, Chao Shen, and Tianwei Zhang. Security matrix for multinodal agents on mobile devices: A systematic and proof of concept study. arXiv preprint Xiv:24407 09295, 2024c.
Yulong Yang, Xinshan Yang, Shuaidong Li, Chenhao Lin, Zhengyu Zhao, Chao Shen, and Tianwei Zhang. Systematic categorization, construction and evaluation of new attacks against multi-modal mobile gui agents, 2025d. URL https://arxiy: wrgg h 092955
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 20744 20757, 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.0362 2022b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023. URL htttp::/aarddv..rr//bbb///
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Eray Yapağeı, Yavuz Alp Sencer Öztürk, and Eray Tüzün. Bugeraft: End-to-end crash bug reproduction using Ilm agents in minccraft. arXiv preprint arXin:2503.2003
2025.
Faraz YazzaaiiBanfssheDaragl and Sam Malck. Deep gui: Black-box gui input generation with deep learning. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 905-916. IEEE, 2021.
Jiaming Ye, Ke Chen, Xiaofci Xic, Lei Ma, Ruochen Huang, Yingfeng Chen, Yinxing Xuc, and Jianjun Zhao. An empirical study of gui widget detection for industrial mobile games. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1427 1437, 2021.
Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, and Tianmin Shu. Realwebassist: A benchmark for long-horizon web assistance with real-world users. Xie preprint orxiv:250 10445, 2025.
167

---

Published in Transactions on Machine Learning Research (06/2025)
Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao, Hao Wang, Yu2222ggttviccchssennjaa Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin, et al. Proagent: From robotic process automation to agentic process automation. arXiv preprint arXiv:2311.1075/ 2023.
Tom Yeh, Sainng Chang, and Robert C Miller. Sikuli: using gui screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology, pp. 183 192,
2009.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. Xiv preprind arXiv:2306. 13549, 2023.
Yiwen Yin, Yu Mei, Chun Yu, Toby Jia-Jun Li, Aamir Khan Jadoon, Sixiang Cheng, Weinan Shi, Mohan Chen, and Yuanchun Shi. From operation to cognition: Automatic modeling coguitive dependencies from user demonstrations for gui task automation. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1-24, 2025.
Juyeon Yoon, Robert Feldt, and Shin Yoo. Intent-driven mobile gui testing with autonomouts large language model agents. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 129-139. IEEE, 2024.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfci Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.0770, 2023.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfci Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision, pp. 240-255. Springer, 2025.
Shengcheng Yu, Chunrong Fang, Ziyuan Tuo, Quanjun Zhang, Chunyang Chen, Zhenyu Chen, and Zhendong Su. Vision-based mobile app gui testing: A survey. ar preprint arXiv:2910. 13518, 2023.
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. R-judge: Benchmarking safety risk awareness for Ilm agents. arXiv preprinl ar Xiv:2401. 10019, 2024.
Sukmin Yun, Haokun Lin, Rusirn Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, et al. Web2code: A large-scale webpage-lo-oode to-code datasel and evaluation framework for multinodal lmss. arXiu preprint arXiv:2406. 200:240000008,, 2024.
Xia Zeng, Dengfeng Li, Wujie Zheng, Fan Xia, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. Automated test input generation for android: are we really there yet. in industrial case? In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, pp. 987-992, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342186. doi: 00055//222229229836683 URL ltttps://doi....o org/10. 1145/2950290. 2983958
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11975-11986, 2023.
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Finc-tuning large vision-language models as mariiinnnmakingggg agents via reinforccment learning. Xr preprint arXiv:2405.1 1022922, 2024.
Xian Zhan, Tianming Liu, Lingling Fan, Li Li, Sen Chen, Xiapu Luo, and Yang Liu. Rescarch on third-party libraries in android apps: A taxonomy and systematic literature review. IEEE Transactions on Software Engineering, (3(10):488144211 2021.
168

---

Published in Transactions on Machine Learning Research (06/2025)
Chaoyun Zhang, Paul Patras, and Hamed Haddadi. Deep learning in mobile and wireless networking: A survey. TEFE Commanications surveys & Iulorials, 21(3):2224223) 2287, 2019.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. UFO: A U1-Focused Agent for Windows OS Interaction. arXiv preprint 2r 07.09999, 2024a.
Chaoyun Zhang, Zicheng Ma, Yuhao Wu, Shilin He, Si Qin, Minghua Ma, Xiaoting Qin, Yu Kang, Yuyi Liang, Xiaoyu Gou, et al. Allhands: Ask me anything on large-scale verbatim feedback via large language models. arXiv preprint arXiv:2403. 15157, 2024b.
Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang. Api agents vs. gui agents: Divergence and convergence. arXiv preprint orXiv:2503.11069 2025a.
Chaoyum Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. Ufo2: The desktop agentos. arXiv preprint arXie:2504114603 2025b.
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users, 2023a. URL https://axxiv..rr www.aiiii.../abba...a. abwpg///b////......// 2312. 13771
Danqing Zhang, Balaji Rama, Jingyi Ni, Shiying He, Fu Zhao, Kunyu Chen, Arnold Chen, and Junyu Cao. Litewebagent: The open-source suite for aim-asedd web-agent applications. oriie preprint arXiv:2503.0900 2025c.
Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, and Kai Yu. Mobile-cnv: Building qualified evaluation benchmarks for Ilm-gui interaction, 2024c. URL http::: //arxiv org/abs/ 200../2335.086
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. xlam: A family of large action models to empower ai agent systems. Xrxiv preprint orXi::24090332 03215, 2024d.
Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, and Jianping Fan. Mobileexperts: A dynamic tool-enabled agent team in mobile devices, 2024e. URL https::/arxiv..ot htrgg/abs22407/ 03913
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-acioon-toough for gui agents. arXiv preprint arXiv:2403.02713, 2024f.
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: aftton-oocctioonnhhhu for gui agents, 2024g. URL http::////xvv..oo/aa//aa
02713
Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, and Junxian He. Breaking the data
gui agents through task generalization. arXiv preprint ar 2rXiv:2504.1122 2025d.
Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui task antomation, 2024h. URL https:
7/rxiiv. www..org/aabaa.. httg//bb//244.... 16054
Liang Zhang, Qin Jin, Haoyang Huang, Dongdong Zhang, and Furu Wei. Respond in my language: Mitigating language inconsistency in response generation based on large language models. In Proceedings of the 62nd Anncaal Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 4177 4192, 2024i.
Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bo- Bo-Chen Yu, Weisong Sun, and Zhenyu Chen. A critical review of large language model on software enginoering: An example from chatgpt and automated program repair. ArXiv, abs/2310.066 0.8899 2023b. URL httts::// www.ntisscooleee....oo www.ooorrr.ss.....
264127977
169

---

Published in Transactions on Machine Learning Research (06/2025)
Ruichen Zhang, Mufan Qiu, Zhen Tan, Mohan Zhang, Vincent Lu, Jie Peng, Kaidi Xu, Leandro Z Agudelo, Peter Qian, and Tianlong Chen. Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small llms. arXiv preprint 0rXie:2502.07942 2025e.
Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen, Xinbe Ma, Muyun Yang. Tiejun Zhao, and Min Zhang. Dynamic planning for Ilm-based graphical user interface automation. arXiv preprint arXiv:2410. 10.000007, 2024j.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint 230. 11222308.109022 10792, 2023c.
Xingxuan Zhang, Jianshong Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang. Shikai Guan, Jiazheng Xu, and Peng Cui. On the out-of-distribution generalization of multimodal large language models. arXiv preprint ar arXir:2402.0650) 065999 2024k.
Xinyu Zhang, Huiyu Xu, Zhongjie Ba, Zhibo Wang, Yuan Hong, Jian Liu, Zhan Qin, and Kui Ren. Privacyasst: Safeguarding user privacy in tool-using large language model agents. IEEE Transactions on Dependable and Secure Computing, 20241.
Yanzhe Zhang, Tao Yu, and Diyi Yang. Attacking vision- laiounlanguage computer agents via pop-ups, 2024m. URL https: http:///axxvvv..aaabb/. http://bs/2411.0222
Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. arXiv preprint orXiv:2408. 000008..1999 15978, 2024n.
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arXiv preprint ar aix::24406 06.006667.788 2024o.
Yudi Zhang, Pei Xiao, Lu Wang, Chaoyun Zhang, Meng Fang, Yali Du, Yevgeniy Puzyrev, Randolph Yao, Si Qin, Qingwei Lin, Mykola Pechenizkiy, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Ruag: generation for large language models, 2024p. URL https: /ttp://aaivv.vv.. org org/abs/
2411.03345
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's song in the ai ocean: A survey on hallucination in large language models, 2023d. URL https: abs/299 01219
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji- Ji-Rong Wen. A survey on the memory mechanism of large language model basod agents. arXiv preprint or 2rXiv:244.11301, 2024q.
Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. "it's a fair game", or is it? examining how users navigate disclosure risks and benefits when using Ilm-based conversational agents. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-26, 2024r.
Zhisong Zhang, Tianqing Fang, Kaixin Ma, Wenhao Yu, Hongming Zhang, Haitao Mi, and Dong Yu. Enhancing web agents with explicit rollback mechanisms. aixio preprint (xXin2254..1788 2025f.
Zhizheng Zhang, Wenxuan Xie, Xiaoyi Zhang, and Yan Lu. Reinforced ui instruction grounding: Towards a generic ui task automation api, 2023e. URL https:///arxn wwteziiv.rrg//bbeee org/abs//311..//////
Zhuohao Zhang, Eldon Schoop, Jeffrey Nichols, Anuj Mahajan, and Amanda Swearngin. From interaction to impact: Towards safer agent through understanding and evaluating mobile ui operation impacts. In Proceedings of the 30th International Conference on Intelligent User Interfaces, pp. 727-744, 2025g.
170

---

Published in Transactions on Machine Learning Research (06/2025)
Zhuosheng Zhang and Aston Zhang. You only look screens: Multimodal chain-of-action agents, 2024. URL
https://arxiv wwt.aawwwv.........../ooeeeeaeee org/abs/ abtpaa/...././//eee/ 11436
Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal 09992
internet agents, 2024s. URL https: httpaaaaavvrrrrrrrraa/aaaa
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Lim agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632-19642, 2024a.
Di Zhao, Longhui Ma, Siwei Wang, Miao Wang, and Zhao Lv. Cola: A scalable multi-agent framework for windows ui task automation. arXiv preprint arXiv:2508.0900 2025a.
Haoren Zhao, Tianyi Chen, and Zhen Wang. On the robustness of gui grounding models against image attacks. arXiv preprint arXiv:2504.04711 2025b.
Henry Hengyuan Zhao, Difei Gao, and Mike Zheng Shou. Worldgui: Dynamic testing for comprehensive desktop gui automation, 2025c. URL
Kangjia Zhao, Jiahui Song, Leigang Sha, HaoZhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, and Jianwei Yin. Gui testing arena: A unified benchmark for advancing autonomous gui testing agent. arXiv preprint ar Xrxi:22412 12. 18 18426, 2024b.
Pengyu Zhao, Zijian Jin, and Ning Cheng. An in-depth survey of large language model-based artificial intelligence agents. arXiv preprint 2rxiv 14965, 2023a.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint 2ix:22303.18222 2023b.
Arman Zharmagambetor, Chuan Guo, Ivan Evtimov, Maya Pavlova, Ruslan Salakhutdinov, and Kama- lika Chaudhuri. Agentdam: Privacy leakage evaluation for autonomous web agents. arXiv preprint Stw:2503.09780 2025.
Boyuan Zheng, Zeyuan Liu, Scott Salisbury, Zheng Du, Xuyan Huang, Qinyuan Zheng, Lee Davis, Michael Lin, Xiaolong Jin, Huan Sun, et al. Agentmonitor: Towards generalist guardrail for web agent.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-Av(ision) is a generalist web agent, if grounded, 2024a. URL https: /taps:///rziv. org/ aba/ 22t.//200..01111
Boyuan Zheng, Boyu Gou, Scott Salisbury, Zheng Du, Huan Sun, and Yu Su. Webolympus: An open platform for web agents on live websites. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 187 197, 2024b.
Boyuan Zheng, Michael Y Fatomi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, al. wkiveweaver: Web agents can self-improve by discovering and honing skills. arXie preprint orXiv:250,.0707, 2025a.
Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang, Vem: Euvironment-free exploration for training gui agent with value environment model. arXie preprint arXiv:2502. 18906, 2025b.
Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. Lifelong learning of large language model based agents: A roadmap. arXio preprint arXiv:2501.0799 2025c.
Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng Yan. Agentstudio: A toolkit for building general virtual agents, 2024c. URL https://arxxi org.obg.aa..o...ooo......... baabs/240..m 17918
171

---

Published in Transactions on Machine Learning Research (06/2025)
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: craeepl prompting with memory for computer control, 2024d. URL httges//Aaaxi.ooor/////22000..1 07863
Li Zhong and Zilong Wang. study on robustness and reliability of large language model code generation. arxiv preprint ar Xix:22908.. 10335, 2023.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations.
Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran Li. Proposer-aggent-evalaaaoo (pae): Autonomous skill discovery for foundation model internet agents, 2024a.
URL http//aaaiiiorrrra///aa////// 13194
Yuqi Zhou, Shuai Wang, Sunhao Dai, Qinglin Jia, Zhaocheng Du, Zhenhua Dong, and Jun Xu. Chop: Mobile operating assistant with constrained figh-ffequency optimized subtask planning, arXiv preprint arXiv:2503.0374 2025.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey efficient inference for large language models. arXiv preprinl arXiv:2404.1220, 2024b.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17114, 2023a.
Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Ningyu Zhang, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Knowagent: Knowledgo-augmented planning for Ilm-based agents. arXiv preprint ar 203.0000, 2024a.
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules. arXiv preprint arXiv23310007064, 2023b.
Zichen Zhu, Hao Tang, Yansi Li, Kunyao Lan, Yixuan Jiang, Hao Zhou, Yixiao Wang, Situo Zhang, Liangtai Sun, Lu Chen, et al. Moba: A two-level agent system for efficient mobile task automation. arXic preprint ar 140.22110...5 2024b.
Mingehen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jurgen Schmidhuber. agent-as-a-judge Evaluate agents with agents. 2024. URL https://api semanticscholar org/ CorpusID:273350000
Daniel Zimmermann and Anne Koziolek. Automating gui-based software testing with gpt-3. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 62 65, 2023a. doi: N,L100//CCTTW5553222222... 200200000...
Daniel Zimmermann and Anne Koziolek. Gui-based software testing: An automated approach using gpt-4 and selenium webdriver. In 2023 38th TEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), pp. 171 174. IEEE, 2023b.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey. Proceedings of the IEEE, 111(3):257 276, 2023.
172


Reviews:
[{"review_id": "IrHSaG22Ki", "reviewer": "Reviewer_MvVq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. Highly comprehensive, with textbook level breath and reference.\n2. It provides takeaways for people to quickly draw insights.\n3. It discussed current works' limitations from multi-perspective.\n4. It provides some suggestions for current works' limitations, thus, points out future direction.\n\n# Weaknesses\n\n(see requested changes)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work surveys large language model brained GUI agents. It covers GUI automation progression and history over the years, contains detailed introduction and survey for each component of LLM brained GUI agents, surveys exemplar LLM GUI agents, data collection methods, available data, key techniques related to build LLM GUI agents, evaluation metrics and benchmarks, application of LLM GUI agents, limitation of current works. It provides summarizing insights for each section, provided authors' suggestion about current limitation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Multi-platform LLM brained GUI agents is surveyed throughout each section, it would be good to highlight this earlier on more in introduction and motivation (even in title) for the survey to be more useful for people.\n2. 1.3 seems redundant given 1.2\n3. I think it make sense to provide a slightly more detailed compare and contrast between this survey and other surveys that addresses bot LLM agent and GUI automation as indicated by Table 2. This should help highlight the value of this survey. (a motivation of why this particular survey is needed)\n4. In 5.7.4, slightly more detail could be beneficial, for example, how is guidance and rules extracted and represented in LLM based GUI system?\n5. Motivation section is cluttered, a more structural motivation section could be more helpful, with each paragraph highlight one aspect.", "broader_impact_concerns": "The survey already touched ethical concern of LLM brained GUI agents and provided suggestions for solution. The survey itself does not raise any concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5UahnAmNVk", "reviewer": "Reviewer_122h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* overall comprehensiveness \n* wide range of papers with up-to-date coverage \n* thorough discussion of challenges and future work\n\nWeaknesses: \n* lack of clear definitions and connections: While superficially the topics seem to be carefully outlined, there are gaps and confusing repetitions. “LLM-Brained” is an ambiguous term and “LLM-powered” could encompass models with visual capabilities, tool use, and more. The terms introduced in the beginning of the paper do not connect consistently with those in the following sections. \n* limited insights on quantitative findings: Evaluation section (Section 9) describes metrics and benchmarks, and the text cites certain papers’ evaluation setups, but the survey does not aggregating how well current agents perform on those benchmarks. For example, it would have been informative to report ranges of success rates on common tasks (e.g. web form filling or app navigation).\n* assumption on reader familiarity: For instance, the term “Large Action Model (LAM)” is introduced as a key idea in Section 8 and highlighted in the research questions​. The authors do explain it (an LLM fine-tuned for GUI actions) later in Section 8.2, but the phrase appears early (even in the abstract​) without prior definition.\n* overwhelmingly lengthy and redundant: The paper is detailed, but it is hard to find relevant information. The paper repeatedly uses terms like “foundation” and “crucial” without sufficient evidence. There is lack of the clarity for the readers, especially with several long tables interspersed in main content.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* Structured Comprehensive Survey: The paper defines eight guiding research questions (RQ1–RQ8) ranging from the historical development of GUI agents, architectural components, representative frameworks, datasets for training, specialized “Large Action Models” (LAMs), evaluation metrics, real-world applications, and future research directions. \n\n* Practical \"cookbook\": the survey distills common design patterns and components of LLM-driven GUI agents. It presents a reference architecture and workflow, as well as enumerates the existing work for data and benchmarks across diverse platforms.\n\n* Analysis of Applications and Impact: The paper explores real-world applications of LLM-brained GUI agents (Section 10), notably in GUI testing and virtual assistants. The survey’s analysis of several applications highlights how LLM-based GUI agents have the potential to democratize GUI automation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "[Critical] Please Clarify/Update:\n\n0. Include a high-level comparison of agent performance or capabilities. add a small summary (in text or a new table) aggregating results from the literature. This is important for the technical contribution, and to show not just what methods exist but how effective they are.\n1. Choose a better name than \"LLM-brained\" that reflects the multimodality of the GUI agents in this survey. \n2. Provide clarity when the term \"UI\" used in the canonical sense, and when it is repurposed for the GUI agent (in sec 5.1 with description of environment, prompt engineering, etc). \n3. Provide a cross reference for such phrases if they appear early without prior definition. For example, the term “Large Action Model (LAM)” is explained (an LLM fine-tuned for GUI actions) later in Section 8.2., it is introduced much earlier and also highlighted in the research questions​. The research questions could have a link to the sections where they are addressed.\n4. Explain the difference between “Agent Instruction” and “System Prompt” in Section 5.3.\n5. Clarify differences between planning, complementary outputs\n6. In Section 5.4. Consider renaming “Action Inference” to “Action Prediction” if that better aligns with the underlying process. \n7. Consider including tools and function calling schemas, perhaps as part of Complementary Information / miscellaneous part of the prompt.\n8. Is Environment Perception in Sec 7.1.1. and Environment Feedback in Sec 5.x the same thing?\n9. Provide what context is not considered in the literature yet, say for deployment considerations and practical integration. for e.g., GUIs change frequently with software updates; an agent might break if a button moves. How do frameworks handle this (do they retrain, rely on robust vision, etc.)?\n\n[good to have] Structural fixes:\n\n1. It'll be better to have a few salient works in main discussion and refer to the detailed table in the appendix. Pruning the papers in each category for the main text will also highlight any key takeaways or insights. The goal should be to maintain comprehensiveness but improve readability by trimming excess detail.\n2.  In Fig 4, update the architecture workflow so that it shows how the user interacts with the agent, including aspects like request clarification, error recovery, and user control resumption. Specify how the agent prompts users for clarification and how it resumes after user intervention.\n3. In Fig 5, provide insights for the readers to understand what challenges and action space each GUI offers, what set of tasks are considered in each setting?\n\n\n[good to have] Minor typos and fixes:\n\n1. In Sec1.3 (Survey Structure): “Section ~2~ 4 for the evolution of LLM-powered GUI agents”​. \n2. In Table 1, ~Trem~ Term for STM\n3. “challenges” is misspelled as “chanllenges”\n4. Formatting issues, like in tables, excessive length due to hyperlinks, the content goes beyond the margin, why have the \"Platform\" column when all are agents for that platform in the table?\n5. Improve figure and table captions, for e.g., Figure 3 (the evolution timeline) could add that \" by year and platform​\" and any key takeaway from it.", "broader_impact_concerns": "While the paper discusses the limitations in Section 11 and list about the several papers for each topic on privacy and ethical concerns, it will benefit from a more detailed discussion of the implications Autonomous GUI agents performing critical operations. \nParticularly, the survey should to highlight different scenarios where we need accountability (if an agent performs an unintended action (e.g., deleting data), it becomes essential to track decisions and provide an explainable audit trail), or privacy (there is a risk that data might be inadvertently exposed during processing, especially when data is sent to external APIs). \n\nCurrently, they are not addressed in the Broader Impact Statement section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pLc2BPJQk6", "reviewer": "Reviewer_Citg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# strengths\n- Covers citations to many systems and prior surveys in the area.\n- Provides examples and visuals to make specific areas and applications clear and concrete for readers.\n\n\n# weaknesses\n- It was not possible to easily tell which work is peer-reviewed vs what is pre-publication. This makes it difficult to assess the strength of research in particular topics.\n- The sections often read more as a sequence of compressed abstracts, instead of offering much insight or perspective. I found the takeaways to be the most helpful parts, yet these were often fairly high level. \n\t- Structurally the paper would benefit from offering the key insights or organizing takeaways at the opening of sections, before offering the literature that supports these claims. It was difficult to read as I was struggling to understand the narrative until reaching the end of each section.\n- The historical review was fairly light and did not offer much insight. It could be cut without any harm to the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# contributions\nThe paper is a survey and focuses on documenting existing efforts across various applications of LLMs and VLMs to interacting with varying GUI enviroments: web interfaces, mobile devices, and desktop applications, including cross-platform. The survey touches on the history of systems to interact with GUIs prior to the use of LLMs or VLMs, the components of these systems, datasets used to train them, evaluation metrics and frameworks, and real-world applications. The conclusion summarizes known challenges and future areas of work.\n\nOverall the paper contributes a collection of references to the key areas and a taxonomy of shared features in each of the topics above.\n\n\n# new knowledge\nAs a survey new knowledge is not a major focus of the work. The primary contribution is offering a shared resource and capturing trends in specific areas.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# critical\n- Which citations are archival publications?\n\t- The paper offers many references to other works and other surveys. But many of these are pre-publication (on arxiv, or at least the citations only reference the arxiv version). \n\t- I worry that I cannot be sure if the systems cited really work, which makes the value of the review lower. At the least the survey should distinguish archival vs non-archival references for readers to make their own informed choice on trusting the references. This could perhaps be through a color code or other symbol that is easily recognized in the text.\n\t- This is a major barrier to using the work as a canonical reference on the state of the field. Unpublished work may not meet peer review standards. For industry systems (like GPT) it is understandable to not have peer-reviewed references as these systems have passed a \"user test\" for quality. The same is not true from non-archival academic systems. \n- The paper should provide a clearer set of taxonomies or sub-categories to orient the reader from the beginning of the paper and each section. It was hard to follow the organization and often only clear after reaching the take-aways at the end of sections.\n- What are the impacts of the growth of more OS-level LLMs and VLMs?\n\t- Siri / Apple Intelligence and Microsoft CoPilot (as an application) are not addressed, despite their ubiquity and growing capabilities. \n\t- How will this impact where LLMs / VLMs can be developed or deployed when systems have pre-existing integrated capabilities?\n\t- This is a pressing topic given the survey focus. At the least these trends should be acknowledge and incorporated into discussions of future directions.\n- (tied to the above) What trends are robust to improving LLMs or VLMs?\n\t- The paper highlights many trends, yet does little to contextualize them to ongoing rapid progress in LLMs and VLMs. It is important to offer some insight on how trends in increasingly baseline VLM or LLM capabilities are solving some problems and not others. This is an important aspect to trends in the area I was seeking from a survey (and part of the TMLR call for surveys).\n\t- Another trend worth discussion is the rise of inference-time computing and reasoning models. I recognize this is a newer development, but it plays into questions about rising baseline model capabilities.\n\n\n# strengthen\n- Are there datasets that include simulators / environments that agents can interact with?\n\t- The evaluation section would benefit from highlighting this particular distinction, as evaluation on offline data is often hard to translate to live systems.\n- The historical review was fairly light and did not offer much insight. It could be cut without any harm to the main paper.\n- The challenges and roadmap would benefit from specifying which areas are unique to GUI agents, as opposed to general to LLMs or VLMs. Many stated areas are general to the LLM and VLM literature.\n- (minor) Many of the summary tables (ex: Table 47, 'Project' column) have text overflowing the boundaries. This should be corrected to be more readable.\n- (minor) The citations seem to be in the wrong format, where names are not in the appropriate parentheses.\n\t- This applies to all the in-text references. Here's a clear case from page 8 (section 2.2 Surveys on LLM Agents): \"Surveys by Guo et al., Guo et al. (2024b) and Han et al., Han et al. (2024) provide comprehensive overviews of the current landscape, challenges, and future directions in this area.\"\n\t\t- \"Guo et al., Guo et al. (2024b)\" should be something like \"Guo et al. (2024b),\"", "broader_impact_concerns": "Section 11 on limitations and challenges addresses several relevant topics including privacy, safety, and ethical and regulatory challenges. The points there are reasonable for the topic. The focus is on user-facing applications, so it is reasonable to focus on the end consumer influences.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers generally agree that this is a timely survey on an important topic. After the rebuttal and revision, the paper is more readable and better presented. The authors should incorporate the reviewer comments into their camera ready version.
Decision: Accept (poster)