Click, Type, Repeat: A Comprehensive Survey on GUI Agents
Authors: Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, Franck Dernoncourt
Accepted at: acl25


Reviewer paper summaries: This paper is a survey on Graphical User Interface (GUI) agents, which includes Static Datasets, Agent Architectures, Agent Training Methods, and Open Problems & Challenges.

This work presents a comprehensive review of agents with a graphical user interface (GUI) as the observation space. Utilizing Large Foundation Models, these agents achieve more automated human-computer interactions by mimicking human behavior in intuitive GUI operations. The article claims to propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. It begins with the traditional reinforcement learning formalism of GUI agent tasks, introduces various datasets and environments, summarizes the architecture and training paradigms of some GUI agents, and concludes with a discussion on the future developments and challenges of these agents.
Summary of strengths: The content is very comprehensive and has researched many relatively recent relevant papers.

1. The manuscript is written in clear and concise English, which helps to understand the concepts related to GUI agents.

2. Section 3 is well-structured and introduces the architecture and training paradigm of GUI agents in a coherent manner. The organization helps the reader grasp the techniques and their applications.

3. The manuscript covers the basic concepts of GUI agents, providing a solid theoretical foundation for the following discussion.
Summary of weaknesses: There is no major weakness, however, I still have some worries.

1.Currently, all the content is textual, lacking an overview or some visual elements. I am concerned that readers might find it difficult to read.

2.As a comprehensive survey, its content is not limited to 8 pages; however, there is a lack of more work in the appendix now.

3.I also notice that there is a similar concurrent work ``GUI Agents with Foundation Models: A Comprehensive Survey''.

1. A significant limitation of this paper is the lack of tables, figures, or other visual aids that would help the reader better understand the concepts and compare existing work. As this is a survey paper, the inclusion of clear and informative figures or tables is necessary, especially when discussing datasets, environments, and evaluation metrics. 

2. While the abstract claims that the authors propose a unified framework for GUI agents, the actual discussion of the framework is vague. In line 58, the paper introduces the framework but fails to provide details on its organization. Simply framing the review as a framework in itself does not constitute a meaningful contribution. Without these details, the paper's contribution seems limited and the core idea of a "unified framework" is not fully realized.

3. Section 4 of the paper contains several inconsistencies. Section 4.1 introduces interfaces for GUI agent.It discuss various interface types such as API and HTML/DOM-based interface. However, according to line 75, the agent should be based on a graphical user interface. Including a discussion of non-GUI interfaces such as APIs and HTML/DOM seems irrelevant to the scope of the paper and creates confusion. Section 4.2 lacks detail and classification, resulting in an incomplete discussion. 

4. The discussion does not delve into key issues such as improving understanding of user intent, ensuring privacy and security, and optimizing inference speed for personalized recommendations. These are important, but the manuscript merely touches on these points without providing actionable insights for future research. 

5. The manuscript lacks some review references and specific paper citations for addressing issues and challenges.
Comments, suggestions and typos: NA

Comments after Author Discussion:
I have read the author's response.

1. It is strongly recommended to include at least one table summarizing the datasets, environments, and evaluation metrics used in the reviewed works. I suggest provide a clearer comparison of existing methods and facilitate reader understanding. If space limitations are an issue, this can be included in an appendix

2. The manuscript should more clearly reiterate the core contributions. Although the abstract mentions a proposal for a unified framework, but not fully explained or distinguished from existing surveys. Authors should clear what makes their proposed framework innovative and how it improves from existing work in the field. 

3. Section 4 discusses the interfaces of the GUI agent. however, this section is confusing, especially with the inclusion of APIs and HTML/DOM-based interfaces, which is not the focus of this article on graphical user interfaces. Redesign this section to focus on relevant interfaces, and provide clearer categorization and examples.

------------------------------------------------------------------------------
Further Assessment Based on Author Response

I appreciate the authors' detailed responses and proposed revisions. After reviewing their clarifications and planned improvements, I acknowledge that these changes will significantly enhance the manuscript. The key improvements include:

1. Addition of Figures and Tables

a) A dataset and environment summary table in Section 3 will improve readability and comparison.

b) A unified framework visualization in Section 4 will clarify key relationships.

c) The structured comparison of perception modalities (Tables 1 & 2) provides a more systematic view of different approaches.

2. Expanded Discussion on Challenges and Open Problems

a) The authors have proposed more actionable research directions for user intent understanding, security/privacy, inference latency, and personalization.

b) These additions will help contextualize the survey within broader research challenges.

Considering these planned improvements, I have adjusted my Overall Assessment from 2.5 to 3.5. I believe that with these revisions, the paper will provide a more structured and comprehensive survey of GUI agents.
Overall assessment scores: 3

3.5
Metareview score: 3
Metareview: This paper provides a comprehensive survey on Graphical User Interface (GUI) agents, focusing on their application in automating human-computer interactions via large foundation models. It presents a detailed taxonomy of GUI agent tasks, architecture, training methods, and open challenges, offering a solid foundation for researchers and practitioners in the field.
Summary of reasons to publish: The paper is well-written, clear, and informative, presenting a thorough review of GUI agents with a useful taxonomy. It effectively covers recent work and key concepts, including the distinction between static vs. dynamic environments and task success vs. step success. Also it highlights important challenges and open problems, which can guide future research.