GUI Agents: A Survey
Authors: name: Dang Nguyen; name: Jian Chen; name: Yu Wang; name: Gang Wu; name: Namyong Park; name: Zhengmian Hu; name: Hanjia Lyu; name: Junda Wu; name: Ryan Aponte; name: Yu Xia; name: Xintong Li; name: Jing Shi; name: Hongjie Chen; name: Viet Dac Lai; name: Zhouhang Xie; name: Sungchul Kim; name: Ruiyi Zhang; name: Tong Yu; name: Mehrab Tanjim; name: Nesreen K. Ahmed; name: Puneet Mathur; name: Seunghyun Yoon; name: Lina Yao; name: Branislav Kveton; name: Thien Huu Nguyen; name: Trung Bui; name: Tianyi Zhou; name: Ryan A. Rossi; name: Franck Dernoncourt
Published: 2024-12-18 04:48:28

Abstract: Graphical User Interface (GUI) agents, powered by Large Foundation Models,
have emerged as a transformative approach to automating human-computer
interaction. These agents autonomously interact with digital systems or
software applications via GUIs, emulating human actions such as clicking,
typing, and navigating visual elements across diverse platforms. Motivated by
the growing interest and fundamental importance of GUI agents, we provide a
comprehensive survey that categorizes their benchmarks, evaluation metrics,
architectures, and training methods. We propose a unified framework that
delineates their perception, reasoning, planning, and acting capabilities.
Furthermore, we identify important open challenges and discuss key future
directions. Finally, this work serves as a basis for practitioners and
researchers to gain an intuitive understanding of current progress, techniques,
benchmarks, and critical open problems that remain to be addressed.