WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation
Authors: name: Henry Hengyuan Zhao; name: Difei Gao; name: Mike Zheng Shou
Published: 2025-02-12 01:06:10

Abstract: Current GUI agents have achieved outstanding performance in GUI element
grounding. However, planning remains highly challenging, especially due to
sensitivity to the initial state of the environment. Specifically, slight
differences in the initial state-such as the target software not being open or
the interface not being in its default state-often lead to planning errors.
This issue is widespread in real user scenarios, but existing benchmarks fail
to evaluate it. In this paper, we present WorldGUI, a novel GUI benchmark that
designs GUI tasks with various initial states to simulate real computer-user
interactions. The benchmark spans a wide range of tasks across 10 popular
software applications, including PowerPoint, VSCode, and Adobe Acrobat. In
addition, to address the challenges of dynamic GUI automation tasks, we propose
GUI-Thinker, a holistic framework, leveraging a critique mechanism, that
effectively manages the unpredictability and complexity of GUI interactions.
Experimental results demonstrate that GUI-Thinker significantly outperforms
Claude-3.5 (Computer Use) by 14.9% in success rate on WorldGUI tasks. This
improvement underscores the effectiveness of our critical-thinking-based
framework in enhancing GUI automation.