The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer
  Use
Authors: name: Siyuan Hu; name: Mingyu Ouyang; name: Difei Gao; name: Mike Zheng Shou
Published: 2024-11-15 16:23:52

Abstract: The recently released model, Claude 3.5 Computer Use, stands out as the first
frontier AI model to offer computer use in public beta as a graphical user
interface (GUI) agent. As an early beta, its capability in the real-world
complex environment remains unknown. In this case study to explore Claude 3.5
Computer Use, we curate and organize a collection of carefully designed tasks
spanning a variety of domains and software. Observations from these cases
demonstrate Claude 3.5 Computer Use's unprecedented ability in end-to-end
language to desktop actions. Along with this study, we provide an
out-of-the-box agent framework for deploying API-based GUI automation models
with easy implementation. Our case studies aim to showcase a groundwork of
capabilities and limitations of Claude 3.5 Computer Use with detailed analyses
and bring to the fore questions about planning, action, and critic, which must
be considered for future improvement. We hope this preliminary exploration will
inspire future research into the GUI agent community. All the test cases in the
paper can be tried through the project:
https://github.com/showlab/computer_use_ootb.