API Agents vs. GUI Agents: Divergence and Convergence
Authors: name: Chaoyun Zhang; name: Shilin He; name: Liqun Li; name: Si Qin; name: Yu Kang; name: Qingwei Lin; name: Dongmei Zhang
Published: 2025-03-14 04:26:21

Abstract: Large language models (LLMs) have evolved beyond simple text generation to
power software agents that directly translate natural language commands into
tangible actions. While API-based LLM agents initially rose to prominence for
their robust automation capabilities and seamless integration with programmatic
endpoints, recent progress in multimodal LLM research has enabled GUI-based LLM
agents that interact with graphical user interfaces in a human-like manner.
Although these two paradigms share the goal of enabling LLM-driven task
automation, they diverge significantly in architectural complexity, development
workflows, and user interaction models.
  This paper presents the first comprehensive comparative study of API-based
and GUI-based LLM agents, systematically analyzing their divergence and
potential convergence. We examine key dimensions and highlight scenarios in
which hybrid approaches can harness their complementary strengths. By proposing
clear decision criteria and illustrating practical use cases, we aim to guide
practitioners and researchers in selecting, combining, or transitioning between
these paradigms. Ultimately, we indicate that continuing innovations in
LLM-based automation are poised to blur the lines between API- and GUI-driven
agents, paving the way for more flexible, adaptive solutions in a wide range of
real-world applications.