Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
Authors: name: Zhen Yang; name: Zi-Yi Dou; name: Di Feng; name: Forrest Huang; name: Anh Nguyen; name: Keen You; name: Omar Attia; name: Yuhao Yang; name: Michael Feng; name: Haotian Zhang; name: Ram Ramrakhya; name: Chao Jia; name: Jeffrey Nichols; name: Alexander Toshev; name: Yinfei Yang; name: Zhe Gan
Published: 2025-09-30 17:13:56

Abstract: Developing autonomous agents that effectively interact with Graphic User
Interfaces (GUIs) remains a challenging open problem, especially for small
on-device models. In this paper, we present Ferret-UI Lite, a compact,
end-to-end GUI agent that operates across diverse platforms, including mobile,
web, and desktop. Utilizing techniques optimized for developing small models,
we build our 3B Ferret-UI Lite agent through curating a diverse GUI data
mixture from real and synthetic sources, strengthening inference-time
performance through chain-of-thought reasoning and visual tool-use, and
reinforcement learning with designed rewards. Ferret-UI Lite achieves
competitive performance with other small-scale GUI agents. In GUI grounding,
Ferret-UI Lite attains scores of 91.6%, 53.3%, and 61.2% on the
ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI
navigation, Ferret-UI Lite achieves success rates of 28.0% on AndroidWorld
and 19.8% on OSWorld. We share our methods and lessons learned from
developing compact, on-device GUI agents.