UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based
  Mobile GUI Agents
Authors: name: Han Xiao; name: Guozhi Wang; name: Yuxiang Chai; name: Zimu Lu; name: Weifeng Lin; name: Hao He; name: Lue Fan; name: Liuyang Bian; name: Rui Hu; name: Liang Liu; name: Shuai Ren; name: Yafei Wen; name: Xiaoxin Chen; name: Aojun Zhou; name: Hongsheng Li
Published: 2025-05-27 17:58:06

Abstract: In this paper, we introduce UI-Genie, a self-improving framework addressing
two key challenges in GUI agents: verification of trajectory outcome is
challenging and high-quality training data are not scalable. These challenges
are addressed by a reward model and a self-improving pipeline, respectively.
The reward model, UI-Genie-RM, features an image-text interleaved architecture
that efficiently pro- cesses historical context and unifies action-level and
task-level rewards. To sup- port the training of UI-Genie-RM, we develop
deliberately-designed data genera- tion strategies including rule-based
verification, controlled trajectory corruption, and hard negative mining. To
address the second challenge, a self-improvement pipeline progressively expands
solvable complex GUI tasks by enhancing both the agent and reward models
through reward-guided exploration and outcome verification in dynamic
environments. For training the model, we generate UI- Genie-RM-517k and
UI-Genie-Agent-16k, establishing the first reward-specific dataset for GUI
agents while demonstrating high-quality synthetic trajectory gen- eration
without manual annotation. Experimental results show that UI-Genie achieves
state-of-the-art performance across multiple GUI agent benchmarks with three
generations of data-model self-improvement. We open-source our complete
framework implementation and generated datasets to facilitate further research
in https://github.com/Euphoria16/UI-Genie.