UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
Authors: name: Zhengxi Lu; name: Jiabo Ye; name: Fei Tang; name: Yongliang Shen; name: Haiyang Xu; name: Ziwei Zheng; name: Weiming Lu; name: Ming Yan; name: Fei Huang; name: Jun Xiao; name: Yueting Zhuang
Published: 2025-09-15 03:24:08

Abstract: Graphical User Interface (GUI) agents have demonstrated remarkable progress
in automating complex user interface interactions through reinforcement
learning. However, current approaches face a fundamental dilemma: offline RL
enables stable training on pre-collected trajectories, but struggles with
multi-step task execution for lack of trajectory-level reward signals; online
RL captures these signals through environment interaction, but suffers from
sparse rewards and prohibitive deployment costs. To address it, we present
Semi-online Reinforcement Learning, a novel paradigm that simulates online RL
on offline trajectories. During each rollout process, we preserve the original
model output within the multi-turn dialogue, where a Patch Module adaptively
recovers the divergence between rollout and expert trajectories. To capture
long-term training signals, Semi-online RL introduces discounted future returns
into the reward computation and optimizes the policy with weighted step-level
and episode-level advantages. We further introduce Semi-Online Performance
(SOP), a metric that aligns better with true online performance, serving as a
practical and effective proxy for real-world evaluation. Experiments show that
ours Semi-online RL achieves SOTA performance among 7B models across four
dynamic benchmarks, with significant gains over the base model (e.g., +12.0% on
AndroidWorld, +23.8% on AITW), demonstrating significant progress in bridging
the gap between offline training efficiency and online multi-turn reasoning.
The code is available at https://github.com/X-PLUG/MobileAgent/tree/main/UI-S1.