UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and
  Precise Inference-Time Grounding
Authors: name: Shuquan Lian; name: Yuhang Wu; name: Jia Ma; name: Zihan Song; name: Bingqi Chen; name: Xiawu Zheng; name: Hui Li
Published: 2025-07-29 17:22:07

Abstract: The emergence of Multimodal Large Language Models (MLLMs) has driven
significant advances in Graphical User Interface (GUI) agent capabilities.
Nevertheless, existing GUI agent training and inference techniques still suffer
from a dilemma for reasoning designs, ineffective reward, and visual noise. To
address these issues, we introduce UI-AGILE, a comprehensive framework
enhancing GUI agents at both the training and inference stages. For training,
we propose a suite of improvements to the Supervised Fine-Tuning (SFT) process:
1) a Continuous Reward function to incentivize high-precision grounding; 2) a
"Simple Thinking" reward to balance planning with speed and grounding accuracy;
and 3) a Cropping-based Resampling strategy to mitigate the sparse reward
problem and improve learning on complex tasks. For inference, we present
Decomposed Grounding with Selection, a novel method that dramatically improves
grounding accuracy on high-resolution displays by breaking the image into
smaller, manageable parts. Experiments show that UI-AGILE achieves the
state-of-the-art performance on two benchmarks ScreenSpot-Pro and
ScreenSpot-v2. For instance, using both our proposed training and inference
enhancement methods brings 23% grounding accuracy improvement over the best
baseline on ScreenSpot-Pro.