UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and
  Precise Inference-Time Grounding
Authors: name: Shuquan Lian, name: Yuhang Wu, name: Jia Ma, name: Zihan Song, name: Bingqi Chen, name: Xiawu Zheng, name: Hui Li

The emergence of Multimodal Large Language Models (MLLMs) has driven
significant advances in Graphical User Interface (GUI) agent capabilities.
Nevertheless, existing GUI agent training and inference techniques still suffer
from a dilemma for reasoning designs, ineffective reward, and visual noise. To
address these issues, we introduce UI-AGILE, a comprehensive framework
enhancing GUI agents at both the training and inference stages. For training,
we propose a suite of improvements to the Supervised Fine-Tuning (SFT) process:
1) a Continuous Reward function to incentivize high-precision grounding; 2) a
"Simple Thinking" reward to balance planning with speed and grounding accuracy;
and 3) a Cropping-based Resampling strategy to mitigate the sparse reward
problem and improve learning on complex tasks. For inference, we present
Decomposed Grounding with Selection, a novel method that dramatically improves
grounding accuracy on high-resolution displays by breaking the image into
smaller, manageable parts. Experiments show that UI-AGILE achieves the
state-of-the-art performance on two benchmarks ScreenSpot-Pro and
ScreenSpot-v2. For instance, using both our proposed training and inference
enhancement methods brings 23% grounding accuracy improvement over the best
baseline on ScreenSpot-Pro.