GUI-G^2: Gaussian Reward Modeling for GUI Grounding
Authors: name: Fei Tang; name: Zhangxuan Gu; name: Zhengxi Lu; name: Xuyang Liu; name: Shuheng Shen; name: Changhua Meng; name: Wen Wang; name: Wenqi Zhang; name: Yongliang Shen; name: Weiming Lu; name: Jun Xiao; name: Yueting Zhuang
Published: 2025-07-21 17:53:42

Abstract: Graphical User Interface (GUI) grounding maps natural language instructions
to precise interface locations for autonomous interaction. Current
reinforcement learning approaches use binary rewards that treat elements as
hit-or-miss targets, creating sparse signals that ignore the continuous nature
of spatial interactions. Motivated by human clicking behavior that naturally
forms Gaussian distributions centered on target elements, we introduce GUI
Gaussian Grounding Rewards (GUI-G^2), a principled reward framework that
models GUI elements as continuous Gaussian distributions across the interface
plane. GUI-G^2 incorporates two synergistic mechanisms: Gaussian point
rewards model precise localization through exponentially decaying distributions
centered on element centroids, while coverage rewards assess spatial alignment
by measuring the overlap between predicted Gaussian distributions and target
regions. To handle diverse element scales, we develop an adaptive variance
mechanism that calibrates reward distributions based on element dimensions.
This framework transforms GUI grounding from sparse binary classification to
dense continuous optimization, where Gaussian distributions generate rich
gradient signals that guide models toward optimal interaction positions.
Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro
benchmarks demonstrate that GUI-G^2, substantially outperforms
state-of-the-art method UI-TARS-72B, with the most significant improvement of
24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides
superior robustness to interface variations and enhanced generalization to
unseen layouts, establishing a new paradigm for spatial reasoning in GUI
interaction tasks.