HyperClick: Advancing Reliable GUI Grounding via Uncertainty Calibration
Authors: name: Shaojie Zhang; name: Pei Fu; name: Ruoceng Zhang; name: Jiahui Yang; name: Anan Du; name: Xiuwen Xi; name: Shaokang Wang; name: Ying Huang; name: Bin Qin; name: Zhenbo Luo; name: Jian Luan
Published: 2025-10-31 08:07:02

Abstract: Autonomous Graphical User Interface (GUI) agents rely on accurate GUI
grounding, which maps language instructions to on-screen coordinates, to
execute user commands. However, current models, whether trained via supervised
fine-tuning (SFT) or reinforcement fine-tuning (RFT), lack self-awareness of
their capability boundaries, leading to overconfidence and unreliable
predictions. We first systematically evaluate probabilistic and verbalized
confidence in general and GUI-specific models, revealing a misalignment between
confidence and actual accuracy, which is particularly critical in dynamic GUI
automation tasks, where single errors can cause task failure. To address this,
we propose HyperClick, a novel framework that enhances reliable GUI grounding
through uncertainty calibration. HyperClick introduces a dual reward mechanism,
combining a binary reward for correct actions with a truncated Gaussian-based
spatial confidence modeling, calibrated using the Brier score. This approach
jointly optimizes grounding accuracy and confidence reliability, fostering
introspective self-criticism. Extensive experiments on seven challenge
benchmarks show that HyperClick achieves state-of-the-art performance while
providing well-calibrated confidence. By enabling explicit confidence
calibration and introspective self-criticism, HyperClick reduces overconfidence
and supports more reliable GUI automation.