HyperClick: Advancing Reliable GUI Grounding via Uncertainty Calibration
Authors: name: Shaojie Zhang, name: Pei Fu, name: Ruoceng Zhang, name: Jiahui Yang, name: Anan Du, name: Xiuwen Xi, name: Shaokang Wang, name: Ying Huang, name: Bin Qin, name: Zhenbo Luo, name: Jian Luan

Autonomous Graphical User Interface (GUI) agents rely on accurate GUI
grounding, which maps language instructions to on-screen coordinates, to
execute user commands. However, current models, whether trained via supervised
fine-tuning (SFT) or reinforcement fine-tuning (RFT), lack self-awareness of
their capability boundaries, leading to overconfidence and unreliable
predictions. We first systematically evaluate probabilistic and verbalized
confidence in general and GUI-specific models, revealing a misalignment between
confidence and actual accuracy, which is particularly critical in dynamic GUI
automation tasks, where single errors can cause task failure. To address this,
we propose HyperClick, a novel framework that enhances reliable GUI grounding
through uncertainty calibration. HyperClick introduces a dual reward mechanism,
combining a binary reward for correct actions with a truncated Gaussian-based
spatial confidence modeling, calibrated using the Brier score. This approach
jointly optimizes grounding accuracy and confidence reliability, fostering
introspective self-criticism. Extensive experiments on seven challenge
benchmarks show that HyperClick achieves state-of-the-art performance while
providing well-calibrated confidence. By enabling explicit confidence
calibration and introspective self-criticism, HyperClick reduces overconfidence
and supports more reliable GUI automation.