Judging with Confidence: Calibrating Autoraters to Preference
  Distributions
Authors: name: Zhuohang Li, name: Xiaowei Li, name: Chengyu Huang, name: Guowang Li, name: Katayoon Goshvadi, name: Bo Dai, name: Dale Schuurmans, name: Paul Zhou, name: Hamid Palangi, name: Yiwen Song, name: Palash Goyal, name: Murat Kantarcioglu, name: Bradley A. Malin, name: Yuan Xue
Organization: Google

The alignment of large language models (LLMs) with human values increasingly
relies on using other LLMs as automated judges, or ``autoraters''. However,
their reliability is limited by a foundational issue: they are trained on
discrete preference labels, forcing a single ground truth onto tasks that are
often subjective, ambiguous, or nuanced. We argue that a reliable autorater
must learn to model the full distribution of preferences defined by a target
population. In this paper, we propose a general framework for calibrating
probabilistic autoraters to any given preference distribution. We formalize the
problem and present two learning methods tailored to different data conditions:
1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a
reinforcement learning approach for sparse, binary labels. Our empirical
results show that finetuning autoraters with a distribution-matching objective
leads to verbalized probability predictions that are better aligned with the
target preference distribution, with improved calibration and significantly
lower positional bias, all while preserving performance on objective tasks.