Annotation-Efficient Universal Honesty Alignment
Authors: name: Shiyu Ni; name: Keping Bi; name: Jiafeng Guo; name: Minghao Tang; name: Jingtong Wu; name: Zengxin Han; name: Xueqi Cheng
Published: 2025-10-20 13:05:22

Abstract: Honesty alignment-the ability of large language models (LLMs) to recognize
their knowledge boundaries and express calibrated confidence-is essential for
trustworthy deployment. Existing methods either rely on training-free
confidence estimation (e.g., token probabilities, self-consistency) or
training-based calibration with correctness annotations. While effective,
achieving universal honesty alignment with training-based calibration requires
costly, large-scale labeling. To support annotation-efficient training, we
introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that
first elicits internal confidence using inexpensive self-consistency
supervision, then calibrates this confidence with a small set of correctness
annotations. To support a large-scale study, we release HonestyBench, a
benchmark covering ten free-form QA datasets with 560k training and 70k
evaluation instances annotated with correctness and self-consistency signals.
Experiments show that EliCal achieves near-optimal alignment with only 1k
correctness annotations (0.18% of full supervision) and better alignment
performance on unseen MMLU tasks than the calibration-only baseline, offering a
scalable solution toward universal honesty alignment in LLMs.