Annotation-Efficient Universal Honesty Alignment
Authors: name: Shiyu Ni, name: Keping Bi, name: Jiafeng Guo, name: Minghao Tang, name: Jingtong Wu, name: Zengxin Han, name: Xueqi Cheng

Honesty alignment-the ability of large language models (LLMs) to recognize
their knowledge boundaries and express calibrated confidence-is essential for
trustworthy deployment. Existing methods either rely on training-free
confidence estimation (e.g., token probabilities, self-consistency) or
training-based calibration with correctness annotations. While effective,
achieving universal honesty alignment with training-based calibration requires
costly, large-scale labeling. To support annotation-efficient training, we
introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that
first elicits internal confidence using inexpensive self-consistency
supervision, then calibrates this confidence with a small set of correctness
annotations. To support a large-scale study, we release HonestyBench, a
benchmark covering ten free-form QA datasets with 560k training and 70k
evaluation instances annotated with correctness and self-consistency signals.
Experiments show that EliCal achieves near-optimal alignment with only 1k
correctness annotations (0.18% of full supervision) and better alignment
performance on unseen MMLU tasks than the calibration-only baseline, offering a
scalable solution toward universal honesty alignment in LLMs.