Generalized Correctness Models: Learning Calibrated and Model-Agnostic
  Correctness Predictors from Historical Patterns
Authors: name: Hanqi Xiao; name: Vaidehi Patil; name: Hyunji Lee; name: Elias Stengel-Eskin; name: Mohit Bansal
Published: 2025-09-29 16:19:01

Abstract: Generating accurate and calibrated confidence estimates is critical for
deploying LLMs in high-stakes or user-facing applications, and remains an open
challenge. Prior research has often framed confidence as a problem of eliciting
a model's "self-knowledge", i.e., the ability of an LLM to judge whether its
own answers are correct; this approach implicitly assumes that there is some
privileged information about the answer's correctness that is accessible to the
model itself. However, our experiments reveal that an LLM attempting to predict
the correctness of its own outputs generally performs no better than an
unrelated LLM. Moreover, we hypothesize that a key factor in building a
"Correctness Model" (CM) is exposure to a target model's historical
predictions. We propose multiple methods to inject this historical correctness
information, creating a Generalized Correctness Model (GCM). We first show that
GCMs can be trained on the correctness data from many LLMs and learn patterns
for correctness prediction applicable across datasets and models. We then use
CMs as a lens for studying the source of correctness prediction ability and its
generalization, systematically controlling their training data and finding that
answer phrasing is a strong predictor for correctness. We further explore
alternative methods of injecting history without training an LLM, finding that
including history as in-context examples can help improve correctness
prediction, and post-hoc calibration can provide complementary reductions in
calibration error. We evaluate GCMs based on Qwen3-8B across 5 model families
and the MMLU and TriviaQA datasets, as well as on a downstream selective
prediction task, finding that reliable LLM confidence estimation is a
generalizable and model-agnostic skill learned by systematically encoding
correctness history rather than a model-specific skill reliant on
self-introspection.