Explaining Sources of Uncertainty in Automated Fact-Checking
Authors: name: Jingyi Sun; name: Greta Warren; name: Irina Shklovski; name: Isabelle Augenstein
Published: 2025-05-23 13:06:43

Abstract: Understanding sources of a model's uncertainty regarding its predictions is
crucial for effective human-AI collaboration. Prior work proposes using
numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain
uncertainty that arises from conflicting evidence, leaving users unable to
resolve disagreements or rely on the output. We introduce CLUE
(Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the
first framework to generate natural language explanations of model uncertainty
by (i) identifying relationships between spans of text that expose
claim-evidence or inter-evidence conflicts and agreements that drive the
model's predictive uncertainty in an unsupervised way, and (ii) generating
explanations via prompting and attention steering that verbalize these critical
interactions. Across three language models and two fact-checking datasets, we
show that CLUE produces explanations that are more faithful to the model's
uncertainty and more consistent with fact-checking decisions than prompting for
uncertainty explanations without span-interaction guidance. Human evaluators
judge our explanations to be more helpful, more informative, less redundant,
and more logically consistent with the input than this baseline. CLUE requires
no fine-tuning or architectural changes, making it plug-and-play for any
white-box language model. By explicitly linking uncertainty to evidence
conflicts, it offers practical support for fact-checking and generalises
readily to other tasks that require reasoning over complex information.