Title: Uncertainty-Aware LLM Probing
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models (LLMs), Probing, Uncertainty estimation (UQ), Out-of-distribution (OOD) detection, Calibration & confidence scoring, Gradient-based sensitivity score, Safety evaluation / guardian models

Abstract: Probing the hidden states of LLMs has been broadly studied in the literature and shown to be an efficient and effective method to elicit linguistic knowledge and even higher-level behaviors from the models; for example, factual accuracy and
uncertainty. However, most evaluations focus on specific short-form question answering scenarios. Further, the applicability of probes has been recently questioned, particularly in out-of-distribution (OOD) scenarios. Given the benefits including
the efficiency of probes and the variety of use cases, we study whether and when
common uncertainty estimation techniques make probes more reliable. Moreover,
we design a novel, robust gradient-based score. We study various established
uncertainty estimation techniques with a focus on probes for LLM correctness. Our
evaluation covers many different LLMs and a variety of task domains with smaller
distribution shifts and OOD scenarios. Most importantly, we find that, in LLM
classifiers such as guardian models (i.e., a type of LLM judge to evaluate prompt
safety), uncertainty estimators considerably improve probe applicability. Generally,
they however only work in domain. We specifically find that none of the methods
studied works reliably under smaller distribution shifts. Overall, our score ranks
among the best performers, provides interpretable OOD signals, and turns out to
be useful for data analysis as well.