Title: No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
Primary area: interpretability and explainable AI
Keywords: LLMs, linear probes, activations, correctness, confidence, uncertainty, mechanistic interpretability, probing, generalization, direction

Abstract: Do large language models (LLMs) anticipate when they will answer correctly?  To study this, we extract activations after a question is read but before any tokens are generated, and train linear probes to predict whether the model’s forthcoming answer will be correct. Across three open‐source model families ranging from 7 to 70 billion parameters, projections on this “in-advance correctness direction’’ trained on generic trivia questions predict success in distribution and on diverse out‐of‐distribution knowledge datasets, indicating a deeper signal than dataset-specific spurious features, and outperforming black‐box baselines and verbalised predicted confidence. Predictive power saturates in intermediate layers and, notably, generalisation falters on questions requiring mathematical reasoning. Moreover, for models responding “I don’t know’’, doing so strongly correlates with the probe score, indicating that the same direction also captures confidence. By complementing previous results on truthfulness and other behaviours obtained with probes and sparse auto-encoders, our work contributes essential findings to elucidate LLM internals.

TL;DR: By reading only question-time activations, we show that a tiny linear probe predicts if the next answer will be correct for certain questions on a wide range of families and model sizes.