Estimating Knowledge in Large Language Models Without Generating a
  Single Token
Authors: name: Daniela Gottesman; name: Mor Geva
Published: 2024-06-18 14:45:50

Abstract: To evaluate knowledge in large language models (LLMs), current methods query
the model and then evaluate its generated responses. In this work, we ask
whether evaluation can be done before the model has generated any
text. Concretely, is it possible to estimate how knowledgeable a model is about
a certain entity, only from its internal computation? We study this question
with two tasks: given a subject entity, the goal is to predict (a) the ability
of the model to answer common questions about the entity, and (b) the
factuality of responses generated by the model about the entity. Experiments
with a variety of LLMs show that KEEN, a simple probe trained over internal
subject representations, succeeds at both tasks - strongly correlating with
both the QA accuracy of the model per-subject and FActScore, a recent
factuality metric in open-ended generation. Moreover, KEEN naturally aligns
with the model's hedging behavior and faithfully reflects changes in the
model's knowledge after fine-tuning. Lastly, we show a more interpretable yet
equally performant variant of KEEN, which highlights a small set of tokens that
correlates with the model's lack of knowledge. Being simple and lightweight,
KEEN can be leveraged to identify gaps and clusters of entity knowledge in
LLMs, and guide decisions such as augmenting queries with retrieval.