Do I Know This Entity? Knowledge Awareness and Hallucinations in
  Language Models
Authors: name: Javier Ferrando; name: Oscar Obeso; name: Senthooran Rajamanoharan; name: Neel Nanda
Published: 2024-11-21 16:05:58

Abstract: Hallucinations in large language models are a widespread problem, yet the
mechanisms behind whether models will hallucinate are poorly understood,
limiting our ability to solve this problem. Using sparse autoencoders as an
interpretability tool, we discover that a key part of these mechanisms is
entity recognition, where the model detects if an entity is one it can recall
facts about. Sparse autoencoders uncover meaningful directions in the
representation space, these detect whether the model recognizes an entity, e.g.
detecting it doesn't know about an athlete or a movie. This suggests that
models can have self-knowledge: internal representations about their own
capabilities. These directions are causally relevant: capable of steering the
model to refuse to answer questions about known entities, or to hallucinate
attributes of unknown entities when it would otherwise refuse. We demonstrate
that despite the sparse autoencoders being trained on the base model, these
directions have a causal effect on the chat model's refusal behavior,
suggesting that chat finetuning has repurposed this existing mechanism.
Furthermore, we provide an initial exploration into the mechanistic role of
these directions in the model, finding that they disrupt the attention of
downstream heads that typically move entity attributes to the final token.