Confidence Regulation Neurons in Language Models
Authors: name: Alessandro Stolfo; name: Ben Wu; name: Wes Gurnee; name: Yonatan Belinkov; name: Xingyi Song; name: Mrinmaya Sachan; name: Neel Nanda
Published: 2024-06-24 01:31:03

Abstract: Despite their widespread use, the mechanisms by which large language models
(LLMs) represent and regulate uncertainty in next-token predictions remain
largely unexplored. This study investigates two critical components believed to
influence this uncertainty: the recently discovered entropy neurons and a new
set of components that we term token frequency neurons. Entropy neurons are
characterized by an unusually high weight norm and influence the final layer
normalization (LayerNorm) scale to effectively scale down the logits. Our work
shows that entropy neurons operate by writing onto an unembedding null space,
allowing them to impact the residual stream norm with minimal direct effect on
the logits themselves. We observe the presence of entropy neurons across a
range of models, up to 7 billion parameters. On the other hand, token frequency
neurons, which we discover and describe here for the first time, boost or
suppress each token's logit proportionally to its log frequency, thereby
shifting the output distribution towards or away from the unigram distribution.
Finally, we present a detailed case study where entropy neurons actively manage
confidence in the setting of induction, i.e. detecting and continuing repeated
subsequences.