Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
Authors: name: Sheridan Feucht, name: David Atkinson, name: Byron Wallace, name: David Bau

LLMs process text as sequences of tokens that roughly correspond to words,
where less common words are represented by multiple tokens. However, individual
tokens are often semantically unrelated to the meanings of the words/concepts
they comprise. For example, Llama-2-7b's tokenizer splits the word
"northeastern" into the tokens ['_n', 'ort', 'he', 'astern'], none of which
correspond to semantically meaningful units like "north" or "east." Similarly,
the overall meanings of named entities like "Neil Young" and multi-word
expressions like "break a leg" cannot be directly inferred from their
constituent tokens. Mechanistically, how do LLMs convert such arbitrary groups
of tokens into useful higher-level representations? In this work, we find that
last token representations of named entities and multi-token words exhibit a
pronounced "erasure" effect, where information about previous and current
tokens is rapidly forgotten in early layers. Using this observation, we propose
a method to "read out" the implicit vocabulary of an autoregressive LLM by
examining differences in token representations across layers, and present
results of this method for Llama-2-7b and Llama-3-8B. To our knowledge, this is
the first attempt to probe the implicit vocabulary of an LLM.