From Tokens to Words: On the Inner Lexicon of LLMs

## Abstract

Abstract Natural language is composed of words, but modern LLMs process sub-words as input. A natural question raised by this discrepancy is whether LLMs encode words internally, and if so how. We present evidence that LLMs engage in an intrinsic detokenization process, where sub-word sequences are combined into coherent word representations. Our experiments show that this process takes place primarily within the early and middle layers of the model. They also show that it is robust to non-morphemic splits, typos and perhaps importantly—to out-of-vocabulary words: when feeding the inner representation of such words to the model as input vectors, it can “understand” them despite never seeing them during training. Our findings suggest that LLMs maintain a latent vocabulary beyond the tokenizer’s scope. These insights provide a practical, finetuning-free application for expanding the vocabulary of pre-trained models. By enabling the addition of new vocabulary words, we reduce input length and inference iterations, which reduces both space and model latency, with little to no loss in model accuracy.

## 1 Introduction

Large language models (LLMs) rely heavily on tokenization methods such as byte-pair encoding (BPE; Sennrich et al., ).
Such methods often split words into multiple tokens,
potentially disrupting their morphological structure (Batsuren et al., ).(^1^11For instance, the word “unhappiness” might be tokenized into “un,” “h,” and “appiness” (see Fig. 1).)
Typos and other perturbations can also lead to large variations in the tokens that represent a word (Kaushal & Mahowald, ).
Nonetheless, LLMs exhibit a remarkable ability to recover word meaning (Cao et al., ). This raises important questions
about how models internally compose meaningful word representations from tokens,
a process referred to as detokenization (Elhage et al., ; Gurnee et al., ).

In this work, we seek to understand the detokenization mechanism in LLMs. We consider two cases: words that are not part of the model BPE vocabulary, and are thus split into multiple sub-words; and single-token, in-vocabulary words that we artificially split into multiple tokens.
In both cases, our experiments indicate a word-level detokenization process in LLMs, which occurs mainly in the early to middle layers.
Our results hint that language models hold a *latent vocabulary* or *inner lexicon*, which they access to identify words from token sequences.(^2^22This process might resemble the mental lexicon in humans (Aitchison, ; Marslen-Wilson et al., ).)

We begin by examining if internal representations of token sequences reflect whether a sequence of tokens comprises a word (Section 3).
We probe the model’s hidden representations (Conneau et al., ) of both multi-token real English words and gibberish *nonwords* (Frisch et al., ). We observe that the representations of words and nonwords substantially diverge in middle layers—a simple k-nearest neighbors classifier achieves a 89% accuracy on the task of discriminating between both groups.
Overall, these results suggest that models hold a concept of recognized words.

We next explore the mechanism through which models reconstruct cohesive word representations from sub-word tokens (Section 4). To do so, we use techniques that interpret a token’s hidden states and decode them into natural language (Belrose et al., ; Ghandeharioun et al., ).
We find that for both multi-token and single-token words, in most cases, the last token can be decoded as the full word after being processed by 3–5 layers,
with some words requiring up to 15 layers.
Interestingly, 23% of the multi-token words are never successfully decoded from their last token’s hidden states, hinting this inner lexicon does not cover all words.

Figure: Figure 1: Left: The sub-word detokenization process in LLMs. From bottom to top: (a) Tokenization and Embedding: The input string is tokenized using a sub-word tokenizer (e.g., BPE) and converted into vector embeddings; (b) Token Aggregation: the attention mechanism is used to merge sub-word tokens into the final sub-word token; (c) Word Retrieval: The model retrieves the full word representation from an internal dictionary in its feedforward (FFN) layers. This representation is added to the residual stream, until it takes over the word’s hidden representation. Right: When taking this hidden representation and patching into another prompt, the model interprets it as the original word. In this example, the model is prompted to repeat the (single vector) hidden representation of the word “unhappiness” (originally represented as three tokens), and is able to “understand” it by regenerating the original three tokens.
Refer to caption: extracted/5916017/figures/illustration_paper.png

We then turn to explore how models assemble these full word representations (Section 5).
We first interpret the feedforward network (FFN) updates in the vocabulary space (Geva et al., ; Todd et al., ). We show that in 85% of the evaluated words, an FFN update promoting the full word’s concept is written to the last token’s residual stream (Geva et al., ; Merullo et al., ). Importantly, this happens in the layers before the word’s representation emerges, indicating that models retrieve the reconstructed word representation from their FFN weights.
Second, we compare attention patterns between standard single-token words to how multi-token words attend to their preceding sub-word tokens. We find that the last token in multi-token words attends significantly *more* to its previous (sub-word) tokens in layers 1–2 compared to single-token words, and significantly *less* in the following layers. This suggests that models initially aggregate information from previous tokens, and later almost ignore them.

Our findings have concrete applications (Section 6); they allow us to expand the model input and output BPE vocabulary with the fused representation of multi-token words found in our experiments. Importantly, this expansion requires no update to model parameters. Applying our approach to multi-token words found in English Wikipedia data (Merity et al., ), we find that the model successfully uses the new vocabulary entries both as inputs and during generation; its language model performance is maintained, and even slightly improves. This demonstrates the potential to dramatically reduce both input and output sequence length, and inference costs accordingly, especially in languages where the ratio of tokens per word is high (Ahia et al., ; Petrov et al., ).

Overall, our results establish that word-level detokenization is a core process in LLMs, and provide evidence of how it unfolds across model layers. Beyond improving our understanding of the internal mechanisms driving LLMs, our work lays the foundation for practical applications, particularly in terms of optimizing token management and reducing computational costs.(^3^33Our code will be released here: [https://github.com/schwartz-lab-NLP/Tokens2Words](URL))

## 2 Related Work

#### Tokenization

Sub-word tokenization algorithms (Wu, ; Kudo, ) are the standard for pre-processing text in modern LLMs. The most widely used method is byte pair encoding (BPE; Sennrich et al., ), which keeps frequent words intact and splits rare words into multiple sub-words.
Recent studies propose ways to improve tokenization to ease word reconstruction (Provilkov et al., ; Hofmann et al., ; Yehezkel & Pinter, ; Bauwens & Delobelle, ) or analyze how tokenization affects model performance (Bostrom & Durrett, ; Church, ; Klein & Tsarfaty, ; Zouhar et al., ; Schmidt et al., ). To the best of our knowledge, we are the first to thoroughly investigate how LLMs internally reconstruct word representations.

#### Detokenization and stages of inference

Recent studies suggested that early LLM layers integrate local context to map raw token embeddings into coherent concepts or entities, a process called *detokenization*, but their observations were derived from specific case studies (Elhage et al., ; Lad et al., ). Prior work probed transformer models to explore how predictions are constructed over the course of the model, showing early layers provide local syntactic information (Tenney et al., ; Vulić et al., ; Durrani et al., ).
Ferrando & Voita () analyzed token attributions in Llama2 (Touvron et al., ), and observed attention heads that promote sub-word merging.
Our work focuses on word-level detokenization, and goes beyond previous efforts to provide an in-depth analysis of how word representations are assembled from multiple tokens.

#### Interpreting the residual stream

Recent methods for interpreting the intermediate states of LLMs draw on a *residual stream* perspective of transformer architectures (Vaswani et al., ): the hidden state acts as an information stream along the layers, from which information is read at each layer, and new information is added through residual connections (Elhage et al., ). Thus, hidden states in any layer can be projected into the model’s vocabulary space, treating the hidden state as if it were the output of the last layer (nostalgebraist, ; Dar et al., ; Belrose et al., ; Yom Din et al., ).
Similarly, Ghandeharioun et al. () proposed to decode information from hidden representations into natural language, by patching (Zhang & Nanda, ) it into a prompt that encourages the model to *verbalize* the encoded information.
We use both approaches to inspect how token representations evolve across layers.

#### LLM memories

Our hypothesis that models maintain an *inner lexicon* is motivated by recent work showing feedforward networks (FFN) layers in transformers act as key-value memories that encode factual and linguistic knowledge (Geva et al., ; Meng et al., ; Dai et al., ). Particularly, FFN layers were shown to enrich entity tokens with associated information (Meng et al., ; Geva et al., ) and promote relevant concepts in vocabulary space to build up predictions (Geva et al., ), similar to our finding that word representations are pulled from FFN layers before emerging in the hidden state of the word’s last token. Concurrently to our work, Feucht et al. () showed that models tend to “forget” the contents of early tokens in the last token representations of multi-token words, but not in single-token words. They also hypothesize the existence of an implicit vocabulary in LLMs, but do not further analyze the detokenization process. Combined with our results, their findings suggest that full word information retrieved from FFN layers “deletes” the existing representation rather than enriching it.

## 3 A Motivating Observation: LLMs Can Tell Words from Non-Words

One of our key hypotheses in this work is that LLMs hold an internal lexicon of words, which is different from the BPE lexicon.
As a result, we begin by asking whether LLMs, when processing a sequence of tokens, capture some notion of whether or not this sequence forms a word.

To address this question, we construct a balanced dataset containing two groups: one with real English words, and another with artificially generated, meaningless *nonwords* (Frisch et al., ). Both groups are tokenized using the Llama2 tokenizer (Touvron et al., ).
The word dataset consists of 10,000 distinct words sampled from the Gutenberg corpus (Gerlach & Font-Clos, ), with 53% of the words containing two tokens, 37.3% containing three tokens, and the rest four tokens.
We generate the nonwords by shuffling tokens from the word dataset, ensuring that the prefix and suffix positions align with the original tokens’ positional probabilities.
This process ensures that nonwords tokens maintain position distribution properties similar to word tokens (Fig. 2(a)).
We next fit a $k$-nearest neighbors ($k$-NN) classifier ($k=4$, using Euclidean distance) on the hidden states of the last tokens of both words and nonwords, for each layer of the Llama2-7B model.
The training set consists of 80% of the dataset, and the remaining 20% are used for evaluation.

Figure: (a) nonwords creation process
Refer to caption: extracted/5916017/figures/dataset_creation_2_levels.png

Fig. 2(b) shows our probing results.
Our results reveal a three-stage pattern in the model’s representation of word and nonword token sequences.
In the model’s first few layers, representations from both groups are relatively indistinguishable and performance is close to chance level. Then, from layers 2 to 6, a clear distinction between the two groups emerges, until the representations remain almost completely distinct in middle layers, between layers 6 and 20. At this point, the probe achieves a stable, high accuracy, peaking at 89% on layer 13.
Finally, performance slightly drops after layer 20.
Importantly, word and nonword representations are much more similar in the representations of earlier tokens, as the probe only reaches 61% classification accuracy. We provide further details in Appendix A, as well as an analysis of wrongly classified words and nonwords.

Our results indicate that language models internally represent words and nonwords differently. This distinction is gradually developed in the model’s early layers and maintained throughout its middle layers.
These findings support the hypothesis that the model performs a detokenization process and suggests where this process occurs.
Building on these results, we next investigate how sub-word tokens are combined into word representations across model layers, and explore the internal mechanisms that facilitate this transformation.

## 4 Extracting Word Identity from LLM Hidden States

We have so far observed that LLMs can differentiate between words and nonwords, suggesting an internal detokenization process specific to word composition.
We next dig into this process, by asking whether we can directly extract word identity from the hidden states of sub-word tokens. We start by considering single-token words (Section 4.1), and then move on to multi-token words (Section 4.2).

### 4.1 Single-Token Words

We first consider in-vocabulary words, which are mapped to single tokens by the tokenizer. Naively, such words don’t tell us much about detokenization, as they are represented using only one token. To address this, we artificially split them into multiple sub-word tokens. E.g., we take the single-token word “cats” and split it into two tokens: “ca” and ”ts”. We hypothesize that if the model performs detokenization, it will represent the last token of the word (“ts”) similarly to the original word token (“cats”). We iterate the WikiText-103 dataset (Merity et al., ) and randomly split each single-token word longer than three characters into 2–5 sub-words tokens. We then feed the model with the new sequence of tokens.

To measure the similarity between the representation of the final token and original word, we apply the logit lens method (nostalgebraist, ), an interpretability method that maps a hidden representation of a given token to the word whose vector in the output unembedding matrix is most closely aligned with the hidden representation. We note that this technique is typically used to inspect model predictions of the *next* token in intermediate layers. As we aim to identify how LLMs represent the *current* word, we use the input embeddings matrix instead.

We use four LLMs: Llama2-7B, Llama3-8B (Dubey et al., ), Mistral-7B (Jiang et al., ), and Yi-6B (AI et al., ). We report for each layer the rate of retrieval—the proportion of words for which the closest vector in the BPE space is the original (single token) word.

Our results for the Llama2-7B are shown in Fig. 3(a).(^4^44Results for the other models show a similar trend, see Appendix C.) The results show that starting layer at 8, the hidden state of the final token is mapped to the original word with high accuracy, which peaks at more than 80% in layer 15.(^5^55Further, we find that 93.2% of all split words are correctly mapped to the original word in at least one model layer (see Appendix D).) Interestingly, afterwards this accuracy starts to decline, possibly because the model transitions into representing the prediction of the next token.

Our results indicate that LLMs perform a detokenization process in cases where a single-token word is split into multiple tokens. This process assigns the final token with a hidden representation similar to that of the full, single-token word. But how robust is this process? To address this question,
we consider a different way of splitting a word into sub-tokens—by adding typos such as flipping two adjacent characters, deleting a character, and inserting a new character—see Appendix E for more details. Much like our previous experiment, this process also splits the word into multiple tokens, as corrupted words are very rarely found in BPE lexicons. We repeat the same experiment, aiming to study whether the model maps the last token to the original (correctly-spelled) word.

Our results for Llama2-7B (Fig. 3(a)) show a similar trend, though less pronounced: the model correctly matches about 40% of the corrupted words with the original, single-token, uncorrepted word.(^6^66Here, 66% of the words are correctly mapped in at least one layer.) We also note that the trend across layers persists, with accuracy peaking around layer 15.

Figure: (a) Rate of retrieval for single-token words
Refer to caption: extracted/5916017/figures/typos_separations_llama2.png

### 4.2 Multi-Token Words

We have seen that LLMs are able to detokenize single-token words that are artificially broken into sub-words into their correct (single-token) word. This indicates that these words are part of the model’s inner lexicon. But what about multi-token words? By definition, these words do not have a single embedding vector in the BPE vocabulary, so we cannot compare them to any existing vector, and thus cannot use logit lens.

Instead, we use the Patchscopes technique (Ghandeharioun et al., ), which interprets the hidden representation of a model using the model’s language abilities.
In particular, we feed the model with the prompt “Repeat this word twice: 1) $X$ 2)”, where $X$ is the hidden representation of the last word token. Our hypothesis is that if the multi-token word is found in the model internal lexicon, than it will “understand” its representation in the input layer as well, and successfully repeat it by generating all tokens of the original word (see Fig. 1, right).

We evaluate the model by computing the rate of retrieval—the proportion of times it repeats the correct (multi-token) word.
We use the same models and dataset as in the previous experiments, though in this case we do not further split or add typos, as the words are already split into multiple tokens. Particularly, we feed each multi-token word (without any context) to the model, and extract the last token’s representation from each layer. We then use Patchscopes’ prompt in a new run to check if the model regenerates the original multi-token word.

Our Llama2-7B results (Fig. 3(b)) show a striking trend: when feeding the model with vectors from layers 5–7, it is able to repeat the word in 64% of the cases, despite never seeing this vector as input before.(^7^7777.4% of the words are repeated correctly in any layer.) As to the different layers, we observe a similar trend to the single-token results: retrieval rates increase rapidly at some point (here earlier than in previous experiments, around layer 3-4), reach a peak at layer 7, and then start to decline.
Combined, these results suggest that the model
treats multi-tokens words as if they were in the vocabulary but split into sub-word tokens, indicating a latent vocabulary that expands beyond the tokenizer’s limitations. Interestingly, we observe that 22.6% of the multi-token words are not successfully decoded from any internal layer, hinting that they might not be represented in the model’s inner lexicon.

## 5 How Does Detokenization Happen?

We have so far shown that LLMs perform a detokenization step: at some point, the hidden representation of the final token of a given word becomes strikingly similar to the (single-word) vector of that word. This process is robust to artificial splits of single token words, to splits due to typos, and even to multi-token words, which the model can still recognize at the input layer, despite never having seen them there during training.

We next turn to ask how does the model reconstructs full word
representations from sub-word tokens?
We aim to understand the dynamics of this process by analyzing the main transformer components—feedforward network layers (FFN) and the attention mechanism.

### 5.1 Word Retrieval using the Feedforward Mechanism

FFN layers have been shown to serve as key-value memories for storing memory (Geva et al., ; ; Burns et al., ). We hypothesize that this memory might be used to store the inner lexicon as well.
Particularly, we suggest that the model uses the FFNs to refine the representation of the final token, enabling it to retrieve the original word.

To test this hypothesis, we repeat the typos experiment of single-token words (Section 4.1),
but this time applying logit lens to the output of the FFN layers, instead of the hidden representation. Figure 4(a) shows the retrieval rate of the original word.
We observe that the retrieval of the full word concepts from the FFN layers appears a few layers earlier compared to the hidden state, which indicates that they help refine it. Nonetheless, the rate of retrieval appears to be lower in FFNs, which suggests that other factors might come into play in building the hidden representation of words. However, in Fig. 4(b) we plot the cumulative retrieval rate, i.e., for each layer, whether the word was identified in any layer so far. Our results indicate a different conclusion:
the FFN update vectors match the (single-token) input representation of the word 70% of the time in layer, particularly both earlier and more frequently compared to the hidden representation. This suggest a larger variance in the exact layer in which FFNs identify the word compared to the hidden state.
Our results indicate that the FFNs play a pivotal role in
reconstructing word-level information from sub-word tokens.

### 5.2 Token Aggregation

Our results hint that LLMs retrieve words form the FFN layers, which allows the final token to represent these words. But this role of the FFN layers only starts to emerge in layers 3–4. What happens earlier?
We rely on previous work that showed that in the early
layers the model focuses on integrating information from nearby tokens to compose entities (Lad et al., ), and hypothesize that the model starts by aggregating information from the previous sub-word tokens.

Figure: (a) Word retrieval per layer
Refer to caption: extracted/5916017/figures/Llama2_ffn_hidden_state_by_layer.png

To test this, we take all 2-token words in a subset of 1,000 documents from
WikiText-103 (a total of 5,571 words), and feed them (along with their context) to the Llama2-7B model. We then measure the average attention weights of the final token to the prefix tokens in each layer.
As control, we measure the average attention weights assigned by single-token words to their preceding token.

Figure: Figure 5: Attention weights for 2-token words: Early peaks (layers 2–3) show high attention values from the 2nd sub-word token to the first, but these values decline rapidly. Attention from single-token words to their previous token shows a similar trend, though with substantially lower values at first, which become higher later.
Refer to caption: extracted/5916017/figures/tokens_aggregation_2_tokens.png

Our results (Fig. 5) support previous findings (Lad et al., )—the attention to previous tokens is high
in the first 2–3 layers, but then declines sharply (up to $\sim$90% decline).(^8^88Experiments with 3- and 4-token words show a similar trend, see Appendix B.)
Considering the attention from single-word tokens to their previous token, we also observe this peak phenomenon, but importantly—here the initial peak is far lower than in our multi-token words(^9^99The diverging patterns between single and multi-token words are statistically significant; see Appendix B.). However, we also note that the following values are in turn higher. These results suggest that LLM strongly attend to their sub-word tokens at first, but then largely ignore them.

Our results suggest that the model performs a detokenization process
by first aggregating information from the prefix tokens into the final token’s
hidden representation. It then refines the representation
of the final token in the FFN layers to retrieve the full word. This
two-stage process of token aggregation and concept retrieval provides
insight into the internal mechanisms of LLMs and how they handle sub-word
tokens to reconstruct word-level representations.

## 6 Expanding LLM Vocabulary without Finetuning

We have shown that language models internally fuse multi-token words into single-token representations, which models can decode to generate the complete word. This raises the question whether models can make more general use of such fused representations when processing textual input, by using them in place of the original, multi-token embeddings. Conversely, models were shown to foreshadow several future tokens in a single hidden state without being explicitly trained to do so (Pal et al., ); can we leverage the fused representations to enable models to generate multi-token words in a *single* inference step?

We next explore whether such vocabulary expansion is possible without any updates to the model’s parameters.(^10^1010We do add new entries to the model’s embedding and unembedding matrices, but these are inherently required to expand the vocabulary and represent the new vocabulary words.)
The potential of such methods is high, as even tokenizers intentionally trained for multilingual support result in extremely longer tokenization lengths for non-English languages (up to 13x for some languages), affecting cost and latency (Ahia et al., ; Sengupta et al., ; Petrov et al., ).
Still, previous attempts to expand tokenizer vocabulary are limited in number, and require further training on large corpora (Kim et al., ; Zhao et al., ) or attempt to force alignment between languages in embedding space (Wang et al., ), rather than rely on the internal detokenization mechanism of LLMs.

We first experiment with expanding the input embeddings matrix $E$, leaving the output unembedding matrix $U$ unchanged. To extract fused representations, we begin with the same approach as in Section 4.2: Given a multi-token word, we run it through the model using a prompt template $P$.(^11^1111We use the template “x x x x” where x is the new vocabulary word; preliminary experiments showed it provides better results than the template “x”, possibly due to the effect a token’s position has on its representation.)
We then identify the first layer in which the last token is successfully decoded into the full word (Ghandeharioun et al., ). We extract the hidden state of the last token in that layer, and add it as the word’s fused representation to $E$. If the word is never successfully decoded from the representation in any of the layers, we don’t attempt to add it to the vocabulary, as we assume it is not in the model’s inner lexicon.
We experiment with two approaches: identity, where we use the fused representation as is; and linear, where we first transform it with a linear map $T_{\ell,E}$.
We learn $T_{\ell,E}$ by fitting a linear regression from the layer $\ell$ hidden states of all single-token words when passed with the template $P$, to their corresponding input embeddings. Importantly, $T_{\ell,E}$ is both not part of the original model, and does not depend on the new, originally multi-token words we add to the vocabulary.

**Table 1: Performance of finetuning-free vocabulary expansion for Llama2-7B on WikiText-103. Different rows correspond to baselines (first two rows) and proposed approaches (last three). Methods with $E$ add new multi-token words as single-token entries in the embedding matrix, while those with $U$ add new entries to the unembedding matrix. Column groups correspond to evaluations of language model performance on different token groups: those already in the vocabulary (left) and the newly added entries (right, relevant only when expanding $U$). Results marked with $\ast$ are for predicting the 1st token of a multi-token word (rather than predicting the new entry of the complete word). Our best approach (Fused-Linear) leverages the fused representations emerging within models after detokenization, to add new vocabulary entries for multi-token words. We show it maintains performance on the original vocabulary, and enables the model to successfully utilize entries in $U$ that were never observed during training.**
|  | Original vocabulary tokens | Newly added words |  |  |
| --- | --- | --- | --- | --- |
|  | Top-1 Acc. | Mean Reciprocal Rank | Top-1 Acc. | Mean Reciprocal Rank |
| Original vocabulary | 57.7% | 0.679 | 47.5%^∗ | 0.556^∗ |
| Last token embedding ($E$) | 54.5% | 0.647 | – | – |
| Fused-Identity ($E$) | 53.5% | 0.638 | – | – |
| Fused-Linear ($E$) | 56.7% | 0.669 | – | – |
| Fused-Linear ($E$ + $U$) | 56.7% | 0.667 | 19.5% | 0.598 |

We apply our approach to expand the vocabulary of Llama2-7B with all multi-token words found in the WikiText-103 test set, resulting in 4.3k newly added words. To examine whether the model can correctly use the new entries, we evaluate it in a language modeling setup, when all newly added words are replaced with their fused single-token embeddings, similar to our setup in Section 4.2 (see righthand side in Fig. 1).
We compare our approach to two baselines: using the original vocabulary without any new entries; and using the unprocessed last token embedding of each new word as its new vocabulary entry (instead of the fused representations).
For evaluation, we cannot use perplexity, the standard LM measure, as the different approaches have different vocabularies, which influences perplexity scores and prevents a fair comparison. Instead, we measure the token-level top-1 accuracy and the mean reciprocal rank of the ground truth tokens; naturally, we do not report the ranking of sub-word tokens forming the newly added words, as they only serve as ground truth in our baseline.

We report our results in Table 1.
Examining model performance when using the fused representations in the identity approach, we find that while the model can successfully decode the full word information when explicitly prompted to do so, it fails to utilize the fused representation for inference, and achieves slightly worse performance than when using the unprocessed last token embeddings.
However, using the linear approach by first applying $T_{\ell,E}$ transforms the fused representation into a usable form for initial layers, as we find the model can then utilize the newly added words and maintains its performance.(^12^1212A possible explanation for the importance of $T_{\ell,E}$ is that input embeddings have very different distributions from intermediate hidden states, as LLMs are extremely sensitive to 1st layer ablations (Lad et al., ).)

Given the model’s ability to use fused representations as inputs, we next enable the model to generate multi-token words in a single inference step, by adding new entries to the unembedding matrix $U$ derived from their fused representations. In light of our previous results, we only experiment with the linear approach. We follow the same steps as before, this time applying linear maps $T_{\ell,U}$ which we learn by fitting linear regressions from the layer $\ell$ hidden states of all existing single-token words, to their corresponding rows in the unembedding matrix.
We evaluate whether the model chooses to generate the new vocabulary entries, or rather the first token of the original multi-token words (as it does when using the unedited vocabulary).

Surprisingly, we find the model assigns, on average, higher rank to the new vocabulary entries over their original 1st sub-word tokens in the appropriate contexts (Table 1, last row). While the top-1 accuracy is higher for the 1st sub-word tokens in the unedited vocabulary, we find that when new vocabulary entries are not assigned high ranks, the original sub-word will often be predicted. We further observe that in 7.1% of examples, the new entry achieves the top-1 rank even though the original sub-word in the baseline does not.
Notably, this also does not compromise performance on existing vocabulary words. Our results suggest that the model can sometimes try to generate the fused representation, but as it does not exist in its (original) unembedding vocabulary, the first sub-word token is selected instead.
Overall, our findings show potential to substantially decrease inference times for LLMs. In future work, we will explore their application to multilingual LLMs (Petrov et al., ).

## 7 Conclusion

The ability of LLMs to comprehend and generate language relies on intricate internal processes, and understanding these mechanisms is crucial for improving model performance as well as making them more efficient. In this work, we unraveled the word detokenization process, shedding light on how models internally transform fragmented sub-word tokens into coherent word representations formed at the last token.
Our results indicate that this mechanism manifests in early to middle layers, where models attempt to reconstruct words by mapping them to an inner lexicon using their FFN layers. We provided evidence this lexicon is more exhaustive than the tokenizer’s vocabulary, and could facilitate the model’s ability to recognize words even amidst noise.

Our work also unlocks practical avenues for optimizing tokenization, as well as the speed and cost of inference. We demonstrated one such application and presented a finetuning-free method to expand the vocabulary of LLMs. We hope our work paves the way for more efficient and versatile models.

## Acknowledgements

This work was supported in part by the Israel Science Foundation (grant no. 2045/21) and by NSF-BSF grant 2020793. In addition, we thank Iddo Yosha for his valuable feedback.

## Appendix A Words vs. Nonwords Analysis

To further analyze results from Section 3, we conduct a failure analysis,
focusing on false negatives (FN) and false positives (FP).
The former, valid words misclassified as gibberish, are
often rare or complex words, suggesting that the model’s internal
vocabulary may lack representations for less common words.
On the other hand, false positives—where gibberish is misclassified as a
word—typically involved nonwords that closely resemble valid words,
likely due to shared sub-word structures.
See Table 2 for a few examples.

Figure: (a) accuracy postfix and last token
Refer to caption: extracted/5916017/figures/words_nonwords_prefix_vs_postfix.png

**Table 2: Examples of false negative (FN) and false positive (FP) for the word vs. nonword experiment (Section 3).**
| Original Word | Predicted Value | True Value | Status |
| --- | --- | --- | --- |
| _Unitarianism | gibberish | word | FN |
| _killy | gibberish | word | FN |
| _quadruropic | word | gibberish | FP |
| _nonwith | word | gibberish | FP |

## Appendix B Statistical Analysis of Token Aggregation in Multi-token Words

We repeat the multi-token experiments Section 5.2 for 3- and 4-token words. Results Fig. 7 show a very similar trend to the 2-token words (Fig. 5).

We present the detailed statistical analysis conducted of our experiments in Section 5.2 to
examine token aggregation in multi-token words
compared to single-token words (control group). The objective
is to investigate whether the model exhibits significantly
different attention patterns between these groups, indicative
of the detokenization process.

**Table 3: Results of one-sided t-tests comparing attention weights between multi-token and single-token words across layers over Llama2-7B model.**
| Layer | p-value (Multi $>$ Single) | p-value (Single $>$ Multi) | Significance |
| --- | --- | --- | --- |
| 0 | $9.999\times 10^{-1}$ | $5.891\times 10^{-5}$ | Single-token $>$ Multi-token (***) |
| 1 | $0.000$ | $1.000$ | Multi-token $>$ Single-token (***) |
| 2 | $0.000$ | $1.000$ | Multi-token $>$ Single-token (***) |
| 3 | $1.000$ | $1.252\times 10^{-273}$ | Single-token $>$ Multi-token (***) |
| 4 | $1.000$ | $0.000$ | Single-token $>$ Multi-token (***) |
| 5 | $1.000$ | $4.466\times 10^{-242}$ | Single-token $>$ Multi-token (***) |
| 6 | $1.000$ | $0.000$ | Single-token $>$ Multi-token (***) |
| 7 | $1.000$ | $0.000$ | Single-token $>$ Multi-token (***) |
| 8 | $1.000$ | $0.000$ | Single-token $>$ Multi-token (***) |
| 9 | $1.000$ | $0.000$ | Single-token $>$ Multi-token (***) |
| 10 | $1.000$ | $0.000$ | Single-token $>$ Multi-token (***) |
| 11 | $1.000$ | $7.847\times 10^{-87}$ | Single-token $>$ Multi-token (***) |
| 12 | $1.000$ | $6.307\times 10^{-303}$ | Single-token $>$ Multi-token (***) |
| 13 | $1.000$ | $1.417\times 10^{-255}$ | Single-token $>$ Multi-token (***) |
| 14 | $1.000$ | $2.397\times 10^{-172}$ | Single-token $>$ Multi-token (***) |
| 15 | $1.000$ | $0.000$ | Single-token $>$ Multi-token (***) |
| 16 | $1.000$ | $5.857\times 10^{-25}$ | Single-token $>$ Multi-token (***) |
| 17 | $1.000$ | $1.414\times 10^{-152}$ | Single-token $>$ Multi-token (***) |
| 18 | $5.646\times 10^{-5}$ | $9.999\times 10^{-1}$ | Multi-token $>$ Single-token (***) |
| 19 | $1.000$ | $7.919\times 10^{-161}$ | Single-token $>$ Multi-token (***) |
| 20 | $1.000$ | $2.036\times 10^{-286}$ | Single-token $>$ Multi-token (***) |
| 21 | $1.399\times 10^{-190}$ | $1.000$ | Multi-token $>$ Single-token (***) |
| 22 | $1.000$ | $3.647\times 10^{-44}$ | Single-token $>$ Multi-token (***) |
| 23 | $1.000$ | $6.008\times 10^{-13}$ | Single-token $>$ Multi-token (***) |
| 24 | $1.000$ | $1.062\times 10^{-15}$ | Single-token $>$ Multi-token (***) |
| 25 | $5.274\times 10^{-68}$ | $1.000$ | Multi-token $>$ Single-token (***) |
| 26 | $1.000$ | $3.670\times 10^{-12}$ | Single-token $>$ Multi-token (***) |
| 27 | $0.000$ | $1.000$ | Multi-token $>$ Single-token (***) |
| 28 | $1.000$ | $2.446\times 10^{-10}$ | Single-token $>$ Multi-token (***) |
| 29 | $1.438\times 10^{-202}$ | $1.000$ | Multi-token $>$ Single-token (***) |
| 30 | $9.506\times 10^{-99}$ | $1.000$ | Multi-token $>$ Single-token (***) |
| 31 | $6.731\times 10^{-1}$ | $3.270\times 10^{-1}$ | Not significant (ns) |

To determine significance, we performed two one-sided t-tests
per layer for the Llama2-7B case:
(1) Testing whether attention to prefix tokens in multi-token words is higher than to previous tokens in single-token words.
(2) Testing whether attention to previous tokens in single-token words is higher than to prefix tokens in multi-token words.

Table 3 shows the p-values and significance levels for each layer. Significance levels are denoted as ns (not significant), and *** ($p<0.001$).

Figure: (a) Llama2-7B Attention weights for 3-tokens words
Refer to caption: extracted/5916017/figures/tokens_aggregation_3_tokens.png

The results reveal that in layers 1 and 2, attention to prefix
tokens in multi-token words is significantly higher than in single-token words,
suggesting the early phase of Token Aggregation.
From Layers 3 to 17, attention to single-token words is higher, indicating a shift in
attention focus from prefix attention in relation regular close tokens.
Notably, there are intermittent increases in attention to prefix tokens at Layers
18, 21, 25, 27, 29, and 30, possibly signaling a transfer into a prediction ensembling
phase in which the attention in general is less important (Artzy & Schwartz, )
and therefore we don’t see a coherent
pattern.

In conclusion, the attention mechanisms differ significantly between multi-token and single-token words,
confirming that the detokenization process involves initial amplification of attention to
sub-word tokens followed by a reduction as the model processes word representations.

## Appendix C Word Retrieval for Single-Token and Multi-Token Words Across All Models Comparison

This section presents a detailed comparison of word retrieval performance (Section 4) across several models (Llama2-7B, Llama3-8B, Yi-6B, and Mistral-7B) for both single-token and multi-token words. The evaluation focuses on how effectively each model retrieves the original word across layers, especially in challenging cases like artificially separated single-token words, typos, and multi-token words.

Figure: Figure 8: Layer-wise word retrieval rates for single-token and multi-token words across all models.
Refer to caption: extracted/5916017/figures/word_retrieval_all_experimants.png

Across all experiments, we observe a similar trend where the retrieval performance increases over the first several layers, peaks around the middle layers, and then decreases in the later layers. The main difference across models lies in the peak performance, especially in cases involving typos and multi-token words, where more advanced models such as Llama3-8B and Mistral-7B demonstrate superior performance in reconstructing the original word representations.

## Appendix D Cumulative Word Retrieval for Single-Token and Multi-Token Words

In this section, we present the results of cumulative word retrieval (Section 4.1) across various models, focusing on both single-token and multi-token words. For each model, we analyze the ability of the LLM to retrieve the original word from sub-word tokens across its layers. The cumulative retrieval is calculated as the proportion of words that are successfully retrieved at each layer, with the percentage increasing as more words are recovered throughout the model’s layers.

Single-Token Words:
The cumulative word retrieval for single-token words—those that are artificially split into sub-word tokens (via typographical errors or manual splits)—shows a rapid increase in retrieval success in the early layers. For Llama2-7B, for instance, cumulative retrieval reaches 93.2% for words split by manual intervention, and 66% for words affected by typos by the middle layers. This pattern is observed across models, with retrieval generally peaking around layers 15-20.

Multi-Token Words:
For multi-token words, which are naturally split due to being out-of-vocabulary for the tokenizer, the cumulative retrieval process follows a similar trajectory. However, in models like Llama2-7B, the retrieval peaks earlier in the model, with a cumulative retrieval rate of 77.41%. Other models like Llama3 and Yi show higher cumulative retrieval rates, suggesting improved efficiency in handling multi-token words, potentially due to larger model capacities and internal dictionaries.

The similarity in cumulative retrieval between single-token and multi-token words suggests that LLMs treat out-of-vocabulary words in a manner similar to sub-word-tokenized words, accessing a latent vocabulary to reconstruct full word representations.

Figure: Figure 9: Cumulative word retrieval for single-token and multi-token words across all models.
Refer to caption: extracted/5916017/figures/cumulative_all_experiments.png

## Appendix E Introducing Typos for Single-Token Words

In this section, we describe the process of introducing typos into single-token words to split them into multiple tokens (Section 4.1). The modification applies to words longer than four characters and involves randomly performing one of three operations: substituting two characters, deleting a character, or inserting a new character. By introducing these slight variations, the word becomes unfamiliar to the tokenizer, causing it to be divided into multiple smaller tokens during tokenization. Particularly, this process results in splitting words into 2–5 tokens. Table 4 shows examples of the different splits.

**Table 4: Examples of the different typos we consider, exemplified by perturbing the single-token word “development” (Section 4.1).**
| Description | perturbed | . new tokens |
| --- | --- | --- |
| Substitution of two characters | develpoment | [‘de’, ‘vel’, ‘p’, ‘oment’] |
| Deletion of one character | develoment | [‘de’, ‘vel’, ‘oment’] |
| Insertion of one character | develfopment | [‘dev’, ‘elf’, ‘op’, ‘ment’] |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: If the model wants to generate some multitoken word that it represents in its 'internal dictionary' is it "planning" multiple tokens ahead? Why or why not?
soundness: 4
strengths: This paper answers particular unanswered questions surrounding "detokenization", which has been repeatedly observed and discussed without being properly studied. These are important for observations around, for example, stages of inference in language models.

Interpretability results on early layers of LMs are often lacking, as vocab projections are much easier to perform at later layers. This work provides interesting and convincing results for one role early layers take on in these models, which is indeed different from the roles of later layers.

The vocab expansion experiments are a nice proof of concept, and could be expanded on in the future to decrease inference times.

The results on typos are interesting and to my knowledge, novel
summary: This paper explores the process in which models transform tokens, which often split long words into subwords (e.g., "un" "h" "appiness"), into higher level representations for the full word through "detokenization". Detokenization has been observed in LMs before, but has not been directly studied extensively. This work shows that LMs can recognize when a word is part of a larger word, and show that early attention fuses subwords together (in the last token of the word), and uses early MLP layers to then recall the full word from multiple subwords in an "internal dictionary" (e.g., representing "unhappiness" as a single vector internally even though it is not in the tokenizer). The authors then show that this can be used to expand a model's tokenizer by including the hidden 'internal dictionary' representation as an input token. This works to some extent.

Overall, this paper enhances our understanding of early layer processing in language models, and provides a path towards enhancing models to reduce inference time.
weaknesses: The evidence for a third stage of processing in Figure 2b is a little sparse. These results are only for one model, and the degree to which accuracy drops is not substantial enough to obviously be due to a difference in processing altogether. These results could be made stronger by including results for more models. As a motivating example, it is fine, but perhaps isn't the best use of that space if this point can't be made more strongly.

Typos:

L370: "form"

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. How were tokens assembled into nonwords in sec 3? I am missing detail here which could be useful in understanding the method. I also do not understand what it means to "fit" a KNN classifier (which is non-parametric) -- were there representations used which were different from those taken from the model hidden states?
2. There was a claim made that the proposed method in section 6 can improve inference-time costs, though I cannot find any experiments or numbers for this in the paper. Can the authors point me to or provide any information about this? Thank you.
soundness: 3
strengths: 1. This paper analyzes the process of detokenization across transformer network layers via a series of targeted experiments. It builds an intuitive understanding that agrees with many prior works in layer-based analysis.

2. The paper proposes an interesting method for training-free expansion of the model vocabulary by leveraging the insights into internal word representations. This method is shown to be effective in limited experiments. See below in "weaknesses" for further thoughts on this.

3. The writing is clear, but sometimes too abstract (see weakness 5).

This paper shows very solid work and I greatly appreciate the thorough breadth of exploration, though it could possibly be more effective to focus on fewer areas. I want to emphasize that I enjoyed reading the paper and believe it will be strong after some revision, including reworking the claims and focusing more on the novel contributions which are begun later in the paper. I believe it would be more impactful to explore sec 6 in more depth; see weakness 4 below.
summary: This paper analyzes the process of latent detokenization inside the transformer-based LM forward pass, as it occurs across network layers. It shows that models are able to recognize words from pretraining even when they are noised with slight token variations or artificially split across multiple tokens. These experiments are different from earlier works, but ultimately show very similar findings about hierarchical processing in transformers. Using these findings, a novel method is briefly introduced to leverage the internal states of merged tokens to automatically expand the token vocabulary, which can hypothetically improve inference costs with fewer lookups. This method appears initially effective, but could be explored more.
weaknesses: 1. The concept of an inner lexicon is interesting, but not novel as is claimed in this work. The idea follows implicitly from prior work in the memorization of training data, and explicitly in works about tokenization, such as the introduction of BPE (which is discussed greatly in this paper). It is the stated goal of subword tokenizers to enable learning a vocabulary of words and concepts which is larger than the vocabulary of concrete tokens through the process of token combination. It is nice to see these findings reproduced and analyzed, but they are not new.

2. The experiment in section 3, which motivates the idea of an inner lexicon, is not very strongly designed. Why are nonwords created by randomizing tokens, and not by some other method on the morphological level or otherwise something more linguistically motivated? Resulting nonwords do not seem to follow English conventional morphology (eg. the nonword "chha") and this could make it trivial to distinguish words from nonwords. Prior work has shown LLM sensitivity to word frequency in training corpora, and this experiment seems to reproduce those findings. This experiment seems to me to show that LLMs can distinguish easy cases such as "chha" which are very dissimilar to real words, and predictably struggles with more difficult cases that more closely resemble real words (see appendix) but there doesn't seem to be strong evidence that the LLM representation is doing more than locating words on a gradient based on their prior likelihood of appearing in the pretraining corpus. This fact is fairly well established at this point.

3. The experiments in the paper seem mostly sound and reasonable, but their novelty is overstated. Several of the earlier experiments in particular build on each other to show that early and intermediate layers in the network are responsible for aggregating and disambiguating word representations (sec 4 and 5). However, these findings may be seen to be subsumed by many prior works in the analysis of syntactic and semantic composition of tokens across transformer layers (see section 4 in [1] for many citations).

4. The paper may have been too ambitious in scope. The first several experiments were good reproductions of findings. The last experiment was novel to me, and it would have been interesting to expand on it more deeply. However, it did not require many of the earlier experiments in order to understand it, which took up most of the room in the paper. Other reviewers may have different opinions, but mine is that the paper would be more valuable if it explored the final research question more deeply, and provided more concrete findings for it. For example, can we estimate a size/contents of the inner lexicon? Does this lexicon scale with model capacity and/or training size? Can we provide some guarantees or estimates about the boundaries of the method of finetuning-free vocabulary expansion? For what kinds of words is this method effective and when is it ineffective?

5. There were many smaller experiments given in the paper, and this resulted in important implementation details being often omitted. For example, experiments often hinge on model memory of tokens from training, and the natural distributions of those tokens in the corpora, but details about how words/tokens were sampled in the tests (such as construction of nonwords) were not often given in enough detail to reproduce experiments. I would expect there to be significant influence of such distributions on test outcomes, so these details are important.


[1] Anna Rogers, Olga Kovaleva, Anna Rumshisky; A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 2020.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: * The experiments in 4.1 focus on logit lens. What about cosine similarity or more direct measrues of similarity?
soundness: 3
strengths: This paper addresses a crucial question: how can language models construct symbolic representations of entire words when their input comes from tokenizers that often fragment words in ways that disregard their morphological structure? Specifically, the authors investigate whether LMs internally form representations of morphological units that help bridge the gap between the tokenized input and the naturally holistic nature of words in language. Through experiments, the paper presents some evidence that whole-word representations emerge within the model’s hidden states, even when it processes fragmented word tokens. Additionally, the writing is clear, and the experiments are easy to replicate.
summary: In this paper, the authors investigate how LMs internally reconstruct word-level representations from sub-word tokens, a process they term "detokenization". They provide evidence that Lms can inherently combine sub-words into hidden representations which can be mapped into coherent words, even for out-of-vocabulary items, across the early to middle model layers. By probing LMs on both known words and artificial nonwords, they show that the model forms distinct representations for these categories, suggesting an "inner lexicon" that extends beyond tokenized inputs. The findings reveal that this detokenization mechanism leverages feedforward layers and attention patterns to generate whole word representations, which could, in theory, improve vocabulary flexibility without finetuning (though this is not shown in practice).
weaknesses: I believe there is a disparity between the paper’s claims and the experimental evidence provided to support them. Specifically, some of the experiments lend themselves to alternative interpretations, which could be clarified with additional baselines or experiments. The paper claims is that model come up with an "internal lexicon" that create hidden representations of "virtual" words, even when fed, e.g., word pieces as input. This is a claim on the computation carried out by the model, i.e., it is implied that there are some modules whose explicit computation is forming this internal lexicon. I am not sure that the experiments provide sufficient evidence for this claim:

* First, the "motivating experiment" in Section 3 lacks sufficient controls. The authors demonstrate that there is a linear separation in the hidden state between the representations of actual multi-token words and fictional ones created by randomly mixing word pieces. However, this separation could simply reflect the model's ability to distinguish between linguistically valid English morphology and nonconforming sequences, rather than providing evidence of "internal detokenization." For instance, an alternative hypothesis is that the model has learned distributional cues—such as suffixes like "ing" rarely appearing at the beginning of a word—which causes out-of-distribution effects in the hidden states when encountering atypical token sequences.

* In Section 4.1, the authors hypothesize that "if the model performs detokenization, it will represent the last token of a word similarly to the original word token." However, even if such similarity is observed, it could be attributed to the distributional properties of language rather than any explicit "detokenization" process. For instance, in the example provided in the paper where "cats" is split into "ca" and "ts," it is plausible that the pretraining corpus contains instances where this split occurs unnaturally, such as in URLs like "catsanddogs.com" (an actual website) or in cases with typos. Such occurrences might push the representation of "ca ts" closer to that of "cats" without requiring an explicit detokenization step. Furthermore, it is known that such similarities exists also in word2vec methods like Glove, and it is difficult to argue that any explicit detokenization happens there. 

* In Section 4.2, the authors feed the hidden state of the last token of a multi-token word into the model and prompt it to repeat the word. Instances where the model accurately reproduces the entire word are taken as evidence that it has stored the multi-token word in an "internal lexicon." However, a key baseline is missing: including phrases that are not single words, such as "repeat this word: rainy day." The observed results could simply reflect the model's tendency to form contextualized representations that transfer information across tokens, rather than indicating an internalized whole-word representation.

* Finally, the paper’s closing sections aim to illuminate the model's internal computations and the supposed formation of an internal lexicon. While the results provide some evidence of contextualization in the feedforward layers, it's unclear to me whether they genuinely support the existence of an internal detokenization process. Intervention-based experiments could strengthen this claim. For example, could we identify a subset of parameters where ablation specifically impairs performance on multi-token words without affecting single-token words? Or could linear concept erasure techniques reveal a subspace whose neutralization removes all distinctions between multi-token and single-token representations?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: For section 5, the feedforward mechanism, why only FFN output are measured please? Does it make sense to also measure the residual part please?
For section 6:
- For all original vocabulary tokens, all models perform less well than the model with original vocabulary? Are there examples to illustrate these and some hints where models fall short?
- What would be the full newly token accuracy for the model with original vocabulary?
- With such techniques, what would be an estimated inference speed gain? For input embedding as well as for output embedding?
soundness: 2
strengths: The paper presents a significant amount of content and materials while being easy to follow. The paper incorporates appropriately the related works so that it is relatively straightforward to situate the paper in the literature. Concretely, I think the paper has made the following contributions:
- Through techniques such as logic lens and patchscope, the paper demonstrates convincingly where the model performs the detokenization process by presenting clearly such studies.
- The paper shows that FFN serves to combine subword information in section 5.
- In the final section, the paper shows how the understanding can help transformer decoding in practice. The paper adds word embeddings both in input matrix and output matrix and show that the model can accelerate the inference while maintaining a good performance.
summary: The paper studies how subwords are detokenized through transformers into the original word for further processing and understanding (where LLM is able to distinguish between words and non-words are shown are shown in a preliminary study in this work). In this research direction, the paper makes the following contributions:
- The paper shows that detoenization process happens in the beginning-middle layers using techniques using techniques such as logic lens (single token) and patchscope (multi-token)
- The paper then carry on experiments suggesting that the detokenization happens within FFN layers
- Leveraging the above results, the paper shows that that transformer efficiency can be enhanced by introducing "decodable" token embeddings; the paper examines both input embeddings and output embeddings.
weaknesses: While the paper presents a complete study (with no missing component) in the detokenization study, it feels that the paper can still be further enhanced with some more in-depth studies, some of the questions I have put in the questions section but in general:
- The cumulative curve shows that FFN indeed contributes to detokenization. What about other components? Are there any hints/patterns that the authors observe in the detokenization process (e.g. what words are first detokenized)?
- Cumulative rate saturates at around 0.7 shown in the figure. What about the rest 30%? Are these limitations for the measured model? Do better models (e.g. llama3) perform better at these?
- More details will help section 6 and I list some of them in the questions section. I think these are just some of the questions that a common reader would have after reading the paper. I think the results in this section may be of practical importance and deserve to be enhanced with more empirical results.

Meta-review: This paper describes the process by which language models "detokenize" subword-level tokenization. Based on the observation that this process is robust to the addition of out of vocabulary words, they also propose a method for expanding model vocabulary.

Pros: Convincing experiments.

Cons: Some findings, and even the experiments accompanying them, are already made in the existing background literature, even from years ago. (Specifically the detokenization process involving the middle layers of the model.) The authors need an extended literature review in response to these concerns about lack of novelty. Tone of the paper exaggerates the result by describing their findings as the model's "internal lexicon" and is unjustifiably specific about the intuition of the process.

Decision: Accept (Poster)