From Tokens to Words: On the Inner Lexicon of LLMs
Authors: Guy Kaplan
Matanel Oren
Yuval Reif
Roy Schwartz
Venue: ICLR 2025 Poster (2025)
Keywords: Detokenization
Large Language Models
LLM
Byte-Pair Encoding
BPE
Subword Tokens
Word Reconstruction
Latent Lexicon
Inner Dictionary
Token Aggregation
Feed-Forward Networks
FFNs
Out-of-Vocabulary Words
Efficiency
Tokenization
Language Model Optimization
Primary area: interpretability and explainable AI
TL;DR: We provide evidence that LLMs use an inner lexicon to reconstruct words from sub-word tokens. We thoroughly analyze this detokenization process to understand how LLMs manage words internally, and demonstrate the potential gains in efficiency.

Abstract: Natural language is composed of words, but modern large language models (LLMs) process sub-words as input. A natural question raised by this discrepancy is whether LLMs encode words internally, and if so how. We present evidence that LLMs engage in an intrinsic detokenization process, where subword sequences are combined into coherent whole-word representations at their last token. Our experiments show that this process primarily takes place within the early and middle layers of the model. We further demonstrate its robustness to arbitrary splits (e.g., “cats” to “ca” and “ts”), typos, and importantly—to out-of-vocabulary words: when feeding the last token internal representations of such words to the model as input, it can “understand” them as the complete word despite never seeing such representations as input during training. Our findings suggest that LLMs maintain a latent vocabulary beyond the tokenizer’s scope. These insights provide a practical, finetuning-free application for expanding the vocabulary of pre-trained models. By enabling the addition of new vocabulary words, we reduce input length and inference iterations, which reduces both space and model latency, with little to no loss in model accuracy.

Published as a conference paper at ICLR 2025
FROM TOKENS TO WORDS: ON THE INNER LEXICON OF LLMS
Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz
The Hebrew University of Jerusalem {guy. kaplan3, matanel............m oren, yuval. reifreiifff roy roy.saaataaa.. huji. ac. .1
ABSTRACT
Natural language is composed of words, but modern large language mod- els (LLMs) process sub-words as input. A natural question raised by this dis- crepancy is whether LLMs encode words internally, and if so how. We present evidence that L.MS engage in an intrinsic detokenization process, where sub- word sequences are combined into coherent whole-word representations at their last token. Our experiments show that this process primarily takes place within the early and middle layers of the model. We further demonstrate its robustness to arbitrary splits (e.g., "cats" to "ca" and "ts"), typos, and importantly to out- of af-vocabulary words: when feeding the last token internal representations of such words to the model input, it can "understand" them as the complete word de- spite never seeing such representations as input during training. Our findings sug- gest that LLMs maintain latent vocabulary beyond the tokenizer's scope. These insights provide a practical, finetuning-fre ffre application for expanding the vocabu- lary of pre-trained models. By enabling the addition of new vocabulary words, we reduce input length and inference iterations, which reduces both space and model latency, with little to no loss in model accuracy
INTRODUCTION
Large language models (LLMs) rely heavily on tokenization methods such as byte-pair encod- ing (BPE; Sennrich et al. 2016). Such methods often split words into multiple tokens, potentially disrupting their morphological structure (Batsuren et al. 2024 Typos and other perturbations also lead large variations in the tokens that represent word (Kaushal & Mahowald 2022) Nonetheless, LLMs exhibit a remarkable ability to recover word meaning (Cao et al. 2023). This raises important questions about how models internally compose meaningful word representations
from tokens, a process referred to as detokenization (Elhage et al. 2022 Gurnee et al. 2023
In this work, we seek understand the detokenization mechanism in LLMs. We consider two cases: words that are not part of the model's BPE vocabulary, and are thus split into multiple sub-words; and single-token, in- -n-coubbulrry words that we artificially split into multiple tokens. In both cases, our experiments indicate a word-level detokenization process in LLMs, which occurs mainly in the early to middle layers. Our results hint that language models hold latent vocabulary or inner lexicon, which they access to identify words from token sequences
We begin by examining if internal representations of token sequences reflect whether or not a se- quence of tokens comprises word (Sec. 3.. We probe the model's hidden representations (Conneau al. 2018) of both multi-token real English words and gibberish nonwords (Frisch et al. 2000). We observe that the representations of words and nonwords substantially diverge in middle layers-a simple k-nearest neighbors classifier achieves a 89% accuracy on the task of discriminating between the two groups. Overall, these results suggest that models hold concept of recognized words.
We next explore the mechanism through which models reconstruct cohesive word representations from sub-word tokens (Sec. 4h. To do so, use techniques that interpret token's hidden states and
We release our code at httpt:s::/// pestts::/gittiie wwg.ghhh.b.... com/ schwartz- NLb-NLP Lab--MM//Toenns2Rerrd /Tokens2orrds
2 Forr instance, the word "unhappiness" might be tokenized into "un," "h." and "appiness" (see Fig. This process might resemble the mental lexicon in humans (Aitchison 2012 Marslen-Wiin Wilson et al. 1994

---

Published as a conference paper at ICLR 2025
generated
appiness
PPN
attantion
Ict FRN (ward retrakval)
folion aggr cgatin))
(a) embbed
embed
0
appiness
pept
this
NA
The sub-word detokenization process in LLMs. From bottom to top: (a) Tok- enization and Embedding: The input string tokenized using a sub-word tokenizer (e.g., BPE) and converted into vector embeddings;; (b) Token Aggregation: The attention mechanism relays information from the word's previous tokens ("un", "h") into its final sub-word representation ("ap- piness"); (c) Word Retrieval: The model retrieves the full word representation from implicit internal lexicon its feedforward (FFN) layers. This representation added to the residual stream, until it takes over the word's 's hidden representation.
Figure 1: Left:
Right: When taking this hidden representation and patching it into another prompt, the model in- terprets it as the original word. In this example, the model is prompted to repeat the (single vector) hidden representation of the word "unhappiness" (originally represented as three tokens), and is able "understand" by regenerating the original three tokens.
decode them into natural language (Belrose et al. 2023 Ghandeharioun et 2024 We find that for both multi-token and single-token words, in most cases, the last token can be decoded as the full word after being processed by 3-5 layers, with some words requiring up to layers. Interestingly, 23% of the multi-token words are never successfully decoded from their last token's hidden states, hinting this inner lexicon does not cover words.
We then turn to explore how models assemble these full word representations (Sec. 5). We first interpret the feedforward network (FFN) updates in the vocabulary space (Geva et al. 2021 Todd al. 2024). We show that in 85% of the evaluated words, FFN update promoting the full word concept is written to the last token's ' residual stream (Geva et al, 2022; Merullo et al. 2024). Importantly, this happens in the layers before the full word's representation emerges, indicating that models retrieve the reconstructed word representation from their FFN weights. Second, we compare attention patterns between standard single-token words how multi-token words attend to their preceding sub-word tokens. We find that the last token in multi-token words attends significantly more its previous tokens (which are sub-words of the same word) layers 1-2 compared to single-token words (where these previous tokens represent other words), and significantly less in the following layers. This suggests that models initially aggregate information from previous tokens, and later almost ignore them. See Fig. T for an overview.
Our findings have concrete applications (Sec. 6 they allow to expand the model input and out- put BPE vocabulary with the fused representation of multi-token words found in our experiments. Importantly, this expansion requires no update to model parameters. Applying our approach to multi-token words found in English Wikipedia data (Merity al.) 2017). find that the model successfully uses the new vocabulary entries both as inputs and during generation: its language model performance is maintained, and even slightly improves. This demonstrates the potential to dramatically reduce both input and output sequence length, and inference costs accordingly, espe- cially in languages where the ratio of tokens per word is high (Ahia al. 2023
Petrov al. 2023).
Overall, our results establish that word-level detokenization is a core process in LLMs, and provide evidence of how it unfolds across model layers. Beyond improving our understanding of the internal
2

---

Published as a conference paper at ICLR 2025
mechanisms driving LLMs, work lays the foundation for practical applications, particularly in optimizing token management and reducing computational costs.
RELATED WORK
Tokenization Sub-word tokenization algorithms (Wu) 2016 Kudo 2018) are the standard for pre- processing text in modern LLMs. The most widely used method is byte-pair encoding (BPE: [Sen- nrich et al., 2016), which keeps frequent words intact and splits rare ones into multiple sub-words. Recent studies proposed ways to improve tokenization to consider word structure (Provilkov et al.) 2020 Hofmann et al. [0222 Yehezkel Pinter 2023 Bauwens Delobelle 2024) or analyze how tokenization affects model performance (Bostrom & Durret 2020| Church) 2020 Klein & Tsarfaty)
2020 Zouhar et al. 2023 Schmidt et al. 2024). To the best of our knowledge, we are the first to
thoroughly investigate how LLMs internally reconstruct word representations.
Detokenization and stages of inference Harly L.M layers have been shown to integrate local context and map raw token embeddings into representations of concepts or entities process called detokenization. However, such observations were based on specific case studies (Elhage et al.
2022 Lad et al. 2024). More generally, recent work showed early layers provide local syntactic information (Tenney et al., 2019) Vulić et al., 2020 Durrani et al. 2020; Sajjad et al. 2022) or
focus on extracting information from previous tokens (Ben Artzy & Schwartz 2024). Ferrando & Voita (2024) analyzed token attributions in LLMs, observing attention heads that promote sub-word
merging (Correia et al., 2019). Our work focuses on word-level detokenization, and goes beyond previous efforts to provide an in-depth analysis of how word representations are assembled from multiple tokens.
Interpreting the residual stream Recent methods for interpreting the intermediate states of LLMs draw on a residual stream perspective: the hidden state acts as an information stream along the layers, from which information is read each layer, and new information is added through residual connections (Elhage et al. 2022). Thus, hidden states at any layer can be projected into the model's s vocabulary space, treating the hidden state if it were the output of the last layer (nostalge- braist 2020 Dar et af.] 2023 Belrose et al.) [0033 Yom Din et al.] 2024). Similarly, Ghandehhrriou al. 2024) proposed to decode information from hidden representations into natural language, by patching (Zhagg & Nanda 2024) it into a prompt that encourages the model to verbalize the encoded information. We use both approaches inspect how token representations evolve across layers.
LLM memories The idea of an inner lexicon aligns with recent work showing feedforward net- works (FFN) layers in transformers act key-value memories that encode factual and linguistic knowledge (Geva et al., [021] Meng et al.) [022b] Dai et al.] 2002).. Particularly, FFNs were shown to enrich entity tokens with associated information (Meng et al. 2022bb Geva et al., [203] and pro- mote relevant concepts in vocabulary space to build up predictions (Geva et al. 2022). Our work expands on these findings, showing that word representations are pulled from FFN layers before emerging in the hidden state of the word's 's last token.
Inner lexicon structure Inspired by studies on how concepts are encoded in LLMs PPar et al. 2024 2023) and gradually promoted throughout their layers (Geva et al. 2022 Merullo et al. 2024). we consider the inner lexicon a "soft" lexicon, which (a) combines multiple vectors to form word representations (rather than key-value dictionary); and (b) is unique, i.e., a word might be stored and retrieved in more than one layer. Concurrently, /eucht et al. (2024) present evidence for an implicit vocabulary in LLMs, showing that models "forget" preceding tokens in multi-token words or murd entities, but remember previous tokens when processing single-token words.
3 A MOTIVATING OBSERVATION: LLMS CAN TELL WORDS FROM NON-WORDS
One of our key hypotheses in this work is that LLMs hold an internal lexicon of words. which is different from the BPE lexicon. We begin by asking whether LLMs, when processing a sequence of tokens, capture some notion of whether not this sequence forms word.
3

---

Published as a conference paper at ICLR 2025
Words Dataset
Un
appiness
Unhappiness
Takenizer
udd
ha
0.9
Buddha cathouse
parultiraaa
ouse
0.8
Mixing
Accuracy
0.7
Un
udd
ouse
non-words
Dalaset
ha
Tokenizer
Unuddouse
05
chiha
appiness
bathappibess
10
15
Layer
(a) Nonword creation process
(b) Classification results
Figure 2: Our word vs. nonword probing classification experiments. 2ab Dataset creation process.
Top: words from the Gutenberg corpus tokenized using the Llama2 tokenizer. Bottom: nonwords are generated by shuffling tokens while maintaining their positions within the word; (2b) Classifica- tion results of words vs. nonwords. Using the last token shows strong results (reaching up to 90% accuracy), which sharply rise in early layers (3-7), peak in the middle (13), and decrease in later layers (20-32). Using the penultimate token shows substantially lower scores, suggesting that the high classification accuracy is specifically tied to the presence of a complete word rather than token co-occurrence patterns.
To address this question, we construct a balanced dataset containing two groups: one with real English words, and another with artificially generated, meaningless Ronwords (Frisch et al. 2000). Both groups are tokenized using the Llama2 tokenizer (Touvron et al. 2023). The word dataset consists of 10,000 distinct words sampled from the Gutenberg corpus (Gerlach & Font-Clos 2018), with 53% of the words containing two tokens, 37.3% containing three tokens, and the rest four tokens. We generate the nonwords by shuffling tokens from the word dataset, ensuring that the prefix and suffix positions align with the original tokens' positional probabilities. For example, the token "ing". extracted from the final position of real words in the dataset (e.g.. "directing" tokenized as "direct", "ing"), is retained as a suffix in the nonwords dataset. This process ensures that nonword tokens maintain position distribution properties similar to word tokens, preserving natural positional patterns and mitigating potential distributional biases (Fig. 2a). We next apply a k-nearest neighbors (k-NN) probing classifier (A- 4, using Euclidean distance) on the hidden states of the last tokens of both words and nonwords, for each layer of the Llama2-7B model. The training set consists of 80% of the dataset, and the remaining 20% are used for evaluation.
Our results (Fig. 2b blue) reveal three-stage pattern in the model's representation of word and nonword token sequences. In the model's first few layers, representations from both groups are relatively indistinguishable and accuracy is close to chance level. Then, from layers 2 to 6, clear distinction between the two groups emerges, until the representations are almost completely separate in middle layers, between layers 6 and 20. At this point, the probe achieves stable, high accuracy, peaking at 89% on layer 13. Finally, accuracy slightly drops after layer 20
Our results indicate the LLMs distinguish between words and nonwords, But this distinction can be attributed to the distributional properties of words: it might be the case that models are recogniz- ing common sequences of tokens, rather than identifying whole words. To this hypothesis, we repeat the same experiment, this time using the penultimate token representation, for words three tokens or longer. By definition, such words also as frequently co-occur with the initial sub-word tokens in their words as the final tokens. Our results (Fig. 2b orange line) show that the probe only reaches 61% classification accuracy, indicating that the high classification accuracy does not stem from token co-occurrence, but is tied to the presence of whole word. See App. A for an analysis of misclassified words and nonwords.
WWe perform a similar experiment using dataset of morphologically plausible nonwords (ARC Nonword Database; Rastle al. 2002). which shows similar trend. See App.
4

---

Published as a conference paper at ICLR 2025
Overall, our results show that language models internally represent words and nonwords differently. This distinction is gradually developed in the model's early layers and maintained throughout its middle layers. These findings support the hypothesis that the model performs detokenization pro- cess and suggests where this process occurs. Building on these results, we next investigate how sub-word tokens combined into word representations across model layers, and explore the inter- nal mechanisms that facilitate this transformation.
4 EXTRACTING WORD IDENIITY FROM LLM HIDDEN STATES
We have so far observed that LLMs can differentiate between words and nonwords, suggesting an internal detokenization process specific to word composition. We next dig into this process, by asking whether we directly extract word identity from the hidden states of sub-word tokens. We
start by considering single-token words (Sec. 4.1, and then move on to multi-token words (Sec. 4.2)
4.1 SINGLE-TOKER WORDS
We first consider in-vocabulary words, which are mapped to single tokens by the tokenizer. Naively, such words don't 't tell us much about detokenization, as they are represented using only one token. To address this, we artificially split them into multiple sub-word tokens. For example, we take the single-token word "cats" and split it into two tokens: "ca" and "ts". We hypothesize that if the model performs detokenization, it will represent the last token of the word ("ts") similarly to the original word token ("cats"). We iterate the WIKITEXT- 103 dataset (Merity et al., (017) and randomly split each single-token word longer than three characters into 2-5 sub-words tokens. We then feed the model the new sequence of tokens preceded by the last 100 tokens that came before each split word in the original text as context F
To measure the similarity between the representation of the final token and original word, we apply the logit lens method (nostalgebraiss, 2020), an interpretability method that maps hidden represen- tation of a given token to the word whose vector in the output unembedding matrix is most closely aligned with the hidden representation. We note that this technique is typically used inspect model predictions of the next token in intermediate layers. As we aim to identify how LLMs represent the current word, we use the imput embeddings matrix rather than the unembedding one
We study four LLMs: Llama2-7B, Llama3-8B (Dubey et al. 2024). Mistral-7B (Fiang et al. 2023), and Yi-6B (AI et al. 2024). For each layer, we report the rate of retrieval the proportion of words for which the closest vector in the vocabulary space is the original (single token) word.
Our results for the Llama2-7B are shown in Fig. 3a Starting at layer 8, the hidden state of the final token is mapped to the original word with high accuracy, which peaks at more than 80% in layer 15. Interestingly, this accuracy then starts to decline, possibly because the model transitions into representing the prediction of the next token. We also find that 93.2% of all split words are correctly mapped to the original word in at least one model layer (see App. C.I
Our results indicate that LLMS perform detokenization process in cases where single- token word is split into multiple tokens. This process assigns the final token with a hidden representation similar to that of the full, single-token word. But how robust is this process? To address this question, we consider a different way of splitting a word into mab-tokens-addin randomly flip two adjacent characters, delete character, or insert new character (see App. for more details). Much like our previous experiment, this process splits the word into multiple tokens, as corrupted words are rarely found in BPE lexicons. We repeat the same experiment as above, aiming to study whether models map the last token to the original (correctly-spelle word.
typos. For each word, we
'We observe similar trends, in this and in other experiments in this section, when passing the split words without their context, albeit with slightly lower rates of retrieval.
We also compute a second measure, cosine similarity, to compare between the hidden representations and input embeddings. We find similar results to those obtained using logit lens. For details, see App. Results for the other models (App. C show similar trend.
5

---

Published as a conference paper at ICLR 2025
(a) Rate of retrieval for single-token words
(b) Rate of retrieval for multi tokens words
5,00
0.0
Rala of original WOMS estional
0.6
0.4
Rate of original
0.4
0.2
0.2
0.0
20
25
30
Layer
Layer
Figure 3: (3a Logit lens rate of retrieval of single-token words artificially split (blue line) and split due to typos (orange). both cases, we see an increasing rate of retrieval after the 4th layer, peaking in the middle (16-17 layers) and then dropping: (3b) Patchscopes rate of retrieval for multi-token words. In this case results peak in the 5th layer and then start to decline.
Our results for Llama2-7B (Fig. 3a show similar trend, though less pronounced: the model cor- rectly matches about 40% of corrupted words with the original, single-token, uncorrupted word We also note that the trend across layers persists, with accuracy peaking around layer 15.
4.2 MULTI-TOKEN WORDS
We have seen that LLMs are able to detokenize single-token words that are artificially broken into sub-words back into their correct (single-token) word. This indicates that these words are part of the model's i inner lexicon. But what about multi-token words? By definition, these words do not have a cannot compare them to any existing vector,
single embedding vector in the BPE vocabulary, and thus cannot use logit lens.
Instead, we use the Patchscopes technique (Ghandeharioun et al. 2024), which interprets the hidden representation of a model using the model's language abilities. In particular, we feed the model with the prompt 'Repeat this word twice: 1) X 2)". where X is the hidden representation of the last word token. Our hypothesis that if the multi-toke token word is found the model's s internal lexicon, then it will "understand" its representation in the input layer well, and successfully repeat it by generating all tokens of the original word (see Fig. right).
We evaluate the model by computing the rate of retrieval- the proportion of times it generates the correct (multi-token) word. We use the same models and dataset as in the previous experiments, though in this case do further split add typos, the words are already split into multiple tokens. Particularly, we feed each multi-token word (along with its 100-tokens context, as before) the model, and extract the last token's 's representation from each layer. We then use Patchscopes' prompt in new run check if the model regenerates the original multi-token word.
Our Llama2-7B results (Fig- 3b) show a striking trend: when feeding the model with vectors from layers 5-7, it is able to repeat the word in 64% of the cases, despite never seeing this vector input before 9 As to the different layers, we observe a similar trend the single-token results: retrieval rates increase rapidly at some point (here earlier than in previous experiments, around layer 3-4), reach a peak at layer 7, and then start to decline. Combined, these results suggest that the model treats multi-tokens words as if they were in the vocabulary but split into sub-word tokens, indicating a latent vocabulary that expands beyond the tokenizer's 's limitations. Interestingly, we observe that 22.6% of the multi-token words are successfully decoded from any internal layer, hinting that they might not be represented in the model's 's inner lexicon.
'Here, 66% of the words are correctly mapped in at least one layer. Overall, 77.4% of the words are repeated correctly in at least one layer.

---

Published as a conference paper at ICLR 2025
(a) Word retrieval per layer
(b) Cumulative word retrieval per layer
Nidde FRN
0.7
0.5
Cumulative rate of retrieval
0.6
Rate of retrieval
0.4
0.5
0.3
0.4
0.3
0.2
0.2
0.1
0.1
0.0
0.0
5
10
15 Layer
20
25
30
0
5
10
15 Layer
25
30
Figure 4: Word retrieval in FFN layers in the hidden states. (4a) Retrieval rates across layers. The FFN values peak before word retrieval begins in the hidden layers. (4b) Its cumulative version, showing word retrieval occurs earlier and more frequently in FFNs than in the hidden representation.
5 HOW DOES DETOKENIZATION HAPPEN?
We have so far shown that LLMs perform detokenization step: some point, the hidden repre- sentation of the final token of a given word becomes strikingly similar to the (single-tokenx) vector of that word. This process is robust to artificial splits of single token words, to splits due to typos, and even to multi-token words, which the model can still recognize at the input layer, despite never having seen them there during training.
We next turn to ask how does the model reconstruct full word representations from sub-word tokens? We aim understand the dynamics of this process by analyzing the main transformer components: feedforward network layers (FFN) and the attention mechanism.
5.1
WORD RETRIEVAL USING THE FEEDFORWARD MECHANISM
FFN layers have been shown to serve as key-value tables for storing memory (Geva et al. 2022 2021 Meng et al., 2022b). We hypothesize that this memory might be used store the inner lexicon as well. Particularly, we suggest that the model uses the FFNs to refine the representation of final token, enabling it to retrieve the original word.
To test this hypothesis, we repeat the typos experiment of single-token words (Sec. 4.1 but this time applying logit lens to the output of the FFN layers, instead of the hidden representation. Figure 4a shows the retrieval rate of the original word. We observe that the retrieval of the full word concepts from the FFN layers occurs a few layers earlier compared to the hidden state, which indicates that they help refine it. Nonetheless, the rate of retrieval appears be lower in FFNs, which suggests words. However,
that other factors might come into play in building the hidden representation in Fig. 4b we plot the cumulative retrieval rate, i.e., for each layer, whether the word is identified in any layer so far. Our results indicate a different conclusion: the FFN update vectors match the (single-token) input representation of the word 70% of the time in any layer, particularly both earlier and more frequently compared to the hidden representation. This hints that FFNs indeed play substantial role in building the internal word representations in LLMs.
We further investigate the role FFNs play in detokenization through ablation experiments on FFN updates Llama2-7B. Specifically, we test whether these updates are necessary for the word rep- resentation to emerge in the residual stream. To do so, we artificially split single-token words in similar setting to Sec. I this time focusing on derived words, formed by adding a suffix word 10
root We split these back to two parts, for example, we divide "eating" to "eat" and "ing". so that processing the suffix token essential to reconstructing the word correctly. Using logit lens, we ex-
We examine words with three common suffixes: "ing", "ion", and "est".

---

Published as a conference paper at ICLR 2025
amine the FFN updates to the suffix token's residual stream, and selectively ablate those associated with the original single-token word (~5% of layers). As a control, for each word, we ablate an equal number of random FFN updates. Our results (Fig. 12 in App. E) show that removing the updates carrying full-word representations dramatically reduces retrieval rates-from 85% without ablation to just 18%. In contrast, ablating random FFN updates has little no effect.
This indicates that FFN updates are essential for a detokenized representation to emerge in the final token. But does this affect the model's ability to "understand" the word, particularly when predicting the next token? To investigate this, we evaluate the model's ability to retrieve the capital cities of countries, using the prompt "The capital of [COUNTRY] is --- We use single-token country names artificially split into two tokens, and follow the previous ablation protocol. We observe notable patterns: randomly ablating FFN updates reduces performance from 88% to 74% likely due to disrupting the model's factual recall mechanism (Meng et al. (022b). However, specifically canceling updates detected as country's name (similarly occurring in only ~5% of layers) causes sharp drop to 41%. Overall, our results suggest that FFNs are central to detokenization, and actively reconstruct full-word meanings in LLMs.
5.2 TOKEN AGGREGATION
Our results suggest that LLMs the FFN layers to store and retrieve word representations, which are accessed in the final token to reconstruct the complete word. However, this role of the FFN layers only begins to emerge around layers 3-4. What happens earlier? We build on previous work showing that early layers primarily integrate information from nearby tokens to compose en- tities (Lad et al., (002)). and hypothesize that the model starts by aggregating information from the previous wurd tokens.
To test this, we extract all two-token words a subset of 1,000 WIKITEXT- 103 documents (a total of 5,571 words), and feed them, along with their context, to Llama2-7B. We then measure the average attention weights of the final token to the
Average attention weight
0.10
Mull-tokznn
Singl-iiiee pire tokon atterton
0.08
prefix token in each layer. As control, also measure the average attention weights assigned by single-token words to their preceding token.
0.06
0.04
Our results (Fig. 5) support previous find-
0.02
ings (Lad et al. 2024) the attention previ-
ous tokens is high in the first 2-3 layers. but then declines sharply (by up ~90% For single-token words, we observe a similar atten- tion pattern the previous token in the first lay- ers, but importantly the initial peak is signifi- cantly lower than multi-token words 13 Still, later layers, the attention weights of single-token words are in turn higher than for multi-token words. These results suggest LLMs strongly at- tend to the preceding sub-word tokens multi- token words at first, but then largely ignore them.
20
Layer
Figure 5: Attention weights for 2-token words: Early peaks (layers 2-3) show high attention values from the second sub-word token to the first, but these values decline rapidly. Attention from single-token words to their previous token shows similar trend, though with substantially lower values at first, which become higher later.
Altogether, our results suggest that LLMs perform a detokenization process by first aggregating in- formation from the prefix tokens into the final token's hidden representation, and then refining the representation of the final token using the FFN layers to retrieve the full word's concept represen- tation. This two-stage process of token aggregation and concept retrieval provides insight into the mechanisms LLMs use to handle sub-word tokens and reconstruct word-level representations.
We note that baseline performance when passing country names without artificial splits is 95%. 12 Experiments with 3- and 4-token words show similar trend, see App. B The diverging patterns between single and multi-token words are statistically significant; see App.
8

---

Published as a conference paper at ICLR 2025
6 EXPANDING LLM VOCABULARY WITHOUT FINETUNING
We have shown that language models internally fuse multi-token words into a single-token repre- sentation, and that they can further "read" these representations as inputs, and decode the original multi-token words. This raises the question: can models use these fused representations instead of the original multi-token inputs to encode input prompts using less tokens and reduce compu- tation? Similarly, models were shown to implicitly predict several future tokens in a single hidden state without being explicitly trained to do so (Pal et al., 2023); ); can we leverage these representations to enable models to predict multi-token words in single inference step?
Motivated by our findings, we explore whether we can expand the model's vocabulary with new input and output embeddings for originally multi-token words, without any updates to model pa- rameters [4 This goal of practical importance, especially for low-resource languages and domains: even tokenizers built for multilingual support often produce substantially longer token sequences for non- non-English languages up 13 times longer than equivalent English texts iiss-mtmpptnggg inference cost and speed (Ahia et al. 2023 Sengupta et al.) 2023: Petrov et al. 2023 Still, prior attempts to expand tokenizer vocabulary lary post- hoc are limited in number, and require substantial additional train- ing (Kim et al.) 22224] Zhao et al. 2024). In contrast, we propose to detect words the model "knows" and successfully detokenizes to single vector, and use this to obtain new token embeddings.
Word retrievel
"develop"!
U
H₁
Tu.
U
H
Embed
H
TE.
W Σ
E
dev
lop
E
(1) Extract detokenized
(2) Learn + apply token-agnostic maps from hidden layers to E and U
(3) Train linear maps to jointly refine all new entries È and Ü
representations
Figure 6: Our 3-step method to expand LLM vocabulary without updates to core model parameters.
Our framework for vocabulary expansion follows 3-step process (Fig. 6). Given a multi-token word w that we would like to add to the vocabulary, and the model's is original input embedding and output unembedding matrices E and U with hidden dimension d, we (1) extract a detokenized, single-token representation r for w using a similar approach to Sec. 4.2 pass wi as input to the model and apply PATCHSCOPES (Ghandeharioun et al. 2024) with prompt A to the last token's hidden states at all layers. We then identify É, the earliest layer at which the hidden state is successfully decoded into the full word, and set that hidden state. Next, we (2.1) learn set of linear maps Te, and Te, to project hidden states from layer l of the model to the embedding and unembedding spaces, based only on the existing in- in-vocabulary tokens Then, (2.2) for each detokenized representation taken from layer f, we apply Ty, F and Te, to obtain the initial entries in the embedding and unembedding matrices to represent the word, é and ú. Finally, after computing the initial entries for all new words, we (3) refine the new representations d x d all-zeros matrices We and Wo, and set the new entries as e = é Wge and u = û W. We train the refinement matrices WE and Wo jointly in a short continued pretraining run, while keeping all other parameters frozen. Finally, we compute the final and u, and use these to represent the new word in the expanded vocabulary. Importantly, if word is never successfully decoded from the hidden states in any of the layers in step (1), we assume it is not found in the model's 's inner lexicon, and therefore do not add it to the vocabulary.
obtain the final entries: we initialize two
Wee do add new entries to the model's embedding and unembedding matrices, but these are intrinsically required to expand the vocabulary and represent the new vocahulary words.
For P., we use the template "x x xx" where is the patched representation of the new word.
We learn T', and Te. by fitting an orthogonal procrustes transformation from the layer & hidden states of all in- vocabulary tokens (when passed as inputs on their own), to their corresponding embeddings or unem- beddings (see App. App.Gfre for details). Importantly, these projections only rely on the model's 's existing vocabulary, and do not depend on which multi-token words we choose to add to the vocabulary.

---

Published as a conference paper at ICLR 2025
WIKITEXT-10: Mean Embed.
PuBMan Mean Embood.
WikI40B-AMiic Mcan Einbed.
Original
Ours
Original
Ours
Original
Ours
New words
token
0.071
0.171
0.123
0.180
0.117
0,402
original or new token
0.322
D.178
0.284
0.280
0.171
0.259
D413
0.119
0,414
All words
0.522
0.473
0.519
0.517
0.479
0,511
0.535
0.211
0.532
Table 1: Token-level accuracy of finetuning-free vocabulary expansion for Llama2-7B on three datasets. We compare our method using detokenized representations (Ours) to the original model (Original) and to an expansion baseline using the word's average token embeddings from E and U (Mean Embed.; /ee et al. 2022). Accuracy is reported for newly added words, where we distinguish between correctly predicting the new tokens (top row) and predicting either the word's original first token, the unexpanded model would, or its new token (middle row). We also report overall performance on all tokens (last row). Our method enables the frozen model to use the newly added input and output embeddings, while maintaining overall model performance.
We apply our approach to LLAMA2-7B and experiment with three datasets: WIKITEXT- 103 (Mer- iity et al. 2017), abstracts of biomedical articles from PUBMED (Xiong et al. 2024). and the Arabic split of WIKI40B (Guo et al.] 2020). For each dataset, expand the model's vocabulary with all multi-toker words that appear at least m times in the test set T7 We then learn the refinement matrices WE and Wy using 20M tokens from the train set We evaluate models in next-word prediction setup and compare our approach against two baselines: the original model with an un- modified vocabulary (original); and an expanded model that follows our framework but uses the mean embeddings of the word's tokens in E and U to initialize its new representation, instead of the detokenized representations r, following Gee al. (2022) mean embedding). For evaluation, per- plexity is not suitable metric, as differences in vocabulary between the approaches skew perplexity scores and prevent a fair comparison. Instead, we measure the model's token-level top-1 accuracy when it is given each token's ' previous context.
Our results (Table T indicate that models can generalize to new vocabulary entries surprisingly well when these are initialized with their own detokenized representations. Unlike the mean embeddings expansion baseline, which both struggles with using new tokens and degrades overall performance, our method allows the model to successfully integrate new vocabulary words while preserving its accuracy on existing words. This effect is particularly pronounced in Arabic WIK140B, where the model almost always selects new tokens instead of the original tokenization (40.2% 41.3%), demonstrating the potential of our approach for multilingual and domain domain-specific applications.
Our method also shows potential to reduce the number of tokens processed during encoding and inference: across the three datasets, we observe a reduction in the total number of tokens processed during encoding by 10.5% 14.5% (see Table App. H. while maintaining model performance. In future work, we will further explore the application of our framework to multilingual adapta- tion (Petrov et al.
2023 Alabi et al. (022) and continual domain-adaptive pretraining (Ke et al.
2023
Yıldız et al.
2024 Gururangan et al. 2020
7 CONCLUSION
The ability of LLMs to comprehend and generate language relies on intricate internal processes, and understanding these mechanisms is crucial for improving model performance and efficiency. In this work, we unraveled the word detokenization process, shedding light on how models internally transform fragmented sub-word tokens into coherent word representations formed at the last token. Our results indicate that this mechanism manifests in early middle layers, where models attempt to reconstruct words by mapping them to an inner lexicon using their FFN layers. We provided evidence this lexicon is more exhaustive than the tokenizer's vocabulary, and could help models to recognize words even amidst noise.
Our work also unlocks practical avenues for optimizing tokenization, well as the speed and cost of inference. We demonstrated one such application and presented a finetuning frret method to expand the vocabulary of LLMs. We hope our work paves the way for more efficient and versatile models.
17 We use m = 1,5, 50 for WIKITEXT- 103, PUBMED and Arabic WIK140B respectively.
We use sequence length of 512 and train 10,000 sequences, taking up 30 minutes on single GPU.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the Isracl Science Foundation (grant no. 2045/21) and by NSF- BSF grant 2020793. We are grateful to Iddo Yosha for his valuable insights and thank the anonymous reviewers for their constructive feedback.
REFERENCES
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Confer- on Empirical Methods in Natural Language Processing, pp. 9904 9923, Singapore, Decem- ber 2023, Association for Computational Linguistics. doi: 10. 10.18653/11/2223.mmml maanllp-main..... URL whttts://accaaaete
ogy. org/ 2023. emn p- ma n. 61
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024. htt.httsss///xiiv... abs/2405 04652
Jean Aitchison. Words in the mind: An introduction to the mental lexicon. John Wiley & Sons, 2012.
Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. Adapting pre- trained language models to African languages via multilingual adaptive fine-tuning. In Nico- letta Calzolari, Chu-Ren Huang. Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Ni- anwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 4336-4349, Gyeongju, Republic of Korea, October 2022. Inter- national Committee on Computational Linguistics. URL https://aclanteee ogy. orgg/
382/
Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, and Gábor Bella. Evaluating subword tokenization: Alien subword com- position and oov generalization challenge. arXiv preprint arXiv:2404. 13292, 2024.
Thomas Bauwens and Pieter Delobelle. Bpe-knockout: Pruning pre-existing bpe tokenisers with backwards-compatibn morphological semi-supervision. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023. URL
08112
Amit Ben Artzy and Roy Schwartz. Attend first, consolidate later: in different LLM layers. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet. Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Ana- lyzing and Interpreting Neural Networks for NLP., pp. 177-184, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 120..//////2224bbbbcbboommo https: /accc///cccaaa antho Logy. org/ 2024 bl acwboxn
On the importance of attention
URL
Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pretrain- ing. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Com- putational Linguisties: EMNLP 2020, pp. 4617-4624, Online, November 2020. Association
for Computational Linguistics. doi: 10.18653/v1/2020.fmm findinng-eennl..44444 URL https:
//oclan
www.llogy.rgg//2220 org////0/0000 wtt.:2.........io. findings emnlp.. 414
Qi Cao, Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. Unnatural error correction: GPT-4 can almost perfectly handle unnatural scrambled text. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language
11

---

Published as a conference paper at ICLR 2025
Processing, pp. 8898 8913,, Singapore, December 2023. Association for Computational Linguis- tics. doi: http://..2223..mmmmmmaaaaaa URL monnppmmain 550
Kenneth Ward Church. Emerging trends: Subwords, seriously? Natural Language Engineering, 26 (3):7753322, 2020.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single S& !# vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao (eds.). Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/1/6//////1118//1 1111888-8 1198.
URL http:////claaanooooooooooooe
1198
Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. Adaptively sparse transformers. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 2174-2184, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 10.18653/11/119-1222 URL https: hwtpt:tt///cccc. http:://aaaaathhho org/ 1223/
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (cds.), Proceedings of the 60th Annual Meeting of Association for Computational Linguistics (Volume I: Long Papers), pp. $493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.
URL https: lttps://aclanthologg .org////yyorr/////ooooooo
2022.ac1- long......... 581
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 18124-16170 Toronto, Canada, July 2023. Association for Computational Linguis- 1/06633/11/22333aalloong
tics. doi: acl- acl-long. 893
URL https: attps::/aaclanthologyy.. org/2033.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neu- rons in pre-trained language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 4865-4880, Online, November 2020. Association for Computational Linguis- https::: //acl//ccccaa
tics. doi: 10. 2ttp://11122000m/mmmaaaaal
acc http:///lcaaaallog....o.ooo org/ 2020.
emn 1 p-mmp-maaa- in. 395
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer EIShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Ka- mal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodde Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Ka- davath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCan- dlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022. https//ttansfommerccrriiii pub/2022/solu//ndeer ootpaaasoo.ee..iiiiiiooooooooood html.
Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. arXiv preprint arXiv:2403.008: 000844 2024.
Sheridan Feucht, David Atkinson, Byron C Wallace, and David Bau. Token erasure as a footprint of implicit vocabulary items in LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, pp.
Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 1/22443/11222..mmmllpl. main.5. URL /tttts://accaaaaeee http:///laaanhooogg..ooooo oxq/ 2024 emn m...mmain.5337777
12

---

Published as a conference paper at ICLR 2025
Stefan A. Frisch, Nathan R. Large, and David B. Pisoni. Perception of wordlikeness: Effects of seg- ment probability and length on the processing of nonwords. Journal of memory and language, 42 4:481-496, 2000.
httht /ttp//ap.... www.stteaatttttcceeeeeeee http://oo.....oooooooooooooaa1a.
org/ CorpusID:173695
Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. Fast vocabulary transfer for language model compression. In Yunyao Li and Angeliki Lazaridou (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 409-416, Abu Dhabi, UAE, December 2022. Association for Computational Linguistics. 2t0p555/11222222mmmm-nndnssttt/
doi: 10.18653/11/20022eemm
URL https: https///aaaaatthloooooeo ootthelooy.org/2022..
emn 1ommpp-nndie wtw.p-indsutt..444
Martin Gerlach and Francesc Font-Clos. standardized project gutenberg corpus for statistical analysis of natural language and quantitative linguistics, 2018. URL htt.s:ts///aaiiiiii org/
abs/1812.0800 08092
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics, doi: 10, 18653/81/202/ www.:0/..mmmmmmmmmmmmmmmmmm main.446. URL https://acl ant.aaaaanntttaccooooo wownnthooog..... org/ 2021 emnlp-ma 1 n.. 446
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 20.18633/v1/0000 00665//112222.mmmmmmmin... URL
https://ach awtaaatothooooooo wwt.orgg/2200 2022. emnlp-ma 3
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson.
Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 12216 6-12255, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2023 wem.:3mmmmlpmmmmee m5in.751. URL
emm.pp 751
Asma Ghandeharioun, Avi Caciularu, Adam Pearce. Lucas Dixon, and Mor Geva. Patchscopes: A unifying frammwork for inspecting hidden representations of language models. In Proc. of ICML, volume abss/440..06111 002.0 2024. URL http:atsss//xx.....rmm org. http://b///0000 02102
Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40B: Multilingual lan- guage model dataset. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceed- ings of the Twelfth Language Resources and Evaluation Conference, pp. 2440-2455 2452, Marseille, France, May 2020, European Language Resources Association. ISBN 979- 10-95546--4444, URL httpps://cclamthooooy.... org
htt.:/aaaaaatoooooooorr0//0..... 1recc 1ea.eecc...777 1.2977
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsi- mas. Finding neurons in haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: http///ppeneeiieeeeeeeeeeee net forum?
id JYs1R9IMJJ
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342 8360, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v wttp:////222eeee acl- main. 740. URL
https: //t::///cc antho www.:olooy............. wrw.ooorg/200.. acl -aann....... 740/|
13

---

Published as a conference paper at ICLR 2025
Valentin Hofmann, Hinrich Schuetze, and Janet Pierrchumbert. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 385-393, Dublin, Ire- land, May 2022. Association for Computational Linguistics. doi: 10. 1
URL httpp::/aclaanthoooyy org. 2022 acl short. 43
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv.org oogg abs/ 2310000822222
Ayush Kaushal and Kyle Mahowald. What do tokens know about their characters and how do they know it? In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2487-2507, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 16333/6112222.maaa0 nww.l- main. 179. htthttts:: a/ccl//ccccccc wwt.hoolgy.............. org 2022. www.caacllaaa. in.ii...... 179
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre- training of language models. In The Eleventh International Conference on Leurning Representa- tions, 2023. URL http::://pppeeeeeeeeietee
GDIItaI30
Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. Efficient and effective vocabulary expansion towards multilingual large language models. arXiv preprint arXiv:2402.14714 2024.
Stav Klein and Reut Tsarfaty. Getting the### life out of living: How adequate word-pieces for modelling complex morphology? In Proceedings of the 17th SIGMORPHON workshop on computational research in phonetics, phonology, and morphology, pp. 204 209, 2020.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/ P10-1007. URLthttss: httpa/ttsss//aaaaaaalllaaoo///oe hacla/aaaaaoooooooooro///-----11
Vedang Lad, Wes Gumnee, and Max Tegmark. The remarkable robustness of llms: Stages of infer- ence?, 2024. /
19384
William Marslen- -iiloo,, Lorraine Tyler, Rachelle Waksler, and Lianne Older. Morphology and meaning in the english mental lexicon. Psychological review, 101(1):3, 1994.
Fanwang Meng, Michael Richer, Alireza Tehrani, Jonathan La, Taewon David Kim, Paul W Ayers, and Farnaz similarity between matrices. Computer Physics Communications, 276: 108334, 2022a.
Procrustes: A python library to find transformations that maximize the
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in gpt. Advances in Neural Information Processing Systems, 17359-17377 2022b.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference on Learning Representations, 2017. URL https:
w/oppnrevvew..o
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple Word2Vec- style vector arithmetic. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Asxociation for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 5030-5047 Mexico City. Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/11/024. wwwplloyy.ogg/224..rom naacl lon..on...2.. 281
naacl-long.284 URL
nostalgebraist.
interpreting GPT: the logit lens. LessWrong, 2020.
URL
http:://wwwweessrro.te http://www...oooooottt////tttaaa
interprating
14

---

Published as a conference paper at ICLR 2025
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. Future lens: Antici- pating subsequent tokens from a single hidden state. In Jing Jiang, David Reitter, and Shumin Deng (eds.), Proceedings of the 27th Conference Computational Natural Language Learn- ing (CoNLLL,, pp. 548-560, Singapore, December 2023. Association for Computational Lin- URL https: //acl www.aanooooo.gg///000.
guisties. doi: 10. con11-1.37
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierar- chical concepts in large language models. In ICML 2024 Workshop on Mechanistic Interprelabil- ity, 2024. URL htttps:://peerreieeeeeeee www.eeveeeeeveeeeeeeeeeeee ew net
Kiho Park, Yo Joong Choe, Yibo Jiang. and Victor Veitch. The representation geometry of features and hierarchy in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL htt.https: : http///ppee/eeeeeee net wttpee///ooooommoo///22e2222
Alcksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. Advances in Neural Information Processing Systems, 36, 2023.
Ivan Provilkov, Dmitrii Emelianenko, and Flena Voita. BPE-dropout: Simple and effective subword regularization. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.). Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1882-1892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 1/2600//12020.acll main. 170. htthttss: acl aww.acoooaototoo.ooooooo ooroorrg///0// www.amalmmma..177.
Kathleen Rastle, Jonathan Harrington, and Max Coltheart. 358,534 nonwords: the arc nonword
database. The Quarterly journal of experimental psychology. A, Human experimental psychology, 55:1339-62, 11 2002. doi: 10. 00.18800222498804440009999
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Khan, and Jia Xu. Analyzing en- coded concepts in transformer language models. In Marine Carpuat, Marie-Catherine de Marn- effe, and Ivan Vladimir Meza Ruiz (cds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 3082-3101, Seattle, United States, July 2022. Association for Computational Linguis- URL httphtttpe///ae..atooooooo.orrrp
tics. doi: haacc-ma in. 2257|
Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. Tokenization is more than compression. arXiv preprint arXiv:2402.18376, 2024.
Peter H Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia. Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kam- boj. Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hee- tor Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. Jais and jais-chat: Arabic- centric foundation and instruction- open generative large language models, 2023. URL
49149 arXiv:2308. 16149.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/P11 1162.
URL htt.httts::///
antho ogy.oogg....1111111 www.or/g//.11116...1
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluis Märquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 118653/v1/P19-11111 14522 URL https:
7/acl Torclanntollo www.annooooooooooooooooooooeoeee
15

---

Published as a conference paper at ICLR 2025
Eric Todd, Millicent Li, Arnb Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Tweifth International Conference on Learning Representations, 2024. URL https: settps://openreeieww. htwp://ww...tt..o....ooooo/oaaaa Awy
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL htt.hhtt:///// //tp:v//rxvvvvo 2ttg/abba atrrg/abs/2300.. 200.:/237..c0 0928889288888888
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. Probing pretrained language models for lexical semantics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirieal Methods in Natural Language Processing (EMNLP), pp- 7222-7240, Online, November 2020. Association for Computational Linguistics. doi: 10. 201118651/22220 10001660/1100000.mmmmmmmmmm 2020. www.pmnmmmne ma n. 586
URL
ogy. org/
Yonghui Wu. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08/1 08144, 2016.
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking autreval-augmented generation for medicine. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 6233 6251, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/11/2224.ffnnim 372. URL BtRhhtts::/ acl antaoloooyy...... wwg. 2024. fwnniiginnniii.....nn.rnn.nnoo acl 3727
Shaked Yehezkel and Yuval Pinter. Incorporating context into subword vocabularies. In Andreas Vlachos and Isabelle Augenstein (eds.). Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 623-635, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi:
URL
org/ 202g://222.cmm www.lmaalaaa...... wtt.://rro/22...cmmmmmmmm..
Çagatay Yıldız, Nishaanth Kanna Ravichandran, Prishruit Punia, Matthias Bethge, and Beyza Ermis. Investigating continual pretraining in large language models: Insights and implications. arXiv preprint ar arXiv:2402.17400 2024.
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short- cutting transformers with linear transformations. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu- ation (LREC- 2024), pp. 9615-9625 Torino, Italia, May 2024. ELRA and ICCL. URL ht
wtg wwwc-maaiiiiie 840
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metries and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpp//wwwwwpp./cccooe htt.:/iwwwwweeeeeeeeeeeeeeeeeeee fest/forrnne oftp://wwww......ooooooo..//aa/.
Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.0 2024.
16

---

Published as a conference paper at ICLR 2025
Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. Tok- enization and the noiseless channel. In Anna Rogers, Jordan Boyd-Graben and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5184-5207, Toronto, Canada, July 2023. Association for Compu- tational Linguistics. doi: 10.1 10.18853//122223.cc-h wwt.2222...a--ooo..22. URL https: attpss:/acllantholol
284
17

---

Original Word Predicted Value Published as a conference paper at ICLR 2025
A WORDS VS. NONWORDS ANALYSIS
To further analyze our results from Sec. 3 we conduct failure analysis, focusing on false negatives (FN) and false positives (FP). Our analysis indicates that the former-valle words misclassified as gibberish are often rare and complex words, suggesting that the model's internal vocabulary may lack representations for infrequent words. On the other hand, false positives-wher gibberish is misclassified as a word word-typpo cetp-tyiicllly involves nonwords that closely resemble valid words, likely due to shared sub-word structures. See Tab, for few examples.
True Value
Status
Unitaaiinism _killy -uadruropic
gibberish gibberish word word
word word
FN
FN
gibberish
FP
nonwith
gibberish FP
Table 2: Examples of false negative (FN) and false positive (FP) for the word vs. nonword experi- ment (Sec. 3).
We further conduct additional experiments to compare the performance our custom nonword dataset with the iinuuitticallymmoivvaedd ARC Nonword Database (Rastle et al.] 2002), which con- tains morphologically plausible nonwords designed to resemble real words to humans. Our re- sults (Fig. 7 indicate higher differentiability on the ARC dataset compared
our own dataset, which highlights that our nonword creation procedure effectively mitigates potential biases. This result strengthens our conclusions: the last token provides significant cues for distinguishing words from nonwords, particularly in the middle layers of the model after several layers of processing.
Accuracy Layer
0.
0.8
Accuracy
07
06
0.5
ARC Nomword Database (Last Token)
Token)
Our dataset (Penultimate Token))
10 15
20
25
30
Layer
Figure 7: Accuracy comparison by layer for distinguishing words from nonwords. The ARC Non- word Database (blue line) exhibits the highest accuracy across layers, potentially due to its mor- phologically plausible nonwords. In contrast, our dataset (orange line, last token) achieves slightly lower accuracy, demonstrating the effectiveness of our bias-mitigated nonword generation process. The penultimate token from our dataset (green line) shows significantly lower accuracy, highlighting the importance of the last token in distinguishing words from nonwords.
In another experiment, we evaluate the penultimate token representation against nonword tokens for words of length three more tokens. For example, in word like "unhappiness," we use the inner token "h" (c.g., am-h-appiness). Unlike the final tokens, penultimate tokens are poorly distinguish- able from nonwords, achieving significantly lower accuracies. These results suggest that the model's
18

---

Published as a conference paper at ICLR 2025
ability to separate words from nonwords is highly dependent the position and completeness of token representations.
Finally, expand the comparison of our dataset across three additional moddls--Llama3--e Mistral-7B, and Yi-6B. Our results across these models (Fig. ( mirror the trends observed with Llama2-7B, with aceuracy improving layers deepened but peaking and gradually declining in later layers. This consistency further supports our findings that the model's ability to classify words vs. nonwords relies on nuanced patterns in internal token representations, unaffected by co- occurrence biases or morphological artifacts.
Moan True Status by Layer for Different Models
LLaMA2-78
Mistral-78
0.7
Actuniyy
06
0.5
5
10
15 20 25 30
Figure 8: Mean true status accuracy by layer for distinguishing words from nonwords across mul- tiple models (Llama2-7BB Llama3-8B, Mistral-7B, Yi-6B) using our dataset. All models exhibit similar trends: accuracy improves across initial layers, peaks in the middle layers, and declines in deeper layers. This pattern demonstrates the robustness our findings across different model archi- tectures and sizes.
B STATISTICAL ANALYSIS OF TOKEN AGGREGATION IN MULTI-TOKEN WORDS
We repeat the multi-token experiments Sec. 5.2 for 3- and 4-token words. Our results (Fig. 9 show a very similar trend to 2-token words (Fig.
We further present the detailed statistical analysis conducted of experiments in Sec. 5.2 amine token aggregation in multi-token words compared to single-token words (control group). The objective is investigate whether the model exhibits significantly different attention patterns between these groups, indicative of the detokenization process.
To determine significance, perform two one-sided t-tests per layer for the Llama2-7B case: (1) Testing whether attention to prefix tokens in multi-token words is higher than to previous tokens in single-token words. (2) Testing whether attention to previous tokens in single-token tokenn-okeen words is higher than to prefix tokens in multi-token words. Table shows the p-values and significance levels for each layer. Significance levels are denoted as ns (not significant), and *** (p < 0.001).
The results reveal that in layers and 2, attention to prefix tokens in multi-token words is signifi- cantly higher than in single-token words, suggesting the early phase of Token Aggregation. From Layers 3 to 17, attention to single-token words is higher, indicating shift in attention focus from prefix attention in relation regular close tokens. Notably, there are intermittent increases in attention to prefix tokens at Layers 18, 21, 25, 27, 29, and 30, possibly signaling transfer into prediction ensembling phase which the attention in general is less important (Ben Artzy & Schwartz 2024) and therefore don't coherent pattern.
19

---

Published as a conference paper at ICLR 2025
Layer
p-value (Multi
Single)
p-value (Single Multi)
Significance
0
9.999 x 10
5.891 x 10-5
Single-token > Multi-token (***)
1
0.000
1.000
Multi-token >> Single-token (***)
2
0,000
1.000
Multi-token > Single-token (*#*)
3
1.000
1 1.252 x 10
Single-token > Multi-token (*e*)
4
1.000
0,000
Single-token Multi-token (***)
5
1,000
4.466 x 10
Single-token Multi-token (*4*)
6
1.000
0.000
Single-token Multi-token (*#0)
7
1.000
0.000
Single-token Multi-token (***)
8
1,000
0.000
Single-token Multi-token (***)
9
1,000
0.000
Single-token Multi-token (***)
10
1.000
0.000
Single-token Multi-token (*##)
11
1.000
7.847 10 10-87
Single-token Multi-token (*40)
12
1,000
6.307 X 10 10-309
Single-token > Multi-token (*4*²)
13
1.000
1.417 X 10 255
Single-token Multi-token (*0*)
14
1.000
2.397 x 10 10-172
Single-token > Multi-token (*4#2)
15
1,000
0,000
Single-token > Multi-token (#4:2)
16
1.000
5.857 x 10
Single-token > Multi-token (*9*)
17
1,000
1,414 X 10
Single-token > Multi-token (#4*)
18
5.646 10-5
9.999 X 10-1
Multi-token Single-token (***)
19
1.000
7.919 10-161
Single-token Multi-token (*e0)
20
1,000
2.036 x 286
Single-token > Multi-token (*4*))
21
1 1.399 10-190
1.000
Multi-token Single-token (*0*)
22
1.000
3.647 10 10-44
Single-token > Multi-token (*4#2
23
1.000
6.008 x 10-13
Single-token > Multi-token (*0*)
24
1,000
1.062 10-15
Single-token Multi-token (***)
25
5.274 10 10-68
1.000
Multi-token Single-token (#4#)
26
1.000
3.670 10
Single-token > Multi-token (***)
27
0.000
1,000
Multi tokti-token >> Single-token (***)
28
1.000
2.446 20
Single-token > Multi-token (*e0)
29
1.438 x 10 202
1.000
Multi-token >> Single-token (***)
30
9.506 x 10
1.000
Multi-token > Single-token (*e*)
31
6.731 10
3.270 x 10
Not significant (ns)
Table 3: Results of one-sided t-tests comparing attention weights between multi-token and single- token words across layers over Llama2-7B model.
(a) Llama2-7B Attention weights for 3-tokens words
(b) Llama2-7B Attention weights for 4-tokens words
0.07
0.05
Average attention waight
Avarage attantion waight
0.06
0.05
0.04
0.04
0.03
0.03
0.02
0.02
0.01
0.01
15
30
20
Layer
Layer
Figure 9: Analysis for 3- and 4-token words for Llama2-7B. The higher attention pattern in lay- 1-2, while lower values are observed afterwards, is consistent with our results for 2-token words (Fig. 5)
In conclusion, the attention mechanism differs significantly between multi-token and single-token words, indicating that the detokenization process involves initial amplification of attention to sub- word tokens followed by reduction the model obtains whole-word representations.
20

---

Published as a conference paper at ICLR 2025
C WORD RETRIEVAL FOR SINGLE-TOKEN AND MULTI-TOKEN WORDS ACROSS MODELS
This section presents detailed comparison of word retrieval performance (Sec. across several models (Llama2-7B, Llama3-8B, Yi-6B, and Mistral-7B) for both single-token and multi-token words. The evaluation focuses how effectively each model retrieves the original word across lay- ers, especially in challenging cases like artificially separated single-token words, typos, and multi- token words.
Across all experiments (Fig. T0p. we observe a similar trend where the retrieval rate increases over the first several layers, peaks around the middle layers, and then decreases in the later layers. The main difference across models lies in the peak performance, especially in cases involving typos and multi-token words, where more advanced models such as Llama3-8B and Mistral-7B demonstrate superior performance in reconstructing the original word representations.
Lugil Lens Seperaled Tokae Words
Logil Leni Of Typus Token Words
Poliul iecopes Mulli Tohens Words
Liamaz
Mistral
0.8
0.5
0.8
Rata Orginal word Fodr
08
0.8
0.8
94
9.4
Liama3 Mutrrl
30
30
Layor
Layer
Layer
Figure 10: Layer-wise word retrieval rates for single-token and multi-token words across all models.
C.1 CUMULATIVE WORD RETRIEVAL FOR SINGLE-TORER AND MULTI-TOKEN WORDS
Fig. [1 shows the results of cumulative word retrieval (Sec. across various models, focusing on both single-token and multi-token words. For each model, we analyze the ability of the LLM to retrieve the original word from sub-word tokens across its layers. The cumulative retrieval is calculated as the proportion words that are successfully retrieved at each layer, with the percentage increasing more words are recovered throughout the model's 's layers.
Single-token words The cumulative word retrieval for single-token words-those that are artifi- cially split into sub-word tokens (via typographical errors or manual splits)-shows a rapid increase in retrieval success the early layers. For Llama2-7B, for instance, cumulative retrieval reaches 93.2% for words split by manual intervention, and 66% for words affected by typos by the middle layers. This pattern is observed across models, with retrieval generally peaking around layers 15-20.
Multi-token words For multi-token words, which are naturally split due to being out-of- vocabulary for the tokenizer, the cumulative retrieval process follows similar trajectory. However, in models like Llama2-7B, the retrieval peaks earlier in the model, with a cumulative retrieval rate of 77.41% %. Other models like Llama3 and Yi show higher cumulative retrieval rates, suggesting improved efficiency handling multi-token words, potentially due to larger model capacities and internal dictionaries.
21

---

Published as a conference paper at ICLR 2025
Canodctio mand Txpecked Mood
Canadii hall "yoe N6c Pusk
Lear
Lowe
Figure 11:
Cumulative word retrieval for single-token and multi-token words across all models.
The similarity in cumulative retrieval between single-token and multi-token words suggests that LLMs treat out-of-vocabulary words in a manner similar to sub-word-tokenized words, accessing a latent vocabulary reconstruct full word representations.
D INTRODUCING TYPOS FOR SINGLE-TOKER WORDS
In this section, we describe the process of introducing typos into single-token words to split them into multiple tokens 1. The modification applies to words longer than four characters and involves randomly performing one three operations: substituting two characters, deleting a char- acter, or inserting new character. By introducing these slight variations, the word becomes un- familiar to the tokenizer, causing it to be divided into multiple smaller tokens during tokenization. Particularly, this process results in splitting words into 2-5 tokens. Table 4 shows examples of the different splits.
Description
perturbed
new tokens
Substitution of two characters
develpoment develoment develfopment
['de', 'vel', 'p', 'oment'] ['de". 'vel', 'oment'] ('dev", 'elf", 'op'. 'ment']
Deletion one character Insertion of one character
Table 4: Examples of the different typos we consider, exemplified by perturbing the single-token word "development" (Sec. 4.1.
E ABLATION EXPERIMENT ON SUFFIX-SPLIT WORDS
In Sec. 5 test the role of FFN layers detokenization, we conduct an intervention- experi- ment measuring how ablations to FFN updates affect word retrieval. We run this experiment on all single-token words from WIKITEXT-101 that end with one of three common suffixes: "ing." "ion," or "est". Each word is then artificially split to two parts the root word and the suffix. For example, the word "eating" " is split into "eat" and "ing". while "connection" " is split into "connect" and "ion". This ensures that processing the suffix token is necessary reconstruct the identity of the word, and reduces possible effects of strong distributional artifacts of token co-occurrence.
Using logit lens, identify FFN layers where the update to the residual stream decoded the original single-token word. We then ablate these layers by zeroing out their updates to the residual stream. As control, ablate the same proportion (5%) of random FFN layers.
Our results, shown in Fig. 12 reveal that ablating the identified FFN layers lead to sharp drop in word retrieval rates, effectively disrupting the detokenization process. In contrast, ablating random layers has little to no effect on retrieval accuracy. This indicates that the FFN layers play a critical role in reconstructing word representations rather than simply enhancing contextualization.
22

---

Published as a conference paper at ICLR 2025
Ablated FFN regular retrieval using logit lens
0.8
0.6
Rate of Retrivval
ablated
regular
0.4
control
0.2
0.0
5 10
15
20
25
30
Layer
Comparison of retrieval rates using logit lens for suffix-split words under three condi- tions: ablation of identified FFN layers (blue line), regular retrieval (orange line), and control with random FFN layer ablation (green line). Ablation critical layers causes a sharp drop retrieval accuracy, highlighting their importance in detokenization.
Figure 12:
F COMPARISION BETWEEN LOGIT LENS AND COSINE SIMILARITY
To validate our use of the logit lens, we repeat the artificial split-word experiment using cosine similarity. We use the same setting artifical splits based on suffixes detailed above in App.
For the logit lens, we adapt the standard imple- mentation measure similarity between the hid- den representation of the final token and the in- put embedding space (vocabulary space), rather than the output embedding space typically used to predict the next token. This adjustment allows us to directly evaluate the alignment between the hidden states and the embeddings of the original words in the vocabulary. In parallel, cosine sim- ilarity is used to compute the similarity between the same hidden states and the embeddings of the original words in the vocabulary space.
with (ing, est)
34
logi lens siilarty
Our results, shown in Fig. 13. demonstrate that both methods produce nearly identical patterns. In both cases, retrieval accuracy peaks in the mid- dle layers, where the hidden representation of the suffix token aligns most closely with the original word. Beyond these middle layers, retrieval ac- curacy declines, likely reflecting the model's shift toward representing the prediction of the next to- ken rather than maintaining the full representa- tion of the current word.
20
15
20
25
30
Figure 13: Comparison of retrieval accuracy for split-word experiments using either logit lens (blue line) and cosine similarity (orange line) across model layers.
These findings reinforce the validity of using the logit lens when adapted to the input embedding space, streamlined approach for analyzing model representations. This approach results in similar trends to cosine similarity while offering a more interpretable and direct framework for measuring alignment with the vocabulary space.
23

---

Published as a conference paper at ICLR 2025
G LEARNING THE LINEAR MAPS TE. AND TEu
To expand the model's vocabulary without modifying its core parameters, we construct linear trans- formations that map hidden states at different layers the model's embedding and unembedding spaces. By learning these transformations solely from the model's existing vocabulary, we can infer new token representations without modifying any of the model's core weights. This section details our method for learning these mappings.
G. EXTRACTING MODEL REPRESENTATIONS
Given a pretrained language model with input embedding matrix E E RVdd and output unembed- ding (LM head) matrix U E RVxd where V is the vocabulary size and d is the hidden dimension, we extract the representations used for learning the transformations as follows:
We extract the model'' embedding matrix E and LM head matrix before making any modifications to the vocabulary.
Embedding and Unembedding Matrices:
Hidden States Across Layers: For each token t in the model's original vocabulary, we pass t as a single-token input and record its hidden state at every layer of the model. Let hz(t) denote the hidden state token & layer L.
G.2 LEARNING THE LINEAR MAPPINGS
For each layer E, aim to learn two linear transformations:
Te, that maps hidden states to input embeddings.
Te that maps hidden states unembedding representations.
We learn these mappings as orthogonal Procrustes problems (Schönemann 1966). which seek to find the best orthogonal transformation aligning two sets of vectors. Specifically, for each layer t, solve:
T = arg min T
[[The(t) - eell?, subject to TTT=I
LEV
TE = arg min Σ
[The(t) - u1l?, -ull subject to TT
(2)
LEV
where e, and 2, are the embedding and unembedding vectors of token t, respectively. The con- straints enforce that each transformation preserves distances and does not distort the structure of the space. our experiments, we use the Python implementation of Meng al. (2022a).
G.3 NORMALIZATION WITH RMS SCALING
To preserve the relative magnitudes of embedding and unembedding entries, normalize all rep- resentations using their root mean square (RMS) norm (Zhang & Sennrich 2019). Specifically:
1. Preprocessing Training Representations: Before fitting the Procrustes transformations. we normalize all hidden states, embeddings, and unembedding vectors by dividing each vector æ by its RMS norm:
Unoem
||a||RMS
where
[ar||RMMs
d
2. Applying the Learned Maps: When using the trained transformations on new detokenized representations apply the following steps:
(a) Normalize by its RMS norm:
Tnorm TT
(b) Apply the learned transformation: è = T Enorm-
norm- î = T' Toom.
(c) Rescale by the mean RMS of the target space:
Billee||RMA
ú. EB[| ||RMM]
24

---

Published as a conference paper at ICLR 2025
This ensures that the embeddings and unembeddings maintain scale consistent with the orig- inal vocabulary.
H EFFICIENCY GAINS FROM VOCABULARY EXPANSION
Beyond improving model performance on newly added words, our vocabulary expansion method rectly reduces the number of tokens required to encode input text, and could lead to further potential efficiency gains in inference.
Table 5 summarizes the average reduc- tions in sequence length achieved by en- coding texts from each domain using the expanded vocabulary instead of the origi- nal vocabulary in the three datasets. Us- ing our method, we attempt to expand Llama2-7B's vocabulary with all multi- token words appearing at least m times in the test set, where we use m 1, 5, 50 for WIKITEXT-103, PUBMED, and Ara- WIKI40B, respectively.
Attempted
New Words
Token Reduction
WIKITEXT-101 PUBMED
14.1k
10.1k
10.5%
9.5k
5.4k
13.5%
Wix140B-Arabic
Table Reduction average sequence length when encoding text with the expanded vocabulary for Llama2-7B. # Attempted the number of multi-token words tested for expansion, while New Words is those detected as detokenized using Patchscopes.
We find that the token reduction rates depend on the success rate of extracting detokenized represen- tations using Patchscopes (which in turn depend on the textual domain and language). WIKITEXT- 103 achieves 10.5% reduction in token count, with 72.9% of attempted words successfully verted into new vocabulary entries. PUBMED shows higher token savings rate (13.5%) despite a lower success rate (58.3%), its vocabulary expansion targets terms. The largest efficiency gain is observed in Arabic WIK140B, where encoding with the ex- panded vocabulary reduces token count by 14.5%, highlighting the method's potential for languages with inherently longer token sequences. However, the success rate of identifying detokenized rep- resentations in WIK140B is lower (16.0%), suggesting room for improvement in expanding vocab- ularies for morphologically rich languages, and suggesting many Arabic words are not stored in the model's s inner lexicon.
domain-specing multi-token
These results suggest that post-hoc vocabulary expansion can significantly reduce the computational cost of inference, particularly for non-English languages and domain domain-specific texts, without requir- ing any modifications to the model's core parameters.
25


Reviews:
[{"review_id": "hmShQyhCZm", "reviewer": "Reviewer_rVMS", "summary": "This paper explores the process in which models transform tokens, which often split long words into subwords (e.g., \"un\" \"h\" \"appiness\"), into higher level representations for the full word through \"detokenization\". Detokenization has been observed in LMs before, but has not been directly studied extensively. This work shows that LMs can recognize when a word is part of a larger word, and show that early attention fuses subwords together (in the last token of the word), and uses early MLP layers to then recall the full word from multiple subwords in an \"internal dictionary\" (e.g., representing \"unhappiness\" as a single vector internally even though it is not in the tokenizer). The authors then show that this can be used to expand a model's tokenizer by including the hidden 'internal dictionary' representation as an input token. This works to some extent.\n\nOverall, this paper enhances our understanding of early layer processing in language models, and provides a path towards enhancing models to reduce inference time.", "questions": "If the model wants to generate some multitoken word that it represents in its 'internal dictionary' is it \"planning\" multiple tokens ahead? Why or why not?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper answers particular unanswered questions surrounding \"detokenization\", which has been repeatedly observed and discussed without being properly studied. These are important for observations around, for example, stages of inference in language models.\n\nInterpretability results on early layers of LMs are often lacking, as vocab projections are much easier to perform at later layers. This work provides interesting and convincing results for one role early layers take on in these models, which is indeed different from the roles of later layers.\n\nThe vocab expansion experiments are a nice proof of concept, and could be expanded on in the future to decrease inference times.\n\nThe results on typos are interesting and to my knowledge, novel", "weaknesses": "The evidence for a third stage of processing in Figure 2b is a little sparse. These results are only for one model, and the degree to which accuracy drops is not substantial enough to obviously be due to a difference in processing altogether. These results could be made stronger by including results for more models. As a motivating example, it is fine, but perhaps isn't the best use of that space if this point can't be made more strongly.\n\nTypos:\n\nL370: \"form\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2FiQWuVlL", "reviewer": "Reviewer_ChVq", "summary": "This paper analyzes the process of latent detokenization inside the transformer-based LM forward pass, as it occurs across network layers. It shows that models are able to recognize words from pretraining even when they are noised with slight token variations or artificially split across multiple tokens. These experiments are different from earlier works, but ultimately show very similar findings about hierarchical processing in transformers. Using these findings, a novel method is briefly introduced to leverage the internal states of merged tokens to automatically expand the token vocabulary, which can hypothetically improve inference costs with fewer lookups. This method appears initially effective, but could be explored more.", "questions": "1. How were tokens assembled into nonwords in sec 3? I am missing detail here which could be useful in understanding the method. I also do not understand what it means to \"fit\" a KNN classifier (which is non-parametric) -- were there representations used which were different from those taken from the model hidden states?\n2. There was a claim made that the proposed method in section 6 can improve inference-time costs, though I cannot find any experiments or numbers for this in the paper. Can the authors point me to or provide any information about this? Thank you.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper analyzes the process of detokenization across transformer network layers via a series of targeted experiments. It builds an intuitive understanding that agrees with many prior works in layer-based analysis.\n\n2. The paper proposes an interesting method for training-free expansion of the model vocabulary by leveraging the insights into internal word representations. This method is shown to be effective in limited experiments. See below in \"weaknesses\" for further thoughts on this.\n\n3. The writing is clear, but sometimes too abstract (see weakness 5).\n\nThis paper shows very solid work and I greatly appreciate the thorough breadth of exploration, though it could possibly be more effective to focus on fewer areas. I want to emphasize that I enjoyed reading the paper and believe it will be strong after some revision, including reworking the claims and focusing more on the novel contributions which are begun later in the paper. I believe it would be more impactful to explore sec 6 in more depth; see weakness 4 below.", "weaknesses": "1. The concept of an inner lexicon is interesting, but not novel as is claimed in this work. The idea follows implicitly from prior work in the memorization of training data, and explicitly in works about tokenization, such as the introduction of BPE (which is discussed greatly in this paper). It is the stated goal of subword tokenizers to enable learning a vocabulary of words and concepts which is larger than the vocabulary of concrete tokens through the process of token combination. It is nice to see these findings reproduced and analyzed, but they are not new.\n\n2. The experiment in section 3, which motivates the idea of an inner lexicon, is not very strongly designed. Why are nonwords created by randomizing tokens, and not by some other method on the morphological level or otherwise something more linguistically motivated? Resulting nonwords do not seem to follow English conventional morphology (eg. the nonword \"chha\") and this could make it trivial to distinguish words from nonwords. Prior work has shown LLM sensitivity to word frequency in training corpora, and this experiment seems to reproduce those findings. This experiment seems to me to show that LLMs can distinguish easy cases such as \"chha\" which are very dissimilar to real words, and predictably struggles with more difficult cases that more closely resemble real words (see appendix) but there doesn't seem to be strong evidence that the LLM representation is doing more than locating words on a gradient based on their prior likelihood of appearing in the pretraining corpus. This fact is fairly well established at this point.\n\n3. The experiments in the paper seem mostly sound and reasonable, but their novelty is overstated. Several of the earlier experiments in particular build on each other to show that early and intermediate layers in the network are responsible for aggregating and disambiguating word representations (sec 4 and 5). However, these findings may be seen to be subsumed by many prior works in the analysis of syntactic and semantic composition of tokens across transformer layers (see section 4 in [1] for many citations).\n\n4. The paper may have been too ambitious in scope. The first several experiments were good reproductions of findings. The last experiment was novel to me, and it would have been interesting to expand on it more deeply. However, it did not require many of the earlier experiments in order to understand it, which took up most of the room in the paper. Other reviewers may have different opinions, but mine is that the paper would be more valuable if it explored the final research question more deeply, and provided more concrete findings for it. For example, can we estimate a size/contents of the inner lexicon? Does this lexicon scale with model capacity and/or training size? Can we provide some guarantees or estimates about the boundaries of the method of finetuning-free vocabulary expansion? For what kinds of words is this method effective and when is it ineffective?\n\n5. There were many smaller experiments given in the paper, and this resulted in important implementation details being often omitted. For example, experiments often hinge on model memory of tokens from training, and the natural distributions of those tokens in the corpora, but details about how words/tokens were sampled in the tests (such as construction of nonwords) were not often given in enough detail to reproduce experiments. I would expect there to be significant influence of such distributions on test outcomes, so these details are important.\n\n\n[1] Anna Rogers, Olga Kovaleva, Anna Rumshisky; A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTOp0sG5Qx", "reviewer": "Reviewer_NB7M", "summary": "In this paper, the authors investigate how LMs internally reconstruct word-level representations from sub-word tokens, a process they term \"detokenization\". They provide evidence that Lms can inherently combine sub-words into hidden representations which can be mapped into coherent words, even for out-of-vocabulary items, across the early to middle model layers. By probing LMs on both known words and artificial nonwords, they show that the model forms distinct representations for these categories, suggesting an \"inner lexicon\" that extends beyond tokenized inputs. The findings reveal that this detokenization mechanism leverages feedforward layers and attention patterns to generate whole word representations, which could, in theory, improve vocabulary flexibility without finetuning (though this is not shown in practice).", "questions": "* The experiments in 4.1 focus on logit lens. What about cosine similarity or more direct measrues of similarity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper addresses a crucial question: how can language models construct symbolic representations of entire words when their input comes from tokenizers that often fragment words in ways that disregard their morphological structure? Specifically, the authors investigate whether LMs internally form representations of morphological units that help bridge the gap between the tokenized input and the naturally holistic nature of words in language. Through experiments, the paper presents some evidence that whole-word representations emerge within the model’s hidden states, even when it processes fragmented word tokens. Additionally, the writing is clear, and the experiments are easy to replicate.", "weaknesses": "I believe there is a disparity between the paper’s claims and the experimental evidence provided to support them. Specifically, some of the experiments lend themselves to alternative interpretations, which could be clarified with additional baselines or experiments. The paper claims is that model come up with an \"internal lexicon\" that create hidden representations of \"virtual\" words, even when fed, e.g., word pieces as input. This is a claim on the computation carried out by the model, i.e., it is implied that there are some modules whose explicit computation is forming this internal lexicon. I am not sure that the experiments provide sufficient evidence for this claim:\n\n* First, the \"motivating experiment\" in Section 3 lacks sufficient controls. The authors demonstrate that there is a linear separation in the hidden state between the representations of actual multi-token words and fictional ones created by randomly mixing word pieces. However, this separation could simply reflect the model's ability to distinguish between linguistically valid English morphology and nonconforming sequences, rather than providing evidence of \"internal detokenization.\" For instance, an alternative hypothesis is that the model has learned distributional cues—such as suffixes like \"ing\" rarely appearing at the beginning of a word—which causes out-of-distribution effects in the hidden states when encountering atypical token sequences.\n\n* In Section 4.1, the authors hypothesize that \"if the model performs detokenization, it will represent the last token of a word similarly to the original word token.\" However, even if such similarity is observed, it could be attributed to the distributional properties of language rather than any explicit \"detokenization\" process. For instance, in the example provided in the paper where \"cats\" is split into \"ca\" and \"ts,\" it is plausible that the pretraining corpus contains instances where this split occurs unnaturally, such as in URLs like \"catsanddogs.com\" (an actual website) or in cases with typos. Such occurrences might push the representation of \"ca ts\" closer to that of \"cats\" without requiring an explicit detokenization step. Furthermore, it is known that such similarities exists also in word2vec methods like Glove, and it is difficult to argue that any explicit detokenization happens there. \n\n* In Section 4.2, the authors feed the hidden state of the last token of a multi-token word into the model and prompt it to repeat the word. Instances where the model accurately reproduces the entire word are taken as evidence that it has stored the multi-token word in an \"internal lexicon.\" However, a key baseline is missing: including phrases that are not single words, such as \"repeat this word: rainy day.\" The observed results could simply reflect the model's tendency to form contextualized representations that transfer information across tokens, rather than indicating an internalized whole-word representation.\n\n* Finally, the paper’s closing sections aim to illuminate the model's internal computations and the supposed formation of an internal lexicon. While the results provide some evidence of contextualization in the feedforward layers, it's unclear to me whether they genuinely support the existence of an internal detokenization process. Intervention-based experiments could strengthen this claim. For example, could we identify a subset of parameters where ablation specifically impairs performance on multi-token words without affecting single-token words? Or could linear concept erasure techniques reveal a subspace whose neutralization removes all distinctions between multi-token and single-token representations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9HFRRLkMfg", "reviewer": "Reviewer_t1G5", "summary": "The paper studies how subwords are detokenized through transformers into the original word for further processing and understanding (where LLM is able to distinguish between words and non-words are shown are shown in a preliminary study in this work). In this research direction, the paper makes the following contributions:\n- The paper shows that detoenization process happens in the beginning-middle layers using techniques using techniques such as logic lens (single token) and patchscope (multi-token)\n- The paper then carry on experiments suggesting that the detokenization happens within FFN layers\n- Leveraging the above results, the paper shows that that transformer efficiency can be enhanced by introducing \"decodable\" token embeddings; the paper examines both input embeddings and output embeddings.", "questions": "For section 5, the feedforward mechanism, why only FFN output are measured please? Does it make sense to also measure the residual part please?\nFor section 6:\n- For all original vocabulary tokens, all models perform less well than the model with original vocabulary? Are there examples to illustrate these and some hints where models fall short?\n- What would be the full newly token accuracy for the model with original vocabulary?\n- With such techniques, what would be an estimated inference speed gain? For input embedding as well as for output embedding?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a significant amount of content and materials while being easy to follow. The paper incorporates appropriately the related works so that it is relatively straightforward to situate the paper in the literature. Concretely, I think the paper has made the following contributions:\n- Through techniques such as logic lens and patchscope, the paper demonstrates convincingly where the model performs the detokenization process by presenting clearly such studies.\n- The paper shows that FFN serves to combine subword information in section 5.\n- In the final section, the paper shows how the understanding can help transformer decoding in practice. The paper adds word embeddings both in input matrix and output matrix and show that the model can accelerate the inference while maintaining a good performance.", "weaknesses": "While the paper presents a complete study (with no missing component) in the detokenization study, it feels that the paper can still be further enhanced with some more in-depth studies, some of the questions I have put in the questions section but in general:\n- The cumulative curve shows that FFN indeed contributes to detokenization. What about other components? Are there any hints/patterns that the authors observe in the detokenization process (e.g. what words are first detokenized)?\n- Cumulative rate saturates at around 0.7 shown in the figure. What about the rest 30%? Are these limitations for the measured model? Do better models (e.g. llama3) perform better at these?\n- More details will help section 6 and I list some of them in the questions section. I think these are just some of the questions that a common reader would have after reading the paper. I think the results in this section may be of practical importance and deserve to be enhanced with more empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)