Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
Authors: Brian Siyuan Zheng
Alisa Liu
Orevaoghene Ahia
Jonathan Hayase
Yejin Choi
Noah A. Smith
Venue: NeurIPS 2025 spotlight ()
Keywords: tokenization
language models
robustness
Primary area: evaluation
TL;DR: Language models are surprisingly robust to non-canonical tokenizations of the input, which can even lead to improved performance

Abstract: Modern tokenizers employ deterministic algorithms to map text into a single ``canonical" token sequence, yet the same string can be encoded as many non-canonical tokenizations using the language model vocabulary, including tokenizing by character. In this paper, we investigate the robustness of LMs to input encoded with non-canonical tokenizations entirely unseen during training. Surprisingly, when evaluated across 20 benchmarks, we find that instruction-tuned models retain up to 93.4\% of their original performance when given a randomly sampled tokenization, and 90.8\% with character-level tokenization.  We find that overall stronger models tend to be more robust, and that robustness diminishes as the tokenization departs farther from the canonical form.  Motivated by these results, we identify settings where non-canonical tokenization schemes can \textit{improve} performance, finding that character‑level segmentation improves string manipulation and code understanding tasks by up to 15\%, and right‑aligned digit grouping enhances large‑number arithmetic by over 33\%. Finally, we investigate the source of this robustness, finding that it arises in the instruction-tuning phase. We provide evidence that both base and post-trained models grasp the semantics of non-canonical tokenizations (perceiving them as containing misspellings). However, base models try to mimic the imagined mistakes and degenerate into nonsensical output, while post-trained models are committed to fluent responses. Overall, our findings suggest that models are less committed to their tokenizer than previously believed, and highlight the promise of intervening on tokenization at inference time to boost language model performance.

Broken Tokens? Your Language Model can Tokenizations
Secretly Handle No
n
Ca
noni
cal
Brian Siyuan Zheng v
Alisa Liu
Orevaoghene Ahia Noah A. Smith +
Jonathan Hayase
Yejin Choi
University of Washington
Allen Institute for Al
Stanford University
zhengbrêcs. washington.co edu edu
Abstract
Modern tokenizers employ deterministic algorithms to map text into a single "canonical" token sequence, yet the same string can be encoded many non- canonical tokenizations using the tokenizer vocabulary. this work, we investigate the robustness of LMs to text encoded with non-canonical tokenizations entirely unseen during training. Surprisingly, when evaluated across 20 benchmarks, we find that instruction-tune tunde models retain up to 93.4% of their original performance when given a randomly sampled tokenization, and 90.8% with character-level tokenization. We that overall stronger models tend to be more robust, and robustness diminishes as the tokenization departs farther from the canonical form. Motivated by these results, then identify settings where non-canonical tokeniza- tion schemes can improve performance, finding that character-level segmentation improves string manipulation and code understanding tasks by up +14%, and right-aligned digit grouping enhances large-number arithmetic by +33%. Finally, we investigate the source of this robustness, finding that it arises in the instruction- tuning phase. We show that while both base and post-trained models grasp the semantics of canoncall tokenizations (perceiving them as containing mis- spellings), base models try to mimic the imagined mistakes and degenerate into nonsensical output, while post-trained models are committed fluent responses. Overall, our findings suggest that models are less tied to their tokenizer than pre- viously believed, and demonstrate the promise of intervening on tokenization inference time to boost performance
Introduction
Tokenizers segment text into sequence of discrete tokens in the language model's (LM) vocabulary. Most of today's LMs use deterministic subword tokenization, which produces single canonical token sequence for given piece of text, and further, for each whitespace-delimited word. One commonly discussed limitation of this approach is that, by mapping byte strings to symbolic token IDs, the orthographic makeup of tokens is obscured harmful for LM understanding of numbers [47 64 61 and morphologically rich languages 2 25 and has motivated efforts to model text directly at the byte level 12 70
the LM [22 18). This can be especially
To shed more light on this perceived limitation, in this work we study whether LMs can adapt at inference time, without any additional training, to different tokenization scheme than the one they were trained with. While the tokenizer deterministically outputs canonical tokenization of any text into tokens (usually by applying an ordered list of merge rules), non-canonical tokenizations of the same text using the same vocabulary are generally possible (see example in Figure Here,
Code is available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Canonical Tokenization
Describg
Seattle
Meather
Seattle
Meather
being Herr
Original
Rondom
Quite
unique
and
Cemper
owental
Multiple Choice Benchmarks
general
overvien
Accumacy
Altemative Tokenization
Random
C Sea
000
Hic Chatenge
cont
imnggred
Commenternaaal
Seattle
weather
being
quite
itpredictabee
and overvien
change
able
Here
seneral
Short Answer Benchmarks
Accundry
0
C
1
3
8
x
0
Seattle
5
weather
15
known
for
being
100
MMTTH
CUTF
DROO
qui
uuique]
and
change
able
Here
general
Figure I: Left: An example of how LLAMA- 3.1-..--1......... ----INSRUCTT responds when given canonically tokenized imput, versus random tokenization and character-level tokenization. The responses are surprisingly similar, demonstrating their ability to handle non-canonical tokenizations. Moreover, LMs generally respond with correctly tokenized output regardless of the tokenization scheme used for the context. Right: Performance of QWEN- 2.5-7B-INSTRUS across various benchmarks and tokenization schemes. Much of the original performance is preserved when given non-canonical tokenizations.
we evaluate how LMs trained with deterministic tokenizers behave when given non-canonical tokenizations of text. Surprisingly, we find that instruction-tuuned LMs across many model famiiia are extremely robust to non-canonical tokenizations ($2). For example, when evaluated across 20 benchmarks, QWEN-2.5-7B- --NT-BIII retains 93.4% of its original performance when presented with random non-canonical tokenization, and 90.8% when presented with character-level tokens (see Figure l). Thus, far from not understanding the makeup of their tokens, LMs are able to compose token sequences in entirely new ways at inference time [33
This leads an intriguing possibility: LMS can process non-canonical tokenizations, can use different tokenization schemes at inference time to improve performance? For instance, prior work has found that better segmentation of large numbers can improve accuracy on arithmetic [6] [56]. Indeed, we identify several settings where non-canonical tokenization schemes improve performance for LLAMA-3. I-NNSTRUCT (3): character-level tokenization brings up to + +14% improvement on string manipulation and code understanding tasks, perhaps by granting LMs more direct access to orthographic cues. Meanwhile, right-aligned digit groups, which provide consistent grouping of digits by powers of thousand, improves arithmetic on large numbers by +33%. These performance gains are achieved without any model finetuning, pointing to the promise of tokenization as a means of inference-ting time control.
Finally, we investigate the origins of model robustness to non-canonical tokenizations multiple model families, we find that pretrained-one LMs consistently fail to produce fluent continua- tions given canconnnically tokenized context. By studying models at different stages of post- post-training. we identify that robustness arises during the supervised instruction- then ablate differences between pretraining and SFT procedures and find that the separation of the instruction and response as distinct turns of conversation is key (4.2). From here, we provide evidence for a plausible explanation: while both base and post-trained trained models grasp the semantics of non-canonical tokenizations, they also perceive them containing misspellings (54.2) Base models attempt to mimic the imagined mistakes and degenerate into nonsense, whereas post-trained models not bound by the style of the instruction and thus able to produce fluent responses.
Across
(SFT) phase (64.1). We
Overall, despite being trained with deterministic tokenization, instruction-tum commodate new tokenizations at inference time, suggesting that LMs are less constrained by their tokenizer than previously believed 45). Moreover, in settings where different representations text are beneficial, we can intervene on tokenization at inference time for performance gains. We hope our work sheds new light on the discussion of strengths and limitations of tokenization, and points to the possibility of dynamically finding the optimal representation of text after pretraining.
LMs readily ac-

---

Table Evaluated across many benchmarks, models are surprisingly robust to non-canonical tokenizations of the context. We show the absolute drop in performance when given randomly sampled non-canonical tokenization (Rand A) and character-level tokenization (Char A), relative to the canonical (Canon) tokenization. We also summarize the model's ability to retain performance across benchmarks and tokenization strategies (bottom).
QWEN-2.55 2.3-7B 1-77--ISSTRUT
LLAMA-3.1 1-B--B--BB 1-8B-INNSRRE INSTRUCT
OLMO-2-77 2-7B-INSTRUCH
Benchmark
Canon
Rand A
Char
Canon
Rand A
Char A
Canon
Rand
Char
Maltiple choice (MC)
ARC-C
86.4
-1.80
-2.60
76.2
-14.10
-22.40
77.0
-37.40
-44.0
ARC-E
94.4
-2.20
-3.60
-12.60
-21.50
-37.00
-49.00
COPA
97.0
-1.80
-7.40
97.2
-9.60
14.80
93.8
-21.40
-33.60
Winogrande
46.0
-4600
-3.00
59.6
+2.00
-5.00
58.6
-7.80
-7.00
Winograd
72.4
16.80
26.60
74.4
-9.40
-13.00
72.4
-8.60
-19.00
CSQA
85.6
-11.40
20.00
-1,60
OpenbookQA
87.2
-7.00
-7.80
82.0
-13880
200200
76.2
-30,80
-40.80
PIQA
87.0
-6.00
-14.00
84.0
-12.60
-18.40
78.2
-17.40
-25.00
MMLU
71.7
-3.70
-7.30
68.2
-11.60
-24.00
-6633
-29.10
BoolQ
84.8
-5.00
-4.80
86.2
-19,20
-17.20
71.0
-4,00
-9.20
HellaSwag
79.6
-6.20
-7.40
68.6
-14.20
-23.40
68.0
-26.80
-39.80
Short onswer (SA)
WikidataQA
81.2
-7.60
44.00
78.6
-2240
18.00
73.2
28.80
-32.20
TOFU
77.8
+0.00
-1.70
82.1
+1.70
+0.80
82.9
12.80
-23.90
TriviaQA
70.0
-1.00
-3.80
76.6
-9.0
-33660
70.0
22.20
-34,80
JeopardyQA
49.4
-5.60
-8.00
43.6
-2.20
-10.20
42,6
-21.60
-24.20
AlpucaEval
50.0
-3,70
-1.80
50.0
-5,70
-7.50
50.0
-2.10
-11.30
MATH
53.9
-2.40
-2.70
32.0
-420
-9.70
22.7
-5.20
-9.20
70.0
GSM8K
87.3
-6.10
-8.50
82.0
-11.70
-16.00
-23.10
-35,80
DROP
88.2
0.80
+0.00
88.8
-1.60
-5.00
77.0
-5.60
-7.60
Overall
Avg MC Retention (%)
92.4
89. 89.219.3
85. 6...
76.8
71.2
59.2
Avg SA Retention (%)
94.6+3.97
92.7+66.5
90.3 2..78
82. 82.7+9.05
75 16.4
65.4
Avg Overall Retention (%)
03.4
90.8
7:000
79.4.0.00
73.1
62.0g17.b
2 Language Models are Robust to Non-Canonical Tokenizations
In our main experiments, we evaluate the robustness of LMs to non-canonical tokenizations by comparing their performance on downstream tasks when given different tokenizations of the input.
2.1 Background
Most LMs today, and all the models we study, use the Byte-Pair Encoding (BPE) 58 algorithm for tokenization. The BPE tokenizer is learned by splitting corpus of text into bytes, which form the initial vocabulary, then iteratively merging the most frequent pair of tokens into a new token that is added to the vocabulary. To encode in the same order. As result, a BPE tokenizer always produces the same token sequence for the same text. Further, because BPE tokens do word is always represented with the same token or token sequence.
text, it is split into bytes, and the learned merges applied
whitespace boundaries, the
thitespace-dellmited
A natural observation is that given tokenizer vocabulary, there exist many token sequences that decode to the same text. For instance, caat could be tokenized as [- cat], [-, cat]. [-, c, at], [.., c. t], etc. In general, the number of non-canonical tokenizations grows exponentially with the length of the text. Many previous works have argued that the probability of a string should be calculated the sum of probabilities of all possible tokenizations 9 20). However, less attention has been paid to how non-canonical tokenizations affect LMs generative settings.
2.2 Setup
We consider two non-canonical tokenization schemes. (1) Random tokenization produces tokeniza- tion (uniformly at random) from the set of tokenizations more granular than the canonical one. This can be achieved by recursively splitting individual tokens into valid pair of tokens, similarly to

---

100
Benchmark
Relative RelativeChang Change
Evac
HellaSwag
90
GSMBK
MMLU
ARC Challenge
PIGA
ARC Easy
Winagrad
Boo
WinoGrende
CommonSSensaaA
TrivisQA
COPA
DROP
60-
CUTE
Meen Performence
Canonical
1.5-20
2.0-25
2.5-3.0
3.0-3.5 3.5-4.0
Length Ratio
Figure 2: Model performance generally declines as the tokenization becomes more granular. We achieve variation in tokenization length using different values of in BPE-dropout, and group tokenizations into buckets based on how many times longer it than the canonical tokenization.
660; the pseudocode and proof of correctness is provided in Appendix A (2) Character-level tokenization decomposes the string into character tokens, i.e., using no subword token from the vocabulary. For text containing only English letters and punctuation (where each character exactly one byte), this produces the most granular possible tokenization.
We consider three models, LLAMA-3. 1-8B INSTRUCT 43
1A002--B-INSTRUCE
48 and
Q [53], which we evaluate on 20 benchmarks shown in Table for further description of the datasets and evaluation setup.
Please see
BB.]
2.3
Results
Shown Table while random tokenization consistently leads to worse performance compared to the canonical tokenization, the effect is small. On average across benchmarks, QWEN-2.5 retains 93.4% of its performance when given random tokenization, followed by LLAMA-3. 87.7% and OLMO-2 at 73.1%. %. The performance drops further with character-level tokenization, with retention of 90.8%, 79.4%, and 62.0% for three models, respectively. This ranking of models in terms of retention is consistent with their ranking in absolute accuracy (under canonical tokenization), suggesting that stronger models are generally more robust to non-canonical tokenization strategies.
We also observe that all models retain performance better short answer (SA) benchmarks (where the model generates an output in free-form) compared to multiple choice (MC) benchmarks (where the model is instructed to directly output the correct answer choice). In addition, LMs consistently produce correct token sequences even when conditioning on con-canonical tokenizations. We hypothesize that, in the SA setting, models benefit from eventually conditioning recent correctly- tokenized context.
2.4 Analysis: How does granularity of the tokenization affect robustness?
We next study whether tokenization fineggrineddnss correlates in general with model robustness. We measure the mme-grainedness a given non-canonical tokenization by how many times longer it is (in tokens) than the canonical tokenization, which call the "length ratio." Finer-grained tokenizations have higher ratios, while coarser ones have ratios closer to We produce tokenizations with diverse length ratios by applying BPE dropout [52) with 0.2,,
0.2. ..., 0.9). which controls the probability with which each merge is dropped. (High leads to finer- finer-grained grained segmentations, and p 0.0 corresponds to conventional BPE.)
Figure shoows the relationship between the length ratio and the average performance retention relative to canonical tokenization, with finer-grained tokenization generally leading to worse performance. When performance retention is averaged across tasks, the negative correlation statistically significant under Kendall's with 0.003.

---

Table Examples from tasks we construct where non-canonical tokenizations lead to improved performance for LLAMA-3 1-------.--B--AA. 1-7B-INSTRUCT (3).
Counting characters: Count the number of the letter 'r' in the word strawberry.
Acronyms: Come up with sequence of words where the first letters would form this acronym: isman
Codeline Description: What does the following code (code block)
A. Counts paths from a point to reach Origin
Program check if matrix symmetric
Longest subsequence from an array of pairs having first element increasing and second element decreasing
D. Count the number of strings in an array whose distinct characters are less than equal to M
Arithmetic: 8492079913 + 4877278482 =
3 Can non-canonical tokenizations improve model performance?
If can process non-canonical tokenizations, this points the exciting possibility that tokenization schemes can be modified completely at inference-time. This would be useful if, in certain settings, there exists better representation of text than what the tokenizer produces. In this section, we develop suite tasks that intuitively require understanding the orthography of the text, and show that 1...AAA-.1-8B 1-1---BBINSS INSTRUCT performs better under non-canonical tokenization schemes.
3.1 Tasks
Please Table 2 for an example question in each task and Table B.2 for further details on dataset construction. For all tasks except Arithmetic, we use character-leva tokenization.
Counting Characters This task asks the model to count the number of occurrences of the most common letter in 5-10 character tokens in LL.MM-331 1ss vocabulary, and contains 1001 samples.
Acronyms This task asks models generate list of words whose first letters form given acronym. We construct 3594 5-letter acronyms by sampling each letter uniformly at random from the alphabet.
Code Description For a more real-world application, we construct a task where the model is given code snippet and asked to identify the function of the code in natural language from four MC options. The setup is inspired by the Codeline Description task from BIG-Bench (5), but to increase the difficulty
more complex code snippets and corresponding natural language descriptions from XLCCST [74]. To collect incorrect answers, we sample three other code descriptions from the dataset. This task contains 4800 samples across programming languages.
Arithmetic Prior work has suggested that arithmetic is difficult for LMs in part due to poor segmentation of digits [47] 64 We curate a simple arithmetic dataset by constructing addition and subtraction tasks for 10 digit numbers. Here, use different segmentation strategy. The LLAMA-3.1 tokenizer segments numbers into groups of three left-to-right (e.g., 1000000 is encoded ["100", "000", "O"J), due the pretokenization regular expression looking for matches greedily from the left. Inspired by [61], we instead segment digits into groups of three right-to-left (e.g., ["1", "000", "000"]). This task contains 1000 addition and subtraction questions in total.
3.2 Results
Shown Table in all the tasks we construct, the non-canonical tokenization strategy leads to substantially better performance compared to the canonical tokenization. In particular, we observe a + 14.33 improvement on code description and +33.7% on arithmetic. Our results show that the tokenization scheme used in training is not necessarily the optimal one at inference-time, and
5

---

Table On several tasks, LLAMA-331-8B-INNTTE
achieves better performance when using a non-canonical tokenization scheme. For the first four tasks, the input is tokenized at the character level; for Arithmetic, we segment digits into groups of three digits from right left (instead of the usual left to right). On all tasks, we observe large performance improvement from using the alternative tokenization scheme.
Task
Canonical
Alternative
A
Counting Characters
66.5
73.5
+6.99
Acronyms
49.7
57.4
+7.74
Code Description
68.6
82.9
+14.3
Arithmetic
36.5
70.2
+33.7
replacing them with intuitively meaningful tokenizations can bring substantial performance gains. We leave automatically identifying the optimal tokenization as promising direction for future work.
4 Investigating the Source of Robustness
Thus far, experiments have used post-trained "instruct" models. In this section, find that pretrained-onle models are actually unable to produce fluent continuations of unusually tokenized context ( perform ablations to identify the conditions enabling robustness ($4.2), and finally provide support for an explanation of why generative robustness arises during post-training $4.3
4.1
When does robustness appear in model training?
We first quantify the robustness models at different stages of the model development pipeline by using the OLMO2 and TULU3 (36) model families which include the base, SFT, DPO, and final instruct models. For simplicity, we focus on AlpacaEval and use character-level tokenization. For base models, we construct the prompt by placing the instruction in question-answer template (Question: [instruction] pAnswerr). We define three simple measures of generation quality.
Spelling We measure the proportion of (whitespace-deliitee)) words in the generation that can be found in collection of the top 10,000 most common English words
Grammaticality We use LonguageTool's grammar checker count the number of grammatical mistakes, which we normalize by the number of words in the generation and subtract from 1 to produce grammaticality score where higher better.
Win rate To measure overall generation quality, use alpaca eealleeallleom -pt4 as LM judge in the AlpacaEval framework and report the win rate of the generation given alternative against canonical tokenizations of the context. Unlike the previous two metrics, this measures not only the quality of the generation but also its relevance to the context.
the base models of OLMO2 and LLAMA-3.1 both unable to produce sensible output conditioned on character-level tokenizations of context, scoring at best 0.317 on spelling and 0.260 on grammaticality, Qualitatively, generations
Shown in Figure 3
extremely difficult to parse and often involve odd character substitutions and repetitions (e.g., Yoou, haviin). Despite this, they sometimes reflect an understanding of the prompt. Consider, for example,
Question: like to host guests at my home from time to time [...] Can you give me a recipe for Canjeero?
Answer: I aam glade tio hear tio hear tio hear tio hear that yoou enjoy haviin gauests yoour hoome tio keeep tio keeep keeep

---

Tulu Robustness Between Posttraining Stages
OLMo Robustness
Between Posttraining Stages
Spalling
Scelling
0.8
Grammar
Grammer
Winrate
wit
0.6
0.4
0.2
0.0
Base
SFT
DPO
Instruct
Bass
DPO
Instruct
Figure 3: Pretrained-onle models completely fail to generate coherent output conditioned on non-canonical tokenizations of context; robustness is gained in the SFT stage. We evaluate the spelling, grammaticality, and AlpacaEval win rate of model generations. Note that since TULU3 uses LLAMA-3 as the base model, base scores are computed using LLAMA-3's s base model scores.
In contrast, the post-trained models are more robust across all three metrics, with much of the improvement coming from the SFT stage alone.
4.2 Why do instruction-tuned models become robust?
We first replicate the finding from $4.1 that SFT yields robustness to non-canonical tokenizations by finetuning the LLAMA-3.2-1 1----1--BB base model on the TULU 3 SFT PERSONAS INSTRUCTION FOLLOWING dataset. Then, we perform the following interventions on the SFT training data and procedure to shed light the possible source.
Gradient over full sequence SFT on instruction instruction-response pairs conventionally uses a loss mask the instruction tokens, so that only the response tokens contribute the loss. We remove this loss mask and instead compute gradients over the entire instruction and response.
Question/answer template We replace the chat template with simple question-ansee answer template, Question: [instruction] Answer: (response). both for training and evaluation.
Removing the chat template We remove the chat template by concatenating the instruction and response without any special formatting. In evaluation, we again provide the instruction alone.
Removing the instruction After SFT training, the LM's goal is no longer to continue a given text prefix, but rather generate response the given instruction. To ablate the nature of the data itself. we take only the responses from the SFT data, and randomly split each into a new "prompt" and "response," which format with the SFT template At test time, we similarly provide an incomplete response within "instruction" tags. Since the purpose of the passage is generally inferrable from the first few words of the gold response ("Sure, here's 's a recipe for Kubdari..." "), we able evaluate generated responses under the same AlpacaEval framework.
Our results shown in Figure 4 We replicate the finding that SFT (No ablation) leads the model to be able to handle non-canonical tokenizations. This persists when computing gradients over the entire instruction and response (Full gradient) that the training procedure matches regular pretraining. Replacing the original chat template with simple question-answer template (QA template) also maintains model robustness. However, the usage of template is crucial when directly concatenating the instruction and response (Removing chat template), the model fails to produce coherent generations, with the spelling score dropping from 0.786 in the no ablation setting to 0.0698. Inserting the chat template into vvetraining-stye data (Removing the instruction) also does not yield robustness, with spelling and grammaticality scores remaining low at 0.181 and 0.158,
*We match the instruction length distribution by counting the number of tokens n in the original instruction. and formatting the first tokens of the response as the "instruction."
7

---

Ablation Experiments
Spelling
0.8
Grammar
Win rate
Score
0.6
0.2
0.0
No Ablation
Full Gradient
QA Template
Removing Chat Template
Removing
Instruction
Sequence
Figure 4: Ablations on the SFT training data and procedure indicate that the separation of the context and expected continuation as different turns of dialogue demarcated with special token is key robustness to non-canonical tokenizations.
Table 4: Both base and instruct models from the LLAMA-3. 1-8B family recognize words represented with non-canonical tokenizations (performing well Word Repeat), but incorrectly perceive that there are misspellings (performing at random on Identifying Misspellings).
Word Repeat
Identifying Misspellings
Base model Instruct model
90.8
48.2
92.0
55.8
respectively. Overall, these findings suggest that in order for the LM to generate fluent continuations given non-canonical tokenizations, the context and expected continuation need to represent separate turns of dialogue, and additionally, be demarcated with a special template.
4.3 Disentangling understanding from generation
One plausible explanation for our findings thus far is that both base and instruction-tunet models grasp the semantics of non-canonical tokenizations, yet falsely perceive them as containing misspellings. While base LMs attempt faithfully continue these mistakes and degenerate into nonsensical output, instruction-tuned models are trained provide fluent responses regardless of the instruction, leading to the results observed in $4.1 To test this hypothesis, we construct two simple tests:
Word Repeat: To determine if model perceives the meaning of word with non-canonical tokenization, we prompt the model to repeat given word (while correcting any typos).
2. Identifying Misspellings: To determine if a model perceives misspelling, we ask it to identify the word with misspelling among two options: a (correctly tokenized) misspelling word and an non-canonical tokenization of that word (correctly spelled).
Results are shown in Table 4 Consistent with our hypothesis, we find that both the base and instruct models from the LLAMA-3. 1-B family score highly (> 90%) on Word Repeat. This means that the base model, despite its poor performance in 14.1 actually recognizes the correct form of non- canonical tokenizations as well as its post-trained counterpart.
addition, both models perform at random when asked to distinguish non-canonn non-canonical tokenization from true misspellings. In other words, the instruct model produces fluent responses (2) while interpreting the instruction as heavily misspelled! While instruct models evidently overcome this, the base model likely attempts to mimic the (perceived) idiosyncratic surface form, thus producing nonsensical (yet sometimes relevant) outputs.

---

5
Related Work
The extent to which LMs are limited by their tokenization is topic of much debate, with the story evolving as LMs become larger and more capable.
Character-level understanding in tokenizer- models It is commonly argued that tokeniza- tion obscures orthographic information about tokens from the LM, leading to unexpected failures [ [ 67]. As result, there have been many efforts towards linguistically-nnool infoormedd tokenization that
make derivational, compound, and morphological boundaries within words explicit [35 25 26 71
Similarly, BPE-dropout [52] and related methods [60] introduce variation in training in how given string is tokenized in order to make models more robust to rare, misspelled, or unseen words.
However, there is other evidence that LMs naturally overcome these limitations. For instance, token embeddings have been found to robustly encode character-level information, especially larger models 34 127. This may be because word variants that do not share tokens in common (consider e.g.. [- didctiooaayl] and [iiction, aries], as tokenized by GPT-2) incentivize the model to learn spelling general solution to understanding their relations 34 Other works argue that LMs maintain an implicit vocabulary, and can compose arbitrary token sequences (including non-canonical ones) into useful higher-level representations [99 33 Even in domains like biomedical text where terms are highly agglutinative, using tokenizers that segment on meaningful components does not lead to improved models (30). Recent works have even found that coarser superword tokenization 40] [57, which capture common word sequences in a single token, preserve character-level understanding while providing benefits in compression and downstream performance.
Our work informs this conversation by showing that LMs can effectively leverage character-level knowledge of their tokens and glean potential benefits of improved representation inference time.
Partial token problem A related but distinct problem is the partial token problem (also known as tokenization bias or the prompt boundary problem) where the prompt ends with the prefix of a valid token, causing the model to assign unexpectedly low probability to the completion of that token. Many works have found that this continues to compromise a serious failure mode for frontier LMs [69, 51 66 22]. In particular, DEEPSEEK v3 (15 aims to improve robustness to partial punctuation tokens by randomly splitting some proportion of multi- multi-punctuation tokens into smaller tokens during training. though they do not present experiments with this ablation. We note that these results not inconsistent with ours together, they suggest that while models are very unlikely to generate non-canonical tokenizations, they can nonetheless understand them in the context history.
Non-canonical tokenizations It has long been recognized that there are many possible ways to segment a string into tokens with a fixed vocabulary [10], which in principle should be considered in the calculation of a string's 's likelihood 7 g 20 Previous work has briefly touched on non-canonical tokenization in the context self-supervised evaluation (28) and defense against adversarial attacks [29. In contemporaneous work, Geh et al. [21] also show that non-canonical tokenizations can be constructed adversarially to trigger unsafe completions. In contrast, we provide a more systematic study of LM robustness using benchmark evaluations and additionally study its source.
Somewhat relatedly, other works have provided algorithms for sampling at the character- or byte-level from tokenizer-baat -ased LMs [50 65 3 22). Together, these directions suggest that despite being trained with one deterministic tokenization scheme, LMs can both condition on and produce token sequences over different (sub)vocabulary.
6 Conclusion
Despite being trained with deterministic tokenization algorithms, we show that instruction- language models are surprisingly robust to token sequences not seen in training. In certain domains, such as arithmetic or code, more intuitively meaningful tokenizations can even be swapped-in in at inference time for improved performance. We analyze the source of this robustness, and find that while the base and instruct models both perceive the semanties of non-canonical tokenizations, only instruct models are capable of providing fluent continuations. Our work demonstrates a way in which LMs are not necessarily tied to tokenizer they were trained with, and highlights the potential of finding more optimal representations of text after pretraining.

---

Acknowledgments
We would like to thank Joseph An and Ricky Koppolu, as well as the broader UW NLP community, for helpful conversations about this work. AL and JH are supported by the NSF Graduate Research Fellowship.
References
[1] O. Ahia, S. Kumar, H. Gonen, V. Hofmann, T. Limisiewicz, Y. Tsvetkov, and N. A. Smith. MAGNET: Improving the multilingual fairness of language models with adaptive gradient- based tokenization. In The eightt Annual Conference Neural Information Processing Systems, 2024. URL https:://opeereeiiiiieee wttp://ooooni/i-111e0e-a
[2] C. Arnet and B. Bergen. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607-6623, 2025.
[3] B. Athiwaratkun, S. Wang, M. Shang, Y. Tian, Z. Wang. S. K. Gonugondla, S. K. Gouda, R. Kwiatkowski, R. Nallapati, P. Bhatia, and B. Xiang. Token alignment via character matching for subword completion. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 15725-15738, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics, doi: 10. 18.188/3/v 1/2024, http:/...2n22n........ findings-acl.929)
URL https: paattps:/accantteee http://aaaaaaatoooa.ocooo wrg/2024. wtt.2222....nnnne findings- acl..29
[4] T. Bauwens and P. Delobelle. BPE-knockout:
Pruning pre-existing BPE tokenisers with backpwardscomppttbble morphological semi-supervision. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pages Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 1/2224 htwp:://.2222.2llll.e.
naacl-long. lonel-lon...4... 324. URL
324
[5] B. bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
https://openreviex. nwt.:/runfffrtrooee itt.://eew....eeooooomiiiiitiaa. Tiy.iddsyoososs
[6] Y. Bisk. R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical com- monsense in natural language. Thirty-Fourn Fourth AAAI Conference on Artificial Intelligence. 2020.
[7] K. Cao and L. Rimell. You should evaluate your language model on marginal likelihood over tokenisations. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 2104 2114, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/20211 www.eemmmapaaaae main. URL httpt:tts////cll /tLL:tttpp///aaaaattoooogg org/ 2021. emn www.emmmmmmmm.....mmmmmmmmmmmaam 161
[8] Y. Chai, Y. Fang, Q. Peng, and X. Li. Tokenization falling short: On subword robustness in large language models. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors. Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1582-1599, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10.18653/v 1/2024. findings-emmlp.86
wwwpaaatttooooooooo///2/22/ wrg// findings----nneeee emn emnle.nn.... 8686
[9] N. Chirkova, G. Kruszewski, J. Rozen, and M. Dymetman. Should you marginalize over possible tokenizations? A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 1-12, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: URL https: http:///aaaaahoo//////// arg/2023 acl-short.i
[10] K. W. Church. Emerging trends: Subwords, seriously? Natural Language Engineering, 26(3):
375-382, 2020, doi: 10.1011// 150101777113111322220000000
10

---

[11] C. Clark, K. Lee, M.-W. Chang. T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1300. URL laapp:://aclanthooog
org/019-1300
[12] J. H. Clark, D. Garrette. I. Turc, and J. Wieting. Canine: Pre-training efficient tokenization- free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73-91, 2022. doi: 10.1162/tacl. a 00448. (RRLLattps://alaaattollooo org/ 2022.tac1 tac1-1.5
[13] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https 05457
'arxiv.org/aal
[14] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. httthttts:///rrtttt http:///aaaiii.oo/aaa/////////// wttp:oog///////o/eoeo 2210.. 14168
[15] DeepSeck-AL. Deepseek-v3 technical report, 2025. URL 19437
[16] D. Dua. Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume (Long and Short Papers), pages 2368-2268 2378- Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1211866////N19-12200 URL oraeloyyy.oogg////
[17] Y. Dubois, X. Li, R. Taori, T. Zhang. I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
[18] L. Edman, H. Schmid, and A. Fraser. CUTE: Measuring LLMS' understanding of their tokens. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3017-3026, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10. 10.11866601 (0018663/11/2224emmmpp mma.4/4...mmmmmmimm17...... URL hawp://ooo.a org//org//00 2024. emnlp monnp-mminn.. 177
[19] S. Feucht, D. Atkinson, B. C. Wallace, and D. Bau. Token erasure as a footprint of implicit vocabulary items in LLMs. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9727-9739, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 18.18653/ v *12024.emnlp-main.54. URL RttLhhtps:/// httpe://acaaaatthto anthology or2/2024. emnlp-maiieeeee maammain...mm 543
[20] R. Geh, H. Zhang, K. Ahmed, B. Wang, and G. Van Den Broeck. Where is the signal in tokenization space? In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3966-3979, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics. doi: 10.18653/
v1/2024.cmnlp-mail mawemmmmlllmaaaammmmmmmmmmmmmaem 230. URL http::///aaaatthooooo org/2024. emnlp manlp-main... wwin.ann..mmmm.......... 230
[21] R. L. Geh, Z. Shao, and G. V. den Broeck. Adversarial tokenization, 2025. URL https 7/arxiv. barxiv..rg/aaee abwgg//bs//533...m 02174
[22] Hayase, A. Liu, N. A. Smith, and Oh. Sampling from your language model one byte at a time. arXiv preprint arXiv:2506. /4/23, 2025.
wwt.:///////bb//// 2506 14123
[23] D. Hendryeks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding. In International Conference on Learning Representations, 2021.

---

[24] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the MATH dataset. Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL Bdyy
[25] V. Hofmann, J. Pierrehumbert, and H. Schütze. Superbizarre is not superb: Derivational morphology improves BERT's interpretation of complex words. In C. Zong. F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 3594-3608, Online, Aug, 2021. Asso- ciation for Computational Linguisties. doi: 10.18653ww122221.ac--onngg //aclanthology. org/ 2021. moll-lnng... 279
URL https
[26] V. Hofmann, H. Schuetze, and J. Pierrehumbert. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computationnal Linguistics (Volume 2: Short Papers), pages
Dublin, Ireland, May URL
2022. Association for Computational Linguisties. doi: (0.118653/11/2022..clhhhh https://aclanthollee
acl-short.43
[27] I. Itzhak and O. Levy. Models in spelling bee: Language models implicitly learn the character composition of tokens. In M. Carpuat, M.-C. de Marneffe, and V. Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5061 15068, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 URL httpt:::/ta/aattaaaaaa
org/2022 mwwcl-main....ee 373
[28] N. Jain, K. Saifullah, Y. Wen, J. Kirchenbauer, M. Shu, A. Saha, M. Goldblum. J. Geiping, and Goldstein. Bring your own datal supervisd evaluation for large language models, 2023. URL https://arxiv..o wwwparwww....raaaaaeoe wttp://.rrrrbbb///////// http:/bbb/2200.... 13651
[29] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P. yeh Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023. htttps:::: :/apss/////xivi/ wwwaaiioog/aaaaaeaeaaaaaaoo abs 2230999../300.....m 2309.00614
[30] B. Jimenez Gutierrez, H. Sun, and Su. Biomedical language models are robust to sub- optimal tokenization. In D. Demner-fushman, S. Ananiadou, and K. Cohen, editors, The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 350- 362, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v 2023.bionlp-1. 1.32. URL htttps::/aclatthlll.... orgg 2023. http://...rrr/2.....ollp--p--- bionlp-1.32
[31] M. Joshi, E. Choi, D. S. Weld, and L.. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL. htt.htttpp//rxii.or/////// 03551
[32]
Kaggle. 200,000+ jeoparyy! questions. 2019. http:t/s:: httttpp:/wwww.kaggleeee .aaglle.com/datsstts//
[33] G. Kaplan, M. Oren, Y. Reif, and R. Schwartz. From tokens to words: On the inner lexicon of LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openreviess www.le/w..eeeoooommmeomo
[34] A. Kaushal and K. Mahowald. What do tokens know about their characters and how do they know it? In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2487-2507, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. https://aallatthooogy wt2.
URL
naacl http//2222nnaccmmmllla. 179
[35] S. Klein and R. Tsarfaty. Getting the ##life out of living: How adequate are word-pieces for modelling complex morphology? G. Nicolai, K. Gorman, and R. Cotterell, editors, Proceedings of the 17th SIGMORPHON Workshop Computational Research in Phonetics,
12

---

Phonology, and Morphology, pages 209-209, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/1//222.iiiggi
1.24. URL httpp:://aclaaatoollggg org/
2020. sigmorphon-1 24
[36] N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang. R. L. Bras, O. Tafjord,
C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL htttps//arxivi wwwp.....o/bb///11111111 15124]
[37] H. J. Levesque, E. Davis, and L. Morgenstern The winograd schema challenge. In Proceedings of the Thirteenth International Conference Principles of Knowledge Representation and Reasoning, page 552-561, AAAI Press, 2012.
[38] H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 11260-11 1285, Bangkok, Thailand, Aug. 2024. Association for Computational org/
Linguistics. doi: 10.186533v112000 1/022/..nin/ ac1.671. URL 2024. finddrgsnnnaalll...
[39] T. Limisiewiez, T. Blevins, H. Gonen, O. Ahia, and I.. Zettlemoyer. MYTE: Morphology-driven byte encoding for better and fairer multilingual language modeling. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15059-15076, Bangkok, Thailand, Aug. 2024. Association for Computational Linguisties. doi: 10. N0.186633/11/204.aclllooo URL https: http://ccaattooogy...ooo 200gg/2224.ccoo acl- mong..lngg... 804
[40] A. Liu, Hayase, V. Hofmann, S. Oh, Smith, and Y. Choi. SuperBPE: Space travel for language models. arXiv preprint arXiv: 13423, 2025. URL https:s///arrx org/ /bs/
13423
[41] S. Lundberg. ken healing, 2023.
Prompt boundaries and to- data--s-ddddaassiiii----- scieec// unlearning for llms, 2024. URL pooogpgoooouaarreeeeaooooooeeaer
The art of prompt design:
URL
the araaattt ces-prromt-desssgee
[42] P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lidton
Tofu: A task of fictitious
[43] Meta. The Ilama herd of models, 2024. URL//ttps///axxivoo///bbb
21783
[44] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering, 2018. URL 02789
[45] B. Minixhofer, E. Ponti, and I. Vulié. Zero-shot tokenizer transfer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL ltttps://operrev// bet//forum?iddBBBBbbe..
[46] P. Nawrot, J. Chorowski, A. Laneucki, and E. M. Ponti. Efficient transformers with dynamic token pooling. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6403-6417, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 00.18653/v120033 2ttp:////223.a.-loogg333333 acl-long.353 URL
prg/2023 httg:2223.acl-oon.. acl-long. 353
[47] R. Nogueira, Z. Jiang, and J. Lin. Investigatiing the limitations of transformers with simple arithmetic tasks, 2021. URL https://arxxv..rrt httpg//bbb/22222 13019
[48] T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, T. Anderson, D. Atkinson, F. Brahman, C. Clark, P. Dasigi, N. Dziri, M. Guerquin, H. Ivison, P. W. Koh, Liu, S. Malik, W. Merrill, I. J. V. Miranda, J. Morrison, T. Murray, C. Nam, V. Pyatkin, A. Rangapur, M. Schmitz, S. S..kjosbbbrgg D. Wadden, C. Wilhelm, M. Wilson, I.. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi. 2 olmo 2 furious, 2024. URL https: / arxiv. org/abs/////eeeeoooee...ooeeeert 2501 00656
13

---

[49] A. Pagnoni, R. Pasunuru, P. Rodriguez, J. Nguyen, B. Muller, M. Li, C. Zhou, L. Yu, J. Weston, L. Zettlemoyer, G. Ghosh, M. Lewis, A. Holtzman, and S. Iyer. Byte latent transformer: Patches scale better than tokens, 2024. URL
[50] B. Phan, M. Havasi, M. Muckley, and K. Ullrich. Understanding and mitigating tokenization bias in language models, 2024. URL https::/aazziv..rgggggggo/////
16829
[51] B. Phan, B. Amos, Gat, M. Havasi, M. Muckley, and K. Ullrich. Exact byte-level probabilities from tokenized language models for fim-tasks and model ensembles, 2025. URL https
// //aaxxiv.coo orgoooggggo /t///bb/////// 09303
[52] Provilkov, D. Emelianenko, and E. Voita. BPE-dropout: Simple and effective subword regularization. In D. Jurafsky, J. Chai, N. Schluter, and Tetreauut, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882-1892, Online, July 2020. Association for Computational Linguistics. doi: 10.18663/w1/2220.cc-mmaa 170. URL
org wrrg/2020.co wctg/2220.acll11100/ aca.-00..mmmmii..m...
[53] Qwen. Qwen2.5 technical report, 2025. URL
[54] M. Roemmele, C. A. Bejan, and A. S. Gordon. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI Spring Symposium on Logi- cal Formalizations of Commonsense Reasoning. Stanford University, Mar. 2011. URL
http://ictt usc. ottp////////.aaa...oooooo//oo//a pdf
X20Am%
[55] K. Sakaguchi, R. L. Bras, C. Bhagavatulla, and Y. Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, Aug. 2021. ISSN 0001-0782. URL
attps:///ol
org/10. 1145/3474381
[56] A. Sathe, D. Aggarwal, and S. Sitaram. Improving consistency in LLM inference using probabilistic tokenization. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 4766-4778, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 979-8-89177--95-7. URL 002/2025. findings-naacl.2222 268/
[57] C. W. Schmidt, V. Reddy, C. Tanner, and Y. Pinter. Boundless byte pair encoding: Breaking the pre-tokenization barrier, 2025. URL attps::/aarxiv.orr//ab/////// 00178
[58] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In K. Erk and N. A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10. 10.1886//11P16111 URL paattps://aclanthollin www.anhhoooooooooooooo--o wttg//6-g/11/11
[59] F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay. S. Ruder, D. Zhou, Das, and Wei. Language models are multilingual chain-of-thooe of-thhughtt reasoners, 2022. URL httpttts::// //tss////rxvv org/ abs 2210. 03057
[60] A. Sims, C. Lu, K. Kaleb, J. N. Foerster, and Y. W. Teh. StochasTok: Improving fine-grained subword understanding in LLMs. In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025. URL
[61] A. K. Singh and D. Strouse. Tokenization counts: the impact of tokenization on arithmetic in frontier llms, 2024. URL https:////:: http://aaaxxivooorrb/b///a//
14903
[62] A. Talmor, J. Herzig, N. Lourie, and J. Berant. CommonsenseQA: question answering challenge targeting commonsense knowledge. In J. Burstein, C. Doran, and T. Solorio, edi- tors, Proceedings of the 2019 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4155 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 18.18653/v 10.18653/11/N19-1440 1/21.-
URL https: a/cl anthology. 0rg/X19-1421
14

---

[63] Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. In International Conference on Learning Representations, 2022. URL https //openreview iepeerreiee..ne//foumm22222
[64] A. Thawani, J. Pujara, F. Ilievski, and P. Szekely. Representing numbers in NLP: a survey and vision. In K. Toutanova, A. Rumshisky, L. Zettlemoyer. D. Hakkani- Tur, I. Belt- agy. Bethard, R. Cotterell.
Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-656, Online, June 2021. Associa- tion for Computational Linguistics. doi:
URL https
/aclanthology 22g/2021 naacl mwa...in...m.................... 53
[65] T. Vieira, B. LeBrun, M. Giulianelli, J. L. Gastaldi,
B. DuSell, J. Terilla, T. J. O' O'Donnell, and R. Cotterell. From language models over tokens to language models over characters, 2024.
URL httth: http://rxxii..orraa///111.22... 2412//000/
[66] T. Vieira, B. LeBrun, M. Giulianelli, J. L. Gastaldi, B. DuSell, J. Terilla, T. O'Donnell and R. Cotterell. From language models over tokens language models over characters. In Forty- second International Conference on Machine Learning, 2025. URL https::tp/////eeeeeeeii het/fforunPdd=$$00oolll
[67] D. Wang, Y. Li, J. Jiang, Z. Ding, G. Jiang, J. Liang, and D. Yang. Tokenization matters! degrading large language models through challenging their tokenization, 2024. URL https 7/arxiv. //arxiv.org// org/orgg//....oooo..ooo.o. aa/aab///240 2405. 17067
[68] J. Wang, T. Gangavarapu, J. N. Yan, and A. M. Rush. Mambabyte: Token-free selective state space model. In First Conference on Language Modeling, 2024. http:/ttpp///o net -iimmmdddddxssseee
[69] H. Xu, A. Liu, J. Hayase, Y. Choi, and N. A. Smith. Are you going to finish that? practical study the partial token problem, 2026. URL /ttttp::/aaxiv.orgg...
232223
[70] L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. ByT5: Towards token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306 2022. doi: 10.1 10.1162/tacl att/:/tcll.000000 a.0461. URL
org 2022. tacl-1.17
[71] S. Yehezkel and Y. Pinter. Incorporating context into subword vocabularies. In A. Vlachos and I. Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 623-635, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 1:65://1//0233.ecc-mmmim Ectps://aclantee
URL
wwtg/222...... org/ 2023. wac.lmalaiin...
[72] L. Yu, D. Simig, C. Flaherty, A. Aghajanyan, L. Zettlemoyer, and M. Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. In Shirty-seventh Conference on Neural Information Processing Systems, 2023. /tllhtttss///perrrriii fet.rruummme//oor
id=JIm02V9Xpz
[73] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can machine really finish your sentence? In A. Korhonen, D. Traum, and L. Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19-1472. URL http:s:://allathooogy.oooo 1472
[74] M. Zhu, A. Jain, K. Suresh, Ravindran, S. Tipimeni, and C. K. Reddy, Xlcost: benchmark dataset for cross-lingual code intelligence, 2022. URL attps://ee //arxiv org/abs//200 08474
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide support in the paper for all claims in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly identify questions that we leave to future work and differentiate hypotheses from claims supported by evidence.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
Our proof in the appendix is complete and provides the full set of assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
This information will be provided in the appendix for the appendix deadline.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We plan to open-source our code shortly after submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
mublic//uudee//Cdeetttm www.ooobbbbiioooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ Lttipp.cc/pullic//uiice http://pibbppcuuuued////dddttaia /CodeSubmeiiiccec ietammmsssion ssisnn oeeiiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
This information will be provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide information about standard deviation and statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have CI. the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We will provide this information in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::::////ris//eeeee cwterrisscccpuuuie
Answer: [Yes]
Justification: Our research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work has immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No data from the paper has high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the creators and comply with the license of all assets used the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssithcooee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [Yes]
Justification:
We will release the datasets we created.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
There are no experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There are no experiments with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required. this research?
Answer: [NA]
21

---

Justification:
We did not use LLMs to conduct this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy httpas/s//eeerrppoccc///oorreere for what should or should not be described.
22

---

A Random Non-Canonical Tokenization
In this section, we provide our algorithm for producing a random non-canonical tokenization, and proof that each con-canoiical tokenization that is more fine- fine-grained than the canonical one has equal probability of being output.
A.1 Algorithm
We will split each token in canonical tokenization into smaller tokens (that each exists in the tokenizer's vocabulary). We formulate our problem as: Given a valid token i, and set vocabulary V. construct a sequence of tokens seq using tokens that exist in V and form t when concatenated together. We produce seq using recursive algorithm. Since there can be many possible seq for each I, we need to randomly choose one and guarantee that each possible seg is chosen with equal probability. We achieve this by considering recursion as producing a tree. Each path down the tree corresponds to one possible way to segment L. Each node of the tree represents segmentation state where we have chosen some number of sub-tokens. At each node, we weigh the choice of which child node visit by the number of leaves in the sub-tree that is rooted each child node. This guarantees that each path down the tree is chosen with equal probability since the number of paths down tree is equal to the number of leaf nodes in that tree. The pscudocode for the algorithm is in
A.2 Proof
Goal: To prove that the random segmentation algorithm chooses one valid segmentation from all possible valid segmentations with uniform probability.
Notation: Let (ii) denote the number of valid segmentation completions (i.e., the number of leaves in the recursive tree) for the substring starting at index i. In particular, W W(tokenl) = 1. Note that (i)) is calculated by the memoized recursive function countSegmentss which calculates the number of leaves of the subtree rooted at i.
Base Case: Consider the node corresponding to |token| (the end of the token). Here, there is exactly one valid segmentation (the empty segmentation), so the algorithm returns it with probability 1. That is, every segmentation (in this case, the only one) is chosen with probability
1.
W W(ttoken)
Thus, the base case holds.
Inductive Hypothesis: Assume that for any node corresponding to an index j with j > (i.e., deeper in the recursion tree), every complete segmentation (leaf) in the subtree rooted at j is chosen with probability
1
WG)"
Inductive Step: Now consider node corresponding to index (with i < [loken)). Suppose that from there are k valid branches corresponding to choosing substrings that end at indices ji.j2......J where for each j we have i < j s [token| and the substring tokenji j] is in the vocabulary. By definition,
k
(
Σ W(.).
r=1
The algorithm selects the branch from i to a specific child j with probability
W
P(i +
W(1)
Once branch i + j is chosen, by the inductive hypothesis every complete segmentation (leaf) the subtree rooted at is chosen with probability
W (
23

---

Thus, the probability P(S) of obtaining a particular complete segmentation V that starts at i by first taking the branch - and then following specific path the subtree rooted at j is
WG) W(1) WG)
1 W
P(S)
Since the factor (J) cancels, the probability P(S) independent of the particular child j chosen. Conclusion: By the inductive step, every complete segmentation (leaf) in the subtree rooted at any index i is chosen with probability w(t) In particular, when 0 (the start of the token), every valid
segmentation of the entire token is selected with uniform probability w(ay This completes the proof.
Algorithm Random Token Segmentation
1: function COUNTSEGMENTS(star))
Cached (using memoization)
2: if start = [token| then
3: return 1
Reached end; valid segmentation
4: end
5: total +-
6: for end +- start + 1 [token| do
7: substring <-
end]
8:
substring E vocabulary then
9:
total t- total COUNTSEGMENTS(end)
10: end
11: end for
12: return total
end function
14:
function BUILDSEGMENTS(sGarr)
if starl = |Loken] then
16:
return Ø end if
b Empty segmentation
18:
validSegments +
weights +
20:
for end +- start + to |token] do substring c token (etensstattt... end] if substring E cocabulary then segCount + COUNSSECMENTT(())) if segCount > 0 then
22:
23:
24:
Append subslring to validSegments
26:
end Append segCount to weights
27:
28: end if
29: end for
30: if validSegments is empty then
31:
return
32:
end if
33:
SegmenSegmeet <
weights)
34:
return [chosen.Seyment) || WOUIDDEEMENNTS(lee + Seemenn & Concatenate
chosen segment with segmentation of remaining token
end function
36:
procedure SEGMENTTOKEN(toeen. vocabulary)
if COUNTSEGMENTS(O) then
38:
return Ø else
No valid segmentation exists
40:
return BUILDSEGMEETT(()
end if
42: end procedure
24

---

B Evaluation Details
B.1 General benchmarks
For short-answer benchmarks, the system prompt is:
You are a helpful assistant.
For multiple-choice benchmarks, the system prompt is:
You are helpful assistant. For the following multiple choice questions, return the answer only, without any additional reasoning or explanation.
MATH MATH is dataset composed of fairly difficult, competition level math problems (24). The test set is composed of short answer problem that describe some scenario and asks the model to output a mathematically correct answer.
GSM8K GSM8K dataset consisting of relatively simple math questions that would appear in grade school math exams []4]. For GSM8K, the evaluations were done in the same manner as MATH.
MMLU MMLU is benchmarks comprising of multiple choice questions from wide variety of subjects. 23 We sampled 500 questions from MMLU for our evaluation. We instructed the model to only output one answer each question without any explanation.
Alpaca Eval Alpaca Eval is an evaluation benchmark where generations from language models against given prompts are compared and judged by an annotator model. [17] The metric used was raw winrate of the perturbed model as judged by a language model. The annotator used was alpaca_eva evalleelllllaaee eval-veaaetteeeeeeeeeeeeeeeeee gp14, which has been shown to have the highest Spearman and Pearson correlation coefficient with human annotators.
ARC Challenge and ARC Easy Contains multiple choice questions with four options each, taken from grade school science exams (13). ARC Easy is tests basic science knowledge while ARC Challenge requires some procedural reasoning.
BoolQ Contains true or false questions along with context passage that provides the answer the question. 11
CommonsenseQA Contains multiple choice questions with five options each that requires common sense knowledge answer. 62
COPA
Contains multiple choice questions with two options each that tests knowledge of cause and
effect. 5
CUTE Contains questions that require the model to manipulate sentence-level, word-level, and character-level structure for strings. 18
DROP contains questions that potentially require reasoning multiple pieces of information present in given passage. [66]
HellaSwag contains multiples choice questions with four options each that asks for the most natural continuation some given context. 773
Jeopardy QA
contains short answer questions from the "Jeopardy!" game show.
32
OpenbookQA contains multiple choice questions with four options each that require some multi- step and common sense reasoning. 44
25

---

Table 5: System prompt for tasks in $3 See Table for example instructions.
Counting characters: You are helpful assistant. The following prompt will ask you to return sequence of words. Only return the sequence, separated by spaces. provide any additional
explanation.
programming assistant trained to analyze and interpret code enippets. When provided with code snippet and (A, B, C, or D), your task is to evaluate the code, determine its behavior, and select the answer that best describes this behavior. Your reaponse must be a single letter: A, B, C, D. Do not provide explanations or additional text unless explicitly requested.
Code Deseription: You
answer choices
computational assistant trained to evaluate arithmetic operations. When provided with arithmetic expreasion, calculate the reault and round to the nearest integer. Respond only with the rounded result, without any additional text or explanation.
Arithmetic:
You
PIQA contains multiple choice questions that require reasoning about the physical world. 6
TriviaQA
contains short answer questions that requires knowledge of the world. 31
contains multiple choice questions with two options that asks determine what a pro- noun might refer to. Answering these questions require knowledge commen sense and surrounding context. 37
Winograd
Winogrande contains questions in the same format of Winograd but there are more questions and the questions are harder. (55)
TOFU contains general short answer questions that tests the model's ability to process world knowledge. This is the retain the task of fictitious unlearning dataset. 442
WikidataQA require models to complete factual statements.
S
B.2 Constructed Benchmarks
In this section, we provide more detail on how datasets we use in $3 are constructed.
Count Characters Task The prompt asks the model to count the number of occurrences of given character in a 10-character word; we always use the most frequently occurring character. Evaluation was done, similar to GSM and MATH, by finding the last number in the generated response. Generations without any numbers are considered incorrect.
Generate Acronym Task
The model is asked to generate a sequence of words whose first letters form randomly sampled five character string. For evaluation, we take the first character of each cettespace-delimicee word and check if matches the desired acronym.
Codeline Description Task The model is asked to comprehend piece of code and choose the best description from four options.
The model is asked to perform addition or subtraction with digit numbers. We regex extract numbers from the generation, which then compared the ground truth answer.
Arithmetic Task
B.3 Metries of generation quality
Here provide additional details on the metrics defined in $4.1
26

---

Table 6: Data format of ablations $4.2
No ablation: <luerr >Provide detailed analysis of Candace Parker's defensive techniques in her recent games, excluding the words "aggressive" and "blocking", the format of sports commentary script. <lassistant|> [Sporta Commentary Script]
[Opening Scene...
QA Template: Question: Provide detailed analysis Candace Parker's defensive techniques in her recent games, excluding the words "aggressive" and "blocking", in the format of sports commentary script. Answer: [Sports Commentary Script]
[Opening Scene...
Removing the chat template: Provide detailed analysis of Candace Parker's defensive techniques in her recent games, excluding the words "aggressive" and "blocking", in the format of sports commentary script. [Sports Commentary Script] [Opening Scene...
Removing the instruction: < </user|> [ Commentary Script] [0pening Scene:
packed basketball arena, with fans eagerly awaiting the analysis of Candace Parker's recent performances the court.]
Commentator Welcome back, basketball fans! <lassistant/>Tonighh we're diving into the defensive prowess of Candace Parker...
Spelling We the top 10000 most frequently appearing English words in Google's trillion word corpus. We only consider words with more than one character. This is because sometimes base models will repeatedly generate the same letter, and since all English letters are in the word list, the generation would receive high score.
Grammaticality One drawback with this evaluation method is that oftentimes the model would repeat the same letter over and over again, start counting numbers. In both of these cases, there are no detected grammar mistakes, however they are still obviously gibberish. Therefore, we only calculate grammaticality scores for generations that receive score 2 0.5 spelling; otherwise, give a grammaticality score of O.
Win rate Similar to evaluation in $2 we also used alpaca_ avaca.eval.ce -pt-- as the evaluator and report raw win rate. In 4.1 the win rate is calculated against generations conditioned input with canonical tokenization. In 4.2 the win rate against generations from the No Ablation setting when also given character-level tokenization. By construction, the win rate of the No Ablation setting itself is 50%.
B.4
Ablation Settings
For ablations on the data format, see examples formatted data in Table 6 Our finetuning code was forked from sllenai/ppenninstruutt instruct The exact finetune recipe is given below:
Setup: 8 L40S GPUs
Gradient Accumulation Steps: 20
Per Device Train Batch Size: 2
Mixed Predision: bf16
Max Seq Length: 4096
Learning Rate: 5e-06
LR Scheduler Type: Linear
Warmup Ratio: 0.03
Weight Decay: 0
Epochs:
Seed: 123
27

---

Table 7: OWWN-2.5. 2.5-7B-INSTRUCT is robust to character-level tokenization of Chinese text.
Benchmark
Canon
Char
Chinese MMLU
77.8
74.2
Chinese GSM
78.8
76.8
B.5 Disentangling understanding from generation
For these tasks, we use 500 words randomly sampled from Google's 10000 English word lis
Word Repeat An example prompt shown below.
Repeat each word directly, while correcting any typos.
Question: guarantees
Answer: guarantees
Question: revelation (character-level tokenization) Answer:
Identifying Misspellings We obtain the misspelled word by randomly adding, removing, or substituting single character from the word. An example prompt is shown below.
Question:
Which of the two words contains misspelling? Respond directly with the answer option.
Question:
A. guarantees
B. garantees
Answer:
(9 more context examples]
Question:
A. farmer (character-level tokenization)
B. farme (canonical tokenization)
C Additional Results
C.1 Evaluation on Chinese Benchmarks
We also investigate how robust language models given character-level tokenization of text in Chinese as evaluated on two tasks, Chinese GSM [59] (part multilingual GSM benchmarks) and Chinese MMLU 38) Note that since each Chinese character is usually represented with three bytes under UTF-8 encoding, this is not equivalent to byte-level tokenization. We focus on QWEN- LLAMA-3.1-8 1-88BIISTTUCT and OLMo-2-7B- 1--2---I-NSTUCCT do not officially support Chinese. As shown in Table 7 we observe similar robustness on Chinese, with performance dropping by only 3% each task.
ggogle-10000-engl... txt
28


Reviews:
[{"review_id": "ksRG580Smw", "reviewer": "Reviewer_guJs", "summary": "This paper studies how LLMs trained with a single, deterministic BPE tokenizer work when are fed input that is tokenized in “non-canonical” ways in inference time. Non-canonical tokenization ranges from random alternative BPE to pure character-level splits. Across 20 benchmarks the authors show that performance drops surprisingly little, with instruction-tuned models retaining more accuracy than their pre-train–only counterparts. Moreover, they uncover scenarios where deliberately altering the tokenization improves results: character-level splits boost string-manipulation and code tasks, while right-aligned digit grouping markedly improves large-number arithmetic.", "questions": "The study focuses on non-canonical tokenization at inference. Could you clarify why you do not explore randomized tokenization during pre-training, as in DeepSeek-V3 (https://arxiv.org/pdf/2412.19437, Sec. 4.1)? What challenges or trade-offs led you to focus solely on inference-time?", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The work focuses on an important yet under-explored aspect of LLMs—how BPE tokenization choices at inference time affect model behavior. This paper reveals unexpected flexibility and generalizatibity of LLMs.\n2. The comparison of instruction-tuned vs. base models is very insightful. By comparing Llama-3 with its SFT-fine-tuned variant, the paper clearly shows that instruction tuning significantly increases robustness to tokenization change, showing the practical value of SFT.\n3. This paper finds a simple yet effective number tokenization trick. The right-aligned digit grouping technique is very easy to implement and delivers substantial gains on arithmetic tasks, offering a “free” performance boost.\n\nWeaknesses:\n1. This paper has limited model diversity in the SFT analysis. Section 3 compares only one base/SFT pair (Llama-3 vs. Llama-3-Instruct), leaving open whether the observed gap generalizes to other architectures and scales.\n2. This paper misses related work on the same topic. Existing work (e.g., DeepSeek-V3, see question 1) already uses randomized tokenization during training; the paper neither acknowledges this clearly nor tests such models. I suggest adding DeepSeek-V3 to experiments, to see how a small percentage of random tokenization in pretraining stage can make a difference.\n3. The tri-color bar plots in Figures 3 & 4 occupy significant space yet convey highly correlated and redundant information: showing one bar instead of 3 bars make little difference to the conclusions drawn; a more compact format (e.g., a table with a few numbers) could communicate the key numbers more efficiently.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the “important” and “unexpected” finding that LLMs are robust to non-canonical tokenizations of text, and furthermore, can see “substantial gains” from R2L tokenization in arithmetic.\n\n\n**Model diversity in SFT analysis**\n\nWe wish to clarify which section the reviewer is referring to. §3 (mentioned by the reviewer) identifies tasks where non-canonical tokenization leads to better performance, and uses only instruction-tuned models, not base/chat pairs. §4 (perhaps what the reviewer intended) does use base/chat pairs to identify the source of robustness, but we study both $\\texttt{Olmo2}$ and $\\texttt{Tulu3}$ model families. We would be happy to run additional experiments with clarification from the reviewer!\n\n**Related work in $\\texttt{Deepseekv3}$**\n\nWe thank the reviewer for this important reference that we were not aware of. Indeed, $\\texttt{Deepseekv3}$ trains with non-deterministic tokenization, randomly dropping merges of punctuation tokens like \\n\\n to overcome the “prompt boundary problem” (so that the model can generate \\n when conditioned on \\n). However, we regret to share that we were not able to run $\\texttt{Deepseekv3}$, which has 671B parameters, with our compute budget. We considered distilled versions of $\\texttt{Deepseekv3}$, but this would not test the same hypothesis, since the distilled versions themselves were not trained with non-deterministic tokenization. We will definitely discuss $\\texttt{Deepseekv3}$ in the next revision of the paper.\n\n**More compact format for Figures 3 & 4**\n\nThank you for the suggestion, we plan to present this data in a table as suggested for the next revision.\n\n**On random tokenizations during pretraining**\n\nIt would be very interesting to control for the extent of randomness in tokenization during pretraining and measure its impact on models’ robustness to non-canonical tokenizations. However, pretraining from scratch is not possible with our compute budget, and moreover, our paper focuses instead on demonstrating the unexpected finding that even with deterministic tokenization, models are surprisingly robust to non-canonical tokenizations. We believe the reviewer’s suggestion is a promising direction for future work, as non-deterministic tokenization during pretraining may enable further inference-time gains from adaptive tokenization."}, {"review_id": "1tY6yG9HJl", "reviewer": "Reviewer_oPqB", "summary": "The paper investigates how instruction-tuned language models handle non-canonical tokenizations of input text. Through systematic evaluation on 20 benchmarks and three model families, the authors show that models retain over 90 percent of their performance under random or character-level tokenizations. They further demonstrate that swapping in alternative tokenizations such as character-level segmentation for string tasks or right-aligned digit grouping for arithmetic can yield substantial gains (up to +33 percent) without additional training. Finally, an analysis of model stages reveals that robustness to non-canonical tokenizations emerges during the supervised instruction-tuning phase.", "questions": "- Would be good to see if fine-tuning on some non-canonical scheme (e.g. right-to-left digit grouping) further boosts improvements.\n- Would be good to see some studies on non-latin languages.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Addresses the under-explored tokenizer, with a large-scale empirical study on diverse tasks.\n- Demonstrates opportunities for performance gains via simple inference-time interventions with a creative and diverse task suite.\n- Rigorous analysis pinpoints instruction-tuning as the key driver of robustness, supported by ablations on training data and procedure.\n\nWeaknesses:\n\n- Non-canonical tokenizations do not show performance gain across the board. Would be great to have an adaptive tokenization scheme with consistent improvements.", "was_revised": true, "final_justification": "This paper provides meaningful insight to the often overlooked tokenization in LLMs. My questions have been addressed with additional insightful experiments.", "rebuttal": "We thank the reviewer for recognizing our “systematic evaluation” of model robustness to non-canonical tokenizations, the “performance gains” delivered by “simple inference-time interventions,” and “rigorous analysis” into the source of robustness. We hope you find that our response addresses your concerns.\n\n**On the possibility of automatically adapting tokenization schemes**\n\nThe reviewer is correct that, in this work, we do not find a single adaptive tokenization strategy that improves performance across the board. The goal of the work was not to improve tokenization, but instead demonstrate that LMs can handle non-canonical tokenization, which is surprising and important because these tokenizations are provably out-of-distribution. We believe that automatically finding the best segmentation of text at inference-time is a promising direction for future work.\n\n**Finetuning on non-canonical tokenizations**\n\nFollowing the reviewer’s suggestion, we explore whether finetuning on a non-canonical tokenization scheme, which has proven useful at inference-time, can further improve downstream performance. In particular, we finetune the $\\texttt{Llama-3.2-1B}$ base model on 30K random examples from the Tulu3 SFT mixture: one with L2R digit grouping, and one with R2L grouping. Then, we compare the following settings: (1) the L2R-trained model with L2R inference, (2) L2R-trained model with R2L inference, and (3) R2L-trained model with R2L inference. We expect (3) to outperform (2) due to the additional opportunity to adapt to R2L tokenization during training. We evaluate these models on 1,000 instances of addition and subtraction of 4 - 8 digit numbers. We find that (1) gets 2%, (2) gets 24%, and (3) gets 12.3%, counter to our hypothesis. We conjecture that this is because (3) learns to predict two different representations of numbers between pretraining and finetuning, and when generating output, divides probability among R2L and L2R representations. In contrast, (2) continues to generate numbers L2R, even when conditioned on R2L representation of digits.\n\n**Evaluations on non-Latin languages**\n\nThank you for pointing out the value of additional evaluations on non-Latin-script languages! This is indeed a point that we overlooked. We did additional evaluations on Chinese with $\\texttt{Qwen-2.5}$ (since $\\texttt{Llama3}$ and $\\texttt{OLMo2}$ do not officially support Chinese), comparing character-level tokenization and canonical tokenization. We evaluate on Chinese MMLU and Chinese GSM. Shown in the table below, $\\texttt{Qwen-2.5}$ is robust to character-level tokens in Chinese. We plan to expand these results and include them in the next revision.\n\n|             | **Normal** | **Char** |\n| ----------- | ---------- | -------- |\n| **Chinese MMLU**       | 77.8       | 74.2     |\n| **Chinese GSM** | 78.8       | 76.8     |"}, {"review_id": "0Y0lriENj4", "reviewer": "Reviewer_7SJp", "summary": "This work studies the problem of tokenization. In particular, they show that instruction finetuned LLM can handle broken tokenization, i.e. encodings that does not follow the BPE such as character-level id. They studies the problem through a systematic evaluation and also show that broken tokens can also help LLM improve its performance in certain character-level aware task.", "questions": "The experimental design in Section 3 lacks persuasiveness, primarily due to the suboptimal digit tokenization strategy employed by Llama3. It remains uncertain whether the proposed approach generalizes effectively to other large language models (LLMs). Does the strategy hold for tokenizers that do not merge digits?\n\nIn Section 3, why were 10-digit numbers specifically chosen for the arithmetic task? Is there a particular rationale behind this decision? Also, the Counting Characters dataset size contains only 331 samples, which makes the evaluation less convincing. \n\nIt seems like that Qwen and Llama3 share many common tokens (correct me if I'm wrong). Do these findings extend to other models, such as Llama2/Mistral? It would be more compelling if the authors could demonstrate similar results for task-specific models like CodeLlama, where spelling errors could significantly impact performance. \n\nRegarding the findings in Section 4.2, when does this robustness emerge? Specifically, how many training epochs are required for this robustness to emerge, and how does this align with the reported results? Also, I wonder if we can take the idea from Section 3 to improve the performance during the SFT stage? From Figure 1, it seems like the model always output canonical encodings, I wonder if the author has further insight why this is the case? Does your finding in Section 3 applies for Llama3-1B model in Section 4.2?\n\nFigure 4 suggests that the chat template is critical for enabling the model to become \"character\" aware. Can you provide further insights into why this might be the case? My understanding is that the template encourages the model to learn from fragmented or broken tokens. This approach seems similar to techniques used in code completion LLMs, where random token insertion is employed to train models to handle incomplete or broken tokens effectively.\n\nOverall, although I find the message the paper's conveying is interesting, I feel like the experiments conducted do not fully investigate the phenomenon.", "limitations": "While the paper presents interesting observations, it would be more informative for readers if additional details about the experimental setup were provided. See Questions.\n\nI'll happily increase my score if the authors address my questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: The paper's message is interesting and can provide further insights for future development of tokenization.\n\nWeakness: Certain experiments are not very convincing and open up questions. See Questions.", "was_revised": true, "final_justification": "While the paper leaves some open questions on why this phenomenon emerges during SFT finetuning phase (with chat template), this work can potentially offers many interesting following up work in this area.", "rebuttal": "We thank the reviewer for recognizing that our paper presents “interesting observations,” as well as the detailed comments and insightful questions. We hope you find that our response addresses your concerns.\n\n**On testing models that do not merge digits**\n\nLMs whose tokenizers do not merge digits will not have multi-digit tokens in the vocabulary, making it impossible to evaluate right-to-left tokenization (which chunks digits into groups of 3, e.g., 1000 $\\rightarrow$ [`1`, `000`]) at inference-time.\n\nWe also want to mention that the best representation of digits is still an open question. For instance, [[1](https://arxiv.org/abs/2102.13019)] found that character-level tokenization is still suboptimal and underperforms “fixed-character” tokenization, which is somewhat similar to R2L because it ensures consistent digit positions. In addition, grouping digits in chunks of 3 is more common among tokenizers of frontier LMs today.\n\n**Expanding the test sets in Section 3**\n\nThank you for identifying the limitation of small test sets. Following the reviewer’s suggestion, we expand the test sets for Arithmetic and Counting Characters to >1000 examples. For Arithmetic, we include numbers with 5 to 15 digits. We find that right-to-left tokenization of digits still significantly outperforms the canonical left-to-right tokenization, achieving 30.8% compared to 9.8% accuracy. We also expand the Counting Characters dataset by considering tokens of 5 to 10 characters in Llama3’s vocabulary. Again, we replicate the finding that character-level tokenization leads to better performance, achieving 73.5% compared to 66.5% for canonical tokenization. We will update the test sets and corresponding results in the next revision.\n\n**On evaluating models with tokenizers that differ more from $\\texttt{Qwen2}$ & $\\texttt{Llama3}$**\n\nFollowing the reviewer’s suggestion, we perform evaluations on $\\texttt{Mistral-7B-Instruct-v0.3}$ and $\\texttt{Qwen2.5-Coder-7B-Instruct}$. Note that we do not use $\\texttt{Llama2}$ or $\\texttt{CodeLlama}$ (based on $\\texttt{Llama2}$) due to very poor performance, even with canonical tokenization. The findings are consistent with those reported in the paper — $\\texttt{QwenCoder}$ retains 94% of performance and $\\texttt{Mistral}$ retains 80% of performance with random tokenization.\n\n$\\texttt{QwenCoder}$ Average Retention: Random: 94.1%, Char: 89.6%\n\n| Dataset    | Normal | Random |  Char  |\n| ---------- | :----: | :----: | :----: |\n| ARC-C      |  0.816 |  0.772 |  0.712 |\n| ARC-E      |  0.906 |  0.860 |  0.806 |\n| COPA       |  0.932 |  0.898 |  0.902 |\n| Winogrande |  0.544 |  0.538 |  0.522 |\n| Winograd   |  0.722 |  0.676 |  0.634 |\n| CSQA       |  0.794 |  0.714 |  0.664 |\n| OpenbookQA |  0.810 |  0.700 |  0.650 |\n| PIQA       |  0.842 |  0.798 |  0.734 |\n| MMLU       |  0.656 |  0.558 |  0.546 |\n| BoolQ      |  0.826 |  0.802 |  0.792 |\n| HellaSwag  |  0.760 |  0.676 |  0.596 |\n| WikidataQA |  0.552 |  0.630 |  0.576 |\n| TOFU       |  0.795 |  0.778 |  0.769 |\n| TriviaQA   |  0.510 |  0.384 |  0.354 |\n| JeopardyQA |  0.252 |  0.176 |  0.158 |\n| AlpacaEval |  0.500 | 0.5923 | 0.5293 |\n| MATH       |  0.488 |  0.448 |  0.456 |\n| CUTE       |  0.457 |  0.467 |  0.500 |\n| GSM8K      |  0.804 |  0.741 |  0.701 |\n| DROP       |  0.814 |  0.816 |  0.786 |\n\n\n$\\texttt{Mistral}$ Average Retention: Random: 80.3%, Char: 70.5%\n\n| Dataset    | Normal | Random |  Char  |\n| ---------- | :----: | :----: | :----: |\n| ARC-C      |  0.710 |  0.512 |  0.438 |\n| ARC-E      |  0.846 |  0.642 |  0.538 |\n| COPA       |  0.944 |  0.840 |  0.766 |\n| Winogrande |  0.520 |  0.454 |  0.406 |\n| Winograd   |  0.622 |  0.526 |  0.436 |\n| CSQA       |  0.724 |  0.480 |  0.454 |\n| OpenbookQA |  0.742 |  0.506 |  0.438 |\n| PIQA       |  0.824 |  0.668 |  0.612 |\n| MMLU       |  0.576 |  0.428 |  0.382 |\n| BoolQ      |  0.857 |  0.754 |  0.734 |\n| HellaSwag  |  0.704 |  0.460 |  0.390 |\n| WikidataQA |  0.820 |  0.652 |  0.542 |\n| TOFU       |  0.846 |  0.803 |  0.718 |\n| TriviaQA   |  0.760 |  0.576 |  0.498 |\n| JeopardyQA |  0.446 |  0.240 |  0.180 |\n| AlpacaEval |  0.500 | 0.4391 | 0.3552 |\n| MATH       |  0.106 |  0.096 |  0.093 |\n| CUTE       |  0.286 |  0.271 |  0.205 |\n| GSM8K      |  0.491 |  0.406 |  0.356 |\n| DROP       |  0.798 |  0.752 |  0.728 |\n\n**How many training epochs are required for robustness to emerge?**\n\nThank you for the question, we should have stated explicitly that we trained for two epochs, which we will include in the next revision. To observe how robustness varies with the amount of SFT training, we also evaluate the epoch 1 checkpoint on our robustness metrics (spelling, grammaticality, and win rate). We find that the 1-epoch checkpoint’s performance lies between the base model and the 2-epoch model. This suggests that robustness increases during the SFT process.\n\n**Why do models always produce canonical tokenizations?**\n\nWe hypothesize that models are able to understand non-canonical tokenizations in its context because it has the opportunity to “re-aggregate” information over multi-token words into the last token’s hidden representation. This process has been observed in how models process multi-token entities (e.g., [`␣New`, `␣York`]) and multi-token words [[1](https://arxiv.org/abs/2406.20086), [2](https://arxiv.org/abs/2410.05864), [3](https://arxiv.org/abs/2406.19384)]. This learned process of “internal retokenization” in early layers likely grants the robustness to non-canonical tokenizations.\n\nIn contrast, models are not trained to predict non-canonical token sequences. Of course, mathematically, models must place non-zero probability on non-canonical token sequences [[4](https://arxiv.org/abs/2109.02550), [5](https://arxiv.org/abs/2306.17757)]. This means that over many samples, models should sometimes produce non-canonical tokenizations (regardless of whether the input is canonically tokenized). However, we qualitatively observe that generations are overwhelmingly canonical.\n\n**Applying more optimal tokenizations during SFT**\n\nWe interpret the reviewer’s suggestion (“take the idea from §3 to improve the performance during the SFT stage”) as training with a non-canonical tokenizations during SFT to further improve upon the gains that we found at inference-time alone.\n\nTo test this, we finetune the $\\texttt{Llama-3.2-1B}$ base model on 30K random examples from the $\\texttt{Tulu3}$ SFT mixture: one with L2R digit grouping, and one with R2L grouping. Then, we compare the following settings: (1) the L2R-trained model with L2R inference, (2) L2R-trained model with R2L inference, and (3) R2L-trained model with R2L inference. We expect (3) to outperform (2) due to the additional opportunity to adapt to R2L tokenization during training. We evaluate these models on 1,000 instances of addition and subtraction of 4 - 8 digit numbers. We find that (1) gets 2%, (2) gets 24%, and (3) gets 12.3%, counter to our hypothesis. We conjecture that this is because (3) learns to predict two different representations of numbers between pretraining and finetuning, and when generating output, divides probability among R2L and L2R representations. In contrast, (2) continues to generate numbers L2R, even when conditioned on R2L representation of digits.\n\n**Evaluating alternative tokenization schemes on $\\texttt{Llama-3.2-1B-Instruct}$**\n\nFollowing the reviewer’s suggestion, we investigate whether the performance improvements we identify in §3 for $\\texttt{Llama-3.1-8B-Instruct}$ also apply to $\\texttt{Llama-3.2-1B-Instruct}$. On the Arithmetic task (expanded to include 1000 examples with arithmetic for 4-8 digit numbers instead of 5-15 digit numbers to account for weaker model abilities), we find that R2L segmentation achieved 32.7% accuracy, a substantial improvement over 9.7% with canonical tokenization.\n\nHowever, for the other four tasks (Common Morpheme, Codeline Description, Acronym, and Counting Characters tasks), Llama-3.2-1B-Instruct consistently performs worse when presented with character-level tokens. We hypothesize this is because, as we observed, weaker models are less robust to non-canonical tokenizations compared to stronger models. \n\n**Why is the chat template crucial to robustness?**\n\nWe believe that the chat template is crucial for indicating a new turn of conversation, “freeing” the model from continuing the perceived misspellings in the context. Indeed, we show in §4.2 that it does not need to be an actual “chat template,” and a generic “Question:/Answer:” template that separates the user and model turn produces the same effect. \n\n**Regarding more detailed documentation**\n\nThank you for raising this important point. We plan on updating the manuscript to include more details about the experimental setup, alongside the exact script/settings that we used to train/evaluate our models and our benchmarks."}, {"review_id": "daOSduyVXp", "reviewer": "Reviewer_2MzU", "summary": "This paper presents a study about how non-canonical tokenization of the input text affects language models’ ability to provide correct outputs. First, the authors show the instruction-tuned versions of Qwen-2.5-7B, Llama-3.1-8B, and Olmo-2-7B, on a wide set of benchmarks, show only a small performance degradation due to non-canonical tokenization, and that their performance is negatively correlated with the granularity of the tokenization (i.e., the more tokens are used to express the same input, the lower the performance). Then, the authors show that non-canonical tokenization can lead to higher model performance in tasks such as counting characters, identifying common morphemes in different words, and performing arithmetic calculations. Finally, the authors find that the instruction-tuned models’ robustness to different tokenizations is mainly due to the SFT stage of training and in particular to the introduction of the chat template that distinguishes separate turns in the model-user dialogue.", "questions": "1. The robustness of language models to non-canonical tokenizations is likely due to the presence of misspellings in the pre-training data. For instance, GPT-4o’s tokenizer has a single token for “Seattle”, but splits it in different ways depending types (e.g., “Seat-le” or “Se-att-lle”). I wonder if the authors looked into whether a high number of misspelling for a specific word in the training data might lead to better”understanding” (in the sanse described in Section 4.3) of the non-canonically-tokenized word.\n2. In Section 4.1, can you provide the value for the “Spelling” and “Grammaticality” metrics for the canonical tokenization?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is very clearly written, with well-structured experiments and thorough analyses.\n- The findings offer practical insights for researchers working on tokenization, model robustness, and internal representations of language.\n- The observation that non-canonical tokenization can improve performance in certain tasks is novel and may inspire future work on task-aware or adaptive tokenization methods.\n\n**Weaknesses**\n- The paper could benefit from a deeper discussion of *why* language models are able to interpret non-canonically-tokenized inputs effectively. See Question 1 below.", "was_revised": true, "final_justification": "I will maintain my score, as I think the paper presents an interesting study and should be accepted.", "rebuttal": "We thank the reviewer for their recognition of the “thorough analysis” in our experiments, the “novelty” and “practical insights” of our findings, and the potential to “inspire future work on adaptive tokenization.” \n\n**Are LMs robust to non-canonical tokenizations due to misspellings?**\n\nWe thank the reviewer for this insightful hypothesis. Because it would be difficult to automatically identify misspellings of words, we designed a similar experiment to study this question. In particular, we consider the diversity of ways in which a token is internally segmented (when appearing within a larger word), and measure whether that is correlated with the model’s understanding of the spelling of that token. Consider the token `bio` — when appearing as a substring in, e.g., “microbiology,” it may be segmented internally, e.g., as [`microbi`, `ology`], with the token boundary appearing between “bi” and “o.” We hypothesize that LMs will better understand tokens that are segmented in more unique ways. \n\nTo test this, we use tokens from the vocabulary of $\\texttt{Llama-3.1-8B-Instruct}$ which are 5-15 characters long and occur 50-500 times in a 13GB sample of The Pile. From there, we construct two subsets of tokens: those that are segmented in at least 5 different ways, and those that are consistently segmented the same way. We then evaluate the model on intra-token understanding using task constructions from CUTE. Shown in the table below, we find that the model consistently understands commonly segmented tokens better. We believe that this indirectly provides support for the reviewer’s hypothesis that misspellings would also lead to better understanding, as it similarly splits a string into a different sequence of subtokens.\n\n| Variation         | Commonly Segmented | Rarely Segmented |\n| ----------------- | :----------------: | :--------------: |\n| Char Deletion     |        **38.8**       |       20.5       |\n| Char Substitution |        **18.0**        |        4.1       |\n| Char Swapping     |        **16.2**        |        8.2       |\n| Spell             |        **82.5**        |       54.8       |\n| Inverse Spell     |        **60.2**        |       36.3       |\n\n\n**Spelling and grammaticality comparison for canonical tokenization in §4.1**\n\nWe provide a comparison to canonical tokenizations below. We see that the performance of SFT, DPO, Instruct models are similar to those given non-canonical tokenizations (Figure 3), whereas the Base model performs much better given canonical compared to non-canonical tokenizations. This is consistent with what we expect, given that robustness to non-canonical tokenization arises after the SFT stage.\n\n*Spelling*\n\n| Model | Base | SFT  | DPO  | Instruct |\n| ----- | ---- | ---- | ---- | -------- |\n| OLMo  | 61.3 | 79.6 | 78.3 | 77.5     |\n| TULU  | 67.6 | 78.5 | 79.4 | 79.7     |\n\n*Grammaticality*\n\n| Model | Base | SFT  | DPO  | Instruct |\n| ----- | ---- | ---- | ---- | -------- |\n| OLMo  | 72.4 | 92.9 | 92.9 | 92.4     |\n| TULU  | 78.2 | 91.8 | 95.4 | 95.5     |"}]

Author rebuttal:

Meta-review: This paper presents a thorough study of how well LMs handle non-canonical tokenizations of input text. This is an important, but under-explored problem, and the findings are surprising: LMs are surprisingly robust. This opens up the door to swap in alternative tokenizations at inference time to boost performance, as is shown in the paper, which presents an exciting direction for future work.

Reviewers agree that this paper studies an important topic, that the experiments conducted are thorough, and that the findings are (very) interesting. There remains some uncertainty about precisely the role SFT plays for this behavior to emergence, and how to best leverage the insight for adaptive tokenization at inference time. However, these can be addressed in future work and should not stand in the way of publication.
Decision: Accept (spotlight)