StochasTok: Improving Fine-Grained Subword Understanding in LLMs
Authors: Anya Sims
Thomas Foster
T. Duy Nguyen-Hien
Klara Kaleb
Joseph Lee
Jakob Nicolaus Foerster
Yee Whye Teh
Cong Lu
Venue: ICLR 2026 Poster (2026)
Keywords: language models
tokenization
pretraining
finetuning
subword understanding
Primary area: foundation or frontier models, including LLMs
TL;DR: A simple stochastic tokenization method—randomly splitting tokens before pretraining—that dramatically improves fine-grained, subword-level understanding in language models without any compromise to benchmark performance or increase in training cost.

Abstract: Subword-level understanding is integral to numerous tasks, including understanding multi-digit numbers, spelling mistakes, abbreviations, rhyming, and wordplay. Despite this, current large language models (LLMs) still struggle disproportionally with seemingly simple subword-level tasks, like counting the number of 'r's in 'strawberry'. A key factor behind these failures is tokenization, which obscures the fine-grained structure of words. Current alternatives, such as character-level and dropout tokenization methods, significantly increase computational costs and provide inconsistent improvements. In this paper, we revisit tokenization and introduce StochasTok, a simple, efficient stochastic tokenization scheme that randomly splits tokens during training, allowing LLMs to ‘see’ their internal structure. Our experiments show that pretraining with StochasTok substantially improves LLMs’ downstream performance across multiple subword-level language games, including character counting, substring identification, and math tasks. Furthermore, StochasTok’s simplicity allows seamless integration at any stage of the training pipeline, and we demonstrate that post-training with StochasTok can instill improved subword understanding into existing pretrained models, thus avoiding costly pretraining from scratch. These dramatic improvements achieved with a minimal change suggest StochasTok holds exciting potential when applied to larger, more capable models.

Published as a conference paper at ICLR 2026
STOCHASTOK: IMPROVING FINE-GRAINED SUBWORD UNDERSTANDING IN LLMs
Anya Sims
Thom Foster
FLAIR, University of Oxford anya.. sims@statts ac.uk
FLAIR, University of Oxford
Tuan-Duy H. Nguyen
Klara Kaleb University of Oxford
Joseph Lee University of Oxford
National University of Singapore
Jakob N. Foerster
Yee Whye Teh University of Oxford
Cong Luu University of British Columbia
FLAIR, University of Oxford
ABSTRACT
Subword-level understanding is integral numerous tasks, including understand- ing multi-digt numbers, spelling mistakes, abbreviations, rhyming, and wordplay. Despite this, current large language models (LL.Ms) still struggle disproportionally with simple subword-level tasks like How many 'r's in 'strawberry'! ? A key factor behind these failures is tokenization which obscures the fine-grained structure of words. Current alternatives, such character-level and dropout tokenization methods, significantly increase computational costs and provide inconsistent im- provements. In this paper revisit tokenization and introduce STOCHASTOK. simple, efficient stochastic tokenization scheme that randomly splits tokens during training, allowing LLMs 'see' their internal structure. Our experiments show that pretraining with STOCHAS substantially improves L.MS' downstream performance across multiple subword-level language games, including character counting, substring identification, and math tasks. Furthermore, STOCHASTOR s simplicity allows seamless integration at any stage of the training pipeline; and we demonstrate that post- post-training with STOCHASTOK can instill improved subword understanding into existing pretrained models, thus avoiding costly pretraining from scratch. These dramatic improvements achieved with a minimal change sug- gest STOCHASTOK holds exciting potential when applied larger, more capable models. Code apen-sourced gi gitbub. com/ anyas ims st sccastook.
INTRODUCTION
Large language models (LLMs) have achieved remarkable progress on wide range of tasks (Achiam et al., 2023; Team et al., 2023; Dubey et al., 2024). However, their reliance on tokenization (Sennrich al.. 2016) obscures how humans naturally perceive language. For example, while humans see "book" and 'cook' as differing by single letter, when training L.Mss, we always treat these words as distinct token IDs!. This makes subword-level tasks such as How many 'r's in 'strawberry' ? difficult, even for current state-of-the-art LLMs. Whilst some advanced reasoning models, such as OpenAl's ol (Jaech et al., 2024), have recently started to show promise, it has required a vast increase in model size and training complexity that seems disproportionate to the simplicity of such questions. In the arts, this shortcoming impacts wordplay. rhyming, and understanding etymology, while in the sciences, it is needed for handling multi-digit numbers, chemical formulae, and mathematical equations. Moreover, these failures highlight a fundamental inability of LLMS to understand how humans perceive language, essential aspect of effective communication with humans.
This limitation in standard tokenizers has motivated research into stochastic tokenization, where 'stochastic tokenization' refers methods in which the same text may be encoded as multiple possible
c.g., "hook'=33 =3092 and 'cook's c in the GPT-4o and GPT-4o mini models (Hurst al., 2024).

---

Published as a conference paper at ICLR 2026
Pretraining Tokenizer
Normalized Accuracy
Deterministic
Figure 1: STOCHASTOK pretraining al- lows the learned representations cap- ture the fine-grained details of how humans "see" language. This is demonstrated as models pretrained with
StochasTok (p=0.05)
StochasTok (p-0.1)
0.6
BPE-dropout
Pretraining
0.4
SToCHASTok be finetuned swer language game questions with no compromise to ability in other domains.
0.2
Validation
Iteration
token sequences. well-known existing method BrE-dropout (Provilkov et al., 2020), which adds randomness by skipping BPE merge steps. In this work, we propose a simpler, more flexible, and more effective alternative: rather than modifying the original tokenization process, we instead allow LLMs to directly 'see' inside tokens by randomly splitting them into equivalent pairs of smaller tokens with some small probability.
Our experiments show that adding this minimal additional preprocessing step significantly alters the model's representations, allowing them to capture subtoken-level morphological structure. Compared prior stochastic tokenization methods (Provilkov al., 2020; Kudo, 2018), find STOCHASTOK to be significantly more effective, while also having strong practical advantages of being faster, simpler, compatible with any base tokenizer, and applicable post-hoc to existing pretrained models.
We demonstrate three main results. Firstly, language models pretrained with STOCIAATTO quickly adapt near-perfect accuracy on several language game tasks (such as 'Which word has the most e' "s'' or 'Which word is the shortest?"), while models pretrained with deterministic tokenization or BPE-dropout struggle (see Figure 1). We test this on two sets of language game tasks: (I) LangGame- our novel set of subword understanding tasks, and (2) the CUTE benchmark of language manipulation tasks (Edman al., 2024). Secondly, we show that STOCHASTOW enables models to grok multi-digit addition, a dramatic change in learning behavior compared to BPF-dropout or deterministically trained models (Lee et al., 2023). Thirdly, since STOCHASTOK with existing pretrained models, we demonstrate that it can be used to 'retrofit' larger existing pretrained models with improved subword understanding. thus mitigating the need to pretrain from scratch. In summary, SToCHASTOK provides stark performance improvement with minimal cost or implementation changes, and we believe our results at the modest scale have potential for major impact LLM ability when used pretrain finetune larger, more capable models.
compatible
BACKGROUND
Tokenization (Sennrich et al., 2016)-the process of converting raw text into tokens-serves serves two essential roles in the LLM pipeline. Firstly, it converts text into sequence of integers enable processing by the LLM. Secondly, it compresses sequences of characters into shorter sequences of tokens, which increases both performance and computational efficiency.
Standard Deterministic Tokenization. A tokenizer consists of two main components: vo- cabulary, and an encoding function for converting text into a sequence of token IDs. The decod- ing procedure shared by all tokenizers simply maps token IDs back text strings. For instance, with vocabulary (0: The, 1 2:at, S: the sequence [0, 1, 2, 3, 2] decodes to "The_cat_sat"
The main tokenizers are Byte-Pair Encoding (BPE; Sennrich et al. (2016)) and Unigram (Kudo, 2018). BPE constructed by starting with individual character tokens and iteratively merging the most frequent adjacent token pairs in training dataset, yielding a fixed-size vocabulary and hierarchical set of merge rules. For encoding, text is initially split into character-level tokens, and the merge rules applied repeatedly until no further merges are possible. In contrast, Unigram starts with a large candidate vocabulary and iteratively prunes tokens that least increase the dataset's log seg-likelihood under a unigram model, using the Viterbi (Viterbi, 1967) and EM (Dempster et al., 1977) algorithms to compute and optimize token probabilities. For encoding, the tokenization with the highest probability under the learned unigram model is selected using the Viterbi algorithm. BPE
2

---

Published as a conference paper at ICLR 2026
currently the choice of most SOTA LLMs (Groeneveld al., 2024; Dubey al., 2024; Team et al., 2024; Jiang et al., 2023; Abdin et al., 2024; Guo et al., 2025; Yang et al., 2024; Biderman et al., 2023) due to having much lower memory requirements than Unigram.
Stochastic Tokenization. BPE and Unigram deterministic tokenizers, meaning the same input text always produces the same tokenization. We define stochastic tokenization as any tokenizer whose encoding function may produce multiple alternative tokenizations for the same input. With rocab=(0:e) 1;x, 2:a, 3 3:m, 4:p, 5 6:exam, 7:ple, 8: exaxmplee; ), for example, the word 'example' might be mapped any of [8], [6, [0, 1, 2, 3, 9, 5, 0], etc., since the decoding procedure (identical to deterministic tokeniz- ers)will map each of these back to the text 'example'.
The two main prior stochastic tokenization methods are Subword Regularization and BPE-dropout. Subword Regularization (Kudo, 2018) extends Unigram by sampling from alternative tokenizations according to learned unigram model probabilities. However, this adds complexity and computa- tional overhead to the already expensive Unigram procedure, and introduces intricacies involving overlapping candidates, beam tuning. and numerical stability. BPE-dropout (Provilkov al., 2020) introduces stochasticity by randomly omitting some merge operations of BPE during encoding. Unfortunately, this results in different vocabulary from the original BPE tokenizer, preventing easy application to pretrained models. also incurs additional drawbacks such as higher computational costs, unwanted tokenization dependence text length, and is only compatible with BPE. In our experiments we therefore compare to BPE, the defacto standard in SOTA LLMs, and BPE-dropout, the only prior BPF-compatible stochastic variant (see Section 8).
STOCHASTOK
Different possible tokenizations for the same text using STOCHASTOR:
text 'An exampl aentence'
vocab" (AR, Lexample,
ple,
ample.
Random seed
Random seed 2
Random seed
Initial tokenization:
An example sentence)
example sentence]
An example sentence
After expand step:
example
An example sentence
example sentence
After expand
An example sentence
An example sentence
[
An example sentence
Underlined denotes the token sampled for expansion. Bold denotes the equivalent of tokens after expansion.
Figure 2: STOCHASTOK involves iteratively sampling tokens to 'expand' into equivalent pairs of tokens in the vocabulary, resulting in multiple possible tokenizations for the text. The exposure to alternative tokenizations enables LLMs naturally learn about lnn-grained subtoken-leved morphological composition tokens.
In this section, describe STOCHASTOK, simple, lightweight, stochastic tokenization scheme that, unlike prior work, is compatible with any base tokenizer or pretrained model.
SToCHASTok involves two steps:
1. Tokenize with the base tokenizer to get list of token_ids.
2. Iteratively apply 'expand' steps in which token is sampled random and (if possible) split into pair of equivalent tokens in the vocabulary (as depicted in Figure 2). This is repeated for p en (token_ids. iterations, where is hyperparameter.
PIn BPF, intermediate tokens not present in the final tokenized training dataset are removed from the vocahulary, meaning RPF-dropout can produce tokens outside the original vocabulary.
3

---

Published as a conference paper at ICLR 2026
In Step 2. if no equivalent pairs of tokens exist for the sampled token (e.g., if the token is already a single character), then the expand step is skipped. Full pseudocode is given in Section A.3, and further illustrative examples in Section A.4. Through this repeated token re-segmentation the model is exposed to many alternative tokenizations; for example, the word [example] may appear in the dataset as any of: [example], [exam] ple], [ex] ample], [ex|am) am/ple], [e Ix am am/plel, etc, thus allowing it to learn the fine-grained structure of words.
SToCHASToR has several practical advantages:
* Cheap and efficient. STOCHASTOK is considerably cheaper than existing methods both in terms of memory and compute. Rather than re-tokenizing from scratch, data can be tokenized once and cheaply expanded for varying numbers of "expand steps' to achieve different levels of stochasticity.
Compatible with any tokenizer. Unlike BPE-dropout or Subword Regularization, STOCHASTOK can be applied to any base tokenizer (BPE, Unigram, WordPiece, etc.)
without requiring any knowledge of the base tokenizer itself.
Extremely simple. STOCHASTOK is simply a lightweight post-processing step after tok- enization. Everything else-including the training loop romp-remains unchanged.
Preserves original vocabulary. Perhaps most significantly, STOCIASTOK maintains the original tokenizer vocabulary, thus allowing straightforward application to any stage of the LLM pipeline. In Section 4 we apply STOCHASTOk during pretraining and switch it off seamlessly for downstream finetuning, while in Section 6, we apply STOCHASTOO after pretraining to instill subword understanding into existing pretrained models.
Robust to hyperparameter choice. STOCHASTOK is robust to hyperparameter choice (see Figure 5) and hence does not require careful tuning. By default we use p 0.1, and show similar effectiveness with 0.05 and other values.
In the following sections, we demonstrate STOCHIASTOK's empirical advantages. Firstly, we show that pretraining with STOCHAS dramatically improves downstream performance on language game tasks, while being (a) extremely robust to hyperparameter choice and (b) exhibiting out- of of-distribution generalization properties (Section 4). Next, examine math tasks and find that models trained with STOCHASTOR quickly grok multi-digit addition-and moreover generalize to unseen test tokenization schemes- whereas models trained with existing tokenizers struggle, even when tested with the matching tokenizer (see Section 5). We then apply STOCHASTOR to existing pretrained models and demonstrate that it can be used 'retrofit' improved subtoken understanding into larger deterministically pretrained models (Section 6). Finally, we provide insights into the internal mechanisms of SToCHASToK-tran models compared to models trained with standard tokenization (Section 7).
4 STOCHASTOK PRETRAINING ENABLES SUCCESS IN LANGUAGE GAMES
Setup. In this section, we look at the effect of STOCHASTOK when applied during pretraining. We build the baseline open-source setup of Hillier al. (2024) (a 50M-parameter model, using GPT-2 BPE tokenizer, trained on the OpenWeb compare four models: (1) Pretrained with standard deterministic tokenization, (2) Pretrained with STOCHASTOK, (3) Pretrained with BPE-dropout, and (4) No pretraining. Firstly, in Figure 3, we verify that STOCHASTOK requires no compromise in original language modeling performance (see Section C. for benchmark details).
dataset-see Section C. for full details). We
Task
Question
Answer
1
Letter
Which word the most letter 'o's? The options arc: I reason, step, continent, their).
continent check
2
Contains
Which choice contains 'ec"? The option words are:
children, require, check].
3
Starts
Which option string starts with 'moo? ? The available options: I case, ask, month, event].
month lead
Finss
What option word ends with 'nd"? The option words are: [osst lead, south, sun].
5
Longest
Which string is the longest? The available choices: wild, dear, had, section].
section
Shortest
Which is shortest? The possible option works: thought, joh, circle, nothing].
job
Table 1: We introduce "anngGmm,," novel dataset consisting of six question types testing fine fine-grained subword- level understanding.
4

---

Published as a conference paper at ICLR 2026
ARC
BLIMP
HellaSwag
Winagrad
0.5
0.01
Normalize Accuracy
0.06
0,02
0.15
0.1
-0.02
0.40
10k
20k
30k
0
10K
20c
30k
10k
20k
30k
10k
20k
30k.
Iteration
Iteration
Iteration
Iteration
StachasTok (p=0.05)
StochasTokk (p=0.1)
Datarministic BPE dropout
StachasTok (p-0.02)
StochasTok (p-0.013
Figure 3: We first verify that STOCHASTOK does not compromise test performance across a wide variety of standard language understanding benchmarks.
Performance on Language Game Tasks. We
Pretraining Tokenizer
now finetune each of the base models above two sets of language game tasks: (1) LangGame,
Normalized Accuracy 2. 0.2
Deterministic
(pm0.05)
0.6
StochasTok (p=0.1)
and (2) CUTE. LangGame is novel dataset consisting of six different tasks, including iden- tifying word lengths, substrings, and individual letters. Examples are shown in Table 1, and additional detail is given Section B. The CUTE benchmark contains further language ma- nipulation tasks (Edman al., 2024) (see Sec- tion B.2 for examples). Critically, each model finetuned identically, using deterministic BPE tokenization.
BPE-dropout
No Pretraining
Train
Iteration
Figure 4: Pretraining with STOCHASTOX enables sig- nificantly higher performance on the CUTE language manipulation tasks (in addition to the LangGame tasks- Figure 1). (For "normalized accuracy," 0 is random guessing and perfect.)
Figure shows performance on the LangGame questions. We observe that the models pretrained with SToCHAsTok quickly achieve near wear-perfect accuracy, while the models pretrained with deterministic tokenization or no pretraining are unable to reach high accuracy. This suggests that, as well as the token-level structure learned with deterministic tokenization, STOCHASTOK enables models to additionally capture subtoken-level fine-grained morphological structure. The prior method of BPE-dropout gives some the benefits of stochastic tokenization, but still performs significantly worse than STOCHASTOR in addition to being significantly more complex. In Figure 4, we see that STOCHASTOX gives similar stark performance difference on the CUTE language manipulation benchmark, thus giving further evidence that STOCHAS significantly changes the representations of the model to enable fine- mne-graieed character-level manipulation.
Robust to Hyperparameter Choice and OOD Questions. In addition to significant performance increases on both language game benchmarks, we find that the benefits of stochastic tokenization robust over an order of magnitude range of the hyperparameter (see Figure 5). Furthermore, we find that this skill is learned in a way that enables the model to generalize to set of holdout language game question types which the train/validation questions all involve identifying sub- strings/prcixxesssufiisss where the substring/prefix/sufhii is always less than or equal to half the answer length, while in the holdout set the substring/pretix/suuiii is always longer than half the
Pretraining Tokenizer
Validsioo Acourscy
na
Dateministic
StochasTok (p=0.05)
0.8
StochasTok (p-0.3)
Normwlized
0.6
R4
Valicitioo
Ron-zoen zero
0.001
0.02
0.1
Stochesticity Level
Iteration
Figure 5: STOCHASTOK is effective over a wide range of stochasticity lev- cls (log x-scale), meaning is robust hyperparameter choice.
Figure 6: Models pretrained with STOCHASTOK successfully gen- cralize to out-of-distributtin language game questions, while those pretrained deterministically exhibit significant generalization gap (and much lower in-distribution performance).
5

---

Published as a conference paper at ICLR 2026
answer length. In Figure 6, observe that models pretrained with stochastic tokenization gen- eralize pear-perfectly while the deterministic ceanizatiio-pretraine equivalent has a significant generalization gap in addition to much lower in-distribution performance.
Transfers to Larger Models. Next, we verify that these findings transfer larger settings by applying STOCHASTOK to modded-manogp" baseline (Jordan et al., 2024a). This setup has a differ- ent architecture and model size of GPT-2
Pretraining Tokenizer
Deterministic
StochasTok (p=0.05)
StochasTok (p=0.1)
1..0
BFE-dropout
with 275M parameters, different train- ing dataset Web Penedo et al. (2024)), and different optimizer (Muon Jordan
Normallzed
0.4
No Pretraining
0.2
Train
Validation
et al. (2024b)). In Figure 7, we see that SToCHASToK gives a similar performance benefit in this larger setting, suggesting that SToCHASToK scales larger models.
Iteration
Figure STOCHASTOK enables improved LangGame performance in larger models.
STOCHASTOK ENABLES LLMS TO GROK MATH TASKS
In addition to language game-type tasks, tokenization also poses difficulties in learning math, due to obscuring the relation between numbers, for example in GPT-4o (Hurst et al., 2024), the numbers '2' "20", '200', '201' are tokenized as 17, 455, 1179, 667 respectively. This poses such a significant additional difficulty for language models that prior works commonly
tricks like adding 's between every character (to force tokenization to keep each digit separate), or using custom character-level tokenizers for digits sidestep the issue (Zhang et al., 2024; Power al., 2022; Lee et al., 2023).
We hypothesize that since STOCHASTOK improves sub-token level awareness, it may also help in learning multi-digit math tasks. To test this, we train on the task of multi-digit addition starting from the 50M-parameter setup Hillier et al. (2024). Examples of the questions are given in Section B.3. We compare the performance of models trained with: (1) standard deterministic tokenization, (2) BrE-dropout, (3) STOCHASTOK, and (4) character-level tokenization. Figure 8, for each of the four models we plot the accuracy with the question tokenized with each of the four methods.
In Figure 8 left, we see as expected that the character-level-rraiee model quickly achieves near- perfect accuracy when the questions are tokenized character-wise (and gets near-zero accuracy when the questions are tokenized differently). Figure 8 middle-left and middle-right, we see that the models trained with standard deterministic tokenization and BPE-dropout struggle to grok the task, appearing to slowly learn examples with the accuracy increasing linearly, even with the matching question tokenization. By contrast, in Figure 8 right, the model trained with STOCHASTOR quickly groks the task and reaches near-perfect accuracy, not just when the question is tokenized with the matching tokenizer, but also when the question is tokenized with any of the other three tokenizers that were unseen during training. This suggests that STOCHASTOk significantly enhances a model's ability understand relationships between multi-digit numbers.
Tokenization:
Character-level
Deterministic
BPE-dropout
StochasTok
Train Accuracy
Evaluation Tokenizetion:
0.5
aPF-dropout
StochesTok
Cherocler-Level
10k
10k
Val Accuracy
0.5
Iteration
Itration
Itoration
Figure 8: STOCHASTOK allows models to grok multi-digit addition. Unlike training with character-level or deterministic BPE tokenizers, training with SToCHASToK achieves near perfectt validation accuracy even when tested with questions tokenized with methods not seen during training.

---

Published as a conference paper at ICLR 2026
STOCHASTOK CAN INSTILL SUBWORD UNDERSTANDING INTO EXISTING PRETRAINED MODELS
Pretraining often prohibitively expensive. In this section, we therefore investigate whether StochasTok can be used to instill improved subword understanding into models that have already been pretrained with an alternative tokenization method, offering a more cost-effective alternative to full retraining from scratch. For our first experiment, Section 4, which was trained for 30k iterations on Open WexText using deterministic BPE. We call this the 'base model.' We then continue to train for additional 2k iterations OpenWebTex with STOCHASToN tokenization, which we refer to as continued pretraining (CPT). As a control, we also perform CPT with standard deterministic BPE. As before, we then try finetuning tasks. In Figure 9, we show that small amount of CPT is sufficient to enable the models to fit the language game questions near-perfectly, significantly higher than all of the controls. This suggests that the 2k steps of CPT with STOCHASTOK were effective in instilling subword understanding into the pretrained model.
start with the 50M-parames model from
the LangGame
Larger Pretrained Models. Next, we test this larger open-source model. In Figure 10, we compare the ability of GPT-2 (Radford et al., 2019) to fit the language game tasks with (1) no additional pretraining, (2) 7 iterations of CPT with deterministic BPE, and (3) 7k iterations of continued pretraining with STOCHASTOK. CPT with deterministic BPE has no effect on the ability to learn the LangGame tasks, whilst STOCHASTOK again allows the model to reach significantly higher accuracy.
Normalized Accuracy
Normalized Accuracy
0.5
%
500
1.5k 2k
Iteration
0
Base model
Iteration
Base model CPT deterministic
Base model CPT with StochasTok
GPT2 + CPT with determinstic BPE
Pretrained with StochasTok
GPT2 + CPT with StochasTok
Figure 9: A small amount of continued pretrain- ing (CPT) with STOCHASTOK significantly im- proves subword awareness in 50M-parameter eeeemmnistically-peetaaii baseline.
Figure 10: The effectiveness of STOCHASTOR in con- tinued pretraining (CPT) transfers the larger setting, enabling the pretrained GPT-2 model fit language game tasks.
7 ANALYSIS
Finally, we present analysis of how STOCIISSTOK enables the improvements in subword-level understanding. In Figure 11, we show completions when prompted with different tokenizations of the same prompt. We find that-as expected the responses from the model trained with STOCHASTOK are much more consistent across different prompt tokenizations, while the standard tokenization- trained model quickly breaks down when exposed to alternative tokenizations.
Different prompt tokernizations The chef sighed and put the The chat sigh ed do ut he et signed ind put he The chef sighed and
Deterministic Training knife his mouth
StechasTok Training dichi the table dish
on the table and seld and then he turned
oust ache the dish the table dish
sighed lP ut The ctef ished an the
dish on the tabie and then
dish the table
Figure 11: Generations given multiple different tokenizations of the same prompt. We find the STOCHASTOK trained model be consistent. while the standard-trained model breaks down when prompted with alternative tokenizations, showing STOCHASTI ToK improves tokenization robustness. More examples are provided Section D.I.
7

---

Published as a conference paper at ICLR 2026
Deterministic Tokenization
StochasTok
Words
coulc
20
depind
carly
forward
10
Grygen
post
segment
story
-10
while
whose
-20
-10
Figure 12: STOCHASTOs visibly results in the internal representations for alternative tokenizations of the same words being much more closely aligned.
0.9
Next, in Figure 12, we visualize the inter- nal representations, both with and without stochastic tokenization. We fit PCA model on the embeddings of the top Ik most com-
mon words and then plot the results for alter-
Normalizen Mean Distance
0.8
native tokenizations of the same words, us- ing random sample of 20 words. We ob-
0.75
HHHNN
serve that, when using stochastic tokeniza-
tion, the embeddings for alternative tokeniza- tions of the same word are significantly more closely aligned and visibly capture subword- level structure.
0.7
0.65
For more quantitative measure of this, in Figure 13, we plot how the mean distance be- tween representations of alternative tokeniza- tions of the same word evolves through the transformer layers. We observe that when trained with STOCHASTOK, each layer maps alternative tokenizations progressively closer to the same representation, while the determin- istically pretrained model does not have this behavior.
0.6
Deterministic Training
0.55
StochasTok Training
0.5
leyer u
layer
layer
toyer
layer 4
layer
layer 6
layer
Figure STOCHASTON -rr-taddd models progressively map equivalent tokenizations closer together.
RELATED WORK
Subtoken-level understanding. Numerous papers have studied LLMs' surprisingly poor ability on subword-level tasks (Xu & Ma, 2024; Fu al., 2024: Zhang al., 2024: Shin & Kaneko, 2024; Edman et al., 2024: Marjich et al., 2025; Kaushal Mahowald, 2022; Itzhak Levy, 2021, inter alia). However, solving these tasks remains challenging, despite improvements to core capabilities and reasoning in other measured benchmarks.
Stochastic Tokenization. Stochastic variants have been proposed for many tokenizers, including BPE-dropout for BPE (see Section 2), Max.Macch-dropout (Hiraoka, 2022) for WordPiece (Schuster & Nakajima, 2012), LCP-dropout (Nonaka et al., 2022) for LCP (Cormode & Muthukrishnan, 2002), and Subword Regularization and STM (Hiraoka et al., 2019) for Unigram (see Section 2). These prior methods are all -akenizer-specific,, for example MaxMatch-dropout randomly omits the longest
'The activations after the final attention layer the position of the last token for each word.
8

---

Published as a conference paper at ICLR 2026
next subword when tokenizing with WordPiece, while LCP-dropout adds stochasticity by randomly partitioning the input before applying LCP tokenization. Similarly, Subword Regularization and STM rely Unigram's 's unigram model for calculating tokenization probabilities using the FFBS or Viterbi algorithms (Scott, 2002; Viterbi, 1967), (but rather than choosing the highest probability tokenization, they instead sample from this distribution). Therefore, since almost all current LLMS use BPE tokenization, these methods are almost never applicable.
BPE-dropout is, therefore, the relevant baseline. As described in Section 3, compared BPE-dropout, SToCHASTOK has several practical advantages: Firstly, to apply BPE-dropout, we require access to the exact merge hierarchy of the BPE tokenizer. By contrast. STOCHASTOK can be easily applied to any base tokenizer without any knowledge of the base tokenizer itself (it only requires knowledge of the model's vocabulary- property of the model). Secondly, STOCHASTOK can be applied at any stage of the L.M pipeline, even to pretrained models, since it preserves the same vocabulary as the original tokenizer. contrast, switching between BPE and BPE-dropout changes the possible vocabulary. leading either to out-of- vocaa-lvocabulary tokens requiring change to the model. Finally, SToCUASTOK is essentially a lightweight processing step after tokenization, meaning it can be used in conjunction with fast, compiled implementations of base tokenizers. By contrast, BrE-dropout requires tokenizing from scratch and compiled implementations of BPE-dropout for predefined BPE tokenizers (i.e., pre-specified vocabulary and merge hierarchy) not readily available, thus often making BPE-dropout prohibitively expensive.
Byte-level models. An alternative line of work in improving character-level understanding is byte- level or 'tokenizer-free'' models, which operate directly on characters. This approach removes the inductive bias imposed by tokenizers' vocabularies and naturally handles unusual words and typos. However, the naive approach is prohibitively inefficient due to increased sequence lengths. As a result, approaches such as hierarchical architectures, local convolutions, patching mechanisms, or auxiliary losses, are necessary to bring down the effective sequence lengths (Al-Rfou et al., 2019; Clark et al., 2022; Yu al., 2023; Pagnoni et al., 2024). However, these come at the cost of added complexity and still substantially higher computational requirements (Xue et al., 2022; Nawrot et al., 2022). Consequently, tokenization- based models currently remain more compute-efficiet and more practical in general. With SToCHASTOK we enable models to get the benefits of byte-level understanding without needing to move to an alternate framework.
DISCUSSION AND FUTURE WORK
While there are adoption costs with any changes to the LLM pipeline, STOCHASTOK minimizes these through its simplicity, wide compatibility, and demonstrated ability to be applied to existing pretrained models. Looking ahead, a valuable addition would be to apply STOCHASTOK 's on a larger scale to investigate other potential benefits, such as greater robustness to spelling mistakes and other general improvements. In this paper, we focus only on English, and it would also be interesting explore the effect of STOCHASTOK on languages with different alphabets, structure, and levels of morphology. Finally, combining STOCHAS with recent orthogonal advances in tokenization, such as Liu et al. (2025), represents another promising direction for future research.
10 CONCLUSION
Our experiments demonstrate that incorporating STOCHASTOK any stage of training dramatically enhances language models' ability to represent subword-level structures central to human language perception. Tokenization has recently received less attention than other methods, such as finetuning and prompting techniques, since its position at the start of the pretraining pipeline often makes experimentation prohibitively expensive. Our work shows that tokenization modifications can be exceptionally effective, not only at the pre-training stage but also in the continued pre-training and post-training stages. Our efficient, cheap changes can help fix pervasive idiosyncrasies and lead to significant improvements in language understanding. Given the stark performance benefits demonstrated here, we are excited to assess the impact of STOCHASTOK on more challenging tasks such coding, algebra, scientific reasoning when applied to more capable models. We hope work encourages renewed exploration of tokenization schemes to bridge the gap between human and machine language perception.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank the contributors of Open WebText and the maintainers of making their resources publicly available under the MIT License. AS supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Machine Learning (EP/S023151/1) YWT's research is supported by the Ministry Digital Development and Information (MDDI) under the Singapore Global AI Visiting Professorship Program (Award No. AIVP-2024-002).
for
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. /42/9, 2024. 3
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.087771 2023. 1
Rami AI-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-leve language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3159-3166, 2019.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023. 3
Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Compu- tational Linguisties, 10:73-91, 2022. 9
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1oX3vv1800305557,, 2018. 20
Graham Cormode and S. Muthukrishnan. The string edit distance matching problem with moves.
ACM Trans. Algorithms, 2002. 8
AP Dempster, NM Laird, and DB Rubin. Maximum likelihood from incomplete data via the algorithm. Journal of the royal statistical society: series B (methodological), 1977. 2
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama herd of models. arXiv preprint arXhv:240.221783 2024. 1,3
Lukas Edman, Helmut Schmid, and Alexander Fraser. Cute: Measuring Ilms' understanding of their tokens. arXiv preprint 2rXhv:2499.15552 2024.
18
Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. Why do large language models (Ilms) struggle to count letters? arXiv preprint 27x1:2212.118666.000 2024. 8
Aaron Gokaslan and Vanya Cohen, Openwebtext corpus. OpenWebTextCorpus 2019. 20
io/
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, al. Olmo: Accelerat- ing the science of language models. arXiv preprint arXiv:2402.00833 2024. 3
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025. 3
10

---

Published as a conference paper at ICLR 2026
Dylan Hillier, Leon Guertler, Cheston Tan, Palaash Agrawal, Chen Ruirui, and Bobby Cheng. Super tiny language models. arXiv preprint arXiv:2405.14159 2024. 4, 6, 18, 20
Tatsuya Hiraoka, Maxmatch-dropoutt Subword regularization for wordpiece. arXiv preprint 02t1:22209.04226, 2022. 8
Tatsuya Hiraoka, Hiroyuki Shindo, and Yuji Matsumoto. Stochastic tokenization with a language model for neural text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1620-1629, 2019. 8
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 1,
Itay Itzhak and Omer Levy. Models in spelling bee: Language models implicitly learn the character composition of tokens. arXiv preprint arXiv:2108, 1119, 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024. 1
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310.06825 2023.
Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fembeeurbsky.social.. Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @Grad62304977. modded-nanogpt: Speedrunning the nanogpt baseline, 2024a. URL https:t////// httpt:p:///gttbbbb/// com/kellerJordan modded modded-nanogpt. 6, 20, 21
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024b. URL https:
//kellerjordan, github. ic http//psss/ss///eeee /wwn/. 6
Ayush Kaushal and Kyle Mahowald. What do tokens know about their characters and how do they know it? arXiv preprint wwXiv:2206.02608 2022.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple sub- word candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 66-75, 2018. 2,
Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023. 2, 6, 19
Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A Smith, and Yejin Choi. Superbpe: Space travel for language models. arXiv preprint arXiv:2503. 2025. 9
Raja Marjieh, Veniamin Veselovsky, Thomas Griffiths, and Ilia Sucholutsky. What is number, that a large language model may know it? arXiv preprint arXiv:2502.01540, 2025. 8
Piotr Nawrot, Jan Chorowski, Adrian łaácucki, and Edoardo M Ponti. Efficient transformers with dynamic token pooling. arXiv preprint 2rXiv:2211.0977 2022. 9
Keita Nonaka, Kazutaka Yamanouchi, Tomohiro Tsuyoshi Okita, Kazutaka Shimada, and Hiroshi Sakamoto. A compression- based multiple subword segmentation for neural machine translation. Electronics, 2022. 8
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu. Jason Weston, Luke Zettlemoyer, et al. Byte latent transformer: Patches scale better than tokens, arXiv preprint -4Xi:22422.09871 2024. 9

---

Published as a conference paper at ICLR 2026
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:/ opwnrevieweew..eooe.e net.//ee///eeem forum? idd=m6ckn2QaG. 6, 20
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Mísra. Grokking: General- ization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177,2000 2022.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. Bpe-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1882-1892, 2020. 2,
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
20
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5149-5152, 2012. 8, 15
Steven Scott. Bayesian methods for hidden markov models. Journal of the American Statistical Association, 2002. 9
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistiess pp. 1715-1725, 2016. 1.2
Andrew Shin and Kunitake Kaneko. Large language models lack understanding of character compo- sition of words. arXiv preprint arXiv:2405. 1357, 2024. 8
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. 2023. I
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 24080:2408.00118 2024. 3
A. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 1967. 2, 9
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, pp. 377-392, 2020. 20
Nan Xu and Xuezhe Ma. Llm the genius paradox: A linguistic and math expert's 's struggle with simple
word-based counting problems. arXiv preprint arXiv:2410. /4166, 2024. 8
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022. 9
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 242.:1541..51155. 2024. 3
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 16:78808-78823 2023. 9
12

---

Published as a conference paper at ICLR 2026
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 20
Xiang Zhang, Juntai Cao, and Chenyu You. Counting ability of large language models and impact of tokenization, arXiv preprint arXiv:2410. 19730, 2024, 6, 8
13

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIAL
TABLE OF CONTENTS
A Tokenizers
15
A.1 BPE Tokenization
15
A.2 Unigram Tokenization
15
A.3 SToCHASToK Tokenization Pseudocode
15
A.4 STOCHASTOK Tokenization Another Illustrative Example
16
Language Game and Math Datasets
17
B.1 LangGame....
17
B.2 CUTE Benchmark
18
B.3 Multi-Digit Addition
19
Training Setups
20
C.I 50M Parameter Model Setup
20
C.2 275M Parameter Model Setup
20
C,3 GPT-2 Continued Pretraining Setup
20
Analysis Details
22
D.I Different Prompt Completions Setup
22
D.2 Embedding Visualization Setup
22
D.3 Distance Over Layers Visualization Setup
22
14

---

Published as a conference paper at ICLR 2026
A TOKENIZERS
A. BPE TOKENIZATION
Construction
The tokenizer is constructed by initializing the vocabulary as individual characters and then iteratively adding the most frequent adjacent token pair in the 'training dataset' until the desired vocabulary size reached. This yields a vocabulary and hierarchy of merge rules.
Encoding
The dataset is initially tokenized as individual characters. Pairs of tokens are then merged according to the hierarchy of merge rules until there are no more merges available."
Decoding
The text strings corresponding to each token ID are simply looked up and joined together.
A.2 UNIGRAM TOKENIZATION
Construction
In contrast to BPE, Unigram starts with large candidate vocabulary of possible subword units and removes elements to get down the desired vocabulary size. Tokens are removed from the vocabulary by modeling the dataset as Unigram model and removing the token that results in the smallest increase logeliieldoood of the dataset considering all possible tokenizations. This relies on using the Viterbi algorithm to compute probabilities of all possible tokenizations. It also relies on using the Expectation-Man Maximization (EM) optimize the vocabulary and the probability of the dataset simultancously. The result is vocabulary and corresponding probabilities of each token (i.e.. a Unigram model of the dataset).
Encoding
All possible tokenizations are considered, and the one with the highest probability under the unigram model is chosen. This involves using the Viterbi algorithm to find the highest probability tokenization.
Decoding
Same as BPE: The text strings corresponding to each token ID simply looked up and joined
together.
A.3 STOCHASTOK TOKENIZATION PSEUDOCODE
Algorithm STOCHASTOR: Construction of sp1 its
1: Require: Tokenizer (e.g. tiktoken's GPT-2 tokenizer)
<- Tokenizer vocabulary
3: splits + {
Initialize an empty dictionary
4: for each token s in V do
5: t cncode(s)
Get the token id Initialize empty list for this token
6: splita(t] []]
each possible split index from to len(s) do
8:
81,82 82 < sf: 1]. s[:
Split string into two substrings If both substrings are in the vocab Add this possible split
9:
if s1 and $2 in then
10:
t, encode(s1), encode(s1) (
12: end (ilits((.append((1..
splits
13: end for
14: end for
'WordPiece (Schuster & Nakajima, 2012) can be seen variant of BPE with merges during encoding chosen by token length rather than the original merge rules.
15

---

Published as a conference paper at ICLR 2026
Algorithm STOCHASTOK:
Tokenization
1: Require: Tokenizer
Require: text: The input text to tokenize
3: Require: splits: Dictionary of possible splits for each token
4: Require: expand_prpp: Expansion proportion (e.g. 0.01)
5: tokenized Tokenizer(textt)
Apply standard tokenization
6: num num_to_expand wew.okenntteeeeeeeeoooe zed) expand_prop
for in 1... _ww__o__epaaaeteeeeeee
8: i randomlnteger(11 (entokknni zed))
Choose a random position
9: I < tckenized[i]
10: in splits and splits[[] empty then
(4,,2) t- randonChiicc(spli:ss))
Replace with random split
12:
tokenized <- tokenized[1 : 1]+ [1, + tokenizedji +
13: end if
14: end for
returm: tokenized
A.4 STOCHASTOK TOKENIZATION ANOTHER ILLUSTRATIVE EXAMPLE
Example vocabulary of base tokenizer:
vocabulary [_, h, u, gr b, m, hu, ug, hug, bug]
Build token_splite which, for each token, contains list of all possible pairs of component tokens that are themselves in the vocabulary.
token_splits
ug: [ (u, g)
hu: [ [h, ],
hug: (h,, ug) (hu, g)
bug: (b, ug)
ugs: [ (ug, s)
Examples of possible expansions:
original: [hug] + all possible expansions:
[hu g], [h ugl, [h
g]
original: [bug] + all possible expansions:
[b ug], [b g]
original: [m ug] + all possible expansions:
[m u g]
16

---

Published as a conference paper at ICLR 2026
B LANGUAGE GAME AND MATH DATASETS
In this section, we provide details of each of the three evaluation datasets: LangGame, CUTE, and multi-digit addition.
B. LANGGAME
We create new benchmark, "LangGame," to test subword-level understanding in LLMs. LangGame a multiple-choice based dataset, allowing for easy evaluation, and is suitable for small models. Here, we describe its construction in detail. The language game consists of six types of questions:
1. Which word has the most letter s?
2. Which word contains '#'s?
3. Which word starts with '#'s?
4. Which word ends with '#'s?
5. Which word is longest?
6. Which word is shortest?
We include multiple phrasings for each type question by constructing the question with a template and randomly replacing the placeholders.
Question template:
" "<WHICHD<WORD> <question>?
<TED><OTTOON>><RR>>>>>>>>
<options>.
Answer:
<answer>."
Synonyms for placeholders:
<WHTCH>: ["Which", "What"]
<WORD>: [" word", "", string", option", choice", option word", " option string"]
<THE>: ["The", "The possible", "The available"]
<OPTIONS>: |" options", choices", option words", option strings") <ARE>: [" are", ""]
This results in 2 x 7 3 x 4 x 2 = 336 possible phrasings for each question.
Question strings are then chosen from:
"has the most letter <AUX>/8?"
"contains <<UXX>", "starts with <<AU>>" "ends with <<AD>>", "is the Longest",
shortest",
Option words and answers are sampled randomly from the top English words, and sub-strings for the "contai ns" "starts wi with", and "ends with" question types are sampled randomly from the answer with length and S the answer length, and we generate 10k train and 1 validation examples. For the experiments in Figure 6, for the train and validation sets, substring lengths are > half the answer word length, and for the holdout set, substring lengths are < half the answer word length. An example of each type of question given in Table 1.
We evaluate accuracy based on whether the probability of the correct option is the highest compared to all the alternative options in the question, but additionally when looking at generations, find that the SToCHASTOK Toitesssut--nnetuueddd models generate the correct answer over all other possible next tokens.
17

---

Published as a conference paper at ICLR 2026
B.2 CUTE BENCHMARK
We also evaluate on the Character-level Understanding of Tokens Evaluation (CUTE) benchmark (Eid- man et al., 2024). CUTE contains 14 question types:
Task
Question
Answer
Spelling Inverse Spolling
Spell out the word: there
there
Write the word that is spolled our (oo spaces): t there "there"?
there
Contains Char
Contains Word
Is there "the' in 'the sky is blue"?
Yes
Orthograhicc
Closer
"happy": glad apply?
apply
Semantic
More semantically related to 'happy": glad apply? Add "t" after every "e" in 'there'
glad
Char Insertion
thehreh
Word Insertion
Add 'is' after every 'the' in 'the sky bluc'
the is blue
Char Delction
Delete every 'e" in 'there"
the
Delete every 'the' in sky
sky blue
11
Char Substitution
Replace every "e" with 'a' in 'there'
thara
Substitution
Replace every
sky blue"
13
Char Swapping
Swap 't' and in 'there'
thete
14
Word Swapping
Swap 'the" and 'is' in 'the sky is blue"
sky the bloe
Table 2: Examples of the CUTF benchmark language composition, similarity, and manipulation tasks.
We the eight subword-leve question types (types 1, 2, 3, 5, 7, 9, 11, and 13). The original benchmark was designed for zero-shot evaluation of full-scale industrial models, and hence, it only includes set. To evaluate our smaller pre-instruction finetuning models, we require additional training examples for finetuning, hence we generate more questions for each of the eight types. We generate questions by randomly sampling words from the top 1k English words. Consistent with the multiple-choice format of the open-source baseline code (Hillier et al., 2024), we also create incorrect answer options. For questions where the answer an option in the question (question types and 5), the incorrect options are the other options in the question (e.g., Yes/No). For questions where the answer is a word (question type 2), the incorrect options are other randomly sampled words from other top 1k English words. Finally, for the remaining question types where the answer is sequence of letters (question types 1. 7, 8, 11, 13), the incorrect options are generated by substituting and reordering letters in the correct answer. Results on each of the individual CUTE tasks over training are shown Figure 14.
18

---

Published as a conference paper at ICLR 2026
CUTE Benchmark Tasks
Spelling
Inverse Spelling
Normllred Accuraty
0.2
0.0
0.0
500
1000
1600
2000
2500
3000
500
1000
1900
2000
2000
3000
Contains Character
Character Insertion
Normalzed Acaroty
0.8
0.8
0.6
0.4
0.4
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Character Deletion
Character Substitution
1.0
1.0
Nomalizd Accurary
0.0
Character Swapping
Orthographic Distance
Normalized Arturaty
0.6
0.4
0.4
0.2
0.2
0.0
0.0
500
1000
1500
2000
2500
3000
500
1000
1500 ller serratio
2000
2500
3000
Iteration
[Train] [Val]
Deterministic BPE-dropout
Stochas Tok (p=0.05) 00)
No Protraining
Stochas Tok (p=0.1)
Figure 14: Performance on each of the tasks within the CUTE benchmark over training. (Accuracy normalized that random guessing is zero.)
B.3 MULTI-DIGIT ADDITION
For the multi-digit addition experiments, sampled pairs of integers up 1000. The answer is reversed as per the procedure in Lee et al. (2023), and we then train on a stream of exam- ples, e.g., S 151+687-838
747+303-0501 752+917-9661 857+579=6341 $ .... with the setup described in Section C.1.
328+869-7911
19

---

Published as a conference paper at ICLR 2026
C TRAINING SETUPS
In this section, we provide full details of the training setups used in the paper. For STOCHASTOK's hyperparameter p. we find that careful tuning is not required and that any value between 0.01 and 0.2 gives good performance. Throughout the paper, we show results with = 0.1 (and also include p 0.05 in some places effectively an extra seed). For BPE-dropout, we use 0.1 as suggested in the original paper.
C.I 50M PARAMETER MODEL SETUP
We build on the baseline 50M-parameter model setup in the open-source HapperTiyLLagguggeMMaee repo (Hillier et al., 2024), which is trained on the OpenWebText dataset (Gokaslan & Cohen, 2019) and uses the GPT-2 BPE tokenizer from the tikt oken library. The pretraining benchmarks evaluated on (see Figure et al., 2019), Winograd (Sakaguchi et al., 2021). The full set hyperparameters for pretraining are given Table 3.
ARC (Clark al., 2018 1), Blimp (Warstadt al., 2020), HellaSwag wag (Zellers
Model
number of layers
8
Ifn type
SwiGLU
ffn dimension
1320
number of attention heads
16
group size
hidden dim
512
tokenizer type
gpi2
vocab size
50257
max context window
$12
Desitional_encoding.o type
RoPE
Training
batch_sipe size iterations
30000
warmup iterations
5000
dropout
0.1
Optimizer
optimizer
AdamW
Ir min Ir
6.0e-04
6.0e-05
Ir sccheuulerrr
Cosine
weight_decay
0.1
Table 3: The baseline setup in Hillier al. (2024) 50M-parameter transformer LLM.
For fine-tuning (as in Figure 4), we train for a further 3k iterations with a learning rate of 1.0e-04 on the LangGame CUTE datasets. For continued pretraining (as in Figure 9) we similarly train for further 3k iterations with learning rate 1.0e-04 on Open WebText.
C.2 275M PARAMETER MODEL SETUP
For 275M parameter model, we follow Jordan et al. (2024a), training on Feb (Penedo et al., 2024) with the hyperparameter setup given in Table 4.
C.3 GPT-2 CONTINUED PRETRAINING SETUP
We initialize the model from the publicly available pretrained weights and architecture on Huggingface https:: / /huggin ngf
co oopenai- community: trmmmuuiitg/tp/22 2pt2. For the continued pretraining, we train for 7k steps with a constant learning rate of 1.0e - 4 and batch size of 128. For the finetuning on LangGame tasks presented in Figure 10, we finetune for 2k steps, again with constant learning rate of 1.0e 3 and a batch size of 512.
githuh.com/geenniikkkkenn
20

---

Published as a conference paper at ICLR 2026
Model
number of layers
12
ffn type
ReLU
dimension
768
number of attention heads
6
head dimension
128
tokenizer type
gpx2
vocab_ size
50257
max context window positional encoding type
1024
RoPE
Trainng
batch size total iterations
60000
cooldown frac
0,4
Optimizer
weights optimizer head, embeddings, biases optimizer
Muon
AdamW
head Ir embeddings Ir
0.044
0.12
biases weights
0.008
0.01
Table 4: The baseline setup in Jordan (2024a) 7775Mpprameetr transformer LLM. The changes made the baseline are training for iterations (as opposed to 1770 iterations of original baseline, since baseline config was set up as a demo) and reducing all the learning rates by a factor of 5 (needed to stabilize training of all models when training for longer).
Lose
3.5
3.5
500
Iterction
1002
1502
2009
2.
2000
4030
6000
Figure 15: Training loss on Open OpenWehTex. WebText during continued pretraining for the 50M STLM base model and GPT-2.
21

---

Published as a conference paper at ICLR 2026
D ANALYSIS DETAILS
In the following section, we provide additional details and results of the visualizations in Section 7.
D.1 DIFFERENT PROMPT COMPLETIONS SETUP
Further examples of completions from multiple different tokenizations of the same prompts are given in Figure 16. The prompts are generated by GPT-4o.. We find that the deterministic tokenization- trained model very sensitive to prompt tokenization and quickly breaks down when given alternative tokenizations of the same prompt. By contrast, the SToCHASToK-traings model is much more robust to prompt tokenization.
D.2 EMBEDDING VISUALIZATION SETUP
As described in the main text, the activations for word are taken as the residual stream activations after the final transformer layer. If the word is tokenized into multiple tokens, we use the position of the final token. We use the standard procedure of normalizing zero mean and unit standard deviation before fitting the PCA model.
D.3 DISTANCE OVER LAYERS VISUALIZATION SETUP
In Figure 13, we plot the mean distance between embeddings of different tokenizations of the same word over the layers of the model. For normalization to allow comparison between different models, we first normalize all embeddings to have unit length. We then evaluate the average distance between embeddings for pairs of different words in the model, and we divide by this average distance metric.
Different tokenizations
Deterministic Training
StochesTok Training
Deep
the forent
they foundl
ismall smail aadd own and the water Is small small and owm and the water is evidence the crowd was cheering
small
white
Isiled
Deep
the
forest
they tou ndi
small
white
tallec
small
people
(who
HP forest
Pay found
small
Deep the forest they fou nd
small
white
talled
found
small
talled
She stepped anto the stage aand the
crowd was sbant
She ped onto the stage and
the crowd cheering crowd siient
steoped onto
ittge and
stage
crowd was
She stepped on to the stape Indi the
First time saw him
crowd was abant
She aped onto the
In
crowd siient
steoped onto the stage and the
sound the
crowd was
crwshed legainst
building and
well (he builking and
the picture the wave i cr ashhed agaiin st 1 he
the wind and the wind well and the wind blew
the pictune the wares ashed again st
uug
wall and the wind blow
the
wall
ground
H the picture the wewves creehed ageinst
well
well and the ground shook
the picture waves crashed again ss he
and the wind blew out
wall and the ground shook
The scientist carefully adjasted the microscops searching for the source of the light the date
the most the mosil common and mooi common common The scient 1st car stully acqusted the microscope so arching for tno mirst timo a decade the most common and most common
The wcientist whlll adjusted microscope neerching
and most cummon
scient
carefully adjusted microscope arching
common
The scient 1st car stuly aduussed th 8 micross ope searching for the sourcee of the gas !!
tha most common and most common common frush The story
eluty edjusted microse ope arching truth
shaped
The journeist pressed record rreddy caosure the joumallst pressed record ready capture
true Iideriity
truth The
The journalit pres Bed rccord ready capt ure the
world
The story
T he journal bit pres sed reaord ready to papture the essemoe of the universe eosencs of the story
The jourueiss presseed reoord reedy
into woord of Inuth The story
jourallet pressed record ready capture
The story
22

---

chef sighed and UR the The cher Ighed put the Published as a conference paper at ICLR 2026
train departed he remambered
had
the
Gar
boen
train
thm
train dep arted
he remembered
been
the
trainn
For
the
departed
remembered he had
Ieen
the same
boat
the
the
deperted romem
ad
flw (words
train
for
As
the train departed he remembered had
boon
in the
car
boon
the
train
for
A
the train departed ramai bared be bw
Ited
the
inamn
mmn
boeen
tth
train
for
After
years
training she finelly reached the
16
(Shm
wes
point
whetee
xhhe
could
Ise soom
After
years
raiinn she finally reached he
reat
SShe
10
After
training finally
ached the
mark She
littl
point
whhoe
she
years
training she finlly reachod
age
After V ears of training she finally
reached the
point where (sha) could see the
agai of 18
After
training whe Boaily reached
The detective examined the crima scene looking for
clues and found that (hhe suxpect who (was (in the
T he det scilve ecemined the crime scere [ookinn fo The det ective examine the Th e desective exomines the crime scene lo oidnn for
tel to (the crime scene
the suspects and found thet the suspects
scere looking
sseeme the
few minutes and thhnn
clued to the Identity of the clues to the iddenity of the new way tt read
T ha detective examine d the crtme s8 looking 1 or detective
and the pollice
ciking
found
He opened the ancient book and discovered
newt (wwy of thinking inbout the
opened ancient ba and discovered
large llarge and very
read Bible
opened
an
and disce vered
and Ite
new Iinn
He apence
the ancient book and discovered
homm the ancient world of now kindl magic
openee the anckent boo k and iscovered
naw read
openeed
discovered
years ago was expected to Hi tha city
way
the heart of the eity storm
Fow hundded people
city
of city swreppng through wass eepeeted to Hit the city and the storm and
hundred people
In the hear of Uhe city
storm
protest and protests erupted
th he art of the city storm heaat city
few hundred people
hundred people
city sto rm
we: city of IMaa
violence bear unleassed
The spaceship evared above
planst
and
orbit
planet was still the distance
The spaces hip overed above the planet
he
wws
E/e first
see
176
planet wes
little more then
The speceship overed above the planel lad
Uhe planet Iwe
planet was still in
Uhe cissance
The hip overed above
planet the
planet
atmosphere land
planet was
little
than
The spacos hpp Diverd above the planct
8 at henoa and
planet was
little more than
The speces hip IN owardd above the plane
and the
two story building was
Iwol story building (wos
the The che sigh 8c and ut the T ha ch of sighad Inc put Highed
Highed
knife hife moutt
words
cish on the table
on the table and sald
bust 2cha on tha
and
turned
worrss not
dish on the table
dish on the tablc Figure 16: Example responses with different tokenizations.
and then
dish 1he tablc
23


Reviews:
[{"review_id": "oAR41j6Szh", "reviewer": "Reviewer_dC4j", "summary": "The paper introduces a new tokenization method STOCHASTOK to improve subword-level understanding of language models. The method is simple, efficient, and compatible to existing tokenization methods. Experiments show that pretraining models with STOCHASTOK improves model perofrmance on language game tasks and multi-digit addition. The paper also demonstrates that applying STOCHASTOK during fine-tuning can enhance the subword understanding of already pretrained models.", "questions": "1. The experiments only reports results for STOCHASTOK with p<=0.1, is this an intentional design choice? How does performance change with higher p?\n\n2. Does STOCHASTOK improves model robustness to spelling errors or out-of-vocabulary words?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clear writing:** the paper is well-structured, clearly written, and easy to follow\n2. **Method simplicity and effectiveness:** the proposed method is simple, efficient, and shows promising performance improvements across tasks. \n3. **Comprehensive experiments:** the experiments cover multiple training settings and analyses, and consistenly demonstrate the method's advantages.", "weaknesses": "1. **Limited model scale**: the experiments are conducted only on 50M-parameter models and GPT-2. While these show reasonable performance on simple language tasks (BLIMP & ARC), it remains unlearn whether the improvements generalize to larger models. It would be valuable to see results on larger (1B/7B) models. I understand such experiments are complex and do NOT expect them for the rebuttal. \n\n2. **Limited evaluation scope**: the current evaluations focus mainly on artificial tasks such as word games and digit addition. It would strengthen the paper to provide more results and discussions on  improvements on more realistic tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EHYeRh7EDm", "reviewer": "Reviewer_HeMs", "summary": "This paper introduces StochasTok, a novel stochastic tokenization method designed to address a key limitation of large language models: their poor performance on subword-level reasoning tasks. Standard tokenizers treat words as opaque symbols, obscuring the internal character structure and making tasks like counting letters or performing multi-digit arithmetic surprisingly difficult for even state-of-the-art models.\n\nThe proposed method operates as a lightweight post-processing step that randomly splits tokens into smaller, valid subtokens from the existing vocabulary, thereby exposing the model to the morphological composition of words during training. The authors demonstrate through extensive experimentation that StochasTok significantly enhances subword reasoning capabilities. Their approach consistently and substantially outperforms strong baselines in both pretraining and finetuning scenarios, achieving near-perfect accuracy on language game benchmarks like LangGame and CUTE, and enabling models to rapidly learn complex tasks such as multi-digit addition.", "questions": "1. I acknowledge the contributions of this work. However, from my understanding, StochasTok appears to be a special case of BPE-dropout? Is it true that for every tokenization generated by StochasTok, there exists an equivalent BPE-dropout tokenization? If so, could the authors provide an intuitive explanation for why StochasTok performs so much better than BPE-dropout in the experiments? Are there key differences in how the stochasticity is applied or how the model learns from these variations that lead to the significant performance gap? Could the authors provide a more detailed comparative analysis between the two?\n2. The paper states that \"In BPE, intermediate tokens not present in the final tokenized training dataset are removed from the vocabulary, meaning BPE-dropout can produce tokens outside the original vocabulary\", which I'm not sure. If true, how does it encode user inputs during inference, given this vocabulary mismatch?\n3. In Figure 1, the \"no pretraining\" model shows a rapid increase in both training and validation accuracy very early in training, followed by a sudden decrease. What is the authors' explanation for this phenomenon?\n4. In Section 5, which focuses on multi-digit addition, what was the range of digits for the numbers used in the training and validation sets?\n5. Could you please report performance on individual subtasks of both LangGame and CUTE (e.g., \"Inverse Spelling\", \"Char Deletion\")? This would provide clearer insights into the model's capabilities.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed StochasTok approach offers an elegantly simple yet effective solution to the fundamental limitation of subword understanding in LLMs. Its implementation as either a pretraining enhancement or a lightweight finetuning step makes it highly practical and accessible.\n2. The authors provide comprehensive validation across multiple domains - from language games (LangGame, CUTE) to mathematical reasoning (multi-digit addition) - demonstrating the method's versatility and robust performance gains.\n3. The paper provides some insights into the method's internal mechanisms through embedding visualizations", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIfA4mYDsi", "reviewer": "Reviewer_eKHe", "summary": "The paper proposes a new stochastic tokenization approach called StochasTok. StochasTok allows for flexible, alternative tokenizations of the same token. Results on various datasets including language games, pre-training small-scale models (50-275M parameters) offer large performance gains.", "questions": "- Perhaps experimenting with slightly larger models might offer a clearer picture if the gains from your method are similar in larger LMs (e.g. ~1B).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written and easy to read.\n* Tokenization is a really important and often neglected area of LMs.\n* The method proposed is simple and works well\n* The paper offers a comprehensive analysis and discussion of experiments and results.", "weaknesses": "* Really minor (without trying to be \"Reviewer 2\"): the models tested are really small for 2025 standards. There is a risk that the gains might not generalize to larger scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LO0V288lvk", "reviewer": "Reviewer_ZDYH", "summary": "The paper proposes STOCHASTOK, a simple stochastic tokenization scheme that preserves the original vocabulary and introduces *training-time only* reversible splits of existing tokens to expose subword/character structure. The approach aims to improve tokenization invariance without changing model architecture or deployment-time tokenization. Empirically, it delivers consistent gains on curated subword “language games” and cross-tokenizer multi-digit addition while keeping standard LM metrics roughly unchanged on small/medium models; qualitative analyses (e.g., embedding alignment across segmentations) support the intuition. The idea is elegant and easy to integrate (especially for CPT), but evidence on larger models and real-world tasks is currently limited.", "questions": "1. **Scale-up sanity check**: Can you run short-budget CPT on one **4B** and one **7B** model (e.g., 3–10k steps, \\(p \\in \\{0.05, 0.1\\}\\)) and report subword tasks, a small **MMLU** slice (no-regression), and **throughput/length/VRAM**?  \n2. **“Real” math**: On **GSM8K** (dev is fine), compare deterministic vs **BPE-dropout** vs **STOCHASTOK** under **equal compute**, including training curves and final accuracy; also test cross-tokenizer transfer at this scale.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Simplicity & compatibility**: Fixed vocabulary, tokenizer-agnostic, and *train-time only* noise makes it a low-friction drop-in (particularly for CPT) with minimal code churn.  \n2. **Clear signal on subword skills**: Reproducible improvements on subword-aware tasks and cross-tokenizer arithmetic transfer, with no obvious small-scale regressions on general LM metrics.  \n3. **Practical deployment story**: Inference remains deterministic, avoiding train–test segmentation mismatch common to methods that alter the tokenizer itself.  \n4. **Plausible mechanism**: Training with multiple valid segmentations encourages segmentation-invariant internal features; analyses suggest layer-wise convergence toward shared representations.", "weaknesses": "1. **External validity at scale**: No results on ≥4B/7B models; claims of robustness and easy integration would be stronger with short-budget CPT evidence at those scales.  \n2. **Evaluation scope**: “Real” math (e.g., GSM8K/MATH) and broader tasks (MMLU/BBH/code/RC) are missing; current gains mostly establish tokenization invariance rather than end-task improvements.  \n3. **Compute/cost parity**: No end-to-end cost curves (length inflation, throughput, VRAM, wall-time) or budget-matched comparisons, making Pareto efficiency unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)