Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
Authors: Craig W Schmidt
Varshini Reddy
Chris Tanner
Yuval Pinter
Venue: COLM 2025 (2025)
Keywords: tokenization
Byte Pair Encoding
BPE
subword tokenization
TL;DR: We propose an extension to BPE that allows pretokens to merge into superwords, leading to a more uniform token distribution and better compression.

Abstract: Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. While this process encourages having full, individual words as tokens, it introduces a fundamental limitation in most tokenization algorithms such as Byte Pair Encoding (BPE). Specifically, pre-tokenization causes the distribution of tokens in a corpus to heavily skew towards common, full-length words. This skewed distribution limits the benefits of expanding to larger vocabularies, since the additional tokens appear with progressively lower counts. To overcome this barrier, we propose BoundlessBPE, a modified BPE algorithm that relaxes the pretoken boundary constraint. Our approach selectively merges two complete pretokens into a larger unit we term a superword. Superwords are not necessarily semantically cohesive. For example, the pretokens " of" and " the" might be combined to form the superword " of the". This merging strategy results in a substantially more uniform distribution of tokens across a corpus than standard BPE, and compresses text more effectively, with an approximate 20% increase in bytes per token.

Published as a conference paper at COLM 2025
Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
Craig W. Schmidt, Varshini Reddy & Chris Tanner*
Kensho Technologies Cambridge, MA 02138, USA schmidt, varshini bogolu, chris, Jokesh.ooee
Yuval Pinter
Department of Computer Science Ben-Gurion University of the Negev
Beer Sheva, Israel uvpêcs. bogecs.bu..at bgu. ac.il il
Abstract
Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. While this process encourages having full, individual words as tokens, it introduces a fundamental limitation in most tokenization algorithms such as Byte Pair Encoding (BPE). Specifically, pre- tokenization causes the distribution of tokens in a corpus to heavily skew towards common, full-length words. This skewed distribution limits the benefits of expanding to larger vocabularies, since the additional tokens appear with progressively lower counts. To overcome this barrier, we pro- pose BOUNDLESSBPE, modified BPE algorithm that relaxes the pretoken boundary constraint. Our approach selectively merges two complete preto- kens into larger unit we term superword. Superwords are not necessarily semantically cohesive. For example, the pretokens _of and the might be combined to form the superword _,o__theee This merging strategy results in a substantially more uniform distribution of tokens across a corpus than standard BPE, and compresses text more effectively, with an approximate 20% increase in bytes per token.
1 Introduction
Pre-tokenization is crucial step in preparing text for language models, helping to aligm token boundaries to meaningful linguistic units. A document is first broken into chunks called pretokens using regular expression,' which are then tokenized separately. Each pretoken may be tokenized into two or more subword tokens, or used exactly matches the entire pretoken. Many common words end as single token (Reddy et al., 2025), more than 90% according to our analysis (Section 2). Under such circumstances, differences in the tokenizer itself can only manifest in the small percentage of remaining pretokens. This high overlap in resulting tokens shared across different tokenizers can explain why tokenizers have been found to perform quite similarly on downstream tasks, with no statistically superior approach (Schmidt et al., 2024). Selecting a larger vocabulary size tends to exacerbate this issue, since the fowerffreueenc tokens added to the vocabulary later in the training process do not substantially change the resulting token distribution in the training corpus.
single token that
'Chris Tanner is also affiliated with MIT in Cambridge, MA, USA.
'See Section 3.5 and Appendix A for discussion of the particular pre-tokenizer regular expression used.
1

---

Published as a conference paper at COLM 2025
Zouhar et al. (2023) suggest that tokenizers exhibiting more uniform distribution of token frequencies across a corpus tend to yield superior performance in language models. They suggest avoiding tokens which exhibit high frequency while carrying little semantic content, such as individual bytes, as well as very rare tokens that lack sufficient contextual information for effective learning. However, current pre-tokenization methodologies offer limited control over this distribution, as most pretokens are mapped to single tokens, often encompassing common whole words.
With the goal of obtaining a more uniform distribution of tokens, we propose a modification to the standard pre-tokenization approach that still allows us to retain the benefits of contextual cohesion. Specifically, we introduce superwords, tokens composed of an n-gram of words, to supplement subwords and words in the vocabulary. The notion of superwords extends beyond semantically cohesive units like New York City often identified through metrics like Pointwise Mutual Information (PMI; Fano & Wintringham, 1961; Church & Hanks, 1989). Instead, they serve to distribute the most common words like the into range of common n--rams like _at and __the__huuse, thereby lowering the frequency of the most common tokens in corpus. As we will demonstrate, a substantial number of such common n-grams is found by our approach, enabling more effective utilization of larger vocabularies.
In Section 3, we introduce Encoding (BPE) algorithm (Sennrich et al., 2016; Gage, 1994). The key modification involves incorporating the concept of a supermerge. During the tokenizer training process, we permit the merging of two adjacent pretokens if each is represented by single token, in which case, the resulting merge is considered a superword. The supermerge operation lends itself to seamless integration into the standard BPE training procedure: at each step of training, the merge or permissible supermerge that occurs the most frequently is performed. We also implement a variation of the deletion technique from PickyBPF (Chizhov al., 2024) to remove low-frequency intermediate tokens from the vocabulary. These deletions contribute to a more uniform distribution at the lower end of the frequency spectrum by freeing up space taken by intermediate tokens for use by more common tokens.
ESSPPE, an extension to the widely adopted Byte Pair
The efficacy of our proposed approach is presented Section 4, where we use an evaluation corpus to compare the token frequency distributions produced by BOUNDLESSBPE and several commonly-used baseline tokenizers. The results demonstrate that BOUNDLESSBPE yields more uniform token distribution, consequently achieving minimum of 21 % increase in Rényi efficiency (Zouhar et al., 2023) compared to the baselines. Since our method exhibits improved compression, with at least 19.7% increase in overall bytes per token, it requires fewer tokens for language model inference.
Limitations of pre-tokenizers
Pre-tokenization processing step shared by most tokenizers and it plays vital role in how tokenizer vocabularies are formed. Regular expressions commonly used in pre- tokenization permit spaces only as the first byte in a pretoken (for example, _the), which supports the alignment of the final tokens with word boundaries. While the objective is to generate tokens that correspond closely to meaningful linguistic units, this alignment does not mandate strict one-to-one mapping between tokens and complete words. Instead, it prioritizes the prevention of tokens containing parts of two adjacent words, thereby enabling better downstream performance. For instance, Schmidt et al. (2024) demonstrated that entirely omitting pre-tokenization resulted in the poorest downstream performance among 18 evaluated tokenizers.
Due to the Zipfian nature of text, where a few words occur with very high frequency and many words occur rarely, the pretokens of common words are often efficiently represented as single token that exactly matches the entire pretoken. We used a 5GB portion of MiniPILE (Kaddour, 2023) as our out-of-sample evaluation corpus to illustrate this.? Using
We train tokenizers on the first 170,721 documents of MiniPile, totalling 1GB. The remaining 829,279 documents form the 5GB out-of-sample evaluation corpus used throughout this paper.
2

---

Published as a conference paper at COLM 2025
UnigramLM
WordPlece
1.000
Fraction Prettllee with Single tolion
0.975
0.950
0.925
0.900
0.875
0.850
0.825
0.800
40960
65536
98304
131072
Vocabulary Size
Figure 1: Proportion of pretokens in the evaluation corpus that are represented as full tokens by different tokenization methods (BPE, UnigramLM, WordPieec) across varying vocabulary sizes.
the widely adopted BPE, WordPiece (Wu et al., 2016; Schuster & Nakajima, 2012) and UnigramLM (Kudo, 2018) tokenizers, 3 with vocabulary sizes of 40,960, 65,536, 98,304, and 131,072, we calculated the proportion of pretokens that were ultimately represented by single token. As shown in Figure 1, BPE and WordPiece, both of which are bottom-up merge- based tokenizers, achieve a single-token representation for well over 90% of pretokens, while UnigramLM, a top-down ablation-based tokenizer, achieves 82-87%. This highlights how these tokenization methods effectively handle the most frequent words in the vocabulary, and it direct consequence of the Zipfian distribution inherent in natural language.
The pre-tokenization step is thus directly determining the vast majority of the tokens, leaving the tokenization training very little ability modify the token distribution. The phrase 'To be or not to be' composed of very common words will be tokenized by our baseline tokenizers as
['To', be', or', not', to', be ']
simply due pre-tokenization. To change the tokenization substantially, we need a way to overcome the fact that a pretoken's 's role during tokenizer training effectively ends once it becomes a single token. For common words, this happens very early in the process.
3 BOUNDLESSBPE
We introduce BOUNDLESSBPE, a tokenization algorithm that allows adjacent pretokens, each represented as single token, to be merged into a superword token. For the Shakespearean example above, our tokenization training process will continue beyond pretoken boundaries, yielding the tokens:4
['To be',
not', to be ]..
3.1 Tokenizer training
Standard BPE training starts with every pretoken in the training data split into individual bytes, which are the initial tokens. For example, 'Tip the hat' has four pretokens, each containing single-byte initial tokens:
[['T', 'i', 'p'J, [' 'o', 'f'J, [' 't', 'h', 'e'J, ['', 'h', 'a', 't']].
BPE employs pairwise merge rules, such as (' (''tt")) -> t', where token pair is merged to form a single token. At each iteration, the algorithm finds the pair of adjacent
We use the Hugging Face implementations: ht https:/ httbb com com/ http/huggingffce.. teacee/tokeiizerss (BoUNDLESSBP examples throughout this paper with vocabulary of 131,072.
3

---

Published as a conference paper at COLM 2025
tokens with the highest frequency of occurrence across the training data. This merge rule with the maximum adjacency count is added to a list of merge rules (used during inference), and the newly-combined token is added to the vocabulary. This pair of tokens is replaced by the combined token throughout the training data, giving:
[['T', 'i', 'p'], [' 'o', 'f'J, [' t', 'h', 'e'l, ['', 'h', 'a', 't']].
This process continues until the vocabulary reaches the desired size. During inference, applicable merge rules applied to document in the same order they were found in training, until no further merges are possible, giving the final tokenization.
BSSBE is an extension of this standard BPE training process, with two key mod- ifications. First, two adjacent pretokens that are currently tokenized with a single token (identical to the pretoken itself) are allowed to merge into combined superword. We call these merges supermerges.. For example, the adjacent pretokens [' of and [ the ' each consisting of a single token:
[['T', 'i', 'p'J, [' of'J, [' the I, [' h', 'at'll, ]],
can be combined into the superword of the J, which would consist of a single token. A superword can subsequently be merged with other fully-merged pretokens into longer superwords. Supermerges often involve pairs of common words, such as of the', in the', to the', on the', and and the'. One consequence of this behavior reduction in the count of the most common tokens such as the' and of", as some of their occurrences are allocated to various superwords.
Our second modification to the standard BPE algorithm is the inclusion of variant of PickyBPE deletions (Chizhov et al., 2024) to eliminate intermediate tokens that primarily serve components of more frequent, larger tokens. After regular merge operation, we use PickyBPE's Intersection over Self (IoS) metric to assess the utility of deletion: we compute the ratio of the frequency with which the merged tokens appear consecutively to the individual frequency of each constituent token. If the computed loS value exceeds threshold, 0.9, the constituent token is removed from the vocabulary. This deletion step is effective in eliminating low-count tokens because a token with high loS predominantly occurs as part of the newly formed token, thereby freeing up space within the desired vocabulary size for higher-frequency tokens. For example, after merging bet' with 'ween' the token 'ween' will be deleted, since its IoS 0.9618 0.9.
For the sake of simplicity, we do not implement deletions following supermerges, as it would require additional bookkeeping of the initial states. 6 In the original PickyBPE implementation, deleted token is split back into the pair that formed it. We adopt more aggressive approach where a deleted token is decomposed back into single bytes within pretoken. This allows single bytes to potentially recombine into frguueyfreyueccy tokens, at the expense of more merge rules.
At any given iteration in the tokenizer training process, we therefore have three potential operations: regular merge, supermerge, or standard deletion. Both regular merges and supermerges with highest frequency are identified, and the operation with the higher score performed. After each regular merge selected, one or both merged tokens may then be deleted according to their respective loS values.
"To avoid combining different pretoken categories, such as digits and words, we restrict the merging process to pretokens matching the byte regex rb"^[ -=azz==[[[aaA regex matches words with optional initial spaces, contractions, and snake case variables. See Section and Appendix for more detailed explanation of the pre-tokenization regex. Note that numbers are already segmented into groups of in a right-to-leff manner, following the suggestion of Singh Strouse (2024), so we avoid combining these digit groups into longer superwords to preserve their intended format. Furthermore, as discussed in Appendix A.1, we opted not to combine punctuation with words.
"a-zA-Z]*$) This
'In our training run with vocabulary size of 2 131,072, there would only have been 164 deletions in the 1038 supermerges, compared to 1,987 deletions for the 89,778 regular merges. Single bytes are never deleted, as they are necessary avoid the generation of unknown tokens.
4

---

Published as a conference paper at COLM 2025
3.2 Inference procedure
Once a tokenizer has been trained, it can be used to tokenize a new document, a process often referred to as segmentation inference. A trained BOUNDLESSBPE tokenizer consists of separate dictionaries for regular merge rules, supermerge rules, and regular deletion rules. Each rule has unique index representing the order in which it was added during the training process. Each step of inference involves identifying possible operations that can be performed given the current tokenization of the document. Each pair of adjacent tokens within each pretoken is compared to the merge rules. Each pair of adjacent pretokens containing single tokens is compared to the supermerge rules. Finally, each individual token is compared to the deletion rules. The operation with the smallest index, meaning it was the first to added during the training process, is performed everywhere it occurs within the document. Listing 1 gives example of this tokenization process for the phrase 'Tip of the hat':
[['', ' 'p [' 1, 'o', 'f', [' " 't', 'h', eel, [' ' 'h', la', 't']]
[['T', '1' 'p'], (',, 'o', 'f'], [' t', 'h', 'l'l, [ 'h', 'a', 't]]]
[['T', 'i', 'i', ["', '0', 'f'], [' he 1, ['', 'h', 'a', 't]]
'o',
[['T', 'i', (p],,
[' the J, [' "h,', 'a', 't
[['T', 'i', 'p', '0', 'f',, [' the'] ], [' 'h', at ']]
[['T', 'i', 'p'], o', 'f',, the'l, [' 1, 'at']]
[['T', "i', 'p', of o, the'', [' ' 'h', 'at'll
[['T', 'i', 'p'l,
[' the' 1, [' h h', 'at']]
[['T', 'i', 'p'], of the I1 [' h', 'at']]
10 [['T', ip '1, [' of t,e'], [' h', 'at']]
11 [['T', 'ip'], D' of the'l, [' hat ']]
12 [[' Tip '], [' of the'l, [' h]t']]
Listing 1: Inference example
The data is initialized on line 1 with a list of 4 pretokens, each containing list of individual bytes as the initial tokens. Line 2 shows the results of the first merge of (" 't'), which had the lowest index of any operation. This process of selecting the lowest index merge, supermerge, or deletion continues until line where [' of'] and [' the ' appear next to each other, and each is single token. At this point, supermerge becomes a valid option. On line 8, a regular merge (' ', 'h') is performed next due to a lower index, but then the first supermerge creating [' of the ] happens on line 9. The tokenization process concludes on line 12 after the application of a few more merges.
3.3 Efficient training implementation
In the representation shown in Section 3.2, a document is segmented into pretokens, with each pretoken containing one or more tokens. While conceptually straightforward and directly applicable to tokenizer training, this representation computationally inefficient. See Appendix B for description of several techniques used to speed up training time. We release an open source version of the BOUNDLESSBPE training and inference code.?
3.4 Training dynamics
Figure 2 shows the logarithm of the count of each selected merge or supermerge during the training process for BOUND, along with the count of each selected merge for standard BPE and PickyBPE, each employing two distinct pretookenization regular expressions. The addition of supermerges as an available choice allows the counts to decrease at a slower rate compared to the baseline methods. As we will demonstrate, this
"httpss//gitthee https////g.ttttcccc/////stttttte http:///esshotttccccoooieeee///e
5

---

Published as a conference paper at COLM 2025
Reguler
Supermerge
BoundessBPE,
BPE, GPT2 PATTERN rages
BPC, BOUNDLESS PATTERN
GPT2 PATTERN
log10mma counte of morge
RccyaPE, BOUNDLESS, ragex
Trection
0
20000
40000
50000
80000
100000
120000
140000
0
20000
40000
50000
80000
100000
120000
Marge namber
Narge number
Figure Logarithm of the maximum count of each selected merge, up to a vocabulary size of 131,072.
Figure 3: Average fraction merges (lower) and supermerges (upper) over intervals of 8,192 merges, up to a vocabulary size of 131,072.
results in more uniform token distribution characterized by tail of tokens with higher frequencies. The other four curves are essentially indistinguishable, which is a result of so many pretokens ending up as single tokens, as described in Section 2.
Figure 3 shows the fraction of regular merges and supermerges across each successive group of 8,192 merges. A supermerge cannot form until both candidate pretokens are a single tokens. For example, [' of and [' the' need to be represented as single tokens of and the', respectively, before they are eligible for a supermerge. As a result, early in the process regular merges constitute majority of operations, and over time the proportion of supermerges grows. Supermerges were 31.3% total merges over the entire range to vocabulary size of 131,072. In the rightmost interval, supermerges constitute 44.4% of operations. Thus, the BOUNDLESSBPE tokenizer uses substantial number of superwords, breaking through the pre-tokenization barrier.
3.5 Superwords enable improved pre-tokenization
The WHNNLESS_PPATERSS regex in Listing 2 contains number of improvements over
isting pre-tokenization regex patterns. 8 One notable improvement is the better han- dling of names in code, which is directly enabled by the supermerges in BOUND- LESSBPE. The variable name 'MMHHttpRequest' is composed of three sub-components: 'XML', 'Http', and 'Request', which can be identified by capitalization conventions. The B-2 to B-6 branches of Listing 2 work together to break variable and function names into smaller pretokens based on capitalization. These sub-components can then be recombined via supermerges as their co-occurrence counts warrant. The example "MMLHttpReqeest snake_cass camelCase CONSTANT' is pre-tokenized as:
['XML', 'Http', 'Request', snake', '_case', camel', 'Case', CONSTANT') J.
Using the the commonly used GPT2_ 6 pre-tokenization segments on the full names, unable utilize the príor knowledge encoded the sub-components:
[' "XMLHtttpRqqest'.
snake', '_', 'case', camelCase', CONSTANT ].
Conversely, using BOUNDLESS_PATTE PATTERN with standard BPE would always result in the final tokens being the individual sub-components. With the supermerges BOUNDLESSBPP provides, it is possible both align with sub-components and to recombine some of the full names.
We discuss existing regular expressions and compare them in Appendix A.

---

Published as a conference paper at COLM 2025
BOINDLESSPPATTERN = "]".join([
r"
B-1, contraction
""_(2:>N<LL)\ppMM>>>>
B-2, snake_case
r" r" ((C:DDD((UNN00000)2222))DD)))))) r" ???::N<(u)>p())>>++
# B-3, words
B-4, words
B-5, words
r" "CC:://e(LL//(c///////1000010000 r" FCC/://wwwerr0000033002200000000
B-6, words
(?:AP(N))P(MM>)
B-7
r" ((::INoPP2ND(S3)110009)+
8-8, punct and symbols
B-9, whitespace
B-10, sep or control
"Ap[M]*"
B-11, leftover marks
1)
Listing 2: Pre-tokenization regular expressions (regex)
4 Token distribution
We have seen that pre-tokenization influences more than 90% of the tokens in the vocabulary, which makes it difficult for algorithms like BPE to alter the distribution of token occurrences. However, supermerges offer means to overcome this barrier.
The left column of Figure 4 shows the log counts for each token, sorted in descending order of frequency along the x-axis, in our evaluation corpus. We observe that the tail of BOUNDLESSBPE's token frequency chart is substantially higher than that of the three baseline tokenizers for two vocabulary sizes, 10 and it has far fewer tokens that don't appear combination of higher counts for
in the evaluation corpus (plotted at zero). This is due final merges mentioned above and the deletion of intermediate tokens, which makes room for additional useful tokens. The right column of Figure presents zoomed-in view of this same measure for the 250 tokens with the largest counts in each vocabulary, showing that supermerges have indeed succeeded in reducing the counts overly-geners tokens.
Figure 5 shows the fraction of the vocabulary that is used at least once on the same evaluation corpus. Over 97.5 97.5% tokens in BOUNDLESSBPE's vocabulary are found in the evaluation corpus, across all four vocabulary sizes, while baseline tokenizers' vocabularies include between 82-95%.
Finally, quantify the token distributions over the corpus using two metrics: (1) the Rényi efficiency metric (Zouhar et al., 2023), which indicates how uniform a token distribution is; and (2) compression rate of the evaluation corpus. Figure & shows that the BOUNDLE BPE Rényi efficiency is at least 21 % above that of the baselines, using the a = 2.5 value recommended by Zouhar al. (2023). There is later work pointing out limitations of Rényi efficiency in predicting downstream performance (Cognetta et al., 2024b). However, our goal was to design system with more uniform distribution of tokens, and it does provides quantitative measure of the uniformity.
Figure shows that BOUNDLESSBPE provides at least 19.7% increase in overall bytes per token for the 5GB evaluation corpus compared to baseline tokenizers. Compression rate continues to increase at larger vocabularies, indicating it is able to effectively use the additional vocabulary space. While the effect of compression on downstream performance is unclear (Goldman et al., 2024; Ali et al., 2024; Schmidt et al., 2024; Gallé, 2019), having more bytes per token can speed up language model inference, as fewer tokens are needed to process or generate a given text.
'The baseline tokenizers used the GPT2_ described Appendix as the pre-tokenization regex, chosen because it is the default for the Huggingface tokenizer, and hence very commonly used. 10see Figure in Appendix C for two additional vocabulary sizes.
7

---

Published as a conference paper at COLM 2025
Vocabulary size: 40,960
Distribution
Common Tokens
8-
8.0
BPE
BPE
UnigramLM
UnigramLM
WordPlece
7.5
Werd
6-
BoundlessBrE
BoundlessBrE
log1 (100count+11
5-
log3 1100counnt11)
7.0
6.5
6.0
2-
1-
5.5
0-
5.0
10000
20000
30000
40000
50
100
150
200
250
Sorted by Count
Sorted
Vocabulary size: 131,072
Taken Distribution
250 Most Common Tokens
8-
BPE
BPE
7-
UnigramLM
UnigramLM
WordPlece
WardPlace
BoundlessBPE
BoundlessBPE
5-
log10lcount+11
logtolcount--)
6.0
5.5
0-
5.0
20000 40000 60000
80000 1000C0 120000
0
50
100
150
200
250
Tokens Sorted by Count
Takens Sorted by Count
Figure 4: The left column is the log1o(count, frequent on the x-axis our evaluation corpus. The +1 is to allow plotting of 0 counts. The right column shows a zoomed-in view the 250 most common tokens.
(count + 1) for each token, sorted from most to least
BPE
UnigramLM
WordPiece -
Boundles:BPE
1.000
0.975
Fraction of Vocalo Used
0.950
0.925
0.875
0.850
0.825
0.800
40960
65536
98304
131072
Vecabulary
Figure 5: Fraction of vocabulary used at least once in evaluation corpus, across different tokenization methods and vocabulary sizes. A higher fraction suggests the vocabulary has more useful tokens for representing unseen data.
8

---

GPT40 Published as a conference paper at COLM 2025
-d--
BPC
Unig MMLM
VerdPlece
DoundlesBPEE
BPC
UrigramLN
WordPece
BoundlessBPE
0.50
Blay Piiceety
0.40
0.30
35
0.2
42980
b533b
1E304
121072
40260
550.36
38.14
132072
Vicabulary Size
Vocabulayy Size
Figure 6: Rényi efficiency, calculated over evaluation corpus, with a = 2.5. Tokenizers with higher efficiency are generally desir- able.
Figure Bytes per token, calculated over evaluation corpus. A higher value implies better compression, which can lead to effi- cient training and faster inference.
Ablation study
combined supermerges with a new pre-tokenization regular expression, BOUNDLESS PAATEEN... It also used different form of PickyBPE deletions, where deleted tokens were split into individual bytes before being allowed to re-merge. Our motivation was that the more aggressive decomposition could allow the deleted token to form more compressed tokens. In contrast, Chizhov et al. (2024) break deleted token into the tokens that created it in a pairwise merge.
BOUNDLESSBPE
Vocab Size
Regex Pattern
PickyBPE Bytes Per Token
Rényi Efficiency
40,960
none
4.472
0.4921
40,960
GPT4o
original ours
4.497
0.4920
40,960
GPT4o
4.492
0.4920
40,960
Boundless Boundless
none
4.165
0.4243
40,960
original
4.188
0.4242
40,960
Boundless
ours
4.171
0.4242
131,072
GPT4o
none
5.072
0.4300
131,072
GPT4o
original
5.071
0,4300
131,072
GPT4o
ours
5.075
0,4302
131,072
Boundless Boundless Boundless
none
4.635
0.3684
131,072
original 4.648
0.3682
131,072
ours
4.645
0.3682
Table Effect of regular expression pattern and PickyBPE style on intrinsic measures
Table gives an ablation study that disentangles these changes." 11 We consider both forms of PickyBPE deletions, as well as no deletions. We compare our BOUNDLESS_PATTER to modern baseline of the GPT4o regular expression." 12 We see that the differences in the types of deletions had only a very small effect on either of our intrinsic measures. Deletions may still be important for downstream performance, since low frequency tokens are harder to learn (Su et al., 2024; Yu et al., 2022). The GPT4o regex increased bytes per token by 0.307-0.438. Like F-2 of GPT4_PAG PATTERN discussed in Appendix A, GPT4o allows single
1eSe Appendix D for the same results in two additional vocabulary sizes. The ablations were performed with slightly different merge eligibility regex r"* (?=, (?: 'u2019]) that will perform better with Unicode text.
L101-L114

---

Published as a conference paper at COLM 2025
non-alphanumeric character to combine with word tokens, compressing this case more than DOUNDLESS_PATTERS which was designed to keep character classes more separate.
6 Related work
Impact of pre-tokenization Velayuthan & Sarveswaran (2025) emphasize the importance of pre-tokenization relative to the tokenizer in achieving egalitarian tokenization across languages. They observe that pre-tokenization limits achievable compression, since each pretoken must contain at least one token. Thus, the number of of pretokens is a lower bound the number of tokens. Dagan et al. (2024) also showed the substantial impact of pre-tokenization regex choices on tokenizer compression and downstream performance. Furthermore, Wegmann al. (2025) demonstrate that pre-tokenization has a stronger impact on downstream task performance than that of vocabulary size and training corpus variations.
Multi-word tokens Prior work has explored incorporating larger linguistic units into vocabularies. Salehi et al. (2015); Liu et al. (2025); Huang et al. (2025) highlight the benefits of multi-word tokens for compression, training cost, and model performance. Similarly, Otani et al. (2020) show representation improvements using multi-word expressions (MWE's) in multilingual settings. Kumar & &Thawani (2022) found that adding high-I'MI MWE's improved performance of machine translation better than high-frequency subword or whole- word tokens. Gee et al. (2023) introduced a multi-word tokenizer that augments a standard BPE vocabulary with MWE's by representing frequent 11-grams single token.
In concurrent work, Liu et al. (2025) proposed SuperBPE, an enhancement to BPE that employs two-pass tokenization strategy to obtain multi-word tokens, which they also term superwords. Their method involves an initial BPE training phase with pre-tokenization, conducted up to a vocabulary size T, where T represents a hyperparameter they call the transition point. This phase is followed by second BPE training pass that resumes from the first but omits pre-tokenization, thereby enabling the formation of superwords to populate the remainder of the vocabulary. In contrast to SuperBPE, BOUNDLESSBPE operates in single pass rather than in separate stages, allowing both standard merges and supermerges to occur at the same points based on their respective frequencies. Our approach thus obviates the need for transition point, avoiding hyperparameter search and speeding up tokenizer training. Additionally, our method offers control over which pretokens can be merged together, preventing different types of pretokens, such as words and punctuation, from merging.
See Appendix E for additional related work.
7 Conclusion
While natural language processing has achieved significant advancements in performance over the past decade, certain design choices remain static, such as tokenization algorithms and pre-tokenization regular expressions. Pre-tokenization exerts considerable influence over a corpus token distribution, with standard pre-tokenization methods fixing over 90% of words to be represented single tokens. To address the current limitations in achieving a uniform token distribution over a corpus, we introduce two key contributions. First, we present BOUNDLESSBPE, a modified BPE training process that enables the combination of adjacent full pre-tokens into superwords. The incorporation of superwords yields enhanced compression, quantified by an increase in bytes per token, and more uniform distribution of token frequencies across corpus. Second, we propose BOUNDLESS_PATTER tokenization regular expression designed to work with superwords, resulting in improved tokenization, particularly for code and named entities, when compared to standard regular expressions such as the one used by GPT-2. Based on prior work (Zouhar et al., 2023; Liu et al., 2025) we hypothesize that the intrinsic performance improvements shown by BOUNDLESSBPE will have a positive impact on the downstream performance of language models.
novel pre-
10

---

Published as a conference paper at COLM 2025
Acknowledgments
Many thanks to Ilya Yudkovich and Mike Arov at Kensho Technologies for their technical assistance. Thanks also Seth Ebner, Charles Lovering, and Michael Krumdick Kensho Technologies for many helpful comments and discussions. This research was supported in part by the Israel Science Foundation (grant No. 1166/23).
References
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Jo- hannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Or- tiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. Tokenizer choice for LLM training: Negligible or crucial? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3907-3924 Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 / w11/2224..1nlvvv fo0d2..indings-aaacc... anthoooggy.y.ooo .wrpgg...rrg/g/ org/ http://22....n.nnnnnaaaaaeee findings-nacll.. 247/.
URL https://acl https://aaclntthloogy
Thomas Bauwens and Pieter Delobelle. BPE-kmockout: Pruning pre-existing BPE tokenis- ers with backwards-compatble morphological semi-supervision. In Kevin Duh, He- lena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), Pp. 5810-5832, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: org/ 2024. naacl-long 324/.
htt//2222.aacclloe
URL
Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pre- training. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4617-4624, Online, November 2020. Associ- ation for Computational Linguistics. doi: 10.18653/ www.//2..indiigs--eeeeeeeo findings-emnij 414. 414. URL https://aclanthholggg
findings-enninn 414/.
Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, and Ivan P. Yamshchikov. BPE gets picky: Efficient vocabulary refinement during tokenizer training. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16587-16604, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1 w22//224.mmllpmainn. URL https: / http://ccaaaahooooooo org/ 2204..222..22 mamin.. m911.995/.
Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. In 27th Annual Meeting of the Association for Compulational Linguistics, pp. 76-83, Vancouver, British Columbia, Canada, June 1989. Association for Computational Linguistics. doi: 10.3115 10.3115/981623.1
URL https: / /t//a/aaaaanollooorrrrre wrg/ 1010/.
Marco Cognetta, Tatsuya Hiraoka, Rico Sennrich, Yuval Pinter, and Naoaki Okazaki. An analysis of BPE vocabulary trimming in neural machine translation. In Shabnam Tafreshi, Arjun Akula, João Sedoc, Aleksandr Drozd, Anna Rogers, and Anna Rumshisky (eds.), Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, pp. 48-50, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.insights-11 URL ttttps://aalanthhooloo
org/2024. insights-1.7/.
Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. Two counterex- amples to tokenization and the noiseless channel. In Nicoletta Calzolari, Min-Yer Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 16897-16906, Torino, Italia, May 2024b. ELRA and ICCL. URL
org/ 2024. drec-main, m
Gautier Dagan, Gabriel Synnaeve, and Baptiste Roziere. Getting the most out of your tokenizer for pre-training and domain adaptation. In Ruslan Salakhutdinov, Zico Kolter,
11

---

Published as a conference paper at COLM 2025
Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 9784-9805. PMLR, 21-27 Jul 2024. URL https:/ happs://pree http:////ooeeeeinn mwr. waa..prees/v2335 2a/3///aaa22aa... html. html.
Robert M Fano and WT Wintringham. Transmission of information, 1961.
Philip Gage. A new algorithm for data compression. C Users 12(2):2338, February 1994.
ISSN 0898-9788.
Matthias Gallé. Investigating the effectiveness of BPE: The power of shorter sequences. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1375-1381, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1 D19-1141. URL https: hatp//aaaaannoooggyoooooo org/D19-1141/
Leonidas Gee, Leonardo Rigutini, Marco Ernandes, and Andrea Zugarini. Multi-word tokenization for sequence compression. In Mingxuan Wang and Imed Zitouni (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 612-621, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 /11 v http:/2222.22pmiiiiii. mmn.:..iinnnnn..s....... URL https: haaps:///aclanthologyyy org/ 2023. emnlp-industry.5k
Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty: Unpacking tokenization: Evaluating text compression and its correlation with model per- formance. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Asso- ciation for Computational Linguistics: ACL 2024, pp. 2274-2286, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ vt/20/11222200// 1tt//224.indings--clllll URL https:tt///// /tttps://aclaathhtt http://aaaaaatoooyoooooo. org/2024. findings-acl acl. 134/.
Kyle Gorman and Yuval Pinter. Don't 't touch my diacritics, 2025. URL httphtt//axxiv..or abs/2410. 24140.
Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Xun Zhou. Over-tokenized transformer: Vocabulary is generally worth scaling, 2025. URL 16975.
Jean Kaddour. The minipile challenge for data-efficient language models, 2023. URL https://arxiv.orgg/ 230..08422.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Iryna Gurevych and Yusuke Miyao (eds.), Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational URL https: https:////ccaaahhoogoo// org/ P18-1007/.
Linguistics. doi:
Dipesh Kumar and Avijit Thawani. BPE beyond word boundary: How NOT use multi word expressions in neural machine translation. In Shabnam Tafreshi, João Sedoc, Anna Rogers, Aleksandr Drozd, Anna Rumshisky, and Arjun Akula (eds.), Proceedings of the Third Workshop on Insights from Negative Results in NLP, pp. 172-179, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: URL
org/ 2022. insights-1.24.
Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Hui Chen, Peng Liu, Jungong Han, and Guiguang Ding. Scaffold-bpe: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal, 2024. URL https:/ http//axiivii.//// www.://///bb//44..... 2404./44444888 17808.
Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. Superbpe: Space travel for language models, 2025. URL https://arxiv org/abs/ 2503. 13423.
12

---

Published as a conference paper at COLM 2025
Naoki Otani, Satoru Ozaki, Xingyuan Zhao, Yucen Li, Micaelah St Johns, and Lori Levin. Pre-tokenization of multi-word expressions in cross-lingual word embeddings. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 4451 14644,44,4446 Online, November 2020. Association for Computational Linguistics. doi: 10.1 emnlp-main. 3000-aaai..........
emnlp-main.360. URL https: https:///aaaaathollgg.ooooooo org/2820. hmm/2220.emmmmmainn
Varshini Reddy, Craig W Schmidt, Yuval Pinter, and Chris Tanner. How much is enough? the diminishing returns of tokenization training data. arXio preprint ar arXii::502.20273, 2025.
Bahar Salehi, Paul Cook, and Timothy Baldwin. A word embedding approach to predicting the compositionality of multiword expressions. In Rada Mihalcea, Joyce Chai, and Anoop Sarkar (eds.), Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 977-983, Denver, Colorado, MayJune 2015. Association for Computational Linguistics. doi: 10. harpaaahhooooy.orr///-11999
3115/v1/N15-1099 URL
Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. Tokenization is more than compression. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, PP. 678-702, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ http:/22242emmmmm /2024..mmmmmmmmam4440 URL https: / /ttp//aaaaaaaaa acl.nnollggy.....ooomooooooooooe orgoorg///// 2024. emn momnlpmmiin... 40/.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5149-5152, 2012. doi: 10000///CASS..21226655555
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi:
11/11661162.. URL https://aallntthooogg/ http://aaaaaooooooooo///111111// org/P16-1162/.
Aaditya K. Singh and DJ Strouse. Tokenization counts: the impact of tokenization arithmetic in frontier llms, 2024. URL htt.::ttt//axxii.orrg////// 14903. 14903.
Zhenpeng Su, Zijia Lin, Baixue Baixue, Hui Chen, Songlin Hu, Wei Zhou, Guiguang Ding, and Xing W. MiLe loss: a new loss for mitigating the bias of learning difficulties in generative language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 250-262, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1, 4024.findings-nacc... 18.. URL httpp::/aclaanthologg/ org/2024. findings-nand 18/.
Menan Velayuthan and Kengatharaiyer Sarveswaran. Egalitarian language representation in language models: It all begins with tokenizers. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 5987-5996, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. variation, 2025. URL htwpollooyyorggg/222/ org/ coling main. 400/.
Anna Wegmann, Dong Nguyen, and David Jurgens. Tokenization is sensitive to language 2ttpaaas//2222..11 15343.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the wrw..bs////b/ gap between human and machine translation, 2016. URL https: / http://awwww..aaaasasass/e 08144.
13

---

Published as a conference paper at COLM 2025
Sangwon Yu, Jongyoon Song, Heeseung Kim, Seongmin Lee, Woo-Jong Ryu, and Sungroh Yoon. Rare tokens degenerate all tokens: Improving neural text generation via adaptive gradient gating for rare token embeddinggs In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Amnual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 29-45, Dublin, Treland, May 2022. Association for Computational Linguistics. doi: 10.18653/ 10.18653/1/2022..2 /wt////222222-looogg https://aclantthll./ wrg/2022..c// mong-loog...
URL
Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cot- terell. Tokenization and the noiseless channel. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 5184-5207, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 / w1/2023..cl-long 284. URL htttps://aclattiiii hathoooloaaa org/ 2023. acl-long..288 284/.
A Pre-tokenizer regular expression
Pre-tokenization applies a regular expression (regex) to each document to form pretokens, which are then each tokenized separately. The regex in this section make extensive use of Unicode categories. These are not supported by the re library used by default in python, instead requiring the more powerful regex library. 13 In the notation of regex, Unicode can be divided into the categories given in Table 2.
Category
Description
Ap(L)
Letters
Ap(N)
Numbers
Ap[Z]
Whitespace and other separators
Ap(S}
Symbols
\p[P}
Punctuation
Ap[C}
Control characters
Ap(M)
Combining marks (diacritical marks, etc.)
Table 2: Unicode Character Categories
Any regular expression can be used for pre-tokenization, provided that it matches all of the text in given Unicode string. Thus, at least one branch of the regex must match each of these Unicode categories.
Previous regex used without much emperical justification. Dagan et al. (2024) and Wegmann et al. (2025) are among the first to more systematically investigate the effect of the regex on downstream performance.
number of regex are shown in Listing 3. It includes the regex for GPT-2 and GPT-4," 14 the Punct regex (Dagan et al., 2024), and the proposed regex for BOUNDLESSBPE Each of the regular expression branches shown in the labeled lists are combined together with the operator into single regex. In many of the regex, an initial r" ?" indicates an optional initial space, while final "[Arrn]** indicates zero or more line endings.
A.1 Separation of Unicode Classes
One open question with the pre-tokenization regex is how much the Unicode character classes should be kept separate. The GPT2_PATTERN pattern kept them very separate. The GPT4_PATTERN pattern moved away from that with pattern in line F-2 that allowed any single character besides letter, digit, or line ending to come before word. It also combined line endings with other characters in F-4. To study if this was a good idea, the Punct pattern
1tttpss: prpppio.rg/rrojec//regx/
14aaen from https:: http://ghtwb.....kaaaaatttmnnnnp
14

---

Published as a conference paper at COLM 2025
import regex as re
GPT2_PATTERN "I". join
r" (?: [P:dm]]] |11 [velre)) ?\p[L)+", r" ?\p[N]+",
# T-1, English contractions
T-2, words
# T-3, digits
?[<SSN((>>D<N))>> ("!Ss)" r"sst"
T-4, not letters, digits, or whitespace
# T-5, all-but-last whitespace
T-6, whitespace
))
@PT4_PATTERN "I" "I". joinn
r" (?i: [sddt]]ll]] (ree")", PCCAAAAnDDLLNNN(NIHHIHHDDLL r" ("Np(N)(1,3)", 2CNNSppLL)N((()>=<<MmM ("!s) r"tst"
# F-1, English contractions
F-2, words, opt non-alphanumeric
# F-3, digits
F-4, not letters, digits, or whitespace
r"As*[\rin]",,
# F-5, whitespace with line-ending
F-6, all-but-last whitespace
# F-7, all whitespace
})
PUNCT_ JUNCT_PATTERN [[ ?Xp[L]+", r" ("NP(N)(1, 3CNNSIDELLN((01IC<NW)0> r"NsM[\rin]*" ("!SS)?! r"Ist"
# P-1, words
# P-2, digits
# P-3, not letters, digits, whitespace
# P-4, whitespace with line-ending
P-5, all-but-last whitespace
# P-6, whitespace
})
BOUNDLESS
PATTERN -ATTEE
.oin
# B-1, contraction, allow curly apostrophe r" ??: (OCL)\p(M)*)+(>Nu200)) (P(L)))))())))))))))) # B-2, snake_cas case, with underscore at front
r"_(?: ((::Np(L)))(MM))))"
# B-3, Uppercase, followed by uppercase and lowercase letter
r" ?(? (CMN>/p((\X(M)))))>>>>>>>>> (CC))))))))))((((((((((()")))))) (?: Ap(LI))((M))))) # B-4, optional uppercase, and one or more lowercase
r" 2C22,pC(UU)NMMMH00222:::LLLLLL) B-5, all uppercase acronym CONSTANT
r" ((2:Np(LU)\D(M)>))++
B-6, titlecase, modifier, or uncased letters r" ((C::booLtV)o((um))))o)ooll)v>>)
B-7, numbers
r" (I>N)(N))((D))))1333333???
(WCA://(((AAAANNMMMMM")E))C:CC"0 (S)",,
# B-8, punctuation and symbols r" "((:L\o(PPXoSS))oo(>>>>>>> # B-9, whitespace
r ""AASSrrhHHH(M/222HM>MAMMrr
# B-10, separator or control
r" "(?:IN66ZZNDCC)))0)>>>>"
# B-11, leftover marks, just for bad utf-8 r"MP(M)*"
})
Listing 3: Comparison of pre-tokenization regular expressions (regex)
15

---

Published as a conference paper at COLM 2025
(Dagan et al., 2024) returned to only allowing space before word with P-1. They found an improvement for SUNCT_PATTERS over GPT4_PATTERS at a vocabulary size of 32k, but no significant difference at vocabulary size of 80k. Wegmann et al. (2025) also had mixed results on this question. They found the GPT2_PATTERN with the highest separation was best for tasks requiring robustness to language variation, while more mixing was beneficial to tasks requiring sensitivity to language variation.
In the face of mixed evidence, we keep our classes well separated, in the hope that this will give more of the word based tokens that can be combined by super merges.
A.2 Combining Marks and Unicode Normalization
The T-2, F-2 and P-1 word patterns all have flaw in the handling of combining marks in the \p(M) class. These patterns only match letters, which then end up in a separate pretoken from any combining marks modify that letter. For example, é, which can be represented as 'e\u0301' becomes two separate pretokens. The BOUNDLESS_PATTERN patterns keep all combining marks with their base character. The pattern r" (?: ((:>P(L))p(M))) ", for example, letter and zero or more combining marks. These are wrapped in a non-capturing group, so the unit can be treated like single character. Combining marks are possible with all other Unicode classes, so this approach is used across the BOUNDLESSPPATTESS The final branch B-11 of r"Np[M]+" matches combining marks without base letter. For example the combining acute accent '\u0301 by itself is a valid Unicode code point, but is linguistically ill-formed without base character.
patterns.
The other approach to fix this problem is Unicode normalization. The é can also be written as single pre-composed character '\00eg9' Unicode normalization can be used to convert between these forms. Gorman & Pinter (2025) show that normalization can fix this type of problems with combining marks. However, Dagan et al. (2024, App. D) argue against normalization, as it is usually non-reversible. Our approach of always keeping combining reversible way.
marks with their base characters solves the problem
Velayuthan & Sarveswaran (2025) present a more general problem with extended grapheme clusters in Tamil, Sinhala, and Hindi being broken up by pre-tokenization. Some (but not all) graphemes are formed by base character and one or more combining marks, so this would keep some of their graphemes intact.
A.3 Code Related Pretokens
Patterns B-2 B-6 collectively provide a new form of token alignment for variable and function names in code. Programmers follow strict case conventions that allow the patterns to find the individual words within the names, in a programming language independent way. The example XMLHtttReqqess snake_case camelCase CONSTANT' becomes:
['XML', 'Http', 'Request', snake', "_case', camel', 'Case', CONSTANT') '].
Note that these are aligned with parts of the names. With regular BPE, breaking variable names up pre-ookenization would be prevent the full names from becoming tokens. However, with superword merges these more aligned pretokens have the opportunity to recombine and form the complete variable names. Less common variable names will become several tokens. This is one example of how having supermerges allows more extensive pretoken alignment.
A.4 Whitespace
The existing whitespace regex (T-5 to T-6, F-5 to F-7, and P-4 to P-6) have small variations, but all use negative lookahead to cause the match to backtrack one space.
With the example 'Hello world
AnAn An and GPT4_PATTERN we get: C'Hello', ', world', An\n An J..
16

---

Published as a conference paper at COLM 2025
The F-6 pattern matches the first three of the four spaces between Hello and world. Then F-2 picks off the third space in world'. The F-5 matches multiple lines of whitespace ending in line-ending, and then finally F-6 picks up the remaining trailing whitespace the final line.
Breaking the last space off from longer runs of whitespace increases the number of words preceded by a space. However, runs of multiple spaces are often encountered in the context of code. Especially for space sensitive languages like python, splitting four spaces into three and one would seem to increase the difficulty of coding tasks. Similarly, having white space span multiple lines, in this example, seems to make coding tasks more difficult. So in contrast to the other approaches, B-9 keeps runs of whitespace together, and breaks multiple lines of whitespace into separate tokens with one or more line endings. For the example we would have:
C'Hello',
', 'world', An'n', An', '].
This is an area that would benefit from further experimentation.
A.5 Numbers
Singh & Strouse (2024) take a detailed look at the tokenization numbers. Early models used patterns like T-3, that just used whichever numbers were found by BPE. As they describe, models then switched to runs of) 1 to 3 digits (F-3, P-2) or to using single digits. By adding commas to numbers in the input context enforce right-to-left tokenization they saw dramatic decrease in arithmetic errors. However, this can be done directly with regex B-7 without the need for inserting commas, so that '1234567' becomes the pretokens ['1', '234', '567'3 ].
A.6 Contractions
The T-1 and F-1 patterns match English specific contraction endings like 've or '11. Dislik- ing the English-specific nature, this was omitted from Punct. Pattern B-1 is more language- independent. keeps the ending of the contraction together with the word as a single pretoken. Thus, any word containing straight or curly apostrophe internal to the word is matched, like C' d'est or J J'ai
B Efficient implementation
In the representation shown in Section 3.2, document divided into pretokens, each of which contains one or more tokens. This is simple conceptually and can be used directly for training, but is extremely slow. Tokenizer training routines commonly use the trick of aggregating the pretokens produced over large training corpus, and use the aggregate counts when calculating pairwise merge counts. Thus the pretoken [' the is only ever tokenized once even though it might appear hundreds of thousands of times. This speedup is crucial for performance reasons. 15
This allows a faster implementation of BOUNDLESSBPE. We keep two separate sets of the training data. We pre-tokenize and tally up the frequency of the pretokens for regular merges. The aggregation here results in more than 10x speedup on the training time. This first set of data is initialized with single bytes, and merges are selected according to the total aggregated counts.
We have separate second copy of the training data for the supermerges, where each document is initially broken into pretokens, which will combined into superwords. Since most documents are distinct, no aggregation can help speed up supermerges.
15 have our own implementation of BPE with deletions using this speedup, based on Andrej Karpathy's minbpe library. https://gt //gs::///ittt thub. http:///ggtuuuu.caaaaaaatttttttt com/karpathhy/iinbe
17

---

Published as a conference paper at COLM 2025
These two representations related through the process of unlocking pretokens. The first subword representation proceeds as regular BPE. However, when a pretoken there is reduced to single token, we say that that pretoken has been unlocked.
The second superword representation can only consider a pairwise merge when two adjacent words unlocked. After new pretoken is unlocked, the counts of the superword must be updated accordingly. We track the pairwise and single counts of each representation. For speed, we dynamically compute the changes in counts that result from performing each merge. Despite these optimizations, dataset due the lack of aggregation for supermerges. For comparison, training Hugginface BPE on the same data, vocabulary size, and machine takes 59s.
still took 4.7 CPU days to train our 1GB training
C Token distribution at additional vocabulary sizes
Figure 8 shows the same plot Figure 4 at the vocabulary sizes of 65,536 and 98,304. The trends are largely the same as in Figure 4, with BOUNDLES$BPE having higher counts at the low frequency end of the distribution compared to baselines, and lower frequencies for the most common tokens. Both these are desirable to have more uniform distribution.
Vocabulary size: 65,536
Token Distribution
250 Most Common Tokens
8.0
BPE
BPE
UnigramLM
UnigramLM
WordPiece
7.5
WordPiece
6-
BoundlessBPE
BoundlessBPE
7.0
5-
0g10(count-11
og10icount
4-
3-
2-
1
5.5
10000
20000
30000
40000
50000 60000
100
150
200
250
Takens Sorted by Count
Takens Sorted by Count
Vocabulary size: 98,304
Token Distribution
250 Most Common Tokens
8.0
UnigramLM
UnigramLM
WordPiece
7.5
WordPiece
BoundlessBPE
BoundlessBPE
7.0
agg0tcount+11
log10/count-))
4-
6.5
3-
6.0
2-
1
5.5
0-
5.0
20000
40000
60000
80000
100000
50
100
150
200
250
Takens Sortad Count
Takens Sortad Count
Figure 8: Left column is the (count + 1) for each token, sorted from most to least frequent on the x-axis on our evaluation corpus. The +1 is to allow plotting counts. Right column shows zoomed-in view of the 250 most common tokens.
18

---

Published as a conference paper at COLM 2025
D Ablation results at additional vocabulary sizes
Vocab Size
Regex Pattern
PickyBPE
Bytes Per Token
Rényi Efficiency
65,536
GPT4o
none
4.722
0.4652
65,536
GPT4o
original
4.722
0,4651
65,536
GPT40
ours
4.735
0.4651
65,536
Boundless
none
4.358
0.3996
65,536
Boundless
original
4.366
0.3995
65,536
Boundless
ours
4.370
0,3996
98,304
GPT4o
none
4.927
0.4440
98, 98,304
GPT4o
original
4.917
0.4438
98,304
GPT40
ours
4.953
0.4438
98,304
Boundless Boundless Boundless
none
4.549
0.3807
98,304
original ours
4.530
0.3806
98,304
4.542
0.3805
Table 3: Effect of regular expression pattern and PickyBPE style intrinsic measures for vocabulary sizes of 65,536 and 98,304
E
Additional related work
Foundational Subword Tokenization Algorithms Byte level subword tokenization has become fundamental component in modern NLP, balancing vocabulary size with mor- phology and out-of-vocabulary handling, Byte Pair Encoding (BPE; Sennrich et al., 2016) iteratively merges the pair of adjacent tokens with the highest count build a vocabulary. WordPiece (Schuster & Nakajima, 2012) is similar to BPE, except that merges are selected according to their Pointwise Mutual Information (PMI). Kudo (2018) introduced the Uni- gramLM tokenizer, a top-down approach that starts with a large vocabulary and prunes tokens based on their contribution to sequence likelihood according to a unigram language model.
Removal of Intermediate Tokens The pairwise nature of BPE merges mean some scaffold tokens added to the vocabulary simply serve as bridge more popular token, but are not often used on their own. Bostrom & Durrett (2020) observed dead zone of such tokens in both UnigramLM and BPE vocabularies. The proportion of such tokens were found to be higher in the case of BPE, motivating research into their removal and vocabulary refinement. In Lian et al. (2024) scaffold tokens are marked during a training process, and split into components as postprocessing step during inference. Chizhov et al. (2024) integrate the deletion step into the BPE training process, where can affect later merge decisions. Thus ordered combination of merges and deletions occur during inference. We find Chizhov et al. (2024)'s approach particularly compelling due to its direct integration into BPE training. In contrast, Bauwens & Delobelle (2024) focused on post-processing step to remove merge rules decreasing morphological alignment. We opted against a purely post- processing approach like Cognetta et al. (2024a) to maintain tighter control over vocabulary construction during training.
19


Reviews:
[{"review_id": "pr51DQ35Ei", "reviewer": "Reviewer_DcHp", "summary": "The paper is generally of good quality. It presents a well-motivated approach to improving BPE tokenization. The empirical evaluation, though focused on intrinsic metrics, is conducted thoroughly across multiple vocabulary sizes and compared against standard baselines. The discussion of pre-tokenization regex patterns in the appendix is detailed and insightful.\nThe paper is also very well-written and generally clear. The core concept of \"supermerges\" (the boundless part) is well-explained, with many examples to aid understanding.\nThe central idea of allowing merges across pre-token boundaries (\"supermerges\") within a BPE framework is relatively novel. Though similar approaches have been investigates, the lack of additional parameters is a plus.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method offers a clear and intuitive way to address the limitations of standard pre-tokenization in BPE. It also appears to improve on some other recent methods by requiring less complex setup. The paper is very well-presented, with clear explanations and illustrative examples that make the concepts accessible. Furthermore, the authors engage thoughtfully with both recent and historical literature in the field.", "reasons_to_reject": "The paper has one primary weakness that may make it not quite ready for publication at COLM: a significant lack of detailed ablation studies. This makes it difficult to understand the precise impact of each individual modification introduced by the authors.\n\nThe authors combines several changes: (1) the core \"superword\" merges, (2) a new variant of PickyBPE deletions, and (3) two new regular expressions (the main BOUNDLESS_PATTERN for pre-tokenization, and a separate regex that constrains which pretokens can form supermerges). Without ablations, it's hard to tell which of these changes are responsible for the observed improvements in token distribution and compression.\n\nAdditionally, the configuration of the \"standard BPE\" baseline used for comparison is not consistently clear. The paper sometimes implies the use of OpenAI's GPT-2 regex pattern, but its choice over the latest one (GPT-4o) is not clear. \n\nGiven that the paper focuses exclusively on tokenizer-level metrics and does not include downstream model performance, a detailed ablation study is even more critical. Such a study would allow readers to better interpret the results and understand the value of each proposed change. An ideal study would incrementally add features to a clearly defined, strong baseline (e.g., standard BPE with a modern regex like GPT-4o's). While this would likely require significant additional work and a resubmission, it would greatly strengthen the paper and clarify the benefits of each component.", "questions_to_authors": "#### PickyBPE\n\nThe paper should provide a clearer rationale for its specific variant of PickyBPE, particularly the decision to decompose deleted tokens into bytes rather than their constituent BPE tokens (as in the original PickyBPE). What results or hypotheses led to this choice? The concern that this aggressive decomposition could lead to re-learning identical merges also needs to be addressed more directly.\n#### Supermerges restriction pattern\n\nThe regex used to restrict which pretokens can form supermerges is very influential but discussed only in a footnote. Its design choices, such as using [a-zA-Z] instead of a more Unicode-aware \\p{L}\\p{M}* (which is used in the main BOUNDLESS_PATTERN), should be justified. This current pattern appears to limit supermerges to English-like words. The special handling for snake_case variables also seems somewhat arbitrary when other common mixed-text patterns are not explicitly included.\n\n#### Multilinguality\n\nThe experiments are conducted on MiniPile, an English-only dataset. This is a notable limitation, as most modern LLMs are multilingual. The paper should clearly state this limitation and ideally discuss how the proposed system (especially supermerges and the capitalization-sensitive BOUNDLESS_PATTERN) might perform or interact with languages that have different orthographic conventions, such as those without spaces. Throughout the paper there are also subtle overstatements with respect to languages such as \"Pattern B-1 takes a different language-independent approach\".\n\n#### Smaller comments\n* Figure 2 is difficult to read due to overlapping lines. Perhaps plotting a relative metric or using a different visualization approach could improve clarity.\n- While comprehensive, the WordPiece and Unigram results might be less critical, in light of their lack of recent use, and generally subpar performance by the authors' metrics.\n * \"no aggregation can help speed up supermerges\" seems an overly strong claim, even if not researching or implementing such novel aggregations is reasonable.\n* \"it still took 4.7 CPU days to train our 1GB training\". lacks context. How does this compare to the training time for the baseline BPE on the same hardware and dataset?\n* \"The T-2, F-2and P-1 word patterns all have a flaw in the handling of combining marks in the \\p{M}class\" is correct, but also outdated in light of the GPT-4o update.\n* \"Any regular expression can be used for pre-tokenization, provided that it matches all of the text in a given Unicode string. Thus, at least one branch of the regex must match each of these Unicode categories.\" This assumes a 'findall' approach. It might be worth noting that split-based approaches, where non-matched spans become pretokens (as used by e.g. DeepSeek), also exist.\n* \"The other approach to fix this problem is Unicode normalization\" Note that though this is fairly accurate for western languages, not all marks can be removed by Unicode normalization.\n* \"# B-11, leftover marks, just for bad utf-8\" - is inaccurate. These are valid standalone Unicode combining marks. The paper's other discussion referring to them as \"isolated marks\" or \"linguistically ill-formed without a base character\" is correct.\n* For the superwords pattern, I believe ```\\$``` should just be ```$```", "extra_scores": {}}, {"review_id": "rFeZTwwGyI", "reviewer": "Reviewer_bq5M", "summary": "This submission proposes and analyzes a modification of the popular BPE tokenization algorithm. While the original BPE does not merge across pre-tokenization boundaries (most typically: whitespace), the key idea of the proposed algorithm is to allow \"super-merges\" of pre-tokens, e.g., merging \"_of\" and \"_the\" into \"_of_the\", thereby creating what the authors call \"superwords\". After a thorough motivation and description of the proposed algorithm, the algorithm is shown to outperform all common subword tokenization methods in an extensive, but only intrinsic evaluation, which looks at metrics such compression rate and Renyi efficiency.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is simple, well-motivated, and makes sense, and the resulting tokenizer -- at least in the intrinsic evaluation -- performs very well.\n2. The paper is written very clearly and also serves as a nice introduction to recent work in this area\n3. The intrinsic evaluation is thorough and convincing (in terms of comparing tokenizers on their own)", "reasons_to_reject": "1. The evaluation is only intrinsic, i.e., there is no evaluation of the proposed algorithm in terms of language modeling or downstream task performance. One of the measures used, Renyi efficiency, may be one of the better metrics for predicting relative downstream task performance, but  as Cognetta et al. (2024), note \"Although useful, the predictive power of this metric is not perfect\". This submission would be *much* stronger if it would show the benefit of superwords in an extrinsic evaluation, maybe by training -- at the very least -- a NanoGPT-style proof-of-concept. I know that this relates to availability of computing resources, but I still find it difficult to give a higher score without any extrinsic evaluation.\n\n\nReference: Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. 2024. Two Counterexamples to Tokenization and the Noiseless Channel. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16897–16906, Torino, Italia. ELRA and ICCL.", "questions_to_authors": "I'm wondering how BoundlessBPE compares to omitting pre-tokenization. In the paper, you say \n> Schmidt et al. (2024) demonstrated that entirely omitting pre-tokenization resulted in the poorest downstream performance among 18 evaluated tokenizers.\n\nBut looking at Table 1 in Schmidt et al. (2024), it appears that the \"no pre-tokenization\" setting is only compared for their proposed algorithm, but not for any of the other ones.\n\nHaving played around with the --split_by_whitespace=False option offered by SentencePiece, which allows to efficiently merge across whitespace boundaries, I noticed that the resulting vocabulary contains many merges of two, three, or even more pretokens. So I'm wondering how BoundlessBPE compares to naive BPE without pre-tokenization.\n\nConcurrent work (SuperBPE, Liu et al. 2025) has a short paragraph on this, but I'm curious what your take on naive BPE without pre-tokenization is.", "extra_scores": {}}, {"review_id": "COAlK9atLh", "reviewer": "Reviewer_sXwb", "summary": "The authors suggest to extend BPE with \"superword\" merges and provide some experimental evaluations that hint that the proposed tokenization is more efficient.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's a result that would be particularly interesting in the context of \"tokenization vs. tokenless\" debate. The proposed idea is elegant and the evaluations hit towards its usability.", "reasons_to_reject": "The evaluation of the proposed tokenization algorithm is not fully conclusive. The proposed method is a combination of two ideas: removal or undertrained tokens (similar to Chizhov et al. 2024) and superword tokens. There are several papers that suggest superword tokenizations too:\n\nhttps://arxiv.org/pdf/2503.13423\nhttps://ieeexplore.ieee.org/abstract/document/10815724/\nhttps://arxiv.org/abs/2211.11041\n\nIt seems natural to provide some form of an ablation study to illustrate if the combination of those two ideas has some particular synergistic effect as well as to compare the proposed solution with the BPE alternatives mentioned in the paper.\n\nSome minor corrections:\n\"Lian et al. (2024) and Chizhov et al. (2024) proposed strategies to eliminate these low-frequency tokens formed as intermediate steps. We find Chizhov et al. (2024)’s approach particularly compelling due to its direct integration into BPE training.\"\n\nTo the best of my knowledge, both proposed methods integrate the updates into the BPE training direclty, however, Picky BPE has better inference since it takes into consideration the sequence of removals and merges explicitly.", "questions_to_authors": "1. How would the proposed method affect multilingual models?\n\n2. Could you, please, clarify your logic here:\n\n\"We first pretrain 8B models with BPE and SuperBPE tokenizers. We use the OLMO2 7B (OLMo et al., 2024) training configuration, including the model architecture, training hyperparameters, and pretraining corpus, but reduce the total number of training steps to correspond to ∼330B tokens (compared to 4T) for the sake of compute.\"\n\nFor reference, out of the recent models Qwen 3 600m parameters has 6T tokenizer. Could you explain your reasoning in detail here? Is there any reason why reduced amount of tokens makes your results more conclusive?", "extra_scores": {}}, {"review_id": "l2W8K6BxQZ", "reviewer": "Reviewer_SuJi", "summary": "This paper highlights the limitation of common pre-tokenization practices, namely splitting on whitespaces and punctuation. The authors propose a modified BPE algorithm to address this issue. While I highlight some concerns about the validity and generalizability of the results, I feel that this proposed approach to tokenization is exciting and a valuable contribution to the literature. As the authors plan to release their code, they are enabling others to study the features and benefits of the proposed tokenization algorithm.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The combination of superword tokenization with vocabulary tripping via the IoS metric is extremely interesting. One potential downside of superword merges is that it could create long tokens which are rarely seen. Additionally, as superword tokens will be longer, they will require more merges, increasing the chance of useless intermediate tokens. This is reflected in the lower number of unreachable* tokens.\n* This seems like one of the most significant proposed modifications of the BPE algorithm, which is exciting and contributes to a growing line of work seeking more optimal tokenization algorithms.\n\n*I recommend using the term ‘unreachable’ or mentioning its usage in https://arxiv.org/pdf/2405.05417.", "reasons_to_reject": "* The authors speculate in the conclusion that they believe this tokenization approach would result in improved downstream performance; however the authors do not train a model in order to test this. The authors could try to estimate this using average sentence log probability (proposed in https://arxiv.org/pdf/2109.07306), which was shown to be highly predictive of downstream performance. Currently, it is unclear whether changes to tokenization would make significant performance changes, especially as Schmidt et al. 2024 (https://arxiv.org/pdf/2402.18376) show that in many cases changes to tokenization lead to minimal changes in downstream performance. \n* The results are only for English and are only on one dataset (presumably, though this isn’t stated). The results would be more convincing if the authors trained multiple tokenizers on different datasets (or even subsets of the same large dataset) in order to demonstrate that these effects are robust and not specific to any one dataset. \n* Some key methodological details remain unclear. What data is the tokenizer trained on? And what is it evaluated on? Is it the same data mentioned in Section 2?\n* I think the way that Renyi entropy is used and discussed is potentially problematic, as it does not always correspond with better performance (https://arxiv.org/pdf/2402.14614; https://arxiv.org/pdf/2408.11443). The authors should include more discussion so as not to give the impression that Renyi entropy can be completely trusted as a metric predictive of better performance.", "questions_to_authors": "* Given the lack of downstream task evaluations, I would have liked to see more analyses of the tokenizer vocabulary. The authors report a small number of examples of different types. For example, how many of the superword tokens start with '_the'? How many of the superword tokens are full phrases (as opposed to parts of phrases missing one or more subword tokens)? This could be tested by looking at the tokenized corpus and seeing how many times a superword token is preceded by or followed by a space, effectively testing whether BoundlessBPE successfully creates full-phrase superword tokens. Other work (https://aclanthology.org/2023.eacl-main.45/, https://arxiv.org/pdf/2409.04599?) analyzes the number or proportion of word-initial tokens. This type of analysis could also make the paper richer. \n* Is there a way to evaluate the nature of the multi-word tokens? How much of the time is it cases like '_of_the', versus awkward part-phrases? How many are named entities like 'New York City' versus more functional ones like '_of_the'?\n* How much slower is implementation this than Hugging Face’s BPE implementation? The authors report that training took 4.7 CPU days, which seems very long. Is it possible to overcome this so that it can be used for a language model?\n* Is this tokenization method sensitive to the quantity of training data? Based on the appendix, the authors trained the tokenizer on 1GB of data. Recent work shows that up to around 150GB there is a benefit to training on more data (https://arxiv.org/pdf/2502.20273; note: the authors cite this paper, but do not discuss that aspect of the paper). \n\nMisc:\n* L175 open source → open-source", "extra_scores": {}}, {"review_id": "9ApfeBBz7p", "reviewer": "Reviewer_s1uu", "summary": "The submission proposes a modification to the standard pre-tokenization approach. The paper introduces superwords, which are tokens of n-gram. The goal is to lower the frequency of the most common tokens in a corpus, enabling more effective utilization of larger vocab. The method is called BOUNDLESSBPE, which performs the most frequent merges during BPE training. Deletion is also considered. Experiments show that the proposed method yield a more uniform token distribution and improved compression.\n\nQuality: The submission is well motivated and executed for the methodology part. The experiments look very brief unfortunately.\n\nClarity: The reviewer finds the submission written clearly.\n\nOriginality: The reviewer is ok with this dimension. The method is simple which may need deep insight.\n\nSignificance: The paper may have good potential significance given the importance of tokenizations. \n\nPlease see detailed pros and cons below.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Pre-tokenization is an important component of language models. The paper provides good motivations on the limitations of existing methods, and the proposed method is simple and sensible.\n\n2. The proposed method is simple, requiring non-trivial insight to the problem.", "reasons_to_reject": "Overall the reviewer feels the experiments of the submission is lacking.\n\n1. The evaluations are limited to tokenization itself, not any downstream tasks using LM. The experiments are essentially just validated on one small dataset. These limit the significance of the work at its current form.\n\n2. Baselines compared are limited to the classical ones (Wordpiece, BPE, UnigramLM) but not more competitive and recent ones. It's unclear why PickyBPE is discussed, borrowed from, but not directly and fairly compared with.\n\n3. Experiments are limited to EN. This seems to be very limited for a tokenization work.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This tokenization paper proposes to lower the impact of pretokenization, by allowing complete pretokens to be merged (but not smaller units, a major difference of just applying BPE without pretokenization). The authors engaged with the (excellent!) reviews and the discussion reads like a journal submission condensed in timeline. With the additional remarks and experiments the paper will be much stronger

The main drawbacks are:

- lack of dissociation of what modification drove what gains. *This has been addressed during the rebuttal phase*

- no evidence of impact of the final model. This is a general issue with tokenization proposal, and while a pity, it is a lot to ask for such a paper to train even small LLMs (and even then, there will always be the doubt if this extrapolates to larger models)

- lack of multilingual experiments. This seems a major issue, and the main reason for my `Maybe` rating for what is otherwise an excellent paper
Decision: Accept