Learn Your Tokens: Word-Pooled Tokenization for Language Modeling
Authors: Avijit Thawani
Saurabh Ghanekar
Xiaoyuan Zhu
Jay Pujara
Venue: EMNLP 2023 Findings (2023)
Keywords: Tokenization
Language Modeling
BPE
Subword
Segmentation
Numeracy
Multilingual
TL;DR: A word-pooled end-to-end tokenizer: more robust than subword, more efficient than byte/char-level; outperforms both by over 300% on language modeling.

Abstract: Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as 'ing' or whole words. 
Recent literature has repeatedly shown the limitations of such a tokenization strategy, particularly for documents not written in English and for representing numbers.
On the other extreme, byte/character-level language models are much less restricted but suffer from increased sequence description lengths and a subsequent quadratic expansion in self-attention computation.
Recent attempts to compress and limit these context lengths with fixed size convolutions is helpful but completely ignores the word boundary.
This paper considers an alternative 'learn your tokens' scheme which utilizes the word boundary to pool bytes/characters into word representations, which are fed to the primary language model, before again decoding individual characters/bytes per word in parallel.
We find that our moderately expressive and moderately fast end-to-end tokenizer outperform by over `300%` both subwords and byte/character models over the intrinsic language modeling metric of next-word prediction across datasets. It particularly outshines on rare words, outperforming by a factor of 30! We extensively study the language modeling  setup for all three categories of tokenizers and theoretically analyze how our end-to-end models can also be a strong trade-off in efficiency and robustness.

Learn Your Tokens:
Word-Pooled Tokenization for Language Modeling
Avijit Thawani dawani@ussc.. edu
Saurabh Ghanekar
Xiaoyuan Zhu USC
Jay Pujara
USC
USC/ / ISI
Abstract
Efficiency
Expressivity
Accuracy
Subword
High
Low
Mid
Language models typically tokenize text into subwords, using deterministic, hand- engineered heuristic of combining characters into longer surface-level strings such 'ing" whole words. Recent literature has repeat- edly shown the limitations of such a tokeniza- tion strategy, particularly for documents written in English and for representing num- bers.
Byte/
Low
High
Low
Char
cByte/
Mid
Mid
High
eChar
Table 1: Trade-off's involved when choosing tokenizers:
Subword vs Bytes/Characters eByte/eChar (ours).
other extreme, byte/character- level language models much less restricted but suffer from increased sequence descrip- tion lengths and a subsequent quadratic ex- pansion in self-attention computation. Recent attempts to compress and limit these context lengths with fixed size convolutions is help- ful but completely ignores the word boundary. This paper considers an alternative 'learn your tokens' scheme which utilizes the word bound- ary to pool bytes/characters into word repre- sentations, which are fed to the primary lan- guage model, before again decoding individual characters/bytes per word in parallel. We find our moderately expressive and moderately fast end-to-end tokenizers outperform by over 300% both subwords and byte/character mod- els over the intrinsic language modeling metric of next-word prediction across datasets. It par- ticularly outshines on rare words, outperform- ing by factor of 30! We extensively study the language modeling setup for all three cate- gories of tokenizers and theoretically analyze how our end-to-end models can also be strong trade-off' in efficiency and robustness. [Github].
n-gram occurrences or part-of-speech tags or other representations of syntax. All of these pipelines have over time been replaced with end-to-end learn- ing using recurrent neural networks (RNNs) transformers, however the tokenization schemes remain static, deterministie, and manually engi- neered.
State-of-the-art approaches include subword kenization schemes such as WordPiece (Wu et al., 2016), Byte Pair Encoding or BPE (Sennrich et al., 2016), and Unigram (Kudo, 2018), all of which are statistical methods for preprocessing a large unlabeled corpus of text to yield a fixed vocabu- lary, midway between characters or bytes at one end and whole words at the other. This results in convenient trade-off in sequence description length while avoiding the UNK token, that is, fallback mechanism for handling rare words. However, it is not obvious why these hand-engineered algorithms would be the optimal forms of tokenization and whether there exists a possibility for end-to-end models to also include this crucial stage of the NL.P pipeline.
1 Introduction
Almost all natural language processing (NLP) be- gins with tokenization (Mielke et al., 2021). Se- quences of characters (mostly deterministically) segmented into discrete tokens, each of which has enormous vocabulary
Recent work has shown countless limitations with subword embeddings. Several languages con- tain diverse morphological features whereas sub- word segmentation is mostly apt at only identifying suffixes and prefixes (Clark et al., 2022). Technical domains such as biomedical documents often need to pre-train their own tokenizer for improved vocab- ulary (Boecking et al., 2022). Finally, numbers are
a lookup embedding matrix. Statistical NLP methods, similar to other forms machine learning at the time, relied on feature extraction from these tokens, in the form of

---

often inconsistently segmented into subwords, lead- ing decreased arithmetic (Wallace al., 2019) and estimation (Thawani et al., 2021b) skills. The extent of these numeric limitations is so dire that GPT-4 (OpenAI, 2023) has an explicit workaround of adding all numbers from 0 to 999 as individual tokens to the model's vocabulary.
b
E08
S
t
E08
Transformer Decoder (Step 3)
Transformer Decoder (Step 3)
b
y
CLS
a
Recently, several language models have been proposed which remove the tokenizer vocabulary entirely, beginning with a character (El Boukkouri et al., 2020) or byte-level (Xue al., 2022) vocab- ulary and often compressing them into fixed units of around four tokens each (Tay et al., 2021; Yu et al., 2023; Clark et al., 2022). While these zero- assumption methods are useful in compressing text and consequently expand context windows, they completely ignore the word boundary. Besides, the so-called 'tokenizer-free' byte-based models are not entirely fias-free since the Unicode-8 encod- ing they use is itself biased towards representing Latin scripts with a single byte each, whereas some scripts' like Bammum (Africa), Meetei (India), and Cherokee (North America) may require four bytes to represent single character.
Transformer Word-Level Decoder (Step 2)
cus,
Transformer Encoder (Step 1)
Transformer Encoder (Step
e
O
y
Figure I: Overview of our proposed simple end-to- end tokenized autoregressive language model. A trans- former encoder compresses the variable number base units (here, characters) into n=1 CLS tokens per word. Dotted characters are the previously predicted tokens at inference, when training they are the ground truth.
The concept of words is fundamental feature of nearly all human languages, including those written in Chinese or Japanese scripts that do explic- itly delineate words by whitespaces. This paper empirically studies the case where tokenizers lose their subword segmentation algorithms but utilize the word boundary for a multi-level model with added efficiency. More concretely, we use the word boundary to compress the base tokens of bytes or characters into word representations, which are then fed into the underlying language model (here, a small version of GPT (Radford al., 2018)).
ary to compress an autoregressive language model's base tokens.
3.
We report over 300% gains in language mod- eling capabilities over multiple languages and datasets, against both subwords and charac- ter/byte models, and by factor of 30 rare words.
4. We theoretically analyze strengths and weak- nesses of word-compressed tokenization scheme, which carries insights for the lan- guage modeling community.
Our end-to-end learned tokenization undoubt- edly has several limitations. It not faster than subwords. It does not allow characters/bytes within one word to directly attend to those in another word. It relies on the word boundary. which is not straightforward to find for most internet-scale datasets. Nevertheless, deee-dive into tokenizers for language modeling offers the following contributions:
We will publicly release all code (see supplemen- tary material) and checkpoints upon acceptance.
believe this empirical
Method
Figure 1 pictorially depicts our proposed language model architecture. Our end-to-end tokenization strategy is straightforward word pooling method which uses transformer encoder (Step 1) to pool in the base tokens (characters or bytes) into fixed number of embeddings per word. This is analogous to how CLS embeddings are often used to pool the embeddings an entire sentence or any text
1. We compare different tokenizer strategies for language modeling on multiple facets and on fair footing across languages.
2. We are the first to explicitly use word bound-
Phttps://un code. woree wttg/rrodooie roadmaps/bmp/

---

sequence in BERT-like transformer encoders. In our case, we have the equivalent of fixed number of CLS tokens? prepended to each word that store the meaning of the entire word.
Subword segmentation maps the base units de- terministically into fewer subwords per word, i.c. ....... mi where mf S ma, the number of subwords that the ith word is decom- posed into. Following this determinsite process, a subword model predicts the next subword as:
wi = cp. ..... C. +
Next. (Step 2) the pooled per-word embeddings are passed onto the main language model, in our case, a vanilla transformer decoder like GPT (Rad- ford al., 2018). Finally, the contextualized word embeddings are fed through another transformer decoder to autoregressively decode the next word, one base unit (character or byte) at a time. Note that we call this method an end-to-end 'tokenizer' since it compresses the many units into few em- beddings per word, just like subwords, except the compression is learned from scratch. Finally, at de- coding stage (Step 3), the contextualized word rep- resentations are unrolled with another transformer decoder to autoregressively predict one base token (character/byte) at a time."
8/ Decoder -0. 50 mi
Our end-to-end models instead follow a three- step process to (1) pool basc units into a fixed set of embeddings per word, (2) autoregressively pre- dicting the next word embedding, and (3) autore- gressively predicting individual unit embeddings per word:
CLS: E c.t)
(1)
CLS; Mecoder((CSS. CLSI, CLS
(2)
c = Deeoderr(CLS)
d
(3)
Note how we achieve our purported trade-off between subwords and byte/character models. The CLS representations learnt are unconstrained by deterministic mapping as in subwords. They are also efficient to compute and decode from, since the first and last steps only allow intra-word attention. For a tokenizer-free model, roughly 80% of the memory bottleneck* is spent on tokens from one word attending to tokens on another word, which we contest is of questionable importance relative to the overhead incurred.
Here, Encoder refers to a transformer BERT-like encoder and Decoder refers to transformer GPT- like decoder. From an implementation standpoint, we prefix fixed number (n 1 or 4 in this pa- per) of CLS tokens to every word before passing it through a transformer encoder. The word-level con- textualized representations obtained on the other end are collectively depicted here as w;-
Figure 2 a visualization of how our end-to-end model saves on self-attention computation bottle- neck by only allowing intra-word attention at the first step, before allowing contextualization of in- formation across the word boundary in step us- ing the base decoder model. Finally step 3 again restricts the individual characters/bytes to be pre- dicted using only the single word-level predicted embeddings."
Formally, we begin with a sequence of words wo, W1, Wn, each of which is comprised of an ordered set of base units (character/bytes) wy cf.c...... c. where m, + 1 is the length of the ith word. The task is autoregressive language modeling, i.e. given the previously seen words Wo; wh...., W;-1 as well as the previously seen pre-
units in w; (the current word): cl.,
Experiments
dict the next unit d/-
There are numerous NLP tasks that can bene- fit from improved tokenization, such as Machine Translation, Question Answering, and Text Clas- sification. However, the scope of our preliminary analysis is not to cast wide net over every down- stream application. Instead, we choose to analyze in depth the most commonly used pre-training task in NLP i.e. language modeling.
Character/byte level models ignore the word- boundary and directly model the task as:
d = -Decoder(co
mi
"This paper uses 4 CLS tokens per word, except in Section 4.2 where we ablate with CLS per word.
'This autoregressive decoder can also be replaced by a aotoeeooreeeesivvee transformer which emits the entire word O(1) time. Our initial experiments with such vanilla setup performed much worse than autoregressive models (in line with prior work), therefore we leave this to future work.
We pretrain autoregressive language models from scratch using different tokenizers described in
"In Figure 2, this is the difference in blue attention blocks depicted in Figure hetween Byte/Char-level models and our intra-word attention.
'Note that our current implementation has minor devia- tions from the shown simplistic figure. Refer to Section 3.4 for details.

---

e CLS, a CLS J
cLS;
g
a
job
CLS
Step 3 Decoder
CLS
CLS
j
CLS,CLS,CLS,
CLS,
Ours
Step 2: Main LM
CLS,
CLS
Byte/Character-Ievel
CLS, CLS, cLs, o b
CLS get a job
CLS,
Step
get
Encoder
CLS,
job
Subword-level
b
Figure 2: Self-attention visualized across (1) Byte/Char-level models, (2) Subword/Word-leww models, and (3) Our proposed end-to-end tokenization modules (word encoder; base LM decoder; word decoder) with character base. Blue blocks indicate self attention mask. symbol indicates prepended CLS token per word.
the previous section, on different datasets described in Section 3.2.
Dataset
Size (MBs)
Words (Mil.)
Chars/ Word
English
4.7
1.34
5,46
3.1 Models
French
5.1
1.55
5.18
We report results over the following tokenizers: Subword: pretrained BPE vocabulary used by GPT-2 and GPT-3. Byte: a pretrained byte-level cabulary as implemented in ByT5 Xue al. (2022). Character: coppus-specific vocabulary learnt from each dataset, with fallback to UNK for characters unseen in training. eByte/eChar: Our end-to-end tokenized models which begin with the above Byte/Character vocabularies, but are com- pressed into CLS representations described in Section 2.
Russian
7.5
1.18
6.39
Numeracy 6.6
1.35
5.09
Table 2: Statistics for our language modeling datasets.
See Section for more details.
in different font sizes to choose between light and bulky books. We control for this information parity by fixing the number of characters in the available context 192 for each tokenizer and each dataset. Subword models will then be allowed to access 192//N subwords where N is the average number of characters per subword.
There can be countless ways to make for a 'fair" comparison across tokenizers. We train all models on all datasets for the same number of total epochs. We also focus on letting the models access the same context window size, i.e. amount of information available predict the next of tokens. Different tokenizers can use vastly different memory sizes to the same amount of information. This is analogous to how the same book can be published
3.2 Datasets
Our proposed method requires access to word boundary signal, which can either be obtained from clean natural language corpus, or by running a preprocessing pipeline on an unclean corpus to

---

Tokenizer
Acc (%)
Mem (GBs)
Params (Mil.)
Acc (%)
Mem (BBs)
Params (Mil.)
Acc (%)
Mem
Params
(GBs)
(Mil.)
Language
English
French
Russian
Subword
14.37
0.55
76.8
41.20
1.50
76.8
8.31
1.49
76.8
Byte
13.69
0.53
25.7
17.39
0.54
25.7
12.76
0.53
25.7
Char
13.68
0.54
26.3
16.95
0.53
25.7
10.01
0.54
26.1
eByte cChar
44.17
3.84
38.7
46.44
6.01
38.7
35.00
4.92
38.7
42.94
2.94
39.2
47.06
3.59
38.7
37.15
3.95
39.0
Table 3: Word Prediction Accuracies (Ace %) for different languages and tokenizers. See Section 4.1 for details.
ter nonlinguistic tokens such as URLs meta- data. We chose the former avoid confounding our results with a layer of preprocessing decisions. Therefore, our datasets are smaller but cleaner than the large-scale mC4 and OSCAR datasets typically used for training large language models.
fix the number of context be exactly 192 char- acters and report the accuracy of predicting the next word (over held-out validation data from the same corpus as the training sct). When estimating num- bers, we report magnitude- typically reported in the literature (Thawani et al.. 2021a: Berg- Kirkpatrick and Spokoyny, 2020): the order-of- -aaei-tu--m Exponent Accuracy (EAcc: whether the number of digits are the same in the ground truth and predicted number) and Median Absolute Percentage Error (MdAPE: median of 100|æ - u1/y where is the prediction and y is the ground truth number).
metrics that are
Our choice of languages depended on the avail- ability of large enough corpus of clean data. We also deliberately avoid Chinese and Japanese pora since segmenting them into words would re- quire an additional, possibly confounding step of segmentation through an off-the-shelf model.
Concretely, here are the four datasets we pre- train and evaluate our language models on:
3.4 Implementation
1. English: We randomly sample 10.000 paragraphs from the comprehensions of SQuAD2.0 (Rajpurkar et al.. 2016) dataset.
Every model (using different tokenizer) is pre- trained from scratch on every dataset described above. We report the aforementioned metrics on the individual test set from each corpus. Our base language model is decoder-only transformer called minGPT with layers. For our end-to-end models, the main language model (Step 2) remains the same with 8 layers like the others, whereas the word-encoded (Step 1) and word-decoder (Step 3) both shallow transformers (encoder and decoder respectively) with layers each. They use padding tokens to make each word of equal length for ease of training. We use trained absolute positional em- beddings for all models, and the end-to-end models use it thrice one for each step. We pretrain all models on all datasets from scratch for 100 epochs.
2. French: We randomly sample 10.000 paragraphs from the comprehensions of SQuAD_FR (Cattan et al., 2021) dataset.
3. Russian: We randomly sample 10,000 para- graphs from the reading passages of the SberQuAD (Efimov al., 2020) dataset.
4. Numeracy: We sample 60,000 rows of number-annotated sentences from Wiki- Convert (Thawani et al., 2021a), itself derived from the English Wikipedia. The task to estimate these numbers approximately using the preceding words as context.
We set the learning rate 0.0001, batch size to 2, and block size 192. We used Adam W our optimizer and trained our models on NVIDIA A -CI--CCI- GPUs. With this configuration. training each model variant for 100 epochs took an average of 52 hours.
Table 2 presents statistics for the datasets that we use. The average dataset consists of 7.4M charac- ters (676 unique) and 1.4M words (102k unique).
3.3 Metries
Since the models have different vocabularies, we can not compare their perplexity scores. Instead,
Phttps://gtt wttphh/.www..aaaaay...aaaaa

---

Lang Tok
CLS
4 CLS
A%
A Mem
Tokenizer
Rare
Frequent
en eByte
31
44
42%
0.02
Subword
0.11
7.20
eChar
29
43
48%
0.02
Byte
0.00
4.36
Char
0.28
9.84
fr
eByte eChar
31
46
48%
0.02
fr
34
47
38%
0.02
eByte 5.90
42.90
eChar 6.78
44.17
ru eByte
26
35
35%
0.02
ru
eChar
29
37
28%
0.02
Table 5: Case study: Word Prediction Accuracies for Russian across tokenizers, stratified by Rare and Fre- quent words. See Section 4.3 for details.
Table 4: Word Prediction Accuracies for different rep- resentative power (number of prefix CLS tokens) per word in our end-to-end byte/char-tokenized (Tok) mod- els. Up to 45% higher prediction scores are available for marginal increase in memory (Mem in GBs) of about 20 MBs. See Section 4.2 for details.
Tokenizer
% Num t
EAcc t
MdAPE +
Subword
20.0
44.8
95.72
Byte
39.9
40.5
99.00
Char
42.8
46.6
92.5
4
Results
eByte
47.5
49.9
88.37
4.1
Main Results
eChar
46.7
45.6
90.0
Our main results are summarized in Table 3. Next word prediction accuracies over different datasets show that given a fixed context window, our end- to-end tokenized language models perform much better (up to 300% from 14% to 44% on English) all datasets than both the default BPE subwords well the tokenizer-free character and byte models. This does come at doubling of GPU memory requirements, due to the additional word- level modules our architecture.
Table 6: Number Estimation results on Numeracy dataset across tokenizers. % Num the percentage of times the model predicts a number, over which the next two metrics are calculated. EAce = Exponent Ac- curacy. MdAPE Median Absolute Percentage Error. See Section 4.4 for details.
empirical performance gains achieved by subword models. Hence, we report in Table 5 the word pre- diction accuracies for rare words (those seen less than 10 times in the training dataset) as well as frequent ones (those seen more than 45 times). We find our end-to-end models outperform by factor of 5-7 on frequent words and over 30 times rare words!
4.2 Representation Power
Here we ablate the representative power available for word-pooling of character- or byte-level embed- dings. This hyperparameter is controlled simply by adding different (yet fixed) number of prefix CLS tokens per word before encoding via transformer. Table 4 shows the word prediction accuracies and relative jumps when the number of prefix CLS to- kens per word is increased from to We notice a huge jump for every model, with the trade-off in sequence description length. Note, however, that the memory usage does not jump by more than 20 MBs. Similarly, the number of parameters also in- creases (not shown in table) by only 300K (0.7%) for both eByte and eChar models.
4.4 Number Estimation
We further evaluate a representative subset of to- kenizers on WikiConvert number estimation task. Table again reports that the ability of end-to-end- tokenized eByte/eChar is far better than both sub- word as well as Byte/Char models.
Efficiency Analysis
4.3 Predicting Rare Words
One of the primary motivations for subword tok- enization is their ability to compositionally create rarer words using other frequently occurring sub- words. Wolleb et al. (2023) recently show that such compositionality is significant contribution to the
Here, determine the theoretical training and inferencee/eeneration speed-up accessible by com- pressing words using our end-to-end tokenizer as opposed to tokenizer- free methods, while also com- paring against the more efficient subword models.

---

Dataset
En
Fr
Ru
Ru (rare)
Ru (freg)
Numeracy [3]
Metric
Next Word Prediction Accuracy
EAcct
MdAPE.
Subword
14.37
41.20
8.31
0.11
7.20
44.8
95.7
Byte
13.69
17.39
12.76
0.00
4.36
40.5
99.0
Char
13.68
16.95
10.01
0.28
9.84
46.6
92.5
eByte 44.17
46.44
35.00
5.90
42.90
49.9
88.4
eChar
42.94
47.06
37.15
6.78
44.17
45.6
90.0
5.1 Training Speed-up
(this our default setup in this paper). Let B" be the optimal batch size that we wish to calculate and be the average number of characters per word (roughly 5.5 for English). Note that we retain T characters as our context window, therefore the average number of words per batch sequence will be T/c. The memory footprint of activations would then be (LDB"Te)/ for the word encoder (and same for word decoder) and (LDB"T")/C for the primary (word level) language model.
Assume M total memory budget available (say. in GBs) and context window of T characters per batch. Also assume the tokenizer-free model (henceforth referred to base/baseline) is de- coder transformer with L layers and D dimensions. The memory footprint would most significantly de- pend on the number of activations stored which can be estimated as M LDBT2 where the batch size. Given a fixed memory budget of M and required context size of T characters, we can find our optimal batch size as:
This leads to the optimal batch size:
M B" LDT(c/2 + T/2)
M
B LDT2
and the number training steps to be:
Assuming the training corpus comprises of N characters, the number of training iterations re- quired is:
X" N
NDL (c/2+T/c")
B"T
M
Finally, we estimate our proposed speedup total training time as
N NDLT BT
X
=
M
T
X/X" c/2 T/c2
Next, for subwords, a similar batch size can be estimated as:
Plugging in c = 5.5 as a conservative number characters per word and T 192 context window length, we get a 6.8x speed-up in training steps, which is only marginally less than the subword speed-up (7.8x) relative to character level langauge model.
M
B'
LDT2/8
where s is the number of characters per subword (roughly 2.8 for our three languages). Substituting to find the number of training steps:
5.2 Generation Speed-up
NDLT BT 2
X'=
Another unique advantage of our end-to-end tok- enized model is in generation, which is also par- allelized per word. A character/byte model must generate one token a time, then feed the predicted token back into the input and run the forward loop again for autoregressively generating the next. As- suming the L layers of a decoder take seconds for the forward iteration of GPT-like decoder, the generation speed for such a character based model will be 1/t characters per second.
The training speed-up of subword model is there- fore estimated to be X/X' 52 7.8x.
Finally, we calculate the analogous number of training steps required for one epoch of our end- to-end character-tokenizee model. We assume L/4 word-encoder layers, primary LM (word level) layers, and L/4 word-decoder layers for simplicity
'The other components such as parameter variables and hackpropagated gradients can be approximated away.
Real values: En 5.5, Fr 5.2, Ru 6.4.

---

Method
Citation
Compress?
Generate?
Learnt?
Word level?
GPT
Radford et al. (2018)
Lookup
Yes
No
Yes
ByT3
Xue et al. (2022)
None
Yes
No
No
MANTa
Godey et al. (2022)
Segment
Yes
Yes
No
RetVec
Bursztein et al. (2023)
Conv.
No
Yes
Yes
FastText
Bojanowski et al. (2017)
Conv.
No
Yes
Yes
ELMo
Peters et al. (2018)
Conv.
No
Yes
Yes
CharBERT
El Boukkouri et (2020)
Conv.
No
Yes
Yes
CharFormer
Tay et al. (2021)
Conv.
No
Yes
No
LOBEF-nCF
Sreedhar et al. (2022)
None
Yes
Yes
No
LOBEF-WSF
Sreedhar et al. (2022)
None
Yes
Yes
Yes
CANINE
Clark et al. (2022)
Conv.
Yes
Yes
No
MegaByte
Yu et al. (2023)
Dense
Yes
Yes
No
Ours
Attn.
Yes
Yes
Yes
Literature Review existing tokenization methods along several dimensions. Compress? Is the input string chunked into bigger units? Generate? Whether not the model can generate new unseen tokens? Learnt? Is the tokenization learnt end-to-end with other parameters? Word Boundary? Is the word boundary considered or treated as just another token? Conv: Convolution. Attn: Attention.
Table 7:
Subword models benefit from having tokens that are longer (roughly 2.8 characters/subwordd for the three languages consider), therefore they can generate speed of 2.8/t characters per second.
Character/Byte-level ByT5 (Xue et al., 2022), CANINE (Clark et al., 2022), and SubChar (Si et al., 2021) propose using very small fixed-length units such as characters, bytes, glyph strokes in- stead of dynamic-length subwords or words. This often comes the expense of larger sequence lengths and more compute requirements, especially for a transformer architecture which typically has a complexity of (n²)) in number of input tokens.
With very coarse assumption, our end-to-end character model with 1.4 word-encoder layers and decoder layers (ignore the L/1 word-decoder layers for now) will require 5t/4 seconds to gener- ate the representation of one word at time. The next step can then be parallelized (with trade-off in memory consumption) to both autoregressively go generating the next word representation in other 51/1 seconds, well autoregressively gen- erating one character time using this predicted word representation. This word-level decoder that emits characters currently has L/4 layers so a crude assumption would mean 1/1 seconds per character. Therefore, at steady state, the word-decoder will take 5.5t/4 seconds to generate the average 5.5 characters per word, while the next word will be ready for decoding simultaneously in just 5t/4 sec- onds. Thus, the generation speed is 4/t characters per second, i.e. roughly 50% better than subwords and four times as fast as tokenizer-fin models.
Beyond word level CodeBPE (Chirkova and Troshin, 2022) and Multi Word Expressions (Ku- mar and Thawani, 2022; Zaninello and Birch, 2020; Rikters and Bojar, 2017) show promise in yet larger tokens that cross word boundaries, e.g., vocab- ulary with single tokens for the strings "for range" or "New York City" respectively..
Visual segmentation Yet another line of work (Rust al., 2022: Salesky et al., 2021) renders text as images before feeding them CNNs, do- ing away with tokenization altogether and showing gains robustness to spelling or printing errors.
Learnt subword segmentation Finally, some methods (Mofijul Islam al., 2022: Kaushal and Mahowald, 2022; Pinter al., 2021; Tay al., 2021; Provilkov et al., 2020: Wang al., 2021) parameterize the process tokenization by pooling character n n-grams or randomly choosing one of the many ways to segment a given word.
6 Related Work
Some recent work has challenged the subword tok- enization schemes. Table highlights the different kinds of tokenizations existing in prior work and positions our work uniquely among them.

---

A recent preprint on machine translation by Sreedhar al. (2022) proposes method called WSF, perhaps closest to ours, except that they only use the word boundary fusion at encoder stage. Our independent analysis focuses on language model- ing instead and also generates text in parallel using end-to-end attention based tokenization.
boundary signal.
To summarize, we concede that our proposed end-to-end models are not yet ready for adoption at scale with large language models trained on raw internet data. However, we expect our analyses to encourage insightful conversation in the com- munity about (1) the spectrum between subwords and character/byte models, as well as on (2) the role of word boundary a meaningful signal in tokenization and language modeling.
7 Conclusion
Subword tokenization is efficient but too rigid and deterministic. Character/Byte-ler models on the other hand are too expressive, which leads inef- ficient training and inference. We propose word- boundary-informe tokenizer that efficiently and robustly performs language modeling in a hicrar- chical, end-to-end, learned model. We show that it outperforms by over 300% both extremes: sub- words and character/byte models. We also analyze its trade-offss in training and inference efficiency. Despite its many flaws including reliance boundary signal and moderate efficiency as well as moderate expressiveness, we expect this prelimi- nary study to pose an interesting trade-off tokeniza- tion for truly end-to-end language modeling. Our code is released on Github.
Acknowledgements
This work was funded by the Defense Ad- vanced Research Projects Agency with award N66001 1924033. The authors acknowledge the Center for Advanced Research Computing (CARC) at the University of Southern California for provid- ing computing resources that have contributed the research results reported within this publication. We also thank Google for their generous support. We appreciate the anonymous reviewers at EMNLP 2023 for helping us refine this paper.
word
References
Taylor Berg-Kirkpatrick and Daniel Spokoyny. 2020. An empirical investigation of contextualized number prediction. In Proceddings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4754 4764, Online, Association for Computational Linguistics.
8 Limitations
We repeatedly highlight our work's limitations throughout the paper: our proposed end-to-end to- kenization is neither faster than subwords nor as expressive as character/hyte-leww models. Instead, we propose reasonable trade-off between the two extremes.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hy- land, Maria Wetscherek. Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. 2022. Making the most of text semantics to improve biomedical lision-language processing. In Computer Vision- ECCV 2022: 17th Eurppan Conference, Tel Aviy, Is- rael, October 23-27, 2022, Proceedings, Part XXXVI, pages 1-21. Springer.
We also do not cast a wide net on many down- stream tasks like machine translation or question answering, hence lack a comparison with other models like CharFormer, CANINE, and Local Byte Fusion. We instead focus solely on the intrinsic metrics of language modeling, across multiple lan- guages as well as on number estimation.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the associa- tion for computational linguistics, 5:135-146.
Our choice of languages limited by availability of high quality 'natural language' corpora, unlike the internet-scale language modeling data which we observed are filled with examples of long strings like URLs and unsegmented strings. We do not preprocessing pipelines (except simple punc- tuation cleanup) avoid confounding results with the choice of such heuristics. This unfortunately prevents us from experimenting with other high resource languages like Chinese and Japanese, cor- pora of which do not implicitly have a whitespace
Elie Bursztein, Marina Zhang, Owen Vallis, Xinyu Jia, and Alexey Kurakin. 2023. Retvee: Resilient and efficient text vectorizer.
Oralic Cattan, Christophe Servan, and Sophie Rosset. 2021. On the Usability of Transformers-bass mod- els for a French Question-Answeriing task. In Recent Advances in Natural Language Processing (RANLP), Varna, Bulgaria.
Nadezhda Chirkova and Sergey Troshin. 2022. Codebpe Investigating subtokenization options for

---

large language model pretraining on source code. In Deep Learning for Code Workshop.
Md Mofijul Islam, Gustavo Aguilar, Pragaash Pon- nusamy, Clint Solomon Mathialagan, Chengyuan Ma. and Chenlei Guo. 2022. A vocabulary-free multilin- gual neural tokenizer for end-to-end task learning. In Proceedings of the 7th Workshop on Representa- tion Learning for NI.P, pages 91-99, Dublin, Ireland. Association for Computational Linguisties.
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training efficient tokenization-free encoder for language representa- tion. Transactions of the Association for Computa- tional Linguisties, 10:73-91.
OpenAL. 2023. Gpt-4 technical report.
Pavel Efimov, Andrey Chertok. Leonid Boytsov, and Pavel Braslavski. 2020. Sberquad russian reading comprehension dataset: Description and analysis. In Experimental IR Meets Multilinguality, Multimodal- ity, and Interaction, pages 3-15, Springer Interna- tional Publishing.
Matthew E Peters, Mark Neumann, Mohit Iyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of NAACL-HLT, pages 2227-2237.
Yuval Pinter, Amanda Stent, Mark Dredze, and Jacob Eisenstein. 2021. Learning to look inside: Augment- ing token-based encoders with character-level infor- mation.
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun ichi Tsu- jii 2020. CharacterBERRT Reconciling ELMo and BERT for word-level opea-vocabulary representa- tions from characters. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 6903-6915, Barcelona, Spain (Online). International Committee Computational Linguis- tics.
Ivan Provilkov. Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1882-1892, Online, Association for Computational Linguistics.
Nathan Godey, Roman Castagné, Éric de la Clergerie, and Benoît Sagot. 2022. MANTa: Efficient gradient- based tokenization for end-to-end robust language modeling. Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 2859- 2870, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page arXiv:1606.05250.
Ayush Kaushal and Kyle Mahowald. 2022. What do tokens know about their characters and how do they know it? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2487-2507, Seattle, United States. Association for Computational Linguistics.
Matiss Rikters and Ondřej Bojar. 2017. Paying attention to multi-word expressions in neural machine transla- tion. In Proceedings of Machine Translation Summit XVI: Research Track, pages 86 95, Nagoya Japan.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Eliz- abeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2022. Language modelling with pixels.
Taku Kudo. 2018 Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Elizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7235-7252, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Dipesh Kumar and Avijit Thawani. 2022. BPE beyond word boundary: How NOT to use multi word expres- sions in neural machine translation. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 172-179, Dublin, Ireland. As- sociation for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1723
Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoit Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nip. arXiv preprini
Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2021. Shuowen-jiezi: Linguistically informed to- kenizers for chinese language model pretraining. arXiv preprint 20X.v:2106.00400.

---

Makesh Narsimhan Sreedhar, Xiangpeng Wan. Yu-Jie Cheng, and Junjie Hu. 2022. Local byte fusion for neural machine translation. ArXiv, abs/2205. 1490.
2023. Megabyte: Predicting million- dillion-byte se- quences with multiscale transformers. ar preprint arXiv:2305.07185
Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient- based subword tokenization. International Con- ference on Learning Representations.
Andrea Zaninello and Alexandra Birch. 2020. Multi- word expression aware neural machine translation. Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 38 6-3255,, Marseille, France. European Language Resources Association.
Avijit Thawani, Jay Pujara, and Filip Ilievski. 2021a. Numeracy enhances the literacy of language models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6960-6967, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021b. Representing numbers in NLP: survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-656, Online. As- sociation for Computational Linguistics.
Erie Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know num- bers? probing numeracy in embeddings. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP--JCNIPP) pages 5307-5315, Hong Kong, China. Association for Computational Linguis- tics.
Xinyi Wang. Sebastian Ruder, and Graham Neubig. 2021. Multi-view subword regularization. In Pro- ceedings of the 202/ Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473-482, Online. Association for Computational Lin- guistics.
Benoist Wolleb, Romain Silvestri, Giorgos Vernikos, Ljiljana Dolamic, and Andrei Popescu-Belis 2023. Assessing the importance of frequency versus com- positionality for subword-bus based tokenization nmt.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's ' neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv: 1609.08144.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards foken-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics, 10:291-306.
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis.


Reviews:
[{"review_id": "SrZFXWZZ2p", "reviewer": "Reviewer_Exn2", "summary": "", "questions": "As I'm understanding Section 3.4, when you use expressions such as \"Our base language model is\", it makes me think that your are using a different architecture for your language model, instead of using the same one replacing only the tokenization strategy. Is that so?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments! We appreciate your feedback that Section 3.4 should be reworded for more clarity. In the meanwhile, we want to emphasize that **evaluation was indeed done under the same conditions for all tokenizers**. Our reference to “the base language model” refers to the underlying transformer decoder architecture, which remains the same for all strategies (all different tokenizers).\n\nWe understand the concern around scalability and generalizability of our results. Despite our findings being consistently across multiple languages and multiple metrics, we note that it is natural for a new tokenization method to undergo years of empirical experimentation across tasks, models, and languages before wide adoption, e.g., the now-ubiquitous subwords were once introduced in papers working exclusively on machine translation on a few language pairs. This paper is, therefore, a preliminary work to very specifically study the difference in language modeling capabilities across several tokenization strategies.\n\nWe hope we have clarified the concerns around the soundness of our work by reiterating the fair experimental setup across experiments. Regarding excitement around the scalability and generalizability of our results, we request that you consider the **resources required to compare tokenization in language models by pretraining each variant from scratch**. Given a modest computation budget, we have tried our best to contribute insightful results across tokenizers for the community to learn from, and scale up our experiments across models and languages in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novel strategy that could benefit the language model community.", "reasons_to_reject": "* We are uncertain of the scalability to broader data and languages.", "questions_to_authors": "As I'm understanding Section 3.4, when you use expressions such as \"Our base language model is\", it makes me think that your are using a different architecture for your language model, instead of using the same one replacing only the tokenization strategy. Is that so?", "paper_topic_and_main_contributions": "This work proposes a new tokenization scheme for language modeling. This scheme uses a Transformer encoder to pool in the base tokens (either characters or bytes) into a fixed number of embeddings per word. Those embeddings are, then, passed onto the main language model (in this case, a vanilla GPT-like Transformer decoder) and, after that, fed trough a Transformer decoder to autoregressively decode the next word.\n\nThe proposal is evaluated in comparison to other well-known strategies (subword-level, byte-level and character-level) over language modeling, showcasing significant improvements in terms of accuracy (with an increase of memory consumption as a trade-off). However, I'm not convinced that the evaluation has been done under the same conditions for all strategies (e.g., Section 3.1 mentions that the subword strategy is the BPE vocabulary from GPT-2 and GPT-3) nor I am sure that the language model implementation is the same for all strategies (though this may just be a problem with the description in Section 3.4).\n\nAs mentioned at the limitations section, both the scalability to broader data and the use of other languages are a main concern.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 086 and 087: (Tay et al., 2021; Yu et al., 2023; Clark et al., 2022) -> (Tay et al., 2021; Clark et al., 2022; Yu et al., 2023)\nl. 227: byte level -> byte-level\nl. 228: Xue et al. (2022) -> (Xue et al., 2022)\nl. 250: \"192//N\" is not standard notation\nl. 312 and 319: use em-dash instead of dash\nTable 6: dastaset -> dataset\nl. 530: using -> use"}}, {"review_id": "3sPutUpQk2", "reviewer": "Reviewer_kEqs", "summary": "", "questions": "some corrections:\n\nWhat is word accuracy? please explain your metric. Also is it used in other papers? if yes please add citations.\n\nYour improvement is not exactly 300% so either say it is 314% or just that is shows a 3 fold increase.\n\nline :whereas some African scripts may require four bytes to represent a single character [citation  missing, or at least an example of such language]\n\nline 529: We do not using\n\nline 298: we report magnitude-based metrics that are typically reported in the literature [missing reference]. Also what is EAcc and MdAPE ? please explain briefly", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and interesting ideas for follow-up experiments!\n\nAs you mention, it will be interesting to compare our results with MegaByte’s fixed-length compression, which we are already working on. We would nevertheless stress that it was unrealistic to include such results in the current submission, given that **[MegaByte](https://arxiv.org/abs/2305.07185) is an unpublished preprint submitted to arxiv less than 5 weeks before the EMNLP deadline**. We included MegaByte in our Related Work section merely to taxonomize the literature review and to highlight the vast interest shown by the NLP community in the search for alternative end-to-end tokenization strategies.\n\nWe will be happy to further describe our metrics like word prediction and numeracy metrics described in L300 and Table 6: “EAcc = Exponent Accuracy. MdAPE = Median Absolute Percentage Error.” Word prediction evaluates whether the next characters/bytes/subwords predicted include the exact next ground truth word. Exponent Accuracy is an order-of-magnitude accuracy (whether the number of digits are the same in the ground truth and predicted number). Median Absolute Percentage Error is formulated as the median of 100|x-y|/y where x is the prediction and y is the ground truth number. We will cite relevant prior work for these metrics: \n- Spokoyny, Daniel, and Taylor Berg-Kirkpatrick. \"[An empirical investigation of contextualized number prediction](https://aclanthology.org/2020.emnlp-main.385/).\" \n- Thawani, Avijit, Jay Pujara, and Filip Ilievski. \"[Numeracy enhances the literacy of language models](https://aclanthology.org/2021.emnlp-main.557/).\" \n\nWe will also make sure to correct other minor typos and paraphrasing errors that you point out.\nWe will also make sure to correct other minor typos and paraphrasing errors that you point out. Bamum is an example of an African script that suffers from using many bytes merely because it is placed far in the [Basic Multilingual Plane](https://unicode.org/roadmaps/bmp/). Other disadvantaged scripts are Meetei (India) and Cherokee (North America).\n\nFinally, we would like to challenge the assertion that our proposed end-to-end models are unfairly trained on word prediction. All model variants were trained for the same number of epochs and the number of epochs was determined by looking at validation loss curves (which we could add to the camera ready appendix for justification). The training proceeds with negative log likelihood loss over the base units: characters for eChar and Char; bytes for eByte and Byte; and subwords for subword model. There is no added signal that unfairly helps our proposed end-to-end tokenized models excel at the task of word prediction.\n\nWe acknowledge (as in our Limitations section) that this paper does not convincingly conclude which tokenizer must be used for training language models in general. Our datasets are too simple and small, and we do not cast a wide net on downstream tasks. Nevertheless, our hope is that **our systematic analysis of tokenizers will benefit the community**, given how difficult it is to compare such variants due to the need for pretraining language models from scratch. \n\nWe hope we have sufficiently answered your queries about the soundness of our approach. Regarding the excitement/novelty of our approach, we *kindly request that you compare our work against only non-contemporaneous work* (published over three months before the EMNLP deadline).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clearly written and simple to understand.", "reasons_to_reject": "The model used is typically a megabyte with dynamic size tokens that respect the word boundaries. Even though the idea is interesting, the paper is not using it in a convincing manner. Your model is trained to decode the real words for many epochs (100) on a very small dataset. Such a model is expected to have learned a good embedding for all words in the corpus, even for rare words. It is not surprising that such a model will outperform subword and character based model that have not been trained to decode exactly the complete words. As this paper does not prove that the method is of any use in real word scenario (machine translation, sentence generation,...), it is difficult to understand the take away from this paper.\n\nI think this paper is too weak to be accepted at EMNLP. Yet, an idea that would be interesting to pursue would be to compare your model to Megabytes that always pool 4 characters without taking into account the word boundaries. If you manage to show that dynamic size tokens are better than fixed-size ones, you would have an interesting results to publish.", "questions_to_authors": "some corrections:\n\nWhat is word accuracy? please explain your metric. Also is it used in other papers? if yes please add citations.\n\nYour improvement is not exactly 300% so either say it is 314% or just that is shows a 3 fold increase.\n\nline :whereas some African scripts may require four bytes to represent a single character [citation  missing, or at least an example of such language]\n\nline 529: We do not using\n\nline 298: we report magnitude-based metrics that are typically reported in the literature [missing reference]. Also what is EAcc and MdAPE ? please explain briefly", "paper_topic_and_main_contributions": "This paper uses very small text dataset and make a statement on what kind of tokenizer that is more suited for language modelling. The authors propose to pool word representations into the LM and decoding them at the character/byte level. The method is said to improve word accuracy across dataset and works particularly well on rare words.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hjsr3p1Jfx", "reviewer": "Reviewer_nfbm", "summary": "", "questions": "n/a", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoroughly review our paper and provide insightful comments! \n\nWe specifically appreciate your consideration of limited resources when evaluating the limited scale of our experiments and instead judging our work on the soundness and novelty of the research questions considered.\n\nHere are the eChar and Char results as requested for Table 6:\n- eChar: 46.7 (% Num), 45.6 (EAcc), 90.0 (MdAPE). \n- Char: 42.8 (% Num), 46.6 (EAcc), 92.5 (MdAPE). \n\nWe will be happy to include them in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses a very relevant and timely subject, since tokenization is the first step of any NLP task and in particular the first step before applying the now ubiquitous LLMs. The paper makes a good point that LLMs learn tasks almost completely end-to-end, except for tokenization, and gives good explanations of the problems this causes.\n\nThe paper is clearly written, easy to follow, and the experimental methodology is sound.\n\nThe results are very good. The proposed method strikes a balance between efficiency and expressivity, while being very good (much better than the subword and char/byte-based alternatives) at next word prediction.\n\nExperiments are rather wide in terms of covering various languages, and doing specific experiments about arithmetic, a weak point of other tokenization schemes.\n\nThe paper is very sincere and clear about the limitations of the approach.", "reasons_to_reject": "Experiments are all done on a single autoregressive language model, which is relatively small, and there is no guarantee that the results will generalize to other language models or that they will scale. I don't consider this to be a real reason to reject as not everyone has the resources to try other/larger models.\n", "questions_to_authors": "n/a", "paper_topic_and_main_contributions": "This paper presents a new tokenization technique for language modeling, which is an alternative to both subword tokenization and raw character or byte-based tokenization. The proposed method learns tokens by using word boundaries to pool characters or bytes into a word representation (in particular, the character or byte representations of each word are compresed into a constant number n of CLS tokens, the paper tries n=1 and n=4). This scheme has better efficiency than byte/char tokenization, albeit worse than subword tokenization (because attention is computed between bytes within the same word, but not those in different words), and is also a middle ground in terms of expressivity. The most salient result is language modeling accuracy, with the proposed scheme outperforming both subword and byte/char-based tokenization by a large margin, and even more so on rare words (by a factor of 30 in this case).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None found.", "Typos_Grammar_Style_and_Presentation_Improvements": "Why is eChar missing from Table 6?\n\nLine 24: outperform -> outperforms\n\nLine 256: pipleine -> pipeline", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose a new tokenization method for language modeling alternative to other sub-word tokenization or byte/char-level methods. Reviewers all acknowledged the merits of the proposed approach but also highlighted the need for a more controlled setup when it comes to the granularity of the learning signal and how it compares with the other methods, since this could be a confounder needs studying and further explanations.
Decision: Accept-Findings