Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models
Authors: Pit Neitemeier
Björn Deiseroth
Constantin Eichenberg
Lukas Balles
Venue: ICLR 2025 Poster (2025)
Keywords: transformer
autoregressive
generative
language modelling
tokenizer-free
byte-level
hierarchical
Primary area: foundation or frontier models, including LLMs

Abstract: Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the de-facto standard, they present challenges such as large vocabularies, limited adaptability to new domains or languages, and sensitivity to spelling errors and variations. To overcome these limitations, we investigate a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. It employs a lightweight character-level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters via a compact character-level decoder. This method retains the sequence compression benefits of word-level tokenization without relying on a rigid, predefined vocabulary. We demonstrate, at scales up to 7 billion parameters, that hierarchical transformers match the downstream task performance of subword-tokenizer-based models while exhibiting significantly greater robustness to input perturbations. Additionally, during continued pretraining on an out-of-domain language, our model trains almost twice as fast, achieves superior performance on the target language, and retains more of its previously learned knowledge. Hierarchical transformers pave the way for NLP systems that are more robust, flexible, and generalizable across languages and domains.

Published as a conference paper at ICLR 2025
HIERARCHICAL AUTOREGRESSIVE TRANSFORMERS: COMBINING BYTE- AND WORD-LEVEL PROCESSING FOR ROBUST, ADAPTABLE LANGUAGE MODELS
Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg & Lukas Balles
Aleph Alpha Research Heidelberg, Germany <f <firstname> <astttnmm--eleeee-aaappp------
ABSTRACT
Tokenization is fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tok- enizers have become the de-facto standard, they present challenges such as large vocabularies, limited adaptability to new domains or languages, and sensitivity spelling errors and variations. To overcome these limitations, we investigate a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. It employs a lightweight character- level encoder to convert character sequences into word embeddings, which are then processed by word-level backbone model and decoded back into characters via a compact character-level decoder. This method retains the sequence com- pression benefits of word-level tokenization without relying on a rigid, predefined vocabulary. We demonstrate, at scales up to 7 billion parameters, that hierarchi- cal transformers match the downstream task performance based models while exhibiting significantly greater robustness to input perturba- tions. Additionally, during continued pretraining on an out-of-domain language, model trains almost twice fast, achieves superior performance the get language, and retains more of its previously learned knowledge. Hierarchical transformers pave the way for NLP systems that are more robust, flexible, and generalizable across languages and domains.
subword-tokeniz
INTRODUCTION
Tokenization plays a fundamental role in natural language processing (NLP) as it breaks down text into units that computational models can process. Two fundamental approaches character-level and word-level tokenization. While character-level tokenization uses the "atomic" units of text and enjoys small vocabulary size, it leads to long sequences with high computational and memory cost. Conversely, word-level tokenization leads to short sequences but suffers from extremely large vocabulary sizes and the inability process out-of-vocabulary words.
Subword tokenization has emerged as compromise between these two extremes and has become the standard. Common subword tokenizers are trained--separately from the model-on a reference corpus of text. For example, Byte Pair Encoding (BPE; Gage 1994 Sennrich et al.) 2016) builds vocabulary starting from individual bytes and iteratively merging adjacent pairs tokens that oceur most frequently in the corpus until the desired vocabulary size is reached. The resulting subword vocabulary leads to good sequence length compression on the reference corpus, while maintaining the ability handle out-of-vocabulal words using a byte fallback.
However, subword tokenizers come with several downsides. First, contemporary models routinely use vocabulary sizes in the hundreds of thousands, making the corresponding embedding matrices and output heads extremely large. For instance, for the 8B model of the Llama-3 family (Dubey et al. 2024), with vocabulary size of 128k, embedding and head account for roughly 13% of the model's 's parameter footprint. Secondly, the tokenizer is fitted a separate step and not included in the end-to-end learning process of the model. This becomes problematic when pretrained model

---

Published as a conference paper at ICLR 2025
Inference
Training
Input Text
Character Inputs
WI
Recumson
Encoder
Encoder
Encoder
Encoder
Encoder
Word Embeddings
Backbone
Backbone
Word Predictions
Decoder
Decoder
Decoder
Character Logits
Charactee Recuraion
Targets
Figure 1: Schematic of the proposed hierarchical architecture. The input text is first split into words, with each word prepended by a special token [W]. These words are passed through the encoder and the activation the position of the [W] token is selected as the word embedding e'. The sequence of word embeddings ef is then processed by the backbone to produce abstract predictive word embeddings p'. The decoder then maps p' to probabilites for the characters of the next word. During inference, given text with partial word, complete words are processed by the encoder and backbone. The decoder then recursively completes the remaining characters of the incomplete word and the completed word enters the encoder in word recursion.
applied to, or finetuned on text from different domains or languages to which the tokenizer not attuned (see, e.g., Ali et al. 2024 Petrov et al. 2023 Deiseroth et al.] (0224.. Finally, spelling
mistakes or variations can lead to drastically different token sequences for semantically close inputs and thereby degrade model performance.
To address these shortcomings, we investigate a hierarchical architecture as shown in Figure I which combines character-level and word-level processing. We first split the text into words. The characters of each word are processed by a small character-level encoder module, which maps them to word embedding. The resulting sequence of word embeddings is then processed by a larger backbone model. The outputs of the backbone are treated as abstract "predictive" word embeddings and are decoded back to characters by another small character-level decoder module. As will demonstrate, the character-lewe modules can be kept very small, having fewer parameters than the token-level embedding and head they replace. Encoder, backbone and decoder are transformer mod- els and the entire system can be trained end-to-end, without the need for a fixed, trained tokenizer.
Our core contributions are:
We revisit, refine and thoroughly investigate hierarchical architecture for autoregressive lan- guage modelling that combines character- and word-level processing. Its design climinates the need for fixed word or subword-level vocabulary, and does not require separate tokenizer train- ing. Through careful computational cost analysis and comprehensive architecture sweeps, identify optimal hierarchical model configurations across various compute budgets.
We conduct extensive experiments, comparing our proposed model against state-of-the-art sub- word tokenizer-basse based models and competing hierarchical approach in compute-matcher exper- iments on identical data. We demonstrate that our approach scales effectively to the 7B scale, consistently matching the performance of the baseline models.
We demonstrate that our proposed model is significantly more robust to perturbations of its input.
Finally, we show that our model enjoys superior finetunability out-of-distributtion data, such as new languages or domains, outperforming tokenizer-bacce architectures which struggle with adaption to new vocabularies. In particular, during continued pretraining unseen language, our model achieves superior performance on the target language, retains more its previously acquired knowledge, while training almost twice as fast.
2

---

Published as a conference paper at ICLR 2025
The remainder of this paper is organised as follows. Section Oectioon2introde introduces our hierarchical architec- ture, including a detailed computational cost analysis. Related work is discussed in Section 3 In Section 4 present experimental setup and results. Section Section5connee concludes the paper with final remarks and potential directions for future research.
HIERARCHICAL AUTOREGRESSIVE TRANSFORMERS
We now introduce our hierarchical architecture, which is refinement and simplification of similar architectures proposed in prior works, see Section 3 Our approach relies on a splitting rule that partitions text into sequences of words. Specifically, consisting of V/8 = 256 distinct v and split the text at Unicode whitespace characters, which are appended the previous word. A text can then be represented (w".... ......,w) with w E [V&]=. being a word of length 4. Importantly, the splitting rule is the only non-trainable processing step in our method. We argue that, for natural text in (alphabetic) languages, whitespace splitting adequate. However, our hierarchical architecture is agnostic to the type of splitting rule, allowing for alternatives that may be more appropriate for different languages or domains.
UTF-8 byteq
base alphahet,
2.1 HIERARCHICAL ARCHITECTURE
Our architecture consists of three main components:
An encoder E: Nxd + Nxd, bidirectional transformer operating the character embeddings within each word.
A backbone B: Nx + RNx,, causal transformer operating word embeddings.
A decoder N + R N a causal transformer with a language modelling head, operating character level and outputting next-character prediction logits.
In addition, we have character embedding C: [Vi] - Ied and two projection matrices We e Dxd and WD Rd D mapping between the (smaller) character-level dimension d and the (larger) word-level dimension D.
A document (w w is processed by the model explained in the following and depicted in Figure Following Devlin al. (2019), each word is prepended with a special token [W] and its characters are embedded via C,
(1) Then, it is passed through the encoder. The output corresponding the [W] token, i.e., the first entry in the sequence dimension, is selected as a word embedding:
x = ((w E Rdd
X(WI = C(W)) Rd.
e' = [[( xi....,xx x1,]] E Rdd
The resulting sequence of word embeddings is projected to the backbone dimension, passed through the backbone and projected back the decoder dimension:
e² = Wee' E ER,, p²
..... ,
D'=WWp' E Ed.
The output for the i-th word, p', is treated predictive word embedding, to be decoded into sequence of characters matchting the next word. To that end, during training, we concatenate p¹ with the character embeddings of the next word and map them through the decoder, resulting in sequence next-character prediction logits:
13 D(p'
E ER
(4)
We train on the loss
1 841+1
Σ
(5)
where L denotes character-level coss-entropy loss and set the final prediction target of each word to w 1 [W], indicating the end of the word.
In the following, we use the terms character and byte interchangeably, but want to highlight that our approach is not specific to byte-level modelling.
2UTF8 has unused byte values, which exploit for our special tokens, keeping the vocab size 256
3

---

Published as a conference paper at ICLR 2025
2.2 INFERENCE
Inference in hierarchical architecture proceeds in nested loop, shown the left side of Figure To generate a new word, we pass the context sequence through encoder and backbone to produce predictive next-word embedding. To materialize the next word in characters, autoregressive loop of the decoder module. When a word is completed, as indicated by the prediction of [W] token, appended to the input and the process is repeated.
2.3 COMPUTATIONAL AND MEMORY COST
We now discuss the computational cost and memory footprint of our proposed architecture, contrast- ing it with baseline model using subword tokenizer and corresponding embedding matrix and language modelling head. We assume the baseline model uses phackbone parameters in its backbone and phend in the backbone and pehur
baseline in its embedding and head. For the hierarchical model, we assume phackhone hicrarchical parameters
hicrarchical in each of the character-level modules.
Computational Cost. We present simplified computational cost analysis under the assumption that the cost of sequence length and is the number of non-embedding parameters. This a standard simplification based the observation that feed-forward FLOPs dominate attention FLOPs for typical settings. An exact comparison, factoring in attention FLOPs, may be found Appendix A.2
pass through a transformer is proportional to SP,, where S is the
Consider document of S characters. The computational cost of the two models depends heavily on the length of the sequence passed through the backbone, i... the number of words Sw and tokens ST contained in the document, respectively. The baseline model processes sequence of length Sr, passing it through the backbone and the output head, incurring total cost of
Chaseline
ST P
phackhone
+ ST phead baseline
(6)
In the hierarchical architecture, the backbone processes a sequence length Sw. Additionally, the two character-level models each process the sequence of length S plus additional Sw word separator tokens for total cost of
Chierarchical = Sw phackhone
hicrarchical + 2(S + Sw) I pccar hicrarchical-
Our experiments will show that small en-/decoder modules are viable. For instance, our 3B- scale experiment, the computational cost of our two character-level modules roughly equals that of the language modelling head of the baseline. Additionally, words represent a coarser unit than subwords; pretraining dataset exhibits ratio of Sw ss 0.698,, see Figure (left). Consequently, for a given cost budget. our hierarchical model will be able to operate with a larger backbone. We will revisit this for our compute-matchen matched experiments in Section
Memory. For compute-matched models, hierarchical architecture will have larger parameter footprint, see Table terms of activation memory, we have to distinguish between training and inference. During training, all activations have to be stored. Our hierarchical model shrinks the number of activations in the backbone by a factor of Sw/Sr and avoids large logit tensors, but operates with slightly larger backbone and needs to store additional activations in the character- level modules. Since the exact size the activation memory depends on the caching strategy of the auto-differentiation framework, forego detailed comparison.
Inference. Using KV caching (Pope et al.) (023), it is possible to achieve near parity of FLOPs during training and inference. When comparing compute-matched models, we argue that hier- archical architecture has a modest advantage in wall-clock time performance, see Appendix A.4 Additionally we propose a scheme to further optimise inference performance using cached word embeddings. The relevant quantity for memory consumption at inference time is the size of the KV cache, which can be bottleneck in high-throughhut inference systems. Here, hierarchical models lead to smaller memory load, detailed in Appendix A.5
4

---

Published as a conference paper at ICLR 2025
OCILM-Bascline
Occiglot-DE
RPW RPT 12 050.5
BPW 7--7.25,
1.8
0.75
BPT p=2 66, o=:0.20
Ganviion
Giussian Fit
Density
Density
1.2
0.50
0.25
0.6
0.00
0.0
2
10 11
14
33
6
10
11
12
13
14
15
Byles
Bytes
Figure 2: Bytes per word (BPW) and bytes per token (BPT) statistics, showing that words are coarser unit than subword tokens. The tokenizer has been fitted to the DCLM-baseline dataset (left). On a dataset to which the tokenizer is not attuned, such as the German Occiglot dataset (right), the tokenizer "fragments" and BPT drops significantly.
3 RELATED WORK
The granularity of computational models for natural language has been a prominent question in NLP research since its inception (see, e.g., review by Mielke et al. [02T). For many years, the question of character- vs word-level modelling has dominated the discussion, until subword tok- enization methods, such Byte-Pair Encoding (BPE, [Sennrich et al., 2016), have emerged middle ground, balancing the flexibility of characters with the semantic coherence of words. Over time, subword tokenizers have become the dominant approach, gaining widespread adoption. The current landscape of NL.P largely treats subword tokens as fundamental, indivisible units that are determined through a preprocessing step prior to model training.
Some prior works have augmented word or subword token embeddings with character-leved infor- mation (e.g.. Ma al., 2020; El Boukkouri et al., 2020) Aguilar al. (02T). More recently, the T-FREE method (Deiseroth et al. (024) operates at the word level while incorporating character information through specialized embedding and output layers. These approaches have either not tackled generative modelling or still require fixed vocabulary for generation.
Another line of prior work aims to enable character- or byte-level language modelling at contem- porary scales. While some attempts have been made to train purely character-level transformers (AI-Rfou et al. 2019 Choe et al. 2019 Xue et al., 20022 these have ultimately not kept pace with level models. Moving closer the present work, some authors have presented "hybrid" approaches, e.g., using downsampling mechanisms (Charformer: Tay et al. 2022), or cross-attention with latent "bottleneck" sequence (Perceiver AR; Hawthorne al. 2022) to internally condense the large sequence lengths generated by character-level models. In the following, we describe in detail the most closely- -eelatd works.
Sun et al. 2023) use a hierarchical character-word architecture for BERT-style masked language modelling. As input to the decoder, the authors concatenate the backbone output for the i-th word with the per-character encoder outputs for the same word. We devise a generative variant of this architecture. We introduce a shift by one, concatenating with embeddings of the next word and use the raw character embeddings rather than the encoder outputs, since the bidirectional encoder would otherwise "leak" information about future characters. Our work also substantially scales up this modelling paradigm, experimenting with models up to the 7B parameter scale, compared to Sun cet al. (2023) who use models around the 100M scale.
The MegaByte architecture (Yu et al. 2023) uses backbone-decoder architecture for generative modelling. Instead of splitting byte sequences into words, MegaByte chunks into fixed-size patches of subsequent bytes. Their architecture does not backbone is simply the concatenation of the embeddings of the bytes within patch. Note that this restricts the architecture the use of fixed-size patches. At the decoder, the backbone output is
encoder module; the input to the
5

---

Published as a conference paper at ICLR 2025
added to the byte sequences, which is prepended with padding token. Yu et al. (2023) experiment with language models up to 320M parameters for the baseline model. We compare our approach experimentally to MegaByte below.
Thawani ct al. ) propose the hicrarchical architecture most closely related to the present work. A notable difference is that they prepend not one but four [W] tokens each word in order in- crease model capacity when going from encoder to backbone. This incurs drastically higher cost in en- and decoder compared to our approach of incrasing the hidden dimension. (Thawani et al. (2023) experiment with models up to 77M parameters on datasets with fewer than 10M characters and context window of only 192 characters (or the token equivalent thereof). Unfortunately. their exper- iments are not compute-matcher and, by our calculation, assign 4x more compute to the hierarchical architecture compared to the baseline.
Finally, in work that appeared concurrently with the preparation of the present paper, Slagle (2024) propose a byte-level model that applies additional Transformer layers to a subset of the imput bytes. They investigate both a fixed-size spacing well as split rule that marks only certain bytes "split bytes", including whitespaces and punctuation. The byte-level layers are not restricted to individual words/chunks and instead use sliding window attention, precluding inference-time performance im- Experiments are
provement via caching (Appendix models trained on 80B bytes and do not include downstream evaluations. None of the above papers investigate robustness or finetunability.
and scaled up to the IB
EXPERIMENTS
We proceed with experimental investigation of the proposed method.
Models. All models are based on the Llama architecture (ouvron et al. 2023) with a fixed atten- tion head size of 128. For the baseline model and the backbone of our hierarchical architecture, use Llama's ' default 1:1 "aspect ratio", i.e., the number of heads equals the number of layers. The design of the character-level encoder and decoder modules is discussed in Section The baseline model uses a BPE tokenizer with a vocabulary size of 64k, fitted on our pretraining data.
Data. We perform our main experiments using the Maselllee dataset (Li et al. 2024). which a curated English-only pretraining dataset. Hyperparameter sweeps and ablations conducted early on in the project used the well-established Fineweb dataset (Penedo al. 2024). For our continued pretraining experiment, we use the German portion of the Occiglot Fineweb v0.5 dataset (Brack ct al. 2024). Dataloading handled on a byte basis to guarantee that both models get see the exact same data during training. We enforce a maximum document length of 16, 384 bytes, corresponding to roughly 4k tokens or 2.7k words. We load batches of 1024 16 384 bytes, packing together documents of varying lengths with appropriate attention mask reset. In our pretraining experiments, we train for 72k steps, which comes down to a total training set size of roughly 1.2 trillion bytes.
Hyperparameters. We use the Adam W optimiser (Loshchllov & Hutter 2019) using 111 = 0.9, 32 0.95, & = 10-8 and weight decay coefficient over 500 steps followed by a cosine decay to 10% of its peak value. We did not tune learning rates individually for each model but instead opted to use a well-established heuristic, scaling the learning rate inversely proportional to model width, see Appendix B.I
0.1. The learning rate warmed up
Eval Metrics. We focus on downstream evaluations as the primary metric for comparison, see Appendix B.4 for description of our eval suite. In addition, we report more immediate metric for pretraining performance. Since we compare models making byte-level and subword-leved pre- dictions, this requires some extra care. We use accuracy aggregated at the word level, explained in Appendix B.2
4.1 HIBRARCHICAL ARCHITECTURE SWEEP
Our hierarchical model consists of encoder, backbone, and decoder. Given total compute budget, we now conduct a series of experiments to determine optimal sizes (number heads and layers) of these three modules. These experiments used the Fineweb dataset at budget of 14.4k steps.

---

Published as a conference paper at ICLR 2025
Baseline IB
Baseline 3B
Baseline IB
Baseline 3B
39.9%
Word Accurccy
Byte Accuracy
38.5%
85.5%
38.0%
85.0%
37.9%
Deodeee Size
layers headk layers
10 heads layers
36.0%
84.0%
Normalized FLOPs
Normalized FLOPs
Figure 3: Word and byte accuracy for hierarchical models with different encoder/decoder sizes. Each candidate encoder/decoder size has been trained with backhone sizes ranging from Ta = H, = 16 LaB H 30. The horizontal axis shows training compute (Eq. (7) normalized by the highest value. The vertical gray lines indicate the compute required by the baseline models. Small character- level modules tend yield better word accuracy at the tested compute budgets.
Given backhone size (with Ta Hr mentioned ahove), have four degrees of freedom- the number of layers (Te, Taa) and heads (H,, Ha) in encoder and decoder-which is too many for an ex- conducted some preliminary experiments, detailed in Appendix [B.3]
haustive sweep. Therefore, to constrain the search space. Specifically, we concluded to use the same architecture for encoder and decoder (Le La. H. Ha) and fix their aspect ratio at Hl. 2Le-
The crucial remaining question is how to size en- and decoder relative to the backbone. From preliminary experiments. isolated three candidate sizes: (He, Le) E {(6, 3), (8, 4). (10, 5)}. We then trained each candidate with multiple backbone sizes, ranging from La = Hp 16 to La H,, = 30. The result are depicted Fig. where we report both byte-level and word-level aceuracy. Strikingly, these two metrics paint sharply different pictures, with the former favoring larger byte- level modules and the latter favoring a larger backbone. However, the relative differences byte accuracy are very small. Moreover, byte accuracy be improved by merely making the decoder better at completing words given the first few characters, which does not improve word accuracy.
We adopted word accuracy, which we hypothesized to be better predictor of model quality, as our guiding metric. This lead us to choose the (6. 3) configuration for the smaller two model sizes (IB and 3B-scale). We opted to the slightly larger (8, 4) configuration for our 7B-scale model, based on the the observed trend that larger encoder/decoder sizes catch up for larger backbones. The backbone sizes for these models set by compute-matching the baseline, which discuss next.
4.2 COMPUTE-MATCHHS MODELS
For the following architecture comparison, we decided to compare compute-matched models, meaning models that require (on average) the same amount of compute to process document from the pretraining dataset. Since compute is the primary driver of training and inference cost, we believe this approach ensures fair comparison between architectures, We first set the sizes for the encoder and decoder modules of our hierar- chical architecture based on the considerations in the previous section. For each scale, we then size the backbone of the hierarchical model to match the compute required by the baseline. For the compute matching, used the exact computa- tional cost analysis, including attention FLOPs, as explained in Appendix A.2 and the statistics of pretraining dataset, depicted in Fig. Details the compute matching methodology may be
matched model configu- rations, showing number of attention Heads, Layers, and Parameter count.
Table 2:
Scale
Tokenizer Baseline
Hierarchical (our)
Backbone
En-Idecoder
H
I.
P
H
L
P
H
I.
P
1B
16
16
1.1B
18
18
1.1B
6
3
23M
3B
24
24
3.1B
28
28
4.3B
6
3
24M
7B
32
32
7.0B
36
36
9.2B
8
4
55M

---

Published as a conference paper at ICLR 2025
Table 1: Results of our pretraining experiments, showing accuracy on the pretraining dataset as well as scores on established eval tasks in the zero-shot setting. Generally, our hierarchical model performs on par with the tokenizer baseline within each compute-matchi notable wins for the hierarchical model on the Lambada (LBD) eval task, where it outperforms the baseline by a relative margin of up to 68% (at the 7B scale).
scale. There are some
DCLM Word Ace
DCLM Byle Ace
OpenBook QA
MMLU
LBD OAI
LBD OALC
L,0
LRD
ARCC
ARCIE
ThiviaQA
TPQA
WinGGr
HellaSwng
WIC
WebQQ
PIQA
BoolQ
XNLI
Model
1B
Hierarchica (whilespace)
35.5
30.5
37.5
Baseline
3533
27.6
57.7
53.7
0.8
31.7
68.0
25.8
15.3
26.8
00.1
46.2
30.3
6.4
75,9
62.6
33,9
MegaByte (8-byte split)
80.0
25.1
31.3
2.1
41.0
0.7
23.4
54.9
20.2
2.9
30.7
53.0
38.9
52.0
1.5
68.8
35.8
34,4
Hierarchical (8 hyte split)
21.0
3B
Hicrarchical (whitespace)
37.8
2x.7 70.7
64.2
18.037.1
18.0
72.3
29.0
24.2
29.9
64.
33.2
151.1
9.6
75.4
69.8
34.5
Baseline
39.8
30.0
50.0
34.1
7B
Hicrarchical (whitespace)
39.0
32.0
73.8
43.1
67.4
28.6
44.2
76.6
30.8
33.1
31.5
69.0
36.
47.3
118
12.0
40.5
Baseline
392
67.9
25.6
66.0
24.4
46.8
79.4
33.4
32.4
28.6
49.1
79.6
73.3
37.4
found in Appendix A.3 The resulting models are listed in Table 2 Thanks to efficient batched
implementation of the character-level models using flash attention (Dao et al. 2022), we also see comparable step durations for our compute matched models (see Appendix A.3
4.3 PRETRAINING RESULTS
In this section, we compare our hierarchical architecture with a subword tokenizer baseline for pre- training the DCLM-Baseline dataset. All models are trained from scratch. We compare compute- matched models at three scales, corresponding to IB, 3B, and 7B baselines. Table presents byte- and word-level accuracy, well as zero-shot performance across 17 standard downstream tasks. Across scales and evaluation tasks, both architectures perform similarly, with a few notable differ- ences. At the 1B scale, the tokenizer-based model holds modest advantage this gap disappears at larger scales. On the other hand, the hierarchical model consistently outper- forms the baseline on the Lambada evaluation, with relative margin of up to 68%.
TriviaQA, though
Comparison MegaByte At the IB scale, we also conducted a comparison with MegaByte Yu
et al. 2023),, another hierarchical byte-level generative architecture, as detailed in Section 3 We
use model configuration from the original paper, which is compute-matched with our IB-scale (14
layers in the backbone with hidden dimension D 2048, 18 layers in the decoder with hidden size 1024 and learning rate 10 10-4). The results Table 1 show that MegaByte underperforms across all but one evaluation tasks. To further investigate. our hierarchical architecture with MegaByte's fixed 8-byte splitting, labeled as Ilierarchical (8-byte split) in the table. This variant significantly improves over MegaByte (c.g., 2.6 ppt increase in byte accuracy), showing that our hierarchical architecture is more performant even when using the same splitting rule. However, the hierarchical architecture with whitespace splitting still comes out top, suggesting that a semantically meaningful splitting a valuable inductive bias for the model.
conducted
ablation combining
4.4 ROBUSTNESS AGAINST INPUT PERTURBATIONS
Next, we evaluate the robustness of hierarchical and baseline models against perturbations of the inputs. We conduct this experiment on a subset of five eval tasks, for which the two architectures showed similar performance. We apply perturbations to the prompt of each item of the dataset and measure the change in average accuracy compared to each model's performance on the original (unperturbed) golden answer. The perturbations include permuting, randomizing, of the characters per word, as well as changing the prompt to all caps. The results are depicted in Figure 4 We see that the hierarchical model is significantly more robust than the tokenizer-based
deleting 10%
8

---

Published as a conference paper at ICLR 2025
1B
3B
7B
1B
3B
7B
1B
3B
7B
1B
3B
7B
0.0%
-2.0%
Accuracy
4.0%
Hieecrchical
-6.0%
10% permuted
10% replaced
10% deleted
ALL CAPS
Figure 4: Average change in accuracy across the MMLU, OpenBookQA, Are Challenge and Hel- laSwag eval tasks for perturbations the prompt.
model across perturbation types and scales. In particular for the all-caps perturbation, the baseline model suffers 3 times larger drop in accuracy. Individual results and more details may be found in Appendix [B.]] Table shows some illustrative examples, comparing the two models' completions to perturbed prompts.
Table Example completions of perturbed prompts.
Prompt
Hierarchical (our)
Baseline
Incomplete words
The quick brown jumps the
zy dog.
la la
Unce upon ti
me. there was [...]
ime, there was [..]]
Spelling variations
The quick brown fox
jumps over the lazy dog
jumpp over the lazy dog
THE QUICK BROWN FOX
JUMPED OVER THE LAZY DOG
by John Updike
The quick hrwon
fox jumps over lazy dog.
-out the 1980s, [...]
4.5 ADAPTATION ON CROSS-LINGUAL CONTINUED PRETRAINING
After pretraining the English-only DCLM-
Baseline dataset, we continue training the 3B-scale models on the German Occiglot dataset (Brack et al., 2024) to test adaption to a shift in data distribution. We re-warm the learning rate to half of its initial value and train for 20k steps with otherwise identical settings as the pretraining runs. We conducted down- stream evaluations every 2000 steps on set of tasks for which both English and German ver- sions are available, see Appendix [.6 The re- sults shown in Figure 5 The hierarchical model consistently achieves higher average ac- curacy on the German evaluations while also re- taining better scores on the English tasks. Since the tokenizer operates
42.5%
Hienschical Baseline
Average Accuracy
40.0%
Engtish
30.0%
27.5%
German
25.0%
22.5%
5000
10000
15000
20000
Steps
Figure 5: Continued pretraining on Occiglot Ger- man. We show average eval scores on English and German. The hierarchical architecture adapts better to the new language, while also retaining higher scores on English evals. At an equal FLOP budget
rigid vocabulary, is unable to adapt to the new domain and must resort to byte fallback or combine tokens sta- tistically unfounded ways. We attribute the per- formance difference to the resulting larger dis- tribution shift in the input of the tokenizer based model.
only able to train the baseline for half as many steps due to requiring almost two times the compute per document the german dataset. Nevertheless, we continued training the baseline for more steps, depicted dotted lines.
It is important to note that the models compared here have been compute-matched matched based on the statisties of the English DCLM dataset. As shown in Figure 2 (right), the word and token statistics shift drastically when switching to the German dataset, with bytes-per bytes-per-token decreasing substantially due tokenizer fragmentation. Consequently, the token sequence length grows much larger than

---

Published as a conference paper at ICLR 2025
the word sequence length, making the continued pretraining of the hierarchical model 1.9 times faster than the baseline model, or equivalently being able to train on almost twice the data for the hierarchical model.
CONCLUSION
We presented a hierarchical autoregressive transformer architecture that integrates character-level and word-level processing. Our approach retains the sequence length compression of word-level tokenization, while removing the need for a rigid, predefined vocabulary. Through extensive exper- iments, including models scaled up to 7 billion parameters, we demonstrated that the hierarchical architecture matches the downstream task performance of computed-matede els, while significantly improving robustness to input perturbations and continued pretraining dutti--distribution data, such as previously undersampled languages. These findings highlight the potential of hierarchical transformers enhance flexibility, robustness, and generalization across diverse NLP tasks and domains.
tokenizer-basee mod-
Limitations. The whitespace splitting we used above tailored to alphabetic languages rather than logographic languages like Chinese, where characters represent entire words morphemes. These languages may benefit from a custom splitting rule to group bytes into semantically meaning- ful units. A similar concern holds for domains like mathematical writing code, where whitespace splitting might not yield an optimal chunking. In follow-up experiments, presented in Appendix C obtained promising results using "universal" splitter based on the Unicode standard. Secondly, as discussed above, for a compute-matched model, the hierarchical architecture will have higher parameter footprint. During inference, this may be offset by a reduced size of the KV cache.
Outlook. Our work be extended in various ways. First, one could experiment with different models for encoder and decoder. For example, the small character vocabulary, may facilitate multi- token prediction (Gloeckle et al. (024) with multiple output heads. Further, with few characters per word, a text diffusion model (e.g.. Li et al. 2022) could be used as decoder. Finally, one could investigate additional levels of hierarchy, such sentences paragraphs, which may improve long context generation abilities.
REFERENCES
Gustavo Aguilar, Bryan McCann, Tong Niu, Nazneen Rajani, Nitish Shirish Keskar, and Thamar Solorio. Char2Subword: Extending the subword embedding space using robust character com- positionality. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1640-1651, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 00.18633/v1/222/ htt.:///........nseeeeeeee Findings- em..p..mmmpp...... 141
URL https::/aaelantbaloo...oo//////
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the Thirty-Third AAAI Con- ference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelli- gence Conference and Ninth AAAI Symposiun on Educational Advances Artificial Intelligence, (AA/19/AATT19/AAAM
doi: 10. 1609/aii httt:ttt:: /o.. wrr.//rggo...... 609 aaai. v33101 330133155
AAAI Press, 2019. ISBN 978-1-57735-800
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Schulze Buschhoff, Charvi Jain, Alexan- der Amo Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Hem. Tokenizer choice for LLM training: Negligible or crucial? In Kevin Duh, Helena Gómez- Adorno, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 3907-3924. Association for Com- putational Linguistics, 2024. doi: 00.186633V1/22224.F1 fid
NAACL.247. URL https: wtw.nddiinnnnaann ngs ngs-naaci .247
7/doi. oacllorg/1000 100
10

---

Published as a conference paper at ICLR 2025
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing Freebase from crestion-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL ps: www. ac www.:www........................
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. In AAAI Conference on Artificial Intelligence, 2019.
URL htttpp::/aapi..emaa..iicic httpsch........ooooooooooo...222 http:/Ccoppu11::0220000
org.
Manuel Brack, Malte Ostendorff, and Pedro Ortiz. Occiglot fineweb v0.5, 2024. URL https:
wwwwweb--wweee v0..
Dokook Choe, Rami Al-Rfou, Mandy Guo, Heeyoung Lee, and Noah Constant. Bridging the gap for tokenizer-free language models. arXiv preprint arXiv: arXiv:
2019.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computationall Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 2924-2936, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18633/1//1111330000 URL org/N19 1300
htttps:/acclanthr
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. CoRR, abs/ 803.05457, 2018. URL httt::/tpa//iii/ wwwp:/i...rr//ooooo org
05457
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2018.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference
Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http::/pappee wips. cw/pape fi httplees/pppp///2222////// 77d55cc3222Fdaaaaa3333a33
ht.l]
Bjöm Deiseroth, Manuel Brack, Patrick Schramowski. Kristian Kersting, and Samuel Weinbach. T-FREE: tokenizer free generative Ilms via sparse representations for memory-efficient embed-
dings. CoRR, 192223306.11223,, 2024. doi: AAS500/ARXIV.2406.19222 URL https: //doi. 48550 2406. 19223
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLTT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N199 1423. URL
1423
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina I Lobanova, Emily Dinan, Eric Michael
11

---

Published as a conference paper at ICLR 2025
Smith, Filip Radenovic, and Frank Zhang et al. The llama 3 herd of models, 2024. URL http:///bb//24..21111 22788
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne. Hiroshi Noji, Pierre Zweigenbaum, and aunic Tsujii. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Pro- ceedings of the 28th International Conference on Computational Linguistics, pp. 6903-6915, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguis tics. doi: 10. http://///////2/ooooooooiiiiooo/ main-mmai..00 URL
coling maagg-main... 609
Philip Gage, A new algorithm for data compression. The C Users Journal, 12(2):23-38, 1994.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. bttps::t:///eooodoo wtw.://oorrrr//ooooooooo 12608602]
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better faster large language models via multi-token prediction. In Forty-first International Con- ference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. htttt::://peereiieeeeee wttp:///oo..nnae///cceeeeeeeeee j102
Curtis Hawthorne, Andrew Jaegle, Catalina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew M. Botvinick, Ian Simon, Hannah Shea- han, Neil Zeghidour, Jean-Baal Baptiste Alayrac, João Carreira, and Jesse H. Engel. General purposs,, long-context autoregressive modeling with perceiver AR. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Con- ference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, vol- ume 162 of Proceedings of Machine Learning Research, pp. 8535-8558. PMLR, 2022. URL www.:.al......r....m...
https::/proceest
mwm. press / vvl 62 w2/hawthhonne222 html.
Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and 256gb open-source legal dataset, 2022. URL htttttttss///xiii.ccom
abs/ 2207. 00220
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- jmI 3CmQ
view.net, 2021. URLLbttps:://ppeneee//
net
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Assoccition for Com- putational Linguistics. doi: 1.111855/v1/P/// 1147. URL https: httpss///llatthoooyy..o org/
P17-1 47
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muen- nighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M. Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Raghavi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song, Sujay Sanghavi, Far- tash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo. Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev. Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lm: In search of the generation of training sets for
12

---

Published as a conference paper at ICLR 2025
language models. CoRR, abs/2406.11794, 2024. doi: 10-8553/AARXIIV2400.1744 URL http:://doooorgg//// 1ttp:/...4855//xxxv..22.
11794
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Anmual Conference on Neural Information Process- ing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http::/ httt:://papers.ooe www.ppprss.sss....p.. wtwppps.cpppppp..pppppppppp wetrrrffieesppppee wtt.llsepppppp/........ 9d6a www.eeoooooaaaaats-aeeeaea casttract.oo wwn--oroneeeecoooe -anteaa-- ht.ml
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic hu- man falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Pupers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 3214-3252. Asso- ciation for Computational Linguistics, 2022. doi:
URL
aaaoi.oorg/100 1ttp:///1..88533122222200 wwt.::22.accloooooo 229
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. www.:ettt///aa.. d= CqY7
OpenReview.net, 2019.
Wentao Ma. Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. CharBERT: Character-aware pre-trained language model. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference 39-50, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguisties. doi: (0.18653/11/200000 http:/22200.oonmmmmaa.aa main.4. URL httts://aclanttht
Computational Linguisties, pp-
org/
Sabrina Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. arXiv preprint 12Xiv:21/2.10500 2021.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2331-2391, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DI8- 1260. URL
OpenAl. Mmmlu, 2024. wtt.h//es//nnggggg.......//testa
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Com- putational Linguistics. doi: 10.18655/v//111-1144 URL https::/aclaatthol////// org
P16-111 1144
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Guilherme Penedo, Hynek Kydlicek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data scale. CoRR, abs/2406. 17557, 2024. doi: 0044850/ARXIV.2466.. htttp:://doi..... wwwp:/.....1......0.. 188500/axxii0 www....www........ 2406. 2406.17557
175557776..7777. URL
Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model to- kenizers introduce unfairness between languages. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 36963-36990. Curran Associates, Inc., 2023. URL
13

---

Published as a conference paper at ICLR 2025
Mohammad Taher Pilehvar and Jose Camacch-Coolldos... WiC: the word-in-context dataset for eval- uating context-sensitive meaning representations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies, Volume (Long and Short 1 guistics. doi: 10. (0.18653/1//N19-1222 |BL_Ls/ppe////aaaatooooooooooo//
Papers), pp. 1267-1273, Minneapolis, Minnesota, June 2019. Association for Computational Lin-
Bjöm Plüster. Germanbenchmark, 2024. GermanBenchmark
URL stta:://gitaub..com//jott/
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Dawn Song, Michael Carbin, and Tianqi Chen (eds.), Proceedings of the Sixth Conference Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023, mlsys.org. 2023.
patppeee2211eeeppp///222333 URL https: httppss//procceednggs.ott httppigg.....nsssooor.ree.eeirie wttpoorgrgrpoaoieeee
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9) 106, August 2021. ISSN 0001-0782. doi: 10. 14/34443444388. URL
http///oooo/////.////// 1145/347438/
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguisties, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Pa- pers. The Association for Computer Linguisties, 2016. doi: 10.18653/V/P11-1116. URL
https://doi. 1ttp:/....1633/11p11pp.p 1162
Kevin Slagle. Spacebyte: Towards deleting tokenization from large language modeling. arXiv
preprint arXiv:2404. /4408, 2024.
Li Sun, Florian Luisier, Kayhan Batmanghelich, Dinei Florencio, and Cha Zhang. From characters to words: Hierarchical pre-trained language model for open-vocabulary language understanding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3605- 3620, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 18653vV// 2023.acl long.loo..... URL htthttt:::2 ac www.annnhhloogoo aooooooooo........oo........ 200
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Prakash Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL htttps://opt /pppr//v/eeeeeeeewee net forum?id- BRnrRO1FFF
Avijit Thawani, Saurabh Ghanckar, Xiaoyuan Zhu, and Jay Pujara. Learn your tokens: Word- pooled tokenization for language modeling. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9883-9893, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v12023.. wwgp://....ggg/2/o.... f fwwfiiggnnniiii..iiinninnnnnnn emms.....ppp......
findings-emnlp.662. URL
666
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023. doi: 10.48550/ARXIV.2302.11 https: http://oo..ggg/////// 4e..4455555 48540/ wtt...20............... 13971
URL
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü,, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun
14

---

Published as a conference paper at ICLR 2025
Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuan- hai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng. Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou. Skywork: A more open bilingual foundation model, 2023.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguisties, 10:291-306, 2022. doi: 10.1162/taci 00461. URL
Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. predicting million-byte sequences with multiscale transformers. Alice Oh, Tristan Naumann. Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023. htt.http://pppeeeee n nips wttpppcccppppppeeeeeee files//ppper22233/aaas Ctnterenceettmml
MEGABYTE:
E8178 B
Abstract Abstrrrtttt-ttttt---tt--a-- oatrra/eeeaaatatttoooorrrreeeeon
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 4791- 4800. Association for Computational Linguistics. 2019, doi: 10.18663/V1/119-4422 URL https: / /ww.i 10trg//...1633///pplll 1472
15

---

Published as a conference paper at ICLR 2025
A MODEL DETAILS
A.1 ARCHITECTURE DETAILS
Splitting Rule We split whitespace characters, as per the Unicode standard, which includes spaces, tabs, newlines, et cetera. All consecutive whitespace characters are appended to the previous word. Our approach is agnostic to the type of splitting rule. For instance, in Section[4] we experiment with a fixed-size splitting, in Yu et al. (2023). Other splitting rules me be adequate for non- alphabetic languages or domains like mathematical writing or code. We leave this to future work.
End of Document We append final dummy word to the end of each document indicate its end. This dummy word consists of a single special token [S] and, like every word, is prepended with a [W] token during processing. We treat [S] as termination token during inference. During training, we omit the final [W] as prediction target for that word, since [S] already indicates termination.
A.2 EXACT COMPUTATIONAL COST COMPARISON
We give a detailed description of the computational cost, extending the simplified analysis presented in Section 2.3 Following common practice, we count the number of multiplications in the operations involved in a forward pass through the model, covering only matrix multiplications and ignoring biases, normalization layers, activation functions and other minor operations.
Consider a forward pass through a single transformer laver with hidden size D and sequence length S. In standard Llama architecture (Touvron et al. 2023), this layer has 12.2² parameters. This includes key, query. value and attention dense-out weight matrices shape D D and three matrices of shape D D for the SwiGLU-MLP. The matrix multiplications in the forward pass require one multiplication for each weight,
(s D) 12SD?
Additionally, factor in the multiplication of the query and key matrix, well the multi- plication of the attention matrix with the value matrix, resulting in an additional cost of
Catin (S, D) 252 D.
Now consider a document consisting of S characters, which get split into Sr tokens and Sw words, respectively. For simplicity, we assume that all words are of the same length S/Sw.
Baseline The baseline model processes sequence of length ST. passing through the backbone and the output head, incurring a total cost of
Obaseline
I. hackhome baseline
Cir(ST, D Dhuckhone))
(Feed-forwart
I bnseline
Caun (St, Dbuckbbue))
(Attention)
(10)
+ ST Daseline
Vr
(LM Head)
Hierarchical Architecture In the hierarchical architecture, the backbone processes sequence of length Sw. The two byte-level models process each word in the document, each of which is of length + S/Sw due to the appended word separator tokens. Additionally, we have the two linear projections at the intersection of backbone and encoder/decoder, each of which incur cost of D bierarchical ietraciica per character. Finally, the encoder has a character-level LM head, which adds Ieracciccal BB. In total
Dehar
cost of Dehar
C Chieraccical
=
I backbone hierarchical
Cn((S: D Deeeatemnca))
(Backbone forworr
+ L backbone
Cauu (Sw, baackbone
(Backbone attention)
+ 2L cbat
hierarctical
Sw Cir(1 S/Sw,
Char Deharroee (eeennitical
(En/decoder (eed-forward)
(11)
2L char
Sw Cattn (1+ S/Sw.
thar
thar
(En/decoder attention)
hierurctical
+ 2Sw titrrrcticcal Ietracciical
backbome chtr
(Linear projections)
+ (S + Sw) hierarchical VB
chur
(Decoder LM head)
16

---

Published as a conference paper at ICLR 2025
Scale Hierarchical Bascline
IB
0.9s
0.9s
3B
2.3s
1.6s
7B
4.5s
4.7s
Table 4: Average step durations on 256 H100s at a global batch size of 1024 sequences with 16384 bytes of data
A.3 COMPUTE MATCHING
We randomly sample 10, 000 documents from our pretraining dataset. For a given model configu- approximate its average cost by
ration, be it a baseline model hierarchical model, averaging the cost formulae (Eq. 10 or 11 respectively) over our sample documents. The cost only depends on the document lengths in characters, tokens, and words.
The baseline model parameters are fixed, so we can compute its average cost. For the hierarchical model, encoder and decoder size are fixed and the only variable parameter is the number of heads and layers in the backbone. We compute average cost for possible backhone sizes and choose the size whose average cost matches that of the baseline model as closely as possible. The resulting hierarchical model configurations are listed in Table 2 Since the number of backbone heads/layers an integer quantity, the matching not exact, but the relative deviation smaller than 5% across all configurations.
The average step durations for the models during pretraining on 256 H100s are shown in Table
A.4 INFERENCE- PERFORMANCE
We briefly discuss the inference-time performance of our hierarchical model compared to compute- matched baseline model. We assume we use KV caching and ignore attention FLOPs. Then the average FLOPs required to generate S bytes is matched
Stusellnne SCenccoder
heerarcheca + Sw Chockbone
hierarchical
+ SCdecoder
Iierarchical'
(12)
Wall-clock time is proportional to cost if all architectures process single tokens, i.e.,
Sylaweline Rs cycoddr
+ 5w backbone + Sydcoder hiirarchical-
S
(13)
However, in an inference setting, the hierarchical architecture has the advantage that the encoder will always ever be queried with entire words, not single characters. Since the wall-clock time of forward pass in KVeecheed inference typically dominated by I/O operations, the encoder will take a similar time to process an entire word as it would to process single character. Hence, in terms of wall-clock time, we will have
SyShsseline
VS
Sw Sw(tencn (nerrrrchial +
backbone 'iierrcciical
St
deecder nieranchical
(14)
This advantage could be further expanded with a simple inference-time performance optimisation of our hierarchical model. After training. could extract and store the word embeddings (i.e., encoder outputs) for the Vw most common words in a reference corpus. One could probably cover 2 95% of words with very modest vocabulary size. For words in this vocabulary, the encoder stage is then replaced with an (11) lookup. Likewise, for the decoder, one could store predictive word embeddings for the words in the corpus. This match could be used for speculative decoding or even be accepted as is if the match is "good enough". We leave the implementation and evaluation of this performance optimisation future work.
A.S INFERENCE- TIME MEMORY
The size of the KV cache is proportional to SLD, where S is the sequence length, L is the num- ber of layers and is the hidden dimension. For the hierarchical model, one would only cache activations in the backbone in any practical setting. Encoder and decoder operate very small sequence lengths at inference time, where KV caching would not yield wall-clock time speedups, also Appendix A.4. Hence, compare ST baseline
BacakboeeDaaaae backbonc to Sw T, Tiieurchiccl Ierrachicall backkoon For
17

---

Published as a conference paper at ICLR 2025
compute-matched models, using ST and Sw from Figure 2 the size of the KV cache would be reduced by 13%, depending on the exact configuration. Note that this reduction would be far more pronounced for out-of-domain inference, where ST und Sw can shift drastically.
B EXPERIMENT DETAILS
B. TRAINING SETTINGS
Following Dubey et al. (2024), the peak learning rate is set to Irg2 = 3 10 10-4 for the 32 head model and scaled with model size terms of number of heads as Ir(H) H Ir32- For the hierarchical model, we use the number of heads in the backbone; we briefly verified that the heuristic is adequate for the hierarchical architecture. Since is not tailored hierarchical model, there might be room for further improvement.
32
B.2 WORD-LEVEL ACCURACY
For given input sequence F1.T and segment Tist, we denote the model's prediction as m(x1:)) [0, 1] B1, which a vector of predictive probabilities for the next byte T1+1- Byte-level accuracy
Anyle (xI:T) T
Σ (m(oneee -max maxm())))))
(15)
Now assume the byte sequence is segmented into words, given by set of indices $1.... Sw E [T] indicating the first character of word. The next-byte probabilities auto-regressively imply next- word probabilities, for which we can compute a word-level accuracy as
W-1
8mm-1 -2
Aword(D1e1 S1:W) W
II t=6w
(m(F2u)aw) = max maxm()1))))
(16)
B.3 HIERARCHICAL ARCHITECTURE SWEEP
Aspect Ratio We first ran an experiment concerning the heads:layers aspect ratio encoder and decoder, wanting to understand whether it may be beneficial to deviate from the 1:1 ratio used in standard Llama architecture. For this experiment, we tied the sizes encoder and decoder (La = Le- Ha H.) and tested different values for the aspect ratio. We hypothesized that "wider" model may be beneficial and tested aspect ratios 1:1, 3:2, and 2:1. The result is depicted Figure Since the result didn't 't show any considerable difference between different aspect ratios, went with 2:1 based on the intuition that the change hidden size between character- and word-level modules should be limited when scaling larger backbones.
Encoder//ecoder Balance Next, we ran an experiment to decide how encoder and decoder should be sized relative to each other. We used H. Ha 8 and allocated total number of L, + La 8 layers to distributed between encoder and decoder. As shown Figure the best word accuracy is achieved by an even number of layers in the two modules. Similar as in Figure 3 byte-level accuracy favors a larger decoder module.
B.4 EVALUATION TASKS
We use set of established downstream evaluation tasks, implemented in evaluation suites like the Eleuther AI eval harness (Gao et al., 2024). In the following, we give brief descriptions as well as citations. The descriptions quotes either from the original paper from Li al. (2024).
MMLU (Hendrycks et al. 2021 is 4-way multiple choice question answering dataset that covers 57 different domains and tasks, evaluating both world knowledge and problem solving capabilities.
18

---

Published as a conference paper at ICLR 2025
$5.055
Word Accuracy Byte Accuracy
82.7%
Accuracy
$4.3%
82.5%
al
33.0%
$4.0%
Aspect Ratio 1.5:1
Aspect Ratio 1.0:1
Aspect Ratio 1.0:1
$3.59
32.8%
0.4
0.9 Normalized FLOPs
0.6
1.0
(2.6)
(3.5)
(4.40
(5,3)
(6,2)
(Enooder LendeerLLyyee Layers, Decoder I Layers)
Figure 6: Scale up of the character-leved module size different aspect ratios and fixed backbone size of 16 heads and layers. The x-axis depicts compute cost (Eq. 7). nor- malized by the largest value.
Figure 7: -ecoder- balance. A fixed number of character-level layers is dis- tributed between encoder and decoder. The backbone is fixed.
LBD: LAMBADA (Paperno et al. 2016) is collection of narratives where human is not able if they are only given the final
able to guess the final word of the narrative, sentence. To perform well on this task requires the model to attend to context from the full narrative and cannot simply rely on the local context.
The ARC easy and ARC challenge datasets (Clark al. 2018) contain four-way multiple choice questions taken from grade 3-9 science exams, where questions in the easy dataset require knowledge of basic science, and the challenge questions require some procedural reasoning.
ARC:
OpenBook QA (Mihaylov et al. 2018 is 4-way multiple choice question answering dataset that requires the model to use multi-step reasoning and commonsense knowledge.
TriviaQA (Joshi et al. 2017) an open-ended question answering dataset that evaluates the world knowledge of model.
TFQA: TruthfulQA (Lin et al. (2022) a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. Questions are crafted so that some humans would answer falsely due a false belief misconception. To perform well, models must avoid generating false answers learned from imitating human texts.
WinoGr: The Winogrande dataset (Sakaguchi et al. 2021 extends the Winograd Schema Challenge dataset by expanding the dataset
wider variety of domains.
HellaSwag (Zellers et al., 2019) is a 4-way multiple choice commonsense reasoning dataset, where the model is required to understand implicit context and common knowl- edge in order to correctly select the continuation to a context.
WiC: Word in Context (Pilehvar & CaaachooCollade evaluation of context-sensitivy word embeddings. WiC is framed as a binary classification task. Each instance in WiC has target word w, either verb or noun, for which two contexts are provided. Each of these contexts triggers specific meaning of w. The task is to identify the occurrences of w the two contexts correspond to same meaning or not. In fact, the dataset can also be viewed as application of Word Sense Disambiguation in practise.
2019]. WiC is benchmark for the
WebQs:
The Web Questions dataset (Berant al. 2013) consists of 6,642 question/answer
pairs. The questions are supposed to be answerable by Freebase, a large knowledge graph. The questions are mostly centered around a single named entity. The questions are popular ones asked on the web (at least in 2013).
PIQA (Bisk al.) 2019) a binary multiple choice question answering dataset that re- physical commonsense reasoning to answer correctly.
quires the model
BoolQ (Clark et al. 2019 is binary question answering dataset where the model is ex-
pected answer questions about relevant passages.
19

---

Published as a conference paper at ICLR 2025
XNLI (Coneaau et al. 2018) is subset of few thousand examples from MNLI which has been translated into a different languages (some low-ish resource). As with MNLI, the goal is to predict textual entailment (does sentence A imply/conttradic//cittte sentence B) and a classification task (given two sentences, predict one of three labels).
B.5 DETAILS ON ROBUSTNESS EVALUATIONS
Figure 8 shows the complete robustness results, initially reported in Section 4.4 displayed separately for each eval task.
B.6 DETAILS ON CONTINUED PRETRAINING EXPERIMENT
We average over the following benchmarks, for which we have English and German versions:
HellaSwag, machine translated (Zellers al.
2019
Lüster
2024)
Arc Challenge, machine translated (Clark al.
2018
Plüster
2024
Truthfulqa, machine translated (Lin et al.
2022
Plüster
2024)
Lambada OpenAl, machine translated (Gao et al.
2024
MMLU, Human translated (MMMLU) (Hendrycks et al.
2021
OpenAl
2024
20

---

Published as a conference paper at ICLR 2025
Robustness of ARC C to 10% perturbations
7B
3B
1B
7B
0.0%
A
-2.0%
Hierarchical
Baseline
8.00%
10% permuted
10% replaced
10% deleted
ALL C.ASS
Robustness of HellaSwag to 10% perturbations
1B
3B
7B
1B
3B
7B
1B
3B
7B
1B
3B
7B
Hierarchical
Baseline
10% permuted
10% replaced
10% deleted
ALL CAPS
Robustness of MMLU to 10% perturbations
1B
7B
0.0%
1.0%
A
-2.0%
3.0%
4,0%
Hierarchical
Basellne
10%
permuted
10% replaced
10% deleted
ALL CAPS
Robustness of OpenBook QA 10% perturbations
1B
3B
7B
1B
3B
7B
1B
3B
7B
1B
3B
7B
0.0%
-4,0%
Hieerrctucal
-6.0%
Basclliee
10% permuted
10% replaced
10% deleted
ALL CAPS
Figure 8: Robustness results.
21

---

Published as a conference paper at ICLR 2025
Table Results of pretraining experiments with the Unicode splitter, showing scores on established
eval tasks in the zero-shot setting. The Unicode splitter outperforms the whitespace splitter on the majority of eval tasks.
OpenBook QA
LBD LBD OAIC
LBD OAD
TrivinQA
HellaSwag
MMLU
LBD
LBD
ARCC
ARCIE
TFQA
WinoGr
WIC
WehQs
PIQA
BoolQ
XNLI
Model
Hicrarchical (Unicode)
31.0
70.9
65.7
18.3
39.0
73.4
27.8
25.9
27.8
65.9
52.8
11.0
69.1
37.7
Hicrarchical (Whitespace)
28.7
70.7
28.4
64.2
18.0
37.1
72.3
29.0
24.2
64.1
53.2
51.1
9.6
69.8
34.5
Baseline
29.1
65.8
39.8
72.9
24.0
65.7
53.3
7.3
77.5
69.7
C UNICODE SPLITTER
In main experiments, used simple whitespace splitting rule, which showed competitive performance on natural text in english and german. As discussed in Section whitespace splitting not suited for languages without explicit word separators (Chinese, Japanes, Korean) and code, where punctuation used as separator: To address these limitations we experimented with alter- native split rules and found promising universal rule in the Unicode standard for word boundaries, which we detail and evaluate in this section.
C.1
SPLIT RULE
The Unicode Standard, in particular Unicode Standard Annex #29, provides "guidelines for deter- mining default segmentation boundaries between certain significant text elements", which includes word segmentation. This also covers word boundaries for non-alphabetic languages. We use the uni seg Python package to split text into words according to this standard. In addition, we have found it to be beneficial to split text punctuation. To improve sequence compression, merge leading whitespaces and trailing punctuation into words. In the following, we refer to this splitting rule the Unicode splitter.
C.2 PRETRAINING RESULTS
We repeat our --aselinee pretraining experiments using the Unicode splitter. All training details stay described in Section 4 The byte-per-word statistics of the new splitter result in the same flop-matched model sizes described Section 4.2 The results are shown in Table where we see that the Unicode splitter outperforms the whitespace splitter the majority of eval tasks.
C.3 CROSS-LINGUAL CONTINUED PRETRAINING ON CHINESE DATA
Next, test lings-li adaptation a non-alphabetic language, we perform continued pretrain- ing on the Skypile dataset (Wei et al. 2023). which is Chinese pretraining dataset. This experiment has been done using the 3B model scale using 5k steps; all other experimental details match those described in the German experiment presented in Section 4.5) We compare the hierarchical archi- tecture using the Unicode splitter to our tokenizer baseline.
Since we did not have access to Chinese-language downstream evals, we report bits byte (bpb) D The learning curves are depicted in Figure 9 and the final bits per byte on held-out portion of SykPile is shown in 6 We see that the hierarchical architecture reduces bpb more rapidly and reaches significantly better values within the assigned training step budget.
https: /tt///niiegggggeee py. www.dreerdddds.e.eeem.o
http://iiieg.yr..reteerreeeeseee i www.iiio/eeeeeseeeeeeeeeeeeeesee http://////aatttt//////teeeeee/e
For input document x define BPB(z) == N. Σ Σm log, P(x, (i)), where N₂ is the number of UTF-8 bytes in the text. N is the number of elements processed by the model (Equal to No for the hierarchical model and equal to the number of tokens in the tokenizer based model) and P. is the next item probability as predicted by the model.
22

---

Published as a conference paper at ICLR 2025
Table 6: Bits per byte on held-out portion of SkyPile after after continued pretraining.
Model
Bits per Byte
Tokenizer Baseline
0,94
Hierarchical (Unicode)
0,80
Hierarchical (Whitespace + Unicode)
0.80
As in the corresponding experiment using German data, the tokenizer baseline also needs substan- tially more compute for the same number of training steps, due to tokenizer fragmentation. Since the tokenizer is attuned to Chinese, it almost always goes into byte fallback, degrading to an average bytes-per-token of just 1.02, compared to 4.29 bytes-per-word for Unicode splitter. Overall, the tokenizer version incurs 2.3 times the computational cost.
Fig. 10 shows variant of the previous experiment, where we exchange the splitting rule after the initial (English-only) pretraining experiment. That is. the learning curve labeled Whitespace + Unicode has been pretrained on DCLM using the whitespace splitter and fine-tuned on Skypile using the Unicode splitter. For reference, we also display the learning curve of the previous experiment, which uses the Unicode splitter throughout. We can see that the model adapts to the change in splitting rule rapidly, catching up with the "continuous" run within approximately 1000 training steps. This rapid adaptation suggests that the backbone learns language-agnosee representations, while the encoder/decoder components can effectively map new byte sequences into this established embedding space.
Baseline (Tokenizer)
Hicrarchical (Whitespace + Unicode)
Hierarchical (Unicode)
Hierarchical (Unicode)
Bits per Byte
Bits Byte
1.4
1.2
0.8
0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Training Step
Training Step
Figure 9: Learning curve in bits per byte (bpb) for continued pretraining on the Chinese Skyp- ile dataset. Using the new Unicode splitter, the hierarchical architecture adapts to the new language more rapidly and achieves better bpb within the assigned step budget. Note that the baseline requires 2.3 times more compute for the same number of training steps.
Figure 10: Learning curve in bits per byte (bpb) for continued pretraining on the Chinese Skyp- ile dataset. Both variants use the Unicode split- ter for the continued pretraining stage, but have been trained using different splitters in the ini- tial English-only pretraining phase. The vari- ant that undergoes a change splitting rule adapts quickly and achieves the same final per- formance.
C.4 INFERENCE PERFORMANCE ON OUT-OF--ISTRIIUTTINN DATA
We also applied the models pretrained on DCLM to duttof-distribution text without any additional training. We use the GitHub Code Open WebMath (Paster al. 2023), Pile of Law (Henderson et al. 2022), and SkyPile datasets. We report bits per byte in Table 7 Overall, there are no signifi- cant differences between the hierarchial and the tokenizer-based model. However,
also computed inference FLOPs on these -ut-tf-distribution datasets, shown in Table g As in our continued pre- training experiments, see significant advantage for the hierarchical model.
http://///oiiiieee..////aeeeeeer code
23

---

Published as a conference paper at ICLR 2025
Table 7: Bits per byte (bpb) of the DCLM -reet-rrrr models out-of-distribution data without any additional adaptation. The two models perform on par.
Model
GitHuu Code Open WethWahaath Pile of Law |SkyPile
Hicrarchical (Unicode)
0.66
0.81
0.66
1.74
Tokenizer Baseline
0.62
0.81
0.70
1.72
Table 8: Difference in compute required to process intoo--distribution documents, expressed as the ratio between the FLOPs required by the tokenizer baseline and the hierarchical model. The tokenizer-based model requires significantly more compute due to tokenizer fragmentation.
Dataset
FLOP ratio Baseline/Hierrrchiical
Pile of Law
1.30
SkyPile
2.32
Open Web Math
1.28
GitHub Code
1.72
24


Reviews:
[{"review_id": "2IMJpbLsF1", "reviewer": "Reviewer_Kkdq", "summary": "The paper proposed a technique for tokenizer-free language modeling. They design hierarchical processing to aggregate character-level embeddings into word-level embeddings. They achieved similar performance as models with word-level tokenizers while being more robust to input pertubation. They improved the continued pretraining speed when switching domain from English to German while maintaining the performance.", "questions": "- I'm curious how the model works when doing inference in the same language but when the text is from a different distrubtion (out-of-domain).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A method of tokenizer-free langauge modelling that is more robust to input perturbation and input domain shift.", "weaknesses": "- The proposed method only works for character-based languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "osjzELsauD", "reviewer": "Reviewer_KFMw", "summary": "I had wrongly entered the review of another paper - I have now corrected this. I apologise and commend the authors for having tracked down the correct review! \n\nThey propose a hierarchical transformer model which does not rely on a tokeniser to preprocess the dataset into tokens of a fixed sized vocabulary. Instead it proposes an encoder decoder which takes words as a sequence of bytes and encodes and decodes the byte sequences into a word embedding which is then processed by a backbone transformer model. They show that this hierarchical model is theoretically computationally efficient in terms of FLOPs and performs well across a number of tasks. Furthermore they show that it is more robust errors than the baseline which rely on the fixed tokenisation from the original training corpus.", "questions": "Why not discuss in more depth how this work relates to other work on byte/pixel/character based models? This is a fundamental shift in the transformer paradigm and has a lot of implications. More discussion here would have been welcome. Eg. Mielke, Sabrina J., et al. \"Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP.\" Computing Research Repository (arXiv) (2021).\n\nHow would your model cope with languages without whitespace characters as work separations eg. Mandarine? How would your model handle CKY languages with tiny vocabularies, or multilingual models which include CKY and many other scripts?\n\n4.2 It was not clear what the relationship between the backbone and the encoder was. You say you keep the backbone to the 1:1 aspect ratio - what did you mean here? And if you chose the encode to be (8,4) what does this mean for the backbone/encoder? You have 8 heads and 4 layers on the encoder and decoder - but what do you have on the backbone - not clear!\n\nIn table 2 you show compute matched settings for the baseline and you model. They only match in terms of parameters for the first 1.1B model, for 3.1B your model is 4B, and for 7B your model is 9.2B which are both much bigger. Does this mean that they comparisons in Table 1 are unfair on the baseline and overestimate the advantage of your hierarchical model?\n\nThere is no explanation of the test sets in Table 1. In particular what is Lambada and why does the hierarchical model perform so well on it and what does this mean?\n\nI like Figure 4! It clearly shows that the hierarchical model is more robust.\n\nYou do not explain the German Occiglot dataset - why is this considered out of distribution? The Llama model has likely seen quite a bit of German data.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model is well motivated as being a good balance between word and byte level representations. \nThey scale these up to 7B which other hierarchical work did not \nThey show their model is better than models which do not take word boundaries into account", "weaknesses": "They do not situate the work in the long history of hierarchical models, and character and byte based literature. Why is this paper the one that will convince us to move away from sub-word units? The case is not clearly made.\nThere are important details that they do not explain clearly like the dimensions of the backbone model, the test sets and their experimental design. They do not deal with the case of CKY languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MwvsnizFpy", "reviewer": "Reviewer_PbBe", "summary": "The authors propose a method to avoid large lookup table in word-level tokenization by utilizing a small character-level lookup mechanism. They employ a light weight character level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters. Compared with subword tokenization, word-level tokenization with character-level lookup reduces the lengths of sequences processed by the backbone LLM and allows a larger number of parameters under the same computation budget.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. Compared with subword tokenization, word-level tokenization reduces the lengths of sequences processed by the backbone LLM and allows a larger number of parameters under the same computation budget.\n2. Experiments show the proposed method is more robust to input perturbation and cross-lingual continued pretraining.", "weaknesses": "1. **I disagree with the claim that this method is tokenization-free.** Actually, it does employ a word-level tokenization approach. It heavily relies on a space-based tokenization mechanism. The backbone autoregressive model operates on word-level embeddings.\n2. **The primary contribution of this work seems to be more like \"how to avoid a large lookup table in word-level tokenization.\" The proposed method addresses this issue by utilizing a small character-level lookup table.** It aggregates the embeddings of individual characters within a word during encoding and employs an autoregressive model to decode each character from a word-level embedding during inference.\n3. Clearly, this method cannot be applied to languages without using an alphabet system (e.g., Chinese, Japanese) or without a space tokenization mechanism (e.g., Arabic, Thai). Therefore, the adaptation experiments in Section 4.6 is not very convincing, which chooses languages from the same language family.\n4. The contribution appears incremental compared to Megabyte [1]. **While Megabyte employs a fixed patch size, this work utilizes a predefined tokenization mechanism to segment each patch. However, this predefined tokenization approach reduces the flexibility of the method, preventing it from being considered tokenization-free.**\n\n[1]  MEGABYTE: predicting million-byte sequences with multiscale transformers.\n\n**Update**\n\n**I have carefully reviewed the authors' response and acknowledge the experimental results presented in the paper. However, I think the technical content to be highly similar to previously published work [2]. The only difference lies in that [1] uses subword as a higher-level unit but this submission uses word. Both needs predefined segmentation methods. Therefore, I have decided on a final rating of 5.**\n\n[2] Learn Your Tokens: Word-Pooled Tokenization for Language Modeling, EMNLP 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)