Title: SpaceByte: Towards Deleting Tokenization from Large Language Modeling
Venue: NeurIPS 2024
Accepted: yes
Keywords: byte level language model

model architecture

tokenization

efficient pretraining
Abstract: Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity. To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling. SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in the middle of the layers. We find that performance is significantly improved by applying these larger blocks only after certain bytes, such as space characters, which typically denote word boundaries. Our experiments show that for a fixed training and inference compute budget, SpaceByte outperforms other byte-level architectures and roughly matches the performance of tokenized Transformer architectures.

SpaceByte: Towards Deleting Tokenization from Large Language Modeling

Abstract

Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased characterlevel modeling performance, and increased modeling complexity. To address these disadvantages without sacrificing performance, we propose SpaceByte, a novel byte-level decoder architecture that closes the performance gap between byte-level and subword autoregressive language modeling. SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in the middle of the layers. We find that performance is significantly improved by applying these larger blocks only after certain bytes, such as space characters, which typically denote word boundaries. Our experiments show that for a fixed training and inference compute budget, SpaceByte outperforms other byte-level architectures and roughly matches the performance of tokenized Transformer architectures.

1 Introduction

Most language models are trained using tokenization, which partitions text into tokens that typically consist of words or subwords. Tokenization is useful because it significantly decreases the inference and training computational costs of large language models. However, tokenization also imposes several disadvantages, including performance penalties for languages that are priortizes less by the tokenizer [1-3]; increased vulnerability to adversarial attacks [4]; and worse character-level modeling performance [5, 6], and additional model complexity. 1

Recently, MegaByte [7], MambaByte [6], and more [8-12] have been proposed as new byte-level autoregressive language models that model bytes instead of tokens. (See [13-21] for encoder and encoder-decoder byte-level modeling.) To address the longer context size resulting from modeling bytes instead of tokens, MegaByte uses multiscale modeling [22-24], while MambaByte uses Mamba blocks [25] instead of Transformer blocks. But although MegaByte and MambaByte have been shown to perform better than a standard byte-level Transformer, to our knowledge, no byte-level autoregressive large language model architecture has been shown to match the performance of tokenized models when controlling for compute costs.

In this work, we study the performance of byte-level and subword-level autoregressive models when trained using a fixed compute budget. We measure the performance in terms of the cross entropy (measured in bits-per-byte), which has been shown to be a strong predictor of down-stream performance [26]. In addition to controlling for training compute, we also control for inference compute costs (measured in FLOPs). We find that byte-level Transformer and MegaByte models can require roughly 10 times more training FLOPs to achieve the same performance as a subword-

_

s

p

a

c

e

_

i

s

_

_

a

l

l

_

y

s

p

a

c

e

_

i

s

_

a

l

l

embedding

local

(1)

L

⨉

local transformer blocks

local

(2)

L

⨉

local transformer blocks

global

L

⨉

global transformer blocks

de-embedding

level Transformer. To close this substantial performance gap, we propose a new byte-level decoder architecture: SpaceByte.

SpaceByte also utilizes multiscale modeling to improve efficiency by grouping bytes into patches. But unlike MegaByte, which uses a fixed patch size, SpaceByte uses a simple rule to dynamically partition the bytes into patches that are aligned with word and other language boundaries. Our compute-controlled experiments show that this simple modification is crucial for performance, allowing SpaceByte to outperform other byte-level architectures and roughly match the performance of subword Transformers across a variety of text modalities.

Our experiments are performed on datasets consisting of English books, LaTeX formatted arXiv papers, and open-source code. For other data modalities, SpaceByte with our simple patching rule might not be as effective.

2 SpaceByte

The SpaceByte architecture is summarized in Figure 1. In a nutshell, SpaceByte can be thought of as a byte-level Transformer model, but with extra 'global' transformer blocks (with a larger model dimension) inserted in the middle, which are only applied a fraction of the time. While the MegaByte architecture applies the global transformer blocks every P ∼ 8 bytes, we hypothesize that this fixed spacing hinders performance. Our intuition is that the first character of a word is typically significantly harder to predict than the following characters. We therefore expect that performance can be improved by applying the global blocks primarily at word boundaries.

Global Block Insertion Rule In this work, we consider a very simple rule to dynamically decide when to apply the global blocks. We assume that the text bytes are encoded using the UTF-8 encoding. We define a byte to be spacelike if the byte does not encode a letter, number, or UTF-8 continuation byte 2 . We apply the global blocks after any spacelike byte that is not preceded by another spacelike byte (and after any BOS token). See Figure 2 for examples.

The most common spacelike byte is the space character. Thus, the global blocks are applied most frequently to predict the first character of a word, which we expect is the hardest character to predict [27] in a given word. With fixed patch size (e.g. as in MegaByte), the global blocks are typically inserted in the middle a word, which we expect is inefficient because predicting the rest of the word could likely be more efficiently accomplished using the local blocks. We define continuation bytes to be spacelike so that languages that do not use spaces between words can still benefit from the global blocks between multi-byte characters (e.g. Chinese characters consists of three bytes in UTF-8).

PG-19:

Figure 2: Examples of patch boundaries from datasets that we study. Spacelike bytes are underlined and colored blue. Patches boundaries are drawn above the text. Each patch ends after a spacelike byte that is not preceded by another spacelike byte. Consequently, each patch begins with zero or more spacelike bytes, followed by one or more non-spacelike bytes, and ends with a single spacelike byte. The global blocks predict the first character of each patch. The downward arrow ( ↓ ) denotes a newline byte. The left and right quotation characters, (') and (') in the PG-19 example, are encoded using three bytes in UTF-8. The first of the three bytes is spacelike, while the later two bytes are UTF-8 continuation bytes, which are not spacelike and are each denoted using a bullet point (·) above.

Although this very simple 'spacelike' rule is likely not the optimal rule, we find that it works surprisingly well in practice for English text, LaTeX formatted papers, and code. Nevertheless, a critical future direction is to optimize [28, 14, 9] better rules using data rather than our simple heuristic.

Important Details Since the global blocks are not applied as often as the local transformer blocks, it is advantageous to use a larger model dimension for the global transformer blocks. To increase the dimensions of an activation vector before the global blocks, we simply pad the activation vector with zeros. To decrease the dimension, we truncate the activation vector.

In our experiments, we use a significantly larger context size than the model dimension D local of the local transformer blocks. To prevent the attention mechanism from dominating the compute costs for the local model, we use an attention window [29-31] of length D local for the local transformer blocks. The global blocks use a global attention that attends to all other global blocks.

See Appendix C for pseudocode. Additional details specific to our experiments are provided in Sections 4.1 and 4.2 and Appendix A.

3 Related Work

The most straight-forward consequence of modeling bytes instead of subword tokens is that the length of a sequence typically increases by about a factor of four. This increased sequence length increases the training and inference compute cost for modeling a given long sequence of text for a Transformer due to the quadratic scaling of attention.

MegaByte The MegaByte architecture strives to use multiscale Transformer modeling to lessen these performance issues. In particular, MegaByte groups bytes into patches of a fixed patch size P . Each patch of bytes is vectorized and then fed into a 'global' Transformer model. The output of the global model is then fed into a 'local' Transformer model that autoregressively outputs byte-level logits. [7]

For a context size of T bytes, MegaByte's global Transformer model compresses the context into only T/P patches, which can significantly decrease the compute cost for modeling long sequences. Similar to Yu et al. [7], we also find that MegaByte outperforms a standard byte-level Transformer. However, we find that MegaByte's performance is remarkably close to a stronger byte-level Transformer baseline that simply uses a sliding window attention mechanism [29-31] to increase the context size without increasing the compute costs.

Yu et al. [7] do not compare MegaByte to subword-level Transformer in compute controlled experiments. In our compute controlled experiments, we find that MegaByte's performance significantly lags behind a subword-level Transformer.

Compared to MegaByte, SpaceByte makes the crucial change that patches are dynamically sized to be commensurate with the text, e.g. with word boundaries. We also add an additional local model before the global model (while MegaByte only utilizes a single local model after the global model) to help the model deal with the dynamical patch sizes. We also use significantly longer attention windows for our local models. We find that these changes allow SpaceByte to significantly improve upon the performance of MegaByte and roughly match the performance of subword-level Transformers.

MambaByte The MambaByte architecture [6] takes an alternative approach to avoiding the quadratic compute scaling of the attention mechanism by replacing the Transformer block with a Mamba block [25]. Wang et al. [6] find that their byte-level MambaByte models outperform byte-level Transformer and byte-level MegaByte models. They perform one controlled experiment with a subword model, where they find that MambaByte slightly outperforms Mamba (using tokens) when controlling for the amount of model parameters and training data (which was 14 epochs of the PG-19 dataset). But this experiment was not controlled for compute as MambaByte was trained using roughly four times as much compute than Mamba. We view the Mamba and MambaByte architectures as complementary to our work, as the Mamba block could be integrated into SpaceByte (or MegaByte) in place of Transformer blocks.

Layer Skipping SpaceByte could be though of as a Transformer that employs a novel kind of text-dependent layer skipping [32-38] on the middle layers.

Word Boundary Prior works have shown utility in using word boundaries to partition patches for autoregressive multi-scale byte-level modeling [9, 8, 11] (and also [18] for encoder-decoder modeling). However, these works did not compare autoregressive byte-level models to subwordlevel models, nor did they identity a patch partitioning rule that could generically be applied to UTF-8 encoded text. Our primary contributions beyond these prior works is to show how to scale word-boundary byte-level modeling to more diverse text modalities while roughly matching the performance of subword-level models in compute-controlled experiments.

Nawrot et al. [9] and Fleshman and Van Durme [11] make use of the Hourglass Transformer architecture [23]. The SpaceByte architecture is similar to the Hourglass Transformer, except SpaceByte uses a simpler technique for shortening and upscaling the activations before and after the global blocks, and SpaceByte uses a sliding window attention [29-31] in the local blocks to improve performance for long context sizes.

4 Experiment Setup

Our experiments compare the performance of our byte-level SpaceByte architecture to subwordlevel Transformer and byte-level Transformer and MegaByte architectures. To fairly compare the performance between the byte and subword level models, we measure the cross-entropy of the test dataset in terms of bits-per-byte. 3 Given the substantial variation in inference compute costs across the models we study, we also compare their inference compute costs to provide a more comprehensive evaluation. We use FLOPs-per-byte as a simple software and hardware-independent proxy for inference compute costs, which is the number of FLOPs (see Appendix A.1) required to model a byte of text. 4

Note that by controlling for both total training compute and FLOPs-per-byte, we are also controlling for the amount of training data since ( bytes trained ) = ( training FLOPs ) / ( training FLOPs-per-byte ) . The FLOPs-per-byte during training is equal to three times the FLOPs-per-byte during inference (due to the backwards pass during training).

We therefore study the Pareto frontier of lowest bits-per-byte and lowest FLOPs-per-byte. We train all models using a compute-controlled setup, using either 10 18 or 10 19 FLOPs. In order to effectively explore this Pareto frontier, we train models using a grid of different model dimensions and numbers of layers, as specified in Appendix B.3.

Datasets Following the MegaByte [7] and MambaByte [6] experiments, we benchmark our models on a diverse range of long-form datasets: PG-19 (English-language books written before 1919) [41]; arXiv (papers from ArXiv written in LaTeX, extracted from the arXiv component of The Pile [42]); and Github (open-source code repositories, extracted from the Github component of The Pile [42]).

4.1 Models

The models we study tend to perform best when the compute cost is roughly evenly split between the attention and feedforward layers. To ensure this, we fix the context size (or attention window) to be equal to the model dimension for every layer. We detail our model setup below.

Notation For all models, we use T to denote the context length, and D to be the model dimension (of the global model for SpaceByte and MegaByte).

For SpaceByte and MegaByte, D local is the dimension of the local model, and T global is the maximum context size for the global model. The patch size P is the number of bytes between global blocks. If the patch size is fixed (which is always the case for MegaByte), we naturally set the context size to be T = PT global .

Below, we describe each of the model architectures that we compare in our experiments.

SpaceByte We fix the global context size and global model dimension to be equal, T global = D , and we set the local attention window W local equal to the local model dimension, W local = D local. For the PG-19 and arXiv datasets, the average patch size is roughly 6, so we take T = 6 T global for these datasets; for the Github dataset, the average patch size is roughly 8, so we instead take T = 8 T global for the Github dataset.

For simplicity, we fix the number of global transformer blocks L global to be equal to the total number of local blocks, L (1) local + L (2) local , and we evenly split the number of local blocks before ( L (1) local ) and after ( L (2) local ) the global blocks, i.e. we fix L (1) local = L (2) local = 1 2 L global .

SpaceByte (fixed patches) To clearly demonstrate the utility of dynamically aligning patch boundaries in SpaceByte, we also train a simplified version SpaceByte where the patches all have a fixed size. In order to roughly match SpaceByte's average patch size, we take the fixed patch size to be P = 6 for all datasets except for the Github dataset, for which we use P = 8 . We again use T global = D and T = PT global .

Byte-level Transformer For a simple baseline comparison (following Yu et al. [7]), we train bytelevel Transformer models. We take the context size to be equal to the model dimension, T = D .

Note that in our setup, a Transformer with model dimension D only sees a context size of D , which is significantly smaller than the context size of PD for SpaceByte (and MegaByte) with patch size P .

Byte-level Transformer (Window Attention) Since a shorter context is a significant disadvantage for long-form datasets, we also compare against a stronger Transformer baseline that uses a sliding window attention [29-31] to efficiently increase the context size without increasing compute costs. We train each window attention enhanced Transformer using a context size T = PD and a sliding attention window size equal to D , with P = 6 for all datasets except for the Github dataset for which P = 8 . 5

MegaByte We also compare to MegaByte [7]. Although MegaByte was originally trained using a patch size of P = 8 , we found that a patch size of P = 4 was often better in our setup. We thus include both of these patch sizes (4 and 8) in our hyperparameter grid for MegaByte. For simplicity, we fix the number of layers in the global and local blocks to be equal, L global = L local, which is close

to what was used by Yu et al. [7]. Similar to SpaceByte, we set the context size to T = PD , where D is the global model dimension.

Subword Transformers Our most important baseline is the standard subword Transformer. We train subword Transformers using two different tokenizers (both with a vocabulary size of 50,257): (1) the GPT2 tokenizer [43], and (2) a SentencePiece [44] tokenizer using a byte-pair-encoding model [45] that was separately trained for each dataset. As usual, we set the context size to be equal to the model dimension, T = D .

4.2 More Details

We use fairly standard Pre-LN [46, 30, 47] Transformer [48] blocks with no bias terms. Since MegaByte uses Rotary Position Embedding (RoPE) [49], we also use RoPE for all models (which slightly improves the loss). To prevent loss divergences during training, we use qk-layernorm [50-52] (which we strongly recommend) for all models; i.e. we add an extra layer-normalization to the query and key vectors in the self-attention layers.

All hyperparameters have been carefully tuned using grid and random searches. See Appendices A and B for more details. 6

5 Results

We now present our experimental data comparing the different model architectures in computecontrolled settings. Figure 3 plots the Pareto frontier of lowest cross-entropy bits-per-byte and lowest FLOPs-per-byte (i.e. inference compute cost) for each architecture and training compute budget. We assume that the Pareto frontier is convex. Thus, for each architecture and compute budget, we perform a grid search over model dimension and number of layers; we then draw a piecewise-linear line connecting the best (i.e. minimal subset of) models such that all other models (not shown in figure) lie above and to the right of the line. Table 1 summarizes the results for the lowest overall bits-per-byte for each architecture.

Across all datasets, training compute budgets, and inference compute budgets (i.e. FLOPs-per-byte), SpaceByte significantly outperforms all other byte-level architectures. SpaceByte also consistently outperforms the subword Transformer when using GPT2 tokens, and by a wide margin on the arXiv and Github datasets. SpaceByte roughly matches the performance of the most competitive baseline, the subword Transformer using the SentencePiece tokenizer, with SpaceByte performing slightly better on the arXiv and Github datasets. Figure 3 also suggests that SpaceByte's performance improves faster than the subword Transformer as the training compute budget increases.

Byte-level architectures other than SpaceByte perform significantly worse than SpaceByte or the SentencePiece Transformer. For example, for PG-19, the next best byte-level architecture is

10 7

10 8

inference FLOPs-per-byte

1.0

1.1

1.1

1.2

1.2

1.2

1.3

1.4

1.4

cross-entropy (bits-per-byte)

subword-level:

Transformer

(GPT2 tokenizer)

Transformer

(SentencePiece)

byte-level:

Transformer

Transformer

(Window Attention)

MegaByte

SpaceByte

(fixed

P

)

SpaceByte

(a) PG-19 dataset

10 7

10 8

inference FLOPs-per-byte

1.0

0.8

0.9

cross-entropy (bits-per-byte)

subword-level:

Transformer

(GPT2 tokenizer)

Transformer

(SentencePiece)

byte-level:

Transformer

Transformer

(Window Attention)

MegaByte

SpaceByte

(fixed

P

)

SpaceByte

(b) arXiv dataset

10 7

10 8

inference FLOPs-per-byte

1.0

0.5

0.6

0.7

0.8

0.9

cross-entropy (bits-per-byte)

subword-level:

Transformer

(GPT2 tokenizer)

Transformer

(SentencePiece)

byte-level:

Transformer

Transformer

(Window Attention)

MegaByte

SpaceByte

(fixed

P

)

SpaceByte

(c) Github dataset

† These models used slightly different datasets for training and/or testing. For MambaByte-353M, we estimate that this difference very roughly amounts to an extra 3% statistical error.

MegaByte; however, MegaByte trained using 10 19 FLOPs (thick green line in Figure 3a) performs worse across nearly the entire Pareto frontier than the SentencePiece Transformer trained using only 10% as many training FLOPs (thin black line). Although the standard byte-level transformer (which is the primary baseline used by Yu et al. [7], blue in Figure 3) performs significantly worse than the other byte-level models, we note that by simply using a sliding window attention mechanism to increase the context size to more closely match that of the other byte-level models, this stronger baseline (purple) performs almost as well as MegaByte. Nevertheless, SpaceByte still significantly outperforms this stronger baseline.

To verify the importance of dynamic patch sizes for SpaceByte's performance, we compare SpaceByte to a variant of SpaceByte with fixed patch sizes (orange in Figure 3). We observe that fixing the patch size significantly degrades the performance of SpaceByte.

Note that on the arXiv and Github datasets, the subword Transformer performs significantly worse when using GPT2 tokens (which were trained on WebText [43]) than SentencePiece tokens (which were trained using the specific dataset). This exemplifies the bias that tokenization can introduce on data distributions different from what the tokenizer was trained on.

6 Comparison with Other Works

We also compare SpaceByte performance to byte-level models trained in other works. Yu et al. [7] trained Transformer, PerceiverAR, and MegaByte models, each using the same amount of compute, FLOPs-per-byte, and data (80B bytes). Wang et al. [6] additionally trained a MambaByte model using the same FLOPs-per-byte but only 30B bytes of data. We train SpaceByte-793M+184M ( D = 1536 , D local = 768 , L local = 26 , L global = 28 ) using roughly the same inference FLOPs-per-byte (728M) but also only 30B bytes of data (following Wang et al. [6]). Training these models thus requires roughly 3 × 728 MFLOPs-per-byte × 30 B bytes ≈ 6 . 5 × 10 19 FLOPS, where the factor of three comes from converting inference FLOPs-per-byte to training FLOPs-per-byte (which additionally requires a backwards pass). For this experiment, we set the context size of SpaceByte to 8192 bytes to follow the prior works. See Appendix A for more details.

Wealso train subword Transformer-1B ( D = 1536 ) models using the SentencePiece tokenizer (except for the Stories dataset, for which we use the GPT2 tokenizer). The average number of bytes per token

for the PG-19, Stories, arXiv, and Github datasets are 4.05, 4.39, 3.73, and 3.31, respectively. To match the FLOPs-per-byte of the subword Transformer-1B models to the byte-level models, we set the number of layers to 40, 44, 37, or 31, for Transformer-1B on these four respective datasets.

Results are shown in Table 2. We show experiments for the PG-19 [41], Stories [53], arXiv (extracted from The Pile [42]), and Github (extracted from The Pile [42]) datasets. 7 Yu et al. [7] used proprietary 'arXiv' and 'Code' datasets, which we do not have access to. Following Wang et al. [6], we compare Yu et al. [7]'s results to the similar (but likely slightly different) arXiv and Github components of The Pile [42]. However, Wang et al. [6] use their own test splits to evaluate MambaByte-353M on Stories, arXiv, and Github. Due to the rather small test splits ( ∼ 100 MBfor the arXiv and Github datasets), this difference can be significant. For example, the validation (and test) bits-per-byte for SpaceByte-793M+184M on the Stories, arXiv, and Github datasets are 0.877 (0.833), 0.658 (0.663) and 0.397 (0.411), which differ by +5% , -1% , and -3% , respectively. Given this variation, the bits-per-byte of MambaByte-353M and SpaceByte-793M+184M are not statistically different on the arXiv or Github datasets.

Overall, we find that SpaceByte outperforms the byte-level models trained in other works. SpaceByte outperforms MegaByte, even though MegaByte was trained using 2.7 times as much compute and data. Moreover, SpaceByte's performance is competitive with the subword Transformer-1B.

7 Conclusion

We have proposed a new byte-level Transformer decoder architecture, SpaceByte. Our computecontrolled experiments show that SpaceByte outperforms all other byte-level architectures and roughly matches the performance of sub-word level Transformers.

Limitations SpaceByte uses a simple byte partitioning rule that relies on 'spacelike' bytes, such as spaces which typically denote word boundaries. As such, SpaceByte should not be expected to perform well on arbitrary sequences of bytes, such as images or audio. Some languages, such as Chinese, do not use spaces between words. SpaceByte is somewhat robust to these languages, since e.g. Chinese characters are encoded using three bytes in UTF-8, which SpaceByte will group together. However, our preliminary experiments suggest that SpaceByte performs worse than subword transformers on Chinese text. It would therefore be desirable to improve upon and generalize SpaceByte's global block insertion rule.

The variable spacing between global blocks makes it more challenging to design and implement an efficient batched inference sampling algorithm for SpaceByte.

Future Work SpaceByte uses multiscale modeling where the local model operates on bytes while the global model typically operates on words. Another natural extension of our work is to try recursively applying multiscale modeling at even longer scales, such as the sentence or paragraph level. It would also be fruitful to investigate if Mamba blocks [25] could further improve SpaceByte's performance.

Acknowledgments and Disclosure of Funding

We thank Tushaar Gangavarapu, Junxiong Wang, and Lili Yu for helpful conversations. This work was supported in part by the NSF Campus Cyberinfrastructure grant CC* Compute: Interactive Data Analysis Platform OAC-2019007 and by Rice University's Center for Research Computing (CRC).

References

J. Pourmostafa Roshan Sharami, D. Shterionov, and P. Spronck. A Systematic Analysis of Vocabulary and BPE Settings for Optimal Fine-tuning of NMT: A Case Study of In-domain Translation. March 2023. doi: 10.48550/arXiv.2303.00722.

Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=78yDLKi95p .

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 9904-9923, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.614. URL https://aclanthology.org/2023.emnlp-main.614 .

Jessica Rumbelow and mwatkins. Solidgoldmagikarp (plus, prompt generation), 2023. URL https://www. lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation .

Peter Welinder, May 2022. URL https://twitter.com/npew/status/1525900849888866307 .

Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. MambaByte: Token-free Selective State Space Model. January 2024. doi: 10.48550/arXiv.2401.13660.

Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MegaByte: Predicting Million-byte Sequences with Multiscale Transformers. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=JTmO2V9Xpz .

Avijit Thawani, Saurabh Ghanekar, Xiaoyuan Zhu, and Jay Pujara. Learn your tokens: Word-pooled tokenization for language modeling. In Conference on Empirical Methods in Natural Language Processing , 2023. URL https://openreview.net/forum?id=O9zrG7NB3X .

Piotr Nawrot, Jan Chorowski, Adrian Ła´ ncucki, and Edoardo M. Ponti. Efficient Transformers with Dynamic Token Pooling. art. arXiv:2211.09761, November 2022. doi: 10.48550/arXiv.2211.09761.

Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, and Noah Constant. Training LLMs over Neurally Compressed Text. art. arXiv:2404.03626, April 2024.

William Fleshman and Benjamin Van Durme. Toucan: Token-Aware Character Level Language Modeling. arXiv e-prints , art. arXiv:2311.08620, November 2023. doi: 10.48550/arXiv.2311.08620.

Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, and Luke Zettlemoyer. MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling. arXiv e-prints , art. arXiv:2403.10691, March 2024. doi: 10.48550/arXiv.2403.10691.

Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics , 10:291-306, 2022. doi: 10.1162/tacl_a_00461. URL https://aclanthology.org/2022.tacl-1.17 .

Nathan Godey, Roman Castagné, Éric de la Clergerie, and Benoît Sagot. MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling. art. arXiv:2212.07284, December 2022. doi: 10.48550/arXiv.2212.07284.

Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. art. arXiv:2103.06874, March 2021. doi: 10.48550/arXiv.2103.06874.

Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun'ichi Tsujii. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In International Committee on Computational Linguistics , pages 69036915, Barcelona, Spain (Online), December 2020. doi: 10.18653/v1/2020.coling-main.609. URL https://aclanthology.org/2020.coling-main.609 .

Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. art. arXiv:2106.12672, June 2021. doi: 10.48550/arXiv.2106.12672.

Lukas Edman, Antonio Toral, and Gertjan van Noord. Subword-Delimited Downsampling for Better Character-Level Translation. art. arXiv:2212.01304, December 2022. doi: 10.48550/arXiv.2212.01304.

Makesh Narsimhan Sreedhar, Xiangpeng Wan, Yu Cheng, and Junjie Hu. Local byte fusion for neural machine translation. In Association for Computational Linguistics , pages 7199-7214, Toronto, Canada, July 2023. doi: 10.18653/v1/2023.acl-long.397. URL https://aclanthology.org/2023.acl-long.397 .

Lukas Edman, Gabriele Sarti, Antonio Toral, Gertjan van Noord, and Arianna Bisazza. Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation. art. arXiv:2302.14220, February 2023. doi: 10.48550/arXiv.2302.14220.

Kris Cao. What is the best recipe for character-level encoder-only modelling? art. arXiv:2305.05461, May 2023. doi: 10.48550/arXiv.2305.05461.

Sandeep Subramanian, Ronan Collobert, Marc'Aurelio Ranzato, and Y-Lan Boureau. Multi-scale Transformer Language Models. art. arXiv:2005.00581, May 2020. doi: 10.48550/arXiv.2005.00581.

Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Association for Computational Linguistics , pages 1559-1571, Seattle, United States, July 2022. doi: 10.18653/v1/2022. findings-naacl.117. URL https://aclanthology.org/2022.findings-naacl.117 .

Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V. Le. Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing. art. arXiv:2006.03236, June 2020. doi: 10.48550/arXiv. 2006.03236.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://openreview.net/forum?id=AL1fq05o7H .

Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression Represents Intelligence Linearly. art. arXiv:2404.09937, April 2024. doi: 10.48550/arXiv.2404.09937.

Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. AAAI, 2019. doi: 10.1609/aaai.v33i01.33013159. URL https: //doi.org/10.1609/aaai.v33i01.33013159 .

Md Mofijul Islam, Gustavo Aguilar, Pragaash Ponnusamy, Clint Solomon Mathialagan, Chengyuan Ma, and Chenlei Guo. A vocabulary-free multilingual neural tokenizer for end-to-end task learning. Association for Computational Linguistics, May 2022. doi: 10.18653/v1/2022.repl4nlp-1.10. URL https://aclanthology.org/2022.repl4nlp-1.10 .

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long-Document Transformer. April 2020. doi: 10.48550/arXiv.2004.05150.

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating Long Sequences with Sparse Transformers. April 2019. doi: 10.48550/arXiv.1904.10509.

Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Association for Computational Linguistics , pages 5493-5505, August 2021. doi: 10.18653/v1/2021. acl-long.427. URL https://aclanthology.org/2021.acl-long.427 .

Alex Graves. Adaptive Computation Time for Recurrent Neural Networks. art. arXiv:1603.08983, March 2016. doi: 10.48550/arXiv.1603.08983.

Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. SkipNet: Learning Dynamic Routing in Convolutional Networks. art. arXiv:1711.09485, November 2017. doi: 10.48550/arXiv.1711. 09485.

Andreas Veit and Serge Belongie. Convolutional Networks with Adaptive Inference Graphs. art. arXiv:1711.11503, November 2017. doi: 10.48550/arXiv.1711.11503.

Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogerio Feris. BlockDrop: Dynamic Inference Paths in Residual Networks. art. arXiv:1711.08393, November 2017. doi: 10.48550/arXiv.1711.08393.

Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. In Association for Computational Linguistics , pages 4962-4979, Online and Punta Cana, Dominican Republic, November 2021. doi: 10.18653/v1/2021.emnlp-main.406. URL https://aclanthology.org/2021.emnlp-main.406 .

Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic Neural Networks: A Survey. art. arXiv:2102.04906, February 2021. doi: 10.48550/arXiv.2102.04906.

David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. art. arXiv:2404.02258, April 2024.

Noam Shazeer. Fast Transformer Decoding: One Write-Head is All You Need. art. arXiv:1911.02150, November 2019. doi: 10.48550/arXiv.1911.02150.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Association for Computational Linguistics , pages 4895-4901, Singapore, December 2023. doi: 10.18653/v1/2023.emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298 .

Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=SylKikSYDH .

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. December 2020. doi: 10.48550/arXiv.2101.00027.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. URL https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_multitask_learners.pdf .

Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Association for Computational Linguistics , pages 66-71, nov 2018. doi: 10.18653/v1/D18-2012. URL https://aclanthology.org/D18-2012 .

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Association for Computational Linguistics , pages 1715-1725, August 2016. doi: 10.18653/v1/P16-1162. URL https://aclanthology.org/P16-1162 .

Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=ByxZX20qFQ .

Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Association for Computational Linguistics , pages 1810-1822, July 2019. doi: 10.18653/v1/P19-1176. URL https://aclanthology.org/P19-1176 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30, 2017.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. April 2021. doi: 10.48550/arXiv.2104.09864.

Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4246-4253, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.379. URL https://aclanthology.org/2020.findings-emnlp.379 .

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Paveti´ c, Dustin Tran, Thomas Kipf, Mario Luˇ ci´ c, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling Vision Transformers to 22 Billion Parameters. February 2023. doi: 10.48550/arXiv.2302.05442.

Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D CoReyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=d8w0pmvXbZ .

Trieu H. Trinh and Quoc V. Le. A Simple Method for Commonsense Reasoning. art. arXiv:1806.02847, June 2018. doi: 10.48550/arXiv.1806.02847.

Ofir Press and Lior Wolf. Using the Output Embedding to Improve Language Models. August 2016. doi: 10.48550/arXiv.1608.05859.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 .

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training Recurrent Neural Networks. art. arXiv:1211.5063, November 2012. doi: 10.48550/arXiv.1211.5063.

Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memoryefficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=H4DqfPSibmx .

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=mZn2Xyh9Ec .

Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. The Depth-to-Width Interplay in Self-Attention. art. arXiv:2006.12467, June 2020. doi: 10.48550/arXiv.2006.12467.

A Model Details

Hyperparameters for models shown in Table 1 and Table 2 are summarized in Table 3 and Table 4, respectively. In Figure 4, we show another perspective of Figure 3 where we plot the bits-per-byte vs the bytes trained divided by the number of model parameters.

For the subword models (but not the byte-level models), we tie the input embedding weights with the output linear matrix weights [54]. In self-attention layers, we use a key dimension equal to 64. Although we apply RoPE embeddings, we also included trained position embeddings.

SpaceByte For SpaceByte, we include trained position embeddings just before the first local transformer block, and just before the first global transformer block.

Just like the other models, SpaceByte is trained using a fixed context size of T bytes. At the same time, we also fix the maximum global context size of T global patches. However, the number of patches in a given context of T bytes is usually not exactly equal to T global. To handle this mismatch during training, if the number of patches from applying the patching rule (see e.g. Figure 2) to a context of T bytes is greater T global, then we simply ignore the bytes within these extra patches when calculating the cross entropy. Alternatively, if the number of patches is less than T global, then we pad the activations for the global transformer blocks with zeroes and ignore the output of these global blocks. Thus, the input activations to the global blocks is always a tensor of same shape for each iteration. This discrepancy between the maximal global context size T global and the actual number of patches results in a small fracton of wasted compute during training, which we roughly minimize by roughly tuning T/T global. See Appendix C for pseudocode.

During inference, the model must stop predicting tokens before either the max number of bytes ( T ) or the max number of patches ( T global) is reached.

A.1 FLOPs

The inference FLOPs-per-byte is the number of FLOPs required to output a byte of text during inference. We calculate the FLOPs-per-byte as the FLOPs-per-token divided by the average number of bytes per token (which is equal to 1 for byte-level models).

The FLOPs-per-token is the number of FLOPs required to output a token of text during inference (or byte of text for byte-level models). The FLOPs-per-token for the various architectures is shown in Table 6.

Notation For all architectures, T is the context length; D is the model dimension (of the global model for SpaceByte and MegaByte); e ff = 4 is the model dimension expansion factor for feed-forward layers; and V is the vocabulary size (which is 256 for byte-level models and 50257 for our subword models). For the transformer architecture, L is the number of transformer blocks, and W is the attention window size (which is equal to T if window attention is not used). For SpaceByte and MegaByte, D local is the dimension of the local model; L local is the number of local transformer blocks; and L global is the number of global transformer blocks. For SpaceByte, T global is the maximum context size for the global model, and W local (which we set to D local) is the attention window size for the local blocks. For MegaByte, P is the patch size.

B Training Details

B.1 Data

Each dataset prepared by downloaded it from Hugging Face 8 , concatenating sequences together, and separating sequences with a special BOS token. When preparing a training sample with context size T , we uniformly and randomly sample a sub-sequence of length T from the concatenated dataset. If a BOS token is found in this subset, we align the context with the first BOS token found; i.e. we take the context to be the first BOS token followed by the next T -1 tokens in the concatenated dataset. If a BOS token is not found in the subset, we prepend a BOS token to the context. The context window is always full and always begins with a BOS token.

For the SpaceByte models, we always insert global blocks after a BOS token. A valid UTF-8 encoding never makes use of the byte values 254 or 255. We use 255 to encode the BOS token.

B.2 Training

All models are trained using AdamW [55] with β 1 = 0 . 9 , β 2 = 0 . 98 , batch size 64, weight decay of 0.01, and gradient clipping [56] with a maximum norm of 1.0. Trainable parameters are randomly initialized using a normal distribution with standard deviation σ init = 1 for all parameters except for linear weight matrices, which are initialized with standard deviation of σ init = 1 / √ d in, where d in is the input dimension for the linear layer. We scale the learning rate for each parameter by its initialization standard deviation σ init .

With this setup, we found in our early hyperparameter search experiments that the optimal max learning rate for all models is approximately γ = 0 . 005 B -0 . 5 = 0 . 000625 , where B = 64 is the batch size. We therefore used γ = 0 . 000625 as the max learning rate for all models trained in this work. We applied a linear learning rate warmup over the first 1% of training iterations. We also multiply the learning rate by a 'half-cosine' learning rate decay function cos( πx/ 2) , where 0 ≤ x ≤ 1 is the fraction of training iterations completed. 9

Each model was trained using PyTorch on a single 40GB Nvidia A40 and A100 GPUs with mixedprecision (bfloat16 and float32) training and FlashAttention [57, 58]. SpaceByte-793M+184M took the longest to train, requiring about 10 days on an A100 GPU. 10

10 2

10 3

trained bytes / parameters (non-embedding)

1.0

1.1

1.1

1.2

1.2

1.2

1.3

1.4

1.4

cross-entropy (bits-per-byte)

subword-level:

Transformer

(GPT2 tokenizer)

Transformer

(SentencePiece)

byte-level:

Transformer

Transformer

(Window Attention)

MegaByte

SpaceByte

(fixed

P

)

SpaceByte

(a) PG-19 dataset

10 1

10 2

10 3

trained bytes / parameters (non-embedding)

1.0

0.8

0.9

cross-entropy (bits-per-byte)

subword-level:

Transformer

(GPT2 tokenizer)

Transformer

(SentencePiece)

byte-level:

Transformer

Transformer

(Window Attention)

MegaByte

SpaceByte

(fixed

P

)

SpaceByte

(b) arXiv dataset

10 1

10 2

10 3

trained bytes / parameters (non-embedding)

1.0

0.5

0.6

0.7

0.8

0.9

cross-entropy (bits-per-byte)

subword-level:

Transformer

(GPT2 tokenizer)

Transformer

(SentencePiece)

byte-level:

Transformer

Transformer

(Window Attention)

MegaByte

SpaceByte

(fixed

P

)

SpaceByte

(c) Github dataset

B.3 Hyperparameter Grid

We train models using a grid of different model dimensions and numbers of layers. In our early small-scale experiments, we found that the hyperparameter grid described below effectively explores the bits-per-byte and FLOPs-per-byte Pareto frontier for all models. To simplify the hyperparameter grid, we restrict ourselves to model dimensions and layer numbers to half-powers of two, i.e. a power of two times 1 or 3 2 .

For models trained using 10 18 FLOPs, we train model dimensions D ∈ { 384 , 512 , 768 } . For models trained using 10 19 FLOPs, we train model dimensions D ∈ { 512 , 768 , 1024 } .

For SpaceByte and MegaByte, D is the global model dimension. The local model dimension is chosen from D local ∈ { 1 2 D, 3 4 D } if D is a power of two, or D local ∈ { 1 2 D, 2 3 D } if D is a power of two times 3 2 . However, in order to avoid excessively low FLOP utilization, we restrict D local ≥ 256 (or D local ≥ 384 ) for models trained using 10 18 FLOPs (or 10 19 FLOPs).

To set the number of layers, we roughly follow Levine et al. [59], who found that the compute-optimal number of layers for a Transformer roughly follows L ∼ 12 . 5 log 2 ( D/ 154) . We round this number to the nearest half-power of two to obtain L D , for which L 384 = 16 , L 512 = 24 , L 768 = 32 , and L 1024 = 32 . For Transformer models, we choose the number of layers from L ∈ { 1 2 L D , L D } .

For SpaceByte and MegaByte models, we choose the number of local and global layers from L local = L global ∈ { 3 8 L D , 1 2 L D } if L D is a power of two, or L local = L global ∈ { 1 3 L D , 1 2 L D } if L D is a power of two times 3 2 .

C Pseudocode

See Listing 1 for Pytorch pseudocode for the SpaceByte forward method. The implementation of SpaceByte that we used in our experiments can be found at github.com/kjslag/spacebyte.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims are summarized in the abstract, and restated in the introduction. Our abstract makes three claims, which we enumerate in the following quote:

'SpaceByte consists of a byte-level Transformer model, but with extra larger transformer blocks inserted in the middle of the layers. (1) We find that performance is significantly improved by applying these larger blocks only after certain bytes, such as space characters, which typically denote word boundaries. (2) Our experiments show that for a fixed training and inference compute budget, SpaceByte outperforms other byte-level architectures and (3) roughly matches the performance of tokenized Transformer architectures.'

These claims are substantiated in Sections 5 and 6. The first claim is supported by Figure 3 and Table 1, which compare SpaceByte to 'SpaceByte (fixed P ),' which is a variant of SpaceByte that uses fixed patch sizes instead of applying the larger blocks only after certain bytes. The final two claims are also supported by Figure 3 and Table 1, along with Table 2. At the end of our introduction, we note that our results could be limited to the text modalities (English text, LaTex formatted arXiv papers, and source code) that we study.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: At the end of our introduction, we note that our results could be limited to the text modalities (English text, LaTex formatted arXiv papers, and source code) that we study. We emphasize in Section 2 and our conclusion that generalizing SpaceByte's performance to other text modalities (by improving the global block insertion rule) is an important direction for future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be

used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical results were presented.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Open source code, job execution scripts, and a jupyter notebook for fully reproducing our results are available at github.com/kjslag/spacebyte.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Open access to code and thorough instructions for reproducing our experiments are available at github.com/kjslag/spacebyte. This includes code for the models, training, and bits-per-byte evaluation, and a requirements.txt for pip.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setting is described in Section 4. The models studied are summarized in Sections 4.1, 4.2, and 6. Additional model details are provided in Appendix A. Training details are provided in Appendix B. Any additional details are available in our released source code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Tables 1 and 2 give upper bounds for the 1-sigma statistical error and clarify that these statistical errors account for the finite number of evaluation samples. We also estimate the statistical uncertainty due to the different test splits used by other works in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resource requirements are detailed at the end of Appendix B.2, along with an additional compute estimate for preliminary and failed experiments (in a footnote). Models trained in our main experiments used 10 18 or 10 19 FLOPs for training, as specified in Section 4. When comparing to other works, training each model required 6 . 5 × 10 19 FLOPs, as specified in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Regarding the data-related concerns, we omit comparisons to the Books3 dataset since this dataset is no longer publically available and because it is unclear to us if Books3 is legal under fair use, as stated in Footnote 7.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We briefly mention in the abstract and introduction that tokenization can increase the vulnerability to adversarial attacks. Lessening this adversarial vulnerability was one of our motivations. We are not aware of any potential negative societal impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We include citations to all datasets that we use. We believe that we have properly respected the licenses and terms of use for all code and datasets that we used. Notably, we avoided the Books3 dataset for this reason, since Books3 may not be respecting the licenses of the works from which it derives. Code attributions and licenses are provided in the 'license' directory of our released source code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only asset introduced is source code. We provide detailed instructions for using this code to reproduce our results in the 'reproduce' directory of the source code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: Human subjects were not involved in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: Human subjects were not involved in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a byte-level architecture called SpaceByte that involves local blocks (lower dimension, windowed attention) and global blocks (higher dimension, global attention) where the global blocks are between chunks of local blocks and only selectively applied. The global blocks are applied to \"spacelike\" characters (such as an actual space character), with the intuition being that at such points predicting the next character (like start of word) would be harder. The authors run extensive experiments to show that their proposed architecture outperforms previous byte-level architectures and in fact performs similarly to subword-level transformer baselines.\n\nSTRENGTHS:\n**Originality**: While this work seems to share much in common with past byte-level architecture work, the intuitive idea of some bytes being harder to predict (in a rule predictable way) is very natural and useful (and first presented in this work, as far as I am aware). The architecture itself is also novel.\n\n**Quality**: The authors run a lot of experiments to demonstrate the capabilities of their architecture.\n\n**Clarity**: The paper was clear, down-to-earth, and easy to understand.\n\n**Significance**: This paper shows that byte-level architectures can be competitive with subword tokenization transformers in a flop-matched setting.\n\nWEAKNESSES:\n* As a main weakness I'd say that even though this method beats other byte-level approaches and is comparable to subword tokenization, I don't see a compelling reason to use it over standard tokenization. In the Introduction the authors mention several downsides of tokenization, but don't have results that focus on these issues per se. I'm also a bit confused about the \"additional modeling complexity (line 20)\" point. It seems like tokenization is actually simpler.\n* It would have been nice to see some ablations on the architecture (for example, the local -> global -> local layer ordering seems a little arbitrary).\n* It would have been nice to see some modifications of the global rule. For example, only \"space\" characters like actual space and newline vs the more broad definition of \"space\" used in the paper.\n* It might be nice to include more info on what the \"spacelike\" tokens actually look like (what are the most common \"spacelike\" tokens and what percentage of all \"spacelike\" tokens do these make up).\n\nQUESTIONS:\nPlease see weaknesses for my questions.\n\nLIMITATIONS:\nYes. The paper does a good job of this.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SpaceByte, a byte-level Transformer model that incorporates larger transformer blocks at specific byte boundaries to enhance performance in language modeling tasks. While the approach bears similarities to previous work (e.g. MegaByte), it demonstrates improved performance over traditional tokenization models. The study focuses on specific languages, acknowledging limitations in generalizability, particularly in languages like Chinese, which do not use spaces between words.\n\nSTRENGTHS:\n1. The paper presents a novel approach in utilizing byte-level Transformer models with specific block insertions to enhance language modeling performance.\n\n2. The study provides insights into the limitations of tokenization models and the potential benefits of byte-level architectures.\n\n3. The experimental methodology is well-documented, controlling for compute costs and providing detailed training details in the appendices.\n\nWEAKNESSES:\n1. The novelty of the SpaceByte approach may be limited by similarities to previous models.\n\n2. The study's focus on specific languages, without detailed experiments on broader language datasets, limits the generalizability of the findings.\n\n3. The paper could benefit from a more extensive discussion on the unique contributions of SpaceByte compared to existing models.\n\nQUESTIONS:\n1. How do the authors plan to address the limitations in generalizability to languages beyond the ones studied in the current work?\n\n2. Are there plans to conduct experiments on a more diverse set of language datasets to further validate the effectiveness of SpaceByte in different linguistic contexts?\n\nLIMITATIONS:\nThe authors have acknowledged the limitations of their work, particularly in terms of language generalizability. To enhance the impact of the study, it would be beneficial for the authors to consider conducting experiments on a broader range of language datasets to strengthen the validity and applicability of SpaceByte across various linguistic domains.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nByte-level modeling allows transformers to circumvent subword tokenization, thus avoiding the many weaknesses introduced by tokenization. However those models are not very performant compared to subword-based transformers. This paper proposes a novel architecture named *SpaceByte*. The idea is to add an extra wide layer (i.e. \"global block\") between regular transformer layers (i.e. \"local block\") if and only if \"the byte does not encode a letter, number, or UTF-8 continuation byte\". Results show that when normalizing for training FLOPs, SpaceByte achieves the best PPL on multiple datasets. Plus, at different model dimensions and number of layers (?), SpaceByte always achieve the optimal or near-optimal Pareto optimality between PPL and inference FLOPs-per-byte.\n\nSTRENGTHS:\n1. The idea is simple and well-motivated\n2. Experiment results show that it is competitive when compared subword-based models\n\nWEAKNESSES:\n1. While normalizing for FLOPs, I think there is some potential that the proposal will make the transformer blocks significantly harder to batch (due to the input-dependent dynamic structure of the model), hence the clock-time of the inference might actually be slower with the same FLOPs. It would be good to see some discussions on that.\n2. There is no ablations as to whether the global blocks actually need larger dimension.\n3. Presentation of the paper could be improved. See my question/suggestion in the next section.\n\nQUESTIONS:\n### Presentation Suggestions\n\n1. It's worth explaining more about how context lengths work differently for global vs. local blocks in Section 2. I'm also not very sure why the larger context length is necessary.\n2. I'd suggest switching Section 2 and 3 and talk about related work first.\n3. Three suggestions for Figure 3 -- a. make it clear in the caption that you are changing inference FLOPs with different model dimensions & layers (per line 206); b. use dashed/solid lines rather than thin/thick lines; c. mark the pareto frontier for each subfigures.\n\n### Questions\n\n1. Is the FLOP limits you imposed enough to make the models converge adequately on those datasets? It might have been the case that your model simply converges faster, but if trained longer, doesn't work as well as, e.g. sub-word models (which weakens your case).\n2. Why set the dimension and context length equal?\n3. I'm very confused what's the difference between section 6 and 5. How does the results in section 6 further complements the one in section 5?\n4. In Table 2, you only have one context size, but don't global blocks have larger context size? What context size is being reported then?\n\nLIMITATIONS:\nSee weakness point 1. I'm relatively confident that this could be a potential limitation that needs to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper proposes SpaceByte, a byte-level decoder architecture for language modeling. As opposed to comparable models such as MegaByte, SpaceByte applies global transformer blocks after space-like characters, not after patches of a fixed size. The authors show that this approach leads to a substantially improved performance: compared to several other byte-level decoder architectures (including MegaByte), SpaceByte is the only one that matches or even exceeds the performance of subword-level models trained with the same compute budget.

STRENGTHS:
The proposed SpaceByte architecture is novel. The experimental setup is rigorous --- I think the authors did a great job in (a) evaluating a range of different architectures and (b) ensuring a fair comparison by controlling compute costs and using bits-per-byte as the evaluation measure. The results show clear performance improvements for SpaceByte, highlighting its advantages compared to other byte-level decoder architectures. Overall, I liked the paper very much and think that it should be accepted.

WEAKNESSES:
The main weakness that I see is that the authors only evaluate the different architectures using bits-per-byte, not any downstream task. To become a real alternative to subword-level models in practice, it would be important to show that the similar language modeling performance translates to a similar downstream task performance. While the authors cite work indicating that this might be the case (Huang et al., 2024), without actual experiments it is unclear whether this holds for the examined architectures as well.

QUESTIONS:
Is there any specific reason you did not include evaluations on downstream tasks?

LIMITATIONS:
Yes, the authors discuss limitations as part of the conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-16