Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models

Authors: Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg, Lukas Balles
Primary area: foundation or frontier models, including LLMs
Keywords: transformer, autoregressive, generative, language modelling, tokenizer-free, byte-level, hierarchical

Abstract:
Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the de-facto standard, they present challenges such as large vocabularies, limited adaptability to new domains or languages, and sensitivity to spelling errors and variations. To overcome these limitations, we investigate a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. It employs a lightweight character-level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters via a compact character-level decoder. This method retains the sequence compression benefits of word-level tokenization without relying on a rigid, predefined vocabulary. We demonstrate, at scales up to 7 billion parameters, that hierarchical transformers match the downstream task performance of subword-tokenizer-based models while exhibiting significantly greater robustness to input perturbations. Additionally, during continued pretraining on an out-of-domain language, our model trains almost twice as fast, achieves superior performance on the target language, and retains more of its previously learned knowledge. Hierarchical transformers pave the way for NLP systems that are more robust, flexible, and generalizable across languages and domains.

Decision: accept (poster)
Reviewer scores: 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1730654495375

thread_type:
official_review

<thread_object>
creator: reviewer_pbbe

<content>
summary:
The authors propose a method to avoid large lookup table in word-level tokenization by utilizing a small character-level lookup mechanism. They employ a light weight character level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters. Compared with subword tokenization, word-level tokenization with character-level lookup reduces the lengths of sequences processed by the backbone LLM and allows a larger number of parameters under the same computation budget.

strengths:
1. Compared with subword tokenization, word-level tokenization reduces the lengths of sequences processed by the backbone LLM and allows a larger number of parameters under the same computation budget.
2. Experiments show the proposed method is more robust to input perturbation and cross-lingual continued pretraining.

weaknesses:
1. **I disagree with the claim that this method is tokenization-free.** Actually, it does employ a word-level tokenization approach. It heavily relies on a space-based tokenization mechanism. The backbone autoregressive model operates on word-level embeddings.
2. **The primary contribution of this work seems to be more like "how to avoid a large lookup table in word-level tokenization." The proposed method addresses this issue by utilizing a small character-level lookup table.** It aggregates the embeddings of individual characters within a word during encoding and employs an autoregressive model to decode each character from a word-level embedding during inference.
3. Clearly, this method cannot be applied to languages without using an alphabet system (e.g., Chinese, Japanese) or without a space tokenization mechanism (e.g., Arabic, Thai). Therefore, the adaptation experiments in Section 4.6 is not very convincing, which chooses languages from the same language family.
4. The contribution appears incremental compared to Megabyte [1]. **While Megabyte employs a fixed patch size, this work utilizes a predefined tokenization mechanism to segment each patch. However, this predefined tokenization approach reduces the flexibility of the method, preventing it from being considered tokenization-free.**

[1]  MEGABYTE: predicting million-byte sequences with multiscale transformers.

**Update**

**I have carefully reviewed the authors' response and acknowledge the experimental results presented in the paper. However, I think the technical content to be highly similar to previously published work [2]. The only difference lies in that [1] uses subword as a higher-level unit but this submission uses word. Both needs predefined segmentation methods. Therefore, I have decided on a final rating of 5.**

[2] Learn Your Tokens: Word-Pooled Tokenization for Language Modeling, EMNLP 2023

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730750934641

thread_type:
official_review

<thread_object>
creator: reviewer_kfmw

<content>
summary:
I had wrongly entered the review of another paper - I have now corrected this. I apologise and commend the authors for having tracked down the correct review! 

They propose a hierarchical transformer model which does not rely on a tokeniser to preprocess the dataset into tokens of a fixed sized vocabulary. Instead it proposes an encoder decoder which takes words as a sequence of bytes and encodes and decodes the byte sequences into a word embedding which is then processed by a backbone transformer model. They show that this hierarchical model is theoretically computationally efficient in terms of FLOPs and performs well across a number of tasks. Furthermore they show that it is more robust errors than the baseline which rely on the fixed tokenisation from the original training corpus.

strengths:
The model is well motivated as being a good balance between word and byte level representations. 
They scale these up to 7B which other hierarchical work did not 
They show their model is better than models which do not take word boundaries into account

weaknesses:
They do not situate the work in the long history of hierarchical models, and character and byte based literature. Why is this paper the one that will convince us to move away from sub-word units? The case is not clearly made.
There are important details that they do not explain clearly like the dimensions of the backbone model, the test sets and their experimental design. They do not deal with the case of CKY languages.

questions:
Why not discuss in more depth how this work relates to other work on byte/pixel/character based models? This is a fundamental shift in the transformer paradigm and has a lot of implications. More discussion here would have been welcome. Eg. Mielke, Sabrina J., et al. "Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP." Computing Research Repository (arXiv) (2021).

How would your model cope with languages without whitespace characters as work separations eg. Mandarine? How would your model handle CKY languages with tiny vocabularies, or multilingual models which include CKY and many other scripts?

4.2 It was not clear what the relationship between the backbone and the encoder was. You say you keep the backbone to the 1:1 aspect ratio - what did you mean here? And if you chose the encode to be (8,4) what does this mean for the backbone/encoder? You have 8 heads and 4 layers on the encoder and decoder - but what do you have on the backbone - not clear!

In table 2 you show compute matched settings for the baseline and you model. They only match in terms of parameters for the first 1.1B model, for 3.1B your model is 4B, and for 7B your model is 9.2B which are both much bigger. Does this mean that they comparisons in Table 1 are unfair on the baseline and overestimate the advantage of your hierarchical model?

There is no explanation of the test sets in Table 1. In particular what is Lambada and why does the hierarchical model perform so well on it and what does this mean?

I like Figure 4! It clearly shows that the hierarchical model is more robust.

You do not explain the German Occiglot dataset - why is this considered out of distribution? The Llama model has likely seen quite a bit of German data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730757413390

thread_type:
official_review

<thread_object>
creator: reviewer_kkdq

<content>
summary:
The paper proposed a technique for tokenizer-free language modeling. They design hierarchical processing to aggregate character-level embeddings into word-level embeddings. They achieved similar performance as models with word-level tokenizers while being more robust to input pertubation. They improved the continued pretraining speed when switching domain from English to German while maintaining the performance.

strengths:
- A method of tokenizer-free langauge modelling that is more robust to input perturbation and input domain shift.

weaknesses:
- The proposed method only works for character-based languages.

questions:
- I'm curious how the model works when doing inference in the same language but when the text is from a different distrubtion (out-of-domain).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732532460276

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Title Change

comment:
Dear AC,

Reviewer PbBe expressed the concern that assigning the attribute _tokenizer-free_ to our proposed method may be misleading, because it relies on a word splitting rule. While we don't necessarily agree, we are open to changing the title of the paper to avoid confusion.

To keep the change minimal, we would propose to simply replace _tokenizer-free_ with _vocabulary-free_.

We weren't sure whether a title change during the rebuttal phase is possible, so we didn't make the change yet. If our paper gets accepted, please advise how to proceed.

Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a hierarchical language model that avoids traditional subword tokenizers by operating at the word level while internally processing characters with a lightweight character-level encoder/decoder.


**Strengths** (1) While the idea is not new, the experimental results show clear advantages of using such a hierarchical setup, which improves modeling efficiency and robustness over perturbations; (2) The paper demonstrates improved adaptability when moving from one domain/language to another.

**Weaknesses** (1) Clear confusion in the claim. The reviewer mentioned while the paper claims "tokenizer-free", it still relies on whitespace for splitting words, and thus it is still a rule-based tokenization. Also, such a choice makes it not possible to generalize to other scripts/languages without explicit whitespace such as Chinese;
(2) Some reviewers argued the limited comparison and discussion against prior works with very similar concepts;
(3) Some reviewers raised the confusion about the parameter counts and compute matching.


**Decision** 
It is a paper receiving mixed reviews especially after the rebuttal phase, making it hard to make decisions.

On one hand, I **agree** with PbBe about the claim of "tokenizer-free" are **misleading** (even the change of vocabulary-free is not correct), and the technical contributions are very limited considering its similarity with many prior works. More importantly, after checking the paper, it also seems to lack the necessary discussion of the potential inflexibility of modeling words in characters with variable lengths (e.g., efficient attention with masks). 

On the other hand, all the reviewers acknowledge the efforts of the authors for the detailed experiments and the evidence shown from the benefits. All the reviewers raised their scores from the initial rating. 

I am **personally leaning toward a rejection**; however, considering the champions from Kkdq and KFMw who increased their scores, I mark the case as **borderline accept**, and will leave SAC to make the final decision.