ICLR 2025

Published as a conference paper at ICLR 2025
MRT5: DYNAMIC TOKEN MERGING FOR EFFICIENT
BYTE-LEVEL LANGUAGE MODELS
Julie Kallini, Shikhar Murty, Christopher D. Manning, Christopher Potts, R´obert Csord´as
Stanford University
{kallini, jsmurty, manning, cgpotts, rcsordas}@stanford.edu
ABSTRACT
Models that rely on subword tokenization have significant drawbacks, such as sen-
sitivity to character-level noise like spelling errors and inconsistent compression
rates across different languages and scripts. While character- or byte-level models
like ByT5 attempt to address these concerns, they have not gained widespread
adoption—processing raw byte streams without tokenization results in signifi-
cantly longer sequence lengths, making training and inference inefficient. This
work introduces MrT5 (MergeT5), a more efficient variant of ByT5 that inte-
grates a token deletion mechanism in its encoder to dynamically shorten the input
sequence length. After processing through a fixed number of encoder layers, a
learned delete gate determines which tokens are to be removed and which are to
be retained for subsequent layers. MrT5 effectively “merges” critical information
from deleted tokens into a more compact sequence, leveraging contextual infor-
mation from the remaining tokens. In continued pre-training experiments, we
find that MrT5 can achieve significant gains in inference runtime with minimal
effect on performance, as measured by bits-per-byte. Additionally, with multilin-
gual training, MrT5 adapts to the orthographic characteristics of each language,
learning language-specific compression rates. Furthermore, MrT5 shows compa-
rable accuracy to ByT5 on downstream evaluations such as XNLI, TyDi QA, and
character-level tasks while reducing sequence lengths by up to 75%. Our approach
presents a solution to the practical limitations of existing byte-level models.
https://github.com/jkallini/mrt5
1
INTRODUCTION
Subword tokenization, typically via algorithms such as byte-pair e...