MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
Authors: Julie Kallini
Shikhar Murty
Christopher D Manning
Christopher Potts
Róbert Csordás
Venue: ICLR 2025 Poster (2025)
Keywords: NLP
ByT5
T5
tokenization
byte-level language models
character-level language models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: MrT5 improves the efficiency of byte-level models like ByT5 by introducing a token deletion mechanism in its encoder, reducing the sequence length during processing.

Abstract: Models that rely on subword tokenization have significant drawbacks, such as sensitivity to character-level noise like spelling errors and inconsistent compression rates across different languages and scripts. While character- or byte-level models like ByT5 attempt to address these concerns, they have not gained widespread adoption—processing raw byte streams without tokenization results in significantly longer sequence lengths, making training and inference inefficient. This work introduces MrT5 (MergeT5), a more efficient variant of ByT5 that integrates a token deletion mechanism in its encoder to dynamically shorten the input sequence length. After processing through a fixed number of encoder layers, a learned delete gate determines which tokens are to be removed and which are to be retained for subsequent layers. MrT5 effectively "merges" critical information from deleted tokens into a more compact sequence, leveraging contextual information from the remaining tokens. In continued pre-training experiments, we find that MrT5 can achieve significant gains in inference runtime with minimal effect on performance, as measured by bits-per-byte. Additionally, with multilingual training, MrT5 adapts to the orthographic characteristics of each language, learning language-specific compression rates. Furthermore, MrT5 shows comparable accuracy to ByT5 on downstream evaluations such as XNLI, TyDi QA, and character-level tasks while reducing sequence lengths by up to 75%. Our approach presents a solution to the practical limitations of existing byte-level models.

Published as a conference paper at ICLR 2025
MRT5: DYNAMIC TOKEN MERGING FOR EFFICIENT BYTE-LEVEL LANGUAGE MODELS
Julie Kallini, Shikhar Murty, Christopher D. Manning, Christopher Potts, Róbert Csordás
Stanford University (kallini, jsmurty, manning, cgpotts, restodas)@stannford.
ABSTRACT
Models that rely on subword tokenization have significant drawbacks, such sen- sitivity to character-level noise like spelling errors and inconsistent compression rates across different languages and scripts. While character- or byte-level models like ByT5 attempt to address these concerns, they have not gained widespread adoption-prooessinng raw byte streams without tokenization results in signifi- cantly longer sequence lengths, making training and inference inefficient. This work introduces MrT5 (MergeTT5), more efficient variant of ByT5 that inte- grates token deletion mechanism in its encoder to dynamically shorten the input sequence length. After processing through a fixed number of encoder layers, a learned delete gate determines which tokens to be removed and which are be retained for subsequent layers. MrT5 effectively "merges" critical information from deleted tokens into more compact sequence, leveraging contextual infor- mation from the remaining tokens. In continued pre-training experiments, we find that MrT5 can achieve significant gains in inference runtime with minimal effect on performance, as measured by bits-per-byte. Additionally, with multilin- gual training, MrT5 adapts the orthographic characteristics of each language, learning language-specing compression rates. Furthermore, MrT5 shows compa- rable accuracy ByT5 on downstream evaluations such as XNLI, TyDi QA, and character-level tasks while reducing sequence lengths by up to 75%. Our approach presents a solution to the practical limitations of existing byte-level models.
https://github. com/ wwtllkaalllnniii likkaallii/mrtt
INTRODUCTION
Subword tokenization, typically via algorithms such as byte-pair encoding (Sennrich et al., 2016) or SentencePiece (Kudo & Richardson, 2018), fundamental text preprocessing step that has become ubiquitous in modern large language models (LLMs). Subword tokenizers divide text into meaningful units known as tokens, which closely resemble words or parts of words. Tokenization can be seen as a form of compression, since it reduces the sequence length of the input passed to the compute-intensive Transformer (Vaswani et al., 2017). However, subword tokenizers have several drawbacks. For example, they not very robust to character-level noise and manipulations, such as spelling errors (Kaushal & Mahowald, 2022; Huang et al., 2023); they directly impact how models process digits and perform arithmetic (Singh & Strouse, 2024; Zhou al., 2024); and they have dis- proportionate compression rates for different languages and scripts (Ahia et al., 2023; Petrov et al., 2023). In addition, current language model APIs charge users per-token, and such discrepancies can cause users of certain languages to be overcharged due to poorer compression.
As alternative subword models, tokenization-free models skip the tokenization preprocessing step entirely by passing the raw character or byte stream directly as input. However, character- or byte-level sequences tend be significantly longer than tokenized text sequences, which limits the practical utility of tokenization-fres models. For example, ByT5 (Xue et al., 2022), byte-level counterpart of mT55 (Xue et al., 2021), is competitive with mT5 on a number of tasks, but it has much slower pre-training and inference runtime. Most other tokenization-free models explicitly
See also Andrej Karpathy's tweets tokenization: 7759996551378940038 https://x. http:////ccmckoooot/////aa//////

---

Published as a conference paper at ICLR 2025
Training: soft deletion via soft attn. masking
Testing:
hard deletion via column
X
[
[-0.1, 1,
-28. 6,
-1.4]
Delete Gate
h
h
Figure MrT5's encoder during training and testing. During training, fully-differentiable soft deletion masks out tokens using the output of MrT5's delete gate. During testing, hard deletion removes columns from the computation, which reduces the sequence length and leads to efficiency gains. In this visual, the delete gate is placed at layer 2, but the gate placement may be tuned.
downsample or pool representations reduce the sequence length (Clark al., 2022; Tay al., 2022; Nawrot et al., 2022; 2023). Many of these architectures rely on fixed-span downsampling methods; however, meaningful units of text often span variable number of bytes or characters. While some models identify variable-length spans, they introduce substantial modifications to the standard Transformer architecture, making it difficult to adapt existing pre-traie pre-trained models.
In this work, we explore the question: can we make an existing byte-level model more efficient? We propose MrT5 (Merge'T5), variant the ByT5 architecture that addresses its inefficiencies while maintaining its performance (Section 3). MrT5 dynamically merges its encoder's 's input into shorter sequence using a token deletion gating mechanism at fixed, early encoder layer, as shown in Figure 1. By allowing the first few encoder layers to process the entire sequence, the encoder creates contextualized representations of the tokens. When the gating mechanism then deletes a subset the tokens, those that remain keep the contextual information about those that were removed, allowing information to be implicitly merged into a shorter sequence. During training, we use a deletion regularizer with tunable weight that can adjust the amount of deletion MrT5 performs. MrT5 effectively learns to merge relevant tokens and delete extraneous ones in a completely unsupervised manner, by optimizing the balance between the regularization objective and language modeling.
We first train several MrT5 models on diagnostic tasks (Section 4) and find that MrT5 not only drops irrelevant tokens but also compresses relevant context using task-specific deletion patterns. Next, we perform continued pre-training experiments by fine-tuning the MrTS gating mechanism on top of pre-trained ByT5 (Section 5). Our results show that MrT5 achieves lower bits-per-byte than both random and fixed token deletion baselines, well as pooling-bar based alternatives, at the same compression rates. With multilingual training, MrT5 adjusts each language's 's orthographic features, learning optimal compression rates specific to each language. Finally, in multilingual and character-level benchmarks (Section 6), MrT5 achieves comparable accuracy to ByT5 while cut- ting the sequence length by up to 75%, significantly improving inference runtimes. Our approach improves on the main limitations ByT5, representing a significant step toward the adoption of byte-level language models and the elimination of subword tokenization from modern NLP.
2 BACKGROUND: TOKENIZAATION-RRES MODELS
Soft Tokenization and Downsampling Methods. Many character- or byte-level models employ "soft tokenization" explicit downsampling to shorten input sequences. Here, focus spe- cialized Transformer architectures. CANINE (Clark et al., 2022), character-leved counterpart to mBERT (Devlin al., 2019), uses convolutional downsampling before feeding inputs 12-layer
2

---

Published as a conference paper at ICLR 2025
Transformer encoder. Charformer (Tay et al., 2022) learns gradient-based block scoring fune- tion to pool byte embeddings for efficient training and inference. Islam et al. (2022) propose a vocabulary-free free neural tokenizer trained via supervision from a heuristic- subword tokenizer.
MegaByte (Yu et al., 2023) scales byte-level decoders to long-contenn tasks by segmenting sequences into fixed-length "patches," though these may not align with meaningful units of text. SpaceByte (Slagle, 2024) applies global Transformer blocks to specific byte types, such as spaces, but these bytes are also not chosen dynamically. Hourglass Transformers (Nawrot et al., 2022) incorporate hierarchical downsampling and upsampling at different layers within decoder models, and the ar- chitecture was later extended with a gradient- gradient-based tokenization module that dynamically pools characters using a boundary predictor (Nawrot et al., 2023; Ahia et al., 2024). MANTa (Anagnos- tidis et al., 2024) also learns token boundaries via sliding window attention. More recently, the Byte Latent Transformer (BLT, Pagnoni et al., 2024) and EvaByte (Zheng et al., 2025) have shown that tokenizer-fi frreieerrrreeeeeee models can scale more efficiently than subword L.Mss. BLT utilizes a dynamic boundary predictor based on byte entropies to segment sequences into variably sized patches, and EvaByte uses multibyte prediction and linear attention to improve scalability and decoding speed.
ByT5. This paper focuses on ByT5 (Xue et al., 2022), a byte-level sequence-to-sequeee former architecture designed as counterpart to mT5 (Xue et al., 2021), 1, the multilingual extension of (Raffel al., 2020). Unlike the models discussed previously, ByT5 uses no soft tokenization or downsampling steps to reduce the sequence length. To compensate for the loss of the parameters that would normally be used for subword embeddings, ByT5 has a "heavy" encoder with more lay- than the decoder. While ByT5 matches outperforms mT5 variety of downstream tasks, its heavy encoder, large model and feed forward dimensionalities, and short input sequence length (1024 bytes) make it quite inefficient. ByT5 requires 1.2 times more operations than mT55, and it can be up to 10 times slower at inference, depending on the encoder's 's input length.
Trans-
MrT5 modifies the ByT5 architecture to make it more efficient. Unlike previous work, MrT5's deletion gating mechanism can be applied to pre-trained model with minimal fine-tuning and few additional parameters, or models trained from scratch. For further related work on adjacent topics like early exit models, long-context, and additional tokenization-free architectures, see Appendix A.
3 THE MRT5 MODEL ARCHITECTURE
MrT5 introduces unique deletion gating mechanism: in fixed encoder layer, delete gate selects which tokens to keep for further processing and which to discard, effectively merging information into a shorter sequence. We apply this gating at a single layer for three reasons: (I) avoid the over- head of executing the deletion algorithm multiple times, (2) because performance stabilizes after early layers (Section 7), and (3) because early deletion yields the greatest computational savings.
3.1 DELETION GATING MECHANNIS
The MrT5 deletion gating mechanism is inspired by existing architectures with gating mechanisms such as Long-Shor Term Memory (LSTMs, Hochreiter & Schmidhuber, 1997), Gated Recurrent Units (GRUs, Cho et al., 2014), and Mixture-of-Exppets (MoFs, Shazeer et al., 2017). MrT5's 's delete gate is placed after the output of fixed encoder layer i and is defined by the following function:
G = ko (LayerNorm (H,) W + 1Nb)
(I)
where H, E Nxduuti are the hidden states output by layer W Rdmaax1; b E R; G E NN1 k is large negative constant; N is the encoder input sequence length; dmodel is ByT5's hidden state/model dimensionality; and 1N E RN application of layer normalization. Following the T5 architecture, this is implemented as rool mean square layer normalization (RMSNorm, Zhang & Sennrich, 2019). The gating activation function rescaled and translated sigmoid function, bounded between A and O. In our experiments, use = -30. During training, our experiments include adding Gumbel noise to the gate's 's logits to encourage exploration of gate values. However, we found that the model performs well even without it, making this step optional. See Appendix B for the Gumbel noise formulas.
vector of ones. LayerNorm (H:) denotes the
MrT5's delete gate introduces only 2dmodel + additional parameters in total; dmodel for the weight vector W, dmmdel for the layer normalization, and one for the bias term. This makes the method highly parameter-efficien..
3

---

Published as a conference paper at ICLR 2025
Soft and Hard Deletion. During training, MrT5 deletes tokens seftly, where the outputs of the gating mechanism G are applied as a soft attention mask. These outputs are added directly to the self-attention mechanism of the subsequent encoder layers, as well as to the cross-attention layers between the decoder and encoder. The modified attention mechanism for each attention head is defined
QK + + V Vihead
SoftDelleionnAAteetion(... K, V) softmax
(2)
where dhead is the dimensionality of the query, key, and value vectors for single attention head: Q, K, V E Neduu and 1N E R-Nx1 is vector of ones. token sequence position i E [1, with G, ~ 0 will not be masked, whereas token at position j * E [1, N] with G; ~ k will be masked, since is a large negative constant. Though soft deletion does not reduce the sequence length, we apply it during training to emulate the effect of token deletion while being fully differentiable. To see efficiency gains during inference, we apply hard deletion, where the hidden states removed from the sequence, determined by a hard threshold; set this threshold to be 2. half of the range of the delete gate's output.
In batch, different samples may have different numbers of tokens deleted. With hard deletion, the new sequence length is set by the longest remaining sequence, and shorter ones are padded. Since TS uses relative position biases within the attention mechanism of each layer, deletion and padding is performed on both the hidden states and position biases. For a theoretical analysis of MrT5's compute savings with hard deletion, see Appendix C.
3.2 GATE REGULARIZER
MrT5 allows deletion rates to be adjusted using a tunable regularizer loss:
1 N
I
LG = N=
(3)
This loss is the average of the gate output values, which encourages them to be more negative (i.e. closer to k. the minimum gate value). In other words, as this loss decreases, the number of deleted tokens increases. The total loss is defined as the sum L = LCe + aCg, where LcE is the cross- entropy loss. Varying the hyperparameter a allows the MrT5 model to delete more or fewer tokens.
Optimizing for Specific Deletion Ratio. Setting a by hand allows the model to dynamically discover the deletion ratio depending on the difficulty of the task. However, we can also optimize for a specific ratio of deleted tokens using an algorithm that resembles the proportional-interne controller (PI controller) of classical control theory. We additionally use an exponential moving average on the P-term. Let's call the target deletion ratio E [0, 1]. the proportion of deleted tokens in the current batch & E [0, 1]. and the regularization hyperparameter for the current batch a₁. We
update as follows:
Pr+1 = 7P: + +1 (1- -8)
(4)
i++1 i, + 8 - dz
(5)
ar+1 = clamp (kppPt1 + kzūc-1)
(6)
where clamp(x) = max(æ, 0). We initialize Po = and in = O. In all of our experiments, we use y 0.9. For most experiments (unless otherwise noted), we found that using kp = 0.5 and k; = le-5 worked well in practice. This method easier to use than manually setting a and allows to change dynamically as the model undergoes phase transitions during training, resulting in more stable learning. Besides our diagnostic task models, all MrT5 models are trained with this PI controller.
Softmax It is possible for all elements of G to equal the minimum gate value such that G; k for all E [1, Nj. This would satisfy the gate regularizer but fail to act as an attention mask, since adding the same value to all elements of the input to a standard softmax function does not affect its output. To help avoid this scenario, we use softmax, (Miller, 2023) in the attention mechanism: cxp(x;)
(softmax,(*)) (x)); 1 E3 exp(x;)
4

---

Published as a conference paper at ICLR 2025
Table I: Diagnostic tasks with example input and target sequences. our experiments, sequences are 64 characters/tokens long, including a start and end token. Legend: vowels to remove, vowels to keep, sequences to replace.
Task
Input
Target
Simple Vowel Removal
z=KBreJcexx..1gh2S1os
JKR.JCBxG100225
Contextual Vowel Removal Sequence Merge
EoubXgaWbi:Ogi KJAMIDABC22XKcniAABCS
FOubXgtVbogImnl
KjAxIpOZCxBeniDs
With the softmax function, if G; k for all E [1, NJ. as A: becomes negative, the sum of attention scores approaches zero. This eliminates the failure case using tokens that appear to be all deleted. For consistency, we use softmax; in all attention modules for MrT5 and baseline ByT5 models.
When training from scratch, we also found that attention scores could inflate to counteract the mask- ing effect of the delete gate; we found that a simple attention score regularizer mitigated this issue, as described in Appendix D.
SIMULATIONS
We first train tiny 15M-parameter MrT5 and T5 models with encoder layers and 3 decoder layers from scratch on three diagnostic tasks: simple vowel removal task, a contextual vowel removal task, and sequence merge task. The purpose of these experiments is verify that the architecture behaves as intended, particularly with regard to the merging patterns it learns. Does MrT5 merely drop tokens when they are irrelevant the output, does effectively merge relevant context into a shorter sequence? These results help set the stage for our continued pre-training experiments in Section 5 and our downstream task evaluations in Section
Diagnostic Task Specifications. Each of our three diagnostic tasks is variant of copy task, designed to assess MrT5's ability to identify unimportant or redundant information in the input sequence merge relevant information from some tokens into other tokens. Input sequences 64 bytes long and are comprised of random lowercase and uppercase English characters, plus start and end tokens. Example inputs and labels are provided in Table 1.
1. Simple Vowel Removal: Generate a copy of the input token sequence, except for any vow- els. We expect MrT5 delete vowels, which occur with 19% probability. Thus, the optimal sequence length decrease is 19%.
2. Contextual Vowel Removal: Generate copy of the input token sequence, except for any vowels that follow a lowercase consonant. We increase the probability of vowels such that, average, they comprise 40% of the sequence, and about 18% of the sequence is comprised of vowels that follow a lowercase consonant. MrT5 learns the relevant deletion pattern, we would expect a sequence length decrease of 18%.
Sequence Merge: Generate a copy the input token sequence, and translate any occurrence of the character sequence 'ABC' into the character 'D', 'ABC' sequences are inserted into the input randomly and occur about times per sequence on average. If MrT5 merges 'ABC' into single token, we would expect it to drop 10 tokens average, which is 15.6% of the sequence.
In these experiments, we do not use controller to enforce specific deletion ratio. Instead, we train multiple models with different fixed a values to observe the various deletion patterns that naturally emerge. In all models, we place the delete gate at the middle encoder layer = 2. For further model architecture and training configurations
for the simulations, Appendix E.
Table 2 presents the performance of several MrT5 models that use different regularizer a values trained on each of our three diagnostic tasks. In all three diagnostic tasks, MrT5 models leamed to selectively drop tokens in way that suited each task's specific requirements. This allowed them to achieve the same performance as T5 models, while processing shorter sequences. For instance, in the simple vowel removal task, the MrT5 models selectively dropped vowels but kept consonants intact. These findings suggest that MrT5 can create complex deletion strategies that exploit patterns redundancies in the input.
Results.
5

---

Published as a conference paper at ICLR 2025
Table 2: Diagnostic task performance for T5 and MrT5 with different deletion strategies. Token- level accuracy measures the percentage of correctly predicted tokens, averaged across sequences; sequence-level aceuracy measures the percentage of sequences with all tokens predicted correctly. MrT5 learns to delete around the optimal rate for each task while maintaining performance.
Task
Model
Token-level Accuracy
Seq-level Accuracy
Seq. Length Reduction (%)
Description of Deleted Tokens
Shmple Vowel
I5 (a le 4)
100.00
99.93
0.80
Removal
MrTS Mr15 16-3)
100.00
99.92
18.58
All vowels.
100.00
99.92
Start tokens vowels.
Contextual
15
100.00
99.91
0.00
Nowel Removal
Mr15 (a le-3)
100.00
99.78
1.56
Only start tekcus,
MrT5 (a= 1c- 2)
99.99
99.72
Sturt tokens and vowels lowercase consomants.
Sequence
13
100.00
99 84
0.00
Merge
MrT5 (a le- 2)
99.99
99.44
10.15
Start tokens and most instances of 'B'. All 'B's, 'C' 'ABC" sequences.
MrTS (a- 1.5c 2)
99.98
99.19
17.37
5 CONTINUED PRE-TRAINING
In our main set of experiments, we train MrT5 models on the ByT5 span corruption task. In this pre-training objective, spans of tokens in unlabeled text data are replaced with a single sentinel token ID per span, and the model must fill in the missing tokens. For ByT5 and MrT5, these are spans of bytes, and the masks can potentially interfere with word boundaries.
For continued pre-training, we use the multilingual C4 (mC4) corpus (Raffel al., 2020; Xue et al., 2021). We train all MrT5 and baseline models on 15 typologically diverse languages: English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, Hindi, Swahili, and Urdu. Finally, we examine how each model's performance correlates with se- quence length reduction rates, both overall and for each language. Following related work (Yu et al., 2023; Pagnoni et al., 2024), compare models using bits-per-byte (BPB), tokenizer-agmostic alternative to perplexity defined as BPB = LCe x loga(e).
Models. We train several MrT5 models with different PI controller target deletion ratios: E (0.3, 0.4, 0.5, 0.6, 0.7}. This is continued pre-training setup; we load the pre-trained weights and use the architecture settings of ByT5 Small (300M parameters) and only randomly initialize the weights of MrT5's gating mechanism. Based on sweep of different layers, place the gating mechanism after encoder layer = 3 (see Section 7). In addition to the MrT5 models, we also train several baselines:
1. ByT5 baseline: A ByT5 Small architecture with softmax,, but without deletion. ?
2. Random baseline: We implement and train set of models with random gating mechanism, where the choice of how the tokens are deleted is random; some number of gate values are set to k, and the rest set to O. In our experiments, average deletion rates: 30%, 40%, 50%, 60%, and 70%.
train five random models with different
3. Fixed baseline: We implement and train set of models that delete the ends of words. We train five models that delete different percentages of the ends words: 30%, 40% 50%, 60%, and 70%. For details on the implementation of this baseline,
Appendix F.1.
4. Boundary Predictor (BP) baseline: We implement and train a set of models that integrate an unsupervised boundary predictor module, inspired by the Hourglass Transformer architecture (Nawrot et al., 2022; 2023), which detects segment boundaries for mean-pooling representa- tions. To generate models with different compression rates, we adjust the prior probability of boundary, p. across different training runs: E {0.3, 0.4, 0.5, 0.6, 0.7}. Further details on the implementation of this baseline can be found in Appendix F.2.
5. Convolutional Pooling (CP) baseline: We implement and train a set of models that use a strided ID convolutional layer to downsample the sequence, similar to CANINE (Clark al., 2022), To generate models with different compression rates, we vary the stride r of the con- volutional layer: r (2,3,4] For a sequence of length N, there will be N downsampled representations.
"The validation loss for ByT5 with softmax, (0.7919) and the loss for unaltered ByT5 (0.7917) were comparable, further motivating our of softmax,. Training settings described in Appendix E.2.

---

Published as a conference paper at ICLR 2025
Mr13
Fixed, 70%
By 15
Random Deletion
Fixed, 60%
Fixed Deletion
R om 20% p
BP, p=03
Boundary Predictor with Mean Pooling (BP)
Bits siis-perbyl (13735
1.4
Convolutional Pooling (CT)
Fived, 50%
BP, p=0.5
CP Random, 60%
CP, rm
BP, p=0.6
1.3
Fixed, 40%
Cr,
BP, PP.p-0.7
Fixed, 30%
Random, 50%
1.2
Random, 40%
MrI5,
Random,
MrT5,
6-0.7
MrT5, 6-0.3
MrT5, 6=0.4
MrT5, d=0.5
8=0.6
By15
10
40
50
60
20
Sequence Length Reduction (%)
Figure 2: Span corruption BPB vs. sequence length reduction for each MrT5 and baseline model. MrT5 models consistently have much lower BPB than the baselines, and are generally competitive with unmodified ByT5, even where they achieve very large sequence length reductions.
Table 3: Average inference runtime for single sequence (in milliseconds) for ByT5 and each MrT5 model. Percentage decrease in runtime relative ByT5 is displayed in parentheses.
MrT5
Model
ByT5
0.5
0.6
0.7
Runtime (ms)
44.29
34.99 (4 20.98%;
31.95 (L 27.85%)
29.99 (4 32.27%) 28.24 (1 36.23%)
24.48 (4 44.72%)
For the random and fixed baselines, the gating mechanism is placed at layer 3, like the MrT5 models. For the BP and CP baselines, place the corresponding downsampling modules at layer == 3 as well. All models use softmax; in their attention mechanisms. The ByT5 baseline serves lower bound the best possible span corruption BPB, since it does not reduce the sequence length. For further details on model architectures, train/test dataset preparation, and optmization, see Appendix E.2.
BPB Sequence Length Reduction. First, we compare span corruption BPB across MrT5 and baseline models with varying sequence length reduction rates. Figure 2 illustrates the relationship between BPB and sequence length reduction for each model. These results aggregated across all languages by averaging the results per language; for language-specific plots, see Figure 6 in Appendix G. As expected, ByT5 achieves the lowest BPB but does not reduce the sequence length. While MrT5 models generally have higher BPB than ByT5, they consistently outperform other baselines at comparable sequence length reduction rates. For example, MrT5 with 0.5 reduces the sequence by about 50%, similar to the 50% random baseline and CP baseline with 2, yet achieves the lowest BPB. This trend holds across all MrT5 models, demonstrating the effectiveness of its gating mechanism in balancing sequence length reduction and BPB minimization.
Runtime Speedup. MrT5 models with higher sequence reduction rate also have faster inference runtime, shown in Table 3. partieular, MrT5 models that reduce the sequence length by 50% or more can achieve 30% speedup greater, with our implementation. These results show that hard deletion improves the efficiency of MrT5 when compared to ByT5, and this speedup can be tuned.
While our main experiments use 300M-parameter models, we also trained MrT5 at larger 1.23B- parameter model size and observed even greater efficiency gains over ByT5, with a 44.6% reduction in runtime at 49.5% decrease in sequence length. Additionally, the gap in BPB between MrT5 and ByT5 diminished this scale, indicating that MrT5's deletion mechanism becomes even more effective in larger models. For the details of these larger-model experiments, Appendix H.

---

Published as a conference paper at ICLR 2025
Per-language Evaluation. We analyze how a single MrT5 model performs across 15 train- ing languages. Figure plots test set BPB against sequence length reduction for MrT5 (with
14
Chinese
Swahili
Bits-per-bypt byte (BP7)
12
English
German
Spanish
Turkish
ò 0.5) for each language, alongside ByT5's BPB as
Vietnamese
French
1.0
baseline. The results show that MrT5 adapts its deletion rates to each language, with only a minor increase in BPB com- pared to ByT5. More than half of the languages exhibit sequence reduction rate above 50%. In languages with more information-dense scripts, such Chinese, MrT5 still performs substantial deletions but at lower rate than for less dense scripts. These findings highlight MrT5's ability to learn and ap- ply language-specifee context- aware deletions when trained on multilingual data.
Urdu
Arabic
0.8
Hindi
MrI5
Greek
Bulearian
0.6
ByT5
Thai
33
45
50
86
Sequence Length Reduction (%)
Figure 3: Average test set BPB vs. sequence length reduction
for MrT5 (6 0.5) across each of the 15 languages. ByT5 is shown for BPB comparison only (it does not reduce the sequence length). MrT5 learns language-specific sequence length reduc- tion rates and achieves over 50% reduction in many languages with minimal effect on the BPB.
6 DOWNSTREAM TASK EVALUATIONS
We next assess MrT5's performance on downstream tasks, specifically, two cross-lingual bench- marks (XNLI and TyDi QA) and two character-level tasks (Spelling Correction and Word Search). The two multilingual tasks evaluate MrT5's ability to understand high-level semantics and retrieval in a multilingual setting, while the character-level tasks test whether it retains its sensitivity to character-level manipulations.
Cross-lingual Benchmarks. We first test the cross-lingual capabilities of MrT5 using the Cross- lingual Natural Language Inference (XNLI) corpus (Conneau et al., 2018), benchmark for cross- lingual sentence classification with 5,000 parallel examples in 15 languages. This a cross-lingual zero-shot transfer task; models are fine-tuned on the English MultiNLI corpus (Williams et al., 2018) only, and are tested on multiple languages. The test set contains the same 15 languages our models are trained on. Our second cross-lingual evaluation employs the TyDi QA Gold Passage Task (TYDiQA-GoldP, Clark et al., 2020). In this task, given a passage that is guaranteed to contain the answer, the model must generate the correct answer to the question. TyDiQA-GoldP covers nine typologically diverse languages and comprises 204K question-answer pairs: Arabic, Bengali, English, Finnish, Indonesian, Korean, Russian, Swahili, and Telugu.
For each task, we fine-tune four models: the baseline ByT5 model, MrT5 with d = 0.5, a BP baseline with p = 0.5, and CP baseline with r Each model fine-tuned on top of its corresponding model from our previous continued pre-training experiment, ensuring that MrT5 and the BP/CP baselines maintain the same target compression rate established during pre-training (enforced by 8, p. r. depending on the model). This allows the models preserve their sequence length reductions and efficiency improvements while adapting to downstream tasks. Additional training details can be found in Appendix E.3.
Table 4 presents the results on both XNLI and TyDiQA-GoldP For the XNLI task, MrT5 is the su- perior model; it is the most efficient in terms of both sequence length reduction and runtime speedup (55.60% and 45.13 13%, respectively), and it achieves the highest task accuracy (65.31 1)), outper- forming ByT5. The XNLI task illustrates the efficiency of MrT5's fast encoder; 50% deletion rate results in substantial gains in runtime. This improvement can be attributed to the setup of the XNLI task, which requires large input sequences (up to 1,024 tokens) and short decoder sequences (a single token for classification). On the TyDiQA-GoldP task, MrTS once again proves to be the
8

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation metrics for XNLI and TyDiQA-GoldP. XNLI task performance is measured by accuracy (chance: 33%), and TyDiQA-GoldP by EM/FI scores. MrT5 achieves shorter sequences and faster runtimes than ByT5 while maintaining similar performance. BP and CP fall short in both efficiency and task performance. See Table 8 and Table 9 in Appendix for per-language metrics.
Task Performance
Rontime Decrease (%)
Seq. Len. Reduction (%)
Languag
By
MrTS
BP
CP
MrT5
BP
CP
MrTS
BP
CP
XNLI
64.72
65.31
59.37
49.94
45.13
33.22
36.32
55.60
53,95
50.07
TyDiQA-GoldP 69.90/79.58
68. 21/773
40.59/54.04
54.99/65.85
33.31
13.38
31.97
47.48
22.55
50.05
Table 5: Evaluation metrics for character-level tasks. MrT5 provides the best efficiency improve- ments (runtime and sequence length reduction), while maintaining competitive accuracy with ByT5. See Table 10 and Table 11 in Appendix for metrics on individual test splits for each task.
Seq.-Level Accuracy (%)
Runtime Decrease (%)
Seq. Len. Reduction (%)
Task
ByT5
MrT5
BP
CP
MrT5
BP
CP
MrT5
BP
CP
Spelling Correction
66.73
63.18
54.81
57.29
22.30
16.48
23.13
50.54
49.30
50.00
Word Search
76.61
72.02
74.33
68.23
59.29
44.18
53.29
74.53
69.78
75.00
most efficient model, outperforming the alternative baselines in terms of runtime savings (33.31 %).
Moreover, MrT5 comes closest to matching ByT5's exact match (EM) and F1 scores, with 68.27 EM and 77.73 F1, demonstrating that it retains much of the original model's 's accuracy while significantly improving efficiency.
Character-level Tasks. We fine-tune and evaluate MrT5 and baseline models on the Spelling Cor- rection with Context and Word Search character-level tasks from Huang et al. (2023). In the Spelling Correction task, the input is a sentence containing a spelling error, and the goal is to generate the same sentence with the error corrected. In the Word Search task, the input follows the format definitions letters, and the objective is to identify the substring in letters that, when re- versed, matches the given defini tion. We selected these two tasks from the suite because they both require understanding meaning and context and involve processing longer sequence lengths.
We follow the same fine-tuning approach used for the multilingual benchmarks, using models ini- tialized from our previous continued pre-training experiment. For Spelling Correction, we fine-tune MrT5 with å = 0.5, BP with p = 0.5, and CP with r 2, maintaining the same target compression rates from pre-tnaining. However, for Word Search, we adjust the compression parameters to further optimize efficiency, using MrT5 with 5 0.7, a BP baseline with p 0.3, and CP baseline with r = 4. We evaluate on all test splits from Huang et al. (2023), which are designed to rigorously assess a model's ability to integrate meaning and context in its predictions. Further training details can be found in Appendix E.3.
Table 5 displays the test set results for each character-level task. MrT5 stands out the most efficient model, achieving notable reductions in both sequence length (50.54% for spelling correction and 74.53% for word search) and runtime (22.30% and 59.29%, respectively). Despite these optimiza- tions, it maintains competitive sequence-level accuracy (63.18% for spelling correction and 72.02% for word search), making it well-balanced choice between speed and performance. These results show that MrT5's method of sequence merging effectively preserves its sensitivity to character-level information.
7 ANALYSIS
Per-sample Sequence Length Reduction. We present per-sample analysis of bits-per-byte and sequence length reduction for several MrTS models and random baselines with different sequence length reduction rates. We take sample of 1,000 English sentences from the mC4 test set and cal- culate the percent increase in BPB on a per-sample basis, using ByT5's BPB as the baseline (i.e. the percent increase between MrT5's BPB and ByT5's BPB for individual samples). For each sample, we also get the sequence length reduction. Across five MrT5 models with different deletion rates, found very weak/negligible correlation between the percent increase in BPB and the sequence

---

Published as a conference paper at ICLR 2025
length reduction (average correlation of 0.103). This reflects what would expect from the MrT5 models; for an individual sample, MrT5 learns when it can delete more tokens without incur- ring large increase in the loss. In contrast, for five random models with different sequence length reduction rates, we found a moderate positive correlation (average correlation of = 0.295). When the random model removes ther support the observation that the MrT5 models more strategically and contextually delete tokens compared to the baselines. See Figure 7 in Appendix J for plots presenting the correlations for each individual MrT5 and random baseline model.
tokens, it is more likely to degrade performance. These results fur-
Gate Placement. We present an analy- sis of MrT5 models with delete gates placed at various encoder layers. To max- imize efficiency,
is ideal position the delete gate at the earliest possible en- coder layer. However, placing the gate too early reduces the contextual information in the token representations, leading to more significant performance degradation com-
1.040
Average Runtime (ms)
Bits- Bits-per-byte (BPB)
1.035
1.030
pared to ByTs. We trained several MrT5 models with a target deletion ratio of 0.5, all using the same training setup and architecture described in Section 5. The only variable was the placement of the delete gate, as shown in Figure 4. Infer- ence runtime is lower and test set BPB is higher when the gate placed in early lay- ers. Since our goal is to place the gate as early as possible, we selected layer 3 as the optimal point for gate placement, since its BPB is comparable to layers 4 and 5. This analysis provides further ev- idence that MrT5 merges information into fewer tokens, since deletion earlier, less contextual layers results a higher BPB.
1.025
1.020
2
3
Layer of Delete Gate
Figure 4: BPB and inference runtime for a single se- quence for MrTS models with delete gates at different layers (1 E [l. All MrT5 models are trained with a PI controller with a target deletion ratio of 5 = 0.5. BPB is consistently higher in early layers; since the gate should be placed as early as possible, we select layer 3 as optimal.
The gate placement sets upper bound on the compute savings achievable with our model. We provide a detailed analysis of MrT5's theo- retical compute savings in Appendix C. With typical hyperparameters of tasks, achieve a maximum speedup of around three times with reasonable deletion rates.
CONCLUSION
In this paper, we introduce MrT5 (Merge'T5), variant of the ByT5 architecture designed to ad- dress the inefficiencies of byte-level language modeling. MrT5's token deletion mechanism forces the model to merge input tokens into more compact sequence, allowing for computational savings while preserving model performance. Our diagnostic experiments demonstrate that MrT5 effec- tively merges relevant context into a shorter sequence using strategies that align with task-specific objectives. In our continued pre-training experiments, MrT5 outperforms alternative deletion base- methods, and with multilingual training, MrT5 achieves language-
lines and downsampling/poolle specific sequence length reduction rates with minimal impact on performance. Our model learns very fast: the continued pre-training requires only a few thousand additional training steps. Further- more, MrT5 maintains competitive accuracy with ByT5 downstream multilingual benchmarks and character-lewe
tasks while improving inference runtimes. This demonstrates MrT5's capac- ity to handle tasks requiring semantic information, while still effectively processing character-level details- the main advantage of byte-level modeling. Our work takes a significant step toward the viability of byte-level language models and eliminating the need for subword tokenization.
When averaging correlation coefficients, we apply Fisher's transformation to stabilize the variance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank the members of the Stanford NI.P Group and Jurafsky Lab for help- ful feedback and discussions at various stages of this project. We would also like to thank Sabah Kallini for suggesting the name "MrT5." Julie Kallini is supported by a National Science Foundation Graduate Research Fellowship under grant number DGE-2146755. This work is supported in part by grant from the Laboratory Directed Research and Development program at Sandia National Laboratories. Sandia National Laboratories is multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia LLC, a wholly owned subsidiary of Honeywell International Inc. for the U.S. Department of Energy's National Nuclear Security Ad- ministration under contract DE-NA0003525.
REPRODUCIBILITY STATEMENT
Steps for reproducing each of our experiments are detailed in Appendix E. Descriptions of the model architectures and training provided in Appendix E.1; details of the model architectures, span corruption data preprocessing steps, and training conigurratons/yypeppraameee for continued pre-training are provided in Ap- pendix E.2: and training conigurriion/hyyyepprmmmeer for tine-tuning on the multilingual and character-level downstream tasks are provided in Appendix E.3. We provide our source code at https://githut com/ jaallini/arat.......1
pasguratioon/yyppprraammee for our diagnostic task experiments are
REFERENCES
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? Tokenization in the era of commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pp. ber 2023. Association for Computational Linguistics. doi: 10. (0.18653/w1/2223.emms m1in.611. URL https:tt////..oom http://ccaaathooyyyyy.oo org/ 2023. emw.p-mainnnnnmm 614.
Singapore, Decem-
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Valentin Hofmann, Tomasz Limisiewiez, Yulia Tsvetkov, and Noah A. Smith. MAGNET: Improving the multilingual fairesss of language models with adaptive gradient-han based tokenization. arXiv preprint arXiv: 2407.08818, 2024.
Sotiris Anagnostidis, Dario Pavilo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hof- mann. Dynamic context pruning for efficient and interpretable autoregressive transformers. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024, Curran Associates Inc.
Miguel Ballesteros, Chris Dyer, and Noah A. Smith. Improved transition-based parsing by modeling characters instead of words with LSTMs. In Lluís Mårquez. Chris Callison-Burch and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Process- ing. pp. 349-359, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.11663/11/DD1-1044) URL /atta:://aclaatholoogg org org/D15-101//
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3432, 2013.
Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. Revisiting character-based neural machine translation with capacity and compression. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pp. 4295-4305, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-1461. URL
www.h//olloooo.oooo1eoo/ 1461/.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN incoder-decode for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans

---

Published as a conference paper at ICLR 2025
(eds.). Proceedings of the 2014 Conference Empirical Methods in Natural Language Pro- cessing (EMNLP). PP. 1724-1734, Doha, Qatar, October 2014. Association for Computational Linguisties. doi: 10.3115////114-1177 URL
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-seelke question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 2020.
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wicting. CANINE: Pre-training effi- cient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73-91, 2022. doi: 10.1162/tacl 00448. URL https:
tac1-1.5.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2018.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspect- ing functional modularity through differentiable weight masks. In International Conference on Learning Representations, Virtual only, May 2021.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Minneapolis, Minnesota, June 2019. Association for Computa- tional Linguistics. doi: 10. 10.18653/v1/N19. 1423. URL https: http:///aaaaaaloaaaa www.oo/g///////ooeee
Short Papers), pp.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In Inter- national Conference on Learning Representations, Virtual only, May 2020.
Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. Multilingual language pro- cessing from bytes. In Kevin Knight, Ani Nenkova, and Owen Ranbow (eds.). Proceed- ings of the 2016 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 1296-1306, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/ URL https:/ httpss///alaathooogyyy. www.aannooooooooo///...--e-- 1155//
Albert Gu and Tri Dao. Mamba: Ainear-time sequence modeling with selective state spaces. arXiv preprint arXiv: 2312.00752, 2023,
Curtis Hawthorne, Andrew Jaegle, Cätälina Cangea, Sebastian Borgeaud, Charlic Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, Hannah Sheahan, Neil Zeghidour, Jean-Baptiste Alayrac, Joao Carreira, and Jesse Engel. General- context autoregressive modeling with Perceiver AR. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Inter- national Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research. pp. 3355- PMLR, 17-23 Jul 2022. URL v162/hawthorne22a.. html.
long-
mlr..ress/ mlr, press/
Geoffrey Hinton. Neural networks for machine learning. Coursera, video lectures., 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8): doi: 19977999...199.99.. //doii.ogg/0.11100 1162//ee0... 1997. 1735.
1735-1780, nov 1997. ISSN 0899-76677
URL https:
Jing Huang, Zhengxuan Wu, Kyle Mahowald, and Christopher Potts. Inducing character-level
structure in subword-based language models with type-level interchange intervention training. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Associa- tion for Computational Linguistics: ACL 2023, pp. 12163-12180. Toronto, Canada, July 2023. Sindings-acl.700 URL
Association for Computational Linguistics. doi: https://aclantholoogy. org/2023 http://22......nnnge.e findings-acl. 770.
12

---

Published as a conference paper at ICLR 2025
Kazuki Irie, Pavel Golik, Ralf Schlüter, and Hermann Ney. Investigations byte-level convolu- tional neural networks for language modeling in low resource speech recognition. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5740-5744, 2017. doi: 10. 10.109//CASPP.201..11 7775222.....
Md Mofijul Islam, Gustavo Aguilar, Pragaash Ponnusamy, Clint Solomon Mathialagan, Chengyuan Ma, and Chenlei Guo. vocabulary-free multilingual neural tokenizer for end-to-end task learn- ing. In Spandana Gella, He He, Bodhisattwa Prasad Majumder, Burcu Can, Eleonora Giunchiglia, Samuel Cahyawijaya, Sewon Min. Maximilian Mozes, Xiang Lorraine Li, Isabelle Augenstein, Anna Rogers, Kyunghyun Cho, Edward Grefenstette, Laura Rimell, and Chris Dyer (eds.), Pro- ceedings of the 7th Workshop on Representation Learning for NLP, pp. 91-99, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10.18653/v 1211665///002222pppzllppppp. URL
o 2022. repp4nlp-11101
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrizzaionn with gumbel-softmax. In International Conference on Learning Representations, 2017.
Ayush Kaushal and Kyle Mahowald. What do tokens know about their characters and how do they know it? In Marine Carpuat, Marie-Catheri de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, pp. 2487-2507, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.186633/1/2022..mmm naacl- main. 179. URL htwps:ttp////m https/////caaaahhoooo.yooo or org/2222. naacl-mate 179.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. Character-awarr neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligecee AAAI' 16, pp. 2741-2749. AAAI Press, 2016.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/1/DD8-222 10-222 URL htt.h:tt////eeeee /tttp:://acclanhhllog///
Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, and Luke Zettlemoyer. MYTE: Morphology-driver byte encoding for better and fairer multilingual language modeling. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15059-15076, Bangkok. Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ aon-.cc--oon.8000 URL https: //tp///aaaaaatooyyyyoooooo org/ 2024. acl-long 884/.
Wentao Ma. Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. CharBERT: Character-aware pre-trained language model. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp- 39-50, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguisties. doi: N0.18653/1120220 10.186633112000.ccoinng main.4. URL https::/accaattholog/ org//020. coling-main,
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous re- laxation of discrete random variables. In International Conference on Learning Representations, 2017.
Evan Miller. Attention off by one, Jul 2023. cctention-is-off-by-ooo html.
URL https://ww.evaamillee
Piotr Nawrot, Szymon Tworkowski, Michat Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Ma- rine Carpuat, CatieeCatherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022. pp. 1559-15711 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 1/265///w/2222.fmmlll URL htttps://aclaathheee wrg.gy.rrr/22222 (tepe22fffinnnggge findings-naac! 117.
naacl. 117
13

---

Published as a conference paper at ICLR 2025
Piotr Nawrot, Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. Efficient transformers with dynamic token pooling. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6403-6417, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 1..18653/1122223aalll acl-long.355 353.
URL https://acllntt httph/olccaaatoooo...oooo org/2023.
Artidoro Pagnoni, Ram Pasunuru. Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtz- man, and Srinivasan lyer. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv: 24/2.09871, 2024.
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. arXiv preprint arXhv:2305.15422 2023.
Guanghui Qin and Benjamin Van Durme. Nugget: Neural agglomerative embeddings of text. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28337-28350. PMLR, 23-29 Jul 2023. mlr, cetess/v222/qinne
URL https::tt////roeediiiii
Guanghui Qin, Corby Rosset, Bthan C. Chau, Nikhil Rao, and Benjamin Van Durme. Nugget 2D: Dynamic contextual compression for scaling decoder-only language models. arXiv preprint arXiv:2310.02409, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Jorunal of Machine Learning Research (JMLR), 11:14011-140s 2020.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. Language modelling with pixels. In The Eleventh International Conference Learning Representations, 2023. URL https: http///wppppeeeee//eeeeeeee//pep wtt.:ettooooummmidds/ssee
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, New Orleans, LA, USA, November 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715- Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10118853/11/166 1162.
URL https: / woclanntooooyyy wrwp:16-1111111
Noam Shazeer, * AAzalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of-expertt layer. In International Conference on Learning Representations, 2017. URL https://openrevies net/forum?id=B1-e ckMDQqg.
Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shatiq Joty. Discovering the gems in early layers: Accelerating long-context Ilms with 1000x input token reduction. arXiv preprint arXiv: 2409. 174227 2024.
Aaditya K. Singh and DJ Strouse. 'Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs. arXiv preprint 240. 2024.
Kevin Slagle. SpaceByte: Towards deleting tokenization from large language modeling. arXiv preprint arXiv:2404.1 14408, 2024.
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient- based subword tokenization. In International Conference on Learning Representations, 2022. URL https: /ttpp//pp//////////eeeeeeeeeeeee //tppprreiii....ttfoormddddeeee net/forummidsJttReee 10EFN.
14

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.0370. 2017.
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M. Rush. MambaByte: Token-free selective state space model. arXiv preprint arXiv: 2401. 13660, 2024.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume I (Long Pupers), pp. 1112-1122, 122.. Association for Computational Linguistics, 2018. URL httgg//ntooooog/11---11
http://accwwit
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 2246-2251, Virtual only, July 2020.
Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. BERxiT: Early exiting for BERT with better fine-tuning and extension to regression. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pp. 91-104, Virtual only, April 2021.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguisties. doi: 10.18653/v 1/202/11122111 .naacl www.aaamaannnee. main.41. URL https: / / hatph/oooccattoooyooooooooooo org/2021. 2021. maacl-main.44
Linting Xue, Aditya Barua, Noah Constant, Rami AI-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022. doi: 10.1 00461. URL https://aclantholl/
wwt. tac1-1.17.
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. arXiv preprint arXiv:2305.07185 2023.
Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada, 2019, URL htLphttp:://// ptttps://openrevieee net/ (iteeeencee/ppffidddSifiaae QBBAff.
Lin Zheng, Xueliang Zhao, Guangtao Wang, Chen Wu, David Dong, Angela Wang. Mingran Wang, Yun Du, Haige Bo, Amol Sharma, Bo Li, Kejie Zhang, Changran Hu, Urmish Thakker, and Lingpeng Kong. Evabyte: Efficient byte-level language models at scale, 2025. URL https: //hkunlp. gi thub. ht/blogg/225//eabytttt.
Zhejian Zhou, Jayu Wang, Dahua Lin, and Kai Chen. Scaling behavior for large language mod- els regarding numeral systems: An example using pythia. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 3806-3820, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/200 1/2024 cinl.ng. URL https::/aclaatthlllos .rrg/ 2024. 2024.findings findings-emmlp.221 218/.
15

---

Published as a conference paper at ICLR 2025
A FURTHER RELATED WORK
Early Exit Models. Our architecture related to early-exit methods proposed for autoregressive Transformers (Elbayad al., 2020; Schuster et al., 2022) and BERT (Xin et al., 2020; 2021). In contrast to previous approaches. our method is fully differentiable and does not require special training considerations calculating the entropy of the final classifier, and the deletion decisions are made in a single layer, making the model efficient and easy to use.
Long-context Models. Other work has addressed long-context modeling more broadly. Perceiver AR (Hawthorne al., 2022) uses cross-attention to map inputs smaller set of latent vectors, enabling autoregressive modeling over sequences up to 100K tokens. However, it was not evaluated on character-level tasks language tasks without tokenization. Other solutions include Nugget (Qin & Van Durme, 2023; Qin et al., 2023), which encodes the whole sentence, but passes only a dynamic subset of the embeddings to the decoder. This approach does save compute on the encoder side. Recently, GemFilter (Shi et al., 2024) has improved inference runtimes in decoder-only models by using early LLM layers filter to select input tokens to be processed by the full model; while this method requires no additional training, it requires two forward passes during inference.
Additional Tokenization-free Models. Before the Transformer, character-level LSTMs, with their compact vocabularies, were effective across range of NLP tasks, particularly in morphologically rich languages (Gillick et al., 2016; Kim et al., 2016; Ballesteros et al., 2015; Cherry et al., 2018). Building this idea, CharBERT (Ma al., 2020) enhances subword models with character-level embeddings from a bidirectional GRU and a noise-aware pretraining task to improve robustness. More recently, MambaByte (Wang al., 2024) extends the Mamba architecture (Gu & Dao. 2023) to byte-level inputs, offering fully tokenization-free alternative. Tokenization-fie approaches have also been explored in other modalities, including speech (Irie et al., 2017) and rendered as images (Rust et al., 2023).
B GUMBEL-SIGMOID
The Gumbel-Sigmoid a special case Gumbel-Softtax (Jang et al., 2017; Maddison et al., 2017) for binary choices. Please refer to Csordás et al. (2021) for more details. Given the logit / IR and random samples U1, U2 ~ Uniform(0, 1), Gumbel-Sigmoid can be calculated
log (log log u2))
where (I)
- is the standard logistic sigmoid and - log (log /1/ log u2) the Gumbel noise. We use this soft formulation of the Gumbel-Sigmonl without discretization.
C THEORETICAL COMPUTE SAVINGS
Here we analyze the theoretical amount of compute used by MrT5 given a deletion rate. Let's call the average length of the input sequence NE and the average length of the output sequence Np. The width of the residual is dmodel, the dimension of the up- op-projection in the MLP dr, the encoder has Lg layers and the decoder has LD layers. The deletion occurs after the (g layer, and the average proportion of deleted tokens is 8. We assume that the total size of the head projections is equal to dmudee, as typical for Transformers (dhead Nueul, dmnode).. Then, we can approximate the total number of muuttply--accmullan operations (MACs) for the model as follows. Before deletion, the self-attention the encoder uses N Ng MACs for both the Q, K, and the output projections and N dmmodel MACs for both the A QKT T and AV projections. The MLP layer uses Ne additional MACs. Thus, the total number of MACs used per layer is 4N g + 2% + N dd. This much compute is used for the first (G layers, after which the sequence length is reduced to NE 8) for the remaining LE lG layers. Thus, the encoder uses
rencoder MACS = (I.g lG) (1 - 5) (4N + 2onaell (1 - 5) NNonnen Neccmmedat + la (4N + 2N ddootee + N Nedmmdedd)))
The MACs used by the decoder can be calculated similarly, but additionally the cross attention has to be taken into account. The cross attention uses NE o)dEnode MACS for the K and V projections,
16

---

Published as a conference paper at ICLR 2025
NDdmadd MACs for the and output projections, and -1-8) ( MACs for the attention matrix itself.
N drdcoddrr LD (4Ndd + Bedndel + N DND 2N (o0g11--dppeee 8))clmeeet MACs
(10)
2.N 2N + 2(1 D
Note that d 0 corresponds the baseline ByTs. For MrT5 Small, dmodell 1472, di 3584, IE = 12, LD = 4, lG = 3, Ng 1024, and ND = 189. Given these numbers, we plot the total reduction in compute (compared ByT5) as a function of ó in Fig. 5.
Compute ByT5
1.00
0.75
0.50
0.25
0.00
0.2
0.6
0.8
1.0
Deletion ratio (6)
Figure Reduction in the total amount of compute as function of the deletion ratio
D REGULARIZZTION OF ATTENTION SCORES
Directly adding the delete gate values to the attention scores in Equation (2) can lead to unin- tended inflation the attention scores. This inflation would arise when training MrT5 from scratch, counteracting the intended masking effect of the delete gate values and allowing deleted tokens to influence the encoder and cross-attentions. To mitigate this issue, we introduce regularization term designed prevent the attention scores from inflating excessively, thereby preserving the delete gate's 's intended function.
Let E and C denote the number of encoder layers and cross-attention layers, respectively. We define lG < E as the encoder layer where the delete gate is placed in MrT5.
For any encoder layer E flg...., E). let Qe and K, denote the query and key matrices, respec- tively. Similarly, for any cross- cross-attention layer E {1...., C). let Qe and K, denote the correspond- ing query and key matrices. The attention scores for these layers are computed as:
S. = Q.KT e1
Ve (lg...., E} S. = Q.K. Vc (1....,C)
(11)
(12)
To control attention score inflation, we first apply a clamping operation at a minimum threshold m, compute the mean of the clamped scores, and subtract m:
(S))
max(s, m)
(13)
where is an element the scores matrix S, and |S] represents the number of elements in the matrix. Finally, we compute the aggregate regularization loss across all encoder and cross-altention layers, excluding encoder layers before lg:
E Σ
Es = E- C- 1)+
(S.) + Σ ((S.)
(14)
This final result Ls represents aggregate measure over all encoder and cross-attention layers. We use minimum threshold m is to avoid over-penalizing attention scores, which could negatively impact model performance. Instead of regularizing all attention scores, only aim regularize
17

---

Published as a conference paper at ICLR 2025
excessively large values. We found that with 5.0, could mitigate attention score inflation when training from scratch, and continued pre-training experiments, the regularizer had no effect on MrT5 with deletion (6 0.0) compared to the baseline ByT5 model. This shows that the regularization can target excessively large attention scores, without affecting the keys and queries in way that deteriorates the performance of the model.
The final loss is calculated as C = LCE + aLG + .Lss, where LCE is the cross entropy loss, and La is the delete gate loss defined in Equation (3). We can vary the strength of the regularization by tuning the parameter B. We found 8 = 5.0 to work well in practice, and this is the value we apply in all experiments.
E EXPERIMENTAL DETAILS
E. SIMULATION DETAILS
Model Architectures. We train our diagnostic models with encoder layers and 3 decoder layers, and we use dæ = 1024, dmoddl = 512, and attention heads in each layer. We use softmax, for all TS and MrT5 models. Other architectural settings match the standard ByT5 Small, resulting in an architecture with 15M parameters (5% of ByT5 Small's parameter count). All models use softmaxi in their attention mechanisms.
batch size of 128 examples and sequence length of 64 tokens, and train each model for a total of 30,000 gradient steps. We use the Adam optimizer with a learning rate that linearly warms up le-4 over 3,000 steps and linearly decays. For MrT5 models, the delete gate's s regularizer is enabled at 10,000 steps. We set a constant regularizer a throughout training and use attention score regularization, as described in Appendix D.
Optimization. We
E.2 CONTINUED PRE-TRAINING DETAILS
Model Architectures. All MrT5 and baseline models use the model configuration of standard ByT5 Small, which has dr = 3584, dmodel = 1472, 12 encoder layers, 4 decoder layers, 6 attention heads in each layer, and 300M total parameters. MrT5's gating mechanism introduces an additional 2,945 parameters; the boundary predictor baseline's 's downsampling module introduces additional 5M parameters; and the convolutional pooling baseline's 's downsampling module introduces and ad- ditional 6.5M parameters. All models use softmax; in their attention mechanisms.
Data. When training on the span corruption objective, calculate the corrupted spans such that the average masked span length is 20 tokens with a noise density of 33%-tha is, 15% of tokens the sequence are masked out ful-folloing the specification outlined in the ByT5 paper. To avoid training models for multiple epochs, we ensure that the samples drawn from the mC4 corpus are sufficiently large. Additionally, extract equal-sized samples for each language (in terms of bytes) from the mC4 training split.
For evaluation, sample each language's test set from its mC4 validation split. Each language is tested on 10,000 examples, except for Swahili and Urdu, which only have 2,800 and 9,300 examples in their validation splits, respectively. We sample disjoint sample of 16,000 examples of English C4 to use as a validation set during training.
Optimization. We train each model for 5,000 gradient steps over batches of 220 tokens (i.c. an encoder sequence length of 1024 with effective batch size of 1024). We the AdamW opti- mizer with an initial learning rate of le-4 with linear decay and no warmup. For MrT5, we use a controller to optimize different deletion ratios and use attention score regularization, described in Appendix D.
At test time, we an eval batch size of 214 tokens (i.e. an encoder sequence length of 1024 with batch size of 16). We use the last model checkpoint at step 5,000 for all evaluations.
E.3 DOWNSTREAM TASK DETAILS
Models. All MrT5 and baseline models are initialized from corresponding model in our continued pre-training experiments and inherit the configuration a standard ByT5 Small. This confi guration
18

---

Published as a conference paper at ICLR 2025
includes du 3584, dmodel 1472, 12 encoder layers, 4 decoder layers, 6 attention heads per layer, and total of 300M parameters. All models employ softmax, in their attention mechanisms.
The specific model selected for initialization is based on the desired compression rate for fine-tuning. For instance, since we aim for 50% compression XNLI, we initialize fine-tuning from the MrT5 model pre-trained with 0.5, and continue fine-tune with that target deletion ratio.
Data. For XNLI, all models are trained on the English MNLI training set and evaluated the full XNLI test set in English, as well as zero-shot in 14 additional languages. For TyDiQA-GoldB the multilinguul training set is split into 80% for training and 20% for validation, and evaluation is performed on the separate TyDiQA-GoldP test set. For the character-level tasks, we use the provided train/validation sets for training and evaluate each task's s respective test split.
Optimization. For all downstream tasks, we fine-tune all MrT5 and baseline models using the A optimizer with cosine learning rate schedule and no warmup. MrT5 models use PI controller to achieve target deletion ratio and attention Appendix Training and evaluation details for each task are summarized in Table 6.
regularization, as described in
Table 6: Fine-tuning and evaluation details for the XNLI, TyDiQA-GoldP, Spelling Correction, and Word Search downstream tasks. The maximum sequence length shown is for the encoder. LR denotes the initial learning rate used during fine-tuning.
Task
Steps
Epochs
Batch Size
Max Seq. Length
L.R
PI Controller
XNLI
6,000
sr782
512
1,024
le-4 4
0.5, hp = 0.5, k. 1e-5
TyDiQA-Golde
6,000
==3.07
512
2,048
5e-5
0.5, kp = 0.5, k. le-5
Spelling Correction
207,690
35
16
64
3e-5
0.5, 0.5. kp 0.5, k; 1e-5
Word Scarch
319,670
65
16
128
5e-4
8 = 0.7, ky 1e-3,
1e-6
For XNLI, we evaluation batch size of 16; for the character-level tasks, batch size of 64; and for TyDiQA-Goldi we use an eval batch size of 1 (since the answers are generated by the model for each sequence). For each task, the eval maximum sequence length matches the training sequence length. The final checkpoint is used for evaluation. For XNLI and TyDiQA-GoldP, this is step 6,000; for Spelling Correction, this is step 200,000; and for Word Search, this is step 310,000.
evaluation
F BASELINE IMPLEMENTATION DETAILS
F.1 DESCRIPTION OF FIXED DELETION BASELINES
The fixed deletion baseline deletes tokens at layer = 3 using deterministic rules based on the token identity. All tokens/columns corresponding to whitespace, punctuation, and symbolic characters identified: \,, n,
i, <, 5 % &, 
. These separator tokens are used to locate word boundaries. Then, based on the fixed deletion percentage. the delete gate will drop the tokens/columns corresponding to the ends of words. For example, if the target percentage 50%, the delete gate will remove the tokens corresponding to the final two characters of five-letter word, and the final three characters of six-letter word.
),
/,
=, ?, ?, e
F.2 DESCRIPTION OF BOUNDARY PREDICTOR BASELINES
As an additional baseline, we implement a version of T5 that downsamples a sequence using a Gumbel-Sigmoid boundary predictor, enabling end-to-end learnable segmentation, similar to Hour- glass Transformers (Nawrot et al., 2023). This method allows adaptive sequence compression by mean-pooling adjacent tokens into variable-length segments, which subsequently processed at a reduced resolution. We first summarize the boundary prediction and downsampling approach of Nawrot et (2023), followed by a description of the modifications we made to adapt the approach to our setting.
Boundary Prediction via Gumbel-Sigmoie The Hourglass Transformer employs a boundary pre- dictor to learn segment boundaries in autoregressive fashion. The goal is to find sequence of
19

---

Published as a conference paper at ICLR 2025
segment boundaries b {0, 10% where N is the input sequence length. Consider two tokens 2, and T4+1, where E [1 is the index of the token It in the sequence. Given the hidden state h, of Zt, a multi-layer perceptron with parameters & produces the probability of a boundary b, between token 24 and token 24+1;
b = = 1) 1)= = a (h₂)).
(15)
To enable gradient-basee -baeed optimization while sampling discrete boundary indicators, the boundary predictor uses the cumbell-iggoide trick with relaxed Bernoulli distribution:
1/r
bru
b log
Uniform(0,)
(16)
(1 b.)(1 m)
b. 1 2.55
(17)
where 1, is the indicator function. Following Nawrot et al. (2023), we use the straight-through estimator (Hinton, 2012; Bengio et al., 2013) make this operation differentiable:
b >.0.5 [pe]aup b.
(18)
where Hsop is an operator for preventing backward gradient flow. Here, u introduces stochasticity, and temperature parameter controlling the sharpness of boundary decisions.
Segment Pooling via Algebraic Manipulation. Once boundaries b, are determined, tokens within the same segment are aggregated using mean pooling into a shortened sequence length S = + E. be. This is computed via binary assignment matrix B N where B;, 1 if token i is assigned to segment j. and 0 otherwise. The pooled segment representations S E Sxdont (where denagee is the model dimensionality) are computed via weighted averaging of the hidden states H IR N X
s (B
H Ø (B
I I
(19)
where Ø denotes element-wise (Hadamard) division and 1 E RR Xddssa is matrix of ones. The shortened sequence S is then processed by subsequent Transformer layers.
Regularization via Binomial Prior. To prevent trivial segmentations (i.e. excessively fine coarse partitions), a Binomial prior is imposed on the expected number of segment boundaries:
Binomial(k; N.p)
p0 p)
(20)
1:
where 7: E, by is the total number of boundaries, N is the sequence length, and is tunable prior controlling the expected segmentation rate.
Adaptations to the T5 Setting. The boundary predictor method used in Hourglass Transformers required several modifications to be applicable in experimental setting, as it originally de- signed for autoregressive architectures. To enable its use with an encoder model, we introduced the following adaptations:
1. Removal of null-group representations: In the original method, the last segment of the se- quence removed, and a null-group representation is prepended to sequences to prevent po- tential data leakage from the future to the past in autoregressive setting. However, in an encoder model, such representations are unnecessary and were found to degrade performance. We therefore excluded them.
2. Handling padded sequences: In encoder model, sequences of varying lengths are often batched together with padding tokens. To prevent padding tokens from being included in pool- ing, we identify the position
the first pad token in each sequence and enforce boundary at
position i (i.e. setting 1 = 1). Additionally, any boundaries predicted beyond this posi- tion are zeroed out. This ensures that the final downsampled segment includes only non- non-padding tokens, preventing pad tokens from being pooled into the shortened representations.
3. Updating the attention mask: Token pooling reduces the sequence length dynamically, and each sequence within batch might have different numbers of segment boundaries. To ensure that attention operates only valid pooled representations, we construct new attention
20

---

Published as a conference paper at ICLR 2025
mask that aligns with the shortened sequence. Specifically, for each sequence in the batch, any positions beyond the number of shortened segments are masked, preventing the model from attending to extrancous representations.
4. Handling relative position biases: that incorporate relative position biases, which are used in the attention mechanisms of each T5 layer. To address this, we designed solution that uses the position bias of the first token within a span as the position bias of the entire shortened segment.
The original boundary predictor does not support models
In our experiments, we place the boundary predictor module at encoder layer = 3, the same layer at which MrT5's delete gate placed. Subsequent encoder layers operate on the mean-pooled hidden states S. Following Nawrot et al. (2023), we set the Gumbel temperature to r = 0.5 for all experiments. We vary the prior probability of boundary p across training runs to obtain models with different compression rates.
G PER-LANGUAGE SPAN CORRUPTION EVALUATIONS
Figure contains the span corruption BPB across several MrT5 and baseline models with varying sequence length reduction rates. Evaluations are performed in each of the 15 training languages individually. The shapes and colors correspond to the same models evaluated in Figure 2.
H CONTINUED PRE-TRAINING EXPERIMENTS WITH LARGE MODELS
We present additional continued pre-training experiments using larger model sizes. Specifically, we train one ByT5 model and one MrT5 model, both initialized from the pre-trained 1.23-billion parameter ByT5 Large. While ByT5 Small consists of 12 encoder layers and decoder layers, ByT5 Large has a much deeper architecture with 36 encoder layers and 12 decoder layers. ByT5 Large also has dn 3840, dmmodel 1536, and 16 attention heads in each layer. MrT5 Large's gating mechanism introduces an additional 3,073 parameters. In MrT5, the delete gate is applied at layer I == 3, and all other data and training configurations remain consistent with those used in our primary continued pre-training experiments, as detailed in Appendix E.2. We employ a PI controller to target deletion ratio of 8 0.5. We evaluate on a smaller batch size of 4 examples (2 tokens) using test set containing 2500 examples in each of the 15 languages.
The results in Table 7 underscore the effectiveness of MrT5 at a larger model scale. Notably, the gap in bets-per-byte (BPB) between ByT5 and MrT5 is significantly reduced compared to smaller- scale models, suggesting that MrT5's performance greatly improves as model capacity increases. At the same time, the efficiency gains become even more pronounced; a 49.5% decrease in sequence length results 44.6% reduction inference runtime. This improved efficiency can be attributed to the deeper encoder architecture, where only 3 out of MrT5's 36 encoder layers process the full se- quence, significantly reducing computational overhead. These findings suggest that MrT5's deletion mechanism scales effectively with larger models, offering appealing trade-off between efficiency and modeling performance in high-resource settings.
Table 7: Performance comparison of ByT5 Large and MrT5 Large (6 0.5).
Bits-per-byte (BPB)
Average Runtime (ms)
MrTS Runtime Decrease (%)
MrT5 Seq. Len. Reduction (%)
ByT5
MrT5
ByT5
MrTS
0.70
0.74
177.27
98.21
44.6
49.5
I ADDITIONAL DOWNSTREAM TASK EVALUATIONS
Table 8 contains XNLI evaluation metrics for MrT5 and all baseline models for each of the 15 XNLI languages. Table contains the TyDiQA-Gol PoiiAG evaluation metrics for all MrT5 and baseline models for each of the TyDi QA languages. Table 10 contains evaluations all test splits for the Spelling Conrection with Context task. Table contains evaluations on all test splits for the Word Search task.
21

---

Published as a conference paper at ICLR 2025
1.3
Sheepedyyi (829)
2.3
Sttepe dayte
100
1.
Sequence Length Reduction (S)
Sequence Langth Reduction (%)
Sequence Length Reduction (S)
(a) Arabic.
(h) Bulgarian.
(c) Chinese.
Bite Bit-purrdyta (87P)
Bilegun-byle
1.6
1.4
1.4
1:2
Sequenoe Length Reduction (3))
Sequinee Length Reduction, (%)
Sequenee Length Reduction (34)
(d) English.
(e) French.
(I) German.
Bitegerryye (B155)
4.0
1.0
0.5
20
40
60
Sequence Length Reduction (%5)
Sequence Length Reduction (%)
Sequenoe Length Reduction (%5)
(g) Greek.
(h) Hindi.
(i) Russian.
Btrepeyyt Sppeettyyiiii (878)
1.4
1.4
0.8
20
40
Sequence Langth Reduction (%4
Sequence Length Reduction
Sequeenee Length Reduction (%4
Spanish.
(k) Swahili.
(l) Thai.
Eiteeperrerte (S375)
Bitepan
1.4
1.3
1.1
Bite
Siquenae Length Reductioe
Sequince Length Reduction,
Siquince Length Reductive
(m) Turkish.
(n) Urdu.
Vietnamese.
Figure 6: BPB vs. sequence length reduction for each MrT5 and baseline model, evaluated on span
corruption test sets for 15 languages individually. Legend: * MrT5; ByT5; X random
deletion baseline; = fixed deletion baseline; boundary predictor (BP) baseline with mean
pooling; = convolutional pooling (BP) baseline.
22

---

Published as a conference paper at ICLR 2025
Table 8: Per-language XNLI evaluation metrics for ByT5, MrT5, BP, and CP models. Except for English, all evaluations are zero-shot.
Accuracy (%)
Average Runtime (ms)
Runtime Decrease (%)
Seq.
Len. Reduction (%)
Language
ByT5
MrT5
BP
CP
By
MrT5
BP
CP
MrT5
BP
CP
MrT5
BP
CP
English
80.30
80.20
78.90
73.53
9:01
5.62
7.33
5.90
37.6 97.64
18.63
34.50
50.22
43.38
50.10
French
73.93
73.03
69.02
55.19
10.76
6.67
8.21
7.02
38.04
23.69
34.75
50.21
48.50
50.08
Spanish
74.85
74.25
69.78
59.78
10.15
6.15
7.79
6.63
39.44
23.29
34.63
51.88
48.11
50.09
German
69.58
69.70
63.61
50.66
10:44
6.77
8.01
6.82
35.12
23.28
34.66
47.04
47.97
50.08
Greak
64.73
65.03
57.33
46.55
18.81
9.32
10.75
11.79
50.48
42.85
37.32
63.65
65,51
50.05
Bulgarian
67.47
68.14
61.88
52.79
17.37
8.63
10.78
10.96
50.30
37.96
36.89
63.57
50.49
50.05
Russian
64.21
66.25
60.48
51,46
17,80
8.72
10.85
11.20
50.99
39.03
37.07
64.32
61.27
50.06
Turkish
58.88
47.31
9.77
6.22
6,40
36.37
16.85
34.49
48.45
50.08
Arabic
62.97
63.61
57.29
49.34
14.00
7.82
9.57
8.96
44.15
31.63
36.01
57.98
$4.38
50.06
58.64
47.92
12.54
8,10
Thai
55.45
58.02
46.19
43.55
24.95
11.87
12.34
15.42
52.45
50.57
38.23
67.08
74.89
50.04
Chinese
62.12
60.34
57.92
44.17
8.07
4.9%
5.29
38.89
13.67
34.44
48.03
40.04
50.12
Hindi
55.41
57.15
48.22
41.24
24.19
11.51
14.48
14.96
52.43
40.14
38.17
66.96
66.49
50.04
Swahili
60.04
57.92
55.85
44.57
9.14
5.92
6.01
35.23
18.16
34.23
47.34
44.03
50,10
Urdu
50.76
56.31
46.59
41.08
15.49
8.83
10.47
9.85
43.01
32.39
36.44
55.87
56.87
50.06
All Languages
64.72
65.31
59.37
49.94
14.17
7.77
9.02
45.13
33.22
36.32
55.60
53,95
50.07
Table 9: Per-language TyDiQA-GoldP evaluation metrics for ByT5, MrT5, BP, and CP models. When runtime relative to ByT5 is not reduced, the percent decrease in runtime is omitted (-).
Language
Exact
(%)
Average Runtime (ms))
Runtime Decrease
Seq.
Reduction (%)
Byl5
MrTS
BP
ByT5
MrT5
65
00.59/71.41
31,90443.22
36.37
19.62
Arsiii
60 00277/112
77759
60:69775.4
30.13
33.30
14.94
33.27
Dengali
55.73167.22 222
56.5466600
23.01/31411
38.94551.2 24
77.87
38.88
57.42
47.51
50.06
26.25
38.99
66.68
43.21
50.01
Toluga
77 891
77.4388487
29.98/42.40 40
59.04866 78
57.22
29:96
44.84
35.20
47.65
21.63
37.60
65.33
36.34
50003
Finnish
69 18/78.85 92
67. 907667670 97
45.65/58.5 58
54 24.22265.77 76
26.31
22.28
25.97
19.71
15:31
1.30
25.09
31.19
13.88
$0.05
Swahil:
77 99.96/5552 29
76.55862.77
61,33/1955
60.92267.89 889
19.00
17.04
20.34
16.92
10.33
10:93
40.68
5.39
50.09
Kovena
58. 7177666.22 26
$7.6186..0 66
$188840.07
38 78.04444.67 67
51.37
25. 28
29.42
22.30
19.43
6.22
28.91
52:77
17.00
50.04
Indonesian
75 58/58/44.2 23
73, 1083,19
50.09/65.11
61. 61.06472.10 10
28.76
22,16
27.04
20.81
22.93
5.96
27.64
39:27
16.51
$0.06
English
63.04/73.50
62.50770
48 4W.41/57.79
Languages
69.9079.58
68.27/7.75
40.59/44.04
54,99K5.85 85
24.83
33.31
22.55
50005
Table 10: Evaluation metrics for all splits for the Spelling Correction with Context character-level task. The "Dependent" split requires incorporating context to correct the spelling error; the "Inde- pendent" split does not.
Spelling Correction
Level Accurucy (%)
Average Runtime (ms)
Runtime Decreuse (%)
Sey. Len, Reduction (%)
Test Split
ByT5
M/T5
BP
ByT5
M/T5
CP
MrT5
BP
CP
M/T5
Dependent
49.41
44.20
36.51
37.40
3.86
2.98
3.20
2.94
22.39
16.99
23.72
50.85
49.34
50.00
Indeependent
82.11
80.04
71.08
74.96
3.90
3.04
3.28
3.02
22.05
16.04
22.62
50.26
49.25
50.00
All Splits
66.73
63.18
54.81
57.29
3.88
3.01
3.24
2.98
22.30
16.48
23.13
50.54
49.30
50.00
Table 11: Evaluation metrics for all splits for the Word Search character-level task. The "OOV" split contains hidden words with mT5 tokenization not seen in the training split (this does not apply to our work, since we only train byte-level models, not subword models); the "Paraphrase" split contains definitions from The Online Plain Text English Dictionary, testing the ability to understand context; the "Overlap" split contains overlapping hidden words; and the "Paraphrase + Overlap" split contains both paraphrased definitions and overlapping hidden words.
Word Search
LeveL Accuracy (%)
Average Runtime (ins)
Runtime Decreuse (%)
Len. Reduction (%)
Test Split
ByT5
M:15
BP
CP
ByT5
MrTS
BP
CP
MrT5
BP
CP
MrT5
BP
COV
78.49
73.96
78.42
71.25
6.56
2.63
3.62
3.02
59.90
44.79
$3.98
71.77
69.95
75.00
Paraphrase
81.51
72.30
2.75
3.75
59.39
53.52
Overlap
77.31
72.73
77.41
74.86
6.77
2.76
3.80
3.18
59.19
43.92
$3.06
77.60
69
69.80
75.00
Paraphrase Overlup
60.37
55.48
57.01
51.89
6.76
2.77
3.80
3.18
59.06
43.82
53.00
69.78
75.00
All Splits
76.61
72.02
74.33
68.23
6.75
2.75
3.77
3.15
59.29
44.18
53.29
74.53
69.78
75.00
23

---

Published as a conference paper at ICLR 2025
J ADDITIONAL PER-SAMPLE ANALYSES
Figure shows per-sample correlation plots between the increase in BPB and the sequence length reduction for five MrT5 models and five random baseline models with different deletion rates. The percent increase in BPB is measured relative to ByT5's BPB for that example.
100
6=0.3 (r:0.14, p=0.000)
100
R 30.0% (r=0.28, p=0.000)
100
MrT5, 6-0.4 (r-0. 13, p-0.000)
100
Random, (r-0.30, p=0.000)
Increase in BPB (%)
Increase BPB (%)
100
MrT5, 6-0.5 (=:0.33 p=0.000)
100
Random, 50.0% (r=0.32, p=0,000
100
Mr15, 6-0.6 (f-0.08, p-0.007)
100
Random, 60.0% (r-0.29,, p=0,000)
100
Mr15, 6=0.7 (r=0.33, p=0.316)
100
Random, 70.0% (r=0.28, p=0.000)
80
20
Sequence Length Reduction (%)
Sequence Length Reduction (%)
(a) MrT5 models. All models show very weak to negli- gible correlation between BPB increase and sequence length reduction, showing that MrT5 can delete at dif- ferent percentages depending on the sequence, without incurring significant performance drop.
(b) Random baseline models. All models show a weak to moderate positive correlation between BPB increase and sequence length reduction.
Figure 7: Percent increase in span corruption BPB vs. sequence length reduction for each (a) MrT5 model and (b) random baseline model. Percent increase is calculated using ByT5's BPB as baseline for particular sample. Each point represents single sample.
24


Reviews:
[{"review_id": "GJKl8htrn0", "reviewer": "Reviewer_T4gu", "summary": "This paper presents MrT5 (Merge-T5), a more efficient variant of ByT5 that introduces dynamic sequence shortening through a learned token delete gate after the first few encoder layers. Though it is called Merge-T5, no tokens are merged together, only deleted or preserved. \nThe method is a lightweight addition to a typical transformer architecture, with fully-differentiable soft deletion used during training via attention-masking and a regularizer with a tuned weight to adjust the degree of deletion, and hard deletion at inference time. \nPerformance is shown to be comparable to ByT5 (+/- 2% accuracy) while reducing sequences by up to 80% and runtime up to ~55%. \nOnly 4,000 steps are necessary to adapt a ByT5 model to a MrT5 model. \n\nExperiments are well designed and illustrate the properties of the method in different settings.  \nFirst, synthetic tasks (vowel deletion, token merging) are used to understand learned deletions with controlled settings and small 31M parameter models.  \nNext, they introduce token deletion in continued pre-training of ByT5-small both on English and multilingually, ablating components of the mechanism and different degrees of deletion. The task is span corruption loss, which for byte-level input may affect character and word boundaries. \nMrT5 is (generally) able to delete bytes in other scripts even if they have been unseen (if the model has been pretrained on English only), but results in slightly higher losses than ByT5. Multilingual pretraining reduces but does not remove the loss difference between MrT5 and ByT5. For Chinese, though, token deletion does not occur zero-shot.  \n--> It is not clear the significance of the differences in CE in Figure 2 & 3 here - it would be nice if this could be made clear.  \nFinally, MrT5 is evaluated on downstream tasks, XNLI and 2 character-level tasks.  \nFor English, MrT5 outperforms ByT5 with a reduction in sequence length and inference time of ~50%, while averaged across all languages there is a slight performance degradation of ~2% (though a significant speedup).   \nOn two English character-level tasks (contextual spelling correction and word search), MrT5 again saw ~2% accuracy drop compared to ByT5 in exchange for a 30-55% runtime decrease.  \nWhen comparing different layer placements, it seems that a middle layer (3) balances training stability and deletion level, allowing contextual representations to be learned before deleting tokens while still pruning sufficient tokens for efficiency gains. \n\nIt would be good if it were made clearer what the possible performance cost of introducing the deletion gate is, and some sense of the variability across languages, on explicit task performance as well as the loss presented in Figure 3.", "questions": "Questions: \n- I'd like to see the 'all languages' split for the downstream tasks discussed in the main text to match the CE analysis -> is it the case for example that you see a performance hit and no sequence length reduction for Chinese when the deletion gate is used zero-shot on a task, following Figure 3? Essentially, it would be nice to clearly state what the possible performance cost of introducing the deletion gate is, and some sense of the variability across languages, on explicit task performance as well as the loss presented in Figure 3. \n  - There is not much difference in performance between MrT5 and ByT5 for Chinese XNLI in Appendix Table 6, and there is ~20% length reduction - why do you think there is a difference to Chinese in the top half of Figure 3 with only a ~2% length reduction? Similarly, for Swahili, there seemed to be a significant difference in CE in Figure 3 zero-shot, but a smaller gap in Table 6. Why do you think that is? I suggest addressing these inconsistencies in the main text and providing possible explanations for the differences between training and downstream task performance. \n\nSuggested citations:  \n- [Cherry et al 2018](https://arxiv.org/abs/1808.09943) learns to delete characters for temporal compression with character-level models in MT\n- [Limisiewicz et al 2024](https://arxiv.org/pdf/2403.10691) uses morphologically inspired compression for byte sequences to create MyT5, a ByT5 variant \n\n\nPresentation nits:\n- L99: \"the main the limitations\" -> \"the main limitations\"\n- How α was set was not super clear to me. L196: \"For most of our experiments, we set α by hand, which allows the model to dynamically set the deletion ratio [based on the loss].\" This paragraph says α was set by hand most of the time, but that it is easier to allow α to dynamically change. Do any of the experiments in the main text allow α to change? (It doesn't seem so?) Why not, if it is easier?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A straightforward and lightweight addition to ByT5 models which can provide significant efficiency improvements (up to 80%) with a small trade-off in accuracy (~2%) compared to ByT5 (and often small improvements for English).", "weaknesses": "Efficiency gains (though generally significant) can come at a slight performance cost for non-English languages, and it is not clear how much variance in this cost there may be. \n\n- Without continued training in non-English scripts, there can be performance drops for non-English languages with less efficiency improvements. Figure 3 suggests that for Chinese for example, there may be almost no reduction in seq length zero-shot for the span corruption task, and some languages have relatively large drops in CE compared to ByT5\n- For the downstream experiments, the results are reported in aggregate for non-English languages in the main text. While the results are broken out by languages in the Appendix, a mention of the variance across languages and a comment on the relationship between these results and those in the previous section would make it clearer what the potential trade-offs are and when they arise", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pf1CpVp3bn", "reviewer": "Reviewer_uaeo", "summary": "This work builds on ByT5 and introduces a new token deletion mechanism that dynamically determines how to remove unnecessary tokens from the sequence without compromising performance.\n\nThe authors incorporate additional neural layers to learn the optimal token deletion process. During training, a loss function is introduced to encourage the model to progressively delete tokens (softly) or to achieve a specific ratio. During inference, the layers identify and discard unimportant tokens in a hard manner.\n\nControlled experiments on synthetic tasks demonstrate that the method, MrT5, can effectively learn to compress the input sequence. Further results on downstream tasks, such as XNLI, also support the authors' claims.", "questions": "If possible, I am interested in comparing the effectiveness of additional parameters introduced by MrT5 in learning to delete tokens.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* A simple yet effective method that enables models to dynamically learn how to delete tokens from byte-level inputs.\n* Controlled experiments and results on downstream tasks support the authors' claims.\n* MrT5 demonstrates competitive inference speed compared to ByT5.", "weaknesses": "* If I understand correctly, during training, the byte tokens are deleted softly, meaning there are still significant burdens for byte-level language models given that standard attention has quadratic time complexity, which limits their scalability to larger sizes.\n* The experiments presented utilize moderate model sizes, which may constrain the overall persuasiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vdn57MI8mJ", "reviewer": "Reviewer_91Fx", "summary": "This work improves on top of ByT5 by adding a delete gate at a certain layer to remove unimportant tokens. The proposed model trains a soft deletion layer which is fully differentiable, and then uses the same layer for hard (i.e., discrete) deletion during inference. The model only requires unsupervised data for training. \n\nThe model is evaluated on monolingual and cross-lingual pretraining tasks, showing a trade-off between cross-entropy loss and sequence length reduction. On two evaluation tasks, the model is also shown to be competitive with ByT5 while reducing sequence length.", "questions": "Suggestions:\n\n• 034-035: \"... via algorithms such as byte-pair encoding (Sennrich et al., 2016) or SentencePiece (Kudo & Richardson, 2018) ...\"\nConsider rephrasing this sentence as BPE is part of SentencePiece.\n\n• Consider discussing the relation between this work and GemFilter (https://arxiv.org/pdf/2409.17422) as both pertain to token deletion.\n\n• The proposed method seems readily transferrable to decoder-only models, which is the most widely used architecture nowadays. Would like to see some experiments, or at least some discussions about this direction.\n\n• 150-151: \"(1) we want to avoid the overhead of executing the deletion algorithm multiple times;\"\nThis motivation is better discussed from the perspective of \"trade-off\". If executing the algorithm multiple times can reduce the number of tokens/positions to process in later layers even more without compromising generation quality, then there is no reason to not do it.\n\n• The regularizer loss only seems to encourage the increase of the number of deleted tokens, but does not encourage the gate output to converge to the extreme values (i.e., the min or max gate value). This could make hard deletion during inference less effective because merely setting a threshold may delete some \"somewhat useful\" tokens. Please refer to a very old paper on NALU to see how they do this: https://arxiv.org/pdf/1808.00508. Preferably this work can discuss the motivation of why or why not there is no such regularization term.\n\n• It's an interesting choice to combine experimental setup and results into one section, but I still think it's better to present them separately.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "• Clear and illustrative figure\n\n• Very well-written and easy to read\n\n• Demonstrate that the proposed method performs competitively with ByT5 on XNLI and Spelling Correction.\n\n• The soft and hard deletion switch can be considered novel, at least for applying to this line of work.", "weaknesses": "• The baselines for MrT5 are not properly constructed (Section 5). To see how well MrT5 does, one should compare it with other orthogonal methods/models (e.g., pooling) that reduce sequence length and see how much increase in x-entropy loss they incur.\n\n• For downstream tasks we should compare with non-byte-level models to see the gap: how far are we in terms of accuracy? What's the run-time comparison after this token deletion optimization? These questions are left unanswered in the paper.\n\n• Just evaluating on XNLI and Spelling Correction is not enough to claim that the model is stronger than ByT5, let alone comparing comprehensively with models equipped with traditional tokenizers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lczxVOZVAN", "reviewer": "Reviewer_cEPL", "summary": "This paper introduces MrT5, a ByT5 variant incorporating a learned delete gate for dynamically reducing the byte sequence length during the encoding process. By removing tokens from the sequence, the model is encouraged to implicitly merge the information from the deleted tokens into those which remain. The authors find that this reduction in sequence length leads to significant gains in runtime efficiency at inference time, with minimal differences to baseline performance. They show the deletion mechanism can reduce sequence length in zero-shot scenarios with new languages and see further improvements when training with multi-lingual data.", "questions": "**Q1** When training from scratch, what prevents the model from learning to set G=-30 (minimizing gating loss) and inflating QK^T for all tokens to offset the impact? It seems this would give the model additional capacity without increased loss.\n\n**Q2** Does the specific value of k impact results when fine-tuning existing models, or does k=-30 seem to always work well?\n\n**Q3** If the models are trained until convergence does the performance gap between MrT5 and ByT5 increase?\n\n**Q4** Why was softmax_1 used in the ByT5 baseline? Wouldn’t the unmodified ByT5 perform better?\n\n**Q5** The paragraph beginning at Line 180 suggests that a sequence’s compression rate depends on the least compressed sequence in the batch. Are the rates reported in the paper based on this batch-dependent compression rate or the rate you’d get with a batch size of 1?\n\n**Q6** Similarly, are the performance metrics reported using this explanation of batch processing, or are they reported as if sequences were fully compressed?\n\n**Q7** In Figure 3, why is the English-only trained model doing better than English on many of the other languages?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality**\nMost previous work in this area focus on algorithms for learning “what to keep” when performing sequence compression. This paper provides an interesting alternative by framing the problem as learning “what can be removed”. The proposed deletion mechanism can be added to existing architectures, making their solution easy to integrate into existing models via fine-tuning.\n\n**Quality**\nThe authors conducted several levels of experimentation including simple tasks for gaining an intuition about the mechanism of their approach, a more difficult span completion task to compare against baselines, and down-stream tasks to assess cross-lingual semantic understanding and sensitivity to character-level manipulations.\n\n**Clarity**\nThe paper is well-organized and clearly written. The ideas, methodology, and findings were easy to follow and understand.\n\n**Significance**\nThe paper focuses on improving tokenization-free language models, which is a very relevant topic of interest to the community. Tokenization leads to several issues in LMs, including those listed by the authors as motivation: sensitivity to character-level noise (spelling errors), number representation for mathematical reasoning, and inconsistent compression rates across languages.", "weaknesses": "**1** The results are sensitive to the $/alpha$ hyperparameter. The gate regularization loss does not directly relate $\\alpha$ to the desired compression rate, leading the authors to manually tune $\\alpha$ or use the proposed P-controller to optimize for a desired compression during training. Previous work such as those cited by the authors (Nawrot et al., 2023) use a binomial loss to directly incorporate the desired compression rate into the regularization term. The same approach could be used here to remove the additional complexity.\n\n**2** The location of the gate in a specific network also needs to be tuned. The authors show that for implicit merging of contextualized tokens the gate needs to be placed later in the network, reducing the cost-savings of the approach.\nThe results of the synthetic experiments were not convincing. The deletion gate seemed to only target the correct characters when sequence length was significantly reduced, resulting in a steep drop in sequence-level accuracy.\n\n**3** There were no comparisons with other token-merging models in any of the experiments. It would have been nice to see a head-to-head comparison of this approach with the unsupervised boundary predictor from the cited (Nawrot et al., 2023) paper or the faster Toucan model from (Fleshman and Van Durme, 2023). Stronger baselines are needed throughout.\n\n **4** The models aren’t trained to convergence (footnote 4), and the potential 15% improvement in the baseline through continued training might not translate to MrT5, increasing the performance gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)