MrT5: Dynamic Token Merging for Efficient Byte-level Language Models

## Abstract

Abstract Models that rely on subword tokenization have significant drawbacks, such as sensitivity to character-level noise like spelling errors and inconsistent compression rates across different languages and scripts.
While character- or byte-level models like ByT5 attempt to address these concerns, they have not gained widespread adoption—processing raw byte streams without tokenization results in significantly longer sequence lengths, making training and inference inefficient.
This work introduces MrT5 ( M e r ge T5 ), a more efficient variant of ByT5 that integrates a token deletion mechanism in its encoder to dynamically shorten the input sequence length. After processing through a fixed number of encoder layers, a learnt delete gate determines which tokens are to be removed and which are to be retained for subsequent layers. MrT5 effectively “merges” critical information from deleted tokens into a more compact sequence, leveraging contextual information from the remaining tokens.
In continued pre-training experiments, we find that MrT5 can achieve significant gains in inference runtime with minimal effect on performance. When trained on English text, MrT5 demonstrates the capability to transfer its deletion feature zero-shot across several languages, with significant additional improvements following multilingual training.
Furthermore, MrT5 shows comparable accuracy to ByT5 on downstream evaluations such as XNLI and character-level tasks while reducing sequence lengths by up to 80%.
Our approach presents a solution to the practical limitations of existing byte-level models. https://github.com/jkallini/mrt5

## 1 Introduction

*Subword tokenization*, typically via algorithms such as byte-pair encoding (Sennrich et al., ) or SentencePiece (Kudo & Richardson, ), is a fundamental text preprocessing step that has become ubiquitous in modern language models. Subword tokenizers divide text into meaningful units known as *tokens*, which closely resemble words or parts of words. Tokenization can be seen as a form of compression, since it reduces the sequence length of the input passed to the compute-intensive Transformer (Vaswani et al., ).
However, subword tokenizers have several drawbacks. For example, they are not very robust to character-level noise and manipulations, such as spelling errors (Kaushal & Mahowald, ; Huang et al., ); they directly impact how models process digits and perform arithmetic (Singh & Strouse, ); and they have disproportionate compression rates for different languages and scripts (Ahia et al., ; Petrov et al., ). In addition, current language model APIs charge users per-token, and such discrepancies can cause users of certain languages to be overcharged due to poorer compression.(^1^11See also Andrej Karpathy’s tweets on tokenization: [https://x.com/karpathy/status/1759996551378940395](https://x.com/karpathy/status/1759996551378940395); [https://x.com/karpathy/status/1657949234535211009](https://x.com/karpathy/status/1657949234535211009))

As an alternative to subword models, *tokenization-free* models skip the tokenization preprocessing step entirely by passing the raw character or byte stream directly as input.
However, character- or byte-level sequences tend to be significantly longer than tokenized text sequences, which limits the practical utility of tokenization-free models.
For example, ByT5 (Xue et al., ), a byte-level counterpart of mT5 (Xue et al., ), is competitive with mT5 on a number of tasks, but it has a much slower pre-training and inference runtime.
Most other tokenization-free models perform explicit downsampling or pooling to reduce the sequence length (Clark et al., ; Tay et al., ). However, relevant units of meaning usually span a variable number of bytes/characters. Many of these methods also introduce significant alterations to the standard Transformer architecture that cannot be used to easily adapt existing pre-trained models.

In this work, we propose MrT5 (MergeT5), a variant of the ByT5 architecture that helps address its inefficiencies while allowing more flexibility than fixed-span downsampling methods (Section 3).
MrT5 dynamically merges its encoder’s input into a shorter sequence using a token deletion gating mechanism at a fixed, early encoder layer, as shown in Figure 1.
By allowing the first few encoder layers to process the entire sequence, the encoder creates contextualized representations of the tokens. When the gating mechanism then deletes a subset of the tokens, those that remain keep the contextual
information about those that were removed, allowing information to be implicitly merged into a shorter sequence. During training, we use a deletion regularizer with a tunable weight that can adjust the amount of deletion MrT5 performs. MrT5 effectively learns to merge relevant tokens and delete extraneous ones in a completely unsupervised manner, by optimizing the balance between the regularization objective and language modeling.

We first train several MrT5 models on diagnostic tasks (Section 4); we find that MrT5 not only drops tokens that are irrelevant to the tasks, but also compresses the relevant context into a shorter sequence by applying meaningful deletion patterns tailored to the tasks.
Next, we perform continued pre-training experiments by fine-tuning the MrT5 gating mechanism on top of the pre-trained ByT5 Small (Section 5). Our results show that MrT5 outperforms random and fixed token deletion baselines in terms of span corruption loss while removing an equivalent percentage of tokens.
We also conduct zero-shot tests across 15 diverse languages, showing that MrT5 trained only on English can apply its deletion mechanism to several languages; we also find that we can further improve MrT5’s deletion rates across languages with different scripts through multilingual training. In our final set of experiments, we evaluate MrT5 on XNLI and character-level tasks (Section 6), showing that MrT5 achieves comparable accuracy to ByT5 while significantly improving inference runtimes by reducing the sequence length by up to 80%.
Our approach improves on the main limitations of ByT5, presenting a significant step toward the adoption of byte-level models and the elimination of subword tokenization from modern language models.

Figure: Figure 1: MrT5’s encoder during training and testing. During training, fully-differentiable *soft deletion* masks out tokens using the output of MrT5’s delete gate. During testing, *hard deletion* removes columns from the computation, which reduces the sequence length and leads to efficiency gains. In this visual, the delete gate is placed at layer 2, but the gate placement may be tuned.
Refer to caption: x1.png

## 2 Background: Tokenization-Free Models

Several specialized architectures for character- or byte-level modeling have employed explicit downsampling steps to reduce their input sequence lengths.
For example, CANINE (Clark et al., ) is a character-level model trained on the same languages as mBERT (Devlin et al., ), and it uses convolutional downsampling to reduce the sequence before feeding it to a 12-layer Transformer encoder stack. Charformer (Tay et al., ) is another byte-level encoder-decoder model that learns a gradient-based “soft tokenization” using a block scoring function to select the byte embeddings to pool together for more efficient training and inference.
More recently, MegaByte (Yu et al., ) demonstrated potential in scaling byte-level decoders to long-context tasks by segmenting sequences into fixed-length “patches” upon which local Transformer blocks are applied. However, these patches may not represent meaningful units of text. SpaceByte (Slagle, ) takes a similar approach, adding larger global Transformer blocks to specific byte types, like spaces, though these bytes are also not chosen dynamically.

This paper focuses on ByT5 (Xue et al., ), a byte-level sequence-to-sequence Transformer architecture that serves as a counterpart to mT5 (Xue et al., ), the multilingual version of T5 (Raffel et al., ). ByT5 requires significantly fewer parameters for its vocabulary matrix (which is comprised of only 256 embeddings). However, to compensate for the loss of these parameters, ByT5 has a “heavy” encoder with a larger number of layers than the decoder. While ByT5 shows impressive performance on a variety of downstream tasks,
its heavy encoder, large model and feed forward dimensionalities, and short input sequence length (1024 bytes) make it quite inefficient. The architecture itself requires about 1.2 times more operations than mT5, which contributes to a 33% longer pre-training wall clock time for ByT5, even though it was trained on only a quarter of the data used for mT5. In terms of inference speed on downstream tasks, ByT5 can be up to 10 times slower than mT5, depending on the input sequence length.

MrT5 modifies the ByT5 architecture to make it more efficient. Unlike previous work, MrT5’s deletion gating mechanism can be added to a pre-trained model like ByT5 with minimal fine-tuning using a small number of additional parameters. The gating can also be applied to models trained from scratch. While we focus here on byte-level modeling, our approach can also be applied to subword models, complementing existing work on long-context modeling (described in Appendix A).

## 3 The MrT5 Model Architecture

The unique aspect of MrT5 is its deletion gating mechanism: after a fixed encoder layer, a *delete gate* determines which tokens in the sequence should be kept to be processed by later layers, and which tokens may be deleted, thereby “merging” information into a shorter sequence. Our choice to delete tokens only in a single layer has three motivations: (1) we want to avoid the overhead of executing the deletion algorithm multiple times; (2) we observe that both the performance and the number of deleted tokens stabilize after a few initial layers (see Section 7); and (3) in terms of minimizing computation costs, the deletion is most beneficial if done in an early layer.

### 3.1 Deletion Gating Mechanism

The MrT5 deletion gating mechanism is inspired by existing architectures with gating mechanisms such as Long-Short Term Memory (LSTMs, Hochreiter & Schmidhuber, ), Gated Recurrent Units (GRUs, Cho et al., ), and Mixture-of-Experts (MoEs, Shazeer et al., ).
MrT5’s delete gate is placed after the output of a fixed encoder layer $l$ and is defined by the following function:

$$ $\displaystyle\mathbf{G}$ $\displaystyle=k\sigma(\mathbf{H}_{l}\mathbf{W}+\mathbf{1}_{N}b)$ (1) $$

where $\mathbf{H}_{l}\in\mathbb{R}^{N\times d}$ are the hidden states output by layer $l$; $\mathbf{W}\in\mathbb{R}^{d\times 1}$; $b\in\mathbb{R}$; $\mathbf{G}\in\mathbb{R}^{N\times 1}$; $k$ is a large negative constant; $N$ is the encoder input sequence length; $d$ is ByT5’s hidden state/model dimensionality; and $\mathbf{1}_{N}\in\mathbb{R}^{N\times 1}$ is a vector of ones.
The gating activation function is a rescaled and translated sigmoid function, bounded between $k$ and 0. In our experiments, we use $k=-30$.

#### Hard and Soft Deletion.

During training, MrT5 deletes tokens *softly*, where the outputs of the gating mechanism $\mathbf{G}$ are applied as a soft attention mask. The outputs are added directly to the
self-attention mechanism of the subsequent encoder layers, as well as the cross attentions between the decoder and encoder. As an example, the hidden states of encoder layer $l+1$ are defined as:

$$ $\displaystyle\mathbf{H}_{l+1}$ $\displaystyle=\mathrm{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d }}+\mathbf{1}_{N}\mathbf{G}^{\top}\right)\mathbf{V}$ (2) $$

where $\mathbf{Q},\mathbf{K},\mathbf{V},\mathbf{H}_{l+1}\in\mathbb{R}^{N\times d}$ and $\mathbf{1}_{N}\in\mathbb{R}^{N\times 1}$ is a vector of ones.
A token at sequence position $i\in[1,N]$ and $\mathbf{G}_{i}\approx 0$ will not be masked, whereas a token at sequence position $j\neq i\in[1,N]$ and $\mathbf{G}_{j}\approx k$ will be masked, since $k$ is a large negative constant.
Though soft deletion does not reduce the sequence length, we apply it during training to emulate the effect of token deletion while being fully differentiable. To see efficiency gains during inference, we apply *hard deletion*, where the hidden states are removed from the sequence, determined by a hard threshold; we set this threshold to be $\frac{k}{2}$, half of the range of the delete gate’s output.

For different samples in a given batch, different numbers of tokens may be deleted; when applying hard deletion, the new sequence length is determined by the example in the batch with the largest number of remaining tokens, and the other examples are padded to the new sequence length. In addition to deleting and padding the hidden states, since the T5 architecture uses relative position biases at each layer, the deletion and padding is also performed on the position biases. For a theoretical analysis of the compute savings gained by MrT5’s hard deletion, see Appendix C.

### 3.2 Gate Regularizer

MrT5 allows deletion rates to be adjusted using a tunable regularizer loss:

$$ $\displaystyle\mathcal{L}_{\mathbf{G}}$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\mathbf{G}_{i}$ (3) $$

This loss is the average of the gate output values, which encourages them to be more negative (i.e. closer to $k$, the minimum gate value). In other words, as this loss decreases, the number of deleted tokens increases. The total loss is defined as the sum
$\mathcal{L}=\mathcal{L}_{\text{CE}}+\alpha\mathcal{L}_{\mathbf{G}}$, where $\mathcal{L}_{\text{CE}}$ is the cross entropy loss.
Varying the hyperparameter $\alpha$ allows the MrT5 model to delete more or fewer tokens.

#### Optimizing a Specific Deletion Ratio.

For most of our experiments, we set $\alpha$ by hand, which allows the model to dynamically discover the deletion ratio depending on the difficulty of the task. Alternatively, we can optimize for a specific ratio of deleted tokens. This can be done using an algorithm that resembles the proportional controller (P-controller) from classical control theory. Let’s call the proportion of deleted tokens in the current batch $\delta_{t}\in[0,1]$, the regularization hyperparameter for the current batch $\alpha_{t}$ and the target deletion ratio $\hat{\delta}$. We update $\alpha$ as follows:

$$ $\displaystyle\alpha_{t+1}$ $\displaystyle=\mathrm{clamp}\left(\alpha_{t}+k_{p}(\hat{\delta}-\delta_{t})\right)$ (4) $$

where $\mathrm{clamp}(x)=\max(x,0)$. We found that using $k_{p}=10^{-6}$ and $\alpha_{0}=0.0$ and updating $\alpha$ after every 10 training steps worked well in practice. This method is easier to use than manually setting $\alpha$ and allows $\alpha$ to change dynamically as the model undergoes phase transitions during training, resulting in more stable learning.

#### Softmax 1 .

It is possible for all elements of $\mathbf{G}$ to equal the minimum gate value such that $\mathbf{G}_{i}=k$ for all $i\in[1,N]$. This $\mathbf{G}$ would satisfy the gate regularizer but fail to act as an attention mask, since adding the same value to all elements of the input to a standard $\mathrm{softmax}$ function does not affect its output. To help avoid this scenario, we use $\mathrm{softmax}_{1}$ (Miller, ) in the attention mechanism:

$$ $\displaystyle(\mathrm{softmax}_{1}(\mathbf{x}))_{i}=\frac{\exp(\mathbf{x}_{i}) }{1+\sum_{j}\exp(\mathbf{x}_{j})}$ (5) $$

With the $\mathrm{softmax}_{1}$ function, if $\mathbf{G}_{i}=k$ for all $i\in[1,N]$, as $k$ becomes negative, the sum of attention scores approaches zero.
This eliminates the failure case of using tokens that appear to be all deleted. For consistency, we use $\mathrm{softmax}_{1}$
for both MrT5 and baseline ByT5 models.

## 4 Simulations

We first train tiny 31M-parameter MrT5 and T5 models with 9 encoder layers and 3 decoder layers from scratch on three diagnostic tasks: a simple vowel removal task, a contextual vowel removal task, and a sequence merge task. The purpose of these experiments is to verify that the architecture behaves as intended, particularly with regard to the merging patterns it learns. Does MrT5 merely drop tokens when they are irrelevant to the output, or does it effectively merge relevant context into a shorter sequence? These results help set the stage for our continued pre-training experiments in Section 5 and our downstream task evaluations in Section 6.

#### Diagnostic Task Specifications.

Each of our three diagnostic tasks is a variant of a copy task, designed to assess MrT5’s ability to identify unimportant or redundant information in the input sequence or merge relevant information from some tokens into other tokens. Input sequences are comprised of random lowercase and uppercase English characters, plus start and end tokens. Example inputs and labels are provided in Table 1.

- 1.
Simple Vowel Removal: generate a copy of the input token sequence, except for any vowels. We expect MrT5 to delete vowels, which occur with 19% probability. Thus, the optimal sequence length decrease is 19%.
- 2.
Contextual Vowel Removal: generate a copy of the input token sequence, except for any vowels that follow a lowercase consonant. We increase the probability of vowels such that, on average, they comprise 40% of the sequence, and about 18% of the sequence is comprised of vowels that follow a lowercase consonant. If MrT5 learns the relevant deletion pattern, we would expect a sequence length decrease of 18%.
- 3.
Sequence Merge: generate a copy of the input token sequence, and translate any occurrence of the character sequence ABC into the character D. ABC sequences are inserted into the input randomly and occur about 10 times per sequence on average. If MrT5 merges ABC into a single token, we would expect it to drop 20 tokens on average, which is 15.6% of the sequence.

For the model architecture and training configurations we use for the simulations, see Section B.1.

**Table 1: Diagnostic tasks with example input and target sequences. In our experiments, sequences are 128 characters/tokens long, including a start and end token. Legend: vowels to remove, vowels to keep, sequences to replace.**
| Task | Input | Target |
| --- | --- | --- |
| Simple Vowel Removal | zEKRreJcBxGUJQbZSIos | zKRrJcBxGJQbZSs |
| Contextual Vowel Removal | EOubXgaYVbiOgiIrEnld | EOubXgYVbOgIrnld |
| Sequence Merge | KjAxIpABCZCxBcniABCs | KjAxIpDZCxBcniDs |

#### Results.

Table 2 presents the performance of several MrT5 models that use different regularizer $\alpha$ values trained on each of our three diagnostic tasks. We vary $\alpha$ across model training runs to examine the different deletion patterns that emerge.
In each of the three diagnostic tasks, several MrT5 models with optimal deletion rates outperformed or nearly matched the T5 model while using shorter sequence lengths. We also observed that some MrT5 models, when tuned with the optimal value of $\alpha$, developed token-dropping strategies that exactly aligned with the specific requirements of the tasks. For instance, in the simple vowel removal task, the MrT5 models that selectively dropped vowels but kept consonants intact showed improved performance.(^2^22For tasks requiring more contextual information, we found that these models have too much capacity for the diagnostic tasks, and even the T5 models only started to develop non-trivial attention patterns in very late layers. We found that placing the delete gate at later task layers allowed MrT5 to develop more contextual token deletion solutions.) These findings suggest that MrT5 can create complex deletion strategies that exploit patterns or redundancies in the input.

**(a) Models trained on the simple vowel removal task. MrT5 solves this task effectively with a non-trivial deletion rate, but too much deletion—-beyond the optimal rate of 19%-—results in a drop in performance.**
| Model | Token-level | Seq.-level | Seq. Length | Delete Gate | Description of |
| --- | --- | --- | --- | --- | --- |
| Accuracy (%) | Accuracy (%) | Reduction (%) | Layer | Deleted Tokens |  |
| T5 | 99.97 | 96.44 | 0.00 | — | — |
| MrT5 ($\alpha=0$) | 99.99 | 99.64 | 18.94 | 3 | Most vowels, and no consonants. |
| MrT5 ($\alpha=1\text{e--}4$) | 99.75 | 77.51 | 51.13 | 3 | All vowels and many consonants. |

## 5 Continued Pre-training

In our main set of experiments, we train MrT5 models on the ByT5 span corruption task. In this pre-training objective, spans of tokens in unlabeled text data are replaced with a single *sentinel* token ID per span, and the model must fill in the missing tokens. For ByT5 and MrT5, these are spans of bytes, and the masks can potentially interfere with word boundaries.

We use the English C4 corpus (Raffel et al., ) as well as the multilingual C4 (mC4) corpus (Xue et al., ) for continued pre-training.
We first train several English-only MrT5 and baseline models on C4, and we analyze the performance differences for models with different deletion rates. We also train a multilingual MrT5 and ByT5 model on 15 typologically diverse languages from the mC4 corpus: English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, Hindi, Swahili, and Urdu. We then compare monolingual and multilingual MrT5 in cross-lingual evaluations, analyzing the zero-shot transfer of English MrT5 and evaluating the performance gains achieved through multilingual training.

#### Models.

We train several MrT5 models, varying the $\alpha$ hyperparameter of the deletion regularizer. In our experiments, we found that $\alpha\in[5\text{e--}3,1.5\text{e--}2]$ resulted in a broad range of deletion rates.
This is a continued pre-training setup; we load the pre-trained weights and use the architecture settings of ByT5 Small (300M parameters) and only randomly initialize the weights of MrT5’s gating mechanism. Based on a sweep of different layers, we place the gating mechanism after encoder layer $l=3$ (see Section 7). In addition to the MrT5 models, we also train several baselines:

- 1.
ByT5 baseline: A ByT5 Small architecture with $\mathrm{softmax}_{1}$, but without deletion.(^3^33We found that the loss for ByT5 with $\mathrm{softmax}_{1}$ (0.7805) and the loss for an unaltered ByT5 (0.7815) were comparable, further motivating our use of $\mathrm{softmax}_{1}$. Model training settings are described in Section B.2.)
- 2.
Random baseline: We implement and train a set of models with a random gating mechanism, where the choice of how the tokens are deleted is random; some number of gate values are set to $k$, and the rest are set to 0. In our experiments, we train five random models with different average deletion rates: 15%, 40% 50%, 60%, and 70%.
- 3.
Fixed baseline: We implement and train a set of models that delete the ends of words. We train five models that delete different percentages of the ends of words: 15%, 40% 50%, 60%, and 70%. For details on the implementation of this baseline, see Section B.4.

For the random and fixed baselines, the gating mechanism is placed at layer $l=3$, like the MrT5 models. All models use $\mathrm{softmax}_{1}$ in their attention mechanisms. The ByT5 baseline serves as a lower bound on the best possible span corruption loss, since it does not reduce the sequence length. For further details on model architectures, dataset preparation, and optimization, see Section B.2.

#### Monolingual Results.

Our first set of results are for the monolingual MrT5 models trained only on English data. We evaluate span corruption loss on a held-out test set of 10,000 examples.
Figure 2 illustrates the relationship between span corruption cross entropy loss and sequence length reduction for each MrT5 model compared to their respective baselines. As anticipated, ByT5 achieves the lowest loss; however, it does not reduce sequence length. While the MrT5 models, along with the fixed and random baselines, exhibit higher losses overall, the MrT5 models consistently achieve lower losses than the other baselines at comparable deletion rates. For instance, MrT5 with $\alpha=8\text{e--}3$ removes approximately 57% of tokens, which is a similar deletion rate to the 60% fixed and random baselines, yet MrT5 achieves the lowest loss. This pattern is consistent across all MrT5 models and baselines, showing the effectiveness of MrT5’s gating mechanism in reducing the sequence length while minimizing loss.

Figure: Figure 2: Span corruption cross entropy loss vs. sequence length reduction for each MrT5 and baseline model. MrT5 models consistently have much lower loss than the baselines, and are generally competitive with unmodified ByT5, even where they achieve very large sequence length reductions.
Refer to caption: x2.png

MrT5 models with a higher deletion rate also have a faster inference runtime, as shown in Table 3. In particular, MrT5 models that reduce the sequence length by more than 50% can achieve 25% speedup or greater, with our implementation.
These results show that hard deletion improves the efficiency of MrT5 when compared to ByT5, and this speedup can be tuned.

**Table 3: Average inference runtime for a single sequence (in milliseconds) for ByT5 and each MrT5 model. Percentage decrease in runtime relative to ByT5 is displayed in parentheses.**
| Model | ByT5 | MrT5 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\alpha=5\text{e--}3$ | $\alpha=6\text{e--}3$ | $\alpha=8\text{e--}3$ | $\alpha=1\text{e--}2$ | $\alpha=1.4\text{e--}2$ |  |  |
| Runtime (ms) | 56.27 | 53.98 ($\downarrow$ 4.08%) | 43.92 ($\downarrow$ 21.96%) | 40.78 ($\downarrow$ 27.53%) | 39.62 ($\downarrow$ 29.59%) | 33.81 ($\downarrow$ 39.92%) |

#### Multilingual Results.

In our second set of results, we evaluate MrT5 and ByT5 models on 15 languages. For the monolingual models trained on English only, these are zero-shot evaluations. Each language’s test set is sampled from its mC4 validation split and contains 10,000 examples, except for Swahili and Urdu, which only have 2,800 and 9,300 examples, respectively.
Figure 3 shows the test set span corruption loss versus sequence length reduction for a monolingual MrT5 (with constant $\alpha=1\text{e--}2$) and a multilingual MrT5 model (with constant $\alpha=1.2\text{e--}2$) across 15 languages. We choose to compare these two models because they learned similar deletion rates for English ($\approx$63%). We also include the corresponding ByT5 models’ losses, trained on the same data, as a baseline.

The monolingual MrT5 model achieves the highest sequence length reduction for English, as expected, and performs moderately well in zero-shot settings for Latin-script languages (e.g. Spanish, French, Vietnamese, and Russian), with reduction rates above 40%. However, it struggles with sequence length reduction for Chinese, which uses a completely different script. In contrast, the multilingual MrT5 significantly improves the sequence length reduction across all languages, with most languages achieving 60% reduction; at the same time, the loss increase compared to ByT5 shrinks across all languages. These results demonstrate that the MrT5 architecture can learn to perform contextual deletions and transfer this capability across languages, but its effectiveness in doing so, especially across different scripts, is greatly enhanced when trained with multilingual data.

Figure: Figure 3: Average test set span corruption loss vs. sequence length reduction for monolingual (top) and multilingual (bottom) MrT5 models across 15 languages. ByT5 is shown for loss comparison only (it does not reduce the sequence length). MrT5 can transfer its deletion feature to new languages when trained in English, but performs best with multilingual training. A multilingual MrT5 model achieves over 50% sequence length reduction in most languages with minimal effect on the loss.
Refer to caption: x3.png

## 6 Downstream Task Evaluations

We next assess MrT5’s performance on downstream tasks, specifically XNLI and two character-level tasks. XNLI evaluates MrT5’s ability to understand semantic relationships between sentences, while the character-level tasks test whether it retains its sensitivity to character-level manipulations.

#### Cross-lingual Natural Language Inference.

We first test our multilingual MrT5 model using the Cross-lingual Natural Language Inference (XNLI) corpus (Conneau et al., ), a benchmark for cross-lingual sentence classification with 5,000 parallel examples in 15 languages. These are the same 15 languages our multilingual MrT5 model was trained on. We selected XNLI for testing because the ByT5 authors noted it as one of the downstream tasks with the worst inference runtimes, primarily due to the long input sequences; ByT5 was 6.4 to 9.5 times slower than mT5. We fine-tune two models on the English MultiNLI corpus (Williams et al., ): our multilingual MrT5 model ($\alpha=1.2\text{e--}2$) and the baseline multilingual ByT5 model. For training details, see Section B.3.

Table 4 presents the evaluation metrics for the ByT5 and MrT5 models on the XNLI English test split, as well as their average performance across all languages.
On English, MrT5 outperforms ByT5’s accuracy while reducing the sequence length by 52.56% and cutting down the inference runtime by 38.0%. Across languages, MrT5 significantly decreases both the sequence length and runtime, with a small accuracy decrease of just 1.7% compared to ByT5. The XNLI task illustrates the efficiency of MrT5’s fast encoder; a 50% deletion rate results in substantial gains in runtime. This improvement can be attributed to the setup of the XNLI task, which requires large input sequences (up to 1,024 tokens) and short decoder sequences (a single token for classification).(^4^44We note that the accuracy of our ByT5 Small model on XNLI (51.3%) falls short of the accuracy reported in the ByT5 paper (69.1%). We attribute this discrepancy primarily to the significantly longer training duration used in their fine-tuning setup—262,144 steps, compared to our 4,000 steps.)

**Table 4: XNLI evaluation metrics for ByT5 and MrT5 models. Except for English, all evaluations are zero-shot. Chance accuracy is 33%. MrT5 achieves significantly reduced sequence lengths and faster runtimes compared to ByT5 while maintaining comparable accuracy, and it even outperforms ByT5 on English. See Table 6 in Appendix D for metrics on each language individually.**
| Language | Accuracy (%) | Average Runtime (ms) | MrT5 Runtime<br>Decrease (%) | MrT5 Seq. Len.<br>Reduction (%) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ByT5 | MrT5 | ByT5 | MrT5 |  |  |  |
| English | 76.47 | 78.88 | 8.95 | 5.55 | 38.00 | 52.56 |
| All Languages | 51.34 | 49.63 | 14.25 | 8.19 | 42.50 | 46.82 |

#### Character-level Tasks.

We fine-tune and evaluate an English MrT5 model on the *Spelling Correction with Context* and *Word Search* character-level tasks from Huang et al. (). In the Spelling Correction task, the input is a sentence containing a spelling error, and the goal is to generate the same sentence with the error corrected. In the Word Search task, the input follows the format definition: letters, and the objective is to identify the substring in letters that, when reversed, matches the given definition. We selected these two tasks from the suite because they both require understanding meaning and context and involve processing longer sequence lengths.
For each character-level task, we fine-tune two models: our English-only MrT5 model ($\alpha=1\text{e--}2$) and the baseline English-only ByT5 model. See Section B.3 for fine-tuning optimization details.
We evaluate on all test splits from Huang et al. (), which are designed to rigorously assess a model’s ability to integrate meaning and context in its predictions.

Table 5 displays the test set results for each character-level task. For both tasks, MrT5 reduces the sequence length by over 70%, speeding up the inference runtime by 32.9% for the Spelling Correction task and and 55.15% for the Word Search task. At the same time, MrT5 maintains accuracy scores that are competitive with ByT5. These results show that MrT5’s method of sequence merging effectively preserves its sensitivity to character-level information.

**Table 5: Character-level task metrics for ByT5 and MrT5 models. MrT5 significantly reduces runtime and sequence lengths across both tasks, with a minimal accuracy trade-off in spelling correction and comparable accuracy in word search. See Table 7 and Table 8 in Appendix D for metrics on individual test splits for each task.**
| Task | Seq.-level Accuracy (%) | Average Runtime (ms) | MrT5 Runtime<br>Decrease (%) | MrT5 Seq. Len.<br>Reduction (%) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ByT5 | MrT5 | ByT5 | MrT5 |  |  |  |
| Spelling Correction | 58.19 | 56.07 | 3.25 | 2.18 | 32.90 | 78.88 |
| Word Search | 75.37 | 74.30 | 4.95 | 2.22 | 55.15 | 73.91 |

## 7 Analysis

#### Per-sample Sequence Length Reduction.

We present a per-sample analysis of the cross entropy loss and sequence length reduction for the English MrT5 models and random baselines. We take a sample of 1,000 English sentences from the mC4 test set and calculate the percent increase in loss on a per-sample basis, using the English ByT5’s loss as the baseline (i.e. the percent increase between the MrT5 model’s loss and ByT5’s loss for individual samples).
For each sample, we also get the sequence length reduction.
Across five MrT5 models with different deletion rates, we found no correlation between the percent increase in the loss and the percentage of tokens deleted (average correlation of $r=-0.014$).(^5^55When averaging correlation coefficients, we apply Fisher’s Z transformation to stabilize the variance.) This reflects what we would expect from the MrT5 models; for an individual sample, MrT5 learns when it can delete more tokens without incurring a large increase in the loss.
In contrast, for five random models with different sequence length reduction rates, we found a moderate positive correlation (average correlation of $r=0.298$). When the random model removes more tokens, it is more likely to cause an increase in loss. These results further support the observation that the MrT5 models more strategically and contextually delete tokens compared to the baselines. See Figure 6 in Appendix E for plots presenting the correlations for each individual MrT5 and random baseline model.

#### Gate Placement.

We present an analysis of MrT5 models with delete gates placed at various encoder layers. To maximize efficiency, it is ideal to position the delete gate in the earliest encoder layer possible. However, placing the gate too early reduces the contextual information in the token representations, leading to more significant performance degradation compared to ByT5. We trained several English-only MrT5 models, all using the same training setup and architecture as described in Section 5. The only variable was the placement of the delete gate, as shown in Figure 4. We perform two experiments: one in which all models are trained with a fixed regularizer $\alpha=1\text{e--}2$ (Figure 4(a)), and another where all models are trained with a P-controller that targets $\approx$50% deletion (Figure 4(b)). In both setups, the loss is higher when the gate is placed in early layers.
Since our goal is to place the gate as early as possible, we selected layer 3 as the optimal point for gate placement, since its loss and deletion rate is comparable to layers 4 and 5. This analysis provides further evidence that MrT5 merges information into fewer tokens, since deletion at earlier, less contextual layers results in a higher loss.

The gate placement sets an upper bound on the compute savings achievable with our model. We provide a detailed analysis of MrT5’s theoretical compute savings in Appendix C. With typical hyperparameters from our tasks, we can achieve a maximum speedup of around three times with reasonable deletion rates.

Figure: (a) MrT5s trained with fixed $\alpha=1\text{e--}2$.
Refer to caption: x4.png

## 8 Conclusion

In this paper, we introduce MrT5 (MergeT5), a variant of the ByT5 architecture designed to address the inefficiencies of byte-level language modeling. MrT5’s token deletion mechanism forces the model to merge input tokens into a more compact sequence, allowing for computational savings while preserving model performance. Our diagnostic experiments demonstrate that MrT5 effectively merges relevant context into a shorter sequence using strategies that align with task-specific objectives. In our continued pre-training experiments, MrT5 outperforms both random and fixed deletion baselines when trained in English, and with multilingual training, MrT5 achieves over 50% reduction in sequence length across multiple languages with minimal impact on the loss. Our model learns very fast: the continued pre-training requires only a few thousand additional training steps.
Furthermore, MrT5 maintains competitive accuracy with ByT5 on downstream tasks such as XNLI and character-level manipulations while improving inference runtimes. This demonstrates MrT5’s capacity to handle tasks requiring semantic information, while still effectively processing character-level details–the main advantage of byte-level modeling.
Our work takes a significant step toward the viability of byte-level language models and eliminating the need for subword tokenization.

## Acknowledgments

The authors would like to thank the members of the Stanford NLP Group and Jurafsky Lab for helpful feedback and discussions at various stages of this project. Julie Kallini is supported by a National Science Foundation Graduate Research Fellowship under grant number DGE-2146755. This work is supported in part by a grant from the Laboratory Directed Research and Development program at Sandia National Laboratories. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia LLC, a wholly owned subsidiary of Honeywell International Inc. for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

## Reproducibility Statement

Steps for reproducing each of our experiments are detailed in Appendix B. Descriptions of the model architectures and training configurations/hyperparameters for our diagnostic task experiments are provided in Section B.1; details of the model architectures, span corruption data preprocessing steps, and training configurations/hyperparameters for continued pre-training are provided in Section B.2; and training configurations/hyperparameters for fine-tuning on the XNLI and character-level downstream tasks are provided in Section B.3.
We provide our source code at [https://github.com/jkallini/mrt5](https://github.com/jkallini/mrt5).

## Appendix A Further Related Work

#### Early Exit Models.

Our model is closely related to early-exit methods proposed for autoregressive Transformers (Elbayad et al., ; Schuster et al., ) and BERT (Xin et al., ; ). In contrast to previous approaches, our method is fully differentiable and does not require special training considerations or calculating the entropy of the final classifier, and the deletion decisions are made in a single layer, making the model efficient and easy to use.

#### Long-context Subword Models.

Other work has attempted to address issues with long sequences in Transformer models more generally. For example, Hierarchical Transformers (Nawrot et al., ) add several layers of downsampling and upsampling to handle long sequences in decoder models. Follow-up work has implemented dynamic pooling using boundary predictors, but these usually involve a supervised training step (Nawrot et al., ). Perceiver AR (Hawthorne et al., ) is an autoregressive model that employs cross-attention to condense inputs into a smaller set of latent representations. This approach enables it to handle attention over large input sequences of up to a hundred thousand tokens. However, the authors did not specifically evaluate the model on character-level tasks or language tasks without tokenization. Other solutions include Nugget (Qin & Van Durme, ; Qin et al., ), which encodes the whole sentence, but passes only a dynamic subset of the embeddings to the decoder. This approach does not save compute on the encoder side.

## Appendix B Experimental Details

### B.1 Simulation Details

#### Model Architectures.

We train our diagnostic models with 9 encoder layers and 3 decoder layers, following the 3:1 ratio of encoder to decoder layers in ByT5, and we use $d_{\text{ff}}=1024$ and $d_{\text{model}}=512$. We use $\mathrm{softmax}_{1}$ for all T5 and MrT5 models.
Other architectural settings match the standard ByT5 Small, resulting in an architecture with 31M parameters (10% of ByT5 Small’s parameter count).

#### Optimization.

We use a batch size of 128 examples and a sequence length of 128 tokens, and we train each model for a total of 20,000 gradient steps.
We use the AdamW optimizer with a learning rate that linearly warms up to 1e–4 over 1,000 steps and linearly decays.
For MrT5 models, the delete gate’s regularizer is enabled half-way through training, at 10,000 steps. We set a constant regularizer $\alpha$ throughout training.

### B.2 Continued Pre-training Details

#### Model Architectures.

All MrT5 and baseline models use the model configuration of a standard ByT5 Small, which has $d_{\text{ff}}=3584$, $d_{\text{model}}=1472$, 12 encoder layers, 4 decoder layers, and 300M total parameters. The only difference for MrT5 and the baselines is the additional delete gate and the use of $\mathrm{softmax}_{1}$ in the attention mechanisms.

#### Data.

When training on the span corruption objective, we calculate the corrupted spans such that the average masked span length is 20 tokens with a noise density of 15% (i.e. 15% of tokens in the sequence are masked out), following the specification in the ByT5 paper.
For both monolingual and multilingual model training, we ensure that the samples of the mC4 corpus are sufficiently large to avoid training the models for multiple epochs. In the case of multilingual training, we extract equal-sized samples for each language from the mC4 training split.

#### Optimization.

We train each model for 3,000 gradient steps over batches of $2^{20}$ tokens (i.e. an encoder sequence length of 1024 with an effective batch size of 1024).
We use the AdamW optimizer with an initial learning rate of 1e–4 with linear decay and no warmup. As with the simulations from the previous section, we keep the gate regularizer $\alpha$ constant throughout training to allow the model to discover its own deletion rate, but we vary $\alpha$ across runs. Since we are continuing to train on ByT5’s pre-training objective, we do not delay the regularizer.

At test time, we use an eval batch size of $2^{14}$ tokens (i.e. an encoder sequence length of 1024 with a batch size of 16). We use use the last model checkpoint at step 3,000 for all evaluations.

### B.3 Downstream Task Details

#### XNLI Training Details.

We train all models for 4,000 gradient steps ($\approx$10.43 epochs) with a batch size of 1,024 and a maximum sequence length of 1,024 tokens.
We use the AdamW optimizer with an initial learning rate of 1e–3 that linearly decays with no warmup. For MrT5, we apply a P-controller to achieve a target deletion ratio $\hat{\delta}=0.5$, aiming for a sequence length reduction of $\approx$50%, and we delay the regularizer until 500 steps into fine-tuning. The controller parameters are $k_{p}=10^{-6}$ and $\alpha_{0}=0.0$.

We evaluate both models on the XNLI dataset, testing them in English and conducting zero-shot evaluations in the 14 additional languages. For evaluation, we use a batch size of 16 and a maximum sequence length of 1,024 tokens.

#### Character-level Task Training Details.

For the Spelling Correction task, we train for 200,000 gradient steps ($\approx$33.7 epochs); for the Word Search task, we train for 300,000 steps ($\approx$ 61.0 epochs). When training MrT5, we apply a P-controller to achieve a target deletion ratio. On the Spelling Correction task, we set the target deletion ratio $\hat{\delta}=0.6$, aiming for a sequence length reduction of $\approx$60% (although the model learned to delete closer to $\approx$80% of tokens). On the Word Search task, we set a target deletion ratio $\hat{\delta}=0.7$, aiming for a sequence length reduction of $\approx$70%. We delay the regularizer until 10,000 steps into fine-tuning for both tasks.

For both tasks, we use a batch size of 16 examples, and we use the AdamW optimizer with an initial learning rate of 5e–4 that linearly decays with no warmup. For evaluation, we use a batch size of 16. The Spelling Correction and Word Search tasks have a maximum input sequence length of 64 and 128 tokens, respectively.

### B.4 Description of Fixed Deletion Baselines

The fixed deletion baseline deletes tokens at layer $l=3$ using deterministic rules based on the token identity. All tokens/columns corresponding to whitespace, punctuation, and symbolic characters are identified: \t,
\n,
␣, !,
",
#,
$,
%,
&,
’,
(,
),
⁢,
+,
,,
-,
.,
/,
:,
;,
<,
=,
>,
?,
@,
[,
\,
],
,̂
_,
,̀
{,
|,
},
~,

.
These separator tokens are used to locate word boundaries. Then, based on the fixed deletion percentage, the delete gate will drop the tokens/columns corresponding to the ends of words. For example, if the target percentage is 50%, the delete gate will remove the tokens corresponding to the final two characters of a five letter word, and the final three characters of a six character word.

## Appendix C Theoretical Compute Savings

Here we analyze the theoretical amount of compute used by MrT5 given a deletion rate. Let’s call the average length of the input sequence $N_{E}$ and the average length of the output sequence $N_{D}$. The width of the residual is $d_{\text{model}}$, the dimension of the up-projection in the MLP is $d_{\text{ff}}$, our encoder has $L_{E}$ and the decoder has $L_{D}$ layers. The deletion occurs after the $L_{\text{del}}$ layer and the average proportion of deleted tokens is $\delta$. We assume that the total size of the head projections is equals to $d_{\text{model}}$, as typical for Transformers ($d_{\text{head}}*N_{\text{heads}}=d_{\text{model}}$). Then, we can approximate the total number of multiply-accumulate operations (MACs) for the model as follows. Before deletion, the self attention in the encoder uses $N_{E}d_{\text{model}}^{2}$ MACs for both the Q, K, V and the output projections and $N_{E}^{2}d_{\text{model}}$ MACs for both the $\mathbf{A}=\mathbf{Q}\mathbf{K}^{T}$ and $\mathbf{A}\mathbf{V}$ projections. The MLP layer uses $N_{E}d_{\text{model}}d_{\text{ff}}$ additional MACs. Thus, the total number of MACs used per layer is $4N_{E}d_{\text{model}}^{2}+2N_{E}^{2}d_{\text{model}}+N_{E}d_{\text{model}}d_{
\text{ff}}$. This much compute is used for the first $L_{\text{del}}$ layers, after which the sequence length is reduced to $N_{E}(1-\delta)$ for the remaining $L_{E}-L_{\text{del}}$ layers. Thus, the encoder uses

$$ $\begin{split}N_{\text{MACs}}^{\text{encoder}}=\left(L_{E}-L_{\text{del}}\right )\left(1-\delta\right)\left(4N_{E}d_{\text{model}}^{2}+2N_{E}^{2}d_{\text{ model}}(1-\delta)+N_{E}d_{\text{model}}d_{\text{ff}}\right)+\\ L_{\text{del}}\left(4N_{E}d_{\text{model}}^{2}+2N_{E}^{2}d_{\text{model}}+N_{E }d_{\text{model}}d_{\text{ff}}\right)\end{split}$ (6) $$

The MACs used by the decoder can be calculated similarly, but additionally the cross attention has to be taken into account. The cross attention uses $N_{E}(1-\delta)d_{\text{model}}^{2}$ MACs for the K and V projections, $N_{D}d_{\text{model}}^{2}$ MACs for the Q and output projections, and $(1-\delta)N_{E}N_{D}d_{\text{model}}$ MACs for the attention matrix itself.

$$ $\begin{split}N_{\text{MACs}}^{\text{decoder}}=L_{D}\left(4N_{D}d_{\text{model} }^{2}+2N_{D}^{2}d_{\text{model}}+N_{D}d_{\text{model}}d_{\text{ff}}+2N_{E}(1- \delta)d_{\text{model}}^{2}+\right.\\ \left.2N_{D}d_{\text{model}}^{2}+2(1-\delta)N_{E}N_{D}d_{\text{model}}\right) \end{split}$ (7) $$

Note that $\delta=0$ corresponds to the baseline ByT5. For MrT5, $d_{\text{model}}=1472,d_{\text{ff}}=3584,L_{E}=12,L_{D}=4,L_{\text{del}}=3,N_{
E}=1024,N_{D}=189$. Given these numbers, we plot the total reduction in compute (compared to ByT5) as a function of $\delta$ in Fig. 5.

Figure: Figure 5: Reduction in the total amount of compute as a function of the deletion ratio $\delta$.
Refer to caption: x6.png

## Appendix D Additional Downstream Task Evaluations

Table 6 contains XNLI evaluation metrics for ByT5 and MrT5 for each of the 15 languages. Table 7 contains evaluations on all test splits for the Spelling Correction with Context task. Table 8 contains evaluation on all test splits for the Word Search task. T5 results for the character-level tasks are from the subword tokenization baseline reported in (Huang et al., ).

**Table 6: Per-language XNLI evaluation metrics for ByT5 and MrT5 models. Except for English, all evaluations are zero-shot.**
| Language | Accuracy (%) | Average Runtime (ms) | MrT5 Runtime<br>Decrease (%) | MrT5 Seq. Len.<br>Reduction (%) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ByT5 | MrT5 | ByT5 | MrT5 |  |  |  |
| English | 76.47 | 78.88 | 8.95 | 5.55 | 38.0 | 52.56 |
| French | 53.90 | 52.25 | 10.75 | 6.66 | 38.0 | 47.58 |
| Spanish | 55.99 | 53.84 | 10.13 | 6.09 | 39.9 | 50.67 |
| German | 45.68 | 46.20 | 10.42 | 6.59 | 36.8 | 47.61 |
| Greek | 52.01 | 52.33 | 19.08 | 9.77 | 48.8 | 54.79 |
| Bulgarian | 56.31 | 53.16 | 17.58 | 9.68 | 44.9 | 47.82 |
| Russian | 55.35 | 55.85 | 18.02 | 9.95 | 44.8 | 47.95 |
| Turkish | 44.86 | 43.31 | 9.73 | 6.03 | 38.0 | 46.72 |
| Arabic | 51.89 | 47.29 | 14.12 | 8.31 | 41.1 | 43.78 |
| Vietnamese | 48.63 | 47.39 | 12.60 | 7.67 | 39.2 | 46.51 |
| Thai | 48.35 | 44.57 | 25.28 | 11.77 | 53.5 | 61.07 |
| Chinese | 50.70 | 50.50 | 7.94 | 5.83 | 26.7 | 24.96 |
| Hindi | 43.87 | 39.99 | 24.50 | 13.68 | 44.2 | 43.56 |
| Swahili | 40.64 | 38.42 | 9.08 | 5.62 | 38.1 | 48.48 |
| Urdu | 45.48 | 40.45 | 15.63 | 9.77 | 37.5 | 38.32 |
| All Languages | 51.34 | 49.63 | 14.25 | 8.19 | 42.5 | 46.82 |

**Table 7: Evaluation metrics for all splits for the Spelling Correction with Context character-level task. The “Dependent” split requires incorporating context to correct the spelling error; the “Independent” split does not.**
| Spelling Correction<br>Test Split | Seq.-Level Accuracy (%) | Average Runtime (ms) | MrT5 Runtime<br>Decrease (%) | MrT5 Seq. Len.<br>Reduction (%) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ByT5 | MrT5 | T5 | ByT5 | MrT5 |  |  |  |
| Dependent | 37.63 | 35.05 | 36.59 | 3.28 | 2.26 | 31.21 | 78.77 |
| Independent | 76.45 | 74.76 | 58.11 | 3.21 | 2.11 | 34.44 | 78.98 |
| All Splits | 58.19 | 56.07 | 47.98 | 3.25 | 2.18 | 32.90 | 78.88 |

**Table 8: Evaluation metrics for all splits for the Word Search character-level task. The “OOV” split contains hidden words with mT5 tokenization not seen in the training split (this does not apply to our work, since we only train byte-level models, not subword models); the “Paraphrase” split contains definitions from *The Online Plain Text English Dictionary*, testing the ability to understand context; the “Overlap” split contains overlapping hidden words; and the “Paraphrase + Overlap” split contains both paraphrased definitions and overlapping hidden words.**
| Word Search<br>Test Split | Seq.-Level Accuracy (%) | Average Runtime (ms) | MrT5 Runtime<br>Decrease (%) | MrT5 Seq. Len.<br>Reduction (%) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ByT5 | MrT5 | T5 | ByT5 | MrT5 |  |  |  |
| OOV | 78.05 | 79.99 | 24.07 | 5.20 | 2.55 | 50.95 | 72.52 |
| Paraphrase | 83.67 | 82.06 | 71.17 | 4.89 | 2.17 | 55.58 | 72.27 |
| Overlap | 77.58 | 76.22 | 93.65 | 4.94 | 2.20 | 55.55 | 76.14 |
| Paraphrase + Overlap | 58.46 | 57.60 | 64.13 | 4.95 | 2.21 | 55.41 | 73.95 |
| All Splits | 75.37 | 74.30 | 72.90 | 4.95 | 2.22 | 55.15 | 73.91 |

## Appendix E Additional Per-sample Analyses

Figure 6 shows per-sample correlation plots between the cross entropy loss percent increase and the sequence length reduction for five MrT5 models and five random baseline models with different deletion rates.

Figure: (a) MrT5 models. All models show no correlation between loss increase and sequence length reduction, showing the MrT5 can deletes at different percentages depending on the sequence, without incurring a loss increase.
Refer to caption: x7.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions: 
- I'd like to see the 'all languages' split for the downstream tasks discussed in the main text to match the CE analysis -> is it the case for example that you see a performance hit and no sequence length reduction for Chinese when the deletion gate is used zero-shot on a task, following Figure 3? Essentially, it would be nice to clearly state what the possible performance cost of introducing the deletion gate is, and some sense of the variability across languages, on explicit task performance as well as the loss presented in Figure 3. 
  - There is not much difference in performance between MrT5 and ByT5 for Chinese XNLI in Appendix Table 6, and there is ~20% length reduction - why do you think there is a difference to Chinese in the top half of Figure 3 with only a ~2% length reduction? Similarly, for Swahili, there seemed to be a significant difference in CE in Figure 3 zero-shot, but a smaller gap in Table 6. Why do you think that is? I suggest addressing these inconsistencies in the main text and providing possible explanations for the differences between training and downstream task performance. 

Suggested citations:  
- [Cherry et al 2018](https://arxiv.org/abs/1808.09943) learns to delete characters for temporal compression with character-level models in MT
- [Limisiewicz et al 2024](https://arxiv.org/pdf/2403.10691) uses morphologically inspired compression for byte sequences to create MyT5, a ByT5 variant 


Presentation nits:
- L99: "the main the limitations" -> "the main limitations"
- How α was set was not super clear to me. L196: "For most of our experiments, we set α by hand, which allows the model to dynamically set the deletion ratio [based on the loss]." This paragraph says α was set by hand most of the time, but that it is easier to allow α to dynamically change. Do any of the experiments in the main text allow α to change? (It doesn't seem so?) Why not, if it is easier?
soundness: 3
strengths: A straightforward and lightweight addition to ByT5 models which can provide significant efficiency improvements (up to 80%) with a small trade-off in accuracy (~2%) compared to ByT5 (and often small improvements for English).
summary: This paper presents MrT5 (Merge-T5), a more efficient variant of ByT5 that introduces dynamic sequence shortening through a learned token delete gate after the first few encoder layers. Though it is called Merge-T5, no tokens are merged together, only deleted or preserved. 
The method is a lightweight addition to a typical transformer architecture, with fully-differentiable soft deletion used during training via attention-masking and a regularizer with a tuned weight to adjust the degree of deletion, and hard deletion at inference time. 
Performance is shown to be comparable to ByT5 (+/- 2% accuracy) while reducing sequences by up to 80% and runtime up to ~55%. 
Only 4,000 steps are necessary to adapt a ByT5 model to a MrT5 model. 

Experiments are well designed and illustrate the properties of the method in different settings.  
First, synthetic tasks (vowel deletion, token merging) are used to understand learned deletions with controlled settings and small 31M parameter models.  
Next, they introduce token deletion in continued pre-training of ByT5-small both on English and multilingually, ablating components of the mechanism and different degrees of deletion. The task is span corruption loss, which for byte-level input may affect character and word boundaries. 
MrT5 is (generally) able to delete bytes in other scripts even if they have been unseen (if the model has been pretrained on English only), but results in slightly higher losses than ByT5. Multilingual pretraining reduces but does not remove the loss difference between MrT5 and ByT5. For Chinese, though, token deletion does not occur zero-shot.  
--> It is not clear the significance of the differences in CE in Figure 2 & 3 here - it would be nice if this could be made clear.  
Finally, MrT5 is evaluated on downstream tasks, XNLI and 2 character-level tasks.  
For English, MrT5 outperforms ByT5 with a reduction in sequence length and inference time of ~50%, while averaged across all languages there is a slight performance degradation of ~2% (though a significant speedup).   
On two English character-level tasks (contextual spelling correction and word search), MrT5 again saw ~2% accuracy drop compared to ByT5 in exchange for a 30-55% runtime decrease.  
When comparing different layer placements, it seems that a middle layer (3) balances training stability and deletion level, allowing contextual representations to be learned before deleting tokens while still pruning sufficient tokens for efficiency gains. 

It would be good if it were made clearer what the possible performance cost of introducing the deletion gate is, and some sense of the variability across languages, on explicit task performance as well as the loss presented in Figure 3.
weaknesses: Efficiency gains (though generally significant) can come at a slight performance cost for non-English languages, and it is not clear how much variance in this cost there may be. 

- Without continued training in non-English scripts, there can be performance drops for non-English languages with less efficiency improvements. Figure 3 suggests that for Chinese for example, there may be almost no reduction in seq length zero-shot for the span corruption task, and some languages have relatively large drops in CE compared to ByT5
- For the downstream experiments, the results are reported in aggregate for non-English languages in the main text. While the results are broken out by languages in the Appendix, a mention of the variance across languages and a comment on the relationship between these results and those in the previous section would make it clearer what the potential trade-offs are and when they arise

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: If possible, I am interested in comparing the effectiveness of additional parameters introduced by MrT5 in learning to delete tokens.
soundness: 3
strengths: * A simple yet effective method that enables models to dynamically learn how to delete tokens from byte-level inputs.
* Controlled experiments and results on downstream tasks support the authors' claims.
* MrT5 demonstrates competitive inference speed compared to ByT5.
summary: This work builds on ByT5 and introduces a new token deletion mechanism that dynamically determines how to remove unnecessary tokens from the sequence without compromising performance.

The authors incorporate additional neural layers to learn the optimal token deletion process. During training, a loss function is introduced to encourage the model to progressively delete tokens (softly) or to achieve a specific ratio. During inference, the layers identify and discard unimportant tokens in a hard manner.

Controlled experiments on synthetic tasks demonstrate that the method, MrT5, can effectively learn to compress the input sequence. Further results on downstream tasks, such as XNLI, also support the authors' claims.
weaknesses: * If I understand correctly, during training, the byte tokens are deleted softly, meaning there are still significant burdens for byte-level language models given that standard attention has quadratic time complexity, which limits their scalability to larger sizes.
* The experiments presented utilize moderate model sizes, which may constrain the overall persuasiveness of the proposed method.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: Suggestions:

• 034-035: "... via algorithms such as byte-pair encoding (Sennrich et al., 2016) or SentencePiece (Kudo & Richardson, 2018) ..."
Consider rephrasing this sentence as BPE is part of SentencePiece.

• Consider discussing the relation between this work and GemFilter (https://arxiv.org/pdf/2409.17422) as both pertain to token deletion.

• The proposed method seems readily transferrable to decoder-only models, which is the most widely used architecture nowadays. Would like to see some experiments, or at least some discussions about this direction.

• 150-151: "(1) we want to avoid the overhead of executing the deletion algorithm multiple times;"
This motivation is better discussed from the perspective of "trade-off". If executing the algorithm multiple times can reduce the number of tokens/positions to process in later layers even more without compromising generation quality, then there is no reason to not do it.

• The regularizer loss only seems to encourage the increase of the number of deleted tokens, but does not encourage the gate output to converge to the extreme values (i.e., the min or max gate value). This could make hard deletion during inference less effective because merely setting a threshold may delete some "somewhat useful" tokens. Please refer to a very old paper on NALU to see how they do this: https://arxiv.org/pdf/1808.00508. Preferably this work can discuss the motivation of why or why not there is no such regularization term.

• It's an interesting choice to combine experimental setup and results into one section, but I still think it's better to present them separately.
soundness: 2
strengths: • Clear and illustrative figure

• Very well-written and easy to read

• Demonstrate that the proposed method performs competitively with ByT5 on XNLI and Spelling Correction.

• The soft and hard deletion switch can be considered novel, at least for applying to this line of work.
summary: This work improves on top of ByT5 by adding a delete gate at a certain layer to remove unimportant tokens. The proposed model trains a soft deletion layer which is fully differentiable, and then uses the same layer for hard (i.e., discrete) deletion during inference. The model only requires unsupervised data for training. 

The model is evaluated on monolingual and cross-lingual pretraining tasks, showing a trade-off between cross-entropy loss and sequence length reduction. On two evaluation tasks, the model is also shown to be competitive with ByT5 while reducing sequence length.
weaknesses: • The baselines for MrT5 are not properly constructed (Section 5). To see how well MrT5 does, one should compare it with other orthogonal methods/models (e.g., pooling) that reduce sequence length and see how much increase in x-entropy loss they incur.

• For downstream tasks we should compare with non-byte-level models to see the gap: how far are we in terms of accuracy? What's the run-time comparison after this token deletion optimization? These questions are left unanswered in the paper.

• Just evaluating on XNLI and Spelling Correction is not enough to claim that the model is stronger than ByT5, let alone comparing comprehensively with models equipped with traditional tokenizers.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: **Q1** When training from scratch, what prevents the model from learning to set G=-30 (minimizing gating loss) and inflating QK^T for all tokens to offset the impact? It seems this would give the model additional capacity without increased loss.

**Q2** Does the specific value of k impact results when fine-tuning existing models, or does k=-30 seem to always work well?

**Q3** If the models are trained until convergence does the performance gap between MrT5 and ByT5 increase?

**Q4** Why was softmax_1 used in the ByT5 baseline? Wouldn’t the unmodified ByT5 perform better?

**Q5** The paragraph beginning at Line 180 suggests that a sequence’s compression rate depends on the least compressed sequence in the batch. Are the rates reported in the paper based on this batch-dependent compression rate or the rate you’d get with a batch size of 1?

**Q6** Similarly, are the performance metrics reported using this explanation of batch processing, or are they reported as if sequences were fully compressed?

**Q7** In Figure 3, why is the English-only trained model doing better than English on many of the other languages?
soundness: 2
strengths: **Originality**
Most previous work in this area focus on algorithms for learning “what to keep” when performing sequence compression. This paper provides an interesting alternative by framing the problem as learning “what can be removed”. The proposed deletion mechanism can be added to existing architectures, making their solution easy to integrate into existing models via fine-tuning.

**Quality**
The authors conducted several levels of experimentation including simple tasks for gaining an intuition about the mechanism of their approach, a more difficult span completion task to compare against baselines, and down-stream tasks to assess cross-lingual semantic understanding and sensitivity to character-level manipulations.

**Clarity**
The paper is well-organized and clearly written. The ideas, methodology, and findings were easy to follow and understand.

**Significance**
The paper focuses on improving tokenization-free language models, which is a very relevant topic of interest to the community. Tokenization leads to several issues in LMs, including those listed by the authors as motivation: sensitivity to character-level noise (spelling errors), number representation for mathematical reasoning, and inconsistent compression rates across languages.
summary: This paper introduces MrT5, a ByT5 variant incorporating a learned delete gate for dynamically reducing the byte sequence length during the encoding process. By removing tokens from the sequence, the model is encouraged to implicitly merge the information from the deleted tokens into those which remain. The authors find that this reduction in sequence length leads to significant gains in runtime efficiency at inference time, with minimal differences to baseline performance. They show the deletion mechanism can reduce sequence length in zero-shot scenarios with new languages and see further improvements when training with multi-lingual data.
weaknesses: **1** The results are sensitive to the $/alpha$ hyperparameter. The gate regularization loss does not directly relate $\alpha$ to the desired compression rate, leading the authors to manually tune $\alpha$ or use the proposed P-controller to optimize for a desired compression during training. Previous work such as those cited by the authors (Nawrot et al., 2023) use a binomial loss to directly incorporate the desired compression rate into the regularization term. The same approach could be used here to remove the additional complexity.

**2** The location of the gate in a specific network also needs to be tuned. The authors show that for implicit merging of contextualized tokens the gate needs to be placed later in the network, reducing the cost-savings of the approach.
The results of the synthetic experiments were not convincing. The deletion gate seemed to only target the correct characters when sequence length was significantly reduced, resulting in a steep drop in sequence-level accuracy.

**3** There were no comparisons with other token-merging models in any of the experiments. It would have been nice to see a head-to-head comparison of this approach with the unsupervised boundary predictor from the cited (Nawrot et al., 2023) paper or the faster Toucan model from (Fleshman and Van Durme, 2023). Stronger baselines are needed throughout.

 **4** The models aren’t trained to convergence (footnote 4), and the potential 15% improvement in the baseline through continued training might not translate to MrT5, increasing the performance gap.

Meta-review: This paper introduces MrT5, an extension of the ByT5 architecture that incorporates a "delete gate" mechanism to dynamically remove unnecessary tokens from the input sequence. The gate learns to selectively discard tokens during the encoding process, leading to significant reductions in sequence length and inference time. The authors study the effectiveness of MrT5 on various tasks, including XNLI and character-level tasks, while maintaining competitive performance with ByT5.

The reviewers have praised the novelty and the efficiency of the approach and the clarity of the paper. The reviewers raised issues including limited baselines to which the authors have provided extensive comparisons. They also have questioned scope of evaluation, and limited/non-uniform improvement for non-English languages. The authors again have provided acceptable responses. While the reviewers have not raised their scores, I believe the rebuttal sufficiently addresses most of the concerns raised in the reviews and thus, I recommend acceptance. I encourage the authors to include all the new results in their revised draft.

Decision: Accept (Poster)