MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
Authors: Orevaoghene Ahia
Sachin Kumar
Hila Gonen
Valentin Hofmann
Tomasz Limisiewicz
Yulia Tsvetkov
Noah A. Smith
Venue: NeurIPS 2024 poster (2024)
Keywords: tokenization
multilingual LMs
over-segmentation
fariness
Primary area: natural_language_processing
TL;DR: We develop gradient based tokenizers that promote uniform segmentation granularity across languages in multilingual LMs.

Abstract: In multilingual settings, non-Latin scripts and low-resource languages are usually disadvantaged in terms of language models’ utility, efficiency, and cost. Specifically, previous studies have reported multiple modeling biases that the current tokenization algorithms introduce to non-Latin script languages, the main one being over-segmentation. In this work, we propose MAGNET— multilingual adaptive gradient-based tokenization—to reduce over-segmentation via adaptive gradient-based subword tokenization. MAGNET learns to predict segment boundaries between byte tokens in a sequence via sub-modules within the model, which act as internal boundary predictors (tokenizers). Previous gradient-based tokenization methods aimed for uniform compression across sequences by integrating a single boundary predictor during training and optimizing it end-to-end through stochastic reparameterization alongside the next token prediction objective. However, this approach still results in over-segmentation for non-Latin script languages in multilingual settings. In contrast, MAGNET offers a customizable architecture where byte-level sequences are routed through language-script-specific predictors, each optimized for its respective language script. This modularity enforces equitable segmentation granularity across different language scripts compared to previous methods. Through extensive experiments, we demonstrate that in addition to reducing segmentation disparities, MAGNET also enables faster language modeling and improves downstream utility.

MAGNET Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
Orevaoghene Ahia'
Sachin Kumar Hila Gonen'
Valentin Hofmann?
Tomasz Limisiewicz* Yulia Tsvetkov!
Noah A. Smith
'University of Washington 2 Allen Institute for AI 'The Ohio State University
*Charles University
bahiaãca washington.eduu
Abstract
In multilingual settings, non-Latin scripts and low-resource languages are usually disadvantaged in terms of language models' utility, efficiency, and cost. Specifi- cally, previous studies have reported multiple modeling biases that the current to- kenization algorithms introduce to non-Latin script languages, the main one being werr-eegmentation. In this work, we propose gradient- based tokenization-to reduce over-segmentation via adaptive gradient- based subword tokenization. MAGNET learns predict segment boundaries between byte tokens in sequence via sub-modules within the model, which act as internal boundary predictors (tokenizers). Previous gradient-based tokenization methods aimed for uniform compression across sequences by integrating a single boundary predictor during training and optimizing end-to-end through stochastic reparame- terization alongside the next token prediction objective. However, this approach still results in wer-segmennation for non-Latin script languages in multilingual settings. In contrast, MAGNET offers customizable architecture where byte-level sequences are routed through
SAANET-mutilingues adaptive
predictors, each optimized for its respec- tive language script. This modularity enforces equitable segmentation granularity across different language scripts compared to previous methods. Through extensive experiments, we demonstrate that in addition reducing segmentation disparities, MAGNET also enables faster language modelling and improves downstream utility.
1 Introduction
Despite the proliferation of generative language models (LMs) English, their non-English counterparts are far from being widely adopted. While multilingual LMs offer several advantages such resource efficiency and cross-lingual generalization, the performance disparities across languages remain a significant challenge. Previous work has largely attributed these disparities to training data imbalances across languages [43 34 29 24). Recent work, however, highlights that tokenization- the way input text is segmented
considerably degrade not only model performance but also training and inference costs on account of overly fragmenting certain languages and scripts B 33 Subword segmentation
algorithms used to build LM tokenizers (28 39
typically segment the training corpus relying on frequency statistics alone. Due to data imbalances, they obtain high compression in high-resource wra-ffragmented. This issue disproportionately affects
22 40]
languages, while majority of languages non-Latin scripts covering languages spoken by billions of people, which are not only less frequent in such corpora, but require up 4x bytes represent the same information.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Prior Work
MAGNET
Transformer Block
Transformer Block
ENGLISH Assesmally basw cancelled their rumsining roarett on their foor
MAGNET Coree E Larda
BPE
DTP
Transformer Block
Thansformar Block
RUSSLAN: Tpynna
Acrusmiih ORLuIIATa pertocuecci KOnIGGem d COUCH HOpI
MAGNET fire Cromn
BPE
Ird B
Snd
d Sand ferta
DTP
Latin predictor
Oyr predictor
Indic predictor
Boundary predictor
TELEGU:
2
bines
Schd
MAGNET
Transformer Block
Transforer Block
BPB
Mono Sequinos
Laiin scquence
Cyrl Sequencc
Indic SOCuChoc
DTP
Figure I: MAGNET routes byte-level sequences via language-scripl specific boundary predictors. These predictors infer boundaries leading equitable segmentation across languages. Prior work infers boundaries with a single predictor across languages and leads to wer-seementttiinn
To address these challenges, prior work has instead proposed to build tokenizer-free models by directly training on character or byte sequences [45] 4]. Operating on smaller or finer-grained segments leads to significantly higher compute- and memory-intense modeling, caused by much longer sequences. To alleviate this issue, recent work introduced a tokenization "layer" within the model architecture by pooling fixed or dynamic length contiguous character representations into smaller sets of patch representations 30 9 41 [1 [5], resulting in models optimized with a sequence "compression rate" mind. While these can improve efficiency, they are mostly suited for character-level modeling for scripts whose characters can be mapped to single Unicode codepoint. Due to the extensive number of codepoints Unicode, character-based vocabularies can be extremely large for multilingual models. issues similar to those experienced with subword models may arise if we only included codepoints in the training data (28) When extending these methods to training byte-level multilingual models, observe that the disparities in fragmentation rates across languages persist. For instance, the English text "Fellow wrestlers also paid tribute to Luna." and its Telugu equivalent, "5"s Sog duggrreew S-ar e ho &. contain 43 and 148 UTF-8 bytes, respectively. A fixed compression ratio for both languages will result in the Telugu text getting over fragmented with requiring close to 3x more tokens than English.
Since many of those characters may never appear during training. "out-off-vcabulla"
In this work, we propose MAGNET (multilingual adaptive gradient-based tokenization) to reduce this disparity in tokenizer-free multilingual LMs. Our goal is to obtain end-to-end multilingual language modeling with gradient-basse subword tokenization that results in high and similar sequence compression across languages with varying scripts. We leverage hourglass transformers [30 31 to efficiently route byte-level sequences through language-script specific internal boundary predictors trained infer word boundaries between byte tokens in sequence. These boundary predictors are trained end-to-end through The inferred boundaries are then used to pool representations of contiguous tokens in the same segment, after which the pooled (narrow) representation is passed into the rest of the transformer block. Unlike previous gradient- based tokenization approaches that apply the same compression rate to all languages in the pretraining data by incorporating a single boundary predictor within their model architectures, MAGNET employs modularity. It incorporates multiple language-scring specific predictors to achieve equitable segmentation granularity across different language scripts. We test the effectiveness of MAGNET on equitable fragmentation, model efficiency, and downstream task performance across nine typologically different languages, comparing to byte-level models without compression and existing gradient- based tokenization models [3I. Our extensive experiments demonstrate that our approach MAGNET results in more equitable tokenization when compared to hebworddtokknizerr. byte-level tokenizers and previous gradient- based tokenization models. This in turn leads to faster modelling and competitive performance across downstream-tase
stochastic reparameterisation 20 27
Another disparity with character- tokenizers is that Chinese-Japanese Korean scripts high number Unicode codepoints. "Code and data are publicly
available
at https: com/ ceoo//rrevaaaaa//
nagnet-tokenizati.o

---

2 MAGNET: Multilingual Adaptive GradieNt-basEd Tokenization
Our goal is to build multilingual byte-level language model with equitable segmentation across languages. We propose to achieve this by dedicating a separate module within the model for each writing script, to serve as an internal tokenizer for languages that use that script. Our proposed model, called MAGNET, builds on hourglass transformers [30 31], an architecture that was introduced to efficiently handle long sequences in tokenizer-Fi free models. We make several simple but important modifications to this architecture in order to obtain equitable segmentation across languages, while maintaining a high quality of multilingual modeling. In what follows, we explain the main concepts of hourglass transformers, and then introduce the modifications we make to accommodate equitable multilingual modeling.
2.1 Background: Hourglass Transformers
The hourglass transformer [31 30 is hierarchical architecture for efficiently handling long sequences. The architecture has three main components, each consisting of one or more transformer layers: A tokenization submodule which takes as input byte sequence and outputs a segmentation, a language modeling submodule that takes as input the predicted segments or tokens and is then trained to perform next token prediction, and an upsampling module that takes as input the hidden representations of the segmentations and converts them back byte sequence on which a typical language modeling loss can be applied. Considering this model blackbox, it still performs byte-level language modeling, however, requires significantly less compute thanks to the tokenization submodule.
Gradient-base Tokenization This submodule performs two steps. First, the given input sequence .i,.... NN (where each xy is byte our case) is encoded using a small transformer network (with causal attention) to produce sequence of hidden vectors hf ...h, hh- Next, boundary predictor takes as input each he and predicts a scalar value between 0 and 1, indicating the probability position t to be the end of segment. It is implemented as
b. p(b. 1) (MLP,
where MLP indicates a multi-layer perceptron and is the sigmoid function. To convert the soft probabilities to hard segment predictions, Bernoulli distribution is sampled from, defined by b,. Since the sampling operation will make the process non-diffcrentiable, hard Gumbel -igmood is used, a stochastic reparameteriization of the Bernoulli distribution, following Nawrot et al. BI:
bru b. = sigmoid log (1 b,)(1 u)
u ~ Uniform (0,1)
where is a hhpee--paaee puper-parameter. Since this module is differentiable, the segmentations are learned during training of the full model This module is referred to as "gradient-base" based tokenization."
Language Modeling Given sequence of segment boundaries b, E {0, 1} from the boundary predictor, this submodule first pools the hidden states belonging to the same segment by averaging them form a sequence of representations hf..... ......... h Let f......t tt indicate the positions at which a boundary is sampled, i.e., for any contiguous par ty. fj+1. the sequence #1,+1...
forms a "token" ending at position tj+1 The input representation of this "token" is defined as 111+ hP, 1, -t= hT, These representations are then passed through the middle block of transformer layers (with causal attention) to obtain another sequence of hidden representations hM ..... A.M. From the perspective of subword-tokent tokennizektznn based language model, this module is equivalent to the transformer blocks without the input and output embedding layers.
Upsampling This module converts h.M probabilities over byte vocabulary. This involves, first, upsampling the output of the middle block to the original resolution by duplication followed by skip
Nawrot et al. 31 also explore learning the segmentations using supervision from predefined word or subword boundaries. However, it is not viable solution for all languages and does not resolve the unfaimess issues. P, M, and T denote representations in middle transformer black, after pooling and at the token level. 'The first token is defined 20... St, and the last token Ity +1 2N.
3

---

connection: N. = hM
+ 1 These vectors are further passed through a small transformer
-12
network followed by an unembedding layer and a softmax to get a probability distribution over which language modeling loss (cross entropy) can be computed. To prevent the boundary predictor from collapsing and trivially predicting each position t as a boundary, Nawrot et al. [31] propose adding regularizer to the LM objective: - log Binomial( I, k)) where,
(N 1: (
(B; =
8* 31- and k Σ by.
(3)
Here 8 E [0, 1] is a hyperparameter, is defines the number of predicted segments or tokens. Intuitively. this loss nudges the model find a k close to B1 which is the mode of the Binomial distribution, In other words, / allows to control the compression rate of the input sequence to approximately x. Setting it as I will lead to every position being predicted as boundary whereas setting it to 0 will cause no boundaries to be predicted.
2.2 Adaptive Gradient-Ban Based Tokenization via Multiple Boundary Predictors
To encode the same information, different languages require different number of bytes, owing to their well as restrictions imposed by Unicode mappings. where non-Latin languages (e.g.. Indian languages) may require up to 4 bytes per character f In multilingual models, setting the same compression rate (via ()) for all languages will lead to text in some languages getting segmented into much longer sequences
different inherent efficiencies 3 25 33
[ see Equation (3). This disparity contributes to higher compute and memory costs for such languages as well as poorer model performance for downstream tasks [3] This issue parallels subword tokenizers where languages with higher segmentation rates get disadvantaged due to longer context-lengle requirements to perform the same task and poorer in in-context learning performance since the same context length fits fewer training instances than other languages leading to unfairness [5]
33
To make tokenization more equitable, we propose MAGNET, which efficiently learns to segment sequences across languages and language scripts with similar granularity. As part of creating equitable segmentation, we aim to efficiently maximize sequence compression, without having a negative impact on downstream performance across languages.
Introducing multiple gradient-based tokenizers To achieve this, we propose a modification to the model architecture that enables the processing of multiple language scripts. Each script has its own boundary predictor trained with distinct Binomial priors 3 determined based on the scripts' Unicode encoding and also tailored a desired compression rate. This allows us to achieve similar fragmentation rates across languages, due to variations in compression. The input sequence is tagged with its scrip 10 and we infer the segmentation by routing it through the appropriate boundary predictor. The remainder of the model architecture remains the same.
Determining /3 for equitable tokenizaton We the binomial priors for each boundary predictor to control the rates of the resulting segmentations for the different scripts. Since we want to impose equitable lengths across languages, we set the different 3 according to the following process. First, we choose an anchor language L for each script in our training corpus and define a quantity byle-to-word ratio R for this script as follows. Let X' (xi.....XD be sample of text sequences in language from our training corpus with x,| denoting the byte-length and countwerd (x;) the number of word IT in sequence x,, We define the average byte-to- to-rrd ratio R over X' as:
R D
E
|x,]
countwents (X:)
'The hidden vectors are shifted by one in order to perform next token prediction. /https: h/mp:::/ee/ wik i iedii i org/ hog//wkk /wwkkk iikki// /UTF-8
"Modeling characters directly may alleviate this issue; however, character-level multilingual models can explode vocabulary sizes.
As is the case generally in multilingual modeling, also without boundary predictors.
10 Determining the script given text sequence is trivial, we assume every sequence contains a single script.
ll For the purposes of this work, words are defined by whitespace boundaries.

---

We then set the prior 6s for the corresponding script S to be 1/R. Our final training objective over single instance is as follows:
(script((x) S) log Binomial k)
where is the indicator function and script(.) is function assigning writing script to sequence of bytes x, such assignment can be easily obtained based on codepoint definitions in Unicode.
3 Experimental Setup
3.1 Language Modeling
Pretraining Data We pretrain all models on nine languages (English, Spanish, French, Russian, Ukramian, Belarusian, Telugu, Bengali and Hindi) divided into three groups, written with distinct scripts: Latin, Cyrillic, and Indic (Brahmic) scripts. Our choice of selection is based on the linguistic diversity of these languages and the availability of data for downstream evaluation. Our pretraining data is obtained We present the statistics for each language in Appendix C
from the OSCAR dataset [22
Baselines We compare MAGNET against Dynamic Token Pooling [3]] which infers boundaries with fixed binomial prior 3 for every sequence, irrespective of the language script. This model is referred as DTP in the rest of the paper. DTP has a single boundary predictor; we train two versions of this baseline with the binomial prior /3 as 0.2 and 0.1 respectively yielding 5x and 10x compression respectively. We also compare against a byte-level decoder language model. To ensure fair model comparisons, this model has similar architecture DTP, but without any sequence compression.
MAGNET configurations We compute the byte-word-ratios choosing English, Russian and Telugu as anchor languages for each the language script, based on initial explorations. The FLO- RES [66 dataset is used for this purpose and the resulting ratios are approximately 5x, 10x, and 20x for English, Russian, and Telugu, re- spectively. Based on these ratios, we train five MAGNET models with different binomial prior combinations maintaining the ratio but adjusting the multipliers. First, to optimize for word-lewe level boundary segmentation, we use the original byte- to-word ratio configuration, i.e., 5x compression for Latin, 10x compression for Cyrillic and 20 compression for Indic languages within the same model. The second configuration; (1, 2, 4) x is the average bytes-to-dee toce-ttoeharratter ratio for English, Russian and Telugu. Hence, using this configura- tion optimizes for fair myte-level-modelling with character-level granularity. The third configuration (3, 12)x x is based on hypothesis that it would lead to fair subword-segmentation boundaries. Fi- nally, since we apply very high compression on Indic languages, we empirically test two additional configurations (5, 10, 13)x and (5, 10, 15) x with a reduced compression rate for Indic languages.
Table Binomial prior choice for each MAGNET and DTP model configuration. These combinations of binomial priors determine the compression rate of sequences per language script. While DTP uses fixed priors for all languages, MAGNET is dynamic and script-specific.
Binomial Prier
Configuration
Latin
Cyrillic
Indic
DTP 3x
0.2
0.2
0.2
DTP 10x
MAGNET 21 4)x
0.5
0.25
MAGNET 6, 12)x
0.33
0.17
0.083
MAGNET (5, 10, 13)x
0.2
0.10
0.076
MAGNET (5, 10, 13)x
0.10
0.066
MAGNET (5, 10, 20)
0.2
0.10
0.05
Subword Tokenizer Training To compare the segmentation derived from MAGNET traditional subword tokenizers, create byte-level byte pair encoding (BPE) vocabularies containing 50K, 100K and 250K subword units on our pretraining data. We employ a-sampling to train the tokenizers, typically used to improve representation of low-resource languages 10 That is, we sample documents for each
language according to multinomial distribution with probabilities {9:).=....N. where: q: Σ Σ
with p; >و This increases tokens for low-resource languages and has been shown to reduce bias towards high reshures ones. We consider 0.5, 0.3 consistent with 10 12]
5

---

Downstream Datasets To demonstrate the effectiveness of MAGNET, we evaluate by finetuning our trained models on several question answering and classification tasks. Specifically, we evaluate on XQuAD (question answering) G6. XNLI II (natural language inference) [1] PAWS-X (paraphrase
detection) [46] from XTREME [9]. and SIB 200 (the topic classification) 2 We provide a detailed
language coverage across all tasks in Table 5] In addition, to test how adaptation capabilities of
MAGNET, we evaluate on dialectal tasks, specifically ILI [48] the Enyo-Aryan Language Identification (ILI) shared task and HaSCoSVa-2022 [7]. hatespeech detection Spanish dialects. We provide
finetuning details in Appendix
3.2 Analyzing segmentation across models
The objective of this analysis is to compare segmentation granularity across different approaches. That is, measure whether the same amount of information is conveyed through similar token counts across various languages. Following previous work [33 31, we conduct this analysis with the parallel corpus FLORES-200 [66] focusing the nine languages in our pretraining data.
For byte-level models, segmentation done by converting each sentence to raw UTF-8 bytes and computing the average number of bytes per sentence. With the subword tokenizer, each sentence is segmented using the tokenizer we trained in 3.1 and the average number of resulting tokens computed across all sentences. As for paadiettbbaed methods like DTP and our MAGNET, we feed each sentence into the model and retrieve a sequence of boundary predictions from the boundary predictor layers. The count of positive predictions determines the number of tokens per sentence.
4 Results
The goal of MAGNET to learn equitable segmentation during training while maintaining high quality downstream performance. Idcally, we expect that MAGNET results in higher compression for the non-Latin script languages, hence balancing segmentation granularity across all languages. This, in turn, should improve modeling efficiency by reducing computational costs training and inference time.
4.1 MAGNET results in equitable segmentation across language scripts.
We analyze the segmentation granularity, contrasting our method with byte-level, subword tokenization, and DTP described in $3.2 Our results in in/iiuree 2]show that MAGNET models produce similar segmentation rates for all languages. The improvement is particularly noticeable in non-Latin script languages that are most susceptible ver-segmentation with the basclines.
Tokenizer
MAGNETIS
80
MAGNETIS 10, 20
Adw..55.BPE..
Aeeogee narber tahers
BPS 30k
RPT
80
BPE.2K0
First, we compare byte-level segmentation to MAG- NET with the (1. 2, 4) segmentation configuration. As described $3.1 Indic languages have approxi- mately four byte code-points to character, Cyrillic languages have approximately two, and many Latin languages have a one-to-one mapping. Therefore, we expect that training MAGNET with the (1, 2,4))x con- figuration will result in equitable byte-level modeling across all of these languages. Appendix Figure Ta shows that MAGNET (1, 2, 4) results 3x drop in the average number of tokens for the Indic languages, and close 2x drop for Cyrillic, while the Latin lan- guages are not affected. Next, we compare segmen- tation between DTP 5x, MAGNET at (5, 10, 13)x, MAGNET at (5, 10, 20)x and without alpha sampling (see $3.T). We find that MAGNET models result in the most equitable segmen- tation across all languages. In fact, we measure drop close to 5x in the average number of tokens for the Indic languages compared to DTP and the subword tokenizers. Notably, we also see that subword tokenizer with a large vocabulary size is required to achieve lower segmentation rate on Cyrillic
40-
20
Figure 2: Average number of tokens after seg- menting the FLORES dataset. Subword tok- enizers and DTP result in over-segmentation in non-Latin script languages, while MAGNET closes the gap.
at vocabulary sizes 50k, 100k and 250k with and

---

languages, whereas for Indic languages, even with large vocabulary and a sampling, we observe a pronounced disparity. This contrasts with findings from previous work that a sampling alleviates tokenization disparities [0O Overall, these results suggest that MAGNET learns equitable segmentation across languages with diverse scripts, while fixed segmentation models like DTP and byte-based subword tokenizers are sub-optimal and very likely to result in over-segmentation.
4.2 MAGNET maintains performance on downstream tasks.
Our goal is to enforce equitable segmentation while maintaining model performance across tasks. In Table 2] we present results for the best- performing MAGNET model compared to DTP and byte-level models (we provide comparisons to all MAGNET models in $5.1) Overall, We find that MAGNET models perform better than DTP but are competitive with the byte-level models while being considerably faster and requiring less compute. MAGNET(3, 6, 12)x performs best on PAWS-X and SIB, while (1, 4)x performs best on XNLI and XQUAD.
en
ru
hi
bn
Avg
Byte
24.72
71:09
67:13
02.74
DTP 5x
72.91
70.19
70
66.73
62 89
86 87
66
68 16
Motel
DTP 10x
71:92
90
67.00
62.52
00.36
00:47
67.33
MAGNET(1124) x
24-44
70.63
67 47
64.3 01
MAGNETES T(3, 12)
74.42
70.83
71
71:35
67.85
61.80
66.33
84
68.35
MAGNET(S, 10, 20)
23.57
71.70
67.90
Figure 3: Language-speeific accuracy on the XNLI task across byte-level, and all DTP and MAGNET models. Results are mostly competitive between byte-level and MAGNET models on Latin languages and Russian.
We report segguagg-speciffie results the XNLI dataset in Figure 3] We see better results with where
the MAGNET models even in some the models in comparison optimize for a similar segmentation. For instance, on Spanish, MAG-
NET at (5, 10, 15)x outperforms DTP at 5x. On Indic languages, MAGNET models are generally competitive with the DTP models. We provide more analysis on the trade-off's between downstream performance and segmentation in $5.1 Results on dialectal tasks are reported in $2 We see competitive results across all models on all the tasks, suggesting that there is also no negative impact result of adapting the models to their respective dialects.
Table 2: The average performance (accuracy) on downstream tasks on all languages across different models. We present results for the rest-performing MAGNET model: (33,6, 6, 12) for PAWS-X and SIB), ((1, 2, 4) x for XQUAD and XNLI). Bold indicates the best overall performance Table provides more detailed
results.
Model
XNLI
PAWS-X
SIB
XQUAD
Hascova
ILI
Byte-level
68.68
82.18
71.05
44.62
87.38
89.24
DTP5x
68. 16
81.29
69.17
43.31
86.87
89.17
DTP10x
67.31
75.99
67.83
35.83
87.62
88.72
MAGNET
68.74
85.41
71.43
44.61
87.25
89.27
(a) In- lanaang tasks.
(b) Dialectal tasks.
4.3 MAGNET results in more efficient models.
Comparing inference times across all models, we expect that models which optimize for fixed compres- sion like DTP would be only efficient for Latin languages because of their lower byte-to-word ratio. Hence, we anticipate that our routing strategy with the MAGNET models would result in an efficiency gain for non-Latin script languages. In Figure 4| we plot the inference time per language XQUAD, relative to the inference time of the byte-level models. We show that MAGNET has a shorter inference time than the byte-level models, comparable to DTP for English and Russian and slightly lower for Hindi and Russian. If we assume the optimal compression rates for English and Russian to be 5x and 10x, respectively, using DTP model with a fixed compression rate for both languages requires training two separate monolingual models to obtain the ideal compression rate for each. However, training
7

---

single MAGNET (5, 10, 20) x model that dynamically achieves 5x compression rate for English and 10x compression rate for Russian results in a lower inference time for both.
Inference time (relative)
1.0
Bytc-loyt Food (xc
9.F%
comprmstoon (104)
MAGNET().24)
MAGNET(S.
0.5
MAGNET(S.
0.580
0:34
0.20
5:384
0.99
0.30 0.29k
0.32c
0:38+ 0.34A
0.422
0.0
hi
Figure 4: Inference time per language in XQUAD, relative to the byte-level model. MAGNET's inference time is shorter than the byte-level model and comparable to DTP for most of the languages.
5 Analysis and Discussion
5.1 Trade-off between downstream performance and equitable segmentation
Previous studies have reported correlations be- tween compression and model performance [14].
We empirically investigate these tradeoff's by comparing downstream performance across dif- ferent MAGNET configurations defined in Ta- ble I In the results reported in Table 3] we find that the configurations that pertorm best are [1, 2, 4) and MAGNET(3, 6, 12) x. These configurations are equivalent to fair byte- level and subword modelling across all languages. We also report the average task performance per script at various compression rates in Figure 5] Here we are not looking to compare performance across language scripts, but rather assess perfor- mance across different compression rates within each language script. Our results show that there little to no drop in performance for Latin and Cyrillic languages as compression increases moderately. However, for Indic languages, an average of 5% drop in performance as compression increases.
Latin Indic
Average task performance
80
60
MAGNET
40
11
Compression rate
Figure 5: Average task performance vs compression trade off across language scripts.
Table 3: Results (accuracy) from ablations across all MAGNET configurations.
Model
XNLII
SIB PAWSX
XQUAD
Hascova
11.1
MAGNET(1, 2, x
68.74
83.30
71.02
44.61
86.75
88.62
MAGNET (3, 6, x2)x
68.35
85.41
71.43
43.72
87.13
88.57
MAGNET (5, 10, 20)
67.50
80.44
71.13
41.06
87.25
89.27
MAGNET (5, 10, 13)
66.28
79.85
70.64
41.92
87.88
88.35
MAGNET (5, 10, 15)
67.96
84.79
68.79
42.81
88.37
88.67
(u) In-language tasks
(b) Dialectal tasks
5.2
What is the granularity of segmentation across different compression rates?
In 1 we highlight that the binomial prior is essential for determining the granularity the seg- mentation derived from the boundary predictor. To intrinsically validate that MAGNET indeed learns

---

segmentations of similar lengths across different languages, we manually analyze examples from the SIB corpus, comparing them with DTP at 5x and 10x. As shown in Table 4JDTP at 5 produces word-level segmentation for all Latin languages while producing subword-level segmentation for Cyrillic and Indic languages. At 10x, we see word-level segmentation for Cyrllic languages, phrase-level segmentation for Latin languages and a mix of subword and word-level segmentation on Indic languages. To achieve word- level segmentation for all languages, DTP requires training three separate models. However MAGNET alleviates this requirement by producing similar segmentation granularity across all the languages. For comparison, the segmentation granularity of the BPE tokenizer is highly sub-optimal for Indic languages as shown in Appendix (Table 4| While the BPE tokenizer produces word-level segmentations for Latin and Cyrillic languages, it produces character-level segmentation for Indic languages. MAGNET, on the other hand, finds a good balance of segmentation granularity across languages.
Table 4: Segmentation of English, Spanish, Hindi, Russian, Ukranian and Telugu examples from the SIB corpus. While other models produce different segmentation granularity across languages, MAGNET consistently produces similar segmentation granularity across languages.
Lang
Text
DIF
DTP J0x
MAGNETIS 16, 201
BPE 100k
This will allow pla--
This will allow players
This allow players
This allow players
wwll paaerr
crs. contrel acioms
to can Inel aationlls (ladd
control actions and mune-
to con I astions and
contted llactions and move-
and aovemneees in video games by moving the de- vies thruggh the sir. ja-
I movemen
I in vileo pimes by ll moving I the II desice Il thnugh the Lair.
mems ia valco I games by moving the device I through I the
movemen is in vileo gimes by moving I the
meas in videoll gamell byll mosingl the device I through the [eeilsir I.
device thouugh I the Fair
Esio perminit godores controlar ac- ciones y moximies- los en los videojeggss. trres del movimiento del disposthvo por el
lla la laxil acciun flex y Immesimien- Ins en los |I videnjuegmc
Esto penmitiaa T II godores contolor ciunes y los movimientos en lost sideojnggn I. travss del movimienio del disgosi tio il por el aire.
Es Ti
Esel parmainil Kal ja- pidored conmmlari lasl ac- cinned
jupddurss
Jugadores
trdar las
acciones ily movinimn los I en los videojaggos, lla trovss de II movimien to de depositivo il por i II
movimientedl onl lost videojsegus all
il travss del I movimiento I del dispositvoo I poe I dlll
travill dell moximientol dell dipositivnl porti ell aiee..
10
Goconrs og shoos do anyor badd 1 Mlayss daaddg diddia sced oiu daras warres Koyagua
21d1 1s 16Plg"
adtsoraTa aal sing salis- as le foon is fasstt all "kguff du aaras aasiflllscceee diu I'd Todd Test he wore ladl
20sens l e dsl i borrbaa T5e I' aaddd Teu and ata scedd ud la addced If aaddaadd it ue mbcre (eot be la do feal "auc?
isci i lss" lo ddd let isst
Ieet! olladi doliid 14
1t 18mll of af (id)
1sT lahtaal lessi ixil Thesf I atd't lofl kas Toll
soll of I doud ofl usll or lolle asl (e
d gl 1ss'1
sorr do iall iddel iied bb list la 10%
*add Tagload lodt
lot willl
aall delallde fat Io Tiat
astilledd 1e'l lusibllsf
lil isfi allooll list
hn
de Balled al par ge wet Fannmiinn wht vener site garee ligtt sec dit segifit êen.
TT
at
a
8 Sened ai || of 8 an uy Tangee a of 84 "
||G
Jiet lIte
of
| [
d' ogr ste
d' og wet Raanfa |
||et
ast
ds
Ra
a|
|| | a ct as ||
as oo quue ofe n camn
||
com
cem do
de cjer wer |||] dn
|
||
aft
11
oz
|| oig chr w| ( || of
caff t | dm.
||e
||
sif
se gaftt e d...
||d
|
I|ch
[1
ser
lle
live
DiospeacrBO 201e
o cpe Jars
|o
Blocpea || crso || x and-
apeacrson: || amie
pex CTRON 2010-
Mosiii yerpoteera
[] x
yerpo
|| na II yerpoit II arpo- xM || avorye Il yopssums
xe moss || yerpodterna zeuuss yerpoderna |||||
a noagyxe итроки
Rem a || so aay-
crna a soagyse
noagyxe || xx rpoum
noapyxe || arrposa cMo-
emoryt yopasners,
se || m rpo sa
cuurys || yrgonnem.
ryr II yopauterss II ven-
|I ryr yopa || jun
mo.
||
Il nelieron I| soon || is you-
geltem ||xr sun || jupe..
ermor |] um || II monn II
Javeeeeeeea peourpax
bects
mon
zeen II
see mommu || mnsgeou
emueAA |I all muse
pa
rpax
pox II.
uat aeo a rpa I|
He miteneertt rpan- UpM sоитроss vus
Hopop |
cok Rl] aa || oue Mores a up you no
1f || antcomeusn. || span- Ue ankeamours || rpan- Ue | sabeanee Jm
uus || kowepo || m || man wwx moorpponn 1su sis sie- Ajaan || ra ll pyxsver |I
rpus |loM sourpumn
niwmn pyxaum
squn I/mwa up [ooyyoi.-
un pysasee I sipool-
wax jioer ma || pyx
Laaeiipax, pysaavus
pux y yosaii.
sinesirpa (l. руyаноиs rpas,
pyssso tu apr--
[aso lly vigeu |ir Ipax
upucepiit y nosippi.
Il npverptic y nosixpi.
expin || musiipp.
pyx [ааон upu-
crpla
[[>
[] nouirpi |.
5.3 Does segmentation significantly change after finetuning?
We investigate the effects of fine-tuning on the resulting segmentation boundaries across various down- stream tasks. Essentially, we inspect how the boundary prediction changes after fine-tuning for each downstream task. We find that there are no differences in segmentation before and after fine-tuning despite updating the parameters of the boundary predictors. While there are a few instances where the segmentation of the fine-tuned model is different than that of the pretrained model, there is no clear evidence of the segmentation changing drastically after fine-tuning. In Table 7 in the Appendix, we present two examples from the SIB dataset where there is a slight change in segmentation. We found no indication that any observed changes contribute to or hurt task performance.
9

---

Related Work
Overcoming segmentation disparities in subword tokenization In multilingual settings, subword tokenizers have proven to be prone to vee--eegmnttaion, due to the data-driven nature of the BPE algorithm 39.. Previous work [3 12 44 has attempted to address data imbalance issues in subword tokenizers by over-sampling low-resouree languages. Our work shows that this only alleviates the bias on certain scripts and doesn't solve the problem. Other studies 36 17] have also shown that tokenization in transformers remains biased in favor of high-resource languages. Wang et al. 42 enforce models to use smaller subword units high-resource languages to make segmentation fairer. Some works 23 & suggest training multilingual tokenizers language clusters mitigate segmentation disparities, however, this leads to expanded vocabularies. Despite these attempts, evident that the training objectives of subword tokenizers do not effectively align with those of language modeling.
Tokenizer-free language models Language modelling over bytes 45 and pixels 37
38
26
has become desirable, as it removes complicated preprocessing pipelines in modelling. Xue al. introduced ByT5, a tokenizer free variant T5 [35] that processes text at the byte level. However byte-level encoding wer-fraaments non-Latin script languages resulting in overly long sequences. Since byte or character sequences usually result in longer sequences, previous work [30 41 47
45
(1, 15 on
tokenizer-fiee free LMs has introduced novel model architectures to mitigate the computationa overhead of processing raw character or byte text directly. These methods [9] 41 47 30] end up segmenting raw sequences into fixed/yynamic-size patches, which is not suitable for modelling non-Latin scripts.
7 Conclusion
In this work, we introduce MAGNET, gradient- based tokenization method to learn equitable seg- mentation across languages scripts in byte-level multilingual models. MAGNET dynamically routes byte-level sequences through language-scripl script-ppeccficc internal boundary predictors trained to infer word boundaries through stochastic reparameterisation. We show that MAGNET enables us to learn token representations with the same granularity across languages compared to vanilla byte-level models and previous gradient- baseed tokenization approaches. Our analysis demonstrates that while there are indeed downstream performance trade-off's as result of MAGNET inducing high compression script languages, we are still able to maintain downstream performance quality. Overall, our results hold promise for future research equitable segmentation and text processing more generally.
non-
Acknowledgments
We would like to thank the UW NLP community for valuable discussions of this work. We are grateful to Farhan Samir and Alisa Liu for discussions experiments and analysis. This work supported in part by NSF ISS 21 13530. We also gratefully acknowledge support from the National Science Foundation under CAREER Grant No. IIS2 142319,,9,, NSF grants No. I1S2125201 and ISS2003977 and gift funding from Google, MSR, and OpenAl.
10

---

References
[1] J. Acs. Exploring bert's vocabulary. Blog Post, 2019.
[2] D. Adelani, H. Liu, X. Shen, N. Vassilyev, Alabi, Y. Mao, Gao, and E.-S. Lee. SIB- 200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Y. Graham and M. Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226-245, St. Julian's, Malta, Mar. 2024. Association for Computational Linguistics. URL org/ 2024 weaellllllong. 14
ptttps:/accantthoe
[3] O. Ahia, S. Kumar, H. Gonen, J. Kasai, D. Mortensen, N. Smith, and Y. Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904-9923, Singapore, Dec. 2023. Association for Computational Linguistics. doi: (0.18653//1/2023.emme www.eeemmnnmmmmmiiieeem.e erq/2023.0 emn emmlp-mainn... 614,
URL
[4] R. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones. Character-level language modeling with deeper atte-attention. In AAAI Conference on Artificial Intelligence, 2018. URL, https
/api. www.sttttttcchheoeeoeee
otg/Corpus 52004855.
[5] C. Amett, P. D. Riviere, T. A. Chang, and S. Trott. Different tokenization schemes lead to comparable performance in spanish number agreement. ArXiv, abs/2403. 2024. URL https: semant icscholar org/ I
[6] M. Artetxe, S. Ruder, and D. Yogatama. On the cross lross-lingual transferability of monolingual representations. CoRR, abs/1910. 1856, 2019.
[7] G. Castillo-lópez, A. Riabi, and D. Seddah. Analyzing zero-shot transfer scenarios across Span- ish variants for hate speech detection. In Y. Scherrer, T. Jauhiainen, N. Ljubešic, P. Nakov, J. Tiedemann, and M. Zampieri, editors, Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 1-13, Dubrovnik, Croatia, May 2023. Asso- ciation for Computational Linguistics. doi: 10. 118653//120223.warriaaaa
URL hhttts
7 Maclanthology.coo org/ 20..
vardial-1.1
[8] H. W. Chung, D. Garrette, K. Tan, and Riesa. Improving multilingual models with language- clustered vocabularies. In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
4536-4546, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/
v1/2020. (tt.0000eemllmmmmi..... 36..
ogy org/ borg/202 2020. emn emmnp-ma
[9] J. H. Clark, D. Garrette, I. Ture, and J. Wieting. Canine: Pre-training efficient tokenization-fiee free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73-91, 2022. doi: 10. 00448. URL hitps: http:///aaaaahoooogoooo org//o/// 2022 Eac1-1.5.
[10] A. CONNEAU and G. Lample. Cross-lingudt language model pretraining. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL wttp:ttps///tt. hhttps::/prrceeeddiii ngs www.eeiinsnnnnnnnnn............. wttpiipp...ppppp...ppppp fille// /iles//apper/2019///55 9/FF1 -aap://eccccaaat-ppe....tt.
[11] A. Conneau, R. Rinott, G. Lample, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov. Xnli: Evaluating cross-lingual sentence representations. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
[12] A. Conneau, K. Khandelwal, Goyal, V. Chaudhary, G. Wenzek, Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. Unsupervised cross-lingual representation learning at scale. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1440-8451. Online, July 2020.
11

---

Association for Computational Linguisties. doi: 10.18653///2222.cchhmaaia 747. URL https /acLanttooooyy.orrgg2ZZZZ0 mml.mma.mm.m11
[13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171--186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/NII1 1423. URL /ttp:::/ac.aa.booogogg..
1422
[14) W. Fleshman and B. V. Durme. Toucan: Token-aware character level language modeling. ArXiv, abs/2311 1.08620, 2023. URL htttp:://api.senan ccsho org/ Co Corrps D
265213263
[15] N. Godey, R. Castagné, É. de la Clergerie, and B. Sagot. MANTa: Efficient gradient-based tokenization for end-to-end robust language modeling. In Y. Goldberg. Z. Kozareva, and Y. Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2859-2870, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10. 00.186633/112022.fnnndl orq/2022, www..niinnnnngeee..e2e.m..
URL
[16]
N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan. The Flores- 101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538. 2022. doi: 10.1162/tac_a_0044 URL httpa////////aaaaaaaoo.ooo/////.
K0
[17] Hayase, Liu, Y. Choi, Oh, and N. A. Smith. Data inference: What do bpe tokenizers reveal about their training data?, 2024. URL
wwwp://wwrrgr/////.///o/o..bss 2407 16607
[18] D. Hendryeks and K. Gimpel. Gaussian error linear units (gelus), 2023.
[19] Hu, Ruder, Siddhant, G. Neubig, O. Firat, and M. Johnson. Xtreme: massively multilin- gual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.1 1080, 2020.
[20] Jang, S. Gu, and B. Poole. Categorical reparameterization guth gumbel-softmaax himer In International Conference on Learning Representations, 2017. http::tss//ppeeeeeeee..eett forum? 1d=rkE3y85ee
[21] D. P. Kingma and Ba. Adam: A method for stochastic optimizztion........... CoRR, abs/1412.66980, 2014.
semant semantiisccciteeee..e....cc ccscholar...... wowpoooooooo::
[22] T. Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Mclbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18655100 1111886111111810000 URL http:/:////aaaatholl......
1007
[23] D. Liang, H. Gonen, Mao, R. Hou, N. Goyal, M. Ghazvininejad, L.. Zettlemoyer, and Khabsa. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In
H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13 13122-13152, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10. 12111186611222233222 wwtp::2222..mmmmmmmmm...m.. URL https://
ac aclanthology wwt.:orgg///// 2aeg//023.ccm emn main. 813.
[24] T. Limisiewicz, D. Malkin, and G. Stanovsky. You can have your data and balance too: Towards balanced and efficient multilingual models. In L. Beinborn, K. Goswami, S. Muradoğlu, A. Sorokin, R. Kumar, A. Shcherbakov, E. M. Ponti, R. Cotterell, and E. Vylomova, editors, Proceedings of the 5th Workshop Reseurch in Computational Linguistic Typology and Multilingual NLP, pages 1-11, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 18653/ v1/2023. .... URL ht wttpp::pao................ http:::::///caaaoooooo.ooo wwtp://oooyyrr//222/2
http://0222ssiyy111.1. sigtyp-1.1
12

---

[25] T. Limisiewicz, Т. Blevins, H. Gonen, O. Ahia, and L. Zettlemoyer. Myte: Morphology-driven byte encoding for better and fairer multilingual language modeling. ArXiv, abs/2403. 10691, 2024. httl:s/tt::///t /api semantion icscholar otg/Corpus 26:
[26] J. Lotz, E. Salesky, P. Rust, and D. Elliott. Text rendering strategies for pixel language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10155-10172, Singapore, Dec. 2023. Association for Computational I Linguistics. doi: 10. 222.66///1/2222mmmmmmmmmammaaam
URL https://
acl aclannhollo ogy .rg. 2trg//0233 2023 emn p
628
[27] C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017. URL
https::/opente
net/ fontt/forum?iddi33 JE51,5]
[28] S. J. Mielke, Z. Alyafeai, E. Salesky, C. Raffel, M. Dey, M. Gallé, A. Raja, C. Si, W. Y. Lee, B. Sagot, and S. Tan. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. ArXiv, abs/2112.10500. 10508, 2021. URL https://api
semanticscholar htt.:aa......r/oooooooo0:2//222.
[29] B. Muller, A. Anastasopoulos, B. Sagot, and D. Seddah. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448 442-462.000 Online, June 2021. Association for Computational Linguistics. doi: 10. 10.186633/1/2000 maicl-main.38. URL http:: http:///aaaaaahhaaaooeaa logy. org.
[30] P. Nawrot, Tworkowski, M. Tyrolski, L. Kaiser, Y. Wu, C. Szegedy, and H. Michalewski. Hier- archical transformers
efficient language models. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, Findings of the Association for Computational Linguistics: NAACL 2022, pages 1579, Seattle, United States, July 2022. Association for Computational Linguistics. naacl. URL https: /.l.ttts///allattollloooo/ 2022
doi: 10. 10.186653/1/2022.fmmmmm
Findings naacl. 117,
[31] P. Nawrot, J. Chorowski, Lancucki, and E. M. Ponti. Efficient transformers with dynamic token pooling. In A. Rogers, J. Boyd- Graber,, and N. Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 6403-6417, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. aaclo/200....1111...3....333 URL https:ttp.... http:///aaaaannnll..o..oooooo.aa wrwpt/hoooooyy..o2222332333 http:/2222.ccc1llomm acl-long
3553.
[32] J. Ortiz Su'arez, B. Sagot, and L. Romary. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 16, Mannheim, 2019. Leibniz-Institut frur Deutsche Sprache. doi: 10. 12.1141881 1olo11///spppi 8//is--ub-99211111 URL http://naat rew.-rresoviiiggg de/uras noon de bsz : mh 90215
[33] A. Petrov, La Malfa, H. S. Torr, and Bibi. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems, 2023. URL
https::/arxins org/ abs 2305. 15425)
[34] J. Pfeiffer, I. Vulić, Gurevych, and S. Ruder. UNKs everywhere: Adapting multilingual language models to scripts. In M.-F. Moens, X. Huang. L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pages 101 10186-10203, Online and Punta Cana, Dominican Republic, Nov. 2021. As- sociation for Computational Linguistics. doi: 10. 10.18653/v1/2211. 1011665/w/202.eemm/nnnaa.as wtwp:///2.....mmaai......
URL
https: Sctps://aclanthologs. hwwpellloyoorrg//////e wrgg//22.1.. wwwlp-nnnla.... meallp-main.. 800
[35]
C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, Matena, Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Rex, 21:140:1-14067, 2019. URL https: //ppi. semant icscholan
orq/ Btcc/CozpusI::2443880
13

---

[36] P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume
I: Long Papers), pages 31 18-31335, Online, Aug. 2021. Association for Computational Linguis- tics. doi: 10.188533 2otp/////22...lloo..2522533 243. URL mittps:/aaclantholl.
acl Nan-long.223
[37] P. Rust, J. F. Lotz, E. Bugliarello, E. Salesky, M. de Lhoneux, and D. Elliott. Language modelling with pixels. In The Eleventh International Conference Learning Representations, 2023. URL
[38] E. S.lessyy N. Verma, P. Kochn, and M. Post. Multilingual pixel representations for translation and effective cross-lingual transfer. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13845-13861, Singapore, Singaaeerrr Dec. 2023. Association for Computational Linguistics. doi: 10. 10.18865//11// http:/22233emmmmmmm maall--mai..554.. URL https:: pattpp://aclaaee aaa/aaaaannthttooa wtw.://oooo002......maaaaaaaaaaa emn1p-manmpppa... 854
[39] R. Sennrich, Haddow, and A. Birch. Neural machine translation of rare words with subword units. In K. Erk and N. A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-111 1162. URL
ht https://aces //acl anthol coaethooog....... www.:/ogg/p11111111 1162
[40] X. Song, A. Salcianu, Y. Song. D. Dopson, and D. Zhou. Fast WordPiece tokenization. In M.-F. Moens, X. Huang. L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089-2103, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/ 11/202.emnll-maing
URL https:://///
/aaaaaaaaaaa hwtpa/caaaaattt....cco.. wwtg//22.1..0 emnlp-main
160.
[41] Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer: Fast character transformers via gradient- subword tokenization. In International Conference on Learning Representations, 2022. URL https net/ /oppanrerreeer.nt///ooooooooooop
[42] X. Wang, S. Ruder, and G. Neubig. Multi-view subword regularization. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473-482, Online, June 2021. Association for Computational Linguistics. doi: 00.18633/v1/2022 naacl-main.44 URL /tt.ttpp::/aclaaatoooooo http://acaaanthoooooooo org/
/tt//02//2222 202 naacl waaaaaaaaalaa main....................ammmm..m 40
[43] S. Wu and M. Dredze. Are all languages created equal in multilingual BERT? In Gella,
J. Welbl, M. Rei, F. Petroni, P. Lewis, E. Strubell, M. Seo, and H. Hajishirzi. editors, Proceedings of the 5th Workshop on Representatton Learning for NLP, pages 120-130, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v (BT/S3//11220000 1/200000111//220...4411p- 1.16. URL https
7 /Aclaattho-ogyy.rrr/220Z00 http://ooo.g...0....emmoomo11...
[44] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mT5: massively multilingual pre-trained text-to-text transformer. K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tun I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational I Linguistics. doi: 10..1853//1/222 1/203111/221111 http//221..aaal--ae main.44 URL http:///caaaaahhloooo ogy. org. 2021 naac ma imman..4. 41
[45] L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. ByT5: Towards token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022, doi: 16. a 00461. URL. attps://aclanthoon
14

---

[46] Y. Yang, Y. Zhang, C. Tar, and J. Baldridge. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proc. of EMNLP. 2019.
[47] L. YU, D. Simig, C. Flaherty, A. Aghajanyan, L. Zettlemoyer, and M. Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. http::ss///peeeee.eww net. forum?
id= id=JImoZV9XXp
[48] M. Zampieri, P. Nakov, N. Ljubešié, J. Tiedemann, S. Malmasi, and A. Ali, editors. Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), Santa Fe, New Mexico, USA, Aug. 2018. Association for Computational Linguisties. URL
https: ptttss://acllanholoogy
8 -
15

---

Appendix
A
Limitations
The primary limitation of our work the restricted resources available for the extensive experiments we carried out. This constrained the number of languages included in our pretraining data, the size of the pretraining data itself, and the size of the models. Nonetheless, we hypothesize that our current results will hold true when replicated with larger models, MAGNET is likely to provide even greater benefits when integrated into large models. We leave this future work explore further. Another limitation is the aefformance-comprestti
trade-offs associated with MAGNET, as some languages are sensitive to high compression rates. However we note that this not universal across all tasks. In fact, we argue that MAGNET offers users the flexibility optimize for their desired benefits. Finally. MAGNET also inherits some limitations from previous gradient- based tokenization methods 31
and the vast majority of segmentation methods. This approach may not be suitable for Semitic languages, where morphemes are discontinuous and vowels are interspersed between consonant roots for inflection or sometimes omitted. Computing byte-to-word ratios would be harder in CJK (Chinese, Japanese, and Korean) languages, as they do not mark word boundaries with space. However we note that our choice of languages was not influenced by space separation, but rather by linguistic diversity and the availability of data for downstream evaluation. MAGNET is very flexible and applicable to all languages that can simple proxy to train our boundary predictors
30 41
be expressed with UTF-8, use fite-word-res to learn equitable tokenization. We note that word-word-ratio is not compulsory proxy and for such languages other proxies can be used
B Broader Impacts Statement
In this work, we contribute to promoting equitable segmentation in multilingual language models across various language scripts. Our approach holds promise for enhancing the utility and efficiency of multilingual language models, particularly benefiting low-resourced and non- non-Latin script languages spoken by billions worldwide. We acknowledge limitations of our work in AAppeddix A] and strongly advise against unintended usage of the models. We will release our code and models facilitate further research in this direction.
C Dataset Statistics
C.1 Pretraining data
Our pre-training data is obtained from the OSCAR dataset [32]. Due to resource constraints, we only pretrain our models on subset of this dataset. The distribution of tokens across languages in displayed in Figure 6.
C.2 Downstream data
Table 5: Downstream language and task coverage
Dataset
Task
I Languages
XNLI XQUAD
Natural language inference
en, fr, es, ru, hi, bn te en, es, ru, hi en, fr, en, es, ru, uk, be, bn, te, hi
Question answering
PAWS-X
Adversarial paraphrase identification
SIB-200
Topic classification
D Technical Details
Data Preprocessing For all our datasets, we preprocess all text to raw UTF-8 bytes. In the MAGNET models, we add unique script identifier to the front of every sequence that guides the models to route the sequence to the respective script boundary predictor.
16

---

2.50E+8
2.45E+8
Size (Bytes)
2.40E+8
2.35E+8
2.30E+8
en
fr
es
be
uk
bn
te
hi
Language
Figure 6: Language statisties in the pretraining data.
D.1 Model Hyperparameters
For all our experiments. we use 14-layer hourglass transformers with layers in the first block, 10 layers in the second block and 2 layers in the final block. For every transformer layer, the hidden dimension is 768, the intermediate feed-forward dimension is 3072. Each self atteenteo layer consists of 12 heads. We post-norm architecture, GELU activation function [18 in feedforward layers and the relative attention parametrisation from Transformer XI.. This brings our model's size to z 126M parameters. The boundary predictor is 2-layer MLP that takes in hidden state input and outputs scalar prediction at each time step. We use the Adam optimizer [21] with (31, (32) and parameters as (0.9, 0.98) and le-6, respectively. We use learning rate of 5e-5, warmup ratio of 0.1 and 38, 38,000 training steps, a batch size of 512 distributed across 4 A40 GPUs. Each batch consists of examples concatenated up the maximum sequence length of 512.
D.2 Finetuning
We finetune the pretrained models by appending a linear classification layer and update all parameters during training. The boundary predictors' parameters segmentations are well adapted to each task. We finetune for epochs with a batch size of 32 and the same learning rate and warm-up ratio that we used for pretraining. We report accuracy averaged over runs with different random seeds.
also updated to ensure that the predicted
E Supplementary Results
E.1 Equitable Segmentation at Byte-Level Granularity
In Figure 7 we present plots comparing segmentation granularity between (1.) byte-level segmentation
and MAGNET(1, 4)x, (2) Between DTP 5x, MAGNET (5, 10, 13)x, MAGNET (5, 10. 20) x and heaboord-tokeeinee at vocabulary sizes 50k, 100k and 250k with and without alpha sampling (see section 3.1
E.2
Downstream Tasks
We present language-level results (accuracy) across all tasks and models in Table
17

---

Table 6: Language-level results across all tasks
Model
fr
hi
bn
te
Avg
Byte-Level
74.72
70.89
71.77
67.89
62.89
67.93
66.90
69.12
DTP 5x
73.18
70.19
69.78
66.96
61.79
67.11
66.
66. 76
DTP 10x
71.92
68.05
68.13
67.00
61.26
65.79
65.49
66.38
MAGNET 2,
74.44
70.63
71.99
67.48
64.73
67.44
66.88
69.17
MAGNET 3, 6, 12) x
74.41
70.83
70.78
67.66
61.57
65.32
65.14
67.53
MAGNET (5, 10,20) 20)
73.71
70.23
71.15
66.35
61.72
65.12
65.01
67.13
MAGNET(5. 10, 13) x
73.97
70.38
70.58
66.85
62.42
66.11
65.46
67.97
MAGNET 10,10,11 15) x
73.57
70.82
71.36
67.02
61.89
65,43
64.96
67.86
(a) XNLI
Model
en
es
Avg
en
es
ru
hi
Avg
Byte-Level
85.70
80.45
80.40
82.18
55.61
53.225
40.49
29.18
44.62
DTP 5x
84.13
80.20
79.55
81.29
51.40
49.85
41.62
30.39
43.31
DTP 10x
76.13
76. 10
75.75
75.99
40.68
37.75
38.09
26.79
35.82
MAGNET 4)
87.63
81.73
80.55
83.30
55.94
51.78
41.75
28.98
44.61
MAGNET 6, 12)
87.60
83.98
84.65
85.41
55.32
52.46
41.29
25.82
43.72
MAGNET(5, 10, 20) x
82.05
78.83
80.45
80.44
53.12
50.29
39.61
21.23
41.06
MAGNET(5. 10, x
81.40
78.28
79.10
79.59
53.13
50.16
38.31
26.07
41.92
MAGNET 10, 15)
87.78
83.05
83.55
84.79
53.14
51.88
39.82
26.41
42.81
(b) PAWSX
(c) XQUAD
Model
en
es
fr
ru
be
uk
bn
te
hi
Avg
Byte-Level
76.96
72.06
71.57
75.98 71.07 72.55 63.00
68.63
67. 67.65
71.05
DTP 5x
73.28
73.03
69.60
70.34
73.77
70.83
62.01
66.18
63.48
69.17
DTP 10x
73.28
68.38
68.14
67.89
70.83
67.89
63.24
66.67
64.21
67.83
MAGNET (,, 2)
75.25
73.04
72.55
75.00
70.09
70.59
63.24
66.43
73.04
71.02
MAGNET (3, x2)x
79.17
77.94
75.69
75.00
75.74
73.29
56.62
59.31
56.13
69.87
MAGNET (5, 10, 20)
80.15
78.92
77.70
75.98
76.47
74.02
49.75
53.90
51.96
68.76
MAGNET (0, 13)x
79.17
79.66
73.78
71.57
73.78
71.33
59.07
65.68
61.76
70.64
MAGNET 15,10,
80.64
78.92
74.80
72.30
76.23
72.31
56.37
52.94
54.66
68.79
(d) S1B-200
18

---

100
Tokenizer
Takenizer
Text byte level
DTP
300
MAGNET(1, 2, x)
80
MANINETIS
10, 201
number tokess
250
Aetage number tullos
BPE
3990 2006
BPE.100K
2 JIPE 1006
00+
1.PE,LLL,BPEEE JEPE
150
50
uk
be
bn
te
Language
(a) MAGNET compared to byte-level model
(b) MAGNET compared to DIP and BPE
Figure 7: Average number of tokens after segmenting the FLORES dataset. Evidently subword tokenizers and DTP result in over-segmentation in non-Latin script languages, while MAGNET closes the gap.
Table 7: English and Russian instances from the SIB dataset showing slight changes in segmentation before and after fine-tuning. Segmentation for these examples performed using the MAGNET (5x, 10x, 20x).
Lang
Finetuning Segmentation Last
Pretraining Segmentation mon th || presiden Dall comm
Last month a presidential commission recommended the prior CHP's resignation as part package measures to move the country towards new elections.
month a presiden tial com- mission || recommen || ded || the prior CEP's resign a lion |s part of
Last
ission recommended the prior
CEP's || jesign || ation as part || ||
a package || measures || to move
package
measures || to move towards new lections.
the country || towards || new tions.
country
пропдоо MecAm(e преeи- деитскан Rомиссни рско-
npo IITTOM Mecmee преeнge || npo || muoм Mеснne президде || KOMH CCH || pexo- me ндовата || предня y memy wre |aa комиссно рeкoMe ндоо предилу memy Врeme miоy || избиратеи. номY cope- ry ykmm отeтавRy n каиестве части || naxera mep длли даиож 1018 страma НоRMM выборам.
мендовала предилушемм Временоому ызбнрател--
Врeme ииом нлбирател номy coneтy yimn || oreranxy n ka- sеcrвe || части naxera mep дuи доижк || mии страни к IоBM || выñoрaа
HOMy conery yärи и on- ставку в качестве части nakera mep диии диссееиия страны k ноbM BаGоpaм.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We demonstrate through extensive experiments and analysis that our approach leads to equitable segmentation rates across language scripts.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a "Limitations" section that highlights the shortcomings of our approach such as "extension to Semitic languages" and compute-constrainss
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to- -t-teextttttttt system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimeess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[No]
Justification:
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce The paper doesn't include and theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed description of the model architecture, both visually and in- in-writing. We also provide details about our entire experimental-See
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide a link to our code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
cc/
pub icl wetpudddee//oeeueaaaaeee
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips CC/ pwwppuuliiee wttp:///cpwbbccuuceeeeecmmce CodeeS/odeubaa cemmissionil www.ssssooniinnnn for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and basclines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details about our entire experimental setup inclusing model hyper parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
report
Answer:
[Yes]
Justification:
We report results averaged over 2 runs with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It OK to report I I-igma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g, negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We provided details about the computational resources used in this project
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httphh/wwwwee...ooo.o...oooooooa
Answer: [Yes]
Justification: Our work adheres to the NeurIPS Code of Ethics, there are no harmful impacts that could result from our methods.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethies.
The authors should make to preserve anonymity (e.g., if there special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the appendix we write broader impact statement outlining the positive impacts of our work and discouraging malicious use.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Lieenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and assets properly cited and attributed to the respective creators.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper thoode com dataset: has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: This paper does not release assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codd/moodel part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Thiss This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "rDMlBXnIfu", "reviewer": "Reviewer_stTy", "summary": "This paper presents multilingual adaptive gradient-based tokenization (MAGNET), which aims to reduce over-segmentation in non-Latin script languages in multilingual settings. MAGNET processes byte-level sequences and routes them through language-script-specific predictors, each optimized for its respective script, such as Latin, Cyrillic, and Indic. The segmentation is modeled using a sigmoid function, making it differentiable. Experimental results show that MAGNET can maintain downstream task performance while reducing inference latency.", "questions": "1. Is there any analysis of the segmentation? For example, can the segmentation find phrase boundaries?\n2. Is it possible to incorporate prior domain knowledge (such as word dictionaries or tokenization from other tokenizers) to improve gradient-based tokenization?\n3. Are these three predictors sufficient to achieve good performance across all languages? What's the performance for languages that don't have spaces in the sequence, such as Chinese or Japanese?", "limitations": "Please refer to weaknesses.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The main contribution of MAGNET is its ability to maintain performance on downstream tasks while improving inference time by reducing tokenized sequence length. Compared with byte-level tokenizers, inference is more than twice as fast while maintaining or even improving performance.\n2. The idea is simple and straightforward, making the paper easy to follow.", "weaknesses": "1. The proposed method doesn't conduct experiments on languages with sufficient coverage. The predictors only include Latin, Cyrillic, and Indic, raising concerns about its applicability to languages lacking spaces as word boundaries, such as Chinese and Japanese.\n2. The importance of byte-level tokenization and gradient-based tokenizers as research directions is unclear. The experimental results don't demonstrate the significance of byte-level tokenization in terms of downstream performance and inference latency.\n3. The training objective remains byte-level without segmentation. This raises questions about whether MAGNET still needs to generate very long byte sequences during inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer stTy for taking the time to review our work and noting that MAGNET maintains performance on downstream tasks while improving efficiency and combating over-segmentation. We address your concerns below.\n\n**Sufficient language coverage**\n  - Our choice of languages was not influenced by space separation, but rather by linguistic diversity and the availability of data for downstream evaluation. Whilst we agree that CJK languages like Chinese and Japanese could have been included in our experiments, we were limited by computational resources and future work should extend our current work to these languages.\n - MAGNET is very flexible and applicable to all languages that can be expressed with UTF-8, we use byte-word-ratio as a simple proxy to train our boundary predictors to learn equitable tokenization.  Indeed we cannot use byte-to-word ratio for languages without whitespace like CJK languages, but it's important to note that byte-word-ratio is not a compulsory proxy and for such languages other proxies can be used. Also computing the byte-to-word ratio is quite trivial and doesn’t require a huge amount of text . For CJK languages, one could employ state-of-the-art word-segmentation tools like Jieba to count the number of words before computing the byte-to-word ratio. \n\n**Byte-level and gradient-based tokenization research directions**\n - Several works Ahia et al. (2023), Petrov et al. (2023) have pointed out flaws of subword level tokenization algorithms, particularly over-segmentation in non-Latin script languages. Tokenization in general is an active research area and there has been recent efforts to make it more robust and easily adaptable across languages and data domains Clark et al. (2022), Xue et al. (2022), Tay et al. (2022), Yu et al. (2023). Byte-level models are very relevant because of their high coverage since UTF-8 supports most of the world’s scripts. They have also generally been shown to match or surpass subword-models in performance. Moreover, gradient-based tokenization makes byte/character level models fairer and more efficient. We will highlight this better in our contributions in the final draft.\n\n**Training objective**\n - The training objective is indeed byte-level and slower, and follow-up ideas similar to Fleshman et al. (2023)  can address them. For this reason, we limited our evaluation to text understanding tasks following lots of prior work on tokenization Godey et al. (2022), Clark et al. (2022),Tay et al. (2022) . We believe that future work should focus on text-generation tasks.\n\n**Is there any analysis of the segmentation? For example, can the segmentation find phrase boundaries?**\n  - MAGNET is customizable and the boundary predictors can be trained to learn different granularities of segmentations including phrase segmentations. We provide qualitative analysis of the segmentations learned by MAGNET in comparison to DTP and subword-level models in Table 5 and 6 in the Appendix.\n\n**Is it possible to incorporate prior domain knowledge (such as word dictionaries or tokenization from other tokenizers) to improve gradient-based tokenization?**\n  - Yes, this is very much possible. Although we train our boundary predictors via stochastic reparameterization, we note that one can also incorporate supervision from other sources such as word dictionaries or even BPE tokenizers. DTP Nawrot et al. (2023) experimented with this in their work. \n\n**Are these three predictors sufficient to achieve good performance across all languages?**\n  - We used three predictors in our experiments because the languages we covered belonged to three language subfamilies. First, the number of boundary predictions could be scaled based on user preference. Second, the trained boundary predictors are sufficient to cover languages with properties similar to those in the training data. Beyond languages in our pre-training data, we also conducted downstream experiments on 5 other Indo-Aryan languages. We observed that the predicted segmentations on these languages are close to the word-level segmentations we see in language.   \n\n&nbsp;  \n&nbsp;  \n&nbsp;  \n---\nPetrov, Aleksandar, et al. \"Language model tokenizers introduce unfairness between languages.\" Advances in Neural Information Processing Systems (2023)\n\nAhia, Orevaoghene, et al. \"Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.\n\nClark, Jonathan H., et al. \"Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation.\" Transactions of the Association for Computational Linguistics 10 (2022)\n\nXue, Linting, et al. \"Byt5: Towards a token-free future with pre-trained byte-to-byte models.\" Transactions of the Association for Computational Linguistics 10 (2022)\n\nTay, Yi, et al. \"Charformer: Fast Character Transformers via Gradient-based Subword Tokenization.\" International Conference on Learning Representations.\n\nYu, Lili, et al. \"Megabyte: Predicting million-byte sequences with multiscale transformers.\" Advances in Neural Information Processing Systems 36 (2023) \n\nFleshman, William, and Benjamin Van Durme. \"Toucan: Token-Aware Character Level Language Modeling.\" arXiv preprint arXiv:2311.08620 (2023)\n\nGodey, Nathan, et al. \"MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling.\" EMNLP 2022-The 2022 Conference on Empirical Methods in Natural Language Processing. 2022\n\nNawrot, Piotr, et al. \"Efficient Transformers with Dynamic Token Pooling.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPcI90Jn39", "reviewer": "Reviewer_WSTY", "summary": "In this paper, the authors propose a multilingual adaptive gradient-based tokenization approach to reduce over-segmentation for non-Latin language texts. In particular, they improve the previous Dynamic Token Pooling method by inferring token boundaries with different predictors for different languages. They conduct extensive experiments to demonstrate that not only the over-segmentation issue is reduced, the inference efficiency also increases.", "questions": "In Figure 2, are some lines missing or overlapped?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Apply different boundary predictors for different languages/scripts, which is a natural improvement over the previous approach, DTP\n* The results look reasonable and similar to human tokenization", "weaknesses": "* Basic token prediction tasks or text generation tasks are not included in the evaluation\n* Downstream tasks' performances aren't different\n* Inference time improvement isn't much compared to DTP", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer WSTY for taking time to review our work and noting that our improvements over previous work yield better tokenization. We address your concerns below.\n\n**Basic token prediction tasks or text generation tasks are not included in the evaluation**\n  - We limited our evaluation to text understanding tasks following lots of prior work on tokenization Godey et al. (2022),  Clark et al. (2022), Tay et al. (2022). We believe that future work should focus on text-generation tasks. Text generation tasks are indeed byte-level and slower, and follow-up ideas similar to Fleshman et al. (2023)  can address them. \n\n**Downstream tasks' performances aren't different**\n  - The goal of Magnet is not to outperform byte-level models or DTP, but to encourage multilingual language models to learn equitable segmentations that result in equitable model representations across languages. However, we still see that Magnet outperforms DTP on most tasks. \n\n**Inference time improvement isn't much compared to DTP**\n - Inference time improvements compared to DTP do not appear significant due to some in-effciencies in our implementation. We will invest more time into making the implementation more efficient for the final version of the paper.\n\n&nbsp;  \n&nbsp;  \n&nbsp; \n\nGodey, Nathan, et al. \"MANTa: Efficient Gradient-Based Tokenization for Robust End-to-End Language Modeling.\" EMNLP 2022-The 2022 Conference on Empirical Methods in Natural Language Processing. 2022\n\nClark, Jonathan H., et al. \"Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation.\" Transactions of the Association for Computational Linguistics 10 (2022)\n\nFleshman, William, and Benjamin Van Durme. \"Toucan: Token-Aware Character Level Language Modeling.\" arXiv preprint arXiv:2311.08620 (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9vRnTHoSNF", "reviewer": "Reviewer_KW7H", "summary": "The authors propose MAGNET (multilingual adaptive gradient-based tokenization) to remove the common problem in non-Latin-script languages getting only tokens (subwords assigned in the vocabulary) representing short character sequences as opposed to English getting high-semantic-content tokens.\n\nAs opposed to previous gradient based tokenization strategies which globally minimize compression cost across all sequences, they use language specific segmentation predictors to reduce oversegmentation in non-Latin-script languages, to produce a better multilingual byte-level LM.\n\nWithin a byte-level LM the tokenization acts as an information bottleneck, where the input subword byte sequences have to be predicted again at the output using an \"upsampling module\", while next word prediction takes place over the tokenized-on-the-fly blocks.\n\nThey compare MAGNET to DTP (which doesn't have the same conditioning but otherwise has identical structure) and naive BPE over the vocabulary. Figure 2 shows that the avg # tokens per passage across all languages, as opposed to BPE's severe jump for Hindi, Bengali, and Telegu, and DTP's punishment of both Cyrillic and Indic script languages.", "questions": "N/A", "limitations": "Yes, they mention language-specific issues to expanding their method beyond the alphabetic languages they examined. I would like to see them mention the further potential improvements to using inline language ID prediction (conditioned in training, maybe predicted implicitly in inference?) to control the segmentation module and maybe ensure more equitable performance across the languages.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This tokenization issue comes up a lot in multilingual NLP research. Different segmentation levels between languages can severely hamper LM performance---particularly open source, open weight LM performance---on many cross-lingual tasks including summarization, translation, QA, and reasoning, so fixing this with architectures such as this and scaling them will prove very impactful.\n\nThis simple proposal to fit the hourglass architecture to script-denominated segementation predictors is a simple but useful refinement.\n\nSimple presentation of their method's superior results for simple tokenization, which also generalized to better performance on most multilingual tasks such as XNLI, PAWS-X, and SIB.\n\nThe savings in token cost for passages translate into better inference time as well.", "weaknesses": "Language ID is still an issue here. While they are able to fit these individual segmentation predictors based on implicit language family by script (which can be inferred from position in the UTF-8 table), I think the case could be made that they're basically just kicking the unequal tokenization can down a level. Now words in Telegu might be worse segmented than words in Hindi, etc. Really an inline language identification-conditioned segmentation module would be best here. But that's probably better as a direction for future work anyway.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer KW7H for reviewing our work and noting that MAGNET is very impactful and useful in reducing over-segmentation in multilingual language models. They also note that MAGNET results in better performance and lower inference costs. We appreciate your suggestions and address your concerns below. \n\n**Resolving Language ID issues**\n - Yes, this is a valid point and a great suggestion. Currently, the number of boundary predictors in MAGNET can be scaled based on the linguistic properties that the languages in the pretraining data share. This is straightforward given that the boundary predictor is a small module within the entire model.  However, a language identification-conditioned segmentation module that doesn’t require adding too many individual boundary predictors could also be relevant here, and this would be a great avenue for future work. We will include this discussion in the limitations and future work section of the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwqOEkqeRx", "reviewer": "Reviewer_2xYq", "summary": "This paper proposed MAGNET, a gradient-based tokenization method, to address an over-segmentation issue when handling multilingual text data written in different scripts. MAGNET learns to predict segment boundaries between byte tokens in a text sequence. MAGNET has a customizable architecture where byte-level sequence are routed through language-script-specific predictors, which implements a language-dependent tokenization and thus avoids over-segmentation in non-Latin scripts. The authors conducted extensive experiments on nine languages with distinct scripts. Their results show that the proposed approach works well in the downstream task and moreover contributes to the speed-up at inference time. \n\nThis paper is well organized and clearly motivated. The over-segmentation gets a bigger issue in multilingual model training so the proposed approach will be useful.", "questions": "- I see the proposed approach successfully overcoming segmentation disparities in subword tokenization though what else would be needed to improve the downstream task perfomance? For instance, Figure 3 shows that the byte-level model also achieves good scores across.\n\n- Other interesting languages would be Chinese, Japanese and Korean scripts. Have you ever experimented with the languages? If yes, do you observe anything new?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well organized paper. easily to follow each section. enough description is provided.\n- Extensive results with 9 languages with 3 different scripts.. the authors conducted extensive experiments in the downstream task \n- Successfully reducing the inference time", "weaknesses": "- experiments are carried out in three different scripts. Other interesting languages would be Chinese, Japanese and Korean scripts as another major examples that struggle with the out-of-vocabulary issues.\n- despite addressing the oversegmentation, performance is still similar to that of byte-level or baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 2xYq for taking the time to review our work and noting that our experiments and results are extensive with a successful reduction in inference time. We thank you for your suggestions and address your concerns below. \n\n**Sufficient language coverage**\n   - Our choice of languages was not influenced by space separation, but rather by linguistic diversity and the availability of data for downstream evaluation. Whilst we agree that CJK languages like Chinese and Japanese could have been included in our experiments, we were limited by computational resources and future work should extend our current work to these languages.\n   - MAGNET is very flexible and applicable to all languages that can be expressed with UTF-8, we use byte-word-ratio as a simple proxy to train our boundary predictors to learn equitable tokenization.  Indeed we cannot use byte-to-word ratio for languages without whitespace like CJK languages, but it's important to note that byte-word-ratio is not a compulsory proxy and for such languages other proxies can be used. Also computing the byte-to-word ratio is quite trivial and doesn’t require a huge amount of text. For CJK languages, one could employ state-of-the-art word-segmentation tools like Jieba to count the number of words before computing the byte-to-word ratio. \n\n**Despite addressing the over segmentation, performance is still similar to that of byte-level or baselines**\n   - The goal of MAGNET is not to outperform byte-level models or DTP, but to encourage multilingual language models to learn equitable segmentations that result in equitable model representations across languages.We still, however, see that MAGNET outperforms DTP on most tasks .\n\n**I see the proposed approach successfully overcoming segmentation disparities in subword tokenization though what else would be needed to improve the downstream task performance?**\n  - Due to computational limitations, we are restricted to pretraining on a smaller subset of data compared to state-of-the-art models. We believe that scaling the size of our models and pretraining data could significantly help improving downstream performance alongside overcoming segmentation disparities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tackles a well-known issue in multilingual NLP, where tokenization discrepancies lead to unfair performance across languages. Reviewer KW7H highlights the importance of this: “This tokenization issue comes up a lot in multilingual NLP research. Different segmentation levels between languages can severely hamper LM performance...so fixing this with architectures such as this and scaling them will prove very impactful.” It does so with a simple yet novel approach, MAGNET, which introduces a customizable, gradient-based tokenization method that adapts to different language scripts. The experiments are also strong, as the paper demonstrates nine languages that MAGNET reduces over-segmentation and improves inference time on while maintaining or exceeding downstream task performance compared to baselines. Reviewer 2xYq acknowledges the thoroughness: “Extensive results with 9 languages with 3 different scripts.. the authors conducted extensive experiments in the downstream task.” There were some concerns about the limited language coverage, particularly the absence of CJK languages. I agree with 2xYq and stTy that including them would enhance the technical soundedness of this work, but accept the authors' rationale regarding this issue, "Our choice of languages was not influenced by space separation, but rather by linguistic diversity and the availability of data for downstream evaluation. Whilst we agree that CJK languages like Chinese and Japanese could have been included in our experiments, we were limited by computational resources". Also KM7H makes a good point that "CJK languages are much more well-studied than Indic languages as is; and they also have more information content per byte so segmentation is less of an issue than for alphabetic languages."

Overall the paper received positive reviews, and most concerns were addressed satisfactorily during the rebuttal in my opinion, so I vote to accept the paper.
Decision: Accept (poster)