Zero-Shot Tokenizer Transfer
Authors: Benjamin Minixhofer
Edoardo Ponti
Ivan Vulić
Venue: NeurIPS 2024 poster (2024)
Keywords: tokenization
transfer learning
natural language processing
hypernetworks
zero-shot learning
Primary area: natural_language_processing
TL;DR: We introduce the new problem of Zero-Shot Tokenizer Transfer (using a language model with a tokenizer it has never been trained with), and a first high-performing baseline to solve this problem.

Abstract: Language models (LMs) are bound to their tokenizer, which maps raw text to a sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained primarily on English may still perform well in other natural and programming languages, but have vastly decreased efficiency due to their English-centric tokenizer. To mitigate this, we should be able to swap the original LM tokenizer with an arbitrary one, on the fly, without degrading performance. Hence, in this work we define a new problem: Zero-Shot Tokenizer Transfer (ZeTT). The challenge at the core of ZeTT is finding embeddings for the tokens in the vocabulary of the new tokenizer. Since prior heuristics for initializing embeddings often perform at chance level in a ZeTT setting, we propose a new solution: we train a hypernetwork taking a tokenizer as input and predicting the corresponding embeddings. We empirically demonstrate that the hypernetwork generalizes to new tokenizers both with encoder (e.g., XLM-R) and decoder LLMs (e.g., Mistral-7B). Our method comes close to the original models' performance in cross-lingual and coding tasks while markedly reducing the length of the tokenized sequence. We also find that the remaining gap can be quickly closed by continued training on less than 1B tokens. Finally, we show that a ZeTT hypernetwork trained for a base (L)LM can also be applied to fine-tuned variants without extra training. Overall, our results make substantial strides toward detaching LMs from their tokenizer.

Zero-Shot Tokenizer Transfer
Benjamin Minixhofer [SEP]
Edoardo M. Ponti [CLS] Ivan Vulić [SEP]
[SEP) University of Cambridge
[cLS] [CLS] University of Edinburgh
Abstract
Language models (LMs) bound to their tokenizer, which maps raw text to sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained primarily English may still perform well in other natural and programming languages, but have vastly decreased efficiency due to their English- centric tokenizer. To mitigate this, should be able swap the original LM tokenizer with an arbitrary one, on the fly, without degrading performance. Hence, in this work define new problem: Zero-Shot Tokenizer Transfer (ZeTT). The challenge at the core of ZeTT is finding embeddings for the tokens in the vocabulary of the tokenizer. Since prior heuristics for initializing embeddings often perform at chance level in ZeTT setting, we propose new solution: we train hypernetwork taking a tokenizer as input and predicting the corresponding embeddings. We empirically demonstrate that the hypernetwork generalizes to new tokenizers both with encoder (e.g.. XLM-R) and decoder LLMs (e-g.. Mistral-7B). Our method comes close to the original models' performance in cross-lingual and coding tasks while markedly reducing the length of the tokenized sequence. We also find that the remaining gap can be quickly closed by continued training on less than IB tokens. Finally, we show that a ZeTT hypernetwork trained for base (L)LM can also be applied to fine-tuned variants without extra training. Overall, our results make substantial strides toward detaching LMs from their tokenizer.
Introduction
Language Booel::yypicaan operate on discrete tokens, so they need a means to map text into sequence of tokens, namely lokenizer. The vast majority of contemporary LMs use subword tokenizers (Devlin al., 2019 Jiang al. 20233 Touvron et al. [023 Parmar al. 2024 among others), whereas others use lete- (Xue et al. 2022] Yu et al. 2023 Wang et al. 2024) or character-level tokenizers (Clark et al., 2022 Tay et al. 2022). Regardless
the chosen tokenization 'granularity'. these models share fundamental limitation: once they are trained with particular tokenizer, inference with a different tokenizer is impossible. In other terms, a pre-trained LM is "bound" the tokenizer it was trained with. This has wide-ranging implications: since the focus during pretraining is typically primarily on the English language, the tokenizer often encodes languages besides English (Rust al. 2021) other domains, such code, less efficiently. This leads to large disparities the inference cost between English and non-Eri -on-English text (Ahia et al. 2023 Petroy al. (023). Tokenizers may also be sub-optimal for domains which they used with, e.g. fine-tunings of the Llama models performing subpar on coding tasks (Dagan et al.) 2024). Efficiency and performance
designed to be
only some of the reasons transfer models across tokenizers: methods of interaction between models, such as ensembling (Sagi & Rokach 2018) and model merging /Wortsman al. 2022 Ainsworth et al. 2023 Yaday al. (023). typically assume the same unit of representation (i.c., equivalent tokenization) across models; if two models adopt different
We adopt a broad definition of LMs that also includes models that do not define a prohability distribution over finite-length sequences, such as text encoders.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

text
O T
B Input Embeddinn
L.M.
logits
Tokarinee
Output Entadding
Hlg Hypremetorrk
Figure 1: The hypernetwrkk predicts input and output embeddings based on the tokenizer.
tokenizers. they become unsuitable for ensembling or merging. Problematic artifacts of tokenization
such as 'Glicch tokens' (Land & Bartolo 2024) may also be fixed via transfer to a new tokenizer.
To address these issues, past work developed methods equip LM with new tokenizer by retraining the embedding parameters, and optionally continuing to train the entire model (Artetxe) et al. 2020 de Vries & Nissim, (021). This adaptation can be made faster by initializing the embedding parameters through heuristics (Tran) 2020) Minixhofer et al.) [022 Gee et al. [022] Dobler & de Melo 2023 Liu et al., 2023). In this work, we formulate new problem: given an LM, can we create an embedding matrix on-the-fly for any arbitrary tokenizer, without ever observing data for it? While past work investigated n-shot tokenizer transfer. we refer to this new problem as zero-shot tokenizer transfer (ZeTT), If the performance of the model can be approximately preserved, TTTT effectively "detaches" LMs from the tokenizer they were trained with. We first evaluate the efficacy of prior (heuristic-based approaches for ZcTT, finding that, while heuristics can preserve performance to some extent, there is generally large gap to the original LM performance.
To close this gap. we introduce a new paradigm: We train a hypernetwork on a diverse distribution of tokenizers to predict the embedding parameters for any given tokenizer. By investing in the one-time cost of training the hypernetwork, we aim to subsequently enable effective ZeTT. This proves to be possible: ZeTT via the hypernetwork preserves performance to few percent accuracy in many cases. Furthermore, the hypernetwork can learn to rapidly adapt to a given target tokenizer by continued training on a small amount (<IB) of extra tokens, whereas previous work typically needed hundreds of billions of tokens (Dagan et al. (024). As such, our hypernetwork provides -feheaff solution n-shot tokenizer transfer, while also establishing a competitive baseline for our newly introduced zero-shot tokenizer transfer problem. This unlocks range of new ways to combine language models with tokenizers. For example, in this work, we zero-shot substitute the Mistral-7B tokenizer (Jiang et al. (023) with a tokenizer that encodes code using 10% fewer tokens on average, while preserving functional code generation correctness to approx. 3% (Section (4.2). We also evaluate zero-shot cross-lingual transfer of the multilingual XLM-R encoder model to range of different languages by substituting the XLMMR tokenizer with target-language specific tokenizer and reusing adapters trained for the original XLM-R. This Ieads to a >16% speedup and preserves performance on XNLI (Conncau et al. 2018) to 1% on average. Finally, we show that a hypernetwork trained for base large LM (e.g. Mistral-7B) can also be applied to fine-tunings of the same model (e.g. Mistral-7B ----nntttuuttt... preserving capabilities to a large extent (Section
Background
Tokenizers and Embeddings. Tokenizers operate as a tokenization Iuunciiaaionnuunctinnn T mapping text to sequence of elements the vocabulary V. By the term tokenizer, we henceforth refer to the tuple comprising the two crucial components, (V, T') Importantly, the vocabulary and the tokenization function are distinct components; given some vocabulary. there are many ways to encode text as a sequence of tokens in this vocabulary (e.g. Hofmann et al. 2022 Uzan et al. 2024). After tokenization, the model represents the sequence of tokens via a function Eg : V RT gran (the embeddings). The embeddings assigns a distinct dinns--dimeesional vector (a row of the matrix) to every element V. Embeddings used twice in the language model: once at the input to map tokens again at the output to compute a logit for every token, typically via a dot-product of (t) with the final hidden state of the LM. Embedding parameters may may not be shared between the input and the output(3 2 our method works with both. We denote the entire set of embedding parameters via Φ, denoting input embeddings df and output embeddings doul if necessary.
typically parametrized by matrix á lookup table which
fixed-size vector, and
2 "Some models share the input and the output embedding parameters (e.g. Conneau et al. 2020). this has been shown to be problematic (Chung et al. 2021 and many recent LLMs (e.g. Jiang al. 2023 separate them.

---

Contemporary language models typically use subword tokenizers via BPE (Sennrich et al. 2016 or UnigramLM (Kudo 2018). Subword tokenization a common choice since can represent arbitrary sequences of text (" open-vocabulary" language modeling) while largely retaining the efficiency of word-level models (Mielke et al. 202T). However, there are a number of problems with the (lack of) robustness of subword tokenization (Xue et al., 2022] Golkar et al., (023). A recent strand of work aims to get rid of subword tokenization via byte-level (so-called "token-free") models (Xue) et al. 20222] Yu et al., 2023). However, these models still operate tokens, using the set of 256 bytes as the vocabulary, and UTF-8 as the tokenization function (Mielke et al. 202Tt. lp. In a similar
vein, some models
character-level tokenization (Tay et al., |022|Clark al. [0222, ) optionally learning to pool characters into longer tokens (Nawrot et al. (023). So far, byte- or character
approaches have been unable to supplant subword tokenization due to longer sequences resulting in higher compute requirements, and not necessarily being more robust (Libovický et al., 2022).
Thus, although our approach is applicable any tokenizer, focus experiments subword tokenizers. Specifically, we use the Unigram LM parametrization of the tokenization function, and show that other tokenizers can be converted to this parametrization later in Section [5] UnigramL.M sets I [z) ;= argmaxcec, Eee log p(t) where C2 is the set of all possible decompositions of x in V. This provides a convenient way to represent tokens as a 2-tuple (t. p(t)) E (V,
Embedding Initialization Heuristics. Prior work transfers LMS to a new tokenizer by initializing embedding parameters via heuristic, then continuing to train the embeddings. We denote the original tokenizer as (Va: Ta) and the original embedding parameters as d.. Analogously, the target tokenizer is (V,, T5) with embedding parameters Φ6. FVT (Gee et al. 2022) initializes embeddings for any new token E V/2 as the mean of the embeddings of (t) i.e. the mean of the sequence of embeddings the new token is decomposed into by the previous tokenizer Ta. RAMEN (Tran] 2020), WECHSEL (Minixhofer et al.] (022) and OFA (Liu et al. 2023) require auxiliary embeddings Eaux V Re with /anxx n K [Va and VVa. V.l * D.... They use Eaux to embed tokens in Va and Vú in the same semantic space, then initialize embeddings in F.g, as a weighted average of embeddings in Esg, with weights given by their similarity in F FOCUS (Dobler & de Melo 2023) initializes embeddings of tokens in V. Va as a weighted combination of the overlapping tokens Va n Vú, and copies the embeddings of the overlapping tokens. Weights are again computed using an auxiliary embedding matrix Emax but the only requirement is VMax n V% * [V/.. We use FOCUS as the main baseline since Dobler & de Melo (2023) show it obtains better performance without any training (i.e., zero-shot) than other heuristics, which we also confirm later in Section 4.2
Heuristic-Free Tokenizer Transfer. In addition to heuristics, there is also research into changing the training procedure facilitate n-shot tokenizer transfer. Marchisio et al. (2023) show that forward- and backward- backwarr--ropagating through subset of the model layers is sufficient for learning embeddings for new tokenizer. Chen et al. (2023) find that regularly resetting the embedding parameters during pretraining boosts the speed at which they are relearnt upon transfer. These approaches can be seen as orthogonal to ours. They could be freely combined with our method; leave this to future work.
Hypernetworks networks that predict the parameters of another network (Ha et al. 2017). Prior work uses hypernetworks to predict embeddings for out-of- vocabulary (Pinter al. (201]) or rare words (Schick & Schütze 2019 2020) of word embedding models (Mikolov et al. 2013) and BERT (Devlin et al.) 2019). In contrast, our hypernetwork (i) approaches the more general problem of transferring an arbitrary tokenizer, instead of extending the original tokenizer and (ii) can be applied to encoder and decoder LMs, that is, it is objective-agoosic
Embedding Prediction Hypernetworks.
3 Methodology
3.1
Hypernetwork Training
We aim to find parameters of a hypernetwork Hg : (V,, Tb) - Φb for some pretrained LM. Let Φa and vi be the embedding and inner (non-embedding) parameters of the language model, respectively. C is the loss of the language model as a function of the tokens, the embedding parameters, and the inner parameters, typically:
L(t, Φ.: 1)= = CrossEEttrpyy (E,, (t)), label(t)),
where L. is the language model and label maps the sequence of tokens to corresponding labels, e.g., shifting the sequence in case of standard (autoregressive, causal) language modeling, masking
3

---

Algorithm Hypernetwork training loop for Zero-Shot Tokenizer Transfer
corpus D, tokenizer sample size n, batch size max. token length 1, vocabulary size i noise parameters (H, o. pretrained LM parameters 45, initial hypernetwork parameters 0mn-
Input:
Output: Hypernetwork parameters 0.
1: procedure TRAINHHPPERNETOORK
0uii
3: - queue queue((i,... (1,,.,, 2n ~ D) D-Ceatee pool of n texts (where n 2 m))
4:
5:
for step in train sts_sstesspc do
6:
#1, 3m
7:
q + pop(g, m) PRemove the ceastrecently-addee batch.
8:
q - push q. ,,,, Tm) AAdd the current batch.
9:
10:
1. f substrings( (q. I) &Compute all substrings and their frequency q.
f f E; oNormalize frequencies sum to one.
12:
~~ Lognormal(y,, o2)
13:
for f, f (t, do
14:
p(t) + ( + DAssign a score based on frequency + noise to the substrings. Sort by p(t) descending.
15:
16:
V. - b|: kij >Assemble the top is substrings into the tokenizer.
17:
T. ([(
18:
19:
loss + (x), Ho(,, Tb), v)) &Compute the loss on the m texts in the current batch. update using V8 w.r.t. loss.
20:
the sequence in case of Masked Language Modeling (Devlin et 2019). Importantly, however, we do not make any specific assumptions on L.
Note that the loss of the language model under the original tokenizer T. text is ((Ta(r), das v) We train our hypernetwork to minimize the loss Lo (T)(). Ho(,, Ts), v).. That is, we substitute the original embedding parameters for the hypernet predictions, and substitute the original tokenizer for a tokenizer (V, 75). Figure iillustrate the flow of information.
Defining Distributions over Texts and Tokenizers. We follow standard practice and sample texts uniformly from the training corpus. Tokenizer sampling is not trivial: would like distribution over tokenizers (v,,T)) with high variance to encourage generalization to unseen tokenizers. To this end, we introduce procedure sample diverse set of Unigraml tokenizers. We show later in Section ( that arbitrary tokenizers can be well-approximated via UnigramL.M, motivating this choice.
We initially fill queue q with texts sampled randomly from the training corpus and, every step the training loop, push the m texts in the current batch and remove the m least recently added texts. We then compute all substrings the frequencies to arrive at a final score p(t) for every token Finally, we assemble the tokenizer by taking the top tokens with the highest p(t) the vocabulary and UnigramLM parametrized by p(t) as the tokenization function. The training loop is summarized in Algorithm [ The 'rolling' queue of texts q ensures high variance in the vocabulary, while the Gaussian noise added to the frequencies ensures high variance in the tokenization function.
to length and their frequency in q
3l We add Gaussian noise to
Importantly, the texts and the tokenizer are sampled dependently: the batch of m texts used for training is subset of the texts used for sampling the tokenizer. If they were sampled independently, the probability for a token occur would be p(token) p(token V%) x p(token E æ). Since both these factors are small for rare tokens, p(token) would get vanishingly small this case.
MIMICK-Sty -Style Warmup & Auxiliary Loss. In practice, directly minimizing Co starting from randomly initialized 0 is difficult. Thus, we include a warmup stage where we train the hypernetwork to mimic the embedding parameters of the original tokenizer, akin MIMICK (Pinter al., 2017).
C warmup
Ho (Va, T.) &all2
II practice, implementing q as a queue allows efficiendy eaching the substrings and their probability p(t) this step. They only need be recomputed for the new m texts encountered every batch. "To ensure substrings do not cross word boundaries pretokenize the text before computing substrings.

---

7, Tosertaation Fanction
7 Vocabulary
Hg
A Prodicted Ennirddings
Psperstwork
<peats
HLM,,
Rowerbeds
S
Roreol
Sydood
Fi
HLM,,
F Rowerbooda
L
Smoog
<pials
HLM,
Triden
HLM,,
deccoposee with original tokenizer T, ambed with original ambediingg lig.
compose into new embedaings
Figure 2:
The hypernetwork consists of a language model HLMg learning to compose embeddings under the original tokenization into embedding and amortizes over the tokenization function.
The warmup stage is substantially quicker than the main stage because there is no need to propagate through the main model. We found it prevents divergence in some cases. Afterwards, we add an auxiliary loss, which, for every token the sampled vocabulary V3 that also exists in the original vocabulary Va, penalizes the distance to the corresponding embedding in da.
E
- 6u V [t]ll2
Va n Vol
This penalizes drift from the warmup stage. Combining it with the main loss yields the final loss.
clinal = (T((())),, Ho To), (b) Laux
The hyperparameter weighs the contribution of the auxiliary loss. Since Ha (V.. TT) is also required for the main loss, requires negligible extra computation. The auxiliary loss necessary especially for models with separate input and output embedding matrices as shown in Appendix B
3.2 Hypernetwork Architecture
remains to define the hypernetwork architecture, that is, how map the tokenizer (V,, TT) to the embedding parameters 66. To this end, we represent the new tokens to E V. by decomposing them using the original tokenization function Ta, and embedding them with the original embeddings Eg_ This sequence of embeddings is passed through multiple Transformer layers, plus separate prediction head for the input embeddings and output embeddings Φi, and ogt The hypernetwork thus consists of another language model which applied separately for every token. We refer the hypernetwork's 's language model as HLMg. HLM, can be thought of as learning how to compose the sequence of tokens (P(()which any given token is decomposed into-into one embedding, as illustrated in Figure [ Importantly, we do not take the tokenization function into account. By sampling diverse tokenizers during the training process, we aim for the hypernetwork to learn to produce a single embedding suitable to wide variety of different tokenization functions. We analyze the impact of this choice later in Section 5 We also experiment with hypernetworks which do take the tokenization function into account in Appendix C
On Token Decomposition. The input to the hypernetwork consists of the sequence of tokens (t) that any given token decomposed into. However, this decomposition is not always trivial: for example, T, could be character-level. while the token could be in the vocabulary of a byte-level tokenizer Ty. In this case, i could be any arbitrary sequence of bytes (not necessarily valid UTF-8). To solve this issue, we introduce a procedure to convert tokenizers to the byte level by adding small amount of extra tokens to the vocabulary (c.f. Section 5). This guarantees that Ta can decompose arbitrary tokens. The embeddings of the extra vocabulary are initialized randomly and trainable alongside the hypernetwork parameters.
In the multilingual case, also append element containing a learnable Sanguage-specifie embedding.
5

---

Table I: Accuracy on XNLI when reusing adapters trained for the original XLM-R model with new zero-shot transferred language-specti tokenizers. Also shown are the absolute change in accuracy from applying our hypernetwork (Aaceuracy) and the average decrease in token length of the sanguage-specifte tokenizers over the original tokenizer (Alength).
bg
de
el
en
fr
hi
vi
Avg.
original
68.9
75.6
74.7
73.7
82.3
76.9
76.8
68.4
72.9
63.5
72.2
64.7
73.1
72.6
Lexical
58.7
63.1
65.3
61.7
72.8
68.4
66.7
61.8
62.3
51.8
58.5
60.0
72.0
63.3
FVT
63.9
70.3
70.9
67.4
79.0
73.9
71.9
65.7
67.8
57.1
66.3
61.7
72.9
68.4
OFA
57.3
64.2
67.3
62.8
73.6
68.6
68.4
61.8
63.1
54.8
59.7
59.3
72.3
64.1
FOCUS
64.8
71.0
71.6
67.7
79.6
74.4
72.6
64.5
68.1
55.7
67.3
61.9
72.6
68.6
ours
67.9
73.9
74.1
71.4
81.1
76.2
74.7
67.7
70.7
62.3
68.7
63.2
73.9
71.2
Aaccuracy
-1%
-2%
-1 1%
-2%
- 1%
-1 1%
-2%
-1%
-2%
-1 1
-3%
-2%
+1 %
-1%
Alength
-22%
-14%
-13%
-23%
-9%
-11
9%
12%
-13%
-13%
19%
15%
-9%
-3%
-14%
Table 2: Performance of Mistral-7B-vO.J after zero-shot and n-shot tokenizer transfer (training on 800M tokens). We evaluate transfer the GPT2 tokenizer on natural language benchmarks and transfer to the StarCoder tokenizer on HumanEyalPack. Note that continued training with the original tokenizer (original8800M) 800M) does not consistently improve performance.
Natural Language
Code (pass @ (-> StarCoder Tok.)
(-> GPT2 Tok.)
#shots Method
HumanBivllPack
PIQA
HS
ARC
BoolQ
MMLU
Avg.
Avg.
js
java
original
80.7
81.0
79.5
83.6
59.6
76.9
28.7
20.1
29.3
29.9
32.3
28.1
original @ 800M
82.1
82.7
80.6
80.6
57.8
76.8
31.7
19.5
28.7
27.4
26.2
26.7
FOCUS
69.2
63.8
45.7
60.4
38.8
55.6
21.9
1.8
0.0
20.1
22.6
13.3
0-shot
ours
79.7
77.5
73.0
81.9
53.0
73.0
23,8
17.7
18.9
28.7
26.8
23.2
80CUS@800M
74.8
74.3
72.4
73.3
48.9
68.7
24.4
17.1
22.6
22.6
26.2
22.6
n-shot ours @ 800M
80.9
80.7
77.8
80.7
54.4
74.9
28.0
25.0
26.2
29.9
28.7
27.6
Experiments
4.1 Setup
Data. We use the English subset of the MADLAD-400 corpus (Kudugunta et al. 2023) and code from the StarCoder data (Li et al. (203) for hypernetwork training. The sampling ratio of English to Code 7:3 following Zhang et al, a (2024). For the multilingual hypernetwork, we use a subset of 26 of the languages used in XGLM (Lin et al. 2022) with data from MADLAD-400. We sample languages using a multinomial distribution as in Conneau & Lample 2019)) with a = 0.1. For the n-shot experiments, also train on the StarCoder data, but substitute the English section of the MADLAD corpus for Flan v2 Longpre et al. 2023) sampled as in Soldaini et al. 2024
Evaluation. We use the standard benchmarks PiQA (Bisk et al., 2020), HellaSwag (HS; Zellers
et al. 2019), BoolQ (Clark et al. (211)), MMLU (Hendryeks et al. (0221) and the "easy" subset of ARC (Clark et al.) (2088) for evaluation in English and the synthesis task of HumanEyalPack (Muen nighoff al. (023) for coding evaluation. For multilingual evaluation, we use XNLI (Conneau et al.
2018), XCOPA (Ponti et al. 2020) and MMLU as machine-translated by Lai et al. (2023).
"We exclude languages without whitespace between words since they would require language-specific pretokenizers (e.g. Sun 2012). Although our method is also applicable to this case, leave this to future work. "We use Flan because we observed strong decrease accuracy from continuing to train on MADLAD-400 data (even with the original tokenizer). The training data for most LLMs (including Mistral-7B) is not public, but it is plausible that this decrease stems from higher-quality data mixed in especially towards the end of training in e.g. Groeneveld al. 2024))

---

Table 3: Accuracy of Mistral-7B on XCOPA with language-speccfic tokenizers zero-shot transferred via FOCUS and our hypernetoork. The standard errors are between 2.1% and 2.3%.
ht
id
it
qu
ta
vi
Avg.
original
46.6
51.6
58.0
65.8
48.4
51.4
54.4
56.4
59.0
54.6
FOCUS
52.0
53.0
51.2
49.2
51.4
54.6
54.0
55.2
49.8
52.3
ours
53,4
57.2
60.0
65.6
50.0
57.2
55,8
57.4
57.2
57.1
Aaccuracy
47%
+6%
+2%
0%
+1%
+66
+1% %
+1%
-2%
+33
A Alength
-72%
-42%
-52%
-36%
-54%
-51%
-83%
-57%
-59%
-54%
Table 4: 5-shot accuracy of Mistral-7B on multilingual MMLU with the original tokenizer and language-specifie tokenizers zero-shot transferred via FOCUS and our hypernetwork.
original
FOCUS
Aaccuracy
Alength
German
51.6
26.2
43.7
-8%
-37%
Spanish
53.6
26.2
45.9
-8%
-32%
French
53.6
27.4
44.8
.9%
-30%
Italian
52.5
25.8
42.7
-10%
-36%
Russian
49.9
27.2
35.1
-15%
-47%
Models. To evaluate method, use Mistral-7B (Jiang et al. (2033 the main decoder-style language model and XIMMR (Conneau et al. 2020) as a representative of encoder-style models We
also experiment with the smaller TinyLlama- 1.1B 1. model (Zhang al. 2024 in Appendix H
Tokenizers. We transfer models to the GPT2 tokenizer (Radford et al., 2019) for evaluation on
natural language benchmarks and to the StarCoder tokenizer (Li et al. (023) for evaluation on code
benchmarks For multilingual evaluation, we train language-speeific monolingual tokenizers with a vocabulary size of 50k using SentencePiece (Kudo & Richardson, (018)) and evaluate transfer to these. We also verify that the hypernetwork is robust to the choice of vocabulary size in Appendix B
Hypernetwork training. We train the hypernetwork for 200k steps (10k of which are MIMICK-style warmup) with batch size of 128 and a sequence length of 128 (we find it sufficient to use short sequence lengths) [d For the multilingual decoder-style models, we start from the English Code checkpoint and forgo MIMICK-styLe warmup, keeping other hyperparameters unchanged. We use a RoBERTa-style architecture i.c. bidirectional attention and Post-LayerNorm Transformer layers (Liu cet al., 2019). but use a feedforward dimension of 2x the hidden dimension instead of 4x for the hypernetwork. See Appendix D for full list of hyperparameters.
Continued training details. To keep runtime comparable between training the model with hypernet- work and direct training (without hypernetwork), we run hypernttwrk inference only for subset of k = 16384 tokens in the continued training case. The subset consists of all tokens occurring in the batch, plus a uniform sample of those that do not occur. The language modeling loss is then only computed over this subset of tokens. We found in preliminary experiments that this causes only minor performance degradation. Furthermore, we use the zero-shot predicted embeddings as the target for the auxiliary loss instead of using the original embeddings. This stabilizes training. We train for 50k steps with a batch size of 32 and sequence length of 512, resulting in 'seeing' 819.2M tokens.
4.2 Zero-Shot and n-shot Results
Results for XLM-R are shown in Table We take task adapters trained for the original XLM- model on the English XNLI dataset via Poth et al. (2023) and substitute the tokenizer for our annuuag--specffe one. We compare our hypernetwork against a simple lexical baseline (copying the
"Athhough (decoder-style) LLMs are the centerpiece a large amount of current NLP research, encoder-style
LMs have wide-ranging applications in e.g. retrieval (Khattab & Zaharia 2020 and LLM distillation (Hsieh)
al.
2023) due to their lower computational cost.
chose these tokenizers due to their popularity and comparatively efficient encoding of target domain.
"Triiiin takes around one day for the XLM-R hypernetwork on TPU v3-8 and three days for the Mistral-7B
hypernetwor TPU v4-32 pod.
7

---

Table 5: Single model rating results on MT-Bench of transferring the GPT2 tokenizer using the hypernetwork trained for the base Mistral-7B model. We use as judge. orig. is the original fine-tuned model, base the model with the same tokenizer but embeddings substituted for the base models' embeddings. is the scaling factor for the weight differences in Task Arithmetic (llaarco et al. 2023).
to
original
0-shot
n-shot
Embeddings
orig.
base
FOCUS
ours
ours @ 800
À Score (1 to 10)
0.3
0.5
0.7
7.33
7.48
5.03
6.56
6.59
6.75
6.82
6.77
embeddings of overlapping tokens and initializing the rest randomly), FVT, OFA, and FOCUS (c.f. Section 2). We focus only on FOCUS in the following since it performs best among the baselines. Our hypernetwork consistently outperforms all baselines and preserves accuracy to I 1% on average, losing 3% in the worst case and improving by 1% in the best case, while sequences are on average 14% shorter for the language-specific tokenizers; inference is thus more than 16% faster II We show in Appendix E that these results are robust to the target vocabulary size.
Table 2 shows results on English and Code for Mistral-7B. We find that ZeTT is more challenging in the decoder case: FOCUS performs roughly random in the worst case (-23.2% on BoolQ) and is reduced to 0% pass@1 1 on HumanEval in Python. The hypernetwork goes a long way in closing this gap but still falls behind on some benchmarks. However, continuing train the hypernetwork with the target tokenizer closes the gap almost completely. In fact, continued training on 800M tokens with the StarCoder tokenizer performs better than continued training for the same amount of tokens with the original tokenizer, potentially because the StarCoder tokenizer is more well suited towards code; it results in approx. 10% less tokens on average. Also, notably, continued training with the original tokenizer slightly degrades performance on average; this may be due to a higher-quality data mix used for pretraining Mistral-7B, whereas we use public data sources (c.f. Section 4.1p
Results of the multilingual hypernetwork for Mistral-78 7B are shown in Table 3 and Table 4 On XCOPA, the hypemetwork average improves performance over the original model, while also more than halving sequence length. XCOPA performance is close to random in some languages (e-g. Southern Quechua (qu) and Estonian (et)), so we also evaluate multilingual MMLU. Here, although the hypernetwork clearly outperforms FOCUS (which performs close to random), there is still a substantial gap the original model; this could presumably be fixed via continued training.
4.3 Applying a Hypernetwork trained for a Base Model to Fine-Tuned Models
A large amount of the models used by practitioners are fine-tuned versions of base model e.g. via SFT or RLHF (Ouyang et al. 2022). We now attempt to answer the question: Given a hypernetwrk trained for a base model, can apply this hypernelwork to fine-tuned versions of the same model without any extra training? This would act as a multiplying factor for the hypernetwork's 's applicability. First, we observe that the embedding space of fine-tuned model is compatible with that of the base model: the embeddings of the fine-tuned Mistral-7B- 7h-isal-----sttuutt---l have an average cosine similarity of 98.6% to the corresponding embedding in the base model while the average cosine similarity of the mean embedding vector is 17.4% 13 Embedding compatibility also holds true for other models (Appendix H.. The predictions of hypernetwork trained for a base model can thus be used out-of- tbe-box with fine-tuned models. We verify that this is the case by evaluating Mistral-7B 7aairal----Inttuttt--- Instruct-vO.. transferred to the GPT2 tokenizer on the corrected version of MT- (Zheng et al. 2023). For n-shot transfer, since we train the full model we also need a way to transfer the eeneddddd parameters; we achieve this via Task Arithmetic (Ilharco et al., 2023). Results shown in Table The transferred fine-tuned model performs well, coming within approx. 0.5 score of the original model. Also, curiously, the fine-tuned model with the original tokenizer performs befter when using the embeddings of the (not fine-tuned) base model; this may be a prudent direction for future work.
" (1(1-44))=16%% plus additional speedup due to attention scaling quadratically with sequence length.
We refer to models purely pretrained on the Language Modeling task as base models.
Averaged across the input and the output emheddings.
Using the corrections from https: htt.a//c.w.....aaaeeeaaaoaaaaaaa Benchhans
8

---

Table 6: NLI performance on Farsi (FarsTail; Amirkhami et al. 2023). Dutch (SICK-NL; Wijnholds &
Moortgat, 2021), Aymara and Guarani NAme Ebrahimi
(Ebrahhi et al. 2022). We measure zero-shot
transfer from model trained on English XNLI (c.f. Table 1), except for Sick-NL where we train an adapter on SICK (Marelli al. 2014) since the XNLI adapter underperforms.
Unseen by Hypernet Completely Unseen
Farsi
Dutch
Aymara
Guarani
original
72.4
76.6
40.0
42.4
Lexical
60.5
72.7
38.5
41.7
FVT
65.5
74.8
38.4
39.0
FOCUS
65.3
74.8
37.7
40.7
ours
66.4
77.8
42.9
42.2
Aaccuracy
-6%
+1%
+3%
0%
Alength
-12%
-19%
-36%
-39%
Table 7: Performance of Mistral-7B transferred to the GPT2 tokenizer on English benchmarks. (c.f. Table 2). as well as transferred to a tokenizer containing all words in the evaluation datasets; this converts Mistral-7B to a word-level language model on the evaluation corpora.
PIQA
HS
ARC
BoolQ
MMLU
Avg.
original
80.7
81.0
79.5
83.6
59.6
76.9
FOCUS
69.2
63.8
45.7
60.4
38.8
55.6
GPT2 Tokenizer
ours
79.7
77.5
73.0
81.9
53.0
73.0
Alength
-7.8%
-5.6%
-6.1%
-13.1%
-9.9%
-8.5%
FOCUS
66.8
58.8
51.3
62.6
35.2
54.9
Word Tokenizer
78.9
74.9
73.9
80.9
49.4
71.6
Alength
14.6%
-10.1%
-14.9%
-203%%
-16.8%
-15.3%
5
Discussion
we need a procedure to convert tokenizers to the byte level to ensure that token decomposition is always possible. This is trivial in most cases; the missing bytes just need to be added to the vocabulary. BPE is an exception: here, we need to change the units on which merges merges to assemble the characters used by the tokenizer from their constituent bytes to the beginning of the merge table. This preserves the tokenization in more than 99% of cases (Appendix M
Converting tokenizers byte-level. As per Section 3.2
defined from characters to bytes. We achieve this by adding
Converting tokenizers to UnigramLM, We also introduce procedure to convert arbitrary tokenizers tokenizers using Unigraml .M as the tokenization function. We refer to this process as unigramifying (details in Appendix A). An important assumption of the hypernetwork training is that by using the UnigramLM parametrization with scores distributed as Gaussians we can cover a sufficiently diverse distribution of tokenizers for the hypernttwrk to generalize to c.g. BPE tokenizers. Unigramifying allows us to check if, in principle, this possible. Luckily, we find that is: unigramifying results in minimal performance degradation when substituting the original tokenizer with the corresponding UnigramL.M tokenizer (Appendix J.. Although this does not guarantee that our distribution of tokenizers sufficiently diverse, empirical results suggest it is (cf. Section 4.2).
We believe our conversion methods UnigramL and to byte-level will simplify further research into tokenizer transfer, showing that the wildly heterogeneous landscape of tokenizers can be well approximated via byte-level UnigramLM tokenizers.
What is the effect of amortizing over the tokenization function? As described earlier in Section 3 we 'amortize' over the tokenization function, that is, the tokenization function is not an input to our hypernetwork. We find that the predicted amortized embeddings are robust to the choice of tokenization function. For example, the set of embeddings predicted for the GPT2 vocabulary has low bith-per- for both the original GPT2 tokenization function and a different UnigramI.M tokenization function with scores based on token frequencies (Appendix T). This is not the case

---

for the original GPT2 embeddings: while they (as expected) perform well with the original GPT2 tokenizer, there is significant performance degradation when switching to the frequency- based UnigramL.M M tokenization function. This calls into question prior work copying the embeddings of overlapping tokens for transfer across tokenizers (Dobler de Meloj 2023 Gee et al., 2022, among others), indicating that even if there is an exactly overlapping token in the original tokenizer, it is not necessarily the optimal initialization of the corresponding token in the new tokenizer.
Although we amortize over most of the aspects of the tokenization function, in practice, tokenization functions rely on considerable amount of engineering, so it is not possible to amortize over everything, we discuss remaining assumptions in Appendix[[ [
Analyzing computational overhead. We estimate the FLOPPs per token of multiple hypernetworks in Appendix K Given batch size and sequence length for the main model, and using the hypernetwork to compose k token sequences of length t, the FLOPs per batch will be n x a x FLOPs
token Jhypemet. Taking Mistral-7B as an example with n s = 128, k 32768 and 6 = 7 the FLOPs per batch will be 252T + 30T i.e. 12% overhead from applying the hypernet. Notably, we observed that a hypemnetwrk size of three layers is sufficient, regardless of the main model. so the relative overhead decreases with increased amounts of layers in the main model.
FLOPPs)
loken main + k f x
Generalization to unseen tokens. Although our primary goal is generalization to unseen tokenizers (i..,, tuples (V. T),, the question of how well our hypernetwork can generalize unseen tokens (elements of 1\) presents itself. To answer this question. we test the XLM-R and Mistral-7B hy- pernetworks on out-of-distribtiion vocabularies. Specifically, we test the XLM-R hypernetwork on Farsi and Dutch (which are unseen by the hypernet, but seen by the base model) as well as Aymara and Guarani, which are unseen by both. J the hypernet performs well in this case, even gaining in performance over the model with original embeddings in completely unseen languages. In this setup, up to 40% of the used tokens in the target vocabularies have never been seen during hypernetwork training (we analyze this overlap in detail in Appendix G). The reason for the performance increase from the hypernetwork on unseen languages may be that, under the original tokenization, the embeddings of many tokens occuring in unseen languages are undertrained (c.f. Land & Bartolo 2024), while the embeddings produced by the hypernetwork do not suffer from this issue; future work could investigate this in more detail. For Mistral-7B, we instead transfer to an -ut-of-distribution word-level tokenizer by creating a tokenizer which contains all words which occur in any evaluation corpus (approx. 100k in total). 3.3k words are completely unseen and 13.5k words have been seen in less than 0.1 % of training steps. Still, performance only deteriorates by a small amount and the improvement over FOCUS persists shown in Table 7
Conclusion
We have established Zero-Shot Tokenizer Transfer (ZeTT), the difficult problem of transferring language models to a new tokenizer without any training. We have found that prior heuristics for embedding initialization provide first baseline for ZeTT, but fall short in many cases. To establish much stronger baseline, we introduced a hypernetwork- based approach that closes the gap to a large extent, and can be further improved via continued training on a few (<IB) tokens. Due to preserving the embedding space of the original model, ZeTT can be applied to e.g. reusing adapters trained for the original model with different tokenizer, and transferring fine-tuned models new tokenizer using a hypernetwork trained for the base model. In aggregate. this work is a substantial step towards detaching language models from their tokenizer, increasing their flexibility and reusability.
7 Limitations
The key limitation of our approach is the requirement train a hypernetwork for every base model. Although the hypernetwork only needs to be trained once, doing so is computationally intensive and may not be feasible for many LLM practitioners. Instead, it may be a task LLM providers are better positioned to undertake. Other limitations are the remaining assumptions on the tokenization function (Appendix[i []. and not taking the tokenization function into account (Appendix D). although these limitations do not appear to have substantial impact in practice. Finally, we have limited our scope to experiments on text-only models, but Zero-Shot Tokenizer Transfer could also be beneficial for multimodal models, such models 'perceiving' images speech; we leave this to future work.
10

---

Acknowledgments
This work has been supported by a Royal Society University Research Fellowship 'Inclusive and Sustainable Language Technology for Truly Multilingual World' (no 221137; 2022-) awarded to Ivan Vulić. Research supported with Cloud TPUs from Google's s TPU Research Cloud (TRC). We thank Markus Frohmann, Marcell Fekete and Piotr Nawrot for helpful feedback on draft of this paper, and Arduin Findeis for many valuable discussions during the entirety of this project.
References
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9904 9923. Singapore, December 2023. Association for Computational Linguisties. doi: 18.18653/ 1/t0://112222mmmmm main.614. URL/attps://accaanhhoooogg
grgg/023... emnlp- main. 614
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo per- mutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL
htttps://openreviev..u net wortforumm/iit wowoooooooodoooioo 1d000000m000m111
Hossein Amirkhani, Mohammad AzariJafari, Soroush Faridan- Jaahomm, Zeinab Kouhkan, Zohreh Pourjafari, and Azadeh Amirak. Farstail: persian natural language inference dataset. Soft Computing. 2023. doi: 10. 10.10077/005000002-0999999
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross- lingual transferability of monolingual representations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623-4637, Online, July 2020. Association for Computational Linguistics. doi: 10. http://ww1222..ammmaa..a.m
URI https
//aclanthology.org owwpoolooyooogg////.. wrg/ sel-main 421
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth Fourth AAAI Conference on Artificial Intelligence, 2020.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace Hc, Connor Leaby, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-208B 220: An open-source autoregressive language model. Angela Fan, Suzana Ilie, Thomas Wolf, and Matthias Gallé (eds.), Proceedings of BigSeience Episode #5 Workshop on Challenges & Perspectives in Creating Large Longuage Models, pp. 95-136, virtual+Dublin, May 2022. Association for Computational Linguistics. doi: 1111855311122222iiicct URL httphhttts//aaaaatoollll
1.9.
2022 bigscience-1.9
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http::
com/ google/aa
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifcoluwa Adelani, Pontus Stenetorp. Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. In shirth- Conference on Neural Information Processing Systems, 2023. URL https: net/forum? id= id=jvEb0BxdBX
Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. Rethinking embedding coupling in trained language models. In International Conference on Learning Representations, 2021. URL. httpp:://pperreiieeeee.e
net www.oooruunddpoppo
Ntgpl
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yesino questions. In Jill Burstein. Christy Doran, and Thamar Solorio (cds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 2924-2936, Minncapolis. Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1/113//11/1111111/ URL htttpp:://acaatthll.
wtt///1-/////1/ Nt1/ 1300
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. Canine: Pre- Pre-training an efficient tokenization free encoder for language representation. Transactions of the Association for Computational Linguistics, 10: 73-91, 2022. doi: 10.1162/tacl_5 a 000448. WtRLLtttp://aclattaoo
2rgg/022. t.cl-1.5
11

---

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal. Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. voXiv:/803.05557V/ 2018.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Reygelzimer, d' Alchh-BBue, Fox, and R. Garnett (eds.) Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL /tttps::/ Paper pdf
neurips, cc/ co/paper. paper
files/paper/200 httpp/pppe//////1eeepp//// http://////.1///00100/22222/////
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun 'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2475 -485, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 18.136/31//D/ 1269. URL httplttt///ccaatttoooo
org/ -19699991919
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Oit, Luke Zettlemoyer and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale, In Dan Jurafsky. Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/1120220.co wwt.:0000..l--mii..... --almmmai..7 URL attts::/accaanthol.gg org. 20200
747
Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. Getting the most out of your tokenizer for pre- pre-training and domain adaptation, 2024.
Wietse de Vries and Malvina Nissim. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021, pp. 836-846, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/w1/2221..mmm findiigss-i.i........ URL
aic.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1171-1886, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10.18653/v1/119/ 1423. URL
pettps//aaclanthologgg org /N19911911111 1423
Konstantin Dobler and Gerard de Melo. FOCUS: Effective embedding initialization for monolingual special- ization of multilingual models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13440-13454, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653//1/2223.coom emnlp- main.829. URL httpp://aacaanthooogy wrw.:llloooooo.o////0 2023. emnlp- emnlp-main.......mmmmmmmmmmmmmam mwwn.ainn... 829
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios. Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer. Rolando Coto-Solano, Thang Vu, and Katharina Kann. AmericasNLI: Evaluating zcro-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (cds.). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6279 6299, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 18.18663/v 1/25533/1122222l-lloooo
URL
org/
conglllogg..... 435
Leonidas Gee, Andrea Zugarini. Leonardo Rigutini. and Paolo Torroni. Fast vocabulary transfer for language model compression. In Yunyao Li and Angeliki Lazaridou (cds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 409-416, Abu Dhabi, UAB, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/20222 1t6555/1/2022.mmlp--nndssss hattss://aclanthologyy org 2022. indlptindustzy...
URL
Siavash Golkar, Mariel Pettee, Michael Eickenberg. Alberto Bietti, Miles Cranmer. Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blaneard, Tiberiu Tesilcanu, Kyunghyun Cho, and Shirley Ho. xval: A continuous number encoding for large language models. In NeurIPS 2023 Al for Science Workshop, 2023, URL hhttpp::/oppeee eview. net/fff frrmm KHDdHtttotH
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill,
12

---

Jacob Morrison, Niklas Muennighoff, Aakanksha Naik. Crystal Nam, Matthew E. Peters. Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo. Luca Soldaini, Noah A. Smith, and Hannanch Hajishirzi. Olmo: Accelerating the seience of language models. Preprint, 2024.
David Ha. Andrew M. Dai, and Quoc V. Le. Hypemetworks. In International Conference on Learning Representations, 2017. URL Btt.httts://oppeevviee http:///www.....eeooooooorrttaee
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meering of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 385-393, Dublin, Ireland, May 2022. Association for Computational httphtttt///ccataaaaee
Linguistics. doi: 10. 18536653 acl22.acl-shore
org/ 2022.002.2222 acl- short
43
Cheng- Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujüi, Alex Ratner, Ranjay Krishna. Chen Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Amna Rogers, Jordan Boyd-Graber, Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 8003-8017, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/11/2033 htt.:///////22.....nng//--////// http://accaanhoooo..orroo wwt..//.oorr/22222..ii.n..g.eee. findings-acl....
URL
IBM ILOG. V22.1: User's 's manual for cplex. 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL http::t/opeeeeeeiieee net net/EorummiddSttttaae
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 083611, 2020.
Omar Khattab and Matci Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20. pp. 39-48, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978 0781450380164.
doi: 10.1 411144//97771.3400101 URL WtRI:ttps:://at ///g://r////o
10. 1145/3397271 3401075
Taku Kudo. Subword regularization: Improving neural network translation models with multiple sub- word candidates. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 66 66-75, Melbourne. Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/118- 1007. URL httpas///ellarthoooogg.o////
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.). Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguisties. doi: 10.18653/v1/D18. 2012. http:///ccaamtboooooooo////..1-
Sncha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette- Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat, Madlad-400 A multilingual and document-level large audited dataset, 2023.
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. Yansong Feng and Els Lefever (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 318-327, Singapore, December 2023, Association for Computational Linguistics. doi: 10.186533/1/2023..cm .mnlppdd demo..dm....... URL https //aclanthology.co .r..or////00 2023. emnlp- depp-dem... 28
13

---

Sander Land and Max Bartolo. Fishing for magikarp: Automatically detecting under-trained tokens in large language models, 2024.
Raymond Li. Loubna Ben allal, Yangtian Zi. Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yuc Zhuo, Thomas Wang. Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey. Zhihan Zhang. Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni. Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao. Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavitt Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes. Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research. 2023. ISSN 2835-8856. Reproducibility Certification.
URL hetLittts//opezeeviiee
Jindřich Libovický, Helmut Schmid, and Alexander Fraser. Why don't people use character-level machine translation? In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACI. 2022, pp. 2470-2485, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/w1/0022..2 find.iis.iiinnnn-c...... URL httttttt/accaaaataattt 22022. findings--an 194
org/
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru. Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab. Veselin Stoy- anov, and Xian Li. Few-shot learning with multilingual generative language models. In Yoav Gold- berg. Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. ber 2022, Association for Computational Linguisties. doi: 10. 10653v63/v 1ttl////2222 2200..emnlp-main...m htttp::/aaclanthologg. org/ 2022. mammp-main.ee 616
Abu Dhabi, United Arab Emirates, Decem-
URL
Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schütze. Ofa: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. arXiv preprint arXiv:2311.08844 2023.
Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Shayne Longpre. Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc Le, Barret Zoph. Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688, 2023.
Nlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL htt.l/tttee//oopprrreeeeeeeeee/e http://wwww............oodddddaa
Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe. Mini-model adaptation: Ffficiently extending pretrained models new languages via aligned shallow training. In Anna Rogers, Jordan Boyd-Grabee and Naoaki Okazaki (eds.). Findings of the Association for Computational Linguisticx: ACL 2023, pp. 5474-5490. Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 1/26533/1/222..nnniiii org/2023. findings- acl. 338
338. URL htttp:://aclatthlllle
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerek, Hrafn Loftsson, Bente Macgaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (cds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'I4) 14). pp. 216-223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http:// // www. www.wlree-coo www.eeccc........ wtt.://oceediig//eeeeet////////a http://rrrcooeeaaaaa33/3/PP/./ P63__Ppprrrp
pdf
Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp, 2021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013.
14

---

Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3992-4006, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18833/v1/022.. naacl- 293. URL http::tpps///e htattps://acllanhholggg..
org/2022.22222 www.laacllaaoomm main. 293
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng. Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint 4rXiv:2308.07124 2023.
Piotr Nawrot. Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. Efficient transformers with dynamic token pooling. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the s1 Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6403-6417, Toronto, Canada, July 2023. Association for Computational L Linguistics. doi: 10. (0.18853/v1/2023..l aac.o//......
URL
org 2023. we000.aclll..................... acl- lon..on.....
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL attpa://openreviier net (tt.:/orrrmm.ddTTKA....
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu, Patrick LeGresley, Denys Fridman, Jared Casper, Ashwath Aithal, Oleksii Kuchaiev, Mohammad Shocybi, Jonathan Cohen, and Bryan Catanzaro. Nemotron 15b technical report, 2024.
Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers intro- duce unfairness between languages. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 36963-36990. Curran Associates, Inc., 2023. URL 7
wttp://ieeeaappeee23//////////// httpp//pppr/22///1eee... neuripa. httppacccpppppppppaeeaee
Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. Mimicking word embeddings using subword rnns. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 102- 112, 2017.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Bonnic Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2362-2376, Online, November 2020. Association for Computational Linguistics. doi: 18653/ h/20000eemlp--mmaaieee main.18. bttps:
org/ org/2020. emnlp waalppmmiin....mm 185
Clifton Poth, Hannah Sterz, Indrancil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: A unified library for parameter-efficient and modular transfer learning. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 149-160, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10118655w1/02233.mmmddemmmm 13. URI. attps://aclantholooe .rg/ 2023. emnlp-deme 13
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annuad Meeting of the Association for Computational Linguistics and the Ithh International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 31 18-3135, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2021 acl-llong 243. 243. URL htttps://aclanthiit
org./og///// accl long. 243
Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. 8(4): 2018.
15

---

Timo Schick and Hinrich Schütze. Attentive mimicking; Better word embeddings by attending to informative contexts. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 489 189-494, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 20116633/1/NT/ 1048, URL
org/N19- 1048
Timo Schick and Hinrich Schütze. BERTRAM: Improved word embeddings have big impact on contextualized model performance. In Dan Jurafsky, Joyce Chai. Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3996-4000 Online, July 2020. Association for Computational Linguisties. doi: 10. N0.18853/v1/2000 www.0..cll-mmi....... 368 URL https 7 /aaclanthology... org/2020 acl-main 368
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.). Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1715-1725 Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10. 118635/11/1166 1162. URL. htttp:://aclantaiitt:
org/P16-1162 1162
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke 7ettlemoyer, Noah A. Smith, Hannanch Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, arXiv preprint, 2024.
Junyi Sun. Jieba chinese word segmentation tool. 2012.
Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung. Dara Bahri, Zhen Qin, Simon Baumgartner. Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient- based subword tokenization. In International Conference on Learning Representations, 2022. URL wiw.orrrmnddddBnnnm
httts://openreviier
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu. Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa. Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lcc, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu. Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ke Tran. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07300 2020.
Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter. Greed is all you need: An evaluation of tokenizer inference methods, 2024.
Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model, 2024.
Gijs Wijnholds and Michael Moorigat. SICK-NI: A dataset for Dutch natural language inference. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1474-1479, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022..aa0 mainm 126. URL https //aclanthology.orge.
eacl main. 126
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-22 23998, PMLR, 17-23 Jul 2022. URL https: //proe//ggn.cccinns..... mlr press/ aress/v162// hwwp1/////ootttaa222222 html
16

---

Linting Xue. Aditya Barua. Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByTS: Towards token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022. doi: 10. 162/tacl a 00461. URL
www.oog...r..g/o/o. tacl- 1.17
Prateek Yadav, Derek Tam. Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. httphttttpppeereeeeeeeee wttpe/eweeeeeeerrrr/ddtdtaaaaaaa
Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. In Thirty Conference on Neural Information Processing Systems, 2023. URL http::tt/////oeeeeeeew
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Márquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653/v1/P19- 1472. URL https Lal.aaatooooog.oo///-------
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging Ilm-asa-judge with mt-hench and chathot arena, 2023.
17

---

A Unigramifying: Approximating Arbitrary Tokenizers via UnigramLM
We introduce procedure to convert arbitrary tokenizers to UnigramL.M an optimal (but lossy) way which we refer to as unigramifying. Given text x and the sequence of tokens T(x), for the UnigramLM tokenizer T' to be equivalent to T, it is necessary that T fulfills ΣieT(a) log () > EiEc
(t) for all C' in Cz \ (T()))
Thus, given a corpus of texts X we can formulate a loss
(T(X Σ E ZEX (EC))((E)))
max 0, Σ log (1g
log (T) tET(x)
Σ
IEC
which zero and only if the condition above is satisfied for all texts X. This objective is standard Linear Programming (LP) form and solved via
piecewise linear, so it can be converted an LP solver. In practice, we use the CPLEX v22. 1 (IBM ILOG (022) solver. Since applying the procedure corpus directly would be costly, first pre-tokenize the training corpus, then count the pretokens, and choose the top n 1000000 pretokens as the set X.
GPT2
4.8
GPT2 (untled)
4.6
(untied, no aux, loss)
4.0
3.8
10000
15000
20000
25000
30000 Step
35000
40000
45000
50000
Figure 3: Language modeling loss of GPT2, and GPT2 with untied weight embeddings with and without the auxiliary loss across the first 50k training steps, excluding MIMICK-style warmup.
B Stabilization Effect of the Auxiliary Loss
We found in preliminary experiments that the auxiliary loss is necessary, especially for models that do share embedding parameters between the input and the output (models with untied embeddings). To validate this hypothesis, we conducted an experiment where we manually untied the embeddings of GPT2 i.e. used a separate hypernetwork prediction head for the input and the output embeddings. Although everything else is kept the same, the untied GPT2 model diverges without the auxiliary loss, whereas the original GPT2 trains as expected, even without an auxiliary loss (Figure 3}.
C Non-Amortizing Hypernetworks
We experimented with hypernetworks taking the tokenization function into account by adding sparse inter-token attention blocks between the self-attention and the FFN in every hypernetwork layer. Sparse inter-token attention consists of two attention blocks. The first attention block attends from fixed amount of learnable inter-token embeddings (e.g. 16, each a vector of size dmwste) to the ith token representation of every token sequence passed the hypernetwork. The second block attends from the ith token representation to the inter-token embeddings. This way, we factorize the attention to e.g. one 16 attention and one k 16 attention, instead of the standard
l self-attention
19Thi is not sufficient for equivalence since order is ignored e.g. T(x) = {ab, b) and (()) = {a, 6, ab) fultill the criterion but not equivalent.
18

---

Table 8: Performance of the hypernetwork in bils-per-byt with and without inter-token attention. Sampled Tokenizers are tokenizers sampled during the training loop (c.f. Algorithm en is an English UnigramLM tokenizer. The respective vocabulary sizes are shown in brackets.
Sampled Tokenizers (32k)
GPT-NeoX (50k)
(30k)
ours
1.157
0.902
1.054
ours (+ inter-token attention)
1.118
0.904
1.103
which would be infeasibly slow for typical vocabulary sizes. We only add inter-token attention for the first token in every sequence. This improves performance on the sampled tokenizers, but does not improve performance on 'real-world' tokenizers (Table [): investigating this mismatch is direction for future work.
D Additional Hyperparameters
Hyperparameters for hypernetwork training are shown in Table For continued training, we use the same optimizer. but a sequence length 512, batch size of 32, training for 50k steps and constant learning rate chosen among the set {1e-6, 3e-6, 6.-6, 1e-5, 3e-5] to maximize performance. The chosen learning rate 1e-6 for the runs keeping the original tokenizer (original@ 800M), 6e-6 for continued training starting from FOCUS (FOCUS@ 800M) and 30-6 6 for continued training with the hypernetwork (ours @ 800M).
Table 9: Hypernetwork hyperparameters.
Optimizer (81,8) (2)
AdamW (Loshchilov & Hutter 2019
(0.9,
0.95
weight decay
0.01
Max. global gradient norm
0.1
Sequence length
128
Batch size
128
Steps
200000
of which MIMICK-style warmup steps
10000
MIMICK-style warmup learning rate schedule Main learning rate schedule
warmup to 3-e4
linear warmup to 6e-5 until 10k, then cosine decay to 6e-6
Tokenizer sampling
Vocabulary size
32768
Distribution of noise level & Batch size m
f = In(10 5) "),0
4
2048
Auxiliary loss weight
0.5
Hypereework num. layers
3
max. sequence length
7 (English Code) or (multilingual)
hidden dimension
Sandel
FFN dimension
2dhaoeel
num. attention heads
min min(dmocal 664, 32)
E
Sensitivity to Tokenizer Size
Since the tokenizers experiment with have similar vocabulary sizes (50k for the language-specific tokenizers and for GPT2, 49k for StarCoder tokenizer) we conduct an additional experiment to quantify the sensitivity of the performance of our hypernetwork to the size of the target tokenizer. We find that although there is slight performance degradation when increasing the size of the new tokenizers' vocabulary, the hypemetwork is fairly robust to vocabulary size (Figure 4).
F
Reliance on Vocabulary Overlap
Intuitively, transfer easier the more the target has in common with the source. One way to measure commonality between the original (source) and the target tokenizer is the fraction of tokens of the
19

---

daccuracy
ours
30000
50000
100000
Vocabulary Size
Figure 4: Difference in accuracy to the original XLM-R model on XNLI of our method and FOCUS across vocabularies with size 30k, 50k, and 100k of the new tokenizer.
AucOurity
X
(2-0220)
FOCUS 1P 581
FOCUS (0000)
-10
0:00
0.39
Figure 5: Correlation of the difference in accuracy to the original XLM-R model with Unigram overlap probability p(overlap) (left) and vocabulary overlap (right).
target vocabulary which also exist in the source vocabulary (vocabulary overlap). Performance correlates with vocabulary overlap, but it correlates more strongly with the probability for tokens to overlap: that is, when randomly sampling some token from corpus tokenized with Tú, the probability that this token also exists in the vocabulary of T., We refer to this metric as p(overlap). p(overlap) has higher correlation with the performance of FOCUS, indicating that our hypernetork depends less on overlap (Figure 3).
Table 10: Performance of TinyL 1.1B after zero-shot and n-shot tokenizer transfer (training on 800M tokens), compare Table
Natural Language
Code (pass @ 1) (+ StarCoder Tok.)
(+ GPT2 Tok.)
#shots Method
HumanEvalPack
PIQA
HS
ARC
BoalQ MMLU
Avg.
Avg.
go
Py
cpp
java
original
73.1
59.1
55.2
57.2
25.5
54.0
7.3
6.7
7.3
8.5
7.9
7.5
original @ 800M
73.2
59.5
63.3
65.1
26.3
57.5
9.8
7.3
9.1
8.5
10.4
9.0
FOCUS
60.8
42.1
39.6
56.9
22.9
44.7
4.9
0.6
0.0
3.0
7.9
3.3
0-shot
70.5
55.6
51.4
62.9
23.7
52.8
4.3
5.5
4.3
7.3
3.7
5.0
nt-shot FOCUS@80N0
67.7
52.8
52.7
66.1
25.3
52.9
6.1
6.1
10.4
8.5
8.5
7.9
ours @ 800M
71.4
57.8
59.7
66.1
26.6
56.3
9.1
6.1
11.6
11.0
7.3
9.0
20

---

XUM- Traans
XLM- Tronn (Lnoeen:
M Mittral-70 Thkam
12.
12
20-2
28
20
OFF2
Taur PECTUTT
PUICE Docuring Target
Occuring Tagest
Occaring bagee (irison larguagec)
38.
10"
28.
20.
Wird
Than Perceet
Taker
Figure 6: Analyzing how often the hypernetwork sees the tokens of different target tokenizers during training. Note logarithmic y-scale. We analyze the occurrence for tokens in the target vocabulary (top) and for tokens which occur at least once in the evaluation data (bottom) across target tokenizers in seen languages for XLM-R (left), unseen XLMMR languages (middle) and English Mistral-7B tokenizers (right). The bottom row is more informative w.r.t. how well the hypernetwork generalizes to unseen tokens since tokens which do not occur do not substantially impact evaluation.
Table 11: Single model rating results on MT-Bench of transferring Tiny BB-Chat-v to the GPT2 tokenizer, compare Table II
original
O-shot
n-shot
Embeddings
orig.
base
FOCUS
ours
ours @ 800
0.0
0.3
0.5
0.7
Score to 10)
5.5
5.7
2.7
4.0
4.29
4.63
4.8
4.43
G Reliance on Overlap between Hypernet Training Tokens and Target Tokens
We analyze how often the hypernetwork sees the tokens in the vocabulary of different target tokenizers across multiple settings in Figure 6 We differentiate between tokens which occur in the evaluation data, and tokens which do not; this important since the embeddings of tokens which do occur in the evaluation data will not substantially impact performance. Notably, for XLM-R, >35% of occurring tokens in Greek, Bulgarian and Russian are unseen by the hypernet, even though the hypernet is trained on these languages. This is likely due to the Latin scripts. The hypernet still performs well these languages with an average 2% performance decrease 17% sequence length reduction on XNLI. In total, the HN has seen approx. 200M different tokens during training.
H Additional LLM Results
Zero-shot and n-shot results for TinyLlama-1.18 1.11 shown in Table and and MT-Bench results of transferring TinyLlama-1.18 1BB-ha-tvit in Table [T We observe the same patterns as on Mistral-7B.
I Assumptions on the Tokenization Function
In practice, besides the tokenization algorithm itself (e.g. BPE, UnigramL tokenization functions also contain other steps, in particular pretokenizing text into smaller chunks (usually words) on which to apply the tokenization function (Mielke et al. 2021 In our experiments, assume fixed
21

---

Table 12: Probability of pretokens sampled from the English MADLAD-400 data to be tokenized equivalently to the original tokenization when converting the tokenizer to byte-level (To Byte-Level) or to UnigramLM (Unigramify). Also shown is the LMs bits-per-character when applying the original vs, the corresponding UnigramLM tokenizer. Bits-per- teec-perr can not be measured for conversion to byte-level since extra tokens are added in this process (which there are no embeddings for).
BERT
Mistral-7B BPE
TinyLlama-1.1B
GPT2
Kind
WordPiece
BPE
BBPE
p(preserved) bits per char
100% n/a
100%
100%
100%
Original
0.675
0.747
0.930
p(preserved) Extra Tokens
99.6%
99.9%
99.9%
100%
To Byte-Level
162
522
362
p(preserved) hits per char
99.4%
99.8%
99.8%
99.7%
Unigramify
n/a
0.678
0.750
0.932
Table 13: Bits-per-character of GPT2 with the original tokenizer and the tokenization function being original (left), unigramified (middle) and UnigramLM with scores set to the substring frequency of the tokens (right). We compare the original embeddings with embeddings predicted from our hypernetwork, with without Gaussian noise in the sampling process.
Tokenizer (V, T) (GPT2, GPT2) (GPT2, unigramify(GPT2))
Model Embeddings
(GPT2, UnigramL.M)
original
0.930
0.932
1.005
GPT2 ours (no noise)
ours
0.919
0.920
0.964
0,925
0,926
0.978
pretokenization given by a regular expression based on the regular expression used by GPT2 (Radford et al., (2019), adjusted to not over-segment text in languages using characters in the Unicode Mark category within words (c.g. Hindi and Tamil). We also add a prefix space (i.c., a whitespace at the start of the text to tokenize) if and only if the original tokenizer also uses prefix space. Finally, always add whitespace characters covering sequences of consecutive whitespaces up to 16 characters long similar to [Black al. (2022) ensure code is tokenized efficiently. These light assumptions mostly preserve the generality of our method but could be further relaxed in future work.
J Tokenization Function Amortization and Unigramifying Results
Results measuring the success of unigramifying tokenizers shown in Table 12 the success of amortizing over the tokenization function are shown in Table 13
Results measuring
Table 14:
Parameter count and FLOPs estimates for our hypernetwork (and the corresponding main model) in different setups. The relatively lower computational cost compared parameter count is mainly due to forgoing de-embedding which contributes significantly to FLOPs (Kaplan et al.) 2020).
Model
Hypernet
#params
FLOPs/ token
#params FLOPs / token
GPT2
124M
253M
21M (16%)
4.5M (1.8%)
TinyLlama-I.I
1.1B
2.1G
170M (15%)
33. 33.1M (1.6%) 13 (0.9%)
Mistral-7B
7.2G
15.4G
678M (9%)
K Analyzing FLOPs of the hypernetwork
Estimated FLOPs per token for the hypernet and the corresponding main model are shown in Table 14 We estimate FLOPs on the basis of XLA-compiled instructions using Jax (Bradbury al. 2018
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the abstract and Section D match the results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix D and important hyperparameters in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:
[No]
Code is not submitted alongside this paper but will be provided upon publica- tion.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipps.cc/pulli///uuu guwde//ddee///
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Guidelines:
Detailed hyperparameters are reported in Appendix D
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Quantifying statistical significance would multiply the computational costs, and we perceive it not to be necessary given the margins of improvement over the baselines in our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Hypernetwork training time is reported in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We adhere to all applicable points of the Ethics Guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss implications on fairness across languages in Section |
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any standalone models; released hypernetworks bound to the base model they were trained for, including being bound to the safeguards put on the base model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, were applicable, throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We do not release any new assets besides the trained hypernetworks; the documentation for these will be available upon release.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There are no human subjects involved in the experiments.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There are no human subjects involved in the experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "ViCAuClH4v", "reviewer": "Reviewer_mdD9", "summary": "This paper presents a way to perform \"zero-shot\" construction of embeddings for a new target tokenizer. It proposes a hypernetwork based approach that learns to take in the embeddings of tokens generated by tokenizing the target token with the original tokenizer and produce the embedding of this token. The authors perform analysis on code and multilingual domains, and their method shows promise with respect to FOCUS, a recent embedding transfer method. Overall, the idea is novel, innovative, and seems promising. However, based on my understanding, I have significant concerns with the evaluation settings and some comparisons. I am willing to reconsider my assessment if my concerns on evaluations are adequately addressed.", "questions": "* I cannot seem to find the value that was set for the hyperparameter max token length ‘l’. For a sufficiently large ‘l’ I would assume there would be a significant overlap of the vocabulary produced by UnigramLM during training and the target tokenizer during evaluation, especially when the domain/language is the same.\n* When the target vocabulary size is increased, does the hypernetwork remain the same? If yes, I could see how some tokens might be zero shot transferred as the vocab size grows. However, without comparing the distributions as mentioned earlier, it’s hard to say what exactly the overlap is.\n* How does ZETT compare to FOCUS when extra training is done in the more challenging multilingual setup? Would ZETT be a more attractive choice in this case compared to FOCUS which does not require any prior hypernetwork training?", "limitations": "I believe the weaknesses outlined above need to be addressed, which are not discussed. There is no separate limitations section in the paper as recommended in the checklist guidelines.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The approach is interesting, and the idea of using a hypernetwork to compute new embeddings is innovative.\n* I can see this approach being useful as the tokenizers/vocabulary grows, since the number of parameters will stay constant. However I have some concerns on the ability to generalize (see weaknesses).", "weaknesses": "* I do not think ZETT is truly ‘zero-shot’. While it is important to make a distinction between vocabulary V and tokenization function T, it is likely that a large portion of embeddings are agnostic of T and more reliant on V. For example, I would not expect the embedding of the token ‘car’  to be much different when an LM is trained on two different tokenization functions.\n* Therefore I think it is important to analyze the overlap of the distribution of the tokens. Appendix F analyzes the overlap with respect to the original vocabulary. But it is also important to study the overlap (both vocabulary and segmentations) between what UnigramLM generates during training versus the ‘target’ tokenizer during evaluation to really contextualize whether this approach is useful. \n* The above wouldn’t be as big of a problem if the hypernetwork data did not contain the same languages/domains as that it was trained on. But from what I see all the evaluated languages are seen during hypernetwork training. I assume it will be something similar for code too.\n* I think what would make more sense is to really evaluate on an ‘unseen’ language that uses the same script. Here one can expect a more pronounced distribution shift of both vocabulary and tokenization functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, and for recognizing the strengths of our approach.\n\n__Distinction between ZeTT and generalization to unseen tokens.__ The task we address is Zero-Shot Tokenizer Transfer since the hypernetwork and the base model have not seen the target tokenizer, that is, the specific combination of (V, T), which we want to transfer to. It is expected (and even desired) that there is vocabulary overlap between the tokenizers the hypernet (HN) has seen during training and the target tokenizer; this means that the HN should be able to generate good embeddings for these tokens. Our principal objective is (zero-shot and n-shot) generalization to unseen vocabularies and tokenization functions, not generalization to unseen tokens. Table 11 in Appendix I shows evidence toward successful generalization of this kind: the HN-predicted embeddings are more robust to different choices of tokenization function than off-the-shelf pretrained embeddings. Thank you for pointing out the ambiguity between ZeTT and generalization to unseen tokens in the current version of the paper. We will make sure to clarify the distinction.\n\n__Analyzing token distribution overlap.__ The above being said, we also fully recognize the importance of measuring and reporting generalization to unseen tokens and thank you for bringing this to our attention.\nTo address this concern, we ran experiments to analyze the overlap of the distribution of tokens between the HN training tokenizers and the target tokenizers. We counted how often every token occurs in a training tokenizer during the entire HN training procedure, and used this to analyze how often the tokens in the different target tokenizers have been seen during training. The results are shown in Figure F1 in the attached PDF. We differentiate between tokens which occur in the evaluation data, and tokens which do not; this is important since the embeddings of tokens which do not occur in the evaluation data will not substantially impact performance. Notably, for XLM-R, >35% of occurring tokens in Greek, Bulgarian and Russian are unseen by the HN, even though the HN is trained on these languages. This is likely due to the non-Latin scripts. The hypernetwork still performs well in these languages with an average 2% performance decrease at 17% sequence length reduction on XNLI. In total, the HN has seen ~200M different tokens during training.\n\n__Extra Experiments with new Target Tokenizers.__ To test generalization to unseen tokens and out-of-distribution tokenizers, we have also conducted new experiments on (i) languages which are unseen by the HN but seen by the base model, (ii) languages which are unseen by both HN and base model and (iii) an out-of-distribution English word-level tokenizer. These results are available in the attached PDF.\n\n__Experiments on Unseen Languages.__ We ran additional experiments on XLM-R for two languages which are unseen by the HN but seen by the base model (Farsi and Dutch) and two languages which are unseen by both HN and base model (Aymara and Guarani). Results are shown in Table T1 in the attached PDF. The HN predicted embeddings still perform well in this case. Unseen languages do not necessarily have a high number of unseen tokens (as shown in Figure F1), likely due to the script being a confounding factor. However, for Aymara, where >40% of occurring tokens are unseen, the HN even outperforms the base model, at 36% reduced sequence length. This confirms that the HN generalizes to unseen tokens and languages.\n\n__Experiments on an English word-level tokenizer.__ To further evaluate generalization to out-of-distribution tokenizers, we transferred Mistral-7B to a tokenizer containing all words in the evaluation datasets (~100k words). The resulting model is thus word-level on our evaluation data. Results are shown in Table R2. As expected, there is a slight decrease in accuracy, but the gain over FOCUS persists, and sequence lengths are reduced by up to ~20%. 3.3k words are completely unseen in this setup and 13.5k words have been seen in less than 0.1% of training steps.\n\n__Additional Questions.__\n\n> the value that was set for the hyperparameter max token length ‘l’\n\nThe max. sequence length `l` of the hypernetwork is shown in Table 7, it is 7 for English+Code hypernetworks and 15 for multilingual hypernetworks.\n\n> When the target vocabulary size is increased, does the hypernetwork remain the same?\n\nYes, and we ran additional experiments to compare the distributions between training tokenizers and the target tokenizer. The new experiments show that transfer to large vocabulary sizes (for example, to word-level tokenization) is successful, and that the hypernetwork can generalize to unseen tokens.\n\n > How does ZETT compare to FOCUS when extra training is done in the more challenging multilingual setup? Would ZETT be a more attractive choice in this case compared to FOCUS which does not require any prior hypernetwork training?\n\nDue to the duration of the author response period we were not able to run n-shot transfer experiments in a multilingual setup, however, due to the extent of the gap between FOCUS and the HN in the English+Code setup, we believe it is highly likely this gains of our method would persist in a multilingual n-shot setup.\n\n> There is no separate limitations section in the paper as recommended in the checklist guidelines.\n\nThank you for pointing this out. We discuss limitations e.g. in Appendix E, F, and H. In case of acceptance we will use the extra page to add a dedicated limitations section with pointers to these appendices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xy3vJkYVTh", "reviewer": "Reviewer_EScf", "summary": "The authors propose zero-shot tokenizer transfer, a new task where the goal is adapt language models with unseen tokenizers. They propose to tackle this problem by training a hypernetwork that can directly predict the embeddings for any given tokenizer, and such a hypernetwork is trained by sampling various tokenizations and minimizing a language modeling loss.", "questions": "How long does it take for a trained hypernetwork to predict the embedding of a typical off-the-shelf tokenizer?", "limitations": "Appendix E and H.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose an interesting task of zero-shot tokenizer transfer.\n2. The authors have non-trivial designs for tokenizer sampling and the hypernetwork architecture.", "weaknesses": "1. Training the hypernetwork is time-consuming. However, as the authors mention, this is only a one-time cost.\n2. The pretrained hypernetwork is specific to one particular LLM. As a result, this one-time cost needs to be paid for every LLM that wants to benefit from zero-shot tokenizer transfer. Practically, this may not be as efficient as simply retraining the embedding layer for each model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging feedback!\n\n> The pretrained hypernetwork is specific to one particular LLM. As a result, this one-time cost needs to be paid for every LLM that wants to benefit from zero-shot tokenizer transfer. Practically, this may not be as efficient as simply retraining the embedding layer for each model.\n\nWe made this design choice based on the observation that base LLMs have (relative, considering the pace of the field) longevity, while the number of possible tokenizers which transfer would be useful to is essentially unlimited. Due to this fact, training a hypernetwork quickly becomes more efficient than directly retraining the embedding layer, especially considering that our hypernetwork approach needs to see zero to ~1B tokens in the target tokenization for successful adaptation, while prior work typically needs hundreds of billions of tokens as shown by Dagan et al., 2024.\n\n> How long does it take for a trained hypernetwork to predict the embedding of a typical off-the-shelf tokenizer?\n\nAppendix J provides analysis of the required FLOPs (and thus the speed) of the hypernetwork. For Mistral-7B, the hypernetwork needs approximately 0.9% of the FLOPs of the base model per token. Thus, e.g. transferring to a tokenizer with 32k vocabulary size would take approximately as long as inference of 32k * 0.9% = 288 Mistral-7B tokens (scoring speed, not generation speed), which would take less than a second on most consumer-grade GPUs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ne4l3RAxS", "reviewer": "Reviewer_1Y74", "summary": "The paper presents a novel approach towards separating the tokenizer from the language model (LM). All modern LMs are trained with a fixed tokenizer, which prevents them from generalizing well to unseen or rarely seen tokens. Furthermore, the bounding to a specific tokenizer prevents models trained with different tokenizers from being merged and ensembled. To overcome these issues, the authors propose training a hypernetwork capable of mapping embedding parameters between tokenizers. The authors perform extensive experiments to confirm the effectiveness of the proposed approach, even when the tokenizer is transferred in a zero-shot setting.", "questions": "1. Experiments with TinyLlama show a similar trend to that of the Mistral results. However, I think it's important for the reader to see that the approach works with different models; I would recommend moving Appendix G to the main text.\n\n2. You performed experiments with small language models, up to 7 billion parameters. How do you think your approach would generalize to models with a higher number of parameters, and why?\n\n3. Do you think a similar approach would work for other modalities, such as multimodal models?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The authors describe a zero-shot tokenizer transfer problem and propose a state-of-the-art solution for transferring language models to a new tokenizer without additional training. Unlike popular heuristic-based approaches, the authors propose a novel approach that involves training a hypernetwork. Especially impressive is the attention to the technical details described in the paper, for example, the text sampling strategy or the design of the appropriate loss function. Furthermore, the authors experiment with models of different architectures, different tokenizers, and on a variety of tasks, and verify the effectiveness of the proposed approach.", "weaknesses": "In my opinion, the paper is technically sound and contains an appropriate amount of explanation and evaluation. If anything, I would like to see two additional aspects explored: first, more experiments demonstrating how this approach would scale with increasingly large language models (LLMs) in question; and second, experiments with other open-sourced LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and ideas toward extending our approach!\n\n> Experiments with TinyLlama show a similar trend to that of the Mistral results. However, I think it's important for the reader to see that the approach works with different models; I would recommend moving Appendix G to the main text.\n\nThanks, in case of acceptance we will use the additional page to move the TinyLlama results to the main paper. We are also working on training hypernetworks for more recent LLMs such as Llama 3.1, we will add these results along with the TinyLlama results once the experiments are finished.\n\n> You performed experiments with small language models, up to 7 billion parameters. How do you think your approach would generalize to models with a higher number of parameters, and why?\n\nOur evidence suggests our approach scales favorably with parameter count since the amount of layers in the hypernetwork can mostly stay constant while the amount of layers in the base model increases (Appendix J). The relative overhead of applying the hypernetwork thus decreases with the size of the main model. One challenge when scaling to larger and more recent models is the quality of the data the hypernetwork is trained on. A distillation objective on texts sampled from the LLM could help with this.\n\n> Do you think a similar approach would work for other modalities, such as multimodal models?\n\nYes, applications to multimodal models could be a useful direction for future work. For example, the speech tokenization landscape might be even more heterogeneous than the text tokenization landscape, e.g. HuBERT (Hsu et al., 2021), AudioLM (Boros, 2022) and VALL-E (Wang et al., 2023) all use different ways of converting speech to discrete units. A ZeTT-style approach could help transfer across these different speech tokenizations. The same holds true for images, where ZeTT could be used e.g. to transfer to a different resolution of image patches, or across different image encoders. We will add these directions for future work to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their feedback and reviews.

In response to the concerns about generalization to unseen tokens by reviewer mdD9 we have conducted additional experiments to quantify the overlap between the tokenizers seen during hypernetwork training and the target tokenizers. We also ran new experiments on unseen languages and on an out-of-distribution English word-level tokenizer, all of which exhibit positive results. The results are shown in the attached PDF. Please see the response to reviewer mdD9 for more details.

Meta-review: Tokenizer transfer in general is a very practical and important problem. The paper clearly defines and motivates the ZeTT problem, highlighting its practical significance for cross-lingual and cross-domain applications, as well as for merging and ensembling models trained with different tokenizers. Further, this problem has not been explicitly addressed before to my knowledge, making this a valuable contribution to the NLP community. The paper achieved overall very positive reviews for evaluation, novelty and attention to details, with reviewers saying

Reviewer mdD9: "The authors perform analysis on code and multilingual domains, and their method shows promise with respect to FOCUS, a recent embedding transfer method."
Reviewer EScf: "The authors have non-trivial designs for tokenizer sampling and the hypernetwork architecture."
Reviewer 1Y74: "Especially impressive is the attention to the technical details described in the paper, for example, the text sampling strategy or the design of the appropriate loss function. Furthermore, the authors experiment with models of different architectures, different tokenizers, and on a variety of tasks, and verify the effectiveness of the proposed approach."

I therefore recommend accept.
Decision: Accept (poster)