Zero-Shot Tokenizer Transfer

Authors: benjamin  minixhofer
  sep
edoardo m ponti
ivan  vulić
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:23:44

Abstract: Language models (LMs) are bound to their tokenizer, which maps raw text to a sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained primarily on English may still perform well in other natural and programming languages, but have vastly decreased efficiency due to their English-centric tokenizer. To mitigate this, we should be able to swap the original LM tokenizer with an arbitrary one, on the fly, without degrading performance. Hence, in this work we define a new problem: Zero-Shot Tokenizer Transfer (ZeTT). The challenge at the core of ZeTT is finding embeddings for the tokens in the vocabulary of the new tokenizer. Since prior heuristics for initializing embeddings often perform at chance level in a ZeTT setting, we propose a new solution: we train a hypernetwork taking a tokenizer as input and predicting the corresponding embeddings. We empirically demonstrate that the hypernetwork generalizes to new tokenizers both with encoder (e.g., XLM-R) and decoder LLMs (e.g., Mistral-7B). Our method comes close to the original models' performance in cross-lingual and coding tasks while markedly reducing the length of the tokenized sequence. We also find that the remaining gap can be quickly closed by continued training on less than 1B tokens. Finally, we show that a ZeTT hypernetwork trained for a base (L)LM can also be applied to fine-tuned variants without extra training. Overall, our results make substantial strides toward detaching LMs from their tokenizer.

Main paper: 
 introduction: 
 Language Models 1 typically operate on discrete tokens, so they need a means to map text into a sequence of tokens, namely a tokenizer. The vast majority of contemporary LMs use subword tokenizers (Devlin et al., 2019; Jiang et al., 2023; Touvron et al., 2023; Parmar et al., 2024, among others) , whereas others use byte-level (Xue et al., 2022; Yu et al., 2023; Wang et al., 2024) or character-level tokenizers (Clark et al., 2022; Tay et al., 2022) . Regardless of the chosen tokenization 'granularity', these models share a fundamental limitation: once they are trained with a particular tokenizer, inference with a different tokenizer is impossible. In other terms, a pre-trained LM is "bound" to the tokenizer it was trained with. This has wide-ranging implications: since the focus during pretraining is typically primarily on the English language, the tokenizer often encodes languages besides English (Rust et al., 2021) or other domains, such as code, less efficiently. This leads to large disparities in the inference cost between English and non-English text (Ahia et al., 2023; Petrov et al., 2023) . Tokenizers may also be sub-optimal for domains which they were not designed to be used with, e.g. fine-tunings of the Llama models performing subpar on coding tasks (Dagan et al., 2024) . Efficiency and performance are only some of the reasons to transfer models across tokenizers: methods of interaction between models, such as ensembling (Sagi & Rokach, 2018) and model merging (Wortsman et al., 2022; Ainsworth et al., 2023; Yadav et al., 2023) , typically assume the same unit of representation (i.e., equivalent tokenization) across models; if two models adopt different 1 We adopt a broad definition of LMs that also includes models that do not define a probability distribution over finite-length sequences, such as text encoders.38th Conference on Neural Information Processing Systems (NeurIPS 2024). tokenizers, they become unsuitable for ensembling or merging. Problematic artifacts of tokenization such as 'Glitch tokens' (Land & Bartolo, 2024) may also be fixed via transfer to a new tokenizer.To address these issues, past work developed methods to equip an LM with a new tokenizer by retraining the embedding parameters, and optionally continuing to train the entire model (Artetxe et al., 2020; de Vries & Nissim, 2021) . This adaptation can be made faster by initializing the embedding parameters through heuristics (Tran, 2020; Minixhofer et al., 2022; Gee et al., 2022; Dobler & de Melo, 2023; Liu et al., 2023) . In this work, we formulate a new problem: given an LM, can we create an embedding matrix on-the-fly for any arbitrary tokenizer, without ever observing data for it? While past work investigated n-shot tokenizer transfer, we refer to this new problem as zero-shot tokenizer transfer (ZeTT). If the performance of the model can be approximately preserved, ZeTT effectively "detaches" LMs from the tokenizer they were trained with. We first evaluate the efficacy of prior (heuristic-based) approaches for ZeTT, finding that, while heuristics can preserve performance to some extent, there is generally a large gap to the original LM performance.To close this gap, we introduce a new paradigm: We train a hypernetwork on a diverse distribution of tokenizers to predict the embedding parameters for any given tokenizer. By investing in the one-time cost of training the hypernetwork, we aim to subsequently enable effective ZeTT. This proves to be possible: ZeTT via the hypernetwork preserves performance to a few percent accuracy in many cases. Furthermore, the hypernetwork can learn to rapidly adapt to a given target tokenizer by continued training on a small amount (<1B) of extra tokens, whereas previous work typically needed hundreds of billions of tokens (Dagan et al., 2024) . As such, our hypernetwork provides a state-of-the-art solution to n-shot tokenizer transfer, while also establishing a competitive baseline for our newly introduced zero-shot tokenizer transfer problem. This unlocks a range of new ways to combine language models with tokenizers. For example, in this work, we zero-shot substitute the Mistral-7B tokenizer (Jiang et al., 2023) with a tokenizer that encodes code using 10% fewer tokens on average, while preserving functional code generation correctness to approx. 3% (Section 4.2). We also evaluate zero-shot cross-lingual transfer of the multilingual XLM-R encoder model to a range of different languages by substituting the XLM-R tokenizer with a target-language specific tokenizer and reusing adapters trained for the original XLM-R. This leads to a >16% speedup and preserves performance on XNLI (Conneau et al., 2018) to 1% on average. Finally, we show that a hypernetwork trained for a base large LM (e.g. Mistral-7B) can also be applied to fine-tunings of the same model (e.g. Mistral-7B-Instruct-v0.1), preserving capabilities to a large extent (Section 4.3).Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The claims made in the abstract and Section 1 match the results in Section 4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background: 
 Tokenizers and Embeddings. Tokenizers operate as a tokenization function T mapping a text to a sequence of elements in the vocabulary V. By the term tokenizer, we henceforth refer to the tuple comprising the two crucial components, (V, T ). Importantly, the vocabulary and the tokenization function are distinct components; given some vocabulary, there are many ways to encode text as a sequence of tokens in this vocabulary (e.g. Hofmann et al., 2022; Uzan et al., 2024) . After tokenization, the model represents the sequence of tokens via a function E ϕ : V → R dmodel (the embeddings). The embeddings are typically parametrized by a matrix ϕ as a lookup table which assigns a distinct d model -dimensional vector (a row of the matrix) to every element in V. Embeddings are used twice in the language model: once at the input to map tokens to a fixed-size vector, and again at the output to compute a logit for every token, typically via a dot-product of E ϕ (t) with the final hidden state of the LM. Embedding parameters may or may not be shared between the input and the output;2 our method works with both. We denote the entire set of embedding parameters via ϕ, denoting input embeddings as ϕ in and output embeddings as ϕ out , if necessary.Contemporary language models typically use subword tokenizers via BPE (Sennrich et al., 2016) or UnigramLM (Kudo, 2018) . Subword tokenization is a common choice since it can represent arbitrary sequences of text ("open-vocabulary" language modeling) while largely retaining the efficiency of word-level models (Mielke et al., 2021) . However, there are a number of problems with the (lack of) robustness of subword tokenization (Xue et al., 2022; Golkar et al., 2023) . A recent strand of work aims to get rid of subword tokenization via byte-level (so-called "token-free") models (Xue et al., 2022; Yu et al., 2023) . However, these models still operate on tokens, using the set of 256 bytes as the vocabulary, and UTF-8 as the tokenization function (Mielke et al., 2021) . In a similar vein, some models use character-level tokenization (Tay et al., 2022; Clark et al., 2022) , optionally learning to pool characters into longer tokens (Nawrot et al., 2023) . So far, byte-or character-level approaches have been unable to supplant subword tokenization due to longer sequences resulting in higher compute requirements, and not necessarily being more robust (Libovický et al., 2022) . Thus, although our approach is applicable to any tokenizer, we focus our experiments on subword tokenizers. Specifically, we use the UnigramLM parametrization of the tokenization function, and show that other tokenizers can be converted to this parametrization later in Section 5. UnigramLM sets T (x) := argmax C∈Cx t∈C log p(t) where C x is the set of all possible decompositions of x in V. This provides a convenient way to represent tokens as a 2-tuple (t, p(t)) ∈ (V, R).Embedding Initialization Heuristics. Prior work transfers LMs to a new tokenizer by initializing embedding parameters via a heuristic, then continuing to train the embeddings. We denote the original tokenizer as (V a , T a ) and the original embedding parameters as ϕ a . Analogously, the target tokenizer is (V b , T b ) with embedding parameters ϕ b . FVT (Gee et al., 2022) initializes embeddings for any new token t ∈ V b as the mean of the embeddings of T a (t) i.e. the mean of the sequence of embeddings the new token is decomposed into by the previous tokenizer T a . RAMEN (Tran, 2020) , WECHSEL (Minixhofer et al., 2022) and OFA (Liu et al., 2023) require auxiliary embeddingsE aux : V aux → R daux with |V aux ∩ V a | ̸ ≪ |V a | and |V aux ∩ V b | ̸ ≪ |V b |.They use E aux to embed tokens in V a and V b in the same semantic space, then initialize embeddings in E ϕ b as a weighted average of embeddings in E ϕa with weights given by their similarity in E aux . FOCUS (Dobler & de Melo, 2023) initializes embeddings of tokens in V b \ V a as a weighted combination of the overlapping tokens V a ∩ V b , and copies the embeddings of the overlapping tokens. Weights are again computed using an auxiliary embedding matrix E aux , but the only requirement is|V aux ∩ V b | ̸ ≪ |V b |.We use FOCUS as the main baseline since Dobler & de Melo (2023) show it obtains better performance without any training (i.e., zero-shot) than other heuristics, which we also confirm later in Section 4.2.Heuristic-Free Tokenizer Transfer. In addition to heuristics, there is also research into changing the training procedure to facilitate n-shot tokenizer transfer. Marchisio et al. (2023) show that forwardand backward-propagating through a subset of the model layers is sufficient for learning embeddings for a new tokenizer. Chen et al. (2023) find that regularly resetting the embedding parameters during pretraining boosts the speed at which they are relearnt upon transfer. These approaches can be seen as orthogonal to ours. They could be freely combined with our method; we leave this to future work.Embedding Prediction Hypernetworks. Hypernetworks are networks that predict the parameters of another network (Ha et al., 2017) . Prior work uses hypernetworks to predict embeddings for out-ofvocabulary (Pinter et al., 2017) or rare words (Schick & Schütze, 2019 , 2020) of word embedding models (Mikolov et al., 2013) and BERT (Devlin et al., 2019) . In contrast, our hypernetwork (i) approaches the more general problem of transferring to an arbitrary tokenizer, instead of extending the original tokenizer and (ii) can be applied to encoder and decoder LMs, that is, it is objective-agnostic.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
hypernetwork training: 
 We aim to find parameters θ of a hypernetwork H θ : (V b , T b ) → ϕ b for some pretrained LM. Let ϕ a and ψ be the embedding and inner (non-embedding) parameters of the language model, respectively. L is the loss of the language model as a function of the tokens, the embedding parameters, and the inner parameters, typically:L(t, ϕ a , ψ) = CrossEntropy(LM ψ (E ϕa (t)), label(t)),where LM ψ is the language model and label maps the sequence of tokens to corresponding labels, e.g., shifting the sequence in case of standard (autoregressive, causal) language modeling, or masking Algorithm 1 Hypernetwork training loop for Zero-Shot Tokenizer Transfer Input: corpus D, tokenizer sample size n, batch size m, max. token length l, vocabulary size k, noise parameters (µ, σ), pretrained LM parameters ψ, initial hypernetwork parameters θinit.Output: Hypernetwork parameters θ. 1: procedure TRAINHYPERNETWORK 2:θ ← θinit 3:q ← queue(x1, .., xn ∼ D) ▷Create a pool of n texts (where n ≥ m). 4: 5:for step in train_steps do 6: x1, .., xm ∼ D 7:q ← pop(q, m) ▷Remove the least-recently-added batch. 8:q ← push(q, x1, .., xm) ▷Add the current batch. 9: 10:t, f ← substrings(q, l) ▷Compute all substrings and their frequency in q. 11:f ← f / i fi ▷Normalize frequencies to sum to one. 12:z ∼ Lognormal(µ, σ 2 ) 13:for t, f ∈ (t, f ) do 14: the sequence in case of Masked Language Modeling (Devlin et al., 2019) . Importantly, however, we do not make any specific assumptions on L.p(t) ← f + N (0, z 2 )Note that the loss of the language model under the original tokenizer T a on a text x is L(T a (x), ϕ a , ψ). We train our hypernetwork to minimize the lossL θ (T b (x), H θ (V b , T b ), ψ).That is, we substitute the original embedding parameters for the hypernet predictions, and substitute the original tokenizer for a tokenizer (V b , T b ). Figure 1 illustrates the flow of information.Defining Distributions over Texts and Tokenizers. We follow standard practice and sample texts uniformly from the training corpus. Tokenizer sampling is not as trivial: we would like a distribution over tokenizers (V b , T b ) with high variance to encourage generalization to unseen tokenizers. To this end, we introduce a procedure to sample a diverse set of UnigramLM tokenizers. We show later in Section 5 that arbitrary tokenizers can be well-approximated via UnigramLM, motivating this choice.We initially fill a queue q with n texts sampled randomly from the training corpus and, at every step in the training loop, push the m texts in the current batch and remove the m least recently added texts.We then compute all substrings t up to length l and their frequency in q. 34 We add Gaussian noise to the frequencies to arrive at a final score p(t) for every token t. Finally, we assemble the tokenizer by taking the top k tokens with the highest p(t) as the vocabulary and UnigramLM parametrized by p(t) as the tokenization function. The training loop is summarized in Algorithm 1. The 'rolling' queue of texts q ensures high variance in the vocabulary, while the Gaussian noise added to the frequencies ensures high variance in the tokenization function.Importantly, the texts and the tokenizer are sampled dependently: the batch of m texts used for training is a subset of the n texts used for sampling the tokenizer. If they were sampled independently, the probability for a token to occur would be p(token) ∝ p(token ∈ V b ) × p(token ∈ x). Since both these factors are small for rare tokens, p(token) would get vanishingly small in this case.Warmup & Auxiliary Loss. In practice, directly minimizing L θ starting from randomly initialized θ is difficult. Thus, we include a warmup stage where we train the hypernetwork to mimic the embedding parameters of the original tokenizer, akin to MIMICK (Pinter et al., 2017) . The warmup stage is substantially quicker than the main stage because there is no need to propagate through the main model. We found it prevents divergence in some cases. Afterwards, we add an auxiliary loss, which, for every token in the sampled vocabulary V b that also exists in the original vocabulary V a , penalizes the distance to the corresponding embedding in ϕ a .L warmup θ = ∥H θ (V a , T a ) -ϕ a ∥ 2L aux θ = 1 |V a ∩ V b | t∈|Va∩V b | ∥H θ (V b , T b )[V b [t]] -ϕ a [V a [t]]∥ 2This penalizes drift from the warmup stage. Combining it with the main loss yields the final loss.L final θ = L θ (T b (x), H θ (V b , T b ), ψ) + α • L aux θThe hyperparameter α weighs the contribution of the auxiliary loss. Since H θ (V b , T b ) is also required for the main loss, it requires negligible extra computation. The auxiliary loss is necessary especially for models with separate input and output embedding matrices as shown in Appendix B.It remains to define the hypernetwork architecture, that is, how to map the tokenizer (V b , T b ) to the embedding parameters ϕ b . To this end, we represent the new tokens t b ∈ V b by decomposing them using the original tokenization function T a , and embedding them with the original embeddings E ϕa .5 This sequence of embeddings is passed through multiple Transformer layers, plus a separate prediction head for the input embeddings and output embeddings ϕ in b and ϕ out b . The hypernetwork thus consists of another language model which is applied separately for every token. We refer to the hypernetwork's language model as HLM θ . HLM θ can be thought of as learning how to compose the sequence of tokens T a (t)-which any given token is decomposed into-into one embedding, as illustrated in Figure 2 . Importantly, we do not take the tokenization function into account. By sampling diverse tokenizers during the training process, we aim for the hypernetwork to learn to produce a single embedding suitable to a wide variety of different tokenization functions. We analyze the impact of this choice later in Section 5. We also experiment with hypernetworks which do take the tokenization function into account in Appendix C.On Token Decomposition. The input to the hypernetwork consists of the sequence of tokens T a (t) that any given token is decomposed into. However, this decomposition is not always trivial: for example, T a could be character-level, while the token t could be in the vocabulary of a byte-level tokenizer T b . In this case, t could be any arbitrary sequence of bytes (not necessarily valid UTF-8).To solve this issue, we introduce a procedure to convert tokenizers to the byte level by adding a small amount of extra tokens to the vocabulary (c.f. Section 5). This guarantees that T a can decompose arbitrary tokens. The embeddings of the extra vocabulary are initialized randomly and trainable alongside the hypernetwork parameters.Table 1 : Accuracy on XNLI when reusing adapters trained for the original XLM-R model with new zero-shot transferred language-specific tokenizers. Also shown are the absolute change in accuracy from applying our hypernetwork (∆accuracy) and the average decrease in token length of the language-specific tokenizers over the original tokenizer (∆length). Data. We use the English subset of the MADLAD-400 corpus (Kudugunta et al., 2023) and code from the StarCoder data (Li et al., 2023) for hypernetwork training. The sampling ratio of English to Code is 7:3 following Zhang et al. (2024) . For the multilingual hypernetwork, we use a subset of 26 of the languages used in XGLM (Lin et al., 2022) .6 with data from MADLAD-400. We sample languages using a multinomial distribution as in Conneau & Lample (2019) with α = 0.1. For the n-shot experiments, we also train on the StarCoder data, but substitute the English section of the MADLAD-400 corpus for Flan v2 (Longpre et al., 2023) sampled as in Soldaini et al. (2024). 7 Evaluation. We use the standard benchmarks PiQA (Bisk et al., 2020) , HellaSwag (HS; Zellers et al., 2019) , BoolQ (Clark et al., 2019) , MMLU (Hendrycks et al., 2021) and the "easy" subset of ARC (Clark et al., 2018) for evaluation in English and the synthesis task of HumanEvalPack (Muennighoff et al., 2023) for coding evaluation. For multilingual evaluation, we use XNLI (Conneau et al., 2018) , XCOPA (Ponti et al., 2020) and MMLU as machine-translated by Lai et al. (2023) . Models. To evaluate our method, we use Mistral-7B (Jiang et al., 2023) as the main decoder-style language model and XLM-R (Conneau et al., 2020) as a representative of encoder-style models. 8 We also experiment with the smaller TinyLlama-1.1B model (Zhang et al., 2024) in Appendix H.∆accuracy -1% -2% -1% -2% -1% -1% -2% -1% -2% -1% -3% -2% +1% -1% ∆length -22% -14% -13% -23% -9% -11% -12% -13% -13% -19% -15% -9% -3% -14%Tokenizers. We transfer models to the GPT2 tokenizer (Radford et al., 2019) for evaluation on natural language benchmarks and to the StarCoder tokenizer (Li et al., 2023) for evaluation on code benchmarks.9 For multilingual evaluation, we train language-specific monolingual tokenizers with a vocabulary size of 50k using SentencePiece (Kudo & Richardson, 2018) and evaluate transfer to these. We also verify that the hypernetwork is robust to the choice of vocabulary size in Appendix E.Hypernetwork training. We train the hypernetwork for 200k steps (10k of which are MIMICK-style warmup) with a batch size of 128 and a sequence length of 128 (we find it sufficient to use short sequence lengths). 10 For the multilingual decoder-style models, we start from the English + Code checkpoint and forgo MIMICK-style warmup, keeping other hyperparameters unchanged. We use a RoBERTa-style architecture i.e. bidirectional attention and Post-LayerNorm Transformer layers (Liu et al., 2019) , but use a feedforward dimension of 2x the hidden dimension instead of 4x for the hypernetwork. See Appendix D for a full list of hyperparameters.Continued training details. To keep runtime comparable between training the model with hypernetwork and direct training (without hypernetwork), we run hypernetwork inference only for a subset of k = 16384 tokens in the continued training case. The subset consists of all tokens occurring in the batch, plus a uniform sample of those that do not occur. The language modeling loss is then only computed over this subset of tokens. We found in preliminary experiments that this causes only minor performance degradation. Furthermore, we use the zero-shot predicted embeddings as the target for the auxiliary loss instead of using the original embeddings. This stabilizes training. We train for 50k steps with a batch size of 32 and sequence length of 512, resulting in 'seeing' 819.2M tokens.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [No] Justification: Code is not submitted alongside this paper but will be provided upon publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
zero-shot and n-shot results: 
 Results for XLM-R are shown in Table 1 . We take task adapters trained for the original XLM-R model on the English XNLI dataset via Poth et al. (2023) and substitute the tokenizer for our language-specific one. We compare our hypernetwork against a simple lexical baseline (copying the Table 5 : Single model rating results on MT-Bench of transferring Mistral-7B-Instruct-v0.1 to the GPT2 tokenizer using the hypernetwork trained for the base Mistral-7B model. We use gpt-3.5-turbo-1106 as a judge. orig. is the original fine-tuned model, base the model with the same tokenizer but embeddings substituted for the base models' embeddings. λ is the scaling factor for the weight differences in Task Arithmetic (Ilharco et al., 2023) .original 0-shot n-shot Embeddings orig. base FOCUS ours ours@800 λ ----0.0 0.3 0.5 0.7 Score (1 to 10) 7.33 7.48 5.03 6.56 6.59 6.75 6.82 6.77 embeddings of overlapping tokens and initializing the rest randomly), FVT, OFA, and FOCUS (c.f. Section 2). We focus only on FOCUS in the following since it performs best among the baselines.Our hypernetwork consistently outperforms all baselines and preserves accuracy to 1% on average, losing 3% in the worst case and improving by 1% in the best case, while sequences are on average 14% shorter for the language-specific tokenizers; inference is thus more than 16% faster. 11 We show in Appendix E that these results are robust to the target vocabulary size.Table 2 shows results on English and Code for Mistral-7B. We find that ZeTT is more challenging in the decoder case: FOCUS performs roughly random in the worst case (-23.2% on BoolQ) and is reduced to 0% pass@1 on HumanEval in Python. The hypernetwork goes a long way in closing this gap but still falls behind on some benchmarks. However, continuing to train the hypernetwork with the target tokenizer closes the gap almost completely. In fact, continued training on 800M tokens with the StarCoder tokenizer performs better than continued training for the same amount of tokens with the original tokenizer, potentially because the StarCoder tokenizer is more well suited towards code; it results in approx. 10% less tokens on average. Also, notably, continued training with the original tokenizer slightly degrades performance on average; this may be due to a higher-quality data mix used for pretraining Mistral-7B, whereas we use public data sources (c.f. Section 4.1).Results of the multilingual hypernetwork for Mistral-7B are shown in Table 3 and Table 4 . On XCOPA, the hypernetwork on average improves performance over the original model, while also more than halving sequence length. XCOPA performance is close to random in some languages (e.g. Southern Quechua (qu) and Estonian (et)), so we also evaluate on multilingual MMLU. Here, although the hypernetwork clearly outperforms FOCUS (which performs close to random), there is still a substantial gap to the original model; this could presumably be fixed via continued training.A large amount of the models used by practitioners are fine-tuned versions of base models 12 , e.g. via SFT or RLHF (Ouyang et al., 2022) . We now attempt to answer the question: Given a hypernetwork trained for a base model, can we apply this hypernetwork to fine-tuned versions of the same model without any extra training? This would act as a multiplying factor for the hypernetwork's applicability. First, we observe that the embedding space of a fine-tuned model is compatible with that of the base model: the embeddings of the fine-tuned Mistral-7B-Instruct-v0.1 have an average cosine similarity of 98.6% to the corresponding embedding in the base model while the average cosine similarity of the mean embedding vector is 17.4%. 13 Embedding compatibility also holds true for other models (Appendix H). The predictions of a hypernetwork trained for a base model can thus be used out-ofthe-box with fine-tuned models. We verify that this is the case by evaluating Mistral-7B-Instruct-v0.1 transferred to the GPT2 tokenizer on the corrected 14 version of MT-Bench (Zheng et al., 2023) . For n-shot transfer, since we train the full model we also need a way to transfer the non-embedding parameters; we achieve this via Task Arithmetic (Ilharco et al., 2023) . Results are shown in Table 5 .The transferred fine-tuned model performs well, coming within approx. 0.5 score of the original model. Also, curiously, the fine-tuned model with the original tokenizer performs better when using the embeddings of the (not fine-tuned) base model; this may be a prudent direction for future work.11 1/(1-14%)=16%, plus additional speedup due to attention scaling quadratically with sequence length. 12 We refer to models purely pretrained on the Language Modeling task as base models. 13 Averaged across the input and the output embeddings. 14 Using the corrections from https://github.com/InflectionAI/Inflection-Benchmarks.Table 6 : NLI performance on Farsi (FarsTail; Amirkhani et al., 2023) , Dutch (SICK-NL; Wijnholds & Moortgat, 2021) , Aymara and Guarani (AmericasNLI; Ebrahimi et al., 2022) . We measure zero-shot transfer from a model trained on English XNLI (c.f. Table 1 ), except for Sick-NL where we train an adapter on SICK (Marelli et al., 2014)  
discussion: 
 Converting tokenizers to byte-level. As per Section 3.2, we need a procedure to convert tokenizers to the byte level to ensure that token decomposition is always possible. This is trivial in most cases; the missing bytes just need to be added to the vocabulary. BPE is an exception: here, we need to change the units on which merges are defined from characters to bytes. We achieve this by adding merges to assemble the characters used by the tokenizer from their constituent bytes to the beginning of the merge table. This preserves the tokenization in more than 99% of cases (Appendix J).Converting tokenizers to UnigramLM. We also introduce a procedure to convert arbitrary tokenizers to tokenizers using UnigramLM as the tokenization function. We refer to this process as unigramifying (details in Appendix A). An important assumption of the hypernetwork training is that by using the UnigramLM parametrization with scores distributed as Gaussians we can cover a sufficiently diverse distribution of tokenizers for the hypernetwork to generalize to e.g. BPE tokenizers. Unigramifying allows us to check if, in principle, this is possible. Luckily, we find that it is: unigramifying results in minimal performance degradation when substituting the original tokenizer with the corresponding UnigramLM tokenizer (Appendix J). Although this does not guarantee that our distribution of tokenizers is sufficiently diverse, our empirical results suggest it is (cf. Section 4.2).We believe our conversion methods to UnigramLM and to byte-level will simplify further research into tokenizer transfer, showing that the wildly heterogeneous landscape of tokenizers can be well approximated via byte-level UnigramLM tokenizers.What is the effect of amortizing over the tokenization function? As described earlier in Section 3, we 'amortize' over the tokenization function, that is, the tokenization function is not an input to our hypernetwork. We find that the predicted amortized embeddings are robust to the choice of tokenization function. For example, the set of embeddings predicted for the GPT2 vocabulary has low bits-per-character for both the original GPT2 tokenization function and a different UnigramLM tokenization function with scores based on token frequencies (Appendix J). This is not the case for the original GPT2 embeddings: while they (as expected) perform well with the original GPT2 tokenizer, there is significant performance degradation when switching to the frequency-based UnigramLM tokenization function. This calls into question prior work copying the embeddings of overlapping tokens for transfer across tokenizers (Dobler & de Melo, 2023; Gee et al., 2022, among others) , indicating that even if there is an exactly overlapping token in the original tokenizer, it is not necessarily the optimal initialization of the corresponding token in the new tokenizer.Although we amortize over most of the aspects of the tokenization function, in practice, tokenization functions rely on a considerable amount of engineering, so it is not possible to amortize over everything; we discuss remaining assumptions in Appendix I.Analyzing computational overhead. We estimate the FLOPs per token of multiple hypernetworks in Appendix K. Given a batch size n and sequence length s for the main model, and using the hypernetwork to compose k token sequences of length t, the FLOPs per batch will be n × s × ( FLOPs token ) main + k × t × ( FLOPs token ) hypernet . Taking Mistral-7B as an example with n = s = 128, k = 32768 and t = 7 the FLOPs per batch will be 252T + 30T i.e. a 12% overhead from applying the hypernet. Notably, we observed that a hypernetwork size of three layers is sufficient, regardless of the main model, so the relative overhead decreases with increased amounts of layers in the main model.Generalization to unseen tokens. Although our primary goal is generalization to unseen tokenizers (i.e., tuples (V, T )), the question of how well our hypernetwork can generalize to unseen tokens (elements of V) presents itself. To answer this question, we test the XLM-R and Mistral-7B hypernetworks on out-of-distribution vocabularies. Specifically, we test the XLM-R hypernetwork on Farsi and Dutch (which are unseen by the hypernet, but seen by the base model) as well as Aymara and Guarani, which are unseen by both. Table 6 confirms the hypernet performs well in this case, even gaining in performance over the model with original embeddings in completely unseen languages. In this setup, up to 40% of the used tokens in the target vocabularies have never been seen during hypernetwork training (we analyze this overlap in detail in Appendix G). The reason for the performance increase from the hypernetwork on unseen languages may be that, under the original tokenization, the embeddings of many tokens occuring in unseen languages are undertrained (c.f. Land & Bartolo, 2024) , while the embeddings produced by the hypernetwork do not suffer from this issue; future work could investigate this in more detail. For Mistral-7B, we instead transfer to an out-of-distribution word-level tokenizer by creating a tokenizer which contains all words which occur in any evaluation corpus (approx. 100k in total). 3.3k words are completely unseen and 13.5k words have been seen in less than 0.1% of training steps. Still, performance only deteriorates by a small amount and the improvement over FOCUS persists as shown in Table 7 . 
conclusion: 
 We have established Zero-Shot Tokenizer Transfer (ZeTT), the difficult problem of transferring language models to a new tokenizer without any training. We have found that prior heuristics for embedding initialization provide a first baseline for ZeTT, but fall short in many cases. To establish a much stronger baseline, we introduced a hypernetwork-based approach that closes the gap to a large extent, and can be further improved via continued training on a few (<1B) tokens. Due to preserving the embedding space of the original model, ZeTT can be applied to e.g. reusing adapters trained for the original model with a different tokenizer, and to transferring fine-tuned models to a new tokenizer using a hypernetwork trained for the base model. In aggregate, this work is a substantial step towards detaching language models from their tokenizer, increasing their flexibility and reusability. 
limitations: 
 The key limitation of our approach is the requirement to train a hypernetwork for every base model. Although the hypernetwork only needs to be trained once, doing so is computationally intensive and may not be feasible for many LLM practitioners. Instead, it may be a task LLM providers are better positioned to undertake. Other limitations are the remaining assumptions on the tokenization function (Appendix I), and not taking the tokenization function into account (Appendix J), although these limitations do not appear to have substantial impact in practice. Finally, we have limited our scope to experiments on text-only models, but Zero-Shot Tokenizer Transfer could also be beneficial for multimodal models, such as models 'perceiving' images or speech; we leave this to future work. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We adhere to all applicable points of the Ethics Guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss implications on fairness across languages in Section 1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release any standalone models; released hypernetworks are bound to the base model they were trained for, including being bound to the safeguards put on the base model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, were applicable, throughout the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We introduce a procedure to convert arbitrary tokenizers to UnigramLM in an optimal (but lossy) way which we refer to as unigramifying. Given a text x and the sequence of tokens T (x), for the UnigramLM tokenizer T to be equivalent to T , it is necessary that T fulfills t∈T (x) log p T (t) > t∈C log p T (t) for all C in C x \ {T (x)}.15 Thus, given a corpus of texts X we can formulate a lossL T (X, T ) = x∈X C∈Cx\{T (x)} max   0, t∈C log p T (t) - t∈T (x) log p T (t)  which is zero if and only if the condition above is satisfied for all texts in X. This objective is piecewise linear, so it can be converted to a standard Linear Programming (LP) form and solved via an LP solver. In practice, we use the CPLEX v22.1 (IBM ILOG, 2022) solver. Since applying the procedure to a corpus directly would be costly, we first pre-tokenize the training corpus, then count the pretokens, and choose the top n = 1000000 pretokens as the set X. We found in preliminary experiments that the auxiliary loss is necessary, especially for models that do not share embedding parameters between the input and the output (models with untied embeddings).To validate this hypothesis, we conducted an experiment where we manually untied the embeddings of GPT2 i.e. used a separate hypernetwork prediction head for the input and the output embeddings.Although everything else is kept the same, the untied GPT2 model diverges without the auxiliary loss, whereas the original GPT2 trains as expected, even without an auxiliary loss (Figure 3 ).We experimented with hypernetworks taking the tokenization function into account by adding sparse inter-token attention blocks between the self-attention and the FFN in every hypernetwork layer. Sparse inter-token attention consists of two attention blocks. The first attention block attends from a fixed amount of learnable inter-token embeddings (e.g. 16, each a vector of size d model ) to the ith token representation of every token sequence passed to the hypernetwork. The second block attends from the ith token representation to the inter-token embeddings. This way, we factorize the attention to e.g. one 16 × k attention and one k × 16 attention, instead of the standard k × k self-attention which would be infeasibly slow for typical vocabulary sizes. We only add inter-token attention for the first token in every sequence. This improves performance on the sampled tokenizers, but does not improve performance on 'real-world' tokenizers (Table 8 ); investigating this mismatch is a direction for future work.Hyperparameters for hypernetwork training are shown in Table 9 . For continued training, we use the same optimizer, but a sequence length of 512, batch size of 32, training for 50k steps and a constant learning rate chosen among the set {1e-6, 3e-6, 6e-6, 1e-5, 3e-5} to maximize performance.The chosen learning rate is 1e-6 for the runs keeping the original tokenizer (original@800M), 6e-6 for continued training starting from FOCUS (FOCUS@800M) and 3e-6 for continued training with the hypernetwork (ours@800M). Since the tokenizers we experiment with have similar vocabulary sizes (50k for the language-specific tokenizers and for GPT2, 49k for the StarCoder tokenizer) we conduct an additional experiment to quantify the sensitivity of the performance of our hypernetwork to the size of the target tokenizer. We find that although there is slight performance degradation when increasing the size of the new tokenizers' vocabulary, the hypernetwork is fairly robust to vocabulary size (Figure 4 ).Intuitively, transfer is easier the more the target has in common with the source. One way to measure commonality between the original (source) and the target tokenizer is the fraction of tokens of the target vocabulary which also exist in the source vocabulary (vocabulary overlap). Performance correlates with vocabulary overlap, but it correlates more strongly with the probability for tokens to overlap: that is, when randomly sampling some token from a corpus tokenized with T b , the probability that this token also exists in the vocabulary of T a . We refer to this metric as p(overlap). p(overlap) has higher correlation with the performance of FOCUS, indicating that our hypernetwork depends less on overlap (Figure 5 ). We analyze the occurrence for all tokens in the target vocabulary (top) and for tokens which occur at least once in the evaluation data (bottom) across target tokenizers in seen languages for XLM-R (left), unseen XLM-R languages (middle) and English Mistral-7B tokenizers (right). The bottom row is more informative w.r.t. how well the hypernetwork generalizes to unseen tokens since tokens which do not occur do not substantially impact evaluation.Table 11 : Single model rating results on MT-Bench of transferring TinyLlama-1.1B-Chat-v1.0 to the GPT2 tokenizer, compare Table 11 . We analyze how often the hypernetwork sees the tokens in the vocabulary of different target tokenizers across multiple settings in Figure 6 . We differentiate between tokens which occur in the evaluation data, and tokens which do not; this is important since the embeddings of tokens which do not occur in the evaluation data will not substantially impact performance. Notably, for XLM-R, >35% of occurring tokens in Greek, Bulgarian and Russian are unseen by the hypernet, even though the hypernet is trained on these languages. This is likely due to the non-Latin scripts. The hypernet still performs well in these languages with an average 2% performance decrease at 17% sequence length reduction on XNLI. In total, the HN has seen approx. 200M different tokens during training.Zero-shot and n-shot results for TinyLlama-1.1B are shown in Table 10 and MT-Bench results of transferring TinyLlama-1.1B-Chat-v1.0 in Table 11 . We observe the same patterns as on Mistral-7B.In practice, besides the tokenization algorithm itself (e.g. BPE, UnigramLM) tokenization functions also contain other steps, in particular pretokenizing text into smaller chunks (usually words) on which to apply the tokenization function (Mielke et al., 2021) . In our experiments, we assume fixed pretokenization given by a regular expression based on the regular expression used by GPT2 (Radford et al., 2019) , adjusted to not over-segment text in languages using characters in the Unicode Mark category within words (e.g. Hindi and Tamil). We also add a prefix space (i.e., a whitespace at the start of the text to tokenize) if and only if the original tokenizer also uses a prefix space. Finally, we always add whitespace characters covering sequences of consecutive whitespaces up to 16 characters long similar to Black et al. (2022) to ensure code is tokenized efficiently. These light assumptions mostly preserve the generality of our method but could be further relaxed in future work.Results measuring the success of unigramifying tokenizers are shown in Table 12 . Results measuring the success of amortizing over the tokenization function are shown in Table 13 .Table 14 : Parameter count and FLOPs estimates for our hypernetwork (and the corresponding main model) in different setups. The relatively lower computational cost compared to parameter count is mainly due to forgoing de-embedding which contributes significantly to FLOPs (Kaplan et al., 2020) . Estimated FLOPs per token for the hypernet and the corresponding main model are shown in Table 14 . We estimate FLOPs on the basis of XLA-compiled instructions using Jax (Bradbury et al., 2018) .NeurIPS Paper ChecklistSome models share the input and the output embedding parameters (e.g.Conneau et al., 2020), this has been shown to be problematic(Chung et al., 2021) and many recent LLMs (e.g.Jiang et al., 2023) separate them.In practice, implementing q as a queue allows efficiently caching the substrings and their probability p(t) at this step. They only need to be recomputed for the new m texts encountered in every batch.To ensure substrings do not cross word boundaries we pretokenize the text before computing substrings.In the multilingual case, we also append an element containing a learnable language-specific embedding.We exclude languages without whitespace between words since they would require language-specific pretokenizers (e.g.Sun, 2012). Although our method is also applicable to this case, we leave this to future work.We use Flan v2 because we observed a strong decrease in accuracy from continuing to train on the MADLAD-400 data (even with the original tokenizer). The training data for most LLMs (including Mistral-7B) is not public, but it is plausible that this decrease stems from higher-quality data mixed in especially towards the end of training as in e.g.Groeneveld et al. (2024).Although (decoder-style) LLMs are the centerpiece of a large amount of current NLP research, encoder-style LMs have wide-ranging applications in e.g. retrieval(Khattab & Zaharia, 2020) and LLM distillation(Hsieh et al., 2023) due to their lower computational cost.We chose these tokenizers due to their popularity and comparatively efficient encoding of the target domain.Training takes around one day for the XLM-R hypernetwork on a TPU v3-8 and three days for the Mistral-7B hypernetwork on a TPU v4-32 pod.This is not sufficient for equivalence since order is ignored e.g. T (x) = {ab, a, b} and T (x) = {a, b, ab} fulfill the criterion but are not equivalent. 


Decision: Tokenizer transfer in general is a very practical and important problem. The paper clearly defines and motivates the ZeTT problem, highlighting its practical significance for cross-lingual and cross-domain applications, as well as for merging and ensembling models trained with different tokenizers. Further, this problem has not been explicitly addressed before to my knowledge, making this a valuable contribution to the NLP community. The paper achieved overall very positive reviews for evaluation, novelty and attention to details, with reviewers saying

Reviewer mdD9: "The authors perform analysis on code and multilingual domains, and their method shows promise with respect to FOCUS, a recent embedding transfer method."
Reviewer EScf: "The authors have non-trivial designs for tokenizer sampling and the hypernetwork architecture."
Reviewer 1Y74: "Especially impressive is the attention to the technical details described in the paper, for example, the text sampling strategy or the design of the appropriate loss function. Furthermore, the authors experiment with models of different architectures, different tokenizers, and on a variety of tasks, and verify the effectiveness of the proposed approach."

I therefore recommend accept.

Reviews: review: paper_summary: paper_summary: This paper presents a way to perform "zero-shot" construction of embeddings for a new target tokenizer. It proposes a hypernetwork based approach that learns to take in the embeddings of tokens generated by tokenizing the target token with the original tokenizer and produce the embedding of this token. The authors perform analysis on code and multilingual domains, and their method shows promise with respect to FOCUS, a recent embedding transfer method. Overall, the idea is novel, innovative, and seems promising. However, based on my understanding, I have significant concerns with the evaluation settings and some comparisons. I am willing to reconsider my assessment if my concerns on evaluations are adequately addressed.
strength_weakness: strength_weakness: * I do not think ZETT is truly ‘zero-shot’. While it is important to make a distinction between vocabulary V and tokenization function T, it is likely that a large portion of embeddings are agnostic of T and more reliant on V. For example, I would not expect the embedding of the token ‘car’  to be much different when an LM is trained on two different tokenization functions.
* Therefore I think it is important to analyze the overlap of the distribution of the tokens. Appendix F analyzes the overlap with respect to the original vocabulary. But it is also important to study the overlap (both vocabulary and segmentations) between what UnigramLM generates during training versus the ‘target’ tokenizer during evaluation to really contextualize whether this approach is useful. 
* The above wouldn’t be as big of a problem if the hypernetwork data did not contain the same languages/domains as that it was trained on. But from what I see all the evaluated languages are seen during hypernetwork training. I assume it will be something similar for code too.
* I think what would make more sense is to really evaluate on an ‘unseen’ language that uses the same script. Here one can expect a more pronounced distribution shift of both vocabulary and tokenization functions.
questions: questions: * I cannot seem to find the value that was set for the hyperparameter max token length ‘l’. For a sufficiently large ‘l’ I would assume there would be a significant overlap of the vocabulary produced by UnigramLM during training and the target tokenizer during evaluation, especially when the domain/language is the same.
* When the target vocabulary size is increased, does the hypernetwork remain the same? If yes, I could see how some tokens might be zero shot transferred as the vocab size grows. However, without comparing the distributions as mentioned earlier, it’s hard to say what exactly the overlap is.
* How does ZETT compare to FOCUS when extra training is done in the more challenging multilingual setup? Would ZETT be a more attractive choice in this case compared to FOCUS which does not require any prior hypernetwork training?
limitations: limitations: I believe the weaknesses outlined above need to be addressed, which are not discussed. There is no separate limitations section in the paper as recommended in the checklist guidelines.
paper_summary: paper_summary: The authors propose zero-shot tokenizer transfer, a new task where the goal is adapt language models with unseen tokenizers. They propose to tackle this problem by training a hypernetwork that can directly predict the embeddings for any given tokenizer, and such a hypernetwork is trained by sampling various tokenizations and minimizing a language modeling loss.
strength_weakness: strength_weakness: 1. Training the hypernetwork is time-consuming. However, as the authors mention, this is only a one-time cost.
2. The pretrained hypernetwork is specific to one particular LLM. As a result, this one-time cost needs to be paid for every LLM that wants to benefit from zero-shot tokenizer transfer. Practically, this may not be as efficient as simply retraining the embedding layer for each model.
questions: questions: How long does it take for a trained hypernetwork to predict the embedding of a typical off-the-shelf tokenizer?
limitations: limitations: Appendix E and H.
paper_summary: paper_summary: The paper presents a novel approach towards separating the tokenizer from the language model (LM). All modern LMs are trained with a fixed tokenizer, which prevents them from generalizing well to unseen or rarely seen tokens. Furthermore, the bounding to a specific tokenizer prevents models trained with different tokenizers from being merged and ensembled. To overcome these issues, the authors propose training a hypernetwork capable of mapping embedding parameters between tokenizers. The authors perform extensive experiments to confirm the effectiveness of the proposed approach, even when the tokenizer is transferred in a zero-shot setting.
strength_weakness: strength_weakness: In my opinion, the paper is technically sound and contains an appropriate amount of explanation and evaluation. If anything, I would like to see two additional aspects explored: first, more experiments demonstrating how this approach would scale with increasingly large language models (LLMs) in question; and second, experiments with other open-sourced LLMs.
questions: questions: 1. Experiments with TinyLlama show a similar trend to that of the Mistral results. However, I think it's important for the reader to see that the approach works with different models; I would recommend moving Appendix G to the main text.

2. You performed experiments with small language models, up to 7 billion parameters. How do you think your approach would generalize to models with a higher number of parameters, and why?

3. Do you think a similar approach would work for other modalities, such as multimodal models?
limitations: limitations: Yes

Comments: title: Thank you for your response and addressing my questions.
comment: Thanks for the response. I will maintain my score.
Looking for ward to see new results on latest LLMs, and multimodal expansion in the future.
Thank you for the detailed response. Most of my concerns have been addressed, and I am happy with the proposed revisions. I have increased my score.