zip2zip: Inference-Time Adaptive Tokenization via Online Compression
Authors: Saibo Geng
Nathan Ranchin
Yunzhen Yao
Maxime Peyrard
Chris Wendler
Michael Gastpar
Robert West
Venue: NeurIPS 2025 poster ()
Keywords: tokenization
LLM
language modeling
inference efficiency
large language model
compression
Primary area: deep_learning

Abstract: Tokenization efficiency plays a critical role in the performance and cost of large language models (LLMs), yet most models rely on static tokenizers optimized on general-purpose corpora. These tokenizers’ fixed vocabularies often fail to adapt to domain- or language-specific inputs, leading to longer token sequences and higher computational costs. We introduce zip2zip, a novel method for achieving
context-adaptive tokenization in LLMs at inference time. Leveraging an online data compression algorithm (Lempel–Ziv–Welch), zip2zip dynamically expands its active vocabulary at inference time by continuously replacing fragmented token sequences with more compact hypertokens, which it can immediately output during generation.  In doing so, the model refines its internal tokenization scheme to match
the token distribution of the current context, reducing redundancy and improving representational efficiency. zip2zip consists of three key components: (1) a tokenizer based on Lempel–Ziv–Welch compression that incrementally merges co-occurring tokens into reusable hypertokens on the fly; (2) a dynamic embedding (and unembedding) layer that computes embeddings for newly formed hypertokens at runtime; and (3) a variant of autoregressive language modeling that pretrains the model to handle hypertokenized, compressed text sequences as inputs and outputs. We show that an existing LLM can be uptrained for zip2zip in 10 GPU-hours via parameter-efficient finetuning. The resulting LLM performs test-time adaptation, learning to use hypertokens in unseen contexts and reducing input and output tokens by 15–40%.
Code and models are released at https://github.com/epfl-dlab/zip2zip.

zip2zip: Inference-Time Adaptive Tokenization via Online Compression
Saibo Gengl*
Nathan Ranchin Yunzhen Yao
Maxime Peyrard*
Chris Wendler
Michael Gastpar!
Robert West!
EPFI. Northeastern University
'niversitéé Grenoble Alpes, CNRS, Grenoble INP, LIG
(saibo. geng,
michael. gastpar,, robert.west)ees. west) yunzhen. yao, yao,
maxime aesimee.psrrarddunnnggreeee alp.... fr ch. wh..wennder@@northsstem.mmmm
Abstract
Tokenization efficiency plays critical role in the performance and cost of large language models (LLMMs) yet most models rely on static tokenizers optimized on general-purpose corpora. These tokenizers' fixed vocabularies often fail to adapt to domain- or language-specific inputs, leading to longer token sequences and higher computational costs. We introduce zip2zip, novel method for achieving context- context-adaptive tokenization in LLMs at inference time. Leveraging an online data compression algorithm (Lempel -iaepel-------- Ziv- Ziy-Welch), zip2zip dynamically expands active vocabulary at inference time by continuously replacing fragmented token sequences with more compact hypertokens, which it can immediately output during generation. zip2zip consists three key components: (1) a tokenizer based on Zempel-Ziv-Wellee compression that incrementally merges co-occurring tokens into reusable hypertokens on the fly; (2) a dynamic embedding (and unembedding) layer that computes embeddings for newly formed hypertokens runtime; and (3) a variant of autoregressive language modeling that pretrains the model to handle hypertokenized, compressed text sequences as inputs and outputs. We show that existing LLM can be uptrained for zip2zip in 10 GPU-hours via parameter-efficien finetuning. The resulting LLM performs test-time adaptation, learning to use hypertokens in unseen contexts and reducing input and output tokens by 15-40%. Code and models are released
Introduction
Large language models (LLMs) have shown impressive versatility across broad spectrum of tasks and domains [Brown et al., 2020, Bubeck et al., 2023], including biomedical tests [Nori et al., 2023], mathematical reasoning [Frieder al., 2023], programming [Jiang al., 2024], and multiple human languages. A critical underlying component of this flexibility is the tokenizer, which defines the model's 's vocabulary and governs how text is converted into token sequences fed to the model. The efficiency of the tokenization scheme-i.e how compactly text is represented as tokens has sigmificant impact on model performance. particular, a more compact tokenization yields three key benefits: (1) larger effective context windows; (2) lower computational (and thus monetary) cost; and (3) shorter response times.
Despite its importance, the tokenizers used in most LLMs operate with fixed, static vocabularies obtained by running algorithms such as Byte Pair Encoding [Sennrich al., 2016] over aarge-scale, general-purpose web corpora. While this globally optimized vocabulary performs reasonably well on
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Context Window
Static Vocab
Dynamic Vocab
Codebook
0
1
0
0
BCS tb
2
3
0
2
3
0
2
4
7
B
2
be
Decoding step
Inoe
Continuously growing
5
0
2
4
7
8
10
10
4
noX [t]
4
1o Oe
1
3
0
2
4
15
9
10
11
12
8
8
12
not
0
0
10
11
12
13
14
LOS
LZW
Continuously growing
0
3 or
Special
BOS
E0S
BOS
nOT
10
be 0r not EOS
Hypertatien
Teat
X
Erepty
Figure 1: zip2zip inference process. At each decoding step, the model has growing context composed of both base tokens (blue) and hypertokens (green). The static vocabulary of size remains fixed, while the dynamic vocabulary is continuously expanded by merging co-occurring tokens using LZW compression. The codebook (right) maps hypertoken IDs to their corresponding hase tokens. As decoding progresses, new hypertokens created step (e.g.. "to be", "or not") become immediately available for reuse at step r + 1. Hypertokens are also eligible for merging, enabling the formation of nested hypertokens. The final output sequence (bottom) is reconstructed via LZW decompression.
average, it often fails to adapt to domain-speciff Petrov et al., 2023], where the text distribution diverges significantly from the pretraining data. The resulting mismatch leads to longer token sequences, increasing both memory and compute demands, as well as the end user's cost by a factor of 2-3x when processing domain-specific text [Ahia et al., 2023]. To mitigate this issue, prior work has explored expanding the token vocabulary during domain or language adaptation to improve tokenization efficiency [Wang et al., 2019, Zhao et al., 2024, Kim et al., 2024, Liu al., 2023, 2024a]. While effective, this approach needs be repeated for each target domain or language and requires maintaining separate tokenizers. Mcanwhile, commercial LLM providers trend toward increasing the size of token vocabularies growing from 32K to 128K [Grattafiori et al., 2024] and even up to 200K [Abdin et al., 2024] tokens- to improve overall tokenization efficiency. However, prior work [Dagan et al., 2024, Liang et al., 2023] shows that simply enlarging the vocabulary yields diminishing returns in domain adaptation, and vocabularies past a certain size can potentially degrade model performance [Liang et al., 2023]. These limitations point to a compelling need for an adaptive tokenization mechanism one that can dynamically tailor the vocabulary to the input text at inference time, without retraining the model or maintaining separate tokenizers. Such a mechanism would allow the model to construct new domain-specifee tokens on-the-fly, so as to enhance tokenization efficiency. However, adaptive tokenization poses architectural challenges, as both the embedding layer and the language modeling head in transformer models (Vaswani et al., 2017] are static matrices tied to a fixed vocabulary size.
ianguag--specffie distributions [Ahia et al., 2023,
In this paper, we propose zip2zip (with hat-tip to seq2seq [Sutskever et al., 2014)), a novel building block that brings inference- time adaptive tokenization LLMs. zip2zip comprises three key components: (1) LZW tokenizer: A tokenizer that integrates the Lempel-Ziv-Weldde compression algorithm on top of Byte Pair Encoding (BPE) [Welch, 1984]. By applying the L.ZW compression merging frequently co-occurring token sequences into reusable longer tokens (hypertokens) the resulting tokenization becomes less fragmented and more compact. (2) Dynamic-embedding architecture: An augmentation of the transformer architecture with a lightweight encoder that replaces the static embedding matrix, allowing the model to compute embeddings for newly formed hypertokens on the fly. (3) Pretraining under online token compression: a variant of causal language modeling that trains the model directly on sequences, aligning learning with the inference-tine time (hyper)token distribution. The overall process is illustrated in Figure 1, which shows how the context window, dynamic vocabulary, and codebook evolve together during decoding. The name zip2zip reflects its dual role in achieving compression of both the input tokens (the first zip) and output tokens (the second zip), thereby jointly improving the efficiency of input encoding and output decoding. We conduct continued pretraining on Phi-3-4B and Phi-3. 5-14B to support zip2zip using as few as 100M tokens. The resulting models demonstrate strong inference-time compression capabilities across various domains, achieving 15-40% reductions in sequence length and up to 40% improvements in end-to-end latency.
algorithm to the base token
P LzW is the algorithm used in the ZIP compression tool, which inspired the name zip2zip.

---

To make it easy to upgrade existing LLMs to zipZzip, we release an efficient, open- spen-source im- plementation of the training and inference stack. It includes (1) fast Rust-based LZW tokenizer, (2) drop-in model architecture compatible with HuggingFace Transformers, (3) training pipeline for L.Z.-compression-baeee finetuning. Existing L.LMs can be seamlessly extended with zip2zip, gaining adaptive tokenization capabilities through parameter-effiieet finetuning.
2 zip2zip
2.1 Dynamic Token Vocabulary
To enable dynamic tokenization at inference time, we associate the LLM with a hyper-vocubulary Vn that augments the model's static token vocabulary. Tokens from the original vocabulary V are referred to as base tokens. Each entry in the hyper-vocabulary is a hypertoken, representing merged sequence of base tokens. The total vocabulary for a zipZzip model is the union VUV, At the beginning of each inference session, V/ is initialized as an empty set, and is incrementally populated during decoding by identifying and merging recurring token subsequences in the context window, as illustrated in Figure 1.
Continuous Vocabulary Expansion. As decoding proceeds, zip2zip continuously merges occurring tokens into new hypertokens and recursively applies merging on new hypertokens. This continual expansion allows the model represent longer, recurring sequences of base tokens compactly. Hypertokens are treated as first-class tokens within the model, used interchangeably with base tokens throughout the decoding process. Importantly, this process occurs entirely during inference, without modifying the underlying tokenizer or requiring model retraining.
LZW Algorithm. We implement vocabulary expansion using the Lempel-ZZv-Welch (LZW) compression algorithm- dictionary-based lossless compression method that incrementally builds codebook of variable-length sequences. In our setting, the codebook is initialized with the base token vocabulary V and expands by adding new hypertokens on the fly as recurring token patterns encountered. To control the growth of the dynamically expanding vocabulary, impose maximum merge size M that restricts how many base tokens a single hypertoken can represent. L.ZW particularly well-suited for zip2zip due the following properties:
(1) it is online: hypertokens created at step can be immediately reusable at step / + 1; in contrast, methods like BPE require access the full sequence and operate offline;
(2) it is self-contained: input base tokens can be perfectly reconstructed from the compressed token sequence alone:?
(3) it is unambiguous: when both base tokens and hypertokens are available, which one to use is
consistently determined by the LZW algorithm without ambiguity.
2.2 Hyper-Emmedding and
Hypertokens do not have fixed embedding vectors in the original model's embedding layer (and unembedding layer), as they are not part of the original vocabulary. To compute the embedding of hypertoken, we learn a mapping from the base token embeddings to the hypertoken embedding. We achieve this by introducing hyper- encoder, which of the constituent base tokens as input and outputs the corresponding hypertoken embedding (see Figure 2(a)). Specifically, for a sequence of M base tokens yI:M := -1 -....yM the hyper-encoder So VM + Rdd produces the hypertoken embedding h = fo E IR.. where M is the maximum merge size and dl is the embedding dimension. For hypertokens composed of fewer than M base tokens, pad the input sequence to length M. Since the embedding map for base tokens remains unchanged, the hyper-encoder fo essentially maps the concatenated base token embeddings from an (M x d) -dimensional space to d-dimensional hypertoken embedding vector, performing nonlinear dimensionality reduction. For the output unembedding layer, if the underlying transformer ties the embedding and the unembedding matrices, one reuse the same hyper-encoder to compute the representation used for unembedding. Otherwise, a separate hyper-encoder trained to produce the hypertoken unembedding vectors.
neural network that takes the embeddings
'There is no need to persist or transmit the codebook across inference calls, preserving compatibility with existing LLM libraries and interfaces.
3

---

Base token IDv
Base tokens
Recorufrucied
Target
tokens
Labels
Hyper Enooder
Compeessed
Pyee
11
E. Static Emibedding Matri
Crose Cross-Entropp Entropy
or
Loss
Loso
to
Language Modeling Compressed Space
(c) AutoEncoding
Figure 2: (a) Dynamic embedding: Base tokens are embedded via static LM embedding matrix, while hypertokens (e.g.. "to be" or "to be or") are dynamically composed using hyper-encoder over their constituent base tokens. (b) Language modeling in compressed space: The model is trained to predict compressed token sequences produced by I.ZW, optimizing cross-entropy loss over compressed token IDs. (c) Auto-encoding loss: To ensure hypertokens are semantically consistent with their base-token compositions, the model also learns to reconstruct the original base tokens from the hyper-token via a decoding loss.
2.3 Architecture
We illustrate the zip2zip architecture in Figure 3. The input text is first tokenized into base tokens (STEP 1), which are then passed through an on- line LZW compressing module that compresses the token sequence into a stream of hypertokens (STEP 2). Since hypertokens are not part of the model's original embedding layer, their embedding vectors are computed on-the-fly using the hyper-encoder during inference (STEP 3-4). Once embedded, both base token embed- dings and hypertokens embeddings are passed through the standard trans- former layers of the base model, pro- ducing contextualized hidden states (STEP 5-6). This step is identical to vanilla transformer, with hyperto- kens and base tokens treated equally. At the output unembedding layer, hy- pertoken unembedding vectors (same as the hypertoken embedding vectors in the tied case, and computed by a separate hyper-encoder otherwise) are appended to the original unembed- ding matrix in the language model- ing head (STEP 7). This allows the model to compute joint softmax over the union of the base vocabulary and the hyper vocabulary VUV, (STEP 8). The resulting probability distribution is over VUV, and the sampled token may be either base token a hypertoken (STEP 9). In the next cycle, the newly generated token (STEP 10)- whether base or hyper-is appended to the input sequence, and the process repeats (back to STEP 1). At the end of generation, the hypertoken sequence decompressed via the LZW decoding function into a sequence of base tokens (STEP 11-12). The whole process works in a fully autoregressive way, where newly generated hypertokens will also be merged into new hypertokens for future steps. Furthermore, we highlight two points:
Base Tokens 11. LZW Decompress
0%
S.Sampling
Probability Distribution
1/
Joint
Taken
Login
V, V/
Linear Projection
12
N
Matrs
Dynartic Uneebeddiing
Output State
5. Forward Pass
Lx Transformer Layers
4. Token
Embeddings
hyper)
Hyper Cacoddr
Fetch Embedding
Dynomic
Compress
Neww
Base Tokens
Base LM module
token
hypertoken
zip2zip
module
embedding
token
hidden
Figure 3: zip2zip architecture and pipeline. At inference time, base tokens are compressed into hypertokens using LZW. hyper-eneoder computes embeddings for hypertokens, which are processed by the base LLM. Output representations are projected jointly on base and hyper-unembedding layers, producing joint log- its and sampled tokens, which can be decoded back to base tokens.
conpulaaay-ommpiiiiii newly created hyperto- kens-mus be updated a consistent manner across both the input embedding layer and the output unembedding layer, maintaining a consistent view of the hypertoken set. Failure to update both sides
Consistent Vocabulary Updates. The expanding

---

consistently can result in two types of errors: (1) hypertokens that cannot be decoded, or (2) the model attempting to decode a non-existing hypertoken.
Hyper-Embeddiin Cache. Although hypertoken embeddings are computed on-the-fly, they are context-independent and can thus be cached across inference steps. Similar the transformer's KV-cache, this enables incremental updates: only newly created hypertokens need to be embedded each step. Since the codebook grows linearly with the number of tokens in the context, the total cache size also grows linearly in memory. Thus, the computational cost for hypertoken embeddings remains constant per step i.e. one token embedding computed per step.
2.4 zip2zip Pretraining
Objective. Let 'D denote the target text distribution. Given a language model Te parameterized by 0, standard pretraining seeks to minimize the causal language modeling (CLM) objective (see Figure 2(b)), which corresponds to the expected negative log-probabiligy of data sequences under the model:
min E |- log Ne (y)],
where Tg (y) denotes the probability of the token sequence y under the model ng.
Let be an online compression algorithm (e.g.. LZW), and @ be the parameters the hyper-encoder. Given sequence y ~ D, let C(y) be its compressed form. In zip2zip, aim to optimize the same CLM loss, but over the compressed sequences z. The training objective becomes:
min Ey<D 0,0
log N (C6))) min Bx-e()) [- log No, (z)]. 0.6
Here, we slightly abuse the notation let (z) denote the probability assigned the compressed sequence z, parameterized by the base model weights O and the hyper-encoder parameters Φ.
To construct the compressed dataset C(D), we first tokenize the corpus using standard tokenizer, and then apply the LZW compression algorithm. This preprocessing step is performed once prior to training and can be efficiently parallelized through batching. Compression is applied the document level, meaning that each document is compressed independently. This prevents the compressor from learning patterns across unrelated documents.
Parallelizable Training via Causal Masking. Although hypertokens introduce additional vocabulary dynamics, training remains fully parallelizable. We leverage the standard causal masking mechanism used in language models, allowing the model to predict the next hypertoken- at each position in parallel. To eliminate the need for sequential codebook updates during inference, we precompute a fixed codebook by applying LZW compression to the entire input sequence. This precomputed codebook is then used consistently throughout training to condition token predictions, ensuring efficiency and compatibility with standard training pipelines.
a base token or
Auxiliary Reconstruction Loss. We introduce auxiliary reconstruction objective that encourages a hypertoken embedding to retain sufficient information about its underlying base token sequence (see Figure 2(c)). Specifically, the model is trained to reconstruct the original base token embeddings from the hypertoken embedding. We jointly optimize the language model and the hyper-encoder using combined loss that includes both the standard next-token prediction loss and the auxiliary reconstruction loss. Formally, we optimize:
min Ey-D [- log no.o o + + EyLM (y1as (Fo())))) G.0.W
where fo : LM + Rdd is the hyper-encoder, fv Rd L is the decoder aiming to reconstruct the corresponding base tokens from their hyper-embeddingg and A VM x yoM + R is the reconstruction loss function, such the cross-entropy loss, between the base tokens y1.M and the reconstructed base tokens fw (fe (YIM)). The hyperparameter À 2 O controls the trade-off between the prediction error of the language model and the reconstruction error of the autoencoder. This joint optimization objective encourages the hyper-encoder to learn a compact d-dimensional manifold embedded in the higher-dimensional (M x d)) space of base token embeddings, while the language model Te. learns to predict the next (hyper)token given the preceding context. The reconstruction loss can be viewed as a form of auto-encoding, where the hypertoken acts as a compressed latent representation and reconstruction encourages the preservation of semantic content and the compression to be lossless.
5

---

Adapting Pretrained Language Models. Retraining large language models from scratch is compu- tationally expensive and often infeasible for most research labs. A more economical alternative is to perform continued pretraining (or adaptation) on existing pretrained model weights. The proposed objectives (Equations 2, 3) integrate naturally into this setup. Parameter-effiient methods such as LoRA [Hu et al., 2022] may also be used, which allow selectively updating parts of the base model weights with minimal computational cost.
2.5 Efficiency Advantage
zip2zip improves efficiency by increasing the average token length, thereby reducing the number of tokens required to represent the same text. This compression applies to both imputs (c.g., prompts) and outputs (e.g.. completions). As result, the model performs fewer computations computations--bott in the at- tention mechanism and the feedforward layers-and, more importantly, requires fewer autoregressive decoding steps during inference. Since the latency of large language models is primarily driven by the cost of sequential decoding, reducing the number of output tokens by n% leads to an approximate 1% speedup in decoding latency, which we will demonstrate empirically in Section 3.6. A more detailed discussion of FLOPs is provided Appendix E for completeness.
2.6 Entropy Invariance under Lossless Compression
Before tumingg to empirical results, analyze whether a lossless compression of the data representa- tion can fundamentally alter the achievable performance of model. We show that for any lossless mapping 8, there always exists corresponding transported model distribution in the compressed space that achieves exactly the same (cross-)entropy as in the original space.
Let X be the original alphabet and Z arbitrary alphabet obtained via lossless compressor g: + z which a bijection onto its image. Denote by Px the true distribution over sequences E X', and by Pe the model distribution on the same space. The corresponding push-forward (compressed-space true distribution Pz and model distribution Py are defined as
P2(z) Px (x(()=()))
Py(x) po (z)),
2 2*
Theorem 2.1 (Entropy invariance under lossless compression). If g is lossless (i.e., bijective onto its image), then the total entropy and cross- cross-entropy are invariant under the transformation:
H(z)) = (Px)),
H(Pz.Py) (Px..P)).
A detailed proof is provided in Appendix G.
The theorem implies that the optimal achievable cross-entropy in the compressed representation identical to that in the original domain: for any model family on X", one can always construct corresponding model family on 2" via push-forward that attains the same likelihood. In practice, the training process can be viewed as attempt approximate this transported model through optimization; however, convergence to the target model not guaranteed (see Section 5).
Experiments
To evaluate the effectiveness of zip2zip, we perform continued pretraining on the Phi-3 models (3B and 14B) within the zip2zip framework. We train a single model on general-purpor corpus and evaluate it across four dimensions: (1) token efficiency, (2) language modeling perplexity, (3) downstream task performance, and (4) inference efficiency. This setup allows us to assess how well zip2zip generalizes to diverse domains without any task- or domain-specific fine-tuning. For perplexity and downstream benchmarks, we use the widely adopted Im-evaluation-hareen framework [Gao al., 2024].
3.1 Training Setup
Rather than updating the full model weights, we adopt parameter-efficient finetuning using LoRA [Hu al., 2022]. addition, we train the hyper-embeelding and hyper-unembeddding modules. We set the maximum merge size to M and use two-layer transformer encoder the hyper-encoder.

---

<Python codeo
<Blomadicab>
<Frencho
potter. comoie tren ontion remosay
Pro peurtry, heoe en decorrtie toete teviled onfureey nodiic
conee Elieel meceent Fant de ves clurrt pour cojer
Jomn ooreea by
la winde pemele dee eedecde the can Toagh we seriiing
treets perssentt wochires dn Herte
ontie wntch
arweing ancter chts
tranitation
aition Ler de and shreed dee proans off
panto, lienaiigg ae will prtteretentt eueties serttarllly
docuare
prentiin
aortd 1000 proeniig conatome
these estgrt
ale starode mince
Figure 4: Phi-3.5-zip2zip output examples. Blue: base tokens. Yellow: hypertokens (composed of base
tokens). Orange: hypertokens (composed 3+ base tokens).
Table I: Examples of hypertokens formed by Phi-3.5-zip2zz across three domains
Code Generation
Blomedical
French
torch
iff
Eiffel
Att
ention
Attention
trans
cribed
transcribed
de
1a
de
1a
Head
Heddd
synth
esis
synthesis
Gust
lave
Gustave
dim
kdin
cell
ular
cellular
conrencé
The loss weighting coefficient À was chosen to be 0.1, as justified in Appendix I. Ablation studies on M and the hyper-encoder architecture can be found in Appendix C. For comparison, we also perform continual pretraining of the base model using LoRA under identical training conditions, serving as a baseline (denoted as Cont. Pretraining the tables). The continual pretraining process is highly efficient, requiring approximately 10 H100-GPU hours for 4B-parameter model and up to 40 H100-GPU hours for 14B-parameter model, using only 0.1 billion training tokens. Interestingly, the reconstruction loss converges to near zero during continued pretraining, indicating that the model can almost perfectly recover the original base token sequences from the hypertoken representations. This highlights the learned compression is highly information-pe clormation--reserving Details of the training setup, compute infrastructure, and dataset curation are provided in Appendices and J.
3.2 Qualitative Examples and Hypertoken Patterns
We present several examples (Figure 4 and Table 1) to provide intuition into how the zip2zip model generates text. We see that the model generates a mixture of hypertokens and base tokens in the output (Figure 4). The hypertoken ratio is as high as 40% in the Python code generation example, and 20% in the biomedical text generation example. Many of the hypertokens correspond to semantically meaningful units or domain-specifie terms as shown in Table 1. For more fine-grained visualization of hypertoken with zip2zip, we provide visualizations of token streams in Figure 10 in the appendix.
3.3 Token Efficiency
Given an input text. and tokenizer, we define the token efficiency 1 == Tokens(x] as the average number of bytes represented by each token (also called compression ratio), where Bytes(x) refers to the number of bytes in the UTF-8 encoding of x. This measures how compactly tokenizer encodes input text-higher values of 17 indicate more efficient tokenization. We evaluate token efficiency using the tokenizers of four 2024], Phi-4 [Abdin et al., 2024], and Gemma-3 [Teamm, 2025]-each associated with a different base vocabulary size ranging from 128K to 256K. Token efficiency measured across five representative domains, sampled from publicly available datasets: code [Lozhkov et al., 2024b], math [LI et al., 2024], chat [Ding et al., 2023], multilingual [Penedo al., 2024], and web [Lozhkov et al., 2024a]. Table 2 shows that applying LZW zip2zip consistently improves token efficiency across all tokenizer and domains. Gains are particularly strong in structured domains like code and math-wil gains of 48% and more over the base tokenizer. Interestingly, models with larger vocabulary sizes do not always achieve better token efficiency, suggesting that simply enlarging the vocabulary size is not sufficient to improve it.
Bytes(x)
[Grattafiori et al., 2024], Qwen-2 [Yang et al.,
7

---

Table Token efficiency (bytes per token) across domains for different tokenizers with and without zip2zip.
Tokenizer
Code
Math
Chat
Multilingual
Web
Llama-3-128k [Grattafiori et al., 2024]
4.1
2.7
5.1
3.8
4.6
+zip2zip
6.3 (+54%)
4.0 (+48%)
6.4 (+25%)
4.7 (+24%)
5.4 (+17%)
(w0n-2-150K (Yang et al., 2024]
4.0
2.3
5.1
3.7
4.4
+zip2zip
6.2 (+55%)
(+61%)
(+25%)
4.6 (+24%)
5.2 (+18%)
Phi-4-200K [Abdin et al., 2024]
4.1
2.7
5.4
4.6
4.7
+zip2zip
6.3 (+54%)
4.1 (+52%)
6.7 (424%)
5.5 (+20%)
5.4 (+15%)
Gemma-3-256K [Team, 2025]
4.4
4.5
+zip2zip
5.6 (+70%)
3.7 (+61%)
6.4 (+28%)
5.4 (+23%)
5.4 (+20%)
3.4 Perplexity
We evaluate the perplexity of zip2zip models on four corpora: Wikitext [Merity et al., 2016], The Pile [Gao et al., 2020], and subsets of Paloma [Magnusson al., 2023]: mC4, multilingual subset of C4, and dC4 (aka C4-100D), subset of C4 spanning 100 domains. Given a token sequence x -1.....-.y and model q, perplexity and byte-level perplexity [Radford et al., 2019, Magnusson et al., 2023] are defined as: PPL := (TT (xx))
1/N Byte-PPL (TT (xx;)) 1/8 PPL where the number of UTF-8 bytes of the text, and 77 denotes the token efficiency (i.e., bytes token). Token-level perplexity depends on the tokenization scheme and is unsuitable for cross- tokenizer comparison. We instead report byte-level perplexity, a vocubulary-agnoste metric that normalizes for tokenization differences. Table 3 (right panel) shows that zip2zip models see modest increase in byte-level perplexity, indicating a slight drop in language modeling performance.
Table Two-shot accuracy across seven NLP benchmarks (left) and byte-level perplexity (L) on four corpora using 1024-token context window (right). Standard deviations (bootstrapped) ~ 0.02 across all tasks.
Model
Method
ARC-c
ARC-c
HS
OBQA
PIQA
WG
GSM8K
Wiki
Pile
mC4
dC4
Phi-3.5-4B
Base
0.60
0.46
0.79
0.75
0.82
1.58
1.79
1.88
1.74
Cont. pretrain
0.60
0.82
0.63
0.47
0.82
0.75
0.40
1.59
1.81
1.88
1.74
zip2zip
0.57
0.83
0.61
0.46
0.82
0.75
0.15
1.69
1.95
2,00
1.82
Phi-3-14B
Base
0.62
0.80
0.70
0.51
0.83
0.76
0.84
1.43
1.72
1.82
1.67
Cont. pretrain
0.62
0.66
0.52
0.87
0.80
0.52
1.47
1.79
1.68
zip2zip
0.62
0.86
0.68
0.51
0.85
0.79
0.25
1.56
1.90
1.96
1.75
3.5 Evaluation on NLP Benchmarks
We next evaluate zip2zip's few-shot performance on real-world tasks. We evaluate on seven widely used NLP benchmarks, including ARC-[Challenge, Fasy] [Clark et al., 2018], HellaSwag [Zellers al., 2019], LAMBADA [Paperno al., 2016), OpenbookQA [Mihaylov al., 2018), PIQA [Bisk et al., 2019], Winogrande (Sakaguchi et al., 2019] and GSM8K [Cobbe et al., 2021]. As shown in Table 3. the model mnntinudd-pretrainet with zip2zip performs similarly to the baseline on most tasks. However, GSM8K, where the primary task involves numerical computation, the model exhibits significant degradation. While token-level operations are already known to be challenging for LLMs [Singh and Strouse, 2024], it is possible that adaptive tokenization exacerbates this effect, though further validation is required to confirm this hypothesis.
Multilinguality. To validate the effectiveness of zip2zip on non-English languages, we evaluate the model on machine translation tasks, including WMT14 [Macháček and Bojar, 2014], WMT16 [Bojar al., 2016]. The results, shown in Table 4, indicate a small performance degradation across the BLEU, CHRF, and TER metrics when using zip2zip. However, the drop is relatively minor, suggesting that the model retains strong multilingual capabilities even in the compressed representation. Additional experiments multilingual QA benchmarks are provided in Appendix H.2.

---

Table Machine translation performance on WMT benchmarks. Scores are averaged across both translation
directions. Standard deviations (approximately 1.0 ~ 2.0) are reported in Table 11 in Appendix H.
Model
Method
WMT14 En-Fr
WMT16 En-De
WMT16 En-Ro
BLEUT
CHRF1 TER.
BLEU
CHRFI
TER
BIEUT
CHRFI
TER
Phi-3.5-4B Base
33.6
58.3
53.0
39.2
63.2
47.9
17.7
45.5
73.4
Cont. pretrain
36.5
61.0
51.5
42.3
65.4
44.9
16.7
45.8
79.7
zip2zip
34.1
59.4
54.5
39.7
64.5
48.0
14.3
44.2
93.5
Phi-3-14B
Base
39.1
62.6
49.3
43.1
65.6
44.1
21.3
51.0
70.5
Cont. pretrain
38.9
63.2
48.8
48.4
70.1
39.8
21.8
52.0
68.3
zip2zip
36.4
62.8
51.2
44.8
68.1
42.9
19.5
50.1
72.9
3.6 Inference Efficiency
zip2zip reduces decoding time by lowering the number of tokens that need to be generated. However, introduces additional FLOPs due to the on-the-fly computation syper-embeddiigs by the hyper- implement hyper-embeddiing caching and optimize the
encoder. To address this overhead, computation using custom Triton kernel. We report separate timings for prefilling and decoding across multiple models, with and without zipziip, in Table 5. As show in Table 5, zip2zip achieves significant speedup in all four settings. Both prefilling and decoding times are significantly reduced, with the most substantial gains observed in the 512+256 setting with the Phi-3.5-4B model. Improvements are significantly stronger on datacenter-gree GPUs like the NVIDIA H100 and more modest on consumer hardware (e.g., Apple MI).
Table 5: Throughput (tokens/sec) comparison of the zip2zip framework against the baseline Hugging Transformers generate and MLX generate implementation. Performance is detailed for prefilling and decode phases various context lengths (first value in column headers) combined with 256-token generation length. zip2zip demonstrates notable throughput improvements, in both the prefilling and decoding phases.
256+256
512+256
10244256
2048+256
Setting
Method
Prefill
Decode
Prefil Decode
Prefill
Decode
Prefil Decode
Handware: Apple MI (I6GB RAM)
Base model
165.0
7.3
211.3
7.5
200.9
7.1
196.6
6.8
Phi-3.5-4B
zip2zip
145.5
7.9
231.4
10.1
189.6
233.8
7.3
Relative %
-11.8%
+7.5%
+9.5%
+34.8%
-6.6%
+3.9%
+18.9%
+7.5%
Handware: NVIDIA H100 80GB GPU
Base model
700.9
56.2
1347.2
54.4
2689.4
52.8
4993.2
53.1
Phi-3.5-4B
zip2zip 936.6
2722.1
79.8
4326.7
61.5
9258.1
61.9
Relative %
+33.6%
49.3%
+102.6%
+46.6%
+60.9%
+16.6%
+85,4%
+16.5%
Base model
724.4
44.6
1356.3
43.8
2328.6
3849.5
42.2
Phi-3-14B
zip2rip
1024.6
54.9
1973.0
61.1
3657.0
66.8
7239.1
46.3
Relative %
+41.5%
+23.0%
+45.5%
+39.5%
+57.0%
+48..1 1%
+88. 19.1%
+9.6%
Efficient LZZW-Tokenizer Implementation. zip2zip introduces additional LZZW compression step during inference and a decompression step at the end of generation. As result, the efficiency of LZW-integrated tokenization important overall performance. To minimize overhead, implemented Rust-based zip2zip tokenizer that outperforms the Python version (see Figure 7) and matches the latency of HuggingFace's fast BPE tokenizer.
4 Related Work
Domain-Adapter Tokenizers. Several works have explored tokenizer adaptation by expanding the token vocabulary to better support specific domains or languages. Zhao al. [2024], Kim al. [2024], Liu et al. [2023, 2024a] adapt the Llama tokenizer to Chinese, Korean, and specialized domains such as mental health and law by adding new tokens. However, these approaches yield fixed vocabulary that does not adapt during inference.

---

Input Compression for LLMs. Prompt compression methods such as gist tokens [Mu et al., 2023], selective context [Li et al., 2023], LLMLingua [Jiang et al., 2023], summary vectors [Chevalier et al., 2023], in-context autoencoders [Ge et al., 2024], and others [Wingate et al., 2022] reduce the context length by lossy compression. While their lossy compression nature enables high compression ratios, these prompt compression methods can only compress input tokens, but not the output tokens, although output tokens typically dominate generation time under low-batch workloads. Lester et al. [2024] propose improving language model efficiency by training LLMs directly on text compressed with arithmetic coding.
Transformers with Dynamic Embeddings. Architecture-we zip2zip employs dynamic embed- ding layer built upon transformer blocks. Similar ideas have been explored in prior work aimed at reducing the computational cost of transformers, including Hourglass [Nawrot et al., 2022], dynamic- pooling transformer [Nawrot et al., 2023], MegaByte (Yu al., 2023], Toucan [Fleshman and Durme, 2023], Learn- Your-Token [TTawani et al., 2023], SpaceByte [Slagle, 2024], ZeTT [Minixhofer et al., 2024], dynamic tokenization [Feher et al., 2025], BLT [Pagnoni et al., 2025], and H-Net [Hwang et al., 2025]. These approaches vary in their model architectures and chunking strategies. Dynamic Vocab [Liu et al., 2024b] is probably the closest in terms of conceptual motivation, as also expands the vocabulary dynamically during generation. The main difference lies in the dynamic vocabulary construction algorithm and the model training procedure.
5 Discussion and Limitations
Beyond L.W.. While we adopt L.ZW for dynamic construction of hypertokens, zip2zip is broadly compatible with any online compression algorithm. Future work may explore alternative schemes that provide different trade-offs between compression efficiency and model performance.
Codebook Management Strategy. The LZW algorithm grows the codebook linearly with the number of tokens in the context window. Empirical results show that only about 25% of hypertokens are reused during generation, leaving substantial room for optimization. Two potential improvements (1) pruning or selective retention strategies to reduce unused entries, and (2) codebook prefilling, which could be beneficial if likely tokens can be speculated ahead of input processing.
Optimization Under Lossless Compression. Since zip2zip employs lossless compression, the achievable performance is theoretically invariant under the transformation: a transported model, as described in Section 2.6, can attain identical likelihood to that in the original space. Empirically, however, we observe a mild increase in perplexity under compression (Table 3), indicating that the trained model does not perfectly recover the transporied model. This discrepancy arises from optimization challenges rather than representational limits-
descent may converge more slowly settle in suboptimal regions due to more complex loss landscape. Understanding this optimization difficulty- how the search landscape changes under compression and whether targeted preconditioning or extended training budgets close the gap- remmrmninn an important question for future work.
6 Conclusion
We presented zip2zip, an approach that brings inference-time tokenization to large language models through online token compression. By combining LZZ-bbased sequence compression with dynamic hypertoken embeddings, zip2zip enables compact, adaptive tokenization with lightweight uptraining and little architectural changes. Across multiple domains and languages. it achieves substantial reductions in sequence length and decoding cost while maintaining strong task performance. These results demonstrate that online token compression can serve as a practical path toward dynamic tokenization, pointing to new directions for efficient and adaptable LLM inference.
Acknowledgements
We would like to thank Miao Xiong for proof-reading the paper and providing valuable feedback. We are also grateful to Emre Kıeıman, Jason Fisner, Barun Patra, Ana-Maria Indreias, Xiuying Wei. Julian Minder, Tiago Pimental, Luca Beurer-Kellnee and Aleksei Kudrinskii for their helpful
10

---

input and insightful discussions throughout this project. Special thanks to Zheng Zhou for providing technical support with the computing infrastructure.
West's lab is partly supported by grants from Swiss National Science Foundation (TMSGI2 2112291199 and Grant 200364),, Swiss Data Science Center (P22_08) _8. H2020 (952215), Microsoft Swiss Joint Research Center, and Google, and by generous gifts from Facebook, Google, and Microsoft.
References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi. Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL pettps://arxeet org.orgg//aoooo htt//bbs/24411 2412. 08905.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 9904-9923, Singapore, December 2023. Association for Computational Linguistics. doi: 118653/112203.3 -emnlp- main. 614. URL https: / /a//aaaaaantooogyooooo org 2023//2222.222 emnlp- main. 614/.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2019.
URL https://apppii... www.:/aisaaaaanniio..oomme.e. wwt.shhhl....oooroopuuueeseeeeee wttg//ooppuSD::22295555
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings the 2016 conference machine translation. In Ondfej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Pavel Pecina, Martin Popel, Philipp Koehn, Christof Monz, Matteo Negri, Matt Post, Lucia Specia, Karin Verspoor, Jörg Tiedemann, and Marco Turchi, editors, Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. (0.18653/11W116-23000 wtw//16-2220 2301/.
URL https:://clanthhlllle
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balean, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL htttps://proceeeinngg neurips.cc. paper
pdf.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. URL https:: pettps://aree //twx/awxvvv//////////eee/aaavva htttps///axxiv.org/bb///333333 httg/bbs/2303.. 12712.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829-3846. Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023.. emnlp-main.235 222. URL httpt:tts///c https:::/aclanthhllggeeooooo///e
org/2023. wttg://2...mmmmmmmain......... emnlp-main.232.. 232/.
Peter Clark, Isaac Cowhey, Oren Erzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL https:://// / /wwp/v.xiiiva///a//aaa////eaewaa orgg/bs/////oooooooo...oo.e 054.77

---

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 2021. URL. abs/2110. 14168.
.org/
Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. Getting the most out of your tokenizer for pre-training and domain adaptation. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. Jrg.. 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029-300 3051, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/w1/2023..mmm emmnppmmin.111.. URL ht attps://aclantiien
0rg/2023, http//22..emmmpmmmal... emnlp-main. 183/.
Darius Feher, Ivan Vulié, and Benjamin Minixhofer. Retrofitting large language models with dynamic tokenization. Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers). pages 29866-29883,, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 17998-891176-21--0 doi: 10.18653/v1/2025 acl- longllon. 1444. URL https:/ hwwp:///llaaantoooo...ooomo oww.lloy..oooooooooooooo 2025. acll
William Fleshman and Benjamin Van Durme. Toucan: Token-aware character level language modeling, 2023. URL httpp:://axxi..or/abbs/////
08620.
Simon Frieder, Luca Pinchetti, Chevalier Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewiez, Philipp Petersen, and Julius Berner. Mathematical capabilities of chat- gpt. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances Neural Information Processing Systems, volume 36, pages 27699-27744. Curran Asso- ciates, Inc., 2023. URL http:htttss///cceettteee 581 (1186888996666649992220000002000 www.aaastss_aaddeeeeeeearrrr.... .df.
neur iss. wwtpppppppppppccpctccc cwtpppperfillleeppeeeeeeeeeeeeee
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https: /tattps:///xiiv... www..o/g//b//..../////
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac' h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https::tp//. /ttp://eeeoo.o/o/e
.wwooddooorgg.o org arggrrcordd// 2608602.
Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference Learning Representations, 2024. URL https: p http////ppeeeeeeeeeeeeeeeeeeeeet wwwp:eeww.eeeee.eeeeeeeeeeee f
Leonidas Gee, Leonardo Rigutini, Marco Ernandes, and Andrea Zugarini. Multi-word tokenization for sequence compression. In Mingxuan Wang and Imed Zitouni, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 612-621, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ org/2023. emnlp- hnipp-iunutry... 58/.
11/2023.emnl--nddustty.5553 URL https: httes:://laaaaaaoaoaa
Aaron Grattafiori, Abhimanyu Dubey, et al. The Ilama herd of models, 2024. URL https: //arxiv.org/abs//24 22/1//..
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL
net/
Sukjun Hwang, Brandon Wang, and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling, 2025. URL https: pettps//aareee /tt:::///xiiviiii
htttp:://arxiv.or//bb////// wwwp://s///b//////// http///bb/2207.07555555 079555.
12

---

Huiqiang Jiang, Qianhui Wu, Chin Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Meth- ods Natural Language Processing, pages 13358-13376 Singapore, December 2023. Asso- ciation for Computational Linguistics. doi: 10. 10.186533/ //aclanthology. org.looo..ooooooooooooooo wrgg/023. wttp///2.eemmmmmmmma...... 825.
URL https:
Juyong Jiang, Fan Wang, Jiasi Shen. Sungju Kim, and Sunghun Kim. A survey on large language models for code generation, 2024. URL https: httpps////ari.ooo/a/////////// wwwp://////bb////55/// 00515.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL httpp:///arti:i org hrrggabbs2000... 08361.
Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. Efficient and effective vocabulary expansion towards multilingual large language models, 2024. URL https://aaxiis org/ wwwgg/as///2.....// 14714.
Brian Lester, Jachoon Lee, Alexander A Alemi, Jeffree Pennington, Adam Roberts, Jascha Sohl- Dickstein, and Noah Constant. Training LLMs over neurally compressed text. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreviert wwwpreeiiiiinne/rruuin id=pRvhMSV48t. Featured Certification.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https: http:///wminaaaaaa11..111eeaeeee (https: /gi thub. comm/project-numina pdf), 2024.
//huggingface wwwroress-prizebbloooooooo
t/ httportttttminmnanaaaaeeee
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 6342-6353, Singapore, December 2023. Association for Computational Linguistics. doi: 10. URL https: htt///aallaaaaa acloanoonnoooyyyyo org/2023. 1ts//22.e.mmppmaaa.... momlp-mminn.. 391/ 391/.
0:663//1/2022..emmp-mmaaa
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 13142-13152, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/11/2023. emnlp-main.8113
httpp:://cclatthol.ggg hwtp:/lloooyooooo//2// org/2023. emnlp-main. 813/.
Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A Smith, and Yejin Choi. SuperBPE: Space travel for language models. In Second Conference on Language Modeling, 2025.
URL https:tt////t http:s:://axxi.orrr org/ www..or/g/bbb///0/0/ 2503. 110003..1123.. 13423.
Chengyuan Liu, Shihang Wang, Lizhi Qing, Kun Kuang, Yangyang Kang, Changlong Sun, and Fei Wu. Gold panning in vocabulary: An adaptive method for vocabulary expansion of domain-specific LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7442-7459. Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 18.18//3/v1/ emnlp-main. 8024.emnlp-main.o 424.. URL https: http::///alaaaahhhh/oaa wtt..///.......mpmmi... 4244.
ww..oogg///// 2024.
Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, and Rada Mihalcea. Task- adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15284, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2223.cmm main.944. URL https:///ccles https://aclaantholgg.. org/ 2023. emnlp-main. 944/.
Yanting Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Xiaoling Wang. Generation with dynamic vocabulary. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference Empirical Methody in Natural Language Processing, pages 18931-18945 Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 18653/ v 100.24..mnlp-min..ue 1053. URL https: / http//aallaaaaa awwannholooooyyoooooooooooooollt org/ 2024. emnlp-main. 1053/.
13

---

Jonas F. Lotz, Hendra Setiawan, Stephan Peitz, and Yova Kementchecljbieva. Overcoming vocabulary constraints with pixel-level fallback, 2025. URL htRL.tttss//aaxx...r//aabb/////
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the http://ggggnnaaac..oooottaaeeeea
finest collection of educational content, 2024a. URL FaggiingaaeeF/fineewee
Anton Lozhkov, Raymond Li, et al. Starcoder 2 and the stack v2: The next generation, 2024b.
Matouš Macháček and Ondfej Bojar. Results of the WMT14 metrics shared task. In Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia, editors, Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293-301, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.31 11t111//1///// 14-3336. URL https: httpp::/aalaatthlllee/ org//1133333366 4-3336/.
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, A. Jha, Oyvind Tafjord, Dustin Schwenk, Pete Walsh, Yanai Elazar, Kyle Lo. Dirk Groeneveld, Iz Beltagy, Hanna Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. ArXiv, 266348815.
2023. URL https://app semanticscholar Cus
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL http:///awii.ooooab/////osso
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii, editors, Proceedings of the 2018 Conference Empir- ical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/1/DD18-1266) URL https://acllanthol.g. org/D18 S 1260/.
Benjamin Minixhofer, Edoardo M. Ponti, and Ivan Vulić. Zero-shot tokenizer transfer. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems, volume 37, pages 46791-46818. Curran Asso- ciates, Inc., 2024. URL https::/prooceediinn neur ips httpppaaarle//pppaeaappp wtt.e///ppaeee///////L.. ceecee werffeereeeeee.m pdf.
Jesse MU, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 19327-19352. Curran Asso- http:/apppffe//ppaaeeeee wtt.ee//paae//22///Ii ciates, Inc., 2023. URL httpttt::///ccceediiiiee neuri pdf. 3d77c6dcc7f1 43aa21 httpee//aeeeeeeeeeee-e--reeeeeee
Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Marine Carpuat, Carre-CCaheernee de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Findings of the Association for Computational Linguistics: NAACL 2022, pages 1559-1571 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/11 wwt.2222...nnnsss....mmama naacl. 117. http://aaaaattttt www.acoootnnoooocoooooooooo org/ 2022. findings-naacl. 117/.
URL httpts://saaaaattetttttt
Piotr Nawrot, Jan Chorowski, Adrian Lancucki, and Edoardo Maria Ponti. Efficient transformers with dynamic token pooling. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6403-6417, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1/003//w//2023..cloooogg... acl-long. 353/
URL httpss:/acclattholl.g or 2023.
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 medical challenge problems, 2023. URL https://arxiv.... www.:/o.g//////eee 2
Abraham Toluwase Owodunni, Orevaoghene Ahia, and Sachin Kumar. Flexitokens: Flexible tokenization for evolving language models, 2025. URL https::/aaaae http//aiiii.oog/b/////51//// 12720.
14

---

Artidoro Pagmoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret
Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srini Iyer. Byte latent transformer: Patches scale better than tokens. In
Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad 'ahee Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pages 9238-9258, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 1799-889916-2255 //aclanthology wwwpoolooyoogg/////ee 20wgg/22222.22 acl-long.............. 453/.
doi: 10. 10.18653/v 1/201/..cl-2/oo//// acl--long.455555 URL https:
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v 1/44, 144. URL https: httpt::ttccllaeeae hatps////laaaaaloo///oooo htt.aaannooooooooo//111111111 org//161144/
Guilherme Penedo, Hynek Kydličck, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. Fineweb2: A sparkling update with 1000s of languages, 2024. URL hetttts://uuggigggee http://gggggggae....oooaaattttta /twpaa/ttssssttggggggeeee//////e
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Tom, and Adel Bibi. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems, 2023. URL pettps//aardee wwwpii.oooooooooooooooooeooooee. orgooogg//oooo 2305. 15425.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models unsupervised multitask learners. OpenAl blog, 2019. URL https://api. semanticscholar. org/ Drg//oorpsII:: 60025533.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande. Communi- cations of the ACM, 64:99 106, 2019. URL https: wetpiissmmtticchhoo..oooo org/CorpusID: 198893658.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.1188///111--11 1162.
URL https: hatp:::////llanhhologyy hww.hhoooog.oooog/P....oo org/P16-10 1161//
Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer, Lili Yu, and Chunting Zhou. Cat: Content-adaptive image tokenization, 2025. URL https://arxiv .org/
abs/2501. 031220.
Aaditya K.
Singh and DJ Strouse. Tokenization counts: the impact of tokenization on arithmetic in frontier Ilms, 2024. URL https://arxiv.orrg org/ wwwp://///aaa/////e/ee /402. 14903.
Kevin Slagle. Spacebyte: Towards deleting tokenization from large language modeling. In Proceedings of the 38th Conference Neural Information Processing Systems (NeurIPS 2024), 2024. URL https://proceediing neurips wttpeuriip.cppppppeeeeeeeppmpeee cc/paper__pppeeeecceepeeee htcpc/ppprfeeee//eeeeeepp wtt.es//ppae//222//iL... e1f418450
pdf.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 28th International Conference Neural Information Processing Systems- Volume 2, NIPS' 14, page 3104-3112 12, Cambridge, MA, USA, 2014. MIT Press.
Gemma Team. Gemma technical report, 2025. URL https: https://arxx...r//abbs/////
19786.
Avijit Thawani, Saurabh Ghanekar, Xiaoyuan Zhu, and Jay Pujara. Learn your tokens: Word- ford-p tokenization for language modeling. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguisties: EMNLP 2023, pages 9883-9899 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1/223 2023. ttt.22222..nna... findings-emnlp. 662/
findings-emnlp.662. URL https://aclanthhtt
15

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Neural Information Processing Systems, volume 30. Curran Asso-
Guyon,
editors, Advances httptttttt:::/eeeeee http:///oooeeee...nee.ee www.neuriiscomm wttp:cccppppeee_e (eaper_files/ppprr/211111 ciates, Inc., 2017. https:
3f5ee243547dee91 F c4a845aa-
pdf.
Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu. Improving pre-trained multilingual model with vocabulary expansion. Mohit Bansal and Aline Villavicencio, editors, Proceedings of the 23rd Conference Computational Natural Language Learning (CoNLL), pages 316-327, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/
K19-1030. URL. https: /atp::///lllltata http///cccaaooloo...ooooo hrgpe19rrg/1119// 9- 1030/.
Shumin Wang, Yuexiang Xie, Bolin Ding, Jinyang Gao, and Yanyong Zhang. Language adaptation of large language models: An empirical study on LLaMA2. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31 International Conference on Computational Linguistics, pages 7195- 7208, Abu Dhabi, UAE January 2025, Association for Computational Linguistics. URL https: //aclanthie //aclanthology..o org/ coling- molling-min.... main.480/.
Welch. A technique for perforrrformecc data compression. Computer, 17(6):8-19 19, 1984. doi: 10. 10.9/....9..........
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. Yoav Goldberg, Zoriisaa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Lin-
guistics: EMNLP 2022, pages 5621-5634, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.1888/1/2202 https://aclantholl org/2022. findings-emnln emnlp.e12222 412/.
http:////2...nnnseemmmiiie findinng---mmlp.422. URL
An Yang, Baosong Yang, al. Qwen2 technical report, 2024. URL https://arxiv.org/ //bs//////// 2407.10677
Lili Yu. Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), 2023. URL https:/ http::/..ccciittaa///////eeeeee
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4804 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1472. URL htttps://aclanthhtt http://cccaanttoooooooo///////// wrg/p19-11///. 1472/.
Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. Llama beyond english: An empirical study on language capability transfer, 2024. URL /tttps://arxiv.... abs/2401.01055
A More Related Work
Wang al. [2025], Liu et al. [2024a] conducted studies how effectively expand the vocabulary by better selecting the subset of tokens to add. Multi-Word Tokenizer [Gee et al., 2023] and Su- perBPE [Liu al., 2025] demonstrated that allowing forming tokens beyond word boundaries BPE vocab learning helps to achieve more compact tokenization and even improves model performance. Content-Adaptive Tokenizer (CAT) by [Shen et al., 2025] introduces dynamic image tokenization approach that allocates tokens based on content complexity, achieving improved reconstruction quality and efficiency compared to fixed-size tokenization methods. [Lotz et al., 2025] propose pixel-level fallback encoder that bypasses subword vocabulary limitations by rendering text as images, enabling vocabulary- representations that improve multilingual performance and efficiency in pretrained language models. The FlexiTokens [Owodunni et al., 2025] introduces learnable, byte- level tokenizers that dynamically adapt token boundaries to new domains and languages, reducing vverfframmentttion.
16

---

B Discussions on Merge Size
B.1 An Upper Bound on Merge Size
Proposition B.1. Let T be an input sequence of length N over a finite alphabet. LZW compression algorithm merges substrings by identifying and replacing the most frequent substrings with new symbols, iteratively. Then, the size M of the largest merged unit (i.e., the longest substring created via merging) is bounded above by O(vN).
Proof. Assume that the largest merged unit has size M. This implies that there exists least one merge at level M involving a substring of length M. Furthermore, due to the nature of merge-based algorithms, any merged unit of size k must be composed of previously merged units of smaller sizes (e.g., from sizes k I and I, or similar). Hence, in order to construct a merged unit of size M, the algorithm must have previously created all merged units of sizes through 1-
Thus, the existence of a merged unit of size M implies the existence of merged units of every size such that 1 $ k < M. Each such unit must occur at least once in the input sequence in order to be merged.
Therefore, the total number of characters in T must be at least the sum of the lengths of all merged units from size to M, i...
M(M+1 +1)
N
2
This implies:
M = O(VN).
Thus, the length M of the largest merged unit bounded above by O(vN).
B.2 Relation Between Merge Size and Compression Rate
Definition B.1 (Compression Rate). We define the compression rate as the ratio between the number of tokens after compression (Ncomp) and the number of tokens in the original uncompressed text (Nong), expressed as percentage:
Ncomp
Compression Rate
x 100%.
Norig
A lower compression rate indicates greater reduction in loken count, and thus more effective pression.
The last column of Table shows how the maximum merge size affects compression rate when the context window length is 2048. As M increases, compression rate improves significantly, especially from M = M 3. 3. Beyond that, gains diminish, suggesting M 3 strikes a good balance between efficiency and compression rate.
Table 6: Effect of maximum merge size (M) byte-level perplexity and compression rate. Perplexity is measured for Phi-3.5-4B across four corpora with 1024-token context window. Compression rate is evaluated over the training corpus with a 2048-tole token context. M corresponds to no compression.
M Wiki Pile mC4 dC4 Compression Rate( %)
1.62
1.70
2.00
1.91
100.00
2
1.96
2.21
2.55
2.22
75.30
1.72
1.84
2.15
2.00
71.21
1.71
1.84
2.14
68.93
5
1.72
1.84
2.14
1.99
68.41
Interestingly, the relationship between maximum merge size and training loss in Figure 5 well as perplexity in Table 6 non-monotonic. The baseline case with M (i.e., no zip2zip compres- sion) yields the lowest perplexity overall, which is expected and consistent with prior findings that
17

---

Table 7: Ablation of hyper-encoder architecture on byte-perplexity (4) across four corpora using a 1024-token
context window. Performance improves with increasingly expressive architectures.
Model
Method
Wiki Pile mC4 dC4
Phi-3.5-4B averaging
1.81
1.97
2.29 2.08
1-attention-layer
1.73
1.86
2.16
2.01
1-transformer-laaee
1.71
1.83
2.13 1.99
2-transformer-layer
1.72
1.84
2.15
2.00
compression typically incurs a trade-off' in model performance. Among the compressed settings, the case M performs the worst, with noticeably slower convergence and higher final loss. In contrast, the case M = 3 achieves the best performance within the compressed configurations, striking favorable balance between compression and prediction performance. While M 4 and M also perform reasonably well, they exhibit slightly higher loss than M = 3, suggesting diminishing returns or possible over-compression at larger maximum merge sizes (see Figure 5).
Merge Size#2
Merge Sizen1
Merge Size
Merge Size#4
Merge Size=3
20
15
10
5
Step
1k
2k
3k
4k
Figure 5: Effect of maximum merge size M on zip2zip training loss: = (no compression) achieves the lowest loss overall. Among compressed settings, M = performs best, while M 2 shows the worst convergence. Larger M (4 and 5) yield slightly worse results than M 3.
Table 6 reports the byte-level perplexity across four corpora using a 1024-token context window. The results align closely with the training loss trends observed earlier. Setting M = I (i.e., no compression) consistently achieves the lowest perplexity across all datasets, reaffirming that compression introduces performance trade-off. Notably, M 2 performs the worst across all corpora, exhibiting the highest perplexity values. For merge sizes M 3, MM 4, and M 5, perplexity scores are nearly identical, suggesting that moderate compression can be achieved without significantly sacrificing language modeling quality-providee
= avoided. This consistency across loss and perplexity metrics further supports the robustness of maximum merge size M = 3 as the most effective trade-off point.
C Discussions on Hyper-Encoder Architecture
C.1 Hyper-encoder architecture
We ablate the architecture of the hyper-encoder to evaluate its effect on language modeling perfor- mance, as shown in Table 7. We compare increasingly expressive architectures, starting with simple averaging method that introduces additional parameters. This baseline yields the highest perplexity, highlighting its limited capacity. Adding a single attention layer significantly improves performance, and further gains are observed with l-layer transformer encoder. The 2-layer transformer offers marginal additional benefit, suggesting that a lightweight transformer (1-2 layers) is sufficient for effective hypertoken modeling.
Figure illustrates the effect of hyper-encoder architecture on zip2zip training loss. We observe that the simple averaging method converges the fastest but plateaus at relatively high loss, reflecting its limited capacity. model complexity increases- with attention and transformer layers- the convergence becomes slower, yet the final loss is significantly lower. Notably, the 1-layer and 2-layer
18

---

transformer encoders yield the best performance, demonstrating that additional parameters enable the model better capture structure, albeit at the cost of slower training dynamics.
Averaging
1-Attention-Layer
Laye 1-Transformer-Layer
25
20
15
10
Step
1k
2k
3k
4k
Figure 6: Effect of hyper-ecodder architecture zip2zip training loss. Averaging (no additional parameters) converges quickly but higher loss. As architectural complexity increases- from attention transformer layers convergence becomes slower, but the final loss is lower. This highlights trade-off between training speed and modeling capacity.
D Discussions on Compression
Table 8: Token statistics for Code Generation, Biomedical, and French QA domains.
Stats
Code
Biomedical
French
Original Seq Len
344
322
399
7ip27ip Seq Len
367
Num Hypertoken
44
35
26
Hypertoken Ratio
0.166
0.130
0,071
Compression Rate
0.770
0.839
0.920
Table shows detailed token statistics on illustrative examples across three domains Figure 4, highlighting zip2zip's ability to reduce sequence length and introduce reusable hypertokens with domain-specific efficiency.
Table 9: Sequence length reduction (%) across domains, inferred from the inverse of token efficiency gains in Table
Tokenizer
Code
Math
Chat
Web Multilingual
Llama-3-1222 128K
34.9%
32.5%
20.3%
19.1%
14.8%
Twenn2--150
35.5%
37.8%
20.3%
19.6%
15,4%
Phi-4-200K
34.9%
34.1%
19.4%
16.4%
13.0%
Gemma-3-256K
41.1%
37.8%
21.9%
18.5%
16.7%
Table 9 reports the estimated sequence length reduction across domains, showing that zip2zip consistently shortens token sequences by 13-41%, with the strongest gains observed in structured domains like code and math.
E FLOPs Estimation for zip2zip
Following the assumptions of Kaplan et al. [2020], we estimate training FLOPs (T) as:
I xz 6 Nokens Nparams
where Nyokens is the total number of processed tokens and Nparams the number of trainable parameters. This estimate ignores the quadratic attention cost, assuming:
19

---

12 dmodel & sequence length.
For zip2zip, this becomes:
F22, 22 6- Niokens p Nparams α),
where p is the compression ratio, and a accounts for the overhead of the hyper-encoder applied at the embedding and LM head. The relative FLOPs ratio is then:
F22 (1+ r
Assuming the hyper-encoder mirrors the base model's configuration, we estimate:
IM a =
L
where is the number of hyper-encoder layers, is the maximum merge size, and is the number of base model layers. We illustrate this estimate across several model scales in Table 10, showing that the relative FLOPs overhead from the hyper-module remains modest (typically under 15%).
Model
L
M
IM
T
Transformer-4B
2
0.14
Transformer-7B
32
2
2
0.13
Transformer-70B
80
3
3
0.11
Transformer-400B
128
3
4
0.09
Table 10: Relative FLOPs overhead from the hyper-module across different model sizes.
Discussions on Tokenizer
Figure 7 compares the tokenization and detokenization latencies across different tokenizer con- figurations. The Base Tokenizer corresponds the standard BPE tokenizer implemented by the Hugging Face tokenizers library. The Rust L.ZW Tokenizer represents the end-to-end latency when LZW compression and decompression are applied on top the BPE tokenization. As shown, this configuration introduces only small additional latency in the tokenization process while leaving the detokenization latency virtually unchanged. The Python LZW Tokenizer, in contrast, exhibits significantly higher latency due to Python's 's runtime overhead. Overall, the results indicate that most of the observed latency arises from the BPE segmentation process itself rather than the LZW compression, suggesting that efficient implementations of compression add minimal overhead to tokenization workflows.
G Entropy Invariance under Lossless Transformations
Theorem G.1 (Entropy Invariance under Lossless Compression). Let g: X X" Z' be a bijection onio its image, and let Py and py denote the push-forward distributions of Px and po. respectively:
(z( Px(g (z)),
P+(2) p (2))
Then the total entropy and cross-entropy are invariant under g:
HP2)) HPx),
H(P2.Py) = HP
20

---

Tokenization Latency
Operation
Latency (millisecands)
40
Base
Rud 12W
Python 12B
Tokenizer Type
Figure 7: zip2zip tokenizer latency (ms) vs. HF tokenizer.
X2
x
22
2"
23
X4
74
8: + 2
Figure 8: Lossless compression mapping (bijective) g (e.g.. LZW) from original sequences X" to compressed sequences Z*
Proof. By definition of the push-forward distribution,
H(P)) = Σ Pog
2-
Σ Px(x (z)) (x)) 2E
Σ
Pog
s(7)
Since g is bijection, we can replace the summation by x' E X", which yields
H(P)) =- Σ x
H(x))
An identical argument applies to the cross-entropy by considering Px (x(x) log (x) as a single function, since the invariance property does not depend the specific form of the function being summed:
H(P2.P7) Σ (z
Hence, both entropy and cross-entropy remain invariant under any lossless (bijective) transformation
21

---

H Additional Results
H.1 Machine Translation
We report standard deviations for machine translation results across WMT benchmarks in Table 11, computed using the Im-evaluation-harnes codebase.
Table 11: Machine translation performance WMT benchmarks (BLEU1, CHRFf, TER.) with standard deviations (+) from bootstrapped estimates. Scores are averaged across both directions.
Method
WMT14 En-Fr
WMT16 En-De
En-Ro CHRF
BLEU
CHRF
TER
BLEU
CHRF
TER
BLEU
TER
Phi-3.5-4B
33.612.1
58.3a14
53.0s17
39.2&1.9
63.2a1.6
47.921.8
17.7&1.
45.5a1.3
73.4a2.4
Cont, pretrain
36.5+2.2
61.0s1.6
51.5s18
42.3+18
65.4+1.4
44.9.1.
16.7&1.
45.8
79.7+2.3
zip2zip
34.1+1.9
59.+1.5
64.541.6
48.0#1.9
Phi-3-14B
Base
39.142.0
62.611.4
49.3s1.9
43.1=2.0
65.641..
44.1a1.7
21.341.5
51.0114
70.5+2.2
Cont, pretrain
38.9+2.2
63.2+114
48.8+1.9
48.4+2.0
70.1+1.3
39.8+1.9
21.8+14
$2.0a1.3
68 28.4429
zip2zip
36.4+2.1
62.8+1.5
51.2s18
44.8+2.1
68.1+1.6
42.9#188
19.5#15
50.1#1.3
72.942.6
H.2
Multilingual QA Tasks
We evaluate Phi-3.5-4B on multilingual downstream tasks beyond translation, including TruthfulQA- HellaSwag, Winograd across five languages (French, Spanish, Russian, Chinese, and Arabic). As shown in Table 12, the base model demonstrates strong cross-lingual generalization. Continued pretraining in the vanilla setting slightly reduces performance, likely due to domain drift, while the zip2zip variant performs similarly. Overall, the results indicate that the proposed multilingual adaptation strategy maintains competitive performance across diverse evaluation benchmarks.
Table 12: Evaluation on Multilingual Tasks in addition Translation. We report results on TruthfulQA-2. HellaSwag, and Winograd across languages (FR, ES, RU, ZH, AR) using 1m-
Model
TruthfulQA-2
HellaSwag
Winegrad
Phi-3.5-41
Base
0,47
0.51
0.46
0.41
0.2
0.61
0.66
0.49
NA
0.44
0.70
KA
0.74
0.74
NA
Cont. Pretrinn Vanilla
0.42
0.47
0.46
0.46
0.43
0.54
0.57
0.37
NA
0.27
0.76
KA
0.70
0.1
NA
Cont. Pretrain. zip2rip
0.41
0.46
0.43
0.55
0.58
0.36
NA
0.27
0.77
NA
0.73
0.64
NA
I Technical Details
1.1 Model and Training Configuration
Pretrained Model: mi -eeerr/u--------iu-u----------
Sequence Length: 1024
Total Batch Size: 32,768 tokens
Learning Rate Schedule: Cosine decay
Learning Rate Range: Max = 3e-4, Min le-5
LoRA rank and alpha value: Both 32
Training Steps: 10,000
Validation Interval: Every 100 steps
Checkpoint Interval: Every 500 steps
Pytorch Model Compilation: Enabled
I.2 LoRA Configuration
Rank; 16
Alpha: 16
Target Modules: qkv_proj,, o_proj, gate_proj, down_proj, up_proj
22

---

1.3 Loss Weighting Coefficient
Since both the language modeling loss and the auxiliary reconstruction loss are formulated as cross- entropy objectives over tokens, they are naturally on a comparable scale. For sequence of N base tokens, the number of hypertokens used in the auto-reconstruction loss is upper-bounded by N, while each hypertoken corresponds to at most M base tokens in the reconstruction target. The loss weighting coefficient 2 primarily serves to fine-tune the relative importance the auxiliary objective rather than to correct for scale mismatch. We set 2 = 0.1, which yielded stable training. We did not perform extensive exploration over the value 2, which we leave as an interesting direction for future work.
1.4 System and Libraries
Hardware: 4 NVIDIA A100-SXM4-80GB GPUs, 64-core CPU (128 threads)
Key Libraries:
PyTorch >= 2.5.0
Transformers >= 4. 47.
Datasets <= 3.1.0
Accelerate >= 0.26.0 0
1,5
Compute Resources
We report the compute resources used for training our models in Table 13. All training conducted
on internal servers equipped with NVIDIA H100 GPUs. We estimate GPU-hours by multiplying wall-clock training time by the number of GPUs used. No additional compute was used beyond the reported experiments; we did not perform parameter grid search, large-scale hyperparameter tuning, or exploratory runs that were excluded from the paper.
Table 13: Training compute resources for zip2zip experiments.
Model
GPUs
Time
GPU Type
GPU-Hours
Phi-3-Medium (14B)
15h 46m
NVIDIA H100 80GB
63.0
Phi-3.5-Mini (4B)
2
7h Om
NVIDIA H100 80GB
14.0
I.6 Evaluation
All evaluations complete within hour on a single A100 GPU.
23

---

Data Mixture
To support effective fine-tuning, construct curated dataset with balanced representation across diverse domains, including code, mathematics, dialogue, general web content, and multilingual text. The final dataset contains approximately billion compressed tokens.
Table 14 summarizes the constituent datasets and their respective proportions. A visualization of the dataset composition and sequence length characteristics is shown in Figure 9.
Dataset
Domain
Portion
Hugg Looo et al., 2024a]
Web
20%
et al., 2024b]
Code
25%
AL-MN//uuninnaattt-1111 20244] al., 2023]
Math
20%
Ilagginngaacct////lttaaaaaaa_a et al., 2024]
Chat
20%
HuggingFaceFW/ http://feeineeeeeeeeeeee
Multilingual
15%
Table 14: Training data composition across domains.
The multilingual subset in fineweb-2 includes the following languages: Mandarin Chinese (cmn__ Hani), German (deu_Latn), Japanese (jpn_Jpan), Spanish (spa_Latn)) French (fra_Latn), Italian (ita_Latn) Portuguese (por_Latn), Dutch (nld_Latn), and Arabic (arb_Arab).
Dats Mottow of 20tf
Ohe lerve Diiiildoor Sepwere longh
Figure Left: Proportional breakdown the fine-tuning dataset across five domains. Right: Cumulative distribution of input sequence lengths per domain (log scale). Code and multilingual data exhibit longer tail distributions, indicating greater variability in sequence lengths.
24

---

K Token Stream Visualization
Casirg contlo att.nttenteen wwt.eettteeeeooooooooo wilee
corrig
No.1
No.
No.2
No.2
andd rax selr.
No.3
No.3
Sell is 1003 Eees indectio
No.4
Lper cttentor sestrall
BESIIO sentrng Rannarin
Shares
Cowlls O SO1 oi set
SS intedaats
No.5
No.5
(0 wil Sorop)
2009))
(a) Default Tokenization of some Python code.
(b) The same code with adaptive tokenization.
Ne.
No.1
oug
precusiing
usd.a to
Faettoaioo for uaun No.3
No.2
No.3
No.2
leteet 1o Thail aino Lmotoae soad
ac cueree wee astrg aitiettira strand bollin
vilow +M8
the treirs incoeds eagarse cone pefore Ue Proteclatton world
sapert the koclom before Dwills endution (1) Thellli Borte typ
AeScil weultas lechussed
prcascin No.5
FroweeN No.5
5
ciffee graccly shog
(c) Default Tokenization of some biomedical text.
(d) The same with adaptive tokenization.
No.1
No.1
of C108 502
IeCLENCIO LE MADE art con
(c) Default Tokenization of text in French.
(f) The same text with adaptive tokenization.
Figure 10: Examples comparing default and adaptive tokenization. Dotted-line frames highlight where the differences are most noticeable.
25

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have clearly stated the claims, assumption and contributions we have made in the abstract and introduction.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have included a section called Discussion and Limitations in the main text.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: While we don't theory results. We have tried to formalize our method with formulas but it's not really theoretical contribution. And we have tried to make it clear with empirical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have described how methods works in details and also provide code and specify dataset and experimental details the supplementary material as well as on this http://iuwwuooome//eddelleeeeeee
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide code and data, well detailed experimental settings, in the supplementary material, as well as on this https://github.com// www.:l/...oo...llleee -aaahppp-ddlabbii22zipp.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: It be found in the code included in the supplemental material; also available on this http:ghhwwb..oomppplllaaiiiiiiii We also include the settings of key hyperparameters Appendix I.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For brevity, the main text we report experimental results in tables, and we leave the corresponding standard deviations in Table 11. For the two-shot accuracy, we report the standard deviations in the caption of Table which are approximately always 0.02.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide detailed information the computer resources in Appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:
ines?
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics. and we confirm that the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
27

---

Answer:
[NA]
Justification:
This work does not involve any societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g.. dep.lymmpoy of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not involve any high-risk components such as pretrained generative models web-scraped data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package, dataset, and models, and included the url links when applicable.
Guidelines:
The answer NA means that the paper does not existing assets.
28

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. cwm/caocaaa/ooooo datasts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided our code package and detailed settings, as well as license, in this github repo https///ithub..ommcpl-- httpbbboococppplllaaiiipppppppp
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
29

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this paper does not involve any LLM as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standary components.
Please refer to our LLM policy (https: //teerrrpp.cc//ooffeeeeee//////p for what should or should not be described.
30


Reviews:
[{"review_id": "S7dZoxOAjk", "reviewer": "Reviewer_eQ7D", "summary": "In this paper, they propose running LZW compression on their input, creating a codebook representing common multi-token sequences they call hyper-tokens. They then train a \"hyper-token encoder\" and decoder that creates an embedding representation of those hyper-tokens. These are then used as input/output of the transformer just like normal.\n\nThe hyper-token encoder/decoder is trained just like a language model, but the model should be predicting the hyper-tokens produced via compression instead of multiple steps predicting the constituent base tokens. They also include a reconstruction loss to ensure the base token embeddings can be extracted from the hyper-token.\n\nThey test the resulting models both as language models and on downstream NLP tasks. There is consistent degradation in performance, especially of language modeling, but they achieve much higher throughput due to the reduction in sequence lengths.", "questions": "1) Did you try using the transformers first layer as the hyper-token encoder? Works like https://arxiv.org/abs/2410.05864 suggest that LLMs already create their own aggregated representation of tokens so it could be a useful starting point. If it works so well that the hyper-token finetuning step is not needed it would really boost the impact of the work.\n\n2) Did you try a setting where the hyper-token encoder is a small MLP with the M base token embeddings concatenated in a long vector? This is what seems to be described in the prose (section 2.2) but all the methods mentioned in the ablation in the appendix seem to be using sequence based models?\n\n3) Did you try a setting where the hyper-token encoder is only trained on general pre-training text and then applied frozen to a new domain? The prose makes it seem like the encoder is always fine-tuned on the target domain, which runs counter to a claim about the difference between this and previous vocabulary adaptation. If the zip2zip hyper net parts could be learned once in a general way and then used on many domains while keeping a large speedup, it would really strengthen the results.\n\n4) When looking at perplexity, did you evaluate the models just in terms of the probability it assigned to the hyper-token from LZW compression, or did you account for the probability based on other segmentations, such as using only base tokens?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n* The idea of the method is clearly explained and makes intuitive sense.\n\n* Applying the compression to each input, so the first time you see a common subsequence it isn't compressed but later instances it is allows them to apply in any setting.\n\n* Their improvements in throughput are impressive, especially on H100's\n\n* Their idea fits nicely within the current state of how LLMs are used, their method makes minimal changes before a standard transformer is used, allowing them to apply it to many models easily\n\n* They release fast implementations of their idea for ease-of-use\n\n# Weaknesses\n* The zip2zip introduces a lot of strange edge cases on what the model is really learning that should be discussed more. It seems like it exacerbates issues around multiple segmentations; this makes the task the LLM is learning feel very non-intuitive. For example, in figure 1, we see the first time \"to be or\" appears it needs to be predicted on at a time, then the next time it should predicted as a hyper token and a base token, \"to be\" and \"or\", then the third time it appears it should predicted as a single hypertoken. The fact the model needs to keep changing which representation is uses for an underlying string as it gets further into it's prediction seems like it could have large effects and should be investigated further. For example, does this large overlap in the meanings of output tokens (the hyper tokens and it's base tokens) result in systematic changes to the score distribution over tokens?\n\n* Average reductions in sequence lengths for various datasets/domains should probably be presented somewhere in a table instead of a general \"20–60% reduction\" statement in the prose\n\n* Some of the prose, especially when comparing to precious work on vocabulary adaptation makes it seem like their zip2zip works in an *inference-only* setting, i.e., that it can be applied without any training at all. Similarly, the prose is confusing w.r.t. what was actually done. Some examples include: the description of the hyper-token encoder doesn't match the models used, it isn't clear if the encoder/decoders are only trained on fire-tuning data or if there is another pre-trianing step, etc.\n\n* Most of their results are based on their method having \"relatively minor\" drops in performance. This should be quantified more. How many more examples are wrong after this drop on each dataset? This is especially important given that on byte-perplexity, the one continuous task they evaluate on, their zip2zip models show consistent drops in performance on all datasets.\n\n* Linear Scaling of codebook size. They discuss this in the limitation section, but now that some models have context lengths up to 1M tokens, this seems like a larger issue than present it as.", "was_revised": true, "final_justification": "Good work, even better with the new results in the rebuttal, but not the \"groundbreaking impact\" strong accept requires. Keeping score of accept.", "rebuttal": "We thank the reviewer for their insightful questions, which we address below and are encouraged to hear that they find the work “interesting” and “ potentially high impact\".:\n\n##  W1: Does zip2zip cause tokenization Ambiguity due to  Multi-View Segmentation ?\n\nThe short answer is no. Expanded vocabulary does raise the concern of *tokenization ambiguity*—though this issue also exists in standard BPE, it’s typically less pronounced. Our use of **Lempel–Ziv–Welch (LZW) compression** offers a key advantage here: it defines a **deterministic compression rule**, where repeated sequences evolve predictably (e.g., “to”, “be”, “or” → “to be”, “or” → “to be or”) as more context accumulates. During training, the model sees exactly these patterns, and we hypothesize that it can **internalize this evolving structure**. So *in theory*, there is **no ambiguity**.\n\nIn practice, the model occasionally generates suboptimal hypertokens—e.g., predicting “to be” and “or” separately instead of “to be or”—but we observe a **strong affinity toward longer, reusable hypertokens**. This behavior does introduce a challenge during evaluation: unlike standard models, where one token sequence maps cleanly to output, multiple compressed sequences can decode to the *same* text, making the space of valid predictions **exponentially large**.\n\n**(Additional Results): Improved Perplexity Under Multi-Way Tokenization**\n\nTo better understand the influence of this problem, we implemented a **multi-view perplexity evaluation** that partially resolves the ambiguity. Instead of consdering only the canonical LZW segmentation, we also consider alternative segmentations that decode to the same text. (We prune some sequences—similar to beam search—to avoid exponential explosion.)\n\nThe results are exciting: when accounting for multiple valid segmentations, the **effective perplexity drops significantly**, recovering **over 50% of the gap** relative to the base model. This confirms that much of the “drop” is an artifact of mismatched evaluation, and we’re thrilled by the potential this opens up!\n\n| **Model**  | **Method**        | **Wiki** | **Pile** | **mC4** | **dC4** |\n| ---------- | ----------------- | -------- | -------- | ------- | ------- |\n| Phi-3.5-4B | Base              | 1.62     | 1.88     | 1.94    | 1.77    |\n|            | Cont. finetune    | 1.63     | 1.89     | 1.94    | 1.77    |\n|            | zip2zip           | 1.71     | 2.02     | 2.04    | 1.84    |\n|    **NEW**       | **zip2zip multi-view** | **1.66**     | **1.92**     | **1.97**    | **1.79**    |\n\n\nAn exmaple on how **multi-view perplexity** works\n\n#### base tokenization\n`[to, be, or, not, to, be, or, to, be, or, not]`\n\n#### canonical LZW tokenization\n`[to, be, or, not, tobe, or, tobeor, not]`\n\n#### non-canonical yet valid tokenization\n- `[to, be, or, not, to, be, or, tobeor, not]`\n- `[to, be, or, not, to, be, or, tobe, or, not]`\n- `[to, be, or, not, to, be, or, to,beor, not]`\n- `[to, be, or, not, to, be, or, to,be, or, not]`\n\nMulti-view perplexity evaluation sums the probabilities across all valid tokenizations that decode to the same output. This gives a more accurate picture of how likely the model generate the ground truth text.  To keep the computation tractable and avoid exponential growth in the number of sequences, we branch over **each hypertoken individually**, rather than enumerating all possible joint segmentations as an approximation.\n\n\n\n## W2. Add a Dedicated Table for Sequence Length Reduction instead of an Averaging number\n\nInstead of Sequence Length Reduction, we reported Token Efficiency (bytes/token) in Table 2 because we believe it offers a more intuitive and interpretable metric. Token efficiency, measured in bytes per token, directly reflects how compactly information is encoded. The reduction in sequence length can be derived from these values, as it is inversely proportional to the efficiency gain — higher efficiency means fewer tokens are needed. We’ll clarify this relationship in the revision and add a dedicated table to explicitly report sequence length reductions. We believe including both will provide a more comprehensive view of model efficiency and thanks for the suggestion.\n\n## W3. If zip2zip is inference-time adaptation, why there is a training step ?\n\nZip2zip is 100% designed as **inference-time adaptation**. The reason we have a training step in the paper is not because zip2zip requires continual fine-tuning, but because we wanted to start from a **pretrained LM** (e.g., Phi or LLaMA) rather than training a model from scratch. Since these base models were not originally trained in the compressed zip2zip space, we perform a **one-time adaptation step** to enable them to operate effectively with hyper-tokens. \n\nImportantly, this is **not domain- or dataset-specific fine-tuning**—it's a one-off process. Once done, users can directly benefit from zip2zip’s **inference-time speedups** without any further training.\n\nLooking ahead, our broader vision is that future language models could be **pretrained natively with zip2zip**, enabling truly **training-free adaptive behavior out of the box**, with no additional burden on end users.\n\n## W4. Minor drop on task performnance but more noticeable drop on perplexity ? \n\nOur response to **W1** already touches on this point. Our previous **perplexity evaluation** was unfairly penalizing **zip2zip** due to the **multi-way segmentation** issue—multiple compressed token sequences can decode to the same underlying text, yet our original evaluation only considered the single **LZW-compressed** sequence, neglecting valid alternatives.\n\nThat said, many **NLP tasks**—especially **multiple-choice QA**—are robust to small changes in **language modeling quality**. These tasks often include **redundant cues** or contain clearly **incorrect distractors**, making them less sensitive to modest shifts in perplexity.\n\n\n\n## W5. Linear Scaling of codebook size \n\nWe thank the reviewer for highlighting concerns around codebook growth and inference efficiency. While it is true that the **full LZW codebook expands linearly** with context length, it is important to note that **vocabulary growth is configurable** in practice. Specifically, zip2zip supports a **maximum vocabulary size**, allowing the system to retain only the **most frequent** or **most recent hypertokens**, rather than expanding to the full set of possible entries.\n\nMoreover, even in settings where the full codebook is retained, the **runtime overhead remains modest**. Hyper-embeddings are **incrementally computed and cached** in a manner similar to standard **KV caching** in transformers, introducing only a small, constant-time cost per decoding step. We will clarify this in the revision and include additional ablation results on runtime behavior with varying vocab sizes.\n\n\n## Q1. Using Transformer’s First Layer as Hyper-Token Encoder\n\nWe thank the reviewer for pointing us to this intriguing idea and associated work. We were not previously aware that the **first transformer layer could be co-opted as a hyper-token encoder**, and we’re excited to explore this direction further.\nThe approach appears promising—especially in the context of **eliminating the need for a separate hyper-encoder**. If successful, this could enable a version of zip2zip that operates **directly on top of an arbitrary pretrained LLM without requiring any adaptation phase**, which would mark an important step forward.\n\n\n## Q2. Did you try a setting where the hyper-token encoder is a small MLP?\n\nWe appreciate the suggestion and agree that using a **small MLP** as a hyper-token encoder is an interesting baseline to consider. Our current hyper-encoder is designed to be **general-purpose**, mapping a `d × M` tensor (i.e., a sequence of token embeddings) into a single `d`-dimensional hyper-token representation.. We felt MLP might be **too weak** to capture compositional patterns,  and it can't handle hyertokens with variable size, so we didn’t explore it.\n\n\n## Q3. Did you try training the hyper-token encoder on general pretraining data and using it frozen in new domains? The prose suggests it’s always fine-tuned on the target domain.\n\nYES! All the expreiment results are using frozen pretrained model without adaptation!\n\nTo clarify: the **zip2zip model**, including the **hyper-token encoder**, is trained **only once** on general pretraining data using sequences tokenized with our **LZW-based zip2zip tokenizer**.  Crucially, we do **NOT** perform any **domain-specific fine-tuning** or adaptation of the hyper-encoder. It is kept **frozen** and applied as-is across all downstream tasks and domains. This is fully consistent with our goal of supporting **inference-time adaptation**, and ensures that the same general-purpose hyper-token encoder can be deployed in new domains without retraining.\n\nWe will revise the text to make this design choice more explicit. We thank the reviewer for raising this important point.\n\n---\n\nWe hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions they might have, and are looking forward to their reply."}, {"review_id": "1oS8OKi3Q0", "reviewer": "Reviewer_tWZT", "summary": "This work aims to improve the inference latency of LLMs by expanding the token vocabulary during the inference stage, thereby reducing the input and output sequence length.", "questions": "1 In Figure 5 and Table 7 of the Appendix, M=2 has the worst convergence and perplexity. This is an interesting phenomenon—do you have any possible explanations for this experimental result?\n\n2  In Appendix D (Technical Details), the sequence length is set to 1024. Would increasing the sequence length affect the effectiveness of the results? Additionally, can the proposed methods help the model address issues caused by overly long sequences?\n\n3 The paper lacks discussion on some works that also focus on adaptive tokenization, such as [1] and [2].\n\n\n[1] Enhancing large language models through adaptive tokenizers NeurIPS (2024)\n\n[2] Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond EMNLP (2023)", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1 The paper is well-structured and presents a high degree of originality by approaching faster inference from the perspective of tokenizer design.\n\n2  It provides extensive experimental results to validate the proposed method, along with sufficiently detailed descriptions of the experimental setup.\n\n3 The authors also release the code.\n\nWeaknesses:\n\n1 On certain tasks, such as GSM8K, there is a noticeable drop in performance.\n\n2  This method introduces several additional components, such as the hyper-encoder and online vocabulary expansion, which may pose challenges for real-world deployment.", "was_revised": true, "final_justification": "The authors have clearly addressed the concerns I raised. The paper is supported by extensive experiments and targets practical problems, offering valuable insights for future research in this direction. Therefore, I am raising my score to 5.", "rebuttal": "Thank you for your thoughtful review and for highlighting the originality of our work and the extensive experimental validation.\nBelow we address your questions:\n\n## W1. On certain tasks, such as GSM8K, there is a noticeable drop in performance.\n\nThis is a limitation we acknowledge in the paper, and we do consider it an important direction for future work to better understand and mitigate this gap.\n\n## W2. Additional components, such as the hyper-encoder and online vocabulary expansion pose challenges for real-world deployment.\n\nThank you for highlighting concerns regarding **deployment complexity** due to the added components in our method. While **zip2zip** introduces a **hyper-encoder** and **online vocabulary expansion**, we’ve carefully designed the system to ensure **minimal integration overhead** and **ease of adoption**:\n\n###  Tokenizer Integration\n- Our compression mechanism is fully integrated with the tokenizer stack.\n- The API for `tokenizer.encode`, `tokenizer.decode`, `batch_encode`, and `batch_decode` remains **unchanged** and **compatible** with HuggingFace’s `tokenizers` library.\n- This allows plug-and-play use without altering preprocessing pipelines.\n\n###  Modular Architecture\n- We provide a custom `DynamicEmbedding(nn.Module)` that encapsulates the complexity of embedding standard and hyper-tokens.\n- The **hyper-encoder** logic is cleanly isolated inside this module, which **replaces** only the **embedding** and **lm_head** layers—**all other model components remain untouched**.\n\n### Codebook Management\n- We introduce a `CodebookManager` module that attaches to the model and acts as a **context manager** for tracking and managing **dynamic vocabulary expansion** during inference.\n- It supports **efficient updates** to the codebook and can operate across batch boundaries.\n\n### High Efficiency\n- Embedding computations are backed by a **custom Triton kernel**, yielding performance **comparable to `torch.compile`**.\n- This ensures that zip2zip maintains **high inference throughput** without introducing latency bottlenecks.\n\n---\n\nIn summary, zip2zip is designed to make it a **drop-in enhancement** to existing model stacks, preserving **API compatibility**.\n\n\n## Q1. Any explanation that smaller merge size leads to worse convergence ?\n\nOur hypothesis is that setting M as a hard maximum merge size introduces an abrupt boundary, which might be difficult for the model to learn. In contrast, with larger M, there are fewer interventions, and the compressed sequences behave more like natural LZW outputs. That said, this remains an unverified hypothesis.\n\n## Q2.Can sequence length go easily beyond 1024 (experiment setting)?\n\nThe sequence length of 1024 used in Appendix D was chosen based on practical hardware constraints—it reflects a realistic maximum context length for our available compute during training. We do not think there is anything in the zip2zip framework that would break with longer sequences. One consideration is the linear growth of the hyper-vocabulary  w.r.t. sequence length, which is not a real issue as we explained in our response to reviewer eQ7D in W5. \n\nRegarding issues caused by overly long sequences: Could you clarify which specific challenges you are referring to? (e.g., memory, latency, degradation in attention quality)\nOne of the core advantages of zip2zip is its ability to reduce sequence length through compression, which might mitigate some problems associated with long contexts.\n\n## Q3. Missing discussion on some works about adaptive tokenization, e.g.\n[1] Enhancing large language models through adaptive tokenizers NeurIPS (2024)\n[2] Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond EMNLP (2023)\n\n\nThank you for the pointers. We actually **already** mentioned  [2] in the related work section, but [1] was indeed not previously on our radar. It appears to be a task- or domain-adaptive tokenizer training method—similar in spirit to [2], but likely more effective. We will include [1] in the related work section as a loosely related approach.\n\nThat said, both [1] and [2] focus on **adaptively constructing** the tokenizer’s vocab during training or post training, which remains fixed at inference time, whereas zip2zip introduces a vocab that is **adaptive at inference time**.\n\n\n# Additional Metric&Result: Improved Perplexity via Multi-Segment Evaluation\n\n\nIn light of the  suggestion from Reviewer 1 regarding segmentation ambiguity during evaluation—we implemented a **multi-view perplexity evaluation** that accounts for multiple valid LZW segmentations that decode to the same text.\n\nThis refinement addresses a key challenge: unlike standard tokenizers, zip2zip can produce **multiple compressed sequences for a single target string**, complicating perplexity measurement. Our new evaluation considers alternative segmentations (pruned similarly to beam search to ensure tractability), and we find that it **substantially improves perplexity**, recovering over **50% of the gap** relative to the base model. This suggests the original drop was largely an artifact of canonical segmentation, and we're excited about the implications this has for future evaluations and model development.\n\n| **Model**  | **Method**        | **Wiki** | **Pile** | **mC4** | **dC4** |\n| ---------- | ----------------- | -------- | -------- | ------- | ------- |\n| Phi-3.5-4B | Base              | 1.62     | 1.88     | 1.94    | 1.77    |\n|            | Cont. finetune    | 1.63     | 1.89     | 1.94    | 1.77    |\n|            | zip2zip           | 1.71     | 2.02     | 2.04    | 1.84    |\n|    **NEW**       | **zip2zip multi-view** | **1.66**     | **1.92**     | **1.97**    | **1.79**    |\n\nAn exmaple on how **multi-view perplexity** works\n\n#### base tokenization\n`[to, be, or, not, to, be, or, to, be, or, not]`\n\n#### canonical LZW tokenization\n`[to, be, or, not, tobe, or, tobeor, not]`\n\n#### non-canonical yet valid tokenization  (multi-view)\n- `[to, be, or, not, to, be, or, tobeor, not]`\n- `[to, be, or, not, to, be, or, tobe, or, not]`\n- `[to, be, or, not, to, be, or, to,beor, not]`\n- `[to, be, or, not, to, be, or, to,be, or, not]`\n\nMulti-view perplexity evaluation sums the probabilities across all valid tokenizations that decode to the same output. This gives a more accurate picture of how likely the model generate the ground truth text. To keep the computation tractable and avoid exponential growth in the number of sequences, we branch over **each hypertoken individually**, rather than enumerating all possible joint segmentations as an approximation.\n\n---\n\nWe hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions they might have, and are looking forward to their reply."}, {"review_id": "Yg13SGeE0p", "reviewer": "Reviewer_UZ7U", "summary": "This paper proposes an adaptation in which the tokens are compressed into dynamic “hypertokens”. The paper evaluates performance changes on a variety of standard tasks as well as translation. The proposed method improves inference efficiency, but does negatively impact performance on some tasks.", "questions": "* Given the massive speedup after zipping, why not speed up training too (either through BLT-like changes or through superword tokenizers)?\n* As math was the domain that showed the greatest drop in performance, could you modify the approach so that hyperwords involving numbers cannot be learned? This could prevent long sequences of numbers from being created that the model cannot work with well. I recommend looking into the number (pre-)tokenization literature [3-5].\n\n[3] https://aclanthology.org/2024.emnlp-main.40/ \n\n[4] https://arxiv.org/abs/2402.14903 \n\n[5] https://arxiv.org/abs/2410.19730  \n\nTypos:\n* L59: Phi-3-4B and Phi-3-14B but in the tables, the 4B model is Phi 3.5\n* L225: needs a period after the close parenthesis\n* L319-320: include → including", "limitations": "No. I think it would be helpful for the authors to address the possibility that this method worsens bias and toxicity (e.g. race and gender bias).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThis paper tackles a novel approach to a topic that has been the topic of recent popular work. In contrast to novel architectural approaches, this method proposes adaptation to existing models that speed up inference. \n\nThe paper is very clearly written. Examples provided in Figure 3 and Table 1 are extremely helpful for getting a sense of the impact of the method. \n\nThe methodology seems technically sound. The authors open source their code so it can be used and built on by others, making it a potentially valuable contribution.  \n\n**Weaknesses**\n\n* Referring to this method as “inference-time” adaptation is somewhat misleading, as there is a necessary training stage with a non-negligible compute requirement. \n* Comparing the zip2zip method only to base and continuous fine tuned models (Tables 2, 3, 4) may not be the most helpful comparison. I think a valuable comparison would be to compare to quantization. This would help contextualize the drop in performance and help the reader evaluate how changes in performance compare. \n* Gains in token efficiency are the highest in code and math domains, but math is also the domain that shows the greatest drop in performance. This could suggest that the method is limited by a strong tradeoff between token efficiency and lower performance. This would have to be much better understood in order for other people to adopt the method. As a result, the claim in L358-360 that “zip2zip maintains strong performance across a range of tasks while achieving significant gains in inference efficiency” should be qualified, as improved inference efficiency seems to be correlated with a greater drop in performance. \n* Table 2 shows the improved token efficiency in chat templates and multilingual data, but there are no evaluations in those domains. The authors should add more evaluations in these domains to help understand the relationship between increased token efficiency and changes in task performance. The inclusion of translation task results does not necessarily speak to multilingual performance, as translation is separate from within-language performance on non-English languages.\n* The authors do not make any comparison to superword tokenizers [1-2]. In practice, the hypertokens look very much like superword tokens, so it would be helpful to draw a comparison and indicate the advantage of zip2zip is over static superword tokenization. For example, if you did some other type of standard vocabulary adaptation to turn a model’s tokenizer into a superword tokenizer, would you expect differences in terms of token efficiency or model performance relative to zip2zip?\n\n[1] https://arxiv.org/abs/2503.13423 \n\n[2] https://arxiv.org/abs/2504.00178?", "was_revised": true, "final_justification": "The authors have addressed several of my concerns. I think the points raised in the rebuttal, if added to the paper, will strengthen the arguments. However I still disagree about the use of 'inference-time' in the core framing of the paper. Therefore, I will keep my score as it was.", "rebuttal": "We appreciate the reviewer’s thoughtful feedback and are encouraged by their recognition of our approach’s novelty, clarity, and potential utility through open-sourcing.Below we address your questions:\n\n## W1. Referring to this method as “inference-time” adaptation is somewhat misleading as there is a necessary training stage\n\nWe appreciate the reviewer’s concern regarding the phrasing of “inference-time” adaptation. To clarify: **zip2zip is explicitly designed for inference-time adaptation**. The one-time training step presented in the paper is **not due to a need for continual fine-tuning**, but rather reflects our desire to start from an existing pretrained LM (e.g., Phi or LLaMA), rather than training a model from scratch.\n\nSince these base models are not natively trained in the compressed zip2zip token space, we introduce a **single adaptation phase** to make them compatible with hyper-tokenized inputs. Crucially, this is **not task-specific or domain-specific fine-tuning**, and once performed, it enables downstream users to **deploy zip2zip without any additional training**, benefiting immediately from inference-time speedups.\n\nLooking ahead, our longer-term vision is for future language models to be **pretrained directly in the zip2zip space**, eliminating even this initial adaptation and enabling **fully training-free inference-time compression** from the outset.\n\nWe hope this clarifies our intent behind the term and the design of zip2zip.\n\n## W2. Comparing against quantization to provide a better understanding on efficiency VS performance \n\nWe view **quantization and zip2zip as complementary techniques** that target different axes of model efficiency. While quantization reduces the **precision of model weights and activations**, zip2zip focuses on **input compression and reducing sequence length**, thereby decreasing the number of decoding steps required. These methods are inherently **orthogonal** and can be seamlessly combined.\n\nIn practice, our implementation **trivially supports quantized models** (e.g., 4-bit and 8-bit), and we have observed that **combining zip2zip with quantization yields additive gains in inference efficiency**—without introducing conflicts or degradation in performance. \n\n## W3: On the Tradeoff Between Token Efficiency and Model Performance\n\nWe thank the reviewer for raising this important concern. We would like to share an **updated result** that directly addresses this point.\n\nFollowing a suggestion from Reviewer 1 regarding **segmentation ambiguity during evaluation**, we implemented a **multi-view perplexity evaluation** that accounts for multiple valid LZW segmentations of a given output string. This refinement addresses a key challenge in evaluating zip2zip: unlike standard tokenizations where a string maps to a single token sequence, zip2zip can yield **multiple compressed sequences that decode to the same text**, making canonical perplexity measurements potentially misleading.\n\nTo resolve this, our new evaluation considers alternative valid segmentations (pruned in a beam-search-like fashion for tractability), leading to a **substantial improvement in perplexity**. As shown below, we recover **over 50% of the original gap** compared to the base model, demonstrating that much of the apparent performance drop was an artifact of rigid evaluation rather than a fundamental limitation:\n\n| **Model**     | **Method**         | **Wiki** | **Pile** | **mC4** | **dC4** |\n|---|---|-|-|-|-|\n| Phi-3.5-4B     | Base                | 1.62     | 1.88     | 1.94    | 1.77    |\n|               | Cont. finetune      | 1.63     | 1.89     | 1.94    | 1.77    |\n|               | zip2zip             | 1.71     | 2.02     | 2.04    | 1.84    |\n| **NEW**       | **zip2zip (multi-view)** | **1.66**     | **1.92**     | **1.97**    | **1.79**    |\n\nThese results indicate that the performance-efficiency tradeoff is smaller than originally reported as multi-view evaluation provides a more faithful view of model capability. That said, we acknowledge that a residual gap remains—particularly on reasoning-heavy tasks—and we view this as a valuable direction for future work. \n\nP.S. An exmaple on how **multi-view perplexity** works\n\n#### base tokenization\n`[to, be, or, not, to, be, or, to, be, or, not]`\n\n#### canonical LZW tokenization\n`[to, be, or, not, tobe, or, tobeor, not]`\n\n#### non-canonical yet valid tokenization (multi-view)\n- `[to, be, or, not, to, be, or, tobeor, not]`\n- `[to, be, or, not, to, be, or, tobe, or, not]`\n- `[to, be, or, not, to, be, or, to,beor, not]`\n- `[to, be, or, not, to, be, or, to,be, or, not]`\n\nMulti-view perplexity evaluation sums the probabilities across all valid tokenizations that decode to the same output. This gives a more accurate picture of how likely the model generate the ground truth text. To keep the computation tractable and avoid exponential growth in the number of sequences, we branch over **each hypertoken individually**, rather than enumerating all possible joint segmentations as an approximation.\n\n## W4. Evaluation on Multilingual Tasks In addition to Translation\n\nWe have conducted additional experiments across **multilingual downstream tasks**, focusing on 5 languages (FR, ES, RU, ZH, AR). These include **TruthfulQA-2**, **HellaSwag**, and **Winograd**, all adapted for multilingual evaluation using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).\n\n\n| Model        | Method         | Truthful-QA-2 |        |||         | Hellaswag     |   ||| | Winograd          |      ||           | |\n|--|--|--|-|-|--|---|-|-|-|-|-|-|-|-|-|-|\n| | | FR  | ES     | RU            | ZH            | AR            | FR             | ES            | RU            | ZH            | AR            | FR             | ES            | RU            | ZH            | AR|\n| **Phi-3.5–4B** | Base     | 0.47     | 0.51         | 0.46         | 0.41         | 0.42         | 0.61     | 0.66    | 0.49     | NA            | 0.44      | 0.70    | NA     | 0.74    | 0.74    |NA    |\n|  | Cont. Pretrain. Vanilla | 0.42          | 0.47         | 0.46         | 0.46         | 0.43         | 0.54     | 0.57   | 0.37    | NA            | 0.27     | 0.76    | NA     | 0.70     | 0.61     |NA  |\n|           | Cont Pretrain. zip2zip | 0.41          | 0.46         | 0.47         | 0.43         | 0.40         | 0.55        | 0.55        | 0.36        | NA            | 0.27         | 0.77       | NA            | 0.73        | 0.64        |NA  |\n\n\n1. Continued pretraining (vanilla) serves as a direct comparison point, as it was trained on the same corpus as the zip2zip variant.\n\n2. The “N/A” entries indicate languages that are not currently supported for those tasks in [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/main).\n\n3. The observed trends are consistent with the main task evaluation table reported in the paper\n\n## W5. Comparison to Superword Tokenizers\n\nWe see **zip2zip and superword tokenization as complementary techniques** rather than competing ones. Superword tokenizers aim to build a **stronger base vocabulary** at training time, improving efficiency by capturing frequent patterns statically. In contrast, zip2zip performs **dynamic, context-aware compression at inference time**, adapting on the fly to the specific patterns present uniquely in each input sequence.\n\nWe agree that a comparison to **superword tokenizers** [1–2] would provide valuable insight for readers, especially given the apparent similarity between hypertokens and subword units with extended granularity. Due to time constraints, we were unable to include this comparison within the rebuttal window, but we see this as a **promising direction for follow-up work**. In fact, we are particularly interested in exploring whether zip2zip can be applied **on top of** superword tokenizers to further boost efficiency.\n\n\n## Q1. Does zip2zip speed up training too?\n\n\nYes—zip2zip can also accelerate training. When run on the same corpus, **zip2zip results in shorter effective context windows** due to improved segmentation and **larger token granularity**, allowing more text to be processed per training step. This leads to **faster throughput** and **reduced compute cost** during training.\n\nAn additional practical advantage is that **zip2zip does not require training from scratch**. As demonstrated in our paper, we perform **continued pretraining** on top of an existing model (e.g., Phi or LLaMA) to adapt it to the compressed zip2zip space. In contrast, methods like **BLT** and **H-Net** are incompatible with existing tokenizers and therefore **require full training from scratch**, which significantly increases cost and limits accessibility.\n\n## Q2. Can we disable merges of digit tokens to improve the performance on math ? \n\nThank you for the thoughtful suggestion. We agree that finer-grained control over numeric token merges could be beneficial, especially given the observed performance drop in math-heavy domains. \n\nAs you pointed out, forming long numeric hypertokens can make it harder for the model to reason effectively. In our current implementation, we already support **two control mechanisms** that help mitigate this issue:  \n(1) a **maximum merge size** that limits how long a hypertoken can become, and  \n(2) a **vocabulary cap** that restricts the number of hypertokens maintained during inference.  \n\nThe max merge size, in particular, can help prevent overly long digit sequences from being compressed into a single token. That said, we agree that **more targeted control—such as disabling merges over numeric spans entirely—would be a promising extension**. We appreciate the references to relevant work on number (pre-)tokenization [3–5], and we plan to explore this direction further in future iterations.\n\n---\nWe hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions you might have, and are looking forward to your reply."}]

Author rebuttal:

Meta-review: The paper explores an adaptive tokenization method aimed at reducing the runtime complexity of language models. The technique provided is mentioned to be original, specifically in that it achieves a speedup by modifying the tokenizer, e.g., UZ7U “This paper tackles a novel approach ... proposes adaptation to existing models that speed up inference.”,  tWZT “a high degree of originality by approaching faster inference from the perspective of tokenizer design”. In addition to this advantage related to the novelty, the impact of the method is also quite strong. The methodology is technically sound (UZ7U) and the ideas are easily applicable to many models (eQ7D). The empirical investigation is sufficiently broad (tWZT), giving convincing evidence to the generalizability of the method. 

The reviewers raised a few concerns, most notably that in some cases the tokenization technique causes a drop in performance that is too significant. The authors mitigated this issue, at least partially, in their rebuttal by providing an experiment measuring “multi-view perplexity” showing that the mentioned performance drop is in fact not as severe. Given this response, the remaining concern appears to be minor. Another notable issue raised is the presentation of limitations. The rebuttal included clarifications that should be integrated into the final version.  

I believe that it might take some effort, but the content of the discussion can be integrated into the paper towards a final version, and the resulting paper will be a good fit to NeurIPS
Decision: Accept (poster)