NeurIPS 2025

zip2zip: Inference-Time Adaptive Tokenization via
Online Compression
Saibo Geng1∗
Nathan Ranchin1∗
Yunzhen Yao1
Maxime Peyrard4
Chris Wendler1,2
Michael Gastpar1
Robert West1
1EPFL
2Northeastern University
4Université Grenoble Alpes, CNRS, Grenoble INP, LIG
{saibo.geng, nathan.ranchin, yunzhen.yao, michael.gastpar, robert.west}@epfl.ch
maxime.peyrard@univ-grenoble-alpes.fr
ch.wendler@northeastern.edu
Abstract
Tokenization efficiency plays a critical role in the performance and cost of large
language models (LLMs), yet most models rely on static tokenizers optimized on
general-purpose corpora. These tokenizers’ fixed vocabularies often fail to adapt
to domain- or language-specific inputs, leading to longer token sequences and
higher computational costs. We introduce zip2zip, a novel method for achieving
context-adaptive tokenization in LLMs at inference time. Leveraging an online
data compression algorithm (Lempel–Ziv–Welch), zip2zip dynamically expands
its active vocabulary at inference time by continuously replacing fragmented token
sequences with more compact hypertokens, which it can immediately output during
generation. zip2zip consists of three key components: (1) a tokenizer based on
Lempel–Ziv–Welch compression that incrementally merges co-occurring tokens
into reusable hypertokens on the fly; (2) a dynamic embedding (and unembedding)
layer that computes embeddings for newly formed hypertokens at runtime; and (3)
a variant of autoregressive language modeling that pretrains the model to handle
hypertokenized, compressed text sequences as inputs and outputs. We show that an
existing LLM can be uptrained for zip2zip in 10 GPU-hours via parameter-efficient
finetuning. The resulting LLM performs test-time adaptation, learning to use
hypertokens in unseen contexts and reducing input and output tokens by 15–40%.
Code and models are released at https://github.com/epfl-dlab/zip2zip.
1
Introduction
Large language models (LLMs) have shown impressive versatility across a...