InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression
Authors: Haotian Ye
Qiyuan He
Jiaqi Han
Puheng Li
Jiaojiao Fan
Zekun Hao
Fitsum Reda
Yogesh Balaji
Huayu Chen
Sheng Liu
Angela Yao
James Zou
Stefano Ermon
Haoxiang Wang
Ming-Yu Liu
Venue: ICLR 2026 Oral (2026)
Keywords: discrete tokenization
video representation
eficiency
information theory
Primary area: generative models
TL;DR: This paper introduces InfoTok, an adaptive video tokenizer guided by information theory, which significantly boosts video compression efficiency and reduces computational overhead without degrading visual quality.

Abstract: Accurate and efficient discrete video tokenization is essential for long video sequences processing. Yet, the inherent complexity and variable information density of videos present a significant bottleneck for current tokenizers, which rigidly compress all content at a fixed rate, leading to redundancy or information loss. Drawing inspiration from Shannon's information theory, this paper introduces \alg, a principled framework for adaptive video tokenization. We rigorously prove that existing data-agnostic training methods are suboptimal in representation length, and present a novel evidence lower bound (ELBO)-based algorithm that approaches theoretical optimality. Leveraging this framework, we develop a transformer-based adaptive compressor that enables adaptive tokenization. Empirical results demonstrate state-of-the-art compression performance, saving $20\%$ tokens without influence on performance, and achieving $2.3\times$ compression rates while still outperforming prior heuristic adaptive approaches. By allocating tokens according to informational richness, \alg enables a more compressed yet accurate tokenization for video representation, offering valuable insights for future research.

Published as a conference paper at ICLR 2026
INFOTOK: ADAPTIVE DISCRETE VIDEO TOKENIZER VIA ONFORMATION-THEORTTI COMPRESSION
Haotian Ye!
Qiyuan He
Jiaqi Han?
Puheng Li
Jiaojiao Fan'
Zekun Hao
Fitsum Reda
Yogesh Balaji
Huayu Chen?
Sheng Liu?
Angela Y
James Zou
Stefano Ermon
Haoxiang Wang*
Ming-Yu Yu Liu
NVIDIA
Stanford University
National University of Singapore
https://research.co
ABSTRACT
Accurate and efficient discrete video tokenization is essential for long video quences processing. Yet, the inherent complexity and variable information density of videos present a significant bottleneck for current tokenizers, which rigidly com- press all content at a fixed rate, leading to redundancy or information loss. Drawing inspiration from Shannon's 's information theory, this paper introduces INFOTOK, a principled framework for adaptive video tokenization. We rigorously prove that existing data-agnostic training methods are suboptimal in representation length, and present a novel evidence lower bound (ELBO)-based algorithm that approaches theoretical optimality. Leveraging this framework, we develop a transformer- based adaptive compressor that enables adaptive tokenization. Empirical results demonstrate state-of oftthe--rr compression performance, saving 20% tokens with- out influence on performance, and achieving 2.3x compression rates while still outperforming prior heuristic adaptive approaches. By allocating tokens accord- ing to informational richness, INFOTOK enables a more compressed yet accurate tokenization for video representation, offering valuable insights for future research.
INTRODUCTION
With the growing success of vision foundation models (Agarwal et al., 2025; Zhang et al., 2025). there has been an increasing interest to model the real world with visual observations, e.g., videos. Representing video as discrete tokens offers clear advantages by aligning seamlessly with LLMs and unifying diverse vision tasks, yet discrete video tokenization remains highly challenging, especially for complex visual content. The main difficulty lies in developing efficient and scalable representations for long sequences, which can easily expand to millions of tokens and become intractable for the transformer architecture (Esser al., 2021; Wang al., 2024; Agarwal al., 2025).
discrete tokenizer typically consists of an encoder, a quantizer, and a decoder. Given a video with a certain duration and resolution, the encoder compresses the original video into sequence of embeddings, which are subsequently quantized into discrete tokens. During decoding, the de- coder reconstructs the video from these token sequences. The training objective aims to minimize the discrepancy between the original and reconstructed videos-as an objective achievable if token sequences are allowed be arbitrarily long (e.g., directly preserving all RGB pixel values). Hence, an effective tokenizer inherently acts as compressor, encoding visual signals into compact tokens while preserving essential information for downstream tasks. Unlike images (Van Den Oord et al., 2017; Mentzer et al., 2023b; Tian et al., 2024; Yu et al., 2024), video signals pose unique tokenization challenges due to their high dimensionality, temporal redundancy, and considerable variability in
Work done as intern at NVIDIA and accepted as Oral presentation at ICLR 2026. Code is available here. Correspond haot a a Saanesstaafoooooooooooo atrnd..d edu. stands for equal contribution.

---

Published as a conference paper at ICLR 2026
Input Video
X
Router
Adaptive Compressor
Encoder
Embeiklings
Quantiee
Cownerccom Applications
Reconstruction
N. Diserete Tokeus
De-quentize
Adaplive De-compressor
Decater
Embeddiags
Erio 0.267
Compression kado (Neessred by 8Phal 8,4004
Overall framework of INFOTOK, maps video x into fixed-length embeddings, from which router estimates the number of tokens Nx based on information complexity (section 3.1). An adaptive compressor encodes the embeddings to Nx tokens (section 3.2). For reconstruction, the tokens are decompressed to fixed-length embeddings and decoded back into video. INFOTOK tokenizes based on video complexity: e.g., the stable dog video is compressed more (0.40) than the dynamic cat-fighting video (0.62). Illustration details can be found Appendix A.
Figure 1:
cottmmtttiionhhooettt adaptive video tokenizer. An encoder
information content (Yan et al., 2024; Agarwal et al., 2025). Therefore, developing an efficient, scalable, and principled video tokenizer is crucial for advancing unified multi-moda. models.
As is well established Shannon information theory (Ash, 2012), compared with content content-agnostic agnostic compression that compresses all data identically, content-dependent compression, which determines the compression ratio of a data sample according to its property, is significantly more efficient. However, regardless of architectures and quantization methods, most existing tokenizers use an identical compression rate for all possible videos (Yan et al., 2021; Esser et al., 2021; Yu et al., 2023a). Consequently, fixed amount of tokens could be redundant for simple videos and insufficient for representing complex videos, making downstream video-undersiee understadnnn generation tasks inefficient or intractable. While recent attempts on flexible tokenization (Yan al., 2024) use heuristic training method that enables users to tokenize a video with varying lengths, prove in this paper that their training method is biased, and their inference via trial-and-error length selection is inefficient. result, we seek answer the following principled and challenging problem:
What is theoretically ideal discrete video tokenizer, and how to train this tokenizer in principle?
To answer the question, this paper draws inspiration from Shannon's 's Source Coding Theorem (Shan- non, 1948) and proves that existing tokenizers, either with fixed compression rate or data-agnostic adaptive rates, are biased in the sense that the expected token length is significantly larger than the optimal length in order achieve identical reconstruction quality. In contrast, tokenizing each video in a difficulty- aware manner, i.e., which is closely tied to the frequency of contents based on information theory, can guarantee a netrroltimal compression rate theory. Altogether, our theoretical results suggest principled way for adaptive tokenization.
Inspired by the theorems, we propose novel cntermatiionthhoorete adaptive tokenization framework, named INFOTOK,, that converts existing fixed-compression tokenizer into an adaptive counterpart. As illustrated in fig. 1, it leverages a router to determine the token sequence lengths based on the evidence lower-bound (ELBO) of the negative log-likelihood of input videos, and a transformer- adaptive compressor to compress fixed-length embeddings into a discrete token sequence with length assigned by the router. By bridging to information theory, INFOTOK produce adaptive token sequences with sequence length conditioned on the information complexity of each frame, thereby reducing redundancy and ensuring each token represents similar amount of information.
2

---

Published as a conference paper at ICLR 2026
Extensive experimental results video reconstruction datasets demonstrate the superiority of INFOTOK. Empirically, our comprehensive experiments show that INFOTOK can save approximately 50% tokens without loss of reconstruction quality compared to state-of-the- fixed-length tokenizers, and it can outperform previous adaptive tokenizers by an average compression rate of 2.3x and number of evaluations (NFEs) by x1 We also conduct ablation studies show that INFOTOK aligns with our theoretical intuition well, and present illustrative examples for presentation.
The main contributions of this paper are summarized below.
We prove rigorously based on Shannon's 's information theory that existing tokenizers with fixed or data-agnostic adaptive compression rates are inherently biased and inefficient.
We propose INFOTOK, information -hoooreii adaptive tokenization framework that leverages an ELBO-based router to dynamically decide compression rate, and a transformer-ban based adaptive compressor to compress fixed-length embeddings into adaptive token sequences effectively. Extensive empirical studies validate the superiority of INFOTOK, demonstrating significantly improved token efficiency without compromising quality.
2
ADAPTIVE TOKENIZATION: THE FRAMEWORK
2.1 DISCRETE VIDEO TOKENIZATION
We consider video data' represented of frames, and H, W is the resolution. A discrete video tokenizer T parameterized by Φ, consists of: 1) an encoder & X r NNN that maps the input video into a sequence of continuous latents h (x) with latent dimension K; a quantizer Q R + 2, e.g., VQ (Gray, 1984), FSQ (Mentzer et al., 2023a), or LFQ (Yu et al., 2023b), that quantizes embeddings h into a sequence of discrete tokens z 2(h) (and de-quantizes z back to h = Q (z)), where Z is the codebook with size C; 3) decoder Da NxK + X' that decodes from h back to the video space through x Dh(h), Here, N is the length of the token sequence, and since lengths and resolutions vary from video to video, N is typically represented by c THW. where c is a pre-defined compression fixed-length tokenizers, as when T, H, W is given, the number of tokens is fixed. The tokenizer is optimized by minimizing the reconstruction loss:
X' & (O..... 255 7x557xxXX** where T is the number
rate. We refer to this as fixed-compression
Crocon (T) (x=D. |- [x(2)(
where DD is the video distribution to sample from, and (z(z)x) and (x(x))) are the posteriors for compression and reconstruction parameterized by the encoder &gg. and decoder DDg respectively. Under the standard assumption of VAE (Kingma al., 2019), eq. (1) can be converted into MSE loss on the pixel space. As such, tokenizers are essentially compressors that compress x into dense discrete sequence z.
2.2 UNIFTED FRAMEWORK FOR ADAPTIVE TOKENIZATION
In this section, analyze the inherent limitations of fixed-compresssion discrete tokenizers. We show that such a constraint is sub-optimal due to the varying spatio-temporal complexity across videos. Specifically, different videos contain diverse spatial content (e.g., scenes, objects) and exhibit varying temporal dynamics (c.g., speed, magnitude of motion), resulting in unequal informational demands. From the perspective of information theory, frequent contents are naturally easier to represent than infrequent contents, leading to natural result of varying representation lengths. To better illustrate this sub-optimality. consider an idealized scenario where the tokenizer T can perfectly reconstruct any input video x with non-zero probability p(x) 0. In what follows, we restate the Shannon Source Coding Theorem (Shannon et al., 1959) from Information Theory to ground our argument.
Theorem 2.1 (Shannon Source Coding Theorem (restated) (Shannon al., 1959)). For any tokenizer T with codebook size C. that can fully reconstruct video data x ~ p(x) defined above, we have
[xx=p(x)[NN Z Ho(D) P Ex=p(x)) 1- loge p(x)],
where Nx the token sequence length x assigned by Additionally, there exists an adaptive tokenization that has
Hc(D) S [N2] < Hc(ID) +
'This paper considers the standard RGB pixel space, but our results can be naturally extended beyond. While we focus video, the entire framework applies directly to images. We do not study image tokenization since the amount of tokens used for image representation is relatively small, and the incentive optimize it is insufficient.
3

---

Published as a conference paper at ICLR 2026
In contrast theorem 2.1, it is obvious that any fixed-length tokenizer requires at least logo [D) tokens to fully reconstruct the distribution, which is significantly larger than (c)D) when the data distribution is not uniform distribution. While this is simplified case, it motivates us to design framework for adaptive video tokenizer, which is presented below.
Adaptive video tokenization. Given fixed-length tokenizer T, adaptive video tokenizer Tadnptive = (T, My)) consists of the two extra key components: A router r(Nx/x) (could be deterministic or stochastic) that determines the token length Nx for each video x, and an adaptive compressor My IRN x N+ + IR XX that effectively compresses the information into Nx tokens for more condensed representation. Before fine-grained analysis on each component, we present a pseudo-code of the framework in algorithm 1.
The role of the router is determine an appro- priate token length Nx. which is the most im- portant problem in adaptive tokenization. The adaptive compressor feeds the latent sequence into a network that transforms information ap- propriately and outputs h' with length Nx. The sequence h' is passed through a quantizer to obtain the discrete token sequence For the decoding counterpart, it decompresses using M that conversely transforms the de-
Algorithm 1 Adaptive Tokenizer Training
Input: Encoder Eg, decoder Ds, router r(N]x), adap- tive compressor Ma. video data distribution D.
1: repeat
2:
Sample video x ~
3:
h t (x(x)
Encode
Nx rrN x)
Router b Compressor $ 3.2
Q(h').
h' <- Me N&)
6:
2 + Q(h')
b Quantize P Dequantize
7:
h' + (z)
quantized sequence h' (z) with length
Decompressor $ 3.2
N₂ back h with the original length N. Fi- nally, the reconstructed video is obtained via the decoder x = (b).
9:
X Do(h)
D Decode
10: Optimize Ladaptive per Eq. 4
until converged
Similarly, the adaptive tokenizer is optimized by the reconstruction loss
Crecon (Tadaptive) = Ex-D,N, N (NN.e Nx)), [- & Nx)),
(2)
which depicted in Algorithm Remarkably, we design our adaptive tokenizer framework on top of existing fixed-length tokenizers, ensuring seamless compatibility and enabling it to benefit directly from future advancements in the field. In short, the objective of adaptive tokenization is to leverage its additional flexibility to compress token sequences in principled way without sacrificing performance, thereby representing long video more accurately and efficiently.
2.3 SUB-OPTIMALITY OF EXISTING METHODS
Specifying Nx for each video challenging because adaptiveness is only for flexibility but also for principled compression. As special cases of our framework, several existing tokenizers for images (Duggal al., 2024) and videos (Duggal et al., 2024) propose to train with heuristic and data- agnostic router, where N₂ is sampled uniformly from (1,... N). During inference, the sequence length selected based on the quality. However, uniform selection does not take into account the different information quantities possessed by different videos. Furthermore, the uncertainty in random selection brings additional overhead inference time: To determine the optimal length, one needs to perform search over possible lengths. To rigorously prove why this is biased, we simplify their inference stage and assume that an oracle r" (Nx (x) can directly return the minimal sequence length conditioned on the loss eq. (2) being minimized. The complete proof is deferred to section B.
Theorem 2.2. Assume that r(- r(-(x) is uniform distribution over (1,..., J. For any constant > 1. there exists a data distribution Do and sufficiently large N, such that for any adaptive tokenizer To Taloptive (T*,r, My) that minimizes eq. (2), any r" (-(xx E arg min, Lremm ,r, M)))
Excx((x) Na (No 2 (ID.
Intuition. We briefly describe the underlying intuition of the theorem. While a uniform r requires the model to reconstruct videos simultaneously at different information levels, it does not inject incentives for reducing the expected token length. Consequently, data with different likelihoods are treated indifferently in this loss function, and the tokenizer could be trained to reduce the prediction difficulty when the sampled N₂ is small, even though it has a negative influence to the average token length. As illustrative example, readers consider the tokenization problem of four-data distribution
4

---

Published as a conference paper at ICLR 2026
with probability 2 with codebook size C = 2. While the optimality is to 1, 2, 3, 3 tokens for each video, minimizing the above loss results in all videos consuming 2 tokens, an inefficiency that will exacerbate the distribution becomes more complex and imbalanced.
Theorem 2.2 highlights the intrinsic bias of the data-sgiostic router: even if we manage to minimize the loss during training, the uniform pattern of the router will bias the tokenizer to tokenize data in way that, during inference, its expected token sequence length could be arbitrarily large compared to the optimality. Remarkably, while this theorem is particular for the uniform router and the case when the loss is minimized, it demonstrates the fundamental limitation of data-agnostic router for training.
3 INFOTOK: INFORMATION ADAPTIVE TOKENIZATION
Noticing the limitations of the data-agnostic routers, an intuitive alternative is to add the expected sequence length to the loss function, the goal is to minimize it while maintaining good recon- struction quality. Unfortunately, N₂ is discrete and not directly optimizable, making this purely optimization-based solution intractable. This section proposes novel router that aligns with the information theory, thus circumventing the dilemma of directly optimizing sequence length.
3.1
INFORMATION
ADAPTIVE TOKENIZATION
Token length selection via ELBO. Our theory in section 2.2 has indicated that, in order to achieve an optimal expected token length, N₂ should be proportional the negative log- lieeiikiidood. In other words, the minimized (ID) is realized by setting N₂ - log p(x). Yet, the log- bog-likelhhoo of arbitrary video x is intractable and thus estimation is required. We propose to leverage the Evidence Lower Bound (ELBO) surrogate, which is defined as:
ELBO(x)= (x(x)) [log pe (x z)] DKL l=<())))ll(()))
where (z(xx) and pa (x] are the encoding and decoding distributions, parameterized by the encoder Es and decoder D3 respectively, and p(z) is the prior. Our choice to employ ELBO is motivated as ELBO is provable lower bound the log- likelihood of data, i..., ELBO(x) log p(x). Critically, the model is trained directly to minimize the gap, and the bound becomes tight when the approximate posterior approaches the true posterior, i.e., DKI. [q [z) (z)x) [p (z(x)] == 0, making ELBO reasonable approximation to the log-likelihood.
Specifically, we design the router for token length selection as
ELBO(x)
< = 6(3
E EEBBO((
where / is the average compression factor that measures how much information token should represent, 8()) denotes the delta distribution, and the term E[ELBO(x)) is for normalization. In addition to the theoretical merit that is discussed below, our choice directly approximates the optimal sequence length through eq. (4) and thus does not require extensive probing over different lengths during both training and inference Yan et al. (2024). In practice, eq. (4) can be computed efficiently: we first encode x and decode back to without using the adaptive compressor, so as to compute the reconstruction error, which is the negative ELBO when adding the additional KL term. We then reuse the continuous embedding h and then compute h' = MM (h, N2). which is henceforth quantized into token sequence z with length Nx. As result, only have one additional decoder pass in order to generate the compressed token sequence.
Unsurprisingly, method can achieve near-optimal token allocation, stated below.
Theorem 3.1. For any adaptive tokenizer Tadopiive and B > (EL) if the tokenizer manages to minimize the reconstruction loss, i.e.
with the router r defined as eq. (4)
Talaptive E arg min Crecon (Tadaptive),
Tadagoor
then during inference, Talapiive guarantees that
E Ex-p(x)
N HD( + -- logp(x)).
The proof is deferred to section B. Theorem 3.1 implies that if the tokenizer well-trained to minimize the loss, then the compression rate of INFOTOK is optimal up the approximation emor.
5

---

Published as a conference paper at ICLR 2026
-[]EBOO(x) E[- p(x) by the definition of ELBO. Provided by large-scale neural networks (fixed-length tokenizers), ELBO values are believed to be close enough to the log-likelihoods, thus providing strong theoretical guarantee when 3 is selected appropriately.
Notice that 3 2
In practice, 3 can be chosen up-front based on the computation budget to determine the average amount of tokens used. To further simplify the problems, we can ensemble multiple choices of into one tokenizer, i.e., we allow /3 to be chosen from different values and input & to the adaptive compressor during training. As such, this single compressor can understand the amount of information to put in each token with different 3, thus remaining performant under a diverse range of compression rates. We name this approach INFOTOK --eex with rg finx (NVx x) = 1 EBEB r.g (NNx)) x) for some given B. Empirically, we find that using the reconstruction error itself (without the KI. term) to derive ( (x) is sufficient, as the KL term is approximately proportional to the reconstruction error, and the ratio is similar. During inference, PROTO--Flex automatically tokenizes each video into a token sequence based on the given B, achieving adaptive tokenization.
3.2 ADAPTIVE COMPRESSOR
The remaining question is how to compress h into a shorter h via the adaptive compressor, given token budget Nx. For ideal fixed-length tokenizers where tokens are organized as a ID sequence without geometric bias, merging information into the first Nx tokens and discarding the remaining N Nmax Nx tokens could be solution. However, existing SOTA tokenizers that support multiple res- olutions and video lengths are spatial-temporal sensitive, making the above approach less reasonable (verified in our experiments). In addition, the high-dimensionality of video data naturally poses extra space to consider, e.g., the variability within each video.
Likelihood-based token selection. To address this issue, we introduce a design for the adaptive compressor that takes into account the variability of information content within a video by discarding the tokens with the lowest information content. Aligned with our analysis in section 3, we preserve the top Nx tokens according to their corresponding per-token log-likelihood, which is also approximated via the ELBO values. Specifically, the adaptive compressor computes a binary mask m E {0, 1 where Nx tokens with the lowest ELBO values are 1 and the remaining are O. Interestingly, it does not incur extra network evaluation since the log-likelihooe router r(-lx). The end-to-end reconstruction loss will train the adaptive compressor transform information in tokens to be masked to the remaining positions. To ensure that the video can be decoded at inference time, is stored as part of the discrete token sequence z, leading overhead of approximately 5% in token length, which is worthwhile as we will see in experiments.
term has been computed in the
minimal
Architecture. As general framework, INFOTOK can be seamlessly integrated top of established tokenizer architectures by reusing their encoder &g and decoder Dg. Empirically, we implement INFO- ToK based on Cosmos Discrete Video Tokenizer (Agarwal al., 2025), D--CNN-based fixed-length tokenizer used to initialize Eg and Do. To account for the extra flexibility introduced in the adaptive compressor and decompressor, we additionally initialize the compressor Ma with a stack of multiple Transformer layers to effectively compress h into h', Similarly, the decompressor contains multiple Transformer layers for mapping the condensed representation back to the full-length sequence.
EXPERIMENT
We present experimental results about the superiority of INFOTOK in video tokenization performance in this section. Notice that this paper focuses mainly on adaptive compression that represents videos accurately with fewer tokens. While video generation is an important downstream application, training video generative model extremely peoource-consuminng and is beyond our scope. For more details about training, inference, and resource consumption, please refer to Appendix C.
4.1 EXPERIMENTAL SETTINGS
Dataset. To systematically evaluate different tokenization, we consider two video reconstruction datasets, namely TokenBench proposed in Agarwal et al. (2025), and DAVIS in Caelles et al. (2019), each containing videos with various resolutions and numbers of frames. Since ElasticTok only takes in square 256px video with H = W = 256, we only take the 256px partition of both datasets and crop each video to square shape randomly. All results below are reported our processed datasets for fair comparisons, although this will make baseline results not directly comparable with their paper. Notably, InfoTok can generalize to video with different resolutions, presented in Appendix. D.

---

Published as a conference paper at ICLR 2026
Table Evaluation of fixed length and adaptive tokenizers on TokenBench and DAVIS, We compare INFOTOK
with ElasticTok at two compression levels (0.81, 0.56) by setting our compression rates to theirs.
TokenBeach- 22cnBeechh-2602566
DAVIS-256x256
Tokenization Method
Compression (BPP1s
PSNR1
SSIMI
LPIPS|
FVD.
PSNR1
SSIM1
LPIPS_
FVDJ
Open- Dpsn-MAGVTT--UCF
1.12
25.77
0.762
0.269
121
21.98
0.656
0.317
583
Omni Tokenizer
0.767
0.318
673
Cosmos-DYs Dosmos-DV4x300 DV4x888
1.00
30.01
0.885
0.138
49
25.92
0.793
0.208
404
Elastic
28.26
0.84
24.09
0.318
754
INFOTOX-FiO Flex
0.81
29.86
0.878
0.148
54
25.69
0.781
0.216
441
INFOTOX
0.223
ElasticTok Tok INFOToX-Flex
0.56
27.34
0.813
U.276
23.76
0.714
0.356
930
0.56
29.30
0.857
0.179
24.84
0.742
0.274
581
INPOTOK
0.56
29.27
0.854
0.176
70
24.52
0.738
0.277
540
Original
POMO Comgrasdor DPPIN
14.13 Compasti (Monanal DPP-
FSML
CA
Counus DV
Elusti
Info Tokk- Flax
Figure 2: Reconstructions examples of video with different complexities using different tokenizers. INFO- Tok-Flex can achieve similar PSNR with much higher compression (compared to Cosmos DV), and similar compression rates with better PSNR (compared to ElasticTok).
Evaluations. We evaluate the quality of reconstructed videos by four metrics: the Peak Signal to Noise Ratio (PSNR) (Hore & Ziou, 2010), the Structural Similarity (SSIM) (Hore & Ziou, 2010), the Learned Perceptual Image Patch Similarity (LPIPS) metric (Zhang et al., 2018), and the Fréchet Video Distance (Unterthiner et al., 2019). To evaluate the compression, we adopt the bits-per-pixe (BPP) value that computes how many digital bits are used to represent one pixel (with all color channels) in the whole video. For the sake of presentation, we present BPP with unit of T which represented as BPP 1s (bits-per- 16-peel-) For instance, the number of tokens is c THW for a video with resolution H x W and length T, and the codebook size is C, then BPP 16c log(C).
INFOTOK Details. We use Cosmos Tokenizer (Agarwal et al., 2025) as the fixed-length tokenizer and it to compute the ELBO value of inputs. Given video of shape T x H x W, it uses 256 THW tokens to tokenize the video. We refer our methods using the router r,a (x as INFOTOK and the one using r (Nx x) as INFOTOK-Flex. For router r (Nx|x)) E[ELBO(x)] is computed from the exponential moving average of historical training samples. For router fex (Nx we consider B 25.x55mm 0.5N 0.75.Nmax Nmmxx } to assemble tokenizers with different average compression rates. During inference, we specify an average BPP and / can 1 is the cost of binary mask. The adaptive compressor an eight-layer transformer with a block-causal attention matrix that maintains the causality of the Cosmos Tokenizer. For quantization, we use the Finite Scalar Quantizer (Mentzer et al., 2023b).
Nmax T x #
W'
8 &
be computed as Nma. (BPP16 1g). where 16
Baselines. First, for fixed-compression tokenizers, we evaluate Open-MAGVITY (codebook size C 2 (Yu et al., 2023a), OmniTookeizer (C' = 213 (Wang et al., 2024), and the Cosmos Discrete Video Tokenizer (C 216) (Agarwal al., 2025). Their resulting BPP 16 values are reported in Table 1. For adaptive tokenization, we compare against ElasticTok (2 (Yan et al., 2024), which employs right-to-left random masking on token sequence during training and binary search to select the minimal number of tokens to meet specified reconstruction-los threshold, with BPP ranging from 0.25 to 4. Since Elastic Tok''s s adaptiveness is governed by a loss threshold rather than an average compression rate, we align our methods with their settings.
7

---

Published as a conference paper at ICLR 2026
Origin
PIRR NE Conpasio Prss Wansed 20hall 875
POWR: 1467 Conparca Run [Moonart S0hall
FSM 33 Comateson Bax (Mocara DFP ad7
Ratesst
Compastio Monuee 30hwl
32:88 Compaation (Moraard
Comprisien (Monaaod
Figure 3: Reconstructions examples of video by INFOTOK- Flexx with different compression rates,
Elsstitik InfoTok
0.3-
InfoTok
InfoToll Flex
InfoTok: Flex
LPIPS
27.5-
200
Elastictkk InteTak
0.2
Additional NFSS. Standard NFRS
26.0
IntoTok-Tlas
100
0.4
0a
0.4
0.6
0.0
10
0.6
0.0
1.0
(a) PSNR-Tokean TokenBench
(b) PPPSS-TokeBBenc
(c) BVD-TokenBench
Elsttctio
ElsstcTik InfoTok
25.0
InfoTok: Flex
InfoTok
1500
InfoTok
LPIPS
PVD
ElasticTkk InfoTok-Tlas
1.00
1.00
0.5
0.
0.4
0.5
10
0.4
09
(d) PSNR-DAVIS
(c) LPIPS-DAVIS
(f) FVD-DAVIS
(g) NFEs Overheul
Figure 4: Video tokenization performance of INFOTOK-Fles INFOTOK, and ElasticTok on TokenBench (a-c) and DAVIS (d-f). Quality metrics are plotted against 1PP1s (bits per 16 pixels). Tokenization efficiency measured in the Number of Function Evaluations overhead (additional NFEs standard NFEs J) is shown in (g). InfoTok-Flee
InfoTok achieve superior reconstruction quality with smaller 1PP1s levels. Additionally, INFOTOK is significantly more efficient than Elastic/okk, which requires searching to meet thresholds.
4.2
SUPERIORITY OF INFOTOK
Benchmarking Results. Table compares our methods against several baselines at average com- pression rates of BPP16 0.81 and 0.56. Compared to ElasticTok,, INFOTOK-Flex and INFOTOK achieve notably better reconstruction quality at the same average compression levels, with FVD reduced by
60%, LPIPS by 25 40%, and PSNR by 1.0 2.0 (which is MSE reduced by 20% 36%). Remarkably, INFOTOK can outperform Flastic with BPP 0.81 with lower compression rate (0.56). Compared to reddccompressoon baselines, INFOTOK performs similarly to Cosmos-DV with 20% tokens saved and outperforms the remaining baselines when the average BPP 16 is cut half, again demonstrating the advantages adaptive tokenization.
Token Efficiency with Flexibility. Figure provides a more comprehensive comparison with ElasticTok under different average compression rates. The blue line represents the performance of INFoTOK-Flex trained with r represents INFOTOK trained on different compression rate levels and used at those levels. The red line represents the ElasticTol model with different loss thresholds, resulting in different performance and compression rates. Improvements over ElasticTokk are salient, with similar reconstruction quality (PSNR) and perceptual scores (LPIPS, FVD) achieved by compression rates 2.3x larger (e.g.. PSNR on TokenBench, FVD DAVIS). Notably, the same inference setting, INFOTOK-FIL performs on par with INFOTOK trained with different specific r,s implying that can ensemble models with varying rates while still outperforming FlasticTokk, which is trained with uniform router.
_llex and used at different compression rates 3, and the green line
8

---

Published as a conference paper at ICLR 2026
Ablation on INFOTOK versus optimal search-based strategy determine the token lengths. "Optimal" is strict upper bound our method, yet their performance is extremely close.
Table 2:
TokenBench-256xxe
DAVIS-256x256
Compression (BPP1s 4)
Inference
Method
PSNRT
SSIM
LPIPS.
FVD]
PSNR
SSIM
LPIPS)
FVD1
0.81
INFOToK-Flex
29.86
0.878
0.148
54
25.69
0.782
0.228
441
0.81
Optimal
29.92
0.878
0,148
54
25.79
0.784
0.226
431
0.56
INFOToK-Flex
29.30
0.857
0.179
71
24.84
0.742
0.274
581
0.56
Optimal
29.39
0.856
0.180
74
24.93
0.740
0.276
601
INFOTOK-
26.89
0.782
0.265
155
21.85
0.639
0.384
1066
0.31
Optimal
26.98
0.779
0.267
165
21.97
0.640
0,388
1138
Inference Efficiency. Figure 4g illustrates the advantage of our inference efficiency. To determine the token length for a video, ElasticTok runs a binary search over each 4096-token block sequentially to satisfy the given loss threshold, requiring an additional log2(4096) - = 11 network forward evaluations (NFEs). In contrast, both INFOTOK and INFOTOK need only one additional network evaluation of the decoder compute the ELBO, demonstrating substantial efficiency gains. More details regarding wall-clock inference latency comparison can be found in Appendix D.
4.3 ABLATION STUDY
Ablation routers. To evaluate how well our ELBO-based token allocation performs, we compare to an optimal routing strategy. For each video, it exhaustively evaluates the reconstruction quality 6 16...... 1}, and then solves the optimization problem over the entire dataset to find the optimal performance constrained on the average compression rate. As shown in Table 2, the LLBO-based router achieves similar performance compared with the optimal strategy, aligning well with our theory and demonstrating the feasibility to specify token lengths without brute-force search.
with 1PP1s { 16
Table 3: Ablation results on TokenBench with an average IPP 0.56. (Left) Ablation on adaptive compressors. (Right) Ablation on different variants of adaptive mechanisms across architectures.
Compressor
PSNR
SSIM
LPIPS
FVD
Architecture
Adaptive Mechanism
PSNR
Uniferm (ElasticTok)
27.35
152
R2I.
27.43
0.825
0.207
137
Cosmos
ELBO (InfoTak)
29.30
71
Jump
28.07
0.841
0.173
84
ElusticTok Backhone
Uniferm (ElasticTok) ELBO (InfoTok)
27.21
198
Ours
29.30
0.857
0.179
71
Vision Transformer
28.64
114
Ablation on adaptive compressor. To better understand the ELBO-based compression, we compare with two additional intuitive strategies: simple right-to-left masking used by ElasticTok (R21., and spatially dispersed variant that masks token every four tokens (Jump), in Table 3 (Left). Our compressor that masks tokens by ELBO values yields better reconstruction quality. These findings demonstrate that the compressor design matters in adaptive tokenization.
Ablation on the adaptive mechanism. Unifying the router and adaptive compression, we further compare the whole adaptive mechanism of InfoTokk to ElasticTok. Specifically, we evaluate both the adaptive mechanism of ElasticTokk and InfoTok, to the Cosmos architecture, and a pure ViT architecture. As shown in Table 3 (Right), Info Tok consistently outperforms FlasticTok across both architectures, yielding substantial improvements in both PSNR and FVD. These results confirm the effectiveness and generalization ability of InfoTok.
Reconstruction under different compression rates. While we have shown above that an ELBO- based compression rate is theoretically justified and empirically effective, we are curious how the reconstruction of one video changes as we gradually increase/decrease the compression rate. To this end, we visualize Fig. the reconstruction results with BPP ranging in (0.81, 0.56, 0.31). INFOTOK will reconstruct overall structures of the video when available tokens are limited, such that the MSE loss remains small despite the tne-grained details becoming invisible.
5 RELATED WORKS
Discrete Tokenization has been extensively explored to facilitate generative Al. Typical VQ- VAE (Van Den Oord et al., 2017) and VQGAN (Esser et al., 2021) extend the Variational Autoencoder (VAE) framework (Kingma et al., 2013) by introducing learnable codebook to quantize 2D encoded features into discrete tokens, which are then decoded back to reconstruct the original image. To streamline the quantization process, lookup-free tokenization methods such as FSQ (Mentzer et al.,

---

Published as a conference paper at ICLR 2026
2023b) and LFQ (Yu et al., 2023c) have been proposed. In terms of representation, TiTok (Yu et al., 2024) employs a Transformer- based architecture to learn ID tokens. Building upon this, FlowMo (Sargent et al., 2025) and SelfTok (Pan al., 2025) utilize diffusion decoders conditioned on ID discrete tokens to mitigate discretization errors further. Along the line of video tokenization, there several works, including OminiTokenizer (Wang et al., 2024), Open-MAGVIT2 (Luo et al., 2024), and Cosmos (Agarwal et al., 2025). Unlike images, video data is intrinsically more complex and sparse, making existing tokenization approaches inefficient.
Adaptive Representation has been studied over the past decade, with early works like Nested Dropout (Rippel et al., 2014) and Matryoshka Representation Learning (Kusupati et al., 2022) introducing methods to learn representations at multiple granularities. Recent approaches have applied similar methodologies to tokenization. CAT (Shen et al., 2025) focuses on continuous tokens, using vision-language models to assess image content complexity heuristically to determine token usage. ALIT (Duggal et al., 2024) recurrently distills 2D tokens into 1D latent tokens, enabling flexible tokenization.
(Miwa et al., 2025) and ElasticTook (Yan et al., 2024) employ random nested dropout strategies for tokens, targeting images and videos, respectively. FlexTok (Bachmann et al., 2025) further incorporates diffusion decoder to enhance reconstruction quality under low token counts. However, these works focus on images and rely on heuristic methods for adaptive tokenization such as random masking. As such, their objective functions biased by definition.
6 DISCUSSIONS & LIMITATIONS
Generalizability of INFOTOK. this work has concentrated on the video domain, the core principles of INFOTOK are not intrinsically limited visual sequences. Our framework is founded on information-theoreee compression and utilizes an ELBO-based router to estimate information complexity, a methodology that broadly applicable to any data modality compatible with a VAE- style reconstruction objective. The fundamental challenge of variable information density exists in many other domains. For instance, audio signals often contain significant temporal redundancy (c.g., periods of silence, sustained background noise), where an adaptive approach could yield substantial compression gains by allocating tokens based on acoustic complexity (Zhang et al., 2023; Liu et al., 2025). Likewise, 3D data, such as point clouds meshes, often features sparse data and non-uniform detail, with large, simple surfaces requiring less representational capacity than areas of high geometric complexity (Chen et al., 2025; Lu al., 2025). Therefore, applying INFOTOK develop adaptive tokenizers for audio, 3D, or other structured data presents promising direction for future research.
Limitation of INKOTOK. While INFOTOK demonstrates significant gains in compression efficiency for video reconstruction, this study has a few limitations. First, our principled ELBO-based router introduces a modest computational overhead by requiring one additional decoder pass to estimate the video's 'ss information complexity. Although this is significantly more efficient than the multi-pan search required by prior adaptive methods, further research could explore lighter-weight router mechanisms, such as estimating complexity directly from the encoder's latent representations, to climinate this extra step entirely. Second, our evaluation is primarily focused on reconstruction fidelity (c.g.. PSNR, FVD) proxy for representational quality. As noted, we did not extend our experiments to downstream applications such as video generation or action understanding, which was beyond our current scope due to the substantial computational resources required. Investigating how INFOTOK's ' adaptive token sequences impact the performance and efficiency of large-scale generative or video-understanding models remains a key direction for future work.
7 CONCLUSION
This paper introduced InfoTook, a principled adaptive video tokenizer inspired by Shannon's 's infor- mation theory, to address the suboptimality of fixed-rate video compression prevalent in existing methods. InfoTok utilizes an ELBO-based router to determine token lengths dynamically according to video complexity and employs a transformer baseed adaptive compressor to manage these variable- length representations efficiently. Empirical results demonstrate Tok's significant advantages. We believe that InfoTok offers valuable insights for advancing future research on scalable multimodal models that can both understand and generate long videos.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Video discrete tokenization is pivotal for advancing world models and video generation, offering significant societal benefits by enhancing accessibility and efficiency, However, these advancements also pose ethical challenges, including the potential misuse of generative models to create deceptive content such as deepfakes, which can spread misinformation and infringe on privacy.
REPRODUCIBILITY STATEMENT
We present the full experimental settings in Section 4.1 and provide additional details in Appendix To ensure reproducibility, the anonymous code is included in the supplementary material.
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.08 003575, 2025. 1, 6, 10, 21,
Robert B Ash. Information theory. Courier Corporation, 2012.
Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oğuzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences of flexible length. arXiv preprint arXiv:2502.13966 2025. 10
Sergi Caelles, Jordi Pont-Tusee Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokitsi Maninis, and
Luc Van Gool. The 2019 davis challenge on vos: Unsupervised multi-object segmentation.
arXiv:1905.00737, 2010 2019. 6, 23
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple mrossmodallty teachers. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Putlern Recognition, pp. 13320-13331 2024. 22
Yiwen Chen, Yikai Wang, Yihao Luo, Zhengyi Wang, Zilong Chen, Jun Zhu, Chi Zhang, and Guosheng Lin. Meshanything v2: Artist-created mesh generation with adjacent mesh tokenization. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 13922-13931, 2025. 10
Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tokenization via recurrent allocation. In First Workshop Scalable Optimization for Efficient and Adaptive Foundation Models, 2024, 4, 10
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high righ-ressotttin image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021. 2.9
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400 2024. 22
Robert Gray. Vector quantization. IEEE Assp Magazine, 1(2):4299, 1984.
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pp. 289-305, Springer, 2024. 21, 22
Alain Hore and Djemel Ziou. Image quality metries: Psnr vs. ssim. 2010 20th international conference on pattern recognition, pp. 2366-2369, IEEE, 2010. 7
Diederik Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
11

---

Published as a conference paper at ICLR 2026
Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders, Foundations and Trends in Machine Learning, 12(4):307-392, 2019.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30249, 2022. 10
Chengwei Liu, Haoyin Yan, Shaofei Xue, Xiaotao Liang, Yinghao Liu, Zheng Xue, Gang Song, and Boyang Zhou. Unitok-audio: A unified audio generation framework via generative modeling on discrete codec tokens. arXiv preprint arXiv:2510.26372 2025. 10
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprini
arXiv:1711.05100 2017. 21
Shugi Lu, Haowei Lin, Lin Yao, Zhifeng Gao, Xiaohong Ji, Linfeng Zhang, Guolin Ke, et al. Uni- 3dar: Unified 3d generation and understanding via autoregression on compressed spatial tokens.
arXiv preprint arXiv:2503.162
2025. 10
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang. and Ying Shan. Open-magvita. An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.044/0 2024. 10
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization:
Vq-vae made simple. arXiv preprint arXiv:2309/15505, 2023a. 3
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15500: 2023b. 9,
Keita Miwa, Kento Sasaki, Hidehisa Arai, Tsubasa Takahashi, and Yu Yamaguchi. One-d-piece: Image tokenizer meets quality-controlllaee compression. arXiv preprint arXiv:2501.10064 2025. 10
Kaihang Pan, Wang Lin, Zhongqi Yue, Tenglong Ao, Liyu Jia, Wei Zhao, Juncheng Li, Siliang Tang, and Hanwang Zhang. Generative multimodal pretraining with discrete diffusion timestep tokens. arXiv preprint arXiv:2504. 1104..4660 14666, 2025. 10
Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pp. 1746-1754. PMLR, 2014. 10
Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, and Jiajun Wu. Flow to the mode: Mode-sceking diffuuson autoencoders for state-of-the-and image tokenization. arXiv preprint arXiv:2503.11055,
2025. 10
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379-425) 1948. 2
Claude E Shannon et al. Coding theorems for discrete source with fidelity criterion. IRE Nat. Conv. Rec, 4(142-163):1, 1959. 3
Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer, Lili Yu, and Chunting Zhou. Cat: Content-adaptive image tokenization. arXiv preprint arX1v:2501.03120, 2025. 10
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 17:84839-84863 2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metrie for video generation. 2019.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 1,9
12

---

Published as a conference paper at ICLR 2026
Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference Robot Learning, pp. 1723-1736 PMLR, 2023. 22
Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer:
A joint image-video tokenizer for visual generation. Advances in Neural Information Processing Systems, 17:28281-282999 2024. 7,
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using
vq-vae and transformers, arXiv preprint arXiv:2104, 100:220400011,, 2021. 2
Wilson Yan, Volodymyr Mnih, Aleksandra Faust, Matei Zaharia, Pieter Abbeel, and Hao Liu. Elastictok: Adaptive tokenization for image and video. arXiv preprint arXiv:2410.08368, 2024. 2, 5, 7, 10, 21, 23
Fisher Yu. Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
(636-2645, 2020. 22
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023a. 2,7
Lijun Yu, José Lezama. Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737 2023b. 3
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng. Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint arXiv:2310.0573: 2023c. 10
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 17:128990-128866, 2024. 10
Boqiang Zhang, Kehan Li, Zesen Cheng. Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501. 13106, 2025. 1
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 586-595, 2018. 7
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 2023. 10
13

---

Published as a conference paper at ICLR 2026
Supplementary Material
CONTENTS
Introduction
2
Adaptive Tokenization: The Framework
2.1 Discrete Video Tokenization
2.2 Unified Framework for Adaptive Tokenization
2.3
Sub-optimality of Existing Methods
3
INFOTOK: Information-thherrlis Adaptive Tokenization Information-theorrticc Adaptive Tokenization..
3.1
3.2
Adaptive Compressor
4
Experiment
4.1
Experimental Settings
6
4.2
Superiority of INFOTOK
4.3
Ablation Study
5
Related Works
6
Discussions & Limitations
10
7 Conclusion
10
Illustration of Adaptive Tokenization
15
Proofs
17
B.1 Proof of Theorem 2.2
17
B.2 Proof of Theorem 3.1
Experimental Details
C.1 Training Details.
21
C.2 Implementation Details
21
C,3 Evaluation Details..
22
Additional Experiment Results
23
Use of Large Language Models
23
14

---

Published as a conference paper at ICLR 2026
A ILLUSTRATION OF ADAPTIVE TOKENIZATION
An illustration of how INFOTOK tokenizes videos based on information complexity (figs. 5 and 6). White tokens are those preserved and black tokens are those masked out by INFOTOK. "Token Usage" specifies how many tokens are preserved in this frame.
Frame
Original
Frame 60
Original
IefoTk
Taken
Frame 120 Original
InfoTok
Token Mask
Token Usoge
Frame 180 Original
InfoTok
Token Mosi
Taken Usoge
Frame 240 Original
InfoTak
Token Mask
Token Usoge
Frame 258 Original
InfoTok
Taken Mosk
Taken Usoge
Figure 5: The video starts with brown dog sleeping on the ground. The video is stable and more than 60% of tokens are masked without harming the reproduction quality. Later on, the camera shifts toward another white dog, during which the token length increases to 60% encode new information. As the camera stabilizes again and focuses on the face of the white dog, the token length reduces back 30%, and much of the ground area masked out (because they can be easily inferred from surrounding areas).
15

---

Published as a conference paper at ICLR 2026
Frame Original
InfoTok
Mask
Token Usage
Frame 20 20 riginal
InfoTok
Token Mask
Token Usage
50%
Frame 40 riginal
InfoTok
Token Mask
Token Usage
29%
Frame 60 riginal
InfoTok
Token Mask
Token Usage
17%
Figure 6: The video begins with a wide shot of workspace. The camera highly dynamic as the person moves, requiring higher token usage of 65% to capture the rapidly changing environment. As the person stabilizes their position and focuses their hands on the task at the center the frame, the token usage progressively drops to 50% and then 29%. By the end of the sequence, the background remains static and the movements become more predictable, the token usage further reduces to 17%. masked out, as the model recognizes they contain
Large portions of the peripheral workshop redundant information that can be inferred from previous frames.
16

---

Published as a conference paper at ICLR 2026
B PROOFS
B.1 PROOF OF THEOREM 2.2
Proof. To prove this theorem, first need to understand the structure of the optimal tokenizer that minimizes eq. (2), as well as the corresponding oracle r* used during inference.
Given sampled and Nx. the loss to be minimized can be represented as
Egagg (a Nx)) log Pe x z)] = lo ((M,((x(
Due to the compression conducted by My. multiple videos (that have identical token prefixes) could be mapped to the same sequence, and the decoder is required to minimize the negative log- eigllikeihhodd for them simultaneously. Therefore, we can rewrite the loss as
C = T2 Xx-p((x)))) log Po (x|2)] 2 (P)
In other words, given short token sequence z. the best decoder can do is to predict with probability exactly equaling p(x[z). in which case the loss is the entropy of the marginal distribution of x that has token sequence prefixes z, averaged across all possible z with corresponding probabilities. On the other hand, for arbitrarily powerful decoder, this loss is realizable. Therefore, below only need to deal with the converted loss.
Notice that since we assume the compressor follows generate-then-mask pipeline, the probability that a decoder has input z is
Σ NRca
pix)p(Q(M,(EEEx)..
N
where the last 1 that since we assume N to be sufficiently large, the probability that different videos x, x' mapped to an identical 2 is O. Otherwise, we can always find another | N, such that the longest common prefix of & and ă is not the prefix of any other video that is not mapped a, i.e. fx such that, z C (E N)) # é. In this case, we can alter the mapping of x' to 2, and the loss will decrease because the entropy of P(x|z) )) for any is non-increasing, and for it strictly decreases.
N comes from the uniform router r, and z C 2 means that z is prefix of 2. Notice
Therefore, the optimal encoder and compressor will map each x to a different à with length N, and mask à according to the router's s sample Nx. We denote the mapping from x to 2 as M (x) = ((M (x), N)). Given masked token sequence z with length N₂ sampled by the router, the decoder will predict x with probability proportional to T[ C (x))x((x).
We can therefore model the entire tokenizer as a C-fold tree with depth N, where each leaf corre- sponds to video with probability p(x) corresponds to nothing. For each video there is one and only one path from the root node to it, representing the (uncompressed) token sequence of the video. The depths of leaf nodes are video lengths, which are defined to be the minimal token lengths such that no two videos are mapped to the same node. The entropy of all leaf nodes is O. and the compressed token sequence z) is the entropy of
entropy of an intermediate node (corresponding the marginal distribution over all videos with prefixes z. The loss can be presented as
p(x)
L f= Σ
Σ
H(P(-lz))
N
(x)(x)
Σ
Σ
P(x)
Σ
log
p(x)
p(x) ExCM(3) P()) p(x)
N
Ex p(x)
MCM(X)
M
In the proof below, we can simply assume that C 2, but the proof can be trivially extended to any general C.
First, we rewrite the loss to get an equivalent loss L(T) NL for a tree T. For each leaf node
(corresponding to video x), define (i) as the set of ancestors of (corresponding to M (x)), and define N; as the set of leaf nodes that have node as an ancestor. Consequently, define
p(i)
p(i) p())
H(N,)= Σ
log
p())
WJNN()
17

---

Published as a conference paper at ICLR 2026
Here, p(j) is the sum of probability for an intermediate node j, which equals to Ex( p(X) in the previous definition. Therefore, we have
L(T) -N Σ p()) Σ
Σ
p(i)
p(t) p(i)
log
N
p(j)
JEN(i)
())N()
N Σ p(i) E
H(N))
N
JEN(()
E E ((OH(NN)
WJEN(O
'' (())
p(i)
log p(i)
ΣE(i p(i) E
p(1)
p(3)
k)EN(()
E E -p(1) log p(i)
p(j)
BYEN(()
Σ p(i) logp(i) Σ
p(i)
(()N(()
Σ logp() Σ (1)p(()) log
I10) p(2)) log - p(2j + 1) 1og + +1]
EMOIG (-R(2)
[_((2)) log ((2))
p(2j + 1)
log p(2j + 1)
p(j)
p(j) p(j)
p(j)
where j all intermediate nodes, p(j) is the sum of probabilities of its children, ((j) is the depth of the node (the root node has depth 0), and Hj) is the entropy of the node, i.e.
p(2j + 1) p(2)) + P(2j + 1) p(2)) + p(2j 1)
p(2j)
H (j) = Entropy
Here 2j and 2j + are the left and right child of node j.
Entropy(a, b) lal blog, b).
To better understand the loss, we further separate it in terms of depth:
L(T) Σ
dfs
(5)
d21
where d represents the depth, and
la Σ P()H())-
ja()=dd
Since HJ) E [0, 1]. we have 0 S fa
Σ p() 1
juli=d
18

---

Published as a conference paper at ICLR 2026
Further, an important property here is that
Σn= E P()H(()
d. PHGTFd
p(2)) E- (p(2)) p(2j +
p(2j)
p(2j + log
p(2j + 1)
log p(2j) + p(2) + p(2)) + p(2j + 1)
p(2j) + p(2j + 1)
Σ (v(2g) +p(2j + log (0(2))+ p(jj + 1)) - p(2j)
p(2j 1)
Σ -p(1)
4:d leaf node.
= (DD)
Now we start to prove the main theorem. We let the data distribution ID be as the distribution for 2 videos, with p(j) 2 for j = -....., 2 - I, and p(2 = p(2 1). We aim to prove that the expected depth of the tree T'M minimizing the loss 5 under D goes to infinity M goes to infinity.
We prove by contradiction. If the expected depth doesn't go to infinity as M + oo, then there exists D > 0 such that ((1) < D for any sufficiently large M. In this case, we claim that we can find a tree that yields a smaller loss, by "lifting" a node j with sufficiently small probability and sufficiently deep depth up be sibling of node 1. Specifically, suppose node was at depth d1, it will now be moved one level down to depth dj + 1, with node j as its sibling. The previous sibling of node j will be moved one level up. This j always exists since increase to infinity simultaneously. Specifically, we choose node j such that it has depth larger than 2D, and Pj sufficiently small compared with all p. for i with depth less than or equal 2D.
oc, and the maximum depth of the tree will
Now, let's look at the change to fa after the lifting. Without loss of generality, we assume the root is the lowest common ancestor of node and node j. If not, we can find their lowest common ancestor and do the same analysis on the sub-tree with that ancestor as the root. We assume node is in the left sub-tree, and node j was originally in the right sub-tree. We decompose the change fa into four different parts.
Part one. A:: change to fa, the left sub-tree.
p(1) The branching of node 1 will add an extra term (p(1) + Ap) Entropy formalized below:
"(1)+A,,' ,(1)-,,) ( to Shy
((1)
Ap
A = (p(1) Ap) Entropy
p(1) + Ap p(1) + Ap
p(1)
Ap
p(1) log
p(1) + A, A, log
p(1) + A - =-p(1) log p() p(1) log(p(1) A - Ap log A, + A, log(((11) + Ap)
- =-p(1) log p(1) [ogg((1) + Ap ) Ap log A, + A, [1og((1) + p(1)
Ap
) + t((Ap)
y(1)
- A, log Ap + o(Ap)
Part two. A2: change to fa in the left sub-tree with < < d1.
For each d < di, suppose the ancestor of node 1 at level d originally has two sub-trees with probabilities Pl and p2. After the operation, these two probabilities become pı + Ap and pa respectively. Hence, the change at level d is:
19

---

Published as a conference paper at ICLR 2026
A3 (p1 P2 Ap Entropy
Pt + Ap
P2
(pr P2 P2)topp
P1
p2
P1 + P2 + Ap p1
- + P2 + A
P1
P2
p1
P2
-(p1 p2 + pp) log(p1 + Ap) - (pi + Ap) log(p1 Ap)
- P2 log P2 (p1 + P2) log + p22) + P1 log P1 + log p2
=(p1 - P2 + p) log(p1 + p2) + Ap
log logpi pi + A
(pi + A
P1 + p2
P1
P2 lygog P2 (p1 p2) log(p= + P2) + p1 log P1 P2 log P2 + o(4p)
P1 + P2 + ((A,)
=Aplog
P1
=0(Ap)
Since d d1 upper bounded by D, by summing up change at all level d, we conclude that A2 O44p)
Part three. As: change to fa level O. Suppose the root originally has two sub-trees with probabilities pi and p2. After the operation, these two probabilities become pt + Ap and p2 Ap respectively. Hence, the change at level
(p1 + Entropy P1 + A₂
P2 Ae) (p1 + p2) Entropy
P1
P2
P1 P2
P1 + P2
P1 + P
P
+ P2
=(1 + p2) log(p1 + P2) (p1 + 4p) log(n + 4p)
(p2 Ap) log(p2 4p) (p1 + p2) log(ps P1 + P2) + Pı logp1 P1 + Pa log
=(p1 + log(p1 + P2) (p1 + Ap) logp1+ Ap
P1
(ogg2 Ap (pi + log(11 + p2) + Pı log + pa p2logg2 o(Ap)
(p2 - Ap)
p2
=Ap log
P2
+ o(Dp)
P1
-0(Ap)
Part four. Aς: change to fo in the right sub-tree with 0 < d < 2D.
Similar to Part two, A (Ap))
Part five. As: change to fa in the right sub-tree with d 2 2D.
Suppose node j' is node j's ancestor in the right sub-tree level 2D. Suppose the node j' originally
has probability Po- After the operation, it has probability Po Ap. Similar to the derivation of Ea Sa. we can calculate that
As = (po - Ap) log(po Ap) - (po log po Ap log Ap)
4g) ) Po log + log Ap + o(Ap)
=(po - A log Po
=Ap log Ap + O(Ap)
By all above, since Ap sufficiently small, the only dominant terms among the A's 's the increase of Ap log Ap at level d1. and decrease of A, log Ap at level 2D and below.
Recall that L(I'- Eaz dfa. and 2D Ap log Ap after the operation. This contradicts our assumption that TM is the minimizer of the loss. Therefore, we have the expected depth of the Tar minimizing loss 5 under D goes to infinity as M goes to infinity.
di we'll have loss decrease by at least -(2D
On the other hand, we can construct Hoffman tree with node j on level j. In this case,
2M
lim M-+00
Σ
12
(2 -1) 2.
1m1
20

---

Published as a conference paper at ICLR 2026
Hence, H2(D) O()). This means that for any x. >> there exists ID and sufficiently large N such that (xx- (H) This completes the proof.
B.2 PROOF OF THEOREM 3.1
Proof. Suppose the ELBO for x is (LBO(x) = = < pxx). Since -Ix 2 - log p(x), and Hoffmmnn tree can encode x with Nx log p(x), the minimizer of the adaptive loss has Nx S -lx with N 2 -Ix-
Therefore, we have
S Mx - lx
-Ho(D) + Mx (-xx log p(x))
-Ho(DD) -[[ELBO(x)] - Ex Exxp(x)]- (ocxppc)(lloop(()
<Hc(D) + - Exxp(x) log p(x))
C EXPERIMENTAL DETAILS
C.I TRAINING DETAILS.
Architecture Configuration. The complete hyperparameter settings of the INFOTOK model archi- tecture are provided in Table 4. The adaptive compressor and de-compressor are implemented using standard Vision Transformer (ViT) equipped with 2D Rotary Position Embeddings (RoPE)Heo et al. (2024) to handle videos with different resolutions. Features obtained from the Cosmos encoder are processed by an eight-layer ViT encoder and subsequently compressed using ELBO-guides length selection and likelihood based token selection, as described in Sections 2 and 3.2. The compressed discrete tokens are decoded back into continuous features via an eight-layer ViT decoder and passed through the Cosmos Tokenizer's ss decoder to reconstruct the original video. Despite its effectiveness, the adaptive sompressor/de-comprea INFOTOK, with the base tokenizer accounting for the remaining 85,4 95.4%. This highlights the efficiency and potential of our adaptive tokenization framework.
module constitutes only 14.6% the total parameters of
Training Datasets. For training,
the datasets in Agarwal et al. (2025), in coverage of different visual objects and multiple resolution. During training, to compare fairly with Elastic Yan al. (2024), we train our models on resized videos with 256px (c.g.. 256x256, 256 446 and on), although our method works naturally on different resolutions.
Training Configuration and Computation Resource. The training augmentation follows Cosmos Tokenizer Agarwal et al. (2025). We employ the AdamW Loshchilov & Hutter (2017) optimizer with an initial learning rate 1e - with cosine decay to le 5 for 1e5 steps. The training regime features batch size of with video clips of 33 frames over 2c5 steps. All models presented in this paper trained with 32 H100 GPUs (4 nodes), and our setting will take 4 days to train a model. Notice that each GPU consume data with different resolution simultaneously.
C.2 IMPLEMENTATION DETAILS
Token length selection. To implement ELBO-based token length selection, we address the chal- lenge of computing the expectation of the Evidence Lower Bound (ELBO) in the context of streaming large-scale video datasets, where precomputing ELBO values for all data is impractical. To overcome this, we maintain an exponentially weighted moving average (EMA) of the ELBO over incoming training data, providing real-time adaptation to the data distribution without the need for full dataset traversal. After that, as ELBO(x) shown in Table 4, for the INFOTOK model, we rescale the ratio E[LBOO[X] by a fixed compression factor 3 (e.g., 0.5) to determine the final token length. Conversely, in the INFOTOK-Flex variant,
efficient approximation of its expectation. This approach allows for
21

---

Published as a conference paper at ICLR 2026
Table 4: Architecture Configuration.
Hyperparameter
InfoTok-33
Info Tok-Fless
Adaptive Conpressor/De-cooppesoor
Patch size
Encoder depth
8
Decader depth
8
Attention head
32
Token hidden size
256
ML.P hidden size
512
Positional embedding
2D
RoPE Heo et al. (2024)
Compression Factors
[3]
[0.25, 0.5, 0.75,
Parameter size
18M (14.6%)
Quantizer
Model type
FSQ Mentzer al. (2023b)
Embedding size
Feature level
[8, 8, 8, 5,
Codebook size
64000 (2
Based Tokenizer: Cosmos Agarwal et al. (2025)
Model type
3D Causal CNN (4, 8, 8) 105M (85.4%)
cemporal-spatial compression Parameter size
Total parameter size
123M
we introduce additional flexibility by randomly sampling 3 from the set [0.25, 0.5, 0.75, 1] for each incoming data instance. INFOTOK- enables the model to ensemble different compression levels, enhancing its adaptability to varying content complexities. We clip the minimal token length of each video to 1/16 of maximal token length avoid over-compression and instable training.
Adaptive Compressor. As detailed Section 3.2, we implement ELBO-guided token pruning by removing tokens with the highest log-likelshoods, corresponding to the lowest information content. This process compresses the discrete token sequence the length specified by the ELBO-base based token length selection. To achieve this efficiently, we reuse the pixel-wise ELBO values computed during the length selection phase. These values are aggregated at the patch indicated by compression level (e.g., (4, 8, 8) for Cosmos Base Tokenizer) and mapped back to their corresponding discrete tokens which are temporal and spatial aligned. Notably, this approach introduces negligible computation overhead and facilitates efficient token compression.
C.3 EVALUATION DETAILS
Inference Pipeline. For long video, we separate it as several clips of 33 frames, and reconstruct them one-by-one. During inference, given the average compression rate (measure by BPP16). we firstly determine the average token length used for video clip of 33 frames. For instance, given a clip shape of (33, 256, 256) and targeting BPP16 of 0.5625, the maximum token length is 9216 = 9 32 x 32, the average token length is 4608 9216 x (0.5625 0.0625), where 0.0625 is additional bits carrying the token location information. Then the desired token length of a certain FLBO(x)
video clip æ is 4608 x 3[LBBO[]] where ([E] is approximated by computing the average ELBO of the provided evaluation dataset during inference. Additionally, if there is only limited data provided, it can be directly approximated by ELBO(x) ())mm. which is the exponentially weighted moving average of BLBO obtained during training with smoothing factor 0.99, as mentioned in Section C.2.
Dataset Details. TokenBench Agarwal et al. (2025) is a curated benchmark designed to standardize the evaluation of video tokenizers. It comprises 500 high-resolution, long- cong-duration videos sampled from diverse datasets, including BDD 100K Yu et al. (2020), EgoExo-4D Grauman et al. (2024), BridgeData V2 Walke et al. (2023), and Panda-70M Chen al. (2024). These videos encompass
22

---

Published as a conference paper at ICLR 2026
Table Comparison Cosmos Arch with with- out InfoTokk across resolutions on TokenBench.
Table 6: Inference latency comparison across differ- ent methods.
Model
Resolution
BPP.:
Method
NFRs
Inference Lafency per
Cosmas
0.56
Cosmos TafoTok mechaniam
Cosmos Arch
360p (1:1, 43, 16:9)
1.00
31.13
Cosmus + Elasti Tok mechanism
12
13,45
Cosmos Arch InfoTok
160p (1:1, 4:3, 16:9)
0.56
30.55
56
ElastcTok
12
42.73%
wide range complex scenarios such as autonomous driving, egocentric activities, robotic manipula- tion, and web videos, providing comprehensive evaluation suite for video tokenization methods. For the DAVIS dataset Caelles et al. (2019), we utilize the Test-Dev 2019 split, which consists 30 video sequences featuring various human activities. As mentioned in Section 4.1, all videos resized and cropped into 256px to ensure fair comparison with baselines Yan et al. (2024) only supporting square video data tokenization.
D ADDITIONAL EXPERIMENT RESULTS
Ablation different resolutions. We further evaluate InfoTok across multiple resolutions assess its generalization ability, as shown in Table 5. The results show consistent trend across diverse aspect ratios higher resolutions, indicating that InfoTok generalizes well and adaptively tokenizes videos under varying resolution settings.
Inference latency comparison. We also compare the inference lateney of InfoTok and ElasticTok at the same compression rate (BPP16 = 0.56) on single NVIDIA RTX A5000 GPU, using videos of 33 frames a resolution of 256 x 256. As shown Table 6, InfoTok achieves significantly lower inference latency compared to ElasticTTok.
E USE OF LARGE LANGUAGE MODELS
We use large language models to polish our writing, e.g., to check grammar and improve paper fluency.
23


Reviews:
[{"review_id": "dznJ07O7v2", "reviewer": "Reviewer_5w2J", "summary": "In this paper, an algorithm called InfoTok is proposed for adaptive video tokenization. The proposed algorithm determines the token length based on ELBO for near optimal compression. For compression, InfoTok utilizes the ELBO based token selection. Thus, it can be integrated on the top of existing tokenizer architectures. Also, the proofs for the mathematical theorems are provided. The experimental results show that the proposed algorithm outperforms the existing methods.", "questions": "Though I have carefully reviewed the paper including the appendix, I did not observe any major drawbacks and I believe the proposed algorithm makes a meaningful contribution to adaptive video tokenization. Please see my minor concerns in the weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper clearly describes the proposed algorithm and is easy to follow.\n- This paper provides the proofs for the theorems.\n- The proposed algorithm achieves good performance on various benchmark tests.", "weaknesses": "- It would be helpful if the paper included a discussion of the limitations of the proposed approach. \n- Since the method seems general enough to be applied to images, it would be helpful to explain the rationale for limiting the experiments to videos.\n- typo\n  - L184: an more accurate -> a more accurate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uncZU69V3l", "reviewer": "Reviewer_t2hE", "summary": "This paper proposes InfoTok, an adaptive discrete tokenizer that leverages an information-theoretic formulation to allocate token budgets based on the data's compressibility. The method introduces a principled way to determine token lengths using a normalized ELBO-based router and achieves strong empirical performance with a conceptually simple approach.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well structured, and the mathematical formulation is precise.\n* It achieves competitive or superior performance using a lightweight and interpretable mechanism.\n* The authors provide rigorous justifications connecting ELBO with optimal token length, and the derivation is insightful.\n* The ELBO-based routing and token selection reuse existing encoder-decoder structures and introduce minimal inference cost, which is appealing in practical deployments.", "weaknesses": "* Is the per-token ELBO computed purely based on the encoder-decoder’s end-to-end reconstruction path? Must this be explicitly introduced during training, or can the method be plugged into any VAE-style model without changes? Specifically, can non-VAE tokenizers be adapted to this framework?\n* If N_max is small or the compression ratio is extremely low (e.g., β< 0.1), what are the observed effects on stability and convergence? Would the KL term dominate or explode in such settings, and does it impact model converging or generalization?\n* The method is designed for videos. Would it generalize to other structured data types such as audio and 3D point clouds? Do different modalities affect the ELBO-based router's assumptions or effectiveness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WRDYl8rp2k", "reviewer": "Reviewer_66nG", "summary": "This paper proposes an adaptive visual tokenizer that allocates a more optimal compressed sequence length than existing methods based on information theoretic principles. This paper demonstrates that prior works’ training methods are often suboptimal, imputing bias that makes the adaptive compressions lower-quality despite their adaptability. It further claims that the optimal design for the compressed token length should be proportional to the negative log likelihood of the input, and thus uses the learned ELBO to modulate the input compression factor. The proposed router uses this principle to more optimally allocate tokens for compression, and the top K tokens are selected based on their log likelihood. The results demonstrate that InfoTok achieves better performance at the same compression rates as comparable methods.", "questions": "I don't have any strong suggestions, but would appreciate responses to my above critiques and would like to see them addressed in the next version of the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper provides some much-needed theoretical grounding to the adaptive tokenizer space, and attempts to pin down what `optimal’ adaptive compression should look like.\n\n2) Experiments are very comprehensive and clearly demonstrate that InfoTok achieves better reconstruction at the same compression rate compared to other adaptive tokenizers.\n\n3) While the results are incremental on their own, the entire paper is well structured and motivated, and properly places all the contributions in context, making the result valuable for the community.\n\n4) While the claims in the theorems are initially hard to parse, they are empirically supported by experiments that make the intuition clearer.", "weaknesses": "1) Key definitions are not well highlighted in the text, and the mathematical sections are poorly written and hard to follow. Since definitions are missing from the theorems, the explanation is rather hard to follow. For example, the entropy H is not clearly defined, \\mathbb{D} not defined. Similarly r(N | x) not defined in Alg 1 input. While I may have missed these definitions, I would suggest not burying them in the text and making key components clear and easier for readers to refer to.\n\n2) It would be helpful if there were some visual results demonstrating the differences in InfoTok's reconstruction at different token lengths for the same image, so we could see what inputs are determined to need lower ones and which are determined to need higher ones. The reconstructions shown in Figure 2 are not particularly compelling - the compression results are considerably worse than Cosmos-DV.\n\n3) There are no video samples provided in the supplementary material or project page which makes qualitative analysis of the results challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)