Single-pass Adaptive Image Tokenization for Minimum Program Search
Authors: Shivam Duggal
Sanghyun Byun
William T. Freeman
Antonio Torralba
Phillip Isola
Venue: NeurIPS 2025 poster ()
Keywords: Adaptive Tokenization; Representation Learning; Intelligent Compression;
Primary area: applications
TL;DR: One-Shot Adaptive Visual Tokenizer

Abstract: According to Algorithmic Information Theory (AIT), intelligent representations compress data into the shortest possible program while remaining predictive of its content—exhibiting low Kolmogorov Complexity (KC). In contrast, most visual representation learning systems assign fixed-length representations to all inputs, ignoring variations in complexity or familiarity. Recent adaptive tokenization methods address this by allocating variable-length representations but typically require test-time search over multiple hypotheses to identify the most predictive one. Inspired by KC principles, we propose a one-shot adaptive tokenizer, KARL, that predicts the appropriate number of tokens for an image in a single forward pass, halting once its approximate KC is reached. The token count serves as a proxy for the minimum description length. KARL performs comparably to recent adaptive tokenizers while operating in a one-pass manner. Additionally, we present a conceptual study showing a correlation between adaptive tokenization and core ideas from AIT. We demonstrate that adaptive tokenization not only aligns with KC but also reveals empirical signals approximating AIT concepts such as sophistication and logical depth. Finally, we analyze predicted image complexity and interestingness across axes such as structure vs. noise and in-distribution vs. out-of-distribution familiarity, highlighting alignment with human annotations.

Single-pass Adaptive Image Tokenization for Minimum Program Search
Shivam Duggal
Sanghyun Byun'
William Freeman
Antonio Torralba
Phillip Isola
Massachusetts Institute of Technology
LG Electronics
Abstract
According to Algorithmic Information Theory (AIT) Intelligent representations compress data into the shortest possible program that can reconstruct content, ex- hibiting low Kolmogorov Complexity (KC). In contrast, most visual representation learning systems use fixed-length representations for all inputs, ignoring variations in complexity familiarity. Recent adaptive tokenization methods address this by allocating vuriable-length representations but typically require test-time search over multiple encodings to find the most predictive one. Inspired by Kolmogorov Complexity principles, we propose a single-pan pasi adaptive tokenizer, KARL, which predicts the appropriate number of tokens for image in single forward pass, halting once its approximate KC reached. The token count serves as proxy for the minimum description length. KARL's training procedure closely resembles the Upside-Dove -Diide--own Reinforcement Learning paradigm, as it learns to conditionally predict token halting based desired reconstruction quality. KARL matches the performance of recent adaptive tokenizers while operating in a single pass. We present scaling laws for KARL, analyzing the role of encoder/decoder size, continuous vs. discrete tokenization and more. Additionally, we offer conceptual study drawing an analogy between Adaptive Image Tokenization and Algorithmic Information Theory, examining the predicted image complexity (KC) across axes such as structure vs. noise and in- vs. out-of -iii-r---itttttiioon familiarity revealing alignment with human intuition.
Code: https://githutt wtt.//oo///iiiii www//hhaaaammggagaaaao/oooooooo tokenizer. Keywords: Representation Learning, Adaptive Tokenization, Compression, Algorithmic Information Theory, Kolmogorov Complexity, Upside-Dow RL.
Introduction
Compression lies at the heart of intelligence [1-3]. The universal intelligence framework from Legg and Hutter [4], inspired from the Solomonoff Prior, formalizes general intelligence as an agent's expected performance across wide range of computable tasks environments, each weighted by its simplicity-
precisely, by Kolmogorov complexity [5]. Agents are thus rewarded for performing well on environments that are easier to describe and penalized less for failures on highly complex ones. The core principle behind this framework is formalization of Occam's 's razor [6] inherently more plausible and tend to generalize better, making them
simpler explanations more desirable to optimize. This simplicity bias underlies much of machine classical dimensionality reduction [7, 8] to modern representation learning [9] - all of which aim to capture the underlying regularities in data using as few components as possible. In this view, intelligent data representations capture the structure of their input in the simplest, most compressed form iidn--dyllly with minimal Kolmogorov Complexity (KC) or its practical counterpart, Minimum Description Length (MDL)-whil preserving the information necessary for downstream tasks, effectively performing task-aware lossless compression.
from
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Task: Given input image desired reconntruction quality, output the min. takens required to reconstruct the image.
Matryoshka: Flextok, One-D One-DePiece
Recurrent tokenizer: ALIT
KARL (Ours)
Desired
Recor. Quality
(default=B.05)
Encoder
Encoder
loop till clasied Recon. Quality
Loop sill desired Recon. Quality
Decoder
Decoder
Decoder
X Smaller embeds consirained as subsett of
Kalmogoroy Complexity a ligned. X Requires multiple Encoder & Decoder runs identify smallest #tokens.
Ko Koinogorov Complexity aligned
larger takens refines the inibial Not aligned w/ Prefix KC.
8: inspired.
Single-pess.
X Requires multiple Decoder runs to iderntify smallest Itokens.
Figure 1: Comparing adaptive tokenizers: Maryyoskka-style methods constrain smaller token sets to be subsets of larger ones and require multiple decoder passes. Recurrent approaches like ALIT perform iterative encoder-decoder loops to meet reconstruction quality. In contrast, KARL employs a single-pass encoder guided by Kolmogorov Complexity, avoiding iterative refinement. KARL'S encoder outputs both token embeddings & halting probabilities; the non-halted tokens constitute the minimal program sufficient to reconstruct the image to the target quality using the decoder.
In this work, we focus on visual representation learning from the perspective of maximum compression. Most existing representation learning algorithms- -uchh as SimCLR [10], DINO IJ, MAE [12], VAE [13]. VQGAN [14] - employ objectives like contrastive learning, self-distillation, or reconstruction to map input images into fixed-length embeddings. While these methods have proven effective for downstream tasks such as classification, VQA and image generation, they produce representations of uniform length for all images, regardless of image complexity, familiarity, or task relevance. This uniformity contradicts the principles of Kolmogorov Complexity, which defines the complexity of an object the shortest program to generate it.
Recently, there has been growing interest in adaptive tokenization, which assign variable- length representations different images at test time. These approaches are more aligned with the principle of Occam's 's razor and KC, as they encode each input with variable number of tokens required to preserve essential information. They also resonate with Epicurus's principle [15], which encourages retaining multiple possible explanations - Adaptive tokenization can thus be seen as dynamically selecting among multiple candidate representations depending on image complexity familiarity.
Among adaptive visual tokenizers, two broad families have emerged: Matryoshka-style [16] or prefix- based models (or fail-dropping representations) [17, 18], and recurrent or iterative tokenizers [19], which resemble Turing machines in their step-wise construction of a representation. Fig. I compares these tokenizers. The first category learns large fixed-length embedding for each image, designed so that any prefix-or subset-caa can serve as a valid explanation. While this offers a mechanism to approximate the minimum required representation length, it assumes that all explanations are nested within single. maximal one. This contradicts core property of prefix Kolmogorov complexity [5, 20], which states that the shortest valid description may not be subset of longer ones. Such nesting can thus impose a strong inductive bias that limits alignment with the principles of minimality and intelligent behavior - any additional token utilized for an image has the role of refining the existing representation, rather than enabling the discovery of a potentially more optimal solution under increased memory constraints.
contrast to Matryoshka-styl approaches, recurrent or iterative tokenizers progressively allocate more tokens memory over multiple network iterations to represent an image, typically halting once a target reconstruction loss is reached. Methods such as ALIT [19] and Tok [21] fall into this category. Although these recurrent approaches are more faithful to the principles of Kolmogorov Complexity, both categories of approaches incur a significant practical cost when adapting representation lengths to input images - identifying minimal embedding requires repeatedly rolling out the encoder-decoder pipeline, which limits their efficiency in real-world applications like VLMS & video models, demanding fast and lightweight inference.
Contrary to prior approaches that rely on iterative search to generate maximally compressed represen- tations at test time, we propose KARL- Kolmogorov-Apprxximaaii Representation Learning.

---

Late Latent-Din Distiiatiionn Encoder Image > 1D Toxeros
Latent Anlen-Dieiilatinn Dwordlr 1D Tokens > imige
Compute Racoresti cututiiion Loke 5 linput Reconst))
Lateet- Latent-Deerlllnn Diviilltion Encorder imagn Takens
Latent Latent Divellation Decoder "D Toklere 7 Image
AT Adddd totams ftoal T .T to no)
AND
Hardd
sampled
Estimate Image Complexity (KC)
Learn to Takenize Complexity
Alms (andl favis at) Lossless Compression at Now Input tokens
Learns to hal the deefined complexity by masking extra tobans
Input- Image, Randiamly Sampled takam (AT), Recon. Thrushs 0
Input - Image, (T+ AT tokams), Reconsmuttion Thres = Output Tokun embedddngs ane takan maxcinng probubillty
Output- - Reconsruution loss m iy define KC.
Rt-Shortest Program for Image Recons. loss #T tokens
(C, ItokenSsuper 7) BC. (image, #tokontoaper T AT)
Figure 2: KARL- -a a one-pass adaptive tokenizer inspired from the principles of KC in being able to use only the sufficient token count for an image, halting any extra token beyond the image's s complexity. The training procedure operates by first attempting lossless compression and later using the failed reconstruction loss measures as guiding signal to halt any extra token than provided during the lossless compression attempt. Such training strategy optimizes the network for reconstruction tasks closer to an image's inherent complexity, implicitly guiding it to allocate tokens adaptively.
a novel method for carppeey-comppressed visual representation learning in a single forward pass. Inspired by the principle of Kolmogorov Complexity (KC), KARI, learns to approximate the minimal number of tokens required to reconstruct an image up to a target reconstruction loss. To achieve this, we introduce a loss-conditioned supervision strategy that trains the model to identify and mask unnecessary tokens in a fully differentiable manner. Each training iteration follows a two-phase process resembling Schmidhuber [22]'s upside-down down reinforcement learning (RL) paradigm: first, the model estimates image complexity by attempting near- losslesss reconstruction under a randomly sampled token budget; the resulting reconstruction loss then serves the task condition for the second phase, where the model learns to achieve the same quality using more tokens while halting the surplus ones. This setup enables the model adaptively allocate tokens in a single forward pass by treating the reconstruction loss as a task-specifying reward-akk akin to upside-down Rr. At inference, given image and desired reconstruction quality, KARL'S encoder predicts both token embeddings and halting probabilities. Tokens with high halting probability are excluded from decoding, yielding an efficient, adaptive representation. In effect, KARL implicitly searches for the shortest sufficient program-outpune only the necessary tokens to match the target quality (see Tab. 2).
We compare KARL both quantitatively (Tab. 1, Tab. 2) and qualitatively (Fig. 4, Fig. 15, Fig. 16) against recent adaptive tokenization methods using standard image reconstruction metrics. KARL performs competitively across all single-image metrics LPIPS, SSIM, PSNR, and DreamSim. We also compare these tokenizers in terms of the minimum number of tokens (KC) required to reconstruct the input image to given reconstruction quality (Tab. 2, Fig. 17, Fig. 18).
Beyond introducing KARL as one-step adaptive tokenizer, draw connections between adaptive image tokenization and the framework of Algorithmic Information Theory (AIT) [15]. In particular, the minimal token count predicted by the adaptive tokenizer can be considered an approximation of the incomputable notion of Kolmogorov Complexity (KC). We perform both quantitative and qualitative analyses of KARL's approximation of KC through Fig. 5, Fig. 7, Fig. 17, and Fig. 18. Additionally, KARL'S complexity strong alignment with human annotations of visual complexity (Fig. 14). Beyond Kolmogorov Complexity, we suggest that adaptive tokenizers may also capture higher-order AIT concepts such as Sophistication and Logical Depth, though this remains promising direction for future work. Overall, our work positions adaptive tokenization, KARL in particular, as a means of learning intelligent representations: maximally compressed yet predictive, and in alignment wtih the principles of AIT.
by the number of latent tokens
2 Related Work
central goal in representation learning and tokenization is dimensionality reduction-mappee high- dimensional inputs to lower-dimensiona. but structured manifolds. These compact representations believed to generalize better and are widely used to train more efficient and effective neural networks. As result, compressed visual learning is increasingly important in the current landscape of AI. Focusing primarily on reconstruction-basse tokenizers, the predominant approaches include VAE [13] and VQGAN-style [14] models. Compression in these architectures is typically achieved
3

---

Input & Preprocessing: Input image img: token budget T; desired loss
Convert img to ing_ tokens é 12 Bx x
Estimating image complexity by aiming Lossless Compression under budget T
1: // Step 1: Token Initialization
Initialize E R ggxxxx as Enit_tokens
2: // Step 2: Initial Encoding with Zero Epsilon (Best possible reconstruction under budget T
Zene f Encoder(x, Zo, 0); imgo + Decoder( (exx)
3: Step 3: Encoder maps image + T tokens. Decoder transforms T tokens + image.
Compute Reconstruction Error
60 img. img
I Learning to tokenize complexity by masking additional tokens: Increased token budget T AT
4: // Step 4: Token Re-Initinn ne-Inntialization
Extend the token budget: +- append_ tokens:
E Bx(T+AD))
5: // Step 5: Use the minimum required token count to achieve co: KC =T by design
Zíinaly Wina f Encoder (x, 2 = 60)
6: Step 6: Sample learned token embedding with halting probability, Winal <, 0.75
Zminimal Index ZamiV (Witmal < 0.75)
7: img <- Decoder(Zuiniml))
8: // Step 8: Optimize using Reconstruction Losses & Cross-Entropy on Wiinal
Halting probability Winal for last AT tokens (extra budget), the first T tokens.
Figure 3: Training algorithm. KARL follows an upside-down RL paradigm (a) Estimate Image Complexity stage samples task-defining inputs (image, token budget T, reconstruction error co by attempting lossless compression (e 0). (b) Learn to Tokenize stage trains the mooel--cnnditionel on 6o-to match the same quality using T+AT tokens while halting the extra (rightmost) AT.
through spatial down-samplin in convolution-based encoders or via fixed patch-to-token mappings as vision transformers [23] - after which no further compression occurs and the initial token count is preserved. This rigid patch-token binding limits the ability to compress the input image effectively. The absence of token-level compression may pose challenges for scaling to large vision tasks, such as video generation and understanding. A widely accepted alternative to patch-token binding is Perceiver's [24, 25] ID tokenization strategy, which maps all inputs to fixed-length 1D sequence enabling modality- agnostic compression. Recent works like Titok [26] and Fo [27] follow this paradigm. While these methods achieve better compression than 2D patch- based systems, they allocate fixed token capacity all images.
Intelligent representation learning should aim for maximum compression conditioned on the input and task. Adaptive tokenization approaches [17-19, 21, 28] address this by training tokenizers to map images to a variable number of tokens, enabling per-image adaptive inference. Among these, Matryoshka-styy prominent- they assume that any subset of a full embedding should remain representative. However, discussed earlier, deriving compressed representations striet subsets of longer ones may yield optimal token allocations. An alternative strategy is iterative adaptive tokenization, where tokens (or memory) incrementally allocated over multiple steps. ALIT [19], for example, introduces RNN-based Transformer for adaptive-length image tokenization.
models such FlexTok [17], One- D-Picee [18], and ElasticTokk [21]
Both category of methods, however, require iterative inference involving multiple encoder-decoder passes to assess image complexity and assign representations accordingly. recent work, CAT [29], instead leverages LLM to predict image complexity before selecting the appropriate nested VAE level for compression. In contrast, we propose a single-pase adaptive tokenization strategy that implicitly learns to sample only the required token count per image- -offeringgig an efficient mechanism to estimate input complexity via the approximated minimal program size. Furthermore, draw connections between adaptive image tokenization and foundational concepts in Algorithmic Information Theory (AIT) [15], providing a fresh perspective on visual learning.
3 Kolmogorov-Approxinatiio
Representation Learning
Representation learning seeks to map input observations to compact, meaningful embeddings that re- tain essential information for downstream tasks. Our goal to learn intelligent data representations that are both highly compressed & predictive. We formalize this as a dual optimization problem:

---

Input Image
One-D-Piece-32
ALTT-32
KARL-32
One-D-Fiece-256
AUT-256
KARL-256
Figure 4: Reconstruction Analysis on ImageNet100: KARL and ALIT produce higher-quality reconstructions than One-D-Piece, with KARL achieving strong single-image metries. One-ID-Piece consistently generates blurry images at low token counts satisfying FID metric more.
minimizing reconstruction loss to ensure the latent representation faithfully captures the input minimizing the representation length to achieve maximal compression.
In this work, we focus exclusively reconstruction- based encoder-decoder approaches (image + embedding or tokens + image), as they naturally align with the goal of lossless compression. Moreover, since fixed-length representations violate the core principles of Kolmogorov Complexity (KC), this work centers on variable-length representation learning through adaptive tokenization. Recent adaptive tokenizers rely on iterative encoder-decoder runs inference time identify minimum tokens to reconstruct an image up to a desired quality. In contrast, we pose the following question: Given token budget, can we determine in single shot how many tokens are sufficient to form an intelligent representation-On one that is both maximally compressed and predictive?
Before diving deeper into this question, it's important to acknowledge two fundamental challenges: both identifying the smallest representation length and perfect lossless compression in deep learning models are practically unattainable. We leverage these limitations as a tool in designing tokenizer to perform minimum program search !. We propose a loss-conditioned adaptive tokenization ap- Representation Learning (KARL), which learns
proach, dubbed aalmogorov-Approxinatin to approximate the minimum description length of an image-defined as the minimal number of tokens seaded-subject to given reconstruction threshold. More formally, KARL an encoder- decoder architecture that, at test time, takes an image, a maximum token budget, and satisfiable reconstruction loss threshold input, and outputs only the necessary tokens (i.c., the approximated minimum program size), while masking out the rest. The use of loss-cone poss-onndiioonin with variable reconstruction thresholds can also be interpreted as prompting the model with alternative tasks, since different downstream applications in computer vision require different levels of detail to be preserved. KARL produces variably compressed representations bused on the target loss magnitud--alloctinng more tokens for harder tasks-uligg aliining with the spirit of Universal Intelligence frumework [4].
How do train KARL? We start by outlining the encoder-decoder architecture used for adaptive tokenization, which maps images to ID latent tokens. Building on this, we introduce our loss-conditioned tokenization strategy, which enables training of the proposed tokenizer that learns to approximate the minimal tokens necessary to satisfy given reconstruction threshold.
ID Tokenization: Like most recent adaptive tokenizers and inspired by the Perceiver model [24], KARL learns to distill an input image into ID latent tokens that are not constrained to fixed spatial patches. We first map the image to grid of 2D latent tokens using pretrained VAE / VQGAN, resulting in 256 tokens for a 256 x 256 image. These 2D tokens are concatenated with a set of initialized ID latent tokens along the token dimension and passed to latent distillation encoder-4 mansformer-based encoder that performs full self-attention across all tokens. The goal of the encoder to distill the 2D image tokens into sariable-lengti ID representation. For reconstruction, decoder with the same architecture performs cross cross-attention between the masked 2D token grid
We use minimum program size, MDL, minimum tokens approximated KC or KC interchangeably.
5

---

Input
c-0.0
c-0.0
<-0.0
c-0.05
lossless compression full input tokens.
optimiaing minimization while trying f
ATokem
KG
4
KC
Chesebooni (OOD)
0.14
64
0.06
128
0.00
256
0.02
0.024
224
0.09
112
0.21
54
Mandelleve Fractal
0.07
0.05
128
0.04
256
0.43
0.034
256
0.099
98
0,770
32
Noiso
0.12
0.12
128
0.12
256
0.11
0.111
256
0.110
256
0.110
256
Dog from ImageNer (LD)
0.10
0.08
0.88
0.06
0.060
256
0.059
0.065
224
Green: ive recuth wchen wither +0.01 Stokerss Only fallure occurs for large OOD chessboard.
128 tokens
256 tokens
takens
Figure 5: Estimating KC across structured, noisy, and OOD inputs. Recon, quality and KC vary by image type and token count. The dog (IID) compresses well; the OOD chessboard does not. High Noise and high structure both yield high KC (like theoretical KC), but adaptive tokenizers can distin- guish: for noise, increasing tokens doesn't 't improve ( i.e. A = & (256 tokens) - (32 tokens) = O. For redundant chessboard structure, A turns be too high meaning what's 's actually interesting lies in the mid range suggesting potential to model AIT notions like Sophistication. Logical Depth. Table on the right shows how well KARL minimizes KC under the constraint f S 6 for diff. input c.
(representing positional structure of the original 2D image tokens) and the encoded ID latent tokens, allowing the model reconstruct the image via predicted 2D tokens and VQGAN VAE decoder.
With the goal of training a one-shot adaptive tokenizer, the encoder is tasked with not only predicting the embedding of each ID token, but also estimating halting probability for each token. If token's halting probability exceeds a given threshold, it is considered inactive imactive-excludde excluded from decoding and thus does not contribute to the shortest program of the image. However, we do not have direct supervision for these halting probabilities. Furthermore, learning when to halt or mask using only a reconstruction objective is non-trivial: masking is non-differentiable, and common techniques like reinforcement learning or straight-through estimation [30] are highly unstable, especially alongside latent quantization [14]. How do we train the model to mask out tokens, enabling one-shot prediction of sufficient token count? We address this by framing the problem as an upside-down RL formulation [22] and introducing loss-conditioned training strategy inspired by the principles of Kolmogorov Complexity [5]. Fig. 9, Fig. 3 showcase the computational graph & the training algorithm.
We define the approximate Kolmogorov Complexity (KC) of image under a reconstruction threshold and maximum given token budget, T as:
Loss-Conditioned Training of KARL:
KC, (x, 7) min {t < T' (re .x)) S })
where é is the reconstructed image using only of the available T representation tokens, and L is the reconstruction error (pixel-wise error in our setup). In essence, (x, T)) denotes the smallest token count sufficient to reconstruct the image within the given error bound e, while respecting the token budget T. This provides a formal, task-specific interpretation of KC that inspired our training.
Now, encourage the network to use (fewer) T and successfully mask out the extra (T - f) tokens, we set up the training procedure inspired by the following KC invariance property:
(x, = KC,(z.) T AT)
i.e., increasing the input token budget should not change the KC, an inherent property of the image. This ensures that once the model has learned the min. sufficient number of tokens / reconstruct the image within the given error threshold (e), adding more tokens is not required to repeat the same task.
Training Procedure: To train KARL, each training iteration involves two runs of the encoder-decoder pipeline, reflecting the principle that Kolmogorov Complexity (KC) remains invariant once the model has identified sufficient token count reconstruct the input. In the early 2000s, Vereshchagin and Vitanyi [31] highlighted the role of Kolmogorov Complexity, particularly the Kolmogorov Structure F Function, in lossy compression (see their Example 11.4), providing support for proposed approach.

---

1. Estimating Image Complexity: Given an input image x, the model first attempts near-lossless compression using a given token budget T. It trained minimize a standard objective that combines reconstruction and latent quantization losses:
LECC Lrccon (xccon(xxT. x) + 3 Lquan(Zr)
where XT = Decoder(Encoder/) T. = 0)), and | balances the quantization loss. The condition- ing parameter = O directs the encoder to aim for perfect reconstruction. Let the resulting empirical reconstruction error be recorded as 60 | - x|; this serves as the target reconstruction quality for the second run. By randomly sampling the input budget T in each iteration, we simulate triplets (image, estimated complexity T, reconstruction error co) for all images the batch, which form the basis for learning to tokenize by complexity. Fig. 6 showcases representative batch output from the Estimate Image Complexity (EIC) phase, which serves as input to the subsequent Learning to Tokenize (LTC) phase within the same training iteration.
Note: Eo and Crcon need not be the same. In most experiments, the loss used define the re- to compute training
construction condition (eg) is the image-level & loss, whereas gradients- can vary: it may be loss over reconstructed 2D tokens, image-level é,, or a learned perceptual or adversarial loss. For exact formulations of Lrecon: see Appendix A.2.
2. Learning to Tokenize at Estimated Complexity: In the see- ond run, we take the generated simulated triplet of image, complexity T. and predicted error ep, and increase the encoder's 's input token budget to T + AT. The encoder, conditioned on co. predicts both the token embeddings & halting probabilities w E [0,1]T+AT, in- dicating which tokens are essential for reconstruction. To encourage the model to disregard additional AT tokens, we optimize the loss:
Exemplar Training Tuples / Batch output of Estimate image Complexit phase input of Learning to Takaniao phase
0.3
U.0G
LITC- Lreemn (XT=nn x) + B Lquant (ZT+ + X Lhan(w)
where XT+ = Decoder En (E (x, T + AT, = The encoder is conditioned on the target error obtained from the first run, encouraging it to match the same reconstruction quality while identifying which tokens are unnecessary. During decoding, halted tokens are excluded from the self-attention operation and therefore do not contribute to the minimal program. The halting loss is defined as:
Figure 6: EIC phase provides the task-conditioning (T and co) for the second LTC phase.
( BCE( 0) BCE(WT.T-AT.1
encouraging the model to assign low halting probability to essential tokens (w; 0 for T) and high halting probability to redundant ones (wi == 1 for i > 7'). The total training loss for each iteration is Cac + Cire. The overall training procedure parallels Upside-Dow Down RL, where the first phase generates a task-defining
recoostrnntt quality-analogeee -nallgooss to how Schmidhuber [22) reformulate RL as supervised learning by treating rewards as input conditions. At inference, only the second adaptive tokenization run is executed: given a maximum token budget and target recon. threshold c (default = 0.05), KARL predicts both token embeddings & halting prob. in a single pass.
The procedure induces self-supervised curricu- lum in which the network is only tasked with compressing data to the extent actually reasible-relative to the true complexity of the image. Consider highly complex images, which cannot achieve low error with a small token budget: these are never paired with (low-loss, during the second training run. As a result, the training signal is either a strict zero-loss target (for near-pert sear-perfect reconstruction) or threshold derived from a failed attempt at lossless compression. In both cases, the network is optimized for reconstruction tasks that match the image's 's intrinsic complexity, implicitly guiding it to allocate tokens adaptively.
This training reveals an important phenomenon:
count) condition
4 Adaptive Image Tokenization meets Algorithmic Information Theory
From the perspective of representation learning, several seminal frameworks such as Algorithmic Information Theory (AIT), Solomonoff Induction, and the Universal Intelligence framework center on the principle of simplicity: seeking the shortest possible program to reproduce a given data instance. This principle is formalized as Kolmogorov Complexity (KC). As presented earlier, despite its intractability theory, KC models an interesting signal for representation learning, enabling simpler / compressed representations. In this section, draw analogies between adaptive tokenization & KCC
7

---

KC = 32
KC = 256
KC = 32
RC = 256
Figure 7: Imagenet (IN) from Lens of Kolmogorov Complexity: We bucketed images from different IN synsets based on their predicted approximate complexity KC, with images sampled using KC.=0.07 (x, T = 256) = 32 (row 1) or 256 (row 2). Aligned w/ human observation, KARL allocates fewer tokens to simpler images (row 1). KC of Imagenet Rs 146 tokens/image on average (Tab. 2).
Correlation with intractable Kolmogorov Complexity principle: Kolmogorov Complexity (KC) formalizes the length of the shortest program that can reproduce a given data instance. In the context approximate this notion by treating the number of tokens as a proxy for program length- each token representing a discrete, structured unit of information. Reducing the token count while still representing the input is thus analogous minimizing its description length and hence its Kolmogorov Complexity. Let's understand this analogy in more detail.
of adaptive tokenization,
More formally, "program" for reproducing an image should include both the token sequence and the decoder that maps tokens back the image. The encoder, this view, acts more like a program synthesizer-searchhe
for the token string that satisfies reconstruction objective. Since the decoder is amortized across the dataset and shared across all samples, its contribution to the overall description length can be considered constant, and is therefore ignored in our analysis (we ablate the effect of varying decoder complexity later via Fig. 12 and Fig. 13). Secondly, reproducing data perfectly is not feasible in most deep learning settings. If we were to insist exact reconstruction, the required token sequence- thenecceeeeeeeeee -worramm-woudd grow unbounded in length. Hence, we redefine the goal of "data reproduction" as achieving an image reconstruction within specified error threshold (e.g., l loss or perceptual loss), making minimal program length both tractable and meaningful in practice. This formulation also parallels the use of the Kolmogorov Structure Function (KSF) [31] in lossy compression (see Example II.4), where KSF(æ,, a) denotes the logarithm of the size of the smallest typical set containing such that the Kolmogorov complexity of S (KC(S)) is most a. Here, acts as complexity budget, and the goal is to find a minimal set S that sufficiently "explains" x an idea analogous finding the shortest token program that reconstructs æ within target loss.
Searching the shortest program- Algorithmic Probability frameworks like Solomonoff Induction (SI) aim to find the simplest (i.e., shortest) program that explains observed data by brute-force enumerating over all programs which, when decoded using a universal prefix Turing machine, represent the data (or its prefix). Shorter programs assigned higher probability, making SI formal realization of Occam's 's Razor, an inductive bias toward simplicity. In high-level alignment with SI, modern adaptive tokenizers [19] approximate this search by varying the token budget test time, iteratively increasing token count until the reconstructed image reaches a desired fidelity threshold.
The proposed KARL framework approximates this shortest token subset search through one-shot selection mechanism: predicts which subset of the input tokens is sufficient to reconstruct the image within defined loss threshold (e.g.. l loss or perceptual loss). By conditioning the encoding process on the reconstruction constraint itself, KARL avoids exhaustive sampling multiple hypotheses and instead adapts the representation in single forward pass. This allows KARL to approximate the behavior of Solomonoff-sty inference without the computational burden of iterative decoding.
The reconstruction threshold or the definition of data reproduction can be altered based on the downstream task by fine-tuning or training KARL accordingly. Finally, it important to emphasize that these minimal token representations are only approximations of the formal theories, as deep
8

---

Train Set Approach
L1
Loss x
10 4
LPIPS 4
SSIM 1
Dreamsim
32
128
256
32
128
256
32
128
256
32
128
256
ALIT
1.17
0.87
0.76
0.25
0.20
0.16
0.32
0.40
0.44
0.26
0.14
0.10
IN100
KARL
1.13
0.80
0.70
0.30
0.19
0.14
0.34
0.44
0.49
0.28
0.16
0.11
One- D--eeee
1.34
1.02
0.94
0.32
0.21
0.19
0.32
0.38
0.40
0.29
0.14
0.12
INIK
ALIT
1.13
0.85
0.75
0.15
0.18
0.29
0.33
0.42
0.45
0.24
0.13
0.10
KARL
1.10
0.81
0.68
0.29
0.18
0.14
0.35
0.44
0.50
0.27
0.14
0.11
Table I: Reconstruction Metrics on ImageNet100 using same token count for all images: KARL
performs equally well as ALIT and One- Dieece, while being one-pass adaptive tokenizer.
Tokens Used
Number of Enc Dec Runs 1
LPIPS
SSIM f 0.05
Approach
0.03
0.05
0.09
0.03
0.05
0.09
0.03
0.05
0.09
0.03
0.09
ALIT
251.5
227.2
136.6
7.86x2
7.01X 2
4.21 2
0.16
0.17
0.22
0.44
0.43
0.39
KARL
252.9
229.6
152.5
1 1-1 1
1+1
1+1 1
0.14
0.15
0.21
0.49
0.47
0.42
One-D-Piece
255.1
247.9
178.4
7,97
+11
7.74
5.57 +1
0.19
0.19
0.22
0.40
0,40
0.38
ALIT
250.0
221.3
128.8
7.81
x2
6.91 x
4.02
x
0.15
0.16
0.21
0.45
0.44
0.40
KARL
252.1
225.2
146.2
1
1+1
1+1
0.14
0.16
0.21
0.50
0.49
0.42
Table 2: Recon. Metrics on IN100 sampling variable token count for different images: All
approaches perform equally well. Compared to ALIT & One-D-Piece, KARL can one-pass generate images satisfying recon. loss threshold. First two rows are trained on IN IN100 vs rest on INIK.
learning dearning-based tokenizers provide no guarantees of true minimality. For instance, consider the Mandelbrot fractal shown in Fig. 5, which theoretically has extremely low Kolmogorov Complexity, yet still requires approximately 98 tokens to achieve a reconstruction loss around 0.05.
Factors influencing learned estimates of minimum token count or Kolmogorov Complexity: The proposed measure of image complexity depends on how well the tokenizer maps image to its learned codebook space. Consequently, the estimates are influenced by whether the input is in-distribution (IID) or outoof-distribution (OOD). For example, in Fig. 5, the dog and chessboard images illustrate this difference: despite the chessboard having a lower true KC, a tokenizer trained on ImageNet compresses dog image more rigorously due to its higher similarity to the training data, Although this is an advantageous property of adaptive tokenizers cernniiers-distinusissies distinguishing IID vs OOD data by assigning fewer tokens to IID, the gap with true KC can be reduced by training on larger datasets.
Another key factor is the amount of structure versus noise in the image. Like theoretical KC, our learned complexity measure is sensitive nolme-nKo Complexity does not distinguish between a highly structured image and pure noise; both can exhibit high complexity. Although higher-order AIT concepts such Sophistication and Logical Depth aim make this distinction, detailed exploration of them is beyond our current scope. A preliminary way to separate structure from noise using adaptive tokenizers could be analyze how image reconstruction metrics (e.g.. SSIM, ( change with the number of tokens. For example, while KARL assigns the maximum token count (256) to both a chessboard and a noise image, SSIM improves significantly with token count for the chessboard due to its simple, redundant, and identifiable patterns wwereas SSIM remains largely unchanged for pure noise (Fig. 5). Aligned with the theoretical concepts of sophistication and logistic depth, more interesting images are those that exhibit a mix of predictable and unpredictable structure, often resulting intermediate values of A reconstruction with increasing tokens. For instance, images like dogs or Mandelbrot sets (Fig. 5 rows 3,4) are more interesting to KARL. than a simple chesshoard or pure noise (Fig. 5 rows 1,2). This suggests that the rate of improvement in reconstruction quality as the token budget increases can serve as a heuristic for identifying meaningful structure, further reinforcing the potential alignment between adaptive tokenizers and AIT. A deeper investigation into these higher-order AIT concepts represents promising direction for future research.
5 Experiments & Ablations
Image Reconstruction: We compare KARL against recent adaptive tokenization methods: ALIT-a recurrent tokenizer [19], One-D-Piece-a matryoshka-style method. These baselines are representative of other approaches like Elastic'Tok [21] and FlexTok [17], and enable a fair comparison since all three use a VQGAN-based 2D tokenizer, similar encoder/decoder architectures, token quantization, and GAN-based optimization alllowwiniinnlallnnnnggao minimum program search. See Fig. for a conceptual comparison of KARL against these approaches.
to focus the study specifically on
We perform two types of comparison through Tab. 1 and Tab. 2. Tab. 2 reports reconstruction metrics across varying token counts, using the same token count for all images. All methods perform comparably, with KARL slightly outperforming on the majority of single-image metrics-La LPIPS,

---

Choosing same tokens for all images
Choosing variable tokens for all images
1.20
Dre Duptt- =i] Booh +
One Depti Dac begin
0.4
Eesivel Ene Dapih+& DOed
Bn e Encooph Dep
22.5
DreamSIM
1.23
23.0
DreamS
2:20
17.5
2.10
FID
12
0:19
13
0.16
2333
13.0
12
230
1/8 140 192 224 254
2.5
66
192 # Tokens
16.
126
193
224
254
030
0.11 0.9 0,5
0:5 085
62)
011 029 0.2
045 Edda ent
# Tokens
Recons. Threshold
Reconst. Threshold
Figure 8: Variable-token count allocation enables better scaling laws: Mapping highly complex
image to a fixed, small token count is sub-optimal. The plots on the left use a fixed token count for all images, potentially suffering from such sub-optimality, leading to such poor scaling behavior compared to variable per-image allocation. Scaling Law: Small Enc Large Dec performs the best.
and SSIM-while ALIT leads marginally on DreamSim. We report FID in the supplementary. as it poorly reflects per-image reconstruction and is misaligned with our goal of minimal program search under Kolmogorov Complexity (which seeks programs that represent data, not distributions). As shown in Fig. 4, visual comparisons highlight the strong reconstruction quality of KARL and ALIT over -ne-Deeiece, despite One-D-Piece slightly outperforming on FID.
In the second set of experiments shown in Tab. 2, we allocate a variable number of tokens per image based on a target LI reconstruction loss condition. For ALIT, this requires multiple encoder-decoder runs until the reconstruction target met - about 4x more runs than KARL for 0.09 threshold, and up to 8x more for lower thresholds. One-ID-Piece, like other matryoshka-style methods, produces encodings in a single pass, but identifying the suitable representation still requires rolling out the decoder multiple times (2-4x on average over 5K IN100 images). In contrast, KARL produces a token embedding in a single pass that satisfies the reconstruction loss constraint, with only minor deviations from the target in few cases. Metric-wise performance remains comparable across methods, but KARL offers the most practical one-pass inference. We believe its training strategy can be adapted for downstream and intelligent tasks, such vision-language models (VLMs).
Representation Learning: The minimum pro- gram search for an image depends not only on the latent embedding but also on the decoder, which acts like a programming language. According to the invariance theorem in AIT [15], the true Kol- mogorov Complexity (KC) includes a constant term tied to the decoder's design. The encoder, while not directly affecting KC, serves as the program wynthesizer-maknng its capacity worth studying alongside the decoder. That said, since the encoder, decoder, and quantization codebooks are shared (amortized) across all images, we approximate image complexity using only the latent token count.
Scaling Laws for Kolmogorov-Approxingg
While prior work emphasizes the importance of decoder size [17, 32], we instead investigate the role of encoder capacity by comparing small vs. large encoder models (Fig. 8), keeping the decoder fixed. Please refer to the appendix for other scaling law ablations (Fig. 10, Fig. 11 and Fig. 12, Fig. 13). The left pair of plots allocates the same token count to all images and shows negligible difference reconstruction
between models. In contrast, the right pair allocates variable token counts based loss threshold, revealing a clear advantage for the smaller encoder. The smaller encoder's 's superior performance is intuitive
the task of latent-distillation encoder is to simply distill VQGAN tokens to ID tokens, a task that does not require high capacity. But what enabled the right-pair experiments to surface this difference which the same per-image token allocation strategy failed to identify? We attribute this to the variable token allocation strategy - letting the network choose the optimal token count itself rather than evaluating every image under all token counts. It may even be suboptimal to test a highly complex image w/ fewer tokens. Variable token allocation avoids such sub-optimal tests. Please refer to the Appendix for a lot more experiments.
6 Conclusion
We introduce KARL- one-pass adaptive image tokenization approach that dynamically allocates a variable fraction of the input token budget based on the complexity of an image. KARL introduces a novel training strategy, resembling upside-down reinforcement learning: it first attempts lossless compression, then uses the reconstruction delta as guiding signal to learn when to halt the allocation of additional tokens. Our method achieves performance comparable to recent adaptive tokenization techniques across standard reconstruction metrics. Beyond empirical results, we offer a conceptual bridge between adaptive image tokenization and the seminal framework of Algorithmic Information Theory interrprrrn active token count an approximation of Kolmogorov Complexity.
10

---

7 Acknowledgments
This work was supported by the Defence Science and Technology Agency, Singapore; Navy ONR award N00014-22-1-2744 1-2740, the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number TA8750-19- 1-000--9----000000 and part by the NSF Award 2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions). We are grateful to LG Electronics for a funding gift and to Jung Guack (LG Electronics) for assisting Sanghbyun in rumning KARL on their cluster. We also thank our lab members Chris Hill, Elliott, Erin (TIG), Ishaan, Jyo, Haoyu, Manel, Shamit, Tianwci, Tianyuan, Xingjian, and Yichen.
References
[1] Marcus Hutter. A theory of universal artificial intelligence based on algorithmic complexity, 2000.
[2] José Heraández-Orallo and Neus Minaya-Collado. A formal definition of intelligence based on an intensional variant of algorithmic complexity. 2003, URL attps://apis semanticscholar org/CorpusID 18: 18291484.
[3] Matthew V. Mahoney. Text compression as a test for artificial intelligence. In AAAI/AAА,, 1999. URL https://appl. sewanti cschoolar wwgg/orrpsIDD:. 1023392.
[4] Shane Legg and Mareus Hutter. Universal intelliienne A definition of machine intelligence. CoRR, abs/0711..332) 2007. URL http://aaaxivv.o.a/b////b/a /372..339.
[5] Ming Li and Paul M.B. Vitnyi. An Introduction to Kolmogorov Complexity and Its Applications. Springer Publishing Company, Incorporated, edition, 2008. ISBN 0387339981.
[6] William of Ockham. Book i, chapter 12. In Philotheus Boehner, Gedeon Gál and Stephen F. Brown, editors, Summa Logicae. Franciscan Institute, 1974.
[7] Geoffrey E. Hinton and Ruslan R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313ee,,33333 313 507, 2006. doi: 10. 12.1cc26/cee 1276447.
[8] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations deep network with local denoising criterion. Journal of Machine Learning Research, 11(110)3331-3408, 2010. URL http://jmlr wwwooo///aaa. httg//apprs/vvll http///i1vvvinee///e// Oa. html.
[9] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through aligmment and uniformity on the hypersphere. In Proceedings of the 37th International Confer- ence Machine Learning. ICML'20. '0. JMLR.org. 2020.
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal et al. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[12] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár et al. Masked autoencoders are scalable vision learners. arXiv:2111.06377, 2021.
[13] Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https: //arxiv. org/ hrrg/bb/// /1322. 1312.6114.
[14] Patrick Esser, Robin Rombach and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
[15] Mareus Hutter. Algorithmic information theory: brief non-technical guide the field, 2007.
URL https: :/abp::///rxivv org/ csbs/////70302.

---

[16] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha et al. Matryoshka representation learning. In Advances in Neural Information Processing Systems, December 2022.
[17] Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar et al. FlexTok: Resampling images into Id token sequences of flexible length. arXiv 2025, 2025.
[18] Keita Miwa, Kento Sasaki, Hidehisa Arai, Tsubasa Takahashi and Yu Yamaguchi. One-d-piece: Image tokenizer meets quality-controllable compression, 2025. URL https///: org/ aba/2501.. 1115501.10666. 10064.
[19] Shivam Duggal, Phillip Isola, Antonio Torralba and William T. Freeman. Adaptive length image tokenization via recurrent allocation. In The Thirteenth International Conference on Learning Representations, 2025. URL https: htttps:/oppereeieee wwtperrewwee.neeeeooeeeeeeee net/forumm iididddma id=mb2ryuZ3wz.
[20] Volker Nannen. A short introduction to kolmogorov complexity, 2010. URL pestps://arxxr org/abat 111/00100..1 2400.
[21] Wilson Yan, Matei Zaharia, Volodymyr Mnih, Pieter Abbeel, Aleksandra Faust et al. Elastictok: Adaptive tokenization for image and video. arXiv preprint, 2024.
[22] Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards juutt map them to actions, 2020. URL https::/// / //txx/aavvvv///////////////aaav. orgg abs/ /112.. 1912.02875.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai al. image is worth 16x16 words: Transformers for image recognition at scale. CoRR, 1112000..1199,, 2020. URL https: pestps://arees //arxivv wwt.oogg//////o/ooooooooo ab1///a/s/00.// 11929.
[24] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals et al. Perceiver: General perception with iterative attention, 2021. URL 03206.
[25] Andrew Jacgle, Sebastian Borgeaud, Jean-Baptiss Alayrac, Carl Doersch, Catalin Ionescu et al. Perceiver 10: A general architecture for structured inputs & outputs. CoRR, abs/2107.14795. 2021. URL https: //trxvaa.ivvv///a////a orgg/brg//////eeee abs/2107. 14795.
[26] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers al. An image is worth 32 tokens for reconstruction and generation. arxiv: 2406. 0755007550, 2024.
[27] Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei and Jiajun Wu. Flow to the mode: Mode- seeking diffusion autoencoders for state-of-the ofte-oofthh-arr image tokenization, 2025. URL https: //arxiv. wwwg/ogg//////eoooo//ooooeeeeeee 2203.. 11056.
[28] Tiansheng Wen, Yifei Wang, Zequn Zeng. Zhong Peng. Yudi Su et al. Beyond matryoshka: Revisiting sparse coding for adaptive representation. 2025.
[29] Junhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra, Luke Zettlemoyer et al. Cat: Content-adaptive image tokenization. arXiv preprint XXXv:2501.03120, 2025.
[30] Yoshua Bengio, Nicholas Léonard and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.
[31] Nikolai Vereshchagin and Paul Vitanyi. Kolmogorov's structure functions and model selection,
2004. URL www.hhtts//aartte org/ prrg/abat
[32] Philippe Hansen-Estruch, David Yan, Ching-Yas Yao Chung, Orr Zohar, Jialiang Wang et al. Learnings from scaling visual tokenizers for reconstruction and generation, 2025.
[33] Elham Saraee, Mona Jalal and Margrit Betke. Savoias: A diverse, multi-category visual complexity dataset. arXiv preprint arxiv:18/0001771, 2018.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately summarize our contributions, namely the introduction of one-pass adaptive tokenizer and the conceptual connection between adaptive tokenization and algorithmic information theory (AIT). These are substantiated in the paper: Sections and 5 detail the adaptive tokenizer, while Section 4 explores the analogy with AIT.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations section in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not present any theoritical results, only draw high-level analogies with frameworks like AIT. This is clearly mentioned throughout the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Additional training details would be shared in the Supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper does not introduce any new data. The code will be shared post acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
wwm.iissssooccciooco for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //iiss... wwwpcppwpcc//aeeeecececceeeeetle ceabbicgguides.../e /ttgg/eeeeeeeeeeeeaiiiiiiea SCodeSubmissionl sewiiisssiiiiiiiiicooocioiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We will share these details in the Supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The high cost of training and evaluation large Imagenet dataset, could not compute such statistical measures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We will share these details in the Supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / http:/rrppp.ccpppccie//eeee/eeee Cettiic//eiieeeee ceGiiiieeenn
Answer: [Yes]
Justification: Authors reviewed NeurIPS Code of Ethics and verified the compliance. We do not foresce any potential harms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
We do not foresee immediate societal impacts. Similar to VAEs, which widely used in research and applications without direct societal concerns, our method operates at the representation level and does not pose ethical safety risks in its current form.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not anticipate any risks for misuse. We mainly introduce an alternative VAEs similar or no risk like VAEs.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All codes and datasets used are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Human-labe dataset was borrowed from a prior publication. No human study conducted by the authors.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
The only human-related dataset used is the SAVOIAS dataset, which was collected and released in a prior publication. is publicly available under the Apache License 2.0, and we did not interact with any human subjects in our study.
Justification:
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used as an important, original, or standtadd component of the core methods in this research
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / www.e///eeeee////////2c /LLM) for what should or should not be described.
19

---

A Appendix
A.1 Summarizing the Core Contributions
Single-pass adaptive tokenizer that allocates variable-length representations per image.
Why do we need an adaptive tokenizer that operates in single pass? Modern large-scale models increasingly use hierarchical or token-based representations that offer flexibility in deployment, latency, retrieval cost. Yet, in practice, adaptivity is often reduced to selecting fixed number of tokens across all data, based solely on downstream task requirements. This is inefficient: complex inputs may be underrepresented, and simple ones overrepresented.
We propose an adaptive tokenizer that predicts the number of tokens per image in a single forward pass, guided not only by task constraints but also by the image's intrinsic complexity. Our formulation enforces target reconstruction quality (e.g.. l loss below a threshold), ensuring compact yet sufficient representations. Future extensions may incorporate perceptual similarity metrics or downstream -ask feedback to further improve semantic adaptivity.
Losscconditioned training strategy inspired by KC and upside-down reinforcement learning.
Prior adaptive methods [16, 19] rely iterative inference-time procedures searching over nested embeddings or using recurrent models to identify the minimal token or memory budget required for task. These approaches introduce latency and computational overhead.
In contrast, we propose a loss-conditioned training strategy that enables the model to predict the appropriate number of tokens in a single forward pass. Drawing inspiration from Kolmogorov Complexity and the upside-down reinforcement learning paradigm [22], we treat reconstruction loss as task- speciffiin condition. If an image can be reconstructed with T tokens up to given loss threshold, then using a larger budget (T + AT) should not be necessary. Our training procedure enforces this principle by conditioning the model on the loss obtained with T tokens and optimizing to match that quality using T + AT tokens-while masking out the redundant ones. This eliminates the need for test-time search and allows efficient, adaptive inference.
Conceptual alignment of adaptive representations w/ Algorithmic Information Theory (AIT).
Our third contribution is high-level conceptual bridge between hierarchical adaptive tok- enization and principles from AIT. We draw an intuitive correspondence: training models to halt once sufficient representational capacity reached parallels the core idea of Kolmogorov Complexity. Moreover, the progressive allocation of tokens in hierarchical tokenizers echoes second-order AIT notions such as sophistication and logical depth. We stress that this is a conceptual perspective not formal theoretical result and offer it as a starting point for further exploration of the intersection between AIT and adaptive representation learning.
A.2 Experimental Details
Throughout the paper, we utilize Imagenet or a 100-class subset of Imagenet dataset to train and evaluate our models. The 100-class subset of Imagenet is utilized in multiple prior works, serving good dataset with decent scale (0.1M images) and lesser compute requirements. Human evaluation (Fig. 14) was done on Savioas Dataset [33] which contains human annotations of complexity on several known computer vision datasets.
Datasets:
Compute Requirements: Majority of the training was done single A100 H100 machine with eight 80GB gpus or on machines with equivalent GPU memory (distributed training on 4 H200 gpus).
A.3 Training Procedure
Our training framework follows two-phase formulation as described in the main paper: (1) Esti- mating Image Complexity, where the model attempts near-lossless reconstruction using a randomly- sampled token budget T, and (2) Learning to Tokenize at Estimated Complexity, where the model learns to ignore surplus tokens when provided a larger token budget T + AT, while maintaining the same reconstruction quality. These two phases jointly optimized with complementary objectives:
Lossless Compression Objective: Encourages the model to fully utilize the given token budget to minimize reconstruction and quantization losses.
20

---

Estimate Image Complexity
Learn to Tokenize Com plexiity
Input Image
[BxCxHxW]
Token Embeddlings [Bx(T=AT]HD]
Only Arttun Tokeenoo Jecade iipur
Token Embeddings [RXTxD]
Recons.
θ Enc
θ Dec
Recore.
0 Enc
θ Dec
Quality
Error emRy
Token Halting Proh.
Sampled
Randomly
Tokens [AT+aT]
[Bx(T=AT]xD]
Takens
Figure 9: KARL Training Graph: KARL follows upside-down RL paradigm (a) Estimate Image Complexity stage samples task-defining inputs [image, token budget 'T reconstruction enror 60) by attempting lossless compression (c =). (b) Learn to Tokenize stage trains the model- conditinnn on eo-to match the same quality using T+AT tokens while halting the extra (rightmost) AT. The shared encoder Onee and decoder Oce are updated in each iteration using reconstruction losses in both stages and halting loss in the second.
Minimal Program Search Objective: Encourages the model to identify and suppress unnecessary tokens via halting supervision, enabling it to approximate the shortest sufficient representation.
Together, these objectives guide the model produce variable-length token representations that reflect the intrinsic complexity of each image, enabling one-shot adaptive tokenization without iterative search inference. See Fig. 9, Fig. 3 for the training computational graph and the training algorithm.
Coss-Conditioned Training Details: Our training procedure reflects the upside-down reinforce- ment learning (RL) paradigm: reconstruction loss conditions act as task-specifying rewards that guide the learning process. We initialize a list of discrete loss targets- including low values (0.0, 0.01, 0.02), mid-range values (0.03-0.11) and few larger ones (0.14-0.4) each mapped to an embedding vector of the same dimensionality as the encoder's input tokens. These loss embeddings are concatenated as an additional token to the latent-distillation encoder input.
In each training iteration, a token count is randomly sampled from (2,32.... ..... 256. In the first run, the model attempts near-lossless compression using a small loss condition and token budget T, producing an actual reconstruction error Eg. We then map Eo to the smallest greater discrete loss from the list and use the corresponding embedding to condition the second run, which uses a larger token budget (I' AT, up 256).
The token halting probabilities are supervised using a binary cross-entropy loss: tokens from the first run are labeled "keep" (0) and new tokens are labeled "halt" (1). If no tokens are added (i.e., T= 256, AT = 0), we augment training by randomly sampling loss values that are smaller than the one attained during the lossless run. This ensures the model still receives diverse training pairs of (target loss condition, input token budget), even when no masking occurs in the second stage. This formulation allows the model learn how to tokenize image given task-specific loss condition, effectively converting a token allocation problem into supervised learning task analoooou to how upside-down RL uses rewards as inputs learn policies.
Tab. 3 evaluates the learned tokenizer's ability to meet the input reconstruction loss condi- tion nonn-spcifficall,, how often the reconstructed image satisfies the input loss condition when portion of the input tokens is masked. When no tokens are masked, some images may still exceed the loss threshold due to insufficient max token budget (=256). As shown in the table, only very small fraction of the 5000 IN100 validation images exhibit a reconstruction loss greater than 0.03 beyond the input threshold when masking is applied. For images that utilize only a subset of the 256 input tokens yet exceed the input loss condition (c), the average reconstruction error is only marginally above the threshold. These results validate the efficacy of our training approach.
Details regarding multi-stage training pipeline: Inspired by recent works [26, 19], our overall training is conducted in two distinct stages over the course of training epochs, separate from the two-phase EIC-LTC procedure executed within each training iteration. Specifically, this multi-stage pipeline consists of latent-distillation pretraining followed by GAN-based finetuning stage.
In the latent distillation stage, a pre-trained image tokenizer (VQGAN or VAE) is used to map input images into 2D token grids. The KARL encoder and decoder modules are trained to compress these 2D tokens into ID representations and reconstruct them back, using reconstruction loss over the 2D token space the primary learning signal (thus performing 2D latent tokens - ID distillation..
21

---

Threshold (e)
% of images with recon. loss different from input condition (c)
Avg Err (> only)
>e
6+0.01
<+0.02
> 6+0.03
6+0.04
e+00.05
0.01
0%
0%
0%
0%
0%
0%
0.000
0.03
1%
0%
0%
0%
0%
0%
0,050
0.05
10%
3%
1%
1%
0%
0%
0,060
0.07
25%
12%
6%
4%
2%
1%
0,086
0.09
18%
9%
5%
3%
2%
1%
0.107
0.11
36%
19%
10%
5%
3%
2%
0.126
Table 3: Analysis of the loss-conditioned training strategy for single-pass adaptive tokenization:
Only a small fraction of images exceed the input target reconstruction loss threshold (c) when masking is applied. The columns indicate the fraction of images exceeding c by more than specified margin values. The last column shows the average reconstruction error for these images, which remains only marginally higher than the input threshold. Default e = 0.05 works well at inference.
Both encoder and decoder operate with full self-attention: the encoder attends over the concatenated sequence of image tokens and initialized ID tokens, while the decoder attends over learned ID tokens concatenated with masked 2D tokens (in an inpainting-style setup). For VQGAN tokenizers, we use cross-entropy loss over predicted logits and ground-trutt codebook indices. For VAE tokenizers, we use mean squared error (MSE) loss over the predicted and target token embeddings.
In the second stage, we finetune the model using pixel-level losses: reconstruction loss shifts from the intermediate 2D token space to the final image space. This includes pixel-wise & loss, an adversarial GAN loss, LPIPS perceptual loss, and standard quantization losses (commitment and codebook loss). Following prior work, we apply quantization on a factorized 12-dimensional embedding derived from the encoder output. Finally, given recent success of diffusion srrfusioo-bbsedd autoencoders instead of GANs [27, 17], extending KARL w/ diffusion based objectives will be promising as future work.
A.4 Scaling Laws & Ablations
Most adaptive tokenization works [19, 17] present scaling laws by varying the number of tokens per image, but they typically assign the same token count to all images for a given configuration. As noted in Core Contribution #1, the ideal goal of an adaptive tokenizer is to allocate variable number of tokens per image test time, depending on the content or complexity of each image. To capture both perspectives, we design scaling laws in two ways: (a) uniform token allocation across images, as done in prior work, and (b) variable token allocation based on target reconstruction loss. Additionally, inspired by the Algorithmic Information Theory (AIT) notion of Kolmogorov Complexity, we analyze scaling laws from a KC perspective-studynn how well we can minimize the token count (i.e., program length) per image while preserving recons. quality, target E loss.
Before delving into the scaling laws, we clarify an important detail: ideally, the Kolmogorov Complexity (KC) of a data point includes the full length of the program required to generate it. In our context, this would encompass not only the number of encoded tokens but also the token feature dimensionality, decoder size, and codebook size. However, since these latter components are amortized across many images, we approximate KC using only the token count and refer to this as KC throughout the paper. Note that the encoder is not part of the generative program- it acts as program synthesizer rather than the program itself.
Continuous vs Discrete Tokenization:
We compare continuous and discrete adaptive tokenization approaches at both ID and 2D token levels under both fixed and variable token allocation regimes.
Key observations from Fig. 10 include (new or significant findings highlighted in green)
Continuous ID tokens consistently outperform quantized ID tokens in terms of reconstruction quality (measured by FID) across nearly all token counts counts-echoing echoing observations of prior works.
This performance advantage holds even when token counts are allocated adaptively per image based on a target reconstruction loss.
Continuous ID tokenizers achieve better reconstruction at lower average token counts (i.e.. lower estimated KC), highlighting their efficiency in capturing image complexity.
The choice of 2D tokenizer (continuous VAE vs. discrete VQGAN has minimal impact on the minimum token count required to reach target loss-when the ID tokens are quantized.
22

---

FID @ same token allocation per-image
FID @ variable token allocation per-image
Active Token Utilization @ variable token allocation
Fréchet inceptioo Dist. (FID)
35
Quant Quant
Fréchet Inception Dist. (FID)
25
250
-0- Quart 20 J 10
Con: 20 Cont ID
Con: 20 I Quant 10
Quant 20| Cant
3D
(ontt
Active Token Count (KC)
200
-0- cart 20 Quarr 10
Zone 20 Coen 1D
Quant 20 Conr 10
25
150
15
15
12
100
10
5
32
64
96
128
160
192
224
256
011
0.09
0.07
0
0.03
0.01
011
0.09
0.07
0.b5
0.03 001
# Tokens
Reconstruction Thresholds
Reconstruction Thresholds
Figure 10: Scaling laws for continuous vs discrete tokenization: ContID is most crucial for FID
(Plot & 2). Cont2D (VAE) with ContID leads to shortest representation (KC) on average (Plot 3).
FID @ same token a allocation per-image
FID @ variable token allocation per-image
Active Token Utilization @ variable token allocation
Fréchet Inception Dist. Dist. (FID)
35
Codkebook Size- 4096
1524
Fréchet Inception Dist. (FID)
24-
22-
Codeseeck Sizt 1924
Codebsok Soe 609c
Codetook Size 0192
Active Token Count (KC)
Cookdrask 5:2 6096
Codebsse Sie 1024
Codebaak Saz 6192
20
25
18
16-
20
14-
12
50-
10
160
192
224
0.11
0.59
0.51
0.07
0.55
0.11
# Tokens
Reconstruction Thresholds
Reconstruction Thresholds
Figure 11: Scaling laws for 1D token codebook size: Larger codebooks (4096 and 8192 codes) lead to shorter representation length (KC) average because of code specialization.
However, when ID tokens remain continuous, pairing them with discrete 2D tokenizer (e.g.. VQGAN) leads to higher KC across all target loss levels, potentially because more tokens are required to compensate for the loss of detail at 2D token level.
1D Token Codebook Size Ablations: From Fig. 11, we observe that increasing the codebook size consistently improves performance, as reflected by lower (i.e., better) FID scores. Moreover, larger codebooks (4096 and 8192) not only achieve better image quality, but do so using fewer tokens on average- average-resulting in lower estimated Kolmogorov Complexity (KC) per image. This is intuitive smaller codebooks offer fewer specialized codes, requiring more tokens to achieve the same representational fidelity.
Encoder and Decoder Depth Width Ablations: As highlighted in the main paper and shown in Fig. 12 and Fig. 13, the scaling laws indicate that the best- perfoorrnn architecture features small encoder and a large decoder. This is intuitive: the encoder's role is to distill the already abstracted 2D tokens from the VQGAN into 1D tokens, while the decoder takes on the more challenging task of inpainting 1D tokens into a masked slate.
A key observation is that the variable token allocation strategy enhances the separation between architectural variants. Specifically, it reveals a more pronounced performance gap between the Small Encoder, Large Decoder and Large Encoder, Large Decoder configurations. When the same number of tokens is assigned to all images- regardless of image complexity - high-complexity images may perform poorly under limited token budgets, making it harder to distinguish between architectural strengths. In contrast, when token counts are adaptively allocated per image, each model can adjust token usage to meet reconstruction targets, enabling fairer comparison and clearer differentiation.
The average number of active tokens remains nearly constant across ablations. The Small Encoder, Large Decoder setup (small in both width and depth) performs best under this similar active token
23

---

FID @ same token allocation per-image
FID @ variable token
Active Token Utilization @ variable token allocation
allocation per-image
Fréchet Inception Dist. (FID)
30
25
-0-
-6-
Ern Wathh Dec Whth -5
Enc Withh 5 Dec whtth
End Witth De: Width -
End wwith Dec Witth =
Fréchet Inception Dist. (FID)
27-
25-
Wooth
(Pec Withh Dac Witth
Active Token Count (KC
- Fne with 1. |er Witt
- tni Witt 5 Doc With -
-h- Inc With 5 Des With i
--- withh
With
17-
100
15-
15
12-
10
10-
32
64
96
128
160
192
224
255
0.31
0.09
0.07
0.05
0.03
0.01
0.31
0.09
0.07
0.05
0.03
001
# Tokens
Reconstruction Thresholds
Reconstruction Thresholds
Figure 12: Scaling laws for encoder/decoder width (feature dimensionality): A Small Encoder with Large Decoder consistently achieves the best performance. Notably, variable token allocation per image enables more informative comparison, clearly distinguishing the Small Encoder, Large Decoder configuration from the Large Encoder, Large Decoder variant.
FID @ same token allocation per-image
FID @ variable token allocation per-image
Active Token Utilization variable token allocation
Fréchet Inception Dist. (FID)
End Dupth 5 Der Depth 5
Depth
Dec Depth
Fréchet Inception Dist. (FID)
22
Enc Depit Depth-
Erc Deptt Das Depth
Active Token Count (KC)
250
Enc Depth S Deg Deptn
Deptn -. Dec Depm
Erc Deptt Depch 5
Enc Expti L Depth
25
Depth 1
20-1
Depm
Exc Depth - Ujec Dupth
18-
20
16
150
100
12
50
30-
32
64
96
128
160
192
224
256
0.11 0.09
0.07
0.05
0.03
0.01
0.09 0.11
0.07
0.05
0.03 0.01
# Tokens
Reconstruction Thresholds
Reconstruction Thresholds
Figure 13: Scaling laws for encoder/decoder depth (number of layers): Similar to the width ablation, the Small Encoder with Large Decoder configuration yields the best performance at the same average active token count as other variants. This distinction becomes more evident when using variable token allocation per image, as opposed to fixed token counts.
For the width ablations, both the depth (i.e., number of transformer layers) and the number of heads (in melti-head attention) are held constant. Conversely, in the depth ablations, the width is fixed to a large value, and the number of heads scales proportionally with the depth.
utilization. Note:
A.S Alignment with human measures of complexity
Fig. 14 showcase strong alignment between human labels of complexity and our predicted estimate- min. token count per image or approx KC. We computed KC for images from the Savoias dataset [33] using 6 = 0.05 as input reconstruction condition - KC.=o.os (x. = 256), followed by bucketing them into intervals of 32. Around 47% of the images with complexity labeled 15 (/ 100) by humans are assigned 32 to 64 tokens, while images with human-assigned complexity measures of 60+ get KC Rg 256.
A.6 Additional Comparison with Prior Adaptive Tokenizers
Beyond the comparisons presented in the main paper, further evaluate KARL against ALIT, a recurrent adaptive tokenizer, and One-D- a Matryoshka- basee tokenizer, under both fixed per-image token allocation and variable token allocation regimes.
Fig. 15 and Fig. 16 clearly demonstrate the superior performance of ALIT and KARL over One- D-Piece, which frequently produces blurry reconstructions (see the second column in both figures). Such blurry outputs misleadingly yield good FID scores (see Tab. 6) while performing poorly on per-image metrics such as LPIPS, SSIM, PSNR, and DreamSIM. This observation underscores limitation of FID: it may not be an ideal metric when the focus is on accurate per-immn per-image reconstruc- tion. Furthermore, from Kolmogorov Complexity (KC) perspective, the goal of tokenization is to
24

---

Input Image
6-0.05
c-0.07
E-0.09
Cell (ij) of images
Human Score (i) & KC
100
256
40
# Latent Takens
192-
50
96
64
10
32
15
25
35
45
55
65
75
85
Human Complexity Score
Figure 14: Alignment of learned image complexity with human estimates Savoias [33]: The lop row is marked as less complex (8) by human compared to bottom row (89). Active token counts or KC (Row = 64,32,32 Row 2 = 256,256,1922 192). Heatmap shows large % of high complex images require large token counts, while large % of low complex images utilize token counts from 32 to 64.
produce the shortest representation that faithfully captures the individual data point-not merely the distribution. In this regard, KARL achieves the best performance most per-image reconstruction metrics, slightly outperforming ALIT. Unlike ALIT, which requires multiple recurrent encoder passes, KARL determines the minimal token count in a single forward pass.
Fig. 17 and Fig. 18 compare different adaptive tokenizers under a variable token allocation regime, where each image is assigned different number of tokens based on target reconstruction loss threshold. For One-D- D-Piece, this involves a single encoder pass but multiple decoder runs to find the shortest embedding that achieves reconstruction & loss below the target. In contrast, ALIT performs a joint encoder-decoder pass at each iteration, requiring multiple iterations until the stopping criterion satisfied. See Tab. 2 in the main paper for additional details.
Thanks to its recurrent training and explicit test-time search, ALIT performs best under a relaxed reconstruction criterion of & loss < 0.09 (as shown in Fig. 17). However, KARL performs competi- tively-often surpassing ALIT on per-image metrics such as LPIPS, SSIM, and & loss.
Fig. 18 analyses reconstructions under tighter constraint of & loss < 0.05. KARL not only achieves better or comparable performance on these metries but also does so with equal or fewer tokens, particularly when compared to D-ecce, Based on our experiments, we conjecture that
KARL with an input condition of e 0.05 represents the most practical configuration for adaptive tokenization in real-world settings.
A.7 Evaluating Downstream Tasks
We perform range of downstream evaluation for CLIP text-image alignment (Tab. 8 and Tab. 9), depth estimation (Tab. 10 and Tab. 11), semantic segmentation (Tab. 12 and Tab. 13) and VLM (Tab. 14) tasks, by levering the reconstructions from different adaptive tokenizers. We evaluate both "fixed same-token for each dataset image" setting (Tab. 8, Tab. 10, Tab. 12, Tab. 14) and "variable token" setting based on desired reconstruction loss threshold e (Tab. 9, Tab. 11, Tab. 13). In these tables, the 256-token, = 0.0 column corresponds to the setting where the input token budget is 256 and meaning all tokens are used for lossless reconstruction.
Our findings reveal that ALIT is the strongest adaptive tokenizer across most metrics, using a recurrent transformer that incrementally adds tokens-each step incurring extra FLOPs. KARL generally consistently ranks second, followed by the matryoshka-stylle One-D-Piece. Notably, all three perform comparably, and KARL matches this performance in single pass, highlighting its promise as a efficient, scalable alternative. A compelling future direction is to combine KARL with ALIT-style recurrent refinement-rrefinng minimal representation with more iterative loops.
As shown across CLIP, Depth, and Semantic Segmentation metrics, the performance drop from = 0.0 to 0.03 and 0.05 is hanimal--highighhe reconstruction loss has little to no impact on downstream performance. As expected, the effect of increasing is especially marginal for coarse tasks like CLIP alignment.
the benefit of KARL: allowing slight
Note: Compared to KARL and ALIT, One-D-Piece requires 30+ more tokens to achieve -bounded reconstructions with similar downstream performance (see Tab. 2 in the main paper).
25

---

Overall, the metrics consistently highlight the significance of KARL- minimal or no drop in performance while operating in single pass.
A.8 Linear Probing on learned token representation
We conducted linear probing experiments by training single-layer linear classifier on top of the latent tokens produced by KARL and ALIT. Using only the first 32 ID latent tokens, AL/T achieves 44.8% Top-1 classification accuracy
ImageNet-Ik while KARL attains 41.5%. When both the 2D image tokens and the first 32 ID tokens are used, performance improves 51.6% for ALIT and 45.0% for surLsssrgssnnn with the non-adaptive Titok tokenizer (48.0%).
As noted in the ALIT paper, lmear-probing accuracy improves with more recurrent iterations during caining-potentially due to added training FLOPs and hierarchical gradient flow across RNN layers. This may partly explain ALIT's edge over KARL. Finally, investigating how compression minimality impacts the linear separability of learned tokens is a promising direction for future work.
A.9 Further Details on Computational Requirements
Tab. 2 compares computational complexity in terms of the number of encoder/decoder calls, which architecturo-agnosie and avoids confounds like GPU type or precision. For fair comparison, all baselines the same encoder/decoder architecture. Below are GFlops and wall-clock runtimes observed on single H100 GPU with FP32 precision.
GFlops:
Encoder: KARL = One-D--iece z First iteration of ALIT ~ 80 GFlops.
Decoder: ALIT and One-D-Piece require multiple decoding passes, each GFlops.
ALIT: Requires iterations during training, and 4 iterations at test time for -buunde reconstruction. Each additional ALIT iteration adds ~ 30 GFlops
Wall-clock Time (ms): First encoder decoder pass: ~7 Each additional ALIT encoder iteration: <44 ms.
Figures and Tables the following pages.
26

---

L1
SSIM
LPIPS L
Method
64
128
256
128
32
128
Flextok--ae-quant-11
1.75
1.51
1.20
0.98
0.23
0.27
0.33
0,40
0.53
0.48
0.42
0.36
KARL-vae-quant-I
1.14
0.98
0.84
0.70
0.33
0.38
0.43
0.50
0.40
0.35
0.29
0.23
Table 4: Comparison with Flextok: KARL-S outperforms Flextok-S on L1, SSIM & LPIPS metrics.
L1 x 10
LPIPS +
SSIM
Eval Set
Approach
32
64
256
32
64
256
32
64
256
One-D-Piece
1.37
1.18
0.95
0.48
0.41
0.33
0.33
0.36
0.41
COCO
ALIT
1.16
0.98
0.74
0.42
0.36
0.27
0.34
0.38
0.46
KARL
1.11
0.96
0.64
0.41
0.35
0.23
0.36
0.40
0.53
One-D-Piece
1.13
0.96
0.75
0.31
0.24
0.16
0.45
0.47
0.52
WIT
ALIT
0.92
0.77
0.57
0.37
0.31
0.24
0.46
0.51
0.58
KARL
0.86
0.73
0.49
0.25
0.20
0.12
0.49
0.53
0.65
Table Reconstruction metrics on OOD datasets (Top) Trained on INIK (object-level dataset), evaluated on COCO (scene-level dataset). (Bottom) Trained on IN1K, evaluated on Wikipedia Image Dataset (more OOD). KARI. demonstrates impressive generalization on OOD reconstructions.
FID
variable token regime ({)
FID
same token regime (L)
Approach
0.03
0.05
0.09
32
128
256
One-D-FF Deecce
8.2
8.3
10.5
24.1
9.7
8.2
ALIT
8.4
9.3
13.5
22.6
11.7
8.2
KARL
9.1
10.3
16.2
28.7
13.6
8.8
Table 6: FID Comparison on IN100: While KARL slightly trails prior approaches in FID, outperforms them on single-image metries that better align with the Kolmogorov Complexity (KC) gerspective-faoriing the shortest possible representation for each individual data point rather than alignment with dataset-level statistics. As is well known, FID is not reliable metric for assessing reconstruction quality. Visual results in Fig. 4, Fig. 15, Fig. 16, Fig. 17, and Fig. 18 clearly demonstrate that both ALIT and KARL produce reconstructions that visually superior to One-D-Piece. Notably, KARL is the only single-pass method for adaptive tokenization and achieves the best performance single-image quality metrics. FID could likely be improved through additional training techniques-for instance, using diffusion decoders or separate (well-tuned) GAN discriminators for different representation lengths/token counts, instead of a shared one. Conditioning masking on perceptual losses like LPIPS, DreamSim (instead of & might enhance FID.
FID
J
Method
32
64
128
256
Flextok--ac-quunt-ID KARL-vaa--uant-I1
25.28
21.95
19.58
17.26
33.70
22.36
14.95
9.74
Table 7: Comparison with Flextok on FID metric.
27

---

Approach
32 tokens
64 tokens
256 tokens
One-D-Piece
82.1
86.7
91.5
ALIT
84.6
88.5
92.8
KARL
82.3
91.2
Table 8: Downstream Evaluation - CLIP text-image (fixed token per-image).
Approach
256,
0.00
256,
0.03
256,
0.05
256,
0.09
One-D-Piece
91.5
91.5
89.8
89.6
ALIT
92.8
92.7
92.2
89.6
KARL
91.2
91.7
90.7
87.1
Table 9: Downstream Evaluation - CLIP text-image (input token budget: 256 tokens; subset of tokens selected such that reconstruction loss less than recon. threshold e). Higher is better.
Approach
32 tokens
64 tokens
256, c 0.0
One-D-Piece
2.73
2.11
1.53
ALIT
2.04
1.62
1.17
KARL
2.19
1.74
1.16
Table 10: Downstream Evaluation Depth Estimation (fixed token per-image).
Approach
256, 0.0
256,
0.03
256, c 0.05
256,
0.09
One-D-Piece
1.53
1.53
1.35
1.77
ALIT
1.17
1.18
1.24
1.51
KARL
1.16
1.18
1.32
1.76
Table 11: Downstream Evaluation - Depth Estimation (input token budget: 256 tokens; subset of tokens selected such that recon. loss less than reconstruction threshold e. Lower is better.)
Approach
F-measure @ F-measure @ 64
F-measure @ 256
One-D-Piece
66.46
73.36
79.79
ALIT
70.58
75.96
81.55
KARL
68.09
73.18
80.64
Table 12: Downstream Evaluation Semantic Segmentation Estimation (fixed token per-image).
Approach One-D-Piece
F-measure (e 0.03)
F-measure (e 0.05)
F-measure (e 0.09)
79.77
79.54
77.17
ALIT
80.90
79.95
76.27
KARL
80.49
78.88
73.41
Table 13: Downstream Evaluation Semantic Segmentation (input token budget: 256 tokens: subset of tokens selected such that recon. loss less than reconstruction threshold Higher is better.)
Approach
VLM Score (32 tokens)
VLM Score (64 tokens)
VLM Score (256 tokens)
One-D-Piece
62.4
71.4
76.7
ALIT
64.1
71.6
77.0
KARL
63.8
68.4
77.6
Table 14: VLM (VQA) downstream task. Higher is better.
28

---

Input Image Ohe-D-Piece-33
ALIT-32
KARL-32
SleeDD-Piece2255 ALIT-256
KARL-256
Figure 15: Reconstruction Comparison on IN100 (visualization 1/2): KARL and ALIT produce higher-quality reconstructions than One-D-Piece with KARL achieving strong single-image metrics. One-D-Piece consistently generates blurry images at low token counts satisfying FID metric more.
29

---

Input Image
One-D-Piece-32
ALIT-32
KARL-32
One-D-Piece-255
ALIT-256
KARL-256
Figure 16: Reconstruction Comparison on IN100 (visualization 2/2): KARL and ALIT produce higher-quality reconstructions than One-D-Piece, with KARL achieving strong single-image metrics. One-D-Piece consistently generates blurry images at low token counts satisfying FID metric more.
30

---

Input Image
One-D-Piece
ALIT
KARL
Images
One-ID-Piece
ALIT
KARL
KC (L)
L1x10 (1)
SSIM (t)
LPIPS (1)
11x10
SSIM
LPIPS
KC
L1x 10
SSIM
LPIPS
Row
44
0.88
0.35
0.42
64
0.68
0.41
0.30
64
0.82
0.36
0.36
Row
0.87
0.42
0.27
0.59
0.65
0.26
Row
256
1.07
0.27
0.40
192
0.89
0.31
0.36
224
0.83
0.37
0,30
Row
75
0.90
0.59
0.32
32
0.85
0.58
0.35
64
1.00
0.59
0.37
Figure 17: Comparison of One-D-Piece, ALIT, and KARL with variable token allocation per image satisfying reconstruction & 10 loss < 0.9: At this reconstruction loss threshold, the goal is not perfect reconstruction. but rather to evaluate which method achieves acceptable reconstruction quality within the target constraint while using the fewest tokens (i.e., minimizing KC). ALIT and KARL use fewer tokens than One-D-Piece for acceptable recons., satisfying x 10 loss 0.9.
31

---

Input Image
One-D-Piece
ALIT
KARL
Image
One-D-Piece
ALIT
KARI
KCC (L)
(L)
SSIM (
LPIPS (1)
KC
L1x10
SSIM
LPIPS
KC
11x
SSIM
LPIPS
Row
256
0.81
0.41
0.30
256
0.56
0.47
0.26
256
0.51
0.51
0.22
Row
256
0.69
64
0.47
Row
256
0.66
0.50
0.31
256
0.53
0.53
0.25
224
0.49
0.58
0.22
Row 4
256
0.64
0.64
0.26
192
0.49
0.67
0.22
224
0.48
0.71
0.19
Figure 18: Comparison of One-D-Piece, ALIT, and KARL with variable token allocation per image satisfying reconstruction & 10 loss 0.5: At this reconstruction loss threshold, the goal becomes much closer to near perfect reconstruction while using the fewest tokens (KC). ALIT and KARL often save few tokens from the full 256 token budget, satisfying & x 10 loss < 0.5. Threshold= 0.05 can serve de-facto threshold for majority of vision tasks.
32


Reviews:
[{"review_id": "QOCwSKGx5j", "reviewer": "Reviewer_2QbU", "summary": "This paper introduces a novel one-shot adaptive tokenizer named KARL, designed for vision representation learning. Inspired by principles from Algorithmic Information Theory (AIT), particularly Kolmogorov Complexity, the method aims to allocate variable-length token representations that reflect the intrinsic complexity of each image. Unlike standard approaches that assign fixed-length representations or adaptive methods that rely on test-time search, KARL predicts the appropriate number of tokens for an image in a single forward pass, stopping when an approximation of its Kolmogorov Complexity is reached. The authors demonstrate that KARL achieves competitive performance with recent adaptive tokenization methods while offering greater efficiency. In addition to empirical evaluations, the paper presents a conceptual study linking adaptive tokenization to AIT concepts such as sophistication and logical depth, and shows that KARL’s predictions correlate well with human judgments of image complexity and interestingness.", "questions": "1. **Justify the Use of Image Reconstruction as the Sole Evaluation Task**\nGiven that the goal is to learn meaningful, adaptive representations, why is image reconstruction chosen as the primary (and only) evaluation task? In recent literature, reconstruction is often seen as a proxy task and not a reliable indicator of representation quality for downstream applications. Please clarify why this choice was made, and to what extent you believe it reflects the utility of the learned representations. Adding even a few representative downstream tasks (e.g., classification, segmentation, or vision-language tasks) would significantly strengthen the empirical validation.\n\n2. **Evaluate Generalization to Downstream Tasks**\nTo demonstrate the broader utility of KARL, it would be helpful to test the learned representations on common downstream vision or multimodal tasks. For example, how well do KARL’s token representations perform when used as input to image classification, retrieval, or visual question answering models? If time constraints prevent such additions in this version, please provide a detailed plan or preliminary results, and discuss potential challenges in extending KARL to these settings.\n\n3. **Broaden the Comparison with Other Adaptive Methods**\nCurrently, the comparison with existing adaptive tokenization methods is limited. Could the authors include more recent or diverse baselines, especially those that have been evaluated on vision tasks? A more comprehensive empirical comparison would help position KARL more clearly in the current landscape and clarify its advantages in terms of both performance and efficiency.", "limitations": "The authors have not adequately addressed the limitations or potential negative societal impacts of their work. While the method is theoretically motivated and technically sound, the paper does not include a discussion of its limitations—such as the fact that it is only validated on image reconstruction tasks, which may not generalize to practical vision or multimodal applications.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* The paper is clearly written and easy to follow, with well-structured explanations and motivation.\n\n* The approach is grounded in a solid theoretical perspective, drawing from Algorithmic Information Theory (AIT) to inform the design of a dynamic tokenization strategy for vision models.\n\n* The idea of linking token count to Kolmogorov Complexity is novel and conceptually insightful.\n\n## Weaknesses\n\n* The method is only evaluated on image reconstruction tasks, which are not widely considered strong indicators of representation quality in current vision or multimodal learning research.\n\n* To better validate the effectiveness of the proposed approach, it would be important to test it on downstream tasks, such as image classification, object detection, or vision-language tasks (e.g., VQA).\n\n* Without such downstream evaluations, the current experiments offer limited evidence for the practical utility or generalization ability of the method.", "was_revised": true, "final_justification": "After the authors' rebuttal, my most concerns are solved. And I decide to raise my rating to 4.", "rebuttal": "### **Evaluating Generalization to Downstream Tasks**\n\nWe thank the reviewer for the great suggestion regarding downstream evaluation. We have now conducted a comprehensive set of evaluations. Furthermore, we evaluated both fixed token allocation per image and variable token allocation settings. We are excited to share our findings—\n\n* ALIT is the strongest adaptive tokenizer across most metrics, using a recurrent transformer that incrementally adds tokens—each step incurring extra FLOPs. KARL generally consistently ranks second, followed by the matryoshka-style One-D-Piece. **Notably, all three perform comparably, and KARL matches this performance in a single pass**, highlighting its promise as a effecient, scalable alternative. A compelling future direction is to combine KARL with ALIT-style recurrent refinement—refining minimal representation with more iterative loops.\n\n* In the tables below, the 256-token column corresponds to the setting where the input token budget is 256 and \\\\( \\epsilon = 0.0 \\\\), meaning all tokens are used for lossless reconstruction. As shown across CLIP, Depth, and Semantic Segmentation metrics, the performance drop from \\\\( \\epsilon = 0.0 \\\\) to \\\\( \\epsilon = 0.03 \\\\) and \\\\( \\epsilon = 0.05 \\\\) is minimal—highlighting the benefit of KARL: allowing slight reconstruction loss has little to no impact on downstream performance. As expected, the effect of increasing \\\\( \\epsilon \\\\) is especially marginal for coarse tasks like CLIP alignment.\n\n* Note: Compared to KARL and ALIT, One-D-Piece requires 30+ more tokens to achieve \\\\( \\epsilon = 0.09 \\\\)-bounded reconstructions with similar downstream performance (see Table 2 in the main paper).\n\nOverall, the metrics consistently highlight the significance of KARL—achieving minimal or no drop in performance while operating in a single pass. We hope this addresses the reviewer’s concerns regarding downstream task evaluations, and we would be happy to explore additional tasks upon suggestion.\n\n\n**Evaluating on CLIP Text-Image Downstream Task**\n\n| Approach      | 32 tokens | 64 tokens | 256 tokens | ε = 0.03 | ε = 0.05 | ε = 0.09 |\n|---------------|-------------------------------|-------------------------------|---------------------------------|-----------------------|-----------------------|-----------------------|\n| One-D-Piece   | 82.1                          | 86.7                          | 91.5                            | 91.5                | 89.8                | 89.6                |\n| ALIT          | **84.6**                      | **88.5**                      | **92.8**                        | **92.7**              | **92.2**              | **89.6**              |\n| KARL          | 82.3                          | 86.7                          | 91.2                            | 91.7                 | 90.7                 | 87.1                 |\n\n\n**Evaluating on Depth Estimation Downstream Task**\n\n| Approach      | 32 tokens | 64 tokens | 256 tokens, ε = 0.0 | 256 tokens, ε = 0.03 | 256 tokens, ε = 0.05 | 256 tokens, ε = 0.09 |\n|---------------|---------------------------|----------------------------|-----------------------------|---------------------------|---------------------------|---------------------------|\n| One-D-Piece   | 2.73                      | 2.11                       | 1.53                        | 1.53                      | 1.55                      | 1.77                      |\n| ALIT          | **2.04**                      | **1.62**                   | 1.17                        | 1.18                      | **1.24**                    | **1.51**                  |\n| KARL          | 2.19                  | 1.74                       | **1.16**                    | **1.18**                  | 1.32                  | 1.76                      |\n\n\n**Evaluating on Semantic Segmenatation**\n\n\n| Approach      | F-measure @ 32 | F-measure @ 64 | F-measure @ 256 | F-measure (ε = 0.03) | F-measure (ε = 0.05) | F-measure (ε = 0.09) |\n|---------------|----------------|----------------|------------------|------------------------|------------------------|------------------------|\n| One-D-Piece   | 66.46          | 73.36          | 79.79            | 79.77                 | 79.54                 | **77.17**             |\n| ALIT          | **70.58**      | **75.96**      | **81.55**        | **80.90**             | **79.95**             | 76.27                 |\n| KARL          | 68.09          | 73.18          | 80.64            | 80.49                 | 78.88                 | 73.41                 |\n\n\n**Evaluating on VLM (VQA)**\n\n| Approach      | VLM Score (32 tokens) | VLM Score (64 tokens) | VLM Score (256 tokens) |\n|---------------|-----------------------|------------------------|-------------------------|\n| One-D-Piece   | 62.4                  | 71.4                   | 76.7                    |\n| ALIT          | **64.1**              | **71.6**               | 77.0                    |\n| KARL          | 63.8                  | 68.4                   | **77.6**                |\n\n---\n\n**Linear Probing Experiments:**\n\nWe also conducted linear probing by training a single-layer linear classifier on top of the latent tokens produced by KARL and ALIT. Using only the first 32 1D latent tokens, ALIT achieves 44.8% Top-1 classification accuracy on ImageNet-1K, while KARL attains 41.5%. When both the 2D image tokens and the first 32 1D tokens are used, performance improves to 51.6% for ALIT and 45.0% for KARL—surpassing with the non-adaptive Titok tokenizer (48.0%).\n\nThis highlights a few key observations:\n- As also noted in the ALIT paper, linear probing accuracy improves with more recurrent iterations during training—potentially due to added training FLOPs and hierarchical gradient flow across RNN layers. This may partly explain ALIT’s edge over KARL.\n- Investigating how compression or minimality impacts the linear separability of learned representations is a promising direction for future work.\n- The One-D-Piece paper reports noticeably lower linear probing accuracy (mid-30s). We suspect this may be due to an incorrect probing setup and will re-run the evaluation in our framework to report corrected numbers in the final draft.\n\n\n---\n\n**Justification for Emphasis on Reconstruction Metrics**\n\nOne minor comment, we believe reconstruction metrics still remain a valuable indicator of performance. A compact representation that enables near-lossless or low-\\\\( \\epsilon \\\\)-bounded reconstruction intuitevly will retain the key semantics necessary for most downstream tasks—although *requiring different decoding capacities (exciting direction for future work)*. Moreover, single-image reconstruction metrics also echoes the principles of KC and AIT in general: identifying the shortest program that can reproduce the input data. Finally, we believe thoroughly exploring the connection between compressed representations and downstream performance is a compelling standalone research direction, and is one we are actively interested in.\n\n\n---\n\n**Comparison with More Baselines**\n\nWe compared against two recent strong baselines—ALIT (ICLR 2025) and One-D-Piece (2025). Other notable works include Flextok, ElasticTok, (non-adaptive) FlowMo, Titok, and older methods like VAE and VQGAN. FlexTok and ElasticTok fall under the Matryoshka-style tokenizers, and since ALIT, One-D-Piece, and Titok all adopt a similar 2-stage GAN-based training pipeline with 2D VQGAN/VAE tokens, we chose them for a fair, apples-to-apples comparison focusing on our core contributions.\n\nFurthermore, Flextok models are much large-scale (e.g., trained with REPA-style losses using DINO, which is trained on much broader datasets), making direct comparisons with KARL, ALIT, and One-D-Piece less fair in scope. That said, based on the reviewer’s request, we will include Flextok results in the final version. \n\n---\n\n**Limitations**\n\nWe apologize for missing the limitations section in hindsight.\n\n\n- FID gap despite strong per-image metrics: KARL performs competitively or better than existing tokenizers on all metrics except the distribution-level metric FID. This may be due to KC-guided training favoring the search for the shortest program per image—optimizing for single-image metrics (LPIPS etc)—rather than global distribution alignment. A deeper analysis of this trade-off would be valuable.\n\n- Inability of KC to distinguish noise from structure: Kolmogorov Complexity assigns high complexity to both noisy and highly structured images. Higher-order measures such as Sophistication can make this distinction and form the basis of our proposal for interestingness. However, our current interestingness formulation is a simple delta on KC, not a learned signal, also not our main focus.\n\n- General limitation of all adaptive tokenizers: Current methods, including KARL, are constrained to a maximum token budget per image (e.g., 256 tokens for 256×256 images), limiting granularity.\n- **Future Work:** Elaborate study of linear separability or required size of decoder for different downstream tasks.\n- **Future Work:** Exploring the use of learned compressed tokens to train downstream models—such as vision-language models or video models—is a promising and timely direction for further research. As acknowledged by the reviewer, this is challenging due to compute constraints in an academic setting, specially learning both a tokenizer, doing scaling laws at tokenizer levels and then training downsteam tasks all in one paper.\n\n---\n\nFinally, we hope we have addressed all of the reviewer’s concerns. As acknowledged, the paper already includes several key contributions suitable for NeurIPS:  \n(a) a novel single-pass adaptive tokenizer,  \n(b) a loss-conditioned training scheme inspired by the seminal Upside-Down RL framework,  \n(c.) theoretical grounding via Kolmogorov Complexity, and  \n(d) thorough evaluation—both through scaling laws and now via downstream tasks (thanks to the reviewer)."}, {"review_id": "8FHoUlYTb5", "reviewer": "Reviewer_95ih", "summary": "This paper proposes KARL, a one-pass adaptive tokenizer that predicts the minimal number of image tokens needed for reconstruction, inspired by Kolmogorov Complexity. Unlike prior methods that require iterative search, KARL learns to halt token allocation using reconstruction loss as a signal. It achieves comparable or better performance to existing methods while being more efficient. The authors also connect adaptive tokenization to concepts from Algorithmic Information Theory and introduce an image interestingness score aligned with human judgments.", "questions": "How sensitive is KARL’s performance to the choice of reconstruction loss threshold during training and inference? Would different thresholds lead to significantly different token allocations or interestingness scores?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1: KARL avoids expensive iterative search at test time by learning to allocate tokens in a single forward pass. The number of tokens used per image serves as an intuitive proxy for complexity, aligning with human perception.\n\nS2: The method matches or outperforms baselines on standard image reconstruction metrics. The work draws thoughtful connections between adaptive tokenization and principles from Algorithmic Information Theory and introduced interpretable metrics.\n\nW1: The experiments focus primarily on image reconstruction; it's unclear how well KARL generalizes to other vision tasks.\n\nW2: The paper introduces multiple novelty ideas, but doesn’t ablate their individual contributions or alternatives.\n\nW3: While the paper mentions distribution shifts, there's minimal quantitative analysis of how well the method handles out-of-distribution images.\n\nW4: (minor) The paper lacks error bars which makes it hard to assess the robustness of the performance.", "was_revised": true, "final_justification": "Thanks for the authors' response, and they have addressed most of my concerns - I've raised the score accordingly.", "rebuttal": "### **Evaluation on Downstream Tasks**\n\nThank you for the great suggestion. We have now thoroughly evaluated KARL on multiple downstream tasks—CLIP alignment, depth estimation, semantic segmentation, vision-language modeling (VLM), and linear probing for classification. Our findings show that KARL performs comparably to other tokenizers while operating in a single pass. Additional insights and detailed results are provided in our response to Reviewer 2QbU.\n\n---\n\n### **Quantitative Analysis on OOD Datasets**\n\nThank you for the great suggestion. We evaluated the reconstruction performance of KARL, ALIT, and One-D-Piece on three datasets: ImageNet (in-distribution), COCO (moderately OOD), and Wikipedia Images (highly OOD). This revealed an interesting trend:\n\n* KARL's relative performance gap improves as we move further OOD—from ImageNet to COCO to Wikipedia—suggesting that training for minimality may yield more generalizable abstractions, an idea well-grounded in Algorithmic Information Theory (AIT).\n\n**Trained on Imagenet, Evaluated on COCO**\n\\\\[\n\\\\begin{array}{|c|ccc|ccc|ccc|}\n\\\\hline\n\\\\textbf{Method} & \\\\text{L1} \\\\times \\\\text{10 (32)↓} & \\\\text{LPIPS (32)↓} & \\\\text{SSIM (32)↑} & \\\\text{L1} \\\\times \\\\text{10 (64)↓} & \\\\text{LPIPS (64)↓} & \\\\text{SSIM (64)↑} & \\\\text{L1} \\\\times \\\\text{10 (256)↓} & \\\\text{LPIPS (256)↓} & \\\\text{SSIM (256)↑} \\\\\\\\\n\\\\hline\n\\\\text{One-D-Piece} & 1.372 & 0.482 & 0.333 & 1.178 & 0.414 & 0.356 & 0.945 & 0.326 & 0.407 \\\\\\\\\n\\\\text{ALIT} & 1.163 & 0.419 & 0.339 & 0.984 & 0.357 & 0.383 & 0.737 & 0.272 & 0.464 \\\\\\\\\n\\\\text{KARL} & \\\\textbf{1.112} & \\\\textbf{0.405} & \\\\textbf{0.358} & \\\\textbf{0.956} & \\\\textbf{0.349} & \\\\textbf{0.399} & \\\\textbf{0.636} & \\\\textbf{0.228} & \\\\textbf{0.527} \\\\\\\\\n\\\\hline\n\\\\end{array}\n\\\\]\n\n**Trained on Imagenet, Evaluated on Wikipedia Image Dataset (OOD)**\n\n\\\\[\n\\\\begin{array}{|c|ccc|ccc|ccc|}\n\\\\hline\n\\\\textbf{Method} & \\\\text{L1} \\times \\\\text{10 (32)↓} & \\\\text{LPIPS (32)↓} & \\\\text{SSIM (32)↑} & \\\\text{L1} \\times \\\\text{10 (64)↓} & \\\\text{LPIPS (64)↓} & \\\\text{SSIM (64)↑} & \\\\text{L1} \\times \\\\text{10 (256)↓} & \\\\text{LPIPS (256)↓} & \\\\text{SSIM (256)↑} \\\\\\\\\n\\\\hline\n\\\\text{One-D-Piece} & 1.125 & 0.3122 & 0.4450 & 0.956 & 0.2390 & 0.4725 & 0.747 & 0.1617 & 0.5247 \\\\\\\\\n\\\\text{ALIT} & 0.915 & 0.3674 & 0.4638 & 0.767 & 0.3121 & 0.5063 & 0.567 & 0.2394 & 0.5835 \\\\\\\\\n\\\\text{KARL} & \\\\textbf{0.863} & \\\\textbf{0.2507} & \\\\textbf{0.4915} & \\\\textbf{0.733} & \\\\textbf{0.1950} & \\\\textbf{0.5325} & \\\\textbf{0.485} & \\\\textbf{0.1206} & \\\\textbf{0.6465} \\\\\\\\\n\\\\hline\n\\\\end{array}\n\\\\]\n\n\n\n---\n\n### **More Ablations**\n\nAs acknowleged, we conducted extensive ablations on architectural choices and their effect on learned KC and minimal token counts. This includes variations in encoder/decoder depth and width, continuous vs. discrete tokenization, and quantization codebook size. Thanks to your suggestion, we’ve now also included evaluations on downstream tasks and IID vs. OOD settings. We’re happy to explore additional experiments based on further feedback from the reviewer.\n\n---\n\n### **Choice of Reconstruction Thresholds During Training and Inference**\n\nA key strength of KARL is its principled treatment of reconstruction thresholds. During training, each image is either trained with \\\\( \\epsilon = 0.0 \\\\) (lossless) or with a threshold dynamically set by the previous iteration’s reconstruction quality—effectively aligning token allocation with image complexity. This self-regulating mechanism removes the need for manual threshold tuning, a subtle but powerful feature of KARL's training design that we plan to explore further for more tasks, modalities. Basically, each image is trained for its own inherent comoplexity (or value of epsilon).\n\nAt inference time, the threshold \\\\( \\epsilon \\\\) can be chosen based on the needs of downstream tasks. For example, as shown in our downstream evaluations (see response to Reviewer 2QbU), coarse tasks like CLIP alignment tolerate higher \\\\( \\epsilon \\\\) (e.g., 0.5–0.9) with minimal performance drop, while yielding significant token savings.\n\n---\n\n### **Missing Error Bars**\n\nWe assume the reviewer refers to error bars across seeds and other sources of randomness, including statistical significance testing. This is a valuable suggestion—we will include such analyses in the final version.\n\n---\n\nWe hope our overall contributions—including the novel problem statement of single-pass adaptive tokenization, the loss-conditioned training strategy (inspired by the seminal Upside-Down Reinforcement Learning framework), theoretical grounding in Kolmogorov Complexity and AIT, extensive scaling law analysis, and now, comprehensive downstream evaluations—help address all of your concerns."}, {"review_id": "XhQNtj0iKU", "reviewer": "Reviewer_HAzc", "summary": "This paper introduces KARL, a single-pass adaptive image tokenizer that aims to predict the optimal number of tokens for image representation without requiring iterative search at inference time. The authors draw connections between adaptive tokenization and Algorithmic Information Theory (AIT), proposing a loss-conditioned training strategy that learns to halt token allocation once sufficient representational capacity is reached. The method is evaluated and compared against existing approaches like ALIT and One-D-Piece.", "questions": "- How sensitive is the method to the choice of reconstruction loss thresholds during training?", "limitations": "- No analysis of failure cases or method limitations.\n- Lack of theoretical guarantees about optimality of learned token counts.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- **Practical Contribution**: The single-pass approach addresses a real computational bottleneck in existing adaptive tokenization methods that require iterative search at inference time.\n- **Scaling Law Analysis**: The systematic study of various architectural choices (encoder/decoder size, continuous vs. discrete tokenization) provides useful insights.\n\nWeaknesses:\n- **Weak Theoretical Foundation**: The connection to Algorithmic Information Theory is superficial and lacks a rigorous mathematical formulation. Section 4 reads more like a philosophical discussion than a technical contribution.  This significantly undermines the claimed theoretical contribution. Additionally, the definition of \"interestingness\" and its claimed connection to sophistication and logical depth appear ad-hoc and lack theoretical justification. \n- **Misplaced Content**: A substantial portion of Section 4 discussing AIT concepts, sophistication, and logical depth should be moved to related work. The current structure makes the paper feel disorganized.\n- **Missing Complexity Analysis**: Despite claiming efficiency advantages over iterative methods, the paper provides no computational complexity analysis comparing training time, memory requirements, or actual inference speedup against baseline methods like ALIT and One-D-Piece.\n- **Limited Theoretical Novelty**: The core technical contribution is essentially a masking strategy during training. While practical, this represents an incremental engineering improvement rather than a fundamental advance in adaptive tokenization.\n- **Evaluation Concerns**: (1) The variable token allocation comparison may not be entirely fair since it's the primary design goal of KARL. (2) No statistical significance testing due to computational constraints.", "was_revised": true, "final_justification": "Thanks for the reply from the authors. I would like to raise the rating to 4.", "rebuttal": "We appreciate the reviewer’s appreciation on the practicality of single-pass adaptive tokenization and our system study of scaling laws.\n\n---\n\nWe revised Section 4 to better articulate the connection between adaptive tokenizers and AIT. We believe earlier writing may have caused confusion, which we have now clarified. Specifically, we include analogies to KC and Solomonoff Induction, particularly the idea of searching for the shortest program.\n\nOne key reference, lossy (complexity-constrained) compression has been discussed in seminal work of Kolmogorov Structure Function [1], supporting our theoretical framing.\n\n### **Section 4: Adaptive Image Tokenization meets AIT**\n\n**Correlation with KC:** KC formalizes the length of the **shortest program** that **reproduces a data** instance. In our setting, we approximate this by treating the token count as a proxy for program length—each token encodes structured information. Minimizing tokens while preserving fidelity aligns with minimizing description length, and thus, KC. Let’s understand this analogy in more detail\n\nFormally, the **“program”** includes both token sequence and decoder. Since the decoder is fixed and shared across samples, its cost can be treated as constant, leaving minimal tokens as shortest program. \n\nSecondly, because exact reproduction is infeasible in deep learning, we define **“reproduction”** as achieving reconstruction within an error threshold (e.g., ℓ₁ or perceptual loss). This yields a tractable notion of approximate KC, the foundation of our work. \n\nThis view parallels the **seminal Kolmogorov Structure Function (KSF) [1] (see Example II.4)**, where KSF(x, α) captures the size of smallest set S containing x such that KC(S) ≤ α. Here, α is a complexity budget, and S \"explains\" x—analogous to finding the shortest token sequence that reconstructs x within a target loss.\n\n**Searching the shortest program**— \nAlgorithmic Probability frameworks like Solomonoff Induction (SI) aim to find the shortest program that explains data, assigning higher probability to simpler programs -- by exhaustively enumerating over programs that reproduce the data when decoded by UTM, in increasing order of KC.\n\nAligning w/ SI, adaptive tokenizers (e.g., ALIT) approximate this search by varying token budgets at test time, incrementally increasing tokens until the reconstruction meets a fidelity threshold. KARL approximates this shortest-program search in a single pass: it predicts the minimal subset of tokens needed to meet a reconstruction constraint avoiding iterative decoding.\n\nWe have exciting general response regarding the alignment of adaptive visual tokenization and algorithmic information theory, and hope the writing would clarify reviewer's concern regarding the same.\n\n---\n\n**Lack of theoretical guarantees about optimality of learned token counts**\n  \nWe respectfully clarify that computing the **theoretically optimal number of tokens** per data point is inherently intractable—this is a known limitation of Kolmogorov Complexity (KC) and Solomonoff Induction (SI), not specific to our method. Our work aims to **approximate** KC by treating the minimal number of tokens (under a learned decoder) as a proxy for the shortest program that reproduces an image—an approach, to our knowledge, not previously explored for visual tokenization.\n\nA useful theoretical framing of KARL (like all auto-encoders) is via classical rate-distortion analysis:\n\n\\\\[\nZ_{\\\\text{optimal}} = \\\\min_Z \\\\left[ -H(X \\\\mid Z) + \\\\lambda D(X, \\\\hat{X}(Z)) \\\\right]\n\\\\]\n\n- \\\\( H(X \\\\mid Z) \\\\): conditional entropy—uncertainty in \\\\( X \\\\) given \\\\( Z \\\\).  \n- \\\\( D(X, \\\\hat{X}(Z)) \\\\): reconstruction loss; \\\\( \\\\lambda \\\\) balances compression vs. fidelity.\n---\n\n**Limited Theoretical Novelty**\n\nWhile the high-level idea—masking out excess tokens given a token budget—may seem straightforward, the proposed method introduces several non-trivial innovations that, in our view, contribute meaningfully to the field.\n\n**Theoretical grounding in Kolmogorov Complexity (KC):**\nOur approach is not merely an engineering trick but is guided by foundational principles from Algorithmic Information Theory. In particular, we leverage the invariance property of KC—that KC is an intrinsic property of data and should remain unchanged even when more computational resources (i.e., tokens) are available. This motivates our learning objective: once a minimal token subset sufficient to reconstruct an image is identified, additional tokens should be deemed unnecessary.\n\n**Training innovation via loss-conditioned supervision:**\nNaively applying masking during training is non-differentiable and, in our experiments, fails to converge. Common strategies like straight-through estimation or reinforcement learning introduce complexity and instability. In contrast, our method draws inspiration from the Upside-Down Reinforcement Learning framework, enabling a stable two-step procedure within each training iteration:\n(a) Estimate Complexity — mapping the image to a randomly selected token budget and observing the resulting reconstruction error \\\\( \\\\epsilon \\\\);\n(b) Tokenize at Learned Complexity — using \\\\( \\\\epsilon \\\\) as a conditioning signal to guide the encoder to mask out redundant tokens from an expanded budget. This enables effective one-shot prediction of a minimal token set for a given reconstruction constraint.\n\nOverall, we believe that combining this theoretical motivation with a stable, practical training strategy for single-pass adaptive tokenization represents a meaningful technical contribution. We are encouraged by the broader response, which found the method novel and interesting, especially in the context of visual tokenization.\n\nWe hope this response answers the concern of technical novelty. We apologize it this wasn't clear upfront and we will improve the writing.\n\n---\n**Evaluation Concerns**  \nWe want to clarify that variable token allocation is a central goal of all adaptive tokenizers—not just KARL. For example, ALIT explicitly emphasizes this (Figure 6 of their paper). The key distinction lies in **how** adaptation is achieved.\n\nKARL predicts when to halt tokenization in a single forward pass, conditioned on a reconstruction threshold. In contrast, methods like ALIT rely on test-time search with multiple encoder-decoder passes, using decoder feedback to iteratively minimize token count. While this allows slightly tighter token estimates, it comes with higher FLOPs and latency.\n\nAs a result, ALIT achieves a marginally lower estimated \\\\( \\\\hat{KC} \\\\) than KARL (see Table 2), but at the cost of increased compute. These approaches are complementary and could be combined—for instance, using KARL’s prediction as an efficient initialization for further refinement.\n\n--- \n**Limitations**\n\n- FID gap despite strong per-image metrics: KARL performs competitively or better than existing tokenizers on all metrics except the distribution-level metric FID. This may be due to KC-guided training favoring the search for the shortest program per image—optimizing for single-image metrics (LPIPS, SSIM, PSNR, DreamSim)—rather than global distribution alignment. A deeper analysis of this trade-off would be valuable.\n\n- Inability of KC to distinguish noise from structure: As noted in the paper, Kolmogorov Complexity assigns high complexity to both noisy and highly structured images. Higher-order measures such as Sophistication or Logical Depth can make this distinction and form the basis of our proposal for interestingness. However, our current interestingness formulation is a simple delta on KC, not a learned signal.\n\n- General limitation of all adaptive tokenizers: Current methods, including KARL, are constrained to a maximum token budget per image (e.g., 256 tokens for 256×256 images), limiting granularity.\n\n**Future Work**\nExploring the use of learned compressed tokens to train downstream models—such as vision-language models or video models—is a promising and timely direction for further research.\n\n--- \n**Misplaced Content:** Thank you for the suggestion. In the latest revision, Section 4 focuses only on the connections to Kolmogorov Complexity and Solomonoff Induction. We will move broader background and references to AIT concepts to the Related Work section.\n\n--- \n**Complexity Analysis**\n  \nWe thank the reviwer for bringing this up. We compared computational complexity in terms of the number of encoder/decoder calls in Table 2, which is architecture-agnostic and avoids confounds like GPU type or precision. For fair comparison, all baselines use the same encoder/decoder architecture.\n\nHere are the requested numbers in terms of Glops and executing time on a single H100 GPU with FP32 precision:\n\n- **Encoder GFlops:** KARL ≈ One-D-Piece ≈ First iteration of ALIT ≈ 80 GFlops\n- **ALIT:** Requires 8 iterations during training, and ~4 iterations at test time for \\\\( \\epsilon = 0.9 \\\\)-bounded reconstruction. Each additional ALIT iteration adds ~30 GFlops.  \n- **Decoder GFlops:** ALIT and One-D-Piece require multiple decoding passes, each ≈ 80 GFlops.\n\nWall-clock Time (ms): \n- First encoder or decoder pass: ~7 ms\n- Each additional ALIT encoder iteration: ~4 ms\n\n---\nWe hope we have addressed all concerns—through improved writing, citing the strong reference on Kolmogorov Structure Function (KSF), and highlighting the strong analogy between AIT and adaptive tokenization. Additionally, we clarified how our training strategy emerges naturally from KC principles and aligns with the seminal Upside-Down RL framework, which we believe helps address any concerns regarding technical novelty.\n\nInterestingly, we’ve also heard others point out conceptual parallels between our work and the recent interesting H-Net paper on dynamic tokenization for LLMs—both approaches learn *when to stop* tokenization.\nWe’re grateful for your review and are happy to answer any further questions.\n\n[1] Kolmogorov’s Structure Functions and Model Selection"}, {"review_id": "V7s3rWZeG9", "reviewer": "Reviewer_YZ29", "summary": "The paper proposes a new method to learn a tokenizer that can adjust its output (token) length that depends on the complexity of input data, image specifically. The innovation in this paper is a new training method  that is run after training an initial tokenizer that outputs a probability value for rejecting/keeping a token. The proposed method takes inspiration from algorithmic information theory that states that the optimal program is the shortest possible (\"simplest\") program for a given input. Empirical results are provided that shows that the proposed method, KARL (acronym), is able to work as well as a recurrent baseline while being a one step method.", "questions": "- Please clarify the details of the training algorithm used in the experiments in the main body of the paper\n\n- Please check and ensure that the values in bold are for the best performing method in Table 1. For example, I see on row 2 that KARL is bolded for LPIPS metric but it appears that ALIT's value is lower for 32 tokens.\n\n- In Figure 5, the 2nd plot on the left shows a lower FID compared to the plot on the right(for variable number of tokens). Why are the variable token length methods not leading to similar FID values? I am unsure if the conclusions that small encoder + large decoder is supported in general. Please clarify", "limitations": "Limitations in appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n  - Adaptive tokenizers that can adjust its output length based on input complexity is definitely an interesting area of research. See for example reference [19] that proposes a method called ALIT quoted heavily in the paper that was presented recently.\n\n- The method identifies a drawback with recurrent approaches like ALIT in that they are serial in nature. Instead the paper proposes one approach to shift some of this computation to a second training stage and performs one-shot inference\n\n# Weaknesses\n\n- Algorithmic information theory (AIT) is used to inspire the proposed method. However, the connection does not appear to be very concrete. Kolmogorov Complexity (KC) is approximated via token length. The use of token length by itself is sensible while the connecting to AIT/theoretical KC is vague and somewhat confusing to this reader\n\n- In a similar vein to above, the measure of interestingness via \\kappa is defined using SSIM which is from image processing literature. The use of AIT/KC here again is vague and may not even be necessary to define this measure.\n\n- The proposed method needs T, T{\\prime} and \\epsilon but the details of these values used in experiments are missing in the main paper. I skimmed the appendix but was not able to find the information that i needed for T\\prime. Please clarity this information in the main body of the paper", "was_revised": true, "final_justification": "I read through other reviewers' and responses carefully and will provide a final rating after discussions with reviewers and AC.", "rebuttal": "We appreciate the reviewer’s appreciation on the motivation and novelty of a single-pass adaptive tokenizer. The concerns raised relate to (1) lack of theoretical grounding in Algorithmic Information Theory (AIT), (2) writing clarity. We address both below.\n\n---\n\n**We revised Section 4** to better articulate the connection between adaptive tokenizers and AIT. We believe earlier writing may have caused confusion, which we have now clarified. Specifically, we include analogies to KC and Solomonoff Induction, particularly the idea of searching for the shortest program.\n\nOne key reference, lossy (complexity-constrained) compression has been discussed in seminal work of Kolmogorov Structure Function [1], supporting our theoretical framing.\n\n### **Section 4: Adaptive Image Tokenization meets AIT**\n\n**Correlation with KC:** KC formalizes the length of the **shortest program** that **reproduces a data** instance. In our setting, we approximate this by treating the token count as a proxy for program length—each token encodes structured information. Minimizing tokens while preserving fidelity aligns with minimizing description length, and thus, KC. Let’s understand this analogy in more detail\n\nFormally, the **“program”** includes both token sequence and decoder. Since the decoder is fixed and shared across samples, its cost can be treated as constant, leaving minimal tokens as shortest program. \n\nSecondly, because exact reproduction is infeasible in deep learning, we define **“reproduction”** as achieving reconstruction within an error threshold (e.g., ℓ₁ or perceptual loss). This yields a tractable notion of approximate KC, the foundation of our work. \n\nThis view parallels the seminal Kolmogorov Structure Function (KSF) [1] (see Example II.4), where KSF(x, α) captures the size of smallest set S containing x such that KC(S) ≤ α. Here, α is a complexity budget, and S \"explains\" x—analogous to finding the shortest token sequence that reconstructs x within a target loss.\n\n**Searching the shortest program**— \nAlgorithmic Probability frameworks like Solomonoff Induction (SI) aim to find the shortest program that explains data, assigning higher probability to simpler programs -- by exhaustively enumerating over programs that reproduce the data when decoded by UTM, in increasing order of KC.\n\nAligning w/ SI, adaptive tokenizers (e.g., ALIT) approximate this search by varying token budgets at test time, incrementally increasing tokens until the reconstruction meets a fidelity threshold. KARL approximates this shortest-program search in a single pass: it predicts the minimal subset of tokens needed to meet a reconstruction constraint avoiding iterative decoding.\n\n----\n**Next, we revised the Methods section for clarity. We apologize for earlier confusion.**\n\nThe overall procedure is inspired by Kolmogorov Complexity principles (like invariance w/ additional resources/tokens, token minimization) and the training prodecure aligns with the seminal Upside-Down RL Framework.\n\n### **Section 3: Kolmogorov-Approximating Representation Learning**\n\n\nOur goal is to learn representations that are both compressed and predictive, thus: (a) minimize reconstruction loss, (b) minimize representation length.\n\nWhile existing adaptive tokenizers rely on iterative passes at test time to find the minimal tokens for a desired quality, we ask: Can we predict minimality in a single pass?\n\nWe introduce KARL, a loss-conditioned adaptive tokenizer that approximates the image’s minimum description length—the fewest tokens needed for a target reconstruction threshold. At test time, KARL takes an image, token budget, and loss threshold, and outputs only the necessary tokens, masking out the rest.\n\n**Loss-Conditioned Training of KARL:**\nWe define the approximate Kolmogorov Complexity (\\\\( \\\\hat{\\\\text{KC}} \\\\)) of an image \\\\( x \\\\) under a reconstruction threshold \\\\( \\\\epsilon \\\\) and a maximum given token budget, \\\\( T \\\\) as:\n\\\\[\n\\\\hat{\\\\text{KC}}\\_\\\\epsilon(x, T) = \\\\min \\\\left\\\\{ t \\\\leq T \\\\;\\\\middle|\\\\; \\\\mathcal{L}\\_{\\\\text{rec}}(x, \\\\hat{x}\\_t) \\\\leq \\\\epsilon \\\\right\\\\}\n\\\\]\nHere, \\\\( \\\\hat{x}\\_t \\\\) is the reconstruction from the first \\\\( t \\\\) tokens (out of \\\\( T \\\\)), and \\\\( \\\\mathrm{L}\\_{\\\\text{rec}} \\\\) is the reconstruction error. Intuitively, \\\\( \\\\hat{\\\\text{KC}}_\\\\epsilon(x, T) \\\\) represents the smallest token count sufficient to reconstruct \\\\( x \\\\) within error \\\\( \\\\epsilon \\\\), constrained by the budget \\\\( T \\\\). This defines a practical, task-specific approximation of Kolmogorov Complexity.\n\n\nTo encourage learning such minimal programs, we train KARL under the assumption of **KC invariance**:\n\\\\[\n\\\\hat{\\\\text{KC}}_\\\\epsilon(x, T) = \\\\hat{\\\\text{KC}}_\\\\epsilon(x, T + \\\\Delta T)\n\\\\]\nThat is, **increasing the token budget shouldn't change the minimal token count needed for a given \\( \\\\epsilon \\)**. Once the model learns the sufficient subset, adding tokens should not alter the representation. This principle guides KARL to mask out unnecessary tokens during training.\n\n\n**Training Procedure:**\nEach training iteration runs the encoder-decoder pipeline twice, reflecting the principle that KC remains invariant once the sufficient token count is found for reconstruction. We now describe the two steps performed in each training iteration:\n\n**1. Estimating Image Complexity:**\nGiven an input image \\\\( \\\\mathbf{x} \\\\), the model first attempts near-lossless compression using a token budget \\\\( T \\\\), optimizing:\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{EIC}} = \\\\mathcal{L}_{\\\\text{recon}}(\\\\hat{\\\\mathbf{x}}\\_T, \\\\mathbf{x}) + \\\\beta \\\\, \\\\mathcal{L}\\_{\\\\text{quant}}(\\\\mathbf{z}\\_T)\n\\\\]\n\nwhere \\\\( \\\\hat{\\\\mathbf{x}}_T = \\\\texttt{Decoder}(\\\\texttt{Encoder}(\\\\mathbf{x}, T, \\\\epsilon = 0)) \\\\), and \\\\( \\\\beta \\\\) balances quantization. Here, \\\\( \\\\epsilon = 0 \\\\) enforces near-perfect reconstruction. The resulting reconstruction error \\\\( \\\\epsilon_0 = \\\\left| \\\\hat{\\\\mathbf{x}}_T - \\\\mathbf{x} \\\\right| \\\\) defines the target quality for the second run. Sampling \\\\( T \\\\) randomly each iteration yields a batch of triplets \\{image, token budget \\\\( T \\\\), target error \\\\( \\\\epsilon_0 \\\\)\\} that guide complexity-aware tokenization.\n\n\n\n**2. Learning to Tokenize at Estimated Complexity:**\nIn the second run, given the simulated triplet \\{\\\\mathbf{x}, T, \\\\epsilon\\_0 \\}, the encoder receives an increased token budget \\\\( T + \\\\Delta T \\\\). Conditioned on \\\\( \\\\epsilon_0 \\\\), it outputs both token embeddings and halting probabilities \\\\( \\\\boldsymbol{\\\\omega} \\\\in [0,1]^{T + \\\\Delta T} \\\\), indicating which tokens to retain.\n\nThe model is optimized using:\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{LTC}} = \n\\\\mathcal{L}\\_{\\\\text{recon}}(\\\\hat{\\\\mathbf{x}}\\_{T+\\\\Delta T}, \\\\mathbf{x}) + \\\\beta \\\\, \\\\mathcal{L}\\_{\\\\text{quant}}(\\\\mathbf{z}\\_{T+\\\\Delta T}) + \\\\lambda \\\\, \\\\mathcal{L}\\_{\\\\text{halt}}(\\\\boldsymbol{\\\\omega})\n\\\\]\n\nwhere \\\\( \\\\hat{\\\\mathbf{x}}\\_{T+\\\\Delta T} = \\\\texttt{Decoder}(\\\\texttt{Encoder}(\\\\mathbf{x}, T+\\\\Delta T, \\\\epsilon = \\\\epsilon\\_0)) \\\\).\nThe encoder is trained to achieve the same recon. quality \\\\( \\\\epsilon\\_0 \\\\) while identifying which tokens can be masked out. Halted tokens are excluded from self-attention, thus do not contribute to reconstruction.\n\nThe halting loss is:\n\n\\\\[\n\\\\mathcal{L}\\_{\\\\text{halt}}(\\\\boldsymbol{\\\\omega}) = \n\\\\text{BCE}(\\\\boldsymbol{\\\\omega}\\_{0:T}, \\\\mathbf{0}) + \n\\\\text{BCE}(\\\\boldsymbol{\\\\omega}\\_{T:T+\\\\Delta T}, \\\\mathbf{1})\n\\\\]\n\nencouraging \\\\( \\\\omega_i \\\\approx 0 \\\\) for essential tokens \\\\( i < T \\\\) and \\\\( \\\\omega\\_i \\\\approx 1 \\\\) for surplus tokens \\\\( i \\\\geq T \\\\).\n\nThe full training objective is \\\\( \\\\mathcal{L}\\_{\\\\text{EIC}} + \\\\mathcal{L}\\_{\\\\text{LTC}} \\\\). This parallels Upside-Down RL, where the first run sets a condition (\\\\( \\\\epsilon\\_0 \\\\)) and the second learns to act under it—akin to reframing RL as supervised learning with reward conditioning.\n\nAt inference, only second step is used: given a max token budget & a target error \\\\( \\\\epsilon \\\\), KARL outputs both token embeddings & halting probabilities in a single pass.\n\n\n---\n**Clarification regarding T':** We believe T' was probably not the best choice of symbol; it's replaced with \\\\( \\\\Delta T \\\\) in the text above. As mentioned earlier, in the **Learning to Estimate Complexity** step, for each image in the batch, we *randomly* sample an input token budget \\\\( T \\\\), which results in a \\\\( \\\\epsilon \\\\) error due to lossy compression. Then, for each image, we randomly sample \\\\( \\\\Delta T \\\\) (which is the same as T' in the earlier draft). In the **Tokenize at Complexity** step, the encoder—conditioned on the \\\\( \\\\epsilon \\\\) obtained in the previous step—is tasked to use only the minimal number of tokens from the new input budget of \\\\( T + \\\\Delta \\\\). We then train the network to mask out the extra \\\\( \\\\Delta T \\\\) tokens, since \\\\( T \\\\) was already sufficient for \\\\( \\\\epsilon \\\\)-bounded reconstruction.\n\n---\n**Regarding Interestingness and Connections to AIT** -- Interestingness is deeply rooted in AIT. For example, see Scott Aaronson's blog on \"The First Law of Complexodynamics\" or Peter Bloem’s slides on Sophistication. Measures like sophistication, logical depth, and KSF [1] are all derivative notions based on Kolmogorov Complexity (KC). While we observed early signs that adaptive tokenizers—especially KARL—could estimate sophistication (via the rate of change in reconstruction quality with respect to token count), the main contribution we emphasize is the novel **single-pass adaptive tokenization** and its core analogy with KC.\n\n---\n**Figure 5 query** — The left plot uses \\\\( \\epsilon = 0 \\\\), targeting perfect reconstruction, so FID at 256 tokens is lower. In contrast, the right plot shows KARL stopping early with \\\\( \\epsilon > 0 \\\\), leading to higher FID. We hope this clarifies the difference.\n\n---\n**Table formatting** — Thank you for pointing this out; we will correct it.\n\nWe hope the two revised sections address the reviewer’s concerns."}]

Author rebuttal:

Meta-review: This paper introduces KARL, a novel and efficient single-pass adaptive tokenizer that performs comparably to state-of-the-art iterative methods. The authors successfully addressed initial reviewer concerns by providing comprehensive evaluations on various downstream tasks and clarifying the method's theoretical grounding. The paper's novelty, good empirical validation, and thorough rebuttal make it a good contribution.
Decision: Accept (poster)