Adaptive Length Image Tokenization via Recurrent Allocation
Authors: Shivam Duggal
Phillip Isola
Antonio Torralba
William T. Freeman
Venue: ICLR 2025 Poster (2025)
Keywords: Representation Learning
Image Tokenization
Compression
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Adaptive Visual Tokenizer

Abstract: Current vision systems typically assign fixed-length representations to images, regardless of the information content. This contrasts with human intelligence —and even large language models—which allocate varying representational capacities based on entropy, context and familiarity. Inspired by this, we propose an approach to learn variable-length token representations for 2D images. Our encoder-decoder architecture recursively processes 2D image tokens, distilling them into 1D latent tokens over multiple iterations of recurrent rollouts. Each iteration refines the 2D tokens, updates the existing 1D latent tokens, and adaptively increases representational capacity by adding new tokens. This enables compression of images into a variable number of tokens, ranging from 32 to 256. We validate our tokenizer using reconstruction loss and FID metrics, demonstrating that token count aligns with image entropy, familiarity and downstream task requirements. Recurrent token processing with increasing representational capacity in each iteration shows signs of token specialization, revealing potential for object / part discovery.

Published as a conference paper at ICLR 2025
ADAPTIVE LENGTH IMAGE TOKENIZATION VIA RECURRENT ALLOCATION
Shivam Duggal MIT CSAIL
Phillip Isola
Antonio Torralba
William T. Freeman
ABSTRACT
Current vision systems typically assign fixed-length representations to images, regardless of the information content. This contrasts with human intelligence and even large language models-whice allocate varying representational ca- pacities based on entropy, context and familiarity. Inspired by this, propose an approach to learn variable-length token representations for 2D images. Our encoder-decoder architecture recursively processes 2D image tokens, distilling them into 1D latent tokens over multiple iterations of recurrent rollouts. Each iteration refines the 2D tokens, updates the existing ID latent tokens, and adap- tively increases representational capacity by adding new tokens. This enables compression of images into variable number of tokens, ranging from 32 256. We validate our tokenizer using reconstruction loss and FID metrics, demon- strating that token count aligns with image entropy. familiarity and downstream task requirements. Recurrent token processing with increasing representational capacity in each iteration shows signs of token specialization, revealing poten- tial for object / part discovery. Code available at https:/ httpt/gtttt//.///////// com/ ShivamDuggai44 //wpgg/a/////a///ee//e//paea/aee
INTRODUCTION
Representation learning (Bengio et al., 2013), which involves extracting meaningful and useful information from input observations, is crucial for decision-making. An effective representation should be compact while encoding all relevant information. However, what constitutes "relevant" information varies based on the specific task; for example, a coarse classification task may require a different latent representation compression factor for satisfactory performance compared to a task demanding perfect pixel-level reconstruction, which necessitates denser representations. This notion of useful representation aligns closely with aspects of human intelligence (Legg & Hutter, 2007), particularly the concept of adaptive and auriable-compressibbee representations (Hutter, 2006). Simi- larly, language models can describe content at various levels of abstraction depending on complexity, context (Graves, 2016; Dehghna et al., 2018), and familiarity (Baevski & Auli, 2018). In contrast, most current visual systems, such VAEs, VQGANs, and ViTs (Kingma & Welling, 2022; Esser et al., 2020; Dosovitskiy et al., 2020), generate fixed- size representations for all images. In this work, take step toward learning adaptive and variable-length visual representations, emphasizing that each image requires a different representation capacity (see Sec. 4).
A common framework for learning image embeddings representations is the encoder-decoder approach, where an encoder compresses input data into a compact latent representation, which can later be decoded and compared with the original image learning objective. While there are other encoder-only methods, such as contrastive learning (Chen et al., 2021) and self-distillation (Caron et al., 2021). we focus on encoder-decoder approaches because reconstruction objective intuitively promotes the learning of adaptive representations by capturing varying level-of-details necessary for better reconstruction. The current tfate-oftthe-an (Dosovitskiy et al., 2020) operate in the discrete token space, by encoding images into learned tokens and then decoding them back to image pixels. To generate these tokens, these approaches compress (slightly) the input patch-level and then maintain the number of tokens (= number of patches) throughout the encoder-decoder network depth. Thus, the representation length for all images is fixed to the number of tokens, equivalent to the fixed patch-size decided by the human- human-engineer. Moreover, by having number of tokens equal to number of patches, such approaches are tied to the natural 2D inductive bias of images, preventing any form of adaptive representation or compression
wransformer-based encoder-decoder approaches

---

Published as a conference paper at ICLR 2025
Recurrent Processing Adaptive Token Mem. (loaping multiple recurrent iterations)
Decoding required either for Reconstruction or Optional Dynamic Halting during Training
New ID Tokens (Intialios)
New Takens (Intisiiod)
Latent ID Tokons (F-2)
Latent- SoonttDistiilationn ation Encoder
Latent- Latent-Distille Distillation Encoder
Latent- Latent-Distilltionn Decoder
Factorize & Qwantios
Quazized Laten ID Tokens
Takens (e-)
Image Encoder
Image Tokens (t-1)
Larent
Recontrurted Image
2D Tokere (t-4)
Image Decoder
Recomstraceed Image
2D Takaus ((-2)
Inpur Image
Masked
Takeus
(1-1)
2D
Dynamic Halting: Tokens reconstructed well are masked already info.
Reconstruction on Less with Image 2D Tokens (t=() providees mask (ostional dynamic halting.
from further processing
Figure I: Adaptive Length Image Tokenizer (ALIT): Given an image, we first convert it into 2D image tokens before applying the 2D - ID latent distillation. ALIT recurrently distills 2D image tokens into variable ID latent tokens, with each iteration adding new latent tokens and processing them with the existing 2D image tokens and the old latent tokens. Training focuses on reconstruct- ing 2D image tokens through reverse distillation from latent 1D to masked 2D tokens. Based on token-reconstruution quality, we can optionally mask specific 2D tokens from further processing, enabling dynamic halting per token. Recurrent processing with Adaptive Memory leads to compressible representations, flexible tokenization & specialized tokens focusing on objects/parts.
of different images. Moving away from this inductive bias and with the goal of having modality- agnostic architecture, Google DeepMind proposed Perceiver (Jaegle et al., 2021b;a), transformer- based architecture which distills imput data tokens to a set of fixed ID tokens. This process of latent-token distillation refers to compressing a higher-dimensional input (c.g.. 2D image tokens) into a more compact set of latent variables (1D tokens), capturing the most relevant features. Like Perceiver, also fall into the category of latent-token distillation, where we encode 2D image tokens into much fewer ID latent tokens via self-supervised reconstruction objective. While ID- tokenization of an image overcomes the patch to token constraint and allows much more efficient compression of the input image, more universal tokenizer would be one which adaptively assigns variable tokens to each input based on content entropy, familiarity etc (Sec. 4).
We tackle the challenge of adaptive or variable-length representation learning by ruto-regressively distilling input visual observations into an increasing number of 1D latent tokens. To achieve this, we draw inspiration from foundational works on recurrent computation (Graves, 2016; Dehghani et al., 2018). Recurrent neural networks are often viewed as adaptive thinking modules (Schwarzschild et al., 2021), capable of enhancing the computational requirements of specific input through re- cursive processing with the same neural network architecture. Thus, unlike the Matryoshka style (Kusupati et al., 2022) approach of learning multiple representations of varying lengths simultane- ously in one-go, we adopt a recurrent computing approach for visual representation learning. In our framework, recurrent computing involves recursively distilling an input image 2D image tokens into 1D latent tokens through a shared encoder- decoder architecture until each image token has been sufficiently processed/distille into the latent tokens. At each iteration of this recurrent rollout, we provide additional computational resources in the form of new learnable latent tokens, enabling the model learn adaptive and variable-length representations across different iterations.
We refer to our approach as ALIT (Adaptive Length Image Tokenizer), and train it using self- supervised image reconstruction objective. Credited to the increasing representational capacity, each recurrent update leads to the latent tokens specializing and attending to localized regions, hinting at object/ / part discovery (see, Fig. 7, Fig. 8, Appendix Fig. 15 and Fig. 16). We validate the effective- ness of the learned tokenizer by demonstrating comparable reconstruction metrics (L1 loss and FID) on multiple datasets (IN, COCO, Places, Art-dataset and even randomly selected internet images Fig. 20) and linear probing results on ImageNet-1K, IK, relative to the 2D VQGAN tokenizer (Esser et al., 2020) and fixed-latent ID tokenizer, Titok (Yu et al., 2024), while also allowing for flexible token counts per image. By utilizing variable representations per image, introduce cumulative dataset representations & emphasize the key aspects of effective representations: required capacity aligns w/ image's 's information entropy, familiarity, & knowledge of downstream tasks models.
2

---

Published as a conference paper at ICLR 2025
L:t TitokS-128 Sooks-112 128
VQGAN-256
Ours 32
O2r
Ours 256
Ground Truth
Histogram of # Images L1 Recon, Loss
(IN-ageesNN-1))1 100)
Dun-32 6,118
800-
Oun-178 11:0.008
Number of (IN--)
Ours 256 0.507
1itues.32 L1:0,22
600-
Teck-S-228 [1:: 0.057
VOSAN [L1:0 900
600-
200
5.00
0.10
0.35
0.20
0.25
0.50
L1 Reconstruction Loss
Figure 2: Reconstruction Analysis on ImageNet-1000 100: Our approach outperforms all baselines in terms of reconstruction loss (right). Comparing Row-1 (high complexity) and Row-2 (low com- plexity) demonstrates the effectiveness of adaptive tokenization. Even with fewer tokens, recon- structions maintain reasonable global alignment with ground truth, with an expected loss in detail.
RELATED WORK
The goal of image tokenization is to map high-dimensional 2D images into compressed latent repre- sentations. Modern vision systems often use self- supe-suue learning objectives, such as contrastive learning, self-distillation, in-painting, and generative modeling, to achieve this. Architecturally, these methods typically convert images into 2D feature embeddings using convolutional backbones or transformers after splitting images into patches. However, this approach constrains and tightly binds the processing. compression, and representation capacities to fixed number of processing units, down-sampling steps or patches, regardless of the input image. Several prior works have explored such issues with different motivations, as discuss below
Dynamic Token Processing: Several works (Bolya et al., 2023; Rao et al., 2021; Yin et al., 2022) focus on dynamically processing tokens in the ViT architecture by pruning or merging them across layers. Token Merging (Bolya et al., 2023) accelerates ViT by merging fixed number of tokens per layer, resulting consistent token count for each image. Inspired by ACT (Graves, 2016) and Universal Transformers (Dehghani et al., 2018), Dynamic VIT (Rao et al., 2021) and A-ViT (Yin et al., 2022) adaptively prune tokens or dynamically halt processing for different tokens, with a focus on classification tasks. Concurrent work (Jain et al., 2024) extends this by routing 2D image tokens through different experts, rather than pruning or merging, for classification and image retrieval. Tokens in these works remain tightly coupled to image patches. Our approach also involves dynamic token processing but primarily focuses on distilling images into a variable- compressed 1D) latent space via self-supervised reconstruction objective, allowing each image to have flexible representational capacity beyond patch-based tokens.
Flexible or Variable-Length Representation Learning: Each image has varying levels of de- tail, making a single patch size insufficient for vision transformers. FiexViT (Beyer et al., 2022) uses variable patch sizes for multiple image representations, though theoretically its capacity is still limited by the smallest patch size and 2D token-patch bias. Matryoshka Representation Learning (Kusupati et al., 2022) learns flexible but fixed representations (bounded by the feature dimension) by ensuring low-dimensional subsets of feature vector can perform classification and image re- trieval. Concurrent works (Hu et al., 2024; Cai al., 2024) extend this token space, enforcing sub- tasks. We also learn variable- token representations, sets of tokens to support vision- but unlike static Matryoshka methods, which learn all the representations one go, we focus on recurrent processing & adaptive memory-iterative doors for longer representations (for streaming data) through longer rollouts in future. Recently published arXiv, ElasticTok (Yan et al., 2024) similar to Matryoshka representations- learns variable-length encodings for images and videos by learning a fixed, max-sized representation in one step, then searching for mask to sample subset of this full-length representation.
refining & adding new latent tokens-opening
Latent Tokens or 1D Tokenization: To overcome the 2D token-patch bias, Perceiver (Jaegle et al., 2021b;a) distills 2D image tokens into ID latent tokens not tied to specific patches, aim- agnostic transformers. Similar approaches, such as Recurrent Interface Networks (RIN) (Jabri et al., 2023), AdaTape (Xue et al., 2023), and Titok (Yu et al., 2024), perform 2D-to- ID distillation or read-write operations for generation, recognition, and reconstruction, respectively. RIN and Titok use fixed-length token representations, while AdaTape allows one-time selection of
ing for andasitt-agnostic
3

---

Published as a conference paper at ICLR 2025
-Number Latent Tokens (32, 64, 128, 192, 256)
Ground
Low Complexity Art Hypothesis
Human Predicted Image Complexity
1.0
256
224
0.8
Reconstion dertuuction Loos
Number of Tokens
192
160
0.6
128
0.4
96
64
0.2
32
0.0
15
25
35
45 55
65
75
85
Image Complexity Human Score
Figure 3: Compression vs. Information Entropy Hypothesis on the Out-of- People- Art Dataset: Adaptive tokenization enables analysis of the com-oeieex Art Hypothesis by examining token requirements for images of varying complexity. The plot on the right clearly shows that as (human-annotatedd image complexity increases, so does the need for more computational tokens. More complex images have higher L1 reconstruction loss at fewer token count.
variable-length latent tokens per input image. RIN uniquely performs recurrent read-write between image and latent tokens. Unlike these, we integrate 2D-to-ID distillation with both recurrent pro- cessing and adaptive memory- iitrraively refining image and existing latent tokens, and adaptively adding more latent tokens as new memory. We also enable optional dynamic halting for improved distillation in regions needing further refinement. Beyond the technical differences, we emphasize that each image requires unique representation depending on complexity, familiarity, downstream model/task (Sec. 4). Moreover, the proposed recurrent computing with adaptive memory promotes emergent token specialization for object/part discovery, Fig. 7, Fig. 8, Appendix Fig. 15 and Fig. 16
Relevant works on large-language models include (Goyal et al., 2024; Herel & Mikolov, 2024) which allocate additional compute budget in terms of fixed additional "thinking" tokens, separate from input tokens. They demonstrate improved reasoning capabilities w/ thinking tokens for LLMs.
ADAPTIVE LENGTH IMAGE TOKENIZATION
Tokenization refers to the process of breaking down input data into discrete units or tokens, that are suitable for a specific downstream task. penepol-purpose tokenizers are usually trained with self- supervised objectives such as auto-encoding, next-token prediction, contrastive learning, or self- distillation. the visual domain, prominent tokenizers like VAEs (Kingma & Welling, 2022), VQ- GAN (Esser et al., 2020), and ViT (Dosovitskiy et al., 2020) rely heavily on the 2D spatial inductive bias of images, treating 2D patches as tokens. This approach ties the tokenizer's i architecture closely to the inductive bias of the visual domain and limits its representational capacity to fixed num- ber of tokens based on the number of image patches. The Perceiver line of research (Jacgle et al., 202 1b;a) overcomes the 2D inductive bias limitation, (proposing modallty-agnossiti architecture) while still distilling 2D image tokens into a fixed one-dimensional learned representation. In this work, we argue that each image is unique and warrants different number of tokens. To address this, we propose a novel framework that auto-regressively allocates more representational capacity (i.e., tokens) to an image, allowing for a variable number of tokens for each image at test time. We first outline the core auto-encoding modules - latent- dictiliistillttion encoder and decoder that dis- till 2D images into ID tokens and back, and then introduce our approach of rego-segressive token allocation per image. Fig. 1 provides an overview of Adaptive Length Image Tokenizer (ALIT).
Latent Distillation of 2D Image Tokens to 1D Tokens: We want to map an input image to ID latent tokens. Focusing on the core problem of compressive aaaptive representation learning (and primarily for compute reasons), we first leverage an existing VQGAN image tokenizer to first map an input image to a set of 2D image tokens, K Credited to years of research done on quantized 2D auto-encoders, the pre-trained VQGAN model can map a 256 x 256 image to 16 X 16 2D spatial tokens, without much loss of detail. Each of the 16 x 16 tokens is a pointer to one of the quantized codes in the trained VQGAN codebook. In this section, dimensional image) spatial tokens to a few KiD( < 256) ID tokens. For majority of the experiments, set this atomic (min. token count per image) number, Kip to 32, for ease of experimentation.
distill the K2D (= 256 for 256-
4

---

Published as a conference paper at ICLR 2025
Min Tokens using Token Selection Criteria
TSC: Top X Classification Accuracy
Tokans using Taken Selection Criteria
TSC: Dataset Depth Error < Threshold
A cnkee
3.95
1.0-
Depth Anything V2 1VT14)
93
Top-1 Classification Acc. (9%
(ar 5% max sesset toleeds)
Resher18
Classification Acc. (%)
(es o max clata sel tokerss)
Depth Anyching V2 SVIT
0.8-
3.83
Task of Interest
088
Task of Interest
25
0.7-
0.8
Tra Chseffor on Aor (0) jaing
3.71
0.6-
83
0
0.4-
2.59
0.59
0.2
0.2
2
15
0.00
0.41
0.66
90
1.15
1.40
for Top-X Classification
Abs. ative Depth Error Thresholds
Dataset representation cumulative token count satisfies nice pareto w/ TSC complexity (Top-1 requires more tokens than Top-5)
Matching TSC TOI Improves TOI performance Latent optimized for classification performs better on classification than latent optimized for depth.
Figure 4: Analysing Dataset Representation Capacity: We vary tokens per image using differennt Token Selection Criteria (TSC) Best Top-X Classification Accuracy (Left) and Depth Error < Threshold (Right). We use GT class/depth maps for computing TSC classification/depth errors. We then evaluate Classification Accuracy (Task of Interest, TOI) the dataset reconstructed using different TSCs. X-axis = TSC, Y-axis = Dataset Token Count, Marker-Color = TOI Perf.
2D-1D Distillation - Given K spatial image tokens / features, each of dimension dæp, we append them with KJD latent tokens along the token axis, and pass then through the latent- distillation-encoder, Enc. K KjD are initialized with learned embeddings. The distillation encoder performs joint self-attention on all the tokens and distills K + KID. Although previous works (Jaegle et al., 2021b; Jabri et al., 2022) have experimented with cross-attention, we do not focus on this aspect of the architecture for this work and leave it for future analysis. The distilled latent tokens. KiD, are then passed to the distillation decoder Dec, which appends them to MND masked tokens and performs the reverse task of distilling the latent tokens back to the 2D spatial tokens ie KID + MaD. All inputs to the eesillattionencoodee and distillation-decode added with positional encoding (separate ones for 2D image tokens, ID latent tokens and 2D masked tokens). We factorize and quantize the distilled latent tokens (output of Enc) before passing them to the distillation-decdder, by sampling from learned ID codebook via closest codebook logic, following (Yu et al., 2021; 2024). Among other techniques (Zhu et al., 2024; Huh et al., 2023), we found factorization to be most useful for learning quantized ID codebook.
masked tokens are
Kt! Ktll Enc( K ; KiD 1) M = Dcc([ Map: K ])
}
Latent Distillation 1 1d Iteration
KDD - KID denotes an encoder update to map initialized latent embedding to learned distilled emhedding. Likewise, MTD > MnD denotes reverse distillation using learned latent tokens (Kıp) to map masked 2D tokens to reconstructed image tokens. t=0 to t=1 denotes one encoder update. The main learning objective is reconstruction loss between MaD and K [s] denotes concatenation.
Auto-regressive Framework for Variable Tokenization: In the previous section, we explained the core module for 2D-1ID distillation module. We now describe the regr-seeee rolling of the encoder-decoder distillation module for learning variable tokens per image, with Kt=l as the minimum tokens per image. Multiple works (Dehghani et al., 2018; Graves, 2016) in sequential decision making and natural language processing perform recursive roll-out of the same thinking (Schwarzschild et al., 2021) architecture to provide more computational budget to the input task. In similar vein, perform recurrent processing of the input image with the objective of learning lennth compressed representations. With each roll-out iteration, we not only provide more processing capacity by recursively rolling out the distillation Enc Dec architecture, but also provide additional computational memory in terms of new writeable tokens to better distill image tokens into more ID latents. We now dive into the details.
2 Tooeen-Selection Criteria (TSC) Example- Selecting per image tokens using TSC= Top-> X Classification means identifying the minimum number of tokens for each image such that the corresponding reconstructed image is correctly classified (by comparing against GT label) among the Top X predictions by ResNet-18.
5

---

Published as a conference paper at ICLR 2025
Tokena max talenn)) for RorRBen Love Threshold
1.0
85
10
G0000
14
1.0
12.37
Fréichat Incept dist (FID)
Classification Acc. 186
Depih Error 1
12
0.
0
0.7
07
07
8.53
22.43
05
055
04
0.
12.83
2
900
0.55
0.08
0.08
Recorstruttion Thresholds
Reconstruction Threshold:
Reconstruction Loss Thresholds
Figure 5: TSC-TOI Alignment determines Dataset Representation Capacity: We vary tokens image via "Reconstruction Loss < Threshold" automatic Token Selection Criteria (TSC) and evaluate multiple Tasks of Interest (TOI) on the reconstructions. Since TSC determines dataset token count, strong TSC-TOI alignment enables desired TOI performance at compressed represen- tations- 60% of max-tokens (selected via Recon. TSC) achieve similar perf. on all TOIs as max tokens perf., supporting totppiivee-oeeiiaaiinnnn Fig. 13 /
14 show classification depth TSC plots.
At each iteration of latent distillation, we concatenate the latent tokens from the previous iteration, KET with additional newly initialized tokens (initialized with learned embeddings), K Option- ally, to help the distillation encoder focus on image tokens that were not perfectly distilled in the previous iteration, we apply a masking dynamic halting operation (Mask) to the processed image tokens from the last iteration, KTD This mask is determined by the alignment between reconstructed output M and original image tokens KID. The masked image tokens are then concatenated with the latent tokens and passed through the distillation encoder-decoder, Enc Dec. This process is repeated across multiple iterations. As in single-step distillation, the primary training objective is minimize the reconstruction loss between the new reconstruction, MD and the original image tokens K2D- At each iteration, the distilled latent tokens are factorized and quantized using shared ID codebook tokens learned across different iterations belong to the same embedding space.
K
K
KiD KT
KTT
Mask
M K )
Latent Distillation T + 1hh Iteration
K Enc([
K
K ])
M Dec([ M20 ; KtT+1 ])
In summary, at each iteration of the recurrent rollout, the latent tokens from the previous iteration receive residual updates, while new computational memory (additional latent tokens) is introduced. These tokens give the existing latent tokens the freedom focus specialized regions, leading to sharper & sparser attention, as shown in Fig. 7, Fig. 8, Appendix Fig. 15 and Fig. 16.
Training Procedure: Our training follows multi- stage approach similar (Esser et al., 2020; Yu et al., 2024). We begin by reconstructing pre-trained VQGAN image tokens from distilled latent tokens, using smooth cross-entropy loss on the discrete VQGAN tokens (as explained above). Next, we jointly optimize VQGAN encoder-decoder and the distillation Enc - Dec with an objective to reconstruct image pixels. Following (Esser al., 2020), we introduce GAN loss later the training process, once the image reconstruction quality has reached a satisfactory level across multiple repre- sentations, to further improve photo-realism. In addition recon. and adversarial losses, we apply quantization losses throughout the training to facilitate the learning of 11) latent codebook. Please refer to Appendix Sec. A.6 for more details on the training procedure implementation details.
NOT ALL IMAGES ARE WORTH THE SAME REPRESENTATION
Each image is unique and requires a different number of tokens as representation. Additionally, each image observation have multiple valid representations, echoing Epicurus' notion of multiple explanations. By mapping an image to various quantized latent spaces, the model learns sample different tokens from the training set's codebook, optimizing the reconstruction objective at different levels of computational capacity. In this section, we provide experimental insights into how learning adaptive representations can help support and expand upon these concepts.

---

Published as a conference paper at ICLR 2025
Dopth Estimation NYUv2
Classification Imagenet100 00
VLM LLLVA-
A Top-1 Class, Normalized (%)
Abs Rel Normalized cormallizd
Relative Score wint GPT4-0
A Performance (Normalized)
Abs. Rel Depth Enrr
Top-1 Ac Accuracy (%6
Denseket23
rospooio
qpe aoge
0008
000
0e
7
0
Ads
2 mr
# Latent Takens
# Latent
Tokens
Latent Takens
Figure 6: Tokens vs. Model Strength:
Stronger models outperform weaker ones across all token
counts but are more sensitive to fewer-token reconstructions, showing a sharper perf. drop (~~color) at fewer tokens. In contrast, weaker models (lower GT perf.) can manage with fewer tokens.
Representation Capacity or Compression Depends on Information Entropy Complexity: Schmidhuber's Low Complexity Art theory (Schmidhuber, 1996) correlates an image's ' perceptual complexity with its compressibiliy-- smallest description length of an image often aligns with its complexity. Given that our approach generates multiple compressed representations for given image, we evaluate such correlation between human-labeled complexity estimates (ranging from O to 100) (Sarace et al., 2018) and the L1 reconstruction loss using our adaptive tokenizer varying token capacities. Fig. 3 (left) illustrates reconstructions of completely out-of-distribution (OOD) images from the PeopleART dataset (Westlake et al., 2016), using between 32 and 256 tokens for images of different complexities. These reconstructions are produced using a model trained on ImageNet-100, which contains no art-related images and is significantly different from PeOPleART. As seen, the low-complexity image (top row) is adequately reconstructed with fewer tokens, while the highly complex image (bottom row) requires more tokens for accurate reconstruction. Fur- thermore, the observations: (a) as image complexity increases, reconstructions with fewer tokens result in higher LI errors, necessitating larger memory budget; and (b) at fixed image complexity, increasing the computational budget (i.e., number of tokens) reduces the loss, demonstrating the efficiency of the adaptive representation model. See Appendix Fig. 11 and Fig. 12 for more such results.
correlation plot in Fig. 3 (right) perfectly highlights two
Representation Capacity Depends on Familiarity with the Training Set: Similar to how out- of-syllabus questions require more effort, reconstructing OOD images demands more computational tokens. By learning quantized adaptive representations, our model maps test images to adaptive tokens by sampling from a learned trainset codebook. This enables us to distinguish between in- distribution (IID) and iutrof-distribution (OOD) images. IID images more efficiently recon- structed with fewer tokens, as they can sample familiar representations from the trained codebook, whereas OOD images require more tokens. From Tab. 1, the FID gap between 64 and 256 tokens is smallest on in-distribution ImageNet- 100 validation set (7.92), larger on less in-distribution COCO dataset (12.56), and largest on highly OOD Wikipedia images (23.32). In summary, while all models exhibit performance loss on OOD images, the loss is most pronounced with fewer tokens. However, training larger adaptive tokenizers on larger datasets (eg: LAION) over longer periods may close the distribution gap, as evidenced by improved performance on IN-IK vs IN-100 (Fig. 9 third plot).
Representational Capacity Depends on Downstream Task: We utilize our variable-length rep- resentations to demonstrate how dataset representations vary across downstream tasks. To achieve this, we select different tokens for each image based on specific token-selection criteria (TSC) and analyze the minimum dataset representations required for optimal performance, plotting cumula- tive image token counts as fraction of the total VQGAN tokens. By dataset representation, we mean cumulative token count for the dataset when different images are reconstructed using differ- ent token counts. For TSC=Classifccttion 2, we evaluate the minimum tokens needed for the best I, top-2, top-X accuracy. For TSC=Depth Estimation, we determine the token count nec- essary to achieve per-image relative depth errors below thresholds such 0.2, 0.4, 0.6. Likewise TSC-Reconttruutton Loss, selects per image tokens based on LI recon. loss < certain thresholds. Notably Reconstruction Loss serves as automatic seeff-suee token selection criteria, while Classification / Depth Error as TSC requires GT class-labels / Pseudo-GT depth maps.
In Fig. 4, select minimum tokens based on two criteria: classification (left) and depth (right). The images reconstructed using selected tokens are then assessed using ResNet- 18 for Classification Ac- curacy the mask-of-interee (TOI). The resulting Pareto curves indicate that as token-selection
7

---

Published as a conference paper at ICLR 2025
Mean IOU w/ GT Seg. (IN-S) by thresholding eltn, weights keep Top X% mass.
$7.8
56.3
53.1
47.6
39.0
25.1
40%
50%
60%
70%
80%
90%
Figure 7: Visualizing Latent Token Attention Maps: The learned latent tokens effectively bind to distinct objects, suggesting potential for object discovery as future research. This contrasts with 2D tokenizers, where tokens are strongly biased predefined patches (also see Appendix Fig. 15).
Ground Truth Image
Reconstruction
Taken 5
Token 9
Token 17
Taken 24
Toker 31
Iteration 8 Iteration
Sharper Sparse Attention Locket Neck Floor
Eyes
Role of Recurrence on Latent Tokens: Credited to increasing number of tokens in each iteration, the recurrent update existing latent tokens makes them focus on sparser & more localized regions, leading to improved alignment w/ GT segmentation over iterations (see Tab. 6).
Figure 8:
criteria (TSC) increases in complexity, the representation capacity requirement also rises; for ex- ample, achieving depth error < 0.41 requires more tokens per image than depth error < 1.4. Moreover, optimal performance on task of interest occurs when tokens are selected using the same criteria the downstream task (i.e. TSC=TO1, Best Top-1 Accuracy is achieved when both TSC and TOI are Classífication (Fig. 4, left), compared to when TSC=Depth, TOI=Classificatin (Fig. 4, right)) and (b) selecting tokens based arbitrary thresholds for depth or reconstruction loss has small impact on classification accuracy i.c. classification accuracy with TSC = Depth Loss < 0.41 similar that with TSC Depth Loss 1.40, suggesting that classification requires fewer tokens (as supported by Linear Probing Experiments in Sec. 5). Thus, optimal tokens per image depends on both token-selection criteria & the desired task.
Next, we explore the scenario where the token-selection criteria remain constant, but the task of interest (TOI) varies. Since the TSC enables sampling different dataset representations, the optimal compression of datasel representations for maximum TOI performance depends on the alignment between the TSC and the TOI. other words, dataset representations are more compressible when optimized and tested on similar task. This also aligns with the idea that representational capac- ity is influenced by familiarity with the training data. For instance, Fig. 5, we allocate variable tokens per image using image reconstruction loss below a threshold as the token-selection criteria. The resulting rokestrrconstrucctoon are then evaluated across three tasks: classification accuracy using ResNet-18, depth estimation accuracy using DepthAnything and FID using VGG16 fea- tures. Approximately 60% of the total dataset representations (sampled using Reconstruction Loss as TSC) are sufficient to achieve near-optimal performance across all tasks- classfllassfficatiion depth estimation, and FID, highlighting that reconstruction loss could be a good self sele-uupee objective select tokens per-image. Appendix Fig. 14 and Fig. 13 analyze different Tasks of Interest (TOIs) performance using Depth Error Thresholding and Best Classification Accuracy as TSC respectively.
Representational Capacity Depends on Model Strength or Current Knowledge Fig. 6 exam- ines the relationship between downstream model strength and tokenized representations for tasks such as depth estimation, classification, and vision- languae modeling. Weaker models exhibit smaller performance drops with reduced token counts, shown by the color in Fig. 6, which repre-
8

---

Published as a conference paper at ICLR 2025
Appronch
32
64
96
ImageNet100
128
160
192
224
256
32" 64
COCO
128
256
32 /64
Wikipedia (WIT)
128
256
Titok-L-32
11.60
14.187 18.1
53.577
Titok-B-44
8.22
9.15
42,86
Tito-S-128
8.22
9.15
38.16
VQ-GAN
7.04
7.77
31.27
(INIK)
14.99
11.97
10.17
8.85
Ours-S (IN100)
16.17
9.30
8.55
22.28
14.22
61.77
38.45
Surs-SemilLarge (IN100)
19.70
13.92
11.39
10:41
9.23
8.75
8.22
8.03
Table 1: Reconstruction FID (4) on different evaluation datasets Our method performs com- parably to VQGAN and Titok, despite being amortized over multiple iterations, allowing flexible representations. Longer training on larger datasets, w/ deeper n/w could bridge the gap (see Fig. for scaling laws)'. The numbers below dataset titles are the variable token counts.
sents the perf. gap between GT and reconstructed images. For instance, fewer-token reconstructions have minimal impact on AlexNet (Classification) and the Blip model (vision-language modeling). Fig. 6 VLM experiments support concurrent works of Matryoshka VLMs (Hu et al., 2024; Cai et al,, 2024), further promoting recurrent and adaptive tokenization using VLMs as TSC and TOI.
FURTHER EXPERIMENTS & ABLATIONS
In this section, we present additional experiments to evaluate our adaptive and recursive represen- tations. We begin with image reconstruction experiments, followed by linear probing of learned representations for Top-1 classification. Next, we ablate the use of continuous vs. discrete tok- enizers for reconstruction task and finally analyze the learned tokens, showcasing alignment with GT segmentation (with tokens hinting at object discovery), and the role of recurrent processing in achieving that. For implementation details. please refer to the Appendix Sec. A.6.
Image Reconstruction: We compare our adaptive tokenizer with the fixed-length ID tokenizer Titok (Yu et al., 2024) and 21) tokenizer VQGAN (Esser et al., 2020) based on I.I reconstruction loss and FID scores on ImageNet100, COCO and Wikipedia Image-Tex Text (WTT) datasets 9. The re- construction loss metrics and the number of images at different sampled reconstruction loss thresh- amortize olds are shown in Fig. 2 (right plot). Unlike baselines with fixed-length representations, the learning of variable length representations using a smaller network Titok-L-32 uses 24-layer encoder/decoder to generate 32 tokens, while Ours-S uses the same 8-layers to learn variable (32 to 256) tokens. Despite that, as shown in Fig. 2. our method achieves slightly lower reconstruction loss compared baselines, maintaining significant details of the input image, while the baselines, optimtized for realism with GAN loss, achieve slightly better FID than ours for low-token recon- structions. Through the ablations shown in Fig. 9, we conjecture that multiple factors could enable further improving the slight FID loss (at low tokens) when training an amortized architecture for variable-length tokenization larger models via both network depth and feature dimension (plot & Fig. 17), training on larger dataset IN-1K vs IN-100 (plot 3 & row 5 vs row 6), longer training (plot 3), and most significantly GAN training (Sec. Training Procedure).
Linear Probing for ImageNet-1K Classification: We evaluate the learned adaptive representa- tions on the downstream task of classification by linear probing the latent representations, following standard practices (He et al., 2021). Specifically, we use the output of the latent-distillation encoder (mean pooling both the processed image tokens and 1D latent tokens). Our linear probe perfor- mance is on par with Titok models -oudels-Ours-S322 (after 1st iteration) achieves 49.9% Top-I 1 Accuracy, compared to Titok-S-32's 48.0%, and Ours-SemiLarge achieves 59.5% compared to Titok-L-32's 60.0%, despite layers). Two key observations: (a) Role of Recurrent Processing the single iteration output of Ours-L distillation encoder with 32 latent tokens achieves 55.6% Top-1 Accuracy, while recurrent processing through the encoder 2-3 times improves this to 59.5%. (b) Distilling 2D tokens into 64 to 128 1D tokens is sufficient for the coarse classification task; mean pool of first 64 to 128 latent tokens yield better linear- probbrn classification accuracy than pooling all (max. 256) latent tokens. Thus, the first few tokens capture most of the information required for classification, while additional tokens are crucial for reconstruction refinement. Thus, recurrent processing sharpens/localzze token attention, improving both reconstruction and classification.
model having much smaller network (SemiLarge 16 layers L 24
Unless mentioned, all experiments/ / visualizations were done using IN100 training of -ient-distillation modules w/ base VQGAN/VAE tokenizer trained on INIK. The results of the INK trained model are only in Tah. and Appendix Tab. 2)

---

Published as a conference paper at ICLR 2025
Enc// / Dec Depth Latent Dim
Multi- Saalti--tageeeiee Training
Larger Dataset, Longer Training
Cont, vs Discrete Tokenization
Frichet Inception Dist. (FID)
<en Uitcee to
2
Smat Suge mote roces Sall peuc
se Commee woa cwrattin
4
x
20
25
20
25
Number tokenss
Number Latens tokens
Number tokens
Number Latent tokens
Figure 9: (Plots 1-3) Ablating FID improvements w/ larger n/w, longer training, larger datasets, and multi-stage GAN training. (Plot 4) Ablating quantized vs continuous 2D and 1D tokenizers.
Continuous vs Discrete Tokenizer: We learn adaptive adaptive representations by distilling a discrete VQ- GAN tokenizer into quantized 1D latents. Fig. 9 (plot 4), Appendix Tab. 7 and Appendix Fig. 18 compare continuous vs discrete tokenizers at both 2D image-token and 1D latent-token hierarchies. Using continuous ID latents (i.e., no discrete quantization between distillation encoder/decoder) may improve FID more than a continuous 2D base tokenizer (both VAE and VQGAN with contin- uous ID representations perform well). However, quantized ID representations can achieve more compressed encodings (32 x 12 bits per image for (ize-size codebook) compared to continuous embeddings (32 x 12 x 32 bits). Additionally, quantization could potentially help distinguish OOD from IID test images (see Sec. 4, second point).
Analyzing Learned Tokens for Object Discovery: We analyze the attention maps of learned ID latent tokens 2D image tokens. Fig. shows attention maps corresponding four tokens from the 7th layer of Ours-S distillation-decodee Many latent tokens correspond to semantically meaningful objects or parts, suggesting emergence of object discovery. This contrasts with 2D tokenizers, where each token inductively maps to image patch, and only single class token attention heads (Caron et al., 2021) hold such semantic information. To further investigate this, following DINO (Caron et al., 2021), we computed attention map alignment with ImageNet-S GT segmentation by thresholding the top X% of attention maps as emergent seg maps. With 40% attention, we achieve 57.8 mean IOU, and by adjusting the threshold per image, we reach 71.8 mIOU. Notably, our tokens are not optimized for segmentation. Such high-alignment with segmentation is an emergent property.
Ablating Recurrence on Latent Tokens: We ablate the effect of recurrent updates on latent to- kens. As outlined in Sec. 3, with each iteration of recurrent and adaptive tokenization, previous tokens receive residual updates while new tokens are added. Fig. 8 illustrates how the attention maps of the first 32 tokens evolve after 8 iterations. Initially, under constrained memory/few token settings, these tokens attend broadly to reconstruct the image well, but as more memory/latent tokens are added, they focus on sparser, more meaningful regions. This sharpening, as seen in Fig, 8 token numbers 17, 24, and 31, attending to the eyes, locket, and floor regions, demonstrates the impact of recurrence (see Appendix Fig. 15 and Fig. 16 for more interesting and dense visualizations).
CONCLUSION
In this work, we propose a variable-length tokenizer for 2D images that operates by recurrently pro- cessing 2D image tokens, distilling them into ID latent tokens, and adaptively adding new ID tokens as computational resources for future iterations. This recurrent processing and adaptive computa- tion enable the learned latent tokens to correspond to semantically meaningful objects or parts in the input image. We demonstrate comparable performance on reconstruction metrics and ImageNet- IK linear-probing experiments. Finally, we utilize per-image learned adaptive representations (and cumulative dataset representations) to highlight alignment of the required image representational ca- pacity with - information entropy, familiarity with train set, knowledge of downstream tasks/models.
We believe that recurrently learning variable-length representations could open up intriguing directions, such as large-scale scale video representation learning or horizoo video under- standing, where simply learning fixed-length representations may not even be feasible solution. Other interesting avenues include exploring adaptive and recurrent nature of the proposed tokenizer as test-time thinking units and training for tasks like rision-languse & visual-abstract reasoning (e.g.: ARC (Chollet, 2019)). Variable-toker compression could speed up generative models.
Future Work:
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. CoRR, abs//800..m 10853, 2018. URL httpp:://ariiie org/ abs/ 10853.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE Transactions on Pattern Analysis and Machine Intelligence, 3798)11788-1888, 2013. doi: 10. 10.100//PAM..2013.5.0.
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Komblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flex- ivit: One model for all patch sizes. arXiv preprint arXiv:2212.080133 2022.
2022.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference Learning Repre- sentations, 2023.
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405. 17430, 2024.
Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the International Conference Computer Vision (ICCV), 2021.
Xinlei Chen, Saining Xie, and Kaiming He. empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.020577 2021.
François Chollet. On the measure of intelligence, 2019. URL htt.::ss///axii..rg//////////
1911.0154
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Univer- sal transformers. ArXiv, 0bbs/187.008899 2018. URL https: h/ap::://ap..../ http::/ap...mmmtttccooo.. semanticscholas. org/CorpusID: 49667762.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at
scale. CoRR, abs/2010. 1929, 2020. URL https: http//aavvvvv////// org/ abs 220000000.000 11929.
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis, 2020.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, 2024. URL https: fttps://arxivv org/ alss///b//aao 2dbs/23100 2310 02226.
Alex Graves. Adaptive computation time for recurrent neural networks. ArXiv, abs/1603.0899 2016. URL U.L.attppp/ppp...mmttt..ic
wrg//orprssrr.......oo :t224491.:::22221.... 8224916
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377, 2021.
David Herel and Tomas Mikolov. Thinking tokens for language modeling, 2024. URL https: //arxiv. httaaa///....///// 2405/00055....
Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng. and Kai-We Chang. Matryoshka query transformer for large vision-language models, 2024.
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- through estimator: Overcoming optimization challenges in vector quantized networks. 2023, URL https: /ttp///axvvv///////////e////aatl org/ htcp//bs/22000 08842.
Mareus Hutter. The hutter prize. http://prize.hutterl.mm 2006.

---

Published as a conference paper at ICLR 2025
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation, 2022.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation, 2023. URL https: /ttpvvawvvvv///////////..///.av. http:/wwwwww..aa.a.a/aaaaaa/a wwtpg////g/....a/r///g//g
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A general architecture for structured inputs & outputs. CoRR, abs/2107.14795, 2021a. URL
https: /aaiiv htttps:/aaxxvvoo////////////////
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention, 2021b. URL ptttps://arxiv..o abs/2103. 03206.
Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arnab, and Sujoy Paul. Mixture of nested experts: Adaptive processing of visual tokens, 2024.
URL httpttttp//// /arxivv.v.....///// www.iiiwrg/.....e..........o
org//bs// 2407. 19985.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https:
//wrxiiv.rgg/a//eee. /1312. 6114.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. In Advances in Neural Information Processing Systems, December 2022.
Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. CoRR, abs/0712.3329 2007. URL
org/ aas///b////// 0000001100011 3329.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Informa- fion Processing Systems (NeurIPS), 2021.
Elham Saraee, Mona Jalal, and Margrit Betke. Savoias: A diverse, multi-category visual complexity dataset. arXiv preprint (xXi::8/000077:: 2018
Jürgen Schmidhuber. Low-complexity art. Leonardo, 30(2):97-103, 1996. doi: 10.2307/157641
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn algorithm? generalizing from easy to hard problems with recurrent networks, 2021. URL https://///... http://xxivv../ abs////6..22/.////////////./e//. 04537.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Pro- ceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '21, pp. 2443-2449, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379, doi: 24114553404835.44622 URL
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere, 2022. URL https: http://axvvvv..///e org/ abs/2005..
10242.
Nicholas Westlake, Hongping Cai, and Peter Hall. Detecting people in artwork with cnns. CoRR, abs/1610.08871, 2016. URL http:// // aaaii.iii.. .rgg abaa 08871.
Fuzhao Xue, Valerii Likhosherstov, Anurag Amab, Neil Houlsby, Mostafa Dehghani, and Yang You. Adaptive computation with elastic input sequence, 2023. URL https: /arxivv org/ abs/ 2301 131 95.
12

---

Published as a conference paper at ICLR 2025
Wilson Yan, Matei Zaharia, Volodymyr Mnih, Pieter Abbeel, Aleksandra Faust, and Hao Liu. Elas- tictok: Adaptive tokenization for image and video. arXiv preprint, 2024.
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Moichanov. A-VīT:
Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQ- GAN. CoRR, abs/2110.04622 2021. URL
/atp:////aaavv/aa org/abs /aclbb//1100 2110. 04627.
Qihang Yu, Mark Weber, Xueqing Deng. Xiaohui Shen, Daniel Cremers, and Liang-Chies Chinh Chen. An image is worth 32 tokens for reconstruction and generation. arxiv: 2406.07550, 2024.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vggan to 100,000
with a utilization rate of 99 URL ht tettps:n : arxiv. org.org///aae abs 20../2406.1111 11837.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ACKNOWLEDGMENTS
This work was supported by the Defence Science and Technology Agency, Singapore, MIT/IBM Agreement No. W1771646, MIT/Hyundai Motor Company RD Center Agmt Did 2/22/23, NSF CIF 1955864 (Occlusion and Directional Resolution in Computational Imaging), NSF PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.or org/) and the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number 7A8750-19-2-100
We would like to thank our lab members - Ayush, Congyue, Ishaan, Jyo, Kabir, Manel, Prafull, Shamit, Tianwei, Tianyuan, Yichen for feedback and support & Qihang Yu for questions regarding Titok baseline.
A.2
LIST OF FIGURES
Fiigure 1] ALIT Architecture Figure.
[Figure 10] ALIT Overview
[Figure 2] Reconstruction Comparison on Imagenet100K.
[Figure Figure 11, Figure 12] Low Complexity Art Hypothesis.
[Figure 4]
Dataset Representation Capacity.
[Figure 5] Dataset Representation Capacity using (self-supervised) Automatic TSC.
[Figure 13] Dataset Representation Capacity using Oracle Classification as TSC.
[Figure 14] Dataset Representation Capacity using Oracle Depth Map as TSC.
[Figure 6] Representation Capacity vs Downstream Model Strength.
[Figure 7] Semantic Segmentation Emergence.
[Figure Figure 15, Figure 16] Token Visualization (notice the Object-Token Binding).
[Figure 9] Scaling Laws Ablation.
[Figure 17] Model Size Ablation.
[Figure 18] Discrete vs Continuous Tokenization Ablation.
[Figure 19] DINO ALIT Token Visualization.
[Figure 20] ALIT Internet Images.
[Fiuure 21] ALIT vs Matryoshka-style Encoder w/ ALIT Decoder.
[Figure 22] Role of Recurrence on Token Codebook Utilization.
A.3
LIST OF TABLES
Tabll 1] FID Comparison IN100 (and INIK) with VQGAN & Titok.
[Table 2] Reconstruction LI Loss Comparison on IN100 (and INIK) w/ VQGAN & Titok.
[aable 3] Dense FID-metric Table for Scaling Laws Figure
[Table 4] FID Comparison IN IN100 w/ Matryoshka Vlex recons. baselines.
(Table 5] FID Comparison OOD Coco and WIT w/ Matrysskk--sylle baseline.
[Table 6] Recurrence for token-object binding (unsupervised segmentation emergence).
[Table 7] FID Ablation on Continuous Discrete Tokenization.
[Table 8] Inference time comparison w/ fixed-length Titok.
Taable 9] Gfiops comparison w/ fixed-length Titok.
(Table 10] Ablation 1D Codebook sharing over recurrent timesteps.
Table 11] Ablation of ID Codebook size.
[ablle 12] Ablating dynamic halting of well-reconstrueted image tokens stage l).
14

---

Published as a conference paper at ICLR 2025
Recurrent Token Allocation
32 Token Reconstruction
Adaptive Length Imagge Tokenizer
GPT4
dog long.
There are additional vaa objects.
Depth Estimation
Reconstruction
GPT4
small
green
ginss neaityy
Reconstruction Depth
Depth
Image Dapth
This bathrom features wanity with light, wall, providing platform leading to shower area.
GPT4:
GPT4:
The bathropm has venity with countertop with sink and There's
The image shows bathroom with vanity festuring countertop, faucet dispanser, below: right glats-enclosed shower, and the room has tiled flooring. A wirdow near the shower.
GPT4:
cabinets, wall-nounted wirdow the right and glass-anclosed shower with a step leading in. The tiled floor completes the clean, minimal al design.
diawars
32 Reconstruction Dapth
Depth
Image Dapth
The image shows natural setting with tree, greanery. and dirt path or open area in the foreground, possibly in forest park.
GPT4:
image green a tree foreground with scottered branches.
green, GPT4: image muddy ground. bushy suggesting natural, forested setting.
dry, sandy
Figure 10: Adaptive Length Image Tokenization maps an image to multiple variable-length repre- sentations through a recurrent token allocation process, enabling task-specific sampling. We learn the tokenizer via image reconstruction tation can be optimized for specific tasks (e.g.. fewer tokens for "dog". "leaf", "grass" may suffice for VLM task), reconstruction objective supports learning universal, task-agnostic tokenizer.
self-supervised objective. While compressed represen-
A.4 ADDITIONAL EXPERIMENTAL RESULTS
Low-Con www-Comleexity Art Hypothesis or Representation Capacity ~ Image Entropy Here, we pro- vide additional experimental analysis supporting representation capacity alignment with image en- tropy claim made in the main paper. Sec. 4. We extracted for the MSCOCO and Places2 subsets of the SAVOIAS dataset (Saraee et al., 2018). We leverage model trained on ImageNet-100 to study this hypothesis, though both COCO and Places may be slightly out-of-distribution (OOD) relative to the learned codebook samples. The complexity- reconstruction loss alignment plots in Fig. 12 and Fig. 11 reaffirm the analysis in the paper- increase in image complexity demands more tokens, and more tokens lead to smaller recon. loss.
annotated image complexity scores
Representational capacity depends downstream tasks: In addition Fig. 5 in the main pa- per, where we evaluated Reconstruction Loss Thresholding as a Token Selection Criterion (TSC), GT-oracle-based Classification and Seeudo-GT-oracle-baaee
Depth Error metrics as TSC through Fig. 13 and Fig. 14. For Top-X Classification as TSC, an image receives 32 latent tokens if ResNet- 18's top-X predictions from the 32-token reconstruction include the ground truth class. Key Takeaways Compared to depth, reconstruction loss, and FID as the tasks of interest (TOI), the decrease in classification accuracy with a reduction in dataset representation capacity (cu- mulative tokens per dataset) is relatively smaller due to classification being a coarser task. When TSC & TOI, using different TSCs to sample the same fraction of tokens yields similar TOI perfor- mance. For example, FID and Reconstruction Loss at 40% of max token capacity comparable (compare 2nd 3rd plots of Fig. 13 and Fig. 14), regardless of whether depth or classification was the token-selectios criterion. However, compared to Top-X classification, choosing the granular tasks of depth and reconstruction as TSC provide finer support for token sampling therefore better/ diverse performance range for the TOL. Reconstruction loss serves self-supervised TSC.
further
15

---

Published as a conference paper at ICLR 2025
Comparison with Flexible Embedding Baselines: At the time of submission, Matryoshka (Kusupati et al., 2022) and Flex basxxeT- (Beyer et al., 2022) approaches had only been explored for classification. The application of Matryoshka to reconstruction work (Yan et al., 2024). We implemented Flex for reconstruction by applying variable patch- ing on VQGAN 21 image tokens and designed Marryoshka-style model by selecting variable subset (either 32, 64, 128, or the full 256) of distilled ID tokens during training with no recurrence. Tab. compares ALIT with both approaches-
concurrent ICLR 2025
experiments indicate that both Matryoshka- and Flex VIT-based methods required larger parameter counts to match ALIT's performance. Fig. 21 visually compares the tokens learned by ALIT and the Matryoshka-bae bbased approach. Thanks to its recurrent nature, ALIT refines tokens over multiple iterations, leading to more specialized token bindings. Tab. compares Matryoshka-style model with ALIT on OOD Coco and WIT datasets.
Finally, some more comparison between recurrent (ALIT) and masking-based (Matryoshka) ap- proaches: (a) Compared to the Matryoshka approach, an ALIT model with the same number of parameters utilizes more FLOPs at test time to learn longer representations due to its recurrent na- ture. (b) In Matryoshka and other masking-basee
approaches, the 256-token representation always includes the same first 32 tokens as the 32-token representation, which may not be optimal, as higher token budget could allow for a more efficient utilization of those 32 tokens. (c) Combining ALIT with both Matryoshka and VIT approaches could yield highly efficient tokenizer by leveraging the benefits of flexible 2D patches, adaptability at each recurrent step, and the ability to allocate more FLOPs through recurrence.
Additional Ablations: Tab. 3 and Tab. 7 specify the exact metrics used to generate the plots in Fig. 9. Both the plots and the tables elearly indicate that longer training, larger datasets (IN-100 vs. IN-1K), and multi-stage training are vital for achieving photo-realistic reconstructions. Furthermore, Fig. 17 qualitatively highlights the benefits of using larger models (ALIT-L vs ALIT-B vs ALIT- S) on image reconstruction quality. Fig. 18 visually analyses the use of continuous vs discrete tokenization at both 2D and ID token levels. Continuous ID Tokenization performs best on image- reconstruction task. That said, studying the benefits of quantization for retrieval and long-horizon tasks like visual abstract reasoning, video question answering would be useful. Furthermore, Tab. 10 and Tab. 11 ablate the role of ID latent token codebook sharing and codebook size on reconstruction FID. Tab. 12 ablates dynamic halting procedure (see Fig. 1 or method section to understand where dynamic halting is performed). Finally, Tab. and Tab, describe the computational needs of ALIT in terms of inference time and gtlops.
A.5 ANALYSING THE LEARNED LATENT TOKENS
Emergent Object Binding: Compared to 2D tokenizers, where each token is bound to patch (other than class or global tokens), the learned ID tokens demonstrate signs of object binding and localization. In addition to Fig. 7 and Fig. 8, we showcase more examples of how the learned latent tokens bind to semantically meaningful objects on COCO dataset. For instance, in Fig. 15, note the token-zebra binding in the second example, the near-perfect segmentation of the bicycle symbol on the lane in row 3, the segmentation of the frisbee and human into different tokens in example 4, and the clustering of both plants (as indicated by high attention weights) in the final example.
Comparison with DINO for Emergent Object Binding: Fig. 19 visualizes the tokens learned by DINO and ALIT. Despite being trained with different objectives on different datasets, both rep- resentation learning systems exhibit intriguing emergent object discovery behavior. For DINO, the number of token-attention map visualizations is fixed and determined by the number of attention heads (=6 for DINO-S) multiplied by the number of class tokens (=1). Additionally, since DINO is trained with a self-distillation regime that matches intra-image crop representations, it is primarily optimized for classification. As a result, the class-token attention heads tend to focus regions crucial for image classification (e.g.. in Fig. 19, row 4, the tokens attend either to the human or the phone, aiding in "selfie" classification). In contrast, ALIT employs variable number of tokens, leading to a flexible number of atten-tttention maps and enabling the discovery of multiple ob- jects-notice, for instance, the switch segment detected on the background wall (row 4 of Fig. 19). We believe this strong emergent behavior arises from ALIT's reconstruuction-aase training objec- tive, the discrete nature of its learned representation. and the recurrent refinement of its tokens.
Recurrent Processing of Latent Tokens: We further explore the role of recurrence in enhancing token attention and binding to semantic objects and parts. With each iteration of the model rollout, new tokens are added, allowing existing tokens to focus on specialized tasks rather than covering
16

---

Published as a conference paper at ICLR 2025
-Number Latent Tokens (32, 64, 128, 192, 256]
Graund Truth
Low Complexity Art Hypothesis
Human Predicted Image Complexity
256
Reconstruction Loss
0.8
Number of Tokens
192
160
0.6
128
96
64
32
0.0
15
25
45
55
65
75
85
Image Complexity Human Score
Figure 1I: Compression vs. Entropy Hypothesis on the Places Dataset: Adaptive tokenization enables analysis of Low-Coooooo now-Complexity Art Hypothesis by examining token requirements for images of varying complexity. The plot on the right clearly shows that as image complexity increases, so does the need for more computational tokens. These reconstructions
Imagenet1000 100-rrined model.
Number of Latent Tokens (32, 64, 128, 192, 256) - Ground Truth
Low Complexity Art Hypothesis
Human Prodicted Image Complexity
1.0
256
224
0.8
Number Tokens
Reconsion Derruction
0.6
96
aod C C
64
0.2
32
0.0
Image Complexity
Figure 12: Compression vs. Information Entropy Hypothesis on the MSCOCO Objects
Dataset: Adaptive tokenization enables analysis of Low-Complexity Art Hypothesis by examin- ing token requirements for images varying complexity. The plot on the right clearly shows that image complexity increases, does the need for more computational tokens.
the entire image. Figure 16 illustrates five examples from the COCO dataset, showing how the attention maps selected tokens receive residual updates over eight recurrent iterations, leading to improved object binding. To quantitatively support this enhanced localization, segmentation metrics at different recurrent iterations in Tab. 6. These metrics reinforce the visual results, with mIOU consistently increasing across iterations at various attention masking percentiles.
calculate mIOU
Analyzing the Role of Recurrence in 1D Codebook Utilization: We visualize the frequency with which each code in the learned codebook is sampled across 5K validation images in Fig. 22. As shown in the figure, later iterations, more tokens exhibit higher sampling frequencies (x-axis: token sampling frequency/count, y-axis: number of tokens at that frequency). This suggests that with more recurrent iterations, certain specialized tokens are sampled more frequently, reinforcing their role in capturing emergent object representations. This aligns with the token-object binding visualizations presented throughout the paper.
A.6 EXPERIMENTAL DETAILS
Training Details In this section, we provide additional training details (see Sec. 3 for approach details and training procedure summary). We train ALIT in two phases catten-distillatiinn pre- training and full fine-tuming stage (with gan loss). In the -istt-distillation pre-training stage, we leverage a pre-trained image tokenizer (VQGAN or VAE) which maps an input image to 2D tokens. We only train the latent-distillation encoder-decoder modules in this stage, using image token reconstruction loss as the core-learning objective. With VQGAN as base tokenzier, we use cross-entropy loss comparing predicted logits with the ground-truth VQGAN-codebook index at each 2D token position. We use mean-squre reconstruction loss when using VAE as the base- tokenizer. We unroll the recurrent token allocation procedure for iterations. expanding token memory from 32 (in 1s iteration) to 256 (in 8th) during training. All the recurrent rollouts are trained end-to-end. At each iteration, we process the image-tokens, the existing 1D latent tokens
17

---

Published as a conference paper at ICLR 2025
Min Tokenss lar %% max tokens)) for Dest Top-Tp-- T Clasificcato Accuracy
1.0--
122
1.8-
0.32
Roconstrion LUCT Loss
1.72
Rall Depth Error
00202
Fréccher incept.co
031
033
0.289
35,09
63
0
X Top-X Classification
For Top-X Classfication
X for Top-X Clessifization
Figure 13: Dataset Representation Analysis factor of TSC-TOI Alignment: We vary tokens per image using "Best Top-X Classification" Accuracy as the Token Selection Criteria (TSC) and evaluate different Tasks of Interest (TOI) on the reconstructions. We sample tokens per image such that each reconstructed image is classified correctly under Top-X Classification criteria, and then evaluate different downstream tasks of interests (TOIs) on the reconstructed dataset.
Min Tokenns las of max max tokend)
for Dapah Enor TTrechold
10
32,492
flcattiiionnnn Acc. 176
Reconntruu. Loss
1.52%
101
Fréchet Incept. diet
03
1.000
0.
0.
21.111
7700
3.00
0.41
5.00
CAL 133
3.00 042
050 135 3:40
Abs. Ral Depth Enror Thrasholds
Abs. Rel Dapth Error Thresholds
Abs. Ral Depth Error Thresholds
Figure 14: Dataset Representation a factor of TSC-TOI Alignment: We vary tokens per image using "Depth Loss < some Threshold" as the Token Selection Criteria (TSC) and evaluate different Tasks of Interest (TOI) on the reconstructions. Being a dense task, depth error thresholding at several continuous thresholds provide more granular and diverse support to dataset representation compression and TOI performance, compared to Top-X Classification (X being discrete integer).
and add new latent tokens. During this training phase, we perform dynamic halting of the image tokens in each iteration, allowing the latent-distlllation modules to focus on distilling image tokens which cannot be reconstructed perfectly till current iterations. We use transformer backbones for both latent-distillation encoder and decoder, performing self-attention among 2D image tokens and latent ID tokens. In the next training phase, jointly fine-tune both the base image tokenizer modules and the antent-distillation encoder- decoder modules with losses directly at the pixel-space. The training objectives are pixel-level reconstruction and adversarial losses (gan generator and dis- criminator losses) inspired from VQGAN (Esser et al., 2020) training procedure. We optimize for reconstruction loss for first few epochs, later switching both reconstruction and adversarial losses. We recurrently map an image to variable-tokens (increasing token count by 32 in each iteration) for 8 iterations. However, unlike previous stage, we only compute loss at only iteration, thus we only need to perform recurrent processing of the latent-distillation encoder in each training run, execut- ing the iatent-distilaate the training and the compute memory requirements. No dynamic halting is performed in this phase. Thanks to the gan losses, this phase help boost the photo-realism FID metric of the reconstruc- tions, specially at lower-token counts. For more low-level details, please refer to our codebase https: / //ggtttuub com/ Shi wniiammDuggaa 4/ aadapaateeeeeeeee
decoder and the base tokenizer decoder only once. This helps speed up
Dataset For training the adaptive tokenizer, we mainly utilize Imagenet- (100 classes of ImageNet- 1K used in ( (Wang & Isola, 2022)) and ImageNet-Iee 1K datasets. Fig. 9 showcase the scaling potential of training on larger datasets. For exact details of the 100 classes, please refer to our codebase. Following are pointers to image sources used to create the paper figures and plots-The three images shown in Fig. 10 are from ImageNet- 100 valiation set (dog image), OOD indoor scene image is from NYUV2 dataset (Nathan Silberman & Fergus, 2012) and tree example is from WIT dataset (Srinivasan et al., 2021). Fig. 3. Fig. 11, Fig. 12 all leverage data from SAVOIAS dataset, which again is different from ImageNet- 100 classes. Plots in Fig. 4, Fig. 5, Fig. 14, Fig. 13 all leverage ImageNet- 100 validation set. Fig. 7 and Fig. 8 use ImageNet images for token visu- alzation, Fig. 15 and Fig. 16) use COCO images. Fig. 17 and Fig. 18 showcase randomly sampled images from ImageNet- 100 validation set.
18

---

Published as a conference paper at ICLR 2025
ImageNet100
Approach
64
128
160
192
224
256
Titok-L-32
12.29
Titok-B-64
9.72
Titok-S-128
9.72
VQ-GAN
7.99
Ours-S (INIK)
11,50
9.81
8.78
8,36
8.01
7.65
7,53
7.48
Ours-S (00)
11.77
10.13
9.26
8.77
8.32
7.95
7.81
7.62
Ours-Semil (IN)00)
11.81
10.09
9.16
8.62
8.36
8.01
7.79
7.63
Table Reconstruction L1 Loss (x 100 4) evaluated on IN100 validation set. For last two rows,
our latent-distillation modules are only trained on IN100, rest everything is trained on INIK.
Train Dataset
Train
Train Epochs
Enc// Dec.
Embed. Dim.
Number of Tookens
Stage
Depth
32
64
96
128
160
192
224
256
IN-100
Stage1
200
small (8)
512
71.53
49.18
35.11
26.97
20.31
15.38
10.84
8.77
IN 1000
Stage1
200
base (12)
768
49.88
35.16
25.79
18.29
14.52
11.27
8.77
7.74
IN-100
Stagel
200
semi-large (16)
1024
43.97
27.56
19.94
16.22
13.48
11.19
7.64
IN-100
Stagel
200
small (8)
1024
53.47
35,66
27.51
21.18
16,66
12.94
9.66
7.29
IN-100
Stage2
200
small (8)
1024
22.57
16.17
13,30
11.69
10.22
9.30
8.25
10-100
Stage?
200
base (12)
768
22.15
15.88
12.57
10.44
9.50
8.96
8.39
8.19
Stagel
200
21.55
16.30
13.45
11.33
IN-IK
Stagel
400
small (8)
1024
41 46
28.26
20.90
16.21
13.21
10.74
8.68
7.39
IN-IK
Stagel
semni-lrge (16)
30.84
19.56
12.87
11.90
Table 3: Image Reconstruction Ablations (FID metric): The ablations clearly highlights longer
training, larger dataset, larger network depth helps in improving FID. GAN base-training is critical.
ImageNer100
Approach
128
160
192
224
256
Matryoshka Enc-S w/ ALIT Dec-S
33.11
20.08
15.10
12.84
11.45
10.51
9.90
9.50
Matryoshka Enc-SemiLarge w/ ALIT Dec-S*
32.12
19.25
14.52
12.48
11.23
10.11
9.48
9.14
Matryoshka Ene-SemiLarge w/ ALIT Dec-SemiLarge
22.85
14.51
11.63
10.13
9.23
8.83
8.42
8.11
FIexVIT Enc-S w/ ALIT Dec-S
79.84
39.73
15.16
12.64
FIeXVIT Ene-Semil.ane w/ ALIT Dec-S
25.62
15.77
10.21
7.90
VIT Ene-SemiLarge w/ ALIT Dee-SemiLarge
23.30
14.73
9.92
7.80
Ours-S Ours-SemiLaage
22.57
16.17
13.30
11.69
10.22
9.30
8.55
8.25
19,70
13.92
11.39
10.41
9.23
8.75
8.22
8.03
Table 4: Reconstruction FID (L) comparison on ImageNet100: Matryoshka and VIT were originally proposed for classification tasks. We adapt them for dense reconstruction by employing ALIT-style ID token decoding. We believe that combining Matryoshka-stte style masking and Flex VIT- style variable 2D patch-size approaches with the ALIT-style recurrent method could yield further benefits. (# indicates that the GAN discriminator runs 4 times more than the generator).
COCO
Wikipedia (WIT)
Approach
64 Tokens
128 Tokens
256 Tokens
64 Tokens
128 Tokens 256 Tokens
Matryoshka-I.
23.05
14.43
10.66
62.02
48.21
40.28
Ours-S
22.28
14.22
9.72
61.77
47.91
38.45
Ours-L
21.44
13.64
9.71
60.11
47.52
38.69
Table 5: Reconstruction FID (L) COCO and Wikipedia Image-Text (WIT). All models
trained on Imagenet100. Our method (Ours-S, Ours-L) outperforms Matryoshka-L on the OOD datasets. Here "I"" "Semi-Large".
Attention Threshold.
Recurrent Iteration
Percentile
3
5
6
7
8
40%
56.7
57.4
57.5
57.6
57.6
57.7
57.7
57.8
60%
51.5
52.4
52.5
52.7
52.8
52.9
53.0
53.1
80%
37.1
38.6
38.4
38.6
38.7
38.8
39.0
39.0
Table 6: Recurrent Processing enhances Alignment of Token Attn Maps with GT Segments. Note we did not optimize for segmentation, and token attention maps binding to segments is an emergent property. This binding improves with recurrent processing (under increasing memory constraint). To extract segment from a token altn map, we threshold the top X% of the attn map.
19

---

Published as a conference paper at ICLR 2025
Base 2D
Latent Dimension
Latent
Number of Tokens
Tokenizer
Quantization
32
64
96
128
160
192
224
256
VAE
12
27.90
19.62
16.07
13.95
12.50
11.16
12.62
9.95
VAE
12
22.14
14.15
10.88
9.00
7.68
6.76
6.22
5.66
VQGAN
12
/
22.57
16.17
13.30
11.69
10.22
9.30
8.55
8.25
VOGAN
12
21.77
10.39
8.02
7.38
7.30
7.44
7.36
7.14
Table Continuous vs. Discrete Tokenization: Continuous ID latents (without quantization) are
advantageous for image reconstruction, while quantization provides the benefit of compressed en- coding. All token-object binding visualizations are performed using the VQGAN latent quantiza- tion approach, showcasing the advantages of quantization beyond compression and error bounding.
Number of Iterations
Approach
Iter
Iter 2
Iter
Iter
Iter
Iter 6
Iter 7
Iter 8
Titok-L-32
9.54
Ours-S
6.84
10.40
14.13
18.04
22.03
25.99
30.20
34.74
single H100 GPU with FP32 pre- cision for single-image encoding. The overhead of running an additional iteration of the adaptive tokenizer encoder is actually quite small: 4ms run time and < +30 gflops on a single H100 gpu. Overall, +30 gflops overhead per iteration is very less for modern compute and can be further re- duced by using techniques like KV caching previous layer attentions etc.
Table 8: Iteration-wise Inference Time (ms) Comparison
Number of Iterations
Approach
Iter
Iter 2
Iter 3
Iter 4
Iter 5
Iter 6
Iter 7
Iter 8
Titok-L-32
58.44
Ours-S
83,88
105.45
129.16
155.03
183.04
213.21
245.53
278.00
Table 9: GFlops Comparison on a single H100 GPU for single-image encoding. For 32 tokens, our latent-distillatioo encoder has 8 transformer layers much smaller than Titok-L-32 (with same number of attention heads, feature dim and same number of tokens). Despite that, we have more GFlops because of using VQGAN image encoder small additional patch embedding layer the image 2D embedding. We can switch to Titok-L-32 style image encoding and reduce upto 50 gflops.
Number of tokens
Codebook Size
Codebook Sharing
32
64
96
128
160
192
224
256
4096
X
26.84
18.71
14.71
12.82
10.91
10.02
9.05
8.61
4096
22.57
16.17
13,30
11.69
10.22
9.30
8,55
8.25
Table 10: Efffec of Codebook Sharing on Reconstruction FID (L). Shared codebook leads to (a) more compression in terms of total codes. (b) theoretically allows one to rollout the adaptive tokenizer for much longer rollouts than done during training by having iinestep-invariatt codebook.
Number of tokens
Codebook Size
128
160
224
256
2048
26.65
18.91
14.76
12.20
10.86
10.03
9.49
9.13
4096
22.57
16.17
13,30
11.69
10.22
9,30
8.55
8.25
8196
23.76
16.55
13.11
11.14
9.75
9.17
8.61
8.42
Table 11: Reconstruction FID vs 1D Codebook Size There is not much (marginal) difference in
performance between 4096 and 8196 codebook size, stating that the tokenizer is less sensitive the codebook size. That said, when scaling to large models, scaling up the codebook while maintaining utilization would be intruiging research question. 4096 is usual codebook size for Imagenet.
Number of tokens
Dynamic Halting
32
96
128
160
192
224
256
X
24.38
17.00
13.49
11.53
10.30
9.43
8.76
8.29
22.57
16.17
13,30
11.69
10.22
9.30
8.55
8.25
Table 12: Ablating Dynamic Halting for Reconstruction FID metric. Dynamic Halting is done only in stage of training (and is optional).
20

---

Published as a conference paper at ICLR 2025
Ground Truth Image
First 32 Tokens
Enlarged Token Attentions for Better Visualization
Iteration 8 Iteration
Ber ls
lteration 8 Iteration
Iteration 8 heration
Iteration 8 Iteration
Iteration Iteration
Figure 15: Recurrent processing of Latent Tokens on COCO Validation Set: Recurrent process- ing with increasing computational tokens leads to specialization of attention. We showcase all the first 32 tokens after the 1st & 8th iterations (out of total 256 tokens at the 8t iteration). (Left) A broader comparison b/w iteration I vs 8 attention maps highlight emerging sparsity in attention.
21

---

Published as a conference paper at ICLR 2025
Image
tteration
Iteration
Iteration
teeatioo
Iterationn 5
heration
Iteration
Ieeration 8
Figure 16: Token Visualization over Multiple Recurrent Iteration: Recurrent processing leads to sparse attention & specialization of tokens localised objects, parts etc. For example- recurrence led to near-perfect bed segmentation (row 1), bicycle-sign segmentation (row 2), improved human segmentation (row 3), zebra rebra-instance segmentation (row 4), and potted -lannts segmentation (row 5).
ALIT-S
ALIT-B
ALIT-L
Ground Truth
ALIT-S
ALIT-B
ALIT-L
Ground Truth
128 Tokens 32 Tokens
256 Tokens
ALIT-S
ALIT-B
ALIT-L
Ground Truth
ALIT-S
ALIT-B
ALIT-L
Ground Truth
256 Tokens 128 Tokens 32 Tokens
Figure 17: Effect Model Size on Image Reconstruction Quality. Increasing model size en- hances reconstruction quality. This visualization uses VQGAN as the base tokenizer (discrete 2D) distilled into discrete ID latent tokens. With 256 tokens, the reconstruction quality closely aligns with VQGAN results and approaches ground truth, even for complex images, supporting the use of adaptive representations. (Zoom in for details.)
22

---

Published as a conference paper at ICLR 2025
Quant-2D Quant-1D
Quant-2D Cont-1D
Cont-2D Cont-1D
Ground Truth
Quant-2D Quant-1D
Quant-2D Cont-1D
Cont-2D Cont-1D
Ground Truth
256 Tokens 128 Tokens
Quant-2D Quant-1D
Quant-2D Cont-1D
Cont-2D Cont-1D
Ground Truth
Quant-2D Quant-1D
Quant-2D Cont-1D
Cont-2D Cont-1D
Ground Truth
64 Tokens
16
128 Tokens
256 Tokens
Quant-2D Quant-1D
Quant-2D Cont-1D
Cont-2D Cont-1D
Ground Truth
Quant-2D Quant-1D
Quant-2D Cont-1D
Cont-2D Cont-1D
Ground Truth
256 Tokens 128 Tokens 64 Tokens
Figure 18: Impact of Discrete vs. Continuous Tokenization Image Reconstruction. Contin- uous ID latent representations yield the best reconstructions overall. The ad advantage of using VAE as the base tokenizer (third column of Cont-2D Cont- ID) becomes more significant higher to- ken counts. VQGAN (discrete 2D) combined with continuous ID (second column) also performs well, especially at lower token counts, possibly due to the learning ease the discrete cross-entropy training framework. Note: ALIT-S is used in this visualization, and reconstructions may improve with a larger model. (Zoom in for details.)
23

---

Published as a conference paper at ICLR 2025
DINO-S visualizing all CLS token attn maps (number of heads 6)
ALIT-S visualizing 6 selected attn maps
ALIT-S visualizing all first 32 attn maps
Input Image
Figure 19: DINO vs ALIT Token Visualization: Compared to DINO-S, which has one class to- ken and 6 attention-heads per token, leading to 6 token-attention maps, have arbitrary attention maps (= number of latent tokens). Dino being trained with self-distillation implicitly has some bias towards attending patches which lead to better classification, like "attending to phone and man for implicit selfie representation" ", while ALIT being trained with reconstruction even attends to the small switch on the wall the background behind the man. Note: we are not an alternative to DINO, both approaches have orthogonal contributions.
24

---

Published as a conference paper at ICLR 2025
Input Image (from Internet)
64 Token Reconstruction
128 Token Reconstruction
256 Token Reconstruction
Figure 20: Generalization of ALIT trained on Imagenet100 on Internet Images: ALIT performs quite well even on OOD imagesmost times, with great potential of being an "any-image" generaliz- able adaptive tokenizer with some scaling.
25

---

Published as a conference paper at ICLR 2025
Four Interesting Tokens (Manually Sampled)
All First 32 Tokens
Matryoshka -tayye style
(Large) Encoder w/
ALIT-style Decoder
ALIT-Smale (Ours)
Matryoshka-styee style (Large) Encoder w/
ALIT-style Decoder
ALIl-Small
(Ours)
Comparing ALIT adaptive tokenizer vs Matryoshka-Style tokenizer: Thanks to recurrent processing in presence of additonal compute (a.k.a tokens), ALIT learns more sharper attention maps over multiple iterations. Unlike ALIT where the first 32 tokens imporve over iter- ations, Matyyoshka-style approach learn a fixed-length (static) embedding is ALIT-style decoding with Matryoshka-style encoding. One could also combine Matryoshka- Style encoding with recurrent encoding, which would be interesting future work.
Figure 21:
one go. Note: This
26

---

Published as a conference paper at ICLR 2025
Iteration
Iteration
Iteration (Cum. Tokens 5K)
(Cum. Tokons 32 5K)
(Cum. Tokens 64 * 5K)
Number Codes tomplng count
with that sampp lng count
Namber of Costes
of
60
300
Coce Sampling Count
Code Sampling Count
Code Sampling Count
Iteration
Iteration (Cum. Takens 160 5K)
Iteration 6
(Cum. Takens 128 5K)
(Cum. Tokens 192 5K)
with that samping count
Nunteral oo Cooess smmping
Numtor of Couka
ampling
Numberaf
20 De Sampling
Sampling
Sampling
Iteration 7
Iteration 8
(Cum. Tokens 224 5K)
(Cum. Tokens 256 5K)
Number of Cortes with that sampling coure
Numbe of Goodee with thet sampoiin coant
Coco Sampling Court
Code Sampling Count
(How mony times is code sampled from codebook see a0vo)
(How many time is code is sampled from cocebook ure 4096)
Figure 22: Codebook Utilization at different recurrent iterations by ALIT-Small: As mentioned
before, recurrent processing with adaptive memory (tokens), leads to token specialization. This can also be seen from the fact larger iterations, there When you have more memory tokens, some of the memory can be utilized for specialization. Note almost no code is sampled times, that's just artifact of low x-axis tick sampling. "*5K" cumulative tokens from 5K Imagenet 100 validation images.
number of higher frequency tokens.
27


Reviews:
[{"review_id": "QjKn1jEjQQ", "reviewer": "Reviewer_rXC5", "summary": "This paper addresses a fundamental limitation in current visual representation learning systems: their use of fixed-length representations regardless of image complexity. The authors introduce an Adaptive Visual Tokenizer (AVT) that can generate variable-length representations (ranging from 32 to 256 tokens) through a recursive processing mechanism.\n\nThe key innovation is a recurrent encoder-decoder architecture that progressively distills 2D image tokens into 1D latent tokens over multiple iterations. During each iteration, the system not only refines existing tokens but also introduces new ones, allowing for adaptive representation capacity. This approach enables tokens to specialize and focus on specific image regions, leading to emergent object discovery without explicit supervision.\n\nThe authors provide extensive empirical evidence demonstrating that optimal token counts naturally vary based on several factors: image complexity/entropy, familiarity with the training distribution, downstream task requirements, and the strength of models using these representations. Through experiments on image reconstruction, classification, and depth estimation tasks, they show that their adaptive approach achieves comparable performance to fixed-length tokenizers while offering the flexibility to use fewer tokens when appropriate.\n\nBeyond the technical contribution, the paper makes a broader conceptual contribution by demonstrating how variable-length representations align with human visual processing and information theory principles. The work suggests promising directions for more efficient visual representation learning, particularly for applications like video understanding where fixed-length representations may be insufficient.\n\n-----------\n\nUpdate: Added response and modify score to 6, presentation to 2.", "questions": "1. **Computational Efficiecy and scaling**  \n- Could you provide a detailed analysis of inference time and memory requirements compared to single-pass approaches?\n- How does the recurrent processing overhead scale with a number of iterations?\n- Have you explored early stopping strategies to reduce computation for \"simple\" images?\n- What are the key bottlenecks preventing training on total ImageNet-1K?\n\n2.**Token allocation optimality**  \n- Given task constraints, Is there a theoretical framework for determining the optimal number of tokens?\n- How do you ensure the progressive token addition across iterations efficiently utilizes capacity?\n- Have you explored methods to predict the required token count before processing, similar to adaptive computation time in transformers?\n- Could you provide an empirical analysis showing token utilization across iterations (e.g., are later tokens less \"important\")?\n\n3. **Comparison with specialized architectures**:\n- How does the method compare with recent adaptive patch-size approaches (like FlexViT) on standard benchmarks?\n- Could you provide direct comparisons with self-supervised methods like DINO on standard metrics for object discovery capabilities?\n- Have you explored combining your approach with hierarchical architectures that naturally handle multi-scale features?\n\n4.  **Codebook and quantization design**  \n- What motivated the choice of shared codebook across iterations versus hierarchical/specialized codebooks?\n- How sensitive is the method to codebook size and quantization strategy?\n- Could you analyze codebook utilization across different iterations and image complexities?\n- Have you explored using learned or adaptive quantization strategies?\n\nThese clarifications would help comprehensively assess the method's practical applicability and theoretical foundations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Originality**  \n- Novel integration of recurrent processing with visual tokenization, extending beyond fixed architectures like VQGAN/Perceiver\n- adapting dynamic computation (from NLP/sequential domains) to visual representation learning\n- Fresh perspective on image compression through variable-length tokens, contrasting with traditional fixed-length approaches\n- Original empirical framework for analyzing token requirements across complexity, familiarity, and model capacity\n\n**Quality**  \nThorough empirical validation across multiple dimensions:\n- Reconstruction quality (L1, FID) is competitive with specialized fixed-length tokenizers\n- Linear probing results comparable to larger models (49.9% Top-1 vs Titok's 48.0% with smaller architecture)\n- Systematic ablations of architecture choices (continuous vs discrete, network depth, training strategies)\n- Strong technical foundation combining quantized latent spaces, recurrent processing, and adaptive computation\n- Rigorous analysis of token specialization through attention map visualization and segmentation alignment (57.8 mIOU without explicit supervision)\n\n**Clarity**  \n- Clear architectural progression from base distillation to full adaptive system\n- Well-structured empirical validation of key hypotheses about token requirements\n- Effective use of visualizations to demonstrate token specialization and attention patterns\n- Systematic organization of ablation studies exploring key design choices\n\n**significance**:  \n- Demonstrates feasibility of variable-length visual representations while maintaining performance\n- Introduces techniques potentially valuable for resource-constrained deployment scenarios\n\nThe paper's main strength lies in its systematic challenge to the fixed-length paradigm while maintaining competitive performance, backed by comprehensive empirical analysis.", "weaknesses": "The paper's primary weakness lies in its validation strategy and scaling limitations. While the approach shows promise, the core experiments are limited to ImageNet-100 (rather than full ImageNet-1K) for the adaptive tokenization training, making it difficult to assess the method's capabilities at scale fully. The authors acknowledge this limitation in Table 1's footnote, but don't sufficiently explore whether the performance gap with fixed tokenizers (particularly in FID scores for low token counts) is fundamental to the approach or simply due to training scale.\n\nA second significant weakness is the lack of thorough comparison with specialized architectures. While the paper compares with fixed-length tokenizers (VQGAN, Titok), it omits comparisons with other adaptive approaches like FlexViT (variable patch sizes) or hierarchical architectures that naturally handle multi-scale features. It isn't easy to assess whether the benefits come from the variable-length representations or could be achieved through simpler adaptive mechanisms.\n\nA third weakness is the lack of thorough investigation into computational overhead. The approach's recurrent nature requires multiple passes through the encoder-decoder architecture to generate variable-length representations. While Figure 9 includes some ablations on network depth and training, there's no detailed analysis of inference time complexity or memory requirements compared to single-pass approaches like VQGAN or Titok. This is particularly important given the paper's motivation of efficient representation learning.\n\nFinally, while the adaptive token allocation based on complexity is well-motivated, the paper lacks a formal framework for analyzing the optimality of these allocations. The empirical correlations with human-labeled complexity scores are interesting but don't provide theoretical insights into whether the model is making optimal token allocation decisions. Given task constraints, including some theoretical analysis or bounds on optimal token allocation would significantly strengthen the paper's contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XMurp2znll", "reviewer": "Reviewer_tXpo", "summary": "Representation learning models assign fixed-length representations to samples regardless of their information content. In contrast, the authors aim to present a method for learning variable-length tokenized representations for 2D images.\n\nThe paper introduces the Adaptive Visual Tokenizer (AVT), which recursively distills 2D images into 1D latent tokens through an encoder-decoder model trained using a reconstruction objective. To achieve variable-length tokenized representations, the authors add new tokens at every iteration of the recurrence. The authors allow optional masking of specific 2D tokens that are well-reconstructed, enabling \"dynamic halting\" per token.\n\nThrough experiments with their method, the authors provide several insights relating to the adaptive representation capacity of AVT:\n- More complex images require more tokens to reconstruct.\n- OOD images require more tokens than in-distribution ones to reconstruct.\n- As the downstream criterion for accepting the representation increases in complexity, more tokens are required.\n- Optimal adaptation for performance occurs when the token selection criterion is relevant to the task of interest.\n- Weaker models exhibit smaller relative performance drops with reduced token counts than stronger ones.\n\nThe authors also compare AVT reconstructions and linear probe performance with various baselines. They analyze the impact of choosing continuous vs discrete tokenizers. Analyzing attention maps of 1D latent tokens to 2D image tokens reveals that tokens attend to distinct semantically meaningful objects or parts, suggesting the emergence of object discovery. Finally, the authors show that broad attention of early tokens specializes to focus on sparser, more meaningful regions, when new tokens are introduced as the recurrence proceeds.", "questions": "Currently, the number of new tokens added per iteration seems to be fixed to 32. What does the relationship between the number of tokens added per iteration and the total number of iterations, for a fixed max token count, look like? It would be interesting to see if trade-offs exist, like incurring higher computational costs by performing more iterations leading to significantly smaller \"good enough\" representations (each iteration adds a smaller amount of new tokens, but each token gets many more refinement updates).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper presents thorough analysis of the adaptive representation capacity of the proposed method.\n- The authors show that their method can lead to the emergence of tokens that exhibit useful semantics.", "weaknesses": "Baselines are very limited.\n- The paper is motivated from a representation learning perspective, but there is no comparison with representation learning methods beyond two families of generative models. For vision, methods like CLIP [1], MoCo [2], DINO [3], and MAE [4] (which is also reconstruction-based) should be compared for downstream tasks like linear probe.\n- There is no comparison with other variable-length representation baselines, such as with Matryoshka representation learning [5], despite noting the relevance in the paper.\n\nThe comparisons with current baselines have very mixed results.\n- In Table 1, all baselines perform better than the authors for a specific token count, and many remain better even when compared to the max token count of AVT. The authors note that longer training on larger datasets with deeper networks could bridge the gap. Still, it is not apparent if that would be enough to outperform the baselines, especially ensuring that the training budgets remain fair.\n- Linear probe performance in Section 5 is at par with baselines, but often requiring more tokens and recurrent processing.\n\nDiscussion of gradients\n- There is no discussion of how gradients are treated within the recurrent processing. It is unclear if the authors backpropagate gradients from future iterations all the way back or if they stop the gradient flow between each iteration.\n\nThe insight about weak and strong models is not very convincing.\n- Weaker models exhibiting smaller relative performance drops with reduced token counts could simply indicate that weak models, due to being weak, do not reach the same peak performances as the strong models. This would result in their performance with more tokens remaining similar to their initial performance.\n- In this case, the statement should be that weaker models exhibit lower performance than stronger models, and they reach their performance cap sooner than stronger models as the number of tokens increases, which is not surprising.\n- Looking at the absolute performance metrics can provide some insights here.\n\nDynamic halting is unclear.\n- Figure 1 states that dynamic halting is optional, but a clear discussion of when it should be enabled or disabled is missing.\n- It is unclear if it is enabled for all experiments or some subset.\n\nComplexity of the method.\n- The proposed method is not simple, including multiple stages like learning to reconstruct pre-trained VQGAN tokens, then joint encoder-decoder training, and then GAN loss at some later stage in training.\n\nMinor (no effect on score):\n- \"Variable-length token representations\" is confusing due to ambiguity: it can mean representations per token variable length or a variable number of tokens. Perhaps \"variable-length tokenized representations\" is a less ambiguous term.\n- The footnote marked on Classification on line 397 points to a footnote printed three pages prior, which is confusing and not immediately apparent.\n- Table 1 has elements not adequately labeled or described in the caption: what is the # superscript? The caption should also clarify that the numbers under the dataset names are the number of tokens and not some varying attribute of the datasets.\n- Relevant paper you might want to cite that also learns tokenized representations (fixed-length) where tokens show the emergence of distinct concepts: [6].\n\nReferences:  \n[1] Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, Sutskever.\nLearning Transferable Visual Models from Natural Language Supervision. ICML 2021.  \n[2] Chen, Fan, Girshick, He. Improved Baselines with Momentum Contrastive Learning. CoRR 2020.  \n[3] Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, Joulin. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.  \n[4] He, Chen, Xie, Li, Dollár, Girshick. Masked Autoencoders are Scalable Vision Learners. CVPR 2022.  \n[5] Kusupati, Bhatt, Rege, Wallingford, Sinha, Ramanujan, Howard-Snyder, Chen, Kakade, Jain, Farhadi. Matryoshka Representation Learning. NeurIPS 2022.  \n[6] Vani, Nguyen, Lavoie, Krishna, Courville. SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Vision. ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4NwnZwyWPs", "reviewer": "Reviewer_46Y3", "summary": "The paper explores the concept of adaptive and recursive representations for image processing, focusing on how many tokens are necessary to effectively represent an image. The authors conduct a series of experiments to evaluate their approach, starting with image reconstruction tasks and linear probing for Top-1 classification. They investigate the impact of using continuous versus discrete tokenizers on reconstruction tasks and analyze the learned tokens' alignment with ground truth segmentation, suggesting potential for object discovery. The role of recurrent processing in achieving these results is also examined.\n\nA key contribution of the paper is the demonstration of a correlation between an image's perceptual complexity and its compressibility, as theorized by Schmidhuber's Low Complexity Art theory. The authors show that images with lower complexity can be reconstructed with fewer tokens, while more complex images require a higher number of tokens for accurate reconstruction. This is evidenced by experiments using out-of-distribution images from the PeopleART dataset, where the model trained on ImageNet-100 successfully reconstructs images of varying complexities with different token capacities.\n\nThe paper highlights two main observations: (a) as image complexity increases, using fewer tokens results in higher reconstruction errors, necessitating a larger computational budget; and (b) for a fixed image complexity, increasing the number of tokens reduces the reconstruction loss, showcasing the efficiency of the adaptive representation model. These findings underscore the potential of the proposed approach in efficiently handling images of varying complexities by adjusting the number of tokens used for representation.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper introduces an innovative method for image representation by employing a multi-stage training procedure that combines VQGAN and distillation techniques. The approach of using adaptive and recursive representations, along with the introduction of a shared 1D codebook for token quantization, is a novel contribution to the field. The concept of dynamically halting token processing based on reconstruction quality adds a unique aspect to the methodology, allowing for more efficient and tailored image representation.\n\nQuality: The quality of the research is demonstrated through comprehensive experiments and ablation studies. The paper provides a thorough evaluation of the proposed method, including image reconstruction experiments, linear probing for classification, and analysis of learned tokens for object discovery. The results, such as the high mean IOU achieved in attention map alignment with ImageNet-S GT segmentation, underscore the effectiveness of the approach. The use of GAN loss to enhance photo-realism further attests to the robustness of the training procedure.\n\nClarity: The paper is well-structured and clearly articulates the training procedure and experimental setup. The inclusion of figures, such as the adaptive visual tokenizer diagram, aids in understanding the complex process of 2D to 1D to 2D distillation. However, some sections could benefit from additional explanations, particularly for readers unfamiliar with the underlying concepts of VQGAN and distillation techniques. The appendix is referenced for implementation details, which is helpful for those seeking to replicate the study.\n\nSignificance: The significance of the paper lies in its potential impact on the field of image processing and representation learning. By demonstrating that adaptive representations can lead to emergent properties like object discovery, the research opens new avenues for exploring how different tokenization strategies can enhance image understanding. The ability to achieve high alignment with segmentation tasks without explicit optimization for segmentation highlights the broader applicability of the method in various computer vision tasks.", "weaknesses": "Limited Evaluation Metrics: The evaluation of the model is primarily focused on image reconstruction and classification tasks. While these are important, the paper could benefit from a broader range of evaluation metrics, particularly those that assess the quality and utility of the learned representations in more diverse and complex tasks.\n\nTokenization Approach: The paper discusses the use of continuous vs. discrete tokenizers but does not provide a comprehensive analysis of the advantages and disadvantages of each approach. A more detailed comparison could help in understanding the trade-offs involved and the scenarios where one might be preferred over the other.\n\nSensitivity to Model Strength: The findings suggest that stronger models are more sensitive to fewer-token reconstructions, which could be a limitation in practical applications where computational resources are constrained. The paper could explore strategies to mitigate this sensitivity and improve the robustness of the approach across different model strengths.\n\nGeneralization to Other Tasks: The paper notes that classification tasks do not align well with dense tasks like depth estimation and reconstruction. This indicates a potential limitation in the generalization of the learned representations to various tasks. Further exploration into how these representations can be adapted or extended to perform well across a wider range of tasks would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gq1C12Ed0z", "reviewer": "Reviewer_6BgE", "summary": "This paper discusses a new approach to variable-length visual token representation learning that addresses the limitations of fixed-length representations for images. The method, AVT, processes image tokens recursively, refining them into 1D latent tokens over multiple iterations while adaptively increasing capacity by adding new tokens. The iterative process potentially facilitates object and part discovery based on image entropy and context.", "questions": "1. How to determine how many iterations are needed for each picture? I see that you have done a lot of interesting experiments, but currently, it can only be set manually through experiments. Can it be selected adaptively based on the amount of information, for example, information entropy for the image?\n2. Compared to 2D fixed-length tokenizers like VQGAN, or distilled 1D fixed-length tokenization methods like Perciver. How much calculation and storage space does the AVT method increase? Is it much slower than the above two methods through iterative reconstruction in AVT?\n3. Tab.1 shows that the FID reconstructed through your method is worse than Tiktok and VQGAN. As you explain in line 453, the datasets for training and evaluation are different. Have you tried the experiment under the same situation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed a novel method, AVT, which distills 2D image tokens into 1D variable-length latent tokens, an important topic for Visual Representation Learning.\n2. The experiments are solid and comprehensive.\n3. The writing is clear and concise.", "weaknesses": "1. The reconstruction effect, FID, of their method AVT in Table 1 has no obvious outstanding improvement compared with existing methods, VQGAN and Tiktok, under the same number of tokens.\n2. Experimental results, in section 5 Further Experiments & Ablations, can be displayed in the form of figures and tables, which will make them more intuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)