CAT: Content-Adaptive Image Tokenization
Authors: Junhong Shen
Kushal Tirumala
Michihiro Yasunaga
Ishan Misra
Luke Zettlemoyer
LILI YU
Chunting Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Image tokenization
Primary area: deep_learning
TL;DR: We develop an adaptive image tokenizer that compresses images into variable-sized latent features based on its content complexity.

Abstract: Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity and introducing unnecessary computate overhead for   simpler images. To address this, we propose Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design (1) a caption-based evaluation system that leverages LLMs to predict content complexity and determine the optimal compression ratio for an image, and (2) a novel nested VAE architecture that performs variable-rate compression in a single model.
Trained on images with  varying   complexity, CAT achieves an average of 15% reduction in rFID across seven detail-rich datasets containing text, humans, and complex textures. On natural image datasets like ImageNet and COCO, it  reduces token   usage by 18% while maintaining high-fidelity reconstructions.   We further evaluate CAT on two downstream tasks.  For image classification, CAT consistently improves top-1 accuracy across  five datasets spanning diverse domains. For image generation, it boosts training throughput by 23% on ImageNet, leading to more efficient learning and  improved FIDs over fixed-token baselines.

CAT: Content-Adaptive Image Tokenization
Junhong Shen" Carnegie Mellon University
Kushal Tirumala FAIR at Meta
Michihiro Yasunaga FAIR at Meta
Ishan Misra FAIR at Meta
Luke Zettlemoyer FAIR at Meta
Lili Yu FAIR at Meta
Chunting Zhou' FAIR at Meta
Abstract
Most existing image tokenizers encode images into fixed number of tokens patches, overlooking the inherent variability in image complexity and introducing unnecessary computate overhead for simpler images. To address this, we propose Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design (1) caption-based based evaluation system that leverages LLMs to predict content complexity and determine the optimal compression ratio for an image, and (2) a novel nested VAE architecture that performs variable-rate compression in a single model. Trained on images with varying complexity, CAT achieves an average of 15% reduction in rFID across seven detail- datasets containing text, humans, and complex textures. On natural image datasets like ImageNet and COCO, it reduces token usage by 18% while maintaining high-fidelity reconstruc- tions. We further evaluate CAT' on two downstream tasks. For image classification, CAT consistently improves top-1 accuracy across five datasets spanning diverse do- mains. For image generation, boosts training throughput by 23% on ImageNet, leading more efficient learning and improved FIDs over fixed-token baselines.
1 Introduction
Image tokenizers compress high-resolution images into low-dimensional latent representations, enabling compact and semantically meaningful inputs for downstream tasks such as generation and classification [ Despite their effectiveness, most existing tokenizers operate at fixed com- pression ratio, producing latent representations of uniform length regardless of the image's content. However, natural images exhibit significant variability in complexity, from sparse scenes to densely textured ones, suggesting that fixed-length representations
be both inefficient and suboptimal.
Classical codecs such as JPEG 9 implicitly exploit this variation: when fixing the quality level, they produce different file sizes for images with different frequency characteristics. In contrast, fixed-ratio tokenizers may under-compress simple images, wasting compute on redundant information, or over- compressing complex ones, losing important details. These problems become more prominent when tokenizers are used in large-scale generative pipelines or as feature extractors in downstream tasks.
Several recent works explore dynamic token representation during inference [10 However, these methods typically require access to the input image-an an assumption incompatible with many practical use cases. For example, in image generation with latent diffusion models [12. only the user's text prompt is available, and the number of latent tokens, which significantly influences generation quality, must be specified in advance. Moreover, these methods do not adapt tokenizer training to image complexity, missing an opportunity to optimize for both content and downstream utility.
Work done during internship at Meta. Joint senior author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Compression Selection
Nested VAE for Adaptive Compression
32x: Low Complexity
Bx: High Complexity
Content Complexity Eval via LLM
cleac Any and
Deaksop with acratcc peepounded by
resectul beadth....
16466
312d12
1ba2
Encoder Downsampilnn
Shared Latent Sampling
Decoder Upsampling
Figure 1: Content-Adaptive Tokenization. CAT uses an LLM to determine the compression ratio from the image's s text description and uses a nested VAË to generate latent features by dynamically routing the input.
In this work, we introduce Content-Adaptive Tokenizer (CAT), novel approach that dynamically adjusts representation capacity based on image complexity. CAT combines a caption-driven com- plexity evaluator with nested autoencoder architecture to produce variable-lengle latent features in a single forward pass (Figure D] Specifically, the evaluator uses large language models (LLMs) to predict the optimal compression ratio from textual descriptions. It analyzes the image's caption and answers perception-orientel questions (e.g., "are there human faces or text?") to produce an interpretable complexity score. Based on this score, assign one of three compression ratios to the image: 8x, 16x, or 32x. Empirical results (Section (..2) show that this system is robust across different LLMs and caption styles, providing general mechanism for content-aware adaptation.
To support variable-lengie length representations, we design a nested variational autoencoder (VAE) architec- ture that routes intermediate encoder features to a shared latent block for generating Gaussian parame- ters of different shapes, enabling latent codes at multiple spatial scales. This design allows us to train single model that supports multiple compression levels while maintaining architectural efficiency.
We train CAT on a diverse set of images using extensive evaluations across nine datasets, covering natural scenes (ImageNet [13, COCO [14) human faces (CelebA ([5)), and detail-heavy domains such text (ChartQA [66. GTSRB [77]
compression ratios and conduct
SVHN (88), textures, and satellite imagery. On large-scale natural images, CAT preserves high reconstruction quality while reducing token usage by 18% compared fix-token baselines. On complex images, CAT achieves significantly better reconstruction quiality. improving the rFID by 12% on CelebA, 17% on GTSRB, 20 % on SVHN, and 39 % on ChartQA relative to fixed-token baselines. We also benchmark CAT two critical downstream tasks:
Image classification: CAT achieves the highest linear probing accuracy compared to all fixed- token baselines across five challenging datasets where prior work has shown that zero-shot models perform poorly [9]]. This highlights the quality our content- content-adaptive latent representations. Besides, CAT consistently improves performance in full fine-tuning settings.
Text-to-Image Generation: We integrate CAT into Latent Diffusion Transformers (DiT's) [20]. On flass-conditional ImageNet generation, CAT increases the training throughput by 23%, thus achieving better FIDs than all fixed-ratio tokenizers trained under the same FLOPs. We note that CAT allows
specify the desired token count at inference time, enabling flexible trade-off between computational cost and output quality, with more tokens typically yielding higher fidelity.
In summary, we propose CAT, an efficient and effective image tokenizer that enables content-adaptive compression through an LLM-based evaluator and nested VAE architecture. To the best of our knowl- edge, this is the first work to combine language-guides tokenization with adaptive representation, showing both performance and efficiency gain in image reconstruction, classification, and generation.
2 Related Work
Image tokenization. Existing tokenizers use diverse architectures and encoding schemes. Continu- ous tokenizers often utilize the VAE architecture (2) to generate Gaussian distributions for sampling continuous latent features. Discrete tokenizers like VQ-VAE (2], RQ- VAE (22). MoVQ (23]. MAGVIT-v2 B. and FSQ 7 use quantization techniques to convert latent representations into tokens. VQ-GAN M. ViT-VQGAN 244,, and Efficient-VQGAN 25 further built on adversarial training to improve performance. Beyond methods that tokenize images into 2D grids, 1D tokenizers such as TiTok g are proposed to enhance efficiency. While CAT is designed continuous 2D tokenizer, the proposed adaptive image encoding scheme can be applied to discrete and ID tokenizers.

---

Adaptive compression. Traditional codecs like JPEG [9] for images and H.264 [26] for videos apply varying levels of compression based on the input media, producing files of different sizes.
In deep learning, patch dropout 227 28 patch merging 32]. and sequence packing 33 are
proposed for Vision Transformers 34 Quadformer [35] uses mixed-resolution patches to vary
token count. However, these methods are tailored for visual understanding tasks and cannot be used for generation. A few recent works such as VAR [36] study multi-scale tokenization for generation. Nonetheless, these works are not adaptive to image content.
Adaptive tokenizers for image generation remain relatively underexplored. Elastic 10 employs random masking to drop the tail tokens during training. ALIT [IT] iteratively distills 2D tokens into ID to reduce the token count. DQ- (37] leverages cotormation-dessiii for dynamic representation. However, all of these methods (1) require an input image to determine the token count, limiting their use in generation settings where only text is available; and (2) overlook image complexity during training. In contrast, we enable adaptive compression directly from textual descriptions without observing the image. We also explicitly train the tokenizer with complexity supervision. A concurrent work, TexTok [38], explores language-guider tokenization by supplying the caption embeddings to the VAE. However, it is not designed for adaptive representation.
Multi-scale network design. Our work is also related to designing neural networks for multi-scale feature extraction. Inspired by U-Net [3] and Matryoshka networks 40 [2]
we incorporate skip connections into the VAE to support multi-ratio compression in a single forward pass. Parallel work explores transformer-basee
+ 49 To the best of our knowledge, our nested design offers the simplest yet effective solution for generating multi-scale latents via VAFs without additional architectural or computational overhead, while achieving strong results (Section 4
multi-scale architectures
3 Method
In this section, we present CAT for adaptive image tokenization. We begin by motivating our caption- based evaluator for image complexity estimation. Then, we describe the nested VAE architecture.
3.1 Proof of Concept
How much we actually compress? A key question in this work is to determine how much an image can be compressed without significant loss of quality. To explore this, we study the reconstruc- tion performance of existing tokenizers under various compression ratios. We take the open-source LDM tokenizers with 8x, 16x and 32x compression ratios and compute their reconstruction mean squared eтror (MSE) on 41K COCO 2014 [4] images with resolu- tion 512. We find that for 28.3% of the images, 32x compression results in less than a 0.001 MSE increase compared to 8x, while reducing the token count by factor of 16. We also compute the best MSE among all compression ratios for each image and determine the maximum acceptable compression ratio under a tolerance т (arg max MSE - MSE < T). Fig. (left) shows that 56% of the images can be compressed at least to 16x with negligible (0.0001) increase in MSF That is, a large portion of natural images can be compressed more aggressively to save compute.
1/-0.0001
1ol00.0000
TaleO/CO15
Metric
Pearson r
JPEG
0.31
MSE
0.36
LPIPS
Caption
0.55
Octina commression Rato
Figure 2: Left: Max acceptable compression ra- tios for different T. Right: Correlation with max acceptable compression ratio for 7 0.0015.
2
On the other hand, our visual inspection reveals that images with fine-grained elements like text have significantly worse reconstruction quality 32x compression compared to Sx compression (e.g.. see row 3 and 4 in Figure B) This suggests that more tokens are required to accurately reconstruct low-level details. The above results provide strong motivation for developing an adaptive tokenizer.
Limitations of existing complexity metrics. Next, we want to identify a metric for predicting an image's S optimal compression ratio. We explore traditional codecs, such as the JPEG file size; (2) metrics based on pretrained VAEs, such as the reconstruction MSE and LPIPS distance [50]. We use Stable Diffusion's 's sd-vac-ft-mse [5]] for this
existing options: (1) metrics produced by
released series VAE tokenizers with diverse compression ratios and trained a controlled setting. Most other tokenizers, such as stabilityai/sd- ft- msme, only have one compressed ratio.
2LDM 12
'The average MSE across all images for 8x LDM VAE is 0.0039, 0.0001 tolerance should be acceptable.
3

---

Larow MSEOPEG
Size
16c
32%
Small DEG Size
P
Originl
MSS-D 01221
MSS C 01651
Origina
NSK-0.05310
Originl
C.00003
3.0:106
Origina
MTS C00030
Ma RE2-02000333
Mic 2,00411
Figure 3: Existing metrics can misjudge image complexity. Text-heavy images that are difficult to difficulty to reconstruct (note the distortion in the bottom two rows) are considered as easy by existing metrics.
analysis. We compute these metrics on COCO and analyze their correlation with the maximum acceptable compression ratio under 0.0015 tolerance. However, Table shows that the Pearson r's relatively low. Statistically, these metrics are not highly correlated with an image's complexity.
We also manually inspect images with large JPEG sizes and MSEs. We note that images featuring repetitive patterns, such as grass, forests, and animals like giraffes and zebras consistently show high complexity metrics. Indeed, JPEG compression can be inefficient for images with sharp edges and high contrast. A single-pixel shift in zebra image can toggle pixel values between black and white, significantly increasing the reconstruction error. However, as Figure 3 or MSEs do not always notably affect visual quality. For example, we can easily recognize the zebra and may not perceive the differences resulting from various compression ratios.
(left) show, large JPEG sizes
Conversely. images with small JPEG sizes and low MSEs can have poor fidelity. For example, as Figure 3 (right) shows, distortions in images containing elements sensitive human perception, such as text, numbers, and human faces, can drastically reduce visual quality. Despite this, these images have low reconstruction errors since the critical elements occupy only small portions of the images.
Thus, existing metrics fail to capture details crucial to human perception. In contrast to the predicted complexity, we actually want to use large compression ratio for zebra images, and a small ratio for the phone images. Beyond this, all considered metrics require images as input and cannot be used for text-to-image generation tasks, where no image is available. Given all these limitations, we seek new complexity metric that is independent of pixel data and aligns with human perception. We note that it impractical use the tolcrance-basse based compression ratio in Figure as the complexity metric because it requires at least three model calls to get the MSEs, making it computationally expensive.
3.2 Complexity Evaluation via Captions and L.Mss
Image generation typically involves users providing a prompt that describes the desired image content. Inspired by this use case, we propose to use the text description of an image to evaluate its complexity.
We propose 3-stage evaluation pipeline: (1) obtaining the text description, (2) prompting a LLM for a complexity score, and (3) classifying the score into a compression ratio. The text description includes both the image caption and answers to a set of pre-defined queries of the form "Are there [obj]?" For example, obj E (human faces, text) can be used to align with human perceptions. During training, when image data are available, we use vision language model (VLM) to generate the captions and answers. During inference, users need to provide the necessary textual information.
We prompt language model with the text description to generate an integer score ranging from 1 to 10, where higher scores indicate greater complexity. To ensure scoring consistency, we design a detailed rubric that instructs the model to consider factors including semantic complexity (objects, scenes), visual complexity (color, texture), and perceptual complexity. We also provide in-context examples for each score. The complete prompt is provided in Appendix B
Based on the complexity score, each image is classified into of three compression ratios: 8x, 16x, or 32x, with higher complexity scores corresponding to lower ratios. We choose these options as they are widely used in existing tokenizers [12) and provide meaningful variation in token counts. Then, we implement thresholding scheme to divide the scores into three intervals: [1, a].
(a, 6). and (b, 10), where a < b E z are selected to achieve an average compression ratio of 16x across all training data to enable fair comparison with fixed 16x tokenizers. Formally, denote the training distribution as X, input resolution as T, the compression ratio of an image E X as

---

f(z) E {f1 =8 = =16,fs = =32,, the target average compression ratio as f= 16. We set a, b by:
Σ RED
Ep(f()))
(1)
Ma 1
While multiple threshold configurations can achieve the target average compression ratio, our experi- ments in Section 442 demonstrate that diverse distribution of ratios yields better performance. For the specific training dataset and thresholds used in our experiments, see Section 4
Robustness testing and bias mitigation. To evaluate the robustness of our caption- based complexity evaluator, particularly under different model choices and caption styles, we conduct ablation studies
on diverse datasets such as ImageNet [3], CelebA [3]
and EuroSAT [52] We compare two
pipelines: (1) a unified pipeline using LLa VA 7B
53 for both captioning and scoring in a single
54 for captioning and either Qwen2.5 7B (55] or LLLMA3311 8B Instruct [56] for scoring. Additionally, within the unified pipeline, we test variants that explicitly prompt the model to produce longer versus shorter captions. As shown Appendix Table TT] all configurations result in similar compression distributions. We attribute this robustness to (1) our carefully designed scoring rubries and in-context examples, and (2) the LLM's ability infer content complexity is more influenced by the semantic content of the caption than its wording. These findings suggest that our evaluator robust to model choices and captioning variations. Given its simplicity and efficiency, we adopt the unified pipeline in all subsequent experiments.
pass, and (2) a separated pipeline using InstructBLIP
High correlation with visual complexity. We verify that our caption score provides reliable estimates of the optimal compression ratio. Similar to Scction we compute the correlation between complexity scores and maximum acceptable compression ratios on COCO. Our metric achieves the highest Pearson r among all options (Table 2 and 62.39% exact agreement on compression ratio selection. Manual inspection also confirms that perceptually challenging images receive high complexity scores. We note that during development, we also tested other caption-derives metrics such as caption length, but does not perform as well as LLM-based scoring (Appendix Table 9
Minimal captioning overhead. When text information is available, our pipeline requires only a single LLM call to obtain the complexity score. During training, where only images are provided, we generate the caption, query responses, and complexity score within single inference pass, keeping the evaluation overhead minimal. Given the efficiency of modern inference engines like vLLM [57]. the cost of complexity evaluation is negligible compared to the overall compute required for training the tokenizer and downstream models.
3.3 Nested VAE for Adaptive Comprssion
To reduce the tokenizer's 's training and storage costs, we introduce nested structure the standard VAE architecture [2] to enable multiple compression ratios within a single model. In standard VAE architecture, the encoder consists of multiple downsampling blocks followed by an attention-based middle block. The decoder consists of an attention -ased middle block followed by upsampling blocks. This symmetrical design resembles U-Net (39 and Matryoshka networks [40) for multi-scale feature extraction. Inspired by these works, we leverage the intermediate outputs of the downsampling blocks to enable adaptive compression (Figure I). We describe the proposed architecture below.
Skip connection with channel matching. Denote the feature shape under the largest compression ratio as (c3, f fs ). where C3 is the channel dimension. We observe that, in the standard VAE encoder, the spatial dimension of the intermediate outputs from the downsampling blocks decreases by factor of 2 with each additional block. This means that the output of the second-to-lass downsampling block has shape (c2, T ).), and the output of the third-to-last downsampling block has shape (CI, f.). Then, a natural thought is to directly route these intermediate outputs to the middle block to generate latent features. However, since the channel dimensions of these intermediate outputs vary, we incorporate ResNet blocks (58) for channel matching. Let the latent channel dimension of the VAE be c. Applying channel matching enables us to transform intermediate features of shape (Cns F. F.) to (c, J., f..) for 1, 2, 3. This will be the shape of the latent parameters.
For decoder, we similarly add skip connection with channel matching and use the decoder middle block's ' output as the input to the corresponding upsampling block, i.e., for compression ratio fn, we bypass the first upsampling blocks so the final output has the same size as the original image.
5

---

Shared mean/variance parametrization. Features after channel matching are directed to the middle block generate the latent distribution parameters. In CAT, we share the middle block for all compression ratios to maintain scale consistency of the parameterized mean and variance. The convolutional design of the middle block allows it to process inputs of varying spatial dimensions, as long as the channel dimension is aligned. Thus, for all n. E (1, (1,2,3), the mean Ms,, variance d and samples on of the Gaussian distribution all have shape (c, f.: f.). i.e. the input compressed at fn.
Increasing parameter allocation for shared modules. Images assigned larger compression ratios do not go through the later downsampling blocks and are directed straight to the middle block. The middle block is thus tasked with handling multi-scale features. To improve its capacity, we allocate more parameters to the middle block by increasing the number of attention layers.
3.4 Training
While existing adaptive tokenizers like ElasticTok [T]] do not consider the varying complexity of training data, we explicitly incorporate content complexity into training to learn feature extraction at different gramularity. For each training example, first obtain the compression ratio from the LLM. Then, the image is processed only by the layers dedicated to its compression ratio.
Similar to prior works [ we use joint objective that minimizes reconstruction error, Kullback- Leibler (KL.) divergence, and perceptual loss. Specifically, we use L1 loss for pixel-wise recon- struction. To encourage the encoder output z towards a normal distribution, Kl-regularizztion is added: (=) == [Qo [=|x>) llp())), where is the encoder parameters and p(z) N/ I). The perceptual loss consists of the LPIPS similarity [50) and a loss based on the internal features of the MoCo v2 model [59]. Beyond these, we train our tokenizer in an adversarial manner (60) using a patch-based discriminator vi, which adds GAN loss LGAN ãt, vb). Thus, our overall objective is:
f min max ErEx free (I,, +) + (=) + 7 Spen (i) + SLGAN( $, v)
8 e
where B, 7, are loss weights. We provide our training code in the supplementary material.
4 Image Reconstruction
We first evaluate CAT's reconstruction quality and analyze various design choices via ablation studies.
Training details. We use a nested VAE with six downsampling blocks and 187M parameters. For our main results (Table(3), we set the latent channel to 16. We study its effect as an ablation study in Section 4.2 For training data, we use 380M licensed Shutterstock images with 512x512 resolution. After obtaining the complexity scores, we find that two sets of thresholds, (a, b) E {(4, 7), (2, 8)}. both achieve an average compression ratio of 16x. We select (4, 7) for our main experiments because it leads to a more diverse distribution and better empirical results (see ablation studies in Section 4.2).
All models including the basclines trained using global batch size of 512 on 64 NVIDIA A100 GPUs for 1M steps. Other architecture and training details can be found in Appendix B
Evaluation datasets. We use four representative datasets: COCO 14] and ImageNet [13 for natural images, CelebA 15 and ChartQA [16] for perceptually challenging im- ages. Table shows the compression ratio distributions.
Table 1: Test data distribution. CAT ap- plics larger compression to natural images and smaller ratios to CelebA and ChartQA.
Dotasets
Method
Aug Lazent Ang
We compare against fixed-token baselines that the same VAE architecture but without the nested struc- ture. We further study the effect of caption complexity by training a nested VAE using JPEG size as the complexity metric. All baselines have average 16x compression. To our knowledge, none of existing adaptive tokenizers (e.g., ElasticTok,, ALIT) report quantitative results on the datasets we use, so we do not compare with them.
Baselines.
coco
CAT (Ouns)
JPFC
10%
54%
54%
37%
36%
31.87
92:43
15.79s
16.07s
ImageNes
JPBG
(Ours)
45%
31.24
29.32
17.46s
16.39s
Celebi
CAT (Ours) 17%
JPDG
0S
85%
0%
100%
0%
39.29
16
13.03K
32x
ClanqQ
CAT (Ouns)
JPFC
96%
08
5/6
976
0%
63.02
1661
30.83s
8
ALIT only shows an rFID of 8.03 on ImageNet100. See Appendix E.3 for more baseline details.
4.1 Main Results
Better reconstruction for complex images, higher efficiency for natural images. Table presents the reconstruction FID (rFID), LPIPS, and PSNR [61] on four datasets. Comparing CAT with the

---

Table 3: Reconstruction results. All models have latent channel 16.
Averge
CO00
ImageNet
CelchA
ChanQA
Compression
IAD, LPIPS,
PSNR
HDD,
LPIPS]
PSNR
#FID,
LPIPS,
PSNR
LPIPS]
PSNR
Fixed
1,86
45.36
Fixed
16
Adaptive JPEG
0.72
0.17
30.11
0.51
0.16
30.61
6.57
0.14
36.47
10.17
0.048
31.54
Adaptive
(Ours)
30.19
0.051
Fixed
6.10
Digiinl
LDM
CAT
Figure 4: Qualitative reconstruction examples. We highlight the compression ratio selected by our caption complexity red. On simpler images (top two rows), adjusting the compression ratio does not significantly affect quality. On more complex images (bottom three rows), the impact is substantial.
fixed 16x baseline, our method significantly outperforms the baselines across all metrics on CelebA
and ChartQA, improving the rFID by 12% on CelebA and 39% on ChartQA. Our ChartQA rFID even surpasses the fixed 8x baseline, likely because the LLM evaluator can effectively identify rich visual details these datasets and assign lower compression ratios accordingly (Table IT. On COCO and ImageNet, CAT generally outperforms the baselines, with only slight drop in rFIDD on ImageNet. However, ImageNet, CAT achieves an average compression ratio of 17.46x, which means 18% fewer tokens to represent the dataset compared to the 16x bascline.
Figure 4 shows qualitative examples of progressive reconstruction quality as we manually increase the token count and reduce the compression ratio to represent each image. We highlight the compression ratio predicted by CAT in red. Different visual inputs need different ideal compression ratios. Natural images with fewer objects and simpler patterns can accurately reconstructed at 32x, whereas complex images with visual details require lower compression. Thus, the caption- based CAT reconstruction has comparable quality to the fixed 16x baseline natural images but surpasses it on text-heavy images. We include more visualization in Appendix [.44 We also report CAT's performance with uniform 16x compression Appendix E.5) Lastly, while most prior works do not evaluate side-by-side comparison with popular fixed-ratio tokenizers on ImageNet in Table 2 This table is for
Table 2: ImageNet- 512 reconstruction.
Imageñer
rMD
MEAGIT-VOGAN A
197
Thl-----12 138 B
1.52
LFQ [1
1.22
TexTol
0.25
LUM 12 CAT
0.55
0.46
diverse datasets like ChartQA, we include
reference only due to differing training setups (we refer the readers to Table 3 for fully comparable
baselines). Nonetheless, the results highlight that CAT achieves competitive performance by adapting compression ratios to image content.
Caption complexity outperforms JPEG metric. further compare CAT against training same adaptive architecture but using JPEG size as the complexity metric. Table shows that CAT achieves better rFID, LPIPS, and PSNR across all datasets. While both tokenizers have similar training compression ratio distribution, the test-time compression ratio distribution varies significantly
7

---

Table 4: Ablation on thresholds. The (1, setting with more diverse training distribution achieves generally hetter performance.
Table 6: More reconstruction and classification results. CAT achieves better reconstruction and classification results.
DTD
EuroSAT
GTSRB
SUN397
SVHN
(a, b)
COCO
ImageNet
CelebA
CharQAA
Reconstruction FID (4)
(1.7)
0.65
0.46
1.97
5.27
LDM VAE 16x
7.86
7.04
1.22
1.95
1.76
(2.8)
0.67
0.43
2.58
7.70
Fix 16x
7.29
6.26
1.38
1.95
2.11
CAT (Ours)
7.23
5.45
1.14
1.93
1.69
Table 5: Ablation latent channel. Increas-
Linear Probing Top-/ Accuracy (%,
ing latent channel e improves rFID across four evaluated datasets.
LDM VAE 16x
53.81
75.38
70.29
63,70
64.87
Fix 16x
50.42
78.08
70.07
62.92
65.07
CAT (Ours)
54.51
78.21
71.28
64.16
66.39
coco
ImageNet
CelebA
ChartQA
Fine-Tuning Top-I
Accuracy (%, '1)
1.66
1.10
5.83
9.13
LDM VAE 16x
75.91
97.00
90.61
79.89
83.05
1.03
0.60
4.54
7.95
Fix 16x
71.96
92.46
95.007
78.77
85.93
16
0.65
0.46
1.97
5.27
CAT (Ours) 74.11
98.00
95.32
78.45
86.13
Notably, since JPEG size often cannot capture perceptually important factors (see discussion in Section B.T nearly all images in CelebA and ChartQA are assigned 32x compression. Thus, CAT significantly outperforms JPEG on these two datasets, showing the effectiveness of caption-base metric and LLM evaluation in determining image intrinsic complexity.
(Table D.
4.2 Ablation Studies
Benefits of diverse compression ratios. We explore several design choices for our tokenizer. First, we study how the distribution of compression ratios affects overall reconstruction. To achieve an average compression ratio of 16, could set the thresholds (a, b) to either (4, 7) or (2,8). (4,7)
yields a more diverse distribution of compression ratios, whereas (2, 8) results in a concentrated distribution similar to a fixed 16x tokenizer, making it less interesting. Table 4 shows that (4, 7) produces better reconstruction metrics across all datasets due to the diversity in compression ratios ensures that all parts of the model are fully trained. Hence, adopt (4, 7) as the thresholds for CAT.
Effect of latent channel dimension. We also vary the latent channel e to study its effect. Table shows that larger improves reconstruction results. However, similar to previous work [3] we observe a reconstruction-gennene does not necessarily improve second-stage generative results. We elaborate more on this in Section 0
trade-off: while increasing e is beneficial for reconstruction,
5 Classification on Diverse Datasets
While CAT demonstrates strong reconstruction performance on COCO, ImageNet, CelebA, and ChartQA, these datasets represent only a subset of image domains. To thoroughly assess CAT's capabilities in diverse domains and tasks beyond reconstruction. we follow OpenCLIP [19] and test CAT's classification performance on five datasets that feature details beyond text and human faces:
DTD 664 (textures), EuroSAT [52] (satellite images), GTSRB 17 (traffic signs), SUN397 [65]
(indoor and outdoor scenes), and SVHN 18 (street numbers). We examine two settings: (1) linear probing, where freeze the encoder and train only linear layer top of the latent features; and (2) fine-tuning both the encoder and classification head. See Appendix F for experiment details.
As shown in Table 6 CAT consistently outperforms fixed-ratio baselines across both settings, achiev- ing the best top-1 accuracy on all five datasets in the linear probing setup. This highlights the quality and generalizability of CAT's latent representations, which transfer effectively to downstream classification tasks even without fine-tuning. In addition, CAT maintains strong reconstruction perfor- mance on these datasets, improving the rFID by 13% on EuroSAT, 17% on GTSRB, and 20% on SVHN (Table 6 top rows). These results further confirm that our caption- complexity metric supports learning representations that balance both compression quality and downstream utility.
Image Generation
Lastly, we evaluate CAT on text-to-image generation to show that (1) its adaptive design does not compromise generation quality; (2) by using fewer tokens to represent the training data, CAT actually enables more efficient learning, leading to stronger generative models under the same compute budget.
Setup. Given CAT's ability to produce variable-length token sequences, we integrate with Diffusion Transformer (DiT) (20). which handles adaptive token representations naturally. DiT
8

---

Table 7: ImageNet-512 generation results. All tokenizers have
Table 8: Larger channel is not always better for generation.
average 16x compression ratio. "rFLOPs" means relative FLOPs.
FIDI
sFIDL
1S7
Precision"
Recall
Eval IFLOPST
Fixed
DIT + DDM VAE
10033
16.88
114.44
1.65
0.50
1x
HID
sHD1
1ST
Precisiont
Recall?
DiT + Fixed 16x.
4.78
11.81
187.47
0.72
0.49
1x
5.12
11.12
152.39
0.72
0.48
Adaptive
[X- Tranffrmerr [3]]
DIT + CAT (Ours)
5.11
4.56
10.55
191,99
178.2
0,78
0.49
0,2x
16
4.56
10.31
10.55
191.09
181.03
0.75
0.76
0.49
0.48
takes the noised latent features as input, applies patching for downsampling, and uses a transformer architecture to predict the added noise. Following [20), we work with class-conditional generation on ImageNet-5122 12, leveraging DiT-XL with patch size of 2.
During training. each image processed through the CAT pipeline to determine its compression ratio, resulting in variable-length latent representations tailored to its complexity. That is, each example can have different number of latent tokens. For inference, we obtain a class-level compression ratio by providing a textual description of the form "this is an image of [label]" to the LLM evaluator, which predicts the appropriate compression ratio for that class. For example, if the evaluator suggests a 16x compression, we generate ((12)) )² 256 tokens. where the 2 in the denominator accounts for DiT's patching. The generated tokens are then decoded by CAT to reconstruct 512x512 images. Following (0),, report FID [66], Sliding FID [67], Inception Score [68), precision and recall [69] 50K images generated with 250 DDPM steps and classifier free guidance [70]. We note that we use the same compression ratio per class predicted by LLMs mainly for benchmarking purpose. In reality, CAT allows users to flexibly specify the desired token count at inference time, as we will show later.
Baselines. We compare against DiT-XL paired with the open-source 16x LDM VAE and the fixed 16x tokenizer we trained ourselves. All models are trained on 16 NVIDIA H100 GPUs for 400K steps, using a global token batch size of 262,144, which is equivalent to 1024 images at 16x compression. See Appendix G for more implementation and baselinedetails.
As shown in Table 7 DiT-CAT achieves the best FID, sFID, IS, and precision among all DiT baselines trained with the same computational resources. We attribute this performance to two factors. First, using fewer tokens for simpler images improves processing efficiency, allowing for more extensive training within the same computational budget. In fact, the average token count per training image for DiT-CAT's is 197 19..44. Compared to the 256 tokens used by fixed 16x tokenizers, CAT achieves a 23% reduction in token count, allowing the model to process more images within the same training budget. Second, adaptively allocating representation capacity also enables more effective modeling of complex images, as richer visual details are better preserved through the use of additional tokens.
As discussed earlier, we use our LLM evalua- tor to obtain the generation token count using pre-defined captions mainly because we want to automate the evaluation process. In reality, users can flexibly set the token count based on their computational budget. To explore this, we manu- ally vary the token count during generation with DiT-CAT and observe that FID-50K improves from 5.83 (64 tokens) to 5.02 (256 tokens) and 4.12 (1024 tokens), confirming that more tokens lead to higher-quality images. Qualitative exam- plcs in Figure 5 further support this observation. Thus, CAT enables controllable trade-off be- tween efficiency and generation quality. For more visualization, see Appendix G.4]
Figure 5:
Increasing token count (left~>right) for CAT leads to better image quality and higher complexity.
Lastly, recall that we trained tokenizers with different latent channels in Section[4.]] Table shows the generation performance. While larger e is better for reconstruction, does not benefit generation. CAT with = 8 achieves the best FID across all experiments. This observation agrees with prior work 12 and highlights the importance of choosing an appropriate c. We leave exploring c's impact on downstream task for future work.

---

7 Conclusion
We propose an adaptive image tokenizer, CAT, that allocates different number of tokens to images based on the content complexity derived from text description. Our experiments show that CAT improves the quality and efficiency of image representation on variety of downstream tasks.
Nested VAE is natural and effective extension of the VAE archi- tecture but is constrained to predefined compression ratios that scale by factors of An intriguing direction for future work would be to enable more flexible compression ratios by transitioning to transformer backbones and dynamically adjusting token counts. Besides, images contain diverse global and local information. While CAT addresses global complexity by increasing token alloca- tion for images with intricate details, further efficiency improvements could come through local irizzationaallccattng more tokens specifically to detailed regions while reducing tokens for sim- pler areas. Besides, an ideal pipeline would enable LLMs to automatically identify perception- crritical elements without relying on predefined queries. This would be an important next step for our work.
Limitations and Future Work.
10

---

References
[1]
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
[2] Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2014.
[3] Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer key to visual generation, 2024. URL Btthttps::// //apxii///axvv...// http//aaxvvvg//aaaae.. org. 2310
05737
[4] Junhong Shen, Mikhail Khodak, and Ameet Talwalkar. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[5] Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS-bench-360; Benchmarking neural architecture search on diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022.
[6] Luyao Yuan, Dongruo Zhou, Junhong Shen, Jingdong Gao, Jeffrey L. Chen, Quanquan Gu, Ying Nian Wu, and Song-Chun Zhu. Iterative teacher-aware learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29231 19231-29245, Curran Associates, Inc., 2021. URL http::tttt///tttt http/ttts//pprrceedinggg newrips. /ttpp//pee..iileepepee////////// f FFG00FFFo44Fff661111114444166666
pdf
[7] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple, 2023. URL https:/tt://aatt http///xxvv..oo httgg/ab//3309.000 2309. 15505
[8] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arxiv: 2406.07550, 2024.
[9] G.K. Wallace. The jpeg still picture compression standard. IEEE Transactions Consumer Electronics, 38(1) (8(1):xviiixxxx)) 1992. doi: 125007
[10] Wilson Yan, Matei Zaharia, Volodymyr Mnih, Picter Abbeel, Aleksandra Faust, and Hao Liu. Elastictok: Adaptive tokenization for image and video. arXiv preprint, 2024.
[11] Shivam Duggal, Phillip Isola, Antonio Torralba, and William T. Freeman. Adaptive length image tokenization via recurrent allocation, 2024.
org/ wtgg/b//////e. ht////bs///1111/1 2411.1410 02393
[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[13] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[14] Tsung-Yl Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects context, 2015. URL httppp////aa org/ abs 1405.0312
[15] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV) December 2015.
[16] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL https://arxiv. org/ abs 2203. 10244
[17] Johannes Stallkamp. Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. The 201 International Joint Conference Neural Networks, pages 1453-1460, 2011. doi:
10.1119/ 10.1109/ICNN.201.603333 16013333335.

---

[18] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 1, 2011. URL wtt..atppp/uuulllllsaaaaaooo stw.:fldrrttafrrrrroooo stanford pdf
[19] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. SRRL.httss:/// If you use this software, please cite it as below.
org/ 52281/
zenodo. 5143773
[20] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09774, 2022.
[21] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018. URL pictps://arxiv.oot / 00937
[22] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11513-11522, 2022. URL hattps:/app
semanticscholar: org/ hetg/CorpusI:::200 :47244553
[23] Chuanxia Zheng, Long Tung Vuong, Jianfei Cai, and Dinh Q. Phung. Movq: Modulating quantized vectors for high-fidelity image generation. ArXiv, abs/2209.09002, 2022. URL httpps:/app..
semanti cechoaaatscccoaa...mmm orrcoooa....or///rrtiaD 252367709.
[24] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved yggan. ArXiv, abs/2110.04622 2021. URL https://api. http://ppp..emmaiccccchooooooo/o semanticscholar org/CorpusID: 228582265
[25] Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaiqi Huang. Efficient-vggan: Towards high-resolution image generation with efficient vision transformers. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7334-7343, 2023. URL hww.s: //api. deeemnticsccollee org/ Corp//orrusiD ::: 263830997
[26] T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the h.264/ave video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7): 560-576, 2003. doi: 10. 10.1199/CSVV.2203.88555
[27] Lei Chen, Zhan Tong, Yibing Song, Gangshan Wu, and Limin Wang. Efficient video action detection with token dropout and context refinement, 2023. URL bttppt:ss///rrrttt org/ abs/ 2304.08451
[28] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jic Zhou, and Cho-Jui Hsich. Dynam- ievit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[29] Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[30] Daniel Bolya, Cheng-Yaue Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmann Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.
[31] Junhong Shen and Lin F. Yang. Theoretically principled deep acceleration via nearest neighbor function approximation. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (11):9558-995) 9566. May 2021. doi: (0.1669/aaai.v35// 17711. lattss/////o aaai org/ index. index.php/AMe http:///w....aaa..eeee/ee//////a
[32] Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. Scriheagent: Towards specialized web agents using production-scale workflow data, 2024. URL http://///////eeeaeeeeeeee wrg. httgg//b//21111 15004
12

---

[33] Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver. Robert Geirhos, Ibrahim M. Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey A. Gritsenko, Mario Luvci' and Neil Houlsby. Patch n' pack: Navit, a vision transformer for any aspect ratio and resolution. ArXiv, 0ttp//377.066044 2023. htRpshttss://att semanticscholar. Peg/CorppuSDD 259837358
[34] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. CoRR, abs/2010.11929, 2020. URL hal.atttee//axxx.oo/////////////
11929
[35] Tomer Ronen, Omer Levy, and Avram Golbert. Vision transformers with mixed-resolution tokenization, 2023. URL https: wwtp://or////////.//////// 2304 00287
[36] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction, 2024. URL https://axxxii org/abs/2400 02900
[37] Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 22596-22605, 2023. URL stt.lptppp/ppp...seaaaccccooooo .rgg arg/CorpuIID- 258823089
[38] Kaiwen Zha, Lijun Yu, Alireza Fathi, David Ross, Cordelia Schmid, Dina Katabi, and Xiuye Gu. manguage-guidee image tokenization for generation, 2024. https:ts:://art org/ abs/2412.0579p
[39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. URL http:ttts///axiioorr//////////// 1tt.//555.00000 04597
[40] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Ma- tryoshka representation learning. In Advances Neural Information Processing Systems, December 2022.
[41] Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint 2735 2024.
[42] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh Susskind, and Navdeep Jaitly. Matryoshka diffusion models, 2024. http:/pss//aars...rr//////////1
wwwp:iiiwwrg//aaaaaaaeee abg/abs/2310
15111
[43] Charlie Nash, João Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Mateusz Malinowski. and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models, 2022. URL Setts://arxiin org/ abs 2200b/099299994 09494
[44] Junhong Shen, Liam Li, Lucio Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: align then refine. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[45] Junhong Shen, Tanya Marwah, and Ameet Talwalkar. Ups: Towards foundation models for pde solving via mooss- adaptation. arXiv preprint avxiv:2403.07/87, 2024.
[46] Nicholas Roberts, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, Minwen Deng, Johannes Hog, Alexander Pfefferle, Sushil Ammanaghatta Shivakumar, Arjun Krishnakumar, Yubo Wang, Rhea Sanjay Sukthanker, Frank Hutter, Euxhen Hasanaj. Tien-Dung Le, Mikhail Khodak, Yuriy Nevmy- vaka, Kashif Rasul, Frederic Sala, Anderson Schneider, Junhong Shen, and Evan R. Sparks. Automl decathlon: Diverse tasks, modern methods, and efficiency at scale. In Neural Infor- mation Processing Systems, 2021. URL. http::ttts:///....m :/api:/pppp semanticscholar. .orilarrrrrrr/oe orgg/CorpusID 265536645
13

---

[47] Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song- Chng-ChunnTu Zhu. Emergence of pragmatics from referential game between theory of mind agents, 2021. URL https
[48] Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-Ilm: Repurposing general-purpose IIms for specialized domains, 2024.
[49] Wenbo Hu, Zi-Y Yi Dou, Liumian Harold Li, Amita Kamath, Nanyun Peng, and Kai- Chang. Matryoshka query transformer for large vision-language models, 2024.
[50] Richard Zhang. Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[51] Stability AI. -tese. 2022.
[52] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019. doi: 10.1 10.1109/SS 201...992019
[53] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuming. 2023.
[54] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. URL 06500
[55] An Yang, ... and Zhihao Fan. Qwen2 technical report, 2024. URL https://
abs/ nos//240..15 10671
[56]
Abhimanyu Dubey, and Zhiwei Zhao. The Ilama 3 herd of models, 2024. URL https
'arxiv.org/ wwt.or/g//bb////////////// abs/2407 2221/22787222
[57] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[58] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL
03385
[59] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020. URL (ttps:/tt:///rrt /arxiv.xvvvv../ii org. abs///11 05722
[60] Ian J. Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL
wttp/org//aaabo/ooooese 2661
[61] Alain Horé and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pages 3366,2369, 2010. doi: 10. 109/CPR220100579
[62] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1 11305-11315, 2022. URL https://aps semanticscholar
Org/CorpusID ID: 246680316
[63] Xiaoliang Dai, Ji Hou, Chih Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing image generation models using photogenic needles in a haystack, 2023, URL https: /ttp///avvvv...///av/eeee/eeaa// org/ abs//309./222//./// 15807
14

---

[64] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014. doi: 10. 10.1 10111099...0.14..0.
[65] Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. Int. J. Comput. Vision, 119(1): 3-22, August 2016. ISSN 0920-5691. doi: 10.1007/s11263-1144 //doi. oxg/ 1007/ 111-11266--1--788---1-
URL https
[66] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium, 2018. URL
https://arxis:
htsp/bbs/17000 08500
[67] Xin Ding, Yongwei Wang. Zuheng Xu, William J. Welch, and Z. Jane Wang. Continuous conditional generative adversarial networks: Novel empirical losses and label input mechanisms, 2023. URL httpt: org/ abs//bbsl...
[68] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016. URL https://axxxi org abs/ 11/1.61666000 03498
[69] Tuomas Kynkäääniemmi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models, 2019. UtLpttps::/ org/
abs/1904. 06991
[70] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL https:// arxiv. org
12598
[71] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019. URL pesttss//arrtae http//axxvvvg///aaa
[72] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128 1181 March 2020. ISSN 1573-1405. doi: (/007/s11233-20-0000 http://dx. doi. org/ 10. 10..10000s12666 00000---220--001060000
URL
[73] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
[74] Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, and Mikhail Khodak. Specialized foundation models struggle to beat supervised baselines, 2024. URL htRts:tss:/// //tp://///iv.v.a//aa www.aarivvoorgg/bee. wtt//br/////a//eee/o 02796.
[75] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Korbblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. ArXiv, abw//aos..ooaa 055592, 2022.
15

---

the beach A Broader Impact
This work introduces CAT, content-adaptive tokenizer designed improve the efficiency and flexibility of image modeling. By enabling dynamic compression based on image content, CAT has the potential to reduce computational and energy costs in large-scale generative and classification systems, contributing to more sustainable Al practices. However, as with any generative model, misuse of this technology for creating misleading or harmful content remains concern. We encourage responsible deployment, including proper safeguards and usage guidelines, particularly in sensitive domains such as media generation, surveillance, or automated decision-making.
B
Prompt for LLM Scorer
Our caption complexity pipeline works as follows:
Step 1: Use VLM to generate image description, with the following prompts:
What's 's in the image? - Caption
Are there text or numbers in the image? - Yes/No.
Are there faces in the image? + Yes/No.
Step 2: Use the same VLM or separate LLM to generate the complexity score with the prompt:
Given the description of a 512px image, determine its complexity based on the following
factors:
1. Number of distinct objects
Color variance
3. Texture complexity
4. Foreground and background
5. Symmetry and repetition
Human perception factors, like the presence of human faces or text
You will be given the caption, whether there are text or numbers, and whether there are faces in the image. Assign a complexity score such that a higher number means the image is more complex. Note that text and facial details are intrinsically complex because they are crucial to human perception. Here are some examples for scoring:
Score 1: A plane in sky
Score 2: t-shirt with a emoji on
Score 3: A dog lying on the grass
Score 4: woman skiing in the snow
Score 5; Two kids walking
Score 6: dinning table full of food
Score 7: A close-up shot of old man
Score 8: Many people gathering in the stadium
Score 9: Newspapers graphs with text and numbers
Now determine the complexity for the caption:
Insert caption here]
Insert one of the following based on the Yes/No questions:
There are text visible in the image. There are also facial details.
There are text visible in the image, but there is no human face.
There is obvious text in the image, but there are facial details.
There is no text or human face in the image. ] Respond with "Score: explanations.
of 9", where "q" is a number between and 9, Then provide
Note these two steps can be combined into a single inference call.
C Other Complexity Metrics

---

Table We also tried various other metrics. However, they are less effective (e.g., caption length) compared to our LLM-based score.
Metric
Pearson
JPEG MSE LPIPS
0.31
0.36
0.23
Caption Length CAT (Ours)
0.33
0.55
D Compression Ratio Distributions with Different LLMs
Table 10: To study whether our caption score is robust to LLM choice, we test multiple captioners
and LLMs for scoring. The fact that these combinations generate similar and compression ratio distributions show that our scoring method is robust.
Scoring Model
Avg
InstruciBLIP Llama3. I-8B --BBInnstuct
6%
49%
45%
17.43
ImageNet
InstructBLIP Dween25577-llosttee
28%
LLaVA1.5 7B
29
65%
13%
17.75
LLaVAI.5 7B (Longer Caption)
3%
62%
35%
17.58
InstructBLIP Llama3.1-8D Instruct
CelebA
InstrucBLLIP Qwwen.5-7B------- I-B-inuuuutt
15%
70%
15%
13.83
LLaVA 7B
5%
95%
0%
14.92
LLaVA1.5 (Longer Caption)
InstrucBLLIP (ammi,1--8--nmee
0%
40%
60%
21.57
InstretBBL. Qwen2.5.7B-
LLaVA15 7B
0ri
41%
59%
21.42
LLaVAI.5 7B Caption)
2%
44%
54%
19.76
InstructLLIP Llamm3,1-88 1-88-iiu-tB
3%
30%
67%
20.87
EuroSAT
InstrueBLLPP Qwen2.5.78 InI-nnsttect
39
255
72%
21.57
LLaVAL 7B (Longer Caption)
4%
2.3%
72%
21.19
InstruciBLIP Llama3,1-SB Inst-nruuct
6%
73%
21%
15.82
InstructBL. Oween..--7-Innsrree
78%
20%
LLAVALS 1..77 7B
3%
80%
17%
16.30
LLaVA1.5 (Longer Caption)
E Reconstruction Experiments
E.1
Architecture
We implement the nested VAE similar to the AutoencoderKL implementation in the diffusers library. The network configuration is:
sample_sin _size: 512
in_ In_channels:
out_channels: 3
down down_block__ypeet [DommEncoderBlockk]]] 6
up_ bopbboockttpess [UDDeco x
block_out_channell [64, 128, 256, 256, 512, 512)
layers_per_block: 2
act_fn: silu
latent_chammels: 4/8/16
norm_num_groups 32
mid_block_attention head_dimi dim:
num_layers: 8
The model sizes for different latent channels are shown below. For the discriminator, we use the pretrained StyleGAN 7

---

Nested VAE
4
8
16
# Params (M)
187.45
187.50
187.61
E.2
Training
We use obj E (human faces, text) for our perception-focusee queries. We use the following training configuration:
GPU: 64 NVIDIA A100
Per-GPU batch size: 8
Global batch size: 512
Training steps: 1,000,000
Optimizer: AdamW
1r: 0.0001
betal: 0.9
beta2: 0.95
weight_decay: 0.1
epsilon: 1e-8
gradient_clip: 5.0
Scheduler: constant with 10,000 warmup steps
Loss:
recon_loss_wet weightt 1.0
k1_loss_weight. le-6
1.0
moco moco__osss_weihht: 0.2
_an_loss_weighe 0.5
gan
[ea__loss_starting.see 50,000
The discriminator is trained with the standard GAN loss.
E.3
Baselines
We train fixed compression baselines using the same data, train- ing configuration. and VAE backbone. For smaller compression ratios, c.g., fixed 8x, the last two downsampling blocks and first two upsampling blocks are not used.
For the adaptive JPEG baseline, we use torchvi- sion.io.encode __peepppeeegeepe to transform the images into JPEG file and compute the number of bytes as the complexity metric. Smaller files correspond to larger complexity. To provide a better understanding of this metric, we show Figure 6 the distribution of JPEG sizes on the COCO 2014 test set, with relevant statistics included the caption. Then, based on the JPEG sizes of all images in the Shutterstock training dataset, we set the thresholds (a, b) to (38761, 65837) to categorize the file sizes into three compression ratios. This set of thresholds ensure that the JPEG baseline has the same training compression ratio distribution as CAT.
6002
10000
12000
Figure 6: On COCO 2014 test set, the minimum JPEG size 6128: maximum is 118428; mean is 45474.29; standard deviation is 15037.07.
For LDM VAEs, we follow the instructions in their original repository to use the model checkpoints. Note that LDM VAES are trained on OpenImages dataset (72), which is different from our training data, so hard to fairly compare reconstruction performance. Nonetheless, present their rFIDs on the evaluation datasets in Table [1

---

Table 11: rFIDs for CAT and LDM VAEs.
COCO
ImageNet
CelebA
ChariQA
CAT
0,65
0.46
1.97
5.27
LDM 8x
0.51
0.33
2.83
8.32
LDM 16x
0.53
0.37
3.07
8.49
LDM 32x
0.90
0.62
5.54
10.35
E,4 More Reconstruction Visualization
See Figure 7 in the next page.

---

E,5 Fixing Token Count for CAT
Table 12: Equalizing test- time token counts.
We also evaluate the reconstruction performance under fixed compres- sion ratio (token count) for different datasets. Table [20opp/acemmrrr the reconstruction FID for CAT with vs. fixed 16x compression ratio. We see that adaptive compression based on image complexity outperforms uniform compression using the same architecture in most cases, possibly because error reduction on complex images outweighs the slight error increase on simpler
compression ratio
CAT #FID
Adaptive
Equiliedd 16x
COCO
0.65
0.67
ImageNet
CelebA
197
2.47
ChanQA
5.27
7.27
ones.

---

Oricinal
and Aoentran rél pertiarre Aatehfann Meseeieree val Nariaar Bubee
Figure 7: More CAT reconstruction examples. We highlight the compression ratio selected by our caption complexity in red.
E.6 Full Results for Table Table
To complement Table include the training distribution for different scoring thresholds in
Table 13
To complement Table 5] we include the results of fixed-ratio baseline in Table 14.
Table 13: Compression ratio distribution affects learning outcomes. Both settings have average compression of 16x,, but (4, 7) leads to better distribution diversity and empirical results.
Training Distsbution
Reconaraction FID
5x
16%
32%
Averigg
COCO
ImsgeNei
CdibA
CharQA
0.46
7.70

---

Table 14: Larger latent channel generally improves rFID.
(IDI
COCO
ImageNet
ColehA
ChirQA
1,25
1.32
5.89
9:45
Fixed 16x
1.10
0.61
4.99
8.19
CAT
1,03
0.60
4.54
7.95
16
0.65
0.46
1.97
5.27
F
Classification Experiments
We selected the diverse datasets because they represent out-of-domain distributions where zero-shot models perform poorly 753. Similar to OpenClip [9), initialize simple linear classification head that maps from the tokenizer's maximum latent dimension (under 8x compression) to the number of labels for each task. When the image is compressed with 16x 32x ratio, we zero-pad the latent features to make the length match with the classification head. We evaluate two settings: (1) linear probing, where keep the image encoder frozen and only fine-tune the classification head; (2) full fine-tuning, where we update both the encoder and the classification head. We train both settings for 20 epochs with batch size 64, learning rate le-4 and cosine annealing learning rate schedule with 2 warm-up epochs. We use the AdamW optimizer with weight decay 0.1.
G Generation Experiments
G.1 Architecture
We use DiT-XL architecture with a patchify downsampler and patch size of 2. The model size depends on the latent channel, but generally around 431M parameters. The model TFLOPs 22.0. All baselines reported in Table 7 use = 16.
G.2 Training & Inference
We LLaVA1.5 7B to generate the complexity for ImageNet training images. For 10 of the time, we remove the image class label from the input and train unconditional image generation. The training configuration for DiT is:
GPU: 16 NVIDIA H100
Per-GPU GPU token batch size: 4096 x (equivalent to 64 images for 16x compression ratio)
Global token batch size: 4096 x
Training steps: 400,000
Optimizer: AdamW
1r: 0.0001
betal: 0.9
beta2: 0.95
weight_decan 0.1
epsilon: 1e-8
gradient_clipi 1.0
Scheduler: Cosine
warmup: 4000
cosine_ theta: 1.0
cycle_leeggeee 1.0
Ir wrwminnratioo 0.05
At test time, we use "this is image of [label]' as standardized prompt and manually provide answers the queries to enable automated evaluation. Then, we use Llama3. to obtain the complexity score. DDPM scheduler (diffusers implementation) configuration is:

---

tum_train_timostoos 1000
beta_start: 0.0001
beta_end: 0.02
beta_schedule: squaredcos_cap_VV
predictionn_ype: epsilon
timestepsspacing: leading
num www.inneee_ssteeeeeee steps: 250
All FID-50K and images generated in this paper are using cfg=1.5.
G.3
Baselines
To ensure we train the baseline with the same compute FLOPs, we fix the token batch size and number of training steps for all settings. For pretrained LDM VAE, the scaling factor specified in the model configuration to ensure the input scale and noise scale are similar. For CAT, we use scaling factor of 1.
G.4
More Visualization
See Figure 8
in the end.
G.5 Full Results for Table
To complement Table 8 we include the results of fixed-ratio baseline in Table 15
Table 15: Larger channel is not always better for generation. Contrary to Table we find that increasing channel dimension does not always result in generation gains.
.FID
Precison"
Recall
5.11
10.34
155.8
0.75
0.89
Fixed 15x
1.95
10.39
221.85
0.76
0.51
16 4.78
11.81
157.47
0.72
0.89
5.12
11:12
153.30
0.72
0.45
CAT
4.38
10.31
181.03
0.76
0.45
16
4.56
10.55
191.09
0.75
0.49

---

1024
256
Figure 8: More DiT-CAT generation examples. Increasing token count (left->righ) generally leads better image quality and higher complexity.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes), [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).

---

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] prooidddd proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our contribution is to propose a dynamic tokenizer based on image complexity. These reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See the Conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
10

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
There is thcoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all training and inference configurations, hyperprameters, LLM prompts in the main text and appendix. We also include the code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case
11

---

of a large language model), releasing of a model checkpoint, or other means that are
appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We use open-source benchmarks and release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/niis..c))
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wtw.oodesssmisssooooiissioo
/ //nips.cc/publee httpc//ppwww.pppe/e//eeeeeeeee/e
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, see the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
12

---

The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report numbers for one trail due to the cost associated with each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, specify the GPUs used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::///aurrrp..ppppp/t//taaaaa
Answer: [Yes]
Justification:
The paper follows the NeurIPS Code of Ethics.
Guidelines:
13

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Broader Impact section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., dep.oymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do use data or train models with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
14

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpppppppsataeea thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
15

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We discussed how prompt LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM for what should or should not be described.
16


Reviews:
[{"review_id": "mznE5WDiPG", "reviewer": "Reviewer_BLmN", "summary": "The paper proposes Content-Adaptive Tokenizer (CAT), which dynamically adjusts token counts for images based on content complexity. It uses LLMs to evaluate complexity via captions and a nested VAE for variable-rate compression. CAT reduces token usage by 18% on natural images, improves rFID by 15% on complex datasets, boosts classification accuracy, and enhances generation throughput by 23%. It integrates language-guided evaluation with adaptive architecture for efficient, high-quality image representation.\nThe key components include: (1) an LLM-based system that analyzes image captions and perceptual queries (e.g., presence of text/ faces) to assign optimal compression ratios (8x, 16x, 32x); (2) a nested VAE that routes intermediate features to shared latent blocks, enabling variable-rate compression in a single model. This design allows CAT to encode simpler images with fewer tokens while preserving details in complex ones.", "questions": "Although CAT reduces token consumption, the LLM-based complexity evaluation adds an extra inference step. How does this affect computational budget? Please explain in combination with the experimental data.\n\nThe significance of Adaptive Image Tokenization lies in enabling a flexible trade-off between computational cost and output quality through adaptive token quantity. However, the article does not provide a comparison of computational overhead for CAT, including comparisons with fixed compression ratio methods and existing adaptive tokenization approaches in terms of inference time, video memory occupation, etc.\n\nWhat is the advantage of CAT over Fix 8x in terms of computational overhead, and what additional benefits does it offer? As can be seen from Table 3, Fix 8x significantly outperforms CAT in accuracy. Is the computational overhead advantage of CAT large enough to justify sacrificing Fix 8x's accuracy edge for adopting CAT?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.Combines LLMs with a nested VAE to dynamically adjust token counts based on content complexity inferred from text descriptions, addressing inefficiencies of fixed-token approaches. This allows simpler images to be encoded with fewer tokens while preserving details in complex ones. \n2.The experiments are comprehensive, including comparisons of max acceptable compression ratios, performance comparisons across multiple datasets, ablation studies on model hyperparameters, and ablation studies on LLM models.\n3.Reduces token usage by 18% on natural image datasets (e.g., ImageNet, COCO) while maintaining high reconstruction quality. Achieves an average 15% reduction in rFID on detail-rich datasets, outperforming fixed baselines in preserving fine details like text and faces. \n4.Improves top-1 accuracy in image classification across diverse datasets and boosts training throughput by 23% in text-to-image generation. CAT’s adaptive representations enhance both efficiency and performance in reconstruction, classification, and generation. \n\nWeaknesses\nThe core innovation of the work lies in using prompt-based image complexity prediction from LLMs. However, the design of the tokenizer still adopts a traditional architecture, showing insufficient innovativeness.\nIn the Table 7 experiments related to Image Generation, add a comparison with DiT+Fixed 8x.\nOther questions can be found in the Questions section.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns. While the computational cost of LLM seems too expensive, I think the paper makes an important contribution, and decided to keep my score.", "rebuttal": "Thank you for your thoughtful review and for recognizing the significance for content-aware tokenization. To address your questions regarding baselines, **we add new results for DiT + fixed 8x tokenizer on ImageNet generation. We also update the paper with detailed computational costs analysis**. Please find our response below. \n\n---\n## Weakness 2: Adding Baseline\n\n>  In the Table 7 experiments related to Image Generation, add a comparison with DiT+Fixed 8x. \n\nThank you for pointing this out! We have previously run DiT + fixed 8x (400K training steps) as a baseline. We did not include it in the main text because the **comparison might be unfair---fixed 8x uses 1024 tokens per image, which is nearly 5 times the number of tokens used by CAT (~197 tokens per image)**. However, for reference purposes, we will still update Table 7 to include the following DiT + fixed 8x baseline:\n\n|ImageNet Generation | Tokenizer Compression Ratio | Avg. Tokens | FID↓ |sFID↓ | IS↑  | Precision↑ | Recall↑| Eval FLOPs↓|\n|-|-|--|-|-|-|--|-|--|\n| DiT + Fixed 8x | 8x | 1024  | 4.24 |8.64|204.88 | 0.75  | 0.57     | 4x     |\n| DiT + Fixed 16x | 16x | 256 | 5.89  |11.81| 187.47| 0.72  | 0.49     | 1x   |\n| DiT + CAT (Ours) | 8x / 16x / 32x | 197.44 | 4.56|10.55  | 191.09 | 0.75     | 0.49 | 0.82x    |\n\nDespite using significantly fewer tokens, CAT achieves similar FID (4.56 vs. 4.24) to DiT+fixed 8x. Thus, CAT offers a better efficiency-performance trade-off than both fixed 16x and fixed 8x setups.\n\n---\n\n## Question 1: LLM Evaluation Overhead\n\n> Although CAT reduces token consumption, the LLM-based complexity evaluation adds an extra inference step. How does this affect computational budget? Please explain in combination with the experimental data.\n\nAs noted on L174-L177, we tested two pipelines for complexity evaluation: (1) a unified pipeline using LLaVA1.5 7B for both captioning and scoring in a single pass, and (2) a separated pipeline using InstructBLIP for captioning and Qwen2.5 7B/LLaMA3.1 8B for scoring. Both pipelines achieve similar complexity score distribution due to our carefully designed scoring rubric. Please find the compute analysis on 512x512 images below:\n| Task       | Model Type | # of Model Calls (per image) |  Avg. Wallclock Time on H100 (per image) |\n|-------------|-------------|-------------|----------------------------------|\n| Reconstruction (Image Data Only)    | VLM | 1           | ~1.63 s                         | \n| Reconstruction (Image Data Only)    | Captioner + LLM  | 2           | ~1.84s        |               \n| Generation (Text Data Only)    | LLM | 1           | ~1 s                         |\n\nIn short, the complexity evaluation step  involves a single forward pass through a VLM for reconstruction tasks and a single forward pass through a LLM for generation tasks. At training time, this adds <1% to the total compute budget across 1M steps. At inference time, the average wall-clock time per image is ~1s on NVIDIA H100. When processed in batches, this cost is further amortized, making the overhead negligible relative to the downstream model cost.\n\nWe also note that, as mentioned in Section 6, users can specify token counts manually during generation, which entirely eliminates the need for LLM calls. Reviewer mnGd noted this flexibility: CAT *\"allows users to specify the desired token count at inference time, enabling a flexible trade-off between computational cost and output quality.\"* Overall, the computational benefit from reduced token usage significantly outweighs the minimal overhead from LLM-based scoring.\n\n---\n\n## Question 2: Compute Comparison\n\n> The significance of Adaptive Image Tokenization lies in enabling a flexible trade-off between computational cost and output quality through adaptive token quantity. However, the article does not provide a comparison of computational overhead for CAT, including comparisons with fixed compression ratio methods and existing adaptive tokenization approaches in terms of inference time, video memory occupation, etc.\n\nWe appreciate the reviewer’s suggestion, so here is a detailed computational overhead comparison on 512x512 images.\n\n**Encoding Task**\n| Aspect                  | CAT                                     | Fixed 16x    Baseline        | Fixed 8x  Baseline            | Fixed 32x Baseline           |\n|-------------------------|------------------------------------------|------------------------------|-------------------------------|------------------------------|\n|Adaptive Encoding|✓|x|x|x|\n| Param Count             | 187M       | 169M|    147M            |                   187M |\n| Latent Dim        | Adaptive (avg. 29 on ImageNet)     |  32                  | 64                  | 16                   |\n| Inference Time          | LLM eval (~50ms) + VAE encoding (adaptive) | VAE encoding only            | VAE encoding only  | VAE encoding only  |\n\nWe note that existing adaptive tokenization methods, such as ElasticTok (210M parameters) and FlexTok (286.7M parameters), support multiple compression ratios, but do not include a mechanism for deciding which ratio to apply to a given image. In contrast, CAT provides an explicit LLM-based complexity evaluation system that determines the appropriate compression level per image. As a result, it is difficult to measure or compare their effective compression behavior on a dataset without retraining or manually selecting ratios.\n\n**Generation Task**\n| Aspect                  | DiT + CAT                                     | DiT + Fixed 16x Baseline          | DiT + Fixed 8x Baseline            | DiT + Fixed 32x Baseline          |\n|-------------------------|------------------------------------------|------------------------------|-------------------------------|------------------------------|\n| Token Count             | Adaptive (avg. 197 tokens on ImageNet)     | 256 tokens                   | 1024 tokens                   | 64 tokens                    |\n| Memory Usage            | Lower (due to fewer tokens)               | Medium                       | Higher                       | Lowest                      |\n| Generation FLOPs   | ~0.82x              | 1x            |~4x                       | ~0.25x                       |\n \nWe see that CAT generally reduces token count and thus improves the efficiency over fixed ratio baselines. Moreover, unlike many existing adaptive tokenizers that require an image input for token count estimation, CAT’s text-only evaluation allows it to adapt compression even in generative settings without access to ground-truth images. As CAT’s tokenizer also  requires a single model, we believe our work offers distinct computational advantages over existing approaches for adaptive tokenization.\n \n---\n\n## Question 3: Trade-Off between Quality and Compute\n\n> What is the advantage of CAT over Fix 8x in terms of computational overhead, and what additional benefits does it offer? As can be seen from Table 3, Fix 8x significantly outperforms CAT in accuracy. Is the computational overhead advantage of CAT large enough to justify sacrificing Fix 8x's accuracy edge for adopting CAT?\n\nWe first note that **on complex datasets such as ChartQA, our rFID (5.27) is actually better than the fixed 8x baseline (8.21)**. For natural image datasets like COCO and ImageNet, we clarify the comparison below:\n| Metric                  | Fixed 8x          | CAT (Adaptive)           | Computational advantage of CAT|\n|-------------------------|-------------------|---------------------------|---------|\n| COCO rFID ↓      | 0.48              | 0.65                      | -|\n| ImageNet rFID ↓   | 0.24              | 0.46                      | -|\n| Avg. Compression (COCO) | 8x (Fixed)       | 16.07x (Adaptive)         | ~2x|\n| Avg. Compression (ImageNet) | 8x (Fixed)   | 17.46x (Adaptive)         | ~2.81x|\n| Latent Dim (COCO) | 64x64               | 31x31                      | ~4.26x|\n|Latent Dim (ImageNet) | 64x64            | 29x29                      | ~4.87x|\n\nWhile fixed 8x achieves better reconstruction metrics on natural image datasets, **CAT is 4-5x more compute efficient**. This is because CAT preserves quality where it matters, such as images with fine details like facial features or text. Thus, while fixed 8x offers slightly stronger quality, CAT delivers selective quality at significantly lower computational cost, making it a more efficient and scalable solution. Thus, the key advantage of CAT lies in its adaptivity.\n\n---\n\n## Weakness 1: Tokenizer Architecture Novelty\n\n> The core innovation of the work lies in using prompt-based image complexity prediction from LLMs. However, the design of the tokenizer still adopts a traditional architecture, showing insufficient innovativeness.\n\nThank you for noting our contribution in proposing LLM-based complexity evaluation. However, we believe that the nested VAE architecture used by CAT (Fig. 1) differs from existing VAE architectures in important ways:\n- Skip connections with channel matching: Our design routes intermediate encoder features to shared latent blocks, enabling multiple compression ratios in a single forward pass\n- Shared Gaussian parametrization: The shared middle block maintains scale consistency across different compression levels while processing variable spatial dimensions\n- Parameter sharing: We strategically increase parameters in shared modules to handle multi-scale features effectively\n\nCompared with traditional tokenizer architectures that support only fixed compression and need separate models for different ratios, our nested VAE is a single model that supports multiple compression levels, enabling adaptive compression without the storage and computational overhead of multiple models. This shows our architectural novelty."}, {"review_id": "Ym6PQvdphB", "reviewer": "Reviewer_mS49", "summary": "This paper proposes an adaptive image tokenization strategy, named CAT, that varies the number of tokens to represent an image based on its complexity. For measuring the complexity, they leverage LLMs using captioning and VQA data extracted from the image, and assign a complexity score to the image. A nested VAE architecture is used to represent images with variable compression rates (three compression rates are used: 8x, 16x, 32x). Evaluations are performed on image reconstruction, generation, and classification tasks to show the effectiveness of the approach.", "questions": "- Is MSE a good metric to measure complexity? If not, what is the main take away from Sec. 3.1? \n\n- How do we ensure that the proposed complexity measurement is well-calibrated and better aligned with human preferences?\n\n- What is the generation performance in other benchmarks, e.g. COCO?\n\n- How do you compare against the relevant baselines listed above?\n\n- Will the code and models be released?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Developing adaptive visual tokenizers is an interesting and promising research direction. I also find the papers focus on developing a better measure of image complexity intriguing. The method is well-motivated and explained, and the empirical results seem promising. Overall, I am leaning towards accepting this paper, but there are a few points that require further clarifications:\n\n- I found the first part of Sec. 3.1. a bit confusing. Exactly because of the reason in L118, is MSE a good metric to measure complexity? If not, what is the take away here? \n\n- LLM-based complexity measure proposed in this paper can be a potential solution, however, how do we ensure that it's well-calibrated and better aligned with human preferences? A user study could be useful here. I found the correlation study in Table 2 not fully conclusive.\n\n- I would be curious to see the generation performance on other popular datasets, e.g. COCO.\n\n- Finally, I suggest including comparisons to relevant baselines. Both FlexTok [A] and Semanticist [B] are recent adaptive tokenization strategies that are missing in the comparisons here. While not adaptive, TiTok [C] is also a relevant and recent baseline that can compress images quite substantially. Having these comparisons would be helpful to better understand the effectiveness of the method.\n\nI am happy to hear authors' opinions on these.\n\n[A] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length\n\n[B] “Principal Components” Enable A New Language of Images\n\n[C] An Image is Worth 32 Tokens for Reconstruction and Generation", "was_revised": true, "final_justification": "Thanks authors for their response addressing most of my concerns. I am happy to keep my rating as accept.", "rebuttal": "Thank you for the positive review. We’re encouraged by your statement that “The method is well-motivated and explained, and the empirical results seem promising.” To address your question around generation dataset and related work, **we add new COCO generation comparison and expand the related work section.** Below are detailed responses to your specific questions.\n\n\n---\n\n## Weakness 1 & Question 1: Clarification on MSE\n\n>  I found the first part of Sec. 3.1. a bit confusing. Exactly because of the reason in L118, is MSE a good metric to measure complexity? If not, what is the take away here? \n\n> Is MSE a good metric to measure complexity? If not, what is the main take away from Sec. 3.1?\n\nThanks for asking about our MSE analysis. To clarify: MSE is **not** a good metric to measure perceptual complexity, which is precisely our main takeaway from Section 3.1.\n\nThe proof-of-concept analysis in Section 3.1 serves two purposes. First, we want to demonstrate the potential for adaptive compression: We show that 56% of images can be compressed to 16x with negligible MSE increase, so significant computational savings are possible. Second, we want to highlight the limitations of existing metrics: we show that MSE, JPEG size, and LPIPS all have low Pearson correlations (r ≤ 0.36) with acceptable compression ratio. Besides, images with fine-grained elements (text, faces) may have low MSE but poor visual quality at high compression. Thus, existing metrics can fail to capture perceptual complexity.This motivates our LLM-based approach that considers semantic and perceptual factors rather than pixel-level differences.\n\nWe will clarify this interpretation in the revision to make the takeaway more explicit.\n\n---\n\n## Weakness 2 & Question 2: Calibrating Complexity Score\n\n> LLM-based complexity measure proposed in this paper can be a potential solution, however, how do we ensure that it's well-calibrated and better aligned with human preferences? A user study could be useful here.  \n\n> How do we ensure that the proposed complexity measurement is well-calibrated and better aligned with human preferences?\n\nThank you for asking this. We use a carefully designed scoring rubric (Appendix B) along with in-context examples to ensure the robustness of our evaluation pipeline. In the paper, we have provided several pieces of evidence to show the quality of our complexity measure:\n\n**Quantitatively,** we show that LLM-based scores achieve the highest Pearson r = 0.55 and 62% agreement with the “ideal” compression ratio. We also conducted robustness ablations across LLMs and captioning styles (Appendix Table 10), finding little variation. The strong downstream performance of CAT also shows  the effectiveness of  our LLM-based complexity evaluation  system.\n\n**Qualitatively,** manual inspection confirms perceptually challenging images receive high complexity scores. Examples in Figure 4 also align with intuitive expectations.\n\nWe agree that a user study would be a valuable next step to validate alignment with human perception, and we plan to pursue this in future work. However, conducting such a study requires IRB approval and annotator recruitment, which is not feasible within the rebuttal period. In the meantime, we hope that the above evidence addresses the reviewer’s concerns.\n\n\n---\n\n## Weakness 3 & Question 3: COCO Results\n\n> I would be curious to see the generation performance on other popular datasets, e.g. COCO. \n\n> What is the generation performance in other benchmarks, e.g. COCO?\n\nThank you for this helpful suggestion. To address this, we directly evaluated the ImageNet-trained DiT + CAT   on COCO image generation. We report FID-5K by generating 512x512 images in a zero-shot fashion on 5k prompts from the MS-COCO 2017 validation set. Other diffusion configurations are kept the same as the ImageNet experiment. Note that we do not retrain from scratch using the COCO training data because of the limited time for rebuttal. We will leave retraining for future work.\n\n| Model              | Avg. Tokens | FID ↓ |  \n|--------------------|-------------|---|\n| DiT + LDM VAE    | 256         |   27.31   |\n| DiT + Fixed 16x    | 256         |      26.6     |\n| DiT + CAT (Ours)   | **212**        |    **23.57**   |\n\nWe find that CAT continues to outperform fixed-token baselines, maintaining strong performance while using significantly fewer tokens on average. This result reaffirms our method’s efficacy. We will include full COCO generation results in the final version.\n \n\n \n---\n\n## Weakness 4 & Question 4: Related Work\n\n> Finally, I suggest including comparisons to relevant baselines. Both FlexTok [A] and Semanticist [B] are recent adaptive tokenization strategies that are missing in the comparisons here. While not adaptive, TiTok [C] is also a relevant and recent baseline that can compress images quite substantially. Having these comparisons would be helpful to better understand the effectiveness of the method. / How do you compare against the relevant baselines listed above?\n\n> How do you compare against the relevant baselines listed above?\n\nWe appreciate these pointers. Here's our perspective on the baselines you mentioned:\n\n**FlexTok and Semanticist:** These methods focus on 1D sequence representation. Our work is complementary as it addresses content-adaptive compression in VAE-based 2D tokenizers. In terms of evaluation, these methods do not report results on the datasets used in our paper, such as ImageNet, COCO, or ChartQA. While we are happy to include direct comparisons in the final version, doing so fairly would require retraining their tokenizers at comparable model scale and on the same training datasets as CAT. This is a nontrivial effort that is unfortunately infeasible within the rebuttal period.\n\n**TiTok:** While TiTok achieves impressive compression, it uses a fixed tokenization strategy. We actually include TiTok in our comparison (Table 2) where our adaptive approach (0.46 rFID) outperforms TiTok-B-128 (1.52 rFID) on ImageNet-512.\n\nTo sum up, CAT’s strong results are precisely due to the integration of LM-based complexity evaluation, which none of the existing works study. We will add the above discussion and relevant works to the paper.\n\n---\n\n## Question 5: Code Release\n\n> Will the code and models be released?\n\nYes. We already include our training code in the supplementary material and plan full public release later."}, {"review_id": "9HzJieTlmH", "reviewer": "Reviewer_mnGd", "summary": "This paper introduces the Content-Adaptive Tokenizer (CAT), a novel approach for image tokenization that dynamically adjusts the number of tokens (with predefined 3 scales) based on image complexity derived from text descriptions. Traditional image tokenizers use fixed compression ratios, which are inefficient for simple images and insufficient for complex ones. CAT addresses this by combining a caption-driven complexity evaluator (using LLMs) and a nested Variational Autoencoder (VAE) architecture. Experiments on different datasets and different tasks show the effectiveness of the proposed CAT. However, may major concern is that that the performance gap comes from the help of LLM, which is not used in most of the evaluated methods.", "questions": "1.\tDirectly use a VLM as a scorer to produce the compression ratio without using an LLM. The impact of using and not using LLM should be discussed.\n\n2.\tIn Table 3, the proposed adaptive CAT outperforms the other 16x methods. Are the number of parameters of adaptive CAT and the fixed 16x baseline model the same? The number of parameters and FLOPs are better to be provided in the table.\n\n3.\tWhile evaluating on ChartQA dataset, rFID is not suitable since it is mainly computed by Convolution Neural Networks, which are not dedicated for texts. It is suggested to evaluate the reconstructed text images with some OCR methods, which are designed to recognize the texts in images. \n\n4.\tWhile evaluating CAT on the classification downstream task, the classification head takes the 8x latent features as input. However, the classification heads in evaluated methods take the 16x latent features as input, which means the number of parameters of the classification head in CAT is more than the parameters of other classification heads in the evaluated methods, resulting in an unfair comparison.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper presents the Content-Adaptive Tokenizer (CAT) to dynamically adjusts token counts for images based on text-derived complexity via an LLM evaluator and a nested VAE. Its strengths include innovative adaptive framework that improves reconstruction quality for complex images while reducing token usage on natural images, strong empirical validation across several datasets and different downstream tasks (classification, generation), and robust LLM-driven complexity assessment. \n\nThe main weaknesses involve limited compression flexibility (fixed 8x/16x/32x ratios), reliance on text descriptions that results in more complex usage of the VAE, added LLM inference overhead, and insufficient discussion of LLM biases or environmental impacts in broader implications.", "was_revised": true, "final_justification": "Thank the authors' response. Most of my concerns are solved. I decide to keep my rating as 4 (Boardline accept).", "rebuttal": "Thank you for recognizing CAT’s contributions, including its “robust LLM-driven complexity assessment,” “innovative adaptive framework,” and strong performance across diverse tasks. To address your questions on evaluation, **we add new results regarding ChartQA OCR accuracy. We also perform additional analysis on compute costs.** Please find our detailed response below.\n\n---\n\n## Question 1\n\n> Directly use a VLM as a scorer to produce the compression ratio without using an LLM. The impact of using and not using LLM should be discussed.\n\n\nThis is an excellent idea that we actually explored in our work. As detailed in L174-L177, we evaluated both VLM + LLM pipelines (e.g., InstructBLIP for captioning followed by Qwen 2.5/Llama 3 for scoring) and VLM-only pipelines (e.g., LLaVA-1.5, which handles both captioning and complexity scoring in a single pass). The compute analysis on 512x512 images is shown below:\n\n| Task | Model Type | # of Model Calls (per image) | Avg. Wallclock Time on H100 (per image) |\n|-|--|----|---|\n| Reconstruction (Image Data Only)    | VLM | 1  | ~1.63s  | \n| Reconstruction (Image Data Only)    | Captioner + LLM  | 2 | ~1.84s |               \n| Generation (Text Data Only)    | LLM | 1 | ~1 s  |\n\nIn short, the complexity evaluation step  involves a single forward pass through a VLM for reconstruction tasks and a single forward pass through a LLM for generation tasks. At training time, this adds <1% to the total compute budget across 1M steps. At inference time, the average wall-clock time per image is ~1s on NVIDIA H100. When processed in batches, this cost is further amortized, making the overhead negligible relative to the downstream model cost.\n\nWe also note that, as mentioned in Section 6, users can specify token counts manually during generation, which eliminates the need for LLM calls. Overall, the computational benefit from reduced token usage significantly outweighs the minimal overhead from LLM-based scoring.\n\n---\n## Question 2\n\n> In Table 3, the proposed adaptive CAT outperforms the other 16x methods. Are the number of parameters of adaptive CAT and the fixed 16x baseline model the same? The number of parameters and FLOPs are better to be provided in the table.\n\nWe provide the model size and FLOPs of CAT and baselines below:\n| Aspect  | CAT  | Fixed 16x    Baseline    | Fixed 32x Baseline           |\n|----|-|-|--|\n|Adaptive Encoding|✓|x|x|x|\n| Average Latent Dim on ImageNet       | 29x29    |  32x32                            | 16x16                  |\n| Param Count             | 187M       | 169M      |                   187M |\n|Average FLOPs  (G)  on ImageNet          | 602      | 640      |                   822 |\n\nCAT has a slightly higher parameter count than the fixed 16x baseline due to an additional downsampling block used to support 32x compression. This module processes 16x latent features into a more compact 32x representation, which the fixed 16x model does not require. However, on standard datasets like ImageNet, CAT is more compute-efficient overall, as many images are assigned higher compression ratios. This results in a lower average latent dimension and reduced FLOPs per image compared to the fixed 16x baseline. These savings highlight CAT’s efficiency in both training and inference, driven by its ability to adapt compression based on image content.\n\n---\n## Question 3\n\n> While evaluating on ChartQA dataset, rFID is not suitable since it is mainly computed by Convolution Neural Networks, which are not dedicated for texts. It is suggested to evaluate the reconstructed text images with some OCR methods\n\nThank you for this suggestion! We agree that rFID may not fully capture the semantic fidelity of text in datasets like ChartQA. To address this, we evaluated the reconstructed images using an OCR-based metric. Specifically, we used the image_to_data() function from the PyTesseract library to extract per-word confidence scores, and computed the average OCR confidence across all words in each image. The following results show that CAT achieves higher OCR confidence compared to fixed baselines, further supporting its ability to preserve fine-grained textual content in visual representations. We will include the experiment details and score computing script in the updated paper.\n\n| Model              | Compression Ratio | OCR Confidence ↑ | rFID ↓ |  \n|----|---|--|---|\n| Fixed 8x        | 8x                       | 78.4             | 8.21   |  \n| Fixed 16x        | 16x                       | 71.2             | 8.67   |  \n| Fixed 32x        | 32x               | 65.7             | 10.79  | \n| CAT (Ours)         | 8x / 16x / 32x , avg 8.12x           | **84.6**         | **5.27** |  \n\n---\n\n## Question 4 \n> the classification heads in evaluated methods take the 16x latent features as input, which means the number of parameters of the classification head in CAT is more than the parameters of other classification heads in the evaluated methods, resulting in an unfair comparison.\n\nGreat observation! However, as shown in Appendix Table 10, the majority of images from the classification datasets are compressed at 16x or 32x, with fewer than 5% using the 8x compression ratio. This means that, in practice, most CAT latent features are shorter than the fixed 16x representations. Since we apply zero-padding to standardize input lengths for the classification head (as described in L718), many of the classification head parameters are effectively inactive due to being multiplied by zeros. In this sense, the experimental setup is actually slightly unfavorable to CAT, as a large portion of images are more aggressively compressed, resulting in shorter effective representations.\n\nThat said, we agree that a perfectly fair comparison is inherently challenging due to the adaptive nature of CAT. To address this concern, we will take the following steps in the final version:\n1. Include fixed 8x and fixed 32x baselines in Table 6 for reference\n2. Add the above explanation to the paper to clarify the trade-offs in representation length and classifier parameter usage.\n \n---\n\n## Weakness 1: limited compression flexibility (fixed 8x/16x/32x ratios)\n\nThank you for pointing this out. We have acknowledged this limitation explicitly in Section 7. We chose fixed ratios for CAT mainly for deployment and empirical reasons.\n\n**Deployment-wise,** fixed ratios enable efficient batching on GPUs. With just three discrete levels,CAT already spans a wide compression range (8x-32x), which we found sufficient to differentiate between high-fidelity  and high efficiency settings.\n\n**Empirically,** our analysis in Section 3.1 shows that over 56% of images can be compressed at 16x or higher with negligible quality loss. Table 1 also demonstrates that CAT assigns 8x to nearly all ChartQA images (96%), while applying higher compression to simpler natural images. This reflects meaningful and data-driven differentiation, i.e., the selection of 8x, 16x, and 32x compression are sufficient to prove the effectiveness of our text-driven compression concept.\n\nThat said, we fully agree enabling more flexible compression is a promising direction for future work. We are actively exploring ways to combine our LLM-based complexity scoring with transformer-based tokenizers that support flexible token counts. We are also studying patch-level adaptation, where different regions of an image receive different compression levels. While these extensions are exciting, we believe they build upon, rather than diminish, the core contribution of this paper.  \n\n---\n\n## Weakness 2: reliance on text descriptions that results in more complex usage of the VAE\n \nWe would like to clarify that the LLM-based complexity evaluation is fully decoupled from the VAE architecture. Once the compression ratio is selected, the VAE functions identically to standard tokenizers, using only the relevant subset of encoder and decoder blocks. This design does not add runtime complexity to the VAE itself, and in fact simplifies deployment by supporting multiple compression levels within a single model (see Fig. 1). \n\nWhile the text-based complexity evaluation introduces an additional step in the entire compression pipeline, it provides substantial practical benefits, in particular saving token count and compute for diverse  reconstruction and generation tasks. Moreover, in settings like text-to-image generation where only text descriptions are available at inference time, CAT is the only method to allow context adaptive tokenization, whereas existing adaptive tokenizers all require image inputs and thus are not applicable to generative settings.\n \n\n\n---\n## Weakness 3: added LLM inference overhead\n\nPlease see response to Question 1.\n\n---\n\n## Weakness 4: insufficient discussion of LLM biases or environmental impacts in broader implications.\n\nWe appreciate the reviewer raising this point, and we will expand the broader impact section in the final version to include discussion on LLM biases and environmental impact."}]

Author rebuttal:

Meta-review: The paper proposes to use an LLM to predict how many tokens should be used to compress an image using a VAE architecture. The reviewers appreciate the comprehensive empirical validation across several datasets and different downstream tasks. Reviewer mS49 considers the work interesting and promising, and the focus on developing a better measure of image complexity intriguing. The reviewers, however, had several concerns like the computational overhead of the LLM, the limited compression flexibility, as well as the calibration of the LLM-based complexity measure. The rebuttal addressed these points, but it also showed that the computational overhead is very high. After the rebuttal, the reviewers unanimously recommend the acceptance of the paper.    

The AC agrees with the recommendation of the reviewers. While the current approach has many limitations like the very high computational overhead for inference and limited compression flexibility, the paper proposes an interesting idea and future works might address these limitations.
Decision: Accept (poster)