Title: CAT: Content-Adaptive Image Tokenization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image tokenization
Abstract: Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity and introducing unnecessary computate overhead for   simpler images. To address this, we propose Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design (1) a caption-based evaluation system that leverages LLMs to predict content complexity and determine the optimal compression ratio for an image, and (2) a novel nested VAE architecture that performs variable-rate compression in a single model.
Trained on images with  varying   complexity, CAT achieves an average of 15% reduction in rFID across seven detail-rich datasets containing text, humans, and complex textures. On natural image datasets like ImageNet and COCO, it  reduces token   usage by 18% while maintaining high-fidelity reconstructions.   We further evaluate CAT on two downstream tasks.  For image classification, CAT consistently improves top-1 accuracy across  five datasets spanning diverse domains. For image generation, it boosts training throughput by 23% on ImageNet, leading to more efficient learning and  improved FIDs over fixed-token baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Content-Adaptive Tokenizer (CAT), a novel approach for image tokenization that dynamically adjusts the number of tokens (with predefined 3 scales) based on image complexity derived from text descriptions. Traditional image tokenizers use fixed compression ratios, which are inefficient for simple images and insufficient for complex ones. CAT addresses this by combining a caption-driven complexity evaluator (using LLMs) and a nested Variational Autoencoder (VAE) architecture. Experiments on different datasets and different tasks show the effectiveness of the proposed CAT. However, may major concern is that that the performance gap comes from the help of LLM, which is not used in most of the evaluated methods.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper presents the Content-Adaptive Tokenizer (CAT) to dynamically adjusts token counts for images based on text-derived complexity via an LLM evaluator and a nested VAE. Its strengths include innovative adaptive framework that improves reconstruction quality for complex images while reducing token usage on natural images, strong empirical validation across several datasets and different downstream tasks (classification, generation), and robust LLM-driven complexity assessment. \n\nThe main weaknesses involve limited compression flexibility (fixed 8x/16x/32x ratios), reliance on text descriptions that results in more complex usage of the VAE, added LLM inference overhead, and insufficient discussion of LLM biases or environmental impacts in broader implications.\n\nQUESTIONS:\n1.\tDirectly use a VLM as a scorer to produce the compression ratio without using an LLM. The impact of using and not using LLM should be discussed.\n\n2.\tIn Table 3, the proposed adaptive CAT outperforms the other 16x methods. Are the number of parameters of adaptive CAT and the fixed 16x baseline model the same? The number of parameters and FLOPs are better to be provided in the table.\n\n3.\tWhile evaluating on ChartQA dataset, rFID is not suitable since it is mainly computed by Convolution Neural Networks, which are not dedicated for texts. It is suggested to evaluate the reconstructed text images with some OCR methods, which are designed to recognize the texts in images. \n\n4.\tWhile evaluating CAT on the classification downstream task, the classification head takes the 8x latent features as input. However, the classification heads in evaluated methods take the 16x latent features as input, which means the number of parameters of the classification head in CAT is more than the parameters of other classification heads in the evaluated methods, resulting in an unfair comparison.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adaptive image tokenization strategy, named CAT, that varies the number of tokens to represent an image based on its complexity. For measuring the complexity, they leverage LLMs using captioning and VQA data extracted from the image, and assign a complexity score to the image. A nested VAE architecture is used to represent images with variable compression rates (three compression rates are used: 8x, 16x, 32x). Evaluations are performed on image reconstruction, generation, and classification tasks to show the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nDeveloping adaptive visual tokenizers is an interesting and promising research direction. I also find the papers focus on developing a better measure of image complexity intriguing. The method is well-motivated and explained, and the empirical results seem promising. Overall, I am leaning towards accepting this paper, but there are a few points that require further clarifications:\n\n- I found the first part of Sec. 3.1. a bit confusing. Exactly because of the reason in L118, is MSE a good metric to measure complexity? If not, what is the take away here? \n\n- LLM-based complexity measure proposed in this paper can be a potential solution, however, how do we ensure that it's well-calibrated and better aligned with human preferences? A user study could be useful here. I found the correlation study in Table 2 not fully conclusive.\n\n- I would be curious to see the generation performance on other popular datasets, e.g. COCO.\n\n- Finally, I suggest including comparisons to relevant baselines. Both FlexTok [A] and Semanticist [B] are recent adaptive tokenization strategies that are missing in the comparisons here. While not adaptive, TiTok [C] is also a relevant and recent baseline that can compress images quite substantially. Having these comparisons would be helpful to better understand the effectiveness of the method.\n\nI am happy to hear authors' opinions on these.\n\n[A] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length\n\n[B] “Principal Components” Enable A New Language of Images\n\n[C] An Image is Worth 32 Tokens for Reconstruction and Generation\n\nQUESTIONS:\n- Is MSE a good metric to measure complexity? If not, what is the main take away from Sec. 3.1? \n\n- How do we ensure that the proposed complexity measurement is well-calibrated and better aligned with human preferences?\n\n- What is the generation performance in other benchmarks, e.g. COCO?\n\n- How do you compare against the relevant baselines listed above?\n\n- Will the code and models be released?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Content-Adaptive Tokenizer (CAT), which dynamically adjusts token counts for images based on content complexity. It uses LLMs to evaluate complexity via captions and a nested VAE for variable-rate compression. CAT reduces token usage by 18% on natural images, improves rFID by 15% on complex datasets, boosts classification accuracy, and enhances generation throughput by 23%. It integrates language-guided evaluation with adaptive architecture for efficient, high-quality image representation.\nThe key components include: (1) an LLM-based system that analyzes image captions and perceptual queries (e.g., presence of text/ faces) to assign optimal compression ratios (8x, 16x, 32x); (2) a nested VAE that routes intermediate features to shared latent blocks, enabling variable-rate compression in a single model. This design allows CAT to encode simpler images with fewer tokens while preserving details in complex ones.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.Combines LLMs with a nested VAE to dynamically adjust token counts based on content complexity inferred from text descriptions, addressing inefficiencies of fixed-token approaches. This allows simpler images to be encoded with fewer tokens while preserving details in complex ones. \n2.The experiments are comprehensive, including comparisons of max acceptable compression ratios, performance comparisons across multiple datasets, ablation studies on model hyperparameters, and ablation studies on LLM models.\n3.Reduces token usage by 18% on natural image datasets (e.g., ImageNet, COCO) while maintaining high reconstruction quality. Achieves an average 15% reduction in rFID on detail-rich datasets, outperforming fixed baselines in preserving fine details like text and faces. \n4.Improves top-1 accuracy in image classification across diverse datasets and boosts training throughput by 23% in text-to-image generation. CAT’s adaptive representations enhance both efficiency and performance in reconstruction, classification, and generation. \n\nWeaknesses\nThe core innovation of the work lies in using prompt-based image complexity prediction from LLMs. However, the design of the tokenizer still adopts a traditional architecture, showing insufficient innovativeness.\nIn the Table 7 experiments related to Image Generation, add a comparison with DiT+Fixed 8x.\nOther questions can be found in the Questions section.\n\nQUESTIONS:\nAlthough CAT reduces token consumption, the LLM-based complexity evaluation adds an extra inference step. How does this affect computational budget? Please explain in combination with the experimental data.\n\nThe significance of Adaptive Image Tokenization lies in enabling a flexible trade-off between computational cost and output quality through adaptive token quantity. However, the article does not provide a comparison of computational overhead for CAT, including comparisons with fixed compression ratio methods and existing adaptive tokenization approaches in terms of inference time, video memory occupation, etc.\n\nWhat is the advantage of CAT over Fix 8x in terms of computational overhead, and what additional benefits does it offer? As can be seen from Table 3, Fix 8x significantly outperforms CAT in accuracy. Is the computational overhead advantage of CAT large enough to justify sacrificing Fix 8x's accuracy edge for adopting CAT?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}