Title: VaporTok: RL-Driven Adaptive Video Tokenizer with Prior & Task Awareness
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adaptive video tokenizer

Autoregressive video generation

Probabilistic taildrop

Reinforcement learning
Abstract: Recent advances in visual tokenizers have demonstrated their effectiveness for multimodal large language models and autoregressive generative models. However, most existing visual tokenizers rely on a fixed downsampling rate at a given visual resolution, and consequently produce a constant number of visual tokens, ignoring the fact that visual information of varying complexity warrant different token budgets. Motivated by this observation, we propose an adaptive video tokenizer "VaporTok" with two core contributions:Probabilistic Taildrop: We introduce a novel taildrop mechanism that learns a truncation index sampling distribution conditioned on visual complexity of the video. During both training and inference, the decoder reconstructs videos at adaptive token lengths, allocating more tokens to complex videos and fewer to simpler ones. Parallel Sample GRPO with Vapor Reward: By leveraging the probability distribution produced by probabilistic taildrop, we reformulate the visual tokenization pipeline as a sequential decision process. To optimize this process, we propose a variant of GRPO and a composite reward encompassing token efficiency, reconstruction fidelity, and generative quality, thus enabling metrics-aware adaptive tokenization across diverse objectives. Extensive experiments on standard video generation benchmarks confirm our analysis, showing that our adaptive approach matches or outperforms fixed‐rate baselines and naive taildrop while using fewer tokens.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adaptive video tokenizer that produces varying visual tokens according to input complexity. To achieve this adaptivity, two core designs are introduced: (1) a taildrop mechanismm that truncates video token according to its complexity, (2) a group relative policy optimization (GRPO) module that learns visual tokenization as a sequential decision process. The proposed tokenizer is evaluated in both video reconstruction and generation experiments, demonstrating its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n* This paper studies an imortant problem of adaptive video tokenization. This problem is critical because of the high requirement of video tokenization despite varying complexity of videos.\n* Experimental results show the effectiveness of their proposed method.\n\nWeaknesses:\n\n* Comparison between reinforcement learning and supervised finetuning is insufficient. For example, reinforcement learning has this reward hacking issue and has limited generalizability. Meanwhile, supervised finetuning cannot resolve back-side sears. The authors could conduct additional ablation study to demonstrate the advantages of their method.\n* Hyperparameter sensitivity is not well discussed. While the paper introduces multiple weights within the reward model (in line 244 of page 7), it would complicate the framework and the same set of hyperparameters might not generalize to different scenarios .\n* Lack of human study. Despite that automatic evaluation metrics like PSNR shows some advantages of the proposed method, many video issues such as blurriness and artifacts cannot be fully reflected through automated metrics. Therefore, the authors could conduct additional user study to validate the effectiveness of their model.\n\nQUESTIONS:\nPlease ref to the weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adaptive video tokenizer named VaporTok. It proposes a taildrop mechanism that learns a truncation index sampling distribution conditioned on visual complexity of the video. By doing this,  the decoder reconstructs videos at adaptive token lengths, allocating more tokens to complex videos and fewer to simpler ones. Further, the authors reformulate the visual tokenization pipeline as a sequential decision process and enables metrics-aware adaptive tokenization across diverse objectives. Extensive experiments show the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ The paper proposes an adaptive video tokenization technique adjusting token count based on visual complexity for more efficient sequences.\n\n+ The paper formulates video tokenization as a sequential decision process and uses RL to optimize the tokenizer. The idea is novel.\n\n+ The paper is well written and easy to read.\n\nWeakness:\n\n- My main concerns are about the performance comparison. For example, the Authors can consider compare with more baseline models which also use adaptive tokenizers [R1-R4]. \n\n- More analysis on model complexity needs to be given. For example, will the introduction of adaptive encoding based on reinforcement learning significantly increase the computational complexity and time delay of the model? And will it make the model significantly more difficult to optimize and train to converge?\n\n- More training details shoud to be provided. For example, how much time/GPU hours does it take to train the model? And the use of GPU cards or clusters. Besides, a comparison of the training resources with the baseline models should also be provided.\n\n[R1] Visual lexicon: Rich image features in language space. 2024\n\n[R2]  Elastictok: Adaptive tokenization for image and video. 2025.\n\n[R3] Flextok: Resampling images into 1d token sequences of flexible length. 2024.\n\n[R4] One-d-piece: Image tokenizer meets quality-controllable compression. 2025.\n\nQUESTIONS:\nsee weakness.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper innovatively proposes an adaptive video tokenizer that adjusts the token count: \n1) A probabilistic taildrop module is learned by aligning with a predefined explicit video complexity prior.\n2) In the second stage, the probabilistic taildrop layer is further optimized using reinforcement learning with GRPO, where the probabilistic taildrop module serves as the policy model and is optimized according to multiple rewards.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is well-written and easy to follow. To facilitate the shift from fixed-rate to adaptive downsampling, the authors novelty parameterized and optimized the truncation probabilistic taildrop supervised by a visual prior, then further used GRPO to optimize the truncation of the entire video clips as  a sequential decision process. \n\n2. The experimental results on UCF101 and K600 show that VaporTok+GRPO reduces token length by 75%, demonstrating the effectiveness of the adaptivity token adjusting method. \n\n\nWeaknesses\n1. The expression in abstract line 19, \"matches and outperforms fixed-rate baselines and naive taildrop while using fewer tokens\" is somehow unclear regarding which model is referred to as the \"fixed-rate baseline\" and \"naive taildrop\" in Table 1.  From my review of the codebase, it appears that the implementation closely follows the open-sourced LARP model, so I assume the \"fixed-rate baseline\" refers to LARP. However, VaporTok uses lossy token truncation compression, which introduces a gap compared to LARP in rFVD, from 20 to 53.9(VaporTok) or 66.6(VaporTok-GRPO). The authors do not discuss in the experiment whether this gap is significant or minor.  Simply saying \"competitive\" is not sufficient.\n\n2. In line 276, it says, “From the first two rows in Table 2: taildrop causes a substantial performance drop in rFVD, from 20 to 49.45.”   I’m confused about why VaporTok with Taildrop still maintains a token length of 1024 same as the LARP baseline.  Could you clarify what exactly is being done here and where the performance loss comes from? Specifically, what is the operation involved, and how does it lead to this drop in performance with the same token length?\n\n3. It would be better to include a discussion on the visualization of the Taildrop Probability Query Module after GRPO at different truncation points across various video clips.  Specifically, it is unclear which videos adopted longer or shorter truncation points, and whether the visualizations align with prior assumptions.   This analysis may be beneficial to investigate whether cold-starting the module using the current visual prior is both expected and necessary\n\nQUESTIONS:\nSee weakness above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops VaporTok, a model for adaptive video tokenization that accounts for both content complexity and for downstream tasks like generation. Regarding content complexity, instead of using uniform taildrop (nested dropout), VaporTok learns to predict the number of tokens to use for a video clip based on the content and a visual complexity prior. This information is used during training to better optimize VaporTok to use more tokens for complex scenes and fewer tokens for simple ones.\n\nRegarding awareness of downstream tasks, the authors develop a method for training the taildrop probability query module using GRPO. Learning is driven by four reward signals that capture efficiency (fewer tokens is better), diversity (avoid mode collapse), reconstruction (preserve the autoencoder functionality of the tokenizer), and generation (awareness of the downstream video generation task).\n\nThe authors evaluate VaporTok using the UCF-101 and Kinetics-600 video datasets. They show reasonable gFVD results using 50-65% fewer tokens than previous methods (Table 1). Extensive ablation studies are also performed to understand the impact of different design choices (taildrop vs. probabilistic taildrop, sampling method, etc.).\n\nSTRENGTHS AND WEAKNESSES:\nQuality -- The quality of this paper is generally very high. The methods used make sense, and I see no technical issues with the formulation. The evaluation is quite thorough in terms of ablations, comparing to previous methods, and showing the impact of different reward signals. However, there's a gap in the evaluation related to adaptivity since results are only shown for a single average token count (see the Questions section for more detail).\n\nClarity -- The paper is very clearly written and well-organized. The authors also provide code.\n\nSignificance -- The addresses two problems of interest within video tokenization and generation, and so I think the significance is quite high. First, adaptive tokenization is a very active area of research right now since the memory and computational costs of video modeling (for understanding and generation tasks) is quite high in part because most methods use highly redundant, and thus wasteful, representations. Second, task-aware and joint optimization of multiple components within a generative system is also an active area of research since training components in isolation is unlikely to yield optimal results.\n\nOriginality -- Both of the primary contributions of this paper (probabilistic taildrop and RL for task-aware tokenizer training) are novel as far as I know.\n\nQUESTIONS:\nQ1. How does VaporTok compare to existing methods when using (roughly) the same number of tokens on average? The expectation is that an adaptive method should perform much better than a fixed method on a diverse dataset since it can more efficiently allocate tokens: fewer for simple videos and more for complex videos leading to a net improvement in reconstruction quality.\n\nPerhaps this is shown in Table 2 comparing LARP with 512 tokens to VaporTok with 498 where rFVD is nearly the same and gFVD is slightly betterfor VaporTok? If you agree this is a fair comparison, why isn't there a larger benefit from the adaptive tokenization of VaporTok?\n\nTo better motivate my question, adaptive tokenization is a kind of variable rate compression, and the standard visualization for codecs is a rate-distortion curve. For tokenization, the key graph would show FVD vs. average number of tokens. Importantly, it's very difficult to compare different methods from a single point (one rate and one distortion) if one method doesn't \"dominate\" the other (lower rate and lower distortion). Table 1 doesn't show this since VaporTok has fewer tokens but also worse FVD so it's hard to draw a strong conclusion.\n\n\nQ2. Does VaporTok support different token counts for a single video? I think it does since the taildrop index is sampled during training so some flexibility should exist at inference time. Assuming that's right, I'd expect to see a visualization that shows how reconstructions change as you move from very few to a relatively large number of tokens for a particular video.\n\nSuch a visualization would also make it clear whether early tokens truly encode *semantic* information or if the representation is better described as coarse-to-fine. For example, if you reconstruct a video of a dog with very few tokens, do you get a sharp video of a different dog or a blurry/distorted video of the same dog?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-19"}