VaporTok: RL-Driven Adaptive Video Tokenizer with Prior & Task Awareness
Authors: Yang Minghao
Zechen Bai
Jing Lin
Haoqian Wang
Alex Jinpeng Wang
Venue: NeurIPS 2025 poster ()
Keywords: Adaptive video tokenizer
Autoregressive video generation
Probabilistic taildrop
Reinforcement learning
Primary area: applications
TL;DR: To train an adaptive video tokenizer, we introduce probabilistic taildrop to inject visual complexity prior to the tokenizer and incorporate GRPO for post-training, which further boosts efficiency in a task-aware adaptive manner.

Abstract: Recent advances in visual tokenizers have demonstrated their effectiveness for multimodal large language models and autoregressive generative models. However, most existing visual tokenizers rely on a fixed downsampling rate at a given visual resolution, and consequently produce a constant number of visual tokens, ignoring the fact that visual information of varying complexity warrant different token budgets. Motivated by this observation, we propose an adaptive video tokenizer "VaporTok" with two core contributions:Probabilistic Taildrop: We introduce a novel taildrop mechanism that learns a truncation index sampling distribution conditioned on visual complexity of the video. During both training and inference, the decoder reconstructs videos at adaptive token lengths, allocating more tokens to complex videos and fewer to simpler ones. Parallel Sample GRPO with Vapor Reward: By leveraging the probability distribution produced by probabilistic taildrop, we reformulate the visual tokenization pipeline as a sequential decision process. To optimize this process, we propose a variant of GRPO and a composite reward encompassing token efficiency, reconstruction fidelity, and generative quality, thus enabling metrics-aware adaptive tokenization across diverse objectives. Extensive experiments on standard video generation benchmarks confirm our analysis, showing that our adaptive approach matches or outperforms fixed‐rate baselines and naive taildrop while using fewer tokens.

VaporTok: RL-Driven Adaptive Video Tokenizer with Prior & Task Awareness
Minghao Yang
Zechen Bai
Jing Lin
Haoqian Wang"!
Alex Jinpeng Wang"
Tinnghua University
Nattooall University of Singapore
Nanyang Technological University
*Central South University
Abstract
Recent advances in visual tokenizers have demonstrated their effectiveness for multimodal large language models and autoregressive generative models. However, most existing visual tokenizers rely fixed downsampling rate given visual resolution, and consequently produce a constant number of visual tokens, ignoring the fact that visual information of varying complexity warrant different token budgets. Motivated by this observation, we propose an adaptive video tokenizer "VaporTok" with core contributions: Probabilistic Taildrop: We introduce a novel taildrop mechanism that learns a truncation index sampling distribution conditioned on visual complexity of the video. During both training and inference, the decoder reconstructs videos at adaptive token lengths, allocating more tokens to complex videos and fewer simpler ones. Parallel Sample GRPO with Vapor Reward: By leveraging the probability distribution produced by probabilistic taildrop, we reformulate the visual tokenization pipeline as a sequential decision process. To optimize this process, we propose variant of GRPO and composite reward encompassing token efficiency, reconstruction fidelity, and generative quality, thus enabling metrics-awan adaptive tokenization across diverse objectives. Extensive experiments on standard video generation benchmarks confirm our analysis, showing that our adaptive approach matches outperforms fixed-rate baselines and naive taildrop while using fewer tokens.
Introduction
Visual generative models have undergone rapid advancements recent years, progressing from [2Es[22] and G7 to diffusion models [8). More recently, autoregressive (AR) based approaches 5 52 45 have emerged as prominent direction, demonstrating competitiveness with diffusion models. The superior performance is largely due to the scalability and flexibility of the
AR paradigm, as demonstrated by large language models (LLMs) (58, 46 10]. Similar to LLMs, AR-based visual generative models necessitate visual tokenizer, which is essential for converting image or video data into vector representations that the model can process. Consequently, research into visual tokenizers has become central focus in visual generative models.
Despite their remarkable performance, AR models still face fundamental limitations inherent in the AR paradigm. Specifically, the computational complexity of processing token sequences grows quadratically with their length. In addition, the prediction errors can accumulate progressively as observed in numerous works [ 36]. Intuitively, shorter, more compact visual token sequence can be a favorable option. Existing visual tokenizers [47 60) 29 62) generally output a predetermined attempts adaptive
number of latent tokens for subsequent generation tasks. While there
"Equal contribution. t Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tokenization (e.g., 1 SID, most of them still depend on manually specified range of token counts without an effective prior, limiting their capacity for truly adaptive tokenization.
Another representative limitation in AR-based visual generative models is that the training of tokenizers and AR models are usually divided into two separated stages, making the visual to- kenizer sub-optimal and preventing from generalizing well to downstream tasks. In fact. the above issues are intertwined. Determining visual rep- resentation and its adaptivity should not only satisfy the data prior but also align with downstream task character- istics. On the other hand, the super- vision of visual generation can help optimize both AR generative model and, more importantly, its tokenizer, if prok-propagated properly. This implies that an efficient, adaptive and downstream-awe visual tokenizer is the key to address the AR issues mentioned above.
Yapartia Encoder.
Sugge
Tranction Indi
Ru
ZV
Trasestien bikes 1ush
VaparTol Deceder
Trancation Iuder Puch
Figure I: VaporTok employs two-stage training: the first stage uses visual complexity for supervision, the second stage uses GRPO with multiple task-aware rewards for supervision.
In this work, we introduce Vapor Tok, an adaptive video tokenizer that leverages both data prior and task-related signals to optimize adaptivity. Specifically, we observe that visual data inherently possesses varying degrees of complexity in terms of content across spatial and temporal di- mensions. Consequently, representing simple content with excessive tokens can lead to redundancy. Conversely, complex content may not be adequately captured if represented with too few tokens. Therefore, dynamically adjusting token number based on visual complexity would enable a more faithful alignment between visual information and its representation. Inspired by this, we propose "Probabilistic Taildrop". a method that leverages visual complexity to build a sampling distribution over token counts and then drops the excess tail tokens accordingly. The complexity informed taildrop not only helps mitigate quadratic computation by producing a compact token sequence, but also reduce error accumulation by condensing meaningful information at the head of the sequence.
When tackling the limitation of training disparity between tokenizer and AR model, the main technical challenge is the differentiability of the two models due hard token indexing. Recently, reinforcement learning (RL), particularly (RPO has shown considerable advantages in various domains 10. 13 6 (5, 50 21). Notably, the reward formulation in reinforcement learning does not necessitate differentiability with respect to the parameters the policy model. Therefore, propose to leverage this characteristic to unlock task-aware adaptive tokenizer training. The core idea is that by optimizing the visual tokenizer using RL-based rewards, these non-differentiabe supervisory signals could be effectively transmitted during its training. Although the idea seems intuitive, it non-trivial to achieve. Our approach innovatively formulates visual tokenization as a sequential decision process, which is compatible with RL training framework. To the best of our knowledge, this is the first work that employs RL framework to formulate and train a visual tokenizer for task-aware adaptive tokenization. In addition, we design novel "Vapor Reward" that accommodates multiple supervisory signals into the reward function, providing valuable insights for the community. Our contribution be summarized follows:
Probabilistic Taildrop: An adaptive video tokenization technique adjusting token count based on visual complexity for more efficient sequences.
RL for Task-Aware Tokenizer Training: A novel framework that formulates video tokenization as sequential decision process and uses RL to optimize the tokenizer, making its adaptivity aware of multiple metrics, including downstream AR generation performance.
Vapor Reward: A new multi-signal reward function designed to effectively guide the RL-based tokenizer optimization.
2

---

2 Related work
Adaptive visual tokenizers.
Most existing visual tokenizers 12
29 62
49
16
67 are limited to representing visual features using fixed number of tokens. ignoring the varying complexity visual content. As a result, recent works have shifted toward adaptive tokenization schemes that dynamically adjust the token budgets based on visual complexity. ALIT [ employs a recurrent encoding scheme to progressively assemble the token sequence of an image and the process can be halted. CAT[39] leverages MLLM's complexity analysis of image captions to regress which compression rate to apply. Flex Tok 2 and One-D Deee-ce-Peeeee- [3]] both employ nested dropout to encourage the model to prioritize a coase-too-fine representation without fixed-length constraints. ElasticTok[57] extends the taildrop technique the video domain and adaptively selecting the token count inference based reconstruction-quet threshold. ViLex introduces a novel "visual language" that encodes image tokens after taildrop into the textual token space by self-supervised training on a frozen text-to-image diffusion model. These adaptive methods either pick from fixed set of token counts or use random taildrop with thresholding. yet neither truly adaptive: the first is limited predefined choices, and the second ignores visual complexity during training. In addition, they focus only on reconstruction and neglect the impact of adaptive token selection on downstream performance.
68
20
55
43
visual information in early tokens, yiclding
GRPO in vision domains. Enhancing foundation models via reinforcement learning has become a major research focus. Motivated by the strong inference performance of DeepSeek [T], Group Relative Policy Optimization (GRPO) [37] has demonstrated clear advantages over PPO in both training efficiency and final model quality. In computer vision, core generation and understand- ing tasks, including Visual Question Answering 32 59 Image Grounding 6), Video Question Answering[14] [14, 33), and Visual Generation (50 56
25 are actively investigating the integration
of GRPO to boost existing methods, seeking to transfer the success of GRPO from large language models to vision.
Method
3.1 Probabilistic Taildrop
Conventional taildrop technique (23 simply samples truncation positions from uniform distribution without considering any prior information about visual complexity. This can lead insufficient preser- vation of complex visual content when too few tokens are selected, and to unnecessary redundancy when too many tokens are retained for simpler visuals. In contrast, we propose probabilistic taildrop, which constructs a truncation-sapa sampling distribution informed by the complexity of the visual input. During training, truncation index are drawn from this distribution to perform taildrop. This strategy both preserves the fundamental principle of taildrop taildrop-compeen compressing semantic information into earlier tokens while relegating detailed information to later tokens-and adaptively selects truncation points based on visual priors, thereby achieving efficient yet faithful visual encoding.
To implement probabilistic taildrop, we introduce the Taildrop Probability Query Module in Sec- tion 3. to obtain the taildrop probabilities. To incorporate visual priors into the supervision of these probabilities, we construct a distribution from the visual information regularize the predicted taildrop probabilities, as described in Section 3.1.2
3.1.1 Taildrop Probability Query Module
Given a video E [TTxxxxx3 VaporTok first patchify into a sequence of video tokens P:
P Patchily(V) E R
)xD
where fr, fH, fw are the temporal and spatial downsampling factors. And then P will be concate- nated with K' learnable query tokens Q E RR xDD and the combined sequence will be passed into the encoder:
Zp + Zq = Enc + Q)
+K)XD
(2)
where @ denotes concatenation and Zp denotes the represatation of P, Q after encoder respec- tively. To enable VaporTok to learn distribution for taildrop. we introduce Taildrop Probability
3

---

Video Patches
Encoder Queries
Putchify
Vape VaporTok Encoder
Tolldrop Probability Query
Visual Complexity Prior
Taildrop Probability Query Module
Quantizer
Prier Probakiltty
KL align
Tuildrop Probability
Sample index
Probubilistic Tuildrop
Decoder Queries
VaporTok Decoder
UnPatchify
Figure 2: VaporTokk pipeline: A taildrop probability query module constructs taildrop probability supervised by a video-complexity prior. An index is then sampled according to this probability, and only the tokens preceding that index are retained for reconstruction training.
Query Module consisting of / successive transformer blocks and softmax layer, illustrated in Figure A dedicated taildrop probability query Qtail E R is concatenated with Zp and fed into transformer blocks of Taildrop Probability Query Module
Qtai + Z/, TransformerBlocks (Qtail Zp) F
(1+
Tw
1xD
(3)
P'ail is then passed through an MLP followed by a softmax layer to produce the taildrop probability distribution P:
P
Softmax (MLP (Mta(((ai)) E RRKK
where is the total token counts of latent query ZQ. To enable adaptive token usage, a truncation index t is sampled according the learned taildrop probability P:
I ~~ Categorical(P)
(5)
All latent tokens of Zo whose index exceeds the sampled index t are discarded and only the first t tokens are concatenated with the decoder query M which are then passed to the decoder for reconstruction:
PTD_2p ProbabilisticTaille Taildrop( Zq) Zq, 1a E R
(6)
= Dec (M + PTD_2 ZQ) E RTxxxxxx3
(7)
The first-stage training loss of VaporTok is composed of LI reconstruction loss, LPIPS perceptual
loss (6), GAN [7], quantizer loss and prior loss 48 The detail is provided in the Appendix.
3.1.2 Video Complexity Prior
Implicitly modeling video complexity is impractical. Therefore, explicitly supervise the Taildrop Probability Query Module with Gaussian distribution that contains visual prior information. First, the spatial and temporal complexities are computed separately and then these complexities mapped to a corresponding token count via Equation 9 We then construct Gaussian distribution to supervise the taildrop probabilities, with its mean set the token count obtained from the preceding mapping and its variance determined by K, the total number of encoder query tokens. Specifically, let visual complexity of video be
(8) where SC; and TC, are its spatial and temporal complexities whose details are provided in the Appendix. Then maintain an empirical CDF F' over all observed complexities training set and map each video complexity to token count k; by:
C; SC; x TC,,
k; ss++ 1s+DF(c)l. F(c) E [0.
(9)

---

where s is predefined minimum token count tolerable for reconstruction and A is the maximum increment. This yields whole dataset wise mapping: :
{
[dotaset]
{ki {
Idataset]
(10)
For each training sample i, we construct Gaussian distribution with mean k; and variance K Israle as the prior distribution:
CeeussianPrior, N (1: p Kú, o² K asaale)
(11)
where is the token count caleulated by visual complexity prior for the i-th video, K is the total token counts of Zo. and acaale is hyperparameter to control the trend of the prior distribution.
Then, the loss of sample video i to train the taildrop branch, specifically Taildrop Probability Query Module and Vapor Tok encoder, is calculated as the KL divergence between P; and Gaussian.Piion;
Loss;
KL(P: ||
CerussianPriony
(12)
3.2 Parallel Sample GRPO with Vapor Reward
Since the sampling operation employed in the VaporTok is non-differentiable, it is infeasible to propagate the reconstruction loss to the Probabilistic Taildrop branch through the latent space. Nevertheless, in reinforcement learning, the reward funetion can be treated as arbitrary black box, whose information can be passed to the policy model, even though the reward is not differentiable with respect to the parameters of the policy model. In addition, except for the basic reconstruction feedback, several helpful metries can also be incorporated in reward definition to make the tokenizer more efficient and appropriate for downstream generation task.
To this end, we cast the video tokenization as a sequential decision process in Section 3.2. and introduce Parallel Sampling GRPO in Section 3.2.2 to optimize this process while avoiding mode collapse. Furthermore, in Section 3.2.3 we introduce Vapor Reward, which enables GRPO to refine VaporTok's adaptivity with respect to both reconstruction and generation tasks. Finally, in Section 3.2.4. we define the optimization objective of Parallel Sample GRPO.
3.2.1 Definition of sequential decision process
Given an entire video Ventire E R.NCRPO HxWx3 x3 of length NGRPO. we first partition it into where NVAE is the predefined frame count can be processed by VaporTok encoder at one time. For each video clip Vaúp VaporTok
= NerPo video clips (Veipp V2ip...., Vi ...pp...., Veli)) ... NvaR
encoder computes its latent representation Z and Taildrop Probability Query Module computes its taildrop probability distribution P.. We define the three key components of the sequential decision process in the context of token truncation problem follows:
State S: the current input video clip Viisp E R NNNNExM*X>3
Action A: the truncation index sampled from P, to truncate the latent representation z(( of
current video clip Vaipp
Reward a): we will introduce our proposed Vapor Reward in Section 3.2.3
Then we can get a taildrop probability sequence P2....... P.....,, PL) to sample truncation index for each latent representation z respectively. Also, we define the policy in our pipeline V ) where is the parameter of Taildrop Probability Query Module proposed in 3.1.I Notably, the key difference between Markov Decision Process (MDP) and our proposed sequential decision process that the latter does not satisfy the Markov property, and its transition dynamics P(s' a) are implicitly defined by the entire input video Ventire-
Ta (P: Section
3.2.2 Parallel Sample GRPO
In our setting, the G trajectories within GRPO group share exactly the same state sequence (Vdsp) V2 .2.p Vi ...p VLL and thus the same policy Te (P Valp)) is applied. This leads to
5

---

Group Relative Policy Optimization
Split the entire video into video clips
A
A
Ap
Taildrop Probability Query Module
Advantage Computation
Truncation Index Parallel Sampling
Pua
P
Truncation Index Path
Bificiency
Truncation Index Patk 2
Vapor Reward
Divernity
Reconstrucion
Trancation
Generation
Figure 3: Parallel Sample GRPO pipeline: Split the video into a sequence of clips and apply the taildrop probability query module to generate corresponding taildrop probabilities. Then perform parallel sampling to derive truncation matrix I, compute rewards using the Vapor Reward, and update the parameters of taildrop probability query module via GRPO.
highly similar or even identical sampled truncation index path across all G trajectories within group, especially under greedy or top-k sampling strategies, which in turn causes severe mode collapse. To address this issue, propose Parallel Sample GRPO, variant of GRPO specifically designed for training GRPO in such non-Markovian scenario, which concludes two remedies:
Stochastic parallel sampling. Rather than restricting sampling to the top-k candidates, we draw samples from the whole categorical distribution defined by the taildrop probability distribution P.. The sampling strategy increases the probability of selecting lower probability indexes and allows for greater diversity between trajectories within group. Specifically, a truncation matrix I= [7,]]= E Z is obtained, where each row represents truncation path that specifies how many tokens are used to represent each clip in the video and there are G such truncation paths in total. Besides, to further mitigate the mode collapse introduced by the Man-Maakovian nature of the defination, we augment the reward signal with an exploration bonus defined in Equation [4] which encourages diversity across trajectories and therefore promotes exploration.
3.2.3 Vapor Reward
We define the Vapor Reward to incorporate reconstruction, token- and downstream feedback into the policy model (Taildrop Probability Query Module) while avoiding collapsed sampling paths within a group. The Vapor Reward comprises of following four types of rewards:
Efficiency reward. To encourage the latent representation of entire video Ventire be more efficient, we define the efficiency reward. Specifically, for the i-th path, the efficiency reward is defined as:
officiency -amax
(6) curr
(13)
where Nmax is the maximum permissible token count for the video, which can be calculated as Nmax L K, where K is the total token number of Zo- N(1) is the actual number of tokens retained and can be calculated as (6) = E/ Ji.j
Diversity reward. To encourage exploration and mitigate mode collapse, we define a diversity reward for each path based on how different its sampled index sequence is from others in the same group. Specifically, for the i-th path, the diversity reward is defined as:
( Riversity
Uia + Ingi
(14)
(G - 1)L
where I = [R,g] E ZGXL is the sampled truncation matrix, G is the number of sampled paths, and L the sequence length. The indicator function 1 equal to I its argument true and otherwise. A higher reward is assigned to a path if its sampled indices are more dissimilar from the others.

---

Reconstruction reward. To encourage the refined taildrop probability to preserve the reconstruction ability learned from former stage, we define reconstruction reward. Specifically, for the i-th path, the reconstruction reward is defined as:
R (
reconstruction
Σ
MSE(Va clips V
(15)
For j-th video clip, we use the truncation index path [ to truncate the latent representation and reconstruct the video clip via the VaporTok decoder. Then we compute the mean-sques mean--quared error between reconstructed video clip Viip and ground truth video clip V The reconstruction reward for the i-th path is the negative of the sum of the reconstruction MSE of all video clips of current entire video.
Generation reward. To make the taildrop probability be aware of downstream generation perfor- mance, we define generation reward. The former work LARP impose a lightweight AR prior model to encourage the latent space be more suitable for downstream AR-based generation. Hence, the prior model in LARP 48 reused in our VaporTok (the detail about lightweight ar prior used in Vapor Tok is provided in the Appendix), and its top-5 accuracy latent-toked predictions is employed as the AR generation reward, guiding the taildrop probability query module to improve efficiency without sacrificing downstream generation performance. Specifically, for the i-th path, the generation reward is defined as:
( generation
Accuracyrops (PTD_Z))
(16)
3.2.4 Objective of Parallel Sample GRPO
For each entire input video Ventires a batch of G candidate truncate index sequence (k, (k:) wheee k; (Jr,, I.,,..., Ia.j...., Ir, is sampled from the policy mans and scores by reward:
R Σ
Am R(O
(17)
MEM
where m one element of M = {efficiency, diversity, reconstruction, generation] and the non- negative weights {fm} } control the relative importance of each reward component. To obtain relative advantages, the rewards { are normalized by their mean and standard deviation:
R. mean R1, R2, ..... Ro}
A
(18)
std R1, R2, ..., RG)
The parameter 0 is then updated to maximize the following objective:
(f) = EIWSTo
min Pi A,, clip(p:. 1- 1- 1+)) - DRI(TT Tret) (19)
T
(k; Ventire)
To (k; Ventire)
denotes the clipping threshold, 3 scales the KL-divergence penalty. I = [4,1] zGxL is the sampled truncation matrix, and A, represents the advantage estimate for the ith sample.
4 Experiments
Dataset. We conduct video reconstruction and generation experiments using the Kinetics-6000 and UCF-101 [4I] datasets. In the first stage, we use NVAE 16 frame video clips at a spatial resolution of 128 x 128 for VaporTok training and evaluation following [88]. In the second stage, Parallel Sample GRPO training operates on full NGRPO 80 frame videos and the sequence length optimized by GRPO is L
Nçaea
80 = 5.
NvAR
Implementation details. VaporTok first patchifies the input video into a sequence of tokens. In all experiments, set the patch sizes to fr 4, fH = 8, fw 8, that 16 x 128 x 128 video clip
7

---

Table 1: Comparison of generative video models. VaporTok refers to the evaluation results after Stage training, while VaporTok- Tok-GRPO refers to the evaluation results after Stage 2 training. The reported token counts are the average number of tokens used per video.
Method
#Params
#Tokens
rFVD]
gFVD
Tokenizer Generalor
K600
UCF
Diffissio-bool based generative models with continuous tokenizers
VideoFusion 26
2B
173
HPDM 40
725M
66
MLM generative models with discrete tokenizers
MAGVIT-MLM 61
158M
306M
1024
25
76
MAGVIT-12-MLM
62
307M
1280
8.6
4.3
58
AR generative models with discrete tokenizers
Video 10
9.4B
2065
109.2
626
TATS (13
32M
321M
1024
162
332
MAGVIT-AR 0
158M
306M
1024
25
265
MAGVIT-22-AR 62
840M
1280
8.6
109
OmaiTokenizer 49
82.2M
650M
1280
42
32.9
LARP- 48
173M
632M
1024
20
5.1
57
LARP-512 48
173M
632M
512
53.3
86
Vapor Tok (Ours)
195M
632M
498
53.9
8.3
80
VaporTok-GRPO (Ours)
632M
66.6
10,4
split into x 16 16 1024 patches. The number of encoder query tokens is A: 1024. The quantizer and prior model is set as same as [48] where the factorized codebook is employed of size 8192 with embedding dimension deodebook
and prior model is adapted from small GPT-2 backbone For taildrop probability query module, we set the number of transformer blocks as I and the softmax temperature is set 1.8. Due to the high computational cost of training, we trained for 30 epochs on the UCFI01 and K600 datasets using the pretrained model provided by which required 90 hours on A100 GPUs.
LARP 48
For parallel sample GRPO, we set the group size G= 8, the KL penalty weight | = 0.1, the number of inner iterations μ and the clipping bounds élow 0.2 and thigh 0.28 as in [64] The default reward weights for efficiency, penalty, diversity, reconstruction, and generation are set to 1:1:1:1:1. The GRPO training process uses the UCF101 dataset for single epoch, which takes hours on a single A100 GPU.
For AR generative model, we adopt LLaMA-style transformer 42 In the class-conditional generation task on UCF-101 we prepend a [cls] token to represent the category, and [stop] token to cease the generation process when encountering it. The generation task is trained on the UCF101 dataset for 3000 epochs, which takes 40 hours on A100 GPUs.
Table 2: Comparison of different training techniques.
Table 3: Entropy taildrop probabilities under different GRPO implementation on UCF101 validation set.
Modill
Base
#Takess
Tuildrop
Trildop
Punh
Indes
:FVDj
gFVDj
4PVDV (FVD
LARP 4B
1024
30.00
57.00
2.85
LARP 4M
Vapertba
512
53.25
49.45
86.35
62.34
1.62
Model
Diversity
Reward
Sampling
Paraliel
Sampling
Topk
Entrupy
Vopoctoa
512
sample
$1.94
93.34
1.14
ViporTak
5.11 308
VoporTok
argmnn
1.50
VaporTok-
0.006412
Vapor
298
sample
53.92
80.13
1.48
ViPO
Vapor Tolk
pee sample
73.00
95.41
1.30
ViporTok ToR-CIO-IR
4.13323
4.1 Video reconstruction & generation comparison
On the UCF-101 class-conditional generation benchmark, evaluate LARP against video generation approaches- spanning diffusion- based models, masked- sive methods as in [48]. As shown in Table VaporTok achieves competitive performance with other video generators on the UCF- 101 dataset even when using significantly fewer tokens. Notably, our VaporTok model shows a much smaller gap between gFVD and rFVD than other AR-based video generators. Besides, after GRPO training, we can further reduce the average latent token count from roughly 50% down to about 30% of the original while still preserving reconstruction and generation quality- thus achieving efficient adaptivity.
methods, and autoregres-
8

---

4.2 Comparison of training techniques
To demonstrate the effectiveness of proposed probabilistic taildrop technique, we make a comparison of different type of training strategies and show the evaluation result in Table For naive taildrop, conducted experiments to test whether taildrop can induce a progressively decreasing importance of tokens along the sequence. From first two rows in Table 2 taildrop causes substantial performance drop in rFVD, from 20 to 49,45, yet gFVD almost fully recovers the gap introduced by reconstruction and achieve competitive gFVD 62.34 with 57. This shows that taildrop training can partially close the original gap between reconstruction and generation performance by enforeing a comaantic-todetaan ordering in the latent space, which greatly reduces error accumulation during autoregressive inference. The same conclusion can also be drawn from the third and fourth rows in Table 2 For probabilitstic taildrop, we examine three strategies for obtaining the truncation index from the taildrop probability distribution: taking the argmax, directly sampling, and a pre-sampling variant that only samples from indices before the argmax. Among these strategies, directly sampling from the taildrop probability yields the best performance. achieve the best gFVD 80.13 13 with a similar token count to baseline methods. The argmax approach lacks variability across different lengths for the same video, thereby missing the core advantage of taildrop- structuring the latent space from semantic to detail. Pre- sampling best reflects this advantage, but it tends to reduce the average token count significantly, which slightly compromises reconstruction and generation quality.
4.3 Ablation studies
Impact of different sampling strategies. To verify that Parallel Sample GRPO alleviates mode collapse introduced by non-Markovian setting, we compute the average entropy of the taildrop proba- bilities on UCF101 validation set. The results appear in Table 3 The conclusion that probabilities adjusted by GRPO inevitably become more concentrated can be drawn from the lower entropy compared to that optimized by the prior probability of the first stage as shown the first row in Table 3 This concentration is an unavoidable consequence of mode collapse introduced by our definition of sequential decision process. However, unlike the complete collapse observed with direct topk sampling, our parallel sampling combined with an exploration reward effectively mitigates the issue.
Impact of four rewards in Vapor Reward. During GRPO fine-tuning, we only adjust the taildrop probabil- ity without altering the latent-space distribution, our goal to reduce token usage without degrading generation or reconstruction quality- that is, to achieve an efficiently task-aware adaptive tokenizer.
1:1:0:1
1:1:1:0
1:1:1:1
1:1:0:0
0.75
0.5
Firstly, we isolated the individual effects of the generation reward and reconstruction reward. We conducted three ablations: (a) dropping the generation reward, (b) dropping the reconstruction reward, and (c) dropping both. Fig- ure 4 that relying solely on the generation reward yields a large boost in generation quality at the expense of reconstruction quality, whereas relying solely on the re- construction reward greatly improves reconstruction with negligible impact on generation. If both rewards are re- moved, the model suffers its worst overall performance on both tasks. Notably, using both rewards simultaneously allows steady improvement in both without compromising either. These results confirm that jointly optimizing gener- ation and reconstruction rewards outperforms using either one alone or none at all.
0.45
26
(a) Generation Reward
1:1:0:1 1:1:1:0
111:1
1:1:0:0
-01
0.16
24
(b) Reconstruction Reward
Figure 4: The generation (top) and re- construction (bottom) rewards under dif- ferent reward weights where the weights order efficiency, diversity, reconstruc- tion, and generation.
To rigorously assess all four rewards, we carried out a full four-way ablation, removing each reward in turn. The results are summarized in Table Without the efficiency reward, the performance of reconstruction and generation become stronger, but this leads to an increase in the average token cost, violating our efficiency goal. Leaving out the diversity reward produces comparable task performance but causes the taildrop probability to collapse few fixed indices, undermining true adaptivity. Omitting
9

---

Table 4: Ablation of each rewards where each row "w/o R indicates the model trained without
reward R. The reported token counts are the average number of tokens used per video on the UCF101 validation set.
Missing Reward
#Tokens
rFVDJ
PSNR
gFVDL
MSEJ
ACC1
GRpo
361
66.6
24.49
97.92
4.48
10-3
3.70%
w/o efficiency reward
874
42.2
27.08
60.17
2.52 x 10
4.44%
w/o diversity reward
325
74.9
24.23
100.94
4.77
10-3
3.65%
w/o reconstruction reward
318
73.7
24.19
109.52
4.77
10-3
3.60%
w/o gencration reward
297
77.5
24.04
113.56
4.96
10
3.56%
either the reconstruction generation reward leads to performance drop in corresponding tasks. In conclusion, dropping any single reward prevents the model from maintaining reconstruction and generation quality in efficiently adaptive manner.
4.4 The alignment between FVD and reward
To verify the alignment between the final FVD and the reward, the results of reconstruction and generation under both sampling and argmax settings are shown in the Figure It can be observed that rFVD strongly correlates with the reconstruction reward, while gFVD shows similarly strong correlation with the generation reward. This further supports the validity of our reward design: the reconstruction MSE serves reliable proxy for reconstruction quality, and the top-5 accuracy of the prior model effectively reflects generation quality.
gFVD Top-5 Accuracy
rFVD MSE
150
argmax
argmaax sample
sample
130
eFVD
80-
:10
60
3,1
3.3
3.5
0.0040
0.0045
0.0050
0.0055
Top-5 Accuracy 1
MSE
(a) Alignment between gFVD and ACC
(b) Aligmment between rFVD and MSE
Figure 5: rFVD and MSE exhibit a strong positive correlation, while gFVD and ACC show a clear negative correlation, indicating the effectiveness and rationality of the proposed reward design.
Conclusion and future work
We introduce VaporTok,, an efficient and adaptive video tokenizer with two key innovations. First, our probabilistic taildrop leverages visual complexity dynamically determine truncation indexes, preserving semantic-to-detai token structure. Second, we introduce a parallel sample GRPO strategy guided by the Vapor Reward, unified signal combining token count, reconstruction quality, and generation fidelity, to inject multiple task-related information to VaporTok. Our results show that adaptive tokenization can be effectively learned during training, and demonstrate the effectiveness of GRPO in optimizing tokenizers. However, the present work explores only how to employ GRPO to optimize the truncation probability rather than the entire VAE, and only the generation task is considered as downstream task. Moreover, although various methods have been employed to mitigate mode collapse, the truncation diversity of VaporTok after GRPO training remains notably lower than that after the prior training. Future work will investigate extending GRPO to entire VAE optimization, modeling video tokenization as a complete Markov decision process and applying the framework to
understanding [34. 65. 44 and unified genneration&umddesttnddi scenarios as in (28 9 33 53
10

---

References
[1] Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. arXiv preprint arXiv:2403.06963 2024.
[2] Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini. Oguzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences of flexible length, 2025.
[3] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015,
[4] Joao Carreira, Eric Noland, Andras Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv; 1808.01340, 2018.
[5] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 11315-11325 2022.
[6] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. https:///githbb: http://tthbbb.co///eee-eaeeee/e/ 2025-02-02.
2025. Accessed:
[7] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
[8] Yinda Chen, Haoyuan Shi, Xiaoyu Liu, Te Shi, Ruobing Zhang, Dong Liu, Zhiwei Xiong, and Feng Wu. Tokemunify: Scalable autoregressive visual pre-training with mixture token prediction. arXiv preprint arXiv:2405. 16847, 2024.
[9] Zisheng Chen, Chunwei Wang, Xiuwei Chen, Hang Xu, Jianhua Han, and Xiandan Liang. Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation. arXiv preprint arXiv:2503.06764 2025.
[10] DeepSeck-AL, Deepseck-rl: Incentivizing reasoning capahility in llms via reinforcement learning, 2025.
[11] Shivam Duggal, Phillip Isola, Antonio Torralba, and William T. Freeman. Adaptive length image tokeniza- tion via recurrent allocation, 2024.
[12] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for nigh-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[13] Hugging Face. Open rl: A fully open reproduction of deepseck-rl, January 2025.
[14] Kaituo Feng. Kaixiong Gong. Bohao Li, Zonghao Guo, Yibing Wang. Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning milms, 2025.
[15] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with aine-tgnostic vegan and time-sensitive transformer. In European Conference on Computer Vision, pages 102-118. Springer, 2022.
[16] Yuying Ge, Yizhuo Li, Yixiao Ge, and Ying Shan. Divot: Diffusion powers video tokenizer for comprehen- sion and generation, 2024.
[17] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[19] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation oemerratinnnn via transformers. arXiv preprint arXiv: 15868, 2022.
[20] Huiwon Jang. Sihyun Yu, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Efficient long video tokenization via coordinate-based patch reconstruction, 2025.

---

[21] Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li. Le Zhuo, Shilin Yan, Pheng-Ann Heng. and Hongsheng Li. T2i-r1: Reinforeing image generation with collaborative semantic-level and token-level cot, 2025.
[22] Diederik Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
[23] Aditya Kusupati, Gantavya Bhatt, Aniket Rege. Matthew Wallingford, Aditya Sinha, Vivek Ramanujan. William HowarddSnyyee. Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation 2022.
learning. Advances in Neural Information Processing Systems,
[24] Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng I.i, and Peng Gao. Lumina-mgpt: Illuminate flcxible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint 2408 02657, 2024.
[25] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXii:2505.054770 2025.
[26] Zhengxiong Luo, Dayou Chen, Yingya Zhang. Yan Huang, Liang Wang. Yujum Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint (:Xiv:2303,083200 202..
[27] Zhuoyan Luo, Fengyuan Shi, Yixiao Ge. Yujiu Yang. Limin Wang, and Ying Shan. Open-magvin An open-source project toward democratizing auto-regressive visual generation, 2024.
[28] Chuofan Ma. Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20321, 2025.
[29] Fabian Mentzer, David Minnen, Firikur Agustsson, and Michael Tschannen. Finite scalar quantizaaion: Vq-vae made simple. arXiv preprint arXiv:2309.1555 2023.
[30] Tsvetomila Mihaylova and André FT Martins. Scheduled sampling for transformers. arXiv preprint (eXXvv:996.006333 2019.
[31] Keita Miwa, Kento Sasaki, Hidchisa Arai, Tsuhasa Takahashi, and Yu Yamaguchi. One-d-piece: Image tokenizer meets quality compression, 2025.
[32] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilitics through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025.
[33] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03009,, 2024.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMI., 2021.
[35] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9 2019.
[36] Dhruv Rohatgi, Adam Block, Audrey Huang, Akshay Krishnamurthy, and Dylan J Foster. Computational- statistical tradcoffs at the next-token prediction barrier: Autoregressive and imitation leaminng under misspccification. arXiv preprint arXiv:2502. 124.., 2025.
[37] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. Xiv preprint arXiv:2402.033: 2024.
[38] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. VIm-rl: A stable and generalizable rl-style large lasion-language model, 2025.
[39] Junhong Shen, Kushal Tirumala, Michihiro Yasunaga. Ishan Misra, Luke Zettlemoyer, Lili Yu, and Chunting Zhou. Cat: Content-adaptive image tokenization, 2025.
12

---

[40] Ivan Skorokhodov. Willi Menapace, Aliaksandr Siarohin, and Sergey Tulyakov. Hierarchical patch diffusion models for high-resolution video generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7569-7579, 2024.
[41] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos the wild, arXiv preprint arXiv: 2012.
[42] Peize Sun, Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng. Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation. arXiv preprini arXiv:2406.06525, 2024.
[43] Anni Tang, Tianyu He. Junliang Guo, Xinle Cheng. Li Song, and Jiang Bian. Vidtok: A versatile and open-source video tokenizer. arXiv preprint arXiv:2412. 13067, 2024.
[44] Zineng Tang. Long Lian, Seun Eisape, XuDong Wang, Roei Herzig, Adam Yala, Alane Suhr, Trevor Darrell, and David M. Chan. Tulip: Towards unified language-image pretraining, 2025.
[45] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems. 37:84839-84888 84865, 2024.
[46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar0iv:2302. 10000...997 13971, 2023.
[47] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[48] Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with learned autoregressive generative prior. arXiv preprint arXiv:2410.21264, 2024.
[49] Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: joint vidgeovideo tokenizer for visual generation. Advances Neural Information Processing Systems. 37:28281-28295. 2024.
[50] Junke Wang. Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang. Zuxuan Wu, and Yu- tu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl, 2025.
[51] XuDong Wang, Xingyi Zhou, Alireza Fathi, Trevor Darrell, and Cordelia Schmid. Visual lexicon: Rich image features in language space, 2024.
[52] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu. Zizheng Pan. Wen Liu, Zhenda Xie. Xingkai Yu, Chong Ruan, al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024.
[53] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang. Ligeng Zhu, Enze Xie. Hongxu Yin, Li Yi, Song Han, and Yao Lu. Vila-u: unified foundation model integrating visual understanding and generation, 2025.
[54] Jinheng Xie, Weijia Mao. Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu. Zhijie Chen. Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[55] Yazhou Xing, Yang Fei, Yingging He, Jingye Chen, Jiaxin Xie, Xiaowei Chi, and Qifeng Chen. Large motion video autoencoding with cross-modal video vae. arXiv preprint 2rXiv:2412.17805 2024.
[56] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation, 2025.
[57] Wilson Yan, Volodymyr Mnih, Aleksandra Faust, Matei Zaharia, Pieter Abbeel, and Hao Liu. Elastictok: Adaptive tokenization for image and video, 2025.
[58] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115, 2024.
[59] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng. Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. RI -nevvisis: Advancing generalized multimodal reasoning through gross-modke formalization, 2025.
13

---

[60] Jiahui Yu, Xin Li. Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu. Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint 1xX0v:2110.04627. 2021.
[61] Lijun Yu, Yong Cheng. Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann. Ming-Hsuar Yang. Yuan Hao, Irfan Essa. et Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-
10469, 2023.
[62] Lijun Yu, José Lezama, Nitesh B Gundavarapu. Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion -okeenize is key to visual generation. arXiv preprint arXiv:231000777,, 2023.
[63] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chin Chich Chen. An image worth 32 tokens for reconstruction and generation, 2024.
[64] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue. Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Weng Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025.
[65] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11975- 11966 2023.
[66] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[67] Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. arXiv preprint arXiv:2405.20277 2024.
[68] Ziqin Zhou, Yifan Yang. Yuqing Yang. Tianyu He, Houwen Peng, Kai Qiu, Qi Dai, Lili Qiu, Chong Luo, and Lingqiao Liu. Hitvideo: Hierarchical tokenizers for enhancing text-to-vides generation with autoregressive large language models, 2025.
[69] Xianwei Zhuang, Yuxin Xie, Yufan Deng, Dongchao Yang, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt-v1.1; Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning. 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Reflected in Section contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper do not propose new theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The reproduce details are already included in the experiment section and appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide the code in supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com /ttc: ssiniioiscyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The details are reported in the experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not include error bars the experiments computationally expensive, and existing related literature do not have the convention to report error bras.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
We discuss the details in the first subsection of experiment and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///uuurppcccoooc/caacaaaaaa
Answer: [Yes]
Justification:
We have reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We expect our model to have similar risk as the Lumina-mgpi 24 or other models. We discuss potential mitigation strategies appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cited all code, data, and models used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Included code in supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work do not include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Only used LLMs for editing (e.g., grammar, spelling, and word choice).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: 22tppp///..///oooeeeeee////////0 for what should or should not be described.
21

---

Appendix:
A
Additional Methodological Details about Vapor Tok
A.1
Computation of video spatio&temporal complexity
Given video tensor V RTTxxxWxx each frame is converted grayscale via
(x, v) 0.299 Vz,.,,11 0.587 V1,2,y22 + 0.114 V v,333
(20)
.where = I..... T and (z, v) E {I...., , x {1.....Wi W}.
Spatial Complexity. Define the empirical pixel-value distribution of frame
(v( HW [(2, v) (x( - -//. w= v=0,
0, 1.... 255.
(21)
The Shannon entropy of frame / is
255
H, = --=-2 - (v) log, Pr();
(22)
v=0
and the spatial complexity is the average frame entropy:
T
SC = T
H
(23)
Temporal Complexity. The temporal complexity is defined as the mean absolute difference between consecutive frames:
T-1 H
w
TC (-1))H Σ
ΣΣΣ
90+1 (0+1(x,1)) (x, ge (x, v)
(24)
A.2 AR prior in VaporTok
Inspired by the AR prior model introduced in LARP 48 we integrate a similar lightweight autore- gressive model into VaporTok as shown in Figure 6 This AR model is designed to make latent tokens more compatible with downstream AR generation tasks, and thus its implementation and associated evaluation metries can serve as proxy for downstream generation performance.
Similar to LARP, our lightweight AR model trained jointly with the VaporTok tokenizer an end- to-end manner. Specifically, the model takes the quantized latent token embeddings input, and uses the corresponding codebook IDs as labels. To address the instability caused by the tkining-inferene discrepancy inherent to AR models, Scheduled Sampling Mixing proposed in 3 30 is employed.
The key difference is that the prior model used in VaporTok is trained only on the retained latent tokens after truncation rather than the whole latent space. Moreover, to enable efficient batchwise training, we adopt the attention masking scheme described in Section A.3 within the transformer blocks of the AR prior model.
A.3 Attention mask in reconstucttionngeeeratto pipeline
Attention mask for the VaporTok decoder. During the training of VaporTok, the spatiotemporal complexity of each video sample varies, which results in different taildrop probabilities. Additionally, since sampling performed over the entire probability distribution during training. These two factors lead to varying truncation positions across samples. Consequently, it becomes infeasible to reconstruct all samples within batch using shared decoder input length. To address this issue, we design adaptive attention mask for the VaporTok decoder to accommodate the variable token lengths caused by the probabilistic taildrop. Specifically, as illustrated in Figure aa): For each sample, construct individual attention mask: all tokens from decoder queries M are granted
22

---

Video Patches
Encoder Queries
Codebook Embedding
VaporTok Encoder
Trundsorrieer Block
Trunsforterr Block 2
TPQ Module
AR Prior
Quantizer
Yaildrop Probability
Trans foansr otner Block N
Tuildrop
Codebook
Prior Loss
VaporTok Decoder
Figure AR prior model in VaporTok.
full visibility, while for latent tokens, positions beyond the truncation point are masked out to ensure that dropped tokens do not participate in attention computation. This prevents non in-fommmaatiee tokens from interfering with the training process and allows batchwise training of VaporTok.
Attention mask for the AR prior model. The AR prior model originally adopts causal attention mask, which ensures that later tokens not affect the prediction of earlier tokens. However, if we apply a standard causal mask without modification, tokens before the truncation point can still attend to and influence those after the truncation, which is undesirable. Due to the introduction of taildrop, tokens after the truncation point should not be supervised and influenced by prior tokens during training. To resolve this, we propose a modified attention mask as shown in Figure bb).
Attention mask for the downstream AR generative model. Since the AR prior model serves as a compact abstraction of the downstream AR generative model, the attention mask used in the downstream AR model is identical to that of the AR prior model, which is also illustrated as Figure bb).
A.4 The complete loss function of VaporTok
Lree ALI LLI + Apene Leere + AGAN
LGAN + Acommit Leummit
(25)
N
reppresentation prior
log xx))
(26)
N
Ppobahility peior KL(P GaGGsaaaaPPiiiii Prior)
(27)
Cree is comprised of L1 loss, perceive loss, GAN loss, and commitment loss as traditional VQ
tokenizer. In Equation 26 denotes codebook embedding, y denotes codebookid. In Equation 27 P
denotes taildrop probability and GaussianPrion denotes prior probability. The complete loss of
VaporTok is:
Ceomplete
Lres + Arep frep + Apmi Eepohh_piior
(28)
B
Supplementary Experiments on Vapor Reward
B.1 Penalty reward
In our experiment of argmax sampling probabilistic taildrop, the truncation index always becomes too small to be enough to reconstruct&geeeratte videos, reward to punish such truncation is
23

---

Preserved lateat tolenss
Queriies
Dropped lsteat tokeas
proppodd
Decoder Queries
Preserved
Dropped
Preserved latent tokens
Dropped latent tokens
(a) Attention mask VaporTok decoder
(b) Attention mask for AR modee(PFor&&Geeerrayy
Figure Attention mask for different parts.
introduced in such senario. Specifically, for the i-th path, the penalty reward is defined
R(() penalty
Penalty(Fas), if Tis Nehreshold
(29)
where K Kthreshold is a manually defined threshold specifying the minimum number of tokens tolerable for reconstructing the video clip and Penalty (t,) denotes a penalty function that imposes more punishment when the truncation index becomes smaller. Then we design five way ablation study in argmax sampling strategy of probabilistic taildrop as shown Table
Table 5: Ablation of each rewards. Each row "w/o R" indicates the model trained without reward R. Lower better for rFVD, gFVD and MSE; higher is better for PSNR and prior top-5 accuracy.
Missing Reward
#Tokens
rFVDJ
PSNR1
gFVDJ
MSEL
ACC1
VaporTok-GRPO
299
72.5
24.1
118
4.85 x 10
3.24%
w/o efficiency reward
577
59.4
25.6
88.46
3.89 x 10
3.88%
w/o diversity reward
305
75.4
24.7
105.85
4.80 x 10-3
3.24%
w/o reconstruction reward
272
91.6
23.3
138.58
5.83 x 10-3
3.13%
w/o generation reward
286
83.6
23.8
122.21
5.33 x 10 3
3.17%
w/o penalty reward
58
3267
10.7
3255.59
9.45 x 10
2.73%
B.2 Effect reconstruction&ggeerrttoo reward under different weight
We adopt a baseline weighting of 1:1:1:1:1 for the efficiency, penalty, diversity, reconstruction, and generation rewards respectively and then increased the proportion of each of reconstruc- tion&generttion rewards. As shown Figure as the weights for these two rewards grow, their numerical values also increase, which demonstrates that the higher the combined weight on these two performance rewards, more faithfully the original reconstruction and generation quality is preserved. It is worth noting that if both weights are set too high, performance gains come at the expense of efficiency reward, contradicting our original intent that making tokenizer to be more efficient.

---

Goneration Sword
Pacoetaationn BrARd
1.5
53103
4.08
1154 18031 13303
35
0.14
5.5
4.10
0.33
2
(a) Generation Reward
(b) Reconstruction Reward
Deop
(c) Efficiency Reward
(d) Diversity Reward
Figure 8: (a) generation reward (b) reconstruction reward (c) efficiency reward (d) diversity reward of
differenct reward weight where the order is efficiency, penalty, diversity, reconstruction, generation.
C Analysis about VaporTok
C.1 Mitigating the three core challenges of autoregressive generation
To mitigate the three key limitations of autoregressive (AR) visual generation, we propose a unified solution within the VaporTok framework:
Quadratic complexity with long sequences. We introduce a sparse but sufficient token representation by leveraging visual priors to reduce the number of tokens required for generation. Furthermore, we apply GRPO to adaptively compress the token sequence while preserving downstream task performance much possible.
Error accumulation during AR inference. We propose a probabilistic taildrop training strategy that pushes important tokens toward the beginning of the visual representation. As result, during inference, the model generates the most critical tokens when the accumulation of prediction errors is still minimal, thereby mitigating the impact of error accumulation.
Training gap between the tokenizer and the AR generator. To close this gap. we adopt two complementary strategies. First, we refine the latent space following the approach of LARP 48) to make more suitable for AR generation. Second, to guide the training of the taildrop query module via GRPO. This enables our adaptive tokenizer incorporate downstream task constraints into both the adaptivity mechanism and the latent token representation, thus reducing the mismatch between the tokenizer and the generator.
leverage the same AR-aligned information
C.2 Taildrop training strategy for semantic-to-detail representation
The taildrop training strategy designed to encourage a semantic-to-detal organization the latent token representations. This is primarily achieved through the following mechanism: for the same video sample, train the model using different numbers of tokens for reconstruction. As a result, tokens at the beginning of the sequence are exposed to the reconstruction objective more frequently, while later tokens appear less often during training. Furthermore, even when only a small number of tokens are used, the model is still tasked with reconstructing the entire video. This encourages early tokens to encode more global, semantic information that is sufficient for reconstruction.
In addition, it is worth noting the distinction between the two evaluation metrics used in visual reconstruction and generation: rFVD and gFVD By design, FVD is consistently worse (higher) than rFVD. This is because rFVD evaluates the reconstruction quality using ground-truth codebook ID directly obtained from the encoder, whereas gFVD evaluates the generation quality using codebook ID predicted by the autoregressive model. Brieffy, the only difference between rFVD and gFVD is that
25

---

reconstruction relies on ground-truth ID, while generation depends on autoregressively predicted ID. making the gap between gFVD and rFVD direct indicator of the degree of AR ervor accumulation.
1144
1144
1,0
3.24
1.244
3-83X
60
DVD
40
23
20
LAMP
1101
2004
512
(a) Comparison between LARP and VaporTok with naive taildrop under 1024 and 512 token budgets
(b) Comparison between no technique, naive taildrop, and probabilistic taildrop under 500 token budgets
Figure 9: (a) Naive taildrop reduces the gap between rFVD and gFVD, but leads to drop in recon- struction performance, which in turn results in degraded generation performance. (b) Probabilistic taildrop, by incorporating a visual prior, avoids the reconstruction performance degradation caused by taildrop, while preserving its original ability to reduce the gap between rFVD and gFVD.
To quantify this effect, we report the gFVD/rFVD ratio as shown in Table 2 of the main paper and FigurePaljo measure the discrepancy between reconstruction and generation. We conduct experiments under both 1024-token and 512-token settings. The results show that with taildrop enabled, the gap between gFVD and rFVD becomes smaller, demonstrating the effectiveness of taildrop in mitigating AR error accumulation; specifically, the gFVD/FVD ratio decreases from 2.85 to 1.26 under the 1024-token setting, and from 1.62 to 1.14 under the 512-token setting.
C.3 Difference between naive taildrop and probabilistic taildrop
The distinction between naive and probabilistic taildrop primarily manifests in training and inference:
While naive taildrop uses uniform sampling for truncation during training, probabilistic taildrop samples from learned, informenformed distribution. (We also investigate three specific sampling strategies under the probabilistic framework.)
Training:
Truncation by argmax of taildrop probability: The sequence is truncated at the index corresponding the maximum value in the taildrop probability distribution. While simple, this approach always uses the same number of tokens for given input, limiting the semantic- to-detail effect.
Sampling from the full taildrop probability: The truncation index is sampled from the entire taildrop probability distribution. This allows different truncation lengths for the same input and leads to strong reconstruction performance.
Sampling indices before the argmax index. We sample truncation index from the region before the argmax position, based on the taildrop probability. This also enables varying token counts across samples, though typically results in fewer tokens and slightly degraded reconstruction quality.
Inference: During inference, naive taildrop uses the full set of tokens for decoding (which is the default inference mode used for all reported results in this paper. Alternatively, one may adopt a threshold- harsshold-based token selection strategy, in [57]. In contrast, probabilistic taildrop performs de- coding using all tokens preceding the argmax index of the taildrop probability distribution, achieving an adaptive number of tokens based on input complexity.
worth noting that, among the three sampling strategies for probabilistic taildrop during training (from argmax to sample and then to pre-sample), the degree of cemantic-to-detal structure in the
26

---

Raw Video
Video Patches
Encoder Queries
Video Patches
Encoder Queries
Video
Encoder
Encoder
Encoder
Quantizer
Quantizer
Quantizer
Decoder Queries
Decoder
Decoder
Taildrop
Decoder Queries
Video
Decoder
Putchwise paradigm
(b) Holistic parudigm
(c) Holistic paradigm with tuildrop
Figure 10: Different paradigm of visual tokenizer: (a) The patchwise-token paradigm typically represents each token as encoding information from specific spatial region and usually adopts CNN as the backbone.(b) The holistic-toker paradigm is not constrained by fixed spatial positions and can flexibly adjust the information each token represents based on the training strategy. (c) Taildrop is training technique commonly used in the holistic-token paradigm, enabling the token sequence to exhibit semantie-to-detai property.
tokens used during inference gradually increases because of the frequency of sampling index before argmax index is gradually frequent. In other words, these strategies increasingly mitigate AR error accumulation. This trend is also validated by the experimental results reported Table of the main text the gFVD/FFD ratio becomes smaller.
Furthermore, while naive taildrop helps narrow the gap between reconstruction and generation (rFVD vs. gFVD), its uninformed dropping during training-without accounting for visual priors-results in compromised reconstruction quality. In contrast, our probabilistic taildrop incorporates learned prior distribution, maintaining competitive reconstruction performance (rFVD) and achieving superior generation quality by alleviating AR error accumulation.
C.4 Different visual tokenizer paradigm
Visual tokenizers can be classified according to various criteria, and one particularly informative distinction is how they map visual patches to latent tokens, yielding two families: Patchwise-Tool Token and Holistic-Token.
In the Patchwise-Tokes paradigm as depicted in Figure 10 a) each learned token corresponds one-to-one with a visual patch, thereby preserving the spatial structure imposed. Briefly, given video input V [xTxxxWxx x3 the encoder outputs downsampled feature map
w
Z Enc(V) E
(30)
where fr, fu, fw are the temporal and spatial downsampling factors. The reconstructed video is then obtained as
V = Dec(2) Z) 2T'xHxxx3
(31)
In the Holistic- Token paradigm as depicted in Figure 10 each latent token may assume different semantic roles depending on the training strategy, resulting more flexible representational scope. Different from Patchwise-Tool Token paradigm mainly depends on 3D CNN as the backbone, Holistic- Token paradigm usually employs Transformer blocks as the backbone, so simple patch embedding layer is needed to be conducted to V E T H x3
P = Patchify(V) E R
(32)
27

---

and then P will be concated with K learnable query tokens Q I2 RxXD and the combined sequence will be passed into the encoder:
Zp @ Zo = Enc + Q)
E IR
fw +K)xD
(33)
where @ denotes concatenation and Zp.Zq denotes the represatation of P. Q after encoder respec- H W )x the decoder query of the same shape as tively. During detokenization, M E / TH Tw
the video patches P, will be concatenated with 7 and passed to the decoder to renconstruct the input video.
V = Dec(M @ 2Q) E T
(34)
It is worth noting that the taildrop training strategy is typically employed in holistic-token tokenizers. This is primarily because the query representation is not constrained by fixed spatial regions, allowing it to flexibly adapt to different training objectives and strategies. Moreover, such flexibility enables the model achieve effective representation learning with less training data.
D Supplementary Related Work about Fixed-length Visual Tokenizer
Visual tokenizers for understanding tasks typically rely on contrastive learning, for example: CLIP 34 trains paired image and text encoders with an InfoNCE contrastive loss over matched image-caption pairs, enabling strong zero-shot transfer across diverse vision task; SigLIP replaces CLIP's softmax-based InfoNCE loss with an independent pairwise sigmoid loss, removing the need for global normalization and scaling more efficiently to very large small batch sizes. TULIP[44) augments CLIP-style pretraining with generative data augmentation and unified image-image, text-text, and image- text contrastive objectives plus reconstruction regularization to learn fine-grained visual features without sacrificing semantic alignment.
Whereas visual tokenizers designed for generation usually employ VAE-based architectures: VQ- VAE(47] first introduce vector quantization into VAF, transforming data from continuous spaces into discrete tokens to simplify modeling and circumvent issues of "posterior collapse" in VAE framework; VQ-GAN improves image reconstruction quality by introducing adversarial loss and using Transformer for autoregressive visual generation; FSQ[29) projects representations into a lower dimensional space for quantization into fixed values, while its variant LFQ 27 further simplifies the process by using binary quantized representations. This new kind of quantization method effectively enhances the AR generation paradigm by dramatically enlarging the vocab size and improving the encoding efficiency. Beyond these patch-to-token VAEs, there also VAEs that learn holistic tokens, such as TiTok 63 and LARP(43), by compressing visual information into a holistic query, they eliminate the patch-to-token correspondence constraint, yielding more flexible architecture with inherent compression potential.
Recently, along with the rapid development of the unified model[ 54 53, 69]. there are also several visual tokenizers designed for both generation and understanding, such as TokenFlow [33], SemHiTok [ and UniTok These works focus unifying visual generation and understanding within single visual tokenizer by employing discrete representations and hierarchical or multi- codebook strategies, addressing the differing requirements in token granularity and semantic level between generation and understanding tasks.
While the above methods have shown impressive results for static images. extending visual tokenizers video requires capturing both temporal continuity and spatial detail, presenting new challenges for tokenizer architectures and training paradigms. Recent work addresses this by integrating temporal modeling 49), diffusion-guu guided reconstruction [1)., hierarchical codebooks(68), and coordinate- patch schemes 20 to efficiently compress and faithfully reconstruct long video sequences.
E
Visualization
We present visualizations of VaporTok's reconstruction and final class-based generation results, as shown in the Figure [] and Figure 12
28

---

GT
Reamsstaciion
Resemimmttio
GT
Recontrucioo
GT
CT
Figure 11: Video reconstruction on UCF101.
29

---

Figure 12: Class-based video generation on UCF101.
F
Broader Impacts
Our Adaptive Video Tokenizer is designed exclusively for autoregressive video generation tasks, and not intended for video understanding or classification; by allocating fewer tokens to low-complexity videos and more high-detail videos, method reduces computational and bandwidth costs for real-time generative applications (e.g., interactive video editing, virtual content creation); enables fast, adaptive video synthesis for artistic tools and educational simulators, lowering barriers for non-expert users to generate high-fidelity video content; it facilitates deployment of generative video models on edge devices (c.g., AR/VR headsets, mobile phones) by reducing token sequence length and inference latency; however, improved efficiency in video generation could be misused to produce highly realistic deepfake videos, exacerbating misinformation campaigns; although not designed for recognition, the underlying tokenizer could be adapted to generate misleading synthetic footage for surveillance evasion or identity spoofing; training on unbalanced datasets may lead the tokenizer to allocate token budgets unevenly, causing generative artifacts that disproportionately affect certain demographics; while inference is more efficient, training the dual-branch model remains GPU-intensive, contributing to carbon emissions.
30

---

G
Safeguards
To mitigate potential misuse and harms, we will release detailed model card specifying that use for deceptive or harmful video synthesis (e.g., deepfakes) is prohibited; distribute weights under non- commercial, no-derivatives license (e.g., CC BY-NC-ND) and/or via an API with rate limits rather than open weight download; embed imperceptible watermarks in generated videos for provenance tracking and provide a companion detection model flag synthetic content; maintain a public issue tracker for misuse reports and regularly update the model to address discovered biases or vulnerabilities; publish training logs, compute cost estimates, and carbon-emission metries to inform users of environmental impact.
31


Reviews:
[{"review_id": "IDQB6t8Z6W", "reviewer": "Reviewer_R7pP", "summary": "This paper develops VaporTok, a model for adaptive video tokenization that accounts for both content complexity and for downstream tasks like generation. Regarding content complexity, instead of using uniform taildrop (nested dropout), VaporTok learns to predict the number of tokens to use for a video clip based on the content and a visual complexity prior. This information is used during training to better optimize VaporTok to use more tokens for complex scenes and fewer tokens for simple ones.\n\nRegarding awareness of downstream tasks, the authors develop a method for training the taildrop probability query module using GRPO. Learning is driven by four reward signals that capture efficiency (fewer tokens is better), diversity (avoid mode collapse), reconstruction (preserve the autoencoder functionality of the tokenizer), and generation (awareness of the downstream video generation task).\n\nThe authors evaluate VaporTok using the UCF-101 and Kinetics-600 video datasets. They show reasonable gFVD results using 50-65% fewer tokens than previous methods (Table 1). Extensive ablation studies are also performed to understand the impact of different design choices (taildrop vs. probabilistic taildrop, sampling method, etc.).", "questions": "Q1. How does VaporTok compare to existing methods when using (roughly) the same number of tokens on average? The expectation is that an adaptive method should perform much better than a fixed method on a diverse dataset since it can more efficiently allocate tokens: fewer for simple videos and more for complex videos leading to a net improvement in reconstruction quality.\n\nPerhaps this is shown in Table 2 comparing LARP with 512 tokens to VaporTok with 498 where rFVD is nearly the same and gFVD is slightly betterfor VaporTok? If you agree this is a fair comparison, why isn't there a larger benefit from the adaptive tokenization of VaporTok?\n\nTo better motivate my question, adaptive tokenization is a kind of variable rate compression, and the standard visualization for codecs is a rate-distortion curve. For tokenization, the key graph would show FVD vs. average number of tokens. Importantly, it's very difficult to compare different methods from a single point (one rate and one distortion) if one method doesn't \"dominate\" the other (lower rate and lower distortion). Table 1 doesn't show this since VaporTok has fewer tokens but also worse FVD so it's hard to draw a strong conclusion.\n\n\nQ2. Does VaporTok support different token counts for a single video? I think it does since the taildrop index is sampled during training so some flexibility should exist at inference time. Assuming that's right, I'd expect to see a visualization that shows how reconstructions change as you move from very few to a relatively large number of tokens for a particular video.\n\nSuch a visualization would also make it clear whether early tokens truly encode *semantic* information or if the representation is better described as coarse-to-fine. For example, if you reconstruct a video of a dog with very few tokens, do you get a sharp video of a different dog or a blurry/distorted video of the same dog?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Quality -- The quality of this paper is generally very high. The methods used make sense, and I see no technical issues with the formulation. The evaluation is quite thorough in terms of ablations, comparing to previous methods, and showing the impact of different reward signals. However, there's a gap in the evaluation related to adaptivity since results are only shown for a single average token count (see the Questions section for more detail).\n\nClarity -- The paper is very clearly written and well-organized. The authors also provide code.\n\nSignificance -- The addresses two problems of interest within video tokenization and generation, and so I think the significance is quite high. First, adaptive tokenization is a very active area of research right now since the memory and computational costs of video modeling (for understanding and generation tasks) is quite high in part because most methods use highly redundant, and thus wasteful, representations. Second, task-aware and joint optimization of multiple components within a generative system is also an active area of research since training components in isolation is unlikely to yield optimal results.\n\nOriginality -- Both of the primary contributions of this paper (probabilistic taildrop and RL for task-aware tokenizer training) are novel as far as I know.", "was_revised": true, "final_justification": "Based on the authors' responses to the initial reviews, I increased my rating from \"borderline accept\" to \"accept\".\n\nI think this is justified because adaptive tokenization is a growing research area for generative image and video models, and VaporTok provides a solution to the problem of visual quality degradation due to taildrop, which is a common method for achieving adaptive representations. This approach will be interesting to many researchers in the NeurIPS community.\n\nI don't think that a \"strong accept\" is justified due to some shortcomings in the current method:\n\n1) The semantic-to-detail hierarchy does not appear to extend to the case where very few tokens are used (contrast with Flex\nTok)\n\n2) While the \"probability traildrop\" method presented in the paper compensates for the quality degradation caused by \"naive taildrop\", it's reasonable to believe that an adaptive method will outperform fixed-rate methods (even at moderate-to-high token counts), which is not achieved by VaporTok. So I think probability taildrop is an interesting and important step forward, but there's room for much better solutions to high-quality adaptive tokenization.", "rebuttal": "We thank the reviewer for their recognition of our work and for the constructive comments. We provide our feedback as follows:\n\n---\n\n### **Q1-1: How does VaporTok have advantages over other methods?**\n\nOur probabilistic taildrop was configured with a maximum token length of 1024. However, since we utilize adaptive token count during both training and inference, the actual average number of tokens used is less than 1024, averaging around 500 tokens. To enable a fairer comparison, we primarily compare our approach against the 512-token scheme in this context, as follows: \n\n| Method | Training Technique | Token | rFVD | gFVD | gFVD/rFVD |\n| --- | --- | --- | --- | --- | :---: |\n| LARP | none | 512 | 53.25 | 86.25 | 1.62 |\n| VaporTok | naive taildrop | 512 | 81.94 | 93.34 | 1.14 |\n| VaporTok | probability taildrop | 498 | 53.92 | 80.13 | 1.48 |\n\nOur approach offsets the rFVD degradation caused by taildrop, achieving rFVD on par with the baseline. Furthermore, our probabilistic taildrop also exhibits the semantic-to-detail property, so it delivers superior generation performance and lower gFVD/rFVD than the baseline, demonstrating that we effectively mitigate the training-inference gap of the downstream AR generation model.\n\n---\n\n### **Q1-2: Why isn't there a larger benefit from the adaptive tokenization of VaporTok?**\n\nWhen training a visual tokenizer, both reconstruction and generation metrics are important; however, since our goal is to solve the AR generation problem, we place greater emphasis on generation. Taildrop is inherently unfriendly to reconstruction (as shown by the TiTok vs. One-D-Piece reconstruction results below), but by ordering the latent space from semantic to detail it makes the VAE decoder more robust to later tokens, thereby boosting generation. Our method retains this strong generation performance while cleverly overcoming taildrop’s reconstruction weakness, achieving rFVD results on par with the baseline.\n\nThis degradation in rFVD performance actually occurs in all naive taildrop methods. For example, the comparison between TiTok[1] and One-D-Piece[2] (TiTok trained with naive taildrop), as shown in Table 2 and Figure 4 of the One-D-Piece paper, demonstrates that using the same number of tokens leads to a drop in rFVD, specifically:\n\n- TiTok-L with 32 tokens: rFVD increases from 2.21 (TiTok) to 3.23 (One-D-Piece)\n- TiTok-B with 64 tokens: rFVD increases from 1.71 (TiTok) to 2.39 (One-D-Piece)\n- TiTok-S with 128 tokens: rFVD increases from 1.70 (TiTok) to 1.96 (One-D-Piece)\n\nAs for the causes of this reconstruction performance degradation, they may include the following:\n\n- One-D-Piece attributes this phenomenon to the fact that too few tokens cannot deliver good perceptual quality.  Likewise, applying naive taildrop to compress the full set of 1,024 tokens when representing a 16-frame video may similarly undermine its representational capacity.\n- Moreover, current methods apply taildrop on the discrete latent representations, i.e., they perform VQ, immediately follow it with taildrop, and then reconstruct via the decoder. This incurs two successive levels of information loss (the per-token quantization loss from VQ and the sequence-level information loss from taildrop), making it more difficult to optimize the VQ codebook, thereby leading to decreased rFVD performance. We leave this aspect for future exploration.\n\nOur probabilistic taildrop, by contrast, compensates for the rFVD degradation caused by taildrop by adjusting the number of tokens according to the visual complexity of the data, thereby alleviating this issue. Furthermore, our training approach arranges the video latent space in a semantic-to-detail order and strengthens the decoder’s robustness to later tokens, thereby enhancing generation performance.\n\n[1] An Image is Worth 32 Tokens for Reconstruction and Generation\n\n[2] One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression\n\n---\n\n### **Q2: Does VaporTok’s latent space truly exhibit the semantic-to-detail (or coarse-to-fine) property?**\n\nVaporTok supports reconstruction with varying token counts. (For details on the distinction between probabilistic taildrop during training and inference, see Appendix C.4. In brief, during training with probabilistic taildrop we could employ three different modes to get truncation index—argmax, sample, and pre-sample—whereas at inference, to eliminate randomness, we report results obtained exclusively via argmax sampling.) Also, **by directly setting the truncation index, we can control how many tokens are used for reconstruction**. However, if using a fixed token count for inference, VaporTok will lose the token-efficiency advantages provided by its adaptivity.\n\nDue to this year’s NeurIPS policy, we cannot submit a PDF to display the semantic-to-detail visualization. Instead, we implicitly demonstrate this trend through reconstruction metrics at different token counts. In addition to rFVD, we report MSE, PSNR, and LPIPS of VaporTok and LARP(512) at different token counts. **With an extremely small number of tokens, VaporTok significantly outperforms LARP(512), demonstrating that its head tokens capture strong semantic information and exhibits the semantic-to-detail characteristic.**\n\nThe reconstruction results of V(VaporTok) and L(LARP-512)：\n\n| Token counts | V-MSE | L-MSE | V-PSNR | L-PSNR | V-rFVD | L-rFVD | V-LPIPS | L-LPIPS |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| 16 | 0.0400 | 0.1964 | 14.62 | 7.46 | 1774 | 4020 | 0.484 | 0.789 |\n| 32 | 0.0340 | 0.1431 | 15.36 | 8.94 | 1449 | 3816 | 0.441 | 0.775 |\n| 64 | 0.0210 | 0.1603 | 17.51 | 8.34 | 858 | 3825 | 0.346 | 0.765 |\n| 128 | 0.0113 | 0.1479 | 20.24 | 8.68 | 387 | 3222 | 0.240 | 0.661 |\n| 256 | 0.0058 | 0.0235 | 23.35 | 17.53 | 101 | 521 | 0.155 | 0.325 |\n| 512 | 0.0036 | 0.0031 | 25.40 | 26.10 | 60 | 53 | 0.115 | 0.105 |\n| 1024 | 0.0024 | \\ | 27.19 | \\ | 41 | \\ | 0.09 | \\ |\n\nAs for the qualitative visualizations, our final results are nearly indistinguishable from those shown in One-D-Piece (see Figure 1 of One-D-Piece paper). The critical factor in exhibiting the semantic-to-detail characteristic is using varying token lengths during training: naive taildrop chooses lengths entirely at random, whereas we sample lengths from a learned distribution. Although this sampling can somewhat attenuate the semantic-to-detail effect, we still train with different token lengths , so VaporTok naturally displays the same characteristic with One-D-Piece."}, {"review_id": "YNQzZrK2rU", "reviewer": "Reviewer_FNXp", "summary": "This paper innovatively proposes an adaptive video tokenizer that adjusts the token count: \n1) A probabilistic taildrop module is learned by aligning with a predefined explicit video complexity prior.\n2) In the second stage, the probabilistic taildrop layer is further optimized using reinforcement learning with GRPO, where the probabilistic taildrop module serves as the policy model and is optimized according to multiple rewards.", "questions": "See weakness above", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper is well-written and easy to follow. To facilitate the shift from fixed-rate to adaptive downsampling, the authors novelty parameterized and optimized the truncation probabilistic taildrop supervised by a visual prior, then further used GRPO to optimize the truncation of the entire video clips as  a sequential decision process. \n\n2. The experimental results on UCF101 and K600 show that VaporTok+GRPO reduces token length by 75%, demonstrating the effectiveness of the adaptivity token adjusting method. \n\n\nWeaknesses\n1. The expression in abstract line 19, \"matches and outperforms fixed-rate baselines and naive taildrop while using fewer tokens\" is somehow unclear regarding which model is referred to as the \"fixed-rate baseline\" and \"naive taildrop\" in Table 1.  From my review of the codebase, it appears that the implementation closely follows the open-sourced LARP model, so I assume the \"fixed-rate baseline\" refers to LARP. However, VaporTok uses lossy token truncation compression, which introduces a gap compared to LARP in rFVD, from 20 to 53.9(VaporTok) or 66.6(VaporTok-GRPO). The authors do not discuss in the experiment whether this gap is significant or minor.  Simply saying \"competitive\" is not sufficient.\n\n2. In line 276, it says, “From the first two rows in Table 2: taildrop causes a substantial performance drop in rFVD, from 20 to 49.45.”   I’m confused about why VaporTok with Taildrop still maintains a token length of 1024 same as the LARP baseline.  Could you clarify what exactly is being done here and where the performance loss comes from? Specifically, what is the operation involved, and how does it lead to this drop in performance with the same token length?\n\n3. It would be better to include a discussion on the visualization of the Taildrop Probability Query Module after GRPO at different truncation points across various video clips.  Specifically, it is unclear which videos adopted longer or shorter truncation points, and whether the visualizations align with prior assumptions.   This analysis may be beneficial to investigate whether cold-starting the module using the current visual prior is both expected and necessary", "was_revised": true, "final_justification": "I'm satisfied with the response and keep my original rating.", "rebuttal": "We thank the reviewer for acknowledging the novelty of our method and the effectiveness of its efficiency. Below, we provide our responses to the reviewer’s questions.\n\n---\n\n### **Q1: What do “fixed-rate baseline” and “naive taildrop” refer to in the abstract of the paper?**\n\nThe terms 'fix-rate baseline' and 'naive taildrop' mentioned in the abstract refer to the third and fourth rows in Table 2 of the main paper, respectively. Our probability taildrop experiment was configured with a maximum token length of 1024. However, since we utilize adaptive token count during both training and inference, the actual average number of tokens used is less than 1024, averaging around 500 tokens. The comparison of LARP in table1 of main paper is not a fair comparison, as LARP is using a larger number of tokens, so we have constructed a fairer comparison below (with token counts of around 500). We mitigated the shortcoming of naive taildrop causing rFVD degradation. While achieving rFVD comparable to the original baseline, our approach achieves a better gFVD and lower gFVD/rFVD ratio, demonstrating that we effectively reduced the training & inference discrepancy in the AR model through this method.\n\n| Method | Training Technique | Token | rFVD | gFVD | gFVD/rFVD |\n| --- | --- | --- | --- | --- | :---: |\n| LARP | none | 512 | 53.25 | 86.25 | 1.62 |\n| VaporTok | naive taildrop | 512 | 81.94 | 93.34 | 1.14 |\n| VaporTok | probability taildrop | 498 | 53.92 | 80.13 | 1.48 |\n\n---\n\n### **Q2-1: Why does naive taildrop still use 1,024 tokens during inference?**\n\nIn Appendix C.4, we provide a detailed description of the training and inference procedures for naive taildrop. Specifically：\n\nDuring training, naive taildrop samples a truncation index uniformly at random, discards all subsequent tokens, and feeds the preceding tokens into the decoder for reconstruction. \n\nDuring inference, one may apply certain heuristics to determine how many tokens to use for reconstruction (e.g., the target reconstruction threshold employed by ElasticTok), or simply reconstruct using all tokens. Because reconstruction with the full token set always yields better performance than using only a subset, the results we report for naïve taildrop on the evaluation set are obtained by reconstructing with all 1,024 tokens.\n\n---\n\n### **Q2-2: Why does naive taildrop cause a degradation in rFVD performance?**\n\nLeveraging taildrop’s semantic-to-detail latent-space properties comes with the drawback that naive taildrop reduces rFVD. For example, the comparison between TiTok[1] and One-D-Piece[2] (TiTok trained with naive taildrop), as shown in Table 2 and Figure 4 of the One-D-Piece paper, demonstrates that using the same number of tokens leads to a drop in rFVD.\n\n- TiTok-L with 32 tokens: rFVD increases from 2.21 (TiTok) to 3.23 (One-D-Piece)\n- TiTok-B with 64 tokens: rFVD increases from 1.71 (TiTok) to 2.39 (One-D-Piece)\n- TiTok-S with 128 tokens: rFVD increases from 1.70 (TiTok) to 1.96 (One-D-Piece)\n\nAs for the causes of this reconstruction performance degradation, they may include the following:\n\n- One-D-Piece attributes this phenomenon to the fact that too few tokens cannot deliver good perceptual quality.  Likewise, applying naive taildrop to compress the full set of 1,024 tokens when representing a 16-frame video may similarly undermine its representational capacity.  Our probability taildrop, by contrast, compensates for the rFVD degradation caused by taildrop by adjusting the number of tokens according to the visual complexity of the data, thereby alleviating this issue.\n- Moreover, current methods apply taildrop on the discrete latent representations—i.e., they perform VQ, immediately follow it with taildrop, and then reconstruct via the decoder. This incurs two successive levels of information loss (the per-token quantization loss from VQ and the sequence-level information loss from taildrop), making it more difficult to optimize the VQ codebook. We leave this aspect for future exploration.\n\n[1] An Image is Worth 32 Tokens for Reconstruction and Generation\n\n[2] One-d-piece: Image tokenizer meets quality-controllable compression.\n\n---\n\n### **Q3-1: Does the taildrop probability after GRPO  align with that after first stage only optimized by data prior ?**\n\nDuring the GRPO optimization process, a KL divergence loss term is included to ensure similarity. The KL divergence between the old and new taildrop probability in GRPO loss function ensures that the multi-task-aware taildrop probability remains anchored close to data-prior taildrop probability learned in the first stage.\n\nTo verify the above statement, we compute the similarity between the taildrop probability before and after optimized by GRPO over all samples in the UCF101 validation set using the following definition (higher score indicates that the distributions are more similar)：\n\n\\begin{align}\nS_m &= \\max\\bigl(0,100 - \\min\\bigl(8 |m_a - m_b|\\,100\\bigr)\\bigr)  ,\\quad\\text{where} \\quad m_a= \\mathrm{median}(a), m_b= \\mathrm{median}(b),\\\\\n\\end{align}\n\n\\begin{align}\nS_d &= \\bigl(\\min\\bigl(100\\,100 r_d\\bigr)\\bigr)^{1.2} ,\\quad\\text{where} \\quad \\mathrm{IQR}_a= Q_3(a) - Q_1(a), \\mathrm{IQR}_b= Q_3(b) - Q_1(b), r_d= \\frac{\\min(\\mathrm{IQR}_a,\\mathrm{IQR}_b)}{\\max(\\mathrm{IQR}_a,\\mathrm{IQR}_b)}\n\\end{align}\n\n\\begin{align}\nS_{\\mathrm{KS}} &= \\max\\bigl(0\\,100 - \\min\\bigl(150 D, 100\\bigr)\\bigr) ,\\quad\\text{where} \\quad D = \\sup_x\\lvert F_a(x) - F_b(x)\\rvert\n\\end{align}\n\n\\begin{align}\nS_p &= \\max\\bigl(0\\,100 - \\min\\bigl(30 |p_a - p_b| \\,100\\bigr)\\bigr) ,\\quad\\text{where} \\quad p_a= \\mathrm{peak}(a),  p_b= \\mathrm{peak}(b).\n\\end{align}\n\n\\begin{align}\nSimilarity Score &= \\tfrac{1}{4}\\bigl(S_{m} + S_{d} + S_{\\mathrm{KS}} + S_{p}\\bigr).\n\\end{align}\n\nThe results, shown in the table below, indicate that for the vast majority of samples the two distributions exhibit very high similarity. Samples with lower similarity reflect cases where GRPO has performed more extensive exploration in order to reconcile the four distinct reward components. Perceptually, the smooth Gaussian distribution learned in the first stage undergoes peak attenuation and a shift of its mean toward fewer tokens—thereby increasing the opportunity for more tokens to be sampled and becoming more efficient—which is consistent with GRPO’s exploration&exploitation tradeoff. \n\n| Similarity Score  | Counts |\n| :---: | --- |\n| 90-100 | 644 |\n| 80-90 | 384 |\n| 70-80 | 518 |\n| 60-70 | 581 |\n| 50-60 | 593 |\n| 40-50 | 1063 |\n| 30-40 | 0 |\n| 20-30 | 0 |\n| 10-20 | 0 |\n| 0-10 | 0 |\n\n---\n\n### **Q3-2: Does the taildrop probability after GRPO reflect data simplicity or complexity?**\n\nThe taildrop probabilities after GRPO not only align with those learned in the first stage, but also genuinely reflect the complexity of the data. We apologize that, due to this year’s NeurIPS policy, we are unable to upload visualization figures; however, we can explain this from two perspectives:\n\nFrom a quantitative perspective, the taildrop probabilities learned in the first stage fully capture the data’s complexity and the GRPO-refined taildrop probabilities are highly similar to those initial values, so the GRPO-refined probabilities likewise reflect the complexity of the video data.\n\nFrom a qualitative perspective, for example, in a violin-playing video where primarily only the hands move, the mean taildrop probability is around 200 tokens; by contrast, in a soccer video with complex backgrounds and large-scale motion, the mean rises to approximately 700 tokens.\n\n---\n\n### **Q3-3: Whether cold-starting the module using the current visual prior is both expected and necessary.**\n\nAs noted in our responses to Q3-1 and Q3-2, this data prior serves the role we \"expected\". As for \"necessary\", we provide the following explanation:\n\n- Just as large language models undergo pretraining and supervised fine-tuning before GRPO to establish a strong foundation, we also require a well-initialized distribution to GRPO. This initialization stabilizes GRPO’s exploration and accelerates convergence.\n- Since our reward does not include any data-specific term, GRPO alone only trades off efficiency against reconstruction & generation. Without data-prior constraint in the first stage, the final adaptive distribution would not correlate strongly with the underlying data complexity.\n\nIn summary, we want our tokenizer to be both data-prior-aware and task-metrics-aware, so we inject the data prior in the first stage. Then, in the second stage, the KL divergence term in the GRPO loss guarantees that while the distribution is adjusted toward task objectives, it retains the information encoded by the data prior. These two phases thus complement one another to achieve our overall goal."}, {"review_id": "xBWZiNMpLe", "reviewer": "Reviewer_8AKU", "summary": "This paper proposes an adaptive video tokenizer named VaporTok. It proposes a taildrop mechanism that learns a truncation index sampling distribution conditioned on visual complexity of the video. By doing this,  the decoder reconstructs videos at adaptive token lengths, allocating more tokens to complex videos and fewer to simpler ones. Further, the authors reformulate the visual tokenization pipeline as a sequential decision process and enables metrics-aware adaptive tokenization across diverse objectives. Extensive experiments show the effectiveness of the method.", "questions": "see weakness.", "limitations": "n/a", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n+ The paper proposes an adaptive video tokenization technique adjusting token count based on visual complexity for more efficient sequences.\n\n+ The paper formulates video tokenization as a sequential decision process and uses RL to optimize the tokenizer. The idea is novel.\n\n+ The paper is well written and easy to read.\n\nWeakness:\n\n- My main concerns are about the performance comparison. For example, the Authors can consider compare with more baseline models which also use adaptive tokenizers [R1-R4]. \n\n- More analysis on model complexity needs to be given. For example, will the introduction of adaptive encoding based on reinforcement learning significantly increase the computational complexity and time delay of the model? And will it make the model significantly more difficult to optimize and train to converge?\n\n- More training details shoud to be provided. For example, how much time/GPU hours does it take to train the model? And the use of GPU cards or clusters. Besides, a comparison of the training resources with the baseline models should also be provided.\n\n[R1] Visual lexicon: Rich image features in language space. 2024\n\n[R2]  Elastictok: Adaptive tokenization for image and video. 2025.\n\n[R3] Flextok: Resampling images into 1d token sequences of flexible length. 2024.\n\n[R4] One-d-piece: Image tokenizer meets quality-controllable compression. 2025.", "was_revised": true, "final_justification": "The author response has addressed most of my concerns (baseline comparison, model complexity). So i raise my score to 4.", "rebuttal": "We thank reviewer for the constructive comments. We provide our feedbacks as follows.\n\n---\n\n### **Q1:  Compare with more baseline models which also use adaptive tokenizers.**\n\nWe have already discussed adaptive visual tokenizer approaches such as Vilex[1], FlexTok[2], One-D-Piece[3], and ElasticTok[4] in the related work section of paper; any additional methods we have not mentioned will be included in future versions.\n\nVilex, Flextok, and One-D-Piece are adaptive **image** tokenizers and therefore cannot be directly compared to **video** tokenizers under a single unified metric. Moreover, ElasticTok only addresses the reconstruction task and does not include a generative component, so we compare it solely on reconstruction performance. To broaden our evaluation, we additionally include Cosmos-Tokenizer-DV[5] and a contemporaneous method, AdapTok[6]. The comparison results are shown below:\n\n| Method | Token(Avg.) | rFVD |\n| --- | --- | --- |\n| ElasticTok | 1024 | 390 |\n| ElasticTok | 2048 | 93 |\n| Cosmos-Tokenizer-DV | 1280 | 140 |\n| AdapTok | 512 | 60 |\n| LARP | 512 | 53 |\n| VaporTok | 498 | 54 |\n\nFor AR-style downstream generation tasks, we additionally incorporate Video-LaVIT[7] to complement the gFVD metric. The comparison results are as follows:\n\n| Method | Token(Avg.) | gFVD |\n| --- | --- | --- |\n| MAGVIT-AR | 1024 | 265 |\n| MAGVIT-v2-AR | 1280 | 109 |\n| TATS | 1024 | 332 |\n| OmniTokenizer | 1280 | 191 |\n| CogVideo | 2065 | 626 |\n| Video-LaVIT | 1024 | 281 |\n| LARP | 1024 | 57 |\n| LARP | 512 | 86 |\n| VaporTok | 498 | 80 |\n\n[1] Visual lexicon: Rich image features in language space\n\n[2] Flextok: Resampling images into 1d token sequences of flexible length\n\n[3] One-d-piece: Image tokenizer meets quality-controllable compression\n\n[4] Elastictok: Adaptive tokenization for image and video\n\n[5] Cosmos-Tokenizer: A suite of image and video neural tokenizers\n\n[6] Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space\n\n[7] Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization\n\n---\n\n### **Q2-1: Analysis of the computational complexity of probabilistic taildrop during both training and inference.**\n\nDuring training, VaporTok learns a Taildrop Probability Query Module (22 M parameters) in addition to the original VAE (173 M parameters). This does introduce extra overhead, but only increases the total training parameters by 12.7%. The module is trained in both stage1 and stage2.\n\nDuring inference:\n\n- The Taildrop Probability Query Module runs in parallel with token discretization, adding computation but without noticeably increasing inference time.\n- In the decoder, because VaporTok decodes a shorter sequence than the full length, for single-sample reconstruction one can simply truncate (rather than mask) the input before feeding it to the decoder, achieving baseline-equivalent speed.\n\nWe evaluated inference time on the UCF101 validation set with a batch size of 1, averaging over five runs. The results are as follows:\n\n|  | VaporTok | VaporTok(directly truncate) | Baseline（LARP） |\n| :---: | :---: | :---: | :---: |\n| Encoder time(bs=1) | 6.7ms | 6.3ms | 6.3ms |\n| Decoder time(bs=1) | 7.3ms | 4.0ms | 3.8ms |\n\n---\n\n### **Q2-2: Will the introduction of adaptive encoding based on reinforcement learning make the model significantly more difficult to optimize and train to converge?**\n\nAdaptive encoding based on reinforcement learning does not make the model more difficult to optimize and train to converge. As shown in Appendix C.1, our results demonstrate that after roughly 8,000 steps, the reward values settle into a stable range, so convergence issues do not arise. We analyzed the cause and found that having a good initialization distribution before GRPO is critically important.\n\nIf one trains from scratch and simultaneously optimizes both the truncation distribution with GRPO and the latent‐space distribution, convergence can indeed be problematic. \n\nHowever, VaporTok initializes with a pre-trained weights, at which point the VAE has already learned a strong latent‐space representation. On this foundation, VaporTok learns the taildrop probability from the data prior while concurrently adjusting the latent‐space representation to follow a semantic-to-detail ordering. The data prior is more stable and provides an excellent reference for GRPO; moreover, the KL-divergence term in GRPO’s objective ensures that optimization remains centered around the taildrop probability of first stage, thereby enhancing training stability.\n\n---\n\n### **Q3: More training details should to be provided.**\n\nWe have already detailed the training hardware and durations in Appendix E; we now summarize them as follows:\n\n- **VaporTok-Stage1:** VaporTok initializes with pre-trained weights of LARP and traines for 30 epochs on the UCF101 and K600 datasets, which requires 90 hours on 8 A100 GPUs.\n- **VaporTok-Stage2:** The GRPO training process uses the UCF101 dataset for a single epoch, which takes 3 hours on a single A100 GPU.\n- **Generation:** The generation task is trained on the UCF101 dataset for 3000 epochs, which takes 40 hours on 8 A100 GPUs.\n\nThe baseline model(LARP) is trained for 150 epochs on the UCF101 and K600 datasets and his generation task is also trained on the UCF101 dataset for 3000 epochs, which is as same as ours. However, LARP does not specify the GPU hardware used or the precise duration of training, so we cannot compare training durations with it."}, {"review_id": "uG1NKNd3vB", "reviewer": "Reviewer_T8WN", "summary": "This paper proposes an adaptive video tokenizer that produces varying visual tokens according to input complexity. To achieve this adaptivity, two core designs are introduced: (1) a taildrop mechanismm that truncates video token according to its complexity, (2) a group relative policy optimization (GRPO) module that learns visual tokenization as a sequential decision process. The proposed tokenizer is evaluated in both video reconstruction and generation experiments, demonstrating its effectiveness.", "questions": "Please ref to the weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n* This paper studies an imortant problem of adaptive video tokenization. This problem is critical because of the high requirement of video tokenization despite varying complexity of videos.\n* Experimental results show the effectiveness of their proposed method.\n\nWeaknesses:\n\n* Comparison between reinforcement learning and supervised finetuning is insufficient. For example, reinforcement learning has this reward hacking issue and has limited generalizability. Meanwhile, supervised finetuning cannot resolve back-side sears. The authors could conduct additional ablation study to demonstrate the advantages of their method.\n* Hyperparameter sensitivity is not well discussed. While the paper introduces multiple weights within the reward model (in line 244 of page 7), it would complicate the framework and the same set of hyperparameters might not generalize to different scenarios .\n* Lack of human study. Despite that automatic evaluation metrics like PSNR shows some advantages of the proposed method, many video issues such as blurriness and artifacts cannot be fully reflected through automated metrics. Therefore, the authors could conduct additional user study to validate the effectiveness of their model.", "was_revised": true, "final_justification": "The authors have addressed my concerns. Therefore, I maintain my original rating of acceptance.", "rebuttal": "We appreciate the reviewer’s constructive feedback and have provided our responses below.\n\n---\n\n### **Q1: Comparison between reinforcement learning and supervised finetuning.**\n\nTo address the reviewers’ concerns, we constructed an SFT dataset using the truncation positions derived from first-stage video complexity prior and directly optimized VaporTok’s taildrop probability. In order to ensure a fair comparison, we adjusted the token count for VaporTok-SFT to be comparable to that of VaporTok-GRPO. The comparative results are as follows:\n\n|  | token num(Avg.) | rFVD | gFVD | entropy |\n| --- | :---: | --- | --- | --- |\n| VaporTok | 498 | 53.9 | 80 | 5.11308 |\n| VaporTok-SFT | 361 | 69.4 | 107 | 5.09822 |\n| VaporTok-GRPO | 361  | 66.6 | 98 | 4.11323 |\n\nIt can be seen that, under the same token budget, VaporTok-SFT underperforms VaporTok-GRPO by a considerable margin. The main reason is that during SFT, the gradients from reconstruction and generation cannot be back-propagated to the taildrop probability module, so the taildrop probability is never optimized for either task and just aligns with data prior of the first stage.\n\nBy contrast, VaporTok-GRPO explicitly trades off efficiency against reconstruction & generation, allowing further token-count compression at the cost of only a slight drop in reconstruction/generation performance compared to baseline. In summary, GRPO offers two main advantages over SFT:\n\n1. **Gradient propagation for task objectives.** GRPO circumvents the non-differentiability issue by incorporating reconstruction and generation signals into reward function of GRPO. SFT cannot do this, since one cannot label whether a sampled token sequence benefits generation, and the sampling operation itself blocks precise backpropagation of reconstruction gradients.\n2. **Parallel, stable training and rapid convergence.** GRPO trains on many video clips simultaneously, inherently improving stability and speed of convergence (see Appendix B.2, Figure 3: the reward stabilizes after ∼8 K steps).\n\nFinally, to address the reviewers’ concern about reward hacking observed in reinforcement learning (i.e., mode collapse, where different videos’ taildrop probabilities peak at the same index), we employ two countermeasures—stochastic parallel sampling and a diversity reward—and Table 3 in the main paper confirms their effectiveness via ablation.\n\n---\n\n### **Q2: The discussion about sensitivity and generalization about hyperparameters of the weight of different components in Vapor Reward**\n\nThe purpose of this hyperparameter is simply to rescale the four rewards into comparable ranges so that they can be effectively traded off during the GRPO training.\n\nFor example, when applying GRPO in the LLM or VLM domains, all rewards are typically constrained to [0, 1].  Likewise, we map our model-based rewards (reconstruction and generation) to [−1, 1] and our rule-based rewards (efficiency and diversity) to [0, 10].  We chose these ranges because, without adjusting the video latent-space distribution during GRPO, the model-based rewards fluctuate much more strongly than the rule-based ones—assigning them a smaller numerical span thus stabilizes training.\n\nAfter normalizing the scales, in Figure 4 of Section 4.3 and Figure 3 of Appendix B.2 we performed a simple sensitivity analysis on the two most volatile model-based rewards.  We varied their weights to study the impact on final performance, testing settings from 1:1:1:1 → 1:1:1:0 → 1:1:0:1 → 1:1:0:0, and observed that both rewards are indispensable.  Moreover, increasing their weights (from 1:1:1:1 → 1:1:2:2 → 1:1:3:3 → 1:1:4:4) consistently improves reconstruction and generation quality.  These analyses confirm not only the effectiveness of our reward design but also that, even when an order of magnitude smaller than the rule-based rewards, the model-based terms still exert strong control over GRPO training.\n\nAs for the generalizability of this hyperparameter: different videos do not produce order-of-magnitude differences in their computed rewards.  We sampled 10 % of the Kinetics-600 validation set and evaluated the rewards with our final VaporTok-GRPO; the mean values (below) show no such magnitude disparities. Thus, the same hyperparameters can be applied to different datasets without significant modification.\n\n| Data | UCF101 | K600 sub set  |\n| --- | :---: | :---: |\n| Efficiency reward | 10.89 | 10.20 |\n| Diversity reward | 8.48 | 8.56 |\n| Reconstruction reward | -0.162 | -0.042 |\n| Generation reward | 0.460 | 0.777 |\n\n---\n\n### **Q3: Human study to validate the effectiveness of VaporTok.**\n\nFor reconstruction, we used the same 20 videos and reconstructed them with LARP (512 tokens), VaporTok (avg. 534 tokens), ElasticTok (1024 tokens), and OmniTokenizer (1280 tokens), comparing each against the ground truth.  Participants saw four options and could choose two reconstrction results per video (the first choice as most preferred, the second as next preferred).  Eighteen people participated in the test.\n\nBelow are the voting results, which show that VaporTok, even with a token count comparable to or significantly lower than other methods, still outperforms them.\n\n| Method | First choice | First choice ratio | Second choice | Second choice ratio | Total choice | Overall ratio |\n| --- | --- | --- | --- | --- | --- | --- |\n| VaporTok(Avg. 534) | 125 | 34.7% | 114 | 31.6% | 239 | 33.19% |\n| LARP(512) | 113 | 31.3% | 105 | 29.1% | 218 | 30.27% |\n| OmniTokenizer(1280) | 115 | 31.9% | 130 | 36.1% | 245 | 34.02% |\n| ElasticTok(1024) | 7 | 1.9% | 11 | 3.0% | 18 | 2.5% |\n| Total | 360 | \\ | 360 | \\ | 720 | \\ |\n\nFor generation evaluation, we randomly selected 10 classes from UCF101. For each class, we generated one video with OmniTokenizer, LARP, and VaporTok. Participants then chose which model produced the best result. Eighteen people took part in the test. It can be seen that, with a comparable token count, our method achieves superior user preference.\n\n| Method | Choice | Choice ratio |\n| --- | --- | :---: |\n| OmniTokenizer(1280) | 5 | 2.7% |\n| LARP(512) | 69 | 38.3% |\n| VaporTok(Avg.525) | 106 | 58.8% |\n| Total | 180 | \\ |"}]

Author rebuttal:

Meta-review: The paper proposes VaporTok, an adaptive video tokenizer combining probabilistic taildrop with GRPO to allocate tokens by content complexity and downstream objectives. It received 1 A and 3 BAs. Reviewers largely agree on novelty and practical impact, noting substantial token reductions at comparable reconstruction quality and improved generation metrics at similar average token counts; added analyses (SFT vs GRPO, sensitivity, overhead), a human study, and broader comparisons address earlier concerns about baselines and complexity. One reviewer still points out that the method does not strictly dominate fixed-rate baselines across the full rate–distortion range and that the semantic-to-detail claim is not fully established at very low token counts; however, these are outweighed by the method’s soundness, clarity, code availability, and demonstrated efficiency gains on UCF101/K600. As a result, the AC recommands accept.
Decision: Accept (poster)