Title: VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumption
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Tokenizer

Diffusion Model

Video Generation
Abstract: Modern video generation frameworks based on Latent Diffusion Models suffer from inefficiencies in tokenization due to the Frame-Proportional Information Assumption.
Existing tokenizers provide fixed temporal compression rates, causing the computational cost of the diffusion model to scale linearly with the frame rate.
The paper proposes the Duration-Proportional Information Assumption: the upper bound on the information capacity of a video is proportional to the duration rather than the number of frames.
Based on this insight, the paper introduces VFRTok, a Transformer-based video tokenizer, that enables variable frame rate encoding and decoding through asymmetric frame rate training between the encoder and decoder.
Furthermore, the paper proposes Partial Rotary Position Embeddings (RoPE) to decouple position and content modeling, which groups correlated patches into unified tokens.
The Partial RoPE effectively improves content-awareness, enhancing the video generation capability.
Benefiting from the compact and continuous spatio-temporal representation, VFRTok achieves competitive reconstruction quality and state-of-the-art generation fidelity while using only $1/8$ tokens compared to existing tokenizers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VFRTok, a video tokenizer based on the **duration-proportional information assumption**, aiming to address the inefficiency in current video generation models caused by variations in frame rate. Existing video tokenizers typically use a fixed compression rate, resulting in the computational cost of diffusion models scaling linearly with frame rate. VFRTok introduces a variable frame rate encoding/decoding mechanism and incorporates an improved RoPE (Partial RoPE) to achieve more compact and continuous spatiotemporal representations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The introduction of Partial RoPE cleverly addresses the issue of content modeling interference by positional modeling in Transformers.\n2. The method supports video interpolation at arbitrary frame rates (from 12 FPS to 120 FPS), demonstrating strong generalization ability.\n3. The paper is logically well-structured and contains rich visualizations and figures.\n\n**Weaknesses:**\n\n1. The paper does not discuss in depth how the Duration-Proportional Information Assumption theoretically guarantees the effectiveness of this compression method (e.g., information fidelity, reconstruction error).\n2. Some experimental details (such as optimizer settings, learning rate schedules, regularization methods) are not fully described in the main text, affecting the reproducibility and independent evaluation of the work.\n\nQUESTIONS:\n1. To my knowledge, models such as WFVAE, Hunyuan VAE, and Wan VAE have been trained extensively across different frame rates and can also reconstruct and generate videos at arbitrary frame rates. What advantages does your approach offer compared to these methods?\n2. Since there are multiple choices for frame rate, the number of frames corresponding to the same duration will vary. If the number of learnable tokens is fixed, it may be difficult to adaptively align the output frame count. How is training conducted in this case? Is the number of tokens constant across different frame rates? Does information loss occur when the frame rate is too high?\n3. Is the Duration-Proportional Information Assumption applicable to all types of video?\n4. What are the results of symmetric training across frame rates? This is not analyzed in the paper.\n5. Partial RoPE applies RoPE only to some attention heads. While this improves content modeling, could it potentially weaken the model's ability to accurately model spatiotemporal positions, thereby affecting generation consistency?\n\nLIMITATIONS:\nsee Questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VFRTok, a query-based 1D transformer tokenizer that encodes videos of variable frame rate and same length into the same number of latent tokens. This is achieved by converting the frame index to the timestamp when computing the temporal rotation angles of a 3D RoPE. They further propose Partial RoPE, where they only apply RoPE to a subset of the attention heads. The authors conduct experiments on video reconstruction and generation based on VFRTok.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- VFRTok achieves competitive reconstruction results and sota generation results while converge much faster.\n- Both variable frame rate design and Partial RoPE are novel contributions to video tokenization and easy to implement.\n- Partial RoPE is motivated by empirical observation where the attention is dominated by the grid position prior in Fig. 6.\n- Partial RoPE is empirically validated in ablation study in Tab. 4.\n- VFRTok supports interpolating videos from 12 FPS to 120 FPS.\n\n## Weaknesses\n- Some figures (e.g. Fig. 1-6), while informative, are a bit confusing and needs additional captions or paragraphs in the main paper. \n- More details for the training (expanding L138-L145) and inference are needed. For example, how VFRTok adjust encoding/decoding frame rates during inference? Do VFRTok learn shared or separate sets of learnable tokens for each frame rate?\n- No quantitative results for video interploation in Sec. 4.5.\n\nQUESTIONS:\n- Sec. 4.5 shows that VFRTok can perform frame interpolation. Given a variable frame rate video LDM, can VFRTok be used to speed up inference by letting the LDM to generate videos latents with lower FPS and then use VFRTok to interpolate to higher FPS during or after decoding? If this is possible, this will be a significant contribution to the video generation field.\n- Code is provided in supplementary material. Will the code and model weights be released to the public?\n- From Fig. 3, it seems that only the patch tokens have RoPE while the latent tokens don't. Why such design and has this been ablated?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new video tokenizer for video generation. The method is based on the assumption that a video's information is upper bounded by its duration not the number of frames. The tokenizer is based on recent 1D tokenizers using Transformers that directly patchify pixel cubes into tokens and produce latent tokens with learnable queries. On top of the 1D tokenizer design, the authors introduced 1) the split-head 3D RoPE which was previously used in Flux and QWen models; 2) the position heads and content heads, where the content heads are similar to NoPE. The tokenizer is trained with random pairs of high-to-low or low-to-high FPS rates in the inputs and outputs, effectively learning frame interpolation or sumsampling. \n\nThe proposed tokenizer is tested in video reconstruction and video generation benchmarks, showing better compression ratio and generation quality than grid based video tokenizers, while using few number of latent tokens.\n\nSTRENGTHS AND WEAKNESSES:\n+ Making video tokenizer less dependent on the frame number, which is an artifact of the video production process, seems natural and beneficial for video tokenization. The choice of Transformer-based 1D tokenizer makes this viable. \n+ The design choices 1) and 2) seem to be sound and reasonable.\n+ The training strategy that uses the frame interpolation/subsampling to make model adative to frame rates is very interesting and could generate good impact in the community. \n+ Strong experimental results. \n\nWeaknesses\n\n- The 3D rope design and NoPE usage both lack references to the existing literature. I would recommend the authors to carefully revise the related work section. \n- Video data has three dimensions in general, height, width, and time. In this work with a fixed latent token amount, the time dimension is flexible. But the choice of using 1D tokenizer with fixed latent token count makes it non-trivial to deal with the varying height and width of the video frames.\n\nQUESTIONS:\nMy major question relates to weakness #2: how to deal with varying spatial resolution with the current tokenizer design?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce VFRTok, a transformer-based video tokenizer that trained through asymmetric frame rate training between the encoder and decoder. The length of the token encoded is propotional to the video duration instead of number of frames. In addition, a Partial RoPE is proposed to allow the model's content-aware learning. Experiments demonstrate the high compression rate and good reconstruction/generation quality. VFRTok also shows the potential in video frame interpolation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed VFRTok is a potential solution to the inconsistency of frame rate in video generation data in real world practice, especially In scenarios like game video generation, where the frames in the training data are rendered by computers and it is difficult to keep the frame rate stable during rendering.\n2. The experiment result seems promising. The high compression rate is a critical advantage for downstream applications, such as its use as a tokenizer in video generation models, where efficiency and compactness are essential.\n3. The design of Partial RoPE is reasonable and effective. \n4. When used as a tokenizer for video generation models, VFRTok 's flexibility in handling varying frame rates has the potential to alleviate the burden on the generation model by sharing the responsibility of frame rate control. It may also help mitigate the negative impact of inconsistent frame rates in training data, thereby improving the learning process of the video generation model.\n\nWeakness:\nI suspect that the underlying assumption of duration-proportional information has inherent limitations. This assumption appears valid only when the content within the frames is not moving rapidly. However, in cases where the content is moving quickly, videos with higher frame rates inherently capture significantly more information than those with lower frame rates over the same duration. From this perspective, I am concerned that VFRTok may inadvertently constrain the representation of fast-moving content when applied to video generation tasks. This limitation could restrict the model's ability to accurately capture dynamic scenes, potentially impacting its generation quality in scenarios involving rapid motion.\n\nQUESTIONS:\n1. Is it possible to show evidence that when used as the tokenizer of a video generation model, VFRTok will not limit the contents' movement in video generation compared with other video tokenizers? \n2. Since the Partial RoPE can improve the model's content-awareness, will it facilitate the video generation model to learn the knowledge of movement? For example, does it enable the video generation model to generate videos that better align with physical rules or exhibit more realistic motion patterns?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}