VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumption
Authors: Tianxiong Zhong
Xingye Tian
Boyuan Jiang
Xuebo Wang
Xin Tao
Pengfei Wan
Zhiwei Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Video Tokenizer
Diffusion Model
Video Generation
Primary area: deep_learning

Abstract: Modern video generation frameworks based on Latent Diffusion Models suffer from inefficiencies in tokenization due to the Frame-Proportional Information Assumption.
Existing tokenizers provide fixed temporal compression rates, causing the computational cost of the diffusion model to scale linearly with the frame rate.
The paper proposes the Duration-Proportional Information Assumption: the upper bound on the information capacity of a video is proportional to the duration rather than the number of frames.
Based on this insight, the paper introduces VFRTok, a Transformer-based video tokenizer, that enables variable frame rate encoding and decoding through asymmetric frame rate training between the encoder and decoder.
Furthermore, the paper proposes Partial Rotary Position Embeddings (RoPE) to decouple position and content modeling, which groups correlated patches into unified tokens.
The Partial RoPE effectively improves content-awareness, enhancing the video generation capability.
Benefiting from the compact and continuous spatio-temporal representation, VFRTok achieves competitive reconstruction quality and state-of-the-art generation fidelity while using only $1/8$ tokens compared to existing tokenizers.

VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportinnal Information Assumption
Tianxiong Zhong':
Xingye Tian', Boyuan Jiang", Xuebo Wang"t, Xin Tao?, Pengfei Wan?, Zhiwei Zhang
Beijing Institute of Technology, "Kling Team, Kuaishou Technology inkosizhong@gmail com, (tianxingye, r jjangbbyuan, wangxuebo)/kkuiiieeeee com, {taoxin, wargengffnnnnneeeee (attnngffi@kkuaa Akuaii shou. com, zwzhang@bit. edu. en,
Abstract
Modern video generation frameworks based on Latent Diffusion Models suffer from inefficiencies in tokenization due the Frame-Pree ProporPropllli Information As- sumption. Existing tokenizers provide fixed temporal compression rates, causing the computational cost of the diffusion model to scale linearly with the frame rate. The paper proposes the Awration-Proportionn Information Assumption: the upper bound on the information capacity of a video proportional the duration rather than the number of frames. Based on this insight, the paper intro- duces VFRTok, Transformer-baser video tokenizer, that enables variable frame rate encoding and decoding through asymmetric frame rate training between the encoder and decoder. Furthermore, the paper proposes Partial Rotary Position Embeddings (RoPE) decouple position and content modeling, which groups correlated patches into unified tokens. The Partial RoPE effectively improves enhancing the video generation capability. Benefiting from the compact and continuous spatio-temporal representation, VFRTook achieves competitive reconstruction quality and state-of-the-arr generation fidelity while using only 1/8 tokens compared to existing tokenizers. The code and weights are released at: https: : //g /ttgtgtttuu....////////gg com
Real-world
Other
Other
High fps Video
D
Cosmos
Omni
| Sample
Ours
Ours
LTX
High fps Video
gFVD
VFRTok
200
Sample
Other
Other
Low fps Video
Low fps Video
Ours
Ours
100
11.9 lower
Frames
Enc.
Latents
Dec.
Frames
TFLOPs
Figure I: VFRTok is based on the Duration- Proprooooaooo Information Assumption. The number of tokens for other tokenizers grows with frame rate. VFRTok maintains a fixed length latents tied to video duration and supports asymmetric frame-rate encoding and decoding.
Figure 2: Efficiency-quality trade-off, where lower-left indicates better perfor- mance. VFRTok provides a more effi- cient latent representation.
'This work was conducted during the author's internship at Kling Team. Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
and video generation I8 22 35 II, comprising two main components: a tokenizer and a diffusion model. The tokenizer compresses data from the original high-dimensionas pixel space to low-dimensional latent space, which reduces the training and inference overhead of the Diffusion Transformers (DiT) by quadratic factor. The video tokenizers [ 33 35 41 eliminate intra- and inter-frame redundancy in the video by simultancously compressing both temporal and spatial dimensions.
Recently, Latent Diffusion Model (LDM) is widely used in image 22 24 25 36 38 39
Existing video tokenizers Assumption, which assumes a fixed compression rate for given number of video frames (Figure [ These tokenizers are trained on and designed to generate videos with a fixed frame rate. High frame rate videos require a larger number of tokens for representation, resulting in the number of tokens increase linearly with the frame rate, which significantly increases the computational overhead.
18
33
are built upon the Frame- Prope-Propotiional Information
Video is the result of continuous space-time being sampled uniformly. The amount of observable information in continuous space-time serves as the natural upper bound on the information contained in the video. Intuitively, when the video frame rate increases from 12 frame per second (FPS) to 24 FPS, the change can be clearly observed, whereas the difference between 60 FPS to 120 FPS yields more subtle changes. When a camera samples a motion trajectory x(t) at a sampling frequency f.- the resulting discrete samples can be used to estimate the continuous trajectory. An interpolation algorithm is typically employed to reconstruct the continuous motion trajectory ((1) from these discrete observations. According to interpolation error estimation theory in numerical analysis, the upper bound of the estimation error Emon related to fs as follows:
sup Emaz = sup (x(() -010))--(10 =(t) < C- sup, (e)())) (t)
where k represents the order of accuracy of the interpolation algorithm, sup, (t)) is upper bound on the 6thh derivative of the true trajectory z(t), and C' is a positive constant. Equation (I implies that the information gain diminishes as the frame rate increases.
Motivated by this insight, we propose the Duration- Proportional Information Assumption, which guides the design of compression rates that scale with video duration (Figure DD. Specifically, we introduce Variable Frame Rates video Tokenizer, VFRTok, which is a query- based Transformer tokenizer and enables the encoder and decoder to process different frame rates. VFRTok uses asymmetric frame rate training between the encoder and decoder to learn continuous spatio-temporal representations, that enables the generation of videos at arbitrary frame rates.
Furthermore, we observed that both existing tokenizers I 33 and VFRTok exhibit a grid-based pattern, where each latent token tends attend to fixed spatial location within the temporal sequence. To achieve a more compact representation, we aim to strengthen VFRTok's 's content-awareness. Our analysis illustrates that the latent representations in VFRTok are strongly influenced by the positional prior introduced by Rotary Position Embeddings (RoPE), which hinders effective content modeling. Therefore, we propose Partial RoPE, which applies RoPE only to a subset of the attention heads, encouraging a functional separation between positional and content encoding. Experimental results show that the Partial RoPE effectively enhances the model's capacity for content modeling.
Benefiting from these designs, VFRTok efficiently reconstructs and generates videos. For example, as shown in Figure 2 VFRTok achieves better generation results than existing tokenizers M 33) while requiring 11.9x less computation overhead. Meanwhile, VFRTok natively supports video frame interpolation, enabling frame rates be increased from 12 FPS to 120 FPS. In summary, we highlight the main contributions:
1. We propose the Duration- compression video tokenizer with variable frame rate.
Duration-Proportionn Proportional Information Assumption and the first high-
2. We introduce Partial RoPE mitigate the influence of video patch position priors latent tokens and enhance content-awareness.
3. Experiments show that we can achieve comparable reconstruction and ofate-of-the-art generation while using only 1/8 tokens compared to existing tokenizers.

---

2 Related Work
2.1 Video Tokenizer
Video data consists of series of continuously changing frames and video tokenizers compress the video in both temporal and spatial dimensions. Therefore, video generation task must consider inter- frame consistency to avoid problems such as flickering and jittering. Early LDM [22] directly uses image tokenizers
frame-by-frand compression pattern. In contrast, modern video tokenizers [I] generally use 3D computation modules that include the temporal dimension.
16 [8] 31
33
35 41
Mainstream video tokenizers generally provide compression rate of 4 8 x 8 [16 18 33 35 41)
which meaning a 16-frame video with 256 x 256 resolution can be compressed into 4 x 32 x 32
4096 tokens. A few works propose tokenizers with higher compression rates [ |1 20 B1] For instance, Cosmos Tokenizer [ leverages wavelet transform and provides a series of tokenizers with compression rates from 4 x 8 to 8 x 16 x 16. LTX- VAE [1]]
cascades additional downsample
layers and achieves a compression rate of up to 8 x 32 x 32 through a spatial-to-depth approach.
rely on the Trame-Proportie Information Assumption and process sequences with fixed frame rates. Different from these tok- enizers, which model discrete video frame pixel space, VFRTok models continuous ppatio-tempora information using asymmetric frame rate training strategy between the encoder and decoder.
However, existing video tokenizers 1 16 18
31
33 35
41
2.2 Query-based Visual Tokenizer
Unlike traditional grid-based tokenizers 24 25 query- query-based image tokenizers 5 6 37 encode
images into compact 1D latent representations. These tokenizers employ a Transformer based framework and use learnable latent tokens to query information from the image patches. For instance, TiTok (37 proposes a VQ-based tokenizer and represent an image with 32 tokens for reconstruction and MaskGiT 4 -style generation. SoftVQ-VAA VAE (5) leverages soft categorical posteriors to increase the representation capacity of the latent space, which can be applied to both autoregressive- and diffusion-basee image generation. MAETok (6) further improves the diffusibility 66 36) of vanilla AutoEncoder (AE) by regularizing it with mask modeling and auxiliary decoders. LARP (32) extends the query-based structure to video tokenizer, adapt autoregressive video generative models.
The query-basee image tokenizers 6 37 achieve comparable or better performance than the grid- based image tokenizers 2 [4 25), using significantly fewer tokens. This shows that the query-based structure has the potential to better eliminate redundant information. More importantly, provides the possibility encode variable-length data into fixed-length latent representations.
3 Method
3.1 Architecture
Given a video X E XRFXHXXX3, we want to obtain its compressed representation Z = EXX) e RNxd where N represents the number of tokens, and the reconstructed video X = D(Z) FFx xx3 iillustates the architecture of VFRTok. We adopt the AE architecture and use ViT 9 as the backbone for both the encoder and decoder. On the encoder side, the input video X is
# W patchified into a serials of spatial-temporal patch tokens x E RR "P "ww where pr X PH X Pw the patch size and h is the hidden dimension. Similarly, the decoder recovers the reconstructed F. H patch tokens & E E PP PM Pw xh back to the pixel-space X.
VFRTok is based on the Duration-Propprtiona Information Assumption, which requires a continuous spatio-temporal representation. We adopt a query-based approach 6 37 that encodes video by querying grid-based patch tokens with fixed-length latent tokens, and decodes by reversing this process, where patch tokens query latent representations. To align equal-duration videos of different frame rates, we modify the temporal modeling 3D RoPE [19 28 from frame-index-bate position encoding
33 to a timestamp-based approach, detailed in Section 3.2 Furthermore, as
described Section 3.3 we propose an improved RoPE implementation, Partial RoPF, which applies RoPE to subset of attention heads to decouple positional information from content modeling.
3

---

Patch Tokens
Latent Tokens
Learnable Tokens
Content Heads
Position Heads
3D RoPE
fi
Partial RoPE MHA
Partial RoPE MHA
Approx. Continuous Latents Z
ViT Encoder E
VIT Decoder
Figure 3: VFRTok adopts a query-based VTT architecture. VFRTok models variable-length patch tokens with fixed-length latent tokens, support the encoding and decoding of variable frame rate videos. VFRTok further introduces Partial RoPE, which applies RoPE to a subset of attention heads decouple positional information from content modeling.
Random Select
Random Swap
AutoEncoder
Videos f
Random
Frame Sampling
fz
Training Loss
Videos @ f
1. Fetch source data
2. Create paired data
3. Asym. reconstruction
Figure 4: Asymmetric frame rate training strategy between encoder and decoder. We construct paired high and low frame rate videos to learn continuous spatio-temporal representations.
Duration-Proportional Compression
VFRTok accepts data of any frame rate but with equal duration, learns the continuous spatio-temporal information it represents, and encodes this information into fixed-length latent representations. Specifically, it concatenates learnable latent tokens E RR with the input æ before ViT encoding. The patch tokens are dropped at the bottleneck of the encoder, and only the latent tokens are retained and mapped to the low-dimension compressed representation Z. On the decoder side, Z is first remapped to the hidden feature 2, and concatenated with the learnable patch tokens á. As shown in Figure 3, based on the flexible architecture, VFRTok allows the encoder and decoder to apply different frame rates. Since they represent data of the same duration, videos with higher FPS contain more frames, which in turn translates into more video patches.
To achieve Duration- Prattion-rroottionall compression, we use 3D RoPE 28 to model the positional dependencies among patch tokens and replace the frame-driven rotation angle to timestamp-driven formulation. Specifically, we first generalize RoPE (28 to video data by splitting the channels in each attention head into three parts, which are used to encode the positional information along the temporal, vertical, and horizontal spatial axis. Given patch token at temporal-spatial position (t, j, k) in F H W patch token space, the rotation matrix Re,j guxn can be represented as:
R2
0
0
R
coos 0i
sin 0
Re,j. =
0
RH
0
R =
R
(2)
ssn f
cos 07
0
0
RW
W
R

---

(a) OmniTokenizer 4x32x32 tokens
(b) Cosmos-S 2x16x16 tokens
(c) Ours w/o Partial RoPE 512 tokens
(d) Ours 512 tokens
Figure 5: Visualization of the region affected by a single token. The heat map is an overlay of 100 samples, showing the 1st, 5th, 9th, and 13th frames for each method. The reference lines drawn in the grid-based approaches to indicate the spatial position of the token within the grid.
Samples
Layer-2
Layer-7
Layer-12
Video Patch Tokens
Figure 6: The latent-to-pane attention map from different Transformer layers reveals the interference of position prior on content modeling. Each row represents the intensity of information flow across patch tokens, and rows correspond to different samples.
where represents the number of channels of attention head, 0,.c is the rotation angle of the e-th channel of the i-th token. Specifically, as shown Equation (3), VFRTok directly uses the spatial
index (j. k) } of the patch calculate the rotation angle (A4 k,e }. and converts the temporal index t to timestamp I/So a to compute the corresponding rotation angles Of.--
of.- = C
x 10000 4, 0 j.c = x 10000 Es AW = k x 10000-
(3)
where fs is the frame rate and an optional normalization coefficient. This makes videos of the equal duration share the same maximum rotation angle. Meanwhile, as the frame rate increases, the angular difference between adjacent frames decreases.
As shown in Figure we employ the encoder-decoder asymmetric frame rate training strategy to guide the latents encoding continuous spatio-temporal information. Firstly, we divide the dataset into multiple buckets according to their original frame rate. For each batch, we randomly fetch data with frame rate f. from bucket. Secondly, we randomly select downsampling factor predefined range and extract frames with f. = fa J from the video. The two video sequences are randomly swapped to determine the frame rate VRFTok's s encoder fe and decoder f!P Finally, encodes the video sequence with f and learns to reconstruct the video with f.. The configurations vary across GPUs, enabling VFRTok to learn more continuous spatio-temporal representations.
from
3.3 Partial RoPE
To analyze what information in the video is compressed by each token, we use PCA 14 to ablate latent tokens individually, retain the remaining tokens for reconstruction, and observe the degradation position of the reconstructed video.
AX D(()) D(Z,),, Zi (Zo, ZC,, Z,11, ZN),
(4)
5

---

where (-) is the inverse PCA based on the coefficient corresponding to the i-th token Z, across on batch of samples, and is an element- absolute. As shown in Figure b,C, the latents of existing methods [
33 and the vanilla query-based approach exhibits regular grid property. The token corresponds to pixels at a relatively fixed position across different frames. Existing methods [ are based on grid designs, and their tokens strictly follow the guidance of the grid. In contrast, our query-based approach is expected to provide more content-aware latent expressions. Therefore, we investigate the underlying cause.
We visualize the tatent-to-pacch attention map of the decoder across all layers and heads. Formally, for given attention map A E (MM+))) where M
x
H
x W. is the number of patch
PF
PH
PW
tokens, can be considered as a block matrix:
Apatch E R MMM
Alateut-patee E R RMM
A Apatch--latent E NRM
(5)
Auttnt E NNN
For given sample, we extract each column from Alatent--patch which represents the intensity of information flow across patch tokens. We concatenate together the arrays of multiple samples to form Figure 6 where the rows correspond to samples and the columns correspond to video patch tokens. Figure ( reveals that the attention is dominated by positional prior. Specifically, the attention distribution shows a long-short pattern across different samples, where the long periodicity arises from temporal priors, while short periodicity originates from spatial priors.
To alleviate this problem, we propose Partial RoPE, which divides the attention heads into position heads and content heads. For the position heads, we adopt the original 3D RoPE, while for the content heads, we simply remove the RoPE to adequately learn content information. We use TROPE to control the proportion of position heads. The experimental results demonstrate that Partial RoPE effectively enhances the content-awareness and the generation quality of VFRTok.
4 Experiments
4.1
Setup
Training details. We train VFRTok & using the standard reconstruction objective.
V (Crecon > Epercape)
L= Lreron + d1 pprreept + X2 Ladva Aa
(6)
V Ladv
where Lreenn, Epereept. and Ladv are the LI reconstruction loss, perceptual loss 13 and adversarial loss (1O), respectively, and \ represents adaptive weight. The hyperparameters are set to X = 1 and A2 0.2. The patch size of VFRTok is set to pr X PH X pw 4 x 8 x 8. The Partial RoPE ratio is set TROPE- = 0.5, indicating that 6 of the 12 attention heads employ RoPE. An implicit advantage of VFRTok is that the number of tokens N and channels d can be easily adjusted. To balance generation quality and training efficiency, we provide VFRTok-L and VFRTok-S with the same latent capacity but differ in token count, zL E 52222x33 and 2 E IR 1g12xx125 To evaluate the tokenizers, we modified LightningDiT--X models process video with 2/3s duration, which is 16 frames at f. 24. For fair comparison, the first frame of the image and video joint tokenizers | 33 decoded using the image token, is not counted in the metrics, nor counted in the model calculation costs.
(36 to support video generation. If not specified, the
VFRTok is trained in three-stage manner. First, it is initialized the ImageNet-1 KK { for 30,000 steps with a batch size of 512. Then, it is pre-trained on the K600 dataset (3 for 200,000 steps with batch size of 64, employing asymmetric FPS training f = f < 12, 18, 24, 30 J. Finally, 22 sequences of 120 FPS data from the BVI-HFR dataset (7] are added for 100,000 steps with a batch size of 16, using FPS settings f.. E { 12 6k k 0, 1.... 18] J. Reconstruction evaluation is performed on the K600 B validation set and the UCF101 [26] dataset. VFRTok-L and VFRTok-S are trained on a single node equipped with 8 H800 GPUs, requiring days, respectively. The DìT models (36] are trained and evaluated on the K600 3 and UCF101 226 datasets, respectively, using label-based Classifier-Free Guidance 12 (CFG). To demonstrate the advantages of VFRTok in high
Datasets.
frame rate video generation, the DiT model was also trained on 60 FPS data from the LAVIB [27]
dataset. DiTs (36] are trained for 100,000 steps with a batch size of 128 on UCF101 (26 and
K600 3 datasets, and 50,000 steps with batch size of 48 on LAVID 27 dataset, respectively. The

---

Table Comparison reconstruction performance across multiple datasets for different tokenizers. Gray highlights indicate cases where VFRTok is superior or comparable.
K600
UCFI01
Method
#Tokens
#Dim.
PSNRT
SSIMT
LPIPSI
rFVD
PSNR
SSIMT
LPIPS
rFVD
Omni
4096
29.35
0.9143
0.0573
5.14
28.95
0.9239
0.0505
10.21
Cosmos-I.
4096
16
33.34
0.9284
0.0546
33.42
0,9372
0.0439
5.55
Cosmos-M M 1
2048
16
31.66
0.9068
0.0710
6.77
31.70
0.9177
0.0575
13.67
S 1
512
16
28.46
0,8445
0.1209
70.26
28.26
0.8577
0.1046
104.51
LTX [
128
128
32.04
0.9100
0.0582
22.11
32.02
0.9202
0.0508
35.32
VFRTok-I
512
32
31.63
0.9104
0.0394
4.64
31.54
0.9193
0.0391
13.79
VFRTok-S
128
128
31.55
0.9089
0.0401
6.02
31.50
0.9178
0.0401
15.55
Table 2:
Comparison of unconditional and CFG video gen- eration in terms of gFVDJ and TFLOPs]- Best results are bolded; gray highlights indicate VFRTok-S superiority.
1000
Cosmos-L
Omni
Cosmas-M
VFRTok-L VERTok-S
Cosmos-S
LTX
BFFDD
300
K600
UCF101
Method
TFLOPs
w/o CFG
w/ CFG
w/o CFG
w/ CFG
Omni 33
5.82
521.89
242.54
480.60
88.89
200
Cosmos-L. 1
5.82
620.07
302.58
476.08
75.11
21.6x faster
Cosmos-M 1
2.37
554.95
125.02
497.01
85.22
0.0
0.5
1.0
1.5
2.0
Cosmos-S
1
0.49
569.58
210.21
678.37
191.49
Training TFLOPs
1e8
LTX [1]
0.12
615.28
358.48
735.38
345,82
VFRTok-I.
0.49
323,37
124.78
377.50
71.34
Figure 7: Convergence speed differ- ent tokenizers on UCFI01.
VFRTok-S
0.12
412.97
131.34
443.41
129.55
training cost correlates with the number of latent tokens, ranging from 5 hours using 8 H800 GPUs (VFRTok-S) to 3 days using 16 H800 GPUs (Cosmos-L (I).
Metrics. For the reconstruction task, we use PSNR, SSIM, and LPIPS evaluation. Mcanwhile, we use reconstruction FVD [30] (rFVD) as
perform frame- wise metric. For the generation task, we use generation FVD 30 (gFVD) evaluate frame quality with and without
CFG [12].
We use floating- ppint operations (TFL.Ps) to evaluate the calculation costs of all DiTs.
4.2 Video Reconstruction
Table [ Ireports reconstruction metrics various methods [|]]]] across different datasets 3 26 It shows that all models in the VFRTok family share similar reconstruction quality, attributed to
their shared latent token capacity. VFRTok achieves comparable quality with Cosmos-M [ and
OmniTokenizer [33]. using only 1/4 latent capacity. In comparisone, LTX- VAE (11] performs better than VFRTok on PSNR, but VFRTok achieves significantly better LPIPS 40) and rFVD [30]. J. is
worth noting that Cosmos [1] and LTX-VVE [1I] use larger and licensed training dataset, VFRTok still has the potential to achieve better results.
4.3 Video Generation
As shown in Table 2 VFRTok-L achieves state-of-the-art (SOTA) performance in the unconditional and CFG [12] generation on UCF101 (26] and K600 3 dataset. The overhead of VFRTok-L is only 8.4% of Oonenzkee and Cosmos-L, and 20.6% of Cosmos-M. Furthermore, we use gray highlighting to indicate cases where VFRTok-S is superior. For example, VFRTok-S consistently outperforms VAX [I]] and Cosmos-S [ in all tasks. We show the convergence speed of different methods on the UCF101 26 dataset in Figure 7 where the error bars reflect different CFG scales { 2,3,4 4 Due to the high computational cost DiT (36) when using existing tokenizers [[] 33 we randomly conduct a fair comparison 1,000 samples. The experimental result shows that

---

Figure 8: Qualitative results for video generation with CFG on the UCF101 dataset.
Table 3: Comparison of unconditional video generation on LAVID at 60 FPS.
Table 4: Effectiveness of Partial RoPE.
Reconstruction
Generation
Method
#Tokens
TFLOPS
gFVDJ
TRoPR
PSNR
SSIM1
LPIPS.
rFVD]
gFVD
Omni
3
10240
22.67
375.47
0
12.17
0.2204
0.7897
4652.25
Cosmos-L 11
10240
22.67
1552.04
0.25
29.24
0.8886
0.0545
18.38
199.86
Cosmos-Ne M 1
5120
7.95
1176.80
0.5
30.87
0.9106
0.0451
16.33
147.41
Cosmos-S S M
1280
1.35
863.36
0.75
30.91
0.9094
0.0437
16.17
174.44
VERTok-L
512
0.49
148.68
1
30.85
0.9083
0.0441
16.84
208.42
VFRTok converges significantly faster; for example, VFRTok-L achieves maximum convergence speed of 21.6x that of OmniTokenizer 333. Qualitative results of generation are shown in Figure
We also provide the generation results on the LAVIB 27 60 FPS dataset in Table 3 As shown, all baseline methods [ 333 require 2.5x the tokens represent videos with higher frame rates, resulting in near-quadratic increase in computation cost. For example, for videos of the same duration, the computational cost of DiT based on Omn''ookenizer (33 at 24 FPS 5.82, where as at 60 FPS it increases to 22.67. In contrast, VFRTok maintains a constant token count regardless of frame rate, which not only improves efficiency but also enables significantly faster DiT convergence. As result, VFRTok-L achieves the optimal gFVD 30 score among all compared methods.
4.4 Ablation Study
Partial RoPE. We perform an ablation study on Partial RoPE by training DiT [36 with varying Partial RoPE factors TROPE {0, 0.25, 0.5, 0.75, }.. As shown in Table 4 smaller TROPE fails to provide adequate positional guidance, resulting in poorer reconstruction and generation quality, while higher TRoPE results in excessive positional bias (Figure 66), leading to suboptimal performance. Specifically, we find that when TROPE 0, ViT [2] entirely loses its capacity for positional encoding, rendering incapable of reconstructing or generating videos. We adopt TROP = 0.5 as our default setting, as it achieves optimal balances position priors and content modeling. The visualization of latent-to-patch attention map belonging to the position heads and content heads is illustrated in Figure 9 VFRTok effectively decouples the two patterns, and different samples in the content

---

Position Heads Position Heads
Content Heads
Content Heads
Figure 9: VFRTok decouples positional encoding and content modeling through Partial RoPE. The Position heads share similar patterns across samples (rows) indicating reduced sensitivity to content. The content heads attend distinct patches in different samples (zoom in for details).
Table 5: Symmetric and asymmetric reconstruction performance Adobe240fps dataset. Symm represents a variation of VFRTok which is trained on the symmetric reconstruction task.
Symmetric Reconstruction (PSNR)
Asymmetric Reconstruction (PSNR)
Methods
30fps
60fps
120fps
30 + 60fps
30 - 120fps 60 + 120fps
Symm
25.92
24.71
23.89
22.36
21.60
23.33
Ours 25.87 10.05
24.53 J0.18
23.95
10.03
24.05 11.6 23.56 11.9
23.90 10.66
head exhibit greater diversity in attention patterns. The PCA analysis in Figure () also shows that adopting partial RoPE can increase the variability of video regions influenced by individual tokens.
Asymmetric Symmetric Training. We train a VFRTok variant under symmetric encoding strategy. Specifically, we initialized from the K600 3 pre-trained model in stage for quick verification. In stage 2, which uses both K600 3 and BVI-HFR 7 datasets, disabled asymmetric encoding, forcing both encoder and decoder to operate at the same frame rate. For fair comparison, we adopt Adobe240fps (29) for evaluation. As shown in Table 5 this symmetric training yields a marginally better reconstruction in the symmetric setting but suffers a pronounced degradation under asymmetric encoding. Note that our stage initialization itself used asymmetric frame rates, which partially narrows the gap in asymmetric reconstruction. Overall, these results confirm that asymmetric training is crucial for enabling VFRTok to generate videos arbitrary frame rates.
30 fps
60 fps
120 fps
Figure 10: Results of video frame interpolation from 12 FPS to 30, 60, and 120 FPS. The original 12 FPS frame positions are indicated by triangle markers.
4.5 Video Frame Interpolation
VFRTok designed for L.M,, but also exhibits strong capabilities in Video Frame Interpolation (VFI). We evaluate VFRTok sequences exhibiting large motion from the public UVG (3] dataset.

---

As shown in Figure 10 VFRTok successfully interpolates 12 FPS videos to 30, 60 and 120 FPS. A key advantage of VFRTok its ability perform variable frame rate interpolation, supporting arbitrary input and output frame rates. Performance on VFI can be enhanced by increasing the capacity of the latent space Rdxx xd of VFRTok. Comparison with FLAVV [5] is shown in the Appendix B
5
Conclusion
We propose the Duration-Proportiin Information Assumption, where the upper bound on the ob- servable information capacity of video is proportional to the video duration. Under this assumption, we introduce VFRTok, a Transformer-based video tokenizer capable of encoding and decoding introduce Partial Rotary Position Embeddings to
videos at variable frame rates. Furthermore, decouple positional encoding from content modeling. thereby enhancing content-awareness, ulti- mately improving generation performance. Experiments show that VFRTok achieves comparable reconstruction performance and better generation quality compared to existing tokenizers using 1/8 tokens, while being 11.9x faster. Mcanwhile, VFRTok converges significantly faster than existing works, reducing computational cost by up to 21.6 x1.6x. When the frame rate increases, VFRTok does not require additional denoising tokens in DiT as frame rate increases, further demonstrating its efficiency advantage. Finally. we also find that VFRTok has the potential for video interpolation, capable interpolating 12 FPS videos up to 120 FPS.
Limitations. Dense attention limits VFRTok's s scalability to long videos. Segmenting videos into temporal slices with causal window attention is a potential solution that we leave for future work.
References
[1] Niket Agarwal, Arslan Ali. Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501. 03355,, 2025.
[2] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference IEEE/eeecooferencee computer vision and pattern recognition, pages 22669-22679, 2023.
Chloe Hillier, and Andrew 7isserman. A short note [3] Joao Carreira, Fric Noland, Andras 1808.01444 01340, 2018.
about kinetics-600. arXiv preprint arxiv: 1tw:::188000011
[4] Huiwen Chang, Han Zhang, Lu Jiang. Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 11315-11325, 2022.
[5] Hao Chen, Ze Wang, Xiang Li, Ximeng Sun, Fangyi Chen, Jiang Liu, Jindong Wang. Bhiksha Raj. Zicheng Liu, and Emad Barsoum. Softvq-vac: Efficient I-dimensional continuous tokenizer. arXiv preprint arXiv:2412. 10958, 2024.
[6] Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang. Ze Wang, Zicheng Liu, Difan Zou, and Bhiksha Raj. Masked autnencoders are effective tokenizers diffusion models. arXiv preprint arXiv:2502.0344 03444, 2025.
[7] Duolikun Danier. Fan Zhang, and David R Bull. Bvi-vfi: video quality database for video frame interpolation. IEEE Transactions on Image Processing, 32:6004 1716004-601,,, 2023.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Icce, 2009.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 10.:2.99.1199, 2020.
[10] Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde Fardey Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 199-144, 2020.
10

---

[11] Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Lix-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.0010 2024.
[12] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:220.112598 2022.
[13] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Computer Vision-ECCY ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016. Proceedings, Part II 14, pages 694-711. Springer, 2016.
[14] Ian T Jolliffe. Principal component analysis for special types data, Springer, 2002.
[15] Tarun Kalluri, Deepak Pathak, Manmohan Chandraker. and Du Tran. Flavr: Flow-agnostic video represen- tations for fast frame interpolation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2071 1-222, 2023.
[16] Weijie Kong, Qi Tian, Zijian Zhang. Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint wxii:2412.00300) 2024.
[17] Anders Boesen Lindbo Larsen, Søren Kaae Sonderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning, pages 1558-1566. PMLR. 2016.
[18] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine mnaiiiaieed chinese understanding. arXiv preprint S15i::2005088744 2024.
[19] Zikang Liu, Longteng Guo, Yepeng Tang, Tongtian Yue, Junxian Cai, Kai Ma, Qingbin Liu, Chen, and Jing Liu. Vrope: Rotary position embedding for video large language models. arXiv prepriat arXiv:2502. 1664, 2025.
[20] Guoqing Ma, Haoyang Huang. Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, al. Step-video-12v technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXiv:2502. 10248, 2025.
[21] Nanye Ma, Mark Goldstein, Michael Albergo, Nicholas M Boffi, Eric Vanden- BaadennFijnden, and Saining Xie. Sit: Exploring flow and diffusion-bascd generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23-40. Springer. 2024.
[22] Xin Ma. Yaohui Wang. Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan- Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint aix:v:2401 .00048 2024.
[23] Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/ 50/120fps sequences for video codec analysis and development. In Proceedings of the 11th ACM multimedia systems conference, pages 297-302, 2020.
[24] William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 4195 4205, 2023.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[26] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, arXiv preprint arXiv:1212.0402, 2012.
[27] Alexandros Stergiou. Lavih: A large-scale video interpolation benchmark. arXiv preprint 041x24406.09754, 2024.
[28] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[29] Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. Deep video deblurring for hand-hele helld cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1279 12889,128111111 2017.

---

[30] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[31] Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.
[32] Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with learned autoregressive generative prior. arXiv preprint arXiv:2410.21264, 2024.
[33] Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: joint image-video tokenizer for visual generation. Advances in Neural Information Processing Systems. 17:28281-28295, 2024.
[34] Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang. et al. Videorope: What makes for good video rotary position embedding? arXiv preprint arXiv:2502.00 0573,, 2025.
[35] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06077 2024.
[36] Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501.014 2025.
[37] Qihang Yu, Mark Weber, Xueqing Deng. Xiaohui Shen, Daniel Cremers, and Liang-Chich Chich Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37:128940- 128966. 2024.
[38] Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong. Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers easier than you think. arXiv preprint arXiv:2410.06940 2024.
[39] Kaiwen Zha, Lijun Yu, Alireza Fathi, David A Ross, Cordelia Schmid, Dina Katabi, and Xiuye Gu. gugideddeg
Language-guin guided image tokenization for generation. arXiv preprint arXiv:2412.05796, 2024.
[40] Richard Zhang, Phillip Isola, Alexei Efros, Shechtman, Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pages 586-595, 2018.
[41] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint 12ix:2412.20400 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have stated main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed limitations and possible solutions in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
We provide theoretical assumptions and proofs in Section |
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification:
We present full implementation details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
We provide code and environment description in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp..cppulli//guiii wwwiiccgguie///eeeeeee CoodeSubeiiieeeec ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present all the training and inference details in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the error bars of different methods in Figure 7
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present the compute resources in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte:::///uuis..eeeeeeeee cct.ripsscc....cccccccoooooooocc hwtpppbblic/cccccccccieeeeeeeeee
Answer: [Yes]
Justification:
The experiments do not involve human subjects. We only use public datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is foundational research video tokenizer.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
models
Apache
2.0 license. http::///ppeerrtttooodecc//////p
600
CC BY 4.0
license. htteap///pppeerrrrtooocccccrrrrp MIT license.
https
[/ppeessstttooee.cccmm/aattttt/p DCN. BY-N-SA 4.0 license.
https
//data. bris ac http::aaww...aeaaaeeeee/////r/a/
Non-
Commercial Government Licence for public sector information.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.ppapersttt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Implementation Details
The detailed configuration of VFRTok and LightningDiT are shown in Table 6 and Table 7
Table 6: Training configuration of VFRTok.
Configuration
Value
image resolution
256x256
enc/dec hidden dimension
768
enc/dec #position heads
6
enc/dee #content heads
6
enc/dec #layers
12
enc/dec patch size
4x8x8
enc/dec positional embedding
3D RoPE (video), 1D APE (latent)
optimizer
AdamW
weight decay
le44
optimizer momentum
31, = 0.9, 0.95
global batch size training steps base learning rate learning rate schedule augmentation
512 (stagel), )), 64 (stage2), 16 (stage3)
30k (stagel), 200K (stage2), 100K (stage3)
lc-4 (stagel & stage2), le-5 (stage2) cosine
horizontal flip, center crop
perceptual weight A discriminator
DINOv2-S
discriminator weight A2
0.2
discriminator start discriminator LeCAM
30K
0.001
Table 7: Training and inference configuration of Dight.
Configuration
Value
hidden dimension
1152
#heads
16
#layer
28
patch size
positional embedding
APE
optimizer
AdamW
weight decay
optimizer momentum
$1. 32 0.9, 0.95
global batch size training steps base learning rate
128 (UCFL00/K600), 48 (LAVID)
100K (UCF101/K600), 50K (LAVID) le-4
learning rate schedule
constant
augmentation
center crop
diffusion sampler
Euler
diffusion steps
50
CFG interval start
0.1
timestamp shift
Learnable token details. We adopted the same design existing image ID tokenizers: on the encoder side, we learn N independent latent tokens, whereas on decoder side, we use a single shared token. Although we experimented with replacing the decoder's shared token with fixed- length set of independent tokens, which improves reconstruction fidelity, this change eliminated the decoder's 's ability to flexibly handle variable-frames decoding.
20

---

B Quantitative Results for Video Frame Interpolation
Table shows quantitative video-interpolation results comparing FLAVR 151 and VFRTok. Although FLAVR is a strong intoo-interpolation baseline and outperforms VFRTok on this task, VFRTok was designed primarily for interpolation. First, VFRTok employs an extremely high compression rate to enable efficient video generation, creating tighter bottleneck than dedicated interpolation models. Second, our training set contains only 22 clips at p20fp and no 60fps videos, whereas interpolation models are rypically trained on large-scale, high-frame-rate data. In summary, while VFRTok can perform interpolation, its principal application remains general video generation.
Table 8: Quantitative results for video interpolation.
12 - 24 (2x)
30 + 60 (2x)
- 48 (4x)
+
120 (4x)
15 + 120 (8x)
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
FLAVR
26.05
0.7852
34.22
0.9529
22.17
0.6435
27.59
0.8372
21.70
0.6135
Ours 22.93
0.6724
24.06
0.7324
22.08
0.6433
23.68
0.7215
19.58
0.5279
21


Reviews:
[{"review_id": "H67eL7cUt2", "reviewer": "Reviewer_WTga", "summary": "This paper introduce VFRTok, a transformer-based video tokenizer that trained through asymmetric frame rate training between the encoder and decoder. The length of the token encoded is propotional to the video duration instead of number of frames. In addition, a Partial RoPE is proposed to allow the model's content-aware learning. Experiments demonstrate the high compression rate and good reconstruction/generation quality. VFRTok also shows the potential in video frame interpolation.", "questions": "1. Is it possible to show evidence that when used as the tokenizer of a video generation model, VFRTok will not limit the contents' movement in video generation compared with other video tokenizers? \n2. Since the Partial RoPE can improve the model's content-awareness, will it facilitate the video generation model to learn the knowledge of movement? For example, does it enable the video generation model to generate videos that better align with physical rules or exhibit more realistic motion patterns?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed VFRTok is a potential solution to the inconsistency of frame rate in video generation data in real world practice, especially In scenarios like game video generation, where the frames in the training data are rendered by computers and it is difficult to keep the frame rate stable during rendering.\n2. The experiment result seems promising. The high compression rate is a critical advantage for downstream applications, such as its use as a tokenizer in video generation models, where efficiency and compactness are essential.\n3. The design of Partial RoPE is reasonable and effective. \n4. When used as a tokenizer for video generation models, VFRTok 's flexibility in handling varying frame rates has the potential to alleviate the burden on the generation model by sharing the responsibility of frame rate control. It may also help mitigate the negative impact of inconsistent frame rates in training data, thereby improving the learning process of the video generation model.\n\nWeakness:\nI suspect that the underlying assumption of duration-proportional information has inherent limitations. This assumption appears valid only when the content within the frames is not moving rapidly. However, in cases where the content is moving quickly, videos with higher frame rates inherently capture significantly more information than those with lower frame rates over the same duration. From this perspective, I am concerned that VFRTok may inadvertently constrain the representation of fast-moving content when applied to video generation tasks. This limitation could restrict the model's ability to accurately capture dynamic scenes, potentially impacting its generation quality in scenarios involving rapid motion.", "was_revised": true, "final_justification": "The proposed method is novel and valuable for real-world applications. The Duration-Proportional Information Assumption is a new and valuable perspective for research in video tokenizer.", "rebuttal": "We appreciate your positive feedback on our motivation and architectural design.\nThank you as well for illustrating the application scenario of game video generation.\nBelow, we address each of the noted weaknesses and questions in turn.\n\n# W1 - Duration-Proportional Information Assumption and content-adaptive modeling\n\nWe appreciate your concerns.\nIt is true that fast‑moving videos often convey more information.\nHowever, it is important to note that high frame‑rate video and high‑speed motion are not synonymous: a video’s frame rate and the speed of the motion it depicts are two decoupled factors.\nConsequently, traditional video tokenizers, which hold the Frame-Proportional Information Assumption, face the same limitation.\nGiven a fixed frame rate, a standard tokenizer allocates a fixed number of tokens to represent each frame (often far more than VFRTok).\nAs motion increases, reconstruction quality naturally degrades under this scheme.\n\nMore generally, videos vary in semantic, textural, and motion complexity, and thus should require varying information capacity.\nIn conventional video codecs (*e.g.*, H.264), achieving the same reconstruction quality for two videos of differing complexity typically demands different bitrates.\nAnalogously for tokenizers, the length of the latent representation ought to adapt to a video’s content complexity.\nThis content‑adaptive modeling perspective is orthogonal to the hypothesis explored in our paper.\n\nWhile content‑adaptive tokenizers have been explored in a few works, the key challenge lies in enabling the downstream DiT model to support estimating a video’s complexity from its caption alone.\nWe plan to address this in future work.\n\n# Q1 - Fast motion modeling\n\nVFRTok’s latent capacity is adequate for capturing rapid motion.\nAs illustrated in Figure8, VFRTok successfully generates scenes of fast-paced activities, such as skiing and pull‑ups.\nNotably, the examples shown in Figure~8 are drawn from clips only 2–3 seconds in length, underscoring the model’s ability to represent and recreate high‑speed motion faithfully.\n\n# Q2: Partial RoPE enhances motion modeling\n\nOur analysis shows that incorporating Partial RoPE yields markedly more coherent motion.\nIn particular, with Partial RoPE enabled, objects and human figures exhibit substantially less fragmentation, deformation, and distortion throughout their movements."}, {"review_id": "MuX3auTmHJ", "reviewer": "Reviewer_Zg58", "summary": "This paper proposes a new video tokenizer for video generation. The method is based on the assumption that a video's information is upper bounded by its duration not the number of frames. The tokenizer is based on recent 1D tokenizers using Transformers that directly patchify pixel cubes into tokens and produce latent tokens with learnable queries. On top of the 1D tokenizer design, the authors introduced 1) the split-head 3D RoPE which was previously used in Flux and QWen models; 2) the position heads and content heads, where the content heads are similar to NoPE. The tokenizer is trained with random pairs of high-to-low or low-to-high FPS rates in the inputs and outputs, effectively learning frame interpolation or sumsampling. \n\nThe proposed tokenizer is tested in video reconstruction and video generation benchmarks, showing better compression ratio and generation quality than grid based video tokenizers, while using few number of latent tokens.", "questions": "My major question relates to weakness #2: how to deal with varying spatial resolution with the current tokenizer design?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ Making video tokenizer less dependent on the frame number, which is an artifact of the video production process, seems natural and beneficial for video tokenization. The choice of Transformer-based 1D tokenizer makes this viable. \n+ The design choices 1) and 2) seem to be sound and reasonable.\n+ The training strategy that uses the frame interpolation/subsampling to make model adative to frame rates is very interesting and could generate good impact in the community. \n+ Strong experimental results. \n\nWeaknesses\n\n- The 3D rope design and NoPE usage both lack references to the existing literature. I would recommend the authors to carefully revise the related work section. \n- Video data has three dimensions in general, height, width, and time. In this work with a fixed latent token amount, the time dimension is flexible. But the choice of using 1D tokenizer with fixed latent token count makes it non-trivial to deal with the varying height and width of the video frames.", "was_revised": true, "final_justification": "No other concern raised. Keeping my initial recommendation.", "rebuttal": "We appreciate your positive evaluation of our work’s motivation, design strategy, and experimental findings.\nBelow, we address each of the noted weaknesses and questions in turn.\n\n# W1 - RoPE related works\n\nWe will include more comprehensive references to 3D RoPE and NoPE in the camera‑ready manuscript.\nIt is important to note that existing 3D RoPE follows the same channel‑division strategy as VFRTok: each attention head’s channels are split into three groups, with RoPE applied independently along the temporal, width, and height dimensions (Equation 2).\nTo our knowledge, no other work employs this split-head Partial RoPE approach.\n\n# W2/Q1 - Generalization to resolution\n\nExtending VFRTok to handle variable resolutions is another direction we are exploring.\nA natural extension of our Duration‑Proportional Information Assumption is a Spatial‑Temporal‑Proportional Information Assumption, which would allow videos of differing frame rates and resolutions to be encoded into fixed‑length latents.\nIn practice, we would achieve this simply by applying interpolated RoPE along the spatial dimensions (height and width) for different‑resolution inputs.\n\nThat said, stretching the video representation across three dimensions may lead to capacity inefficiencies in low‑frame‑rate and low‑resolution segments.\nTo mitigate this, we plan to design a sublinear, variable‑compression‑rate video tokenizer in future work."}, {"review_id": "SI9f8sLQvC", "reviewer": "Reviewer_VrQd", "summary": "This paper introduces VFRTok, a query-based 1D transformer tokenizer that encodes videos of variable frame rate and same length into the same number of latent tokens. This is achieved by converting the frame index to the timestamp when computing the temporal rotation angles of a 3D RoPE. They further propose Partial RoPE, where they only apply RoPE to a subset of the attention heads. The authors conduct experiments on video reconstruction and generation based on VFRTok.", "questions": "- Sec. 4.5 shows that VFRTok can perform frame interpolation. Given a variable frame rate video LDM, can VFRTok be used to speed up inference by letting the LDM to generate videos latents with lower FPS and then use VFRTok to interpolate to higher FPS during or after decoding? If this is possible, this will be a significant contribution to the video generation field.\n- Code is provided in supplementary material. Will the code and model weights be released to the public?\n- From Fig. 3, it seems that only the patch tokens have RoPE while the latent tokens don't. Why such design and has this been ablated?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- VFRTok achieves competitive reconstruction results and sota generation results while converge much faster.\n- Both variable frame rate design and Partial RoPE are novel contributions to video tokenization and easy to implement.\n- Partial RoPE is motivated by empirical observation where the attention is dominated by the grid position prior in Fig. 6.\n- Partial RoPE is empirically validated in ablation study in Tab. 4.\n- VFRTok supports interpolating videos from 12 FPS to 120 FPS.\n\n## Weaknesses\n- Some figures (e.g. Fig. 1-6), while informative, are a bit confusing and needs additional captions or paragraphs in the main paper. \n- More details for the training (expanding L138-L145) and inference are needed. For example, how VFRTok adjust encoding/decoding frame rates during inference? Do VFRTok learn shared or separate sets of learnable tokens for each frame rate?\n- No quantitative results for video interploation in Sec. 4.5.", "was_revised": true, "final_justification": "My questions are addressed in the rebuttal. I only have a minor suggestion on improving the clarity of figures and captions (see comment below). I maintain my rating of accept, since the method is a novel contribution and should have high impact in the field, where the number of tokens is a key bottleneck in the training and inference efficiency of video DiTs.", "rebuttal": "We appreciate your positive feedback on the technical novelty and experimental results.\nBelow, we address each of the noted weaknesses and questions in turn.\n\n# W1 - Additional captions\n\nWe will further refine captions to enhance readability.\nFor example, we may modify the caption of Figure 1 to “*The number of tokens for other tokenizers grows with frame rate. VFRTok maintains a fixed number of latent tokens tied to video duration and supports asymmetric frame‑rate encoding and decoding.”*\n\n# W2 - Experimental details\n\nWe will expand on the details of the training.\n\n**Inference details.** During inference, VFRTok supports decoding videos of any frame rate from latent tokens denoised by DiT.\nSpecifically, the VFRTok decoder calculates the number of learnable tokens based on the incoming frame rate and injects the timestamp-based RoPE into each token.\n\n**Learnable token details.** We adopted the same design as existing image 1D tokenizers: on the encoder side, we learn N independent latent tokens, whereas on the decoder side, we use a single shared token.\nAlthough we experimented with replacing the decoder’s shared token with a fixed-length set of independent tokens, which improves reconstruction fidelity, this change eliminated the decoder’s ability to flexibly handle variable‑frame‑rate decoding.\n# W3 - Quantitative results for video interpolation\n\nThe following table shows quantitative video‑interpolation results comparing FLAVR and VFRTok.\nAlthough FLAVR is a strong video‑interpolation baseline and outperforms VFRTok on this task, VFRTok was not designed primarily for interpolation.\nFirst, VFRTok employs an extremely high compression rate to enable efficient video generation, creating a tighter bottleneck than dedicated interpolation models.\nSecond, our training set contains only 22 clips at 120fps and no 60fps videos, whereas interpolation models are typically trained on large-scale, high‑frame‑rate data.\nIn summary, while VFRTok can perform interpolation, its principal application remains general video generation.\n\n| **PSNR** | **12→24** | **30→60** | **12→48** | **30→120** | **15→120** |｜ **SSIM** | **12→24** | **30→60** | **12→48** | **30→120** | **15→120** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| **FLAVR** | 26.05 | 34.22 | 22.17 | 27.59 | 21.70 | ｜**FLAVR** | 0.7852 | 0.9529 | 0.6435 | 0.8372 | 0.6135 |\n| **Ours** | 22.93 | 24.06 | 22.08 | 23.68 | 19.58 | ｜**Ours** | 0.6724 | 0.7324 | 0.6433 | 0.7215 | 0.5279 |\n\n# Q1 - High frame rate video generation\n\nTraditional Frame-Proportional Information Assumption tokenizers must encode high frame rate videos during DiT training to enable high frame rate generation.\nIn contrast, VFRTok’s Duration‑Proportional Information Assumption allows it to map videos of any frame rate into fixed‑length latent sequences.\nBecause the VFRTok decoder converts these fixed‑length latents into variable‑length patch tokens, it can efficiently generate high frame rate videos, even when the DiT model is trained exclusively on low frame rate data.\n\n# Q2 - Release model weights\n\nWe plan to release both the code and model weights concurrent with the camera‑ready version.\n\n# Q3 - Position embedding for latent tokens\n\nWe adopt the architecture of existing 1D image tokenizers by incorporating learnable absolute positional encodings into our latent tokens.\nWe omit RoPE because modeling relative positions between latent tokens is unnecessary.\nUnlike video patch tokens, where proximity in time and space implies similar content, the latent tokens aggregate global information, so adjacent latents need not correspond to semantically related features."}, {"review_id": "NiAUgklJgZ", "reviewer": "Reviewer_bssa", "summary": "This paper proposes VFRTok, a video tokenizer based on the **duration-proportional information assumption**, aiming to address the inefficiency in current video generation models caused by variations in frame rate. Existing video tokenizers typically use a fixed compression rate, resulting in the computational cost of diffusion models scaling linearly with frame rate. VFRTok introduces a variable frame rate encoding/decoding mechanism and incorporates an improved RoPE (Partial RoPE) to achieve more compact and continuous spatiotemporal representations.", "questions": "1. To my knowledge, models such as WFVAE, Hunyuan VAE, and Wan VAE have been trained extensively across different frame rates and can also reconstruct and generate videos at arbitrary frame rates. What advantages does your approach offer compared to these methods?\n2. Since there are multiple choices for frame rate, the number of frames corresponding to the same duration will vary. If the number of learnable tokens is fixed, it may be difficult to adaptively align the output frame count. How is training conducted in this case? Is the number of tokens constant across different frame rates? Does information loss occur when the frame rate is too high?\n3. Is the Duration-Proportional Information Assumption applicable to all types of video?\n4. What are the results of symmetric training across frame rates? This is not analyzed in the paper.\n5. Partial RoPE applies RoPE only to some attention heads. While this improves content modeling, could it potentially weaken the model's ability to accurately model spatiotemporal positions, thereby affecting generation consistency?", "limitations": "see Questions", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The introduction of Partial RoPE cleverly addresses the issue of content modeling interference by positional modeling in Transformers.\n2. The method supports video interpolation at arbitrary frame rates (from 12 FPS to 120 FPS), demonstrating strong generalization ability.\n3. The paper is logically well-structured and contains rich visualizations and figures.\n\n**Weaknesses:**\n\n1. The paper does not discuss in depth how the Duration-Proportional Information Assumption theoretically guarantees the effectiveness of this compression method (e.g., information fidelity, reconstruction error).\n2. Some experimental details (such as optimizer settings, learning rate schedules, regularization methods) are not fully described in the main text, affecting the reproducibility and independent evaluation of the work.", "was_revised": true, "final_justification": "The author has addressed part of my concerns. I will keep my score at Borderline Accept.", "rebuttal": "We appreciate your positive feedback on the architecture design and paper logic.\nBelow, we address each of the noted weaknesses and questions in turn.\n\n# W1/Q2 - Duration-Proportional Information Assumption and practice\n\nThe Duration‑Proportional Information Assumption is motivated by three observations:\n\n- Video information content grows as frame rate increases.\n- However, the rate of that growth diminishes at higher frame rates.\n- As the frame rate approaches infinity, the total information converges to a finite upper bound. (Equation 1)\n\nThe traditional tokenizers uses a fixed spatiotemporal compression rate, which implicitly assumes a linear, unbounded increase in information with frame rate.\nIn contrast, VFRTok instead employs a fixed-length latent sequence to approximate that theoretical upper bound.\n\nIn the table below, we compare reconstruction performance on the Adobe240fps dataset between standard tokenizers and VFRTok across multiple frame rates.\nAs expected, the traditional tokenizers' token count and its reconstruction quality grows linearly with frame rate, implying that it overallocates capacity at high frame rates.\nWhile VFRTok's reconstruction quality degrades modestly at higher frame rates.\nThe experimental result confirms that real video information scales sublinearly.\nWe note that VFRTok was trained primarily on 24fps and 30fps data, with only 22 clips at 120fps; increasing the proportion of high–frame‑rate examples should further improve its fidelity at those rates.\n\n| FPS | Metrics | ｜Omni | Cosmos-L | Cosmos-M | **Cosmos-S** | **Ours** |\n| --- | --- | --- | --- | --- | --- | --- |\n| 24 | #Tokens | ｜4096 | 4096 | 2048 | **512** | **512** |\n| 24 | PSNR | ｜25.38 | 27.69 | 26.06 | **24.06** | **25.85** |\n| 60 | #Tokens | ｜10240 | 10240 | 5120 | **1280** | **512** |\n| 60 | PSNR | ｜25.64 | 29.16 | 27.81 | **25.47** | **24.53** |\n| 120 | #Tokens | ｜20480 | 20480 | 10240 | **2560** | **512** |\n| 120 | PSNR | ｜23.60 | 29.80 | 28.53 | **26.09** | **23.95** |\n\nWhile neither the Frame‑Proportional nor the Duration‑Proportional Information Assumption perfectly captures the sublinear growth of video information with frame rate, the Duration‑Proportional Assumption nonetheless offers clear practical advantages over a purely Frame‑Proportional approach in downstream generation.\nBy fixing the latent length, it enables a single DiT model to denoise and generate videos at arbitrary frame rates, and it delivers better efficiency gains as target frame rate increases.\n\n# W2 - Experimental details\n\nWe will add more experimental details in the appendix in the next version.\nThe configuration of VFRTok and LightningDiT are shown in the following tables.\n\n| VFRTok Configuration | Value |\n| --- | --- |\n| image resolution | 256$\\times$256 |\n| enc/dec hidden dimension | 768 |\n| enc/dec #position heads | 6 |\n| enc/dec #content heads | 6 |\n| enc/dec #layers | 12 |\n| enc/dec patch size | 4$\\times$8$\\times$8 |\n| enc/dec positional embedding | 3D RoPE (video), 1D APE (latent) |\n| optimizer | AdamW |\n| weight decay | 1e-4 |\n| optimizer momentum | $\\beta_1,\\beta_2=0.9,0.95$ |\n| global batch size | 128 (stage1), 32 (stage2) |\n| training steps | 200K (stage1), 100K (stage2) |\n| base learning rate  | 1e-4 (stage1), 1e-5 (stage2) |\n| learning rate schedule | cosine |\n| augmentation | horizontal flip, center crop |\n| perceptual weight $\\lambda_1$ | 1 |\n| discriminator | DINOv2-S |\n| discriminator weight $\\lambda_2$ | 0.2 |\n| discriminator start | 30K |\n| discriminator LeCAM | 1e-3 |\n\n| LightningDiT Configuration | Value |\n| --- | --- |\n| hidden dimension | 1152 |\n| #heads | 16 |\n| #layer | 28 |\n| patch size | 1 |\n| positional embedding | APE |\n| optimizer | AdamW |\n| weight decay | 0 |\n| optimizer momentum | $\\beta_1,\\beta_2=0.9,0.95$ |\n| global batch size | 128 (UCF101/K600), 48 (LAVID) |\n| training steps | 100K (UCF101/K600), 50K (LAVID) |\n| base learning rate | 1e-4 |\n| learning rate schedule | constant |\n| augmentation | center crop |\n| diffusion sampler | Euler |\n| diffusion steps | 50 |\n| CFG interval start | 0.1 |\n| timestamp shift | 2 |\n\n# Q1 - Advantages compared to existing works\n\nAlthough prior methods also train on raw videos at multiple frame rates, their strategies differ fundamentally from VFRTok.\nThey either resample every video to a single, fixed frame rate before encoding or simply ignore frame rate by feeding all frames into the model.\nThe former approach constrains the model to that chosen frame rate.\nWhile the latter enforces a uniform spatiotemporal compression rate regardless of frame rate, forcing high-frame-rate videos to be “slowed down” into unnecessarily many tokens and low-frame-rate videos to be “speed up”.\nBoth of which limit the expressivity of the latent representation.\n\nAs shown in Figure 1, these methods operate under a Frame‑Proportional Information Assumption, where token count grows linearly with frame rate (*e.g.*, a 120 fps clip demands four times the tokens of a 30 fps clip).\nBy contrast, we find that a video’s information content increases sublinearly with frame rate and asymptotically approaches an upper bound (Duration-Proportional Information Assumption).\nVFRTok leverages this insight by using a fixed number of latent tokens to approximate that bound, thus unifying videos of the same duration but different frame rates under one consistent representation.\nSince the downstream DiT model’s computational complexity scales geometrically with the number of latent tokens, VFRTok’s constant-length encoding delivers significant efficiency gains for high-frame-rate video generation.\n\n# Q3 - Applicability\n\nWhen contrasting the Duration‑Proportional Information Assumption with the Frame‑Proportional Information Assumption, we are claiming that allocating information uniformly over time yields statistically better performance than allocating it uniformly over frames.\nHowever, at the sample level, both assumptions share a common limitation: videos exhibit widely varying content complexity.\nSpecifically, videos vary in semantic, textural, and motion complexity, and thus should require varying information capacity.\nIn conventional video codecs (*e.g.*, H.264), achieving the same reconstruction quality for two videos of differing complexity typically demands different bitrates.\nAnalogously for tokenizers, the length of the latent representation ought to adapt to a video’s content complexity.\nThis content‑adaptive modeling perspective is orthogonal to the hypothesis explored in our paper.\n\nWhile content‑adaptive tokenizers have been explored in a few works, the key challenge lies in enabling the downstream DiT model to support estimating a video’s complexity from its caption alone.\nWe plan to address this in future work.\n\n# Q4 - Symmetric training\n\nWe train a VFRTok variant under a symmetric encoding regime.\nSpecifically, we initialized from the K600 pre-trained model in stage 1 for quick verification.\nIn stage 2, which uses both K600 and BVI-HFR datasets, we disabled asymmetric encoding, forcing both encoder and decoder to operate at the same frame rate.\nAs shown in the following table, this symmetric training yields a marginally better reconstruction in the symmetric setting but suffers a pronounced degradation under asymmetric encoding.\nNote that our stage 1 initialization itself used asymmetric frame rates, which partially narrows the gap in asymmetric reconstruction.\nOverall, these results confirm that asymmetric training is crucial for enabling VFRTok to generate videos at arbitrary frame rates.\n| **PSNR** | **30fps** | **60fps** | **120fps** | ｜**PSNR** | **30→60fps** | **30→120fps** | **60→120fps** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| **Symm** | **25.92** | **24.71** | 23.89 | ｜**Symm** | 22.36 | 21.60 | 23.33 |\n| **Ours** | 25.87 $\\downarrow0.05$ | 24.53$\\downarrow0.18$ | **23.95**$\\uparrow0.03$ | ｜**Ours** | **24.05**$\\uparrow1.69$ | **23.56**$\\uparrow1.96$ | **23.90**$\\uparrow0.66$ |\n\n# Q5 - Partial RoPE and generation consistency\n\nOur experiments show that adding Partial RoPE does not compromise temporal consistency.\nAs reported in the following table, we compute the average L2 distance between consecutive frames with and without Partial RoPE.\nThe results reveal only a negligible difference, confirming that Partial RoPE maintains temporal coherence.\n\nMeanwhile, we find that incorporating Partial RoPE yields markedly more coherent motion in the generated videos.\nIn particular, with it enabled, objects and human figures exhibit substantially less fragmentation, deformation, and distortion throughout their movements.\n\n| L2 distance | K600 | UCF101 |\n| --- | --- | --- |\n| w/o Partial RoPE | 13.93 | 12.26 |\n| w/ Partial RoPE | 14.14 | 11.77 |"}]

Author rebuttal:

Meta-review: The authors propose VFRTok, a query-based 1D transformer tokenizer that encodes videos of variable frame rate and same length into the same number of latent tokens. Additionally, the partial RoPE is proposed to encourage a functional separation between positional and content encoding. Promising results are demonstrated on video generation and video interpolation.

Initially, the reviewers raised several concerns, which are briefly outlined below:

* Reviewer WTga: Method's limitations.

* Reviewer Zg58: Lack of citing related works, limitations to deal with varying height and width of the video frames.

* Reviewer VrQd: Refinement on figures, details of training and inference, quantitative results for video interpolation.

* Reviewer bssa: Discussion on the Duration-Proportional Information Assumption, implementation details.

The rebuttal and subsequent author-reviewer discussions effectively addressed the reviewers' concerns. After carefully considering the reviews, rebuttal, and discussion, the AC concurs with the reviewers’ assessment (particularly, regarding VFRTok's potential to handle varying frame rates for video generation) and thus recommends acceptance of the paper. 

Additionally, the AC kindly notes that the 1D image tokens approach [29] has also been extended to video [A] and further improved in subsequent work [B]. A proper discussion and citation of these related studies would strengthen the revision. Finally, the authors are encouraged to incorporate the rebuttal experiments into the manuscript and address the reviewers’ feedback in the final revision.


[A] LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior. ICLR 2025.

[B] Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens. ICCV 2025
Decision: Accept (poster)