TPDiff: Temporal Pyramid Video Diffusion Model
Authors: Lingmin Ran
Mike Zheng Shou
Venue: ICLR 2026 Poster (2026)
Keywords: Video Generation
Diffusion model
Primary area: generative models
TL;DR: TPDiff reduces the training cost of video diffusion models by 50% and accelerates inference by 1.5 times.

Abstract: The development of video diffusion models unveils a significant challenge: the substantial computational demands. To mitigate this challenge, we note that the reverse process of diffusion exhibits an inherent entropy-reducing nature. Given the inter-frame redundancy in video modality, maintaining full frame rates in high-entropy stages is unnecessary. Based on this insight, we propose TPDiff, a unified framework to enhance training and inference efficiency. By dividing diffusion into several stages, our framework progressively increases frame rate along the diffusion process with only the last stage operating on full frame rate, thereby optimizing computational efficiency. To train the multi-stage diffusion model, we introduce a dedicated training framework: stage-wise diffusion. By solving the partitioned probability flow ordinary differential equations (ODE) of diffusion under aligned data and noise, our training strategy is applicable to various diffusion forms and further enhances training efficiency. Comprehensive experimental evaluations validate the generality of our method, demonstrating 50% reduction in training cost and 1.5x improvement in inference efficiency.

Published as a conference paper at ICLR 2026
TPDIFF: TEMPORAL PYRAMID VIDEO DIFFUSION MODEL
Lingmin Ran & Mike Zheng Shou'
Department of Electrical and Computer Engineering National University of Singapore University Hall, Singapore (andy ran, mikeshou) nuss edu.sg
ABSTRACT
The development of video diffusion models unveils a significant challenge: the substantial computational demands. To mitigate this challenge, note that the reverse process of diffusion exhibits an inherent entropy-reducing nature. Given the inter-frame redundancy in video modality, maintaining full frame rates in high- entropy stages is unnecessary. Based on this insight, we propose TPDiff. a unified framework to enhance training and inference efficiency. By dividing diffusion into several stages, our framework progressively increases frame rate along the diffusion process with only the last stage operating on full frame rate, thereby optimizing computational efficiency. To train the multi-stage diffusion model, we introduce a dedicated training framework: stage-wise diffusion. By solving the partitioned probability flow ordinary differential equations (ODE) of diffusion under aligned data and noise, our training strategy is applicable to various diffusion forms and further enhances training efficiency. Comprehensive experimental evaluations validate the generality of our method, demonstrating 50% reduction in training cost and 1.5x improvement in inference efficiency.
INTRODUCTION
With the development of diffusion models, video generation has achieved significant breakthroughs. The most advanced video diffusion models (OpenAl 2024 KuaiShou 2024 Polyak et al.) 2024)
not only enable individuals to engage in artistic creation but also demonstrate immense potential in other fields like robotics (Songming al. (024) and virtual reality (Melech et al. 2023). Despite the powerful performance of video diffusion models, the complexity of jointly modeling spatial and temporal distribution makes their training costs prohibitively high (OpenSora) 2023 Zhuoyi et al.
2024; Weijie et al., (024). Moreover, as the demand for long videos increases, the training and inference costs will continue scale accordingly.
To alleviate this problem, researchers propose series of approaches to increase training and inference efficiency. Show-1 (Zhang et al. 2023) and Lavie (Wang et al., 2023) adopts cascaded framework to model temporal relations at low resolution and apply super-resolution to improve the final video resolution. However, the cascaded structure leads to error accunulation and significantly increases the inference time. SimDA (Zhen et al.) (024)) proposes lightweight model which replaces Atten- tion (Vaswani et al.) (017) with 3D convolution (Du et al. 20155) to model temporal relationship. Although convolution is computationally efficient, DiT (William & Xie (023) demonstrates that attention- based model is scalable and achieves better performance as the volume of data and model parameters increases. Recently, (Yang al.) 2024) introduces an interesting work: pyramid flow. This method proposes spatial pyramid: employs low resolution during the early diffusion steps and gradually increases the resolution as the diffusion process proceeds. It avoids the need to always maintain full resolution, significantly reducing computational costs.
However, pyramid flow has several problems: 1) It only demonstrates its effectiveness under flow matching Xingchao et al. 2022) and does not explore its applicability to other diffusion forms like
"Coresponding Author.

---

Published as a conference paper at ICLR 2026
Denoising step: t=l
Training Emticlnccy
Vonila Diftustan
Some frame rate
rate incressey
FM Ours
inference efficiency
[a] Vanilla Video Diffusion Model
teT
[b) TPDiff
(c] Compartson of our method and vanilla diffusion
Figure I: Overview of our method. Our method employs progressive frame rates, which utilizes full frame rate only in the final stage shown in (a) and (b), thereby largely optimizing computational efficiency in both training and inference shown in (c).
denoising diffusion implicit models (DDIM) (Jiaming et al. 2020). 2) It formulates video generation in an auto-regressive manner which significantly reduces inference speed. 3) The feasibility of modeling temporal relationship in pyramid-like structure remains unexplored.
To solve the problems, we propose TPDiff. general framework to accelerate training and inference speed. Our method inspired by the fact that video is highly redundant modality (Ma et 2019). as consecutive frames often contain minimal variations. Additionally, in a typical diffusion process, latents in the early timesteps contain limited informative content and the temporal relations between frames are weak, which makes maintaining full frame rate throughout this process unnecessary. Based on this insight, we propose temporal pyramid: 1) In our method, the frame rate progressively increases the diffusion process proceeds as shown in Fig, Unlike previous works (Wang et al.) 2023] Zhang et al. 2023) require an additional temporal interpolation network, we adopt a single model handle different frame rates. To achieve this, we divide the diffusion process into multiple stages, with each stage operating at different frame rate. 2) To train the temporal pyramid model, solve the partitioned probability flow ordinary differential equations (ODE) (Yang al. [02T]] Cheng et al. 2022) by leveraging data-noise alignment and reach a unified solution for various types of diffusion. 3) Our experiments show method is generalizable to different diffusion forms, including flow matching and DDIM, accelerating training and inference compared to vanilla diffusion models as shown in Fig. The core contributions of this paper are summarized as follows:
We introduce temporal pyramid video diffusion model, a generalizable framework aiming enhancing the efficiency of training and inference for video diffusion models. By employing progressive frame rates across different stages of the diffusion process, the framework achieves substantial reductions in computational cost.
We design a dedicated training framework: stage-wise diffusion. We solve the decomposed probability flow ODE by aligning noise and data. The solution is applicable to different diffusion forms, enabling flexible and seamless extension to various video generation frameworks.
Our experiments demonstrate that the proposed method can be applied across various diffusion frameworks, achieving performance improvement, 2. faster training and 1.5x faster inference.
2 RELATED WORKS
Generative Video Models. The field of video generation has witnessed significant progress recently due to the advancement of diffusion models (Sohi-Dickstein et al. 2015 DDhariwal & Nichol (021) These models generate videos from text descriptions or images. Most methods develop video models based on powerful text-to-image models like Stable Diffusion (Rombach et al., (021). adding extra layers to capture cross-frame motion and ensure consistency. Among these, Tune-A-Videe (Wul et al. 2023) employs a causal attention module and limits training module to reduce computational costs. AnimateDiff (Guo et al.] 2024) utilizes plug-and- aad--and-payy temporal module to enable video generation on personalized image models (AI). Recently, DiT models (Xin et al. 2024
OpenSora
2023) pushes the boundaries of video generation. Commercial products (OpenAl
2024 KuaiShou
2

---

Published as a conference paper at ICLR 2026
Diffusion Process
Vanilla Diffusion
Ours
Random sampling
Data-noise alignment
t-0
1=0
t-t
t-t
t=t
t-t
t=T
t-T
Temporal interpolation
Noise direction
Diffusion trajectory
Real Data
Noise
(a) Temporal pyramid video diffusion model
(b) Training strategy
(c) Data-noise alignment
Figure Methodology. a) Pipeline of temporal pyramid video diffuusion model. We divide diffusion process into multiple stages with increasing frame rate. In each stage, new frames are initially temporally interpolated from existing frames. b) Our training strategy: stage-wise diffusson. In vanilla diffusion models, the noise direction along the ODE path points toward the real data distribution. In stage-wise diffusion, the noise direction is oriented to the end point of the current stage. c) Data-Noise Alignment. For every training sample, vanilla diffusion training randomly samples noises across the entire noise distribution, resulting in stochastic ODE path during training. In contrast, our method samples noises in the closest range, making the ODE path approximately deterministic during training
2024 Luma 2024 and
works (Weijie et al. 2024 Zhuoyi et al. 2024 OpenSora
2023 demonstrate remarkable performance by scaling up DiT pretraining. Although DiT achieves significant performance improvements, its training cost escalates to an unaffordable level, hindering the development of video generation.
Temporal Pyramid. The complex temporal structure of videos raises a challenge for generation and understanding. SlowFast (Christoph et al., 2019) simplifies video understanding by utilizing an input-level frame pyramid, where frames at different levels are sampled at varying rates. Each level independently processed by separate network, with their mid-level features interactively fused. This combination of the frame pyramid enables SlowFast to efficiently manage the variability of visual tempos. Similarly, DTPN (Da Zhang & Wang.) 2018) employs different frame-per-seone (FPS) sampling to construct pyramidal representation for videos of arbitrary length. Temporal pyramid network (Yang et al. 2020) leverages the feature hierarchy to handle the variance of temporal information. It avoids to learn visual tempos inside single network, and only need frames sampled at a single rate at the input-level. Although the effectiveness of temporal pyramid have been validated in video understanding, its application in generation remains explr-explored.
METHOD
3.1 PRELIMINARY
Denoising Diffusion Implicit Models DDIM (Jiaming et al. 2020) extends DDPMs (Ho et al.
2020) by operating in the latent space. Similar to DDPM, in the forward process, DDIM transforms real data zo into a series of intermediate sample I,, and eventually the Gaussian noise (0, according noise schedule ā.:
Co.D + V1 aee. N(O,
Zt
where ~ [1, 7] and 7 denotes the total timesteps. After adding noise to the latent, we usually train neural network eg predict the added noise. Formally, eg is trained using following objecive:
min Ez, en (0, b> Unifomm (1,7) ||e eg (Iz, 1
(2)
Given a pretrained diffusion model cg. one can generate new data by solving the corresponding probability flow ODE YYagg al. 2021) DDIM is essentially first-order ODE solver, which
3

---

Published as a conference paper at ICLR 2026
formulates denoising process to generate 24-1 from sample 24 via:
v1 aneo catola + vI (x,0
Ft
-t-1 = 04-11
a
where 0t 0r-1
Flow Matching Flow-based generative models aim to learn a velocity field Up that transports Gaussian noise e ~ (0, 1) to the distribution of real data To. Flow matching (Xingchao et al. 2022) adopts linear interpolation between noise and data ze:
(1 t) + te, (0,
(4)
It trains neural network eg match the velocity field and then solves the ODE for given boundary condition e to obtain the flow. The flow matching loss function is as follows:
min Ex (o.1)) Uniform (1.2) [I(e xo) - do (xr, 1)
(5)
3.2 TEMPORAL PYRAMID DIFFUSION
The core module of existing video diffusion models, attention (Vaswani et al. 2017). exhibits quadratic complexity with respect to sequence length. Our goal to reduce the sequence length in video generation and decrease the computational cost. Our method is based on two key insights: 1) There considerable redundancy between consecutive video frames. 2) the early stages of the diffusion process remain at low signal-to-noise ratio (SNR), resulting in minimal information content. suggests that operating at full frame rate during these initial timesteps unnecessary. Based on these insights, propose temporal pyramid video diffusion shown in Fig. (a). Compared to traditional video diffusion model using fixed frame rate, our framework progressively increases the frame rate the denoising proceeds.
In detail, we divide the diffusion process into multiple stages, each characterized by a distinct frame rate, and employ single model to learn data distributions across all stages. We create K stages (1k, (x=1)) where l 12-.. < (K-1 T, T denotes the total timesteps. The frame rate the ttth stage is reduced शーーr of the original one. It ensures that only the last stage operates at full frame rate, thereby optimizing computational efficiency. Despite efficiency, the vanilla diffusion model does not support multi-stage training and inference. Therefore, the remaining challenges are: 1) How to train the multi-stage diffsion model in a unified way. which will be introduced in Section 3.3 and Section 3.4 2) How to perform inference, which will be discussed Section 3.6
3.3 TRAINING STRATEGY
In stage k. denote (sk, Ck) as the start and end timestep, Peg and T.. start and end point. The objective of training is to transport distribution of L.g. to Leg at every stage. To achieve the objective, the key is to obtain stage-wise 1) target, iee. in DDIM and dæs in flow matching, and 2) intermediate di latents 24 where E [sk, ek) (Fu- Yun al. 2024 Hanshu et al. 2024). In the following, will introduce a unified training framework named stage-wise diffusion.
Stage-wise Diffusion To ensure generality, recognizing different diffusion frameworks share we present unified diffusion form:
similar formulation as shown in Equation and Equation 4
Et 7: o
(6)
where the form of 7r and Tr depend on diffusion framework selected. Our derivation is based on Equation without constraining the parameterization of 7) and o,. Considering continuity between stages with distinct frame rates, we obtain e.2 and fe. by:
Boy Yax Up( Uown(ooO 2 2*+1), + t.
Rey. = Yep Down(wo. 2*) + oe.
(8)
where e ~ NO,, I), Down(., 2*) and Up(-, 2¹) are downsampling and upsampling 2* times along temporal axis. We derive the start point of current stage from the end point of preceding stage
4

---

Published as a conference paper at ICLR 2026
in Equation to bridge adjacent stages, which is crucial for inference and will be introduced in Section 3.6 However, this design also leads to boundary distribution shift and we cannot directly obtain trainigg target from Equation 7 and Equation 8 Instead, should compute added noise in every stage with boundary condition Uss and -e.- Fortunately, DPM-Solver (Cheng et al. 2022) derives the relationship between any two points, xy and z diffusion ODE path and this relatronship can also be applied to any stage in our method. Accordingly, in stage k. by replacing with &, and Te with Tt, can express intermediate latent Ie as function of fza:
7t Tea
Tt
Rek Vo
Aug
where Ex < < Sks de In and tx is the inverse function of X Equation consists of two components: a deterministic scaling factor, given by _h, and the exponentially weighted integral Yag of the noise (2))),)). If e( (x) is constant in stage k; denoted as the above integral is equivalent to:
It Ess Yock
/ c -d
Ft
Ysk
)up
(10)
7t S.s + yeck
Tt
Jsk
You
Yt
Yek
While enforcing constant value for C at any stage is challenging, aligmment (Li et al. 2024) to constrain its value within a narrow range. In detail, before adding noise to video, pre-determine the target noise distribution for each video by minimizing the aggregate distance between video-noise pairs as shown in Fig. (c) thereby ensuring data-noise alignment and Equation 9 are approximately equivalent to Equation 10 The alignment process can be implemented using Scipy (Pauli et al. 2020) in one line of code as shown in Algorithm
leverage data-noise
Algorithm Data-Noise Alignment
Require: Video batch random noise
1: assign_mat + scipy. optimiiiie wineer..i... .uw..ssumaaaa www.mmsiiimmeetttoooom E))
2: d' c[assign.mat mat]
Output: é'
Our experiments in Section demonstrate that this approximation is valid and does not compromise the model's performance. Notably, the computational cost introduced by alignment is negligible, and we provide the extra time required for alignment in Appendix A.2
Through data-noise alignment, we can apply Equation T0 any point the stage, including the end
point fex- By substituting / ex and = See into Equation 10 through simple transformation, we arrive at the expression for noise Eg of stage k
Rog
6k
del
7eg
GxL
Jek.
(11)
Tea
TeA
Then can easily get any intermediate point 24 in stage by substituting c into Equation TO Consequently, we can compute the corresponding loss using and eg obtained in our method and optimize model parameters in the same way as vanilla diffusion training. Note that the above derivation does not constrain the expressions of % and d,, making our method applicable to different diffusion frameworks. We also note that the direction e points towards the end point of the current stage rather than the final target in vanilla diffusion models as shown in Fig. (b). By reducing the distance between intermediate points and their target points, our method facilitates the training process and further accelerates model convergence.
3.4 PRACTICAL IMPLEMENTATION
In practice, for diffusion framework whose ODE path is curved, like DDIM, we can substitute Tt Vor and 0, vT az into Equation 10 and Equation 11 to obtain sa and eg- For flow
5

---

Published as a conference paper at ICLR 2026
matching, unlike DDIM which requires the prior distribution to be standard Gaussian distribution, can transport any prior distribution to other distributions. Thus, we can model each stage as a complete flow matching process (Yang al. 2024), resulting in simpler expression:
2t (1 t') fe + taa++tigg ttis,
(12)
where t' t-ek_, And The objective of stage k is: 5x
dr: Sss Sea
(13)
dt'
One aspect pyramid flow overlooks is the noise-data aligment, leading to increased variance in the prior distribution, thereby hindering model convergence. Notably, we model each stage as complete DDIM process, the model fails to converge. This is because it is exceedingly challenging for a single model to fit multiple curved ODE trajectories.
In conclusion, visualize the training process of our method in Algorithm 2
Algorithm> 2 Stage-wise Diffusion
Require: Training dataset D, Number of stages K, Diffusion type DDIM or Flow Matching, Model Eg or vp, Create stages {[s,, Ex)
1: repeat
2: Sample Zy ~ D:
3: Sample stage & from (1,...K, then sample timestep & [sk, ex)
4: Sample noise e' E (0, 7) aligned with 20
5: Add e' to zo and get dee. and Ses
6: if Flow Matching then
7:
24 (1 --'t' f') + t' where L-Ek
8:
Ug = EæL Lex
9:
Compute loss: = 1110(zz) - 2/12
else
Acg. tok
11:
CK
eoz oah
ark
an fer + 0.Ek (ec das
dek.
12:
21 ask
13:
Compute loss: |100(x.)) - 1x end if
14:
15:
Update 0 with gradient-based optimizer using Vol
16:
until Convergence
3.5
STAGE-CONSISTENT POSITIONAL. ENCODING
We employ RoPE (S J 2024) as our positional encoding. A direct application of RoPE at each stage would assign new positional indices after upsampling, causing misalignment between stages. In practice, this misalignment restricts the model to generate only small-amplitude motions.
To overcome this limitation, we propose stage-consistent positional encoding, where each frame index is multiplied by a stage-dependent factor to ensure that the same frame across different stages assigned an identical positional encoding. Note that the first frame in every stage is shared so we do not scale it. Formally, let the diffusion process be divided into m stages. For the n-th frame in the i-th stage, the positional encoding is defined as:
(m. = RoPE(n (mm-))) if n (ROPE(n),
(14)
otherwise
where RoPE(-) denotes the rotary positional encoding function. By maintaining consistent positional encoding across stages, our strategy enables the model to capture both large-amplitude motions, significantly enhancing temporal dynamic degree.
3.6 INFERENCE STRATEGY
After training, we can use standard sampling algorithm (Cheng et al. 2022) to solve the reverse ODE in every stage. However, need to carefully handle the stage continuity. Upon completion

---

Published as a conference paper at ICLR 2026
miking
dirugh
night.
paper alggloee gllds
Figure 3: Qualitative comparison. In each pair of videos, the first presents the results of models trained using vanilla diffusion and the second row shows the results of our method. Video pairs the left generated by MiniFlux-vid-ours and the remaining are generated by Animatediff-ours.
of stage, first upsample G₁ in temporal dimension to double its frame rate via interpolation. Subsequently, we scale Up((..) and inject additional random noise to match the distribution of &s during training. Considering the simplest scenario using nearest temporal upsampling and lowering the effect of noise, we can get Ga, using the equation below (see Appendix A.1 for detailed derivations):
V2Te2 Up((en) + V20r n', n' N(O, NO. [1 T/.
Sau
(15)
+ V2
2
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
We implement our method in both DDIM and flow matching. Since most video diffusion models are bulit upon pretrained image models, experiments are based two image models: MiniFlux (Yang cct al. 2024) and SD1.5 (Rombach et al. 202I). These two models are trained under flow matching and DDIM, respectively. We extend MiniFlux to MiniFlux-vid by finetuning all its parameters video data and we follow AnimateDiff (Guo et al. (0244 to extend SD1.5 to video model. Moreover, demonstrate the generalization ability our approach, we finetune Wan (Wan al. 2025) using
our method. Specifically, we apply LoRA (Hu et al. [202I) to the pretrained Wan and perform training with our temporal pyramid method. The number of stages
and each stage is uniformly partitioned in all experiments. Our experiments are conducted on NVIDIA H100 GPU. The detailed training setting is in Appendix A.4
Dataset We adopt Open VIDIM (Kepan et al. 2024 as dataset. The training video resolutions for MiniFlux-vid, AnimateDiff and Wan are 640x384, 512x512 and 832x480 respectively.
Baselines We compare method with video diffusion models of the same network architecture but trained in the vanilla diffusion framework (i.e., pretrained Animatediff and MiniFlux-vid) to demonstrate that our approach does not lead performance degradation. Note that MiniFlux-vid is trained under the flow matching framework without applying the temporal pyramid, using the same initialization and hyperparameters as our method. To evaluate the effectiveness of approach compared to existing methods, we compare our method with ModelScope (Jiuniu et al. 2023). OpenSora v1.3 (OpenSora) (023). Videocrafter2 (Chen al., (203)) and Cog VideoX SB (Zhuoyi et al. 2024). Furthermore, to compare with other selicinncy-enhanciie approaches, we also compare our method with SimDA (Zhen al. 2024).
Evaluation We evaluate our model from two perspectives: generation quality and efficiency. To evaluate the generation quality, we adopt quantitative metrics from VBench (Huang et al. 2024) to compare method's performance with existing models. For efficiency, visualize the convergence curve to intuitively demonstrate training efficiency. detail, to evaluate the convergence speed, we follow common practice (Jiuniu et al. 2023) to validation videos from MSRVTT (Jun al.
7

---

Published as a conference paper at ICLR 2026
Models
AParams.
Tocal
QS
SC
TF
MS
DD
AQ
HA
SR
TS
OC
ModelScope
Juniu et al.
2023
1.78
75.75
78.05
89.87
98.28
95.79
66.39
52.06
92.40
43.95
25.37
26.18
menSora
OpenSorn
2023
1.1B
75.98
80.14
97.27
99. 12.12
99.17
35.28
60.42
58.20
49 04.4
22.47
20.39
Zhuayi
2024
80.91
82.75
96.78
98.88
97.73
59.86
61.98
98.00
69.90
25.38
26.66
1dcocratee22
96.76
99.77
65.65
25.84
25.47
SumLIA
81.63
94.21
98. 78
98.16
24.95
25.42
RepVideo
[S] et 2025
4B
81.94
82.70
78.91
99.16
98.13
57.78
62.40
98.00
74.74
25.31
26.96
Micch
Team
2024
108
80.13
82.64
70.08
99:40
99.02
61.85
56.94
94,60
69.24
23.65
25.15
MM al. 2025
30B
81.83
84,46
71.28
99.40
99.08
53.06
61.23
94.00
71.47
26.01
27.12
NAGF-12-44.
(1l ct. 2025
4.5B
77.92
80.98
65.68
99.71
99.48
20.28
62.42
84.20
57.76
23.03
25.56
(Pegg
2025
11B
81.71
82.10
80.14
99.40
99.49
20.74
64.33
95.40
76.18
25.91
27.57
AnimateDai
Guo et 2024
1.8B
80.27
82.90
95.30
98.75
97.76
54.83
67.16
92.60
34,60
26.03
27.04
AnmaiceDi Ours
1,88
80.76
81.80
96.87
98.49
98.12
58.71
62.19
94.20
44.18
25.14
27.36
MiniFlux-vid- - Vanilla
9B
81.54
82.14
97.12
99.34
98.67
61.29
60.34
96.60
67.49
25.32
27.42
MiniFlux-vid
82.81
25.94
27.94
Wan WWan et al. 2025
1.38
84.26
84.28
96.34
09:49
97.44
37.76
95.20
76.66
24.17
27.07
Wan Ouus
1.38
84.33
83.74
97.86
99.07
92.19
42.13
73.91
95.4
74.47
25.86
27.19
Table 1: Comparison of video generation quality across baselines and method. #Params. indicates the number of total parameters, QS: Quality Score, SC: Subject Consistency, TF: Temporal Flickering. MS: Motion Smoothness, DD: Dynamic Degree, AQ: Aesthetic Quality, HA: Human Action, SR: Spatial Relationship. TS: Temporal Style, OC: Overall Consistency. Because Wan-Ours is finetuned from pretrained Wan model using LoRA, rather than being trained from scratch, therefore excluded from comparisons with other models.
2016) for zero-shot generation evaluation. We systematically compute the FVD (Thomas et al. 2018 value during training and present the FVD-GPU hours curve to demonstrate the training efficiency of method. We also report the average inference time validate the inference efficiency.
4.2 QUANTITATIVE RESULTS
Red. rate
Train Speedup
Infer
Model
Method
Latency(s)
#Stages
Partition
TS (T)
Align
Speedup
Vanilla
20.79
1-1-1
Yes
$0.76
2.16x
1.49%
MiniFlux-vid
1-1-1
No
79.16
1.75x
1.49x
Ours
12.18 (1.71x)
1-1-1-1
Yes
80.14
1.82x
1.65x
5
1-1-1-1-1
Yes
80.03
1.7lx
1.74x
AnimateDiff
Vanilla
6.01
3
2-1-1
Yes
79.82
1.92x
1.62x
Ours
4.04 (1.49x)
1-1-2
Yes 80.94
1.62x
1.36x
3
1-1-1
Yes 80.12
1 1.38x
1.79x
Vanilla
50.52
1-1-1
4-2
Yes 80.65
2.01x
I.54X
Wan
80.18
1.76x
Ours
27.76 (1.82x)
Table 3: Inference efficiency comparison.
Table 2: Ablation study on temporal pyramid design.
Timestep 30.
Tab. [ shows quantitative comparison between our method and baselines. Compared to existing method, our approach achieves improvements in most aspects, particularly temporal-related metrics such as overall consistency (OC) and dynamic degree (DD). This indicates that vanilla video diffusion models contain substantial temporal redundancy, whereas our method effectively eliminates these redundancies and thereby exhibits stronger temporal modeling capability. Figure. (a) and (b) shows that our method achieves speedup of 2x and 2. 23x in training compared to vanilla diffusion models. This acceleration primarily stems from factors:
8

---

Published as a conference paper at ICLR 2026
baseline
3 stages stages
300
PV0
PVD
154
Speed Up
Speed Up
100
126
100
150
206 20
GPU (c) Ablation on the number of stuges
[a] DDIM
(b) Flow Matching
Figure 4: (a) & (b) Convergence curve of vanilla diffusion and our method on (a) DDIM, (b) Flow Matching. We illustrate the FVD of two methods with different GPU hours consumed. Our method achieves higher training efficiency compared to vanilla approachs. (c) Ablation study of stages. Increasing the number stages leads to improved efficiency.
Noise-data alignment: By aligning noise with data, we reduce the randomness in training. The model learns a nearly deterministic ODE path rather than the expectation of multiple intersecting ODE paths.
2) Reduced average sequence length. Since the computational complexity of attention mechanism scales quadratically with sequence length, our method requires significantly less computational complexity on average. For example, to generate a video of length T. the averaged computational cost of attention modules in our method halved, reducing to { + ())² + (G)²) 0.447
compared 2 vanilla diffusion model. This advantage is also reflected in the faster inference speed as shown in Tab.
4.3 QUALITATIVE RESULT
As shown in Fig. we show qualitative comparison between our method and vanilla video diffussin models. The results generated by our method are presented in the second column and the outputs of our baseline are displayed in the first column. Evidently, our approach is able to generate videos with better semantic accuracy and larger motion. For instance, under prompt "A man talking on Mars". the baseline generates person merely shaking their head without speaking, failing to fully adhere to the prompt. In contrast, our approach accurately generates the specified actions, demonstrating superior alignment with the given prompt. Moreover, for AnimateDiff, the baseline generates videos that are nearly static, whereas our approach achieves motion with a more natural and reasonable amplitude. You find more qualitative results in the supplementary material and Appendix A.3
4.4 ABLATION STUDY
We conduct ablation study to assess the contribution of different design choices in our method. Tab, 2 summarizes the results, measured in terms of the total score (TS) from VBench (Huang et al. 2024).
Frame-rate reduction schedule. Beyond using a fixed reduction rate of 2 across all stages, we experiment with non-uniform reduction schedules such 4-2 and 2-4. Here, x-y denotes non- uniform reduction schedule where the frame rate is reduced by factor of from Stage 1-+2 and by factor of y from Stage 2-+3. A more aggressive reduction at early stages (4-2) yields competitive total score (80.65) and achieves the second-highen training speedup (2.01x), suggesting that aggressively shrinking the sequence length early can reduce compute without severely harming the final quality. Conversely, more aggressive reduction at the last stage (2-4) keeps quality relatively stable but trades part of the inference efficiency (1.68x) against slower refinement. Overall, while non-uniform schedules offer flexibility, the uniform schedule remains the most robust across both quality and compute.
Trade-off between training and inference efficiency. A notable trend is that configurations offering the highest training acceleration do not always provide the highest inference acceleration. For example, the 4-stage and 5-stage variants achieve higher inference speedups (1.65x and 1.74x) due to

---

Published as a conference paper at ICLR 2026
shorter effective sequence lengths in later stages but have slower training convergence compared to the default 3-stage setup. Similarly, using a higher reduction rate of 4 achieves the best inference speedup (1.79x) but the cost of lower training efficiency (1.38x) and mildly degraded quality (20. These results again highlight that overly aggressive downsampling helps inference but slows training and harms temporal fidelity.
Importance of late-stage refinement. Configurations allocating more steps to the last stage (1-1-2) yield the highest TS score (80.94), supporting our observation that high-frame-rate refinement is crucial. In contrast, shifting more steps to the earliest stage (2-1-1) harms quality, likely because coarse www-framerate representations alone cannot capture sufficient temporal dynamics.
Effeetiveness of data-noise alignment. Removing our proposed alignment mechanism consistently reduces performance (80.76 79.16), confirming its critical role in stabilizing the hicrarchical denoising trajectory across variable frame rates.
4.5 ZERO-SHOT LONG VIDEO GENERATION
Although our model is trained on fixed-length video clips, we observe that it can generate longer videos in zero-shot manner, without any retraining or architectural modification. We attribute this capability to the design our temporal pyramid: during training, the model naturally encounters multi-frame-rate trajectories, since each stage operates exposure implicitly equips the model with robustness to varying temporal lengths and enables extrapolation to longer sequences at inference time. We provide qualitative examples well as quantitative evaluation on longer video generation in Sec. A.6
different temporal resolution. This
CONCLUSION
In this paper, we propose a general acceleration framework for video diffusion models. We introduce TPDiff, framework that progressively increases the frame rate along the diffusion process. Moreover, we design a dedicated training framework named stage-wise diffusion, which applicable to any form of diffusion. Our experiments demonstrate that our method accelerates both training and inference on different frameworks.
ACKNOWLEDGE
This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award No: MOE-T2EP20124-00122)
ETHICS STATEMENT
This work adheres to the Code of Ethics. Our study does not involve human subjects, sensitive personal data, applications with foresecable harmful consequences. All datasets used are publicly available and described in detail in Section 4 with careful consideration of potential biases and fairness concerns. The proposed methods are intended solely for academic research, and we have taken steps to ensure compliance with ethical research standards and legal requirements.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of work. Section 4 provides detailed descriptions of the model architecture, training objectives, and evaluation protocols. Additional imple- mentation details, hyperparameter settings are included in the Appendix while the supplementary materials contain further results.
REFERENCES
Sand. ai, Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, W. Q. Zhang, Weifeng Luo, Xiaoyang Kang, Yuchen Sun, Yue Cao, Yunpeng
10

---

Published as a conference paper at ICLR 2026
Huang, Yutong Lin, Yuxin Fang, Zewei Tao, Zheng Zhang. Zhongshu Wang, Zixun Liu, Dai Shi, Guoli Su, Hanwen Sun, Hong Pan, Jie Wang, Jiexin Sheng, Min Cui, Min Hu, Ming Yan, Shucheng Yin, Siran Zhang, Tingting Liu, Xianping Yin, Xiaoyu Yang, Xin Song, Xuan Hu, Yankai Zhang, and Yuqiao Li. Magi-l: Autoregressive video generation at scale. 2505. 1, 2025.
AI. httpp:
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arxiv:2300
2023.
Haoxin Chen, Yong Zhang. Xiaodong Cun, Menghan Xia, Xintao Wang. Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. 2024.
Lu Cheng, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 2022.
Feichtenhofer Christoph, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. Proceedings of the IEEE/CVF international conference on computer vision, 2019.
Xiyang Dai Da Zhang and Yuan-Fang Wang. Dynamic temporal pyramid network: A closer look at multi-scale modeling for activity detection. Asian Conference on Computer Vision, 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arxiv:2105.05233 2021.
Tran Du, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotem- poral features with 3d convolutional networks. Proceedings of the IEEE international conference computer vision, 2015.
Wang Yun, Ling Yang. Zhaoyang Huang, Mengdi Wang, and Hongsheng Li. Rectified diffusion: Straightness is not your need in rectified flow. arXiv preprint arXiv:2410.07303 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. 2024.
Yan Hanshu, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint arXiv:2405.07510, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint (xXiv:210600668) 2021.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, and Yuanhan Zhang et al. Vbench: Comprehensive benchmark suite for video generative models. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2024.
Song Jiaming, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Wang Jiuniu, Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang, and Shiwei Zhang. Mod- elscope ext-to-videe technical report. arXiv preprint arXiv:2308.06571 2023.
Xu Jun, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridg- ing video and language. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
11

---

Published as a conference paper at ICLR 2026
Nan Kepan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-Im A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02337, 2024.
KuaiShou. Kling. https://klingai..
2024 2024.
Yiheng Li, Heyang Jiang, Akio Kodaira, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Immiscible diffusion: Accelerating diffusion training with noise assignment. arXiv preprint arXiv:2406. 12303, 2024.
Luma. macahne
https:/ http///lmaaaabbb... ai www.aaaireaammaaaame.ee. 2024 2024.
Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu. Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang. Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang. Qiaohui Chen, Qinglin Hc, Qiuyan Liang, Quan Sun, Ran Sun. Rui Wang, Shaoliang Pang. Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang, Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng. Xiaojia Liu, Xuan Yang, Yaqi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang, Yilei Wang. Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng. Yuxiang Yang, Zecheng Tang. Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Heung- Yeung Shum. and Daxin Jiang. Step-video-12v technical report: The practice, challenges, and future of video foundation model. 2502. 2025.
Siwei Ma, Xinfeng Zhang, Chuanmin Jia, Zhenghui Zhao, Shiqi Wang, and Shanshe Wang. Image and video compression with neural networks: review. IEEE Transactions on Circuits and Systems for Video Technology, 2019.
Stan Gabriela Ben Melech, Diana Wofk, Estelle Aflalo, Yeao-Yen Tseng, Zhipeng Cai, Michael Paulitsch, and Vasudev Lal. Ldm3d-vr: Latent diffusion model for 3d vr. arXiv preprint arXiv:2311.03226, 2023.
OpenAI. Sora. http:s//ppenaal.ccmmnnnnee/r///
2024.
OpenSora. opensora.
Sora 2023.
Virtanen Pauli, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, and al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature Methods, 2020.
Xiangyu Peng. Zangwei Zheng, Chenhui Shen, Tom Young. Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, Yuhui Wang, Anbang Ye, Gang Ren, Qianran Ma. Wanying Liang, Xiang Lian, Xiwen Wu, Yuting Zhong, Zhuangyan Li, Chaoyu Gong, Guojun Lei, Leijun Cheng, Limin Zhang, Minghao Li, Ruijie Zhang, Silan Hu, Shijie Huang, Xiaokang Wang, Yuanheng Zhao, Yuqi Wang. Ziang Wei, and Yang You. Open-sora 2.0: Training commercial-level video generation model in 200k. arxiv preprint arXiv:2503. 096422 2025.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, and et Movie gen: cast of media foundation models. arXiv:2410. 13720, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. 2021.
Chenyang Si, Weichen Fan, Zhengyao Lv. Ziqi Huang. Yu Qiao, and Ziwei Liu. Repvideo: Rethinking cross-layer representation for video generation. 2501.08994, 2025.
Jascha DicksDickt,,in, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arxiv: /503.03583 2015.
12

---

Published as a conference paper at ICLR 2026
Liu Songming, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-lb: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07866 2024.
Lu Y et al. Su J. Ahmed M. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
Genmo Team. Mochi 1.
2024.
Unterthiner Thomas, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: /812.01717 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arxiv:1006.03762, 2017.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui. Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, 7hen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20311 2025.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang. Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang. Yuwei Guo. Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. Lavie: High-quality video generation with cascaded latent diffusion models. 2023.
Kong Weijie, Qi Tian, Zijian Zhang, Zuozhuo Dai Rox Min, Jin Zhou, Jiangfeng Xiong, and et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024.
Peebles William and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi. Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. 2023.
Ma Xin, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03045 2024.
Liu Xingchao, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 0tti::2009000000 2022.
Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Jin Yang, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song. Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954, 2024.
Song Yang, Jascha Sohl-Dickstein. Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2021.
13

---

Published as a conference paper at ICLR 2026
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint 20w.:2309..818.. 2023.
Xing Zhen, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
Yang Zhuoyi, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. and et al. Cogvideox: Texe-tovvideo diffusion models with expert transformer. arXiv preprint arXiv:2408. 06072, 2024.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DERIVATION
This section provides derivation for Equation. 22 Our derivation primarily follows pyramid
flow Yang et al. 2024), and we extend it to the temporal dimension. According to Equation. and Equation.
Dos (( DD(Dnn((To 2 1)) o2
(16)
Up( ~ (Tees Up( Dowen(To, 2*+1))
1), Tex-1
Spatial pyramid has demonstrate that stages can be smoothly connected by renoising the endpoint of the last stage. Renoising process be expressed
Tsu Up(āeup())))))))eee)ee))) +n',, n ~ NN,,
(17)
Teat1
where the rescaling coefficient ex+1 the noise weight. Additionally, we need match the covariance matrices:
allows the means of these distributions to be matched, and is
T8k
E + 2 =
(18)
Tek+1
we consider the simplest interpolation: nearest neighbor temporal upsampling. Then we can get upsampling and noise's covariance matrix Σ' has the same structure as Σ:
Ebjock
1 1
-
-'
(19)
To ensure >' is semidefinite, 7 E [-1,0, Then we solve Equation. and Equation. 1y considering the equality of their diagonal and non-diagonal elements and get the solution:
Ysp VI
Nek-1
(20)
T To V-y 7op
To reduce the affect of noise, let -1 and substitute it into Equation. 20
we can get:
V2
V20, 2
Tea+1
a
(21)
Tsk + /27
We can finally obtain Equation.
22
22
Up((ec) 4+ V20l
n'
(22)
Tsz + V27
where:
1 -1
n' ~ N(O.
(23)
-1
15

---

Published as a conference paper at ICLR 2026
Waves lapping lightly on a beach.
A goldfish swims in small bowl. buby blinking and shifting their gaze slightly.
A dog is wagging tail and walking on green
Black&write, close-up of a person blinking slowly.
Steam from cup of coffee.
Figure 1: Qualitative results. Results in lines 1-3 are generated by MiniFlux-vid and in lines 4-6 are generated by AnimateDiff. You can find more qualitative videos in our supplementary material.
Batch Size
Latency (ms)
5.1
9.6
17.8
Table 1: Computational overhead of data-noise alignment different batch size.
A.2 COMPUTATIONAL OVERHEAD OF DATA- ALIGNMENT
We evaluate the runtime overhead data-noise alignment under different batch sizes. Notably, the alignment is performed CPUs, incurring additional GPU memory consumption. The evaluation results is shown in Tab.
A.3 QUALITATIVE RESULTS
We show more qualitative results in Fig. You can find more qualitative videos in our supplementary material.
A.4 TRAINING SETTING
For Miniflux-vid and Animatediff training, we utilize the AdamW optimizer with a learning rate of 14 and a batch size of 128. Two models are trained for 50k steps using 8 H100 GPUs. For Wan finetuning, we set LoRA rank to 128, using AdamW optimizer with a learning rate of 5e to finetune for 10k steps using H100 GPUS.
16

---

Published as a conference paper at ICLR 2026
bomuler suiling cnuwy
A cal with hvadlpluner slub.
cut la drinking boar. (al Fallure cases
Jolutinn shulf Zecowshot long gencration
Figure 2: Failure cases and zero-shot long video generation result.
A.5 FAILURE CASE STUDY
We present representative failure cases of our method as shown in Fig. (a). the first row, the generated hamster does not fully adhere realistic anatomy, with noticeable artifacts around the paws. In the second row, the cat's head undergoes an abrupt change in size and appearance after turning, indicating instability in identity preservation across viewpoints. Importantly, these issues are not unique to our method. Similar artifacts such as local geometry distortions, identity drift, or physics-inconsistent motion are widely observed in existing video diffusion models.
A.6 ZERO-SHOT LONG VIDEO GENERATION
Although our Animetediff model is trained 16-frame clips and does not incorporate any explicit mechanism for temporal extrapolation, we find that it naturally supports zero-shot long video generation. At inference time, the same model can produce coherent 32-frau 22-frame video clips without any retraining. We attribute this emergent capability to the structure of our temporal pyramid: each stage operates at progressively higher frame rates, effectively exposing the model to variable temporal resolutions during training. As a result, the model learns frame-to-frame dynamics in a way that is not tied fixed sequence length, making it robust extended temporal horizons time. We provide qualitative long video generation result on Fig. (b).
A.7 LIMITATIONS
In this paper, we identify the temporal redundancy video generation and propose pyramid architecture mitigate it. However, prior works (Christoph et al. 2019 Yang et al. 2024) point out that redundancy exists in both temporal and spatial dimensions. This work does not explore how jointly eliminate temporal and spatial redundancy while preserving generation quality, which leave it as future work.
A.8 USE OF LLMS
In our workflow, large language models (LLMss) are employed solely for the purpose of polishing the writing of our manuscripts. Their role is restricted to improving grammar, clarity, and readability, without influencing the scientific content or experimental results. We emphasize that all conceptual ideas, methodological designs, and analyses are entirely developed by the authors. Thus, the LLM serves only as a language refinement tool rather than contributor the research itself.
17


Reviews:
[{"review_id": "3wdoMBUw3S", "reviewer": "Reviewer_d9CS", "summary": "This paper proposes TPDiff, a temporal pyramid video diffusion framework that progressively increases the frame rate during the diffusion process to reduce computational cost. The method divides the diffusion process into multiple stages, with only the final stage operating at full frame rate. A stage-wise training strategy based on partitioned probability flow ODEs and data-noise alignment is introduced to support multi-stage learning. Experiments show that TPDiff achieves 2× faster training and 1.5× faster inference while maintaining or improving video generation quality on benchmarks like VBench.", "questions": "Can the authors provide a theoretical analysis or intuition for why the temporal pyramid does not degrade motion consistency?\n\nPlease include experiments on longer video generation to assess the method’s scalability.\n\nConsider comparing with more recent efficient video diffusion models to better position TPDiff’s contributions.\n\nThe paper would benefit from a discussion on the trade-off between the number of stages and video quality.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Novelty: The temporal pyramid structure is a novel and intuitive way to exploit temporal redundancy in video generation, extending the spatial pyramid idea to the temporal dimension.\n\nEfficiency: The method significantly reduces both training and inference costs without sacrificing quality, making it highly practical for large-scale video generation.\n\nGenerality: The framework is applicable to multiple diffusion forms (DDIM, Flow Matching) and models (AnimateDiff, MiniFlux-vid, Wan), demonstrating strong generalization.", "weaknesses": "Theoretical grounding: While the stage-wise ODE derivation is sound, the justification for why the temporal pyramid does not harm motion modeling is somewhat heuristic.\n\nLimited video length evaluation: The experiments focus on short video clips; it is unclear how the method scales to longer sequences.\n\nComparison with recent methods: Missing comparisons with other efficient video diffusion methods (e.g., LVD, VideoPoet) limits the context of its advantages.\n\nComplexity of implementation: The need for data-noise alignment and stage-consistent positional encoding adds implementation overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SB47ivVdH3", "reviewer": "Reviewer_xG3P", "summary": "The paper proposes TPDiff, a temporal-pyramid video diffusion framework that aims to improve training and inference efficiency for text-to-video diffusion models. The key idea is to progressively increase the frame rate along the reverse diffusion process: early steps operate on a temporally down-sampled video (to avoid processing redundant, low-SNR frames), while only the final stage runs at the full frame rate. To support this, the authors introduce a stage-wise diffusion training scheme that decomposes the overall ODE into several sub-ODEs, and they show how to obtain the intermediate latents and targets for each stage via data–noise alignment so that one single network can handle all stages. The method is instantiated on both DDIM-style diffusion and flow-matching–style video generation, and evaluated on several video backbones (AnimateDiff, MiniFlux-vid, Wan). Experiments show comparable or slightly better VBench scores while reducing training cost (≈2× speedup) and improving inference latency (≈1.5×)", "questions": "1. For videos with large or non-smooth motion (e.g., camera panning, fast human motion), how often does the temporal upsampling between stages fail? Could you provide a metric or user study that shows interpolation does not become the bottleneck?\n2. The authors should explicitly state the coloring rule (per-backbone improvement? column-wise best?\n3. The training illustration in Fig. 4(b) resembles prior “trajectory-aligned” or “consistency/distillation” style diffusion works, where multiple timesteps are forced to follow the same noise / data direction to reduce variance and enable parameter sharing. What is actually specific to this paper is that the alignment is used to support multi-stage, multi-frame-rate video diffusion. I would suggest the authors clarify this connection and better distinguish their alignment from earlier consistency / rectified-flow / progressive-distillation lines.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Video diffusion indeed suffers from temporal redundancy and from the fact that early denoising steps carry very little signal; using a lower frame rate in these steps is a sensible way to cut quadratic attention cost. Aligning video samples to nearby noises reduces variance of the ODE path and seems to be the reason why multi-stage training can converge, which is a neat implementation detail often ignored in similar works", "weaknesses": "The paper is very close in spirit to pyramid flow / spatial-pyramid diffusion: the new part is mainly “do it in temporal dimension + make one model handle all stages + add data–noise alignment”. Some readers may feel this is more an engineering generalization than a fundamentally new generative formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RXCZdVzypW", "reviewer": "Reviewer_61ci", "summary": "This work introduces TPDiff, a temporal-pyramid scheme that progressively increases the frame rate across denoising stages. Early, high-noise steps operate at lower frame rates (exploiting redundancy), while later steps use full frame rate. A stage-wise training framework solves a partitioned probability-flow ODE with data-noise alignment, making the approach applicable to both flow matching and DDIM; a single model handles multiple frame-rate regimes. The latent is written in unified form $x_t = \\gamma_t x_0 + \\sigma_t \\epsilon$ with $\\epsilon \\sim \\mathcal{N}(0,I)$. The result is better training/inference efficiency with competitive quality.", "questions": "1. How sensitive are results to the number of temporal stages and their schedule?\n\n2. Does the single-model design struggle with extreme motion or long-range dependencies?\n\n3. Can the PF-ODE training be combined with distillation from a full-rate teacher?\n\n4. Any failure cases for long videos ($>$16 s) or high resolutions (e.g., $720\\text{p}$, $1080\\text{p}$)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Intuitive curriculum on temporal redundancy with a single model across frame‑rate regimes.\n\n2. Implementation appears simple and potentially compatible with other accelerations.\n\n3. Some metrics on temporal consistency improve, suggesting a useful inductive bias.", "weaknesses": "1. Comparative breadth. More baselines (esp. temporal interpolation modules and caching) would sharpen the empirical case.\n\n2. Temporal fidelity. Limited analysis of temporal artifacts vs. full-rate baselines.\n\n3. Scalability details. Memory/computation profiles for long videos and high resolutions could be elaborated.\n\n4. Ablations: stage count and frame‑rate schedules lack systematic exploration and guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rqwqFvKuKo", "reviewer": "Reviewer_mH91", "summary": "In this paper, the author argues that maintaining full frame rates in high-entropy is unnecessary considering the nature of inter-frame redundancy in video. The author proposes TPDiff framework to facilitate the training and inference efficiency. Specifically, the proposed method divides the diffusion process into several stages and gradually increases the frame rate. To achieve this goal, the author proposes a stage-wise training strategy that leverages ordinary differential equation of diffusion under aligned data and noise. As a result, the training and inference efficiency can be improved and faster. And the proposed method can be generalized to various types of diffusion methods.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-organized and easy to follow.\n* The motivation for leveraging the inter-frame redundancy to increase the training and inference efficiency is intuitive and the proposed solution is effective.\n* The proposed solution is applicable to various diffusion forms such as DDIM and flow matching, demonstrating flexibility and extensibility across major generative modeling approaches.\n* The proposed method achieves better performance compared to the reference work. Specifically, the results show that the proposed method outperforms vanilla diffusion baselines like MiniFlux-vid and AnimateDiff, achieving improvements in motion smoothness, object fidelity, and spatial coherence as measured by VBench metrics and FVD.\n* The authors added a discussion and experiments on the effect of the number of stages. This improves the completeness of the study.\n* The authors added a detailed analysis of computational overhead, demonstrating that the proposed data-noise alignment incurs negligible extra cost.\n* The authors discuss the data selection and generalization. They use the full OpenVID1M dataset and include experiments on Wan to validate generalization across models, avoiding the potential dataset bias.", "weaknesses": "* While the paper introduces the stage-consistent positional encoding to maintain temporal coherence across stages, the experimental validation of this component is limited. There is no dedicated ablation or quantitative evaluation to isolate its contribution, making it unclear how much improvement comes from this module versus other parts of the framework.\n\n* The computational overhead analysis provides CPU latency data, but there is little discussion of GPU memory usage, scalability with video length, or training cost under large-scale settings. A more comprehensive efficiency breakdown would strengthen the claims of computational efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "28EOxAkgEb", "reviewer": "Reviewer_fhFV", "summary": "The paper proposes a temporal pyramid video diffusion model that accelerates both training and inference. The authors finds that consecutive frames are temporally redundant and that early high-noise steps do not need full frame rate, so generating video in a coarse-to-fine temporal pyramid is intuitive. To make this work in a pyramid format, they introduce a stage-wise diffusion objective that treats each temporal stage as its own denoising trajectory, and also align each training clip with a consistent noise sample to stabilize those trajectories. The authors also extend to support DDIM-style sampling. Experiments show up to around 2× faster training convergence and 1.5–1.8× faster inference while maintaining comparable results with baseline methods.", "questions": "1. Not a limitation but a curious question. How robust is the temporal pyramid schedule? Will the inference break if the user chooses more stages (beyond 3) in the inference for better quality?\n2. Do authors observe any semantic or visual artifacts at stage transitions (such as duplicated frames or sudden velocity jumps)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces a novel stage-wise temporal pyramid diffusion process that reduces frame rate in early noisy steps and restores it later. This aligns well with the intuition that video modality is usually quite redundant, especially between consecutive frames.\n2. The proposed stage-wise diffusion objective and sample-noise alignment seem to solve the main problem in pyramid format modeling shown in the experiment section.\n3. The method is shown to work beyond typical flow matching objectives and extend to DDIM-style objectives\n4. Experiments faster training and inference speed while maintaining similar results with baseline methods.", "weaknesses": "1. The approach depends on a fixed multi-stage pyramid schedule (how frames are downsampled/upsampled across stages), and the ablation study in the paper explores limited variations (3-4 stage) of the fixed schedule. \n2. While data–noise alignment improves determinism and stability for training, will this lead to less diversity in generation?\n3. It would be great to include some visual failure cases, especially related to your ablation studies, as quantitative evaluation suites like VBench tend to be biased, and sometimes can not align with visual perception.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C23AdbgDZF", "reviewer": "Reviewer_uQWS", "summary": "This paper proposes a staged frame-rate reduction strategy to accelerate the training of diffusion-based generative models. The method progressively lowers the temporal resolution during training, aiming to reduce computational cost while preserving generation quality.", "questions": "Q1. Could the authors include comparisons with more recent (2024–2025) approaches in video generation acceleration to contextualize their results?\n\nQ2. It would be beneficial to add experiments involving newer baselines such as Wan 2.2 or other contemporary diffusion acceleration models.\n\nOverall, I would be open to increasing my evaluation if these issues are addressed in the rebuttal. However, given the current limitations in experimental scope and currency, my initial assessment remains conservative.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1. The core idea is intuitive and conceptually easy to grasp.\nS2. The empirical results, based on three baselines, demonstrate consistent improvements and validate the effectiveness of the proposed approach.", "weaknesses": "W1. Lack of comprehensive baselines. The experimental comparisons are rather limited. The field of video diffusion acceleration has seen rapid development since 2023, with numerous relevant approaches proposed recently. However, this paper only compares its method against a vanilla diffusion baseline, without evaluating against other contemporary acceleration techniques. As a result, it is difficult to assess the relative advantage or practical relevance of this approach. Readers cannot determine whether the proposed strategy offers clear benefits over more recent methods.\n\nW2. Outdated experimental setup. Most of the compared methods date back to 2024 or earlier, while the video generation domain evolves extremely fast. Many 2023-era methods are no longer representative of the current state of the art. For instance, the authors compare against Open-Sora 1.3 (2023), but Open-Sora 2.0 was released in March 2025—roughly six months before the submission deadline. This omission raises concerns about the timeliness and completeness of the evaluation. Incorporating more recent baselines would provide a fairer and more convincing assessment of the proposed contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)