History-Guided Video Diffusion
Authors: Kiwhan Song
Boyuan Chen
Max Simchowitz
Yilun Du
Russ Tedrake
Vincent Sitzmann
Venue: ICML 2025 poster (2025)
Keywords: diffusion
video
guidance
generative models
3d
Primary area: applications->computer_vision
TL;DR: History Guidance guides video diffusion models with any set of context frames and significantly enhances video quality

Abstract: Classifier-free guidance (CFG) is a key technique for improving conditional generation in diffusion models, enabling more accurate control while enhancing sample quality. It is natural to extend this technique to video diffusion, which generates video conditioned on a variable number of context frames,  collectively referred to as history. However, we find two key challenges to guiding with variable-length history: architectures that only support fixed-size conditioning, and the empirical observation that CFG-style history dropout performs poorly.  To address this, we propose the Diffusion Forcing Transformer (DFoT), a video diffusion architecture and theoretically grounded training objective that jointly enable conditioning on a flexible number of history frames. We then introduce History Guidance, a family of guidance methods uniquely enabled by DFoT. We show that its simplest form, vanilla history guidance, already significantly improves video generation quality and temporal consistency. A more advanced method, history guidance across time and frequency further enhances motion dynamics, enables compositional generalization to out-of-distribution history, and can stably roll out extremely long videos. Project website: [https://boyuan.space/history-guidance](https://boyuan.space/history-guidance)

History-Guided Video Diffusion
Kiwhan Song
Boyuan Chen
Max Simchowitz 2
Yilun Du
Russ Tedrake' Vincent Sitzmann'
Abstract
ables. Yet, another conditioning variable, namely the entire collection of previous video frames, or history, deserves fur- ther exploration. In this paper, we investigate the following question: Can we use different portions of history variable lengths, subsets of frames, and even different image domain frequencies as form of guidance for video generation? Importantly, CFG with flexible history is incompatible with existing diffusion model architectures and the most obvious fix significantly degrades sample quality (see Section 3).
Classifier-free guidance (CFG) is a key technique for improving conditional generation in diffusion models, enabling more accurate control while en- hancing sample quality. is natural extend this technique to video diffusion, which gener- ates video conditioned on variable number of context frames, collectively referred to as history. However, we find two key challenges to guiding with variable-length history: architectures that only support fixed-size conditioning, and the em- pirical observation that CFG-style history dropout performs poorly. To address this, we propose the Diffusion Foreing Transformer (DFoT), video diffusion architecture and theoretically grounded training objective that jointly enable conditioning on a flexible number of history frames. We then introduce History Guidance, a family of guid- ance methods uniquely enabled by DFoT. We show that its simplest form, vanilla history guid- ance, already significantly improves video gener- ation quality and temporal consistency. A more advanced method, history guidance across time and frequency further enhances motion dynamics, enables compositional generalization to out-of- distribution history, and can stably roll out ex- tremely long videos. Project website: https: //boyuan. www.s hpppcc/hhistor--uudddaaa
To address these limitations, we propose the Diffusion Forc- ing Transformer (DFoT), a video diffusion framework that enables flexible conditioning on any portion of the input history. Extending the "noising-as-mast"" masking" paradigm in Diffusion Forcing (Chen et al., 2024) to non-causal trans- formers, DFoT trains video diffusion models by applying independent noise levels to each frame. During sampling, portions of the history can be selectively masked with noise, enabling flexible conditioning and guidance, For instance, in CFG, the unconditional score corresponds to our model with the entire history masked out. Notably, DFoT is com- patible with existing architectures such as DiT (Peebles & Xie, 2023) and U-ViT (Hoogeboom et al., 2023; 2024) and can be efficiently implemented through fine-tuning of pre-trained video diffusion models.
At sampling time, the DFoT facilitates family of history- conditioned guidance methods, collectively referred to as History Guidance (HG). The simplest of these, Vanilla His- tory Guidance (HG-v), uses an arbitrary length of history as the conditioning variable for CFG. Notably, even this simple method significantly enhances video quality. We fur- ther introduce two advanced methods enabled by the DFoT: Temporal History Guidance (HG-t) and Fractional History Guidance (HG-f). These extend history guidance beyond a special case of CFG. Temporal History Guidance combines scores from different history windows. Fractional History Guidance conditions on history windows corrupted by vary- ing levels of noise, effectively acting as a "low-pass filter" on historical frames. With minor modifications, it can also target specific frequency bandwidths enhance the dynamic degree of generated videos (hence the frequency- -bsed ter- minology). Together, we compose HG-t and HG-f to create comprehensive history guidance paradigm, which we term history guidance across time and frequency (HG-tf).
Introduction
Diffusion models are effective generative models in domains such as image, sound, and video. Critical to their success is classifier-free guidance (CFG) (Ho & Salimans, 2022), which trades off between sample quality and diversity by jointly training a conditional and an unconditional diffusion model and combining their score estimates when sampling.
In the realm of video generative models, CFG commonly relies on either text or image prompts as conditioning vari-
Equal contribution 'MIT *Carnegie Mellon University e 'Harvard University. Correspondence to: Kiwhan Song <ki- whan mii, Boyuan Chen <boyuanc mit.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
The Diffusion Forcing Transformer and associated History

---

Guiied Video Diffusion
Given: Given:single image
Diffusion Forcing Transformer and History Guidance: stably generate than 800 frames with consistent transitions
t=
t-16
24
t=
t=
t=48
t-56
kitchen
250 frames later,
exploring a bedroom:
T 3-3
I=316 tm324
t=352
t=340
1-348
f-356
Indoor outdoor 1-460 40,47,00.
450 frames later, exploring outside:
808
Tw816
tw832
tw840
1w848
Tw856
Figure Diffusion Forcing Transformer with history guidance enables stable rollout of extremely long videos. We visualize 21 frames from 862-frame long navigation video generated by our DFoT model from single image in test set video that the model has never seen before. Best viewed as videos on our project website.
Guidance methods dramatically improve the quality and consistency of video generation, enabling the creation of exceptionally long vidcos through autoregressive extension, outperforming the de facto standard DiT diffusion and per- forming on par with industry models trained with an order of magnitude more compute. In Fig. 1, showcase our method by using history guidance across time and frequency with DFoT to generate 862-frame navigation video from a single image-many times longer than prior results and the maximum video length in the training set.
which enables iterative denoising of data point, gradually transforming it from white noise back to a sample from the original distribution. In practice, the score function is often parameterized as an affine function of alternative objectives such as the noise prediction EØ(x*
(x* ,k) E.
Video Diffusion Models (VDMs). VDMs have enabled the generation of realistic, high-resolution videos (Brooks et al., 2024; Yang et al., 2024; Zheng et al., 2024; Kong et al., 2024). Their success largely attributed to advance- ments such as transferring successful image diffusion mod- els (Singer et al., 2022; Guo al., 2023), scaling data and model (Blattmann al., 2023a), improving transformer- based architectures (Peebles & Xie, 2023; Gupta et al., 2024; Jin et al., 2024), and enhancing computational efficiency through multi-stage approaches like latent VDMs (He et al., 2022: Blattmann et al., 2023b; Ma al., 2024; Yin et al., 2024). Many of these models (Blattmann al., 2023a; Yang et al., 2024) focus on generating videos from single first image. In contrast, model is designed to condition arbitrary length histories, crucial capability for autoregres- sively extending newly generated videos.
Our contributions be summarized as follows: We pro- pose the Diffusion Foreing Transformer (DFoT), competi- tive video diffusion framework that enables sampling-tine conditioning using any portion of history, a capability that difficult to achieve with existing models. 2. We introduce History Guidance (HG), a family of history-conditioned guidance methods enabled by DFoT that significantly en- hance sample consistency, motion dynamics, and visual quality in video diffusion. 3. We empirically demonstrate the state-of-the-art performance and new capabilities en- abled by our method, especially in long video generation. Additionally, we provide a theoretical justification of the training objective through a variational lower bound.
Conditional Diffusion Sampling with Guidance. Classifier-fre guidance (CFG) (Ho & Salimans, 2022) is crucial technique for improving sample quality in diffusion models. CFG jointly trains conditional and unconditional models so so(x,c, s V (x*Ic) and se (x, Ø, k) z V log by randomly dropping out the conditioning c. During sampling, the true conditional score V log Pı (x* c) replaced with the weighted score
Preliminaries and Related Work
Diffusion Models. Diffusion models Sohl-Dickstein al., 2015; Ho et al., 2020; Song et al., 2021) define a forward process that transforms a data distribution into white noise via a stochastic process over increasing noise levels & E [o, 1]: x xxx" toRe, where c~N I). The goal of the model to reverse this process by learning to estimate the score function (x x k)) V logg Pkc (x*) (Vincent, 2011),
Plog (x + w [V loggK (x* lo (I)
where w 2 1 the guidance scale that pushes the sample towards the conditioning. VDMs, CFG is predominantly
2

---

History-Cuided Video Diffusion
DIT
DiT
N
Scale
history
brcoda
Patchily
Patchily
FF
Patchify
Patchify
Patchify
Patchify
Patchify
AdaLN
Embed
Embed
Embed
Embeed
Embed
Scale
noise level
Attention
Embed
AdaL.
k-5.5
naisy generation target
Takens
Condition
noise
level
0.0.6
1=1.0
k=0.2
k=0.55
k=0.7
(a) Conventional Video Diffusion
(b) Diffusion Forcing Transformer
Figure 2. Comparison of the conventional conditional video diffusion models and Diffusion Forcing Transformer. At training time, conventional (a) approaches treat history as part of the conditioning input, first encoded by an separate encoder and then injected to the DiT via Adaptive Layer Norm and scaling, The Diffusion Forcing Transformer (b) instead does not distinguish between history and generation target frames. It trains DIT denoise all frames a sequence, where frames have independently varying noise levels.
used for text guidance (Ho et al., 2022b; Wang et al., 2023). For frame conditioning, "first frame" guidance is common- place in image-to-video models (Blattmann et al., 2023a; Yang et al., 2024), or "fixed set of few frames" (Blattmann et al., 2023b; Gupta et al., 2024; Watson et al., 2025), like- wise in multi-view diffusion models (Gao et al., 2024).
Formally, let xy denote a T-frame video clips with indices T {1, 1,,2.... T)- Define H c T as the indices of history frames used for conditioning, and G T H. as the indices of the frames to be generated. Our objective is to model the conditional distribution p(xg (xx) with a diffusion model.
We aim to extend classifier-free guidance (CFG) to this set- ting. Since the history X# serves as conditioning, sampling can be performed by estimating the following score:
Our work generalizes CFG by enabling guidance with a variable number of conditioning frames and later extends beyond the conventional approach of subtracting uncon- ditioned score similar to prior works in compositional gen- erative models (Du & Kaelbling, 2024; Liu et al., 2022; Du et al., 2023), we compose score from multiple conditioning to combine their behaviors. Additionally, we eliminate the reliance on binary-dropout training, the default mechanism for enabling CFG, which we empirically show performs sub-optimally when extended to history guidance.
V log PA (xx) +w V log Pk (xr1xx)
(xx)
V Vlog pa (x+) (2)
This approach differs from conventional CFG in two ways: 1) The generation xg and conditioning history x belong to the same signal xT. differing only in their indices G, H c T; thus, the generated Xy can be reused conditioning X for generating subsequent frames. 2) The history Xw can be any subset of T, allowing its length to vary. Guiding with history, therefore, requires a model that can estimate both conditional and unconditional scores given arbitrary subsets of video frames. Below, we analyze how these differences present challenges for implementation within the current paradigm of video diffusion models (VDMs).
Diffusion Forcing. Traditionally, diffusion models are trained using uniform noise levels across all tokens. Dif- fusion Forcing (DF) (Chen et al., 2024) proposes training sequence diffusion models with independently varied noise levels per frame. Although DF provides theoretical and empirical support for this approach, their work focuses causal, state-space models. CiusVid (Yin et al., 2024) builds on DF by scaling it to a causal transformer, creating an au- toregressive video foundation model. Our work extends the flexibility of DF by developing both the theory and ar- chitecture for non- con-causal, state-free models, enabling new, unexplored capabilities in video generation.
Architectures with fixed-length conditioning, As shown in Figure 2a, DiT (Peebles & Xie, 2023) or U-Net-based diffusion models (Bao et al., 2023; Rombach et al., 2022) typically inject conditioning using AdaLN (Peebles Xie, 2023; Perez et al., 2018) layers or by concatenating the conditioning with noisy input frames along the channel dimension. This design constrains conditioning to a fixed- size vector. While some models adopt sequence encoders for variable-length conditioning (e.g.. for text inputs), these encoders are often pre-trained (Yang et al., 2024) and cannot share parameters with the diffusion model to encode history frames. Consequently, guidance has been limited to fixed- length and generally short history (Blattmann al., 2023a; Xing et al., 2023; Yang al., 2024; Watson et al., 2025).
3 Challenges when Guiding with History
Video diffusion models are conditional diffusion models p(x(c), where x denotes frames be generated, and represents the conditioning (e.g. text prompt, or a few ob- served prior frames). For simplicity, we refer to the latter as history, even when the observed images could be e.g. a subset of keyframes that are spaced across time. Our dis- cussion of e will focus exclusively on history conditioning and exclude text or other forms of conditioning in notation.
Framewise Binary Dropout performs poorly. Classifier- free guidance is typically implemented using a single net- work that jointly represents the conditional and uncondi-
3

---

Husddr Vidco Diffusion
Denoising X
Differently masked History
log
S
Clean
Cleas
Without History Guidance
Noisy Target
log P
S
S
Fully Masked
DFoT
Fully Masked
+
Vanilla History Guidance
log P
S1
S2
S
Subset Masked
Subwel Masked
Temporal History Guidance
V log P
S
+
S
Partially Masked
Partially Mashoul
Fractional History Guidance
Random noise
(a) Noise as masking
(b) Estimating scores using DFoT
(c) Composing scores
Figure 3. Sampling with DFoT and History Guidance. A DFoT can be used to estimate scores conditioned on differently masked histories using noise as masking. This includes clean (full history), fully masked (unconditional) subset masked (shorter history), or partially masked (low-requency history). These scores can be composed when sampling to obtain a family of History Guidance methods.
tional models. These are trained via binary dropout, where the conditioning variable c is randomly masked during train- ing with a certain probability. History guidance can, in principle, be achieved by randomly dropping out subsets of history frames during training. However, our ablations (Sec. 6.2) reveal that this approach performs poorly. We hypothesize that this is due to inefficient token utilization: although the model processes all [T] frames via attention, only a random subset of [G] frames contribute to the loss. This becomes more pronounced as videos grow longer, mak- ing framewise binary dropout a suboptimal choice.
if E H 0
k: if
This formulation treats history and generated frames as parts of the same input to the transformer, rather than separating history as distinct "conditioning" input (see Figure 2 and Section 3). This unification allows any full-sequence trans- former to be fine-tuned into distory-conditiona model with variable-length history, simply by varying the noise levels within each sequence.
Training: Per-frame Independent Noise Levels. As illus- trated in Figure 2b, instead of setting noise levels to zero for all history frames, we adopt per-frame independent noise levels introduced in Diffussion Forcing (Chen et al., 2024). Each frame x, E X is assigned an independent noise level h E [0, 1]. resulting in random sequences of noise lev- els ky in contrast with Equation 3. The DFoT model is then trained to minimize the following noise prediction loss, where ET denotes noise added to all frames:
4 The Diffusion Forcing Transformer
In this section, we introduce the Diffusion Forcing Trans- former (DFoT), simple yet powerful video diffusion frame- work designed to model score functions associated with different portions of history. This includes variable-length histories, arbitrary subsets of frames, and even history pro- cessed at different image-domain frequencies. DFoT im- proves video generation performance base model even without guidance. By addressing the challenges outlined in Section 3, DFoT further enables guidance with flexible history and a more advanced family of history guidance methods described in Section 5.
(4) Crucially, noise levels selected independently for all frames without distinguishing the past and the future. This enables parallel training while also allowing non-causal conditioning on partially masked future frames. In Ap- pendix A.5, we further discuss a simplified objective when ((ax((H4)) << T and a causal adaptation of our model. In Appendix A. we justify this training objective as optimiz- ing (reweighted) valid Evidence Lower Bound (ELBO) on the expected log-likelihoods:
E
[|ET ED (x kT))
for KT.eT
Noise as Masking. The forward diffusion process turns the t-th frame x, of video sequence into noisy frame x at noise levels k E [0, 1]. One can interpret this as progres- sively masking x, with noise (Chen et al., 2024) x! is clean and hence unmasked, x3 is fully masked and contains no information about the original x,. Intermediate noise levels (0- < k < 1) yield a partially masked frame ,t, retaining a noisy snapshot of the original frame's information.
Theorem 4.1 (Informal). The DFoT training objective (Equation (4)) optimizes reweighting of an Evidence Lower Bound (ELBO) on the expected log-likelihoods.
History as noise-free frames. Denoising generated frames x6 conditioned on history X can be unified under the noise- as masking framework. Specifically, this involves denoising the entire sequence of frames xx. U x with noise levels ky [k,, k2, ... kr] defined as:
Compared to conventional video diffusion methods, where a single noise level E [0, 1] is uniformly applied to all generation frames Xg, our approach provides two key bene- fits: (1) token utilization is improved by computing a loss

---

Hisdory-Guided Vidco Diffusion
Table 1. Comparison with generic diffusion models Kineties- 600. "X", " and "v" indicate whether a model can condition on "single "arbbrryy under approximation." or "ar- bitrary" history. DFoT, both trained from scratch and fine-tuned, outperforms all generic diffusion baselines under the same architec- ture and is on par with industry models trained with more compute
History
Generated
GT
"arbitrary
DFoT (acraicá)
resources (see Appendix C.4).
Flexible?
Method
FVD
f0ne (ine-tuned)
DFoT
MAGVIT-v2 (Yu al., 2023h)
4.3g01
Industry size and compute
W.A.I.TT (Gupta et al., 2024)
3.3
Rolling Diffusion (Ruhe et al., 2024)
5.2
Video Diffusion (Ho et al., 2022b)
16.2.103
SD
MAGVIT (Yu al., 2023a)
9.9=03
X
SD
4.8
Same Architecture
BD
FS
+5.5
BD
6.4
DFoT (scratch)
4.3
Figure 4. Qualitative comparison on Kinetics-600. DFoT (both scratch and fine-tuned) generates higher-quality samples consistent with the history than baselines. FS omitted for poor quality. We show of frames; see Figure 14 for more comparisons.
DFoT (fine-toned from FS)
4.7=00
conditioned on all frames xy instead of a smaller subset; second, (2) this objective places variable history lengths "in-distribution" of the training objective, leading to more flexible use of history lengths as detailed below.
significantly improves generation quality and consistency.
History Guidance Across Time and Frequency. While history guidance has been presented as special case of CFG so far, its full potential extends far beyond CFG. Consider the following generalization of Equation 2:
Sampling: Conditioning on Arbitrary History. Un- like standard VDMs that require fixed-length history dur- ing sampling, DFoT allows conditioning on arbitrary his- tory. To generate xg conditioned on Xx at each sampling step with noise level k; we estimate the conditional score V Vlog Pı: (x²) xxx) by feeding the model noisy x5 and clean history frames x Sampling is then performed using stan- dard score-based sampling schemes such as DDPM (Ho et al., 2020) or DDIM (Song et al., 2020). This flexibil- ity in conditioning enables history guidance and its more advanced variants, described in the next section.
V (xg) + w. [V log Px x ( v (x (5)
where the total score is weighted sum of conditional scores, each conditioned possibly different segments of history [H4). and each masked with a possibly different noise level kH, This formulation enables better generalization than a single score function conditioned on a full long history. By composing scores, each individual score component oper- ates on restricted conditional context, reducing the likeli- hood of being Appendix A.3 provides informal mathematical intuition on why summing conditional scores is permissible.
(Du & Kaelbling, 2024).
History Guidance
Leveraging the flexibility of Diffusion Forcing Transformer (DFoT), we introduce History Guidance (HG), a family of techniques for distory-conditional video generation. These methods enhance generation quality, improve motion dy- namies, enable robustness to out-of-distribution (OOD) his- tories, and unlock novel capabilities such compositional video generation. Please refer to Figure 3 for an overview.
Equation 5 effectively allows us to compose the scores con- ditioned on 1) different history subsequences, and 2) history frames that are partially noisy. We refer to these two prin- cipal axes as time and frequency, which together form a 2D plane of options that we refer to History Guidance across Time and Frequency. For simplicity, we introduce composition along these two axes separately.
Simplest HG: Vanilla History Guidance. The simplest form of HG, referred to as Vanilla History Guidance (HG-v), directly performs classifier-fre, guidance (CFG) with cho- sen history length, following Equation 2. The conditional score for any history H. can be computed described in the previous section. To perform CFG, we need to esti the unconditional score V log Pk (xt)) Notably, the uncon- ditional score is a special case of the conditional score with H. = Ø and can be estimated by masking history frames XH with complete noise. Even this simple form of HG
Time Axis: Temporal History Guidance. Due to the curse of dimensionality, the amount of data that we require to guarantee constant data support grows exponentially with the length of history we wish to condition on. As result, history conditioned models are particularly prone to out- of-distribution (OOD) history without an inductive bias of sparse dependency. Common symptoms include blowing up or overfitting to irrelevant features. To address this, we pro- pose Temporal History Guidance (HG-t), which composes
5

---

History-Guided Video Diffusion
300
0.34
Frame-wiie Quality
Dynamic Degree
1..6
0.39
Consiedenegy
0.92
FVD
0.4
11.88
0.90
200
161.2
0.88
0.3
100.4
Betiee
Botier
2
2
Guidance scale
Guidance scale W
Guidancee scale w
Guidance scale
DFoT
DFoT
HG-v
DFOT HG-f
SD
GT
Figure 5. Various metrics as function of guidance scale w for vanilla and fractional history guidance on Kinetics-600, comparing
against 1 w/o HG), SD, and ground truth (GT). FS is omitted for poor performance (FVD 1040). Vanilla history guidance trades
off dynamics diversity for quality consistency. Fractional history guidance better balances these trade-offs, achieving the best FVD.
Higher Guidance Scale
Quality Consistency
(a) Vanilla history guidance significantly improves frame quality and consistency with an increasing guidance scale. We sample with varying guidance scales (top, without history guidance). 1.5 (middle). and (bottom).
Lower Frequency Scale
Dynamies
(b) Fractional history guidance resolves the issue of static videos, improving dynamics by guiding with lower frequuences. We sample with varying frequency scales, with lxy = (top. vanilla guidance leading static videos), 0.3 (middle), and 0.6 (bottom).
Figure Qualitative results vanilla fractional history guidance Kinetics-600, viewed zooed in. Red box history frames.
scores conditioned on different subsequences of history by setting kn. Equation 5. This composition can be per- formed with either: 1) long and short history ("Hongg Hshoon, }. aiming to trade-off between the two imperfect predictive models, reducing the likelihood of OOD while preserving both long and short-term dependencies, or 2) multiple short, overlapping in-distribution histories (Hahwrt, Haturit ' to simulate conditional distribution the full history.
tionally masking history retains only low-frequency infor- mation (Dieleman (2024), Appendix A.2), allowing high- frequency details (e.g., fine textures and fast motions) to remain unconstrained by guidance. This approach makes videos more dynamic while maintaining consistency, which is mainly associated with low-frequency details. Specifi- cally, the HG-f score given by:
v log pa (x (xx) w log (xglx<))) v log Vlggpa(x))) (6)
Frequency Axis: Fractional History Guidance. We ob- serve that major failure mode of HG-v under high guidance scales is the generation of overly static videos with minimal motion. This occurs because HG-v encourages consistency with history, leading to trivial solution of simply copying the most recent history frame. To address this, we pro- pose Fractional History Guidance (HG-f), which guides the sampling process using fractionally masked history. Frac-
where by (0, 1) controls the degree of masking to fo- cus fower-freyuency details, and w is the guidance scale for the partially masked history x ent history frames could contribute information at different frequency bands, such as cent frames and low- www-freuency motion from earlier frames. While a detailed exploration of sophisticated sampling strate- gies is left future work, our experiments show that even
kH In principle, differ-
details from re-
6

---

Hlistoy-GGuided Vidco Diffusion
Sujaly oOD
00B
2000
1500
SSonee
01.4
22001
Truniing Date
000
3401
DR&T HG+
0
Camera Rotation Augle
50
75
2001
125
1.00
History
Ground Truth DFOT + HG-t DFOT + HG-v
SD
Figure 7. Robust performance of temporal history guidance given OOD history unseen in the training data. Left: Baselines sharply lose performance transitioning from jn-distribution, Slightly OOD, to OOD tasks, while DFoT with HG-t shows minimal drop. Right: Baselines produce blurry, inconsistent frames with artifacts on flightly OOD history and unrecognizable frames on OOD history, whereas DFoT with HG-L generates high-quality, accurate samples. Each frame shown is one of four generated; see Figure for full results.
simple implementations of HG-f significantly improve mo- tion dynamics without sacrificing consistency.
maximum sequence length. Flexible-length conditioning is achieved during sampling via history replacement and reconstruction guidance (Ho et al., 2022b).
6 Experiments
Evaluation. To evaluate the overall video generation perfor- mance encompassing quality and diversity, we use Fréchet Video Distance (FVD, Unterthiner et al. (2018)). For more detailed analysis of video quality, we use VBench (Huang et al., 2024), which provides separate scores for different aspects such as frame quality, consistency, and dynamics. For highly deterministic tasks, we evaluate according to Learned Perceptual Image Patch Similarity (LPIPS, Zhang et al. (2018)), computed frame-wise against the ground truth. Additional experimental details are provided in Appendix C.
We empirically evaluate the performance of the Diffusion Forcing Transformer and history guidance. We first validate the DFoT as a generic video model without history guidance (Sec. 6.2), demonstrating the effectiveness of the modified training objective. Next, we examine the effectiveness and additional capabilities of history guidance (Secs. 6.3 and 6.4). Finally, we showcase very long videos generated by DFoT with history guidance (Sec. 6.5).
6.1 Experimental Setup
6.2 Evaluating the Diffusion Forcing Transformer
Datasets. Throughout our experiments, we train and eval- uate a separate DFoT model for each dataset as follows: Kinetics-600 (Kay et al. (2017), 128x128), a standard video prediction benchmark, RealEstate IOK or RE1OK (Zhou et al. (2018). 256x256), dataset of real-world indoor scenes with camera pose annotations, and Minecraft (Yan et al. (2023), 256x256), a dataset of long-contex, Minecraft navigation videos with discrete actions. We employ Fruit Swapping, an imitation learning task adapted from Diffusion Forc- ing (Chen et al., 2024) to test the combined ability to handle long-term memory and reactive behavior with a physical robot. Details are in Appendix C. We use Kineties-600 for benchmarking and quantitative comparisons, and the other three for studying new applications.
We validate DFoT as a competitive video generative model without history guidance by answering the questions:
Q1: How does DFoT compare to the conventional video diffusion approach in standard video benchmarks?
Q2: Does binary dropout diffusion (BD) perform com-
petitively as an alternative training approach that also supports flexible history?
Q3: Is DFoT empirically flexible enough to handle arbi-
trary sets of history frames?
Q4: Can we fine-tine an existing model into DFoT?
We summarize quantitative and qualitative results in Table I and Figure 4 respectively.
Competitive Performance of DFoT (Q1) without Guid- ance. DFoT outperforms all baselines, including single- task standard diffusion (SD), despite SD being optimized for the eval's specific history length. This demonstrates DFoT's flexibility without sacrificing task-specific perfor- mance, aligning with observations from (Chen et al., 2024).
Baselines. 1) Standard Diffusion (SD): A single-task model trained for specific test history lengths following the stan- dard conditional diffusion setup (Gupta et al., 2024; Watson et al., 2025). 2) Choorot Diffusion (BD): An ab- lative baseline trained with framewise binary dropout for history guidance instead of independent per-frame noise levels. Note that BD requires DFoT's architecture as op- posed to conditioning via adaptive LayerNorm to support flexible history lengths, effectively making it an ablation. 3) Full-Sequence Diffusion with Reconstruction Guidance (FS): An unconditional video diffusion model trained with
Limited Performance of Binary Dropout (Q2). While BD enables flexible history conditioning, it suffers a sig- nificant performance drop compared to SD. Notably, BD produces artifacts and inconsistent generations (Figure 4), highlighting its inefficiency as an alternative to DFoT
7

---

History-Cuided Vidco Diffusion
training objective.
Task 1. Robust to
(OOD) History. We evaluate robustness to OOD histories on RealEstate 10K by creating scenarios with extreme camera rotations be- tween history frames and ask the model to interpolate. Base- lines fail to generalize, producing incoherent generations. In contrast, DFoT with temporal history guidance splits OOD histories into shorter, in-distribution subsequences, composing their scores to maintain both local and global dependencies. This enables DFoT to handle OOD histories effectively, as shown in Figure 7.
Flexibility of DFoT (Q3). We demonstrate DFoT"s flexi- bility by tasking it with various video generation tasks on REIOK, such as future prediction, frame interpolation, and mixed history setups. As shown in Figure 11, DFoT gener- ates consistent, high-quality samples across all tasks.
Fine-tune existing models into DFoT (Q4). As discussed in Sec. 4, an DFoT can be obtained by fine-tuning an exist- ing video diffusion model. We fine-tune the full-sequence model on Kinetics-600 into a DFoT using only 12.5% of the training cost. The fine-tuned model surpasses all base- lines and performs comparably to the DFoT trained from scratch (see Appendix D.1 for detailed analysis). This con- firms the feasibility of fine-tuning large foundation models into DFoT to support history guidance.
Task 2. Long Context Generation. Minecraft is a video dataset that requires long context to achieve good FVD scores. We found generating coherent videos with long contexts often leads to OOD histories. Baselines prioritize consistency with the context at the expense of quality. Our hypothesis is that temporal guidance blends scores from long- congecontext and short-context models, balancing memory retention with robustness to OOD. This strategy improves FVD scores from 97.63 to 79.19, achieving long-term coher- ent high-quality generations. See Appendix D.3 for details.
6.3 Improving Video Generation via History Guidance
We examine the effect of history guidance on video quality in terms of frame-wise quality, frame-to-frame consistency and dynamic degree of generated video. We benchmark 64-frame video generation using sliding window rollout Kinetics-600, challenging setup that requires outstanding consistency to avoid blowing up. Note that this is setup where conventional image-to-video models struggle since they can only condition on the final generated frame to extend the video. We present quantitative and quantitative results in Figures 5 and Figure respectively.
Task 3. Long-horizon yet Reactive Imitation Learning. We test on a robotic manipulation task requiring both long- term memory for object rearrangement and short-term reac- tivity for disturbances. Rach data point in the dataset con- tains either of these two behaviors but never both. Baselines fail to integrate the two behaviors, while DFoT combines full-history scores (for memory) with single-frame scores (for reactivity) using temporal history guidance. This al- lows the robot to recover from disturbances and complete tasks, achieving success rate of 83% while baselines fail to perform the task completely. See Appendix D.4 for details.
Vanilla History Guidance We visualize samples gener- ated with vanilla history guidance with increasing guidance scale in Figure 6a. Stronger history guidance consistently improves frame quality and consistency, which is also re- flected in their corresponding VBench scores in Figures 5b and 5c. In Figure 5a, we obtain the best FVD result with small guidance scale of 1=1.5. Beyond that, FVD increases sharply, indicating a loss of diversity with higher guidance scales, similar to the quality-diversity trade-off of CFG.
6.5 Ultra Long Video Generation
In Figure 1, we present a showcase that utilizes all of this paper's contributions - we extend a single image to an 862- frame video in RE10K. Even the most high-performing prior methods can only roll out for dozens of frames under the same setup. This is made possible by enhanced quality, consistency, and rollout stability through history guidance, plus DFoT's flexibility that enables this. See Appendix C.9 for more details and Appendix D.6 for more samples (Fig- ures 8a to 8d), including notable failures of other models.
Fractional History Guidance. Despite notable quality im- provements, we observe that vanilla history guidance tends to generate static videos at high guidance scales (w 2 3), as illustrated in the top rows of Figure 6b, with significantly less motion than ground truth in Figure 5d. Fractional his- tory guidance resolves this in the bydi-b visualization. We find that guiding with lower frequencies (higher kx) consistently increases dynamics while maintaining quality, as shown in Figure 6b. This further lowers the best FVD of vanilla history guidance (181.6) to 170.4, surpassing FS (1040). SD (247.5), and DFoT without guidance (208.0).
7 Conclusion
Enabling flexible conditioning on different portions of his- tory, the Diffusion Forcing Transformer not only establishes itself as a competitive video diffusion framework but also gives rise to History Guidance, family of powerful history- conditioned guidance methods that significantly enhances video quality, consistency, and motion degree. Additionally, we demonstrate that DFoT can be efficiently fine-tuned from existing models, suggesting future potentials of integrating History Guidance with current foundation models.
6.4 New Abilities via Temporal History Guidance
Temporal history guidance brings new capabilities to DFoT, allowing it to solve tasks impossible for previous models. We discuss three representative tasks.
8

---

History-Guided Video Diffusion
Acknowledgements
Chen, B., Monso, D. M., Du, Y., Simchowitz, M., Tedrake, R., and Sitzmann, V. Diffusion forcing: Next-token pre- diction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 2024.
This work was supported by the National Science Founda- tion under Grant No. 2211259, by the Singapore DSTA under 2ST000FC120300823 (New Representations for Vi- sion, 3D Self-Supervised Learning for Label-Efficient Vi- sion). by the Intelligence Advanced Research Projects Ac- tivity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) under 140D0423C0075, by the Amazon Science Hub, and by the MIT-Google Program for Comput- ing Innovation.
Chen, T. On the importance noise scheduling for diffusion models. arXiv preprini arXiv:2301. 10972, 2023.
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241123666. 2023.
Impact Statement
This paper aims to advance the field of video generative modeling. As video generative model, our approach may enable the creation of longer, higher-quality videos, with potential applications in robotics and other fields. However, we acknowledge the potential risks associated with mis- use, such as the generation of harmful unethical content. We emphasize the importance of ethical considerations and responsible use of this work.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780-8794, 2021.
Dieleman, S.
Diffusion is spectral autoregression, 2024. URL / at/eaa///////aee /02/
html.
Du, Y. and Kaelbling, L. Compositional generative model- ing: single model is not all you need. arXiv preprint arXiv:2402.01 103, 2024.
References
Bao, F. Nie, S., Xue, K., Cao, Li, C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669-22679, 2023.
Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Diele- man, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. S. Reduce, reuse, recycle: Composi- tional generation with energy-based diffusion models and meme. In International conference on machine learning, pp. 8489-8510. PMLR, 2023.
Bellec, P. C. Optimal exponential bounds for aggregation of density estimators. Bernoulli, 23(1):219-248, 2017.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15 1217, 2023a.
Gao, R., Holynski, A., Henzler, P., Brussee, A.. Martin- Brualla, R., Srinivasan, P. P., Barron, J. T., and Poole, B. Cat3d: Create anything in 3d with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer 2023b.
Gervet, T., Xian, Z.,, Gkanatsios, N., and Fragkiadaki, K. Act3d: 3d feature field transformers for multi-task robotic manipulation. In Conference on Robot Learning, pp- 3949-3965. PMLR, 2023.
Vision and Pallern Recognition, pp.
Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-iman -mage diffusion models without specific tuning. arXiv preprint arXiv:2307.0472. 2023.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., et al. Video generation models world simulators. OpenAl Blog, 1:8, 2024.
Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Li, F.- F., Essa, I., Jiang, L., and Lezama, J. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, pp. 493-411. Springer, 2024.
Carreira, J. and Zisserman, A. Quo vadis, action recogni- tion? a new model and the kinetics dataset. In proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Chan, S. et al. Tutorial on diffusion models for imaging and vision. Foundations and Trends@ in Computer Graphics and Vision, 16(4):322-47) 2024.
Hang. T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H., Geng, X., and Guo, B. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEF/CVE

---

History-Guided Video Diffusion
international conference on computer vision, pp. 7441- 7451, 2023.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T. and Laine, S. Analyzing and improving the training dynamics of diffusson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
He, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. La- tent video diffusion models for figh-fidelity long video generation. arXiv preprint urXiv:2211.133 13221, 2022.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan. S., Viola, F,, Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv:1005.06950, 2017.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Ho, J. and Salimans, T. Classifier-fff free diffusion guidance. arXiv preprini arXiv:2207. 12598, 2022.
Kingma, D. P. and Gao, R. Understanding the diffusion
objective as weighted integral of elbos. Advances in Neural Information Processing Systems, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-660 116640-6851,,, 2020.
Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al. Hunyuan- video: A systematic framework for large video generative models. arXiv preprini 24xi:2412.036000 2024.
Ho, J., Chan, W. Saharia, C., Whang. J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
Lin, B., Ge, Y., Cheng, X., Li, Z., Zhu, B., Wang, S., He, X., Ye, Y., Yuan, S., Chen, L.,, et al. Open-sora plan: Open- large video generation model. arXiv preprint arXiv:2412.00131, 2024a.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-866 2022b.
Lin, S., Liu, B., Li, J., and Yang, X. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5404-5411, 2024b.
Hoogeboom, E., Heek, J., and Salimans, T. simple diffu- sion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023.
Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. Compositional visual generation with composable dif- fusion models. In European Conference Computer Vision, pp. 423 439.439.000 Springer, 2022.
Hoogeboom, E., Mensink, T., Heek, J.. Lamerigts, K., Gao, R., and Salimans, T. Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion. arXiv preprint arXiv:2410. 19324, 2024.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 1711 1.051001 2017.
Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.-F, Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.0300 2024.
Huang, S., Wang. Z., Li, P., Jia, B., Liu, T., Zhu, Y., Liang, W., and Zhu, S.-C. Diffusion-basee generation, optimiza- tion, and planning in 3d scenes. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750-16761, 2023.
Mildenhall, B.. Srinivasan, P.P Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. Communica- tions of the ACM, 65(1 (511)999-106 2021.
Huang, Z., He, Y., Yu, J., Zhang, F,, Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q. Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 3162-8171. PMLR, 2021.
Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q. Song, Y., Mu, Y., and Lin, Z. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.052 05504444, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195 4205, 2023.
10

---

History-Guided Video Diffusion
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general con- ditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Vincent, P. A connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661- 1674, 2011.
Wang. J., Yuan, H., Chen, D., Zhang, Y., Wang. X., and Zhang, S. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.0633 06571, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8763. PMLR, 2021.
Watson, D., Chan, W., Martin Bruuala,, R., Ho, J., Tagliasac- chi, A., and Norouzi, M. Novel view synthesis with diffusion models. International Conference on Learning Representations, 2023.
Rigollet, P. and Tsybakov, A. B. Linear and convex aggre- gation of density estimators. Mathematical Methods of Statistics, 16:260-280, 2007.
Watson, D., Saxena, S., Li, L.,, Tagliasacchi, A., and Fleet, D. J. Controlling space and time with diffusion models. International Conference on Learning Representations, 2025.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolutinn image synthesis with latent diffiusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 10684-10695, 2022.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. International Conference on Learning Representations, 2024.
Ruhe, D., Heek, J., Salimans, T., and Hoogeboom, E. Rolling diffusion models. In International Conference on
Machine Learning, pp. 42818 428355. PMLR, 2024.
Xing, J., Xia, M., Zhang, Y., Chen, H., Yu, W., Liu, H., Wang. X., Wong, T.-T., and Shan, Y. Dynamicrafter: Ani- mating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12199 2023.
Salimans, and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint 10X1::2202.00512, 2022.
Yan, W., Hafner, D., James, S., and Abbeel, P. Temporally consistent transformers for video generation. In Inter- national Conference on Machine Learning, pp. 39062- 39098. PMLR, 2023.
Shoemake, Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245-254, 1985.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang. S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. Make-a- video: Text-to-video generation without text-video data. arXiv preprint arXiv: 14792, 2022.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an ex- pert transformer. arXiv preprint arXiv:2408.0600) 2024.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In Proceedings of the Interna- tional Conference on Machine Learning (ICML), 2015.
Yin, T., Zhang, Q., Zhang, R., Freeman, W. T., Durand, F., Shechtman, E., and Huang, X. From slow bidirec- tional to fast causal video generators. arXiv preprint arXiv:241..07772, 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
2023a.
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., et al. Language model beats diffussion key to visual generation. arXiv preprint arX0v:2310.05737, 2023b.
Su. J., Lu. Y., Pan, S., Murtadha, A.,
Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 18: 18/1.//7 2018.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference

---

History-Cuided Video Diffusion
on computer vision and pattern recognition, pp. 586-595, 2018.
Zheng, Z., Peng. X., Yang. T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratiz- ing efficient video production for all. arXiv preprint arXiv:2412.20404, 2024.
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., and Snavely, N. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint NEt5:1800009811 2018.
12

---

Hisddry-Cuided Video Diffusion
A Proofs, Explanations, and Extensions
A.1 Derivation of an ELBO
This section includes derivation of an ELBO corresponding to the DFoT training objective. By taking a sequence modeling perspective, the derivation below streamlines that of the Diffusion Forcing ELBO in (Chen et al., 2024).
Let T denote the index set associated with sequence so that xy (x))ET is the whole sequence. We the notation k EET for the sequence of noise levels. A path p is sequence of noising steps that transition from an unnoised sequence to a noised one. Specifically,
Definition A.1 (Path). We define path as a sequence Jusssn that begins at zero noise k2 (0.0....,0) 0), and terminates at full noise kN = (K, K K....,K) K).
Given a path P. let x = x k denote the sequence with (x IEET. Note that there is nothing intrinsically causal or temporal about the indices f; indeed, we can define noising paths on other objects like trees or graphs. Examples of paths include:
Autoregressive diffusion, where ki is equal to K i S Lj/K], equal to 0 if L > Lj/K] + and equal to j K [i otherwise. This path looks like (O.....0), 0), lexicographically.
O)..., (K, 0, .), 1K,1,0,... .,). 0), increasing
Full-sequence diffusion, where k/ j and N = K; i.c. all points are denoised together.
We can accomodate skips in noiseless, e.g. DDIM, or paths with linearly increasing noise, such as those considered in (Chen et al., 2024).
Typically, we assume that Kf is non-decreasing in j (the noise level is monotonic up to k. = (K)...,K))
The essential property that we require is that our learned model and forward process factor nicely along such paths. is straightforward check that this is indeed the case for Diffusion Foreing Transformer with these monotonic paths:
say that a model pa and forward process q factor along a path p if for any path, = (k2 "k2,...,kk,, ) be path, <x*k"iv xk)) factors as q(xk x²) = IT ((**' 2 xk²²)) and po factors as (x = 11 j=1 Po (xk=- xkk (xck)) ), with po pxc*") not depending on
Definition A.2 (Factoring Property).
When the model factors along paths, general ELBO holds. We first state the general form, then specialize to Diffusion via Gaussian forward processes, and conclude with the proof of the general result.
Theorem A.3. Suppose that (po, ( factor along path (k).... ik.,....k'' ), Then, for some constant C not depending on have
In lnp( 2C C ExRto I (oto)
In Ix
|xk')
Σ
Dxu (polxk x((+)) ll g(xck' | xtt)))
(7)
Consequently, BRIINND, denotes an expectation over paths p (k) ..... kk along which (po, 9) factor, then
Inp(x")) 2 C BRINNDD Exii Ng(x (oko) In px
(oo
x x
g(xk)
|xkit xx0
We now specialize Theorem A.3 to Gaussian diffusion. For now, we focus on the "x-prediction" formulation of diffusion. The "c-prediction", used throughout the main body of the text and the "v-prediction formalism, which is the one used in implementation, can be derived similarly (see Section of (Chan al., 2024) for clean exposition). The following theorem is derived directly by applying standard likelihood and KL-divergence computations for the DDPM (Ho et al., 2020; Chan et al., 2024) to Theorem A.3.
For simplicity, we focus on paths with a single increment (e.g. DDPM), but extending to jumps (e.g. DDIM) is straightforward (albeit more notationally burdensome).
13

---

History-Guided Video Diffusion
Corollary A.4. Consider only paths p for which kj k-- entrywise, and for any and j for which kig k!_
1/= kg increments by one.
g(x k/t
II N(x i 1- 3xx " 12;11)
(8)
t:kg< <
and define ork = (1- ā& = IT aj. Suppose that we parameterize po po(x*', :xk't1 .k) = -N(uo(x ; where furthe,
:x , 0,)
x k)) (1 a xky (1 (1-ag)vā-- Xo x k)), of == (1 - -(oj)))
px k)
- Cij
(zj
Further, let (x)xx' ;x k)) = x0(x x x k) denote the I-block component of X2 xx(xx', :xx²= k)), and suppose
that if =kद ktt1,, then (x :xk' k)) -xe" (i.e., no denoising occurs, we do not re-predict the denoising). Then,
for some distribution Do over paths p along which (po: 9) satisfy the requisite factoring property, and for some constant C
independent of po.
N Σ
Inpo 10 11 C E
Ko.N B, EpRiT
E
9y G (xx)) ;x :x 1k)) x, 4i 1|?
(a-- (1=
where above, we define
Proof of Corollary A.4. The first inequality follows from the standard computations for the "x-prediction" formulation of Diffusion (see Section 2.7 of (Chan et al., 2024) and references therein).
Remark A.5 (Factoring). Observe that forward process in Equation (8) naturally factorizes across all the paths considered in Corollary A.4. While pe (by definition) factors across any single path p. these factorizations may be inconsistent across paths. Enforcing some explicit consistency remains open for future work.
Proof of Theorem A.3. The first step is the standard ELBO trick:
pxx(x"") = In /
XAII p(x p(x koN
)d
In ExaliN moxttei bewyy (f (ee xeko
2 E B vixkki lo In Pixckelioo
(xckii x
where the last step follows from Jensen's inequality. We now expand
In Po N (xckiN x
+ In Po(x
x(k)
po I (x
= =Inpo(x
((ck:
+
(ikii xxki x2e)
(Factoring, Definition A.2)
4
=nnpo ) + In px
(xk)
po(x |
In 8(ck) xte) (xeki- xxo
In
(Bayes' Rule on g)
9xck1
xxko)
(xxi) ski+i xtoo
=Inpo(x In px( (xk)
+ In g(xk
xX()
Σ In po(x x
N-1
(Telescoping)
g(ck)
(xkx x
gxcks selist xoko
=n (x(x
N
In Do Pxc I skiti
g(xkiv x
+ In Ixpo
|xk| +
g(cko xolo++ sekey
(Canceling)
1=1
14

---

History-Guided Video Diffusion
We observe that in px(x and In g(>ck* ( do depend on 0 (recall pe pe (x* kM
is the distribution over noise), so taking
an expectation over the q(-), regard these as a constant C. This yields
N-1
Po(x [x((li In
Inp(x*)) ) 12C+ +.Euu ExILN ma(xctiin boco)
po 1x(()
g(xki x xxc))
N-1
C + E wo(xkin (xk)
In In
(xt') +
Σ
D Dxt. (po |x((t) q(xck)
bektt xx4)) })
A.2 Understanding Frequency Guidance
For simplicity, we focus on 1-dimensional discrete signals with even dimension d, but extending to 2-dimensions is straightforward. We provide a simple mathematical explanation that "noising" feature corresponds to a form of low-pass filtering.
Specifically, we consider a regression setting with features x E Fd and targets y E R. We now study the conditional distribution of y Xa, where x, study the conditional distribution of the Fourier transform of y given a measurement of X We assume that the entries of x can be interpreted as entries in sequence and we interpret this conditional distribution as function of the Fourier transformation, Fx),), of x. Similarly, we define Fm (y). For simplicity, we focus on a 1-d Fourier transform, but analogous statements hold for 2-d features x (e.g. 2-d frames in a video).
is noisy measurement of x. To understand effects in the frequency domain,
We begin by recalling the Fourier transform of vector.
Definition A.6. Let Fa Rd + Rd denote the (real) discrete Fourier transform, specified by
sin (ik://m
k d/2 d/2 < k =
Fa(x)(()=
(9)
x cos(ik/2m))
We note that, by Parseval's theorem, Fa is an isometry:
2 [ = |
(10)
Because Fa is bijective linear mapping, we identify it with an invertible matrix in Rdxd We now characterize the conditional of (m) Fa(x).
Proposition A.7. Let x NO, ΣE), and y X~~ N(Ax, Eg Define x, = + oz, where ~ N(00 I) is independent of y. Define = AFAFF Σg == FIE and S(o) = (Ex + ddo
and E, = Fm Then,
~ ~ N E
Fmlly) Fa(x) N(FF((()) Ex)
The distribution of Fyl))) Xo (or Fmly)) Fa(xx))
N(AS(o)F((xx)) E. A
(11)
Proof. Set & = Fa(x) and & = Faly). As Fa, Fm are linear, we see that & ~ N' E2) and y = N(AF(((x), Ey)
N(FmA(x),
For the last statement, we have that Fa(xx) = x + oF(z)). As 1 Fa is an isometry (i.e orthogonal), we have [E(F I. Thus, oF((z) = odd2, where à ~ Ao Ia) is independent of x,ý. We may now invoke Lemma A.8 to show that Equation (11) describes the distribution of Fmly) Fa(x,). As Fa is a bijection, conditioning on Fa(x,) and x, is equivalent.
15

---

History-Cuided Video Diffusion
Interpretation in Terms for Frequency Attenuation: It is common that natural signals exhibit power-law decay in the frequency domain. As illustration, consider Σn = CDiag( (1= that is, in the Fourier domain, is independent across frequencies and exhibits a power-law decay with exponent a. Then, S(o) is diagonal, and
{
( ) or o² S Ci /d
S(o)u = 1 do?in ~ i- i2 do 1 or 2 Ci /d
also exhibits power law decay. Hence, when conditioning on X the shrinkage operator E(r) attenuates the contribution of the i-th frequency of x, in proportion i- for i-large. Moreover, as a becomes larger, more frequencies are attenuated. In other words, conditioning on noisier examples leads to more aggressive attenuation.
Importantly, there is no intrinsic bias of Gaussian noising towards preferring lower frequencies. Rather, noising serves to regularize away weaker frequencies. For natural images, this corresponds to high frequencies, but may not in other application domains.
Lemma A.8 (Gaussian Conditional Computation). Let x ~ 2) and y x ~ (Ax,, E Define Xa x + oz, where z (O, independent of y. Set S(o) == (E + o²I) 1 Then, the distribution of y Xg N(AS(o))x, By A
Proof. First, we observe that (Xx;)) y) are jointly Gaussian random variables with mean zero We set Σn [ o²I + E,, and Σ1 1By² = n + AE,A Moreover, Σ12 == B[yx,] = AE,, Hence, from the standard formula for Gaussian conditional distributions, we have
y/x, ~ (E12 E11 - En2 E12)
= -AE Eg + -00 Ey AE, AE, -
We may then simplify AT (E, EAA = ACE, - E, ol) - Note that (E, Σ, [E, + 1211)~1,,, (E,- (E, + r² -E + = E' + + Define S(o) = E. (E + 10) We conclude that
y x, ~ MOO~~N((S((),,) E, A
(12)
A.3 Maximum Likelihood Interpretation for Score Addition.
The Diffusion Forcing Transformer achieves history guidance across time and frequency by sampling with linearly weighted diffusion scores conditioned on different history lengths. Though this appears to be purely heuristic, as in classifier-fres guidance, we provide meaningful probabilistic interpretation of the algorithm.
Intuition for guidance via Gaussian MLE, We begin by justifying linearly combining scores in simple Gaussian models. For now, let us assume that the goal is sample q (x)), and the aim to estimate the score s* (xx V In (xx).
We make strong assumption that we have N estimators for the score functions, (x)))) and that errors are Gaussian. Assumption A.9 (Gaussian Errors). We assume that, conditioned on x, the errors E := (x) s 82 s*(xx..... 8n (x) - s* (x)) form a Gaussian vector with mean zero and covariance E(x) E Rddxdd.
Though the assumption is clearly not true in practice, it helps build intuition for the idea. Moreover, given that the reverse process of SDE essentially involves Gaussian predictions, it is plausible to expect that the individual steps of the denoising process model Gaussian distributions, and consequently, errors are "Gaussian-like" (Huang et al., 2023).
Let us now consider the maximum likelihood score estimator in this model. We introduce the notation
I = T Ixxa ... Tixal"
(13)
In this case, we have
(x) (x), (2.))....,
..., sxx)))))))))))) x ~~~ NN(sssss E()).
(14)
16

---

History-Guided Video Diffusion
Let us now characterize the maximum likelihood estimator, gALLE This solves
g (x) = argmax,(x) (1.m (x);
arg max s(x)
cxp - [(x) ((xx)
$1:n - -[s(x))
((
min T -
(é = S1m - -Is(
=(x) 2
arg min(Sx (x) Iss E(())))><(>>>>>>>)) xxx
E(x) ((w((x) [x)).
An exercise in Calculus reveals that
MLLE x (xxx)>==T(>>))) (IT=(x)-11) [T=)())~)) (x.n
(15)
In other words, SMLE is some (x-dependent) linear function of $1:n-
We now describe a couple of special cases:
Case 1: d (x is scalar) scores are independent. this case, Σ(x) has a diagonal inverse, and by positive definiteness, its entries are strictly positive. Thus, letting og denote the diagonal entries of [xx)=, we have 1 [xx) is vector with strictly positive entries (a1 (x). an (x)), and -(x)= Ex= (x) is their sum.
In this case,
g SMLE(x)
Σ E, ag (x))
02
(x(x)
(16)
is convex combination of the various scores.
Case 2: general d (x is scalar) scores are independent, and the errors 8; - s have scaled identity covariance. In this
case, E(x) is block diagonal with scaled-indenity blocks, so we can also show
(x(x) (i(x), (x(x))
MLB (x)
(17)
where o;" are the scalings of the identity blocks.
Now we can examine the specific case of history guidance. Let the pieces of evidences be the n different history segments of different lengths that our model condition on. Diffusion Forcing Transformer is essentially trying to combine these evidences with Maximum A Posteriori (MAP) get overall estimation of the score of future tokens.
Why MLE/ Averaging Works in General? Though the averages derived above hold for Gaussian case, there is very general theory for combining multiple estimators into one called Optimal Aggregation of Estimators (see, e.g. (Rigollet Tsybakov, 2007)). In this case, even beyond Gaussian settings, there are known benefits to optimizing over the convex hull of family of estimators rather than choosing the best single one (see, e.g. (Bellec, (017)). Another rational for combining estimators is that an average of estimators can do better than the best single estimator. Indeed, suppose that you have maps &; X' + [0, 1), and assume that the optimal value (for simplicity) is s (x) 0 (also, scalar for simplicity). Suppose you partition the space into components Xi..... Xn such that
1 nn Pr(x E x|=
{1
x X otherwise
3x(x)
(18)
For any estimator, the expected square error is then
B((())
P(x X²]
(19)
17

---

History-Cuided Video Diffusion
Algorithm 1 Flexible Sampling with DFoT and (optionally) History Guidance
Task: specified by indices H. 9 T H, and history frames Xx.
Input: diffusion process defined by ouk; dk. ok. diffusion sampler S with sampling steps N,
DFoT model So (,)),,, and History Guidance scheme specified by {(Hi, Kit,, wi) wi))i= Sample Xg ~ (N,0 I), then X7 < Xx C Xg
b Sample random noise for generation frames
for n = N,N 1, 1 do
ki N
if E G if E H
kT + (K=) where
ke- =1 1
&T + xT. then replace Xx +- where I0. s sa (RT, KT) for = I.....Iaa do
Fully mask history Estimate unconditional score
ke a
ift 9
by +- (ke) where
k KH: if H. if E H. \ Hi
ke 1
&H. + aks, &H, + Onn, where e <~ 1), K40,2 <- e where e ~ 1))
8T + Xx, then replace
Mask history based on Hi and kw,
3 + so(xT, KT)
Estimate i-th conditional score
end for
$ + 8
+ El
wi (s'
65'-8% -8a)
b. Compose scores
Xg S(x
sgi n N )=1)
Denoise k: N N
a-1
end for Output: Xe
However, for any x. Σ 3,(x) n
1 Σ. I(x X;) 1. Thus.
27
= P(x E Xz} = n
(20)
Because estimators make errors on complementary regions of state space, they work in concert to cancel out errors to reduce overall error.
We suspect history guidance functions in similar fashion: though attending to different history contexts may result in errors for different realizations of past frames, but by averaging all these effects out, we ameliorate total error.
A.4 Sampling with DFoT and History Guidance
DFoT is capable of flexible sampling conditioning on arbitrary history, and is further capable of performing history guidance, a family of guidance methods we propose. In Algorithm 1, we provide a detailed sampling procedure for DFoT and history guidance, where any score-base based sampler such as DDPM (Ho et al., 2020) or DDIM (Song et al., 2020) can be used for S. Importantly, when estimating a score conditioned on a masked history, it is crucial to pass the corresponding noise levels ky and to replace the clean history frames with noisy frames, which are created by diffusing the clean history to the noise levels. This ensures that the model input is consistent with what encounters during training time. Note that Algorithm can be applied given arbitrary history frames. For instance, to extrapolate the history of length to T frames, set H. {l..... - and g = {r + ,,....T)): to interpolate between two frames, set H. = (1,7) and g = (,..... ... 1}. Below we provide several representative examples of how the algorithm is applied:
Conditional Sampling without History Guidance: {(Hs, kH, wi 11 ((H, 1)}
Vanilla History Guidance with a guidance scale w > 1: ((H,, kn, wi) = (HH,0,w)
Temporal History Guidance with / subsequences (H: and guidance scales {wileen {(Hi, kH. wi) [(H,, W
Fractional History Guidance with a guidance scale w and fractional masking level ky: (HH wi)l {(HL, 0, 1), (HL kn. - 1)}
18

---

History-Cuided Vidco Diffusion
A.5 Simplifying Training Objective
Diffusion Forcing (Chen et al., 2024) proposes to train the entire sequence with independent noise per frame. A natural question to ask is whether this mixed objective includes too many tasks compared to what one actually needs. Here we provide some insights from our experiments throughout the project: When the number of frames is small e.g. 10 latent frames, there is no noticeable decrease in training efficiency- Diffusion Forcing seems to converge as fast as standard diffusion from both training and validation curves. However, when we grow the number of latent frames to 50, we start to witness decreased performance at sampling time. While we firmly believe that binary dropout is not the ideal way to achieve objective reduction from our experiments, we believe that one can easily reduce our training objective by only applying independent noise up to the maximum training length one wants to support. In particular, if one wants generate the next 10 frames from previous 1 10 frames, it doesn't 't seem necessary for frame 11 to be independently masked noise from time to time, since we will never need to mask it out for flexible conditioning. In addition, one may want to consider treating the number of history frames as random variable at training time, sampling length first and then applying uniform levels of masking to the history, though independent from the noise level of the generation target. We didn't investigate these simplifications in detail because we simply find Diffusion Forcing's 's training objective very versatile for many of the tasks we want to do, e.g. interpolation, and varying noise level sampling. However, we do believe that these schemes could worth more exploration if one to scale up our method to a much bigger number of context frames.
A.6 Causal Variant
In principle, one can implement DFoT and History Guidance with a causal transformer as well. For example, Caus (Yin et al., 2024) has proved the effectiveness of Diffusion Forcing on fast causal video synthesis and doesn't 't conflict with History Guidance. However, we'd like to highlight that one can also use our non-causal DFoT to achieve causal sampling. Different from traditional transformer- based models, DFoT doesn't 't need to enforce an attention mask to achieve causality. Instead, at generation time, one can mask out the future with white noise to prevent any information from the future from leaking into the neural network. In fact, there might be use cases when one may want some low- owwfrequency information from the future, and then one can fractionally mask out the future via noise as masking to achieve so. On the other hand, the motivation behind causal video diffusion models is often speed and real-time generation using KV caching. In that case, one either needs to train a causal DFoT directly or consult advanced techniques like attention sink (Xiao et al., 2024) to perform windowed attention effectively.
A.7 Incorporating Other Conditioning
Throughout our discussions in the main paper, conditioning is history exclusively, What if one wants to integrate the Diffusion Foreing Transformer into a text-conditioned diffusion model? One claim of the DFoT is that it doesn't require architectural changes so one can fine-tune an existing model into a DFoT model. This is still the case here: if one already has text-conditioned video diffusion model, presumably built to accept such conditioning via adaptive layer norm, one simply take DFoT as an add on to their existing architecture to obtain a DFoT model that accepts both text and history as conditioning. DFoT's 's Figure 2 does not assert that one cannot use an external AdaLN layer with DFoT, but is rather saying no architectural changes is needed.
A.8 Extended Temporal History Guidance
Temporal history guidance addresses the challenge of out-o--distribution (OOD) history by composing scores conditioned on different, shorter history subsequences, which are closer to being in-distribution. However, since the model receives the entire video sequence as input during innnllnggninluudnng both the history and the noisy frames being generated OOD problem can arise throughout the entire video sequence, not just in the history portion. To mitigate this, we propose further decomposing the generation G into generation subsequences G1, G2...., .g C G. In line with the original temporal history guidance, the history H is already decomposed into history subsequences Hi, Hi,H,,,... H2,..., scores conditioned on even shorter, and thus more in-distribution, subsequences in {Ht) x {G Specifically, the composed score is given by:
Hy C H. This allows us to compose
r 1=
V log Pl (X,,, (x.)
(21)
where b denotes frame-wise averaging operation. We refer to this method as Extended Temporal History Guidance, it extends the concept of temporal history guidance by composing both history and generation subsequences. Empirically, we find this method to be more effective than the original temporal history guidance when the video sequence is clearly OOD (e.g., RealEstate 10K OOD history experiment), and thus requires shorter subsequences to be in-distribution.
19

---

Ultra Long Video Generation RealEstatel0K, with HG-v and HG-f
Supplementary Visuals
Before delving into further details, we list extensive figures (Figures 14) that supplement the main paper's content. Detailed descriptions for these figures can be found in Appendix D.
80
t
160
=
240
t=
320
t
t 560
t 640
1= 720
800
(a) Long navigation video generated by DFoT with HG. # frames = 862.
20

---

Ultra Long Video Generation RealEstate 10K, with HG-v and HG-f
80
3
160
1
240
t
t= 320
400
t 480
640
3
1 = 720
t= 800
880
(b) Long navigation video generated by DFoT with HG. # frames = 917.
21

---

Ultra Long Video Generation RealEstate 10K,, with HG-v and HG-f
t= 80
240 t= 160
t =
(c) Long navigation video generated by DFoT with HG. # frames 442.
t
=
240
320
400
(d) Long navigation video generated by DFoT with HG. # frames = 442.
Figure 8. Long navigation videos generated by DFoT with HG-v and HG-f, from single history frame RealEstatel0K, We subsample with a stride of frames for visualization. The videos exhibit consistent transitions navigating while through diverse indoor and outdoor scenes, maintaining high stability over hundreds of frames. This is enabled the improved sample quality and consistency from HG, along with DFoT's flexibility that allows both interpolation and extrapolation.
22

---

DFoT vs. SD: Long Rollout Comparison RealEstate 100,, with HG-v and HG-f
DFoT rolls out further
1=0
t=8 8
t= 16 = 24 = 32
t= 40
E 48
t 56
64
t 72
DFOT HG
TUT
run
13
1050
SD Blows up
1=8
16
24
32
48
t=
SD
Figure 9, Qualitative comparison of DFoT with HG vs. SD on long video generation. Given a single history frame we task both models generate videos of moving straight ahead and visualize them with stride 8 frames. While quickly diverges after 30 frames, DFoT with HG maintains high stability until f = 72 and can roll out further.
23

---

Robustness to Out-of-Distribution History RealEstatelOK, with HG-L
Siicklly OOD History
Generated
Ground Truth
DFoT + HG-t
DFol + HG-v
SD
FS
Ground Truth
DFOT + HG-H
DFoT + HG-V
SD
FS
(a) Given Slightly OOD history with rotation angles [120", 130%). baselines and DFoT with HG-v generate inconsistent frames with artifacts. In contrast, DFoT with HG-t generates consistent videos that highly resemble the ground truth. This the region where HG-t starts showing its generalization with other methods.
24

---

Robustness to Out-of-Distribution History RealEstatelOK, with HG-L
ooD History
Generated
Ground Truth
DFol + HG-H
DFoT + HG-v
SD
FS
Ground Truth
DFOT + HG-t
DFOT + HG-V
SD
FS
(b) Given DOD history, all baselines completely fail yet DFOT with HG-1 still manages to generate high-quality, accurate videos.
Figure 10. Qualitative results of testing robustness to nut-of-distributaa history RealEstatc10K. We provide wide-angle, 4-frame history and models to generate the next 4 frames that interpolate between the history frames. the angle increases, the history becomes more out-of distribbuui,,, and thus we split the results into Slightly OOD and DOD depending on the angle range.
25

---

DFoT's Flexible Sampling on RealEstatel0K, without HG
SY
TIM
Figure 11. An illustration of the empirical flexibility of DFoT, showing ten samples from RealEstate 10K, where single DFoT model infills the missing frames given different history. DFoT successfully generates consistent samples across ten diverse tasks, each varying in the history length from to 6 frames and different timestamps.
26

---

Improved Video Generation on RealEstate with HG-v
Extrapolation
Hightn Guidance Scale
Quality Consistency
Irvilli
Interpolation
Figure 12. Improved video generation quality with vanilla history guidance RealEstate for both extrapolation and interpola- tion tasks. HG-v, with an increasing guidance scale, enhances fidelity and consistency while effectively removing artifacts. Videos are sampled conditioned on two history frames with varying guidance scales 1 (top. without HG-v), (middle), and 3 (bottom). Zoom into the boxed regions) to see notable differences.
27

---

Long Context Generation Minecraft, with HG-L
12
t-63
t=66
68
t=
t=
90
t
t=90
1-99
1-102
t= 05
t- 108
111
t= 111
1=112
t 120
123
t- 1=126
t- 129
32
35
t- 38
1- 141
t- 144
142
(a) Composed Guidance (short + long history) without CFG
1=fl
1=1
I-F
1-5
1-
18
21
1=61
1m66
72
75
78
11
84
1-87
90
1-96
99
t- 102
1-1 100
1- 108
T 111
1114
112
120
129
(b) Conditional Generation (long-history only) without CFG
Figure 13. Visualization of long context generation on Minecraft. We visualize the generation up to the maximum length of the training Given 25 initial frames (red), DFoT with temporal history guidance (upper) can roll stably without blowing up without CFG. In contrast, one can clearly see that without temporal history guidance (lower), conditional generation easily becomes blurry in later frames. This is likely because the shorter-context model less likely fall out of distribution, using its generation power compensate for the unconfident, blurry prediction from the longer-context model.
28

---

DFoT Baselines Kinetics-600, without HG
History
Generated
History
Generated
GT
DFoT
(foe suncdy
DFoT
SD
BD
FS
GT
(sceutch))
DFOT
DFoT
(fine
SD
BD
FS
Figure 14. Additional qualitative comparison on Kinetics-600. We uniformly subsample 6 frames (0, 6,9, 12, 15) from 16-frame videos, conditioned 5-frame histories. Both DFoT variants, scratch and fine-tuned, consistently align with the history, generating high-quality samples that closely resemble the ground truth. In contrast, the baselines, typically ordered as SD > BD > FS, struggle to maintain consistency and often exhibit artifacts.
29

---

Husde Video Diffusion
B Extended Related Work
B.1 History-conditioned Guidance
In this section, we discuss how CFG is employed for guiding with history in video diffusioon models. The most common case is in Image-to-Vided Videeo Diffusion Models (Blattmann et al., 2023a; Xing et al., 2023; Yang et al., 2024), where the model uses the first frame for guidance. Typically, the conditioning frame is incorporated into the architecture by concatenating it channel-wise with each frame to be generated, and additionally, the CLIP (Radford et al., 2021) embedding of the conditioning frame is used for cross-attention.
Few Conditional Video Diffusion Models have pushed the boundary by guiding with.fxed set of few frames. Specifically, VidcoLDM (Blattmann et al., 2023b) uses the first {1,2} frames for guidance, W.A.L.T. (Gupta et al., 2024) guides with the first 2 latent tokens, i.e. {5} frames, and 4DiM (Watson et al., 2025) guides with the first {1, 2, 8] frames. Similarly, in Multi-view Diffusion Models, which is similar to video diffusion models but do not differentiate frame order, CAT3D (Gao et al., 2024) guides with the first (1,3} frames.
Architecturally, these models incorporate history frames in various ways. VideoLDM concatenates a binary mask, indicating whether each history frame is masked, along with all masked history frames, feeding them to every temporal layer using a learnable downsampling encoder. W.A.L.TT simplifies this by directly concatenating the history frames and binary mask to the noisy generation input, omitting the encoder. 4DiM and CAT3D process the entire sequence-both history and generation frames-as as single sequence, with a binary mask concatenated along the channel dimension indicate whether each frame is masked.
In summary, guiding with history in video models has been explored to a limited extent. While these models differ in how they incorporate history frames into the architecture, they all process history frames separately from generated frames, except for 4DiM and CAT3D, leading to inflexibility of guidance. Additionally, these models are trained using CFG-style random dropout of history frames, which categorizes them as special cases of Sinary-Dropon Diffiusion, shown to be suboptimal. These limitations are highlighted in Section 3. In contrast, our work enables guiding with arbitrary, variable-length history frames without the need for binary-dropout training, facilitated by our modified training objective and architecture design.
C Experimental Details
Below, we provide additional details on datasets, architectures, training, evaluation metrics, and protocols for our experi- ments.
C.1 Datasets
Kinetics-600 (Kay et al., 2017) is a widely used benchmark dataset for video generation, featuring 600 classes of approximately 400K action videos. In addition to its role as a standard benchmark, the task is history-conditioned video generation, making it ideal for evaluating our methods. Following prior works, we use a resolution of 128 x 128 pixels. Despite the large volume of videos and their low resolution, generating high-quality samples from the Kinetics-600 dataset is challenging even with large models due to the diversity and complexity of the content, and thus qualifies as our primary benchmark.
RealEstate (Zhou et al., 2018) is a dataset of home walkthrough videos, accompanied by camera pose annotations. While the dataset is predominantly used in novel view synthesis tasks, we utilize for several reasons: 1) The camera poses allow for more controlled evaluation of video models; for instance, we can easily switch between highly stochastic and deterministic tasks by altering the camera poses, 2) The dataset's nature enables the examination of the consistency of generated videos at 3D level, and 3) The dataset's 's relatively smaller size compared to other text-conditioned video datasets makes it more computationally feasible to train our models, while still providing high-resolution videos. We use resolution of 256 x 256 pixels.
Minecraft (Yan et al., 2023) a dataset of Minecraft gameplay videos, where the player randomly navigates using 3 actions: forward, left, and right. The dataset consists of 200K videos, each with length of 300 frames, each frame has a corresponding action label. The dataset is designed in way that good FVD can only be achieved with long context under action conditioned setting. Specifically, the dataset contains many trajectories where the player turns around and visits areas that it had visited before. While the original dataset is 128 x 128 pixels, we train and evaluate on an upsampled version of 256x x 256 pixels, to generate higher-quality samples.
Fruit Swapping is an imitation learning dataset associated with a fruit rearrangement task adopted from Diffusion
30

---

Hustor Video Diffusion
Forcing (Chen et al., 2024). The task involves tabletop setup where an apple and an orange are randomly put in two of the three empty clots. A single-arm robot is tasked with swapping the two fruits' slots using the third, empty slot as shown in Figure 17. The task requires long-horizon memory since one must remember the initial configuration of the slots to determine the final, target configuration. While the three slots provide a discrete state, each slot has a diameter of 15 centimeters and the fruit can be anywhere in the slot as soon as half of its column resides inside the slot. The task is made even harder when an adversarial human deliberately perturbs the fruit within its slot during the task execution - if there are 10 possible locations within each slot, there would already be 10x combinations of waypoints. This requires a robot policy to be reactive to the fruit locations rather than memorizing all possible combinations. The dataset contains 300 expert demonstrations of the entire swapping task collected by model-based planner, during which no disturbance happens. The robot may move an apple from slot 1 to the center of slot 2, move the orange from slot 3 to the center of slot 1, and then move the apple from slot 2 to slot 3. Notably, it had never seen a situation where the apple changed its location from center to edge during the middle of the manipulation due to adversarial humans. In addition, the dataset features 300 additional demonstrations of re- re-grasping, which is a very short recovery behavior when it narrowly misses the fruit. In these re-grasping demonstrations, the robot arm only repositions to grab the missed object without moving to another slot. Therefore, the dataset contains 300 demonstrations that involve moving fruits but no regrasping, and 300 demonstrations of regrasping but no moving fruit. The former has an average length of 540 frames and the later has an average length of around 50 frames.
C.2 Implementation Details
We provide a summary of our implementation details in Table 2 and discuss them below.
Pixel vs. Latent Diffusion. In this work, we validate DFoT and HG using both pixel and latent diffusion models. For Kinetics-600 and Minecraft, we train latent diffusion model enhance computational efficiency. Specifically, for Minecraft, we train Ima (Kingma, 2013) from scratch, which compresses 256 x 256 images into 32 x 32 latents, following the approach of Stable Diffusion (Rombach et al., 2022). For Kinetics-600, we train a chunk-wist Video VAE that compresses {1, 4} x 128 x 128 video chunks into 16 x 16 latents, to more aggressively reduce computational costs. This approach resembles CausalVideo VAF, commonly used in prior works (Yu et al., 2023b; Gupta et al., 2024), which compresses an entire 17x x 128x x 128 video into x 16 x 16 latents via causal convolutions. However, we choose to compress every 4 frames separately to preserve DFoT's flexibility. Moreover, this ensures that consistency is influenced solely by the performance of the diffusion model, not the VAE. We implement the VideoVAE and training procedure following Open-Sora-Plan (Lin et al., 2024a). Lastly, for RealFsttaee we train directly in pixel space, based on the observation that latent diffusion models struggle to correctly follow camera pose conditioning, leading to poor performance on this dataset. Architectures and training details differ significantly between pixel and latent diffusion models, as we discuss in the following sections.
Architecture. We employ the DiT (Peebles & Xie, 2023) and U-ViT (Hoogeboom et al., 2023; 2024) backbones for the latent and pixel diffusion models, respectively. Both are transformer-based architectures; however, the key difference is that DiT's transformer blocks operate at single resolution, whereas U-ViT incorporates multiple resolutions, with transformer blocks residing at each resolution. Due to this difference, we observe that the U-ViT backbone scales better in the pixel space. For improved scalability and temporal consistency, instead of using factorized attention (Ho et al., 2022b), where attention is applied separately to spatial and temporal dimensions, we employ 3D attention that operates on all tokens simultancously. In addition to this, we incorporate 3D RoPE (Su et al., 2023; Gervet et al., 2023) as relative positional encodings for the T, H. W dimensions.
All conditioning inputs, including noise levels, actions, and camera poses, are injected into the model using an Adal.N layer, following (Peebles& & Xie, 2023). For noise levels, since each frame retains independent noise levels in DFoT, an AdaLN layer is applied separately to each token, using the noise level of the corresponding frame. Minecraft actions are converted into one-hot vectors, which are then transformed into embeddings through an MLP layer and added to the noise level embeddings. For camera pose conditioning in RealEstate 10K, we compute the relative camera pose with respect to the first frame. Following the methodologies of 3DiM (Watson et al., 2023) and 4DiM (Watson et al., 2025), this relative pose is then converted into ray origins and directions, which then transformed into 180-dimensional positional embeddings, similar to Nerf (Mildenhall et al., 2021). Across the resolutions of UVV, the camera pose embeddings are spatially downsampled to match the resolution before being injected into the model.
Diffusion. We use a cosine noise schedule (Nichol & Dhariwal, 2021) for all of our diffusion models. For the RealEstate 10K and Minecraft models, we shift the noise schedule to be significantly noisier (Hoogeboom et al., 2023) by a factor of
31

---

21 x 32 x 32 Hustor Video Diffusion
Table 2. Implementation details for DFoT and baseline models.
Kinetics-600
RealEstate 10K
Minecraft
Imitation Learning
VAEs
Input
{1, 4] x 128 x 128
1x 256 x25
Compression (ft, fa)
{1. 4).
Latent channels
16
4
Training steps
600k
50k
Optimizer
Adam
Adam 96
Batch size
64
Learning rate
lc-4
4c-4 x
EMA
0.999
VDMs
Input
17 128 x 128 5 x 16 x 16
8x 256 256 X
50 x 256 256
Latent
50 x 32 x 32 2
X
Frame skip
10 - Max U-ViT 2
15
Backbone
DIT
DIT
Attention UNet
Patch size
Layer types
Transformer
[ResNet 2, Transformer 2] [3,3,6, 6, 20] [128, 256, 376, 1152] 9
Transformer 12
Attention, Conv
Layers
28
8
Hidden size
1152
768
128
Heads
16
12
4
Training steps
640k
500k
200k
100k
Warmup steps
10k
10k
10k
10k
Optimizer
AdamW
AdamW W
AdamW
A
Batch size
192
96
96
64
Learning rate
2e-4
5e-5
le-4
5e-4
Weight decay
1e-2
le-3
1e-3
EMA
0.9999
0.9999
0.9999
Diffusion type
Discrete
Continuous Shifted Cosine 0.125
Discrete
Discrete
Noise schedule
Cosine
Shifted Cosine
Cosine x
Noise schedule shift
X
0.125
Parameterization
v
V
Xo
Sampler
DDIM
DDIM 50
DDIM 50
DDIM
Sampling steps
50
50
0.125, which we find markedly enhances sample quality, especially for RealEstate 1OK. This finding aligns with prior works (Chen, 2023; Hoogeboom et al., 2023) that highlight the importance of adding sufficient noise during training, especially when dealing with highly redundant images, such as those with high resolution. Another important design choice the parameterization of diffusion models. We employ the v-parameterizatioon (Salimans & Ho, 2022) for all models, which has been widely adopted in image and video diffusion models (Ho al., 2022a; Lin et al., 2024b) due to its superior sample quality and quicker convergence, except for the robot model, where we use the su-parameterization. Lastly, to expedite training, we use min-SNR loss reweighting (Hang et al., 2023) for Kinetics and robot learning, and sigmoid loss reweighting (Kingma & Gao, 2023; Hoogeboom et al., 2024) for RealEstate 10K and Minecraft.
Training. We train models for each dataset and for each model class (e.g.. DFoT, SD, etc.), using the same pipeline within each dataset. We apply a frame skip, where training video clips are subsampled by a specific stride: value of for Kinetics-600 for Minecraft, and for Imitation Learning. For RealEstate 10K, from 10 and extending to the maximum frame skip possible within each video, to help the model learn various camera poses. Throughout all training, We employ the AdamW (Loshchilov, 2017) optimizer, with linear warmup and a constant learning rate. Additionally, we utilize fp16 precision for computational efficiency and clip gradients to a maximum norm of 1.0 to stabilize training. For robot imitation learning, we follow the setup in Diffusion Forcing (Chen et al., 2024) where we concatenate actions and the next observation together for diffusion, with the exception that we stack the next 15 actions
an increasing frame skip, starting
32

---

Husder Video Diffusion
together for every video frame.
Sampling. For all experiments, we use the deterministic DDIM (Song et al., 2020) sampler with 50 steps. Sampling with history guidance, which requires multiple scores at every sampling step, is implemented by stacking the corresponding inputs across the batch dimension to compute the scores in parallel. These scores are then composed to obtain the final score for the DDIM update.
Compute Resources. We utilize 12 H100 GPUs for training all of our video diffusion models, with each model requiring approximately 5 days to train under our chosen batch size. One exception is the Robot model, which is trained on 4 RTX4090 GPUs for 4 hours. We note that most of the video models converge in validation metrics with a fraction of our reported total training steps. However, we chose to train them longer because the industry baselines on these datasets (Yu et al., 2023a; Ruhe et al., 2024) are trained for a great number of epochs that are even unmatched by our final training steps. There was no noticeable overfitting throughout the process.
C.3 Evaluation Metries.
Fréchet Video Distance (FVD, Unterthiner et al. (2018)). We employ FVD as the primary evaluation metric for video generation performance. Similar to FID (Heusel et al., 2017), FVD computes the Fréchet distance between the feature distributions of generated and real videos, with features extracted from a pre-trained I3D network (Carreira & Zisserman, 2017). Lower FVD scores indicate better video generation performance. Unlike image-wise metrics such as FIID, FVD evaluates entire video sequences, capturing temporal consistency and dynamics in addition to quality and diversity, making it the most suitable metric for our video generation tasks. Moreover, FVD is computed for the entire video, including both history and generated frames, to assess the consistency between them.
VBench (Huang et al., 2024). We use VBench, evaluation suite designed to assess video generation models in comprehensive manner, when separate evaluation for different aspects of video generation is needed. Among 16 sub-metrics, we focus on 5 metrics to assess three aspects: 1) Frame-wise Quality, calculated as the average of Aesthetic Quality and Imaging Quality, assesses the visual quality of individual frames; 2) (Temporal) Consistency, derived as the average of Subject Consistency and Background Consistency, evaluates the short- and long-term consistency of generated videos; and 3) Dynamic Degree assesses the degree of dynamics, i.e., the amount of motion in the generated videos. All metrics are better when higher, evaluate the generated videos independently without comparison the ground truth, and are computed by averaging over all generated videos.
Learned Perceptual Image Patch Similarity (LPIPS, Zhang et al. (2018)). We use LPIPS as an alternative metric for highly deterministic tasks, where video-wise metrics may not be as sensitive and accurate. LPIPS computes the perceptual similarity between the generated and corresponding ground truth frames, with lower scores indicating higher similarity. We compute LPIPS only for the generated frames, excluding the history frames, to evaluate whether the generated frames are visually similar the ground truth frames.
C.4 Details on Video Generation Benchmark (Section 6.2)
Kinetics-600 Benchmark. We closely follow the experimental setup of prior works (Ho al., 2022b; Yu et al., 2023a;b; Ruhe et al., 2024). On the test split of the dataset, we evaluate the models on a video prediction task, where the model is conditioned on the first 5 history frames and asked to predict the next 11 frames. Since our models, utilizing Video generate 3 future tokens corresponding to 12 frames, we drop the last frame to align with the prediction task. We report the FVD score computed on 50K generated 16- (6-frame videos, using three different random seeds.
Resource Comparison Against Industry-Level Literature Baselines. In Table 1, we show that DFoT not only outperforms generic diffusion baselines trained with the same pipeline but also holds its ground against strong literature baselines, including Video Diffusion (Ho et al., 2022b), MAGVIT (Yu et al., 2023a), MAGVIT-v2 (Yu et al., 2023b), W.A.L.T (Gupta et al., 2024), and Rolling Diffusion (Ruhe et al., 2024). We have selected only the literature for comparison, omitting others for brevity.
baselines from the
critical aspect of our evaluation the comparison of computational resources. Our DFoT is trained with fewer resources compared these industry-level baselines. Specifically, two primary factors affect the performance of diffusion models: network complexity and training batch size. Our DFoT model is a 673M parameter model with DiT backbone, trained with batch size of 196.
(i) Network Complexity. As Video Diffusion and Rolling Diffusion have different backbones from ours, we compare the number of parameters; they are billion- parameerr models, each with 1.1B and 1.2B, significantly larger than our model.
33

---

Husdor Video Diffusion
For MAGVIT, MAGVIT-v2, and W.A.L.T, which are pure transformer models with similar backbones, use Gftops as measure of computational complexity, suggested by (Peebles & Xie, 2023). Our model is of DiT/XL size, whereas the baselines are DiT/L size, making them slightly smaller. In terms of Gflops, our model has & 1.5 times more Gflops compared to these baselines.
(ii) Batch Size. Video Diffussio, MAGVIT, and MAGVIT-v2 are trained with a batch size of 256, while W.A.L.T and Rolling Diffusion are trained with a batch size of 512, which is significantly larger than ours.
When considering both network complexity and training batch size, MAGVIT and MAGVIT-v2 use comparable resources to our model, whereas Video Diffusion, W.A.L.T, and Rolling Diffusion require significantly more resources. Despite this resource disadvantage, DFoT proves to be highly competitive with these strong baselines, It is only slightly behind W.A.L.T, comparable to MAGVIT-v2, and outperforms the rest. This highlights the superior performance of DFoT as a base video diffusion model.
C.5 Details on History Guidance Experiment (Section 6.3)
For the Kinetics-600 rollout experiment, the models generate the next 59 frames using sliding windows, given the first 5 history frames. The sliding windows are applied such that the model is always conditioned on the last 2 latent tokens and generates the next 3 latent tokens. As with the Kinetics-600 benchmark, we drop the last frame to align with the task. We assess the FVD and VBench scores on 1,024 generated 64-frame vidcos.
History Guidance Scheme. To investigate the effect of HG-v and HG-f, we vary guidance scales using an equally spaced set of w E {1.0, 1.5, 2.5, 3.0, 3.5, 4.0} for both methods. For HG-f, we use a fixed fractional masking degree of ky = 0.8, which we find to generate videos with sufficient dynamics.
C.6 Details on OOD History Experiment (Section 6.4, Task
In Task 1 of Section 6.4, we have shown that video diffusion models easily fail to generalize when the conditioning history is OOD, and temporal history guidance resolves this challenge, through a systematic study on RealEstate 10K. Below, we detail the experiment.
What makes a history OOD? As shown in the training data distribution of Figure 7, we find that the rotation angle of the camera poses within single training scene is typically small, rarely exceeding 100 Hence, history with a wider rotation angle. such as 150", is considered OOD. Based this observation, we assign the following tasks to the models: "Given 4- 4-frame history, with varying rotation angles, generate 4 frames that interpolates between these frames."
Evaluation Based on Rotation Angles. We categorize all scenes based on their rotation angles, into the bins of [00. 10%] [10", 20% 2 ... [170", 180%). Based on the statistics of the training scenes, we conceptually classify the bins of [0 10%]. ..... [90-, 100%] as in-distribution, [100%, 110%). .... [130", 140%] as slightly OOD (< 500 training scenes), and [140", 150%), ... OOD (< 100 training scenes). We then randomly select 32 test scenes (or less if the bin contains fewer scenes) from each bin. For each scene, we select 4 equally spaced frames from the beginning and end of it as the history, and designate the target frames as those in between. We evaluate by computing the LPIPS between the generated and target frames, and report the average LPIPS score for each bin, as shown in Figure 7.
History Guidance Scheme. From a full history H. = {0, 1,,,3), we compose scores conditioned on the following two history subsequences: Hi {0, 1, 2) and H2 (1, each with guidance seale of WT = W2 = 2. Additionally, we implement an extended version of temporal history guidance discussed in Appendix A.8, by also composing generation subsequences: 9 = {4, 5, 6) and G2 = {5, 6, 7} chosen from the full generation g = {4, 5, 6, 7}. For the baseline using vanilla history guidance, we apply a guidance scale of w = 2 to the full history H.
C.7 Details on Long Context Generation (Section 6.4, Task 2).
We train a 50-frame DFoT model that can condition on history up to a length of 25 following the simplified objective Appendix A.5. Note that this is equivalent to 100 frames under the original video with frameskip of 2, or one-third of the maximum video length. We sample an initial context of 25 from the dataset and use our trained model auto-regressively diffuse the next 25 frames conditioned on the previous 25. We roll out times, 125 frames in total, converging the maximum video length in the dataset.
History Guidance Scheme. During sampling, we compose the scores from one long-context model and one short-context model, with context lengths of 25 and respectively. Subtracting the unconditioned score doesn't 't play a significant role on this dataset so we proceed to compose the above two scores only, with simple weighting of 50% each.
34

---

Hustor Video Diffusion
C.8 Details on Robot Imitation Learning (Section 6.4, Task 3).
Baselines. We compare against other diffusion-based imitation learning methods using our same architecture and implemen- tation. First, we compare against typical Markovian model, which diffuses the next few actions only based on current observation. Then, we use a variant of this Markovian model, which can see the previous two frames as a short history but still no long-term memory. Notice that these two short history lengths represent the current mainstream approaches (Chi et al., 2023). In addition, we have a third baseline trained to condition on the entire history so far, representing a family of prrisiin-mmkinng as sequence generation methods. For the convenience of notation, we will refer to these baselines as Markov model, 2rame model, and full-history model. All baselines are trained to diffuse actions and next observations jointly.
The Need to Compose Subtrajectories. As we mentioned in the dataset description, robot imitation learning is a sequence task that requires both long-term memory and local reactive behavior. While both are important to the final task's success, a short-context model will trivially fail most of the time since it won't remember which final state to proceed to. Therefore we focus on our experiment design on exploiting the failure mode of long-context models. One predominant failure mode is overfitting since the imitation learning dataset is extremely small, long-co long-context model can attribute action to any coincidental features. For example, all swapping trajectories in the dataset feature the behavior of putting the first fruit in the very center of the initially empty slot and coming back later to move it away from that center location. How should the model determine where it should pick up this fruit? There is little guarantee for it to determine correctly that it shall proceed to move its gripper right above that fruit versus just blindly going to the center. Whenever a human perturbs this fruit from the very center the slot to the edge the slot, an overfitted model will still move to the very center and proceed to grasp air, ignoring the actual location of that fruit. Therefore, theoretically, a full-history model would never be able to react to such perturbation, since had never seen a trajectory with such perturbation and successful trajectory would be out-of-distribution Instead, it needs to mix in some behavior from a local reactive policy to perform the task, leveraging the fact that whenever a long history is distr---iitiiutinnn you can always fall back to a shorter context model and imitate relevant sub-trajectories. Therefore, the only way to solve this task under the adversarial human is to stitch sub-trajectories together while keeping long-term memory.
History Guidance Scheme. To achieve the aforementioned stitched behavior, we compose three diffusion models with a context of 1 frames, frames, and full history. We assign the full-history model with small weight of 0.2, the 1 frame model, and the 4 frame model with weight of 0.45 each. Like Minecraft, we didn't 't find subtracting unconditioned score super important in this task so we omitted it. The frames here refer to the bundle of the next 15 actions and the single future video frame after that as we mentioned earlier in implementation details.
C.9 Details on Ultra Long Video Generation (Section 6.5).
We provide additional details on generating long navigation videos on RealEstate incorporating all advanced techniques associated with DFoT and history guidance. The generation of long navigation videos divided into two phases: (i) a rollout phase, where the model generates a long video using a sliding window approach, and (ii) interpolation phase, where the generated frames are further interpolated to create smooth video. The process is detailed below.
(i) Rollout Phase. During the rollout phase, starting with a single image randomly selected from the dataset, the model generates a long video using a sliding window, where it is conditioned on the last 4 frames to generate the next 4 frames. The first iteration an exception, where the model is conditioned on the single image and generates the next frames. Importantly, navigation cannot rely on the ground truth camera poses for two reasons: 1) videos in the dataset are relatively short (less than 300 frames), so quickly exhaust available camera poses, and 2) the navigation task is highly stochastic, meaning the ground truth camera poses may not align with the generated frames (e.g., moving straight into wall). To address this, we have developed a simple navigation UI, allowing a user to navigate freely in the scene by providing inputs after each sliding window iteration. Specifically, the user can specify the horizontal and vertical angles, relative to the current frame, for the desired navigation direction, as well as the movement distance. This input is converted into a sequence of camera poses, which are then used as conditioning input for the model to sample the next set of frames. This process repeated until the desired video length is achieved.
(ii) Interpolation Phase. Next, in the interpolation phase, leveraging DFoT's flexibility which supports interpolation, we interpolate between the generated frames by factor of 7. Specifically, using every pair of consecutive generated frames as history, we interpolate 6 frames between them. Camera poses for the interpolated frames, which should be given as input to the model, are computed by linearly interpolating the camera poses of the frames at both ends. More specifically, rotation matrices are interpolated using SLERP (Shoemake, 1985), and translation vectors are linearly interpolated.
35

---

Histor Video Diffusion
0.20
0.120
DFoT (scratch)
DFTT (scnatok)
DFoT (fio-ttnnd)
0.118
DF&T (fnnc-wund)
0.18
0.116
Thainnng Loss
Training Loes
[40 FVD: 6.2
(640k) FVD. 4.3
0.16
0.114
0.112
0.11
500)) FVD: 4.7
0.110
0.12
0.108
0.106
0.10
100k
200k
300k
100k
500k
600k
10k 80%
103k
6:1kk
Training Iterations
Training Iterations
(a) A comprehensive view of the training loss curves. DFoT (fine-
(b) A zoomed-in view of the training loss curves. Only after 80k iterations, DFoT (fine-tuned) displays lower training loss than DFoT (scratch) trained for 640k iterations.
tuned) achieves a low training loss early in the iterations and con- verges significantly faster than DFoT (scratch).
Figure 15. Training loss curves for DFoT, trained from scratch and finettuned from the pre-trained FS model, Kinetics-600.
History Guidance Scheme. Finally, we discuss how history guidance is utilized throughout the navigation task. During the sliding window rollout, the default HG scheme is HG-f, which we find to be extremely stable during long rollouts, Specifically, we apply HG-fwin a guidance scale of w = 4 with a fractional masking degree of ky 0.4, chosen to ensure optimal stability. Additionally, we switch to HG-v with a guidance scale of a = 4 for more challenging situations, such as when the model needs to "extrrpolate"" to new areas. This is because HG-v performs better in such challenging scenarios, although is less stable than HG-f, and thus used sparingly. This switch is triggered when the model is asked to change the direction by more than 30", or when the model is asked move further than certain distance. During the interpolation phase, we apply HG-v with small guidance scale of w = 1.5, to ensure the interpolated video is smooth and consistent.
Stabilization. As an additional techinique, we also employ the stabilization technique proposed in Diffusion Forcing (Chen et al., 2024), where the previously generated frames are marked to be slightly noisy at a level of k = 0.02, to prevent error accumulation, thereby further stabilizing the long rollout.
D Additional Experimental Results
In this section, we present additional experimental results to (i) answer potential questions that may provide further insights into our proposed DFoT and HG, and (ii) further elaborate and provide additional samples for Section 6.
D.1 Additional Results on Fine-tuning to DFoT
Below we provide detailed results on fine-tuning a pre-trained full-sequence (FS) model to DFoT, both from training and sampling perspectives.
Training Dynamics. We show the training loss curves of for two variants of DFoT, one trained from scratch for 640k iterations, and the other fine-tuned from the pre-trained FS model for 80k iterations, in Figure 15. We observe that the pre-trained model already provides good initialization for DFoT, the model starts with low training loss and converges rapidly in the early iterations, in Figure 15a. Surprisingly, the fine-tuned model achieves a lower training loss than the model trained from scratch after only 80k iterations, as shown in Figure 15b. Moreover, after 40k iterations, the fine-tuned model exhibits training loss comparable to the model trained from scratch for 405k iterations, which is 100 speedup. This highlights the superior efficiency and ease of training DFoT by fine-tuning from a pre-trained model. While this opens up the possibility of fine-tuning large foundational video diffusion models to DFoT with small computational cost, we leave this as future work.
FVD Metric Evolution. In contrast to the training loss, Figure 15b (or Table 1) shows that the fine-tuned model achieves a slightly higher FVD score than the model trained from scratch, although being highly competitive even after 40k iterations. We attribute this discrepancy to the use of EMA, which is commonly employed in diffusion models to enhance sample quality (Ho et al., 2020; Dhariwal & Nichol, 2021). By default, we use an EMA decay of 0.9999, and thus the model weights used for sampling are affected by the last tens of thousands of training iterations. Therefore, the fnnet model's superior
36

---

Hustor Video Diffusion
280
BD HG-v
260
FVD
240
220
BD
208.0
HG-v
200
196.0
DFoT
Drol + HG
+ HG-v
181.
+ HG-F
Better 1
2
3 Guidance scale w
(a) FVD as function of guidance scale w for DFoT and BD using HG. Both with HG-v, DFoT yields better FVD-w curves than BD
(b) Qualitative comparison of DFoT and BD using HG-v with op- timal guidance scales = 1.5. While DFoT generates consistent, high-quality samples, BD struggles remain consistent with the history frames and produces artifacts. Red box history frames.
and thus achieves lower best FVD score. Applying HG-f, which specific to DFoT. enlarges the performance gap.
Figure 16. History Guidance works better with DFoT than with Binary-Dropout Diffusion (BD).
training loss does not immediately translate to a lower FVD score, but expect it to outperform the model trained from scratch after an additional short training period. While one may consider simply fine- tinettuning the model without EMA to speed up, EMA is crucial for sample quality; for example, at 80k iterations, FVD without EMA is 7.3, significantly higher than the 4.7 with EMA. This suggests that choosing a smaller EMA decay that still guarantees sample quality, through sophisticated strategies such as post hoc EMA tuning (Karras al., 2024), may be a promising direction for future work.
D.2 Ablation Study on Binary-Dropout Diffusion with Vanilla History Guidance
While we have shown that Binary-Dropout Diffusion (BD) performs poorly as a base model (Q2 of Section 6.2). BD still can implement vanilla history guidance due to its binary dropout training. As such, a natural question is: How does BD perform with IIG-v, compared to DFoT? To answer this question, we repeat the Kinetics-600 rollout experiment in Section 6.3 using BD with HG-v, comparing against DFoT with HG. See Figure 16 for the results. We observe that DFoT consistently outperforms BD across all guidance scales except for w = 2.5, as shown in Figure 16a, Under their optimal guidance scales of 1.5, DFoT achieves lower FVD score of 181.6 compared BD's 196.0, and qualitatively, generates more consistent, high-quality samples, as shown in Figure 16b. When using HG-f, which only applicable to DFoT, DFoT further outperforms BD, achieving FVD score of 170.4. These results highlight that DFoT is a better base model for implementing history guidance, both in performance and in variety of guidance methods that can be applied.
D.3 Detailed Results on Long Context Generation (Section 6.4, Task 2)
We calculate the FVD on 1024 samples across all 125 generated frames. simple conditional diffusion model with context full context achieves an FVD of 97.625 while our temporal guidance achieves an FVD of 79.19 (lower is better). We note that while traditionally FVD a bad metric for videos with high intrinsic variance, it's well-suited for our benchmark since both conditionnni and the dataset design constrain the possible variance. We visually observe that Diffusion Forcing Transformer's prediction aligns well with the ground truth semantically over the majority of the frames in video, showing the variance is well- warranaar.. We visualize one randomly picked sample in Figure 13, showing that temporal guidance can maintain high-quality details far into the future even without CFG. In the meanwhile, the long-contex: model without temporal guidance can suffer from the high dimensional context, which makes it much more likely to see outtof-distriuution frames in its history.
D.4 Detailed Results on Long-horizon yet Reactive Imitation Learning (Section 6.4 Task 3)
We examine the success rate of robot imitation learning quantitatively by randomizing the environment 100 times before testing the temporal guidance model as well as its baselines. We found that the Markov baseline fails to perform the task completely as expected since it has trouble sticking to a specific plan it would move away from fruit and then move back halfway since it has no memory. The 4-frame model suffers from the same issue and cannot finish the task. It does react well perturbations on the object and picks up the fruit from time time, showing short context indeed prevents
37

---

Huster Video Diffusion
Figure 17. Visualization of the fruit-swapping task through a D generated video. Two fruits are randomly put within two random slots. The robot is tasked with swapping its slots using the third slot and moving one fruit at time. This task requires long-horizon memory because it needs to remember the initial location of the fruit for the task completion, but also react different fruit locations within each slot, which is combinatorically impossible form the dataset.
overfitting from temporal locality. We found that the full-history model, with the maximum possible memory, performs well whenever there is no human perturbation. However, as soon as the adversarial human perturbs the fruit during the task execution, this policy often blindly goes to the very center of the third slot while the object already moved the edge of the slot. The policy will then proceed to close its gripper, holding nothing, and then move the next slot, thinking has something in its hand. There occasional cases when this doesn't 't happen and the model actually reacts to the adversarial perturbation, although infrequently and only happens to the case then perturbation from the slot center isn't 't too big. Overall this shows that using full-context model naively can make the model suffer from overfitting and one may want to manually emphasize the temporal locality prior. Finally, we tested DFoT composed guidance and found to achieve a much higher success rate of 83%, showing that it's actually stitching the subtrajectories to make decisions, or at least simultaneously borrowing the memory from the full-context model while staying locally reactive using the short-context model. addition, attempted few stronger perturbations such that the adversarial human will deliberately knock off the fruit from the robot's 's gripper when it's closing. We found that temporal guidance even react this by regrasping and eventually finishing the whole swapping task. However, even temporal guidance achieves only 28% to this strong perturbation since it's way too out-of-distribution and may require more data. Qualitatively, we visualize generated robot trajectory with an unseen configuration in Figure 17.
D.5 Additional Qualitative Results
We present additional qualitative results to supplement our main findings in Section 6. Please refer to Figures 8 to 12 and 14 for detailed visual comparisons, which are discussed below.
DFoT vs. Baselines (Section 6.2, Q1). We present additional qualitative comparisons of DFoT against baselines in Figure 14, extension the qualitative results shown in Figure 4. Consistent with the quantitative findings in Table 1, DFoT produces more consistent and higher-quality samples compared to all basclines.
Empirical Flexibility of DFoT (Section 6.2, Q3). As evidence of the empirical flexibility of DFoT, we present additional qualitative results on RealEstate 10K in Figure 11. Our DFoT model successfully generates consistent samples, given histories that vary both in length and timestamps. This highlights the effectiveness of our new training objective, which transforms DFoT into a flexible multi-task model, uniformly achieving high performance across diverse tasks.
Improving Video Generation via History Guidance (Section 6.3). In addition to the results shown in Figure 6a for Kinetics-600, we present further qualitative results on RealEstate 10K in Figure 12, highlighting the effectiveness of vanilla history guidance in improving video generation. With increasing guidance scales, the generated samples exhibit significantly higher frame quality and consistency, likewise to the results on Kinetics-600. This behavior consistent across different tasks extrapolaaoooo and showeasing the broad applicability of history guidance any history-conditionee video generation
38

---

Hustor Video Diffusion
task.
Robustness to Out-of-Distribution (OOD) History (Section 6.4, Task 1). We provide additional qualitative results for Task from Section 6.4, as illustrated in Figure 10. These results demonstrate that HG-t enables DFoT to uniquely remain robust to OOD history. Failure cases clearly observed in baselines show that typically, video diffusion models only perform well when the history is in-distribution. By composing in-distribution short history windows, HG-t can effectively approximate strictly OOD histories that were unseen during training.
D.6 Detailed results on Ultra Long Video Generation (Section 6.5).
We present extended results from Section 6.5 below.
DFoT vs. SD Long Rollout. To begin with, highlight the significant challenges of generating long navigation videos using the RealEstate I0K dataset. Specifically, investigate the performance of SD, the most conventional and competitive baseline. To mitigate the stochastic nature of navigation that complicates comparisons, we evaluate DFoT with HG and SD on a simple navigation task of moving straight, which is almost deterministic. We avoid using interpolation aapplaationaaplcaallllle only to DFoT to ensure a fair comparison. The results, shown in Figure 9, indicate that SD struggles to maintain consistency with the history frame, failing around frame ~30. We attribute this to SD's inferior quality and consistency, along with its inability to recover from small errors during generation. In contrast, DFoT with HG succeeds to stably roll out beyond frame 72. Alongside the qualitative comparison, we note that 4DiM (Watson et al., 2025), an SD model that, our knowledge, produces the longest and highest-qualle videos on RealEstate 10K among the methods in the literature, generates videos with maximum length of 32 frames, which is significantly shorter than our long navigation videos.
More Samples. We present four samples of long navigation videos generated by DFoT with HG Figures 8a to 8d. These samples demonstrate the capability of DFoT with HG to stably generate extremely long videos. The generated videos are notably longer than those in the training dataset, which primarily cover a single room small area, rather than multiple connected rooms areas.
39


Reviews:
[{"review_id": "Ql2MI3e0f2", "reviewer": "Reviewer_uQSd", "summary": "This paper regards the noise in the diffusion process as a form of masking, integrating history frames and generated frames into a unified Diffusion Forcing Transformer (DFoT) framework. By combining different masking strategies for history information, the paper proposes several History Guidance (HG) methods, enhancing generation quality, dynamics, and historical consistency. Experimental results show the advantages of the proposed methods in long video generation and many downstream tasks.\n\nAfter the rebuttal, the authors address my concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on our method's innovation, theoretical support, and evaluation. Below, we address concerns on long video generation and present new results from fine-tuning a 1.3B text-to-video foundation model to DFoT.\n\n> ### **Q1. Advantage of our method in ultra-long video generation**\n\nLong video generation quality is measured by two factors: rollout stability and long-context semantic continuity. Here, we emphasize our method’s exceptional stability, while semantic continuity is discussed in Q2.\n\nStability refers to maintaining video quality as models condition on previously generated, potentially erroneous frames. Achieving stable rollouts via sliding windows has historically been challenging due to compounding errors [1], with even large-scale foundation models often failing. As noted in Appendix D.6, Re10K is a challenging dataset, and the previous state-of-the-art [2] was limited to 32 frames before failure. This underscores why our method's ability to generate nearly 1000 frames marks a significant improvement in long-rollout stability. Furthermore, Appendix D.6 and Figure 9 show DFoT and history guidance are essential for ultra-long video generation. \n\n> ### **Q2a. Coherent transitions but lacking long-context semantic continuity**\n\nWhile we acknowledge that our Re10K demos lack long-context semantic continuity, we emphasize that **this limitation is not inherent to our method, DFoT and History Guidance**, which perform well on Minecraft, designed to test long-context consistency. Instead, the issue arises from limited context window size due to computational constraints and the limited semantic length of the Re10K dataset:\n\n- Limited compute: Our Re10K model is trained with a context window size of only 8 frames, as training with larger context windows becomes increasingly expensive. This results in an overlap of only 4 frames between successive sliding windows, insufficient for long-memory in 3D space.\n- Limited training data: The Re10K dataset consists of semantically short video clips without loop closures, typically covering a single room/area (Appendix D.6). Thus, the model cannot learn to maintain semantic continuity over greater lengths (e.g., multiple rooms/areas).\n\nWe further argue that the **semantic continuity of the generated videos scales with the semantic length of the training data and the context window size**. Beyond strong performance on Minecraft, we are pleased to present new long video generation results in **Figure R1a on [our anonymous website](https://anon-dfot.github.io/)**. We fine-tuned the Wan2.1 T2V-1.3B model [3] to DFoT on a subset of the Panda-70M dataset [4]. With a larger context window size and longer semantic length of the training data, the results are more coherent and semantically consistent than those from Re10K. This demonstrates that the semantic continuity of generated videos using our method can be improved by simply scaling the training data and context window size.\n\n\n> ### **Q2b. Transition mechanisms between different segments in long video generation**\n\nAs detailed in **Appendix C.9**, our Re10K videos are generated by user input of camera poses for each sliding window. For simplicity, we've implemented a basic transition mechanism where the camera pose changes at a constant rate within each sliding window, based on specified distance and rotation angles. This setup causes discontinuous transitions due to velocity changes between sliding windows. We clarify that the **discontinuous transitions stem from the simplicity of our current implementation of camera pose transitions, not a limitation of our method**. Smoother transitions could be achieved with a more advanced mechanism.\n\n> ### **Q3. HG-t's effectiveness in general video generation tasks**\n\n**Section 6.4 demonstrates the practical applicability of HG-t**. We show that HG-t addresses three critical challenges: OOD history, long context generation, and imitation learning. At the core of these tasks is the OOD nature of observed history frames. As noted in **Section 5**, the amount of required training data grows exponentially with the length of conditioning history. Consequently, the OOD problem frequently arises in tasks that require understanding long histories, such as video-to-video generation, and HG-t effectively addresses this core issue. Additionally, the history guidance framework's strength lies in its techniques (HG-v, HG-t, and HG-f) offering unique advantages, allowing the best technique to be chosen based on specific needs of general video generation tasks.\n\n> ### References\n- [1] Qiu et al. \"Freenoise: Tuning-free longer video diffusion via noise rescheduling.\" ICLR 2024.\n- [2] Watson et al. “Controlling space and time with diffusion models.” ICLR 2024.\n- [3] WanTeam et al. \"Wan: Open and Advanced Large-Scale Video Generative Models.\" arXiv 2025.\n- [4] Chen et al. \"Panda-70m: Captioning 70m videos with multiple cross-modality teachers.\" CVPR 2024.", "claims_and_evidence": "This work innovatively introduces the Diffusion Forcing Transformer, integrating generated frames and history conditions within a unified framework. The authors also proposes multiple History Guidance methods, with HG-v already showing performance improvements. Variants HG-t and HG-f further enhance model generalization and dynamic generation capabilities.\n\nHowever, although the paper claims to support flexible-length history information, experiments do not clearly demonstrate the advantage of History Guidance in ultra-long video generation. The incremental extension method for \"unlimited-length\" videos achieves coherent transitions but lacks semantic continuity (e.g., supplementary material video i2v_dfot_long_1.mp4, from 11 to 24 seconds). Although HG-t shows potential in integrating long-term and short-term history contexts, its effectiveness has not been validated in general video generation tasks, limiting the method's practical applicability.", "methods_and_evaluation_criteria": "Yes. The paper proposes DFoT to address high-quality and consistency issues in long video generation. The experiments validate the model’s capabilities using the Kinetics-600, RealEstate10K, Minecraft, and Fruit Swapping datasets, demonstrating its effectiveness in long video generation tasks.", "theoretical_claims": "This paper is not primarily a theoretical contribution. Nevertheless, it provides theoretical support for DFoT’s optimization objectives based on Evidence Lower Bound.", "experimental_designs_or_analyses": "The flexibility and effectiveness of DFoT are evaluated on Kinetics-600, RealEstate10K, Minecraft, and Fruit Swapping datasets, as well as in tasks such as video prediction, frame interpolation, long video generation, and imitation learning.", "relation_to_broader_scientific_literature": "This work provides insights into sementically consistent long video generation.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The supplementary materials are entensive, providing impressive experimental results.", "other_comments_or_suggestions": "None", "questions_for_authors": "I am curious about the transition mechanisms between different segments in long video generation, particularly the role of user interactions (there seems to be significant discontinuity between different segments).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fzBa1nrz79", "reviewer": "Reviewer_mAfS", "summary": "Classifier-free guidance (CFG) greatly improves conditional generation in diffusion models, but applying it to video diffusion—where the number of context frames can vary—introduces significant challenges. Existing architectures often restrict conditioning to a fixed size, and CFG-style history dropout is ineffective. In response, the Diffusion Forcing Transformer (DFoT) and its associated History Guidance methods enable flexible conditioning, substantially enhancing video quality and consistency over potentially very long sequences.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on the novelty and theoretical support of our method, and extensive experimental results. Below, we address your questions by presenting enhanced long video generation achieved through fine-tuning a 1.3B text-to-video foundation model and providing an ablative comparison of the training objectives.\n\n> ### **Q1. Long video generation on dynamic scenes, with SOTA foundation models**\n\nWhile our original results were limited by our compute and data, we are pleased to report positive outcomes by fine-tuning a state-of-the-art video generative model with DFoT using limited compute. We fine-tuned the Wan2.1 T2V-1.3B model [1], a leading SOTA model, to DFoT for only 20K steps on a subset of the Panda-70M dataset [2]. This was achieved with just 18 lines of code changes, thanks to DFoT’s simplicity. Please see **Figure R1 on [our anonymous website](https://anon-dfot.github.io/)** for our enhanced long video generation results. Key findings include:\n\n- **Scalability of DFoT and History Guidance**: Our method's strengths remain robust at larger scales and in real-world applications. First, we present long video generation results (**Figure R1a**), where DFoT stably generates long 217-frame videos via sliding window rollouts. This is substantially longer than the 49-frame context window used during fine-tuning. Additionally, DFoT handles flexible-length history effectively; Figure R1a shows videos generated by initially conditioning on a single frame and then on the previous 13 frames during rollout. Figure R1b demonstrates video generation by interpolating between the first and last frames. Lastly, we present a direct ablation study on the effectiveness of history guidance (**Figure R1b**). Similar to our paper results, history guidance significantly improves the video quality and consistency.\n- **Dynamic scenes**: Unlike our Re10K demonstrations focusing on static scenes, the generated videos in **Figure R1** are dynamic scenes with human/animal actions and moving objects, at higher resolution. This demonstrates our method's capability to perform well in complex, realistic scenarios, beyond static ones.\n\n> ### **Q2. Ablation on independent vs. uniform noise levels**\n\nThere could be multiple interpretations of your suggestion, so we address each of them below:\n\n#### Interpretation 1: Independent vs. uniform noise levels\n\n**Section 6.2**, **Table 1**, and **Figure 4** compare our independent noise level training objective against uniform noise levels. Our independent noise level approach outperforms these baselines using uniform noise levels, such as SD, FS, and BD, both quantitatively (Table 1) and qualitatively (Figure 4). This demonstrates that our proposed training objective offers advantages not only in flexibility (conditioning on arbitrary history frames) but also in performance.\n\n#### Interpretation 2: Fully independent vs. Partially independent noise levels\n\nAlternatively, your question might ask whether DFoT can benefit from training with independent noise levels for some frames and uniform noise levels for others.\n\nTo clarify, our problem statement defines history as **any** subset of the frames. While sampling distinguishes between history and generated frames, training does not, to encompass all possible history conditioning during sampling. Since training with partially independent noise levels would deviate from our problem statement, we discussed it in Appendix A.5 for clarity.\n\nAs detailed in **Appendix A.5**, if only a subset of history conditioning schemes is desired, fully independent noise may not be optimal due to unnecessary complexity as the number of frames increases. Therefore, we explored partially independent noise level training. Given the maximum length of history frames we aim to support, independent noise levels are only applied up to this maximum length, with uniform noise levels assigned to the remaining frames. Our Minecraft model, which processes 50 latent frames, used a simplified training objective with a maximum length of 25 latent frames. This approach improved training efficiency and video generation performance compared to fully independent noise levels. However, we also found that such efficiency gains are not noticeable for fewer frames. Since modern video foundation models [1] often feature around 20 latent frames, partially independent noise does not significantly increase efficiency, aligning with Diffusion Forcing [3]'s finding that temporal complexity is minor compared to visual complexity.\n\n> ### References\n- [1] WanTeam et al. \"Wan: Open and Advanced Large-Scale Video Generative Models.\" arXiv 2025.\n- [2] Chen et al. \"Panda-70m: Captioning 70m videos with multiple cross-modality teachers.\" CVPR 2024.\n- [3] Chen et al. “Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion.” NeurIPS 2024", "claims_and_evidence": "Yes, the claims made in the submission are supported  by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, I checked the theoretical claims of  lower bound evidence.", "experimental_designs_or_analyses": "Yes. Very detailed ablations regardind  history gudiance and diffusion force transformer.", "relation_to_broader_scientific_literature": "Introducing a groundbreaking method of history guidance for extended video generation using diffusion models", "essential_references_not_discussed": "The reference part is good to me.", "other_strengths_and_weaknesses": "Strengths:\n\nProposes a novel strategy for producing extended or even infinite-length videos.\n\nThe idea of training independent noise levels for each frame in a video diffusion framework is innovative and warrants further investigation.\n\nWeaknesses:\n\nIt would be valuable to see an ablation study comparing training with independent noise levels for the history frames versus using a uniform noise level for the generated frames.\n\nThe long video demonstrations primarily focus on static scenes, leaving open questions about performance on lengthy sequences of human actions (e.g., in Kinetics-600).\n\nAlthough the approach seems straightforward to adapt to top-tier video generation models like CogVideo (through additional fine-tuning), I’m curious how well it would perform in creating long videos when combined with state-of-the-art model and the proposed training as well as guidance methods.", "other_comments_or_suggestions": "Please see the above part.", "questions_for_authors": "Please see the above part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fFEalD53GV", "reviewer": "Reviewer_RA3f", "summary": "The paper introduces the Diffusion Forcing Transformer (DFoT), a video diffusion architecture that extends diffusion forcing with a theoretically grounded training objective that enables conditioning on a flexible number of history frames. On top of diffusion forcing using transformer, the authors introduce History Guidance – a family of guidance strategies unique to their architecture where the combination of time and frequency guidance achieves the best in motion dynamics and long-term coherence.\n\n## update after rebuttal\nAfter the rebuttal, most of my concerns have been addressed except for the apparent artifacts in the additional results of model scalability. It is understandable that scaling on the 1.3B model within the rebuttal period with limited resources is hard. Therefore, the scalability of DFoT could be a future direction down the road. Still, this does not affect the fact that this is a strong ICML submission. In light of this, I will keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on the strengths of our method, theoretical justification, and extensive analysis. Below, we address questions on sampling efficiency and scalability/generalizability of DFoT, including new results from fine-tuning a 1.3B text-to-video foundation model (see **[anonymous website](https://anon-dfot.github.io/)** for **Figures R1, R2**):\n\n> ### **Q1. Sampling efficiency of History Guidance**\n\nAs you noted, history guidance requires multiple function evaluations (NFE) per timestep, but this doesn't mean a larger sampling budget is needed. Rather, **Figure R2 on [our anonymous website](https://anon-dfot.github.io/)** shows that **history guidance offers better sampling efficiency than sampling without it**.\n- **Total sampling budget**: Total sampling cost depends not only on NFE per timestep but also on the number of timesteps: $\\text{Total NFE} = \\text{NFE/timestep} \\times \\text{\\\\# timesteps}$\n- **Figure R2**: We compared the performance with/without history guidance under the same total sampling budget (i.e. same total NFE). With the same total NFE, history guidance substantially outperforms sampling without it (mostly HG-f > HG-v > w/o HG). This shows that, despite requiring more NFE per timestep, history guidance can be performed efficiently by reducing the number of timesteps, thereby achieving better performance with the same total sampling cost.\n- **Parallelization**: As you mentioned, even under the same total NFE, history guidance can be performed in parallel by stacking score estimations across batches for faster computation, as we do in our implementation (Appendix C.2 - Sampling).\n\n> ### **Q2. Scalability of DFoT**\n\nWe are pleased to report strong results of scaling up DFoT. We fine-tuned the Wan2.1 T2V-1.3B model [1] to DFoT on a subset of the Panda-70M dataset [2] for 20K steps. We note that this model is text-to-video only, so its history conditioning capability is entirely due to DFoT. Despite limited compute and data, **Figure R1 on [our anonymous website](https://anon-dfot.github.io/)** demonstrates the successful scaling while maintaining its advantages:\n\n- **Handling flexible-length history**: The base Wan2.1 model can only generate videos given a text prompt, not being able to condition on history frames. After fine-tuning, DFoT can effectively handle different history-conditional video generation tasks. Specifically, Figure R1a uses a single frame as the initial conditioning frame and then conditions on the previous 13 frames during rollout, while Figure R1b shows interpolation between the first and last frames.\n- **Long video generation**: Figure R1a shows DFoT's long video generation capability still holds at larger scales. We present 217-frame videos via sliding window rollouts, which is substantially longer than the 49-frame window used during fine-tuning.\n- **History guidance**: Figure R1b compares frame interpolation results with/without history guidance, showing that history guidance improves video quality and consistency even at larger scales.\n\n> ### **Q3. Generalizing to (visual) out-of-domain**\n\nThe video generation results in **Figure R1** demonstrate that DFoT can **generalize to arbitrary in-the-wild visual input**. We fine-tuned on a subset of Panda-70M filtered to include only 190K videos with human actions. Despite this, DFoT generalizes well to various in-the-wild visual inputs from random internet images (not part of the training set), such as animals, nature scenes, and moving objects. This capability is not merely due to the base model, which is text-to-video only; DFoT provides the image-conditioning ability.\n\n> ### **Q4. Beyond visual conditioning (text prompt)**\n\nOur fine-tuning results further demonstrate that **DFoT is highly compatible with other types of conditioning**, such as text prompts. With just 18 lines of code changes, the fine-tuned model can condition on varying-length history. All videos in **Figure R1** are generated by jointly conditioning using both text prompts and history frames, and adhering closely to both. Specifically, we utilize both text guidance and history guidance during sampling, with different guidance scales for each. While it is well-known that text guidance enhances generation results, ablation in **Figure R1b** shows that history guidance further improves the quality and consistency, when combined with text guidance. We believe DFoT's sliding window rollout can further synergize with text prompt conditioning by specifying different text prompts for each sliding window. Additionally, DFoT has already shown compatibility with camera pose conditioning in Re10K experiments, and we believe it can extend to other sequential conditioning (e.g. audio).\n> ### References\n- [1] WanTeam et al. \"Wan: Open and Advanced Large-Scale Video Generative Models.\" arXiv 2025.\n- [2] Chen, Tsai-Shien, et al. \"Panda-70m: Captioning 70m videos with multiple cross-modality teachers.\" CVPR 2024.", "claims_and_evidence": "Claim1:  The most important claim is that the approach enables generating extremely long videos stably, far beyond prior art. This is enabled by rolling out DFoT with the proposed history guidance. The results on the website validate this claim by presenting in-domain visualization results.\n\nClaim2: Compositional generation with long-context is another claim. This is validated on Minecraft video dataset and robotic fruit swapping trajectory.\n\nClaim3: The paper claims DFoT with temporal guidance can handle OOD history inputs (extreme camera poses) that cause other models to fail. This is somehow validated by stress-test on RealEstate10K where abrupt viewpoint changes are introduced between conditioning frames as OOD camera poses. Still, visual domain generalization is not validated.", "methods_and_evaluation_criteria": "The paper introduces the Diffusion Forcing Transformer (DFoT), a video diffusion architecture that extends diffusion forcing with a theoretically grounded training objective that enables conditioning on a flexible number of history frames. Instead of feeding a fixed set of past frames as conditioning, DFoT noises each frame independently during training. On top of diffusion forcing using transformer, the authors introduce History Guidance – a family of guidance strategies unique to their architecture where the combination of time and frequency guidance achieves the best in motion dynamics and long-term coherence.\n\nThe authors benchmark their base model against strong baselines on standard video generation metrics, i.e., FVD and VBench scores. For deterministic task as next frame prediction for robotics, LPIPS is reported. Overall, the evaluation is sufficient and solid.", "theoretical_claims": "The authors claim that their DFoT training objective is theoretically well-founded, specifically that it optimizes a valid likelihood-based objective (rather than being a heuristic) as Theorem 4.1. In specific, the per-frame noise training loss optimizes a reweighting of an Evidence Lower Bound (ELBO) on the expected log-likelihoods.", "experimental_designs_or_analyses": "Overall, the evaluation is sufficient and solid.\n\nThe authors benchmark their base model against strong baselines on standard video generation metrics, i.e., FVD and VBench scores. For deterministic task as next frame prediction for robotics, LPIPS is reported. Datasets are extensively explored, including kinetics, RealEstate10K, Minecraft, and real-world robotic tasks. The analysis and ablation studies are also sound and satisfying.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths are extensively discussed in previous sections. I would mention several weaknesses here:\n\n1. One inherent downside of the history guidance approach is that sampling requires multiple model evaluations per timestep. For CFG (and thus HG-v), you already need two forward passes (conditional & unconditional). For HG-t or HG-f, you might need even more (e.g. one for each history window or each noise level setting) to then combine them. By treating them as a batch might mitigate this issue, but it still takes more computational budget.\n2. The scalability of DFoT is not validated. All the dataset used for training are in relatively limited scale. The potential of DFoT in scaling up is not validated and might be worth exploring in the future.\n3. Only in-domain (visual domain) results are presented for ultra-long video generation. I’m wondering if DFoT can reasonably generalize to arbitrary in-the-wild input.\n4. The method is specialized to using visual history frames as conditioning. It does not experiment with text-to-video or audio-conditioned video, etc. While not exactly a weakness (the paper already has a broad scope), it means the benefits are demonstrated only in the context of visual conditioning. An open question is how DFoT would work if the model also had a text prompt – can it combine text and flexible frame guidance?", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to the Strengths and Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QccJIVHv5X", "reviewer": "Reviewer_2WEt", "summary": "This paper delves into video diffusion models, aiming to extend classifier-free guidance (CFG) to video diffusion with variable-length history frames. The authors identify two key challenges: architectures supporting only fixed-size conditioning and poor performance of CFG-style history dropout. To tackle these, they propose the Diffusion Forcing Transformer (DFoT). DFoT is a video diffusion architecture with a theoretically grounded training objective, enabling flexible conditioning on a varying number of history frames. It expands the “noising-as-masking” paradigm to non-causal transformers and is compatible with existing architectures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our work significant and effective. Below, we address your concerns with further explanation, additional ablations, and results of fine-tuning a video foundation model into DFoT.\n\n> ### **Q1. Practical implications of theoretical results (ELBO)**\n\nThank you for your insightful question. In practice, reweighting terms in diffusion ELBO are often dropped for simplicity, like in the original DDPM paper. Therefore, it’s difficult to compare DFoT’s ELBO and vanilla diffusion’s ELBO in practice since both of them train on simplified objectives. However, our **Appendix C.2 - Diffusion** does provide some practical insights: Higher noise level masks out a greater portion of information from the original sequence. As noted in previous works [1, 2], and in our findings (Appendix C.2), a diffusion model's performance is highly sensitive to signal redundancy. Specifically, when the denoising task contains a lot of redundant signals, training should emphasize the objective at high noise levels. In DFoT, some tasks (e.g. unconditional, image-to-video) provide smaller amounts of conditioning frames and are thus less redundant, while other tasks (e.g. video-to-video) are the opposite. Thus, the latter benefit from higher emphasis on high noise levels. Therefore, it is expected that **as the reweighting $\\omega$ biases towards high noise levels, DFoT's performance shifts towards tasks with more conditioning frames, and vice versa**. Therefore, reweighting and noise schedules should be carefully selected based on this analysis, tailored to the downstream tasks.\n\n> ### **Q2. Trade-offs in long-context generation**\n\nWe present an ablation studying how HG-t trades off long-term memory and quality. We conducted the Minecraft long context generation experiments (**Section 6.4 - Task 2**) with varying weights of combining short and full history. We report the results in **Figure R3 on [our anonymous website](https://anon-dfot.github.io/)**. The Minecraft dataset was specifically designed by TECO [5] to be deterministic under an action-conditioned setting. We, therefore, directly use a deterministic metric LPIPS to measure long-context consistency.  In addition, we use frame-wise FID score as a measure for image-only quality, disregarding consistency. The resulting plot shows a clear reverse trend between quality and long-context memory as the weight for full-context increases, indicating that HG-t can trade off long-horizon memory for improved quality robustness by mixing a short-history model. \n\n> ### **Q3. Video Quality and Diversity**\n\nWe are excited to report enhanced video generation results using a larger DFoT model at higher resolution, featuring complex and diverse scenes. Please see **Figure R1 on [our anonymous website](https://anon-dfot.github.io/)** for the results. We fine-tuned the 1.3B T2V foundation model [3] to DFoT on a subset of the Panda-70M dataset [4] for 20K steps. The base model is text-to-video only, so our reported image-to-video results are largely due to DFoT, not the base model. Here is a summary of the results:\n\n- **Improved video quality, diversity**: The generated videos have a higher resolution (832x480) and feature more dynamic, complex scenes with enhanced quality. Although the original Wan2.1 model was not trained to condition on images and our fine-tuning dataset is limited to human-action videos, our model effectively generalizes to generate diverse scenes from random web images, ranging from human and animal actions to nature scenes and moving objects.\n- **Scalability of DFoT and History Guidance**: Our method effectively scales to larger models and datasets, while maintaining its advantages. It handles flexible-length history (Figure R1a: conditioning on a single frame or on the previous 13 frames during rollout; Figure R1b: interpolation between the first and last frames). DFoT also stably generates long 217-frame videos via sliding window rollouts (Figure R1a), which is substantially longer than the 49-frame context window used during fine-tuning. Lastly, Figure R1b shows that the benefits of history guidance persist at larger scales, significantly enhancing video quality and consistency.\n\nOverall, these results demonstrate that our method, beyond Re10K experiments, can effectively scale to more complex video generation tasks with superior performance, highlighting its practical applicability.\n\n> ### References\n- [1] Chen, T. \"On the importance of noise scheduling for diffusion models.\" arXiv 2023.\n- [2] Hoogeboom et al. \"Simple diffusion: End-to-end diffusion for high resolution images.\" ICML 2023.\n- [3] WanTeam et al. \"Wan: Open and Advanced Large-Scale Video Generative Models.\" arXiv 2025.\n- [4] Chen et al. \"Panda-70m: Captioning 70m videos with multiple cross-modality teachers.\" CVPR 2024.\n- [5] Yan et al., “Temporally Consistent Transformers for Video Generation”", "claims_and_evidence": "DFoT's Performance: The claim that DFoT surpasses baselines in video generation is supported by experiments on datasets like Kinetics 600. Comparing DFoT with standard diffusion (SD), binary-dropout diffusion (BD), and full-sequence diffusion with reconstruction guidance (FS), the authors show that DFoT has better Fréchet Video Distance (FVD) scores and generates higher-quality samples.\n\nHistory Guidance: The effectiveness of History Guidance methods is well-evidenced. For instance, on Kinetics-600, vanilla history guidance improves frame quality and consistency, and fractional history guidance addresses the issue of static videos. In tasks like handling out-of-distribution history on RealEstate10K, DFoT with temporal history guidance outperforms baselines.\n\nLong-Video Generation: DFoT's ability to generate ultra-long videos is demonstrated by creating an 862-frame navigation video on RealEstate10K, far exceeding the capabilities of prior methods.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The authors provide a theoretical justification for the training objective through a variational lower bound. They derive an Evidence Lower Bound (ELBO) corresponding to the DFoT training objective, showing that it optimizes a reweighting of the ELBO on the expected log-likelihoods. This theoretical foundation adds credibility to the DFoT architecture and its training process. However, further discussion on the practical implications of these theoretical results, such as how the reweighting affects DFoT's performance in different video generation tasks, would be beneficial.", "experimental_designs_or_analyses": "Experimental Designs: The experimental designs are sound. The authors conduct experiments on multiple datasets with diverse characteristics, demonstrating the generality of their methods. For example, Kinetics-600 is used for benchmarking and quantitative comparisons, while RealEstate10K, Minecraft, and Fruit Swapping are used to study new applications. The use of sliding window rollout in experiments on Kinetics - 600 to test video generation consistency is a valid approach.\n\nAnalyses: The analyses of the experimental results are comprehensive. The authors report numerical results of evaluation metrics and provide qualitative analyses, such as visualizations of generated videos. This helps readers better understand the performance of DFoT and History Guidance methods. However, in some experiments, like the long-context generation in Minecraft, a more in-depth analysis of the trade-offs between different factors, such as the balance between long-term memory and robustness to out-of-distribution history, would be valuable.", "relation_to_broader_scientific_literature": "Video Diffusion Models: In the field of video diffusion models, the paper addresses the limitations of existing architectures that can only support fixed-length conditioning. It also improves upon the CFG-style history dropout approach, which has been shown to be suboptimal. The proposed methods offer new ways to condition on history frames, enhancing the performance of video diffusion models in terms of quality, consistency, and the ability to generate long videos.", "essential_references_not_discussed": "There are no obvious essential references that are not discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\n1. **Significance**: The research has significant implications. Improving the quality and consistency of video generation, as well as enabling the generation of ultra-long videos, has potential applications in many fields, such as robotics, virtual reality, and video production.\n2. **Effective History Guidance Methods**: The History Guidance (HG) family of methods, enabled by DFoT, offers a powerful way to leverage history in video generation. Vanilla History Guidance (HG-v) alone significantly improves video quality and temporal consistency. More advanced methods which combines Temporal History Guidance (HG-t) and Fractional History Guidance (HG-f), further enhance motion dynamics, enable generalization to out-of-distribution history, and can stably roll out extremely long videos. \n\nWeaknesses:\n\n1. **Video Quality and Diversity**: The generated videos have relatively low resolution and limited scene variety. For example, in RealEstate10K experiments, the navigation videos lack complexity and high-resolution details, restricting practical applications. The author are encouraged to experiment with more diverse dataset or stronger pretrained model such as CogVideoX.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces Diffusion Forcing Transformer (DFoT), integrating history frames and generation via flexible masking strategies (HG-t/f/v) to enhance long video consistency and dynamics. Initial concerns about semantic continuity in ultra-long sequences (e.g., 24-second artifacts in supplementary videos) and limited validation on dynamic human actions (Kinetics-600) were partially mitigated through rebuttal clarifications. The method demonstrates strong empirical results: 862-frame generation on RealEstate10K, improved FVD/LPIPS metrics across Kinetics-600/Minecraft/FruitSwapping, and theoretical grounding via ELBO derivation. Though comparisons with SOTA models (e.g., CogVideo) are absent, the rigorous evaluations on diverse tasks (prediction/interpolation/imitation) justify technical novelty. Thus, I recommend to accept this paper.
Decision: Accept (poster)