Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
Authors: Kunhao Liu
Wenbo Hu
Jiale Xu
Ying Shan
Shijian Lu
Venue: ICLR 2026 Poster (2026)
Keywords: autoregressive video generation
long video generation
real-time video generation
Primary area: generative models
TL;DR: REAL-TIME streaming generation of MULTI-MINUTE videos!

Abstract: Streaming video generation as one fundamental component in interactive world models and neural game engines aims to generate high-quality, low-latency, and temporally coherent long stream videos. However, most existing work suffers from severe error accumulation that often significantly degrades the generated stream videos over long horizons. We design Rolling Forcing, a novel video generation technique that enables streaming long videos with minimal error accumulation. Rolling Forcing comes with three novel designs. First, instead of iteratively sampling individual frames which accelerates error propagation, we design a joint denoising scheme that simultaneously denoises multiple frames with progressively increasing noise levels. This design relaxes the strict causality across adjacent frames, effectively suppressing error growth. Second, we introduce the attention sink mechanism into the long-horizon stream video generation task, which allows the model to keep key–value states of initial frames as a global context anchor and thereby enhances long-term global consistency. Third, we design an efficient training algorithm that enables few-step distillation over largely extended denoising windows. This algorithm operates on non-overlapping windows and mitigates exposure bias conditioned on self-generated histories. Extensive experiments show that Rolling Forcing enables real-time streaming generation of multi-minute videos on a single GPU, with substantially reduced error accumulation.

Published as a conference paper at ICLR 2026
ROLLING FORCING: AUTOREGRESSIVE LONG VIDEO
DIFFUSION IN REAL TIME
Kunhao Liu Wenbo Hu Jiale Xu² Ying Shan? Shijian Lult
Nanyang Technological University ²ARC Lab, Tencent PCG
Streaming real time!
min
min
Skeleton wearing a flower hat and Sunglasses dances in the wild at Sunsee.
longboarder accelerating downhill, carving through
kandheld cámera following dog rumning through a park.
Figure I: Rolling Forcing performs real-ting real-time streaming text-to text-to-video generation at 16 fps on single GPU and is capable of producing malti-minute-long videos with minimal error accumulation. More results, code, and demo can be found at the project page.
ABSTRACT
Streaming video generation, as one fundamental component interactive world models and neural game engines, aims to generate high-quality, low-latency, and temporally coherent long video streams. However, most existing work suffers from severe errо accumulation that often significantly degrades the generated stream videos over long horizons. We design Rolling Forcing, a novel video generation technique that enables streaming long videos with minimal error cumulation. Rolling Forcing comes with three novel designs. First, instead iteratively sampling individual frames, which accelerates design joint denoising scheme that simultaneously denoises multiple frames with progressively increasing noise levels. This design relaxes the strict causality across adjacent frames, effectively suppressing eтrоr growth. Second, we intro- duce the attention sink mechanism into the long-horizon stream video generation task, which allows the model to keep key-value states of initial frames global context anchor and thereby enhances long-term global consistency. Third, we de- sign an efficient training algorithm that enables few-step distillation over largely extended denoising windows. This algorithm operates on non-overlapping win- dows and mitigates exposure bias conditioned on self-generated historics. Ex- tensive experiments show that Rolling Forcing enables real-time streaming gen- eration of multi-minute videos single GPU, with substantially reduced error accumulation.
propagation,
WWork done during internship at ARC Lab, Tencent PCG.
1 "Corrsponniig authors.

---

Published as a conference paper at ICLR 2026
Database
Database
Nolay temporal
context
temporal contect
Global context
Denoised
Consistent
clean hrmm
Sequential
Sequenttol
Sequential
Sequential
Long
Long
Long
Long
(a) History corruption
Planning generation
(c) Self Forcing
Rolling Forcing (ours)
Figure 2: Different paradigms in autoregressive video generation. History corruption (Chen et al., 2024; Guo et al., 2025) in (a) compromises temporal consistency, while planning generation (Zhang & Agrawala, 2025; Xiang et al., 2025) in (b) is incompatible with sequential streaming video gen- eration. Self Forcing (Huang et al., 2025) in (c) can achieve consistent sequential streaming but suffers from severe eπror accumulation while generating long videos. The proposed Rolling Forcing in (d) supports streaming long video generation with superior temporal consistency and minimal error accumulation.
INTRODUCTION
Modern video diffusion models (OpenAI, 2024; Polyak al., 2024) have demonstrated impressive capabilities in generating short video clips with rich detail and coherent motion. However, interac- tive applications such as world models (Bruce al., 2024), neural game engines (Valevski et al., 2024), and immersive XR environments require the ability to stream each frame with minimal la- tency while maintaining visual quality and temporal coherence over long horizons. Unlike offline video generation, where the entire sequence is synthesized together at one go, the streaming video generation operates in an online fashion: frames are generated sequentially and immediately con- sumed by downstream tasks displayed to users. Such online nature imposes unique challenges, as the model must maintain long-horizon consistency while accommodating real-time constraints in an autoregressive manner.
Real-time streaming video generation methods, such as Caus Vid (Yin et al., 2025) and Self Forc- ing (Huang al., 2025) (illustrated in Fig. 2(c)), distill pretrained bidirectional video diffusion model into a fast, causal autoregressive generator. While they enable consistent sequential genera- tion. their strictly causal frame prediction causes each frame inherit errors from its predecessors, allowing small imperfections to compound over long horizons and eventually leading to noticeable drift and quality degradation. Two representative approaches have been explored for improving video generation over long horizons, as illustrated Fig. 2(a.b). The first approach explores history corruption, which injects noise into past frames to reduce over-reliance on histories (Chen al., 2024; Guo et al., 2025). History corruption mitigates drift by narrowing the gap between self- generated and ground-truth context, but deprives the model of clean references and compromises temporal consistency. The second approach explores planning generation by first synthesizing dis- tant key frames and then interpolating intermediates (Zhang Agrawala, 2025; Xiang al., 2025). Anchoring distant frames to the initial context mitigates drift, but the introduced out-of-order sched- ule violates strict sequential emission, which is unsuitable for real-time streaming.
We design Rolling Forcing, an autoregressive long video generation technique that mitigates er- ror accumulation while maintaining real-time performance as illustrated in Fig. 2. Rolling Forcing comes with three new designs. First, instead of iteratively denoising single frame at a time as in most existing work, Rolling Forcing introduces rolling- denoising to process multiple consecutive frames simultancously. Within each window, frames connected by bidirectional at- tention and assigned progressively increasing noise levels. Such mutual refinement corrects local emors before any frame is finalized, thereby suppressing long-horizon drift. In addition, this design
2

---

Published as a conference paper at ICLR 2026
allows us to emit clean frame after each single forward pass, achieving real-time throughput on a single GPU despite a much larger attention window. Second, we adapt the attention sink mech- anism (Xiao et al., 2023) to the streaming video generation task, thereby strengthening long-term global consistency. Specifically, we persist the key-value states of the initial frames as a global context anchor and dynamically adjust their Rotary Position Embeddings (RoPE) (Su et al., 2024), which freezes the relative positions of initial frames to the current denoising frames and prevents excessive offsets. Note that the KV caching is applied to the recent clean frames as well re- duce latency and maintain temporal consistency. Third, we design an efficient training algorithm that enables few-step distillation over the extended denoising windows. This algorithm operates on non-overlapping windows that collectively cover all video frames, mitigating exposure bias by conditioning on self-generated histories during training. Extensive experiments show that Rolling Forcing achieves real-time streaming generation of multi-minute videos on single GPU, with sub- stantially reduced error accumulation as illustrated in Fig. I.
The contributions of this work can be summarized in three key aspects. First, we introduce rolling- window joint denoising technique that processes multiple frames in a single forward pass, enabling mutual refinement while preserving real-time latency. Second, we introduce the attention sink mech- anism into the streaming video generation task, a pioneering effort that enables caching the initial frames as consistent global context for long-term coherence in video generation. Third, we de- sign an efficient training algorithm that operates on non-overlapping windows and conditions on self-gen geff-generated histories, enabling few-step distillation over extended denoising windows and con- currently mitigating exposure bias.
2 RELATED WORK
Bidirectional Video Generation Models. Video generation has advanced rapidly in recent years, with modern approaches mostly adopting the paradigms of denoising diffusion. Video diffusion has been explored in both pixel space (Ho et al., 2022; Singer et al., 2022) and latent space (Blattmann et al, 2023b;a), with architectures evolving from early Tomee-Time U-Nets (Blattmann et al., 2023a; Hong et al., 2022) to more recent DiT-based designs (Peebles & Xie, 2023; Cupta et al., 2024). Significant industrial investment has driven the development of large video diffusion models, leading to several multi-billionn parameter models, including open-source models such as Wan (Wan et al., 2025) and Hunyuan (Kong et al., 2024), and closed-source models such Sora (OpenAI, 2024), Movie- Gen (Polyak et al., 2024), and Seaweed (Seawead et al., 2025). Notably, these models operate as bidirectional video diffusion models, they have access to both past and future frames during denoising. While this bidirectional context enables high-quality synthesis for offline generation, it incompatible with the causality that is necessitated in real-time streaming video generation.
Autoregressive Video Generation Models. To enable long video generation, several studies have extended the generation paradigm from bidirectional to autoregressive, which naturally supports gradual rollout over extended time horizons. Autoregressive models are typically trained with next- token prediction objectives and generate spatiotemporal tokens sequentially at inference time (Bruce et al., 2024; Kondratyuk et al., 2023; Wang et al., 2024; Weissenborn et al., 2019; Yan et al., 202 More recently, a separate line of research combines autoregressive modeling with denoising diffu- sion (Chen et al., 2024; Gu et al., 2025; Guo et al., 2025; Jin et al., 2024; Li et al., 2024; Liu et al., 2024; Weng et al., 2024; Yin et al., 2025; Zhang et al., 2025; Zhang & Agrawala, 2025; Huang et al., 2025; Henschel et al., 2025), where frames are generated one-by-one in an outer loop and each frame is gradually denoised in an inner loop. Within this family, Rolling Diffusion (Ruhe et al., 2024) and its variants (Kim al., 2024; Teng et al., 2025; Sun et al., 2025; Xie et al., 2025; Chen et al., 2025; Teng et al., 2025) merge the outer and inner loops: the diffusion model jointly denoises multiple frames at progressively increasing noise levels. However, these methods mostly suffer from exposure bias and error accumulation when generating long videos. Another line of re- search addresses error accumulation with planning generation (Long et al., 2024; Zhao et al., 2024; Hu et al., 2024; Xie et al., 2024; Zhang & Agrawala, 2025; Bansal et al., 2024; Yang et al., 2024; Xiang et al., 2025), which predicts distant future frames first and then interpolates the intermediate frames. While effective for reducing drift, it breaks the strict sequential order required for real-timo streaming. In contrast, our work enables much longer streaming video generation with minimal error accumulation while addressing exposure bias.
3

---

Published as a conference paper at ICLR 2026
Concurrent and Closed-Source Work. Two concurrent works are also devoted to the streaming generation of long videos. Specifically, StreamDiT (Kodaira et al., 2025) adopts the FIFO-style denoising (Kim et al., 2024) for streaming video generation. It modifies the pretrained model chitecture by introducing micro-steps and window attention, necessitating extensive additional pre- training with large-scale data and computation. In contrast, our method keeps the pretrained model architecture unchanged, can be trained efficiently in only 3,000 steps, and does not require any video data. APT2 (Lin al., 2025b) instead explores adversarial distillation (Lin et al., 2025a) for stream- ing video generation. It denoises videos block-by-blec and involves multiple costly post-training stages, including diffusion adaptation, consistency distillation, adversarial training, and long-vides training. APT2 is trained on one-minute-long videos, whereas ours is trained only on 5-second clips, yet can extend to multi-minute sequences during inference. Note that both StreamDiT and APT2 are closed-source and trained based on internal video diffusion models (Movie-Gen (Polyak et al., 2024) and Seaweed (Seawead et al., 2025)), while our model is trained on public datasets and relies on an open-source model (i.e., Wan2.1 (Wan et al., 2025)) as its foundation.
3 METHODS
3.1 PRELIMINARIES; EXPOSURE BIAS IN AUTOREGRESSIVE VIDEO DIFFUSION MODELS
An autoregressive video diffusion model is a hybrid generative framework that integrates autore- gressive chain-rule decomposition with denoising diffusion for video generation. Formally, given a sequence of N video frames z-:- (x xx ....,,"" ,x"), their joint distribution can be factorized using the chain rule: p(x1-N) = II p(a* x=)). Each conditional distribution p(æ* x²²) is modeled through diffusion process, where each frame is generated by progressively denoising Gaussian noise while conditioning on the previously generated frames. In practice, one may also generate chunk of consecutive frames instead of single frame at each step (Yin al., 2025; Teng et al., 2025; Huang et al., 2025). For clarity, we refer to each chunk simply as a frame in the following text.
Autoregressive video diffusion models are trained either (1) from scratch with frame-wise denois- ing loss or (2) by distilling pretrained bidirectional model. The first approach is trained under the paradigm of Teacher Forcing (TF) or Diffusion Foreing (DF) (Chen et al., 2024). In TF, the conditional distribution for the ith frame at noise level ł, is p(æ,, =²*). where all conditional his- tory frames are the ground-truth clean frames from the training data. While in DF, the conditional distribution is p(z;, x where the history frames are the ground-truth frames corrupted with independent noise levels. Since training relies on ground-trutiy histories while inference relies on the model's s own predictions, a train-test gap known as exposure bias arises (Schmidt, 2019). Mitigating the exposure bias is difficult because the denoising loss requires pairs of model predictions and the corresponding ground truth conditioned on them, which are unavailable.
The second approach of distillation, however, provides a way to bypass the denoising loss and mitigate exposure bias. CausVid (Yin et al., 2025) distills pretrained bidirectional model into few-step causal model. It adopts a Distribution Matching Distillation (DMD) loss (Yin et al., 2024b) that minimizes the reverse KL divergence across randomly sampled timesteps between the smoothed data distribution Pdata and the student generator's 's output distribution Pgen(xe). The gradient of the reverse KL can be approximated as the difference between two score functions:
VOLOMD A E2 (VoKI. (Pgen, [pduta.t)
E.c (((8aan (T (Go(c).)), - Sgen (V (Go(s).t),
dGole) dB
de at), (1)
where V represents the forward diffusion process, c is random Gaussian noise, Go is the generator parameterized by 0, and Saata and 8gen represent the score functions trained on the data and genera- tor's output distribution, respectively. Since training with DMD loss does not require ground-truth image or video data (Yin et al., 2024a), Self Forcing (Huang et al., 2025) mitigates the exposure bias by conditioning each frame previously self- ooff-generated histories during training. However, although exposure bias is alleviated, severe error accumulation still occurs once generation extends beyond the trained temporal window.
4

---

Published as a conference paper at ICLR 2026
Video frames
Denoising window
Denoising window w/o gradiant
paanliinggrrreeee pracess
Temporal conteot KV cache
Denalsing
Global context cache
Figure 3: Illustration the Rolling Forcing denoising process with = 4. Rolling Forcing jointly denoises short window of consecutive frames that assigned progressively higher noise levels and connected by bidirectional attention. The KV cache of recent frames is preserved as temporal context to maintain short-term consistency, while the KV cache of the initial frames is preserved as global context to ensure long-term consistency. During training, only a subset of denoising windows requires gradient computation, as highlighted by the red windows. These windows mutually exclusive yet collectively cover all video frames.
3.2 AUTOREGRESSIVE VIDEO GENERATION VIA ROLLING DIFFUSION WINDOW
In Self Forcing (SF), videos generated Crame-by-framm in strict causal manner. Consider noise schedule (1o = 0, (1,..., f1,...,44 /r = 1000} with total noise levels T + 1. At each denoising step i and frame index i, the model denoises an intermediate noisy frame st conditioned previous clean frames æ and then injects Gaussian noise with lower noise level into the predicted denoised clean frame via the forward diffusion process V. This produces noisy frame -i which will be used as the input the next denoising step. Formally, in SF, the denoising process is achieved v (Go (z) lj. x ()-1), and ætz ~ (O, I). However, this formulation has no
by: zi bidirectional attention between the current denoising frame x² and its history ~² where the strict causality forces every frame to inherit and compound the errors from its predecessors over time.
The proposed Rolling Foreing relaxes this constraint by extending the single-frame frame denoising win- dow into a rolling window spanning multiple frames, as illustrated in Fig. 3. Each denoising window contains consecutive frames with progressively higher noise levels in temporal order, akin Rolling Diffusion (Ruhe et al., 2024). The length of the denoising window Lwin set to the number of de- noising time steps, i.e., Lwin T. To ensure continuity. the next noise level of the ith frame is aligned with the current noise level of the (i 1)th frame, allowing the window to roll forward infinitely. each roll, clean frame is generated, and pure Gaussian noise appended as the next frame to be synthesized. Formally, for the denoising window starting at the ith frame, the denoising distribution Rolling Forcing can defined by:
Po
4++T-1 4+ <1 = V =V(Go(r LI.T, Do x00), 1o:T-1),
(2)
where 4++T-1 denotes the noisy frames the denoising window, and T 1 denotes the window "fr output with each frame denoised to a lower noise level. The generator Go predicts clean frames conditioned on the input noisy frames, their noise levels tı.r, and the clean history frames z <i injects Gaussian noise 6t at noise levels torr1 into the predicted clean frames, producing frames with reduced noise levels.
Since the length of the denoising window equals the number of denoising steps T, which is typi- cally large (i.e., 50) in video diffusion models (Wan et al., 2025), the denoising window itself becomes prohibitively large. To manage such large windows, previous work either processes every frame independently on multiple GPUs (Kim et al., 2024), reduces T' to 30 using few-step samplers (Xie et al., 2025). contrast, we adopt diffusion distillation (Yin et al., 2024b;a), which reduces the number of denoising steps T just 5 while preserving generation quality, thereby mak- ing the denoising windows compact enough to on a single GPU while maintaining real time latency.
5

---

Published as a conference paper at ICLR 2026
3.3 TEMPORAL AND GLOBAL HISTORY CONTEXT
As the clean history frames x accumulate dur- ing generation, handling them directly becomes computationally expensive. To address this, fol- lowing Huang et al. (2025), we cache the key and value states of the history frames, thereby avoiding redundant recomputation when gener- ating new frames, as illustrated in Fig. 3. Note that although the attention within the denoising window is bidirectional, the attention between the frames in the denoising window and the KV cache of history frames remains causal. While KV caching reduces computation, the computa- tional complexity still grows quadratically with the cache size as frames accumulate, and the cache may become large enough cause out-of- memory errors. Given a denoising window start-
Algorithm 1 Rolling Forcing Training
Require: Denoise timesteps (o.44....,,7 Require: Number of video frames N Require: AR diffusion model Ge (returns KV em- beddings via Gb
loop
2: Initialize model output X, <-
3:
Initialize KV cache KV < -0
Initialize 11:7-1 with Go
-1
5: Sample ~ Uniform
13
for L..... N do 1
7:
Sample ftr
(O
8:
Set t 1-647-11 < 21
Ftp
14T-1
Select and apply RoPE KV (Sec. 3.3)
10:
i (mod 7`) then
11:
Enable B computation
ing at the ith frame (er
f1:44 -1 we address this is-
12:
50 46447 +- Golat fI., KV)
sue by retaining only the KV cache of the most
13:
Xa. ap.pap(((a
recent Liem history frames = -- tempo-
14:
Disable gradient computation
ral context to preserve short-term temporal con-
15:
else
sistency. However, relying solely on short-term
16:
s6147 +- Gol fuT, KV) To
history causes a gradual drift of long-range prop- erties of the generated video (like exposure, color
17:
end if
18:
KV append (2 Lo. KV))
tone, white balance, etc.) as generation proceeds.
19:
4TT-1
2-1447-1 + ( 1a4+lat-111 (trr-1)
20:
end for
To maintain long-term global consistency, we
21: Update via DMD loss (Eq. (1))
cache the KV states of the initial Lgoo generated
22:
end loop
frames Zo I: as global context, analogous to at- tention sink tokens in streaming language models (Xiao 2023). The cache sizes Ltem and Lglo are chosen such that the total attention window size matches that of the bidircctional teacher model, i.., Ieem Ligho + Lwwnn Lbsidrectional- However, directly caching the initial frames leads to spilling problems. Modern video diffusion DiTs (Peebles & Xie, 2023) typically use RoPE (Su et al., 2024) for relative positional encoding. As the indices of the denoising frames i 7 relative distance to the initial cached frames grows, eventually exceeding the trained range of RoPE and producing unnatural artifacts. To resolve this, we cache the key states of the global context frames FD 1:L before applying the RoPE transformation. During generation, we dynamically apply RoPE to these cached key states at the effective indices i Ltem - Lglo :t Luem treating them as being positioned immediately before the temporal context frames = Iacm This adjustment preserves a fixed relative position w.r.t. the denoising frames, preventing excessive offsets.
increase, their
3.4 ROLLING FORCING POST-TR.B TRAINING
Rolling Forcing distills a pretrained bidirectional video diffusion model (Wan et al., 2025) to a few- step causal autoregressive generator using the DMD loss (Eq. (1)). As DMD matches the holistic distribution of the entire video sequence to the data distribution
(x I[pp (x 1:W )). the cal- culation of the DMD loss requires predicted clean video ai-F during training. In SF, the predicted clean video is generated by:
(=
2/ = Go t5, 1,1=1,2... 1,2,,...,
(3)
where j ~ Uniform 1, 1....,T T 1} indicates each frame's noise level t, before denoising. For Rolling Forcing, the denoising window consists of multiple frames different noise levels, select the jth frame in each window and combine the selected frames as the predicted clean video:
$ - d (5t (#t-+T-11 = Go T1 x
| i 1,2,
(4)
where j ~ Uniform (0, 1..... T T-11 1} represents both the frame's 's index within the denoising window and the frame's 'ss noise level 1y. However, Eq. (4) incurs T' times higher computational complexity

---

Published as a conference paper at ICLR 2026
Table I: Comparisons with relevant baselines. We compare Rolling Forcing with representative
open-source autoregressive video generation models.
Evaluation Scures f
Model
#Params
Throughuu Latency (FPS)
Quality Drin
(6)
Temporal Flickering Cansisnncy
Subject
Backgoound
Motion
Asshetic
lmuging Quality
Consistoncy
Quality
Non-Sreaming Pack (Zhang & Agrawala, 2025)
13B
0.92
65
99.26
91.65
93.55
99.03
59.93
65.20
3.45
Diffusion Forcing Causal
SkyReels-V2 (Chen et all, 2025)
1.3B
0.49|
112
97.43
89.23
93.45
98.76
61.55
62.90
5.59
MAGI-1 (Tong al., 2025)
0.19!
282
98.21
90.86
59.91
59.87
2.15
Distilled Cansol
Caus Vad (Yin et al., 2025)
1.3B
15.38
0.78
96.84
87.99
89,99
98.09
60.95
66.38
2.18
Self Forcing (Huang et al,, 2025)
1.3B
15.38
0.78
97.49
86.48
90.29
98.47
60.54
68.68
1,66
Rolling Forcing (Ours)
1.3B
15.79
0.76
97.61
92.80
93.71
98.70
62.39
70.75
0.01
Numbers adopted from Huang et (2025).
than Eq. (3), because the query size is T times larger. Given that DMD loss is already computation- ally expensive, this additional cost can easily lead to out-of-memory error even on GPUs with 80G of memory.
To address this issue, instead of backpropagating through every window (which requires gradients for each forward pass), we sample a subset of non-overlapping windows to construct the predicted clean video, as illustrated in Fig. 3. Gradient computation is performed only on these selected windows, significantly reducing memory usage while retaining effective supervision. Formally, the predicted clean video is given by:
{
LAiiTT-1 Go(121tT-1 Sh.T,26") 1i) j (mod T), 1SisN}.
Gg(
(5)
In each iteration, we reduce the number of forward passes requiring gradient computation from N in Eq. (4) to [N/T]] The Rolling Forcing training is il- lustrated in Alg. Similar SF, the input noisy frames 2 2 during training are generated by the model rather than taken from ground truth, thus mitigating the exposure bias. However, unlike Eq. (3) or Eq. (4), where every frame in the predicted clean video # W is denoised from the same noise level ty. the frames in Eq. (5) are denoised from varying noise levels t1.r. Consequently, frames denoised from different noise levels have different quality and clarity, leading to unnatu- ral video 2i:a and camera movement in DMD training. To address this issue, we adopt mixed training strategy that alternates between SF training (Eq, (3)) and Rolling Forcing training (Eq. (5)) with equal probability. The SF objective serves as a regularizer, encouraging the model to produce videos with natural camera movement. The inference adopts the Rolling Forcing paradigm alone as elaborated in Alg. 2.
where j ~ Uniform I, I,..., 1}.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Model. We implement Rolling Forcing with Wan2.1-T2V-1.33 (Wan et. 2025) as our base model, which generates 5s videos at 16 FPS with resolution of 832 x 480. Following Caus (Yin et al., 2025) and Self Foreing (Huang et al., 2025), we first initialize the base model with causal attention masking on 16k ODE solution pairs sampled from the base model. For both ODE initialization and Rolling Forcing training, we sample text prompts from a filtered and LLM-extended version of VidProM (Wang & Yang, 2024). We set T = 5 and perform chunk-wise denoising with each chunk containing latent frames. The model is trained for 3,000 steps with a batch size of 8 and a trained temporal window of 27 latent frames. We use the Adam optimizer for both the generator Go (learning rate 1.5 x 10-6) and the fake score 8gen (learning rate 4.0 x 10-7). The generator is updated every 5 steps of fake score updates.
Evaluation. We adopt the VBench (Huang et al., 2024) quality matrices to evaluate the generation quality over 200 randomly sampled MovieGen (Polyak et al., 2024) prompts, where the matrices
We omit the denoising windows at the start of the video for clarity in Eqs. (2) and (5), where the window has fewer than T frames. Gradient computation is still required if the window index satisfies = (mod T).
7

---

Published as a conference paper at ICLR 2026
Os
308
[-80s
t-90s
120s
Rolling Forcing (Ours)
Self Forcing
CausVid
Rkelsses-
MAGI-1
FruncPack
Rolling Foreing (Ours)
Fereing
CausVid
SkyReels-V2
MAGI-1
FramePack
Figure 4: Qualitative comparisons. We compare Rolling Forcing with representative open-source autoregressive video generation models long video generation.
measure multiple dimensions, including temporal flickering, subject consistency, background con- sistency, motion smoothness, aesthetic quality, and imaging quality. For fairness, all videos for quantitative evaluation are generated with the same length (30s), frame rate (16 fps), and resolution (832> x 480). To assess quality drift in long video generation, following Zhang Agrawala (2025); Yin et al. (2025), we compute the absolute difference imaging quality, Quality between first and the last 5 seconds of each video. The magnitude of A Quality directly reflects the severity of error accumulation. Following Huang et al. (2025), we evaluate real-time performance terms both throughput and latency. Unlike prior work that reports the first-frame latency, we measure latency after the generation process reaches a stable speed.
4.2 COMPARISONS
We compare Rolling Forcing against several relevant open-source video generation models of com- parable scale. Specifically, SkyReels- V2 (Chen et al., 2025) is trained under the Diffusion Forcing paradigm (Chen et al., 2024), which corrupts historical frames during inference to alleviate emror
8

---

Published as a conference paper at ICLR 2026
t=
t- 159
t-30s
Ours full
w/o RF inference
wio RF training
wlo training
w/o attention sink
Figure 5: Ablation studies on rolling diffusion window, mixed training strategy, and attention sink.
accumulation. MAGI-I (Teng et al., 2025) adopts a FIFO-style denoising paradigm (Kim et al., 2024) in both training and inference. We also compare against prior distillation-based approaches, including CiusVid (Yin et al., 2025) and Self Forcing (Huang et al., 2025). Note that SkyReels- V2, Caus Vid, Self Forcing, and our Rolling Forcing are all initialized from the same base model, 1-a2.1-TTV-11.31 (Wan et al., 2025). Additionally, we compare with the non-streaming image- to-video method FramePack (Zhang & Agrawala, 2025), which generates long videos in non- sequential order. We use the first frame generated by our method as the image input for FramePack.
As shown in Table 1, Rolling Forcing achieves the highest overall quality scores. In particular, Quality demonstrating its effectiveness in suppressing error accumula- obtains a substantially lower Aprit. tion. Qualitative comparisons in Fig. 4 further highlight that Rolling Forcing preserves high-fidelity and consistent video quality over 2 minutes of autoregressive generation, while the compared mod- els exhibit pronounced degradation, such as color shifts, artifacts, unnatural motion, etc. In addition, Rolling Forcing achieves real-time generation with sub-second latency, marginally faster than Self Forcing and Caus Vid, thereby establishing its suitability for long-horizon video streaming applica- tions. Notably, Rolling Foreing even outperforms the non-streaming method FramePack in quality, consistency, and drift control, despite FramePack being a 10 times larger image-to-videe model. As shown in Fig. 4, FramePack still suffers from severe error accumulation when generating long videos and tends to generate static outputs. Note that FramePack generates videos in reverse order, thus error accumulation is most severe at the beginning of the videos.
4.3 ABLATION STUDIES
Table 2: Ablation studies. RF refers to Rolling Forcing, and SF refers to Self Forcing.
We conduct ablation studies assess the contribution of several design op- tions, as summarized in Table 2.
Model
Evaluation Scores ?
Quality Drift
Rolling diffusion window. We eval- uate two variants: w/o RF infer- ence and w/o RF training. In w/o RF inference, we remove the rolling denoising window and adopt frame- by-frame denoising during inference, while keeping the same training procedure and model weights as full method. In w/o RF train- ing, the model is trained and inferred entirely with the frame-by- by-ne--b-framee paradigm. As shown in Fig. 5, both variants suffer from noticeable error accumulation within 30s, demonstrating that the rolling window is crucial for suppressing long-term drift.
Temp.
Subi
Back
Moc
Aes.
Img.
w/o inference
95.45
86.01
89.94
97.36
57.59
65.19
5.53
w/o RF training
95.91
87.50
90.86
98.05
60.41
69.24
0.89
w/o training
90.83
83.27
88.14
95.63
55.30
62.00
1.62
w/o attention sink
97.53
83.22
87.99
98.56
58.99
67.30
4.63
Ours full
97.61
92.80
93.71
98.70
62.39
70.75
0.01
Mixed training strategy. To assess its effect, remove the Self Forcing training objective (w/o SF training). As reported in Table 2, this leads to substantial degradation in consistency and overall quality, primarily due to unnatural camera motion.

---

Published as a conference paper at ICLR 2026
Attention sink. Finally, removing the global context frame (w/o attention sink) results in noticeable drift in the generated videos, as illustrated Fig.
CONCLUSION
We presented Rolling Forcing, framework for real-time long-horizon video generation that mit- igates eπror accumulation while sustaining sub-second latency. By introducing rolling- joint denoising strategy, Rolling Forcing enables mutual refinement across consecutive frames, ef- fectively reducing long-term drift. The integration of the attention sink mechanism further enhances global consistency by anchoring initial frames as persistent context, while our efficient training al- gorithm enables few-step distillation over extended denoising windows while mitigating exposure bias. Extensive experiments demonstrate that Rolling Forcing achieves state-of-the-an temporal co- herence and visual fidelity over multi-minute streaming sequences, significantly outperforming prior streaming approaches both quality and efficiency.
ACKNOWLEDGEMENT
This project is funded by the Ministry of Education Singapore, under the Tier-1 project scheme with project number RT18/22.
REFERENCES
Hritik Bansal, Yonatan Bitton, Michal Yarom, Idan Szpektor, Aditya Grover, and Kai-Wei Chang. Tale: Time-aligned captions for multi-scene text-to-video generation. arXiv preprint arXiv:2405.04668 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets, arXiv preprint arX1v:231..55127, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhom, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22563-22575, 2023b.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder. Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-furst International Conference on Machine Learning, 2024.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neurul Information Processing Systems, 17:224811-24123 2024.
Guibin Chen, Dixuan Lin, Jiangping Yang. Chunze Lin, Junchen Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengcheng Ma, et al. Skyreels-v2: Infinite-length film generative model. arXiv preprint arXiv:2504. /374,, 2025.
Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503. 19325, 2025.
Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua Lin, and Lu Jiang. Long context tuning for video generation. arXiv preprint arXiv:2503.10558 2025.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, pp. 393-411. Springer, 2024.
Roberto Henschel. Levon Khachatryan, Hayk Poghosyan, Daniil Hayrapetyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. In Proceedings of the Computer Vision and Patlern Recognition Conference, pp. 2568-2577, 2025.
10

---

Published as a conference paper at ICLR 2026
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02.003 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
Panwen Hu, Jin Jiang, Jianqi Chen, Mingfei Han, Shengcai Liao, Xiaojun Chang, and Xiaodan Liang. Storyagent: Customized storytelling video generation via multi-agent collaboration. arXiv preprint (rxiv:2411.04990 2024.
Xun Huang. Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506. 2025.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang. Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:24 2024.
Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo-diffusion: Generating infinite videos from text without training. Advances in Neural Information Processing Systems, 37: 89834-89868, 2024.
Akio Kodaira, Tingbo Hou, Ji Hou, Masayoshi Tomizuka, and Yue Zhao. Streamdit: Real-time streaming text-to-video generation. arXiv preprint arXiv:2507.03745, 2025.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chane Chiu, et al. Videopoet: A large language model for sero video generation. arXiv preprint arXiv:2312.14125, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. 2024.
Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, and Furu Wei. Arlon: Boosting diffusion transformers with autoregressive models for long video generation. arXiv preprint arXiv:2410.20500 2024.
Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversar- ial post-training for one-step video generation. arXiv preprint arXiv:2501.08.. 250.00836... 2025a.
Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao Xuefeng Xiao, and Lu Jiang. Autoregressive adversarial post-training for real-time interactive video gen- cration. arXiv preprint arXiv:2506.0933: 2025b.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamen Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXfv:2210.02747, 2022.
Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, et al. Mardini: Masked autoregressive diffusion for video generation at scale. arXiv preprint arXiv:2410.2000 2028000 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 03003, 2022.
Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei. Videostudio: Generating consistent-content and multi-scene videos. In European Conference on Computer Vision, pp. 468-485, Springer, 2024.
OpenAl. Sora. https: /ttp///oppeaaa/aaaeeeeeeee comm/ora, 2024.

---

Published as a conference paper at ICLR 2026
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of 2023.
the IEEE/CVF international conference on computer vision, pp.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching Yao Chuang, al. Movie gen: A cast of media founda- tion models. arXiv preprint 10. 13720, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PmLR, 2021.
David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. In International Conference on Machine Learning, 2024.
Florian Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint 1rXiv:1910000292, 2019.
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective training of video generation foundation model. arXiv preprint 2XXv:2504008685 2025.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.
2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Mingzhen Sun, Weining Wang. Gen Li, Jiawei Liu, Jiahui Sun, Wanquan Feng, Shanshan Lao, Si Zhou, Qian Hc, and Jing Liu, Ar-diffusion: Asynchronous video generation with auto-regressive diffusion. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7364- 7373, 2025.
Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, WQ Zhang, Weifeng Luo, al. Magi-1: Autoregressive video generation at scale. arXiv preprint arXiv:2505.13211, 2025.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837, 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojic Mao, Chen-Wet Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint wwXXvv2503.203111 2025.
Wenhao Wang and Yi Yang. Vidprom: A million-scal real prompt-gallery dataset for text-to-video diffusion models. Advances in Neural Information Processing Systems, (1666661166644 2024.
Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint 17X0v22410.02757, 2024.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprini arXiv: 1906.02634, 2019.
Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, et al. Art-v; Auto-regressive text-to-video generation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7395-7405, 2024.
Xunzhi Xiang, Yabo Chen. Guiyu Zhang, Zhongyu Wang, Zhe Gao, Quamming Xiang, Gonghu Shang, Junqi Liu, Haibin Huang. Yang Gao, et al. friro-rroommcro planning for high-quality and parallelized autoregressive long video generation. arXiv preprint arXiv:2508.0333 033334 2025.
12

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.1744 2023.
Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, and Yang Zhou. Progressive autoregressive video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 6322-6332, 2025.
Zhifei Xie, Daniel Tang, Dingwei Tan, Jacques Klein, Tegawend F Bissyand, and Saad Ezzini. Dreamfactory: Pioneering multi-scene long video generation with multi-agent framework. arXiv preprint arXiv:2408. 117848888.878,,, 2024.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 2021.
Dingyi Yang, Chunru Zhan, Ziheng Wang, Biao Wang, Tiezheng Ge. Bo Zheng, and Qin Jin. Syn- chronized video storytelling: Generating video narrations with structured storyline. arXiv preprint MrXiv:2405.140400 2024.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455-47433 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman. Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 6613-6623, 2024b.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang, From slow bidirectional to fast autoregressive video diffusion models. In Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp. 22963-22974, 2025.
Lvmin Zhang and Maneesh Agrawala. Packing input frame context next-frame prediction models for video generation. arXiv preprint arXiv:2504.1 12626. 2025.
Tianyuan Zhang, Sai Bi, Yicong Hong. Kai Zhang, Fujun Luan, Songlin Yang. Kalyan Sunkavalli, William T Freeman, and Hao Tan. Test-time training done right. arXiv preprint arXiv:2505.23884, 2025.
Canyu Zhao, Mingyu Liu, Wen Wang, Weihua Chen, Fan Wang, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequence. arXiv preprint arXiv:2407.16655 2024.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL IMPLEMENTATION DETAILS
KV Cache. We configure the KV cache and de- noising window sizes as Liem 3, Lggo- = 3, and
Algorithm 2 Rolling Forcing Inference
Twin 15 latent frames. When updating the KV cache with GKY, the attention window attends only to recent frames, excluding the global con- text. This design reflects that, apart from the ini- tial frames serving as the global context anchor, other cached frames are retained solely for pre- serving short-term temporal consistency. During inference, we persist the KV states of the global context frames while discarding obsolete tempo- ral frames, thereby maintaining constant memory usage. At the start of the video, when the denois- ing window still includes the first frame, no tem- poral or global context is used.
Require: Denoise timesteps (fa, f1,..., tr Require: Number of video frames N Require:
AR diffusion model Go (returns KV em- beddings via Gb
1: Initialize model output X. <-
2: Initialize KV cache KV +
3: Initialize 2/1.7 11:-11 with Go
4: for i = 1,........
114 do
5: Sample x tT
~ (( 1)
6: Set 11.
1:447-11 t -1TT-1 2 || -4
7: Select and apply RoPE to KV (Sec. 3.3)
8:
a + Ga 5it bu. KV) 2o
9:
X. apppend((tt)
10: KV appana (Gb"
( (Øb. lo KV))
11:
x ++ 1+44-- + V ( la+
(rr-1)
Training. During training, the number of gen- erated frames N is randomly sampled between 21 (the sequence length of the bidirectional teacher
12: end for
model) and 27 latent frames. The DMD loss is computed on the last 21 frames. Since in 1 (Wan et al., 2025) the first VAE-encoded frame is not temporally compressed and thus exhibits different statistics, we decode frames O:N 21 to RGB and re-encode the (N-21) frame into the latent space. This re-encoded frame is then concatenated with latent frames N N-21:N-1 for loss computation. In this way, the first frame is only spatially compressed, ensuring consistency with the statistical distribution of the bidirectional teacher model.
Noise schedule and model parameterization. Following the Wan2.1 I and Self Forcing, we adopt the flow matching framework (Lipman et al., 2022; Liu et al,, 2022), with time step shifting f' ((k, t) = (kt/ 1000) / + (k- ()(t(1000)) 1000 and a shift factor k = 5. The forward process is specified as xt
1000 x + 1000 C, ~ (0, I) with t E [0, 1000]. The data prediction model is given by:
Ge(x, t, c)
Cakip E Cout 08 (Cin It, Croise (t)..).
(6)
We keep the preconditioning coefficients the same as the base models' configuration, i.e., Cskip Gin = Cout = 1 and Croise (t) = f. Our few-step diffusion process employs a uniform 5-step schedule [ts, /4, 13, 12, [1000, 800, 600, 400, 200). We adopt 5-step schedule rather than 4, our method achieves comparable and even slightly faster generation speed than 4-step Self Forcing.
71, 75, 79, 84, 88, 92, Indices of the 200 sampled MovieGen prompts. 0, 10, 15, 24, 30, 34, 38, 44, 48, 53, 60,
98. 103, 108, 112, 116, 122, 126, 131, 137, 142, 146, 150, 157, 165, 171, 176, 182, 188, 196, 200, 207, 211, 215, 219, 225, 229, 233, 237.
242, 246, 250, 256, 261. 267, 272, 277. 284, 288, 294, 299, 303, 308, 312, 317, 321, 327, 331, 336, 340. 344, 348, 353, 357, 362, 367, 372.
376, 380, 388, 392, 396, 400, 408, 415, 423, 428, 433, 437, 446, 452, 456, 460, 469, 473, 477. 482, 491, 495, 502, 507,
515, 521, 525, 529, 533, 540, 544, 548, 553, 558, 569, 574, 578, 585, 590, 598, 602, 609, 614, 619, 626, 632, 636, 641, 647, 651, 657, 661,
666, 671, 677, 681, 686, 690, 695, 699, 704, 708, 712, 717, 722, 726, 730, 734, 739, 743, 747, 752, 756, 761, 766, 772, 776, 781, 786,
795, 799, 803, 808, 812, 816, 820, 825, 829, 834, 838, 845, 849, 855, 860, 865, 870, 875, 880, 884, 888, 892, 897, 904, 908, 915, 924, 928.
933, 937,9 942, 946, 954, 959, 964, 970, 976, 980, 986, 991, 996.
B INTERACTIVE VIDEO STREAMING
In Fig. 6, we demonstrate that Rolling Forcing enables interactive video streaming, allowing users to modify prompts during generation to steer the video content. Implementing this functionality is straightforward:
discard the cross-attention cache of previous text prompts and apply the new prompts in cross-attention. Rolling Forcing is able to perform various changes including subject switch, subject introduction, scene switch, and action switch.
14

---

Published as a conference paper at ICLR 2026
Subject Switch
A horse is rumming in desert. -> Tiger is rumming in desert.
A Skeleton dancing. >> man is daincing.
Subject Introduction
vast mountain valley at morning. sleek Futuristic spacecraft descends over the vidge.
PMPPy riverbank peobles beside >> little zirl light blue the dog.
Scene Switch
A young male longboarder accelerating downhill -> The longboarder glides inéo the forest.
Skier racing down a Steep slope. +> Stars and moons swirting Bround the skier,
Action Switch
An older man playing the piano, >> The old wen eating burger.
young woman with bright Smile, -> The woman is waving her hands enthusiastically
Figure 6: Interactive Video Streaming. Rolling Forcing allows the users to change prompts while streaming to steer the video content.
C EVALUATIONS ON 2-MINUTE VIDEOS
To further evaluate our method's performance in long video generation, we compare it with real- time baselines, iee. CausVid (Yin et al., 2025) and Self Forcing (Huang et al., 2025), on 2-minute videos. As shown in Table 3, Rolling Forcing demonstrates more pronounced advantage in this setting, outperforming both baselines across nearly all quality metrics and exhibiting significantly lower quality drift. We also analyze the CLIP score (Radford et al., 2021) for each time segment over the 2-minute generated videos in Fig. 7. The results show that Rolling Forcing achieves the highest scores across segments, with the smallest score drop time.
15

---

Published as a conference paper at ICLR 2026
Table 3: Evaluations 2-minute videos. We compare Rolling Forcing with real-time baselines and w/o RF training variant.
Model
Evaluation Scores 1
Quality Drift
Temp.
Subj.
Back.
Mot.
Acs.
Img.
Dyn.
Caus (Yin et al., 2025)
96.67
84.69
89.53
98.04
62.16
63.62
52.08
3.35
Self Forcing (Huang et al., 2025)
97.44
71.95
88.73
98.19
50.66
60.03
51.02
14.4
Ours w/o RF training
96.87
84.38
91.93
98.39
58.61
66.55
56.75
1.39
Rolling Forcing (Ours)
96.90
91.47
95.29
98.29
65.21
68.96
57.14
0.49
To further demonstrate the effectiveness of the rolling denoising window strategy, we conduct an ab- lation study on 2-minute videos by removing Rolling Forcing (RF) during training, where the model is trained and inferred entirely under the frame-by-frame paradigm. As shown by the quantitative metries in Table 3, removing the rolling denoising window significantly harms consistency, quality, and drift control. Qualitative comparisons in Fig. 8 also confirm that the rolling denoising window substantially reduces error accumulation, leading to fewer artifacts in long video generation.
Segment
3.29
0.28
CauVVid Sal Forcing Our
Figure 7: CLIP scores comparisons on 2-minute videos.
D VBENCH SCORES ACROSS ALL DIMENSIONS
We conduct a comprehensive evaluation on the full VBench benchmark (Huang et al., 2024) on 30s video clips, using all 946 prompts and covering all 16 metrics reported in Tables 4 and 5. For detailed metric definitions, we refer readers to the VBench paper. All values are computed with the official standardized evaluation scripts. Our method achieves substantial improvements in overall quality, particularly in frame-wise fidelity, and also outperforms distilled baselines on semantic scores.
Table 4: Full VBench quality evaluation on 30s videos.
Model
Subject Consistency
Background Temporal Consistency Flickering
Motion Smoothness
Dynamic Aesthetic
Imaging Quality Quality
Degree
Quality
Score
CausVid (Yin et al., 2025)
89.50
90.00
99.41
98.06
63.88
61.82
65.30
80.89
Self Forcing (Huang et al., 2025)
88.61
89.53
98.90
98.57
68.05
60.60
68.98
81.39
Rolling Foreing (Ours)
94.80
95.69
98.93
98.63
60. 14
62.81
72.31
84.08
16

---

Published as a conference paper at ICLR 2026
t=
t-30s
t=60s
(-90s
t 120s
full
wo RF training
Ours full
w/o RF truining
Figure 8: Ablations on Rolling Foreing. Removing the rolling denoising window significantly harms consistency, quality, and drift control.
Table 5: Full VBench semantic evaluation on 30s videos.
Model
Object Multiple
Color
Sputial Relationship
Scene
Temporal
Appearance
Semantic
Class
Objects
Style
Style
Consistency
CausVid (Yin et al., 2025)
78.56 58.84 81,00 81.02
59.62
31.32
22.51
20.04
23.16
65.85
Self Forcing (Huang et al., 2025)
80.06 62.88
83,00 79.80
74.76
30.59
23.78
20.41
24.80
69.17
Rolling Forcing (Ours)
85.92 64.86 73.00 88.16
78.84
30.52
23.52
19.45
24.56
69.78
E ANALYSIS OF ATTENTION SINK
To better understand the mechanism of the attention sink in streaming video generation, we visualize the model's attention maps Fig. 9. The visualizations reveal that early layers exhibit local attention patterns, while deeper layers allocate more attention weight to the global context. This indicates that the attention sink tokens function effectively by providing essential information to the denoising window.
The reasons for the attention sink phenomenon are complex. Similar to findings LLMs (Xiao et al., 2023), identify several possible explanations for its emergence in streaming video gener- ation: 1) Inherent statistical difference: The latent representation of the first frame has inherently different statistics, is only spatially compressed, unlike subsequent frames which also tem- porally compressed. 2) Information retrieval: The model learns to use the global context as a sink retrieve important information such color tone, white balance, etc. 3) Softmax constraint: As in LLMs, the softmax function requires attention weights to sum to one. The initial tokens may thus serve sink to absorb residual attention weight. This is complex and promising research direction that extends beyond the scope of this work, and we encourage further investigation by the community.
F DYNAMIC ROPE
The placement of RoPE indices for the global context frames critical. Suppose the indices of the denoising window are i:i 7-1, and the temporal context frames are i TTem:i- 1. We investigate several options for assigning indices to the global context frames:
1. immediately preceding the temporal context, Tanem design);
Liglo Tuem -1 (our adopted
2. fixed at 0: without dynamic RoPE adjustment;
3. overlapping with the temporal context, ig 1;;
4. within the denoising window, within d:i + T-1;
17

---

Published as a conference paper at ICLR 2026
Global Context
Globel Context
Globel Context
Attention Map (15x23)
humplaaal Altenton Map (15x22)
Trareeeecel Aitention Hap
$3.9
06
12.0
125
140
13
Layer Head 0
Layer 13 Head 0
Layer 22 Head 0
Figure 9: Visualization of the average attention logits.
5. after the denoising window, beyond T.
Empirically, option produces strong "jumping" artifacts due relative positions exceeding the trained offset range, as shown Fig. 10. Option 3 introduces flickering, as the model confuses global and temporal contexts. Option 4 collapses into static outputs, since the generated frames forced to replicate the global context. Option induces unnatural motion, as the model attempts to converge toward the misplaced global anchor, Among these, only option I yields consistent videos with minimal artifacts.
Framo i-1
Framo
Frame i+1
Figure 10: Fixed RoPE indices produce strong "jumping" artifacts, where the video abruptly resets to the initial frame during streaming.
G HUMAN PREFERENCE STUDIES
We conduct human preference studies where users select their preferred method based error accumulation control and overall video quality. As shown in Fig. 11, our Rolling Forcing receives significantly votes than all other methods, demonstrating its superior generation quality as perceived by human evaluators.
H DISCUSSIONS OF DMD TRAINING
Training efficiency and quality of subset gradient computation. Although would like to conduct an ablation study on subset gradient computation to directly assess its impact on training quality, it is infeasible to remove entirely. The use of subset gradient computation allows Rolling Forcing to trained on 80G GPUs; without it, the theoretical VRAM requirement would be ap- proximately 5 x 80 400G, which impractical to train. Nevertheless, as shown in Table 3,
18

---

Published as a conference paper at ICLR 2026
Less Error Accumulation
Overall Quality
Rolling Foraing (Ours) Self Forcing
Rolling Forcing (Ours) Self Forcing
SkyReels-V2
FramePack
SkyRoela-V2
FramaPack
Figure 11: Human preference studies.
our method significantly outperforms the baselines, demonstrating its high quality. In terms of effi- ciency, our method does require more training iterations than Self Forcing (3000 vs. 600). However, the increased training time has negligible impact on the user experience.
Pros and of subset gradient computation. The primary advantage of subset gradient com- putation is that it enables DMD distillation on the extended rolling denoising windows. The main disadvantage train-test discrepancy: during training, videos are composed of subsets of win- dows, whereas during inference, the final output videos are composed of the first frames from all windows.
Mixed training strategy. Our mixed training strategy involves alternating between Self Forcing (SF) and Rolling Forcing (RF) with equal proba- bility. The model architecture and parameter sizes are identical for both strategies, utilizing the same global and temporal context sizes. The sole dis- tinction lies in the generation process: SF produces videos frame-by-franee whereas RF uses rolling denoising window. Training is stable, as evi- denced by three runs with different random seeds; both the quality drift and quality score remain approximately constant, shown in Table 6.
Table 6: Quality Metrics Across Runs
runl
run2
run3
Quality Drift
0.01
0.04
0.03
Quality Score
84.08
83.24
85,83
LIMITATIONS
While Rolling Forcing substantially suppresses error accumulation in real-time streaming video generation, several limitations remain. First, although the global context helps stabilize long-horizon consistency, frames generated in the middle are discarded once they leave the temporal context. As result, the model retains no memory mid-sequence content, suggesting that incorporating more advanced memory mechanisms is a promising direction for future exploration. Second, training Rolling Forcing is computationally demanding: the enlarged attention window and the DMD loss significantly increase GPU memory usage, which may limit scalability to higher-capacity models. Developing more efficient training or distillation strategies to mitigate these costs therefore an important avenue for future work. Third, mentioned in Huang et al. (2025), the rolling diffusion window may increase latency in interactive applications, as future frames are partially pre-generated before the current frame is finalized. As Rolling Forcing natively supports both inference strategies, future work may consider a mixed inference strategy that interactively switches between frame-by- frame denoising during interaction and rolling denoising otherwise.
19

---

Published as a conference paper at ICLR 2026
J BROADER SOCIETAL IMPACT
This work introduces real-time, long-horizon text-to-video generation, which can broaden access interactive media, live storytelling, and educational tools by enabling continuous and responsive video synthesis. However, the ability to produce realistic long-duration content in real time also heightens risks of misuse, such as generating misleading live streams or amplifying harmful biases over extended outputs. We encourage future research to explore safeguards, including content filter- ing, bias mitigation, and responsible deployment practices, ensure these capabilities are used for positive impact.
20


Reviews:
[{"review_id": "ofWw0qU9zl", "reviewer": "Reviewer_PmgV", "summary": "This paper proposes \"Rolling Forcing,\" a method designed to alleviate error accumulation in long-horizon video generation. It uses a rolling denoising window and introduces an attention sink mechanism to improve temporal coherence and achieve real-time rendering through few-step distillation. The paper includes detailed pseudocode and extensive engineering contributions, clearly explaining the training process.", "questions": "- Could the authors discuss how well their approach handles scenarios with significant changes in video content?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed attention sink clearly helps alleviate error accumulation issues, as demonstrated in Figure 4.\n- Few-step distillation allows real-time rendering, making the approach practical for streaming video generation.\n- The paper includes substantial engineering work, and the provided pseudocode clearly describes the training process in detail.", "weaknesses": "- A significant limitation of the proposed approach is that the use of the attention sink and Rolling Forcing may not generalize well to long videos with drastically changing content. This setup could restrict substantial changes in video content, causing generated videos to appear as repetitive sequences or stitched clips (as evident in all the provided supplementary videos). Additionally, continuity issues arise when key frames do not provide rich enough information.\n- The evaluation is somewhat limited, relying only on metrics from VBench. A broader evaluation would strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CdMvou3Rl9", "reviewer": "Reviewer_zgDS", "summary": "This paper proposed a method to generate long video without drifting error and enables streaming video. Instead of causal diffusion, it uses progressive noise level for denoising. It utilizes attention sink to keep long term video consistency and quality. Finally it proposed a few-step distillation method over extended window to mitigate exposure bias conditioned on self-generated histories.", "questions": "- How does attention sink perform for long video with context dramatically changed?\n- It is not clear how the mixed training strategy (SF + Rolling Forcing) performed practically. Besides, is this training strategy stable? It would be helpful to add a more detailed discussion of it.\n- Since RoPE is very sensitive, does dynamically changing RoPE for attention sink bring any artifacts or instability of the output videos?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- introducing rolling diffusion, i.e., progressive noise level for multiple video latents, into self-forcing pipeline to achieve smoother and lower error drifting denoising framework.\n- history frame use causal attention so as to be saved in KV cache to save inference compute.\n- adopted attention sink with a few global context at the beginning of the sequence. Adjust RoPE dynamically to avoid PE index out of range.\n- novel design of DMD loss on progressive noised latent to save computation cost.\n- the error drifting problem has been improved significantly.", "weaknesses": "- the design of using a subset of backpropogated gradients for DMD loss training alleviate the memory and compute cost. But has the training quality and efficiency been affected at the same time? For example, it is mentioned that the original SF training is still needed to regularize the training. It would be helpful to have a discussion about advantage and disadvantage of it since this is one of the core contribution.\n- the overall design resolves the error drifting significantly, but other metrics, e.g., fps, latency, VBench score, are not improved or improved marginally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0FyFb94U6", "reviewer": "Reviewer_M4Wp", "summary": "This paper introduces Rolling Forcing, a novel framework for real-time autoregressive long-video diffusion. The key idea is to mitigate error accumulation in streaming generation by performing joint denoising over a rolling window of consecutive frames rather than processing single frames sequentially. Three main contributions are proposed: (1) a rolling-window joint denoising strategy enabling mutual refinement across neighboring frames while keeping real-time throughput; (2) integration of an attention-sink mechanism that anchors initial frames as a global context to maintain long-term temporal consistency; and (3) an efficient few-step distillation algorithm over non-overlapping windows that conditions on self-generated histories to reduce exposure bias.\nExperimental results show that Rolling Forcing can stream multi-minute videos at 16 FPS on a single GPU with minimal drift, achieving state-of-the-art performance on the VBench benchmark and outperforming prior autoregressive baselines such as Self Forcing and CausVid in both visual quality and temporal stability.", "questions": "1. What motivates the focus on long video generation, and how is this challenge tackled?  \n2. How does this paper's contribution distinguish itself from the limitations of related works?  \n3. How do the results compare to FramePack?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Rolling forcing is a robust and effective framework. Components like progressively increasing noise levels, attention sink, and few-step distillation enhance long video generation.  \n2. Compared to baselines like causvid and self-forcing, rolling forcing significantly reduces error accumulation and extends video length while maintaining content consistency.", "weaknesses": "The weaknesses primarily stem from the novelty and experiments, as outlined below:  \n1. The proposed components are not entirely novel, as they have been used in video generation. For example, progressively increasing noise is used in Rolling Diffusion[1], MAGI-1[2], and FIFO-Diffusion[3]. Attention sink has been applied in LLMs[4], and step distillation is utilized in self-forcing[5]. This paper seems to integrate these components into an autoregressive video generation paradigm.  \n2. The paper does not compare its approach to related works in autoregressive video generation, such as FramePack[6], which it cites.  \n3. The training length in the paper is set to 27, with a KV cache length of 24 frames. It raises questions about how the attention sink performs with such a short training length.  \n4. In Table 2, the second row (without RF training) achieves strong performance, though still lower than rolling forcing. This raises the question of the key benefits of RF training.\n\n[1] Rolling Diffusion Models.\n[2] MAGI-1: Autoregressive Video Generation at Scale.\n[3] FIFO-Diffusion: Generating Infinite Videos from Text without Training.\n[4] When Attention Sink Emerges in Language Models\n[5] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion\n[6] Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cuI0ni1Y0u", "reviewer": "Reviewer_FrmR", "summary": "The paper introduces Rolling Forcing (RF), a streaming text-to-video diffusion method that reduces long-horizon error accumulation while keeping real-time throughput. Key ideas: (i) a rolling denoising window that jointly denoises several consecutive frames with bidirectional attention inside the window but emits one clean frame per forward pass; (ii) a global/temporal KV cache with dynamic RoPE so early frames act as a stable “attention-sink” anchor over minutes; and (iii) few-step DMD distillation on non-overlapping windows plus a mixed Self-Forcing (SF) objective for more natural motion. On VBench, RF improves quality and notably lowers quality drift vs. open AR baselines, with ablations for each component.", "questions": "Please see weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Effective drift control: Rolling window enables mutual refinement before a frame is finalized; ablations show large Quality_Drift gains.  \n Long-term consistency trick: Global KV cache + dynamic RoPE keeps initial context usable without RoPE over-offset artifacts. \n Real-time demo & interactivity: Sub-second steady-state latency and prompt-switching during streaming.", "weaknesses": "* Attention-sink rationale needs deeper evidence: “Attention sink” comes from LMs; here it’s adapted via KV caching + dynamic RoPE. The paper shows ablations, but provides limited analysis of why the sink phenomenon should emerge in video DiTs. As written, it risks reading as a concept transplant without sufficient mechanistic support.  \n* Motion/scene diversity trade-off: Demos suggest mis-tuning can collapse into static outputs, RF may bias toward stillness to curb drift. More systematic tests compared to baselines on large motion and scene changes are needed, you may use like dynamic degree metric in Vbench.  \n* Evaluation scope: Metrics center on 30 s clips (quality drift measured 0–5 s vs. last 5 s). Given the multi-minute claim, add standardized 2–5 min quantitative runs (and a small human preference study).  \n* Scaling detail: Limited profiling across window/model sizes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)