Title: Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video generation

Autoregressive models

Diffusion models

World models
Abstract: We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models. It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs during inference. Unlike prior methods that denoise future frames based on ground-truth context frames, Self Forcing conditions each frame's generation on previously self-generated outputs by performing autoregressive rollout with key-value (KV) caching during training. This strategy enables supervision through a holistic loss at the video level that directly evaluates the quality of the entire generated sequence, rather than relying solely on traditional frame-wise objectives. To ensure training efficiency, we employ a few-step diffusion model along with a stochastic gradient truncation strategy, effectively balancing computational cost and performance. We further introduce a rolling KV cache mechanism that enables efficient autoregressive video extrapolation. Extensive experiments demonstrate that our approach achieves real-time streaming video generation with sub-second latency on a single GPU, while matching or even surpassing the generation quality of significantly slower and non-causal diffusion models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Self Forcing, a post-training approach for autoregressive video diffusion models. The key idea is to simulate inference-time behavior during training by letting the model condition on its own generated frames rather than ground truth. This helps close the gap between training and inference, which has long been a challenge due to exposure bias. It combines few-step diffusion and stochastic gradient truncation to efficiently expose models to their own prediction errors and optimize a holistic video-level distribution-matching objective.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This paper is the first to introduce autoregressive self-rollout in video diffusion post-training, fully aligning training and inference distributions to eliminate exposure bias.\n2. It uses holistic video-level distribution-matching losses (DMD/SiD/GAN)， so the model learns to correct its own residual errors, improving both visual fidelity and temporal consistency.\n3. It is intuitive why this works: by matching the model’s output to the true distribution at the video (holistic) level, the method enforces a global constraint on cumulative errors.\n4. This paper also demonstrates broad success across VBench, human preference studies, and efficiency benchmarks, outperforming both autoregressive and bidirectional diffusion baselines.\n\n**Weaknesses**\n1. Gradient truncation (only backpropagating the final denoising step) and detaching historical KV embeddings save memory but may weaken the model’s ability to capture long-range, cross-frame dependencies.\n2 .Comparison of DMD, SiD and GAN objectives is limited; the paper does not clearly identify which loss is best for which scenario.\n3 .Key hyperparameters (diffusion steps, truncation point, KV-cache size) are highly sensitive, yet the paper offers little guidance on their tuning.\n\nQUESTIONS:\n1. Did the authors experiment with applying varying noise intensities to individual frames before denoising and then enforcing a holistic distribution-matching constraint?  How do different noise levels impact the mitigation of cumulative errors?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents self-forcing, a diffusion framework for long video generation. Different from prior approaches such as diffusion forcing or teacher forcing, self-forcing aims to remove the gap between training and inference by explicitly doing rollout during training. To reduce the computation and memory burden for this training strategy, the paper introduces several tricks to mitigate this, e.g., using few-step diffusion models or only doing backpropagation for a single random timestep per frame. The paper also adapts the KV caching trick (which has been popular in LLMs) by for long video generation. By incorporating all of these compoments, the paper shows self-forcing can generate long videos in real-time with high quality, outperforming prior long video generation methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is generally well-written and easy to follow.\n- The paper is very well motivated, as the length limit of existing video generation models is one of the biggest bottlenecks for them to be extended to real-world videos.\n- Removing the discrepancy between training and inference is important and a reasonable approach. \n- Some components seem technically novel (e.g., how to handle gradients).\n\n**Weaknesses**\n- There are very limited qualitative comparisons, making it difficult to know how the proposed method generates better long video compared with other baselines. In particular, I wonder how self-forcing can be qualitatively better than diffusion-forcing or teacher-forcing but I cannot find qualitative comparisons. Could the authors provide more comparisons in general (like comparison with other baselines, eg, CausVid) as well as comparison with DF or TF? \n- I think real-time generation seems great, but the provided examples do not seem to be long, as they are just 10 seconds videos. \n- Minor: [1] also deals with chunk-wise autoregressive long video generation and exposure bias but the paper misses discussion. \n- Minor: Could the authors provide memory consumption in training and inference? \n\n[1] Yu et al., MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation, 2025\n\nQUESTIONS:\nPlease see my weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Self-Forcing, a fine-tuning technique to enable autoregressive generation in video diffusion models. The method starts with fine-tuning the base model with causal masking to initialize it with some autoregressive capability. Then it follows with training on autoregressively self-generated data using distribution matching losses to make sure the generated chunks form a plausible temporally consistent video as a whole. KV caching and few-step distillation are used to speed up the training and the inference with the model. Self-forcing achieves better results than the prior work on VBench and human preference.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The motivation and the method itself are clearly presented. \n2. Applying KV caching to video models is a meaningful and valuable contribution.\n3. The quantitaive results are solid.\n\n## Weaknesses\n\n1.The proposed Self-Forcing method is essentially a distillation or fine-tuning technique, whereas Teacher Forcing (TF) and Data Forcing (DF) were originally designed for training from scratch. Presenting these approaches as direct counterparts can be confusing and makes the overall story harder to follow.\n2. The method is presented as addressing exposure bias and enabling long video generation, yet no quantitative long generation results are provided (only some qualitative examples in the supplementary material’s limitations section). Moreover, the authors acknowledge that the model struggles with sequences longer than those seen during training. This seems like an inherrent limitation of the approach, as the distributional losses do not seems to go beyond the native generation window of the base model.\n3. The visual results, aside from the local attention ablation, do not clearly demonstrate the benefits of the proposed method.\n\nQUESTIONS:\n1. I recommend moving the details about the different training objectives to the main paper, as they are essential for understanding key aspects of the method, such as how the pseudo ground truth scores are obtained and how the critic is trained.\n2. Could you clarify how many rollouts are used during training? What is the total number of frames? Does this total align with the native number of frames of the base model or the critic?\n\nI think in the current state the presentation weaknesses overweigh the method's strengths, therefore I am leaning towards rejection. However, I am open to revising my score if the rebuttal adequately addresses my concerns, as I do believe the contributions (the achieved efficiency through KV caching and autoregressive distillation) are noteworthy.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a critical issue in autoregressive video generation setups: error accumulation in the network's predictions, leading to train/test domain gaps. The authors employ an Infinite Nature approach, where instead of teacher forcing (training on ground truth sequences), they use student-forcing or self-forcing, in which the model also observes its own outputs during training. This is challenging because the model must backpropagate through all diffusion time steps and all previous generation steps. To address this, the authors propose several techniques: 1. Use a few step distribution matching backbone and training scheme instead of a full ~50-step SDE/ODE model; 2. Stochastic gradient truncation, where the gradients flow only through the last denoising step of each frame; earlier steps and prior frames are detached. The model works well and achieves 17FPS (on par with its backbone CausVid), and have certain quality improvement.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n-- The problem of error accumulation is a critical and long-standing problem of AR models, and the paper is well-motivated and proposed a reasonable solution.\n\n-- The training is very efficient and could converge within hours.\n\n-- The authors provide thorough empirical section: VBench, human study, latency/throughput, ablations on loss functions and training paradigms.\n\nWeakness:\n\n-- The idea of student forcing/self forcing is not new. Prior autoregressive work such as Infinite Nature did some thorough experiments on such methods.\n\n-- I personally work on this topic and tried the code as soon as it was released. However, I am not certain how much improvement is actually made by the proposed self-forcing training scheme -- efficiency-wise, the FPS and workload is exactly the same as CausVid, so efficiency-based improvements cannot be claimed by this paper; quality-wise, I am also not certain how much improvement is made on top of CausVid, the short 5-sec videos look on par with CausVid, and longer extrapolation still has the drifting issue, and I personally think it is arguably more severe than CausVid, potentially because self-forcing is trying to accommodate and ignore the error, but at the same time it also forgets some portions of the context.\n\nOverall, this is a good paper that should be accepted, as it provides some interesting insights. However, I am not sure how \"functional\" the method actually is — the error accumulation is still evident, and it is hard to argue how much more robust the model gets compared with the backbone. Therefore, I vote for acceptance, but not a strong acceptance.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}