Title: Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation

real-time

interactive

diffusion model

adversarial

gan
Abstract: Existing large-scale video generation models are computationally intensive, preventing adoption in real-time and interactive applications. In this work, we propose autoregressive adversarial post-training (AAPT) to turn a pre-trained latent video diffusion model into
a real-time, interactive, streaming video generator. Our model autoregressively generates a latent frame at a time using a single neural function evaluation (1NFE). The model can stream the result to the user in real time and receive interactive responses as control to generate the next latent frame. Unlike existing approaches, our method explores adversarial training as an effective paradigm for autoregressive generation. This allows us to design a more efficient architecture for one-step generation and to train the model in a student-forcing way to mitigate error accumulation. The adversarial approach also enables us to train the model for long-duration generation fully utilizing the KV cache. As a result, our 8B model achieves real-time, 24fps, nonstop, streaming video generation at 736x416 resolution on a single H100, or 1280x720 on 8xH100 up to a minute long (1440 frames).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a way to turn a pre-trained video diffusion model, trained to generate multiple frames simultaneously to create a fixed length video, to an auto-regressive model that generates the next frame conditioned on the N previous frames.\n\nThe method consists in 3 steps:\n* First the model is adapted to take as input not just the noise, but the concatenation of the previous frame to the noise (channel wise). The model is also adapted to use causal attention (frame by frame), turning it to an auto-regressive model.\n* Then the model is distilled to generate the next frame in one step\n* Then an adversarial loss is used to turn that frame into a higher quality frame, and student-forcing is used so that inference and training do not suffer from a mismatch in distribution that would make inference diverge quickly\n\nThanks to those adaptions, the authors claim that:\n1) the model can generate 24fps real time, with a resolution of 736×416 on a single H100 GPU and 1280×720 on eight H100 GPUs\n2) the model can generate minute long videos thanks to its auto regressive nature\n\nSTRENGTHS AND WEAKNESSES:\nAlthough the idea of one step distillation using GAN loss is not new, this paper introduces some nice tricks that make it better and combine it with auto-regressive generation frame by frame, to make video generation real time.\n\nStrengths are:\n- Great results in terms of generation speed\n- An interesting combination of tricks to speed things up and push quality further (like frame by frame GAN loss, or student forcing)\n- Convincing results, particularly for long 1 minute videos\n\nWeaknesses:\n- The evaluation protocol is relying on VBench-I2V which might not be the best correlated with human judgement on recent models, on qualitative evaluations, but no human evaluations (the golden standard)\n- The benefits of the methods are most seen on long videos, but the videos shown are also very repetitive (there can be no scene change or interesting motion that change the layout too much with this approach)\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on enabling real-time, interactive video generation by proposing Autoregressive Adversarial Post-Training (AAPT), a framework that transforms a pre-trained bidirectional latent video diffusion model into a fast, autoregressive generator. The method integrates block-causal transformer architecture with adversarial training, allowing the model to generate an entire latent frame in a single forward pass while using KV cache for efficiency. To mitigate error accumulation, the authors employ student-forcing adversarial training. Experiments show competitive quality and strong controllability in interactive tasks like pose-driven human and camera-controlled scene generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed method achieves real-time video generation at 24fps and high resolution (736×416 on 1×H100; 1280×720 on 8×H100) using only a single forward pass per latent frame, significantly outperforming existing methods in terms of latency and throughput.\n2. By combining KV cache reuse and a segment-based training strategy, the model is capable of generating one-minute-long videos without stopping, which is notably longer than previous autoregressive diffusion models.\n3. The model demonstrates competitive or state-of-the-art results across three tasks: standard I2V generation (VBench), pose-conditioned human generation, and camera-controlled world exploration, showing both high visual quality and controllability.\n\nWeaknesses\n1. The method extends Diffusion Adversarial Post-Training (APT) [1] to the autoregressive setting, but the novelty is incremental and not clearly disentangled. The key contributions (i.e., causal frame-level generation, student-forcing adversarial training, and efficient use of KV cache) are reasonable adaptations rather than fundamentally new ideas. Besides, the architecture builds heavily upon APT (as the authors themselves note that “Our method extends adversarial post-training (APT) to the autoregressive video generation scenario”). Thus, a more explicit comparison (both conceptual and empirical) to APT would help clarify what is truly novel.\n2. The paper introduces a sliding window attention mechanism to limit memory and compute during long video generation, but it does not explore how the choice of window size $N$ impacts performance. A sensitivity analysis on $N$ would clarify the trade-offs between computational efficiency, temporal consistency, and generation quality.\n3. While the paper reports strong performance on VBench-I2V and other quantitative metrics, it lacks human evaluation of perceptual quality and consistency, especially for the long video and interactive scenarios. Given that many metrics (e.g., FVD, aesthetic scores) are known to be imperfect proxies for human judgment, user studies would strengthen the paper’s claims on quality in real-time applications.\n\n[1] Diffusion adversarial post-training for one-step video generation.\n\nQUESTIONS:\nPlease refer to the Weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Autoregressive Adversarial Post-Training (AAPT), a method to transform pre-trained latent video diffusion models into efficient autoregressive generators for real-time interactive applications. The core innovation lies in leveraging adversarial training to enable single-step, per-frame generation (1NFE) while mitigating error accumulation through student-forcing training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:  \n- This work is written with fluent and clear prose. Rigorous 3-stage training (diffusion adaptation → consistency distillation → adversarial training) enables stable 1NFE generation.  \n- Comprehensive evaluation across 120-frame and 1440-frame generation using VBench-I2V, outperforming diffusion-forcing baselines (CausVid, SkyReel-V2) in throughput/latency .  \n- Ablation studies validate design choices (e.g., student-forcing > teacher-forcing, recycling input necessity).  \n\n**Weaknesses**:  \n- User study is needed to demonstrate effectiveness, although it is not mandatory.If the work already includes human evaluation, please specify details like the number of participants and evaluation methods.\n- Mistake in preparing the manuscript: a paragraph is misplaced (line 224 to line 227), as the author has clarified in the appendix.\n- More concerns about the \"real-time\" and \"interactive\", see the **Questions**.\n\nQUESTIONS:\n**Questions about the \"real-time\"**\n\n*1. The lightweight VAE decoder requires a more detailed explanation.*\n\nAblation studies by module are needed to clarify changes in the model's inference speed. Appendix E mentions that the lightweight VAE decoder achieves a 3x speedup compared to the original decoder. Therefore, it is necessary to demonstrate that the model's real-time performance stems from the main innovation(consistency distillation, adversarial training...) rather than other tricks.\n\n*2. Does the rotary position embeddings (RoPE) impact inference speed?​​*\n\nThis work adopts the ​​MMDiT design​​ and employs a ​​sliding window​​ approach for long-video generation, making ​​RoPE particularly important​​ for maintaining positional awareness. The introduction of RoPE brings additional computational overhead. Please clarify how the inclusion of the ​​RoPE module​​ affects the model's inference speed.\n\n*3. More quantitative/qualitative results are needed to demonstrate the effectiveness of post-training.*\n\nThis work has 3 stage (Diffusion adaptation, Consistency distillation, Adversarial training), but it only compares base diffusion models with 3-stage models, it does not clarify the performance changes at each stage. Many models claim to achieve real-time generation (SkyReel-2, CausVid, Matrix, GameNGen), with most employing distillation techniques (consistency distillation, DMD, etc.). However, in practice, distilled models (1-step, 4-step) often underperform the original model (50-step) in terms of human perceptual quality. This paper utilizes post-training to enhance visual fidelity and reduce error accumulation, but it lacks ablation studies to validate the performance of consistency distillation at 1-step, 4-step, and 8-step settings. Additionally, there is a lack of ablation experiments and visual comparisons (unless demonstrated in the project webpage/videos—please point them out if available) to substantiate the visual improvements brought by **Adversarial Post-Training**.\n\n*4. Why autoregressive step is more efficient than onestep diffusion forcing？*\n\nThe paper mentions that the \"recycling approach is more efficient than one-step diffusion forcing,\" yet diffusion forcing can also employ techniques like sliding windows (as in SkyReel-2) and KV caching (e.g., Magi-1, Playable Game Generation, and later works such as VideoMAR and self-forcing). Could the authors clarify the specific differences in how KV caching operates under these two guidance schemes?\n\n**Questions about the \"interactive\"**\n\n\n​​*5. Underperformance in Pose-Conditioned Human Video Generation.​​*\n\nAll metrics of AAPT fall behind OmniHuman-1 in this task. While this is understandable given AAPT's novel motivation, it remains necessary to clarify the ​​unique advantages​​ of this work. Moreover, this work follows the same approach(extract and encode the human pose) as OmniHuman-1, yet its performance is consistently inferior—which is quite strange.\n\n*6. The memory capability of the interactive process model needs clarification.*\n\nExamples on the demo page show that the model exhibits certain memory lapses (e.g., a man in the desert, where background characters suddenly disappear and reappear when the camera angle changes). The original text uses a sliding window approach for interactive generation, lacking a module to record historical information. However, in the field of interactive generation—such as game world models (The Matrix, GameGenX, Genie) and 3D world generation (Wonderland, WonderWorld)—memory plays a particularly crucial role in interactive systems. Please further supplement this in the \"Related Works\" section.\n\n---\n\nOverall, this paper addresses a very interesting and novel problem. However, most of the algorithms are derived from the baseline APT, and the additional autoregressive step proposed in this paper requires further clarification of its advantages compared to diffusion-forcing to address my concerns about the algorithmic novelty. I hope the authors can resolve my concerns.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of real-time interactive video generation by introducing Autoregressive Adversarial Post-Training (AAPT), a method to convert a pretrained latent diffusion video model into a streaming one-step-per-frame generator. Instead of the usual multi-step diffusion sampling, AAPT produces each new video frame in a single network forward pass (1 NFE) by leveraging a transformer architecture with KV-cache for efficient autoregression. The model is trained with an adversarial objective after initial diffusion training, enabling high-fidelity frame synthesis and student-forced training (training on its own sequential outputs) to mitigate error accumulation over long sequences. Notably, it demonstrates non-stop video streaming up to 1 minute (1440 frames) without quality collapse, significantly outperforming prior one-step or diffusion-forcing methods that were limited to only a few seconds of footage.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** The paper's technical quality is high, offering a novel yet well-founded solution to an important problem. The AAPT approach is **innovative** in combining diffusion models with **adversarial post-training** for video. It builds on recent **diffusion-forcing** ideas but extends them in a new direction. This design fully exploits the transformer's KV cache (like in LLMs) to avoid recomputation, yielding **significantly superior throughput and latency** suitable for real-time use. The empirical results are compelling: the model achieves **state-of-the-art speed** (24 fps at high resolution) far outperforming prior systems like CausVid (which reaches 9.4 fps at lower resolution). It also scales to much longer videos than most diffusion-based methods—*continuous 60+ seconds of video without resets*—whereas competitors like CausVid, SkyReel-V2, and MAGI-1 can only generate in 5-second chunks before needing to restart the generation process. AAPT improves per-frame image fidelity and condition consistency over the diffusion baseline, and achieves the **best overall quality scores** among the compared methods on long video generation. Overall, the authors demonstrate a strong grasp of both the engineering and research aspects, lending credibility to the **originality** of the solution (extending the recent APT idea into an autoregressive domain) and reinforcing the **importance** of the contribution. \n\n**Weaknesses:** The weaknesses of the paper are relatively modest. One concern is that, as with many adversarial generation approaches, **temporal consistency** can suffer slightly; indeed, the authors report a minor drop in temporal quality metrics for AAPT compared to a pure diffusion baseline. There are instances of small **scene or object drift** over very long sequences (e.g., an object that disappears after occlusion, as noted by the authors), indicating that maintaining perfect identity and continuity remains challenging. The proposed solution (AAPT) still relies on a **sliding context window** for simplicity, meaning the generator only looks at a fixed recent history of frames; this could limit long-range coherence and is identified as an area for future improvement. Additionally, the training pipeline is quite complex—it involves diffusion fine-tuning, consistency distillation, and adversarial training with a large 8B parameter model. This raises the **computational cost** and implementation complexity: for example, adversarial post-training on such a large model likely required careful tuning (the paper doesn't detail training stability, which could be non-trivial). Finally, the experiments focus on the image-to-video setting (where the first frame is given); while this is a sensible choice for interactive applications, the method's performance in a pure text-to-video scenario (starting from scratch) is not explicitly demonstrated, a minor omission in scope.\n\nQUESTIONS:\nTraining stability and compute cost: Adversarial post-training on an 8B parameter model is a substantial undertaking. Could the authors provide more insight into the training process? For instance, how long did AAPT training take, and were there any instability issues (mode collapse, divergence) that needed to be overcome? Understanding the computational cost and training stability would help gauge the practicality of reproducing or extending this work (e.g., applying AAPT to other base models or at larger scales).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method that converts a pre-trained bidirectional video diffusion model into a causal generator of a single function evaluation per latent frame, enabling low-latency, real-time video generation.\nThe key contributions include model architecture and training procedure.\n* In terms of model architecture, the bidirectional attention is replaced by block-causal attention with a sliding window. The clean previous frame is concatenated with noise as input, halving the computation compared to diffusion forcing.\n* In terms of training procedure, it involves first finetuning the adapted architecture with teacher-forcing, then performs consistency distillation, and finally adversarial training. The majority of the innovations are concentrated in the adversarial training stage:\n    * The discriminator for the adversarial training stage is initialized from the same generator weights, adapted to output a per-frame logit.\n    * The student predictions are used as conditioning (student-forcing) instead of ground-truth frames (teacher-forcing), reducing error accumulation and encouraging self-improvement.\n    * To generate long videos beyond the training data, the discriminator is applied in an overlapped, chunk-wise manner. To save memory when training for long video generation, the KV cache of the previous chunk is detached.\n\nThis approach achieves state-of-the-art performance at significantly higher speeds. Additionally, it can be fine-tuned to incorporate conditioning signals for human pose and camera control.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n* The proposed method excels across multiple dimensions: high-resolution, high-quality, low-latency, and real-time. These are remarkable achievements.\n* The paper is well written, with a comprehensive background introduction and has every important detail documented.\n* Utilizing student-forcing to fundamentally reduce error accumulation is a principled idea that is well-implemented in the proposed model.\n* Compared to the concurrent work self-forcing [1], the proposed method is simpler to implement and potentially faster, thanks to the 1-step generation design.\n[1] Huang et al., Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion, 2025\n\n### Weaknesses\n* While the proposed method demonstrates strong performance in terms of visual quality and speed, its reliance on sliding window for long-video generation is fundamentally constrained in modeling long-range dependencies. Consequently, it is not clear if the proposed approaches are viable as long-term solutions.\n* There are no visual results provided in the supplementary material.\n\nQUESTIONS:\n* How is student-forcing implemented during adversarial training? It seems that the parallel prediction technique of LLM won’t work for teacher-forcing due to the dependency on previous predictions. Do you pre-sample the student outputs, or chain the sampling process and back-propagate through time?\n* What will happen if the video is generated beyond 60 seconds? How does the failure mode look like?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}