Title: Generative Pre-trained Autoregressive Diffusion Transformer
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation

auto-regressive

diffusion
Abstract: In this work, we present GPDiT, a Generative Pre-trained Autoregressive Diffusion Transformer that unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis, within a continuous latent space. Instead of predicting discrete tokens, GPDiT autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics and semantic consistency across frames. This continuous autoregressive framework not only enhances generation quality but also endows the model with representation capabilities. Additionally, we introduce a lightweight causal attention variant and a parameter-free rotation-based time-conditioning mechanism, improving both the training and inference efficiency. Extensive experiments demonstrate that GPDiT achieves strong performance in video generation quality, video representation ability, and few-shot learning tasks, highlighting its potential as an effective framework for video modeling in continuous space.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GPDiT to unify the advantages of diffusion and autoregressive modeling for long-range video generation task. GPDiT autoregressively produces next latent frames through diffusion loss, while achieving plausible modeling of motion dynamics and consistency across frames. The proposed method improves both generation quality and representation capabilities. Moreover, it also presents a lightweight causal attention and parameter-free rotation-based time-conditioning mechanism to improve training and inference efficiency. Comprehensive experiments show that GPDiT obtains superior performance in video generation, video representation, and few-shot learning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This paper introduces GPDiT with framewise causal attention to improve temporal consistency of long videos, as well as a lightweight causal attention to reduce computational costs.\n2. This paper presents a rotation-based conditioning strategy to provide a parameter-free approache to inject time information.\n3. Extensive experimental results show that the proposed GPDiT obtains comparable performance on video generation and effective extension in few-shot learning tasks.\n \n**Weaknesses**\n1. Lack of comparison with the baselines for popular video generation. The authors only provided comparisons with earlier methods, and are encouraged to include more advanced methods, such as CogVideo-X, HunyuanVideo, Wan.\n2. The core contribution of this paper is the proposal of \"Lightweight Causal Attention\", but the authors did not quantitatively explore the training and sampling efficiency of this technique.\n3. There is a lack of exploration on \"injecting time information\". The authors proposed a new \"Timestep Conditioning Injection\" method in Sec 4.2, but did not conduct ablation studies on this technique.\n\nQUESTIONS:\n1. For Video Few-shot Learning, how many training samples are required for these tasks to acquire in-context learning capabilities, and are there differences between tasks of varying difficulty levels?\n2. Has the author attempted more complex in-context learning tasks, such as image editing tasks?\n3. The method in this paper is implemented based on Wan's VAE, which compresses multi-frame video frames into a single latent. Then, when migrating to image tasks, how are these images encoded?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GPDiT, A unified framework for video generation that combines autoregressive next-token prediction objective with a diffusion-based formulation, diverging from the more common discrete token-based approaches. GPDiT enhances model efficiency during both training and inference through a specialized causal attention mechanism that strategically bypasses attention score computation for (clean) input frames. Furthermore, The authors also propose a parameter-free rotational time conditioning mechanism as an alternative to adaLN-Zero layers used in prior DiT-based models. Two model variants are trained: GPDiT-B (12 layers, 85M parameters) and GPDiT-H (24 layers, 2B parameters) on UCF-101 and MSR-VTT, using sequences of 17 frames.Additionally, a longer variant, GPDiT-H-LONG, is trained with variable-length sequences (17–45 frames). Experimental results consistently demonstrate that GPDiT performs competitively or better than tested methods on both qualitative and quantitative metrics. Beyond video generation, GPDiT exhibits strong capabilities in learning robust representations, enabling various vision-based in-context learning tasks such as colorization, depth prediction, human pose detection, face-to-cartoon synthesis, and style transfer.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- A significant strength of this work is the extensive evaluation of the proposed models beyond just video generation. The authors demonstrate the efficacy of GPDiT in learning high-quality video representations (validated through classification tasks) and its remarkable in-context learning abilities across a diverse range of vision tasks. This highlights the framework's versatility and potential for broader applications.\n- The introduction of a parameter-free, rotation-based time conditioning mechanism is a particularly interesting contribution. Its practical effectiveness, as evidenced by the experimental results, suggests a promising direction for more efficient and robust temporal conditioning in diffusion models.\n\n**Weaknesses**\n- While the underlying formulation of GPDiT is inherently extensible to text-to-video generation, the current work is solely trained and evaluated on visual data. Consequently, the authors are restricted to quantitative metrics like FVD, FID, and IS. This omission prevents a direct comparison with state-of-the-art text-to-video models and misses out on recently proposed and widely adopted prompt-based benchmarks such as VBench and MovieGenBench. Addressing text-to-video capabilities would significantly broaden the paper's impact.\n- The related work section lacks a comprehensive discussion distinguishing GPDiT from highly relevant text-to-video models such as NOVA [7] and CausVid. These models also leverage diffusion losses within an autoregressive framework. A more detailed comparison highlighting the architectural novelties, efficiency gains, or specific contributions of GPDiT over these similar approaches is essential for contextualizing the paper's advancements and demonstrating its unique value.\n\nCausVid:  Yin et al., From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, CVPR 2025 \n\n- The paper contains several vague and inconsistent experimental details that hinder clarity and reproducibility:\n\n(i) Line 236 states that \"GPDiT-H is trained on close-source open-domain dataset,\" which contradicts the earlier impression that models are trained exclusively on UCF-101 and MSR-VTT. Clarification on the specific datasets used for each model version is needed. \n\n(ii) There is an unexplained discrepancy between the qualitative results for video generation (16 frames conditioned on 13 initial frames) and the quantitative results (12-frame generations conditioned on 5 input frames). A clear rationale for these differing evaluation setups should be provided.\n\n**Minor Points** \n- Figure 3 caption: “OF^2 => O(F$^2$) and OF => O(F)\n\nQUESTIONS:\n1. Given that NOVA [7] can also perform image-to-video generation, the experimental evaluation section would greatly benefit from a direct quantitative and qualitative comparison against NOVA. What are the advantages and disadvantages of GPDiT compared to NOVA, especially considering the architectural differences.\n\n2. Table 2 presents evaluations for 12-frame video generation conditioned on 5 input frames. What are the performance and visual quality implications if the models are conditioned on only a single input frame? Does the model maintain coherence and quality under this more challenging setting?\n\n3. As an autoregressive model, the visual quality of generated frames might degrade over time. Could the authors provide an analysis or qualitative examples demonstrating how the visual quality of frames changes as the generation sequence extends, especially for longer video generations?\n\n4. The proposed parameter-free rotation-based time conditioning is a core contribution. How does its effectiveness compare against the adaLN-single layer proposed in PIXART-α [6], particularly concerning performance, stability, and computational overhead?\n\n5. The in-context learning experiments utilize models trained with custom-built SFT data featuring task-specific image pair samples. If the models are primarily trained on single-modality inputs (e.g., sequences of color images, grayscale images, or depth images), how well do they generalize and perform few-shot in-context learning when presented with mixed input types within the same context? This would explore the model's ability to learn and apply representations across different visual modalities.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GPDiT, a novel hybrid framework that unifies autoregressive modeling and diffusion processes in a continuous latent space for long-range video synthesis. Its core contributions are:\n\n1. Continuous Autoregressive Diffusion: Instead of discrete tokens, GPDiT predicts future latent frames autoregressively using a diffusion loss, enabling joint optimization of motion dynamics and semantic consistency across frames.\n2. Efficiency Innovations:\n   - A lightweight causal attention variant reduces computational overhead.\n   - A parameter-free rotation-based time-conditioning mechanism enhances temporal modeling.\n3. Multifaceted Capabilities: Experiments demonstrate GPDiT excels in:\n   - Video generation quality (long-range consistency),\n   - Representation learning (e.g., feature transferability),\n   - Few-shot adaptation tasks.\nThe work establishes continuous latent space as a viable paradigm for scalable video modeling.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nQuality: Comprehensive experimental design covering video generation (UCF-101/MSR-VTT), representation learning (linear probing accuracy), and few-shot multi-task applications (object detection/colorization/style transfer). Appropriate metrics (FVD/FID/IS) and ablation studies validate the efficacy of the lightweight attention and rotation-based conditioning.\n\nOriginality : Two innovative contributions:\n\n1. Continuous latent space AR-diffusion framework: First unification of AR's causal modeling with diffusion's continuous optimization, resolving long-video semantic coherence (vs. discrete token AR models).\n2. Zero-parameter time conditioning: Rotation injection maps noise to complex-plane rotation, replacing adaLN-Zero and significantly reducing parameters.\n\nSignificance: Proposes a unified video generation-understanding framework. Few-shot experiments demonstrate cross-task generalization potential, with multimodal extensibility (Discussion section).\n\n\nWeaknesses:\n\n- Quality Issues: Missing critical baselines: No comparison with state-of-the-art DiT variants or AR-diffusion hybrids, undermining SOTA claims.\n- Significance Limitations: No quantification of lightweight attention's practical speedup, weakening efficiency claims.\n- Clarity: adding algorithm pseudocode would further improve reproducibility.\n\nQUESTIONS:\n1. Quantification of Efficiency Gains\n   - Question: Lightweight attention claims \"improved training/inference efficiency\" but lacks metrics.\n2. Statistical Reliability of Few-Shot Experiments\n   - Question: Using merely 20 videos for few-shot evaluation (detection/colorization) risks overfitting. Was performance verified across data scales?\n3. Generalizability of Rotation Conditioning\n- Question: How does rotation injection handle non-square features (e.g., 768×320 videos)? Does complex-plane rotation require adaptation?\n\nLIMITATIONS:\nYES.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel framework GPDiT that models AR diffusion for long-range video synthesis within a continuous latent space. GPDiT autoregressively predicts future latent frames using a diffusion loss. The paper highlights two key innovations: a lightweight causal attention variant that reduces computational costs by eliminating attention computation between clean frames during training, and a parameter-free rotation-based time-conditioning mechanism that reinterprets noise injection as a rotation in a complex plane, thus removing the need for adaLN-Zero and its associated parameters. Experiments demonstrate GPDiT's strong performance in video generation quality, video representation ability, and few-shot learning tasks, positioning it as an effective framework for continuous video modeling.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The lightweight causal attention and parameter-free rotation-based time-conditioning is novel, addressing critical challenges in streaming video generation.\n- The paper provides clear explanations and illustrations (Figure 2 and 3) of the proposed attention mechanisms and the rotation-based time conditioning. And provides a clear analysis of computational savings from the lightweight causal attention.\n- The experimental results demonstrate competitive performance across multiple tasks: video generation, video representation (validated through linear probing), and few-shot learning (transferring to various downstream tasks with minimal fine-tuning), which presents GPDiT's potential as a unified model for visual understanding and generation.\n\nWeaknesses\n- While the paper mentions the potential for extensibility to multi-modal inputs like language, the current experiments are limited to video. The scale of the experiments (2B model on MSRVTT ) is not large enough. We cannot understand its potential in scaling up further. Showing scaling curves between the number of paramsters/tokens and accuracy is better.\n- The paper lacks a blation study for the parameter-free rotation-based time-conditioning mechanism. A comparison against traditional learned conditioning methods (e.g., adaLN-Zero) essential to validate its effectiveness and efficiency claims.\n- Although the paper discusses the theoretical computational efficiency improvements of the lightweight causal attention, it does not provide empirical validation (e.g., training time, inference speed, or memory usage comparisons) to support these claims.\n- The core challenge in streaming video generation is the error accumulation. There are many other choices to improve the computation efficiency, e.g., sparse attention, consistency distillation, etc. This paper tackles a problem that is not that important. And whether the proposed strategy can be incorporated to the other acceleration strategies remains unknown.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors acknowledge resource constraints limiting larger-scale experiments and the current model's limitation to the video modality, with plans for future multi-modal extensions. The streaming video generation for longer length is still hindered by error accumulation. The computation efficiency problem addressed by this paper is not that important.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}