Generative Pre-trained Autoregressive Diffusion Transformer
Authors: Yuan Zhang
Jiacheng Jiang
Guoqing Ma
Zhiying Lu
Bo Wang
Haoyang Huang
Jianlong Yuan
Nan Duan
Venue: NeurIPS 2025 poster ()
Keywords: video generation
auto-regressive
diffusion
Primary area: deep_learning
TL;DR: We develop a new framework that combines autoregressive modeling and diffusion for video data.

Abstract: In this work, we present GPDiT, a Generative Pre-trained Autoregressive Diffusion Transformer that unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis, within a continuous latent space. Instead of predicting discrete tokens, GPDiT autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics and semantic consistency across frames. This continuous autoregressive framework not only enhances generation quality but also endows the model with representation capabilities. Additionally, we introduce a lightweight causal attention variant and a parameter-free rotation-based time-conditioning mechanism, improving both the training and inference efficiency. Extensive experiments demonstrate that GPDiT achieves strong performance in video generation quality, video representation ability, and few-shot learning tasks, highlighting its potential as an effective framework for video modeling in continuous space.

Generative Pre-trained Autoregressive Diffusion Transformer
Yuan Zhang", Jiacheng Jiang", Guoqing Ma Zhiying Lu Haoyang Huang Jianlong Yuan?
Nan Duan
Peking University
Tsinghua University StepFunn, China
University of Science and Technology of China
Abstract
In this work, we present GPDiT, Generative Pre-trained Autoregressive Diffusion Transformer that unifies the strengths of diffusion and autoregressive modeling for long-range video synthesis, within a continuous latent space. Instead of predict- ing discrete tokens, GPDiT autoregressively predicts future latent frames using a diffusion loss, enabling natural modeling of motion dynamics and semantic consistency across frames. This continuous autoregressive framework not only enhances generation quality but also endows the model with representation capa- bilities. Additionally, we introduce lightweight causal attention variant and a perameter-fres rotation-based time-conditionis mechanism, improving both the training and inference efficiency. Extensive experiments demonstrate that GPDiT achieves strong performance in video generation quality, video representation abil- ity, and few-shot learning tasks, highlighting its potential as an effective framework for video modeling in continuous space.
Introduction
Despite recent Diffusion models have achieved notable success in video generation 2
advancements, existing diffusion-based approaches exhibit limitations in temporal consistency and motion coherence, particularly in long-range generation. key contributing factor is the use of bidirectional attention. This allows future context to influence current predictions, thereby violating the causal structure required for autoregressive generation.
In contrast, autoregressive (AR) modeling has become the de facto paradigm in natural language processing B 32 33]. This method inherently captures the causality in sequences by predicting the next token based previously generated outputs, thereby facilitating both sequence modeling and structural understanding. Inspired by this, recent efforts 7 9 48] have focused on integrating AR modeling with diffusion processes to leverage their complementary strengths. This integration aims to improve temporal coherence and enhance motion continuity in extended video synthesis. Notably, compared to traditional cross-entropy objectives that explicitly train for next-token predic- tion, the combination diffusion ceffussion-based loss and AR modeling offers an implicit path to temporal understanding, serving as natural byproduct rather than specially designed objective.
One line of research investigates the replacement bidirectional attention with causal attention 56 for improved modeling of temporal dependencies. Specifically, the attention computation restricts each noisy token to attend only to preceding clean tokens and itself. This
4 g
10
16
Equal contribution. Techhical leader.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Frame
Denoised
High Level to Level
RAAD
Low Level High Level
Style Transfer
Video Generation
Few-Shot Learning
Figure I: Video Generation and Few-Shot Multitask Learning. The left side of the figure illustrates the model's video generation capability: given set of initial frames, the model can continue the sequence by generating denoised frames. The right side showcases the model's ss multitask learning ability, similar to the approach presented in [33]. After few-shot fine-tuning, the model is capable of performing variety of tasks, such as translating high-level features to low-level features, converting low-level features high-level features, and executing style transfer across video sequences.
architectural design enables more robust generalization to video lengths beyond those seen during training, whereas bidirectional attention often leads to severe quality degradation when extrapolating to longer sequences. Moreover, causal attention is naturally compatible with KV cache, significantly accelerating the generation of long sequences. These advantages are unattainable with traditional bidirectional attention mechanisms.
characterized by the asyn- chronous injection of token-specific noise levels during training facilitate long-range dependency modeling. During inference, this method operates coarse-to-fine manner: first generates early frames and then progressively refines later ones through iterative denoising. While promising, this paradigm still struggles with training instability, with independent frame-level noise schedules often impairing performance compared synchronized alternatives.
Another increasingly studied approach diffusion forcing
To address the aforementioned challenges in suggsee video modeling, we introduce Generative Pre-trained Autoregressive Diffusion Transformer (GPDIT), frame- wise autoregressive diffusion framework. In contrast to discrete token-level autoregressive modeling, GPDiT captures causal dependencies across frames while preserving full attention within each frame, enabling both sequential coherence and intra-frame expressivity. Moreover, GPDIT improves training and inference efficiency through two practical architectural modifications. The first component introduces an attention mechanism that leverages the temporal redundancy of video sequences by eliminating attention computation between clean frames during training, thereby reducing computational cost without compromising generation performance. The second is parameter-fre time-conditioning strategy that reinterprets the noise injection process rotation in the complex plane defined by data and noise components. This design removes the need for adaLN-Zero 30) and its associated parameters, yet still effectively encodes time information. As shown in Figure GPDiT performs well in both video generation and few-shot learning tasks.
Our main contributions can be summarized as follows:
We introduce GPDIT, strong autoregressive video generation framework that leverages framewise causal attention to improve temporal consistency over long durations. To further enhance efficiency, we propose lightweight variant of causal attention that significantly reduces computational costs during both training and inference.
By reinterpreting the forward process diffusion models, we introduce a rotation-based conditioning strategy, offering parameter-free approach inject time information. This

---

lightweight design eliminates the parameters associated with adaLN-Zero while achieving model performance on par with state-of-the DiT-based methods.
Extensive experiments demonstrate that GPDiT achieves competitive performance on video generation benchmarks. Furthermore, evaluations on video representation tasks and few-shot learning tasks show its potential of video understanding capabilities.
2
Related works
Video Diffusion models. Diffusion and flow-based generative models [13 22
23
38
43 50 have
demonstrated unprecedented ability to capture visual concepts and produce high-quality images. Video Diffusion Models [44 is the first work introduce diffusion models for video generation. However, the expense associated with pixel space diffusion and denoising is nontrivial, requiring substantial computational resources. Models such as Magicvideo [55] and LVDM [[]] speed up training and sampling efficiency by compressing high high-dimensional video data into a latent space. Recent efforts have scaled diffusion transformers to substantially larger capacities, demonstrating significantly enhanced generation capabilities and further revolutionizing the field of text-to-video (T2V) synthesis. driving rapid progress across a broad range of applications [5]
25 26 28
[1 45 47P ].
Autoregressive Modeling. An emerging trend is the integration of autoregressive modeling and diffusion models. A key characteristic of this approach is that the model predicts future videos based on previously generated content. Representative works, such as [20 an additional visual tokenizer that maps pixel-space inputs into discrete tokens, which are then fed into language model to generate videos. However, this mapping inherently lossy, leading to
46
52 54 leverage
inferior performance compared to video diffusion models. Recent works [T]24] 40, [6], inspired by Diffusion Foreing (5), allow each video frame to be processed with distinct noise levels, enabling the generation of videas with variable lengths. However, adding noise to antecedent sequences complicates future predictions and degrades performance on discriminative tasks. Recent work [? ] addresses this prediction ambiguity by preserving the clarity of antecedent representations, keeping them nrese-free.
3 Preliminary
3.1 Denoising Diffusion
Diffusion models generate data by progressively corrupting samples from the data distribution with Gaussian noise, eventually transforming them into pure noise, and then learning to invert this process through sequence of denoising steps. Formally, given data distribution Panta (I)), diffusion models apply a stochastic differential equation (SDE) to gradually perturb the data:
dæ, p( t) dt (t) dwey
(I)
where t E [0, I'] for some fixed terminal time T > O, p(:.-) denotes the drift coefficient, (-) is the diffusion coefficient, and (eeor)) represents a standard Brownian motion. Let p. (x) denote the marginal distribution of œr; by construction, the initial distribution satisfies po (x())= Pdatn (x) A
notable property of this SDE is the existence of a corresponding ordinary differential equation (ODE), referred the Probability Flow ODE (8). whose solution trajectories guaranteed to match the time-evolving marginals (z):
dx; = p(xrnt)) - 1 oo( log po dt. (aart))=2000
(2)
4 Generative Pre-trained Autoregressive Diffusion Transformer (GPDiT)
In this section, we present an effective framework that combines autoregressive and diffusion models for video modeling. First, we introduce two variants of the attention mechanism tailored for frame- aware autoregressive diffusion in Section 4.1 Then, we discuss a flexible conditioning strategy designed to handle both clean and noisy frames in Section 4.2 Figure 2 provides an overview of the GPDiT framework, illustrating the inference pipeline, the internal architecture of a GPDiT block, and the rotation-based interpretation of the diffusion process.
3

---

4.1 Attention mechanism
Foeward
Roraae
Ope.
Xt2
GPDIT Block
LeyerNoom
Xy1
0e2
Atteniio
Rutade
811
Xo
stap
stop 2
LayerNocnt
GPDiT framework
GPDIT
Rotate Operation
Figure 2: Left plane: An overview of GPDIT inference. Middle plane: The architecture of typical GPDiT block, where adal NN-Zero is replaced with our rotation-based time conditioning. and causal attention is adopted instead of conventional bidirectional attention. Right plane: An illustration of the rotation-basee view of the diffusion forward process, where the data and noise components evolve through a parameter- free rotation in the complex plane.
Fgg
7g
Co
f1
90
Co
00
710
18g
71g
(c) OF-2 Inference
Y12
71
Co
6
712
OP*2 Training
(b) Training
Co
masked
unmasked
diffusion
current
(d) OF Inference
11g
Figure 3: Illustration of two causal attention variants. Both apply intra-frame full attention and inter- frame causal attention, but differ cross-frame attention handling between clean frames. C; and n; denote clean and noisy frames, respectively.
4.1.1 Vanilla Causal Attention
The traditional bidirectional attention mechanism has been criticized for disrupting temporal coher- ence and failing to maintain consistency in long video modeling. Mcanwhile, most existing models struggle to produce high-quality videos that exceed the frame length they were trained on since the models only learn the joint distribution of fixed-length frames. To alleviate the issues, employ the standard causal attention shown in Figure (a) and (c), where each noisy frame n; can only attends to previous clean frames Cej and itself, while czi also attend to each other. The training objective is:
((0) = -0,
[[e(n,, ces) - -12
A notable advantage of standard causal attention is its compatibility with key-value (KV) caching BT during inference, significantly accelerating generation and shortening the time needed for long video production.
4.1.2 Lightweight Causal Attention
Although the advantages of vanilla causal attention are notable, it presents two major challenges. First, during training, maintaining a clean copy of the noised sequence for attention map computation doubles the memory and computational costs. Second, during inference, the inevitable expansion of

---

the KV cache due to token accumulation in long- lenu-ne prediction imposes prohibitive memory burden.
To address these issues, we propose a lightweight causal attention mechanism that exploits the spatial redundancy in video data. As illustrated in Figures 3 (b) and (d), eliminate attention score computation between clean frames, thus reducing additional operations without compromising model performance. To quantify computational savings, vanilla design. The computational overhead can be decomposed into three components: attention between clean contexts, attention between noisy frames and clean contexts, and self-attention among noisy frames, with computational complexities of O F denotes the number of frames. Since attention between clean frames accounts for nearly half of the total computation, its removal leads to substantial reduction in training cost. Moreover, during inference, achieving (FF) complexity with standard causal attention requires maintaining key-value (KV) cache, resulting in an additional (2FF) memory overhead. In contrast, our method attains O F) inference complexity without incurring extra memory costs, substantially reducing the memory footprint.
analyze the attention complexity the
(() F²), O F²). and ((F), respectively, where
4.2 Re-Thinking Timestep Conditioning Injection
The Adaptive Normalization Layer Zero OadaLN-Zero) has been widely utilized to incorporate timestep and class-label embeddings into diffusion model backbones, introduced in DiT [Q] adar-- is typically designed as an MIPP block to extract class-label embeddings for each transformer block. However, modern tasks in text-to-image, generation involve more complex semantic embeddings. These embeddings are often injected into the model through techniques such as token concatenation along the sequence dimension or cross- attention, leaving the MLP block to primarily handle timestep embeddings. The authors of [6] argue that the adaLN-Zero submodule contributes significantly to the model's parameter count, accounting for an increase of approximately 28%. This considerable overhead has motivated research into more efficient methods for incorporating time conditioning in these models, aiming to reduce the computational cost while maintaining or enhancing performance.
sxx--to-video, and image-to-video
We begin by considering the forward (variance- (variance-peeserving)) diffusion process, given by:
Te Varzo + vI dige,
where E RD is a clean sample drawn from the data distribution, e N 1) represents standard Gaussian noise, and ar E [o, 1). To facilitate our analysis, we reduce the problem to one dimension (D = 1) and reinterpret the forward process rotation in a 2D space. Specifically, we define the rotation angle 8₂ as:
Van,
sin O2 v1 City
such that the forward process becomes:
cos 0:30 + sin 0çc.
To represent this process geometrically, we stack the clean sample æn and the noise into a 2- vector
To
IR. The forward diffusion step then represented as an orthogonal rotation in this
2D space:
a ()
cos Og
sin Og cos 02 E
=
sin 02
R(D))
(1) is its orthogonal complex In this formulation, " represents the usual diffused sample, while x; companion. The clean sample 2g and the noise can be recovered by applying the inverse rotation:
2D (
R(OO)
I
(1)
The model is trained to predict the complex companion xt from the input s using a predefined loss function, which is assumed to be unknown for the current analysis.
5

---

The proposed approach follows the principle of parsimony, applying reverse rotation with the angle 0, = for each block efficiently inject the timestep embedding while incurring no additional computational overhead. Other forms of conditioning, such as text or image conditioning, can be incorporated in the standard manner.
5
Experiments
5.1 Experimental Setups
We conduct experiments in three scenarios: video generation, video representation, and few-shot learning. The results demonstrate that GPDiT exhibits excellent generative and representational capabilities, which are crucial for building a unified model for visual understanding and generation, as well as the ability to transfer downstream tasks with minimal cost and no need for additional modules.
Datasets. For video generation task, UCF-101 33 dataset consists of 13,320 videos across 101 action categories and is widely used for human action recognition, MSR-VTT [49] is a large-scale dataset designed for open-domain video captioning, containing 10,000 video clips from 20 categories, with each clip annotated with 20 English sentences by Amazon Mechanical Turk workers. We assess the capability of GPDiT in video representation the UCF-101 dataset. For the few-shot learning tasks, we construct multiple supervised fine-tuning (SFT) datasets. For each task, we create a SFT dataset with 20 video sequences, each generated by sampling three pairs from set of 40 task-specific image pairs. These tasks include human detection, image colorization, Canny edge-to-image reconstruction, and two style transfer applications.
Evaluations. For video generation, we randomly sample 10,000 videos from UCF-101 and 7,000 videos from MSR-VTT. The Fréchet Video Distance (FVD) [42 is computed for entire videos, while the average Fréchet Inception Distance (FID) [2 and Inception Score (IS) 34] are calculated over individual frames. For the video representation task, top-1 accuracy is reported using linear probing protocol. In the few-shot learning setting, we provide per-task video results along with qualitative analyses.
Implementation details. To ensure fair comparison, we design benchmark model with 80 million parameters based on the ar- chitecture in Table Trained on UCF-101 each video is center-cropped and resized to 256 x x256. The Adam optimizer with a learn- ing rate of lc-4 and a total batch size of 96 across 32 H100 GPUs is used. Training lasts for 400k iterations.
Table I:
Model variants of GPDiT. We follow the model size configurations of DiT (30 and SiT (27]. replacing adal.N-Zero with a parameter-free rotation- based time conditioning.
Models
#Layers Hidden Size MLP #Heads #Params
GPDiT-B GPDiT-H
12
768
3072
12
85M
24
2816
11264
22
2B
We further scale the model to two-billion-
parameter variant, GPDiT-H (see Table [T.. First, we perform a 200k-iteration warm-up using an unconditioned image dataset from LAION-Aesther size of 960. Training continues for another 200k iterations on a mixed image-video dataset, with equal sampling of images and videos, and batch sizes of 256 and 64, respectively. Video frames sampled every three frames and clipped into 17-frame segments. Each image is center-cropped to the resolution closest to the original, with target sizes of 256 x 256, 320, x20x 192, and video to 192x320 Finally, we continue training the GPDiT-H model on a pure video dataset featuring variable video lengths ranging from 17 to 45 frames. This stage lasts for additional 150k iterations, using a reduced learning rate of 2e-5. The resulting model is denoted as GODGT-H-LONG To compress video latents, we employ VAE [43, which reduces four frames into single latent representation.
(35) with learning rate of le-4 and batch
5.2 Video Generation
To evaluate the generalization ability of the GPDIT framework, we conduct experiments on two zero-shot video generation tasks: MSRVTT and UCF-101 using GPDiT-H. The training data does not overlap with the test datasets, allowing us to assess the model's ability to generalize to unseen data. At the same time, to assess its fitting capability, trained the GPDiT-B model on UCF-101

---

and measured its generation performance. For both models, 12-frame video sequences are generated, conditioned on 5 imput frames. The generated results are evaluated using FID, FVD, and metrics. During inference, we apply classifier-free guidance with a scale of 1.2 for the GPDiT-H model and 2.0 for the GPDiT-B model.
Figure 4: Video generation of the subsequent 16 frames conditioned on the initial 13 frames from the MovieGenBencchmat dataset, with the frames sampled at three frame intervals thereafter. For more details, zoom in to observe finer aspects of the generation process.
Table 2:
SSro-Shot performance comparison of video generation on MSRVTT and UCF-101, 12- frame video sequences generated, conditioned on 5 input frames.
Dataset Method
#Data
#Params
FID 4.
FVD 4
IS
MagicVidec
65
10M
36.5
998
LVDM [I]
2M
1.2B
742
ModelScope
45
10M
1.7B
550
PixelDance ?
10M
1.5B
381
MSRVTT
DreamVideo
5.3M+340K
2.0B
149
SnapVideo |?
1.1B
3.9B
8.5
110
GPDiT-H
192M Img + 6.4M Vid
2.0B
7.4
68
GPDiT-H-LONG
24M Vid
2.0B
7.4
64
Magic 55
10M
145.0
Video [S]
5.4M
50.5
Intern Vid
28M
60.3
21.0
Video-IDM 2
10M
4.2B
551
33.5
UCF-101
Make-A-Viden 36
20M
9.7B
367
33.0
Vide [?
1.IB
3.9B
39.0
260
PixelDance [?]
10M
1.5B
49.4
243
42.1
GPDiT-H
192M Img + 6.4M 24M Vid
2.0B
66.5
GPDIT-H-LONG
2.0B
7.9
218
66.6
Main results. Table 2 shows GPDiT achieves a competitive FID of 7.4 and an FVD of 68 on MSRVTT, demonstrating its effectiveness in handling diverse video generation tasks without direct exposure to the test data. Moreover, GPDiT consistently outperforms previous methods both FID and FVD, underscoring its potential handle wide range of unseen video data. On UCF-101, GPDiT also performs well across metrics, with an IS of 66.5, FID of 14.8, and FVD of 243, Notably, GPDiT-H-LONG, trained with 24M video data, achieves the best results with IS of 66.6, FID of 7.9, and FVD of 218, further showcasing the model's 's generalization ability. As shown in Table 3 both GPDiT-B-OR and GPDiT-B OF achieve strong alignment with the UCF-101 distribution. attaining competitive FVD scores of 214 and 216 respectively with only 80M parameters. These results validate GPDiT's effectiveness distribution fitting and its consistent robustness across varying model scales. For visual demonstration, we present the generated videos derived from 13 input frames alongside their corresponding 16-frame extensions on the BovieGenBemch dataset [? ]. shown in Figure 4

---

5.3 Video representation
To assess the model's representation ability, we conduct linear probing experiments with two attention mechanisms, extracting features from various layers of GPDiT-B and GPDiT-H. important to note that GPDîT-B is trained on UCF-101, while GPDiT-H is trained on close-source open- domain dataset, so the representation ability measured both fit and generalized. The probing task is constructed by globally pooling features extracted from the frozen GPDiTmodel and training a logistic layer for the UCF-101 classification task. For each sample, we uniformly select 13 frames, spaced three frames apart, and pass them through the backbone without temporal rotation.
Main results. Figure 5a shows the classification accuracy for two different attention mechanisms on the GPDiT-B model. Notably, OF2 outper- forms OF by a significant margin, highlight- ing the enhanced representation performance when allowing interactions between clean con-
Table 3: FVD comparison of methods trained on UCF-101.
Model
#Params
Type
FVD
ExDDM-K2 S4
119 M
Video-DIT
394
OmniTokknizze
227M
Token-AR
314
text frames. This aligns with intuition, as interac- tions between clean frames enhance the model's ability to understand the content. We also ob- serve that the classification accuracy tends to peak at earlier layers, initially rising shallow layers before gradually declining. This is con- sistent with the classification results presented in REPA (33 where enhanced representation ability strengthens fitting in the shallow layers. This further demonstrates GPDiT's ability to fit and improve representation quality. Figure 5bb illustrates the classification accuracy of GPDiT-H-OF2 across different training steps and layers. As training progresses, the classification accuracy steadily improves. Additionally, since GPDiT-H-OF2 is zero-shot on the UCF-101 dataset, accuracy peaks at the 2/3 layer, which is inconsistent with the results of GPDiT-B. Figure 5c demonstrates the corre- lation between the generation metric (FVD) and classification accuracy for GPDÎT-H-OF2. There a clear positive correlation between generative capability and representational ability, indicating that as training progresses, both the generative performance and the representation ability of GPDiT improve simultaneously.
ACDiT [?
130M
Frame-AR
376
MAGI [?
850M
Frame-AR
FAR 10
130M
Frame-AR
194
GPDiT-B-OF GPDiT-B-OF2
80M
Frame-AR
80M
Frame-AR
214
Attention Varlants
Accuracy
FVD vs. Top Accuracy
100
230
100
90-
225
Accuracy (53
(96)
Accuracy 190
300
80
200
204
Frechet Video
175
40
GPOT---O22 GPOISb-O
25tk stage
30tk stops
PVD [Lower is Better)
15070
400k stops
Top Acouracy Osl
75
101112
222324
100
250
Training Iterations (k)
(a) Classification accuracy of (b) Classification accuracy of GPDiT-B between OF and OF2 GPDIT-H-OF2 across different accuracy across training steps.
(c) FVD of GPDIT-H-ON H-OF2 and Top
attention mechanisms across layers and training iterations.
different layers.
Figure 5: Linear probing performance of GPDIT across different training settings.
5.4
Video Few-shot Learning
The pre-trained GPDIT exhibits strong representational ability, and our AR paradigm enables condi- tioning via sequence concatenation, allowing easy generalization to other tasks without the need for additional modules like VACE [18] or IP-Adapter [51]. This motivates the investigation of the few- shot learning capabilities of pre-trained models across multiple tasks, including grayscale conversion, depth estimation, human detection, image colorization, canny edge-to-image reconstruction, and two style transfer applications. The pretrained GPDiT-H model undergoes 500 iterations of fine-tuning with batch size of 4, optimized to generate transformations conditioned on both input images and
8

---

contextual demonstrations. During testing. the model uses two (source, target) pairs as dynamic conditioning inputs generate the transformed output for an unseen source image.
Main results. Figure and Figure show that GPDiT' able to transfer to multiple downstream tasks after few-shot learning. It is clearly demonstrated that GPDiT easily convert color images to black-and-whie and vice versa. In the Human Detection task, the model accurately distinguishes the number of people and their body skeleton. Additionally, it supports controllable editing by generating controlled instances using edge maps. For instance, Figure shows that the bird generated in the Canny Edge to Image task follows the contours with fine detail. We also explored popular style transfers. such as TikTok-style dice-to-crrtonn transformations and GPT4o-Ghibl art style switches (Figure 7 Additionally, since only 20 shots are needed for few-shot learning, similar to GPT-2, this suggests the potential for larger GPDiT models to exhibit emergent In Context Learning (ICL) abilities, as seen the transition from GPT-2 to GPT-3.
Gray
Depth Estimation
Human Detection
In-coment Examples
Input
Output
OAPN
RRXP
Figure 6: Extracting Low Information from Images: Skeleton, Depth, and Grayscale Transformations.
Image Colorization
Canny Image
Style Transfer
Style Transfer
Inrcontett Exampés
Impott
Outpu
Figure 7: Results of Style Transfer and Conditional Generation: Image Colorization, Bige-to-Image
and Two Style Transfer Tasks.
Discussion
In this work, we present novel framework that unifies autoregressive modeling and diffusion for video generation. Our method incorporates a lightweight attention mechanism that leverages temporal redundancy to reduce computational overhead, as well as a parameter-free conditioning strategy to efficiently inject temporal information. These design choices enable faster training and inference without sacrificing model performance. Extensive experiments demonstrate that our model achieves state-of-the representation, and robust generalization in few-shot multi-task settings, underscoring versatility across various video modeling tasks.
seaation-bseed time-
performance in video generation, competitive results in video

---

Limitations. Due to resource constraints, we are unable to scale our experiments to larger config- urations. In future work, we plan to explore larger-scale models and investigate their potential for in-context learning. While our model demonstrates strong representational capabilities, it is currently limited to the video modality. The design choice of conditioning along the sequence dimension enables seamless integration of other modalities, allowing natural extensibility to multi- inputs, such as language. We intend to explore this unified generative-unddrrsaddng model future research.
References
[1] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563-22575, 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[4] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[5] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusson forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081-24122 2024.
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to- text-to-image synthesis. arXiv preprint arXiv:2310.00426. 2023.
[7] Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14416 14169, 2024.
[8] Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, and Jun Xiao. Vid-gpt: Introducing gpt-style autoregressive generation in video diffusion models. arXiv preprint arXiv:2406.1094 2024.
[9] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In European Conference on Computer Vision, pages 102-118. Springer, 2022.
[10] Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503. 19325, 2025.
[11) Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221, 2022.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840-6855 6851, 2020.
[14] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-88 1186333-646, 2022.
10

---

[15] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.1586. 2022.
[16] Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei-Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive conditional modeling and diffusion transformer. arXiv preprint arXiv:2412.07720, 2024.
[17] Haoyang Huang, Guoqing Ma, Nan Duan, Xing Chen, Changyi Wan, Ranchen Ming, Tianyu Wang. Bo Wang, Zhiying Lu, Aojie Li, al. Step-video-ti2y technical report: A state-of-the-an text-driven image-to-vide generation model. arXiv preprint arXiv:2503.1125/ 2025.
[18] Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598 2025.
[19] Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo- difo-diffusion: Generating infinite videos from text without training. arXiv preprint arXiv:2405. 1473, 2024.
[20] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming- Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023.
[21] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 2024.
[22] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[23] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209. 03003, 2022.
[24] Yaofang Liu, Yumeng Ren, Xiaodong Cun, Aitor Artola, Yang Liu, Tieyong Zeng, Raymond H Chan, and Jean-michel Morel. Redefining temporal modeling in video diffusion: The vectorized timestep approach. arXiv preprint arXiv:2410.0316)) 2024.
[25] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint arXiv:2303.065 0800000 2023.
[26] Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-12v technical report: The practice, challenges, and future of video foundation model. arXiv preprint 1024, 2025.
[27] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Fijnden and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23-40. Springer, 2024.
[28] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.030445 2024.
[29] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05444 2023.
[30] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEF/CV international conference on computer vision, pages 4195-4205, 2023.
[31] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624 2023.

---

[32] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[33] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[34] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
[35] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278 21122288-22294. 2022.
[36] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-videws video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[37] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. arXiv preprint arXiv:2502.06764 2025.
[38] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv: 13466, 2020.
[39] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint 121.v: 2012.
[40] Mingzhen Sun, Weining Wang, Gen Li, Jiawei Liu, Jiahui Sun, Wanquan Feng, Shanshan Lao, Si Zhou, Qian He, and Jing Liu. Ar-diffusion: Asynchronous video generation with auto-regressive diffusion. arXiv preprint arXivv2503.0744 2025.
[41] Mingzhen Sun, Weining Wang, Xinxin Zhu, and Jing Liu. Moso: Decomposing motion, scene and object for video prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18727-18737, 2023.
[42] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[43] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-War Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. et al. Wan: Open and advanced large-scale vidco generative models. arXiv preprint arXiv:2503.2 250::22003.233444 2025.
[44] Cong Wang, Jiaxi Gu, Panwen Hu, Yuanfan Guo, Xiao Dong, Hang Xu, and Xiaodan Liang. Dreamvideo: High-fidelity image-to-video generation with image retention and text guidance. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[45] Jiuniu Wang, Hangjic Yuan, Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint S.NV::2308005777 2023.
[46] Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng. Zuxuan Wu, and Yu-Gang Jiang. Omnitok- enizer: A joint image-video tokenizer for visual generation. Advances in Neural Information Processing Systems. 37:28281-28295 2024.
[47] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma. Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint trXiv:2307.06944 2023.
[48] Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, and Yang Zhou. Progressive autoregressive video diffusion models. arXiv preprint 20Xiv:2410.08151 2024.
12

---

[49] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mst-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[50] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXi:2200..1117, 2021.
[51] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXivv2308.06671, 2023.
[52] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokennizer is key to visual generation. arXiv preprint arXiv:2310005533,2222 2023.
[53] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06322 2024.
[54] Zhicheng Zhang. Junyao Hu, Wentao Cheng. Danda Paudel, and Jufeng Yang. Extdm: Distri- bution extrapolation diffusion model for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19310-19320, 2024.
[55] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.1101 2022.
[56] Deyu Zhou, Quan Sun, Yuang Peng. Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, Xiangyu Zhang, Lionel M Ni, et al. Taming teacher forcing for masked autoregressive video generation. arXiv preprint arXiv:2501.12335 2025.
13

---

A Additional Experimental Details and Results
A.1 Video generation results
We begin by evaluating the effect of our lightweight attention variant on training convergence, in comparison to standard causal attention. As shown in Figure the OF2 model exhibits slight improvement training convergence speed compared to the OF one.
Training Loss Curve
GPDIT-B-OF
0.18
GPDIT-B-OF2
0.16
0.14
Loss
0.12
0.10
0.08
0.06
Ok
50k
100k
150k
200k
250k
300k
350k
400k
Iteration
Figure 8: Training loss comparison of GPDiT-B-OR -BOOF and GPDiT-B-OF
Figure 9 and Figure 10 showcase video prediction results conditioned on camera motion. Each example generated from imput frames and extended by 16 predicted frames, sampled every fourth frame from the MovieGenBench dataset. The top row presents the ground truth, and the bottom row shows the predicted frames. These examples demonstrate the model's 's ability accurately anticipate dynamic camera movements and physical scene transitions.
A.2 Visual Question Answering results
To further validate the discriminative capability of our model's representations, we replace the vision encoder pretrained CLIP in Video-ChatGPT (29) with the first 18 layers of our trained GPDiT-H model as the feature extractor. Correspondingly, we replace the original linear layer with new linear layer matching the output dimensions of GPDiT-H. Following the standard practice of Video- ChatGPT, we freeze the parameters of the feature extractor and train only the linear layer. The training batch size is set to 4, the learning rate is set to 2e-5, and the number of training epochs is 6.
It worth noting that our model is not trained in language modality and solely based video modality for representation learning. Despite this, as illustrated in Table/4 our model achieves an accuracy of 28.2% AetivityNet-QA, compared to the original 35.2%. This result further validates the competitive discriminative capability of our model's representations, demonstrating promising potential for future multi-modality integration and training.
Table 4: Activity Net-QA
Model
Accuracy
Score
Video-ChaIGPT
35.2
2.8
GPDiT-H-OF2-Long
28.2
1.54
14

---

Figure 9: Camera Motion Control Prediction (Camera Rotation).
Figure 10: Camera Motion Control Prediction (First: Far-to-Near, Second: Left-to-Ree tiight..
A.3 More Video Few-shot Learning results
We conducted two sets experiments to evaluate the model's generalization ability. In the first setting, generalization from a known style to the real-face domain, the model was trained on 11 face- style-to-real-face translation tasks using approximately images from of the 11 styles seen during training were introduced, with no overlap between
video samples per style. During testing,
15

---

the training and test sets. After around 20 epochs of training, the model successfully generated realistic face outputs conditioned on previously unseen test images, demonstrating strong intra-style generalization. In the second setting, few-shot generalization from real faces to novel style, the model was trained on real-face-to- 10-style mappings and evaluated on one previously unseen target style. During inference, we provided the corresponding style conditions for this novel style, and the model was able to synthesize outputs that accurately reflected the unseen target style, highlighting its few-shot generalization capability and potential for in-context learning.
Clean Frame
Denoised Frame
Multi Style To Real
Real To
Unseen Style
Figure 11: Qualitative results demonstrating generalization from multiple styles to the real-face domain and few-shot adaptation from real faces to unseen style.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and introduction accurately reflect the paper's contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Primarily covered in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
We do not include theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We explained our model architecture in Section and experiment settings in Section
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We will release code with instructions to reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.SSubmisioooooiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We explained our settings in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Evaluating video models computationally expensive. We adopt the com- monly used approach of generating multiple videos and averaging the measurements; however, we do not provide error bars in our evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We explained our settings in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We do not believe that our work has any harmful consequences as layed out in the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work involves a new model architecture. does not have negative impact on society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite original papers and sources for code and datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release code with detailed documentation and appropriate license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing or human-subjec research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or human-subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used solely for writing, editing, formatting purposes and does not affect the core methodology, scientific rigor, or originality of the research.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23


Reviews:
[{"review_id": "v8kDPgB8PY", "reviewer": "Reviewer_g1aP", "summary": "This paper proposed a novel framework GPDiT that models AR diffusion for long-range video synthesis within a continuous latent space. GPDiT autoregressively predicts future latent frames using a diffusion loss. The paper highlights two key innovations: a lightweight causal attention variant that reduces computational costs by eliminating attention computation between clean frames during training, and a parameter-free rotation-based time-conditioning mechanism that reinterprets noise injection as a rotation in a complex plane, thus removing the need for adaLN-Zero and its associated parameters. Experiments demonstrate GPDiT's strong performance in video generation quality, video representation ability, and few-shot learning tasks, positioning it as an effective framework for continuous video modeling.", "questions": "See weaknesses.", "limitations": "The authors acknowledge resource constraints limiting larger-scale experiments and the current model's limitation to the video modality, with plans for future multi-modal extensions. The streaming video generation for longer length is still hindered by error accumulation. The computation efficiency problem addressed by this paper is not that important.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The lightweight causal attention and parameter-free rotation-based time-conditioning is novel, addressing critical challenges in streaming video generation.\n- The paper provides clear explanations and illustrations (Figure 2 and 3) of the proposed attention mechanisms and the rotation-based time conditioning. And provides a clear analysis of computational savings from the lightweight causal attention.\n- The experimental results demonstrate competitive performance across multiple tasks: video generation, video representation (validated through linear probing), and few-shot learning (transferring to various downstream tasks with minimal fine-tuning), which presents GPDiT's potential as a unified model for visual understanding and generation.\n\nWeaknesses\n- While the paper mentions the potential for extensibility to multi-modal inputs like language, the current experiments are limited to video. The scale of the experiments (2B model on MSRVTT ) is not large enough. We cannot understand its potential in scaling up further. Showing scaling curves between the number of paramsters/tokens and accuracy is better.\n- The paper lacks a blation study for the parameter-free rotation-based time-conditioning mechanism. A comparison against traditional learned conditioning methods (e.g., adaLN-Zero) essential to validate its effectiveness and efficiency claims.\n- Although the paper discusses the theoretical computational efficiency improvements of the lightweight causal attention, it does not provide empirical validation (e.g., training time, inference speed, or memory usage comparisons) to support these claims.\n- The core challenge in streaming video generation is the error accumulation. There are many other choices to improve the computation efficiency, e.g., sparse attention, consistency distillation, etc. This paper tackles a problem that is not that important. And whether the proposed strategy can be incorporated to the other acceleration strategies remains unknown.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns well. I keep my rating.", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below are our point-by-point responses to your comments:\n\n\n> ### W1: Extension to Text-to-Video experiments and Scaling curves\n\nDue to NeurIPS submission restrictions, we are unable to include visualizations of our results. Instead, we provide the loss vs. epoch metrics for the text-to-video (T2V) experiments. Specifically, we report the training loss across epochs for GPDiT-H (2B parameters).\nThe steadily decreasing loss indicates the potential of our model to generalize effectively to T2V tasks. The detailed results are presented in the table below. Due to current limitations in computational resources and time, we are unable to perform full-scale experiments at this stage. \n\n#### Table 1: Text-to-Video Generation with GPDiT-H on the ConceptualCaption12m and Panda7m Dataset \n\n| Iteration | 500 | 1000 | 1500 | 2000 | 2500 | 3000|\n|-----------|-----|-----|-----|-----|-----|-----|\n| Loss      | 0.168 | 0.142 | 0.142 | 0.138 |0.136| 0.139|\n\n\n\n\n> ### W2: Lack of ablation study\n\nTo evaluate the effectiveness of our proposed rotation-based time injection strategy, we conduct an ablation study comparing it with standard AdaLN-Zero and AdaLN-Single [1]. The following results demonstrate that the rotation-based strategy achieves better memory efficiency while maintaining comparable performance to the baseline methods.\n\n\n#### Table 2: Ablation of time-conditioning mechanism  with GPDiT-B on ConceptualCaption12m and Panda7m Dataset \n| Method   | Metric | 10K  | 15K  | 20K  | 25K  |\n|----------|--------|------|------|------|------|\n| Rotation | Loss      | 0.194 | 0.184 | 0.184 | 0.183 |\n|          | Memory | 34G | 34G | 34G | 34G |\n|               | MaxGradNorm | 0.313 | 0.281 | 0.259 | 0.259 |\n| adaLN-Zero    | Loss   | 0.178 | 0.173 | 0.176 | 0.175 |\n|               | Memory | 36G | 36G | 36G | 36G |\n|               | MaxGradNorm | 0.229 | 0.219 | 0.216 | 0.217 |\n| adaLN-single  | Loss   | 0.185 | 0.182 | 0.177 | 0.174 |\n|               | Memory | 34.6G | 34.6G | 34.6G | 3.3G |\n|               | MaxGradNorm | 0.291 | 0.281 | 0.276 | 0.267 |\n\n[1]Pixart-α: Fast training of diffusion transformer for\nphotorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.\n\n> ### W3: Empirical validation of effectiveness\n\nWe conducted experiments on the GPDiT-B model to evaluate the inference efficiency of our proposed method. To evaluate inference efficiency, we configured the denoising process with 30 steps and generated videos with 33 frames. The following table reports the average time required to generate a single video under this setting. \n \n#### Table 1: Empirical validation of effectiveness on GPDiT-B\n| Method              | inference time (second/iter)|\n| ------------------ | --------------------- |\n| O(F) |     3.34             |\n| O(F^2)| 3.68   |\n\n\n\n> ### W4: Error accumulation \n\nWe acknowledge that error accumulation remains a central and unsolved challenge in video generation. By employing teacher forcing and conditioning on clean frames, we suggest our method can alleviate error accumulation in long-range video generation. However, the central contribution of our paper lies in **combining autoregressive and diffusion models in the continuous space**, a direction that has been not fully explored. This unified framework enables us to better model temporal dependencies while maintaining high-quality generation."}, {"review_id": "hld3dz0rw7", "reviewer": "Reviewer_xKDW", "summary": "This paper introduces GPDiT, a novel hybrid framework that unifies autoregressive modeling and diffusion processes in a continuous latent space for long-range video synthesis. Its core contributions are:\n\n1. Continuous Autoregressive Diffusion: Instead of discrete tokens, GPDiT predicts future latent frames autoregressively using a diffusion loss, enabling joint optimization of motion dynamics and semantic consistency across frames.\n2. Efficiency Innovations:\n   - A lightweight causal attention variant reduces computational overhead.\n   - A parameter-free rotation-based time-conditioning mechanism enhances temporal modeling.\n3. Multifaceted Capabilities: Experiments demonstrate GPDiT excels in:\n   - Video generation quality (long-range consistency),\n   - Representation learning (e.g., feature transferability),\n   - Few-shot adaptation tasks.\nThe work establishes continuous latent space as a viable paradigm for scalable video modeling.", "questions": "1. Quantification of Efficiency Gains\n   - Question: Lightweight attention claims \"improved training/inference efficiency\" but lacks metrics.\n2. Statistical Reliability of Few-Shot Experiments\n   - Question: Using merely 20 videos for few-shot evaluation (detection/colorization) risks overfitting. Was performance verified across data scales?\n3. Generalizability of Rotation Conditioning\n- Question: How does rotation injection handle non-square features (e.g., 768×320 videos)? Does complex-plane rotation require adaptation?", "limitations": "YES.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nQuality: Comprehensive experimental design covering video generation (UCF-101/MSR-VTT), representation learning (linear probing accuracy), and few-shot multi-task applications (object detection/colorization/style transfer). Appropriate metrics (FVD/FID/IS) and ablation studies validate the efficacy of the lightweight attention and rotation-based conditioning.\n\nOriginality : Two innovative contributions:\n\n1. Continuous latent space AR-diffusion framework: First unification of AR's causal modeling with diffusion's continuous optimization, resolving long-video semantic coherence (vs. discrete token AR models).\n2. Zero-parameter time conditioning: Rotation injection maps noise to complex-plane rotation, replacing adaLN-Zero and significantly reducing parameters.\n\nSignificance: Proposes a unified video generation-understanding framework. Few-shot experiments demonstrate cross-task generalization potential, with multimodal extensibility (Discussion section).\n\n\nWeaknesses:\n\n- Quality Issues: Missing critical baselines: No comparison with state-of-the-art DiT variants or AR-diffusion hybrids, undermining SOTA claims.\n- Significance Limitations: No quantification of lightweight attention's practical speedup, weakening efficiency claims.\n- Clarity: adding algorithm pseudocode would further improve reproducibility.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below are our point-by-point responses to your comments:\n\n> ### W1：Missing critical baselines\n\nAs our method integrates autoregressive and diffusion components, we conduct a thorough comparison with recent AR-diffusion hybrid baselines, including FAR[1], MAGI[2] and ACDIT[3]. These baselines represent the most recent advances in the field at the time of our study. We emphasize that our evaluations are fair and do not undermine existing state-of-the-art claims.\nAs NOVA[4] supports video-to-video generation, we will include it as a baseline in the revised version to provide a more comprehensive comparison.\n\n[1]Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025.\n\n[2]Taming teacher forcing for masked autoregressive video generation. Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[3]Acdit: Interpolating autoregressive conditional modeling and\ndiffusion transformer. arXiv preprint arXiv:2412.07720, 2024.\n\n[4]Autoregressive video generation without vector\nquantization. arXiv preprint arXiv:2412.14169, 2024.\n\n> ### W2 & Q1: Quantification of lightweight attention's practical speedup \n\nWe conducted experiments on the GPDiT-B model to evaluate the inference efficiency of our proposed method. To evaluate inference efficiency, we configured the denoising process with 30 steps and generated videos with 33 frames. The following table reports the average time required to generate a single video under this setting. \n \n#### Table 1: empirical validation of effectiveness on GPDiT-B\n| Method              | inference time (second/iter)|\n| ------------------ | --------------------- |\n| O(F) |     3.34             |\n| O(F^2)| 3.68   |\n\n\n> ### W3: Algorithm pseudocode\n\nTo improve clarity and reproducibility, we will include the algorithm pseudocode in the revised version.\n\n> ### Q2: Statistical Reliability of Few-Shot Experiments\n\nWe ensure that there is no overlap between the training and testing sets. In few-shot scenarios, using fewer examples aligns with the intended setup, while a larger dataset overall tends to improve model generalization and robustness.\n\n> ### Q3: Generalizability of Rotation Conditioning\n\nFor rotation conditioning, we apply the rotation along the feature dimension (i.e., the last dimension). As a result, it does not affect features with non-square shapes."}, {"review_id": "n5rzRKKRN7", "reviewer": "Reviewer_kYES", "summary": "This paper introduces GPDiT, A unified framework for video generation that combines autoregressive next-token prediction objective with a diffusion-based formulation, diverging from the more common discrete token-based approaches. GPDiT enhances model efficiency during both training and inference through a specialized causal attention mechanism that strategically bypasses attention score computation for (clean) input frames. Furthermore, The authors also propose a parameter-free rotational time conditioning mechanism as an alternative to adaLN-Zero layers used in prior DiT-based models. Two model variants are trained: GPDiT-B (12 layers, 85M parameters) and GPDiT-H (24 layers, 2B parameters) on UCF-101 and MSR-VTT, using sequences of 17 frames.Additionally, a longer variant, GPDiT-H-LONG, is trained with variable-length sequences (17–45 frames). Experimental results consistently demonstrate that GPDiT performs competitively or better than tested methods on both qualitative and quantitative metrics. Beyond video generation, GPDiT exhibits strong capabilities in learning robust representations, enabling various vision-based in-context learning tasks such as colorization, depth prediction, human pose detection, face-to-cartoon synthesis, and style transfer.", "questions": "1. Given that NOVA [7] can also perform image-to-video generation, the experimental evaluation section would greatly benefit from a direct quantitative and qualitative comparison against NOVA. What are the advantages and disadvantages of GPDiT compared to NOVA, especially considering the architectural differences.\n\n2. Table 2 presents evaluations for 12-frame video generation conditioned on 5 input frames. What are the performance and visual quality implications if the models are conditioned on only a single input frame? Does the model maintain coherence and quality under this more challenging setting?\n\n3. As an autoregressive model, the visual quality of generated frames might degrade over time. Could the authors provide an analysis or qualitative examples demonstrating how the visual quality of frames changes as the generation sequence extends, especially for longer video generations?\n\n4. The proposed parameter-free rotation-based time conditioning is a core contribution. How does its effectiveness compare against the adaLN-single layer proposed in PIXART-α [6], particularly concerning performance, stability, and computational overhead?\n\n5. The in-context learning experiments utilize models trained with custom-built SFT data featuring task-specific image pair samples. If the models are primarily trained on single-modality inputs (e.g., sequences of color images, grayscale images, or depth images), how well do they generalize and perform few-shot in-context learning when presented with mixed input types within the same context? This would explore the model's ability to learn and apply representations across different visual modalities.", "limitations": "The authors have adequately discussed the limitations of their approach.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- A significant strength of this work is the extensive evaluation of the proposed models beyond just video generation. The authors demonstrate the efficacy of GPDiT in learning high-quality video representations (validated through classification tasks) and its remarkable in-context learning abilities across a diverse range of vision tasks. This highlights the framework's versatility and potential for broader applications.\n- The introduction of a parameter-free, rotation-based time conditioning mechanism is a particularly interesting contribution. Its practical effectiveness, as evidenced by the experimental results, suggests a promising direction for more efficient and robust temporal conditioning in diffusion models.\n\n**Weaknesses**\n- While the underlying formulation of GPDiT is inherently extensible to text-to-video generation, the current work is solely trained and evaluated on visual data. Consequently, the authors are restricted to quantitative metrics like FVD, FID, and IS. This omission prevents a direct comparison with state-of-the-art text-to-video models and misses out on recently proposed and widely adopted prompt-based benchmarks such as VBench and MovieGenBench. Addressing text-to-video capabilities would significantly broaden the paper's impact.\n- The related work section lacks a comprehensive discussion distinguishing GPDiT from highly relevant text-to-video models such as NOVA [7] and CausVid. These models also leverage diffusion losses within an autoregressive framework. A more detailed comparison highlighting the architectural novelties, efficiency gains, or specific contributions of GPDiT over these similar approaches is essential for contextualizing the paper's advancements and demonstrating its unique value.\n\nCausVid:  Yin et al., From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, CVPR 2025 \n\n- The paper contains several vague and inconsistent experimental details that hinder clarity and reproducibility:\n\n(i) Line 236 states that \"GPDiT-H is trained on close-source open-domain dataset,\" which contradicts the earlier impression that models are trained exclusively on UCF-101 and MSR-VTT. Clarification on the specific datasets used for each model version is needed. \n\n(ii) There is an unexplained discrepancy between the qualitative results for video generation (16 frames conditioned on 13 initial frames) and the quantitative results (12-frame generations conditioned on 5 input frames). A clear rationale for these differing evaluation setups should be provided.\n\n**Minor Points** \n- Figure 3 caption: “OF^2 => O(F$^2$) and OF => O(F)", "was_revised": true, "final_justification": "There are still some small issues regarding the evaluation of some components, but the authors provided extra experiment and comment to most of my questions. Hence, I am now leaning towards acceptance.", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below are our point-by-point responses to your comments:\n> ### W1: Text-to-Video\n\nDue to NeurIPS submission restrictions, we are unable to include visualizations of our results. Instead, we provide the loss vs. epoch metrics for the text-to-video (T2V) experiments. Specifically, we report the training loss across epochs for GPDiT-H (2B parameters).\nThe steadily decreasing loss indicates the potential of our model to generalize effectively to T2V tasks. The detailed results are presented in the table below. \n\n#### Table 1: Text-to-Video Generation with GPDiT-H on the ConceptualCaption12m and Panda7m Dataset \n\n\n| Iteration | 500 | 1000 | 1500 | 2000 | 2500 | 3000|\n|-----------|-----|-----|-----|-----|-----|-----|\n| Loss      | 0.168 | 0.142 | 0.142 | 0.138 |0.136| 0.139|\n\n\n\n> ### W2: Comprehensive discussion with other works\n\nAs our method integrates both autoregressive and diffusion components, we conduct a thorough comparison with recent AR-diffusion hybrid baselines, including FAR [1], MAGI [2], and ACDIT [3], which represent the latest advances in this area at the time of our study.\n\nWhile we acknowledge that a discussion of similarities and differences with NOVA[4] and CausalVid[5] would further highlight the advancements and unique contributions of our method, we note that both NOVA and CausalVid mainly target the text-to-video task. In particular, CausalVid relies on a separate bidirectional teacher model and employs asymmetric distillation to train a causal student. In contrast, our method is entirely self-contained and trained from scratch, requiring no teacher supervision. As NOVA supports video-to-video generation, we will include it as a baseline in the revised version to provide a more comprehensive comparison.\n\n[1]Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025.\n\n[2]Taming teacher forcing for masked autoregressive video generation. Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[3]Acdit: Interpolating autoregressive conditional modeling and\ndiffusion transformer. arXiv preprint arXiv:2412.07720, 2024.\n\n[4]Autoregressive video generation without vector\nquantization. arXiv preprint arXiv:2412.14169, 2024.\n\n[5]CausVid: Yin et al., From Slow Bidirectional to Fast Autoregressive Video Diffusion Models, CVPR 2025\n\n\n> ### W3: Clarity and reproducibility\n\nAs stated in lines 213–214, GPDiT-H is trained on a closed-source open-domain dataset and evaluated in a zero-shot setting on UCF-101 and MSR-VTT, with no overlap between training and test data. Our closed-source dataset is constructed by mixing several publicly available datasets, including Panda-70M, Mixkit, Pixabay, and Pexels for video data, and LAION-400M for image data. We further apply a series of data filtering and cleaning procedures to ensure data quality and relevance.\nRegarding experimental alignment, since our WanVAE[1] compresses 4n+1 frames into n+1 latent frames, our c5p12 setting aligns with c4p12 in FAR[2]. The c13p16 setting is used to explore performance under longer-frame scenarios.\nWe will open-source our code and models, along with detailed instructions for reproduction.\n\n[1]Wan: Open and advanced large-scale video generative\nmodels. arXiv preprint arXiv:2503.20314, 2025.\n\n[2]Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025.\n\n\n> ### Q1: Text-to-Video\n\nsee Weakness 1\n\n> ### Q2: Visual Quality under single frame\n\n In this case, we observe that the generated video tends to remain static for the initial frames. Starting from around the 17th frame, the video typically begins to degrade, leading to incoherent or implausible results. \n\n> ### Q3: Long video generation\n\nWe also evaluated our model on longer video generation tasks (up to 70 frames). We observed that while the model is capable of generating coherent content during the early stages, noticeable degradation in visual quality begins to appear around the 40th frame. This degradation becomes increasingly severe, and by the 50th frame, the generation typically collapses, resulting in implausible or incoherent outputs. Nevertheless, considering that our model was only trained on sequences of up to 33 frames, these results demonstrate a certain degree of extrapolation capability beyond the training horizon.\n\n> ### Q4: Ablation study on time injection strategy\nTo evaluate the effectiveness of our proposed rotation-based time injection strategy, we conduct an ablation study comparing it with standard AdaLN-Zero and AdaLN-Single [1]. The following results demonstrate that the rotation-based strategy achieves better memory efficiency while maintaining comparable performance to the baseline methods.\n\n\n#### Table 2: Ablation of time-conditioning mechanism  with GPDiT-B on ConceptualCaption12m and Panda7m Dataset \n| Method   | Metric | 10K  | 15K  | 20K  | 25K  |\n|----------|--------|------|------|------|------|\n| Rotation | Loss      | 0.194 | 0.184 | 0.184 | 0.183 |\n|          | Memory | 34G | 34G | 34G | 34G |\n|               | MaxGradNorm | 0.313 | 0.281 | 0.259 | 0.259 |\n| adaLN-Zero    | Loss   | 0.178 | 0.173 | 0.176 | 0.175 |\n|               | Memory | 36G | 36G | 36G | 36G |\n|               | MaxGradNorm | 0.229 | 0.219 | 0.216 | 0.217 |\n| adaLN-single  | Loss   | 0.185 | 0.182 | 0.177 | 0.174 |\n|               | Memory | 34.6G | 34.6G | 34.6G | 3.3G |\n|               | MaxGradNorm | 0.291 | 0.281 | 0.276 | 0.267 |\n\n\n[1]Pixart-α: Fast training of diffusion transformer for\nphotorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.\n\n\n> ### Q5: Mixed input types for few-shot learning\n\nThank you for your question. We are not entirely sure we understand your point — could you please clarify it in more detail? Regarding few-shot learning, if a task is not included during training, the model generally lacks the ability to perform it effectively at test time."}, {"review_id": "HHNAMLhb77", "reviewer": "Reviewer_FKsG", "summary": "This paper introduces GPDiT to unify the advantages of diffusion and autoregressive modeling for long-range video generation task. GPDiT autoregressively produces next latent frames through diffusion loss, while achieving plausible modeling of motion dynamics and consistency across frames. The proposed method improves both generation quality and representation capabilities. Moreover, it also presents a lightweight causal attention and parameter-free rotation-based time-conditioning mechanism to improve training and inference efficiency. Comprehensive experiments show that GPDiT obtains superior performance in video generation, video representation, and few-shot learning tasks.", "questions": "1. For Video Few-shot Learning, how many training samples are required for these tasks to acquire in-context learning capabilities, and are there differences between tasks of varying difficulty levels?\n2. Has the author attempted more complex in-context learning tasks, such as image editing tasks?\n3. The method in this paper is implemented based on Wan's VAE, which compresses multi-frame video frames into a single latent. Then, when migrating to image tasks, how are these images encoded?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper introduces GPDiT with framewise causal attention to improve temporal consistency of long videos, as well as a lightweight causal attention to reduce computational costs.\n2. This paper presents a rotation-based conditioning strategy to provide a parameter-free approache to inject time information.\n3. Extensive experimental results show that the proposed GPDiT obtains comparable performance on video generation and effective extension in few-shot learning tasks.\n \n**Weaknesses**\n1. Lack of comparison with the baselines for popular video generation. The authors only provided comparisons with earlier methods, and are encouraged to include more advanced methods, such as CogVideo-X, HunyuanVideo, Wan.\n2. The core contribution of this paper is the proposal of \"Lightweight Causal Attention\", but the authors did not quantitatively explore the training and sampling efficiency of this technique.\n3. There is a lack of exploration on \"injecting time information\". The authors proposed a new \"Timestep Conditioning Injection\" method in Sec 4.2, but did not conduct ablation studies on this technique.", "was_revised": true, "final_justification": "My concerns have been well addressed by the rebuttal, so I am glad to increase my rating to 5.", "rebuttal": "Thank you for taking the time to review our manuscript. We greatly appreciate your valuable feedback. Below are our point-by-point responses to your comments:\n\n> ### W1: Comparison with CogVideo-X, Hunyuan-Video, Wan.\n\nAs our paper primarily focuses on video-to-video gen-\neration, we do not include text-to-video (T2V) meth-\nods such as HunyuanVideo and Wan as baselines.\nHowever, we further extend our method to the T2V\nsetting, and the results are presented in W1 to Reviewer g1aP.\n\nAs our method integrates autoregressive and diffusion components, we conduct a thorough comparison with recent AR-diffusion hybrid baselines, including FAR[1], MAGI[2] and ACDIT[3]. These baselines represent the most recent advances in the field at the time of our study. As NOVA[4] supports video-to-video generation, we will include it as a baseline in the revised version to provide a more comprehensive comparison.\n\n[1]Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025.\n\n[2]Taming teacher forcing for masked autoregressive video generation. Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[3]Acdit: Interpolating autoregressive conditional modeling and\ndiffusion transformer. arXiv preprint arXiv:2412.07720, 2024.\n\n[4]Autoregressive video generation without vector\nquantization. arXiv preprint arXiv:2412.14169, 2024.\n\n\n\n> ### W2: Quantification of lightweight attention's practical speedup\n\n\nWe conducted experiments on the GPDiT-B model to evaluate the inference efficiency of our proposed method. To evaluate inference efficiency, we configured the denoising process with 30 steps and generated videos with 33 frames. The following table reports the average time required to generate a single video under this setting. \n \n#### Table 1: empirical validation of effectiveness on GPDiT-B\n| Method              | inference time (second/iter)|\n| ------------------ | --------------------- |\n| O(F) |     3.34             |\n| O(F^2)| 3.68   |\n\n\n> ### W3: Lack of ablation study\n\nTo evaluate the effectiveness of our proposed rotation-based time injection strategy, we conduct an ablation study comparing it with standard AdaLN-Zero and AdaLN-Single [1]. The following results demonstrate that the rotation-based strategy achieves better memory efficiency while maintaining comparable performance to the baseline methods.\n\n\n#### Table 2: Ablation of time-conditioning mechanism  with GPDiT-B on ConceptualCaption12m and Panda7m Dataset \n| Method   | Metric | 10K  | 15K  | 20K  | 25K  |\n|----------|--------|------|------|------|------|\n| Rotation | Loss      | 0.194 | 0.184 | 0.184 | 0.183 |\n|          | Memory | 34G | 34G | 34G | 34G |\n|               | MaxGradNorm | 0.313 | 0.281 | 0.259 | 0.259 |\n| adaLN-Zero    | Loss   | 0.178 | 0.173 | 0.176 | 0.175 |\n|               | Memory | 36G | 36G | 36G | 36G |\n|               | MaxGradNorm | 0.229 | 0.219 | 0.216 | 0.217 |\n| adaLN-single  | Loss   | 0.185 | 0.182 | 0.177 | 0.174 |\n|               | Memory | 34.6G | 34.6G | 34.6G | 3.3G |\n|               | MaxGradNorm | 0.291 | 0.281 | 0.276 | 0.267 |\n\n\n\n[1]Pixart-α: Fast training of diffusion transformer for\nphotorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.\n\n\n> ### Q1: Video Few-shot Learning\n\nFor video few-shot learning, we sample 20 examples as the training set and additional non-overlapping examples as the test set. Each task is trained for only 500 iterations. This setting ensures a fair evaluation of the model’s generalization ability on unseen data. For tasks such as grayscale conversion, depth estimation, and human detection, the models converge quickly within 100 iterations. In contrast, the two style transfer tasks converge more slowly compared to the above tasks.\n\n> ### Q2: More complex in-context learning tasks\n\nWhile our current work does not cover complex in-context learning tasks, we believe that our model can be extended to more challenging scenarios by incorporating more diverse training data. We plan to evaluate more complex in-context learning tasks and include representative examples in our revised manuscript.\n\n> ### Q3: How to encode image\n\nDuring training, we add the padding to the image to adjust the comreession ratio of Wan's VAE. In our few-shot learning setting, we replicate the single input image four times to construct a 5-frame sequence before feeding it into the VAE."}]

Author rebuttal:

Meta-review: The paper introduces a network architecture for diffusion-based autoregressive long video synthesis. The design integrates per-frame-block latent diffusion with causal conditioning on previous clean blocks. Autoregressive video generation is a key research problem towards enabling interactive, streaming video generation beyond the limitation of a fixed video duration.

All reviewers have praised the novel architecture designs of causal attention and time conditioning for improved efficiency and quality. Extensive experiments span video generation, representation, and few-shot learning. The paper is well written with clear contributions beneficial for the broader community.
Decision: Accept (poster)