Autoregressive Video Generation without Vector Quantization
Authors: Haoge Deng
Ting Pan
Haiwen Diao
Zhengxiong Luo
Yufeng Cui
Huchuan Lu
Shiguang Shan
Yonggang Qi
Xinlong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Autoregressive Generation
Text-to-Video Generation
Continuous-valued Space
Primary area: generative models
TL;DR: An any-to-video autoregressive model that integrates temporal frame-by-frame prediction with spatial set-by-set prediction without discrete tokenizers.

Abstract: This paper presents a novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available at https://github.com/baaivision/NOVA.

Published as a conference paper at ICLR 2025
AUTOREGRESSIVE VIDEO GENERATION WITHOUT VEC- TOR QUANTIZATION
Haoge Deng! Ting Pan Haiwen Diao 4.5+ Zhengxiong Luo Yufeng Cui', Huchuan Lu", Shiguang Shan Yonggang Qi' Xinlong Wang*
Beijjing University of Posts and Telecommunications
2 Key Laboratory of Intelligent Information Processing, ICT, CAS
3 Universityy of Chinese Academy of Sciences
"Dalian University of Technology
Beijing Academy of Artificial Intelligence
The image shows an olderty with
Text Imngge
Text
This video house with rotating perspective...
Text to Video
NOVA
Image
Image Video
Video
Video
Figure 1: NOVA is non-quantized autoregressive model for efficient and flexible visual generation.
ABSTRACT
This paper presents novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-rrame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, train novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-ard image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available https: / /gitt t thub com/ www.bbaaiiiii.ooeeeee...e...iiie iin//VVA..
INTRODUCTION
Autoregressive large language models (LLMs) (Brown et al. (2020); Touvron et al. (2023)) have become a foundational architecture in natural language processing (NLP), exhibiting emerging capa- bilities in in-context learning and long-context reasoning. In autoregressive (AR) vision generation (2021); Ding et (2021); Yu et (2022); Yan al. (2021); Villegas et al. (2022); Kondratyuk et al. (2023); Wang et al. (2024a)) typically transform images or video clips into a discrete-valie token space using vector quantization (Van Den Oord et al. (2017);
domain, prior approaches (Ramesh
* Equual Contribution. This work was done when the first three authors were interns at Beijing Academy of Artificial Intelligence." "Corresponding Author: wangxinlong @ bagi.ac.cn. qiyg @ hupt.edu.cn

---

Published as a conference paper at ICLR 2025
Esser al. (2021)), which are then flattened into sequences for token-by-token prediction. However, it is challenging for vector-quantizee tokenizers to achieve high fidelity and high compression simul- tancously. More tokens are required for a high quality. Thus, the cost increases substantially with higher image resolutions or longer video sequences.
In contrast, video diffusion models (Brooks al. (2024); Kuaishou (2024); Blattmann et al. (2023)) learn with highly compressed video sequences in a compact continuous latent space. However, most of them only learn the joint distribution of fixed-length frames, lacking the flexibility generate videos with varied lengths. More importantly, they do not possess the in-context abilities of autoregressive models, i.e., solving diverse tasks in context with a unified model such as GPT for language.
In this work, we present NOVA, which addresses the issues above and enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-ser prediction. NOVA inspired by Emu3 (Wang al. (2024a)) for autoregressive video and multimodal generation, and MAR (Li et al. (2024b)) for non-quantized autoregressive image generation, which utilizes non-quantized vectors as visual tokens and performs set-by-set autoregressive prediction. While both are non-quantized autoregressive approaches, it is non-trivial at all from MAR to NOVA: 1) NOVA solves the challenges including efficiency, scalability, and mask schedule when learning more complex text-to- rext-to-image generation instead of
generation. 2) NOVA first predicts temporal frames sequentially and then predicts spatial sets within each frame. NOVA is the first to enable non-quantized autoregressive model for video generation.
Specifically, NOVA predicts each frame in a casual order temporally, and predicts each token set in a random order spatially. In this way, text-to-video generation can be regarded as fundamental task that implicitly and comprehensively encompasses various generation tasks (See Figure 1), including text-to-image, amage-to-video, text&image-to-vie to-vidoo, etc. With non-quantized tokenizers and a flexible autoregressive framework, NOVA simultaneously takes advantage of 1) high-fidelity and compact visual compression for low cost in training and inference, and 2) in-context abilities for integrating multiple visual generation tasks in a unified model.
For text-to-video generation, NOVA surpasses autoregressive counterparts in data efficiency, inference speed, and video fluency. while matching the performance of diffusion models of similar scale, e.g., achieving a VBench (Huang et al. (2024)) score of 80.1 with a processing speed of 2.75 FPS!, trained in only 342 GPU days on A100-40G. For text-to-image generation, NOVA achieves a GenEval (Ghosh et al. (2024)) score of 0.75, surpassing previous diffusion models with notably lower training cost, e.g., only 127 GPU days for training this state-of-the-e 0.6B model. Additionally, NOVA also demonstrates strong zero-shot generalization across various contexts. We believe that NOVA paves the way for next-generation video generation, offering possibilities for real-time and infinite video generation, beyond Sora-like video diffusion models.
RELATED WORKS
2.1 DIFFUSION MODELS FOR VISUAL GENERATION
Diffusion models (Ho al. (2020); Song et al. (2020)) have made significant advances in visual generation, including text-to-image tasks (Esser et al. (2024a); Betker et al. (2023a); Baldridge et al. (2024)) and text-to-video tasks (Brooks et al. (2024); Lin et al. (2024); Blattmann et al. (2023)). Image diffusion models typically model the joint distribution of fixed-length tokens in pixel (Ho et al. (2020); Nichol et al. (2021); Hoogeboom et al. (2023)) or latent space (Rombach et al. (2022a); Esser et al. (2024a); Betker et al. (2023a); Chen et al. (2023)). Besides, video diffusion models further introduce temporal layers to capture relationships between a fixed number of video frames. After training, additional tasks and modalities added by incorporating extra inference tricks (Meng et al. (2021)), structure moderation (Blattmann et al. (2023); Esser et al. (2023); Liew et al. (2023)), and adapter layers (Zhang et al. (2023b); Guo al. (2023)). Although these strategies can be composable, they stand in contrast to the autoregressive approaches (Kondratyuk et al. (2023); Hong et al. (2022); Radford (2018); Touvron et al. (2023)), which trains single model end-to-end for multi-task learning.
'The 2.75 FPS is measured on single NVIDIA A A100-40G GPU using batch size of 24.
2

---

Published as a conference paper at ICLR 2025
offering notable context scalability and zero-shot generalizability across diverse application scenarios, especially in extending video generation duration.
2.2 AUTOREGRESSIVE MODELS FOR VISUAL GENERATION
Raster-scan Autoregressive Models are typically implemented on the discrete-valued RGB pix- els (Kalchbrenner et al. (2017); Reed al. (2017)) latent space (Esser et al. (2021); Van Den Oord et al. (2017)), analogous to their language counterparts (Radford et al. (2019); Anil et al. (2023)). Recent studies involve autoregressive transformers generate token sequences in the raster-scan order for image generation (Ramesh et al. (2021); Ding et al. (2021; 2022); Yu et al. (2022); Sun et al. (2024b)), and video generation (Yan et al. (2021); Kondratyuk et al. (2023); Nash et al. (2022); Wang et al. (2024b)). Specifically, VAR (Tian et al. (2024)) introduces next-scale prediction to progressively process the token-by-token sequence across multiple resolutions, leading to improved image quality.
Masked Autoregressive Models further develop masked generative models (Chang et al. (2022)) to introduce a generalized autoregressive concept. They introduce a bidirectional transformer and predict randomly masked tokens by attending unmasked conditions. This makes up for the suboptimal modeling and inefficient inference of sequentially line-by-line strategy, which inspires a series of subsequent works in text-to-image (Chang al. (2023)) and text-to-video generation (Hong et al. (2022); Yu et al. (2023); Villegas et al. (2022)). Particularly, MAR (Li et al. (2024b)) decouples discrete tokenizers from autoregressive models and utilizes a diffusion procedure for per-token probability distributions. It is fully validated in the class-to-image field, holding great potential in the text-to-image domain. However, its application to text-to-video generation intuitively requires a masked autoregressive process across entire video frames, challenging multi-contex, learning and training efficiency. In contrast, our NOVA model breaks down video generation into frame-by- frame-by-frame -rame temporal predictions combined with spatial set-by-set predictions. This allows each frame to act as a meta causal unit, enabling extended video duration and zero-shot generalizability across various contexts. Besides, the subsequent spatial set-of-tokens prediction unlocks the power of bidirectional modeling patterns, enhancing inference efficiency while preserving visual quality and fidelity.
3 METHODOLOGY
We first review two categories of autoregressive video generation in Sec. 3.1. In Sec. 3.2-3.4, we introduce framework pipeline and implementation details of our NOVA, illustrated in Figure 2.
3.1 RETHINKING AUTOREGRESSIVE MODELS FOR VIDEO GENERATION
As mentioned above, we regard text-to-video generation and autoregressive (AR) model as the basic task and means, respectively. We briefly retrospect related technical back- ground. There exist two types of AR video generation approaches: (1) Token-by-token generation via raster- scan order. These studies perform causal per-token pre- diction within video frame sequence (Kondratyuk et al. (2023)), and decode vision tokens sequentially following the raster scan ordering (Wang et (2024a)), which defined as follows:
Table I: Symbology Settings.
N.
The number of all video tokens. The number of all video frames.
F. f
K.
The number of sets in image.
N
IC, 2N) =
II p (x (n-1)...
(1)
72
where C' indicates various condition contexts, e.g., label, text, image, and etc. Note that x,, denotes n-th token of N video raster-scale tokens. contrast, (2) Masked set-by-set generation in a random order treats all tokens within each video frame equally, using a bidirectional transformer decoder for per-set prediction (Yu et al. (2023)). However, this generalized autoregressive (AR) model is trained using synchronous modeling on large, fixed-length video frames, which can lead to poor scalability in context and issues with coherence over longer video durations. Hence, NOVA proposes a novel solution by decoupling per-set generation within a single video frame from the per-frame prediction across the entire video sequence. This allows NOVA to better handle both temporal causality and spatial relationships, providing more flexible and scalable AR framework.
3

---

Published as a conference paper at ICLR 2025
VAB
VAE
5
5
Diffusion MLP
Diffusion MLP
Diffuston MLP
Diffuston MLP
Autoregressive
Spatial Layers
Spetial Layers
Spatial Layers
Spatial Layers
Set Bet-by- Bet-by-Bet Prediction
Temporal Layers
(Autoregressive Frame-by Praaee Prediction)
Text Enooder
<BVV>
S2
A dog with pink collar looking out over the water.
Figure 2: NOVA framework and the inference process.
With text inputs, NOVA performs autore- gressive generation via temporal frame-by-frand prediction and spatial set-by-set prediction. Finally, implement diffusion denoising in castinuous-valuee space.
3.2 TEMPORAL AUTOREGRESSIVE MODELING VIA FRAME- FRAME PREDICTION
Inspired by (Zhuo et al. (2024)), we use pre-trained language model (Javaheripi et al. (2023)) to encode text prompts to features. To better control video dynamics, we use OpenCV (cv2) (Bradski (2000)) to compute the optical flow of sampled video frames. The average flow magnitude is used as a motion score and integrated with the prompt. Besides, we employ open-source 3D variational autoencoder (VAE) (Lin et al. (2024)) with temporal stride of and spatial stride of 8 to encode the video frames to the latent space. We add an additional learnable patch embedding layer with a spatial stride of align channels of latent video the subsequent transformer. Notably, next-token prediction in early AR models seems counter-intuitive for undirected visual patches within a single image and suffers from high latency during inference. In contrast, video frames can naturally be viewed as causal sequence, with each frame acting as meta unit for AR generation. Therefore, we implement block-wise causal masking attention depicted in Figure 3(a), ensuring that each frame can only attend to the text prompts, video flow, and its preceding frames, while allowing all current frame tokens to visible each other as follows:
(PP m. m. B.S,, ..... Sp. = II p P.m. Sy-1),
where P, m indicate text prompts and video flow respectively. Here, Sy denotes the overall tokens of f-th video frame, and represent learnable begin-of-video (BOV) embeddings for predicting the initial video frame, the number of which corresponds to the patch number of one single frame. Note that we add 1-D and 2-D sine-cosine embeddings (Vaswani et al. (2017)) with video frame features to indicate time and position information respectively, which are convenient for temporal and spatial extrapolation. From equation 2, we can reformulate text-to-image and image-to-videe generation p (S1 P. m. B) and p (S, Ø m, B, Si.... S/-1). This generalized causal process can synchronously model the condition contexts for each video frame, greatly enhancing training efficiency, and allowing the kv-cache technology for fast decoding procedure during inference.
3.3 SPATIAL AUTOREGRESSIVE MODELING VIA SET-BY-SET PREDICTION
Inspired by (Chang et al. (2022); Li et al. (2024b)). define each token set with multiple tokens from random directions as a meta causal token Figure 3(b). facilitating a generalized AR process with efficient parallel decoding. Notably, frame as indicator features to assist the spatial layers, gradually decoding randomly masked token sets within the corresponding image. However, this approach resulted in image structure collapse and
tried to utilize the temporal layers' outputs targeting one
4

---

Published as a conference paper at ICLR 2025
Temporal Pertoken vs. Per-frame Prediction
bb) Spatial Per-token vs. Per-set Prediction
Few
Prgeet
Image
Prodicted Taken
Madied Tobexs
Figure 3: Overview of our block-wise temporal and spatial generalized autoregressive attention. Different from per-token generation, NOVA regressively predicts each frame in a casual order across the temporal scale, and predicts each token in a random order across the spatial scale.
inconsistent video fluency over the increasing number of frames. We hypothesize that this occurs because the indicator features from adjacent frames are similar, making it difficult to accurately learn continuous and imperceptible motion changes without explicit modeling. Besides, the indicator features derived from the ground-trute contextual frame during training contribute to weak robustness and stability spatial AR layers against cumulative inference errors.
To address this issue, we introduce a Scaling and Shift Layer that reformulates cross-frame motion changes by learning relative distribution variations within a unified space, rather than directly modeling the unreferenced distribution of the current frame. Notably, we select the BOV-attended output of the temporal layers as the anchor feature set, as it serves as the initial feature set with significantly less noise accumulation than subsequent frame feature sets. Specifically, we first translate the features from current frame set into dimension -iise variance and mean parameters y and / via multi-layer perception (MLP). After that, we affine the normalized features from the anchor set into indicator features Sy, via channel-win -wise scale and shift operation. Specially, we explicitly set y = 1 and / = 0 for the first frame. With unmasked token features, we predict randomly masked visual tokens in set-by-set order through bidirectional paradigm, which can be formulated as follows:
p pS' St.)) ((.K)) II p (S S Sta)) S(,.-I))
where S", denotes the indicator features for generating f-th video frame, and S(F.)) denotes k-th token set of f-th video frame. We add 2-D sine-cosine embeddings with masked and unmasked tokens to indicate their relative position. This generalized spatial AR prediction leverages powerful bidirectional patterns within angle-image tokens and achieves efficient inference with parallel masked decoding. Notably, we incorporate post-norm layers before the residual connections in both temporal and spetial AR layers. Our empirical findings show that this design effectively addresses architectural and optimization challenges that previously hindered stable training in generalized video generation.
3.4 DIFFUSION PROCEDURE DENOISING FOR PER-TOKEN PREDICTION
During training, we import diffusion loss (Li et al. (2024b)) to estimate per-token probability in a continuous-value space. For example, we define one ground-truth token x, and corresponding NOVA's output as Zn, The loss function can be formulated as a denoising criterion:
C(xn | 2n) = Ez,e [||e - c9 (I It. 2m. 1|]
Here c is a Gaussian vector sampled from (0. I), and noisy data of = Vāyz, + -I- where āe a noise schedule (Nichol Dhariwal (2021)) indexed by time step 4. The noise estimator en is multiple MLP blocks parameterized by 0. The notation e (z I, zn) means that this network takes 2h as the input, and is conditional on both i and zn. We follow (Li et al. (2024b)) to sample by times during training for each image.
During inference, we sample x.! from a random Gaussian noise N/ I) and denoise it step-by-step by sequentially sampling 2 2 via - you (xt Vi-a. (1 + o,e, where o, is the noise level at time step f,, and is sampled from the Gaussian distribution (O, I).
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENT
4.1 EXPERIMENT SETUP
Datasets. We involve several diverse, curated, and high-quality datasets to facilitate the training of our NOVA. For text-to-image training, we initially curate 16M image-text pairs sourced from DataComp (Gadre et al. (2024)), COYO (Byeon et al. (2022)), Unsplash (UnsplashTeal (2020)), and JourneyDB (Sun et al. (2024a)). To explore the scaling properties of NOVA, we expanded the dataset to approximately 600M image-text pairs by selecting more images that have a minimum aesthetic score of 5.0 from LAION (Schuhmann et al. (2022)), DataComp and COYO. For text-to-vided training, we select 19M video-text pairs on a subset (Lin et al. (2024)) of Panda-70M (Chen et al. (2024b)) and internal video-text pairs. We further collect 1M of high-resolution video-text pairs from Pexels (PexelsTeam (2014)) to fine-tune our final video generation model. Following (Diao et al. (2024)), we train a caption engine based on Emu2-17B (Sun et al. (2023)) model to create high-quality descriptions for our image and video datasets. The maximum text length is set to 256.
Architectures. We mostly follow (Li et al. (2024b)) to build NOVA'S spatial AR layer and denoising MLP block, including layer sequence of LayerNorm (Lei Ba et al. (2016)). AdaLN (Huang & Belongie (2017)), linear layer, SiLU activation (Elfwing et al. (2018)), and another linear layer. We configure the temporal encoder, spatial encoder, and decoder with 16 layers each, using dimension of 768 (0.3B), 1024 (0.6B) or 1536 (1.4B). The denoising MLP consists of 3 blocks with a dimension of 1280. The spatial layers adopt the encoder-decoder architecture of MAR (Li et al. (2024b)), similar to MAE (He et al. (2022)). Specifically, the encoder processes the visible patches for reconstruction. The decoder further processes visible and masked patches for generation. To capture the image latent features, we employ a pre-trained and frozen VAE from (Lin et al. (2024)). which achieves 4x compression in the temporal dimension and 8 x 8 compression in the spatial dimension. We adopt the masking and diffusion schedulers from (Li et al. (2024b); Nichol & Dhariwal (2021)), using masking ratio between 0.7 and 1.0 during training, and progressively reducing it from 1.0 0 following a cosine schedule (Chang et al. (2023)) during inference. In line with common practice (Ho al. (2020)), we train with 1000-step noise schedule but default to 100 steps for inference.
Training details. NOVA is trained with sixteen A100 (40G) nodes. We utilize the AdamW op- timizer (Loshchilov et al. (2017)) (31 0.9, 32 = 0.95) with a weight decay of 0.02 and a base learning rate of Ic-4 in all experiments. The peak learning rate is adjusted for different batch sizes during training using the scaling rule (Goyal (2017)) : Ir = base x batchsize /56. We train text-to-image models from scratch and then load these weights to train text-to-video models.
Evaluation. We use Ten-Chhpeench (Huang et al. (2023)), GenEval (Ghosh et al. (2024)) and DPG-Bench (Hu et al. (2024) to assess the alignment between the generated images and text condition. We generate image samples for each of the original or rewritten (Wang et al. (2024a)) text prompts. Each image sample has a resolution of 512x512 or 1024 x 1024. We use VBench (Huang et al. (2024)) to evaluate the capacity of text-to-video generation across 16 dimensions. For a given text prompt, we randomly generate 5 samples, each with video size of 33x768 x480. We employ classifier-free guidance (Ho & Salimans (2022)) with a value of 7.0 along with 128 autoregressive steps to enhance the quality of the generated images and videos in all evaluation experiments.
4.2 MAIN RESULTS
NOVA outperforms existing text-to-image models with superior performance and efficiency. In Table 2, we compare NOVA with several recent rext-to-image models, including PixArt-a (Chen et al. (2023)). SD v1/v2 (Rombach et al. (2022b), SDXL (Podell et al. (2023)), DALL-E2 (Ramesh et al. (2022)), DALL-E3 (Betker et al. (2023b)), SD3 (Esser et al. (2024b)), LlamaGien (Sun et al. (2024b) and Emu3 ( (Wang et al. (2024a)). After text-to-image training. NOVA achieves state-of-the art performance on the GenEval benchmark, especially in generating a specified number of targets. Notably, NOVA also achieves leading results on T2l-Compeench and DPG-Bench, excelling at both the small model scale and data scale (requiring only 16% training overhead of the best competitor PixArt-a). Last but not least, our eex-eoo-vieeo model outperforms most specialized text-to-image models, e.g., SD v1/v2, SDXL and DALL-E2. This underscores the robustness and versatility of our model in multi-context scenarios, with text-to-video generation as the fundamental training task.

---

Published as a conference paper at ICLR 2025
Table 2: Text-to-image evaluation on various benchmarks. The best and second-best results are in blue
and
green. The data is from Huang et al. (2023), Wang et al. (2024a) and Esser et al. (2024b).
ModelSpec
12I-CompBench
GenEval
DPG-Bench Overall
Model
*parmms rimages
Color
Shape
Texture Overall Single
Two Counting Colors Position CalorAur
A100 days
Diffasion models
Ait--
0.68
25M
68.86
55.82
70.44
0.48
0.98
0.50
0.44
0.50
B
0.07
71.11
753
SD v1.5
18
24
57.50
57.24
42.19
0.44
0.97
0.38
0.35
0.76
D.14
0.06
63.18
SD v2.1
1B
28
56,91
44.95
19.82
0.50
0.98
0.37
0.44
0.85
0.07
0.17
SDXI.
2.68
63.69
54.08
56.37
0.35
0.98
0.44
0.39
0.85
0.15
0.23
74.65
DALL-E2
6.58
650M
57,50
51,61
63.74
0.52
0.94
0.66
0.49
0.77
0.10
0.19
DALL-KS
81.10
67. 50
80.70
0.67
0.96
0.87
0.47
0.83
D.45
0.45
83,50
SD3
28
0.62
0.98
0.74
0.63
0.67
0.34
0.36
84.10
Autorggessivv modily
LlamaGion
0.8B
60M
0.32
0.71
0.34
0.21
0.58
0.07
0.01
Hmaa3 (+ Rowsierr)
88
79.13
58 .6.46
74.22
0.66
0.99
0.81
0.42
0.80
D.49
0:45
81.60
NOVA NOVA(512X5122 (512x512)
0.6B
16M
70.75
55.98
69.79
0.66
0.98
0.85
0.58
0.83
0.20
0.18
81.76
127
Rewriter Videos
0.6B
16M
61.47
0.75
0.98
0.62
0.5B
36M
71.80
47.86
65.31
0.55
0.98
0.56
0.48
0.75
0.15
0.11
81.77
342
Videos Rewriter
0.6B
36M
59.16
72.45
0.71
0.52
NOVA (1024x 1024)
0.38
600M
73.35
57.28
70.09
0.67
0.98
0.86
0.53
0.84
0.32
0.52
80.60
267
NOVA (1024x1024) 1024)
0.6B
600M
56.99
69.50
0.69
0.56
NOVA (102 (0224x1020)
1.4B
600M
74.30
57.14
70.00
0.71
0.99
0.62
0.85
0.33
0.56
83.01
608
Table 3:
Text-to-video evaluation VBench. We have classified existing video generation methods into different categories for better clarity. The baseline data is sourced from Huang et al. (2024).
#purams
#videos
latency
Total Soure
Quality Score
Semantic Score
Aesthetic Quality
Object Class
Multiple Objjcts
Human Action
Spucial Relationship
Clased-sooree soorce models
Gen-2
80.58
73.03
90.92
89.20
66.91
Kling (2024-07)
81.85
83.39
75.68
61.21
87.24
68.05
93.40
73.03
50.86
Gen-3
82.92
84.11
75.17
63.34
87.81
53.64
96.4
65.09
54.57
Diffusion models (w/ SD in))
LaVie
25M
77.08
78.78
20:31
54.94
96.8
Show-1
4B
10M
78.93
$0.42
72.98
57.35
93.07
45.47
95.60
53.50
47.03
Animateliff-a2
10M
80.27
82:90
67.16
90.90
VideoCrafter.22..
2B
10M
80.44
82.20
73.42
63.13
92.55
40.66
95.00
35.86
55.29
T2V-Turbo (VCT)
28
10M
81.01
82.57
74.76
63.04
93.96
54.65
95.20
38.67
Diffisioo models
OpenSom-v1.1
10M
75.66
77.74
50.12
86.76
40.97
84:20
52.47
OpenSocaPlaa-V1.
IB
4.5M
60s
78.00
80.91
66.38
56.85
76.30
40.35
86.80
53.11
27.17
OpenSon-v1.2
32M
79.76
81.35
73.39
56.85
91.20
68, 36.56
Cog VigeiO
2B
35M
90s
80.91
82.18
75.83
60.82
83.37
62.63
98.00
69.90
51.14
Antorepeesive
Cog
9B
5.4M
67.01
72.06
46.83
38. 18
73.4
18.11
78.20
18:24
28.24
Emu3
80.96
84.09
59.64
86.17
77.71
68.73
NOVA
0.68
20M
12x
78.48
78.96
76.57
54,52
91.36
73.46
91.20
66.37
50.16
0.6B
20M
80.12
80.39
79.05
59.42
92.00
77.52
95.20
77.52
NOVA rivals diffusion text-to-video models and significantly suppresses the AR counterpart. We emphasize that the current version of our NOVA is designed to generate videos at 33 frames and can extend video length through the pre-filling of recently generated frames. We perform a quantitative analysis comparing NOVA against open-source and proprietary text-to-video models. As shown in Table 3, despite its significantly smaller size (0.6B vs. 9B), NOVA remarkably outperoorm Cog Video (Hong et al. (2022)) across a variety of text-to- evaluation metrics. also matches the latest SOTA model Emu3's (Wang et al. (2024a)) performance (80.12 vs. 80.96) with significantly smaller size (0.6B vs. 8B). Additionally, we compared NOVA with state-of-the-art diffusion models. This includes both the closed- source models (Runway (2023); Kuaishou (2024); Runway (2024)), as well as the open- source alternatives (Wang et al. (2023); Zhang et al. (2023a); Guo et al. (2024); Chen et al. (2024a); Li et al. (2024a); Zheng al. (2024); Lin et al. (2024); Yang et al. (2024)).
4.3 QUALITATIVE RESULTS
High-fidelity image and high-fluency video. We present a qualitative comparison of current leading image generation methods in Figure 4. NOVA demonstrates strong visual quality and fidelity across a range of prompts. We present text-to-video visualizations in Figure 5, which highlight NOVA's ability to capture multi-view perspectives, smooth object motion, and stable scene transitions.
7

---

Published as a conference paper at ICLR 2025
(1)
(2)
(8)
(6)
(
SD-XL
PixArt-a
Lumina-Next
NOVA
Figure 4: Text-to-image generation. Text prompts from left right: (1) "A digital artwork of
styled in a whimsical fashion..." ", (2) "A solitary lighthouse standing tall against a backdrop of stormy
seas and dark, rolling clouds", (3) "A vibrant bouquet of wildflowers on rustic wooden table", (4) "A selfie of an old man with a white beard", (5) "A serene, expansive beach with no people", (6) "A
blue apple and green cup." and (7) "A chicken the bottom of balloon."
(1)
Text prompt: : model of 1800s victorian house.
ca
Text prompt: cat wearing sunglasses and working as lifeguard a pool.
Text prompt: drone celebration Christmas and fireworks, starry sky, background.
Text-to-video generation. We highlight the keywords in red color. NOVA follows the text prompts and vividly captures the motion of subjects (i.e., 3D model, cat and fireworks).
Figure 5:
Zero-shot generalization on video extrapolation. By pre-filling generated frames. NOVA can produce videos that surpass the training length. For example, by shifting both the text and BOV embeddings, we generate videos that are up to twice the original length, as shown in Figure 6.
Zero-shot generalization on multiple contexts. By pre-filling the reference image, NOVA can generate videos from images, either with or without accompanying text. In Figure 7, provide qualitative example. show that NOVA can simulate realistic motions without text prompts.
4.4 ABLATION STUDY
Effectiveness of temporal autoregressive modeling. To highlight the advantages of temporal AR, we have facilitated spatial AR to finish video generation task. We observe less subject movement in videos under the same training iterations (Figure 8). Additionally, in zero-shot generalization across various contexts or vidco extrapolation, the network output exhibited more artifacts and temporal inconsistencies. Furthermore, this approach is not compatible with ky-cache acceleration during inference, leading to a linear increase in latency with the number of video frames.
Effectiveness of Scaling and Shift Layer. To capture cross-frame motion changes, employ simple yet effective scaling and shifting layer to explicitly model the relative distribution from the BOV-attended feature space. In Figure (a), demonstrate that this approach significantly reduces
8

---

Published as a conference paper at ICLR 2025
lighthouse ... a vibrant sunset sky... to a warm palette oranges and yellows ... has classio design with a domed top and a lantern room where the light would be housed, ...to intensify.
Figure 6: Zero-shot video extrapolation. We highlight the subjects in red and green respectively. The top images are generated, while the bottom images are extrapolated.
(1) Image To-Vodoee Without Text
23) Image- To- Video With Text
Vo-eedeo
Text-TTo-ae Video
(4) Text-To-Imase
Text casoade of water rushes down a rocky incline, frothing and churning as it descends, surrounded by rugged, layered rock formations.
Figure 7: Zero-shot generalization on multiple contexts. It is evident that NOVA successfully maintains temporal consistency objects, both with and without text. Such as ensuring "water continues to flow smoothly." This highlights NOVA's capability for zero-shot multitasking.
the drift between text-to-image and image-to- video generation losses. As we gradually decrease the inner rank of the MLP, training difficulty increases, leading to a more comprehensive and robust learning process for the network. However, extremely low rank values pose challenges for motion modeling, as they significantly limit the layer's representation capability (Figure 10). The rank is set to 24 by default in all text-to-vidco experiments, resulting in more accurate motion predictions.
Effectiveness Post-Norm Layer. Training large-scale image and video generation models (Ding et al. (2021); ChameleonTeam (2024)) from scratch often poses significant challenges with mixed precision, which is also observed in other visual recognition methods (Liu et al. (2022)). As shown in Figure 9b)), the training process with pre-normalization (Dosovitskiy et al. (2021)) suffers from numerical overflow and variance instability. We attempted various regularization techniques on the residual branch, such stochastic depth (Huang al. (2016)) and residual dropout (Vaswani et al. (2017)). but found them to be less effective. Inspired by (Liu et al. (2022)), we introduce nost-normalizztion and empirically discover that can effectively mitigate the residual accumulation of output embeddings compared to pre-normalization, resulting in a more stable training process.
5 CONCLUSION
In this paper, present NOVA, novel autoregressive model designed for both text-to-image and text-to-video generation. NOVA delivers exceptional image quality and video fluency while significantly minimizing training and inference overhead. Our key designs include temporal frame- by-frame prediction, spatial set-by-set generation, and continuous-span autoregressive modeling across various contexts. Extensive experiments demonstrate that NOVA achieves near-commercial

---

Published as a conference paper at ICLR 2025
frame 1
frame 6
frame 11
frame 16
frame 21
frame 26
W/ TAM
W/O TAM
W/ TAM
W/O TAM
Figure 8: Temporal autoregressive modeling (TAM) for video generation. We highlight the subtle changes in frames generated from the same prompt. Compared to spatial-only autoregressive method, the inclusion of TAM enables NOVA more accurately capture the dynamics of subject movement.
16
Fort
0.26
14
0.22
0.20
0.36
m-16
0.12
molella vikee We' sobotonet
tndeetssates n-24 (Wilee
10000
20000
30000
40000
50000
60000
1.0
20
Taining
BookeD
(a) Parameter decomposition for Scaling and Shift layer.
(b) Normalization layer position.
Figure 9: A Alation studies on NOVA's architecture components. carefully examine the two key stability factors in large-scale video generation training, as illustrated in (a) and (b).
quality in image generation, alongside promising fidelity and fluency in video generation. NOVA paves the way for next- gext-eenerttion video generation and world models. It offers valuable insights and possibilities for real-time and infinite video generation, going beyond Sora-like video diffusion models. As first step, we will continue scalable experiments with larger models and data scaling to explore NOVA's limits in future work.
rank-8
rank-16
rank-B 24
rank-32
w/o scale&ssiftt
First Frame
Last Frame
Figure 10: Visualization of decomposition ranks in the Scaling and Shift layer. The first row displays the results of the first frame, while the second row presents the results of the last frame.
Acknowledgement. We thank Fan Zhang for the valuable advice on Figure 2 of this work, and Xiaotong Li, Jinming Wu, Chengyue Wu, Zhen Li, Quan Sun. Yueze Wang, Jinsheng Wang and Xiaosong Zhang for the insightful discussions. We also acknowledge the support provided by other colleagues at the Beijing Academy of Artificial Intelligence (BAAI) towards this project. This work was supported by the National Key R&D Program of China (2022ZD01 16302), in part by Hainan Provincial Natural Science Foundation of China under No.624LALH00S, and in part by the Program for Youth Innovative Research Team of BUPT under No. YQD02.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Faton-Rosen, et al. Imagen 3. arXiv preprint arXiv:2408. 07009, 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/paperstdalle-3. pdf. 2(3):8, 2023a.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang. Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall---3 pdf. 2(3):8, 2023b.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation mod- els as world simulators, 2024. URL https://openes com/ wtt./ooo///aae/eeee remearch vi dilde--enneatee commeration-modee on-mode sas maaswwoldd-s mulators.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https://githut www/coo////////// katm//kkaabbaii////mmm coyo-dataset, 2022.
ThameleonTeam. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.098/8, 2024.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11322 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024a.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wer wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren. Ming-Hsuan Yang, al. Panda-70m: Captioning 70m videos with multiple cross- mods-mto teachers. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 13320-13331 2024b.

---

Published as a conference paper at ICLR 2025
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang. Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision- lainuae models. arXiv preprint arXiv:2406. 11832, 2024.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 19.19222--9833 2021.
Ming Ding,
Wendi Zheng, Wenyi Hong, and Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35: 16890-16902, 2022.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Stefan Elfwwng, Eiji Uchibe, and Kenji Doya. Sigmoid-weightee linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- nigh-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7346-7356, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024a.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024b.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
Goyal. Accurate large minibatch sg d: training imagenet in hour. arXiv preprint 2017.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307. 2007022307.047255, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representattinss 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 15979-15988, 2022.
Jonathan Ho and Tim Salimans. Classifier-free free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-653 6851, 2020.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale
pretraining for texi-to-video generation via transformers. arXiv preprint 1xXiv:2205.15866 2022.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part IV 14, pp. 646-661. Springer, 2016.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. Tži-compbench: A compre- hensive benchmark for wpendworld compositional text-to-image generation. Advances in Neural Information Processing Systems, 2023.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference computer vision, pp. 1501-1510, 2017.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818. 2024.
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference Machine Learning, pp. 1771-1779. PMLR, 2017.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023.
Kuaishou. Kling ai, 2024. URL httpt:
com/.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints, pp. arXiv-1607, 2016.
Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arXiv preprint drXiv:2405.1887 18750, 2024a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024b.
Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng. Magicedit: High- fidelity and temporally coherent video editing. arXiv preprint arXiv:2308. 14749, 2023.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, al. Open-sora plan: Open- large video generation model, 2024.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong. et al. Swin transformer v2: Scaling up capacity and resolution. Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 22009-12011 2022.
13

---

Published as a conference paper at ICLR 2025
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXi::771..5101. 2017.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint wrXiv:2/08.01073, 2021.
Charlie Nash, Joao Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint 17Xiv:2203.09494,, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, 1lya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffussion models. arXiv preprint arXiv:2112.10744 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-81211 8171. PMLR, 2021.
PexelsTeea. Pexels, royalty-free stock footage website, 2014.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference machine learning, pp. 8821-8831. Pmlr, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061155 1(2)33 2022.
Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo. Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In Interna- tional conference on machine learning, pp. 2912-2921. PMLR, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022b.
Runway. Gen-2: Generate novel videos with text, images or video clips, 2023. URL https: //runwayml com/ wwtp:rreeeeaeeeeeeeeeeeeeeaeeeee gen- 2.
Runway. Gen-3 alpha: A new frontier for video generation, 2024. URL https://runwaynl com/ httm///eeerrc///troooooo. -eaeuiiinnnggn------ alpha.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 35:25278-2523 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011 13446, 2020.
14

---

Published as a conference paper at ICLR 2025
Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 2024a.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 065525,00 2024b.
Quan Sun, Yufeng Cui, Xiaosong Zhang. Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. arXiv: 2312. 13286, 2023.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via sexte prediction. arXiv preprint arXiv:2404.02905, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv: 2307.09288 2023.
UnsplashTeam Unsplash dataset, 2020.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2022.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li. Qiying Yu, al. Emu3: Next-token prediction all you need. arXiv preprint 2409::2409, 18869, 2024a.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15/03, 2023.
Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757,, 2024b.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 10157, 2021.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 2408.2200006672 2024.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5. 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023.
David Junhao Zhang, Jay Zhangjie Wu, Jia- fia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818, 2023a.
15

---

Published as a conference paper at ICLR 2025
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to -ext-to-image diffusion models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3836-3847, 2023b.
Zangwei Zheng, Xiangyu Peng, Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URL https: / wigggtttuu..ooo com http://caaccccc/peeeeoeeee
Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu- Yun Wang, Zhanyu Ma, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit, arXiv preprint urXiv:2406. 18583, 2024.
16


Reviews:
[{"review_id": "cp9lwLxNB0", "reviewer": "Reviewer_GfSG", "summary": "This paper introduces an autoregressive model combined with a diffusion model as the prediction head, enabling video generation without vector quantization. The proposed approach, termed NOVA, maintains the causal property of AR models' temporal frame-by-frame prediction, while leveraging bidirectional modeling within individual frames (spatial set-by-set prediction).  NOVA achieves state-of-the-art text-to-image and text-to-video generation performance with significantly lower training costs and higher inference speed.", "questions": "- From my understanding, the generation of the next frame is achieved by one step of temporal autoregression (stage-1) followed by several steps of spatial autoregression (i.e.,  progressively reducing the masks from 1.0 to 0) (stage-2).  What's the time cost for each of these two stages?\n- As claimed in the paper, the Scaling and Shift Layer effectively reduces the cumulative inference errors. What is the limit of the model's video extrapolation capability? In other words, after how many autoregression steps will the cumulative errors severely affect the frame quality?\n  - A quantitative metric on frame quality vs. the number of autoregression steps might be helpful. Or some qualitative examples that show quality degradation in long-term autoregressive generation.\n- Suggestions:\n  - In Figure 7, better to provide the results of NOVA and compare them with the results of the simple baseline.\n  - Better to add frame ids in the qualitative examples.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- As a follower of MAR (Tianhong Li et al. (2024b)), this paper for the first time lifts the non-quantized AR model to video generation. In contrast to trivially modifying the 2D non-quantized MAR to a 3D version, they design the autoregressive modeling sequentially that integrates first temporal frame-by-frame prediction and then spatial set-by-set within each frame. This facilitates the model's ability of video extrapolation and potential  compatibility with kv-cache acceleration.\n- The model is trained on large-scale text-to-image and text-to-video datasets (trained from scratch) and shows high image and video generation quality compared to existing SOTA models. It will have a great potential contribution to the vision community If the pre-trained checkpoint is released.\n\n- This paper provides valuable empirical design spirits as vilified by their experiments:\n  - Instead of directly assigning the current frame of temporal layers’ outputs to the spatial layer as indicator features (for predicting the next frame), they propose to use the BOV-attended output as an anchor feature and inject the current frame's output via the Scale & Shift of LayerNorm. This technique improves the training stability and alleviates cumulative inference errors. It is a valuable prior (design choice) for subsequent studies on autoregressive long video generation.\n  - They conducted extensive ablation studies to show using post-norm layers before the residual connections is a better design choice for a smoother and more stable training process.\n\nTianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024b.", "weaknesses": "- Unclear training/inference details. \n  1. According to Figure 1. At training time, the model predicts a set of masked tokens of the 2nd frame. At inference time, the model progressively reduces the masked ratio from 1.0 to 0. However, as the 1st and 2nd frames have been generated (as the given conditional frames) in the Fig.1's example, the model should progressively unmask the 3rd frame. There seems to be some inconsistency between training and inference. In other words, for the example in Fig.1, which frame is modeled by $x_n^t$  ?\n  2. It might be helpful to provide a step-by-step explanation of how frames are generated during inference, particularly highlighting any differences from training.\n\n- Unclear video extrapolation setting.\n\n  1. The exact number of frames used during training is not clear. According to all the information available in the paper, the model is trained on samples with 29 frames (Line315: 2.4s x 12 FPS = 28.8 frames. Line 296: the model generates 29 frames for evaluation). But this is not clearly evidenced in the descriptions of training details.\n\n  2. It's unclear how context is handled when generating videos longer than the training length. Suppose that the model was trained on 29 frames. Is the context truncated when the length of video extrapolation exceeds 29 frames? For example, from my perspective, the video extrapolation is like: \n\n    ```\n    ...\n    [x_1,...,x_28]--> x_29 # training length is reached\n    [x_2,...,x_29]--> x_30  # earliest context (x_1) is truncated\n    ...\n    ```\n\n  3. How are the 1D sine-cosine temporal positional embeddings applied for frames beyond the training length? This information would help clarify the model's capabilities and limitations in video extrapolation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AXF1vBg0Au", "reviewer": "Reviewer_nKGc", "summary": "This paper proposes a video generation framework called NOVA, which is based on autoregressive modeling. NOVA performs temporal frame-by-frame prediction and spatial set-by-set prediction. This approach leverages the in-context learning and extrapolation advantages of autoregressive models while maintaining the efficiency of bidirectional modeling. Compared to existing video generative models, NOVA achieves higher data efficiency, faster inference speeds, and about similar video generation quality with fewer parameters.", "questions": "* Suggestion: To make the Scaling and Shift Layer easier to understand, the authors could mention its similarity to FiLM[1] or AdaIN[2].\n* Typo: Line 296 - each with\n\n[1] Perez, et al. \"Film: Visual reasoning with a general conditioning layer.\" AAAI 2018.\\\n[2] Huang, Xun, and Serge Belongie. \"Arbitrary style transfer in real-time with adaptive instance normalization.\" ICCV 2017.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* NOVA achieves state-of-the-art (SOTA) results in text-to-image (T2I) tasks.\n* NOVA shows much faster inference speeds than previous video generative models.\n* The method of combining temporal autoregressive and spatial bidirectional modeling is simple yet effective.\n* The Scaling and Shift Layer is also simple but effective. Also, the analysis of the layer is comprehensive.", "weaknesses": "* While NOVA achieves SOTA in T2I, this aspect feels like a straightforward extension of MAR[1] rather than a novel contribution.\n* For text-to-video (T2V), NOVA uses relatively less data and fewer parameters and has fast inference speeds but falls short in performance. Therefore, it needs further testing about scalability (i.e., if NOVA can match the performance of the open-source models in the main table when scaled up.). \n* There is a question about whether extrapolation is truly unique to autoregressive (AR) models. Diffusion models and bidirectional models could also potentially achieve extrapolation through a sliding window approach, which would need a comparative analysis.\n* The ablation study on frame-by-frame autoregressive modeling lacks clarity, which is critical given the importance of this topic to the authors' main arguments. The qualitative results in Figure 7. appear less convincing when viewed in images, and it’s unclear if NOVA did not have similar subtle limitations. A side-by-side comparison with the same text prompt or inclusion of video examples would be helpful.\n\nIn short, \\\nAlthough the methods presented are simple and novel, the primary claims are not clearly backed by the experiments. The main claims are: (1) NOVA combines the advantages of temporal autoregressive modeling with spatial bidirectional modeling, and (2) NOVA demonstrates data and parameter efficiency in practice. However, these claims are not sufficiently supported by the experimental results.\n\nIf the authors provide clearer support for their claims with additional analysis, I would be happy to raise my score.\n\n[1] Li, et al. \"Autoregressive Image Generation without Vector Quantization.\" arXiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5rP4pMcIpl", "reviewer": "Reviewer_yFXn", "summary": "The paper introduces NOVA, a novel autoregressive (AR) video generation model that leverages non-quantized tokenizers. NOVA aims to combine the advantages of high-fidelity and high-rate visual compression with in-context learning capabilities, enabling it to integrate multiple generative tasks within a unified framework. The model is designed to factorize AR video generation into temporal frame-by-frame prediction and spatial set-by-set prediction, allowing for efficient and effective video generation. The authors claim that NOVA outperforms existing diffusion models in terms of data efficiency, inference speed, and video fluency, while also demonstrating strong zero-shot generalization across various contexts.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) NOVA's framework is well-structured, combining temporal and spatial autoregressive modeling. This dual approach not only enhances the model's efficiency but also its ability to handle multiple generative tasks within a single model, showcasing the potential for in-context learning.\n\n2) The authors provide a thorough evaluation of NOVA, comparing it with state-of-the-art models across various metrics. The results demonstrate that NOVA not only matches but often surpasses the performance of diffusion models, particularly in terms of data efficiency and inference speed.", "weaknesses": "I think the key limitation of this work is the novelty, which seems like an extension of MAR on video generation task.\n\nIn Table 3, I don't see improvements in the proposed on the basis of previous diffusion-based methods. Are AR-based methods really needed for video generation tasks? Could the author clarify this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "73oDHqGKpS", "reviewer": "Reviewer_xK2A", "summary": "This paper proposes a video generation method NOVA, expands MAR[1] from image generation to video generation. However, this process is not smooth sailing. The author encounters problems such as MAR's insufficient ability to model long sequences and poor extrapolation ability. The author proposes a method in the time dimension. Frame-by-frame and set-by-set generated solution strategies in spatial dimensions achieve excellent text-to-image and text-to-video performance.\n\n[1] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024b.", "questions": "In order to ensure fairness when comparing with other methods in Table 2, you should submit the comparison results under the same settings. This is also to ensure that the settings of other methods can be unified when comparing in the future. And you should emphasize whether you conducted the evaluation under zero-shot or the result after fine-tuning on the training set.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. As far as I know, it is the first time to expand MAR into the general generation field (text-to-image, text-to-video, etc.), which is a very good attempt.\n\n2. The article achieves excellent text-to-image performance (on T2I CompBench) and stands out from diffusion models. Although I have some doubts about the evaluation setting and comparison methods described in the article, the results are still excellent.\n\n3. As a pre-training paradigm for generative models, especially video generation, this article consumes less resources and is relatively affordable.", "weaknesses": "1. The article's explanation of the model architecture is vague and difficult to understand:\n\n    ①The paper mentions temporal encoder, spatial encoder, and decoder with 16 layers each on line 260, but the full article does not explain what the encoder and decoder in spatial layers are.\n\n    ②In Figure 1, when Spatial Layers performs mask modeling on S2, it can directly obtain the complete embedding output by S2 after being encoded by Temporal Layers. Isn't this a kind of information leakage? However, in the actual inference generation At the time of S2, there was no known S2. We only had S1 before we get S2.\n\n2. As far as I know, in the evaluation on T2I-CompBench mentioned by the author in lines 292-295 of the article, there are only 300 evaluation prompts for each category, and the others are training prompts. And according to the setting of the original T2I-CompBench paper, each evaluation prompt should generate 10 images.\n\n3. In the comparison of Table 2, there is no comparison with the most advanced diffusion model. At least it should be compared with SD3 [2] and DALL-E3 [3].\n\n4. For the curve in Figure 8(a), loss is optimized best under the image-to-video (w/o scale&shift) setting. Doesn’t this mean that the scale&shift operation is useless? Then why do we need to add this operation? \n\n[2] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024b.\n\n[3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn.openai.com/papers/dall-e-3.pdf, 2(3):8, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)