Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
Authors: Keyu Tian
Yi Jiang
Zehuan Yuan
BINGYUE PENG
Liwei Wang
Venue: NeurIPS 2024 oral (2024)
Keywords: Language Models
Autoregressive Modeling
Scaling Laws
Generative Model
Image Generation
Image Synthesis
Primary area: generative_models
TL;DR: Our VAR (Visual AutoRegressive modeling), for the first time, makes GPT-style autoregressive models surpass Diffusion Transformers, and demonstrates Scaling Laws for image generation with solid evidence.

Abstract: We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine "next-scale prediction" or "next-resolution prediction", diverging from the standard raster-scan "next-token prediction". This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and generalize well: VAR, for the first time, makes GPT-style AR models surpass diffusion transformers in image generation. On ImageNet 256x256 benchmark, VAR significantly improve AR baseline by improving Frechet inception distance (FID) from 18.65 to 1.73, inception score (IS) from 80.4 to 350.2, with around 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot task generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.

Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
Keyu Tian
Yi J Zehuan Yuan Bingyue Peng?, Liwei Wang 13,+
'Center for Data Science, Peking University BBytddance Inc. Sttate Key of General Artificial Intelligence, School of Intelligence Science and Technology. Peking University keyutianesti pku.puu. edu. en, jiangyi.enjoyee enjoysbytedance com,, yuanzehuund@yreddacee. com, bingyue pengebytedance. com, wanglwēpku. edu.
Try and explore our online demo at: https://var.wwe vision Codes and models:
Figure 1: Generated samples from Visual AutoRegressive (VAR) transformers trained on ImageNet. We show 512x512 samples (top), 256 samples (middle), and zero-shot image editing results (bottom).
Abstract
We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning images coarse-to-fine "next-scale prediction" or "next-resolution prediction", diiergingg from the standard raster-scan "next-token prediction".. This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and can generalize well: VAR, for the first time, makes GPT--ttyle AR models surpass diffusion transformers in image generation. On ImageNet 256 x 256 benchmark, VAR significantly improve AR baseline by improving Fréchet inception distance (FID) from 18.65 1.73, inception score (IS) from 80.4 to 350.2, with 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) multiple dimensions including image quality. inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out- out-painting, and editing. These results suggest VAR has initially emulated the two important properties L.LMs: Scaling Laws and zero-shot generalization. We have released all models and codes to promote the exploration AR/VAR models for visual generation and unified learning.
corresponding authors: wanglw @ @kku.edu.cn, yuanzehuan @ bytedayce.ccom. i: project lead
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Three Differentt Autoregressive Generative Models
The
sat
him
Autoregressive Transformer [GPT, LiaMs, PaLM, etc.)
(a) AR: Text generation next-token prediction
AR Transformer (IGPT, VQGAN, Parti...)
Visual Autoregressive Transformer (Our VAR)
AR:
Image generation next-image-token prodiction
(c) VAR: Imaaee generation by next-scale (or next-resolution) prediction
Figure 2: Standard autoregressive modeling (AR) vs. our proposed visual autoregressive modeling (VAR). (a) AR applied to language: sequential text token generation from left to right, word by word; (b) AR applied to images: sequential visual token generation in a raster-scan order, from left to right, top to bottom; (e) VAR for images: multi-scale token maps autoregressively generated from coarse fine scales (lower to higher resolutions). with parallel token generation within each scale. VAR requires multi-scale VQVAE to work.
1 Introduction
The advent of GPT series [66, 67, 15, 63, 1] and more autoregressive (AR) large language models (LLMs) [22, 4, 39, 83, 84, 91, 79, 80] has heralded new epoch in the field of artificial intelligence. These models exhibit promising intelligence in generality and versatility that, despite issues like hallucinations [40], are still considered to take a solid step toward the general artificial intelligence (AGI). At the core of these models is a self-supervised learning strategy predicting the next token in sequence, simple yet profound approach. Studies into the success of these large AR models have highlighted their scalability and generalizabilty: the former, as exemplified by scaling laws [44, 36), allows us to predict large model's performance from smaller ones and thus guides better resource allocation, while the latter, as evidenced by zero-shot and few-shot learning [67, 15], underscores the unsupervised-traine -rainnedd models' adaptability to diverse, unseen tasks. These properties reveal AR models' potential in learning from vast unlabeled data, encapsulating the essence of "AGI".
In parallel, the field of computer vision has been striving to develop large autoregressive world models [59, 58, 6]. aiming to emulate their impressive scalability and generalizability. Trailblazing efforts like VQGAN and DALL-E [30, 68) along with their successors [69, 92, 51, 99] have showcased the potential of AR models in image generation. These models utilize a visual tokenizer to discretize continuous images into grids of 2D tokens, which then flattened to 1D sequence for AR learning (Fig. 2 b). mirroring the process of sequential language modeling (Fig. 2). However, the scaling laws of these models remain underexplored, and more frustratingly, their performance significantly lags behind diffusion models [64, 3, 52], as shown in Fig. 3. In contrast to the remarkable achievements of LLMss. the power of autoregressive models in computer vision appears to be somewhat locked.
Autoregressive modeling requires defining the order of data. Our work reconsiders how to "or- der" an image: Humans typically perceive or create images in hierachical manner, first cap- turing the global structure and then local details. This multi-scale, coarse-to-fie nature suggests an "order" for images. Also inspired by the widespread multi-scale designs [55, 53, 82, 45],
Frechet Inception Disuncee (FID)
0,38
1B
5B
ADM
AR(RQ)
Jow Snverssi betteeeete better Dower
MaskGIT
AR(wgEn)
Gigaran
we define autoregressive learning for images as "next-scale prediction" in Fig. (c), diverging from the conventional "next-token prediction" in Fig. (b). Our approach begins by encoding an image into multi-scale token maps. The au- toregressive process is then started from the 1x1 token map, and progressively expands in resolu- tion: at each step, the transformer predicts the next higher-resolutie all previous ones. We refer to this methodology as Visual Regessgre (VAR) modeling.
RCG
DIT
VAR (ours)
FID lower bound 1.78 (val set)
0.1
I sec
min
Inference Time (butch
Figure Scaling behavior of different model fami- lies on ImageNet 256 256 generation benchmark. The FID of the validation set serves reference lower bound (1.78). VAR with 2B parameters reaches FID of 1.73, surpassing L-DiT with 3B 7B parameters.
token map conditioned on
2

---

VAR directly leverages GPT-2-like transformer architecture [67] for visual autoregressive learning. On the ImageNet 256x256 benchmark. VAR significantly improves its AR baseline, achieving a Fréchet inception distance (FID) of 1.73 and an inception score (IS) of 350.2, with inference speed 20x faster (see Sec. for details). Notably, VAR surpasses the Diffusion Transformer (DiT) - the foundation of leading diffusion systems like Stable Diffusion 3.0 and SORA [29, 14] - in FID/IS, data efficiency, inference speed, and scalability. VAR models also exhibit scaling laws akin to those witnessed in LLMs. Lastly, we showcase VAR's s zero-shot generalization capabilities in tasks like image in in-painting, out-painting, and editing. In summary, our contributions to the community include:
1. A new visual generative framework using a multi-scale autoregressive paradigm with next-scale prediction, offering new insights in autoregressive algorithm design for computer vision.
2. An empirical validation of VAR models' Scaling Laws and zero-shot generalization potential, which initially emulates the appealing properties of large language models (LLMs).
3. A breakthrough in visual autoregressive model performance, making GPT-style autoregressive methods surpass strong diffusion models in image synthesis for the first time"
4. A comprehensive open-souree code suite, including both VQ tokenizer and autoregressive model training pipelines, to help propel the advancement of visual autoregressive learning.
2 Related Work
2.1 Properties of large autoregressive language models
Scaling laws are found and studied in autoregressive language models [44, 36], which describe power-law relationship between the scale of model (or dataset, computation, etc.) and the cross- entropy loss value on the test set. Scaling laws allow
directly predict the performance of larger model from smaller ones [1], thus guiding better resource allocation. More pleasingly, they show that the performance of L.MSs and never saturate, which is considered a key factor in the success of [15, 83, 84, 98, 91, 39J. The success brought by scaling laws has inspired the vision community to explore more similar methods for multimodality understanding and generation [54, 2, 89, 27, 96, 78, 21, 23, 42, 32, 33, 81, 88).
scale well with the growth of model, data, and computation
Zero-shot generalization. Zero-shot generalization [73] refers to the ability of a model, particularly a Large Language Model, to perform tasks that it has not been explicitly trained on. Within the realm of the computer vision, there is burgeoning interest in the zero-shot and in-context learning abilities of foundation models, CLIP [65]. SAM [49], Dinov2 [62]. Innovations like Painter [90] and LVM [6] extend visual prompters [41, 11] achieve in-context learning in vision.
2.2 Visual generation
Raster-scan autoregressive models for visual generation necessitate the encoding of 2D images into ID token sequences. Early endeavors [20, 85] have shown the ability to generate RGB (or grouped) pixels in the standard row-by-row, raster-scan manner. [70] extends [85] by using multiple independent trainable networks to do super-resolution repeatedly. VQGAN [30] advances [20, 85] by doing autoregressive learning in the latent space of VOVAE [86]. It employs GPT-2 decoder-only transformer to generate tokens in the raster-scan order, like how VIT [28] serializes 2D images into 1D patches. VQVAE-2 [69] and RQ- TQ-nssssfrrmrr [51] also follow this raster-scan manner but use extra scales or stacked codes. Parti [93], based on the architecture of ViT-NQGAN [92], scales the transformer to 20B parameters and works well in text-to-image synthesis.
Masked-predictes model. MaskGIT [17] employs a VQ autoencoder and masked prediction trans- former similar to BERT [25, 10, 35] to generate VQ tokens through a greedy algorithm. MagViT ViT [94] adapts this approach videos, and Mag ViT-2 [95] enhances [17, 94] by introducing an improved VQVAE for both images and videos. MUSE [16] further scales MaskGIT to 3B parameters.
Diffusion models' progress has centered around improved learning or sampling [77, 76, 56, 57, 7]. guidance [38, 61], latent learning [71], and architectures [37, 64, 72, 31]. DIT and U-VIT [64, 8] replaces or integrates the U-Net with transformer, and inspires recent image [19, 18] or video synthesis systems [12, 34] including Stable Diffusion 3.0 [29], SORA [14], and Vidu [9].
A related work [95] named "language model beats diffusion" belongs to BERT-style masked-predicee model.
3

---

3 Method
3.1 Preliminary: autoregressive modeling via next-token prediction
Formulation. Consider sequence of discrete tokens (21, 2............................... ,T., 2T), where 2, [V] is an integer from a vocabulary of size V. The next-token autoregressive posits the probability of observing the current token 2; depends only on its prefix (x1, 22. 1)) This unidirectional token dependency assumption allows for the factorization of the sequence x' 's likelihood:
p(z1, 22, 22...,, IT)
II pp( 22,
Training an autoregressive model pa involves optimizing (a | #2, xx-1) 1) over a dataset. This is known the "next-token prediction", and the trained pe can generate new sequences.
Tokenization.
Images are inherently 2D continuous signals. To apply autoregressive modeling to images via next-token prediction, we must: 1) tokenize image into several discrete tokens, and 2) define a 1D order of tokens for unidirectional modeling. For 1), a quantized autoencoder such as [30] is often used convert the image feature map f E RhxxxC to discrete tokens q [V]hxw;
f = {im),
q == 2(f).
where im denotes the raw image, E(.) a encoder, and 2(-) a quantizer. The quantizer typically includes learnable codebook Z E IR Vx containing V vectors. The quantization process q = Q(f)
will map each feature vector f(())) the code index ((..)) of its nearest code in the Euclidean sense:
q (1:) = arg min [|lookup(Z.)) - 1(6.)1112 [V], vE(V]
where lookup( Z, u) means taking the v-th vector in codebook Z. To train the quantized autoencoder, Z is looked up by every (b-z)) to get f, the approximation of original f. Then new image im is reconstructed using the decoder D-)) given f. and compound loss L is minimized:
f lookup( Z, 4),
im =D(f),
(4)
[ |im - imll2 + || - Í1l2 + Xppp(im) + )oCa(im),
(5)
where Lp(-) is a perceptual loss such as LPIPS [97], L- discriminative loss like StyleGAN's discriminator loss [47], and Xp, Åg are loss weights. Once the autoeneoder {E, D) fully trained, it will be used to tokenize images for subsequent training of a unidirectional autoregressive model.
The image tokens in E [V] are arranged in a 2D grid. Unlike natural language sentences with an inherent left-to-right ordering, the order of image tokens must be explicitly defined for unidirectional autoregressive learning. Previous AR methods [30, 92, 51] flatten the 2D grid of q into a ID sequence
(Z1...., x using some strategy such as row-major raster scan, spiral, or z-curve order. Once flattened, they can extract set of sequences æ from the dataset, and then train an autoregressive model to maximize the likelihood in (1) via next-token prediction.
The above approach of tokenizing and flattening enable next-token autoregressive learning on images, but introduces several issues:
Discussion on the weakness of vanilla autoregressive models.
1) Mathematical premise violation. In quantized autoencoders (VQVAEs), the encoder typically produces an image feature map f with inter-dependent feature vectors f( for all j. So x2, after quantization and flattening, the token sequence (2,, ... Ta ) retains bidirectional correlations. This contradicts the unidirectional dependency assumption of autoregressive models, which dictates that each token 3, should only depend on its prefix (æ 22, .,, 1).
2) Inability to perform some zero-shot generalization. Similar issue 1), The unidirectional nature of image autoregressive modeling restricts their generalizability in tasks requiring bidi- rectional reasoning. E.g.. it cannot predict the top part of an image given the bottom part.
3) Structural degradation. The flattening disrupts the spatial locality inherent in image feature maps. For example, the token ((f-)) and its 4 immediate neighbors (+++1.)) (t.++11) are closely correlated due to their proximity. This spatial relationship is compromised in the linear sequence x. where unidirectional constraints diminish these correlations.

---

4) Inefficiency. Generating an image token sequence 2: = (41, 22. ................................ Zaxn) with a conventional self-attention transformer incurs O(n²) autoregressive steps and Onn)) computational cost.
Issues 2) and 3) are evident (see examples above). Regarding issue 1). we present empirical evidence in Appendix C. The proof of issue 3) is detailed in Appendix D. These theoretical and practical limitations call for a rethinking of autoregressive models the context of image generation.
Stage I: Training multi-scale VQVAE on images (to provide the ground truth for training Stuge 2)
Stage 2: Training VAR transformer on tokens
(IS) misuss a start loken with condition infurmaiion)
Cine-Fitropy
Block-wise crusil mask
VA R Transformer (cousal)
S
werd embedding and interpolation
VAE ecooding Multi-scale quantiationn Embedding Decoding
Figure 4: VAR involves two separated training stages. Stage 1: a multi-scale VQ autoencoder encodes an image into K token maps R = (r1, -2,-.., TK) and is trained by compound loss (5). For details on "Multi-scale quantization" and "Embedding", check Algorithm and 2. Stage 2: VAR transformer is trained via next-scale prediction (6): it takes ([s],r), ri, r...... (K-1) input to predict (ri, r2, r3,-... .TK). The attention mask is used training to ensure each re can only attend to rgg. Standard cross-entropy loss is used.
3.2 Visual autoregressive modeling via next-scale prediction
Reformulation. We reconceptualize the autoregressive modeling on images by shifting from "next- token prediction" " to "next-scale prediction" strategy. Here, the autoregressive unit is an entire token map, rather than single token. We start by quantizing a feature map f Rxxxxx into K multi-scale token maps (r1, (2,,?,..... ..,,............................ TK), ). each at increasingly higher resolution hą x wk, culminating in тж matches the original feature map's 's resolution h x w. The autoregressive likelihood is formulated as:
K
p(r1, ... TK)
II p(rk 11,72,....,
(6)
where each autoregressive unit rx E [V] is the token map at scale l: containing ha x we tokens, and the sequence (r1, 72,72..... ......, (k-11 serves as the the "prefix" for ri. During the k-th autoregressive step, all distributions over the hę x wy tokens in r will be generated in parallel, conditioned on Tи 's prefix and associated k-th position embedding map. This "next-scale prediction" methodology is what we define as visual autoregressive modeling (VAR), depicted on the right side of Fig. 4. Note that the training of VAR, a block-wise causal attention mask used ensure that each r'k can only attend to its prefix rsk. During inference, kv-caching can be used and no mask is needed.
Discussion. VAR addresses the previously mentioned three issues as follows:
1) The mathematical premise is satisfied if we constrain each T3 to depend only on its prefix, that is, the process of getting T'k is solely related T'c.- This constraint acceptable the natural, coarse-to-fl to-fine progression characteristics like human visual perception and artistic drawing (as we discussed in Sec. 1). Further details are provided in the Tokenization below.
aligns with
2) The spatial locality is preserved (i) there is flattening operation in VAR, and (ii) tokens in each l'k are fully correlated. The multi-scale design additionally reinforces the spatial structure.
3) The complexity for generating an image with n x n latent is significantly reduced to O(n*), see Appendix for proof. This efficiency gain arises from the parallel token generation in each Ti-
Tokenization. We develope a new multi-scale quantization autoencoder to encode an image to K multi-scale discrete token maps R (r1, r2..r...... .,,K .x) the same architecture as VQGAN [30] but with modified multi-scale quantization layer. The encoding and decoding procedures with residual design on f or are detailed in algorithms I and We empirically find this residual-style design, akin to [51], can perform better than independent interpolation. Algorithm I shows that each rkk would depend only on its prefix (r1, 72, ........ ... 7k-1).
necessary for VAR learning (6). We employ
5

---

Note that a shared codebook Z is utilized across all scales, ensuring that each r4 's tokens belong to the same vocabulary [V]. To address the information loss in upscaling 2 hgg x WK, we use K extra convolution layers (thx ) k=1- No convolution is used after downsampling f to hy x ttk.
Algorithm 1: Multi-scale VQVAE Encoding
Algorithm 2: Malti-scale VQVAE Reconstruction
Inputs: raw image am;;
Inputs: multi-scale token maps
Hyperparameters: sleps K, resolutions
Hypergarameters: steps K, resolutions
(hz, wa wa);-
(h2, wy))mms
f E(im). R R:::
3 f for k 1....
for 1.... -K'
rk Q(interpolate(f hi, wx)):
Fh queue pop(T));
R queue_ queue_pusht rt):
2 lookup ra):
2t lookup( 2, ra):
inteppllaee /u, wx):
26 interpolate(sa. Aic, Wk):
du(a));
f da(=z);
im D(f):
Return: multi-scale tokens
t0 Return: recomstructed image tim;
4 Implementation details
the vanilla VQVAE architecture [30] and a multi- scale quantization scheme with K extra convolutions (0.03M extra parameters). We use a shared codebook for all scales with V = 4096. Following the baseline [30], our tokenizer is also trained on Openlmages [50] with the compound loss (5) and a spatial downsample ratio of 16x.
VAR tokenizer. As aforementioned,
VAR transformer. Our main focus is on VAR algorithm so keep simple model architecture design. We adopt the architecture of standard decoder-only transformers akin to GPT-2 and VQ- GAN [67, 30] with adaptive normalization (AdaLN), which has widespread adoption and proven effectiveness in many visual generative models [47, 48, 46, 75, 74, 43, 64, 19]. For class-conditional synthesis, we use the class embedding the start token [s] and also the condition of AdaLN. We found normalizing queries and keys to unit vectors before attention can stablize the training. We do use advanced techniques SwiGLU MLP, or RMS Norm [83, 84]. Our model shape follows a simple rule like [44] that the width w, head counts h, and drop rate dr are linearly scaled with the depth d as follows:
large language models, such as rotary position embedding (RoPE),
w = 64d,
= d,
0.1 d/24.
Consequently, the main parameter count N of a VAR transformer with depth d is given by
N (d)
d 2w d-Sw² +
d-6w²
18 dw² d3
(8)
self-attention
food-forward
adaptive layemorm
All models are trained with the similar settings: base learning rate of 10-4 per 256 batch size, an AdamW optimizer with 3, = 0.9, 32 = 0.95, decay = 0.05, batch size from 768 to 1024 and training epochs from 200 to 350 (depends model size). The evaluations in Sec. 5 suggest that such a simple model design are capable of scaling and generalizing well.
5 Empirical Results
This section first compares VAR with other image generative model families in Sec. 5.1. Evaluations on the scalability and generalizability of VAR models are presented Sec. 5.2 and Appendix B. For implementation details and ablation study, please see Appendix 4 and Appendix 6.
5.1
State-of-the-art image generation
Setup. We test VAR models with depths 16, 20, 24, and 30 on ImageNet 256x256 and 512x512 conditional generation benchmarks and compare them with the state-of-the- of-the--rt image generation model families. Among all VQVAE-basee AR or VAR models, VQGAN [30] and ours use the same architecture (CNN) and training data (OpenImages [50]) for VQVAE, while ViT-VQGAN [92] uses a ViT autoencoder, and both it and RQTransformer [51] trains the VQVAE directly on ImageNet. The results are summaried Tab. and Tab. 2.
è 'Due to resource limitation, we use single shared adaptive layernorm (AdaLN) acorss all attention blocks in 512 synthesis. In this case, the parameter count would be reduced to around 12dw² + Gw² z 49152 d

---

Table 1: Generative model family comparison on class-conditional ImageNet 256 256. "J" or "1" indicate lower higher values are better. Metrics include Fréchet inception distance (FID), inception score (IS), precision (Pre) and recall (rec). "#Step": number model runs needed generate an image. Wall-clock inference time relative to VAR is reported. Models with the suffix "-re" used rejection sampling. t: taken from MaskGIT [17].
Type
Model
FID
1st
Pret
Reet
#Para
#Step
Time
GAN
BigGAN [13]
6.95
224.5
0.89
0.38
112M
GAN
GigaGIN [43]
3.45
225.5
0.84
0.61
569M
GAN
StyleGan-XL [75]
2.30
265.1
0.78
0.53
166M
0.3 [75]
Diff.
ADM [26]
10.94
101.0
0.69
0.63
554M
250
168 [75]
Diff.
CDM [37]
4.88
158.7
8100
Diff.
LDM-4-G [71]
3.60
247.7
400M
250
Diff.
DiT-L/2 [64]
5.02
167.2
0.75
0.57
458M
250
31
Diff.
DiT-XL/2 [64]
2.27
278.2
0.83
675M
250
45
Diff.
L-DiT-3B [3]
2.10
304.4
0.82
0.60
3.0B
250
Diff.
L-DIT-7B [3]
2.28
316.2
0.83
0.58
7.0B
250
2
Mask.
MaskGIT [17]
6.18
182.1
0.80
0,51
227M
8
0.5 [17]
Mask.
RCG (cond.) [52]
3.49
215.5
502M
20
1.9 [52]
AR
VQVAE-2 2 [69]
31.11
45
0.36
0.57
13.5B
5120
AR
VQGANT [30]
18.65
80.4
0.78
0.26
227M
256
[17]
VQGAN [30]
15.78
74.3
1.4B
256
24
AR
VQGAN-re [30]
5,20
280.3
1.4B
256
24
AR
VITVQ [92]
4.17
175.1
1.7B
1024
>2
AR
VITVQ-re [92]
3.04
227.4
1.7B
1024
>2
AR
RQTran, [51]
7.55
134.0
3.8B
68
21
VAR
VAR-d16
3.30
274.4
0.84
0.51
310M
10
0.4
VAR
VAR-d20
2.57
302.6
0.83
0.56
600M
10
0.5
VAR
VAR-d24
2.09
312.9
0.82
0.59
1.0B
10
0.6
VAR
VAR-d30
1.92
323.1
0.82
0.59
2.0B
10
1
VAR
VAR-d30-Te
1.73
350.2
0.82
0.60
2.0B
10
1
(validation data)
1.78
236.9
0.75
0.67
Overall comparison. In comparison with existing generative approaches including generative adversarial networks (GAN), diffusion models (Diff.), BERT-style masked-prediction models (Mask.), and GPT-style autoregressive models (AR), visual autoregressive (VAR) establishes a new model class. As shown in Tab. 1, VAR not only achieves the best FID/IS but also demonstrates remarkable speed in image generation. VAR also maintains decent precision and recall, confirming its semantic consistency. These advantages hold true on the 512 x512 synthesis benchmark, as detailed in Tab. 2. Notably, VAR significantly advances traditional AR capabilities. To knowledge, this is the first time of autoregressive models outperforming Diffusion transformers, a milestone made possible by VAR's 's resolution of AR limitations discussed in Section 3.
Efficiency comparison. Conventional autore- gressive (AR) models [30, 69, 92, 51] suffer lot from the high computational cost, as the num- ber of image tokens is quadratic to the image resolution. A full autoregressive generation of n² tokens requires O(n²) decoding iterations and O(n²) total computations. In contrast, VAR only requires (log(n)) iterations and O(r²)) to- tal computations. The wall-clock time reported in Tab. also provides empirical evidence that VAR is around 20 times faster than VQGAN and VIT-VGGAN even with more model parame- ters, reaching the speed of efficient GAN models which only require step to generate an image.
Table 2: ImageNet 512 conditional generation. 1: quoted from MaskGIT [17]. "-s": single shared AdaLN layer used due to resource limitation.
Type
Model
FID1
ISt
Time
GAN
BigGAN [13]
8.43
177.9
Diff.
ADM [26]
23.24
101.0
Diff.
DIT-XL/2 [64]
3.04
240.8
81
Mask.
MaskGIT [17]
7.32
156.0
0.5t
AR VAR
VQGAN [30]
26.52
66.8
25
VAR-d36-s
2.63
303.2
Compared with popular diffusion transformer. The VAR model surpasses the recently popular diffusion models Diffusion Transformer (DiT), which serves as the precursor to the latest Stable- Diffusion 3 [29] and SORA [14], in multiple dimensions: 1) In image generation diversity and quality
7

---

(FID and IS), VAR with 2B parameters consistently performs better than DiT-XL/2 [64], L-DiT-3B, and L-DiT-7B [3]. VAR also maintains comparable precision and recall. 2) For inference speed. the DiT-X1/22 requires 45x the wall-clock time compared to VAR, while 3B and 7B models [3] would cost much more. 3) VAR considered more data-efficient, as requires only 350 training epochs compared to DiT-XI/22's 1400. 4) For scalability, Fig. 3 and Tab. 1 show that DiT only obtains marginal or even negative gains beyond 675M parameters. In contrast, the FID and IS of VAR are consistently improved, aligning with the scaling law study in Sec. 5.2. These results establish VAR as potentially a more efficient and scalable model for image generation than models like DiT.
(e)
(9)
(6)
(4)
2.16
1-m (2.5 -I
181
(2.52.5 A0-52E
Tokan emor me (Dast scali, %0
95.5
Errm (5- (5-16"00ma))
Token SEOUT rato (allso 50
97.6
Firm (6-10' Mpnal
Cormis - 0.9933
Conea -0 39255
Corola
Conola 0.9589
Ted locs saall
1.66-
Tox loss (all scals)
1.44
94.9
96.5
1.27
0.97
0.91
91.3
94.3
0.74
0.72
89.6
93.2
10
10"
10"
10"
10"
Model Parmmeters (Billion)
Model Parametoos (Billion)
Model Porameters (Billioe)
Medel Parameten (Billion)
Figure 5: Scaling laws with VAR transformer size N, with power-law fits (dashed) and equations (in legend). Small, near-zero exponents a suggest smooth decline in both test loss I. and token error rate Err when scaling up VAR transformer. Axes are all on a logarithmic scale. The Pearson correlation coefficients near -0.998 signify a strong linear relationship between log(N) vs. (L)(L) or (N) vs. log (Err).
5.2 Power-law scaling laws
Background. Prior research [44,6,,3]] 39, 1] have established that scaling up autoregressive (AR) large language models (LLMs) leads predictable decrease in test loss L. This trend correlates with parameter counts N, training tokens T, and optimal training compute Cmin, following a power-law:
L = (B X)",
(9)
where X can be any of N, T, or Cmin- The exponent a reflects the smoothness of power-law, and L denotes the reducible loss normalized by irreducible loss Lee [36] A logarithmic transformation to L and X will reveal a linear relation between Log(L) and log(
Log(L) = a log(X) + a log B.
(10)
An appealing phenomenon is that both [44] and [36] never observed deviation from these lincar relationships at the higher end of X, although flattening is inevitable as the loss approaches zero.
These observed scaling laws [44, 36, 39, 1] not only validate the scalability of LLMS but also serve as a predictive tool for AR modeling, which facilitates the estimation of performance for larger AR models based on their smaller counterparts, thereby saving resource usage by large model performance forecasting. Given these appealing properties of scaling laws brought by LLMs, their replication in computer vision is therefore of significant interest.
Setup of scaling VAR models. Following the protocols from [44, 36, 39, 11. examine whether our VAR model complies with similar scaling laws. We trained models across 12 different sizes, from 18M to 2B parameters, on the ImageNet training set [24] containing 1.28M images (or 870B image tokens under our VQVAF) per epoch. For models of different sizes, training spanned 200 to 350 epochs, with maximum number of tokens reaching 305 billion. Below focus on the scaling laws with model parameters N and optimal training compute Cmin given sufficient token count T.
Scaling laws with model parameters N. We first investigate the test loss trend as the VAR model size increases. The number of parameters N (d) 73728 d3 for VAR transformer with depth d is specified in (8). We varied d from 6 to 30, yielding 12 models with 18.5M to 2.0B parameters. We assessed the final test cross-entropy loss L and token prediction error rates Err on the ImageNet validation set of 50,000 images [24]. We computed L and Err for both the last seale (at the last next-scale autoregressive step), as well as the global average. Results are plotted in Fig. 5, where we
8

---

3.07
3.35
2.14
Text (last scate)
2.55
Text (al)
1.49
1.69
1.04
1.12
1- (2.2 10 21
[=11.5 10 Cyed
0.72
Comrelaton 0.988
0.74
Correlation
10"
10'
10
10
10"
10'
10"
10"
99.88
Token erove rate: scale, 96)
99.9
Token (alk,
98.1
97.2
96.5
94.6
94.9
Crus
92.1
fronter Car
Errm (8.1 GGea
Errm (4.4 114 C.n) w00
93.2
09997
89.6
Cornistion
10
10
10
10
10'
10'
10'
10'
Training Compute (PFlops)
Triinin Compute (PFlops)
Figure 6: Scaling laws with optimal training compute Cmin- Line color denotes different model sizes. Red dashed lines are power-law fits with equations in legend. Axes are logarithmic scale. Pearson coefficients 0.99 indicate strong linear relationships between log (Cmin) vs. log( L)) or (og(min)) log( Err)).
observed clear power-law scaling trend for L as a function of N, as consistent with [44, 36, 39, 1]. The power-law scaling laws can be expressed
Last (2.0 - and Lavg (2.5 NN
(11)
Although the scaling laws are mainly studied the test loss, also empirically observed similar power-law trends for the token error rate Err:
Erria (4.9 10 and Erravg (6.5 10 -0.010
(12)
These results verify the strong scalability of VAR, by which scaling up VAR transformers can continuously improve the model's test performance.
Scaling laws with optimal training compute Cmin- We then examine the scaling behavior of VAR transformers when increasing training compute C. For each of the 12 models, we traced the test loss and token error rate Err as function of C during training quoted in PFlops (1015 floating-point operations per second). The results plotted in Fig. 6. Here, draw the Pareto frontier of L and Err to highlight the optimal training compute Cmin required to reach a certain value of loss or error.
The fitted power-law scaling laws for L and Err as function of Cmin are:
Last = (2.2 10 CC 0..1
(13)
Lavg (1.5 10 "Cmin) -0.16
(14)
Erras (8.1 10 10021012000 =C - -0.0067
(15)
Errasg (4.4 10 10-2Cmmn) -0.011
(16)
These relations (14, 16) hold across 6 orders of magnitude in Cmins and our findings consistent with those in [44, 36]: when trained with sufficient data, larger VAR transformers are more compute- efficient because they can reach the same level of performance with less computation.
6 Ablation Study
In this study, we aim to verify the effectiveness and efficiency of our proposed VAR framework. Results reported in Tab. 3.
Effectiveness and efficiency of VAR. Starting from the vanilla AR transformer baseline implemented by [17], replace its methodology with our VAR and keep other settings unchanged to get row 2.

---

Table 3: Ablation study of VAR. The first two rows compare GPT-2-style transformers trained under AR or VAR algorithm without any bells and whistles. Subsequent lines show the influence of VAR enhancements. "AdaLN": adaptive layernorm. "CFG": classifier-free guidance. "Attn. Norm": normalizing q and & to unit vectors before attention. "Cost": inference cost relative to the baseline. "A": FID reduction to the baseline.
Description
Para.
Model
AdaLN
Top-k
CFG
Cost
FID
1
AR [30]
227M
AR
X
X
X
1
18.65
0.00
2
AR VAR
207M
VAR-d16
X
0.013
5.22
-13.43
3
A
310M
VAR-d16
V
0.016
4.95
-13.70
4
+Top-k
310M
VAR-d16
/
900
X
0.016
4.64
-14.01
5
+CFG
310M
VAR-d16
V
900
1.5
0.022
3.60
15.05
5
Attm. Norm.
310M
VAR-d16
900
1.5
0.022
3.30
15533
6
+Scale up
2.0B
VAR-d30
900
1.5
0.052
1.73
16.85
VAR achieves way more better FID (18.65 vs. 5.22) with only 0.013x inference wall-clock cost than the AR model, which demonstrates a leap in visual AR model's performance and efficiency.
Component-wise ablation. We further test some key components in VAR. By replacing the standard Layer Normalization (LN) with Adaptive Layer Normalization (AdaLN), VAR starts yielding better FID than baseline. By using the top-k sampling similar to the baseline, VAR's FID is further improved. By using the classifier- free guidance (CFG) with ratio 1.5 and normalizing q and k to unit vectors before attention,
reach the FID of 3.30, which is 15.35 lower to the baseline, and its inference speed is still 45 times faster. We finally scale up VAR size to 2.0B and achieve an FID of 1.73. This 16.85 better than the baseline FID.
7 Limitations and Future Work
In this work, we mainly focus on the design of learning paradigm and keep the VQVAE architecture and training unchanged from the baseline [30] to better justify VAR framework's effectiveness. We expect advancing VQVAE tokenizer [99, 60, 95] as another promising way to enhance autoregressive generative models, which orthogonal to our work. We believe iterating VAR by advanced tokenizer or sampling techniques in these latest work can further improve VAR's performance or speed.
Text- pext-prompt generation is an ongoing direction of our research. Given that model is funda- mentally similar to modern LLMs, it can easily be integrated with them to perform text-to-image generation through either encoder-decoder in-context manner.
Video generation is not implemented this work, but it can be naturally extended. By considering multi-scale video features as 3D pyramids, we can formulate similar "3D next-scale prediction" to generate videos via VAR. Compared to diffusion-basee generators like SORA [14], our method has inherent advantages in temporal consistency integration with LLMs, thus can potentially handle longer temporal dependencies. This makes VAR competitive in the video generation field, because traditional AR models can be too inefficient for video generation due to their extremely high computational complexity and slow inference speed: it becoming prohibitively expensive to generate high- resolutioo videos with traditional AR models, while VAR is capable to solve this. We therefore foresee a promising future for exploiting VAR models in the realm of video generation.
Conclusion
We introduced a new visual generative framework named Visual AutoRegressive modeling (VAR) that 1) theoretically addresses some issues inherent in standard image autoregressive (AR) models, and 2) makes manguage-modelbaseee AR models first surpass strong diffusion models in terms of image quality, diversity, data efficiency, and inference speed. Upon scaling VAR billion parameters, we observed clear power-law relationship between test performance and model parameters or training compute, with Pearson coefficients nearing -0.998, indicating robust framework for performance prediction. These scaling laws and the possibility for zero-shot task generalization, as hallmarks of LLMs, have now been initially verified in our VAR transformer models. We hope our findings and open sources can facilitate a more seamless integration of the substantial successes from the natural language processing domain into computer vision, ultimately contributing to the advancement of powerful multi-moda intelligence.
10

---

Acknowledgements
Liwei Wang was supported by National Science Foundation of China (NSFC92470123, NSFC62276005) and National Science and Technology Major Project (20222D0114902),
A
Visualization of scaling effect
To better understand how VAR models are learning when scaled up, we compare some generated 256 x 256 samples from VAR models of 4 different sizes (depth 6, 16, 26, 30) and different training stages (20%, 60%, 100% total training tokens) in Fig. 7. To keep the content consistent, a same random seed and teacher-forced initial tokens are used. The observed improvements in visual fidelity and soundness are consistent with the scaling laws, as larger transformers are thought able to learn more complex and fine-gra mne-grained image distributions.
B Zero-shot task generalization
Image in- in-painting and out-painting. VAR-d30 is tested. For in- and out-painting, we teacher-force ground truth tokens outside the mask and let the model only generate tokens within the mask. No class label information is injected into the model. The results are visualized in Fig. 8. Without modifications to the network architecture tuning parameters, VAR has achieved decent results on these downstream tasks, substantiating the generalization ability of VAR.
Class-conditional image editing. Following MaskGIT [17] also tested VAR the class- conditional image editing task. Similar to the case of in in-painting, the model is forced to generate tokens only in the bounding box conditional some class label. Fig. shows the model can produce plausible content that fuses well into the surrounding contexts, again verifying the generality of VAR.
C Token dependency in VQVAE
To examine the token dependency in VQVAE [30], we check the attention scores in the self-attention layer before the vector quantization module. We randomly sample 4 256x256 images from the ImageNet validation
for this analysis. Note the self-attention layer in [30] only has head for each image we just plot one attention map. The heat map in Fig. 9 shows the attention scores of each token all other tokens, which indicate a strong, bidirectional dependency among tokens. This not surprising since the VQVAE model, trained to reconstruct images, leverages self-attention layers without any attention mask. Some work [87] has used causal attention self-attention layers of video VAE, but we did not find any image VAE work uses causal self-attention.
D Time complexity of AR and VAR generation
We prove the time complexity of AR and VAR generation.
Lemma D.1. For a standard self-attention transformer, the time complexity of AR generation is Onn)), where h = n and h w are the height and width of the VQ code map, respectively.
Proof. The total number of tokens is h x w n². For the i-th (1 < S n²) autoregressive iteration, the attention scores between each token and all other tokens need to be computed, which requires (12) time. So the total time complexity would
1 (n² (n² + 1)(2n² + 1),
Σ
a
(17)
Which is equivalent to O(n)) basic computation.
For VAR, it needs us to define the resolution sequense (h), w1, h2, .2,... lu, wx) for autoregres- sive generation, where h;, wi are the height and width of the VQ code map at the i-th autoregressive step, and lgg = h, wk w reaches the final resolution. Suppose n& hy wa for all 1 < l: K
11

---

Scaling up training compute C
Scaling up transformer parameters N
WMMA
Figure 7: Scaling model size N and training compute C' improves visual fidelity and soundness. Zoom in for better view. Samples are drawn from VAR models 4 different sizes and different training stages. class labels (from left to right, top to bottom) are: flamingo 130, aretic wolf 270, macaw 88, Siamese cat 284, oscilloscope 688, husky 250, mollymawk 146, volcano 980, and catamaran 484.
and w. for simplicity. We set the resolutions n4 (k-1) where a > is constant such that a! (K-1)
Lemma D.2. For standard self-attention transformer and given hyperparaneter 1, the time complexity of VAR generation O(n²), where w n. and h, w are the height and width of the last (largest) VQ code map, respectively.
12

---

original
generated
In-painting
Out-painting
Class-cond
Editing
Figure 8: Zero-shot evaluation in downstream tasks containing in in-painting, out-painting. and class- conditional editing. results show that VAR can generalize novel downstream tasks without special design and finetuning. Zoom better view.
Figure 9: Token dependency plotted. The normalized heat map of attention scores in the last self-attentice layer of VQGAN encoder is visualized. random 256x 256 images from ImageNet validation set are used.
Proof. Consider the k-th (1 < < K) autoregressive generation. The total number of tokens of current all token maps (r1, (2,,2,... Tk) is:
a2k
2k-1
(18)
0
1
So the time complexity of the k-th autoregressive generation would be:
a 2k
(19)
a -
13

---

By summing all autoregressive generations, we have:
Tog (x)+1
2
(20)
a
(a
1) log n + (a -2aºn² 2n²(n² -1) + 2a² - 1) log a
(21)
(a² ()² + logg a
Onn²).
(22)
This completes the proof.
14

---

BigGAN (FID-6.95)
(FID332
MaskGIT (FID-6.18)
VAR, ours (F99-1.92)
Figure 10: Model comparison on ImageNet 256 256 benchmark. More generated 512x512 samples by VAR can be found in the submitted Supplementary Material zip file.
15

---

KRilit
Figure 11: Some generated 256x 256 samples by VAR trained on ImageNet. More generated 512x512 samples by VAR can be found in the submitted Supplementary Material zip file.
16

---

References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, al. Gpt-4 technical report. arXiv preprint axXi:2303.08772 2023. 2,3, 3,8,9
[2] J.-B, Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican,
M. Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in neural
information processing systems, 35:23716 23736, 2022. 3
[3] Alpha-VILLM,
https: : // tttbtttttubbb.ooo whtpe//AApppaaLLLLLLLMMMAAA Accessory/tree/ http2//3333330001000000aooaaaa12
DiT- ImageNett 2024. 7,8 [4] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin. A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen,
et al. Palm 2 technical report. arXiv preprint arXiv:2305. 2023. 2
[5] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, ct al. Qwen technical
report. arXiv preprint arXiv:2309. 16609, 2023.
[6] Y. Bai, X. Geng, K. Mangalam, A. Bar, A. Yuille, T. Darrell, J. Malik, and A. A. Efros. Sequential
modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312. 00785, 2023. 2, [7] F. Bao, C. L.,, J. Zhu, and B. Zhang. Analytic-dmm: an analytic estimate the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.061 1.6503, 2022.
[8] F. Bao, S. Nie, Xue, Y. Cao, C. Li, Su, and Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22669-22679, 2023.
[9] F. Bao, C. Xiang, G. Yue, G. He, H. Zhu, K. Zheng, M. Zhao, S. Liu, Wang. and J. Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint
arXiv:2405. 044333, 2024. 4
[10] H. Bao, L. Dong. S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint 1xXiv:2166.0855:0
2021.
[11] A. Bar, Y. Gandelsman, T. Darrell, A. Globerson, and A. Efros. Visual prompting via image inpainting.
Advances in Neural Information Processing Systems, 35:25005-25017, 2022. 3
[12] O. Tal. H. Chefer, O. Tov, C. Hermman, Paiss, S. Zada, A. Ephrat, Hur, Li, Michaeli, et al. Lumiere: A space- space-time diffusion model for video generation. arXiv preprint 3XXi:2440112245 2024, 4 [13] A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1006, 2018. 7
[14] T. Brooks, B. Peebles, C. Holmes, W. DePue, Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng. Wang, and A. Ramesh. Video generation models as world simulators. OpenAl, 2024. 3,4, 8, 10 [15] T. Brown, B. Mann. N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2,3
[16] H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint
arXiv:2301.00700 2023. 3
[17] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11315-
11325, 2022. 3, 7, 10, 11
[18] J. Chen, C. Gc, F. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang. P. Luo, H. Lu, and Z. Li. Pixart-\/sigmmn Weak-to- strong training of diffusion transformer for 4k text-to-image generation. arxiv preprint arXiv:2403.0465 04662,, 2024. 4
[19] Chen. J. Yu, C. Ge, L. Yao, E. Xie. Y. Wu, Wang. J. Kwok, Luo, H. Lu, et al. Pixart: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arx.v:2310.004266, 2023.
4,6 6
[20] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, Luan, and I. Sutskever. Generative pretraining from
pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020. 3
[21] Z. Chen, J. Wu, W. Wang. W. Su. G. Chen, S. Xing, Z. Muyan. Q. Zhang, Zhu. L. Lu. et al. Interuvl: Scaling up vision foundation models and aligning for generic visnal-linguistic tasks. arXiv preprint arXiv:23/2. 14238, 2023. 3
[22] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton,
S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning
Research, 24(240):1-113, 2023. 2
[23] X. Dai, J. Hou, C.-Y. Ma, S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint
ar09v:2.309. 15807, 2023. 3
[24] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee. 2009. 8, 9, 22, 23
17

---

[25] J. Devlin. M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[26] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information
processing systems, 34:8780-8794 2021.7 7
[27] R. Dong, C. Han, Y. Peng. Z. Qi. Z. Ge, J. Yang, L.. Zhao, J. Sun, H. Zhou, H. Wei, et al. Dreamllm:
Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309. 1099//14 /1499, 2023, 3 [28] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition
scale. arXiv preprint arXiv:2010. 11999, 2020.
[29] P. Esser. S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer. F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y. Marek, and R. Rombach. Scaling rectified
flow transformers for high-resolution image synthesis, 2024. 3, 4,
[30] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883
2021. 2, 3, 4, 6, 7, 10, 11
[31] S. Gao, P. Zhou, M.-M. Cheng, and S. Yan. Mdtv2: Masked diffusion transformer is strong image synthesizer. arXiv preprint arXiv:2303.
2023, 4
[32] Y. Ge, S. Zhao, Z. Zeng, Y. Ge. C. Li, X. Wang. and Y. Shan. Making llama see and draw with seed tokenizer. arXiv preprini arXiv:2310.01215. 2023. 3
[33] Y. Ge. S. Zhao, Zhu, Y. Ge, K. Yi, L. Song. C. X. Ding, and Y. Shan. Seed-x: Multimodal models with unified multi-gramularitt comprehension and generation. arXiv preprint arXiv:2404.14300 2024. [34] A. Gupta, L. Yu, K. Sohn, X. Gu, M. Hahn. L. Fei-Fei, I. Essa, L. Jiang, and J. Lezama. Photorealistic
video generation with diffusion models, arXiv preprint arXiv:2312.06 066622, 2023, 4
[35] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision Icarners. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 16000-16009, 2022. 3
[36] T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 2020. 2,3, 8,
[37] Ho, C. Saharia, W. Chan, D. Fleet, M. Norouzi, and Salimans. Cascaded diffiusion models for high
fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-2288 2022. 4, 7
[38] J. Ho and T. Salimans. Classifier- free diffusion guidance. arXiv preprint arwii:2207.1255555
2022. 4
[39] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022, 2, 9,
[40] L. Huang. W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q Chen, W. Peng, X. Feng, B. Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint 231 1. 0.03222, 2023. 2
[41] M. Jia, L. Tang. B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning.
In European Conference on Computer Vision, pages 709-727. Springer, 2022. 3
[42] Y. Jin, K. Xu, L. Chen, C. Liao, J. Tan, B. Chen, C. Lei, A. Liu, C. Song, X. Lei, al. Unified language- vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309, 04669, 2023.
[43] M. Kang, J.-Y. Zhu, R. Zhang, J. Park, E. Shechtman, S. Paris, and T. Park. Scaling up gans for text-to- image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124-10134, 2023. 6,7
[44] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08.330 2020. 2, 3, 6, 8,
[45] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability,
and variation. arXiv preprint arXiv:1710. 10196, 2017. 2
[46] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila, Alias-fre generative
adversarial networks. Advances in Neural Information Processing Systems, 34:852-888 863, 2021. [47] T. Karras, S. Laine, and T. Aila. A style based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 4410.
2019. 4, 6
[48] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8110-8119 19, 2020.
[49] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 2023.
[50] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Tont-Tuset, S. Kamali, S. Popov. M. Malloci,
A. Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7)11956 1981, 2020. 6,7 7
18

---

[51] D. Lee. C. Kim, S. Kim. M. Cho. and W.S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11532, 2022. 2,3, 4, 6.7
[52] T. Li, D. Katabi, and K. He. Self-conditoned image generation via generating representations. arXiv preprint arXi::2112.0370) 2023. 2.7
[53] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongic. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2117-2125, 2017. 2
[54] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3
[55] D. G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2. pages 1150-1157. Ieee, 1999.
[56] C. Lu. Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787,
2022. 4
[57] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01099 2022. 4
[58] J. Lu, C. Clark, S. Lee, Z. Zhang. S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi. Unified-io
2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint
arXiv:2312.17172. 2023. 2
[59| J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. Unified-io: A unified model for vision, language,
and multi-modal tasks. arXiv preprint arXiv:2206.089/1 2022. 2
[60] F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen. Finite scalar quantization: Vq-vae made simple.
arXiv preprint arXiv:2309. 15500, 2023. 10
[61] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide:
Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint
arXiv:2112. 10741, 2021, 4
[62] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szaframice, V. Khalidov, P. Fernandez. D. Haziza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint
arXiv:2304.077 07193, 2023,
[63] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural
Information Processing Systems, 35:27730-27744, 2022. 2
[64] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 4195-4205, 2023, 2, 4, 6,8
[65] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin.
J. Clark, et al. Learning transferable visual models from natural language supervision. In International
conference on machine learning, pages 8748 8763. PMLR, 2021. 3
[66] A. Radford, K. Narasimhan,
Salimans, I. Sutskever, et al. Improving language understanding by
generative pre- treitrrining. article, 2018. 2
[67] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models unsupervised
multitask learners. OpenAl blog, 18))9, 2019, 2, 6
[68] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR,
2021. 2
[69] A. Razavi, A. Van den Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019, 3,7
[70] S. Reed, A. Oord, N. Kalchbrenner, S. G. Colmenarejo. Z. Wang. Chen, D. Belov, and N. Freitas. Parallel multiscale autoregressive density estimation. In International conference on machine learning.
pages 1912-2921 PMLR, 2017. 3
[71] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022. 4,7
[72] C. Saharia, W. Chan, S. Saxena, I. Li, J. Whang, E. L.. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language
understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022. 4
[73] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L.. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint (:Xiv:2110.08200 2021. 3
[74] A. Sauer, T. Karras, S. Laine, A. Geiger, and T. Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to- text-to-imant -mmee synthesis. arXiv preprint arXiv:2301.099 095115 2023. 6
[75] A. Sauer, K. Schwarz. and A. Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 101 2022. 6,
19

---

[76] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint urXiv:2010.02502.
2020. 4
[77] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 4
[78] Q. Sun. Q. Yu, Y. Cui, F, Zhang, X. Zhang. Y. Wang. H. Gao, Liu, T. Huang, and X. Wang. Generative pretraining in multimodality. arXiv preprint 0eXXiv23377.02222 2023. 3
[79] Y. Sun, S. Wang. S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao, Y. Lu, al. Ernic 3.0:
Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint urXiv:2107.02133 2021..
[80] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu. R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, 2023. [81] C. Tian, X. Zhu, Y. Xiong, W. Wang. Z. Chen. W. Wang, Y. Chen, L. Lu, T. Lu, J. Zhou, et al. Mm- interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv
et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11
preprint arXiv:2401. 10208. 2024. 3
[82] K. Tian, Y. Jiang, Q. Diao, C. Lin, L. Wang, and Z. Yuan. Designing bert for convolutional networks:
Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.0335 2023. 2
[83] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro,
F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23X2:2302.13971
2023. 3.6 6
[84] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava,
S. Bhosale. et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.08
2023, 2, 3, 6
[85] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation
with pixelcnn decoders. Advances in neural information processing systems, 29, 2016. 3
[86] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information
processing systems, 30, 2017. 3
[87] R. Villegas, M. Babacizadeh, P.-.. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro. J. Kunze, and Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022. 11
[88] H. Wang, H. Tang, L. Jiang, S. Shi, M. F. Naeem, H. Li, B. Schiele, and L. Wang. Git: Towards generalist vision transformer through universal language interface. arXiv preprint urXiv:2403.00 202249009994, 2024. 3 [89] W. Wang. Z. Chen, X. Chen, J. Wu, X. Zhu. G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024.
[90] X. Wang. W. Wang, Y. Cao. C. Shen, and T. Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0830-6839, 2023. 3
[91] B. Workshop. T. L. Scao, A. Fan. C. Akiki, E. Pavlick. S. Ili,, D. Hesslow, R. Castagné, A. S. Luccioni.
F. Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint
arXiv:2211 1,00100,, 2022. 3,3
[92] J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu. Vector-quantized
image modeling with improved vggan. arXiv preprint arXi::21 2021. 2, 2,4,7
[93] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang. B. K. Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint XrXiv22206. 10789, 2(3):5, 2022. 3
194] L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang. Y. Hao, I. Essa, et al. Magvit: Masked generative vidco transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-10469, 2023. 3
[95] L. Yu, Lezama, N. B. Gundavarapu, L.. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta, X. Gu, A. G. Hauptmann, ct al. Language model beats diffusion-tokenizze is key to visual generation. arXiv preprint vrXiv:2310.05737 2023. 3, 10
[96] L. Yu, B. Shi, R. Pasunuru, B. Muller, O. Golovneva, T. Wang, A. Babu, B. Tang, B. Karrer, S. Sheynin, modtl models: Pretraining and instruction tuning. arXiv preprint
ct al. Scaling autoregressive
arXiv:2309.00 0...293..00091,..... 2(3), 2023. 3
1971 R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unrcasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern
recognition, pages 586-595, 2018. 4
[98] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, Dewan, M. Diab, X. Li. X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 2xXi::2005000666 2022. 3 [99] C. Zheng, T.-L. Vuong, J. Cai, and D. Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Syxtems, 35:23412-23425, 2022. 2, 10
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our main contributions are also detailed in Sec. 1. Also see Sec. 5 and Appendix D for more theoretical and experimental evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Sec. 7 for limitations. We also reported a lot about computa- tional efficiency, such as Tab. and Appendix D.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[Yes]
Justification: We detail the assumption and proof of theoretical result on time complexity in Appendix D.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We use publicly-acceee accessable dataset ImageNet [24]. We upload the codes and instructions to recover the results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We use publicly-accesseeee dataset ImageNet [24]. We upload the codes and instructions recover the results. Once the blind review period is finished, we 'Il open-source all codes, instructions, and model checkpoints.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //npsscc... wwt.cc/cccpppcpciieceee. gww.//////////eg// Sociee////odddummaaaaa
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please Sec. 5 and Appendix 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the resource limitation, we do not report error bars. Please note that in Sec. 5 we spent numerous resources (we trained 12 different models) for our scaling law study, which makes it prohibitively to run each experiments for multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the training PFlops in Fig. 6 and speed in Tab. I and Tab. 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / /etn:///nnspseee cc cctppblic////ccccccccccccccllllt httpee/Ethiiiiiiiiieeeeee
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work focuses on academic, publicly-availabl benchmark ImageNet. This work is not related to any private or personal data, and there's 's no explicit negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "vKby2F6ba1", "reviewer": "Reviewer_Hisd", "summary": "This paper introduces next-scale prediction autoregressive models that satisfy mathematical premises (unidirectional dependency of autoregressive model) and preserve the 2D spatial locality. The core method is to develop multiscale VQ-VAE. The proposed method is more efficient than the traditional autoregressive model, requiring only $O(n^4)$ compared to $O(n^6)$ of the raster autoregressive model. Furthermore, this method is proven to follow scaling laws of LLM which guarantee better performance when scaling up the training process.", "questions": "My main concerns are in the method section. I hope the author could provide more method details. See the weakness above.", "limitations": "The limitation discussions are sound and clear to me.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This idea is natural and novel. It successfully solves mathematical premises violation of previous rastering scan autoregressive model. \n2. The power-law scaling law is interesting and encourages follow-up work to scale up models for better performance.\n3. The method’s performance is competitive to the diffusion model and other generative models.\n4. Most of the paper claims seem valid to me.", "weaknesses": "1. The second stage of training VAR transformers is too short and is hard for me to fully understand how it works. I wonder about the details of how to generate $h_w \\times w_h$ tokens in $r_k$ parallel using k-th position embedding map. Is the embedding 1D or 2D embedding ?. How to make sure all tokens in $r_k$ are correlated to each other ?\n2. The highest resolution of the scale $r_K$ is $16 \\times 16$ and there are 10 scales (1,2,3,4,5,6,8,10,13,16). I wonder if there is any motivation to choose these scales.\n3. I think the paper should include the sampling algorithm of autoregressive model with hyper-parameter details such as temperature, top-k, top-p and CFG sampling.\n4. The zero-shot generalisation algorithm should be included in the paper for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Hisd,\n\nMany thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n\n---------------------------\n\n> [W1] The second stage of training VAR transformers is too short and is hard for me to fully understand how it works...\n\n[A2] Thanks for pointing this and we'll detail more on the 2nd stage of VAR method in our manuscript. To generate $h_k\\times w_k$ tokens $r_k$ in $k$-th autoregressive step (like to generate $r_2$ in Figure 4), the input is the previous token map $r_1$. It'll be reshaped to 2D ($1\\times 1$ here), embedded to a 2D feature map, upsampled to $r_2$'s shape $2 \\times 2$, projected to the VAR transformer's hidden dimension, and added by a 2D positional embedding to get $e_1$. So the actual input to the VAR transformer is $e_1$ in Figure 4, and these steps are the details of \"word embedding and up-interpolation\" noted in Figure 4. The detailed implementation can also be found in the code \"codes/models/var.py autoregressive_infer_cfg\" attached in supplementary material.\n\nTo make sure all tokens in $r_k$ are correlated to each other, we don't apply attention mask within $r_k$ to keep a bidirectional attention on them. In other words, $r_k$ can attend to all tokens of $r_{\\le k}$\n\n&nbsp;\n\n> [W2] there are 10 scales (1,2,3,4,5,6,8,10,13,16). I wonder if there is any motivation to choose these scales.\n\n[A2] Yes the chose of scales is a key design of VAR. We choose an *exponential function* $h_k=w_k=\\lfloor a\\cdot b^k\\rfloor$ to get these scales because:\n1) As discussed in appendix F, we can reach a total complexity of $\\mathcal{O}(n^4)$ via an exponential schedule.\n2) We want to increase the number of steps to reach $16\\times 16$ for image quality. An exponential function grows slowly in the early stages and quickly in the later ones. So it allows us to increase steps (mainly in the early stages) without a significant increase in total sequence length.\n\nIn practice, we use $a=1.36$ and $b=1.28$ to get that (1,2,3,4,5,6,8,10,13,16).\n\n&nbsp;\n\n> [W3] I think the paper should include the sampling algorithm of autoregressive model with hyper-parameter details such as temperature, top-k, top-p and CFG sampling.\n\n[A3] We use temperature of $1.0$, top-k of $k=900$, top-p of $p=0.96$, and CFG of $1.25$ (VAR-$d16$) or $1.5$ (the others). We'll add these to our manuscript.\n\n&nbsp;\n\n> [W4] The zero-shot generalisation algorithm should be included in the paper for clarity.\n\n[A4] Thanks for this valuable advice. We'll add a pseudo code for detailing the algorithm of zero-shot generalisation in our manuscript. \nThe algorithms of inpainting, outpainting, and class-condition editing are basically the same.\n\nSpecifically, we would mask out the according region at each scale of (1,2,3,4,5,6,8,10,13,16) given the task, i.e., masking the inner area for inpainting, the outer area for outpainting, and the area we want to edit for editing. During VAR generation, ground-truth non-masked tokens are maintained (like teacher forcing) and we only collect VAR's predictions on those masked regions.\n\n-----------------------------------\n\n**Thank you again for helping us improve the paper and hope our response can resolve your concerns!\nPlease let us know if you have any further questions.\nWe will be actively available until the end of rebuttal period.\nIf you feel your concerns are addressed, please consider reevaluating our work.\nLooking forward to hearing from you :-) !**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F4kWXjoZHy", "reviewer": "Reviewer_oCJa", "summary": "This work proposes a novel approach to image generation using an autoregressive decoder-only transformer model. Rather than decoding in a raster-scan their approach (VAR) decodes scales/resolutions conditioned on previously generated scales, reminiscent of traditional scale pyramids in computer vision. VAR demonstrates competitive performance on ImageNet in terms of generation quality, diversity and inference speed. It also demonstrates scaling laws up to 2.0B parameters.", "questions": "1. I'd like to see a throughput comparison in img/s as the batch size is increased. I'd also like to see some DiT results where a more advanced sampler such as DPM-solver or SA-solver is used.\n2. See above.\n3. Concretely, how is the editing performed? Are certain tokens teacher-forced? If so starting from which scale?", "limitations": "See above", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- I am *very* impressed by the proposed method. It is simple, intuitive and novel. It is not difficult to see why it works well.\n- The empirical results are strong, and since the approach is based on a decoder-only transformer (a tried and tested architecture for autoregressive generation) I would expect it to scale to foundation-model T2I systems easily.\n- The paper is well written and easy to read, with the method/motivation/story communicated clearly to the reader.", "weaknesses": "As the reviewing burden has been heavy for this conference (6 papers) please understand that I can only dedicate so much time to this paper. Thus, I may have made mistakes in my understanding of the paper, and I welcome the authors to correct me if this is the case. \n\n1. The reported inference efficiency is a bit disingenuous. The comparison with DiT uses 250 steps, which is much more than what SotA samplers require. Moreover, diffusion models can be distilled into 1-4 step models that are even faster.  Compared to raster scan autoregressive models, the improvement in latency seems to be due to better use of parallel resources. However, for larger batches/measuring throughput this advantage may fade. \n2. There are some missing details and insight, especially with regards to the multi-scale VQVAE. There are also details that are present in the code that really should be in the paper, such as the number of tokens per image/scale. What is its reconstruction performance of the VQVAE (compared to e.g. the SDXL VAE)? How does one choose the number of scales?  How many codes end up being used over the different scales and do different scales capture similar (spatial) information in the latent space vs the image space? It would also be great to see an ablation like Table 3 for the VQVAE. Also, the code provided doesn't give details to reproduce the VQVAE, only the transformer.\n3. The use of \"zero-shot\" to refer to the model's editing ability is different in nature to zero-shot generalisation in LLMs, so I find the link made in the paper to be a little disingenuous. Moreover, the editing performance doesn't seem to be very strong, with inpainting generation spilling outside of the box. The paper also doesn't give details on how the editing is performed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oCJa,\n\nMany thanks to your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows.\n\n---------------------------\n\n> [W1 Q1, about the efficiency evaluation] I'd like to see a throughput comparison in img/s as the batch size is increased. I'd also like to see some DiT results where a more advanced sampler such as DPM-solver or SA-solver is used.\n\n[A1] Thank you for your suggestions to make our efficiency evaluation more complete and thoughtful!\n\nFirst, we add the throughput comparison with batched inference to our manuscript.\nAll models are tested on a single A100 GPU. VQGAN's results are quoted from [1].\nFrom the results we can see that VAR also **gets a big speedup from batch processing**, similarly to vanilla autoregressive models. We attribute this to the short sequence length of the first few VAR steps (like generating $1\\times 1$ and $2\\times 2$ tokens).\nAfter batching, VAR reaches a throughput of 0.04 $\\text{img/s}$, which is still **4 times faster than batched VQGAN**, even if VAR has a larger model size.\n\n|Model|#Para|Batch size|Throughput$\\downarrow$|FID$\\downarrow$|\n|:-|:-|:-|:-|:-|\n| VQGAN-re     | 1.4B | 1   | 5.76 $\\text{img/s}$  | 5.20 |\n| VQGAN-re     | 1.4B | 100 | 0.16 $\\text{img/s}$  | 5.20 |\n| VAR-$d30$-re | 2.0B | 1   | 0.24 $\\text{img/s}$  | 1.73 |\n| VAR-$d30$-re | 2.0B | 100 | **0.04 $\\text{img/s}$**  | **1.73** |\n-----------------------\nSecond, we add the results of DiT and U-ViT (another transformer-based diffusion model) with fewer diffusion steps ($<50$) to our manuscript.\nThe results are quoted from [2, 3]. From the table one can see that:\n1) Reducing the number of diffusion steps via ODE sampler (like DDIM, DPM-Solver) will result in a large FID rise, especially when reduced to near 10 steps.\n2) Both using 10 steps, VAR-1B is still **faster and better than DiT-675M**. This is also because VAR has a shorter sequence length in the early steps, e.g., the first three VAR steps only generate $1\\times 1 + 2\\times 2 + 3\\times 3 = 14$ tokens.\n3) The Diffusion community has been building up for a long time for efficiency boosting. In contrast, VAR, as a newly proposed method, promises to see more ways to accelerate or distill VAR in the future. We'd like to add this to our Future Work section.\n\n|Model|#Para|#Step|Time$\\downarrow$|FID$\\downarrow$|\n|:-|:-|:-|:-|:-|\n| VAR-$d24$              | 1.0B | 10 | **0.6** | **2.09** |\n| DiT-XL/2 (original)    | 675M |250 | 45      | 2.27 |\n| DiT-XL/2 + DDIM        | 675M |250 | 45      | 2.14 |\n| DiT-XL/2 + DDIM        | 675M | 20 | 2.9     | 4.68 |\n| DiT-XL/2 + DDIM        | 675M | 10 | 1.8     | 12.38|\n| U-ViT-H/2 + DPM-solver | 501M | 20 | 15.6    | 2.53 |\n| U-ViT-H/2 + DPM-solver | 501M | 10 | 7.8     | 3.18 |\n\n> [W2 Q2, details on multi-scale VQVAE]\n\n[A2] We'd like to respond to your queries mentioned in Weakness 2 one by one, and add all the details below to Appendix section.\nPlease see the overall Author Rebuttal part of \"VQVAE concerns\" for specific responses.\n\n&nbsp;\n\n\n> [W3 Q3, The use of \"zero-shot\" to refer to the model's editing ability is different in nature to zero-shot generalisation in LLMs; doesn't give details on how the editing is performed]\n\n[A3] Thanks for your insightful comments. In the field of language processing, it has been verified that every task can be formulated to an autoregressive generation task. This allows a pretrained LLM can generalize to many tasks different to its pretraining task without any finetuning.\nIn our model, we also use the term \"zero-shot\" to emphasize that our model can also do tasks that are different to our pretraining task. We have updated some according descriptions in our paper to make it more clear and accurate.\n\nWe added enough details on how the inpainting/outpainting/editing is performed to our manuscript. You can also find them in the overall Author Rebuttal part.\n\n\n-----------------------------------\n\nMany thanks to Reviewer oCJa for their professional, detailed, and valuable reviews!\nWe have done our best to address each of your concerns and hope our response can resolve them.\nPlease let us know if you have any other questions. We will actively join the discussion until the end of the rebuttal period.\nWe are looking forward to hearing from you :-) !", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hoc8nXAjhw", "reviewer": "Reviewer_yx9H", "summary": "The paper introduces Visual AutoRegressive modeling (VAR), which uses a coarse-to-fine approach for image generation. VAR drastically improves performance, reducing FID from 18.65 to 1.73 and increasing IS from 80.4 to 350.2, with 20x faster inference. It outperforms diffusion transformers in quality, speed, efficiency, and scalability. VAR models show scaling laws like large language models and demonstrate zero-shot generalization in image editing tasks.", "questions": "**About VQVAE**\n\n- What is the rFID of your VQVAE? Can you provide a table comparing your VQVAE and VQVAEs of other works (e.g., VQGAN, MaskGIT)\n\n- What is the pre-training cost of your VQVAE?\n\n- If residual quantization is not used, and instead a multi-scale token map is constructed directly by some ways like: 1) Independently downsample the VAE encoder features to multiple scales, then quantize each scale directly, or 2) Construct multiple resolution ImageNet datasets (e.g., ImageNet 16x16, 32x32…, 256x256) and independently apply vector quantization to each dataset, thereby obtaining low-resolution to high-resolution token maps.Then, can the proposed visual autoregressive modeling still be performed? In other words, is residual quantization an *indispensable* part of VAR modeling? (Ignoring the efficiency or complexity of these alternative tokenization methods)\n- What is the impact on performance if residual quantization is not used?\n\n**Question on Figure 7**\n\nThe apples-to-apples qualitative comparison, like in Figure 7, is common in diffusion-based models because the initial noise is of the same resolution as the final output, allowing significant control over the final image when coupled with deterministic sampling. However, in VAR, the counterpart to the \"initial noise\" is only the \"teacher-forced initial tokens,\" which, if I understand correctly, are of 1x1 size. This suggests only a very loose control over the final image. Given this, why doesn't this result in a situation similar to the \"butterfly effect,\" where identical initial states and random seeds, due to different model configurations, lead to significantly different final outputs after multiple generation iterations?\n\n**Question on zero-shot generation**\n\n- Is only the last resolution of the token map masked and then teacher-forced to generate the masked regions, or is each token map masked?\n- The model hasn't explicitly learned inter-token dependence at the spatial level during training. Could you provide an explanation of why the model can perform zero-shot editing/inpainting, which requires the model to condition on tokens at some spatial positions to generate tokens at other positions?", "limitations": "This paper validates the effectiveness of VAR only in class-conditional generation scenarios. Applying VAR to text-to-image generation is a worthwhile area for future exploration.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- **Solid Motivation**: The scale in vision signals is a natural choice for autoregressive generation. The exploration of autoregression in visual generation is indeed a worthy topic.\n- **Novel Method**: This work is the first to explore a visual generative framework using a multi-scale autoregressive paradigm with next-scale prediction.\n- **Strong Performance**: The paper demonstrates significant advancements in visual autoregressive model performance, with GPT-style autoregressive methods surpassing strong diffusion models in image synthesis for the first time.\n- **Promising Scaling Law**: The paper presents a promising scaling law for the proposed visual autoregressive modeling paradigm.", "weaknesses": "- **Lack of Ablation Study on VQVAE**: There is no ablation study on the newly proposed VQ-VAE model. In Table 3, the performance differences between the first two rows cannot be solely attributed to the model change from AR to VAR, as the VQ-VAE model has also been modified.\n- **Resolution Flexibility**: The resolutions for VAR generation appear to be pre-defined and bound to the VQ-VAE model during its pre-training. Adjusting the number of resolutions or maximum resolution without re-training the VQ-VAE model seems non-trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer yx9H,\n\n---------------------------\n\n> [W1 Q1, about VQVAE] \n\n[A1] We appreciate your detailed comments and will address them one by one.\n\n1. VQVAE rFID: please see the overall Author Rebuttal part of \"VQVAE concerns\"\n2. pre-training cost: please see the overall Author Rebuttal part of \"VQVAE concerns\"\n3. why we didn't choose other multi-scale quantization ways: if the residual quantization is removed and **independently quantization** is used (e.t., independently downsampling VAE features, or independently encoding image in 16x16, 32x32, ..., 256x256),\nthe mathematical premise of unidirectional dependency would be broken. The details can be found in line140 of our paper.\nTo ensure that property, we have to use the **residual quantization** way. We'll add these explanations to our manuscript.\n4. Lack of Ablation Study on VQVAE: please see the overall Author Rebuttal part of \"VQVAE concerns\".\n\n&nbsp;\n\n> [W2] Resolution Flexibility\n\n[A2] Thanks for this detailed comment. We've observed that the multi-scale VQVAE pretrained only on 256px images, can easily generalize to higher resolutions like 512 and 1024. We have added visualizations of this to a new Appendix part.\n\n> [Q2] Question on Figure 7\n\n[A3] Thank you for this professional question.\nIn practice, we: 1) use a fixed sampling seed for every step, and 2) also fixed the first token which intuitively determined the global structure of the image.\nWe found both of them are crucial for the generation consistency.\nWe'll add these details to our manuscript.\n\n&nbsp;\n\n> [Q3] Question on zero-shot generation\n\n[A4] For the details on our zero-shot generalisation algorithm, please see the overall Author Rebuttal part of \"Zero-shot generalisation algorithm\".\nFor your second question, we actually allows full, bidirectional attention among tokens in the same scale $r_k$.\nSo the model can learn inter-token dependence.\n\n&nbsp;\n\n-----------------------------------\n\n**Lastly, thank you so much for helping us improve the paper and appreciate your open discussions! Please let us know if you have any further questions. We are actively available until the end of this rebuttal period. Looking forward to hearing back from you!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GKzcnWCV21", "reviewer": "Reviewer_dFai", "summary": "The paper introduces VAR, a novel autoregressive generative model for images that treats each scale in a multi-resolution feature pyramid as a token. Unlike traditional models that predict the next token from a rasterized grid, VAR predicts the next scale in a multi-resolution grid. This approach demonstrates greater scalability than next-token prediction and extends the well-known scaling laws from language modeling to image generation. Extensive experiments show that VAR outperforms both diffusion and AR baselines while offering improved efficiency in both training and inference.", "questions": "1. Does VAR perform next-scale prediction completely in the latent space? For example, if the image resolution is 256 and VQ-VAE has a latent size of 32, does VAR operate on resolutions up to 32, or does it extend up to the image resolution?\n\n2. Is there a loss term missing in equation (5)? VQ-GAN models typically include a commitment loss and a codebook loss, but it appears VAR's quantizer only uses one of these losses. Could you provide more details on this part?\n\n3. If I understand correctly, when predicting the probabilities for each scale with a transformer, the transformer now needs to estimate a much higher-dimensional distribution (a distribution over the entire grid instead of just one point in the grid). Can you provide some intuition on why the transformer part can handle this prediction task effectively? Is it due to the strong conditioning from the lower scales?\n\n4. Are the features computed by the VAR quantizer also useful for image recognition tasks, or do they perform best in image generation?", "limitations": "The authors have addressed limitations and social impact of the work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper addresses the significant question of bridging the performance gap between autoregressive language models and autoregressive image generation. This makes the topic highly relevant for the community and potentially impactful.\n\n- The method is well-motivated and utilizes well-known building blocks from LLMs. Hence, VAR is an important step toward showing that with a proper scheme, widely used LLM architectures can perform competitively in the Image domain. \n\n- The experimental section is comprehensive, demonstrating VAR's performance and efficiency in image generation on ImageNet 256 and 512. Additionally, the authors provide in-depth discussion of scaling laws for VAR.\n\n- Ablation studies in Appendix D clearly illustrate the contribution of different aspects of VAR to the final model.", "weaknesses": "- While the writing of the paper is clear for the most part, the method section could benefit from better presentation. The authors provide some details on VAR tokenization and training, but I found section 3, especially section 3.2, slightly confusing. I suggest that the authors add more details and clarification on VAR's training process, the residual tokenization, and the workings of the transformer part. Currently, these details are somewhat obscured in Algorithm 1 and 2, and Figure 4.\n\n- Some claims in the paper are slightly exaggerated. For example, in the abstract, the authors mention that VAR brings the FID from 18.65 to 1.73. While this is true, the FID of 18.65 belongs to a relatively weak baseline for AR models. It would be better to rewrite such claims in relation to more realistic baselines, such as RQ-VAE models, which are closer to VAR's methodology\n\n- The baselines used for the diffusion part are also relatively weak. For instance, MDTv2 [1] is a transformer-based diffusion model that achieves an FID of 1.58 on ImageNet 256. Therefore, it would be more appropriate to state that VAR performs \"competitively\" with diffusion models rather than significantly outperforms them.\n\n[1] Gao S, Zhou P, Cheng MM, Yan S. MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer. arXiv preprint arXiv:2303.14389. 2023 Mar 25.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dFai,\n\nMany thanks to your professional, detailed, and valuable reviews. We're going to response to your concerns one by one.\n\n-----------------\n\n> [W1] I suggest that the authors add more details and clarification on VAR's training process, the residual tokenization, and the workings of the transformer part. Currently, these details are somewhat obscured in Algorithm 1 and 2, and Figure 4.\n\n[A1] Thank you for your thorough review and suggestion. We've rearranged our Method part, adding more details on the two different training stages, and make them more closed to each other.\n\n&nbsp;\n\n> [W2] It would be better to rewrite such claims in relation to more realistic baselines, such as RQ-VAE models, which are closer to VAR's methodology.\n\n[A2] We agree that comparing RQ-VAE with VAR can also demonstrates the effectiveness and efficiency of VAR and without any potential overclaims. Also, this would not devalue VAR's novelty and technical contributions. We have updated these according descriptions.\n\n&nbsp;\n\n> [W3] Therefore, it would be more appropriate to state that VAR performs \"competitively\" with diffusion models rather than significantly outperforms them.\n\n[A3] Thanks for this professional advice. We'd first explain why models like MDTv2 was not inclued in  table 1: Since our table 1 mainly focuses on **Generative model family** comparison on class-conditional ImageNet 256x256, we did not include some latest powerful models in it. As also mentioned in line 308, our main focus was on VAR algorithm and we used a plain GPT-2 transformer without SwiGLU, RMSNorm, or Rotary position embedding. So there is still large room to boost VAR. Comparing VAR with other long-optimized diffusion model can be a bit unfair. We have added some explanations and used \"performs competitively\" to describe our VAR.\n\nNonetheless, we added an extra comparison which focuses on comparing VAR with latest, state-of-the-art model to the Appendix. We also updated the Future Work section to see if we can integrate more advanced technique like in MDTv2 or LLaMa to further upgrade VAR.\n\n&nbsp;\n\n-------------------------------\n\n**Thanks for your comments and suggestions, we will add these experiments to our revision. Feel free to let us know if you have any further questions or concerns :-).**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel visual autoregressive model (VAR) that predicts the next image scale rather than rasterizing the image pixel-wise. The VAR model shows strong results in image generation, outperforming existing autoregressive models in efficiency and achieving competitive results with diffusion-based methods.

Strengths:
* Clear Motivation: The motivation behind multiscale image generation is well-articulated, making it a natural and novel extension of existing autoregressive techniques.
* Novelty & Methodology: The paper presents a fresh approach to autoregressive image generation by adopting a coarse-to-fine multiscale framework, which offers new insights into how autoregressive models can be applied in vision tasks.
* Strong Empirical Results: The model demonstrates strong performance on ImageNet in terms of generation quality, diversity, and inference speed, with impressive FID and IS scores. The scaling law properties are promising, showing room for future advancements.
* Efficiency: VAR achieves a significant improvement in inference speed, leveraging its multiscale approach to reduce computational costs while maintaining high-quality image generation.

Weaknesses:
* Missing Ablation Studies: Several reviewers expressed concerns over the lack of ablation studies on the VQ-VAE component, specifically regarding residual quantization and its impact on VAR's performance.
* Clarification of Details: The method section, particularly concerning VAR’s training and the residual tokenization process, could be better clarified. Some key details about the VQVAE's role and scale selection could benefit from deeper discussion in the main paper rather than being relegated to appendices.
* Comparison to Stronger Baselines: Although VAR performs well against the baselines provided, the baselines for diffusion models could have been stronger. Comparisons to models like MDTv2 were suggested to be more appropriate.
* The authors have not verified the scalability of the approach beyond ImageNet, where the performance of AR vs VAR can shift as we scale up the data and model.

The authors provided a comprehensive rebuttal, addressing key concerns raised by the reviewers. They acknowledged the lack of ablation on residual quantization and provided an analysis on why this component is crucial to VAR’s success. Additionally, the authors clarified technical details related to scale selection, VAR’s inference steps, and the interaction between scales in image editing tasks. The revised manuscript will include more details on these points, as well as additional comparisons with state-of-the-art diffusion models. Despite minor weaknesses, the contributions and technical soundness of VAR make it a valuable addition to the field of autoregressive models for image generation. Based on above, I recommend accepting this paper.
Decision: Accept (oral)