Autoregressive Image Generation without Vector Quantization
Authors: Tianhong Li
Yonglong Tian
He Li
Mingyang Deng
Kaiming He
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Image Generation
Autoregressive Models
Diffusion Models
Primary area: generative_models

Abstract: Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens. We observe that while a discrete-valued space can facilitate representing a categorical distribution, it is not a necessity for autoregressive modeling. In this work, we propose to model the per-token probability distribution using a diffusion procedure, which allows us to apply autoregressive models in a continuous-valued space. Rather than using categorical cross-entropy loss, we define a Diffusion Loss function to model the per-token probability. This approach eliminates the need for discrete-valued tokenizers. We evaluate its effectiveness across a wide range of cases, including standard autoregressive models and generalized masked autoregressive (MAR) variants. By removing vector quantization, our image generator achieves strong results while enjoying the speed advantage of sequence modeling. We hope this work will motivate the use of autoregressive generation in other continuous-valued domains and applications. Code is available at [https://github.com/LTH14/mar](https://github.com/LTH14/mar).

Autoregressive Image Generation without Vector Quantization
Tianhong Li
Yonglong Tian
He Li
Mingyang Deng'
Kaiming He
MIIT CSAIL
Google DeepMind
Tsinghua University
Abstract
Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens. We observe that while discrete-valued space can facilitate representing a categorical distribution, it is not necessity for autoregressive modeling. In this work, we propose model the per-token probability distribution using a diffusion procedure, which allows us to apply autoregressive models in continuous- valued space. Rather than using cat- egorical cross-entropy loss, we define a Diffusion Loss function to model the per- token probability. This approach eliminates the need for discrete-valuse valued tokeniz- ers. We evaluate its effectiveness across a wide range cases, including standard autoregressive models and generalized masked autoregressive (MAR) variants. By removing vector quantization, our image generator achieves strong results while enjoying the speed advantage of sequence modeling. We hope this work will moti- vate the use of autoregressive generation in other continuous-valuee domains and applications. Code is available https: //g/ tiu........................ com/LTHA Homm/LTH14/mmam
Introduction
Autoregressive models are currently the de facto solution to generative models in natural language processing [38, 39, 3]. These models predict the next word or token in a sequence based on the previous words as input. Given the discrete nature of languages, the inputs and outputs of these models are in a categorical, discrete-valued space. This prevailing approach has led widespread belief that autoregressive models are inherently linked to discrete representations.
As result, research on generalizing autoregressive models continuous-valued domains-most notably, image generation-has intensely focused on discretizing the data [6, 13, 40J. A commonly adopted strategy is train discrete-valued tokenizer on images, which involves finite vocabulary obtained by vector quantization (VQ) [51, 41]. Autoregressive models are then operated on the discrete-valeee valued token space, analogous their language counterparts.
In this work, we aim to address the following question: Is it necessary for autoregressive models be coupled with vector-quantized representations? We note that the autoregressive nature, iee, "predicting next tokens based on previous ones", is independent of whether the values are discrete continuous. What is needed is to model the per-token probability distribution, which can be measured by a loss function and used to draw samples from. Discrete-valued representations can be conveniently modeled by categorical distribution, but it is not conceptually necessary. If al- ternative models for per-token probability distributions are presented, autoregressive models can be approached without vector quantization.
With this observation, we propose to model the per-token probability distribution by a diffusion procedure operating continuous-valued domains. Our methodology leverages the principles of diffusion models [45, 24, 33, 10] for representing arbitrary probability distributions. Specifically, method autoregressively predicts vector for each token, which serves conditioning for a demoising network (e.g., a small MLP). The denoising diffusion procedure enables us to represent an underlying distribution p(x|z) for the output (Figure I). This small denoising network is
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure 1: Diffusion Loss. Given a continuous-valued to- ken be predicted, the autoregressive model produces a vector which serves as the condition of a denoising dif- fusion network (a small MLP), This offers a way to model the probability distribution p(æ[2z) of this token. This net- work is trained jointly with the autoregressive model by backpropagation. At inference time, with predicted running the reverse diffusion procedure can sample a to- ken following the distribution: z p(x(z). This method eliminates the need for discrete-valued tokenizers.
condition
noisy at
MLP
diffusion loss p(z|z)
trained jointly with the autoregressive model, with continuous- valued tokens as the input and target. Conceptually, this small prediction head, applied to each token, behaves like loss function for measuring the quality of z. We refer to this loss function as Diffission Loss.
Vector-quantized tokenizers difficult to train and are sensitive to gradient approximation strategies [51, 41, 40, 27]. Their recon- struction quality often falls short compared to continuous-vallue counterparts [42]. Our approach allows autoregressive models to enjoy the benefits of higher-quality, non-quantized tokenizers.
Our approach eliminates the need for discrete-valuse tokenizers.
To broaden the scope, we further unify standard autoregressive (AR) models [13] and masked gener- ative models [4, 29] into generalized autoregressive framework (Figure 3). Conceptually, masked
generative models predict multiple output tokens simultaneously in randomized order, while still maintaining the autoregressive nature of "predicting next tokens based on known ones". This leads to masked autoregressive (MAR) model that can be seamlessly used with Diffusion Loss.
We demonstrate by experiments the effectiveness of Diffusion Loss across a wide variety of cases, including AR and MAR models. It eliminates the need for vector- setor-quantized tokenizers and consis- tently improves generation quality. Our loss function can be flexibly applied with different types of tokenizers. Further, our method enjoys the advantage of the fast speed of sequence models. Our MAR model with Diffusion Loss can generate strong FID of < 2.0 on ImageNet 256x Our best model can approach 1 1.55 FID.
rate of 0.3 second per image while achieving
The effectiveness of our method reveals largely uncharted realm of image generation: modeling the interdependence of tokens by autoregression, jointly with the per-token distribution by diffusion. This is in contrast with typical latent diffusion models [42, 37] in which the diffusion process models the joint distribution of all tokens. Given the effectiveness, speed, and flexibility of our method, we hope that the Diffusion Loss will advance autoregressive image generation and be generalized to other domains in future research.
Related Work
Sequence Models for Image Generation. Pioneering efforts autoregressive image models [17, 50, 49, 36, 7, 6] operate on sequences pixels. Autoregression can be performed by RNNs [50], CNNs [49, 7], and, most lately and popularly, Transformers [36, 6]. Motivated by language models, another series of works [51, 41, 13, 40] model images as discrete-valuee tokens. Autoregressive [13, 40] and masked generative models [4, 29] operate the discrete-valuce token space. But discrete tokenizers are difficult to train, which has recently drawn special focus [27, 54, 32).
Related to our work, the recent work on GIVT [48] also focuses on continuous- sequence models. GIVT and our work both reveal the significance and potential of this direction. In GIVT, the token distribution is represented by Gaussian mixture models. It uses a pre-defined number of mixtures, which can limit the types of distributions can represent. In contrast, our method leverages the effectiveness of the diffusion process for modeling arbitrary distributions.
tokens in
Diffusion for Representation Learning. The denoising diffusion process has been explored criterion for visual self self-supervised learning. For example, DiffMAE [53] replaces the L2 loss in the original MAE [21] with a denoising diffusion decoder; DARL [30] trains autoregressive models with denoising diffusion patch decoder. These efforts have been focused on representation learning, rather than image generation. In their scenarios, generating diverse images not a goal; these methods have not presented the capability of generating new images from scratch.

---

Diffusion for Policy Learning. Our work is conceptually related to Diffusion Policy [8] in robotics. In those scenarios, the distribution of taking an action is formulated denoising process on the robot observations, which can be pixels or latents [8, 34]. In image generation, we can think of generating a token as an "action" to take. Despite this conceptual connection, the diversity of the generated samples in robotics is less of core consideration than it is for image generation.
3 Method
In nutshell, our image generation approach is sequence model operated tokenized latent space [6, 13, 40]. But unlike previous methods that are based on vector-quantized tokenizers (e.g., variants of VQ-VAE (51, 13]), aim to continuous-valued tokenizers (e.g., [42]). We propose Diffusion Loss that makes sequence models compatible with continuous-valued tokens.
3.1 Rethinking Discrete- Valued Tokens
To begin with, we revisit the roles of discrete-valued tokens in autoregressive generation models. Denote as x the ground-truie -ruth token to be predicted the next position. With a discrete tokenizer, can be represented as an integer: 0 S < K, with a vocabulary size K. The autoregressive model produces continuous-valuce D-dim vector 2 E TR,,, which then projected by a K K-way classifier matrix W E KR Conceptually, this formulation models categorical probability distribution in the form of p==|2) softmax (1ttmaxxx)::
In the context of generative modeling, this probability distribution must exhibit two essential prop- erties. (i) A loss function that can measure the difference between the estimated and true distribu- tions. In the case of categorical distribution, this can be simply done by the cross-entropy loss. (ii) A sampler that can draw samples from the distribution x ~ p(x|2) at inference time. the case of cat- egorical distribution, this is often implemented as drawing sample from p(z|2) = softmax W 2/T), in which is a temperature that controls the diversity of the samples. Sampling from categorical distribution can be approached by the Gumbel-max method [18] or inverse transform sampling.
This analysis suggests that discrete-valuee tokens stead, is the requirement of modeling a distribution that essential. A discrete-valued token space implies categorical distribution, whose loss function and sampler are simple define. What we actually need are a loss function and its corresponding sampler for distribution modeling.
necessary for autoregressive models. In-
3.2 Diffusion Loss
Denoising diffusion models [24] offer an effective framework to model arbitrary distributions. But unlike common usages of diffusion models for representing the joint distribution of all pixels or all tokens, in our case, the diffusion model is for representing the distribution for each token.
Consider continuous-valued vector Rd which denotes the ground-truth token be predicted at the next position. The autoregressive model produces a vector E RD at this position. Our goal to model a probability distribution of x conditioned z, that is, p(x]2). The loss function and sampler can be defined following the diffusion models [24, 33, 10], described next.
Loss function. Following [24, 33, 10], the loss function of an underlying probability distribution p(z|2) can be formulated as a denoising criterion:
((2,z) E.,: [||e co(xelt, 2)1
Here, s E Rd is a noise vector sampled from I). The noise-corrupted vector Fe is I' va,z + /I- ā,s, where a, defines noise schedule [24, 33]. t is a time step of the noise schedule. The noise estimator Ea. parameterized by is a small MLP network (see Sec. 4). The notation e0(x,]t, z) means that this network takes 2, as the input, and is conditional on both and As per [46, 47], Eqn. (1) conceptually behaves like a form of score matching: it is related to a loss function concerning the score function of p(x122). that is, V log,, p(z[2), 2))2). Diffusion Loss is parameterized loss function, in the same vein as the adversarial loss [15] or perceptual loss [56].
It is worth noticing that the conditioning vector is produced by the autoregressive network: z = f(),, as we will discuss later. The gradient of = = f(-) is propagated from the loss function in Eqn. (1). Conceptually, Eqn. (1) defines loss function for training the network f-)).
3

---

We note that the expectation Eg,[--] in Eqn. (1) is over t. for any given 2. As our denoising network small, we can sample & multiple times for any given This helps improve the utilization of the loss function, without recomputing We sample by 4 times during training for each image.
Sampler. At inference time, it is required to draw samples from the distribution p(a] Sampling
is done via reverse diffusion procedure [24]: Zz-1 Vas [z
vI-a; eo (zy]t,, 2)) + or Here
1-ai
Vas
5 is sampled from the Gaussian distribution (0, I) and de is the noise level at time step t. Starting with Jт ~ (0, I), this procedure produces a sample xp such that xg ~ p(æ[2) [24].
When using categorical distributions (Sec. 3.1), autoregressive models can enjoy the benefit of hav- ing temperature for controlling sample diversity. In fact, existing literature, in both languages and images, has shown that temperature plays a critical role in autoregressive generation. It is desired for the diffusion sampler to offer temperature counterpart. We adopt the temperature sampling pre- sented in [10]. Conceptually, with temperature т, one may want to sample from the (renormalized) probability of p(x)2)) whose score function is V log, p(z]2)) In practice, [10] suggests to either divide Eo by r, or scale the noise by r. We adopt the latter option: we scale oz in the sampler by т. Intuitively, I controls the sample diversity by adjusting the noise variance.
3.3 Diffusion Loss for Autoregressive Models
Next, we describe the autoregressive model with Diffusion Loss for image generation. Given a quence of tokens fx', xx,, ...,, x²) where the superscript 1 < S n specifies an order, autoregressive models [17, 50, 49, 36, 7, 6] formulate the generation problem as "next token prediction":
p(x) x""
II p(x x x",...,,**
(2)
A network is used to represent the conditional probability p(a" xx ,..., "..,,"=1),, ). In our case, x² can be continuous-valued. We can rewrite this formulation in two parts. We first produce a conditioning vector 2' by network (e,g., Transformer [52]) operating
previous tokens: z' = f(x,,... ,.... "..).
Then, we model the probability of the next token by p(z"l22) Diffusion Loss in Equ. (1) can be applied on p(z") (2²). The gradient is backpropagated to 2¹ for updating the parameters of f()).
3.4 Unifying Autoregressive and Masked Generative Models
We show that masked generative models, e.g., MaskGIT [4] and MAGE [29], can be generalized under the broad concept of autoregression, i.e., next token prediction.
Bidirectional attention can perform autoregression. The concept of autoregression is orthogonal
to network architectures: autoregression can be done by RNNs [50], CNNs [49, 71. and Transformers [38, 36, 6]. When using Transformers, although autoregressive models are popularly implemented by causal attention, we show that they can also be done by bidirectional attention. See Figure 2. Note that the goal of autoregression is to predict the next token given the previous tokens; it does not constrain how the previous tokens communicate with the next token.
We can adopt the bidirectional attention implementation as done in Masked Autoencoder (MAE) [21]. See Figure 2(b). Specifically, we first apply an MAE-style encoder' on the known tokens (with positional embedding [52]) Then we concatenate the encoded sequence with mask tokens (with positional embedding added again), and map this sequence with an MAE-style decoder. The positional embedding the mask tokens can let the decoder know which positions to be predicted. Unlike causal attention, here the loss is computed only on the unknown tokens [21].
With the MAF-style trick, we allow all known tokens to see each other, and also allow all unknown tokens to see all known tokens. This full attention introduces better communication across tokens than causal attention. At inference time, we can generate tokens (one or more per step) using this bidirectional formulation, which a form of autoregression. As a compromise, we cannot use the key-value (kv) cache [44] of causal attention to speed up inference. But as we can generate multiple tokens together, we can reduce generation steps to speed up inference. Full attention across tokens can significantly improve the quality and offer better speed/accuracy trade-off.
Herr the terminology of encoder/decoder is in the sense of general Autoencoder, following MAE [21]. It is not related to whether the computation is casual/bidirectionl in Transformers [52].

---

Figure 2: Bidirectional attention can do autoregression. In contrast to conventional wisdom, the broad concept of "autoreeression" (next token prediction) can be done by either causal or bidirectional attention. (a) Causal atten- tion restricts each token to attend only to current/previ- ous tokens. With input shifted by one start token [s], it is valid to compute loss all tokens at training time. (b) Bidirectional attention allows each token to see all to- kens in the sequence. Following MAE [21], mask tokens [m] are applied in a middle layer, with positional embed- ding added. This setup only computes loss unknown tokens, but it allows for full attention capabilities across the sequence, enabling better communication across tokens. This setup can generate tokens one by one at inference time, which is form of autoregression. It also allows to pre- dict multiple tokens simultaneously.
token prediction
(a) causal
(b) bidirectional
Figure Generalized Autoregressive Models. (a) A standard, raster-order autoregressive model predicts one next token based on the previous to- kens. (b) A random-order autoregressive model predicts the next token given random order. behaves like randomly masking out tokens and then predicting one. (c) Masked Autoregressive (MAR) model predicts multiple tokens simultane- ously given a random order, which conceptually analogous to masked generative models [4, 29]. In all cases, the prediction one step can be done by causal or bidirectional attention (Figure 2).
(a) AR, raster order
(b) AR, random order
(c) Masked AR
known//reddccted
prodict at this stop
unknown
Autoregressive models in random orders. To connect to masked generative models [4, 29], we consider autoregressive variant in random orders. The model is given randomly permuted sequence. This random permutation is different for each sample. See Figure 3(b). In this case, the position of the next token to predicted needs be accessible the model. We adopt strategy similar to MAE [21]: we add positional embedding (that corresponds to the unshuffled positions) the decoder layers, which can tell what positions predict. This strategy is applicable for both causal and bidirectional versions.
As shown in Figure 3 (b)(c), random-order autoregression behaves like special form of masked generation, in which one token is generated at a time. We elaborate on this as follows.
Masked autoregressive models. In masked generative modeling [4, 29]. the models predict ran- dom subset of tokens based on known/predicted tokens. This can be formulated as permuting the to- ken sequence by random order, and then predicting multiple tokens based previous tokens. See Figure 3(c). Conceptually, this is an autoregressive procedure, which can be written as estimating the x² ".....""""" 1111), where multiple tokens {x², +++ x!) are to be predicted (i j). We can write this autoregressive model as:
conditional distribution: p(fx", f+1 .... x') ++tt...,']
p( ..... x".,,"))) p(x x")
Ix X1 ......."' x
(3)
Here, X {x², x ... x²) is set of tokens to be predicted at the k-th step, with U this sense, this is essentially "next set-of-tokens prediction", and thus is also a general form of autoregression. We refer to this variant Masked Autoregressive (MAR) models. MAR is a aandom-orde autoregressive model that can predict multiple tokens simultaneously.
( "..,,.),, )").
MAR is conceptually related to MAGE [29]. However, MAR samples tokens by temperature applied on the probability distribution of each token (which is the standard practice in generative language models like GPT). In contrast, MAGE (following MaskGIT [4]) applies a temperature for sampling the locations of the tokens to be predicted: this not fully randomized order, which creates a gap between training-time and inference-time behavior.
5

---

Implementation
This section describes our implementation. We note that the concepts introduced this paper are general and not limited to specific implementations. More detailed specifics are in Appendix B.
4.1 Diffusion Loss
Our diffusion process follows [33]. Our noise schedule has a cosine shape, with 1000 steps at training time; inference time, resampled with fewer steps (by default, 100) [33]. Our denoising network predicts the noise vector e [24]. The loss can optionally include the variational lower bound term Lybb [33]. Diffusion Loss naturally supports classifier-fre guidance (CFG) [23] (detailed in Appendix B).
Diffusion Process.
small MLP consisting of few residual blocks [20] for denoising. Each block sequentially applies a LayerNorm (LN) [1], linear layer, SiLU [12], and another linear layer, merging with a residual connection. By default, The denoising MLP is conditioned on vector 2 produced by the AR/MAR model (see Figure I). The vector z is added the time embedding of the noise schedule time-step which serves as the condition of the MLP in the LN layers via AdaLN [37].
Denoising MLP. We
use blocks and width of 1024 channels.
4.2 Autoregressive and Masked Autoregressive Image Generation
Tokenizer. We use the publicly available tokenizers provided by LDM [42]. Our experiments will involve their VQ-16 16 and KL-16 versions [42]. VQ-16 is VQ-GAN [13], i.e.. VQ-VAE [51] with GAN loss [15] and perceptual loss [56]: KL-16 its counterpart regularized by Kullback-Leille (KL) divergence, without vector quantization. 16 denotes the tokenizer strides.
Transformer. Our architecture follows the Transformer [52] implementation in VIT [11]. Given sequence of tokens from tokenizer, we add positional embedding [52] and append the class tokens [cls]; then we process the sequence by Transformer. By default, our Transformer has 32 blocks and width of 1024, which we refer to the Large size or -L (~400M parameters).
Autoregressive baseline. Causal attention implemented following the common practice of GPT [38] (Figure 2(a)). The input sequence is shifted by one token (here, [cls]). Triangular masking [52] is applied the attention matrix. At inference time, temperature (7) sampling applied. We use kv-cache [44] for efficient inference.
Masked autoregressive models. With bidirectional attention (Figure 2(b)), can predict any number of unknown tokens given any number of known tokens. At training time, we randomly sample masking ratio [21, 29] in [0.7, 1.0]: e.g., 0.7 means 70% tokens are unknown. Because the sampled sequence can be very short, we always pad 64 [cls] tokens at the start of the encoder sequence, which improves the stability and capacity of our encoding. As in Figure 2, mask tokens [m] are introduced in the decoder, with positional embedding added. For simplicity, unlike [21]. we let the encoder and decoder have the same size: each has half of all blocks (e.g., 16 in MAR-L).
At inference, MAR performs "next set-of-tokens prediction". It progressively reduces the masking ratio from 1.0 to 0 with cosine schedule [4, 29]. By default, use 64 steps in this schedule. Temperature (T) sampling is applied. Unlike [4, 291, MAR always uses fully randomized orders.
5 Experiments
We experiment on ImageNet [9] at a resolution of 256x256. We evaluate FID [22] and IS [43], and provide Precision and Recall as references following common practice [10]. We follow the evaluation suite provided by [10].
5.1 Properties of Diffuusin Loss
Diffusion Loss vs. Cross-entropy Loss. We first compare continuous-valued tokens with Diffusion Loss and standard discrete-valued tokens with cross-entropy loss (Table 1). For fair comparisons, the tokenizers ("VQ-166 and "KI.16"") are both downloaded from the LDM codebase [42]. These are popularly used tokenizers (e.g., [13, 42, 37]).

---

causal Table 1: Diffuusinn Loss vs. Cross-entropy Loss. The tokenizers are VQ-16 (discrete) and KL- 16 (continuous), both from the LDM codebase [42] for fair comparisons. Diffusion Loss, with continuous-valued tokens, better than its cross-entropy counterpart with discrete-valued tokens, consistently observed across all variants of AR and MAR. All entries are implemented by us under the same setting: AR/MAR-L (~400M parameters), 400 epochs, ImageNet 256x256.
wlo CPG
w/ CFG
variant
order
direction
preds
loss
FID4
IS
FID
ISt
CrossEnt
19.58
60.8
4.92
227.3
AR
raster
Diff Loss
19.23
62.3
4.69
244.6
CrossEnt
16.22
81.3
4.36
222.7
MAR
rand
causal
Diff Loss
13.07
91.4
4.07
232.4
CrossEnt
8.75
149.6
3.50
280.9
MAR
rand bidirect
Diff Loss
3,43
203.1
1.84
292.7
CrossEnt
8.79
146.1
3.69
278.4
MAR (defanh)
rand bidirect
Diff Loss
3.50
201.4
1.98
290.3
Table 2: Flexibility of Diffusion Loss. Diffusion Loss can support different types of tokenizers.
(i) VQ tokenizers: we treat the continuous-vatued latent before VQ as the tokens. (ii) Tokenizers with a mismatched stride (here, 8): we group 2x2 2 tokens into a new token for sequence modeling. (iii) Consistency Decoder [35], non-VQ tokenizer of different decoder architecture. Here, rFID denotes the reconstruction FID of the tokenizer the ImageNet training set. Settings in this table for all entries: MAR-L, 400 epochs, ImageNet 256 226.. 1; This tokenizer is trained by us on ImugeNeet using [42]'s
code; original ones [42] were urined Openimages.
tokenizer
tokens
wlo CRG
w/ CFG
loss
arch
seq
iFID
FID
ISt
FID4
ISt
[42]
VQ-16
16"
16
5.87
7.82
151.7
3.64
258.5
[42]
KL-16
16²
16²
1.43
3,50
201.4
1.98
290.3
Diff Loss
[42]
KL-8
32
16
1.20
4.33
180.0
2.05
283.9
[35]
Consistency
32a
16
1.30
5.76
170.6
3.23
271.0
[42]
KI.-16
16'
16'
1.22
2.85
214.0
1.97
291.2
The comparisons are in four variants AR/MAR. As shown in Table 1, Diffusion Loss consistently outperforms the cross-entropy counterpart in all cases. Specifically, in MAR (e.g., the default), using Diffusion Loss can reduce FID by relatively ~50%-60% This is because the continuous-value valued KL-16 has smaller compression loss than VQ-16 (discussed next in Table 2), and also because a diffusion process models distributions more effectively than categorical ones.
In the following ablations, unless specified, we follow the "default" MAR setting in Table 1.
Flexibility of Diffusion Loss. One significant advantage of Diffusion Loss is its flexibility with various tokenizers. We compare several publicly available tokenizers in Table 2.
Diffusion Loss can be easily used even given a VQ tokenizer. We simply treat the continuous-valued latent before the VQ layer as the tokens. This variant gives us 7.82 FID (w/o CFG), compared favorably with 8. 8.79 FID (Table 1) of cross-entropy loss using the same VQ tokenizer. This suggests the better capability of diffusion for modeling distributions.
This variant also enables us to compare the VQ-16 and KL-16 tokenizers using the same loss. As shown in Table VQ-16 has much worse reconstruction FID (rFID) than KL-16, which conse- quently leads to a much worse generation FID (e.g.. 7.82 vs. 3.50 in Table 2).
Interestingly. Diffusion Loss also enables to tokenizers with mismatched strides. In Table 2, we study a KL-8 tokenizer whose stride is 8 and output sequence length is 32x32. Without in- group 2x2 2 tokens into new token. Despite the
creasing the sequence length of the generator, mismatch, we are able to obtain decent results, e.g., KL-8 gives us 2.05 FID, vs. KL-16's 1.98 FID. Further, this property allows to investigate other tokenizers, e.g., Consistency Decoder [35], non-VQ tokenizer of a different architecture/stride designed for different goals.
For comprehensiveness, we also train KL-16 tokenizer on ImageNet using the code of [42], noting that the original KI.-16 [42] was trained on OpenImages [28]. The comparison is in the last row of Table 2. We use this tokenizer in the following explorations.
7

---

MLP
w/o CFG
w/ CPG
Table 3: Denoising MLP in Difflusion Loss. The denoising MLP small and efficient. Here, the inference time involves the entire generation model, and the Transformer's size is 407M. Settings: MAR-L, 400 epochs, Im- ageNet 256 x26, MLP blocks.
width
params
FID
1st
FID
1St
inference time
256
2M
3.47
195.3
2.45
274.0
0.286 im.
512
6M
3.24
199.1
2.11
281.0
0.288 s/ im.
1024
21M
2.85
214.0
1.97
291.2
0.288 s/ im.
1536
45M
2.93
207.6
1.91
289.3
0.291 s im.
Figure 4: Sampling steps of Diffusion Loss. We show the FID (left) and IS (right) w.r.t. the number of diffusive sampling steps. Us- ing 100 steps is sufficient to achieve a strong generation quality.
300
4.5
250
Pecppttnn
3.5
1.5
200
400
600
800
1000
200
400
6.0
800
1060
Steps
Steps
Wio CFG
CFG=20
CFG-3.0
(FG.4.0
4.5
250
Figure 5: Temperature of Diffusion Loss. Temperature has clear influence on both FID (left) and IS (right). Just like the temper- ature in discrete-valued autoregression, the temperature here also plays critical role in catinuous-valued autoregression.
4.9
3.5
300
275
250
225
2.0
1.5
2.90
0:35
1.00
1.05
375
0.90
0.35
1.00
1.25
Tamperature
Tempurature
Denoising MLP in Diffusion Loss. We investigate the denoising MLP in Table 3. Even a very small MLP (e.g., 2M) can lead to competitive results. As expected, increasing the MLP width helps improve the generation quality; Note that our default MLP size (1024 width, 21M) adds only ~5% extra parameters to the MAR-L model. During inference, the diffusion sampler has decent cost of ~00% overall running time. In- creasing the MLP width has negligible extra cost in our implementation (Table 3), partially because the main overhead is not about computation but memory communication.
have explored increasing the depth and had similar observations.
Sampling Steps of Diffusion Loss. Our diffusion process follows the common practice of DDPM [24, 10]: we train with 1000-step noise schedule but inference with fewer steps. Figure 4 shows that using 100 diffusion steps at inference is sufficient to achieve a strong generation quality.
Temperature of Diffusion Loss. In the case of cross-entropy loss, the temperature is of central importance. Diffusion Loss also offers a temperature counterpart for controlling the diversity and fidelity. Figure 5 shows the influence of the temperature in the diffusion sampler (see Sec. 3.2) at inference time. The temperature plays an important role our models, similar to the observations on cross-entropy-base -based counterparts (note that the cross-entropy results in Table are with their optimal temperatures).
5.2 Properties of Generalized Autoregressive Models
From AR to MAR. Table is also a comparison on the AR/MAR variants, which we discuss next. First, replacing the raster order in AR with random order has a significant gain, e.g., reducing FID from 19.23 to 13.07 (w/o CFG). Next, replacing the causal attention with the bidirectional counterpart leads to another massive gain, e.g., reducing FID from 13.07 to 3.43 (w/o CFG).
The random-order, bidirectional AR is essentially a form of MAR that predicts one token at a time. Predicting multiple tokens (+> I') at each step can effectively reduce the number of autoregressive steps. In Table 1, we show that the MAR variant with 64 steps slightly trades off generation quality. A more comprehensive trade-off comparison is discussed next.
Speed/accuracy Trade-off. Following MaskGIT [4], our MAR enjoys the flexibility of predicting multiple tokens at time. This is controlled by the number of autoregressive steps at inference time. Figure 6 plots the speed/accuracy tra MAR has a better trade-off than its AR counterpart, noting that AR is with the efficient kv-cache.
With Diffusion Loss, MAR also shows favorable trade-off in comparison with the recently popular Diffusion Transformer (DiT) [37]. As latent diffusion model, DiT models the interdependence
8

---

5.0
Figure 6: Speed/aceuracy trade-off of the gener- ation process. For MAR, curve is obtained by different autoregressive steps (8 to 128). For DIT, curve obtained by different diffusion steps (50, 75, 150, 250) using its official code. We com- pare our implementation of AR and MAR. AR with kv-cache for fast inference. AR/MAR model size L and DiT model size is DiT-XL. The star marker denotes our default MAR setting used in other ablations. We benchmark FID and speed on ImageNet 256x256 using one A100 GPU with a batch size of 256.
AR, CrossEnt
4.5
4.0
3.5
MAR, CrossEnt
3.0
2.5
DIT
2.0
MAR, Diff Loss
(default)
1000
1.5
0.0
0.4
0.8
1.2
1.6
Inference Time (sec/ image)
Table 4: System-level comparison ImageNet 256 conditional generation, Diffusion Loss enables Masked Autoregression achieve leading results in comparison with previous systems.
LDM operates on continuous-valued tokens, though this result uses quantized tokenizer.
CHO
#params
FIDL
1St
Pre.1
Rec.1
FIDJ
1St
Pre.t
Rec.1
pixel-hased
ADM [10]
554M
10.94
101.0
0.69
0.63
4.59
186.7
0.82
0.52
VDM+ [26]
2B
2.40
225.3
2.12
267.7
vector cistor-uantzed tokens
Autoreg. w/ VQGAN [13]
1.4B
15.78
78.3
MaskGIT [4]
227M
6.18
182.1
0.80
0.51
MAGE [29]
230M
6,93
195.8
MAGVIT-v2 [55]
307M
3.65
200.5
1.78
319.4
continuous-ved valued tokens
LDM-4" [42]
400M
10.56
103.5
0.71
0.62
3.60
247.7
0.87
0.48
U-ViT-H/2-G [2]
501M
2.29
263.9
0.82
0.57
DIT-XL/2 [37]
675M
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
DifiTT [19]
1.73
276.5
0.80
0.62
2DT
676M
5.06
155.6
0.72
0.66
1.58
314.7
0.79
0.65
GIVT [48]
304M
5.67
0.75
0.59
3.35
0.84
0.53
MAR-B, Diff Loss
208M
3.48
192.4
0.78
0.58
2.31
281.7
0.82
0.57
MAR-L. DifT Loss
479M
2.60
221.4
0.79
0,60
1.78
296.0
0.81
0,60
MAR-H, Diff Loss
943M
2.35
227.8
0.79
0.62
1.55
303.7
0.81
0.62
across all tokens by the diffusion process. The speed/accuracy trade-off of DiT is mainly controlled by its diffusion steps. Unlike our diffusion process on small MIP, the diffusion process of DiT involves the entire Transformer architecture. Our method more accurate and faster. Notably, our method can generate at rate of < 0.3 second per image with strong FID of 2.0.
5.3 Benchmarking with Previous Systems
We compare with the leading systems in Table 4. We explore various model sizes (see Appendix B) and train for 800 epochs. Similar to autoregressive language models [3], we observe encouraging scaling behavior. Further investigation into scaling could be promising. Regarding metries, report 2.35 FID without CFG, largely outperforming other token-based methods. Our best entry has 1.55 FID and compares favorably with leading systems. Figure 7 shows qualitative results.
6 Discussion and Conclusion
The effectiveness of Diffusion Loss on various autoregressive models suggests new opportunities: modeling the interdependence of tokens by autoregression, jointly with the per-token distribution by diffusion. This is unlike the common usage of diffusion that models the joint distribution of all tokens. Our strong results on image generation suggest that autoregressive models or their exten- sions are powerful tools beyond language modeling. These models do not need to be constrained by vector-quantized representations. We hope our work will motivate the research community to explore sequence models with continuous-valued representations in other domains.

---

ua
Figure 7: Qualitative Results. We show selected examples of class-conditional generation on Ima-
geNet 256 256 using MAR-H with Diffusion Loss.
10

---

Acknowledgements. Tianhong Li was supported by the Mathworks Fellowship during this project.
We thank Congyue Deng and Xinlei Chen for helpful discussion. We thank Google TPU Research Cloud (TRC) for granting access to TPUs, and Google Cloud Platform for supporting GPU re- sources.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv:1607.06450 2016.
[2] Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All worth words: vit backbone for score-based diffusion models. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry. Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language mod- els are few-shot learners. In NeurlPS, 2020.
[4] Huiwen Chang, Han Zhang, Lu Jiang. Ce Liu, and William T Freeman. MaskGIT: Masked generative image Transformer. In CVPR, 2022.
[5] Huiwen Chang, Han Zhang. Jarred Barber, Maschinot, Jose Lezama, Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text- to-image generation via masked generative Transformers. In ICML, 2023.
[6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In /CML, 2020.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Picter Abbeel. PixeISNAIL: An improved autoregres- sive generative model. In ICML, 2018.
[8] Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchtiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. RSS, 2023.
[9] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierar- chical image database. In CVPR, 2009.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NewrlPS, 2021.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit. and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] Stefan Elfwing, Fiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 2018.
[13] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for high-rcsolution image syn- thesis. In CVPR, 2021.
[14] Shanghua Gao. Pan Zhou. Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion Transformer is strong image synthesizer. In /CCV, 2023.
[15] Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[16] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in I hour. arXiv:1706.0261 2017.
[17] Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autoregressive networks. JCML, 2014.
[18] Emil Julius Gumbel. Statistical theory of extreme valuse and some practical applications. Nat. Bur. Standards Appl. Math. Ser. 33, 1954.
[19] Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. DiffIT: Diffusion vision Transformers for image generation. arXiv22312.02139, 2023.

---

[20] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. 2016.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoen- coders are scalable vision learners. In CVPR, 2022.
[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two timel update rule converge to local nash equilibrium. In NIP, 2017.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv: 2022.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[25] Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv:2312.02660 2023.
[26] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the ELBO with simple data augmentation. NeurIPS, 2023.
[27] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. UViM: A unified modeling approach vision with learned guiding codes. NeurIPS, 2022.
[28] Ivan Krasin, Tom Duerig. Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng. Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. 2017.
[29] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. MAGE: Masked generative encoder to unify representation learning and image synthesis. In CVPR, 2023.
[30] Yazhe Li, Jorg Bomschein, and Ting Chen. Denoising autoregressive representation learning. arXiv preprint arXiv:2403.05/11 /996, 2024.
[31] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. /CLR, 2019.
[32] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. In ICLR, 2024.
[33] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[34] Octo Model Team, Dibya Chosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman. You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In RSS, 2024.
[35] OpenAL. Consistency Decoder, 2024. URL https: http:////.p.peeeaccccccccaaaccta
[36] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image Transformer. In /CML, 2018.
[37] William Peebles and Saining Xie. Scalable diffusion models with Transformers. In ICCV, 2023.
[38] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[39] Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[41] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ- VAE-2. In NewrlPS, 2019.
[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[43] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NeurIPS, 2016.
12

---

[44] Noam Shazeer. Fast Transformer decoding: One write-head is all need.
2019. [45] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamies. In ICMI., 2015.
[46] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
[47] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefamo Ermon, and Ben Poole. Score-based generative modeling through stnchastic differential equations. In ICLR, 2021.
[48] Michael Tschannen, Cian Eastwood, and Fabian Mentzer. GIVT: Generative infinite-vocabulary Trans- formers. arXiv:23/2.021166 2023.
[49] Aaron van den Oord, Nal Kalchbrenner, Lasse Espcholt, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with PixelCNN decoders. In NewrIPS, 2016.
[50] Aaron van den Oord, Nal Kalchbrenner, and Koray Kayukcuoglu. Pixel recurrent neural networks. In ICMI., 2016.
[51] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurlPS, 2017.
[52] Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[53] Chen Wei, Karttikeya Mangalam, Po- Yao Huang. Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models masked autoencoders. In 2023,
[54] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Haupt- mann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. MAGVIT: Masked generative vidco Transformer. In CVPR, 2023.
[55] Lijun Yu, José Lezama. Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, David A. Ross Irfan Essa, and Jiang. I Language model heats diffusion-tokenizerr key to visual generation. ICLR, 2024.
[56] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effec- tiveness of deep features as a perceptual metric. In CVPR, 2018
13

---

LINJT
Ours
DIT
Ours
DIT
Ours
DIT
Figure 8: Failure cases. Similar to existing methods, our system can produce results with noticeable artifacts. For each pair, we show MAR-H and DiT-XL's results the same class. The leftmost example of DiT is taken from their paper [37]; the others are obtained from their official code.
A Limitations and Broader Impacts
Limitations. Beyond demonstrating the potential of our method for image generation, this paper acknowledges its limitations.
First of all, image generation system can produce images with noticeable artifacts (Figure 8). This limitation is commonly observed in existing methods, especially when trained on controlled, academic data (e.g., ImageNet). Research-driven models trained on ImageNet still have noticeable gap in visual quality in comparison with commercial models trained on massive data.
Second, our image generation system relies on existing pre-trained tokenizers. The quality of our system can be limited by the quality of these tokenizers. Pre-training better tokenizers beyond the scope of this paper. Nevertheless, hope our work will make it easier to cuntinuoss-valuee tokenizers to be developed in the future.
Last, we note that given the limited computational resources, we have primarily tested our method on the ImageNet benchmark. Further validation needed assess the scalability and robustness of our approach in more diverse and real-world scenarios.
Broader Impacts. Our primary aim is to advance the fundamental research on generative models, and we believe it will be beneficial to this field. An immediate application of our method is to extend it large visual generation models, egg., text-to-image has the potential to significantly reduce the training and inference cost of these large models. At the same time, our method may suggest the opportunity replace traditional loss functions with Diffusion Loss in many applications. On the negative side, our method learns statistics from the training dataset, and as such may reflect the bias in the data; the image generation system may be misused to generate disinformation, which warrants further consideration.
texi-to-video generation. Our approach
B Additional Implementation Details
Classifier-free guidance (CFG). To support CFG [23], training time, the class condition is placed with a dummy class token for 10% of the samples [23]. At inference time, the model is run with the given class token and the dummy token, providing two outputs 2e and zu- The predicted noise e is then modified [23] as: = Ez( it, 2., + w (se(zz)t, ze) - (z Jt, 2.)), where w is the guidance scale. inference time, we use CFG schedule following [5]. We sweep the optimal guidance scale and temperature combination for each model.
Training. By default, the models are trained using the AdamW optimizer [31] for 400 epochs. The weight decay and momenta for AdamW are 0.02 and (0.9, 0.95). We use a batch size of 2048 and learning rate (Ir) of 8e-4. Our models with Diffusion Loss trained with 100-epoch linear Ir warmup [16], followed by a constant [37] Ir schedule. The cross-entropy counterparts are trained with cosine Ir schedule, which works better for them. Following [37, 25]. we maintain the exponential moving average (EMA) of the model parameters with a momentum of 0.9999.
Implementation Details of Table 4. To explore our method's scaling behavior, we study three model sizes described as follows. In addition to MAR-L, we explore a smaller model (MAR-B) and larger model (MAR-H). MAR-B, -L, and -H respectively have 24, 32, 40 Transformer blocks and a width of 768, 1024, and 1280. In Table 4 specifically, the denoising MLP respectively has 6, 8, 12 blocks and a width of 1024, 1280, and 1.36. The training length is increased 800 epochs. At inference time, we run 256 autoregressive steps to achieve the best results.
Pseudo-code of Diffusion Loss. See Algorithm 1.
14

---

Algorithm Diffusion Loss: PyTorch-like Pseudo-code
class DiffusionLoss (Mo.uuuulll)
_iintt_.. _. (depth, width):
# SimpleMLP takes in x_t, timestep, and condition, and outputs predicted noise. self net SimpleMLP (depth, width)
# GauseianDiffusion offers forward and backward functions q_sample and p_eample. self diffusion GaussianDiifuusinn)))
Given condition and ground truth token x, compute loss [self, 2, x):
# sample random noise and timestep
noise torch.randut (x. @xapee timestep torch. rordh.randint(0, self www.ffeiioo.uummteeeeeee..oo x.size(0))
# sample x_t from x_t self diffusicnneeee.. sxammll i timeatep, noise)
# predict noise from x_t noise_pred pred = nel timestep, z)
loss ((nosse_pree noise) 2) (eean()
# optional: loss *= loss_vlb
return loss
Given condition and noise, sample x using reverse diffusion process sample 2, noise):
x noise
for in 1ist (range feel..diffffsin mun_ [:::
x self diffusion dtfffueoon.p.sampleeeeeee....eee net, x, t, 2) return
Pseudo-code illustrating the concept of Diffusion Loss. Here the conditioning vector is the output from the AR/MAR model. The gradient backpropagated For simplicity. here omit the code for inference rescheduling, temperature and the loss term for variational lower bound [10], which can easily incorporated.
Compute Resources. Our training is mainly done on 16 servers with V100 GPUs each. Training 400 epochs MAR-L model takes ~26 days these GPUs. As comparison, training DiT-XL/2 and LDM-4 model for the same number of epochs on this cluster takes 4.6 and 9.5 days, respectively.
C Comparison between MAR and MAGE
MAR (regardless of the loss used) is conceptually related to MAGE [29]. Besides implementation differences (e.g., architecture specifies, hyper-parameters), major conceptual difference between MAR and MAGE is in the scanning order at inference time. In MAGE, following MaskGIT [4], the locations the next tokens to be predicted are determined on-the-fly by the sample confidence at each location, i.e., the more confident locations are more likely to be selected at each step [4, 29]. In contrast, MAR adopts fully randomized order, and its temperature sampling applied each token. Table 5 compares this difference in controlled settings. The first line is our MAR imple- mentation but using MAGE's on-the-fly ordering strategy, which has similar results as the simpler random order counterpart. Fully randomized ordering can make the training and inference process consistent regarding the distribution of orders; it also allows us to adopt token- wise temperature sampling in way similar to autoregressive language models (e.g., GPT [38, 39, 3J).
order on-the-fly random
loss
FIDI
1St
MAR, our impl. MAR, our impl.
CrossEnt
8.72
145.6
CrossEnt
8.79
146.1
MAR, our impl.
random
Diff' Loss
3,50
201.4
Table 5: To compare conceptually with MAGE, we run MAR's inference using the MAGE strategy of determining the order on the fly by confidence sampling across the spatial domain. These entries are all based on the tokenizers provided by the LDM codebase [42].
15

---

Table 6: System-level comparison on ImageNet 512x512 conditional generation. MAR's CFG scale is set to 4.0; other settings follow the MAR-L configuration described in Table 4.
CFG
w/ CPG
#params
FID
1st
FID 1St
pixel-based
ADM [10]
554M
23.24
58.1
7.72
172.7
VDM++ [26]
2B
2.99
232.2
2.65
278.1
vector- qictor-qunttied tokens
MaskGIT [4]
227M
7.32
156,0
MAGVTT-v2 [55]
307M
3.07
213.1
1.91
324.3
confinuous--valued tokens
U-VIT-H/2-G [2]
501M
4.05
263.8
DiT-XL/2 [37]
675M
12.03
105.3
3.04
DiffTT [19]
2.67
252.1
GIVT [48]
304M
8.35
EDM2-XXL [25]
1.5B
1.91
1.81
MAR-I, Loss Loss
48IM
2.74
205.2
1.73
279.9
D Additional Comparisons
D.1 Autoregressive Image Generation Pixel Space
Our MAR+DiffL.oo approach can also be directly applied to model the RGB pixel space without the need for an image tokenizer. demonstrate this, we conducted an experiment on ImageNet 64x64, grouping every 4x4 pixels into single token for the Diffusion Loss to model. A MAR- L+DifiLoss model trained for 400 epochs achieved an FID of 2.93, demonstrating the potential to eliminate the need for tokenizers in autoregressive image generation. However, as commonly observed in the diffusion model literature, directly modeling the pixel space is significantly more computationally expensive than using tokenizer. For MAR+DiffLoss, directly modeling pixels at higher resolutions might require either a much longer sequence length for the autoregressive transformer a substantially larger network for the Diffusion Loss handle larger patches. We leave this exploration for future work.
D.2 ImageNet 512x512
Following previous works, we also report results ImageNet at resolution of 512, compared with leading systems (Table 6). For simplicity, we use the KL-16 tokenizer, which gives a sequence length of 32x32
512x512 image. Other settings follow the MAR-L configuration described Table 4. Our method achieves an FID of 2.74 without CFG and 1.73 with CFG. Our results are competitive with those of previous systems. Due to limited resources, have not trained the larger MAR-H on ImageNet 512x512, which is expected to have better results.
D.3 L2 Loss vs. Diff Loss
A naïve baseline for continuous- tokens is to compute the Mean Squared Error (MSE, i.e,, L2) loss directly between the predictions and the target tokens. In the case of raster-order model, using the L2 loss introduces randomness and thus cannot generate diverse samples. In the of the MAR models with the L2 loss, the only randomness is the sequence order; the prediction at a location is deterministic for any given order. In experiment, have trained MAR model with the L.2 loss, which as expected leads to disastrous FID score (> 100).
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This paper presents autoregresive image generation method without vector quantization.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include theoretical contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: See section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
Code and checkpoints are available at htttpa://githue cot.a////...om///////aa/a/a
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines ptttps://nips.co
public//uiides/odeSSune sonisioooor for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g,, for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss cc/ ptc/public/aae geiccc/guiiee guwdge/dde/eeeeo Cumd/Suudebiii ssi onPolicy)
more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See section and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Following common practice in the generative modeling literature, do not report error bars in this paper because of the heavy computation overheads.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrpp..ee http:://narrrp...cppp..tt////
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification:
We will require the users to adhere to usage guidelines for our released mod-
els.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cite the original assets in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi thcode cww. datasttss has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "bjWj6E32ah", "reviewer": "Reviewer_V3y3", "summary": "This paper introduces a novel approach to image generation using autoregressive models with continuous-valued tokens, challenging the conventional use of discrete vector-quantized tokens. The authors propose a \"Diffusion Loss\" function that models per-token probability distributions using a diffusion process, eliminating the need for vector quantization. They demonstrate the effectiveness of this approach across various autoregressive models, including standard and masked variants. The paper also unifies autoregressive and masked generative models under a generalized framework, showing that bidirectional attention can perform autoregression. The authors implement their method using a small denoising MLP and evaluate it on ImageNet, achieving state-of-the-art results for token-based image generation. They demonstrate the flexibility of Diffusion Loss with various tokenizers and explore its properties, such as temperature control and sampling steps. The approach shows favorable speed-accuracy trade-offs compared to existing methods like Diffusion Transformers. Overall, this work opens up new possibilities for autoregressive modeling in continuous-valued domains, potentially impacting various applications beyond image generation.", "questions": "- Can authors elaborate more on the rationale behind the idea of using small NLP for denoising conditioning on latent code produced by the transformer? Like comparing to diffusion training without conditioning on latent $z$ ?\n- Have you experimented with the same model architecture but applied directly to pixels or image patches, rather than using image tokenizers? How significant is the use of more contextualized tokenized codes compared to working with pixels directly?\n- Is there a theoretical limit to how much continuous-valued tokens can improve performance over discrete ones? Are there any situations where discrete tokens might actually be better? The paper seems to assume continuous tokens are always superior, but is there any proof of this, or are there any counterexamples where this might not be true?\n- Regarding artifact handling, do you think the issues are more related to the diffusion modeling process, or are they inherent limitations of the image tokenizers themselves?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper introduces a novel approach of using continuous-valued tokens in autoregressive models for image generation, challenging the conventional wisdom of discrete vector-quantized tokens. The proposed Diffusion Loss is an innovative way to model per-token probability distributions, bridging the gap between autoregressive and diffusion models. The unification of autoregressive and masked generative models under a generalized framework is a great contribution to the field.\n- The empirical results are robust and comprehensive, demonstrating consistent improvements across various model variants and tokenizers.\n- The ablation studies and analyses of different components (e.g., denoising MLP, sampling steps, temperature) are thorough and insightful.\n- The speed-accuracy trade-off analysis provides a practical perspective on the method's performance.\n- The paper is well-structured and logically organized, guiding the reader through the concept, implementation, and results. The use of figures (especially Figures 2 and 3) effectively illustrates complex concepts like bidirectional attention for autoregression and the generalized autoregressive framework.", "weaknesses": "- While the paper demonstrates good speed-accuracy tradeoffs, I'm concerned about the overall computational complexity, especially during training. The diffusion process adds significant overhead, and training for 400 epochs seems quite intensive. How does the training time and compute requirements compare to other state-of-the-art methods?\n- While the paper explores different model sizes, a more systematic study of how performance scales with model size (similar to studies in language models) could provide valuable insights into the method's potential for further improvements. Usually diffusion models can be more parameter efficient compared to autoregressive models, I'm curious to see if it's still the case for the autoregressive diffusion modeling which is a hybrid between the two.\n- For the interpretability of the continuous-valued tokens or the learned representations. An analysis of what these tokens capture compared to discrete tokens could provide valuable insights into why the method works so well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our novel approach, significant contribution, and comprehensive empirical results. Here, we address the weaknesses (***W***) and questions (***Q***) raised by the reviewer.\n\n***Computational complexity (W1)***\n\n|Methods| Epochs |Training Costs (days)|FID w/o CFG|FID w/ CFG\n|-|-|-|-|-|\n|LDM-4|165|3.93|10.56|3.60|\n|DiT-XL/2|1400|16.1|9.62 |2.27|\n|MAR+CrossEnt|400|2.25|8.79|3.69|\n|MAR+DiffLoss|400|2.57|**3.50**|**1.98**|\n\nThe table above compares the training times of different methods on our cluster of 128 V100 GPUs. While the diffusion loss introduces some training overhead compared to cross-entropy, this overhead is relatively marginal to the computational cost of the backbone autoregressive transformer, resulting in a total training cost increase of only 14%. Additionally, the training cost of our model remains significantly lower than that of representative diffusion-based models such as DiT-XL/2 and LDM-4.\n\n***Performance vs. #Parameters (W2)***\n\nIn Table 4 of the paper, we evaluated MAR's performance with different numbers of parameters and demonstrated promising scaling signals up to 1 billion parameters. Due to resource constraints, we are unable to conduct an extensive systematic evaluation of the scaling behavior and will leave this for future work. MAR-L+DiffLoss achieves an FID of 1.78 with 479M parameters, while a representative diffusion model, DiT-XL/2, achieves an FID of 2.27 with 675M parameters. This demonstrates that MAR+DiffLoss can be more parameter efficient than common diffusion models. This is likely due to the performance improvement brought by DiffLoss, as shown in Table 1 of the paper.\n\n***Continuous tokens vs. discrete tokens (W3, Q3)***\n\nWe provide visualizations of the reconstruction performance using VQ-based and KL-based tokenizers in the attached PDF in the general response. The results clearly demonstrate that the reconstruction quality of VQ-based tokenizers is significantly worse than that of KL-based tokenizers. The KL-16 tokenizer can reconstruct far more details from the original image. For example, the face of the Mona Lisa reconstructed by the KL-16 tokenizer is much better than that reconstructed by the VQ-16 tokenizer.\n\nThis is because VQ-based tokenizers experience much more information loss due to the higher compression ratio of the quantized tokens: for example, a VQ-16 tokenizer with a codebook size of 1024 tokenizes a 256x256 image into a 16x16 sequence of discrete indices, each of which can be represented by 10 bits. In contrast, a KL-16 tokenizer encodes the same image into a 16x16x16 sequence of float numbers. Thus, the compression ratio of the VQ-16 tokenizer is much higher than that of the KL-16 tokenizer, resulting in significantly more information loss. Therefore, the lower bound of information loss by modeling KL-16 continuous tokens using Diffusion Loss is lower compared to modeling VQ-16 discrete token indices using cross-entropy loss. We believe this may explain why Diffusion Loss on continuous-valued tokens can significantly outperform cross-entropy on discrete tokens.\n\nOn the other hand, one possible advantage of discrete-valued tokens is their ability to explicitly compute the probability of each sampled discrete token, which is important for certain sampling methods such as beam search. Although prior works have shown that diffusion models can also compute the explicit probability density on the continuous distribution using a probability-flow ODE [1], such computation is relatively less straightforward.\n\n***The rationale behind using a small MLP conditioning on latent code (Q1)***\n\nWe use a small MLP conditioned on the latent code $z$ to model the probability distribution $p(x|z)$ through a diffusion process. Since the dimension of $x$ is small (16 for KL-16), the MLP can be small yet still accurately model $p(x|z)$. Such conditioning is also key to training the autoregressive transformer, as the diffusion loss is backpropagated to the large autoregressive transformer through $z$. Without conditioning on $z$, the MLP cannot effectively model $p(x|z)$, nor can the autoregressive transformer be properly trained.\n\n***Directly modeling the pixel space (Q2)***\n\nThis is an excellent question, as also pointed out by reviewer mV98. Diffusion Loss does not depend on the tokenizer and can thus be applied directly to pixel space. Given the limited time for the rebuttal, we conducted a preliminary experiment on ImageNet 64x64, grouping every 4x4 pixels into one token for Diffusion Loss to model. A MAR-L+DiffLoss model trained for 400 epochs achieved an FID of 2.93, demonstrating the potential to eliminate the need for tokenizers in autoregressive image generation.\nHowever, as commonly observed in the literature on diffusion models, directly modeling the pixel space is much more computationally expensive than using a tokenizer [2]. For MAR+DiffLoss, directly modeling pixels at a higher resolution might require either a much longer sequence length for the autoregressive transformer or a much larger network for Diffusion Loss to handle larger patches.\n\n***Artifact (Q4)***\n\nWe believe this issue is not primarily due to the KL-based continuous image tokenizer or the diffusion process. Instead, we think it is mostly due to the dataset size. The artifact problem is commonly observed when models are trained only on ImageNet (DiT also exhibits artifacts). Similar tokenizers and diffusion models are widely used in commercial models like Stable Diffusion, which experience far fewer artifacts, likely because they are trained on the much larger LAION-2B dataset. A concrete example is that when trained on LAION-400M, LDM [2] achieves significantly better performance compared to training on ImageNet, even though the model architecture and tokenizer remain almost the same.\n\n[1] Score-Based Generative Modeling through Stochastic Differential Equations\n\n[2] High-Resolution Image Synthesis with Latent Diffusion Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCu2bNK3CO", "reviewer": "Reviewer_xuXX", "summary": "This work challenges the conventional belief that autoregressive (AR) models implemented by a Transformer are best suited for modeling discrete sequences. Instead, it proposes modeling the per-token probability distribution using a diffusion procedure, enabling the application of AR models in a continuous-valued space. By defining a Diffusion Loss function instead of using categorical cross-entropy loss, this approach eliminates the need for discrete-valued tokenizers. Consequently, this framework can directly model unquantized continuous-valued tokens, avoiding information loss and degraded reconstruction performance associated with discretized tokens. Therefore, it is expected to provide better generation quality with the same compression ratio compared to AR models with discrete sequences. Experiments on class-conditional image generation demonstrate that this framework achieves strong results while benefiting from the speed advantage of sequence modeling.", "questions": "#1. In the manuscript, the authors mention that adding 64 CLS tokens at the start of the sequence helps improve the stability and capacity of our encoding when training masked generative models with bidirectional attention. Could you elaborate on why this improves stability? Is it because it sets the minimum sequence length to 64?\n\n#2. \"Vector-quantized tokenizers are difficult to train and are sensitive to gradient approximation strategies.\" Can we say this is difficult to train? While dealing with a large codebook size may require special treatment, such as restarting dead codes, training VQ-VAE or VQ-GAN with a moderate codebook size may not be difficult once one is familiar with the framework. This suggests that there is no inherent art to it.\n\n#3. Does \"16 denotes the tokenizer strides in line 219\" refer to the downsampling factor?\n\n#4. Why is the gap between CrossEnt and Diff Loss in Table 1 smaller for the AR model compared to the gap for the MAR model?\n\n#5. Experiments involving tokenizers with mismatched strides are intriguing. However, I didn't understand why we are considering using these tokenizers, given that KL-8 does not outperform KL-16 despite having the same training and inference costs.\n\n#6. Could you elaborate on why AR with random order improves fidelity compared to AR with a raster scan order?", "limitations": "The limitations of this work are thoroughly discussed in the Appendix. No societal negative impacts have been observed.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "#1. Presentation and story-telling. In my personal experience, while trying to improve VQ-GAN in various ways, I always believed that sticking to discrete tokens was necessary to train AR models. This work has a very compelling motivation to challenge this conventional belief, which many researchers and developers in this field presumably hold. I appreciate how the manuscript effectively describes the motivation and explains why quantization is not essential for training AR models.\n\n#2. This work proposes a novel and interesting idea. This generalizable framework has full potential and could serve as a fundamental building block in multi-modal LMM, effectively handling multi-modal data naturally, rather than relying on adaptor-based approaches. Previous attempts required continuous signals to be vector-quantized before training an autoregressive model with a Transformer architecture. However, this framework overcomes those limitations, advancing the field in various ways.\n\n#3. Simplicity of implementation. The architecture does not contain any specialized components or complex diffusion loss, which increases my confidence in being able to reproduce the results. Furthermore, all important experimental details are thoroughly described in the manuscript.", "weaknesses": "The main weakness of this work is the limited scope of experiments, as the manuscript only presents ImageNet 256x256 class-conditional experiments. However, the authors acknowledge this limitation in the Appendix. Additionally, I am confident that the benefits of diffusion loss validated by ImageNet will be generalizable to other datasets and tasks, including text-to-image generation. Therefore, this weakness does not outweigh the strengths of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of our motivation and proposed method. Here we address the weaknesses (***W***) and questions (***Q***) raised by the reviewer.\n\n***Scope of experiments (W1)***\n\nWe have provided an additional experiment on ImageNet 512x512 in the general response to validate our model’s ability to generalize across different resolutions. Due to resource and time constraints, we are unable to conduct extensive experiments on large-scale text-to-image datasets. However, we agree with the reviewer that the simplicity and standard implementation of the proposed Diffusion Loss make it easy to be generalized to other datasets and tasks, including text-to-image generation.\n\n***Adding CLS tokens (Q1)***\n\nIf no additional padding is used (i.e., only one [CLS] token), the encoder's input sequence can become very short at high masking ratios, sometimes as short as one token when the masking ratio is 100%. This can make the training unstable and hurt the generation performance due to the limited amount of computation spent in the encoder when the masking ratio is high. To address this, we pad the encoder sequence with 64 [CLS] tokens at the beginning, ensuring a minimum sequence length of 64. This padding stabilizes training and enhances the encoder's computation, leading to improved generation performance.\n\n***The difficulty in training VQ-based tokenizers (Q2)***\n\nWe apologize for the confusion. By “difficult to train,” we mean that VQ-based tokenizers typically require much more specialized techniques compared to standard KL-based tokenizers to achieve good reconstruction performance. These techniques include replacing dead codes, L2 normalization on the encoded latent variables, logit-Laplace loss, affine reparameterization of the code vectors, synchronized and alternating training, etc [1, 2]. This also introduces many additional hyper-parameters to tune. With similar specialized techniques and hyper-parameter tuning, the reconstruction quality of VQ-based tokenizers can lag significantly behind their continuous KL-based counterparts, as shown in the attached PDF in the general response. We will clarify this point in the revision.\n\n[1] Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks.\n\n[2] Vector-quantized Image Modeling with Improved VQGAN\n\n***Tokenizer stride (Q3)***\n\nYes, tokenizer stride means the downsampling factor of the tokenizer.\n\n***The gap between CrossEnt and Diff Loss is smaller for the AR model than the MAR model (Q4)***\n\nWe speculate that the generative power of the AR model is limited by its raster order and causal attention mechanism, which are not as well-suited for image generation as random order and bidirectional attention. Thus, the bottleneck for the AR model may lie in the generative capacity of the backbone transformer architecture rather than the nature of the data distribution, whether continuous or discrete. In contrast, the MAR model has sufficient capacity to effectively model the data distribution, with the primary bottleneck being the information loss caused by the VQ tokenizer. This might explain why Diffusion Loss sees larger gains in MAR models than in AR models. A definitive answer to this question requires further extensive experiments, which we leave as future work.\n\n***Tokenizers with mismatched strides (Q5)***\n\nThe purpose of the experiments with KL-16 and KL-8 is to demonstrate the flexibility provided by Diffusion Loss, which decouples the downsampling factor of the tokenizer from the sequence length of the autoregressive transformer. This flexibility eliminates the need to retrain the tokenizer for each sequence length. More importantly, similar to what is seen in DiT, it allows for a customizable choice of the transformer's sequence length, enabling a tailored trade-off between computational costs and performance.\n\n***Random order vs. raster order (Q6)***\n\nWe note that random order does not necessarily improve fidelity when comparing the performance of raster-order and random-order AR models with classifier-free guidance. While the random-order AR models achieve a better FID, they have a worse IS. IS primarily measures fidelity, whereas FID measures both fidelity and diversity. The improvement in FID is likely due to the global randomness introduced by the random order, which enhances the diversity of the generated images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "viFfXEgF8J", "reviewer": "Reviewer_mV98", "summary": "The paper introduces an autoregressive image generation method without vector quantisation for visual. The authors observe that while discrete-valued spaces facilitate representing categorical distributions, they are not necessary for autoregressive modeling. They propose modeling the *per-token* probability distribution using a diffusion process, allowing the application of autoregressive models in a continuous-valued space. This approach eliminates the need for discrete-valued tokenizers and has been experimentally validated across standard autoregressive models and generalized masked autoregressive variants.", "questions": "Please see weaknesses. My Open Questions and Suggestions will not affect my final rating.", "limitations": "The author has stated limitations and social impact.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work is articulated clearly, with well-founded theoretical support for its motivation. The proposed method effectively addresses the existing problems and offers significant inspiration for future research: \n\n* Clear Motivations: Under the AR, there is no intrinsic connection between the method of next token prediction and quantisation. Using discrete distributions, e.g., categorical or multinomial distributions, for image modeling is also counterintuitive.\n\n\n* Original Contributions: The work introduced the Diffusion loss with various types of AR to model the distribution of continuous tokens, thereby eliminating the need for discretisation. Although some existing literatures have used diffusion [1] or continuous token modeling [2], this work is the first to combine diffusion with AR for generative tasks with solid experiment results\n\n\n* Significant Benefits: This work takes a significant step forward in advancing continuous modeling. I personally believe image quantisation is a simplification and compromise to facilitate AR-based generative models, as modeling discrete distributions is much easier. However, quantisation inevitably leads to information loss. The proposed solution effectively mitigates this potential loss. Additionally, continuous representation is more suitable for modeling objective phenomena governed by physical laws, such as images or videos.\n\n\n[1] Li, Yazhe, Jorg Bornschein, and Ting Chen. \"Denoising Autoregressive Representation Learning.\" arXiv preprint arXiv:2403.05196 (2024).\n[2] Tschannen, Michael, Cian Eastwood, and Fabian Mentzer. \"GIVT: Generative infinite-vocabulary transformers.\" arXiv preprint arXiv:2312.02116 (2023).", "weaknesses": "### Concerns\nOverall, this work is good, but the theoretical analysis is somewhat insufficient. \n\nOne concern is that, although quantisation is avoided during generation, the entire process still relies on an encoder/decoder. Can we establish theoretical assumptions to prove that modeling with diffusion loss has lower information loss, or that its upper bound on loss is lower than the version with quantization (although Table 2 has experimentally validated this assumption)? \n\nAnother concern is after replaced Cross entropy, can we obtain an explicit prior distribution or posterior for the whole AR system? Sec 3.1 discussed about CE in traditional AR system is for Cat distribution, but I feel I cannot find similar discussions for Diffusion in the following work. \n\nThe next question is can we model the pixel space directly instead an encoder? Is this limited by the computation cost? \n\n\n### Open Questions\n\nAn open question:  quantisation is not necessary any more for AR, but is the AR still necessary with diffusion loss? Seems image tokens (whether quantised or not) differ significantly from language attributes.  The prior distribution of image tokens might be uniform thus current AR might still face the same drawbacks?\n\nAnother open question is that can we include a decoder in end-to-end training and escape the limitations of the tokenizer?\n\n\n### Suggestions\n\nFor readers, some key links appear to be missing in the reasoning chain, e.g.:\n\nWhy we use Diffusion Loss with noisy input instead of clean image with MSE for continuous modelling?  I think one potential explanation is that the objective of MSE is actually minimizing the divergence between Gaussian and token and it differs significantly from the empirical token distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of the clear motivations, original contributions, and significant benefits of our work. Here we address the weaknesses (***W***) and questions (***Q***)  raised by the reviewer. We are also happy to discuss further if you have any additional questions or confusion.\n\n***Analysis of Information Loss (W1)***\n\nAs pointed out by the reviewer, our experiment results in Table 2 and the attached PDF in the general response demonstrate that the reconstruction quality of VQ-based tokenizers is significantly worse than that of KL-based tokenizers. We believe this is because VQ-based tokenizers experience much more information loss due to their higher compression ratio.\n\nHere, we provide an informal reasoning about the information loss by analyzing the compression ratio of VQ-based and KL-based tokenizers. For example, a VQ-16 tokenizer with a codebook size of 1024 tokenizes a 256x256 image into a 16x16 sequence of discrete indices, each of which can be represented by 10 bits. In contrast, a KL-16 tokenizer encodes the same image into a 16x16x16 sequence of float numbers. Thus, the compression ratio of the VQ-16 tokenizer is much higher than that of the KL-16 tokenizer, resulting in significantly more information loss. Therefore, the lower bound of information loss by modeling KL-16 continuous tokens using Diffusion Loss is lower compared to modeling VQ-16 discrete token indices using cross-entropy loss. \n\nWe also believe a formal analysis could further strengthen the theoretical foundation of Diffusion Loss. However, due to the significant efforts required, we leave this for future work.\n\n***Explicit prior or posterior for the whole AR system (W2)***\n\nOne advantage of using a categorical distribution is that it can explicitly compute the probability of each sampled discrete token, which can then be used to compute the posterior of the entire AR system. While diffusion models are typically implicit generative models, [1] has demonstrated that it is possible to compute the exact likelihood of each continuous token using a probability flow ODE. This could potentially enable computing the posterior of the AR+DiffLoss system, offering an intriguing direction for future exploration.\n\n[1] Score-Based Generative Modeling through Stochastic Differential Equations\n\n***Directly modeling the pixel space (W3)***\n\nThis is an excellent question. Diffusion Loss is independent of the tokenizer and can be directly applied in the pixel space. Given the limited time for the rebuttal, we conduct a preliminary experiment on ImageNet 64x64, where we group every 4x4 pixels into one token for Diffusion Loss to model. A MAR-L+DiffLoss model trained for 400 epochs achieved an FID of 2.93, demonstrating the potential to eliminate the need for tokenizers in autoregressive image generation with diffusion loss. However, as commonly observed in the literature on diffusion models, directly modeling the pixel space is much more computationally expensive than using a tokenizer [2]. For MAR+DiffLoss, directly modeling pixels at a higher resolution might require either a much longer sequence length for the autoregressive transformer or a much larger network for Diffusion Loss to handle larger patches.\n\n[2] High-Resolution Image Synthesis with Latent Diffusion Models\n\n***Is AR still necessary with diffusion loss (Q1)***\n\nWe believe that one major advantage of AR/MAR models is their ability to decompose a complex joint probability distribution $p(x_1, \\cdots, x_n)$ into the product of multiple simpler conditional distributions $\\prod^n_{i=1}  p(x^i~ |~ x^1, ..., x^{i-1})$. Therefore, we believe AR/MAR models will continue to play an important role in modeling complex data distributions, such as high-resolution images or videos. Our paper aims to pave the way for this by eliminating the need for quantization and enabling the autoregressvie modeling of continuous distributions.\n\n***End-to-end training of decoder (Q2)***\n\nWe thank the reviewer for pointing out this possibility. Since Diffusion Loss eliminates the need for vector quantization, the reconstruction loss on pixels can be directly back-propagated to the continuous token output by the AR+DiffLoss part. This makes it possible to train the entire framework in an end-to-end manner. We believe exploring this possibility would be a very interesting future direction.\n\n***Direct regression using MSE loss (Suggestion)***\n\nWe thank the reviewer for the suggestion. A naive baseline to model continuous-valued tokens is to compute the Mean Squared Error (MSE, i.e., L2) loss directly between the predictions and the target tokens. In the case of a raster-order AR model, using the L2 loss introduces no randomness and thus cannot generate diverse samples. In the case of MAR models with L2 loss, the only randomness is the sequence order; the prediction at a location is deterministic for a given order. In our experiment, we trained an MAR model with L2 loss, which, as expected, led to a disastrous FID score ($>$100). We will include this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UUsjbyJJp6", "reviewer": "Reviewer_a8ib", "summary": "This paper proposes an autoregressive modeling method without the use of vector quantization tokens. By using a diffusion procedure to model the next-token probability, it is able to apply autoregressive models in a continuous-valued space. To model the probability of one token $x_i$ given the condition $X_{<i}$, the output feature of $X_{<i}$ is treated as a condition in the diffusion process of $x_i$. Afterwards,  the authors propose to unify masked modeling and autoregressive modeling through a semi-autoregressive way. Comprehensive experiments show the efficacy of the proposed method.", "questions": "Besides the above questions, there are one question concerning the implementation of the method:\n* How do the authors solve the problem of position ambiguity in predicting several tokens at a time in MAR pretraining? To be specific, if the standard ViT structure is used as stated in the paper, when predicting a group of tokens, how does the model know which position is being predicted? This problem has also been stated in [3, 4].\n\n[3] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning, https://openreview.net/forum?id=KUz8QXAgFV\n\n[4] XLNet: Generalized Autoregressive Pretraining for Language Understanding, https://arxiv.org/abs/1906.08237", "limitations": "The authors have stated the limitations in Appendix A.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The idea of integrating the diffusion model and autoregressive model is great.\n* The experiments are comprehensive and they seem to support the conclusion that the proposed method outperform the previous SOTA.", "weaknesses": "Although the experimental results seem great, there exist several non-negligible weaknesses in this paper:\n* The motivation of this paper seems weird. The authors claim that using vector quantization is not a necessity for autoregressive modeling. However, the authors neither provide sufficient examples supporting \"Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens.\" as stated in the abstract, nor demonstrate the unnecessity of vector quantization. Since previous works found it work well, why should we discard it?\n* The proposed method does not match the motivation. The authors first claim that vector quantization is not necessary in autoregressive modeling. Then the authors propose to fomulate the objective with a diffusion process, which is also complicated and hard to converge. So it seems to me that the authors just replace a complicated method with another complicated method.\n* The authors claim that they propose the unification of autoregressive and masked generative models by predicting group by group instead of token by token and in raster order instead of in fixed order. However, these ideas have already been proposed in previous works. The pioneer work should be [1], which first proposed predicting tokens group by group. There are also works on masked image modeling exploring combining masked image modeling and autoregressive modeling [2, 3]. Specifically, [3] explored exactly the same thing in Section 3 as the authors do in the paper. So it seems that this unification is not a novel idea.\n* The proposed method still needs a discrete tokenizer, as shown in the experiment section. Are there any experiment results that do not rely on the discrete tokenizer?\n\n[1] Semi-Autoregressive Neural Machine Translation, EMNLP 2018.\n\n[2] Self-supervision through Random Segments with Autoregressive Coding (RandSAC), ICLR 2023.\n\n[3] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning, https://openreview.net/forum?id=KUz8QXAgFV", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our approach and the comprehensive experimental results. It seems there is a key misunderstanding about our paper that “the proposed method still needs a discrete tokenizer”. In fact, ***ALL*** experimental results in our paper using Diffusion Loss are conducted with ***continuous KL-based tokenizers***, except for the first row in Table 2, where we show that Diffusion Loss can also work well with a discrete VQ-based tokenizer. We believe this misunderstanding may be the primary reason the reviewer questions our motivation and method. We hope the reviewer will reconsider the rating in light of this clarification. Below, we address the specific weaknesses (***W***) and questions (***Q***) raised.\n\n***Motivation (W1)***\n\nThe reviewer questions the motivation of the paper, stating that \"the authors neither provide sufficient examples supporting 'conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens,' nor demonstrate the unnecessity of vector quantization.” We respectfully disagree with these two statements. As stated in L21-25 and 68-70, almost all recent works on autoregressive image generation rely on vector-quantized tokens, starting from VQGAN [1] and DALL-E [2] to MaskGIT [3] and MAGE [4], and more recently, VAR [5] and TiTok [6]. As a side proof, Reviewer xuXX mentioned, \"I always believed that sticking to discrete tokens was necessary to train AR models,\" and Reviewer V3y3 noted, \"the paper introduces a novel approach of using continuous-valued tokens in autoregressive models for image generation, challenging the conventional wisdom of discrete vector-quantized tokens.\" These comments strongly support the notion that \"conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens.\"\n\nMoreover, as mentioned earlier, all major experimental results in our paper using Diffusion Loss DO NOT use VQ-based tokenizers. In Table 1, we show that Diffusion Loss, when used with a continuous KL-16 tokenizer, outperforms its cross-entropy counterpart with a discrete VQ-16 tokenizer, consistently observed across all variants of AR and MAR. This result clearly demonstrates the unnecessity of vector quantization and the superior advantage of using Diffusion Loss to model continuous-valued tokens.\n\n***Complication of Diffusion Loss (W2)***\n\nWe respectfully disagree with the statement that “the diffusion process is complicated and hard to converge.” The intuition behind the diffusion model is quite simple: adding and removing noise. The mathematical formulation is also straightforward, involving a standard SDE equation. Many generative modeling works in recent years have empirically demonstrated that diffusion models are not hard to train and converge, and they have been successfully applied to various domains and problems.\n\nWe also respectfully disagree with the reviewer’s statement that “the authors just replace a complicated method with another complicated method.” We use a simple MLP controlled by only two parameters, depth and width, as the denoising network. We employ a very standard diffusion process from iDDPM [7], which is widely used by common diffusion models such as ADM [7] and DiT [8]. The simplicity of Diffusion Loss is also highly appreciated by other reviewers. Reviewer xuXX states that “the architecture does not contain any specialized components or complex diffusion loss.” Reviewer oAwj notes that “the idea is simple, clean, and neat.” Thus, we believe that the proposed Diffusion Loss is a straightforward yet highly effective module that enables autoregressive image generation on continuous-valued tokens, achieving superior performance compared to traditional autoregressive models on discrete-valued tokens.\n\n***Unification of autoregressive and masked generative models (W3)***\n\nFirst, we want to clarify that the unification of autoregressive (AR) and masked generative (MAR) models is not intended as a novel technical contribution, as both approaches have been thoroughly explored in the literature [1, 3, 4, 9]. Instead, as stated in L47-51, the unification aims to broaden the scope where DiffLoss can be applied and can be helpful.\n\nSecond, the two works [10, 11] mentioned by the reviewer focus on self-supervised representation learning rather than generative models. It's important to note that within the visual generative modeling community, masked generative models are not commonly recognized as autoregressive models. In fact, many of these models are referred to as “non-autoregressive models” in the literature [3, 9]. One goal of this paper is to show that these models still possess the autoregressive characteristic of “predicting next tokens based on known ones,” allowing them to seamlessly use Diffusion Loss. As noted by Reviewer V3y3, “the unification of autoregressive and masked generative models under a generalized framework is a great contribution to the field.”\n\n***Position ambiguity (Q1)***\n\nSimilar to MAE [12], we add a learnable positional embedding to each masked token to indicate its position in the sequence.\n\n[1] Taming Transformers for High-Resolution Image Synthesis\n\n[2] Zero-Shot Text-to-Image Generation\n\n[3] MaskGIT: Masked Generative Image Transformer\n\n[4] MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis\n\n[5] Scalable Image Generation via Next-Scale Prediction\n\n[6] An Image is Worth 32 Tokens for Reconstruction and Generation\n\n[7] Diffusion Models Beat GANs on Image Synthesis\n\n[8] Scalable Diffusion Models with Transformers\n\n[9] Muse: Text-To-Image Generation via Masked Generative Transformers\n\n[10] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning\n\n[11] Self-supervision through Random Segments with Autoregressive Coding (RandSAC)\n\n[12] Masked Autoencoders Are Scalable Vision Learners", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SOTGRNXmpl", "reviewer": "Reviewer_oAwj", "summary": "This paper deals with the task of autoregressive image generation using continuous tokenizers. AR image generation has primarily focused on using discrete tokens, and training discrete tokenizers are quite hard. In this paper, the authors train AR models on continuous tokenizers. The idea is to predict continuous valued latent vectors z using a transformer, and modeling the token distribution using a diffusion model conditioned on the latents z. The use of shallow networks for the diffusion helps the models enjoy the fast inference benefit of AR models. Extensive experiments show that diffusion loss can improve the performance of AR models on Imagenet benchmark using various tokenizers.", "questions": "- Can the authors comment on the design choice of using diffusion loss to model P(x|z)? What is the reason for using diffusion? Did you think about other ways of modeling the distribution. One approach which I can think of is to simply represent P(x|z) as a gaussian similar to VAE. Can such simple distributions suffice?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Use of diffusion loss for AR models is very interesting. The idea is simple, clean and neat. The paper is well written.\n- Experiments are extensively performed on different tokenizers.\n- Performance gains are solid on different settings.", "weaknesses": "- One weakness with this approach is the slow inference speed of diffusion. The authors suggest that a small MLP is sufficient to model the distribution, so it is fast. But looking at the table 3, increasing the model size improves the performance. So, I wonder if in large-scale text-to-image benchmarks, it might be desirable to use large MLPs. If that is the case, the speed could be slower.\n- Another weakness is performing experiments only on Imagenet benchmark. As noted by authors, experiments on Imagenet always have noticeable artifacts. It is understandable that some academic labs might not have resources to run experiments on large-scale benchmarks, so I am not going to penalize for this. But, it would have been really nice to see experiments on large-scale text-to-image benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our clean and neat approach as well as our solid empirical results. Here, we address the weaknesses (***W***) and questions (***Q***) raised by the reviewer.\n\n***Inference speed (W1)***\n\nWe agree that DiffLoss can introduce overhead to the inference process, and there is always a trade-off between a larger MLP with more capacity and increased computation overhead. In our ImageNet experiment, a small MLP with a depth of 3 and a width of 1024 achieved favorable performance (1.97 FID) with marginal computational overhead during inference. For large text-to-image datasets, we assume that increasing the capacity of the autoregressive transformer will be more important than increasing the capacity of the MLP in Diffusion Loss. This is because the role of Diffusion Loss is to model a single token with a few dimensions (e.g., 16), which remains a relatively easy task even on large-scale datasets.\n\n***More experiments (W2)***\n\nWe have provided an additional experiment on ImageNet 512x512 in the general response to validate our model’s ability to generalize across different resolutions. Due to resource and time constraints, we are unable to conduct extensive experiments on large-scale text-to-image datasets. However, similar to reviewer xuXX, we believe that the simplicity and standard implementation of the proposed Diffusion Loss makes it easy to be generalized to other datasets and tasks. \n\n***Other choices to model $p(x|z)$ (Q1)***\n\nThis is an excellent question. Conceptually, $p(x|z)$ can be modeled by any conditional generative model and is not limited to diffusion models. We chose diffusion models due to their simplicity and superior performance, as demonstrated in recent literature. We believe that other generative models, such as VAE, GAN, and consistency models, also have the potential to model $p(x|z)$. Exploring these alternatives would be a very interesting future direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for providing lots of insightful and constructive feedback. We will definitely improve our manuscript accordingly. We are glad to see the commonly recognized strengths highlighted by the reviewers:

1. The presentation of the paper is clear and well-structured (xuXX, mV98, V3y3, oAwj).

2. The paper makes original and significant contributions to the field (xuXX, mV98, V3y3). The introduced idea is great (a8ib), interesting (oAwj), and novel (V3y3, xuXX).

3. The paper provides a clear and compelling motivation to move away from discrete-valued tokenizers and offers a thorough explanation of why quantization is not essential for AR models (xuXX, mV98).

4. The proposed Diffusion Loss is clean, neat (oAwj), and simple to implement (xuXX).

5. The empirical results are solid (mV98, oAwj) and comprehensive (V3y3, a8ib).

We would also like to present additional experimental results on ImageNet 512x512 (Table 1 in the attached PDF), where our MAR-L+DiffLoss model also achieves state-of-the-art performance (1.73 FID with CFG). This demonstrates our model’s ability to generalize and perform well across different resolutions.

In the attached PDF file we further compare the reconstruction performance of VQ-16 and KL-16 tokenizers provided by LDM (Figure 1). The reconstruction performance of KL-16 is clearly better than that of VQ-16, which further highlights the advantage of the proposed DiffLoss which can perform autoregressive modeling on continuous tokens provided by KL-16.

As there are no outstanding common questions, we will address each reviewer’s specific questions in separate responses. We are also happy to continue the discussion if the reviewers have any further questions or concerns.

Meta-review: This paper introduces a novel approach to autoregressive image generation by eliminating the need for discrete vector-quantized tokens, using diffusion loss in continuous-valued spaces. Reviewers broadly agree that the work is a strong and impactful contribution. Reviewer V3y3 highlighted the novelty and potential of the approach to bridge autoregressive and diffusion models, providing a unified framework that could extend to various applications beyond image generation. Reviewer xuXX appreciated the simplicity and straightforward implementation of the method, which increases reproducibility without requiring complex components. The empirical evaluations are robust and comprehensive, demonstrating clear performance gains across different models and settings, as noted by V3y3 and mV98. The paper is also well-organized and clearly presents its contributions, making it accessible to a broad audience. Some concerns were raised about computational complexity and the limited scope of experiments mainly on ImageNet. However, these were effectively addressed by the authors, who demonstrated that the additional overhead is manageable and that the method’s potential extends beyond the datasets used. Overall, the paper offers a significant and well-supported contribution to generative modeling with clear potential for future applications, warranting its acceptance.
Decision: Accept (spotlight)