Autoregressive Image Generation without Vector Quantization

Authors: tianhong  li
yonglong  tian
he  li
mingyang  deng
kaiming  he
mit  csail
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-29 04:30:12

Abstract: Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens. We observe that while a discrete-valued space can facilitate representing a categorical distribution, it is not a necessity for autoregressive modeling. In this work, we propose to model the per-token probability distribution using a diffusion procedure, which allows us to apply autoregressive models in a continuous-valued space. Rather than using categorical cross-entropy loss, we define a Diffusion Loss function to model the per-token probability. This approach eliminates the need for discrete-valued tokenizers. We evaluate its effectiveness across a wide range of cases, including standard autoregressive models and generalized masked autoregressive (MAR) variants. By removing vector quantization, our image generator achieves strong results while enjoying the speed advantage of sequence modeling. We hope this work will motivate the use of autoregressive generation in other continuous-valued domains and applications. Code is available at [https://github.com/LTH14/mar](https://github.com/LTH14/mar).

Main paper: 
 introduction: 
 Autoregressive models are currently the de facto solution to generative models in natural language processing [38, 39, 3] . These models predict the next word or token in a sequence based on the previous words as input. Given the discrete nature of languages, the inputs and outputs of these models are in a categorical, discrete-valued space. This prevailing approach has led to a widespread belief that autoregressive models are inherently linked to discrete representations.As a result, research on generalizing autoregressive models to continuous-valued domains-most notably, image generation-has intensely focused on discretizing the data [6, 13, 40] . A commonly adopted strategy is to train a discrete-valued tokenizer on images, which involves a finite vocabulary obtained by vector quantization (VQ) [51, 41] . Autoregressive models are then operated on the discrete-valued token space, analogous to their language counterparts.In this work, we aim to address the following question: Is it necessary for autoregressive models to be coupled with vector-quantized representations? We note that the autoregressive nature, i.e., "predicting next tokens based on previous ones", is independent of whether the values are discrete or continuous. What is needed is to model the per-token probability distribution, which can be measured by a loss function and used to draw samples from. Discrete-valued representations can be conveniently modeled by a categorical distribution, but it is not conceptually necessary. If alternative models for per-token probability distributions are presented, autoregressive models can be approached without vector quantization.With this observation, we propose to model the per-token probability distribution by a diffusion procedure operating on continuous-valued domains. Our methodology leverages the principles of diffusion models [45, 24, 33, 10] for representing arbitrary probability distributions. Specifically, our method autoregressively predicts a vector z for each token, which serves as a conditioning for a denoising network (e.g., a small MLP). The denoising diffusion procedure enables us to represent an underlying distribution p(x|z) for the output x (Figure 1 ). This small denoising network is 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Given a continuous-valued token x to be predicted, the autoregressive model produces a vector z, which serves as the condition of a denoising diffusion network (a small MLP). This offers a way to model the probability distribution p(x|z) of this token. This network is trained jointly with the autoregressive model by backpropagation. At inference time, with a predicted z, running the reverse diffusion procedure can sample a token following the distribution: x ∼ p(x|z). This method eliminates the need for discrete-valued tokenizers.trained jointly with the autoregressive model, with continuous-valued tokens as the input and target. Conceptually, this small prediction head, applied to each token, behaves like a loss function for measuring the quality of z. We refer to this loss function as Diffusion Loss.Our approach eliminates the need for discrete-valued tokenizers. Vector-quantized tokenizers are difficult to train and are sensitive to gradient approximation strategies [51, 41, 40, 27] . Their reconstruction quality often falls short compared to continuous-valued counterparts [42] . Our approach allows autoregressive models to enjoy the benefits of higher-quality, non-quantized tokenizers.To broaden the scope, we further unify standard autoregressive (AR) models [13] and masked generative models [4, 29] into a generalized autoregressive framework (Figure 3 ). Conceptually, masked generative models predict multiple output tokens simultaneously in a randomized order, while still maintaining the autoregressive nature of "predicting next tokens based on known ones". This leads to a masked autoregressive (MAR) model that can be seamlessly used with Diffusion Loss.We demonstrate by experiments the effectiveness of Diffusion Loss across a wide variety of cases, including AR and MAR models. It eliminates the need for vector-quantized tokenizers and consistently improves generation quality. Our loss function can be flexibly applied with different types of tokenizers. Further, our method enjoys the advantage of the fast speed of sequence models. Our MAR model with Diffusion Loss can generate at a rate of < 0.3 second per image while achieving a strong FID of < 2.0 on ImageNet 256×256. Our best model can approach 1.55 FID.The effectiveness of our method reveals a largely uncharted realm of image generation: modeling the interdependence of tokens by autoregression, jointly with the per-token distribution by diffusion. This is in contrast with typical latent diffusion models [42, 37] in which the diffusion process models the joint distribution of all tokens. Given the effectiveness, speed, and flexibility of our method, we hope that the Diffusion Loss will advance autoregressive image generation and be generalized to other domains in future research. 
related work: 
 Sequence Models for Image Generation. Pioneering efforts on autoregressive image models [17, 50, 49, 36, 7, 6] operate on sequences of pixels. Autoregression can be performed by RNNs [50] , CNNs [49, 7] , and, most lately and popularly, Transformers [36, 6] . Motivated by language models, another series of works [51, 41, 13, 40] model images as discrete-valued tokens. Autoregressive [13, 40] and masked generative models [4, 29] can operate on the discrete-valued token space. But discrete tokenizers are difficult to train, which has recently drawn special focus [27, 54, 32] . Related to our work, the recent work on GIVT [48] also focuses on continuous-valued tokens in sequence models. GIVT and our work both reveal the significance and potential of this direction. In GIVT, the token distribution is represented by Gaussian mixture models. It uses a pre-defined number of mixtures, which can limit the types of distributions it can represent. In contrast, our method leverages the effectiveness of the diffusion process for modeling arbitrary distributions.Diffusion for Representation Learning. The denoising diffusion process has been explored as a criterion for visual self-supervised learning. For example, DiffMAE [53] replaces the L2 loss in the original MAE [21] with a denoising diffusion decoder; DARL [30] trains autoregressive models with a denoising diffusion patch decoder. These efforts have been focused on representation learning, rather than image generation. In their scenarios, generating diverse images is not a goal; these methods have not presented the capability of generating new images from scratch.Diffusion for Policy Learning. Our work is conceptually related to Diffusion Policy [8] in robotics. In those scenarios, the distribution of taking an action is formulated as a denoising process on the robot observations, which can be pixels or latents [8, 34] . In image generation, we can think of generating a token as an "action" to take. Despite this conceptual connection, the diversity of the generated samples in robotics is less of a core consideration than it is for image generation. 
method: 
 In a nutshell, our image generation approach is a sequence model operated on a tokenized latent space [6, 13, 40] . But unlike previous methods that are based on vector-quantized tokenizers (e.g., variants of VQ-VAE [51, 13] ), we aim to use continuous-valued tokenizers (e.g., [42] ). We propose Diffusion Loss that makes sequence models compatible with continuous-valued tokens.To begin with, we revisit the roles of discrete-valued tokens in autoregressive generation models. Denote as x the ground-truth token to be predicted at the next position. With a discrete tokenizer, x can be represented as an integer: 0 ≤ x < K, with a vocabulary size K. The autoregressive model produces a continuous-valued D-dim vector z ∈ R D , which is then projected by a K-way classifier matrix W ∈ R K×D . Conceptually, this formulation models a categorical probability distribution in the form of p(x|z) = softmax(W z).In the context of generative modeling, this probability distribution must exhibit two essential properties. (i) A loss function that can measure the difference between the estimated and true distributions. In the case of categorical distribution, this can be simply done by the cross-entropy loss. (ii) A sampler that can draw samples from the distribution x ∼ p(x|z) at inference time. In the case of categorical distribution, this is often implemented as drawing a sample from p(x|z) = softmax(W z/τ ), in which τ is a temperature that controls the diversity of the samples. Sampling from a categorical distribution can be approached by the Gumbel-max method [18] or inverse transform sampling. This analysis suggests that discrete-valued tokens are not necessary for autoregressive models. Instead, it is the requirement of modeling a distribution that is essential. A discrete-valued token space implies a categorical distribution, whose loss function and sampler are simple to define. What we actually need are a loss function and its corresponding sampler for distribution modeling.Denoising diffusion models [24] offer an effective framework to model arbitrary distributions. But unlike common usages of diffusion models for representing the joint distribution of all pixels or all tokens, in our case, the diffusion model is for representing the distribution for each token.Consider a continuous-valued vector x ∈ R d , which denotes the ground-truth token to be predicted at the next position. The autoregressive model produces a vector z ∈ R D at this position. Our goal is to model a probability distribution of x conditioned on z, that is, p(x|z). The loss function and sampler can be defined following the diffusion models [24, 33, 10] , described next.Loss function. Following [24, 33, 10] , the loss function of an underlying probability distribution p(x|z) can be formulated as a denoising criterion:EQUATIONHere, ε ∈ R d is a noise vector sampled from N (0, I). The noise-corrupted vector x t is x t = √ ᾱt x + √ 1ᾱt ε, where ᾱt defines a noise schedule [24, 33] . t is a time step of the noise schedule. The noise estimator ε θ , parameterized by θ, is a small MLP network (see Sec. 4). The notation ε θ (x t |t, z) means that this network takes x t as the input, and is conditional on both t and z. As per [46, 47] , Eqn. (1) conceptually behaves like a form of score matching: it is related to a loss function concerning the score function of p(x|z), that is, ∇ log x p(x|z). Diffusion Loss is a parameterized loss function, in the same vein as the adversarial loss [15] or perceptual loss [56] .It is worth noticing that the conditioning vector z is produced by the autoregressive network: z = f (•), as we will discuss later. The gradient of z = f (•) is propagated from the loss function in Eqn. (1) . Conceptually, Eqn. (1) defines a loss function for training the network f (•).We note that the expectation E ε,t [•] in Eqn. (1) is over t, for any given z. As our denoising network is small, we can sample t multiple times for any given z. This helps improve the utilization of the loss function, without recomputing z. We sample t by 4 times during training for each image.Sampler. At inference time, it is required to draw samples from the distribution p(x|z). Sampling is done via a reverse diffusion procedure [24] :x t-1 = 1 √ αt x t -1-αt √ 1-ᾱt ε θ (x t |t, z) + σ t δ.Here δ is sampled from the Gaussian distribution N (0, I) and σ t is the noise level at time step t. Starting with x T ∼ N (0, I), this procedure produces a sample x 0 such that x 0 ∼ p(x|z) [24] .When using categorical distributions (Sec. 3.1), autoregressive models can enjoy the benefit of having a temperature τ for controlling sample diversity. In fact, existing literature, in both languages and images, has shown that temperature plays a critical role in autoregressive generation. It is desired for the diffusion sampler to offer a temperature counterpart. We adopt the temperature sampling presented in [10] . Conceptually, with temperature τ , one may want to sample from the (renormalized) probability of p(x|z)1 τ, whose score function is1 τ ∇ log x p(x|z). In practice, [10] suggests to either divide ε θ by τ , or scale the noise by τ . We adopt the latter option: we scale σ t δ in the sampler by τ . Intuitively, τ controls the sample diversity by adjusting the noise variance.Next, we describe the autoregressive model with Diffusion Loss for image generation. Given a sequence of tokens {x 1 , x 2 , ..., x n } where the superscript 1 ≤ i ≤ n specifies an order, autoregressive models [17, 50, 49, 36, 7, 6] formulate the generation problem as "next token prediction":EQUATIONA network is used to represent the conditional probability p(x i | x 1 , ..., x i-1 ). In our case, x i can be continuous-valued. We can rewrite this formulation in two parts. We first produce a conditioning vector z i by a network (e.g., Transformer [52] ) operating on previous tokens:z i = f (x 1 , ..., x i-1).Then, we model the probability of the next token by p(x i |z i ). Diffusion Loss in Eqn. (1) can be applied on p(x i |z i ). The gradient is backpropagated to z i for updating the parameters of f (•).We show that masked generative models, e.g., MaskGIT [4] and MAGE [29] , can be generalized under the broad concept of autoregression, i.e., next token prediction.Bidirectional attention can perform autoregression. The concept of autoregression is orthogonal to network architectures: autoregression can be done by RNNs [50] , CNNs [49, 7] , and Transformers [38, 36, 6] . When using Transformers, although autoregressive models are popularly implemented by causal attention, we show that they can also be done by bidirectional attention. See Figure 2 . Note that the goal of autoregression is to predict the next token given the previous tokens; it does not constrain how the previous tokens communicate with the next token.We can adopt the bidirectional attention implementation as done in Masked Autoencoder (MAE) [21] . See Figure 2(b) . Specifically, we first apply an MAE-style encoder 1 on the known tokens (with positional embedding [52] ). Then we concatenate the encoded sequence with mask tokens (with positional embedding added again), and map this sequence with an MAE-style decoder. The positional embedding on the mask tokens can let the decoder know at which positions are to be predicted. Unlike causal attention, here the loss is computed only on the unknown tokens [21] .With the MAE-style trick, we allow all known tokens to see each other, and also allow all unknown tokens to see all known tokens. This full attention introduces better communication across tokens than causal attention. At inference time, we can generate tokens (one or more per step) using this bidirectional formulation, which is a form of autoregression. As a compromise, we cannot use the key-value (kv) cache [44] of causal attention to speed up inference. But as we can generate multiple tokens together, we can reduce generation steps to speed up inference. Full attention across tokens can significantly improve the quality and offer a better speed/accuracy trade-off. (a) A standard, raster-order autoregressive model predicts one next token based on the previous tokens. (b) A random-order autoregressive model predicts the next token given a random order. It behaves like randomly masking out tokens and then predicting one. (c) A Masked Autoregressive (MAR) model predicts multiple tokens simultaneously given a random order, which is conceptually analogous to masked generative models [4, 29] . In all cases, the prediction of one step can be done by causal or bidirectional attention (Figure 2 ).Autoregressive models in random orders. To connect to masked generative models [4, 29] , we consider an autoregressive variant in random orders. The model is given a randomly permuted sequence. This random permutation is different for each sample. See Figure 3(b) . In this case, the position of the next token to be predicted needs to be accessible to the model. We adopt a strategy similar to MAE [21] : we add positional embedding (that corresponds to the unshuffled positions) to the decoder layers, which can tell what positions to predict. This strategy is applicable for both causal and bidirectional versions.As shown in Figure 3 (b)(c), random-order autoregression behaves like a special form of masked generation, in which one token is generated at a time. We elaborate on this as follows.Masked autoregressive models. In masked generative modeling [4, 29] , the models predict a random subset of tokens based on known/predicted tokens. This can be formulated as permuting the token sequence by a random order, and then predicting multiple tokens based on previous tokens. See Figure 3(c) . Conceptually, this is an autoregressive procedure, which can be written as estimating the conditional distribution: p({x i , x i+1 ..., x j } | x 1 , ..., x i-1 ), where multiple tokens {x i , x i+1 ..., x j } are to be predicted (i ≤ j). We can write this autoregressive model as:EQUATIONHere, X k = {x i , x i+1 ..., x j } is a set of tokens to be predicted at the k-th step, with ∪ k X k = {x 1 , ..., x n }. In this sense, this is essentially "next set-of-tokens prediction", and thus is also a general form of autoregression. We refer to this variant as Masked Autoregressive (MAR) models. MAR is a random-order autoregressive model that can predict multiple tokens simultaneously.MAR is conceptually related to MAGE [29] . However, MAR samples tokens by a temperature τ applied on the probability distribution of each token (which is the standard practice in generative language models like GPT). In contrast, MAGE (following MaskGIT [4] ) applies a temperature for sampling the locations of the tokens to be predicted: this is not a fully randomized order, which creates a gap between training-time and inference-time behavior.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
implementation: 
 This section describes our implementation. We note that the concepts introduced in this paper are general and not limited to specific implementations. More detailed specifics are in Appendix B.Diffusion Process. Our diffusion process follows [33] . Our noise schedule has a cosine shape, with 1000 steps at training time; at inference time, it is resampled with fewer steps (by default, 100) [33] . Our denoising network predicts the noise vector ε [24] . The loss can optionally include the variational lower bound term L vlb [33] . Diffusion Loss naturally supports classifier-free guidance (CFG) [23] (detailed in Appendix B).Denoising MLP. We use a small MLP consisting of a few residual blocks [20] for denoising. Each block sequentially applies a LayerNorm (LN) [1] , a linear layer, SiLU [12] , and another linear layer, merging with a residual connection. By default, we use 3 blocks and a width of 1024 channels.The denoising MLP is conditioned on a vector z produced by the AR/MAR model (see Figure 1 ). The vector z is added to the time embedding of the noise schedule time-step t, which serves as the condition of the MLP in the LN layers via AdaLN [37] .Tokenizer. We use the publicly available tokenizers provided by LDM [42] . Our experiments will involve their VQ-16 and KL-16 versions [42] . VQ-16 is a VQ-GAN [13] , i.e., VQ-VAE [51] with GAN loss [15] and perceptual loss [56] ; KL-16 is its counterpart regularized by Kullback-Leibler (KL) divergence, without vector quantization. 16 denotes the tokenizer strides.Transformer. Our architecture follows the Transformer [52] implementation in ViT [11] . Given a sequence of tokens from a tokenizer, we add positional embedding [52] and append the class tokens [cls]; then we process the sequence by a Transformer. By default, our Transformer has 32 blocks and a width of 1024, which we refer to as the Large size or -L (∼400M parameters).Autoregressive baseline. Causal attention is implemented following the common practice of GPT [38] (Figure 2 (a)). The input sequence is shifted by one token (here, [cls]). Triangular masking [52] is applied to the attention matrix. At inference time, temperature (τ ) sampling is applied. We use kv-cache [44] for efficient inference.Masked autoregressive models. With bidirectional attention (Figure 2 (b)), we can predict any number of unknown tokens given any number of known tokens. At training time, we randomly sample a masking ratio [21, 4, 29] in [0.7, 1.0]: e.g., 0.7 means 70% tokens are unknown. Because the sampled sequence can be very short, we always pad 64 [cls] tokens at the start of the encoder sequence, which improves the stability and capacity of our encoding. As in Figure 2 , mask tokens [m] are introduced in the decoder, with positional embedding added. For simplicity, unlike [21] , we let the encoder and decoder have the same size: each has half of all blocks (e.g., 16 in MAR-L).At inference, MAR performs "next set-of-tokens prediction". It progressively reduces the masking ratio from 1.0 to 0 with a cosine schedule [4, 29] . By default, we use 64 steps in this schedule. Temperature (τ ) sampling is applied. Unlike [4, 29] , MAR always uses fully randomized orders. 
experiments: 
 We experiment on ImageNet [9] at a resolution of 256×256. We evaluate FID [22] and IS [43] , and provide Precision and Recall as references following common practice [10] . We follow the evaluation suite provided by [10] .Diffusion Loss vs. Cross-entropy Loss. We first compare continuous-valued tokens with Diffusion Loss and standard discrete-valued tokens with cross-entropy loss (Table 1 ). For fair comparisons, the tokenizers ("VQ-16" and "KL-16") are both downloaded from the LDM codebase [42] . These are popularly used tokenizers (e.g., [13, 42, 37] ). The comparisons are in four variants of AR/MAR. As shown in Table 1 , Diffusion Loss consistently outperforms the cross-entropy counterpart in all cases. Specifically, in MAR (e.g., the default), using Diffusion Loss can reduce FID by relatively ∼50%-60%. This is because the continuous-valued KL-16 has smaller compression loss than VQ-16 (discussed next in Table 2 ), and also because a diffusion process models distributions more effectively than categorical ones.In the following ablations, unless specified, we follow the "default" MAR setting in Table 1 .Flexibility of Diffusion Loss. One significant advantage of Diffusion Loss is its flexibility with various tokenizers. We compare several publicly available tokenizers in Table 2 .Diffusion Loss can be easily used even given a VQ tokenizer. We simply treat the continuous-valued latent before the VQ layer as the tokens. This variant gives us 7.82 FID (w/o CFG), compared favorably with 8.79 FID (Table 1 ) of cross-entropy loss using the same VQ tokenizer. This suggests the better capability of diffusion for modeling distributions.This variant also enables us to compare the VQ-16 and KL-16 tokenizers using the same loss. As shown in For comprehensiveness, we also train a KL-16 tokenizer on ImageNet using the code of [42] , noting that the original KL-16 in [42] was trained on OpenImages [28] . The comparison is in the last row of Table 2 . We use this tokenizer in the following explorations. Temperature τ has clear influence on both FID (left) and IS (right). Just like the temperature in discrete-valued autoregression, the temperature here also plays a critical role in continuous-valued autoregression.Denoising MLP in Diffusion Loss. We investigate the denoising MLP in Table 3 . Even a very small MLP (e.g., 2M) can lead to competitive results. As expected, increasing the MLP width helps improve the generation quality; we have explored increasing the depth and had similar observations. Note that our default MLP size (1024 width, 21M) adds only ∼5% extra parameters to the MAR-L model. During inference, the diffusion sampler has a decent cost of ∼10% overall running time. Increasing the MLP width has negligible extra cost in our implementation (Table 3 ), partially because the main overhead is not about computation but memory communication.Sampling Steps of Diffusion Loss. Our diffusion process follows the common practice of DDPM [24, 10] : we train with a 1000-step noise schedule but inference with fewer steps. Figure 4 shows that using 100 diffusion steps at inference is sufficient to achieve a strong generation quality.Temperature of Diffusion Loss. In the case of cross-entropy loss, the temperature is of central importance. Diffusion Loss also offers a temperature counterpart for controlling the diversity and fidelity. Figure 5 shows the influence of the temperature τ in the diffusion sampler (see Sec. 3.2) at inference time. The temperature τ plays an important role in our models, similar to the observations on cross-entropy-based counterparts (note that the cross-entropy results in Table 1 are with their optimal temperatures).From AR to MAR. Table 1 is also a comparison on the AR/MAR variants, which we discuss next. First, replacing the raster order in AR with random order has a significant gain, e.g., reducing FID from 19.23 to 13.07 (w/o CFG). Next, replacing the causal attention with the bidirectional counterpart leads to another massive gain, e.g., reducing FID from 13.07 to 3.43 (w/o CFG).The random-order, bidirectional AR is essentially a form of MAR that predicts one token at a time.Predicting multiple tokens ('>1') at each step can effectively reduce the number of autoregressive steps. In Table 1 , we show that the MAR variant with 64 steps slightly trades off generation quality.A more comprehensive trade-off comparison is discussed next.Speed/accuracy Trade-off. Following MaskGIT [4] , our MAR enjoys the flexibility of predicting multiple tokens at a time. This is controlled by the number of autoregressive steps at inference time. Figure 6 plots the speed/accuracy trade-off. MAR has a better trade-off than its AR counterpart, noting that AR is with the efficient kv-cache.With Diffusion Loss, MAR also shows a favorable trade-off in comparison with the recently popular Diffusion Transformer (DiT) [37] . As a latent diffusion model, DiT models the interdependence 6 : Speed/accuracy trade-off of the generation process. For MAR, a curve is obtained by different autoregressive steps (8 to 128). For DiT, a curve is obtained by different diffusion steps (50, 75, 150, 250) using its official code. We compare our implementation of AR and MAR. AR is with kv-cache for fast inference. AR/MAR model size is L and DiT model size is DiT-XL. The star marker denotes our default MAR setting used in other ablations. We benchmark FID and speed on ImageNet 256×256 using one A100 GPU with a batch size of 256. across all tokens by the diffusion process. The speed/accuracy trade-off of DiT is mainly controlled by its diffusion steps. Unlike our diffusion process on a small MLP, the diffusion process of DiT involves the entire Transformer architecture. Our method is more accurate and faster. Notably, our method can generate at a rate of < 0.3 second per image with a strong FID of < 2.0.We compare with the leading systems in Table 4 . We explore various model sizes (see Appendix B) and train for 800 epochs. Similar to autoregressive language models [3] , we observe encouraging scaling behavior. Further investigation into scaling could be promising. Regarding metrics, we report 2.35 FID without CFG, largely outperforming other token-based methods. Our best entry has 1.55 FID and compares favorably with leading systems. Figure 7 shows qualitative results. 
discussion and conclusion: 
 The effectiveness of Diffusion Loss on various autoregressive models suggests new opportunities: modeling the interdependence of tokens by autoregression, jointly with the per-token distribution by diffusion. This is unlike the common usage of diffusion that models the joint distribution of all tokens. Our strong results on image generation suggest that autoregressive models or their extensions are powerful tools beyond language modeling. These models do not need to be constrained by vector-quantized representations. We hope our work will motivate the research community to explore sequence models with continuous-valued representations in other domains. Figure 8 : Failure cases. Similar to existing methods, our system can produce results with noticeable artifacts. For each pair, we show MAR-H and DiT-XL's results of the same class. The leftmost example of DiT is taken from their paper [37] ; the others are obtained from their official code. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Following common practice in the generative modeling literature, we do not report error bars in this paper because of the heavy computation overheads.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: see Appendix A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We will require the users to adhere to usage guidelines for our released models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite the original assets in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Limitations. Beyond demonstrating the potential of our method for image generation, this paper acknowledges its limitations.First of all, our image generation system can produce images with noticeable artifacts (Figure 8 ). This limitation is commonly observed in existing methods, especially when trained on controlled, academic data (e.g., ImageNet). Research-driven models trained on ImageNet still have a noticeable gap in visual quality in comparison with commercial models trained on massive data.Second, our image generation system relies on existing pre-trained tokenizers. The quality of our system can be limited by the quality of these tokenizers. Pre-training better tokenizers is beyond the scope of this paper. Nevertheless, we hope our work will make it easier to use continuous-valued tokenizers to be developed in the future.Last, we note that given the limited computational resources, we have primarily tested our method on the ImageNet benchmark. Further validation is needed to assess the scalability and robustness of our approach in more diverse and real-world scenarios.Broader Impacts. Our primary aim is to advance the fundamental research on generative models, and we believe it will be beneficial to this field. An immediate application of our method is to extend it to large visual generation models, e.g., text-to-image or text-to-video generation. Our approach has the potential to significantly reduce the training and inference cost of these large models. At the same time, our method may suggest the opportunity to replace traditional loss functions with Diffusion Loss in many applications. On the negative side, our method learns statistics from the training dataset, and as such may reflect the bias in the data; the image generation system may be misused to generate disinformation, which warrants further consideration.Classifier-free guidance (CFG). To support CFG [23] , at training time, the class condition is replaced with a dummy class token for 10% of the samples [23] . At inference time, the model is run with the given class token and the dummy token, providing two outputs z c and z u . The predicted noise ε is then modified [23] as:ε = ε θ (x t |t, z u ) + ω • (ε θ (x t |t, z c ) -ε θ (x t |t, z u )),where ω is the guidance scale. At inference time, we use a CFG schedule following [5] . We sweep the optimal guidance scale and temperature combination for each model.Training. By default, the models are trained using the AdamW optimizer [31] for 400 epochs.The weight decay and momenta for AdamW are 0.02 and (0.9, 0.95). We use a batch size of 2048 and a learning rate (lr) of 8e-4. Our models with Diffusion Loss are trained with a 100-epoch linear lr warmup [16] , followed by a constant [37] lr schedule. The cross-entropy counterparts are trained with a cosine lr schedule, which works better for them. Following [37, 25] , we maintain the exponential moving average (EMA) of the model parameters with a momentum of 0.9999. Pseudo-code illustrating the concept of Diffusion Loss. Here the conditioning vector z is the output from the AR/MAR model. The gradient is backpropagated to z. For simplicity, here we omit the code for inference rescheduling, temperature and the loss term for variational lower bound [10] , which can be easily incorporated.Our training is mainly done on 16 servers with 8 V100 GPUs each. Training a 400 epochs MAR-L model takes ∼2.6 days on these GPUs. As a comparison, training a DiT-XL/2 and LDM-4 model for the same number of epochs on this cluster takes 4.6 and 9.5 days, respectively.MAR (regardless of the loss used) is conceptually related to MAGE [29] . Besides implementation differences (e.g., architecture specifics, hyper-parameters), a major conceptual difference between MAR and MAGE is in the scanning order at inference time. In MAGE, following MaskGIT [4] , the locations of the next tokens to be predicted are determined on-the-fly by the sample confidence at each location, i.e., the more confident locations are more likely to be selected at each step [4, 29] . In contrast, MAR adopts a fully randomized order, and its temperature sampling is applied to each token. Table 5 compares this difference in controlled settings. The first line is our MAR implementation but using MAGE's on-the-fly ordering strategy, which has similar results as the simpler random order counterpart. Fully randomized ordering can make the training and inference process consistent regarding the distribution of orders; it also allows us to adopt token-wise temperature sampling in a way similar to autoregressive language models (e.g., GPT [38, 39, 3] ). Table 5 : To compare conceptually with MAGE, we run MAR's inference using the MAGE strategy of determining the order on the fly by confidence sampling across the spatial domain. These entries are all based on the tokenizers provided by the LDM codebase [42] . Our MAR+DiffLoss approach can also be directly applied to model the RGB pixel space without the need for an image tokenizer. To demonstrate this, we conducted an experiment on ImageNet 64×64, grouping every 4×4 pixels into a single token for the Diffusion Loss to model. A MAR-L+DiffLoss model trained for 400 epochs achieved an FID of 2.93, demonstrating the potential to eliminate the need for tokenizers in autoregressive image generation. However, as commonly observed in the diffusion model literature, directly modeling the pixel space is significantly more computationally expensive than using a tokenizer. For MAR+DiffLoss, directly modeling pixels at higher resolutions might require either a much longer sequence length for the autoregressive transformer or a substantially larger network for the Diffusion Loss to handle larger patches. We leave this exploration for future work.Following previous works, we also report results on ImageNet at a resolution of 512×512, compared with leading systems (Table 6 ). For simplicity, we use the KL-16 tokenizer, which gives a sequence length of 32×32 on a 512×512 image. Other settings follow the MAR-L configuration described in Table 4 . Our method achieves an FID of 2.74 without CFG and 1.73 with CFG. Our results are competitive with those of previous systems. Due to limited resources, we have not trained the larger MAR-H on ImageNet 512×512, which is expected to have better results.A naïve baseline for continuous-valued tokens is to compute the Mean Squared Error (MSE, i.e., L2) loss directly between the predictions and the target tokens. In the case of a raster-order AR model, using the L2 loss introduces no randomness and thus cannot generate diverse samples. In the case of the MAR models with the L2 loss, the only randomness is the sequence order; the prediction at a location is deterministic for any given order. In our experiment, we have trained an MAR model with the L2 loss, which as expected leads to a disastrous FID score (>100).NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Here the terminology of encoder/decoder is in the sense of a general Autoencoder, following MAE[21]. It is not related to whether the computation is casual/bidirectional in Transformers[52]. 


Decision: This paper introduces a novel approach to autoregressive image generation by eliminating the need for discrete vector-quantized tokens, using diffusion loss in continuous-valued spaces. Reviewers broadly agree that the work is a strong and impactful contribution. Reviewer V3y3 highlighted the novelty and potential of the approach to bridge autoregressive and diffusion models, providing a unified framework that could extend to various applications beyond image generation. Reviewer xuXX appreciated the simplicity and straightforward implementation of the method, which increases reproducibility without requiring complex components. The empirical evaluations are robust and comprehensive, demonstrating clear performance gains across different models and settings, as noted by V3y3 and mV98. The paper is also well-organized and clearly presents its contributions, making it accessible to a broad audience. Some concerns were raised about computational complexity and the limited scope of experiments mainly on ImageNet. However, these were effectively addressed by the authors, who demonstrated that the additional overhead is manageable and that the method’s potential extends beyond the datasets used. Overall, the paper offers a significant and well-supported contribution to generative modeling with clear potential for future applications, warranting its acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel approach to image generation using autoregressive models with continuous-valued tokens, challenging the conventional use of discrete vector-quantized tokens. The authors propose a "Diffusion Loss" function that models per-token probability distributions using a diffusion process, eliminating the need for vector quantization. They demonstrate the effectiveness of this approach across various autoregressive models, including standard and masked variants. The paper also unifies autoregressive and masked generative models under a generalized framework, showing that bidirectional attention can perform autoregression. The authors implement their method using a small denoising MLP and evaluate it on ImageNet, achieving state-of-the-art results for token-based image generation. They demonstrate the flexibility of Diffusion Loss with various tokenizers and explore its properties, such as temperature control and sampling steps. The approach shows favorable speed-accuracy trade-offs compared to existing methods like Diffusion Transformers. Overall, this work opens up new possibilities for autoregressive modeling in continuous-valued domains, potentially impacting various applications beyond image generation.
strength_weakness: strength_weakness: - While the paper demonstrates good speed-accuracy tradeoffs, I'm concerned about the overall computational complexity, especially during training. The diffusion process adds significant overhead, and training for 400 epochs seems quite intensive. How does the training time and compute requirements compare to other state-of-the-art methods?
- While the paper explores different model sizes, a more systematic study of how performance scales with model size (similar to studies in language models) could provide valuable insights into the method's potential for further improvements. Usually diffusion models can be more parameter efficient compared to autoregressive models, I'm curious to see if it's still the case for the autoregressive diffusion modeling which is a hybrid between the two.
- For the interpretability of the continuous-valued tokens or the learned representations. An analysis of what these tokens capture compared to discrete tokens could provide valuable insights into why the method works so well.
questions: questions: - Can authors elaborate more on the rationale behind the idea of using small NLP for denoising conditioning on latent code produced by the transformer? Like comparing to diffusion training without conditioning on latent $z$ ?
- Have you experimented with the same model architecture but applied directly to pixels or image patches, rather than using image tokenizers? How significant is the use of more contextualized tokenized codes compared to working with pixels directly?
- Is there a theoretical limit to how much continuous-valued tokens can improve performance over discrete ones? Are there any situations where discrete tokens might actually be better? The paper seems to assume continuous tokens are always superior, but is there any proof of this, or are there any counterexamples where this might not be true?
- Regarding artifact handling, do you think the issues are more related to the diffusion modeling process, or are they inherent limitations of the image tokenizers themselves?
limitations: limitations: Yes.
paper_summary: paper_summary: This work challenges the conventional belief that autoregressive (AR) models implemented by a Transformer are best suited for modeling discrete sequences. Instead, it proposes modeling the per-token probability distribution using a diffusion procedure, enabling the application of AR models in a continuous-valued space. By defining a Diffusion Loss function instead of using categorical cross-entropy loss, this approach eliminates the need for discrete-valued tokenizers. Consequently, this framework can directly model unquantized continuous-valued tokens, avoiding information loss and degraded reconstruction performance associated with discretized tokens. Therefore, it is expected to provide better generation quality with the same compression ratio compared to AR models with discrete sequences. Experiments on class-conditional image generation demonstrate that this framework achieves strong results while benefiting from the speed advantage of sequence modeling.
strength_weakness: strength_weakness: The main weakness of this work is the limited scope of experiments, as the manuscript only presents ImageNet 256x256 class-conditional experiments. However, the authors acknowledge this limitation in the Appendix. Additionally, I am confident that the benefits of diffusion loss validated by ImageNet will be generalizable to other datasets and tasks, including text-to-image generation. Therefore, this weakness does not outweigh the strengths of this work.
questions: questions: #1. In the manuscript, the authors mention that adding 64 CLS tokens at the start of the sequence helps improve the stability and capacity of our encoding when training masked generative models with bidirectional attention. Could you elaborate on why this improves stability? Is it because it sets the minimum sequence length to 64?

#2. "Vector-quantized tokenizers are difficult to train and are sensitive to gradient approximation strategies." Can we say this is difficult to train? While dealing with a large codebook size may require special treatment, such as restarting dead codes, training VQ-VAE or VQ-GAN with a moderate codebook size may not be difficult once one is familiar with the framework. This suggests that there is no inherent art to it.

#3. Does "16 denotes the tokenizer strides in line 219" refer to the downsampling factor?

#4. Why is the gap between CrossEnt and Diff Loss in Table 1 smaller for the AR model compared to the gap for the MAR model?

#5. Experiments involving tokenizers with mismatched strides are intriguing. However, I didn't understand why we are considering using these tokenizers, given that KL-8 does not outperform KL-16 despite having the same training and inference costs.

#6. Could you elaborate on why AR with random order improves fidelity compared to AR with a raster scan order?
limitations: limitations: The limitations of this work are thoroughly discussed in the Appendix. No societal negative impacts have been observed.
paper_summary: paper_summary: The paper introduces an autoregressive image generation method without vector quantisation for visual. The authors observe that while discrete-valued spaces facilitate representing categorical distributions, they are not necessary for autoregressive modeling. They propose modeling the *per-token* probability distribution using a diffusion process, allowing the application of autoregressive models in a continuous-valued space. This approach eliminates the need for discrete-valued tokenizers and has been experimentally validated across standard autoregressive models and generalized masked autoregressive variants.
strength_weakness: strength_weakness: ### Concerns
Overall, this work is good, but the theoretical analysis is somewhat insufficient. 

One concern is that, although quantisation is avoided during generation, the entire process still relies on an encoder/decoder. Can we establish theoretical assumptions to prove that modeling with diffusion loss has lower information loss, or that its upper bound on loss is lower than the version with quantization (although Table 2 has experimentally validated this assumption)? 

Another concern is after replaced Cross entropy, can we obtain an explicit prior distribution or posterior for the whole AR system? Sec 3.1 discussed about CE in traditional AR system is for Cat distribution, but I feel I cannot find similar discussions for Diffusion in the following work. 

The next question is can we model the pixel space directly instead an encoder? Is this limited by the computation cost? 


### Open Questions

An open question:  quantisation is not necessary any more for AR, but is the AR still necessary with diffusion loss? Seems image tokens (whether quantised or not) differ significantly from language attributes.  The prior distribution of image tokens might be uniform thus current AR might still face the same drawbacks?

Another open question is that can we include a decoder in end-to-end training and escape the limitations of the tokenizer?


### Suggestions

For readers, some key links appear to be missing in the reasoning chain, e.g.:

Why we use Diffusion Loss with noisy input instead of clean image with MSE for continuous modelling?  I think one potential explanation is that the objective of MSE is actually minimizing the divergence between Gaussian and token and it differs significantly from the empirical token distribution.
questions: questions: Please see weaknesses. My Open Questions and Suggestions will not affect my final rating.
limitations: limitations: The author has stated limitations and social impact.
paper_summary: paper_summary: This paper proposes an autoregressive modeling method without the use of vector quantization tokens. By using a diffusion procedure to model the next-token probability, it is able to apply autoregressive models in a continuous-valued space. To model the probability of one token $x_i$ given the condition $X_{<i}$, the output feature of $X_{<i}$ is treated as a condition in the diffusion process of $x_i$. Afterwards,  the authors propose to unify masked modeling and autoregressive modeling through a semi-autoregressive way. Comprehensive experiments show the efficacy of the proposed method.
strength_weakness: strength_weakness: Although the experimental results seem great, there exist several non-negligible weaknesses in this paper:
* The motivation of this paper seems weird. The authors claim that using vector quantization is not a necessity for autoregressive modeling. However, the authors neither provide sufficient examples supporting "Conventional wisdom holds that autoregressive models for image generation are typically accompanied by vector-quantized tokens." as stated in the abstract, nor demonstrate the unnecessity of vector quantization. Since previous works found it work well, why should we discard it?
* The proposed method does not match the motivation. The authors first claim that vector quantization is not necessary in autoregressive modeling. Then the authors propose to fomulate the objective with a diffusion process, which is also complicated and hard to converge. So it seems to me that the authors just replace a complicated method with another complicated method.
* The authors claim that they propose the unification of autoregressive and masked generative models by predicting group by group instead of token by token and in raster order instead of in fixed order. However, these ideas have already been proposed in previous works. The pioneer work should be [1], which first proposed predicting tokens group by group. There are also works on masked image modeling exploring combining masked image modeling and autoregressive modeling [2, 3]. Specifically, [3] explored exactly the same thing in Section 3 as the authors do in the paper. So it seems that this unification is not a novel idea.
* The proposed method still needs a discrete tokenizer, as shown in the experiment section. Are there any experiment results that do not rely on the discrete tokenizer?

[1] Semi-Autoregressive Neural Machine Translation, EMNLP 2018.

[2] Self-supervision through Random Segments with Autoregressive Coding (RandSAC), ICLR 2023.

[3] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning, https://openreview.net/forum?id=KUz8QXAgFV
questions: questions: Besides the above questions, there are one question concerning the implementation of the method:
* How do the authors solve the problem of position ambiguity in predicting several tokens at a time in MAR pretraining? To be specific, if the standard ViT structure is used as stated in the paper, when predicting a group of tokens, how does the model know which position is being predicted? This problem has also been stated in [3, 4].

[3] Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning, https://openreview.net/forum?id=KUz8QXAgFV

[4] XLNet: Generalized Autoregressive Pretraining for Language Understanding, https://arxiv.org/abs/1906.08237
limitations: limitations: The authors have stated the limitations in Appendix A.
paper_summary: paper_summary: This paper deals with the task of autoregressive image generation using continuous tokenizers. AR image generation has primarily focused on using discrete tokens, and training discrete tokenizers are quite hard. In this paper, the authors train AR models on continuous tokenizers. The idea is to predict continuous valued latent vectors z using a transformer, and modeling the token distribution using a diffusion model conditioned on the latents z. The use of shallow networks for the diffusion helps the models enjoy the fast inference benefit of AR models. Extensive experiments show that diffusion loss can improve the performance of AR models on Imagenet benchmark using various tokenizers.
strength_weakness: strength_weakness: - One weakness with this approach is the slow inference speed of diffusion. The authors suggest that a small MLP is sufficient to model the distribution, so it is fast. But looking at the table 3, increasing the model size improves the performance. So, I wonder if in large-scale text-to-image benchmarks, it might be desirable to use large MLPs. If that is the case, the speed could be slower.
- Another weakness is performing experiments only on Imagenet benchmark. As noted by authors, experiments on Imagenet always have noticeable artifacts. It is understandable that some academic labs might not have resources to run experiments on large-scale benchmarks, so I am not going to penalize for this. But, it would have been really nice to see experiments on large-scale text-to-image benchmarks.
questions: questions: - Can the authors comment on the design choice of using diffusion loss to model P(x|z)? What is the reason for using diffusion? Did you think about other ways of modeling the distribution. One approach which I can think of is to simply represent P(x|z) as a gaussian similar to VAE. Can such simple distributions suffice?
limitations: limitations: N/A