Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

## Abstract

Abstract Scaling up autoregressive models in vision has not proven as beneficial as in large language models.
In this work, we investigate this scaling problem in the context of text-to-image generation, focusing on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed raster order using BERT- or GPT-like transformer architectures.
Our empirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance—measured by FID, GenEval score, and visual quality—follows different trends. Models based on continuous tokens achieve significantly better visual quality than those using discrete tokens. Furthermore, the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared to raster-order models.
Inspired by these findings, we train Fluid, a random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zero-shot FID of 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage future efforts to further bridge the scaling gap between vision and language models.

## 1 Introduction

Scaling laws underpin the unprecedented success of large language models (LLMs). Empirically, increasing the number of parameters in autoregressive models consistently leads to significant performance improvements and the emergence of new capabilities in natural language processing (NLP) tasks (Devlin et al., ; Radford et al., ; Brown et al., ; Kaplan et al., ; Wei et al., ). This empirical relationship has inspired numerous efforts to scale up language models, resulting in the development of many highly capable models (Bubeck et al., ; Team et al., ; Achiam et al., ).

Encouraged by this success, many attempts have been made to adopt and scale up autoregressive models in computer vision, particularly for generative tasks like text-to-image generation (Yu et al., ; ; Bai et al., ; El-Nouby et al., ). However, the performance and visual quality of content generated by these models often fall short compared to other generative models, such as diffusion models (Ho et al., ; Saharia et al., ; Rombach et al., ; Esser et al., ), leaving it unclear whether similar scaling laws apply to the vision domain.

We propose several hypotheses for the performance gap.
First, the vector quantization (VQ) (van den Oord et al., ) step, which is required for most visual autoregressive models, may introduce significant information loss, ultimately limiting model performance. Second, unlike the inherently sequential nature of language, generating visual content might benefit more from a different autoregressive prediction order. Third, there is often a confusion between two levels of generalizability when evaluating scaling laws in vision models: (a) generalization to new data using the same metric as the training loss (commonly referred to as validation loss), and (b) generalization to a new metric or problem different from the training objective, such as FID (Heusel et al., ), the GenEval benchmark (Ghosh et al., ), or visual quality. We hypothesize that power-law scaling (Kaplan et al., ) applies to (a) for autoregressive models on vision data, but not necessarily to (b).

To investigate these hypotheses, we conduct a comprehensive empirical study on the scaling behavior of autoregressive models in the context of text-to-image generation. Specifically, we explore two key factors: whether the model operates on continuous or discrete tokens, and whether tokens are generated in a random or fixed raster order. To this end, we utilize the Diffusion Loss (Li et al., ) to make autoregressive models compatible with continuous tokens. We generalize BERT-like vision model MaskGIT (Chang et al., ) as random-order autoregression, as it conceptually predicts output tokens in a randomized order while retaining the autoregressive nature of “predicting next tokens based on known ones”. We analyze the behavior of four autoregressive variants, each employing different combinations of these two factors. We scale their parameters from 150M to 3B and evaluate their performance using three metrics: validation loss, FID (Heusel et al., ), and GenEval score (Ghosh et al., ). We also inspect the visual quality of the generated images.

Our experiments indicate that VQ-based models, regardless of whether they use a random or fixed raster order, exhibit a slower improvement in FID scores when scaling up model size compared to models operating on continuous tokens. VQ models also produce images of lower visual quality, likely due to information loss introduced by vector quantization.

Furthermore, the token generation order and the associated attention mechanism primarily influence the global structure of the generated image. In the GenEval benchmark, random-order models with bidirectional attention significantly outperform raster-order models with causal attention, particularly when generating multiple objects. Random-order models can readjust the global structure at every prediction step, whereas raster-order models cannot. This suggests that the token generation order plays a crucial role in achieving better text-to-image alignment.

Our experiments also demonstrate that validation loss scales as a power-law with model size, no matter whether the model operates with continuous or discrete tokens. This implies scalable behavior at the level of (a) generalization—generalizing to new data using the same metric as the training loss—which aligns with observations in language models (Kaplan et al., ). However, as for generalizing to a different metric, such as FID or GenEval score, although performance consistently improves with better validation loss, the trend may not follow a strict power-law.

Building on these findings, we scale the Fluid model, *i.e*., random-order model with continuous tokens, up to 10.5B parameters and train it using the WebLI dataset (Chen et al., ). The resulting Fluid 10.5B model achieves a zero-shot FID of 6.16 on MS-COCO and a GenEval overall score of 0.69, comparing favorably with leading text-to-image generative models such as DALL-E 3 (Betker et al., ) and Stable Diffusion 3 (Esser et al., ).
We hope that our empirical findings and positive results could shed light on the scaling behavior for text-to-image generation models and further innovation in this frontier.

## 2 Related Work

#### Text-to-image diffusion models.

The dominant approaches are based on diffusion models. Dall-E 2 (Ramesh et al., ), Imagen (Ho et al., ), and Stable Diffusion (Rombach et al., ) revolutionized text-to-image generation. Most recently, SD v3 (Esser et al., ) and Imagen3 (Baldridge et al., ) can generate realistic images that are hard to tell from real images. However, generating samples is usually computationally expensive due to the multiple forward passes.

#### Autoregressive (AR) models.

While being the defacto model for language modeling, AR models are lagging behind diffusion models for text-to-image generation. AR models (Yu et al., ; Chang et al., ; Yu et al., ) are often used together with discrete tokenizers (van den Oord et al., ; Esser et al., ), which often limit the modeling capability. For example, Parti (Yu et al., ) scales up the model to 20B, which obtains a slightly lower/better FID score of 7.23 on MS-COCO than 7.27 by the 3.4B diffusion-based Imagen. Here we find that, with a continuous tokenizer, our small 369M model can already achieve the same FID score as Parti with 20B model.

Recently Li et al. () challenges the conventional wisdom and replaces the discrete tokenizer with a continuous tokenizer via a diffusion loss. They introduced masked autoregressive (MAR) model obtains strong results for class-conditioning generation on ImageNet. However, scaling MAR models for text-to-image generation is unexplored. Here we empirically study the scaling behavior for MAR and report several findings important for the research community.

#### Scaling language models.

Kaplan et al. () empirically observed that, for language model, the validation loss scales as a power-law with model size, dataset size, and the amount of compute used for training. Hoffmann et al. () discovered that contemporary LLMs are under-trained, and that for compute-optimal training, the model size and the number of training tokens should be scaled equally. Under the same compute budget while using 4x more more data, their 70B Chinchilla outperforms much larger models, such as the 530B Megatron-Turing NLG (Smith et al., ). Wei et al. () found that larger models have emergent abilities that are not present in smaller models.
These observations have inspired significant efforts to scale up language models to trillions of parameters (Achiam et al., ; Team et al., ; Dubey et al., ).

#### Scaling vision models.

Similar scaling law has been obscure for computer vision. For recognition models (Tan, ; Zhai et al., ; He et al., ; Dehghani et al., ), scaling often comes with diminishing returns. For example, Dehghani et al. () scale ViT up from 3B to 22B but only observed 0.25% accuracy increase in ImageNet linear probing. The scaling of generative models is more promising - DiT (Peebles & Xie, ) shows consistent improvement in generation quality when scaling up compute and model size (despite only up to 600M). The follow-up Sora (OpenAI, ) further shows the potential to scale up for video generation. In this paper, we perform a comprehensive empirical study of AR models for text-to-image generation.

## 3 Preliminary: Autoregressive Image Generation

Given a sequence of tokens $\{x^{1},x^{2},...,x^{n}\}$ where the superscript $1\leq i\leq n$ specifies an order, autoregressive models (Gregor et al., ; van den Oord et al., ; ; Parmar et al., ; Chen et al., ; ) formulate the generation problem as “next token prediction”:

$$ $p(x^{1},...,x^{n})=\prod^{n}_{i=1}p(x^{i}\leavevmode\nobreak\ |\leavevmode \nobreak\ x^{1},...,x^{i-1}).$ (1) $$

Following the chain rule, the network is trained to model $p(x^{i}\leavevmode\nobreak\ |\leavevmode\nobreak\ x^{1},...,x^{i-1})$ and generate tokens iteratively.
While all autoregressive models share this fundamental approach, differences in their design can affect the performance. Two key design choices are the representation of $x$, *i.e*., discrete or continuous, and the generation order, which we elaborate below.

#### Discrete vs . continuous tokens.

The goal of an autoregressive model is to estimate $p(x^{i}\leavevmode\nobreak\ |\leavevmode\nobreak\ x^{1},...,x^{i-1})$.
Traditionally, this is done by transforming the image into a set of discrete tokens with a finite vocabulary
and then estimating a categorical distribution over the vocabulary. The training objective is to minimize the cross-entropy loss, and sampling can be efficiently performed using categorical sampling. Most autoregressive image generation models rely on this form of token discretization (Esser et al., ; Chang et al., ; Tian et al., ; Yu et al., ).

However, such discretization often leads to a significant loss of information from the image (Figure 4). Recent work (Li et al., ) has shown the possibility of applying a small diffusion model to approximate the distribution of each image token in a continuous fashion. This approach eliminates the need for vector quantization, and allows modeling images with continuous tokenizers which yield much better reconstruction visual quality. In this paper, we explore the scaling behavior of autoregressive image models on both discrete and continuous tokens.

Figure: Figure 2: Autoregressive models with different orders. (a) A raster-order autoregressive model predicts one next token based on the known ones, implemented using a GPT-like transformer with causal attention. (b) A random-order autoregressive model predicts one or multiple tokens simultaneously given a random order, implemented using a BERT-like transformer with bidirectional attention.
Refer to caption: x2.png

#### Raster Order + GPT vs. Random Order + BERT.

In autoregressive image generation, there are two primary generation orders: raster and random. As illustrated in Figure 2, raster order generates tokens sequentially from left to right, top to bottom. This fixed-order generation is well-suited for a GPT-like transformer architecture, which predicts the next token in a causal manner. In contrast, random order allows multiple tokens to be generated in each step. The selection of these tokens can either be completely random or based on a sampling mechanism that prioritizes tokens with higher predicted confidence scores (Chang et al. (); Li et al. ()).

Each generation order has its pros and cons. Raster order models with GPT-like transformer support fast inference via key-value (kv) caching. However, this causal structure can also introduce performance degradation. On the other hand, random order generation is usually achieved with a BERT-like bidirectional attention mechanism. While this approach prevents the usage of kv-caching, it enables the model to decode multiple tokens at each autoregressive step, allowing global editing.
Despite their individual strengths, it remains unclear in the literature which generation order scales better for text-to-image generation tasks. In this work, we compare the performance and scaling behaviors of raster-order and random-order autoregressive models.

## 4 Implementation

The overall framework of our text-to-image model training is straightforward. An image tokenizer first converts the original image into tokens. These tokens are then partially masked, and a transformer is trained to reconstruct the masked tokens conditioned on the text. Below, we provide a detailed description of each component of our framework (shown in Figure 3).

Figure: Figure 3: Our text-to-image generation framework. A pre-trained image tokenizer converts the image into either discrete or continuous tokens. The text is embedded using a pre-trained T5 encoder, followed by a trainable text aligner. The transformer then takes cross-attention from the text embeddings to predict the missing tokens (only random order model is shown here).
Refer to caption: x3.png

Figure: Figure 4: Reconstruction quality of the tokenizers. Image resolution is 256x256. The discrete tokenizer is significantly worse than the continuous tokenizer.
Refer to caption: extracted/5933431/figures/recon/monalisa/gt.png

Image Tokenizer. We use a pre-trained image tokenizer to encode 256$\times$256 images into a token space. Such a tokenizer can be either discrete or continuous, facilitating different training objectives of the autoregressive model. In our experiments, the discrete tokenizer is an VQGAN model (Esser et al., ) pre-trained on the WebLI dataset (Chen et al., ). We follow Muse (Chang et al., ) to encode each image into 16$\times$16 discrete tokens with a vocabulary size of 8192. For the continuous tokenizer, we adopt a widely-used one from Stable Diffusion (Rombach et al., ), which encodes the image into 32$\times$32 continuous tokens, each containing 4 channels. To be consistent in sequence length with the discrete tokenizer, we further group each 2$\times$2 patch of continuous tokens into a single token, resulting in a final sequence length of 256, with each token containing 16 channels.
As shown in Figure 4, the continuous tokenizer can achieve notably higher reconstruction quality than the discrete one.

Text Encoder. The raw text (maximum length of 128) is tokenized by SentencePiece (Kudo, ),
and embedded through a pre-trained T5-XXL encoder (Raffel et al., ), which has 4.7B parameters and is frozen during training. To further align the text embeddings for image generation, we add a small text aligner consisting of six trainable transformer blocks on top of the T5 embeddings, to extract the final text representation.

Transformer.
After encoding the original image into a sequence of tokens, we use a standard decoder-only transformer model (Vaswani et al., ) for autoregressive generation. Each block consists of three consecutive layers – self-attention, cross-attention, and MLP. The self-attention and MLP layers are only applied to visual tokens, while the cross attention layer takes visual and textual tokens as queries and keys, respectively. As shown in Figure 2, for raster-order models, the transformer predicts the next token based on previous tokens using causal attention for the self-attention block, similar to GPT. In random-order models, unknown tokens are masked by a learnable token, and the transformer predicts these masked tokens using bidirectional attention, similar to BERT.

Output head. For *discrete* tokens, we follow the common practice with autoregressive models. The outputs are transformed into categorical distributions by softmax following a linear layer, whose weights are reused from the input embedding layer. For *continuous* tokens, we apply a six layer light-weight MLP as the diffusion head (Li et al., ) to model the per-token distribution. The embedding dimension of this head is the same as the backbone transformer. The per-token diffusion process follows (Nichol & Dhariwal, ; Li et al., ). The noise schedule has a cosine shape, with
1000 steps at training time; at inference time, it is resampled to 100 steps.

## 5 Experiments

#### Dataset.

We use a subset of the WebLI (Web Language Image) dataset (Chen et al., ) as our training set, which consists of image-text pairs from the web with high scores for both image quality and alt-text relevance. By default, the images are center-cropped and resized to 256$\times$256.

#### Training.

Unless otherwise specified, we use the AdamW optimizer ($\beta_{1}=0.9,\beta_{2}=0.95$) (Loshchilov & Hutter, ) with a weight decay of 0.02 to train each model for 1M steps with a batch size of 2048. This is equivalent to approximately 3 epochs on our dataset. For continuous tokens, we employ a constant learning rate schedule with a 65K-step linear warmup and a maximum learning rate of $1\times 10^{-4}$; for discrete tokens, we use a cosine learning rate schedule as we find it to be better. For training the random-order models, we randomly sample the masking ratio from [0, 1] following a cosine schedule, similar to MaskGIT (Chang et al., ), to mask each image. For all models, exponential moving average of the weights are gathered by a decay rate of 0.9999 and then used for evaluation.

#### Inference.

We follow the practices established by Imagen (Saharia et al., ), Muse (Chang et al., ), and Parti (Yu et al., ) to generate images from text prompts without rejection sampling. For random-order models, we use 64 steps for generation with a cosine schedule (Chang et al., ). To further enhance generation performance, we apply temperature and classifier-free guidance, as is commonly practiced.

#### Evaluation.

We evaluate the scaling behavior of different autoregressive model variants both quantitatively and qualitatively. Quantitatively, we evaluate the validation loss on 30K images from the MS-COCO 2014 training set, as well as two widely-adopted metrics: zero-shot Frechet Inception Distance (FID) on MS-COCO, and the GenEval score (Ghosh et al., ). Inference hyper-parameters, such as temperature and classifier-free guidance, are optimized for each evaluation metric. FID is computed over 30K randomly selected image-text pairs from the MS-COCO 2014 training set, providing a metric that evaluates both the fidelity and diversity of generated images. The GenEval benchmark, on the other hand, measures the model’s ability to generate images that accurately reflect the given prompt. For qualitative evaluation, we generate images from several prompts using each model and compare the visual quality of the generated images.

Figure: Figure 5: Validation loss scales as a power-law with model size. The validation loss is evaluated on 30K images randomly sampled from the MS-COCO 2014 training set. The x and y axes are in log-scale. The change in y is relatively small for each plot, making the log-scale alike linear-scale.
Refer to caption: x4.png

### 5.1 Scaling Behaviors

In this section, we explore how two key design choices in autoregressive image generative models—token representation and generation order—affect performance and scaling behavior. We construct models with different combinations of these two design choices, resulting in four distinct variants of autoregressive image generation models. We also explore the generalizability of these models across different data and evaluation metrics. Our experiments reveal several intriguing properties.

#### Validation losses consistently scale with model size.

In Figure 6, we examine the scaling behavior of the four autoregressive variants in terms of validation loss. We observe a linear relationship between validation loss and model size in the log space, as we increases model size from 150 million to 3 billion parameters. This aligns with the power-law finding in Henighan et al. (). This demonstrates that the improvements in training loss resulting from increased model size generalize well to validation loss on data different from the training data.

#### Random-order models with continuous tokens scale the best in evaluation scores.

In Figure 6, we analyze the scaling behavior of the four autoregressive variants in terms of FID and GenEval overall scores. We find that the improvements observed in validation loss do not always translate linearly to better evaluation metrics, implying that there is no strict power-law relationship between these metrics and model size. For example, raster-order models with discrete tokens (blue line) reach a plateau in both FID and GenEval scores around 1B parameters. Among the four variants, random-order models with continuous tokens (*i.e*., Fluid) show consistent improvements in evaluation metrics up to 3B parameters, achieving the best overall performance. Therefore, we focus on further investigating the scaling behavior of this model.

Figure: Figure 7: Validation losses and evaluation performance scale with increasing training steps and computes. We use random-order models with continuous tokens. Results for other autoregressive variants are included in the appendix. The training compute is computed as model GFLOPs$\times$batch size$\times$training steps$\times$3, where the factor of 3 accounts for the backward pass being approximately twice as compute-intensive as the forward pass.
Refer to caption: x6.png

Figure: Figure 8: Validation loss and evaluation metrics are highly correlated. We use random-order models with continuous tokens. The Pearson correlation coefficients for FID and GenEval scores are 0.917 and -0.931, respectively. We also observe that the linear correlation slightly weakens and becomes less pronounced for the 3.1B model.
Refer to caption: x7.png

#### Random-order models with continuous tokens scale with training computes.

In Figure 7, we plot validation loss, FID, and GenEval scores as functions of total training steps and compute for different model sizes of Fluid. We observe consistent improvements in both validation loss and evaluation performance with increased training steps and compute. However, the benefits from additional training steps saturate around 1M steps, indicating that training smaller models for more steps is less compute-efficient compared to training larger models for fewer steps. This behavior aligns with observations in language models, highlighting the potential for scaling up model sizes with sufficient training.

#### Strong correlation between validation loss and evaluation metrics.

In Figure 8, we plot FID and GenEval scores against validation loss for different model sizes of Fluid and observe a strong correlation. To quantify this, we fit the data points using linear regression. The Pearson correlation coefficients for FID and GenEval scores are 0.917 and -0.931, respectively, indicating a nearly linear relationship between validation loss and these evaluation metrics across model sizes ranging from 150M to 3B(^1^11Since both FID and GenEval scores have lower/upper bounds, this linear relationship cannot hold indefinitely. Future work should explore the limits of this correlation.). Encouraged by this positive trend, we trained a model with 10.5B parameters and a batch size of 4096 for 1M steps, achieving state-of-the-art text-to-image generation performance, as discussed in the next section.

Figure: Figure 9: Visual quality and image-text alignment improves with increasing model size. *Best viewed zoomed-in.* Fluid • achieves the highest visual quality and best image-text alignment.
Refer to caption: x8.png

#### Continuous tokens and large models are crucial for visual quality.

In Figure 9, we compare the visual quality of images generated by the four autoregressive variants. The visual quality of models using discrete tokens is significantly worse than that of models using continuous tokens, *e.g*., the eyes of the corgi is asymmetric for discrete token based models and scaling up can not solve this problem. This limitation is largely because of the discrete tokenizer, which introduces substantial information loss. For instance, even with 3B parameters, the discrete token models cannot generate an accurate Mona Lisa due to poor reconstruction quality of the tokenizer (Figure 4). In contrast, models with continuous tokens produce much higher-quality images.

Additionally, larger models show consistent improvements in both visual quality and image-text alignment. For example, a random-order model with 0.2B parameters struggles to generate “an angry duck doing heavy weightlifting at the gym”, while the same model with 3B parameters can generate the corresponding images successfully. This demonstrates that modeling continuous tokens and increasing model size are crucial for achieving high visual quality in autoregressive image generation models.

### 5.2 Benchmarking with Previous Systems

**Table 1: System-level comparison. Fluid achieves leading results on both MS-COCO zero-shot FID-30K and GenEval benchmark (Ghosh et al., ). ^†: CM3Leon result is reported without retrieval.**
|  |  |  | MS-COCO | GenEval |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | #params | FID-30K$\downarrow$ | Single Obj. | Two Obj. | Counting | Colors | Position | Color Attri. | Overall |
| diffusion model |  |  |  |  |  |  |  |  |  |  |
|  | LDM | 1.4B | 12.64 | 0.92 | 0.29 | 0.23 | 0.70 | 0.02 | 0.05 | 0.37 |
|  | DALL-E 2 | 4.2B | 10.39 | 0.94 | 0.66 | 0.49 | 0.77 | 0.10 | 0.19 | 0.52 |
|  | DALL-E 3 | - | - | 0.96 | 0.87 | 0.47 | 0.83 | 0.43 | 0.45 | 0.67 |
|  | Imagen | 3B | 7.27 | - | - | - | - | - | - | - |
|  | SD3 | 8B | - | 0.98 | 0.84 | 0.66 | 0.74 | 0.40 | 0.43 | 0.68 |
|  | Transfusion | 7.3B | 6.78 | - | - | - | - | - | - | 0.63 |
|  | RAPHAEL | 3B | 6.61 | - | - | - | - | - | - | - |
| autoregressive model |  |  |  |  |  |  |  |  |  |  |
|  | CM3Leon^† | 7B | 10.82 | - | - | - | - | - | - | - |
|  | Show-o | 1.3B | 9.24 | 0.95 | 0.52 | 0.49 | 0.82 | 0.11 | 0.28 | 0.53 |
|  | Muse | 3B | 7.88 | - | - | - | - | - | - | - |
|  | Parti | 20B | 7.23 | - | - | - | - | - | - | - |
|  | Fluid (our work) | 369M | 7.23 | 0.96 | 0.64 | 0.53 | 0.78 | 0.33 | 0.46 | 0.62 |
|  |  | 665M | 6.84 | 0.96 | 0.73 | 0.51 | 0.77 | 0.42 | 0.51 | 0.65 |
|  |  | 1.1B | 6.59 | 0.96 | 0.77 | 0.61 | 0.78 | 0.34 | 0.53 | 0.67 |
|  |  | 3.1B | 6.41 | 0.98 | 0.83 | 0.60 | 0.82 | 0.41 | 0.53 | 0.70 |
|  |  | 10.5B | 6.16 | 0.96 | 0.83 | 0.63 | 0.80 | 0.39 | 0.51 | 0.69 |

In this section, we compare our Fluid, *i.e*., continuous random-order autoregressive model, with leading text-to-image generation systems in Table 1 (Rombach et al., ; Ramesh et al., ; Betker et al., ; Saharia et al., ; Esser et al., ; Zhou et al., ; Xue et al., ; Yu et al., ; Xie et al., ; Chang et al., ; Yu et al., ). The Fluid smallest model, with 369M parameters, achieves a zero-shot FID of 7.23 on MS-COCO and a GenEval overall score of 0.62, matching the performance of many state-of-the-art models with several billion parameters (*e.g*., Parti with 20B parameters only achieves 7.23). The Fluid largest model, with 10.5B parameters, further improves the zero-shot FID on MS-COCO to 6.16 and increases the GenEval overall score to 0.69(^2^22We observe that the GenEval scores plateau for the 10.5B Fluid compared to the 3.1B Fluid; however, it continues to show consistent improvements in visual quality and FID.), with a speed of 1.571 seconds per image per TPU (evaluated on 32 TPU v5 with a batch size of 2048). Detailed model configurations and generation speed results are included in the appendix. We hope these strong results and promising scaling behavior provide valuable insights and support for the scalability of autoregressive models in visual generative modeling.

## 6 Discussion

In this paper, we present an empirical study on the scaling behavior of autoregressive models for text-to-image generation. We investigate two critical design factors: random order versus raster order, and discrete tokens versus continuous tokens. Our results show that random-order models with continuous tokens achieve the best performance and scaling behavior across various evaluation metrics and in terms of visual quality. Building on these findings, we scale up the random-order model with continuous tokens, namely Fluid, to 10.5B parameters, and achieves state-of-the-art text-to-image generation performance.
We hope that our findings and promising results could provide valuable insights into the scaling behavior of autoregressive models for image generation and help bridge the gap between the scaling performance of vision models and language models.

Reproducibility Statement. To aid reproducibility, we have provided the implementation details of our framework in Section 4, training hyper-parameters in Section 5, and model configurations in the Appendix. For the diffusion loss used for continuous tokens, we have strictly followed the open-sourced code of Li et al. ().

Acknowledgements. We would like to express our gratitude to Amy Shen and Alex Rizkowsky for their assistance in securing computational resources, and to Yash Katariya, Ivy Zheng, and Roy Frostig for their valuable insights on JAX-related questions. We also thank Matan Cohen for his support with precomputed T5 embeddings. Our appreciation extends to Han Zhang, Jason Baldridge, Dilip Krishnan, David Salesin and VisCam team for their helpful discussions and valuable feedback.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Is there any specific differences compared to MAR paper?
2.	Any thoughts on why random order generation works better than raster? Why continuous better than discrete? Would love to see more reasoning beyond the empirical results.
3.	How does the training time compare across setups, especially when scaling up?
4.	In the inference time, how does the inference time step influence the model performance?
5.	Is the method scales well for image resolution? For example, finetune the model on a small, but large-resolution dataset. Current diffusion model often works well for much larger resolution like 512 and 1024. The 256^2 image generated in the paper is somehow limited for real application. 
6.	A very important question for me is not confirming the scaling law of AR-based model for image generation(always, larger compute give better results). Rather, we are interested in comparing the diffusion model and the AR in term of the speed of scaling. This paper somehow demonstrates Fluid scales better than causal, discrete-token AR model. If it is possible, a comparing with pure diffusion model is very important.
7.	Are there any failure cases for the model? 
8.	Not exactly a research problem, but will the code be open-sourced? Because this paper is mostly an empirical study, sharing the code and model would be the most important contribution than reading the paper. I personally think this is a key consideration when judging this paper.
One minor suggestion: While using a random scan order prevents the typical use of KV caching for causal sequences, an alternative could be to adopt a structured order, such as a grid-like pattern. This way, you could still cache info from nearby patches and keep some benefits of caching while generating the current patch.
soundness: 3
strengths: Solid Scaling Analysis: The paper does a thorough job of laying out the scaling trends. The consistent improvements with random order and continuous tokens give a clear picture of what’s working.

Concrete study on Toke type and Scan order: The results convincingly show that generating tokens in a random order outperforms the traditional raster order, especially for tasks needing a good global structure. Additionally, switching to continuous tokens noticeably enhances image quality.

This is a solid empirical paper that offers important insights into scaling autoregressive models, mostly on scaling the MAR model (Li et al., 2024). The work holds relevance for both academia and industry, with the potential for meaningful social impact.
summary: The paper explores scaling autoregressive models for text-to-image generation, focusing on two key factors: discrete vs. continuous token representations, and token generation order (raster vs. random). The results show that using continuous tokens and generating in random order lead to better image quality, especially at scale. The new Fluid model, which adopts these strategies, achieves state-of-the-art results on benchmarks like MS-COCO and GenEval.
weaknesses: I value the contribution of the paper. While I am generally positive, we can still find some imperfect points in the paper.

Methodology Largely Follows MAR (Li et al., 2024): The approach in this paper closely follows the method used in MAR, with some modifications to incorporate text condition. While these changes are valuable, the paper doesn’t go far beyond what MAR already established. Also, the discussion of scan order and token type is also a repeat of what has been done in the MAR paper.

Over-Reliance on Empirical Results: While the paper does a great job presenting empirical evidence, there’s a lack of theoretical explanation for why continuous tokens outperform discrete ones. The benefits are demonstrated, but there isn't much discussion on how continuous representations help in practice—whether it’s due to better gradient flow, more expressive capacity, or something else entirely. More in-depth analysis could add substance to the findings.

Scaling Analysis Could Be More Comprehensive: The paper primarily showcases large model almost for sure improve the performance. However, other important factors in scaling, such as the effect of dataset size, are not thoroughly explored. For instance, language model research, like Kaplan et al. (2020), investigates how increasing dataset size influences validation loss, providing insights into the relationship between data and model scaling. In contrast, all the experiments in this paper are conducted on the same subset of the WebLI dataset, leaving the impact of different data sizes unexamined.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: - My questions arise naturally from the "Weaknesses", I believe the authors may have more insights beyond the experiments results provided.
  - What underlying factors drive the observed correlation between metrics and the linear relationship between validation loss and the metrics?
  - What should be considered the most "human-aligned" evaluation criterion for future research?
  - More analysis of the benefits of the continuous tokens and random-order prediction should be explained, especially during the scaling up, e.g., why does random-order prediction yield a larger loss but better visual quality?
- Similar to the "Scaling Law" curves in the language model community, an ablation on the number of training examples across the four configurations could be beneficial.
- Is there any "Emergence" phenomena during training, such as a sudden increase in metric scores?
- I would consider raising my scores if these points are addressed.
soundness: 3
strengths: - Achieve SOTA performance with zero-shot FID of 6.16 on MS-COCO 30K, and an overall score of 0.69 on the GenEval benchmark.
- The paper is well-organized and easy to follow, with robust training data, strategies, and solid experimental setups.
- Extensive experiments reveal clear relations between the four configurations of token types and generation order.
summary: This paper investigates the scaling behaviors of autoregressive models in text-to-image generation, focusing on key factors: token type (discrete vs. continuous) and generation order (random vs. raster). Experiment results demonstrate a power-law relationship between the validation loss and model sizes, and that continuous tokens with random-order generation yield the best performance on evaluation metrics. Leveraging these findings, the authors introduce Fluid, a 10.5B random-order autoregressive model with continuous tokens, outperforming previous models on benchmarks such as FID and GenEval.
weaknesses: - This paper ablated two types of existing technologies, with their scaling behaviors. The experiments are well-designed, and the authors' observations appear reasonable. However, some additional insights are needed.
- I appreciate the authors' efforts in sharing extensive experimental results, but providing an actionable conclusion would add more value than observations alone. If the authors could provide insights or equations that can guide future training of large-scale text-to-image models—such as identifying optimal settings when constrained by FLOPs, image samples, or model size—it would significantly benefit the field.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In the experimental results presented in Figure 6, it seems that with the minimal number of parameters, continuous tokens do not outperform discrete tokens. Would the authors be able to provide further analysis on why this might be the case?

2. In the implementation of continuous tokens, an extra diffusion head is used, whereas no comparable diffusion structure is applied for discrete tokens. If possible, could the authors conduct further ablation studies to examine the impact of this diffusion head?

3. The authors also mention that these techniques vary not only in performance but in efficiency as well. Would it be possible to provide a more detailed comparison of inference efficiency?
soundness: 3
strengths: 1. The writing in the article is clear and easy to understand. 

2. The improvements to the original autoregressive model, such as continuous token and random generation order, are both insightful and empirically effective techniques. These empirical results provide crucial experimental evidence that supports some of the fundamental beliefs about autoregressive image generation.
summary: This work investigates the performance of autoregressive models in generating high-quality images from text prompts, focusing on the use of continuous versus discrete tokens and the impact of token generation order. The authors find that models using continuous tokens produce superior visual quality compared to those with discrete tokens, indicating that continuous representations capture more nuanced information. Additionally, they reveal that random-order generation outperforms fixed raster-order generation, highlighting the significance of the token generation sequence on image quality. Employing various evaluation metrics, the study compares its results with leading diffusion models and autoregressive models, reporting favorable outcomes. Overall, the research provides empirical insights into the scaling behavior of text-to-image generation models, emphasizing the importance of token representation and generation strategy in enhancing autoregressive models' capabilities for high-resolution image synthesis.
weaknesses: 1. Although this article is the first to compare the roles of token representation and generation order together in the autoregressive image generation, the techniques it examines are not novel or originally proposed by the authors. For instance, line 189 notes that the continuous token method is derived from a prior work, and line 287 mentions that the implementation of random order also originates from an earlier study. This work does not optimize or update the technical details of these methods. I highly appreciate the article’s empirical study on autoregressive image generation, but the scope seems limited to experimental evaluation of existing techniques without introducing new methods or innovations.

2. Autoregressive image generation still lacks a standardized technical framework, and both its training and inference processes remain complex and non-standardized. The approach involves various technical details beyond just token representation and generation order. I highly appreciate the authors' efforts to highlight these two factors amidst the complexity. However, different token representations and generation orders introduce other technical variations, making it challenging to ensure absolute fairness in comparisons. This limitation may prevent us from making strictly definitive claims based on the comparative results.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: See weaknesses.
soundness: 4
strengths: 1. This paper successfully scales previous auto-regressive models based on continuous tokens up to the text-to-image setting, which is more complicated and hard-to-train than class-conditioned models.
2. The quantitative results are good given the results in Tab. 1.
3. The paper is well-written and I am able to fully follow the authors' writing.
summary: This paper proposes an auto-regressive text-to-image generation model. The authors specifically explore two aspects, namely random order v.s. raster order and continuous token v.s. discrete token. Through experiments, the authors conclude that random order with continuous tokens is better. Evaluation on MS-COCO and GenEval benchmarks indicate better performance compared with previous diffusion-based and auto-regressive-based models.
weaknesses: 1. It seems that the novelty is limited. The applied techniques are essentially the same as those proposed in [a], which can hardly bring readers new technical insights. The overall feeling of the paper is more like a technical report instead of a research paper.
2. The resolution used in the paper is 256x256, which is relatively small compared with recent models.
3. Also, I am not sure whether the comparison in Tab. 1 is fair because different systems may have been trained on different resolutions.
4. LlamaGen [b] can serve as a recently proposed baseline for comparisons.
5. Lack of qualitative comparisons and efficiency comparisons with recent models. Readers can hardly get the strengths of the proposed model against state-of-the-art ones.


[a] Autoregressive Image Generation without Vector Quantization, Li et al., 2024

[b] Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation, Sun et al., 2024

Meta-review: This paper investigates scaling behavior in autoregressive text-to-image models, focusing on token representation (discrete vs continuous) and generation order (random vs raster). The key findings show that continuous tokens with random-order generation scale most effectively, enabling state-of-the-art results with their 10.5B Fluid model. The paper demonstrates strong empirical results and thorough analysis, with reviews highlighting clear presentation and comprehensive experiments. While initial concerns were raised about novelty and resolution limitations, the authors provided strong responses during rebuttal: They demonstrated their model's effectiveness at 512x512 resolution achieving comparable FID and GenEval scores to 256x256, conducted new experiments showing benefits over recent work like LlamaGen, and clarified their key contribution of being first to systematically study scaling of autoregressive vision models at large scale (up to 10.5B parameters). The review scores (initially 5, 6, 6, 6) improved during discussion as reviewers were convinced by the additional experiments and clarifications. Though some concerns remain about theoretical justification, the thorough empirical validation and clear practical importance of the findings support accepting this paper, which offers valuable guidance for future large-scale autoregressive vision models. Thus I vote to accept this paper.

Decision: Accept (Poster)