Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens
Authors: Lijie Fan
Tianhong Li
Siyang Qin
Yuanzhen Li
Chen Sun
Michael Rubinstein
Deqing Sun
Kaiming He
Yonglong Tian
Venue: ICLR 2025 Poster (2025)
Keywords: generative model
scaling
text-to-image
autoregressive
Primary area: generative models
TL;DR: An Empirical Study of Scaling Autoregressive Text-to-image Generative Models with Discrete and Continuous tokens

Abstract: Scaling up autoregressive models in vision has not proven as beneficial as in large language models.
In this work, we investigate this scaling problem in the context of text-to-image generation, focusing on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed raster order using BERT- or GPT-like transformer architectures. Our empirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends. Models based on continuous tokens achieves significantly better visual quality than those using discrete tokens. Furthermore, the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared to raster-order models.
Inspired by these findings, we train Fluid, a random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zeor-shot FID of 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage future efforts to further bridge the scaling gap between vision and language models.

Published as a conference paper at ICLR 2025
FLUID: SCALING AUTOREGRESSIVE TEXT-TO-IMAGE GENERATIVE MODELS WITH CONTINUOUS TOKENS
Lijie Fan Tianhong Li
Siyang Qin
Yuanzhen Li'
Chen Sun'
Michael Rubinstein
Deqing Sun!
Kaiming He equal contribution, project lead
Yonglong Tian
'Google Deep
MIT
equaa contribution
Figure 1: Samples from our Fluid 10.5B autoregressive model with continuous tokens.
ABSTRACT
Scaling up autoregressive models in vision has not proven beneficial as in large language models. In this work, we investigate this scaling problem the context of text-to-image generation, focusing discrete or continuous tokens, and whether tokens are generated random or fixed raster order using BERT- GPT-like transformer architectures. Our pirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance-measvne coality-folloss different trends. Models based on continuous tokens achieve significantly better visual quality than those using discrete tokens. Furthermore. the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared raster-order models. Inspired by these findings, we train Fluid, random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zero-shot FID of 6. 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage fu- ture efforts to further bridge the scaling gap between vision and language models.
critical factors: whether models use
by FID, GenEval score, and visual

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Scaling laws underpin the unprecedented success of large language models (LLMs). Empirically, in- creasing the number of parameters in autoregressive models consistently leads to significant perfor- mance improvements and the emergence of new capabilities in natural language processing (NLP) tasks (Devlin et al., 2018; Radford et al., 2018; Brown al., 2020; Kaplan et al., 2020; Wei al., 2022). This empirical relationship has inspired numerous efforts to scale up language models, re- sulting in the development of many highly capable models (Bubeck et al., 2023; Team et al., 2023; Achiam et al., 2023).
Encouraged by this success, many attempts have been made to adopt and scale up autoregressive models in computer vision, particularly for generative tasks like text-to-image generation (Yu et al., 2021: 2023: Bai al., 2024; El-Nouby al., 2024). However, the performance and visual quality of content generated by these models often fall short compared to other generative models, such as diffusion models (Ho et al., 2020; Saharia al., 2022; Rombach et al., 2022a; Esser et al., 2024), leaving it unclear whether similar scaling laws apply to the vision domain.
We propose several hypotheses for the performance gap. First, the vector quantization (VQ) (van den Oord et al., 2017) step, which is required for most visual autoregressive models, may introduce sig- nificant information loss, ultimately limiting model performance. Second, unlike the inherently sequential nature of language, generating visual content might benefit more from a different autore- gressive prediction order. Third, there is often a confusion between two levels of generalizability when evaluating scaling laws in vision models: (a) generalization to new data using the same metric as the training loss (commonly referred to as validation loss), and (b) generalization to a new metric or problem different from the training objective, such as FID (Heusel et al., 2017), the GenEval benchmark (Ghosh et al., 2024), or visual quality. We hypothesize that power-law scaling (Kaplan al., 2020) applies to (a) for autoregressive models on vision data, but not necessarily to (b).
To investigate these hypotheses, we conduct a comprehensive empirical study on the scaling behav- ior of autoregressive models the context of text-to-image generation. Specifically, we explore two key factors: whether the model operates on continuous or discrete tokens, and whether tokens are generated in random or fixed raster order. To this end, we utilize the Diffusion Loss (Li et al., 2024) to make autoregressive models compatible with continuous tokens. We generalize BERT-like vision model MaskGIT (Chang et al., 2022) as random-order autoregression, as conceptually pre- dicts output tokens in randomized order while retaining the autoregressive nature of "predicting next tokens based on known ones". We analyze the behavior of four autoregressive variants, each employing different combinations of these two factors. We scale their parameters from 150M to 3B and evaluate their performance using three metrics: validation loss, FII) (Heusel et al., 2017), 7), and GenEval score (Ghosh al., 2024). We also inspect the visual quality of the generated images.
Our experiments indicate that VQ-based models, regardless of whether they use a random or fixed raster order, exhibit slower improvement in FID scores when scaling up model size compared to models operating on continuous tokens. VQ models also produce images of lower visual quality, likely due to information loss introduced by vector quantization.
Furthermore, the token generation order and the associated attention mechanism primarily influence the global structure of the generated image. In the GenEval benchmark, random-order models with bidirectional attention significantly outperform raster-order models with causal attention, particu- larly when generating multiple objects. Random-order models can readjust the global structure at every prediction step, whereas raster-order models cannot. This suggests that the token generation order plays a crucial role achieving better text-to-image alignment.
Our experiments also demonstrate that validation loss scales matter whether the model operates with continuous or discrete tokens. This implies scalable behav- ior at the level of (a) generalizationnggeeeeee loss-whice aligns with observations in language models (Kaplan et al., 2020). However, as for generalizing different metric, such as FID GenEval score, although performance consistently improves with better validation loss, the trend may not follow a strict power-law.
power-law with model size, no
to new data using the same metric the training
Building these findings. scale the Fluid model, i.e., random-order model with continuous tokens, up to 10.5B parameters and train it using the WebLI dataset (Chen et al., 2022). The resulting Fluid 10.5B model achieves zero-shot FID of 6.16 MS-COCO and GenEval overall score of
2

---

Published as a conference paper at ICLR 2025
0.69, comparing favorably with leading rext-to-image generative models such as DALL-E 3 (Betker et al., 2023) and Stable Diffusion 3 (Esser et al., 2024). We hope that our empirical findings and positive results could shed light on the scaling behavior for text-to-image generation models and further innovation in this frontier.
RELATED WORK
Text-to-image diffusion models. The dominant approaches are based on diffusion models. Dall- 2 (Ramesh et al., 2022), Imagen (Ho et al., 2022), and Stable Diffusion (Rombach et al., 2022a) revolutionized text-to-image generation. Most recently, SD v3 (Esser et al., 2024) and Imagen3 (Baldridge et al., 2024) can generate realistic images that are hard to tell from real im- ages. However, generating samples is usually computationally expensive due the multiple forward passes.
Autoregressive (AR) models. While being the defacto model for language modeling, AR models are lagging behind diffusion models for text-to-image generation. AR models (Yu et al., 2022; Chang al., 2023; Li et al., 2023; Yu et al., 2023) are often used together with discrete tokenizers (van den Oord et al.. 2017; Esser et al., 2021), which often limit the modeling capability. For example, Parti (Yu et al., 2022) scales up the model 20B, which obtains a slightly lower/better FID score of 7.23 on MS-COCO than 7.27 by the 3.4B diffusion based Imagen. Here we find that, with continuous tokenizer, our small 369M model can already achieve the same FID score as Parti with 20B model.
Recently Li et al. (2024) challenges the conventional wisdom and replaces the discrete tokenizer with a continuous tokenizer via diffusion loss. They introduced masked autoregressive (MAR) model obtains strong results for glass-conditioning generation on ImageNet. However, scaling MAR models for text-to-image generation is unexplored. Here empirically study the scaling behavior for MAR and report several findings important for the research community.
Scaling language models. Kaplan et al. (2020) empirically observed that, for language model, the validation loss scales as power-law with model size, dataset size, and the amount compute used for training. Hoffmann et al. (2022) discovered that contemporary LLMS are under-trained, and that for compute-optimal training, the model size and the number of training tokens should be scaled equally. Under the same compute budget while using 4x more more data, their 70B Chinchilla outperforms much larger models, such as the 530B Megatron- Turing NLG (Smith et al., 2022). Wei et al. (2022) found that larger models have emergent abilities that not present in smaller models. These observations have inspired significant efforts to scale up language models to trillions of parameters (Achiam et al., 2023; Team et al., 2023; Dubey al., 2024).
Scaling vision models. Similar scaling law has been obscure for computer vision. For recognition models (Tan, 2019; Zhai et al., 2022; He et al., 2022; Dehghani et al., 2023), scaling often comes with diminishing returns. For example, Dehghani et al. (2023) scale ViT up from 3B to 22B but only observed 0.25% accuracy increase in ImageNet linear probing. The scaling of generative models is more promising DiT (Peebles & Xie, 2023) shows consistent improvement in generation quality when scaling up compute and model size (despite only up 600M). The follow-up Sora (OpenAI, 2024) further shows the potential to scale up for video generation. In this paper, we perform comprehensive empirical study of AR models for text-to-image generation.
PRELIMINARY: AUTOREGRESSIVE IMAGE GENERATION
Given a sequence of tokens {ø 2 ..... r²) where the superscript 1 < i < specifies an order, autoregressive models (Gregor et al., 2014; van den Oord al., 201 2016b;a; Parmar al., 2018; Chen et al., 2018; 2020) formulate the generation problem as "next token prediction":
p(x
II p(æ" ..... x 11)
(1)
Following the chain rule, the network is trained to model p(æ" x ...., x² x'1) and generate tokens iteratively. While all autoregressive models share this fundamental approach, differences in their
3

---

Published as a conference paper at ICLR 2025
known/preedicted
to predict this step
unknown
Ed]
[mi
[m]
loss
loss
loss
loss
loss
(a) Raster Order, Causal Attention
(b) Random Order. Bidirectional Attention
Figure 2: Autoregressive models with different orders. (a) A raster-order autoregressive model predicts one next token based on the known ones, implemented using GPT-like transformer with causal attention. (b) A random-order autoregressive model predicts one multiple tokens simul- taneously given a random order, implemented using a BERT-like transformer with bidirectional attention.
design can affect the performance. Two key design choices are the representation of æ, i.e., discrete or continuous, and the generation order, which we elaborate below.
Discrete continuous tokens. The goal of autoregressive model is to estimate p(x' x² x',...,z'"'), Traditionally, this done by transforming the image into a set of discrete tokens with finite vocabulary and then estimating categorical distribution over the vocabulary. The training objective is to minimize the cross-entropy loss, and sampling can be efficiently per- formed using categorical sampling. Most autoregressive image generation models rely on this form of token discretization (Esser et al., 2021; Chang et al., 2022; Tian et al., 2024; Yu et al., 2022).
However, such discretization often leads significant loss of information from the image (Figure 4). Recent work (Li et al., 2024) has shown the possibility of applying a small diffusion model to approximate the distribution of each image token continuous fashion. This approach eliminates the need for vector quantization, and allows modeling images with continuous tokenizers which yield much better reconstruction visual quality. this paper, we explore the scaling behavior of autoregressive image models on both discrete and continuous tokens.
In autoregressive image generation, there are two primary generation orders: raster and random. As illustrated in Figure 2. raster order generates tokens sequentially from left to right, top bottom. This fixed-order generation well-suited for GPT-like transformer architecture, which predicts the next token in causal manner. In contrast, random order allows multiple tokens be generated in each step. The selection of these tokens can either be completely random or based sampling mechanism that prioritizes tokens with higher predicted confidence scores (Chang et al. (2023); Li et al. (2024)).
Raster Order GPT vs. Random Order + BERT.
Each generation order has its pros and cons. Raster order models with GPT-like transformer sup- port fast inference via key-value (kv) caching. However, this causal structure can also introduce performance degradation. On the other hand, random order generation is usually achieved with BERT-like bidirectional attention mechanism. While this approach prevents the usage of ky-caching, enables the model decode multiple tokens each autoregressive step, allowing global editing. Despite their individual strengths, it remains unclear in the literature which generation order scales better for text-to-image generation tasks. In this work, we compare the performance and scaling behaviors of raster-order and random-order autoregressive models.
IMPLEMENTATION
The overall framework of our text-to-image model training is straightforward. An image tokenizer first converts the original image into tokens. These tokens are then partially masked, and trans- former is trained to reconstruct the masked tokens conditioned on the text. Below, we provide detailed description of each component of our framework (shown Figure 3).
4

---

Published as a conference paper at ICLR 2025
"Mona Lisa"
Transformer Blocks
T5 Text Encoder
Self Altention
Frozen
Attention
Image Tokenizer
Trainable
Text Aligner
Output Head
Our text-to-image generation framework. A pre-trained image tokenizer converts the image into either discrete or continuous tokens. The text is embedded using pre-trained T5 en- coder, followed by a trainable text aligner. The transformer then takes cross-attention from the text embeddings to predict the missing tokens (only random order model is shown here).
Figure 3:
Image Tokenizer. We use a pre-trained image tokenizer to encode x56x256 im- ages into token space. Such tok- enizer can be either discrete or continu- ous, facilitating different training objec- tives of the autoregressive model. In our experiments, the discrete tokenizer is an VQGAN model (Esser et al., 2021) pre- trained on the WebLI dataset (Chen et al., 2022). We follow Muse (Chang al., 2023) to encode each image into 16x 16 discrete tokens with vocabulary size of 8192. For the continuous tokenizer, we adopt widely-used one from Stable Diffusion (Rombach et al., 2022b), which encodes the im- age into 32x x2 continuous tokens, each containing 4 channels. To be consistent in sequence length with the discrete tokenizer, we further group each 2x2 patch of continuous tokens into single ken, resulting in a final sequence length of 256, with each token containing 16 channels. As shown Figure 4, the continuous tokenizer can achieve notably higher reconstruction quality than discrete one.
Original
Discrete
Continuous
PSNR: 26.6
PSNR: 31.5
Figure 4: Reconstruction quality of the tokenizers. Image resolution is 256x256. The discrete tokenizer is significantly worse than the continuous tokenizer.
The raw text (maximum length of 128) is tokenized by SentencePiece (Kudo, 2018), Text Encoder.
and embedded through pre-trained T5-XXL encoder (Raffel et al., 2020), which has 4.7B param- eters and is frozen during training. To further align the text embeddings for image generation, we add small text aligner consisting of six trainable transformer blocks on top of the T5 embeddings, to extract the final text representation.
standard decoder-only transformer model (Vaswani et al., 2017) for autoregressive generation. Each block consists of three consecutive layers self-attention, cross-attention, and MLP. The self-attention and MLP layers are only applied to visual tokens, while the cross attention layer takes visual and textual tokens as queries and keys, respectively. As shown Figure for raster-order models, the transformer predicts the next token based on previous tokens using causal attention for the self- attention block, similar to GPT. random-orde -rderr models, unknown tokens are masked by learnable token, and the transformer predicts these masked tokens using bidirectional attention, similar to BERT.
Transformer. After encoding the original image into sequence of tokens,
Output head. For discrete tokens, we follow the common practice with autoregressive models. The outputs are transformed into categorical distributions by softmax following linear layer, whose weights are reused from the input embedding layer. For continuous tokens, apply six layer light- MLP as the diffusion head (Li et al., 2024) to model the por-token distribution. The embedding dimension of this head is the same the backbone transformer. The per-token diffusion process follows (Nichol Dhariwal, 2021: et al., 2024). The noise schedule has a cosine shape, with 1000 steps training time; at inference time, it is resampled to 100 steps.

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
Dataset. We use subset of the WebLI (Web Language Image) dataset (Chen et al., 2022) as training set, which consists of image-text pairs from the web with high scores for both image quality and alt-text relevance. By default, the images center-cropped and resized to 256x256.
Training. Unless otherwise specified, we use the A optimizer (3 0.9, 32 0.95) o
(Loshchilov & Hutter, 2019) with a weight decay of 0.02 to train each model for 1M steps with a batch size of 2048. This is equivalent to approximately epochs on our dataset. For continuous tokens, we employ a constant learning rate schedule with 65K-step linear warmup and a maximum learning rate of 10-4; for discrete tokens, we use a cosine learning rate schedule as we find it to be better. For training the random-order models, randomly sample the masking ratio from [0, 1] following a cosine schedule, similar to MaskGIT (Chang et al., 2022). to mask each image. For all models, exponential moving average of the weights are gathered by decay rate of 0.9999 and then used for evaluation.
Inference. We follow the practices established by Imagen (Saharia et al., 2022), Muse (Chang al., 2023), and Parti (Yu et al., 2022) to generate images from text prompts without rejection sampling. For random-order models, we use 64 steps for generation with cosine schedule (Chang al., 2022). To further enhance generation performance, we apply temperature and classifier-fres guidance, as is commonly practiced.
Evaluation. We evaluate the scaling behavior of different autoregressive model variants both quantitatively and qualitatively. Quantitatively, we evaluate the validation loss on 30K images from the MS-COCO 2014 training set, as well as two widely-adopted metrics: zero-shot Frechet Inception Distance (FID) on MS-COCO, and the GenEval score (Ghosh et al., 2024). Inference hyper-parameters, such as temperature and classifier-fre free guidance, are optimized for each evalu- ation metric. FID is computed over 30K randomly selected image-text pairs from the MS-COCO 2014 training set, providing a metric that evaluates both the fidelity and diversity of generated im- ages. The GenEval benchmark, on the other hand, measures the model's ability generate images that accurately reflect the given prompt. For qualitative evaluation, we generate images from several prompts using each model and compare the visual quality of the generated images.
5.1 SCALING BEHAVIORS
In this section, we explore how two key design choices in autoregressive image generative models- token representation and generation order-afer affect performance and scaling behavior. We construct models with different combinations of these two design choices, resulting in four distinct variants of autoregressive image generation models. We also explore the generalizability of these models across different data and evaluation metrics. Our experiments reveal several intriguing properties.
Validation losses consistently scale with model size. In Figure 5, we examine the scaling behav- ior of the four autoregressive variants terms of validation loss. We observe a linear relationship between validation loss and model size in the log space, we increases model size from 150 mil- lion to billion parameters. This aligns with the power-law finding in Henighan et al. (2020). This demonstrates that the improvements in training loss resulting from increased model size generalize well to validation loss on data different from the training data.
Random-order models with continuous tokens scale the best in evaluation scores. In Figure 6, we analyze the scaling behavior of the four autoregressive variants in terms FID and GenEval overall scores. We find that the improvements observed in validation loss do not always translate linearly to better evaluation metrics, implying that there is no strict power-law relationship between these metrics and model size. For example, raster-order models with discrete tokens (blue line) reach a plateau in both FID and GenEval scores around IB parameters. Among the four variants, random-order models with continuous tokens (i.e., Fluid) show consistent improvements in evalua- tion metrics up to 3B parameters, achieving the best overall performance. Therefore, we focus on further investigating the scaling behavior of this model.

---

Published as a conference paper at ICLR 2025
Raster Order, Discrete
Random Order, Discrete
Raster Order, Continuous
Random Order, Continueus
6.8
6.65
0.255
0.306
Validation Loss
6.7
Validacion Less
0.250
Validation Loss
6.6
6.55
0.245
0.298
6.5
6.50
0.240
6.4
6.45
0.235
0.294
0.1
0.3
1.0
3.0
0.1
0.3
1.0
3.0
0.1
0.3
1.0
3.0
0.1
0.3
1.0
3.0
Parameters
(B)
Parameters
Parameters (B)
Figure 5: Validation loss scales as power-law with model size. The validation loss is evaluated on 30K images randomly sampled from the MS-COCO 2014 training set. The and axes are in log-scale. The change in is relatively small for each plot, making the log-scale alike linear-scale.
Raster Order, Discrete
Random Order, Discrete
Raster Order, Continuous
Random Order, Continueus
0.70
8.5
0.65
8.0
GenEval (overall)
0.60
RD
7.5
7.0
0.50
6.5
0.45
0.1
0.3
1.0
3.0
0.1
0.3
1.0
3.0
Parameters (B)
Parameters
Figure 6: Random-order models using continuous tokens (orange) achieve the best perfor- mance on evaluation metrics. FID (lower is better) evaluated on 30K images randomly sampled from the MS-COCE 2014 training set, while the GenEval overall score (higher is better) is sessed using the 553 prompts provided by the official benchmark, with four images generated for each prompt. Among all models, random-order models continuous tokens consistently show improvement in evaluation metrics as model size increases and achieve the best FID and GenEval scores.
Random-order models with continuous tokens scale with training computes. In Figure 7, plot validation loss, FID, and GenEval scores as functions of total training steps and compute for different model sizes of Fluid. We observe consistent improvements in both validation loss and evaluation performance with increased training steps and compute. However, the benefits from additional training steps saturate around 1M steps, indicating that training smaller models for more steps is less compute-efficien compared to training larger models for fewer steps. This behavior aligns with observations in language models, highlighting the potential for scaling up model sizes with sufficient training.
Strong correlation between validation loss and evaluation metrics. In Figure 8, we plot FID and GenEval scores against validation loss for different model sizes of Fluid and observe strong correlation. To quantify this, we the data points using linear regression. The Pearson correlation coefficients for FID and GenEval scores 0.917 and -0.931, respectively, indicating nearly linear relationship between validation loss and these evaluation metrics across model sizes ranging from 150M 3B Encouraged by this positive trend, we trained model with 10.5B parameters and batch size of 4096 for IM steps, achieving state-of-the text-to-ine text-to-image generation performance, as discussed in the next section.
Siince both FID and GenFval scores have lower/upper bounds, this linear relationship cannot hold indefi- nitely. Future work should explore the limits of this correlation.
7

---

Published as a conference paper at ICLR 2025
0.2B
0.4B
0.7B
1.18
3.1B
0.315
10
0.70-
0.65
Validation LOSS
0.310
GenEval (overall)
0.60-
0.305
FID
0.55
0.300
0.50
0.295
0.45
250
500
750
1000
250
500
750
1000
250
500
750
1000
Training Steps (K)
Training Steps (K)
Training Steps (K)
0.315
Validation Loss
0.310
GenEval (overall)
0.65
0.60
0.305
FID
0.55-
0.50
0.295
0.45-
1020
1021
1022
1021
1022
1021
1022
Training
Training FLOPs
Training FLOPs
Figure 7: Validation losses and evaluation performance scale with increasing training steps and computes. We use random-order models with continuous tokens. Results for other autoregressive variants included in the appendix. The training compute is computed model GFLOPs batch size x training xtepsx3, where the factor of 3 accounts for the backward pass being approximately twice compute-intensive the forward pass.
0.2B
0.4B
0.7B
1.1B
3.1B
10.0
0.75
9.5
0.70
9.0
8.5
0.917
GenEval (overali)
0.65
-0.931
0.60
FID
0.55
0.50
5.5
0.295
0.300
0.305
0.295
0.300
0.305
Validation
Validation
Figure 8: Validation loss and evaluation metrics are highly correlated. We use randon-order models with continuous tokens. The Pearson correlation coefficients for FID and GenEval scores 0.917 and -0.931, respectively. We also observe that the linear correlation slightly weakens and becomes less pronounced for the B. model.
Continuous tokens and large models are crucial for visual quality. In Figure 9, we compare the visual quality of images generated by the four autoregressive variants. The visual quality of models using discrete tokens is significantly worse than that of models using continuous tokens, e.g., the eyes of the corgi is asymmetric for discrete token based models and scaling up not solve this problem. This limitation is largely because the discrete tokenizer, which introduces substantial information loss. For instance, with 3B parameters, the discrete token models cannot generate
8

---

Published as a conference paper at ICLR 2025
Random Order, Continuous, (Fluid)
Raster Order. Continuous.
Random Order. Discrete.
Raster Order. Discrete.
3.18
An angry duck doing heavy weightlfting at the gym.
Mona Lisa winter
corgi.
panda with snow goggles bearding
street
group three teddy bears suit in office celebrating the birthday friend. pizza
photo of smiling snow goggles holding
Figure 9: Visual quality and image-text alignment improves with increasing model size. Best viewed zoomed-in. Fluid achieves the highest visual quality and best image-text alignment.
an accurate Mona Lisa due to poor reconstruction quality of the tokenizer (Figure 4). In contrast, models with continuous tokens produce much higher-quality images.
Additionally, larger models show consistent improvements both visual quality and image-text alignment. For example, random-order model with 0.2B parameters struggles to generate "an angry duck doing heavy weightlifting at the gym", while the same model with 3B parameters can generate the corresponding images successfully. This demonstrates that modeling continuous to- kens and increasing model size are crucial for achieving high visual quality in autoregressive image generation models.

---

Published as a conference paper at ICLR 2025
Table I: System-level comparison. Fluid achieves leading results on both MS-COCO zero-shot
FID-30K and GenEval benchmark (Ghosh et al., 2024). 1: CM3Lcon result is reported without retrieval.
MS-COCO FID-30K
GenEval Colors
#params
Single Obj.
Two Obj.
Counting
Position
Color Attri.
Overall
diffusion model
LDM
1.4B
12.64
0.92
0.23
0.70
0.02
0.05
0.37
D
4.2B
10.39
0.94
0.66
0.49
0.77
0.10
0.19
0.52
DALL-E3
0.96
0.87
0.47
0.83
0.43
0.45
0.67
Imagen
3B
7.27
SD3
8B
0.98
0.84
0.66
0.74
0.40
0.43
0.68
Transfusion
7.38
6.78
0.63
RAPHAEL
3B
6.61
untoregressive model
CM3Leon
7B
10.82
Show-o
1.3B
9.24
0.95
0.52
0.49
0.82
0.11
0.28
0.53
Muse
3B
7.88
Parti
20B
7.23
Fluid (our work)
7.23
0.96
0.64
0.53
0.78
0.33
0.46
0.62
0.73
0.42
0.51
0.65
1.IB
6.59
0.96
0.77
0.61
0.34
0.53
0.67
3.IB
6.41
0.98
0.83
0.60
0.82
0.41
0.53
0.70
10.5B
6.16
0.96
0.83
0.63
0.80
0.39
0.51
0.69
5.2 BENCHMARKING WITH PREVIOUS SYSTEMS
In this section, we compare our Fluid, i.e., continuous random-order autoregressive model, with leading text-to-image generation systems in Table (Rombach al., 2022b; Ramesh et al., 2022; Betker et al., 2023; Saharia et al., 2022; Esser et al., 2024; Zhou et al., 2024; Xue et al., 2024; Yu al., 2023; Xie et al., 2024; Chang al., 2023; Yu al., 2022). The Fluid smallest model, with sero-shot FID of 7.23 on MS-COCO and GenEval overall score of
369M parameters, achieves 0.62, matching the performance of many state-of-the-an models with several billion parameters (e.g., Parti with 20B parameters only achieves 7.23). The Fluid largest model, with 10.5B parameters, further improves the zero-shot FID on MS-COCO to 6.16 and increases the GenEval overall score to 0.692, with speed of 1.571 seconds per image per TPU (evaluated 32 TPU v5 with batch size of 2048). Detailed model configurations and generation speed results are included in the appendix. We hope these strong results and promising scaling behavior provide valuable insights and support for the scalability of autoregressive models in visual generative modeling.
6 DISCUSSION
In this paper, present empirical study on the scaling behavior of autoregressive models for text-to-image generation. We investigate two critical design factors: random order versus raster order, and discrete tokens versus continuous tokens. Our results show that random-order models with continuous tokens achieve the best performance and scaling behavior across various evaluation metrics and in terms of visual quality. Building these findings, we scale up the random-order model with continuous tokens, namely Fluid, to 10.5B parameters, and achieves state-of-the-an text- to-image generation performance. We hope that findings and promising results could provide valuable insights into the scaling behavior of autoregressive models for image generation and help bridge the gap between the scaling performance of vision models and language models.
Reproducibility Statement. To aid reproducibility, we have provided the implementation details of our framework in Section 4, training hyper- parametrrssstrss in Section 5, and model configurations in the appendix. For the diffusion loss used for continuous tokens, we have strictly followed the open-sourced code of et al. (2024).
Acknowledgements. We would like to express our gratitude to Amy Shen and Alex Rizkowsky for their assistance in securing computational resources, and Yash Katariya, Ivy Zheng, and Roy Frostig for their valuable insights on JAX-related questions. We also thank Matan Cohen for his sup- port with precomputed T5 embeddings. Our appreciation extends to Han Zhang, Jason Baldridge, Dilip Krishnan, David Salesin and VisCam team for their helpful discussions and valuable feed- back.
We ohserve that the GenEval scores plateau for the 10.5B Fluid compared to the 3.1B Fluid; however, it continues show consistent improvements in visual quality and FID.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 230707330 2023.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan I. Yuille, Trevor Darrell, Jitendra Malik, and Alexei Efros. Sequential modeling enables scalable learning for large vision models. In CVPR, 2024.
Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuging Du, Zach Eaton-Rosen. et al. Imagen 3. arXiv preprint arXiv:2408.07009, 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et Improving image generation with better captions. OpenAl, 2023.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303.1271,, 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. MaskGIT: Masked genera- tive image Transformer. In CVPR, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative Transformers. In ICML, 2023.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In /CML, 2020.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Picter Abbeel. PixcISNAIL: An improved au- toregressive generative model. In ICML. 2018.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sehastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0677 2022.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos. Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /810.04805, 2018.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.217888 2024.
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large au- toregressive image models. arXiv preprint arXiv:2401.08544 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for high-resolution image synthesis. In CVPR, 2021.
11

---

Published as a conference paper at ICLR 2025
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In NeurlPS, 2024.
Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autoregres- sive networks. In ICML, 2014.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.147000 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jonathan HO, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arX0v:2210.02303, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15555 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Kudo. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint urXiv:/808.066 2018
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. MAGE: Masked generative encoder to unify representation learning and image synthesis. In CVPR, 2023.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
OpenAl. Sora: Creating video from text. https: /tt:://pppnaaa com/ sora, 2024.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image Transformer. In ICML. 2018.
William Peebles and Saining Xie. Scalable diffusion models with Transformers. In ICCV, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. Technical Report, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
12

---

Published as a conference paper at ICLR 2025
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to-image diffusion models with deep language understanding. NewrIPS, 2022.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deep- speed and megatron to train ergattonnturing nlg 530b, a large-scale generative language model. arXiv preprini arXiv:2201.1 19900 2022.
Mingxing Tan. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: /996, 2019.
Gemini Team, Rohan Anil, Sebastian Borgcaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sorieut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint wrXiv:2404.029005 2024.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with PixelCNN decoders. In NeurIPS, 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016b.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. arXiv preprint arXiv:1711.00937 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0766 2022.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou, Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint a4Xiv:2408. /2528, 2024.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. NeurIPS, 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arX1v:2110.04627, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint ar Xrxi::2006. 10789, 2022.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint urXiv:2309.0259/ 2023.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
13

---

Published as a conference paper at ICLR 2025
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 2024.
14


Reviews:
[{"review_id": "CAcnKn54jg", "reviewer": "Reviewer_UT5L", "summary": "The paper explores scaling autoregressive models for text-to-image generation, focusing on two key factors: discrete vs. continuous token representations, and token generation order (raster vs. random). The results show that using continuous tokens and generating in random order lead to better image quality, especially at scale. The new Fluid model, which adopts these strategies, achieves state-of-the-art results on benchmarks like MS-COCO and GenEval.", "questions": "1.\tIs there any specific differences compared to MAR paper?\n2.\tAny thoughts on why random order generation works better than raster? Why continuous better than discrete? Would love to see more reasoning beyond the empirical results.\n3.\tHow does the training time compare across setups, especially when scaling up?\n4.\tIn the inference time, how does the inference time step influence the model performance?\n5.\tIs the method scales well for image resolution? For example, finetune the model on a small, but large-resolution dataset. Current diffusion model often works well for much larger resolution like 512 and 1024. The 256^2 image generated in the paper is somehow limited for real application. \n6.\tA very important question for me is not confirming the scaling law of AR-based model for image generation(always, larger compute give better results). Rather, we are interested in comparing the diffusion model and the AR in term of the speed of scaling. This paper somehow demonstrates Fluid scales better than causal, discrete-token AR model. If it is possible, a comparing with pure diffusion model is very important.\n7.\tAre there any failure cases for the model? \n8.\tNot exactly a research problem, but will the code be open-sourced? Because this paper is mostly an empirical study, sharing the code and model would be the most important contribution than reading the paper. I personally think this is a key consideration when judging this paper.\nOne minor suggestion: While using a random scan order prevents the typical use of KV caching for causal sequences, an alternative could be to adopt a structured order, such as a grid-like pattern. This way, you could still cache info from nearby patches and keep some benefits of caching while generating the current patch.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Solid Scaling Analysis: The paper does a thorough job of laying out the scaling trends. The consistent improvements with random order and continuous tokens give a clear picture of what’s working.\n\nConcrete study on Toke type and Scan order: The results convincingly show that generating tokens in a random order outperforms the traditional raster order, especially for tasks needing a good global structure. Additionally, switching to continuous tokens noticeably enhances image quality.\n\nThis is a solid empirical paper that offers important insights into scaling autoregressive models, mostly on scaling the MAR model (Li et al., 2024). The work holds relevance for both academia and industry, with the potential for meaningful social impact.", "weaknesses": "I value the contribution of the paper. While I am generally positive, we can still find some imperfect points in the paper.\n\nMethodology Largely Follows MAR (Li et al., 2024): The approach in this paper closely follows the method used in MAR, with some modifications to incorporate text condition. While these changes are valuable, the paper doesn’t go far beyond what MAR already established. Also, the discussion of scan order and token type is also a repeat of what has been done in the MAR paper.\n\nOver-Reliance on Empirical Results: While the paper does a great job presenting empirical evidence, there’s a lack of theoretical explanation for why continuous tokens outperform discrete ones. The benefits are demonstrated, but there isn't much discussion on how continuous representations help in practice—whether it’s due to better gradient flow, more expressive capacity, or something else entirely. More in-depth analysis could add substance to the findings.\n\nScaling Analysis Could Be More Comprehensive: The paper primarily showcases large model almost for sure improve the performance. However, other important factors in scaling, such as the effect of dataset size, are not thoroughly explored. For instance, language model research, like Kaplan et al. (2020), investigates how increasing dataset size influences validation loss, providing insights into the relationship between data and model scaling. In contrast, all the experiments in this paper are conducted on the same subset of the WebLI dataset, leaving the impact of different data sizes unexamined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0aL0rcpIiE", "reviewer": "Reviewer_y1Ue", "summary": "This paper investigates the scaling behaviors of autoregressive models in text-to-image generation, focusing on key factors: token type (discrete vs. continuous) and generation order (random vs. raster). Experiment results demonstrate a power-law relationship between the validation loss and model sizes, and that continuous tokens with random-order generation yield the best performance on evaluation metrics. Leveraging these findings, the authors introduce Fluid, a 10.5B random-order autoregressive model with continuous tokens, outperforming previous models on benchmarks such as FID and GenEval.", "questions": "- My questions arise naturally from the \"Weaknesses\", I believe the authors may have more insights beyond the experiments results provided.\n  - What underlying factors drive the observed correlation between metrics and the linear relationship between validation loss and the metrics?\n  - What should be considered the most \"human-aligned\" evaluation criterion for future research?\n  - More analysis of the benefits of the continuous tokens and random-order prediction should be explained, especially during the scaling up, e.g., why does random-order prediction yield a larger loss but better visual quality?\n- Similar to the \"Scaling Law\" curves in the language model community, an ablation on the number of training examples across the four configurations could be beneficial.\n- Is there any \"Emergence\" phenomena during training, such as a sudden increase in metric scores?\n- I would consider raising my scores if these points are addressed.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Achieve SOTA performance with zero-shot FID of 6.16 on MS-COCO 30K, and an overall score of 0.69 on the GenEval benchmark.\n- The paper is well-organized and easy to follow, with robust training data, strategies, and solid experimental setups.\n- Extensive experiments reveal clear relations between the four configurations of token types and generation order.", "weaknesses": "- This paper ablated two types of existing technologies, with their scaling behaviors. The experiments are well-designed, and the authors' observations appear reasonable. However, some additional insights are needed.\n- I appreciate the authors' efforts in sharing extensive experimental results, but providing an actionable conclusion would add more value than observations alone. If the authors could provide insights or equations that can guide future training of large-scale text-to-image models—such as identifying optimal settings when constrained by FLOPs, image samples, or model size—it would significantly benefit the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPtVisSv0q", "reviewer": "Reviewer_QGyS", "summary": "This work investigates the performance of autoregressive models in generating high-quality images from text prompts, focusing on the use of continuous versus discrete tokens and the impact of token generation order. The authors find that models using continuous tokens produce superior visual quality compared to those with discrete tokens, indicating that continuous representations capture more nuanced information. Additionally, they reveal that random-order generation outperforms fixed raster-order generation, highlighting the significance of the token generation sequence on image quality. Employing various evaluation metrics, the study compares its results with leading diffusion models and autoregressive models, reporting favorable outcomes. Overall, the research provides empirical insights into the scaling behavior of text-to-image generation models, emphasizing the importance of token representation and generation strategy in enhancing autoregressive models' capabilities for high-resolution image synthesis.", "questions": "1. In the experimental results presented in Figure 6, it seems that with the minimal number of parameters, continuous tokens do not outperform discrete tokens. Would the authors be able to provide further analysis on why this might be the case?\n\n2. In the implementation of continuous tokens, an extra diffusion head is used, whereas no comparable diffusion structure is applied for discrete tokens. If possible, could the authors conduct further ablation studies to examine the impact of this diffusion head?\n\n3. The authors also mention that these techniques vary not only in performance but in efficiency as well. Would it be possible to provide a more detailed comparison of inference efficiency?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing in the article is clear and easy to understand. \n\n2. The improvements to the original autoregressive model, such as continuous token and random generation order, are both insightful and empirically effective techniques. These empirical results provide crucial experimental evidence that supports some of the fundamental beliefs about autoregressive image generation.", "weaknesses": "1. Although this article is the first to compare the roles of token representation and generation order together in the autoregressive image generation, the techniques it examines are not novel or originally proposed by the authors. For instance, line 189 notes that the continuous token method is derived from a prior work, and line 287 mentions that the implementation of random order also originates from an earlier study. This work does not optimize or update the technical details of these methods. I highly appreciate the article’s empirical study on autoregressive image generation, but the scope seems limited to experimental evaluation of existing techniques without introducing new methods or innovations.\n\n2. Autoregressive image generation still lacks a standardized technical framework, and both its training and inference processes remain complex and non-standardized. The approach involves various technical details beyond just token representation and generation order. I highly appreciate the authors' efforts to highlight these two factors amidst the complexity. However, different token representations and generation orders introduce other technical variations, making it challenging to ensure absolute fairness in comparisons. This limitation may prevent us from making strictly definitive claims based on the comparative results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7OlEmwGuE3", "reviewer": "Reviewer_hShT", "summary": "This paper proposes an auto-regressive text-to-image generation model. The authors specifically explore two aspects, namely random order v.s. raster order and continuous token v.s. discrete token. Through experiments, the authors conclude that random order with continuous tokens is better. Evaluation on MS-COCO and GenEval benchmarks indicate better performance compared with previous diffusion-based and auto-regressive-based models.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. This paper successfully scales previous auto-regressive models based on continuous tokens up to the text-to-image setting, which is more complicated and hard-to-train than class-conditioned models.\n2. The quantitative results are good given the results in Tab. 1.\n3. The paper is well-written and I am able to fully follow the authors' writing.", "weaknesses": "1. It seems that the novelty is limited. The applied techniques are essentially the same as those proposed in [a], which can hardly bring readers new technical insights. The overall feeling of the paper is more like a technical report instead of a research paper.\n2. The resolution used in the paper is 256x256, which is relatively small compared with recent models.\n3. Also, I am not sure whether the comparison in Tab. 1 is fair because different systems may have been trained on different resolutions.\n4. LlamaGen [b] can serve as a recently proposed baseline for comparisons.\n5. Lack of qualitative comparisons and efficiency comparisons with recent models. Readers can hardly get the strengths of the proposed model against state-of-the-art ones.\n\n\n[a] Autoregressive Image Generation without Vector Quantization, Li et al., 2024\n\n[b] Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation, Sun et al., 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)