Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

Authors: Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, Yonglong Tian
Primary area: generative models
Keywords: generative model, scaling, text-to-image, autoregressive

Abstract:
Scaling up autoregressive models in vision has not proven as beneficial as in large language models.
In this work, we investigate this scaling problem in the context of text-to-image generation, focusing on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed raster order using BERT- or GPT-like transformer architectures. Our empirical results show that, while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends. Models based on continuous tokens achieves significantly better visual quality than those using discrete tokens. Furthermore, the generation order and attention mechanisms significantly affect the GenEval score: random-order models achieve notably better GenEval scores compared to raster-order models.
Inspired by these findings, we train Fluid, a random-order autoregressive model on continuous tokens. Fluid 10.5B model achieves a new state-of-the-art zeor-shot FID of 6.16 on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our findings and results will encourage future efforts to further bridge the scaling gap between vision and language models.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729599108891

thread_type:
official_review

<thread_object>
creator: reviewer_hsht

<content>
summary:
This paper proposes an auto-regressive text-to-image generation model. The authors specifically explore two aspects, namely random order v.s. raster order and continuous token v.s. discrete token. Through experiments, the authors conclude that random order with continuous tokens is better. Evaluation on MS-COCO and GenEval benchmarks indicate better performance compared with previous diffusion-based and auto-regressive-based models.

strengths:
1. This paper successfully scales previous auto-regressive models based on continuous tokens up to the text-to-image setting, which is more complicated and hard-to-train than class-conditioned models.
2. The quantitative results are good given the results in Tab. 1.
3. The paper is well-written and I am able to fully follow the authors' writing.

weaknesses:
1. It seems that the novelty is limited. The applied techniques are essentially the same as those proposed in [a], which can hardly bring readers new technical insights. The overall feeling of the paper is more like a technical report instead of a research paper.
2. The resolution used in the paper is 256x256, which is relatively small compared with recent models.
3. Also, I am not sure whether the comparison in Tab. 1 is fair because different systems may have been trained on different resolutions.
4. LlamaGen [b] can serve as a recently proposed baseline for comparisons.
5. Lack of qualitative comparisons and efficiency comparisons with recent models. Readers can hardly get the strengths of the proposed model against state-of-the-art ones.


[a] Autoregressive Image Generation without Vector Quantization, Li et al., 2024

[b] Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation, Sun et al., 2024

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730104380309

thread_type:
official_review

<thread_object>
creator: reviewer_qgys

<content>
summary:
This work investigates the performance of autoregressive models in generating high-quality images from text prompts, focusing on the use of continuous versus discrete tokens and the impact of token generation order. The authors find that models using continuous tokens produce superior visual quality compared to those with discrete tokens, indicating that continuous representations capture more nuanced information. Additionally, they reveal that random-order generation outperforms fixed raster-order generation, highlighting the significance of the token generation sequence on image quality. Employing various evaluation metrics, the study compares its results with leading diffusion models and autoregressive models, reporting favorable outcomes. Overall, the research provides empirical insights into the scaling behavior of text-to-image generation models, emphasizing the importance of token representation and generation strategy in enhancing autoregressive models' capabilities for high-resolution image synthesis.

strengths:
1. The writing in the article is clear and easy to understand. 

2. The improvements to the original autoregressive model, such as continuous token and random generation order, are both insightful and empirically effective techniques. These empirical results provide crucial experimental evidence that supports some of the fundamental beliefs about autoregressive image generation.

weaknesses:
1. Although this article is the first to compare the roles of token representation and generation order together in the autoregressive image generation, the techniques it examines are not novel or originally proposed by the authors. For instance, line 189 notes that the continuous token method is derived from a prior work, and line 287 mentions that the implementation of random order also originates from an earlier study. This work does not optimize or update the technical details of these methods. I highly appreciate the article’s empirical study on autoregressive image generation, but the scope seems limited to experimental evaluation of existing techniques without introducing new methods or innovations.

2. Autoregressive image generation still lacks a standardized technical framework, and both its training and inference processes remain complex and non-standardized. The approach involves various technical details beyond just token representation and generation order. I highly appreciate the authors' efforts to highlight these two factors amidst the complexity. However, different token representations and generation orders introduce other technical variations, making it challenging to ensure absolute fairness in comparisons. This limitation may prevent us from making strictly definitive claims based on the comparative results.

questions:
1. In the experimental results presented in Figure 6, it seems that with the minimal number of parameters, continuous tokens do not outperform discrete tokens. Would the authors be able to provide further analysis on why this might be the case?

2. In the implementation of continuous tokens, an extra diffusion head is used, whereas no comparable diffusion structure is applied for discrete tokens. If possible, could the authors conduct further ablation studies to examine the impact of this diffusion head?

3. The authors also mention that these techniques vary not only in performance but in efficiency as well. Would it be possible to provide a more detailed comparison of inference efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675320726

thread_type:
official_review

<thread_object>
creator: reviewer_y1ue

<content>
summary:
This paper investigates the scaling behaviors of autoregressive models in text-to-image generation, focusing on key factors: token type (discrete vs. continuous) and generation order (random vs. raster). Experiment results demonstrate a power-law relationship between the validation loss and model sizes, and that continuous tokens with random-order generation yield the best performance on evaluation metrics. Leveraging these findings, the authors introduce Fluid, a 10.5B random-order autoregressive model with continuous tokens, outperforming previous models on benchmarks such as FID and GenEval.

strengths:
- Achieve SOTA performance with zero-shot FID of 6.16 on MS-COCO 30K, and an overall score of 0.69 on the GenEval benchmark.
- The paper is well-organized and easy to follow, with robust training data, strategies, and solid experimental setups.
- Extensive experiments reveal clear relations between the four configurations of token types and generation order.

weaknesses:
- This paper ablated two types of existing technologies, with their scaling behaviors. The experiments are well-designed, and the authors' observations appear reasonable. However, some additional insights are needed.
- I appreciate the authors' efforts in sharing extensive experimental results, but providing an actionable conclusion would add more value than observations alone. If the authors could provide insights or equations that can guide future training of large-scale text-to-image models—such as identifying optimal settings when constrained by FLOPs, image samples, or model size—it would significantly benefit the field.

questions:
- My questions arise naturally from the "Weaknesses", I believe the authors may have more insights beyond the experiments results provided.
  - What underlying factors drive the observed correlation between metrics and the linear relationship between validation loss and the metrics?
  - What should be considered the most "human-aligned" evaluation criterion for future research?
  - More analysis of the benefits of the continuous tokens and random-order prediction should be explained, especially during the scaling up, e.g., why does random-order prediction yield a larger loss but better visual quality?
- Similar to the "Scaling Law" curves in the language model community, an ablation on the number of training examples across the four configurations could be beneficial.
- Is there any "Emergence" phenomena during training, such as a sudden increase in metric scores?
- I would consider raising my scores if these points are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714896325

thread_type:
official_review

<thread_object>
creator: reviewer_ut5l

<content>
summary:
The paper explores scaling autoregressive models for text-to-image generation, focusing on two key factors: discrete vs. continuous token representations, and token generation order (raster vs. random). The results show that using continuous tokens and generating in random order lead to better image quality, especially at scale. The new Fluid model, which adopts these strategies, achieves state-of-the-art results on benchmarks like MS-COCO and GenEval.

strengths:
Solid Scaling Analysis: The paper does a thorough job of laying out the scaling trends. The consistent improvements with random order and continuous tokens give a clear picture of what’s working.

Concrete study on Toke type and Scan order: The results convincingly show that generating tokens in a random order outperforms the traditional raster order, especially for tasks needing a good global structure. Additionally, switching to continuous tokens noticeably enhances image quality.

This is a solid empirical paper that offers important insights into scaling autoregressive models, mostly on scaling the MAR model (Li et al., 2024). The work holds relevance for both academia and industry, with the potential for meaningful social impact.

weaknesses:
I value the contribution of the paper. While I am generally positive, we can still find some imperfect points in the paper.

Methodology Largely Follows MAR (Li et al., 2024): The approach in this paper closely follows the method used in MAR, with some modifications to incorporate text condition. While these changes are valuable, the paper doesn’t go far beyond what MAR already established. Also, the discussion of scan order and token type is also a repeat of what has been done in the MAR paper.

Over-Reliance on Empirical Results: While the paper does a great job presenting empirical evidence, there’s a lack of theoretical explanation for why continuous tokens outperform discrete ones. The benefits are demonstrated, but there isn't much discussion on how continuous representations help in practice—whether it’s due to better gradient flow, more expressive capacity, or something else entirely. More in-depth analysis could add substance to the findings.

Scaling Analysis Could Be More Comprehensive: The paper primarily showcases large model almost for sure improve the performance. However, other important factors in scaling, such as the effect of dataset size, are not thoroughly explored. For instance, language model research, like Kaplan et al. (2020), investigates how increasing dataset size influences validation loss, providing insights into the relationship between data and model scaling. In contrast, all the experiments in this paper are conducted on the same subset of the WebLI dataset, leaving the impact of different data sizes unexamined.

questions:
1.	Is there any specific differences compared to MAR paper?
2.	Any thoughts on why random order generation works better than raster? Why continuous better than discrete? Would love to see more reasoning beyond the empirical results.
3.	How does the training time compare across setups, especially when scaling up?
4.	In the inference time, how does the inference time step influence the model performance?
5.	Is the method scales well for image resolution? For example, finetune the model on a small, but large-resolution dataset. Current diffusion model often works well for much larger resolution like 512 and 1024. The 256^2 image generated in the paper is somehow limited for real application. 
6.	A very important question for me is not confirming the scaling law of AR-based model for image generation(always, larger compute give better results). Rather, we are interested in comparing the diffusion model and the AR in term of the speed of scaling. This paper somehow demonstrates Fluid scales better than causal, discrete-token AR model. If it is possible, a comparing with pure diffusion model is very important.
7.	Are there any failure cases for the model? 
8.	Not exactly a research problem, but will the code be open-sourced? Because this paper is mostly an empirical study, sharing the code and model would be the most important contribution than reading the paper. I personally think this is a key consideration when judging this paper.
One minor suggestion: While using a random scan order prevents the typical use of KV caching for causal sequences, an alternative could be to adopt a structured order, such as a grid-like pattern. This way, you could still cache info from nearby patches and keep some benefits of caching while generating the current patch.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732055943396

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response: Contributions and New Experiments

comment:
We sincerely thank all the reviewers for their insightful comments and constructive suggestions to strengthen our work. In addition to responding to specific reviewers, in this response, we would like to highlight our empirical novelty and contributions to the community, as well as the new experiments we have included in the rebuttal.

***Contributions***

Our paper does not aim to propose a novel technique but instead aims to address a critical and long-standing open question: whether autoregressive generative models are scalable in vision and how to achieve this scalability. Before our work, it was common wisdom that diffusion models are the de facto approach for visual generation. We challenge this notion and empirically demonstrate that autoregressive models are equally promising and can scale effectively. To the best of our knowledge, no prior work has systematically demonstrated this finding. Our extensive empirical results provide key design guidance and strong evidence to support autoregressive models in visual generation, opening up new opportunities for future research in this area. We list our key contributions and distinctions from prior works as follows:

1.  Empirical Validation of Scalability:  
    Our primary contribution lies in empirically validating the scalability of autoregressive visual generative models under various design choices. Unlike prior works, which have lacked a systematic exploration of scaling properties, we provide a comprehensive study akin to the progression from GPT-1 and GPT-2 to GPT-3.
    
2.  Extensive Analysis of Key Design Choices:  
   Previous works, such as MAR (Li et al., 2024), have explored scan order and token type comparisons only on small-scale models (e.g., 0.4 billion parameters) and on ImageNet datasets. In contrast, we systematically study the scaling behavior across these design choices—including discrete vs. continuous tokens and random vs. raster order—at significantly larger scales (up to 10.5 billion parameters). Our results reveal previously unknown insights: (1) all four variants of autoregressive models are scalable; (2) continuous tokens with random-order generation scale the best. We also provide extensive visualizations and analyses to thoroughly examine the pros and cons of different configurations. These insights offer valuable guidance for designing scalable autoregressive vision models in the future.
    
3.  Adapting Continuous Tokens for Text-to-Image Generation:  
    We enhance the diffusion head architecture by incorporating cross-attention modules and text embedding layers to handle text-to-image generation tasks. This modification distinguishes our framework from MAR, which does not address text-conditioning in its design.
    
In a nutshell, our paper proposes the first autoregressive text-to-image generation system that scales up to 10.5 billion parameters. Our extensive experiments on two key design choices reveal that random order and continuous tokens play important roles in autoregressive visual generation. These insights from our paper are not only beneficial to text-to-image generation but also broadly impact the computer vision community. Our findings on scalability and scan order could shed light on the development of large vision-language multimodal models, where the autoregressive framework might be a more natural fit than diffusion models. Additionally, our findings on continuous tokens could benefit other visual generative tasks, such as video generation or 3D generation, where discrete tokenization is even more challenging and leads to greater information loss.

***Additional Experiments***

In this rebuttal, we have added more supporting experiments following reviewers’ suggestions. Here we summarize the new experiments that we added below:
-   Scaling with dataset size [UT5L, y1Ue]
-   Performance on higher resolution (512x512) [UT5L, hShT]
-   Comparison with LlamaGen [hShT]
-   Ablation studies on inference steps [UT5L]
-   Comparison of Inference time [QGyS, hShT]

We hope these new experiments can further complement and strengthen the empirical novelty and contributions of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates scaling behavior in autoregressive text-to-image models, focusing on token representation (discrete vs continuous) and generation order (random vs raster). The key findings show that continuous tokens with random-order generation scale most effectively, enabling state-of-the-art results with their 10.5B Fluid model. The paper demonstrates strong empirical results and thorough analysis, with reviews highlighting clear presentation and comprehensive experiments. While initial concerns were raised about novelty and resolution limitations, the authors provided strong responses during rebuttal: They demonstrated their model's effectiveness at 512x512 resolution achieving comparable FID and GenEval scores to 256x256, conducted new experiments showing benefits over recent work like LlamaGen, and clarified their key contribution of being first to systematically study scaling of autoregressive vision models at large scale (up to 10.5B parameters). The review scores (initially 5, 6, 6, 6) improved during discussion as reviewers were convinced by the additional experiments and clarifications. Though some concerns remain about theoretical justification, the thorough empirical validation and clear practical importance of the findings support accepting this paper, which offers valuable guidance for future large-scale autoregressive vision models. Thus I vote to accept this paper.