Scaling Laws for Diffusion Transformers

Authors: Zhengyang Liang, Hao He, Ceyuan Yang, Bo Dai
Primary area: generative models
Keywords: Scaling Laws, Diffusion Models, Transformers, Generative Models

Abstract:
Diffusion transformers (DiT) have already achieved appealing synthesis and scaling properties in content recreation, \emph{e.g.,} image and video generation. 

However, scaling laws of DiT are less explored, which usually offer precise predictions regarding optimal model size and data requirements given a specific compute budget.

Therefore, experiments across a broad range of compute budgets, from \texttt{1e17} to \texttt{6e18} FLOPs are conducted to confirm the existence of scaling laws in DiT \emph{for the first time}. Concretely, the loss of pretraining DiT also follows a power-law relationship with the involved compute.

Based on the scaling law, we can not only determine the optimal model size and required data but also accurately predict the text-to-image generation loss given a model with 1B parameters and a compute budget of \texttt{1e21} FLOPs.

Additionally, we also demonstrate that the trend of pretraining loss matches the generation performances (\emph{e.g.,} FID), even across various datasets, which complements the mapping from compute to synthesis quality and thus provides a predictable benchmark that assesses model performance and data quality at a reduced cost.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730032457648

thread_type:
official_review

<thread_object>
creator: reviewer_cvmz

<content>
summary:
This work tries to reveal the scaling law of Diffusion Transformer in the field of text-to-image generation. It is based on in-context dit and rectified flow matching, and performs a series of experiments between 1e17 and 6e18 flops. By this, it finds the power law relationship between the objective function, model parameters and compute. It also uses the loss of a 1B dit at the compute of 1e21 flops to confirm the correctness of the equation. At last, it conducts a comparative test between in-context and cross-attention dit, to show that the scaling law can help to indicate the effectiveness of the model architecture's design.

strengths:
1. This work first proposes the scaling law of Diffusion Transformer in text-to-image generation. And it gets several formulated equations between loss, parameters and compute, which is confirmed by an additional experiment.
2.  It shows that different metrics (loss, VLB, likelihood) has similar trends with regard to compute budget. And the performance of generation measured by FID does, too.
3.  It indicates the application of the scaling law: serving as a benchmark for measuring the effectiveness of model architectures.

weaknesses:
1. The range of compute budget for experiments is a bit narrow. The count of data points is a little few for drawing a curve. According to "Scaling Laws for Autoregressive Generative Modeling"[1], they did experiments from 1e14 to 1e23 flops.
2. This work's experiment is not as sufficient as the scaling law for AR[1]. Maybe it needs some curves like [1].
3. The base setting of model architecture is not good. Recently, popular settings of in-context dit mainly refer to SD3[2] and FLUX, which only concat text and image as input but use timesteps for adaptive layernorm. This setting is important and may lead to suboptimal conclusion. 
4. It said that "scaling laws can serve as a reliable and predictable benchmark for evaluating the effectiveness of both model architectures and datasets". But it misses experiments to show how to use scaling law to choose training datasets.
5. Did not indicate the resolution of training datasets. Different resolutions may affect the conclusion.

[1] Scaling Laws for Autoregressive Generative Modeling
[2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

questions:
1. Some images in the paper only have one or two curves. Does this imply that it is already experimenting with the optimal model parameters calculated by the scaling law?
2. Why don't you use bigger models for base experiments? Common diffusion models, such as SD3 and FLUX, has 10B parameters or so. Only doing experiments with models less than 1B may is not general enough.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518840019

thread_type:
official_review

<thread_object>
creator: reviewer_cvdq

<content>
summary:
This work studies the scaling laws of diffusion transformer for text-to-image generation. It tries to unveil the relationship between the training budget and the optimal model size/training data quantity. Under careful designed experimental settings, the disclosed scaling laws served as a predictor for the model size/performance under given a specific training budget.

strengths:
This is the first study of scaling laws in DiT based text-to-image generation models. It disclosed some relationships between training budget and model size/training data quantity under a specific setting. The results might be useful for the community when designing future DiT based text-to-image generation.

weaknesses:
Experiment results are somewhat limited. All experiments were done with a very small dataset and small models (<= 1B parameters). These settings are quite different from many DiT based text-to-image models in terms of data quantity and model size. Whether the conclusion will hold for different setting is unknown to us.

questions:
please refer to the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537288137

thread_type:
official_review

<thread_object>
creator: reviewer_ssc5

<content>
summary:
This research investigates the scaling characteristics of diffusion transformers, focusing on a range from ($1 \times 10^{17}$) to ($6 \times 10^{18}$) Flops. The study enhances the understanding of how computational effort maps to synthesis quality.

strengths:
1. Good Presentation: The figures clearly present the properties of the scaling model and the associated FLOPs, making the analysis easy to follow and understand.

2. Predictable Benchmarking: The introduction of a predictable benchmark for assessing model performance and data quality significantly reduces costs and aids in optimizing computational resources.

weaknesses:
1. The largest model in the study only has 1 billion parameters. While I agree with the authors' statement that "scaling laws help us make informed decisions about resource allocation to maximize computational efficiency," a model size of 1 billion parameters is too small to verify the significance of this scaling law. We need guidance for larger model sizes, especially considering that current models, such as the Flux model, already have 8 billion parameters.

2. The discussion of reference work is not accurate. Line 097 makes claims about the masked diffusion transformers but fails to mention the first Masked Diffusion Transformer (MDT), both versions v1 and v2.

questions:
Are there any properties by scaling DiT across different dimensions (like channels, layers, etc)? The paper should explore how scaling DiT in various dimensions affects its properties and performance. This could provide valuable insights into the scaling behavior and help in designing more efficient models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730594301744

thread_type:
official_review

<thread_object>
creator: reviewer_2p8z

<content>
summary:
This paper explore the scaling laws of DiT (for text-to-image generation). The authors experiment with different computation budget and find the optimal allocation (i.e., the lowest training loss) of model parameters and training data amount for each budget. Subsequently, they plot two relationship curves: 1) parameter-FLOPs and 2) token (seen in training)-FLOPs. The conclusion is the training loss follows a power-law relationship with the computation budget. Based on this, the author claim that they can predict the best configurations of model parameter and training data for a given (larger) budget.

strengths:
1. While some similar reports have been seen in LLMs, to my best knowledge this is the first analysis for text-to-image DiTs. The problem this paper aims to address is also a hot topic troubling the community. A good question is proposed in this paper.
2. If the conclusion is correct, then given the model size and training data, it is possible to predict in which training iteration the model will achieve the best performance. This would alleviate a lot of resource consumption in the heavy task of DiT training.
3. The experimental methods are well designed.

weaknesses:
1. I don't know why the authors use FLOPs and token numbers as the indicators. This makes it difficult for me to intuitively estimate the computation budgets and data. In practice, GPU hours and image amount are more common and intuitive. 
2. I'm concerned about the power-law relationship shown in Fig 1 and 3, though the linear is so clear. How can we determine that this is not by coincidence (due to the structure of the model as well as the training data)? Even if this law is objectively universal, At what model size will it fail? For DiT, 1B is still a small scale. Can I really estimate 5B/10B if I only evaluate two points with small model sizes and get the regression line?
3. The training loss is not equal to FID, and more different from general performance. Of course the model does not perform well at large loss values, but it is agnostic at small values. In addition, when the budget is fixed, the number of iterations for the large model decreases. What happened if a larger learning rate is used in this case? The current setting does not seem fair.
4. The authors should improve the writing for easy understanding. For example, I can guess that the tokens in Fig. 1 refer to the number of tokens DiT has processed during the training, it still confuses me about the figure (even though the concept is explained in Sec 3.2)

questions:
Please also check the concerns in Weaknesses.
1. As mentioned in W.1, why not use GPU hours and text-image pairs?
2. How can we use it in practice? Can we only estimate two points at small scale and predict the configuration at large scale? I have no confidence in it.
3. Can this conclusion be validated based on the present DiT works? They usually have the model versions with different parameters.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732688940630

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to common concerns

comment:
We thank all reviewers for providing constructive advice. There are some common concerns raised by multiple reviewers. We respond to the common concerns here. 

Q1: **Model size**

A1: 
Regarding the model size, our experiments focus on models around 1B parameters. This aligns with the majority of open-sourced diffusion models, which are typically under 10B parameters, placing our model within the same order of magnitude. The primary goal of our paper is not to train a SOTA-sized diffusion model but rather to demonstrate the existence of scaling laws for diffusion models and provide a guideline on how to compute and utilize the scaling laws.
To address the reviewer’s concerns, we suggest drawing a parallel to the development of scaling laws in the LLM domain. Scaling laws are a developing field, and the exploration of their existence and limitations has historically been incremental. For example, the initial verification of scaling laws for LLMs was conducted with models at the 600M parameter scale [1]. Over time, this expanded to tens of billions of parameters and eventually to the current 400B or more scale [2, 3]. Notably, when the first paper on scaling laws for LLM[1] was published, the largest model, GPT-3[4], had 175B parameters, creating a size ratio of **175B/0.6B ≈ 291** between SOTA models and the scaling law benchmarks. 

According to most reviewers’ comments and the best of our knowledge, our work is the first work to verify the existence of scaling laws in Diffusion Transformers. Most open-sourced diffusion models today are below 10B parameters, and our 1B-scale models are much closer to SOTA in this domain compared with the first few works in LLM, falling within the same order of magnitude. Scaling law validation is a step-by-step process, and the scale of models used increases over time as the field progresses.
To put this into perspective, training a 3B diffusion model would require approximately 1.8T data tokens and close to a month of training, while a 10B model would require 6T data tokens and several months of training. This level of resource demand far exceeds the affordance of a research project. Our work focuses on proving the existence of scaling laws and providing a guideline, not on training SOTA diffusion models. Larger-scale scaling law validation, like in the LLM field, will require further exploration in future studies. 

Q2: **Model architecture.**

A2: First, based on prior research on large language models (LLMs)[1](Figure 5.), the existence of scaling laws for transformer-based models is generally independent of the specific architectural design. While the model architecture may influence the coefficients and exponents of the scaling laws, it does not affect their existence.
To address the reviewer’s concern, we conducted a series of experiments to validate our conclusions. In the main text, we have already demonstrated this through experiments on PixArt [5], and we further provide predictions of the loss in **Supplementary Section G.2**. Additionally, we extended our validation to the latest DiT architecture by evaluating its performance under the Flux budget. The results are included in **Supplementary Section G.1** as well. In both cases, as shown in the figures provided, there is a clear linear relationship between the compute budget and the loss, which strongly supports the presence of scaling laws.

[1] Scaling Laws for Neural Language Models. Kaplan et al. https://arxiv.org/abs/2001.08361

[2] GPT-4 Technical Report. OpenAI. https://arxiv.org/abs/2303.08774 

[3] The Llama 3 Herd of Models. Dubey et al. https://arxiv.org/abs/2407.21783

[4] Language Models are Few-Shot Learner. Brown et al. https://arxiv.org/abs/2005.14165

[5] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. Chen et al. https://arxiv.org/abs/2310.00426
</content>
</thread_object>

</thread>
<thread>
creation date:
1733300334633

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for carefully reading our paper and providing insightful comments. We are grateful for the recognition of our work as “the first scaling laws for DiT” (Reviewer 2p8z, CVDQ) and for its practical value in “DiT training” (Reviewer 2p8z, ssc5, CVDQ, CvMZ). The acknowledgment of our contributions to providing a “predictable benchmark” (Reviewer ssc5, CvMz), “alleviating resource consumption,” and employing “well-designed methods” (Reviewer 2p8z) is highly appreciated. Additionally, we are pleased that the presentation quality has been described as “good” (Reviewer ssc5).
In response to the reviewers’ feedback, we have made modifications to our manuscript, marking all revisions in blue for clarity. These updates include

* To address the reviewers’ concern about whether the scaling laws apply to modern DiT, we conducted experiments on modern model architecture FLUX and Pixart. (Supp Section G.1 and G.2. Reviewer 2p8z, CvMz).

* To demonstrate that the learning rate will not affect the existence of scaling laws and that a larger learning rate for models with more parameters is not optimal, we conduct a learning rate sensitivity analysis. (Supp Section G.5, reviewer 2p8z.)

* To complete the predictable benchmark, we add experiments to show how we can asses data using scaling laws (Supp Section G.6. Reviewer CvMz)

* To eliminate the reviewer’s concern about the resolution dependency, we conduct experiments on 512 resolution. (Supp Section G.3. Reviewer CvMz)

* We added a more accurate discussion on related work of masked diffusion transformers. (Supp Section G.4. Reviewer ssc5)

* Reviewers’ main concern of the paper lies in the model size, we have responded in the common concerns part. 

We hope our response can help address the reviewers’ concerns raised before and during the discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
### **Summary**
This paper investigates scaling laws for Diffusion Transformers (DiT) in the context of text-to-image generation. The authors conduct experiments to reveal a power-law relationship between training loss and compute budget and claim these findings can inform optimal model sizes and data requirements for future research. Additionally, the authors demonstrate that scaling laws can serve as benchmarks for evaluating data and model architectures.

While the topic is highly relevant to the community, the submission falls short of acceptance standards due to the following major concerns:


### **Strengths**
1. **Novelty**:  This is one of the first studies to explore scaling laws for DiT, which has potential implications for future research and resource optimization.
2. **Relevance**:  Scaling laws are an important topic in machine learning, and applying these insights to diffusion models aligns well with the current state of research.
3. **Clarity of Presentation**:   Visualizations and explanations of scaling relationships are clear and well-organized.
4. **Practical Implications**:   The idea of using scaling laws to assess data and model quality is valuable.


### **Weaknesses**
1. **Limited Experimental Scope**:
   - The experiments are conducted with models capped at 1B parameters, far smaller than the typical scale for state-of-the-art diffusion models (~10B). This raises concerns about the generalizability of the findings.
   - The compute budget range (spanning three orders of magnitude) is narrow compared to existing works in the domain, which span seven or more orders of magnitude.

2. **Insufficient Baseline Comparisons**:
   - There is no comparison with state-of-the-art diffusion models, such as those using more modern architectures (e.g., FLUX or SD3).
   - The analysis does not provide meaningful insights into how these scaling laws hold under diverse architectural or data settings, limiting the work’s relevance to current practical setups.

3. **Questionable Use of Metrics**:
   - While FLOPs and tokens are reasonable metrics in theoretical studies, they lack practical interpretability. The absence of GPU hours or image counts as supplementary references makes the findings less accessible to practitioners.
   - The connection between training loss and downstream generation metrics (e.g., FID) is weakly substantiated, raising concerns about the reliability of conclusions drawn from loss trends alone.


### **Recommendation**
The work is novel and timely, but the experimental design and scope are insufficient to support the authors’ claims. The findings are limited in their generalizability and applicability to modern diffusion models, and key methodological concerns remain unaddressed. While this paper provides an initial exploration of scaling laws for diffusion transformers, it does not quite address the concerns of reviewers in its current form.