FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification
Authors: Jingfeng Yao
Cheng Wang
Wenyu Liu
Xinggang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative Model
Diffusion Transformers
Flow Matching
Primary area: generative_models

Abstract: Diffusion Transformers (DiT) have attracted significant attention in research. However, they suffer from a slow convergence rate. In this paper, we aim to accelerate DiT training without any architectural modification. We identify the following issues in the training process: firstly, certain training strategies do not consistently perform well across different data. Secondly, the effectiveness of supervision at specific timesteps is limited. In response, we propose the following contributions: (1) We introduce a new perspective for interpreting the failure of the strategies. Specifically, we slightly extend the definition of Signal-to-Noise Ratio (SNR) and suggest observing the Probability Density Function (PDF) of SNR to understand the essence of the data robustness of the strategy. (2) We conduct numerous experiments and report over one hundred experimental results to empirically summarize a unified accelerating strategy from the perspective of PDF. (3) We develop a new supervision method that further accelerates the training process of DiT. Based on them, we propose FasterDiT, an exceedingly simple and practicable design strategy. With few lines of code modifications, it achieves 2.30 FID on ImageNet at 256x256 resolution with 1000 iterations, which is comparable to DiT (2.27 FID) but 7 times faster in training.

DisterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification
Jingfeng Yao, Cheng Wang, Wenyu Liu, Xinggang Wang" School of EIC, Huazhong University of Science and Technology Wuhan 430074, China
{jfyao, wangchust, wyliu, xgwang))hust. edu.
Abstract
Diffusion Transformers (DIT) have attracted significant attention in research. How- ever, they suffer from a slow convergence rate. In this paper, we aim to accelerate DiT training without any architectural modification. We identify the following issues the training process: firstly, certain training strategies do not consistently perform well across different data. Secondly, the effectiveness of supervision specific timesteps is limited. In response, we propose the following contributions: perspective for interpreting the failure of the strategies.
(1) We introduce Specifically, we slightly extend the definition of Signal-to- to-ioss Ratio (SNR) and suggest observing the Probability Density Function (PDF) of SNR understand the essence the data robustness of the strategy. (2) We conduct numerous experi- ments and report one hundred experimental results empirically summarize a unified accelerating strategy from the perspective of PDF. (3) We develop a new supervision method that further accelerates the training process of DiT. Based on them, we propose FasterDiT, an exceedingly simple and practicable design strategy. With few lines of code modifications, it achieves 2.30 FID on ImageNet 256x256 resolution with 1000 iterations, which is comparable to DiT (2.27 FID) but 7x faster in training.
Introduction
With the advent of Sora 36), its foundational model, Diffusion Transformers (DiT) 377. has ignited extensive research interest. DiT is characterized by its remarkable flexibility and scalability,
demonstrating exceptional capabilities in both image 37 32 16 9 g and video generation [3 33
177. However, similar the challenges faced with Vision Transformers [15). DiT is associated with high training costs. Its convergence rate remains slow, necessitating over 4700 GPU hours training on ImageNet generation tasks 256 resolution 2 This significant computational demand highlights the need for enhancing training efficiency for large-scale training.
One effective method for improving training is modulate the Signal-to- Noiso Ratio (SNR) dis- tribution across different timesteps during training. Denoising generative models (22 30 create a conversion from noise to data and methodically transfer noise into data the timestep. denoted t, progresses. this conversion process, the SNR gradually increases from zero to infinity. Given a generative process I, a4 + d,C. Assuming that the input data x, ideally normally distributed
with variance of one, the SNR is typically defined as the ratio of variances 2 [24 34 40 20).
During one training step. for pair of input data and noise, we randomly select one for training. Modulating noise scheduling (35 25 10), loss weight [34] 20 and timestep sampling strategy [16
"ooreesponding author: xgwang @ hhust.ddu.cn Testing with H800 GPUs
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1.0-1.9
sto=2.3
Method
Arch.
Train Steps
FID-50K]
Same strategy but
FID-10k
opposite trend?
DiT 37
400k
19.5
SIT
DIT-XL-2
FasterDiT (Ours)
11.9 (-5.3)
SiT 32
200k
27.1
DiT-XL-2
FasterDiT (Ours)
200k
17.5 (-9.6)
lognormt0.5,
Sampla Stratagles
Figure 1: (Left) Problem Setting. We find the same sampling strategy gets different performances with different data. (Right) Performance of Di. We improve Diffusion Transformers (DiT) training speed by large margin without any architecture modification.
are common methods to modify the distribution of SNR during training. They are proven to be effective ways to improve training efficiency and effectiveness.
However, these methods not always work across different data. For example, Stable Diffusion (SD3) [66 proposes timestep sampling strategy by logit normal function (lognorm) [T] for better training. Lognorm sampling is an excellent pioneering strategy and has been proven effective in subse- quent work [17]. But it also has some limitations. According to SD3 exploration, lognorm substantially outperforms lognorm(0.5, 0.6) in terms of Frechet Inception Distance (FID) results discover that this conclusion is particular solution specific
1)
on ImageNet [I3]. Nevertheless, to the training data used. As shown in Figure I as we continuously reduce the signal strength, the FID results for lognorm(-0.55 1) progressively worsen, eventually becoming the poorest outcome. This underscores the necessity of evaluating the robustness of such methods across varying data conditions.
In this paper, we aim to provide a more comprehensive perspective for interpreting the problem. Our first contribution is to suggest interpreting the performance robustness with the perspective of the Probability Density Function (PDF) of SNR during training. Specifically, these methods typically modulate the distribution of SNR based the timestep Although the same timestep has the same
relative SNR as Lt the absolute SNR at the step actually increases with the enhancement of the data signal [5 10). We believe that the original definition of SNR only reflects the relative hignal-t-noiss ratio in the same data distribution. However, the distribution of data-related absolute SNR during the training process is the key factor determining the effectiveness of the training. Hence, we slightly extend the definition of SNR and visualize its probability density function.
Our second contribution is conduct extensive experiments and report about one hundred experi- ment results in our paper to empirically analyze the association between training performance and robustness with PDF We analyze the differences in data robustness among commonly used basic We find there is a trade-off between method robustness and performance, and suggest that the training process can be designed more intuitively from PDF perspective.
pipelines on DIT 37 32).
The other method to improve training is modify Our third contribution is the introduction of a new supervision method for velocity prediction- based approaches. With the same schedule, different prediction targets 34 or different supervised methods 14] may also lead to different training results. Recently, velocity has been demonstrated to be a superior predictive target. Specifically, in addition to the traditional Mean Squared Error (MSE) loss, we have incorporated supervision of the velocity direction. We have found that this method is conceptually simple and significantly accelerates the training process.
prediction or supervision [4
To sum up. in this paper, we propose a new perspective on training the SNR PDF to offer straightfor- ward interpretation of the efficiency in training generative models (Section (2. We conduct extensive experiments and report near one hundred experimental results, aiming to empirically derive insights into training performance, robustness, and their correlations with the SNR PDF (Section 3 Subse- quently, we apply these observations to refine the DiT process and, together with introducing a novel supervision method, developed FasterDiT to significantly enhance training efficiency (Section 4 FasterDiT achieves an FID of 2.30 on ImageNet at resolution of 256, comparable to the original

---

DiT's FID of 2.27, yet achieves convergence seven times faster. We hope that our explorations contribute valuable insights to future research in generative model training.
2 Probability Density Function of SNR during Training
We hypothesize that the distribution of attention to different Signal-to-Noise Ratios (SNRs) during training is crucial determinant of training efficiency and effectiveness. Here, two issues need be addressed. Firstly, the previous definition of SNR as ignore the influence of data signal intensity,
Secondly, unified and intuitive approach is required to analyze the training SNR distribution. Therefore, in this section, we propose slight modification the existing definition of SNR. Subsequently, we utilize the Probability Density Function (PDF) of SNR during training to integrate noise scheduling, loss weighting, and timestep sampling strategies into cohesive framework.
which is proven to be important during training 25
35
2.1 Preliminary
For case of comprehension, we give a brief introduction to the formulation and training pipeline of generative models 22 35 29 30). Since our focus is the SNR distribution, universally consider flow matching and the diffusion model from a high-level perspective, similar to previous work [16]. Given data x, ~ X and Gaussian noise N(0, 1). define the transport between noise and data as equation]] [ In a discrete diffusion process 222 35 t is an integer from 0 to 1000. In
continuous flow process 29 30], L is continuous value between [0, 1]. Or, and o, coefficients related to I defined differently according to schedules.
Te Cg + 0zC
During the training process, noised data 2, and timestep t are input to the generative model. The model is required to predict the specific target (noise, velocity, and so on). For example, in the most commonly used DDPM [22 pipeline, the prediction target is noise and the loss function is defined as equation Only single I; will be trained in each iteration for one image, instead of the whole timesteps. Therefore, some methods choose to change the sampling of t [16] or give different loss
weights 20 35 according to t.
Lo = (e =<-<elxre <a( 1))²
2.2 Formulate Probability Density Function of SNR
highlights the importance of SNR. Typically, they assume that the data is an ideal normalization distribution and define the SNR as the ratio of variances (Equation(3. This definition is independent of data and only associated with the coefficients in timestep I.
Data-dependent SNR
Previous work [2
34
40
20
SNRpree (1)
However, the distribution of actual data is indescribable. It is closely linked to the nature of the image itself. In this study, introduce coefficient, K ([),, associated with the image I that directly scales the signal. The value of K(T) is influenced by various image characteristics such as the range of high and low frequencies, resolution, variance, and so on. It is known that the larger the variance of an image, the higher its SNR, illustrated as Equation { Further, for a specific dataset, we assume that the change in variance has little effect on the other nature of the image, i.c. std? is an independent K(J) variant in K(T), and approximate std as a constant C(T).
SNR(1) K(I) a K(T) x sld?
(4)
K(T) std ot std? of
sta
SNR(1)
C(T)
(5)
of
Our goal is to explore how variations in the SNR of different data lead changes in training. However, constantly altering the training data is neither quantifiable nor convenient. The advantage of the above
3

---

Algorithm Estimate the Probability Density Function of SNR
1: Samples N timestep uniformly
2: Calculate St(() and (t) for each
3: Calculate ft(t) = 2 f((t) (t)
(1) for each
4: Estimate ft(t)
5: Samples N timestep from ft(t)
6: Compute SNR(t) = (()) str
for each sampled
of
7: Logarithmize IgBNR(dB) 10lg(C(T)) + 20lgstd + 10lg o7
8: Estimate the probability density function of SNR(t)
conversion (Equation 5) is that C(I) variations. This simplifies the scheme of probing the robustness of the data and allows us to not need change the training data repeatedly.
the same dataset with different std simulate different
Weightings Consider a training process with the timestep sampling function ft( and loss weight function ft).. For most cases, ft) and fi(t) non-negative. To unify these into single distribution of I, we define a new probability density function fx(1) as:
fa(t)= (t())&(() fo ft)) dt
(6)
PDF of SNR Now, try to get the probability density function of SNR during training. It is independent with timestep f. Assume distribution SN R(t)) ~ Y and follows the distribution of (t)... Solving for the distribution of Y can be transformed into problem of probability transformation. Mathematically, it could be defined as Equation 7
fv( f((g(v)) g(v) dy
Estimate of Probability Density Function of SNR In practice, the functions above are not always available. Hence, we use large amounts of discrete samples to approximate Equation 7 Here we provide its estimation algorithm. For all training processes, we visualize their PDFs in the way shown in Algorithm
Now, we have developed simple and feasible method for generating the PDF of SNR. It directly reflects the level of attention to different SNR values during the training process. Specifically, the higher the value of the PDF, the more frequently the corresponding SNR is trained during training. Next, we will conduct extensive experiments to analyze its relationship with different training data and training strategies.
3 What can we learn from SNR PDF?
Back to the problem we discuss in Section Figrue I We argue that strategies that are solely based on / tend to focus only relatively high or low noise conditions in the generation process but are limited considering the absolute Signal-to-Noise Ratio (SNR) with different signal intensity. They provide awesome solutions to improve the training but lack comprehensive evaluation. In this section, we will explore this issue from the perspective of the Probability Density Function (PDF) of SNR, aiming to identify commonalities in these strategies and to delineate comprehensive design space.
3.1 Experiment Settings
We introduce our experiment settings first. We mainly adopt the Diffusion Transformer (DiT) training pipeline from two previous important works, DiT (37) and SiT [32]. We choose four different noise scheduling from them, including DDPM with linear beta schedule (22). DDPM with cosine beta
schedule 355, flow matching with linear schedule (Rectified Flow) 30 and flow matching with cosine schedule 32).

---

Robust Performance
Sensitive Performance
Pletber Porfommmni
Different Data Signal Interaity
6
Figure Robustness of Different Noise Schedules. By scaling input different standard deviations, we compare the data robustness of four schedules (22] 35 29, 32). including diffusion and flow matching, Note that set the prediction target as noise for a fair comparison. We find that different data signal intensities lead to different generative performances and different schedules have different robustness.
Broader SNR Focus
Narrow SNR Focus
STR Core duuing Uocktrru
PDT ranting larger
60
(:)
(C
6D
Figure 3: SNR PDF of different noise schedules [22 35 29 32J. The figure illustrates the signal-to-
noise ratio (SNR) probability density functions (PDFs) for various schedules and standard deviations (see Section
In the subsequent experiments, we initially set the prediction target of each noise schedule noise (epsilon) to ensure fair comparison. Subsequently, we modify the input strength by scaling the data various standard deviations (std). We then shift the prediction target flow matching to velocity the same as its original setting, to compare the similarities and differences between speed prediction and noise prediction outcomes. Each experiment was conducted on ImageNet [3] at resolution of 128. We train each model for 100,000 iterations and assess their performance using the FID-10k metric for comparative analysis. Each experiment has been conducted with H800 GPUs
Our primary observational objectives are twofold: (1) Performance and robustness under different signal intensities. (2) Potential connections between the state of the PDF and performance and robustness.
3.2 Insights from PDF
Different data signal intensities lead to different training effects. In Figure 2 the prediction target for all schedules is uniformly set to noise to ensure a fair comparison. The curves demonstrate how different input intensities impact training outcomes. Notably, the performance of single noise schedule fluctuates with changes in data intensity. For instance, the DDPM with linear beta scheduler achieves an FID 65.38 at std 0.3 and improves significantly to 47.04 std 2.5 (Figure 2
Different schedules exhibit significant differences in data robustness. In Figure as the intensity of the data varies, distinct schedules demonstrate significantly different robustness profiles. Specifically, with increasing standard deviation, the range of the FID for the DDPM-linear 22 schedule is observed at 18.03 (Figure 2).. In contrast, the DDPM-cosine (35) schedule exhibits considerably larger FID range of 50.02 between schedules.
(Fi.2eeFiuree) Figure 2 highlighting substantial variability performance
Schedule with a broad SNR focus could have robust performance. Figure 3 we present the SNR PDFs as outlined in Section 2 For each noise schedule, we display three PDF curves corresponding to different signal intensities: low (sid 0.3), high (sid = 3.3), and the level that results in the optimal FID. These PDF curves visually represent the training process's focus
5

---

Fork
concomtrating
bothrr perforanne upper bound but less robust
4D LT
impling function
6
10
68)
Figure 4: Influence of Weghting Dring Training. We use lognorm(O, 1) as Stable Diffusion3 [16] The essence of this approach to enhance the local focus of the PDF during the training process. This increases the upper bound of the training, but it also reduces the robustness of the training process to variations in the data.
1D-14
concentrating sampling Funcctoon
betber performance upper bound but less zobust
5.00
Fua rode pracii
6
(e)
(C)
60
Figure 5: Flow Matching with v-prediction. We evaluate the robustness of commonly used flow matching 32 with v-prediction on ImageNet. (1) We find flow matching with v-prediction gets
more robust performance than noise-prediction. (2) There still exists a trade-off between performance and robustness.
across various noise interpeaksvals. Empirical analysis of the curve shapes across different noise schedules reveals that greater variance in the PDF, indicating broader consideration of noise intervals (Figure b), associated with improved stability
schedule.
The optimal SNR ranges for different schedules seem be similar. Direct observation of the standard deviations across different schedules does not readily lead to consistent conclusions. For instance, in Figure 2 while the DDPM- -inear schedule (Figure 2a) achieves optimal generative performance at an std of approximately 2.5, the DDPM-Cosine schedule (Figure (2)) peaks at an sld of 0.9. However, insights might be gleaned from analyzing the PDF curves. In our experiments, as depicted in Figure 3 with unified prediction target, better FID performance correlates with the mean the PDF falling within the designated gray area. This observation offers a novel perspective and enriches understanding of the previously established results.
There is a trade-off between performance and robustness. The primary purpose of employing weighting during the training process to intensify the focus on specific SNR levels. We implement logit normal function (Jognorm) [ to adjust the timestep sampling for two distinct schedules (Figure &b),, with results displayed in Figure 4 Our analysis reveals that while well-defined sampling strategy can enhance performance, it may also introduce risks. For instance, the DDPM- Linear schedule achieves an optimal FID of 47.04, demonstrating robust performance across a range of 18.02. However, when we intensify the focus using lognorm (Figure 4p), although the best FID improves to 36.9, it also results in more variable outcomes, with a performance range widening to 80 (Figure ()).
Flow matching with v-prediction gets more robust performance. Recently, flow matching [2 30 has been regarded more concise and efficient pipeline for generative models. In the previous discussion, to facilitate a fair comparison, we set the prediction target on noise prediction. In Figure 5 We find flow matching with v-prediction achieves a more stable output performance, demonstrating enhanced robustness in data stability. For instance, Linear Flow exhibits range in FID of 17.54 when using noise prediction, compared 11.79 when using v-prediction (Figure 5p). This characteristic is even more pronounced in Flow with cosine schedule, where the ranges are 34.76 and 12.39, respectively (Figure 5b)) Additionally, we find that the overall performance v-prediction generally surpasses that of noise prediction.

---

Algorithm 2 FasterDiT Training
def lognorn(zunO, signart, size-Nons);
find a properstart standard deviation to shift PDF
get logit nornal distribution samples scipy morn. rva (locrmm, scale-eigns, size-size)
# transform 0 to samples 1 (1 up. sppeex((aaaplee) return samples
while training:
30d
data shifting data: (B, C, B, W)
(a) Standard Deviation Modulating, Initially, we determine an appropriate range for the stan- dard deviation.
# noise: (B, E, V)
data (target_std data_atd) noise torch. rand nlike (data)
multi-step
velocity direction loss
FID-50k
2. concentrating
balance
# tiuestep: (B, 1ognorn(0, 1, t.shapa[o]) x_t t*data (1-t)=noise pred (x_t, t, y)
150k
200k
400k
21.4
17.5
11.5
27.2
22.7
15.8
32.5
27.1
18.5
# improved supervision
v data moise
(b) Ablations for FasterDiT Training, Both multiple-step balance and velocity direction loss significantly enhance training efficiency.
loss (pred ())a2.neaa()) cosine weeineeimilariityppree v, din-1) .een)()
(acksard))
Figure 7: Training Details. Our training pipeline involves only minimal modifications to the code.
The trade-off still exists in flow matching with v-prediction. Furthermore, observe that although v-prediction is generally more robust, there remains a trade-off between performance and stability. As shown in Figure5.&0 when used lognorm concentrate the focus of Linear-Flow, its performance resembled that of noise prediction, where the upper limit improved but stability decreased. We hypothesize that such trade-offs may be broadly prevalent.
To sum up, from the perspective of the PDF, efficient training requires satisfying two conditions: (1) the PDF should have concentrated focus; (2) the focus of SNR needs to fall within the correct range. Based on the simple observation, we try to improve DiT training.
4 Improving DiT Training
In this paper, objective is not to devise novel model architectures for achieving state-of-the-art outcomes. Instead, we aim to explore simpler, more interpretable, and universally applicable training approach for Diffusion Transformers (DiTs) 37
In Section Bl we demonstrate trade-off between performance and robustness in the DiT training process via signal-to-nois (SNR) probability density function (PDF) analysis. To expedite training, we should have concentrated PDF that focuses on the right SNR during training. Initially, we select flow matching (30] with v-prediction as our noise schedule, owing to its robustness and superior performance. We then adjust the PDF to focus on the optimal SNR during training by modulating the standard deviation (std) of training data. Further, we employ logit normal function []] to sharpen the focus on the adjusted regions. Finally, we introduce new, straightforward supervisory strategy that significantly enhances training outcomes.
4.1 Improving Multiple Step Balance
Here, we first modulate the standard deviation from 0.5 to 1.2 as illustrated in Figure (left). (left). During this sweep, we refrain from employing additional techniques such as weighting. shows that when sid is around 0.70, the DiT training gets better generative performance. Empirically, we set the
target sld to 0.82, which near the proper and stays consistent with previous work's setting However, argue that when the input data changes, e.g. diffeerent resolution, the choice of std will
7

---

Method
Model
Training Iters
FID.
sFID
ISt
Prec.
Rec.
BigGAN
4
BigGAN-deep
6.95
7.36
171.4
0.87
0.28
MaskGIT
MaskGIT
x36
6.18
182.1
ADM-G 14
ADM
x250
4.59
5.25
186.70
0.82
0.52
CDM 223
CDM
4.88
158.71
RIN 26
RIN
3.42
Simple Diffusion
25
U-Net
251
3.76
Simple Diffusion
U-VIT-L
500kx2048
2.77
LDM-4-G 40
LDM
178k 1200
3.60
247.67
0.87
0,48
U-VIT-G
U-VIT
300kx 1024
3,40
StyleGAN 42
StyleGAN-XL
2,30
4.02
265.12
0.78
0.53
MDI-G 18
MDT
2500kx256
2.15
4.52
249.27
0.82
0.58
DiT 37
7000kx256
9.62
6.85
121.50
0.67
0.67
SiT 32
7000kx256
8.61
6.32
131.65
0.68
0.67
DiT-XL/2
1060kx256
8.72
5.23
121.17
0.68
0.67
FasterDiT
2000kx256
7.91
5.46
131.27
0.67
0.69
DiT (cfg=1.55 37
7000kx256
2.27
4.60
278.24
0.83
0.57
SiT (cfg=1.5) 32
7000kx256
2.06
4.50
270.27
0.82
0.59
DIT-XL/2
1000kx256
2.30
4.80
249.34
0.82
0.58
FasterDiT (cfg=1.5)
2000kx250
2.03
4.63
263.95
0.81
0.60
Table Performance of FisterDiT on ImageNet 256x256. Employing the identical architecture as DiT (37) FasterDiT achieves comparable performance with an FID of 2.30, requires only 1,000k iterations to converge.
differ. The strategy is similar to previous work [10], but we discuss it in new perspective of training SNR PDF.
Then we improve the performance by a haman-made concentration for PDF. Specifically, we use logit normal function [1] for timestep sampling, while and sigma
0 and respectively.
Notably, mu is set to 0 because the PDF already properly shifted (see Figure 3). It could help avoid instability discussed in Section D
and improve the training performance (see Figure (left) (b)).
4.2 Improving Single Step Supervision
Effective single-step supervision is essential in training generative models. For instance, employing different prediction targets can result diverse predictive outcomes [34]. the original DiT model, strategy that simultancously predicts noise and sigma is utilized to maximize performance (37).
Typically, the Mean Squared Error (MSE) loss function employed for supervising these targets.
#-1 W
(h,w) (h.wr)
Σ
gt
pred
La = HW
(8)
(h,w) ||0 (h,ww) gt
h=
Recently, in the context of flow matching specific physical significance, representing the flow rate from noise to data. Based on this, we hypothesize that supervising the direction of velocity could serve effective supervision strategy. Consequently, in this paper, we not only use the Mean Squared Error (MSE) to supervise velocity predictions but also apply cosine similarity we present velocity direction loss as shown in Equation while H, W denotes the height and width of predicted latent by DiT. We demonstrate that this combined supervisory approach significantly enhances the model's 'ss convergence rate (see Figure (left) (b))
29 30
the prediction target of velocity acquires more
further supervise the directionality of velocity. Here
4.3 Comparison with Previous Methods
Owing resource limitations, we train FasterDiT on ImageNet 256 resolution [3 for 2,000k iterations and benchmark it against current advanced generative models (see Table I.. Our experiment

---

Figure 8: Visualization Results. We present visualization results for FasterDiT--X1/2 after training for 1,000k iterations, with CFG set to 4.0.
has been conducted with H800 GPUs. Notably, the focus of our work is to explore training strategies beyond structural enhancements. Thus, FasterDiT utilizes the identical architecture DiT [32]. As illustrated in Section Figure [
FasterDiT achieves an FID-50k score of 11.9 at 400k iterations, markedly outperforming the original DiT model (FID-50K 19.5), and its enhanced counterpart, SiT (FID-50k 17.2) (32). Utilizing the same architecture as DiT (37], FasterDiT attains similar performance level with FID score of 2.30 but converges in just 1,000k iterations.
4.4 Performance on Higher Resolution Images
We evaluate FasterDiT on higher resolution generation tasks to explore its resolution generalization capabilities. Specifically, apply approach to DiT-B/2 and DiT-L/2 models for ImageNet generation at a 512x512 resolution. The results, presented in Table 2 indicate that FasterDiT consistently achieves faster convergence across all configurations. Notably, after 200k training iterations, our method improves the FID-10k performance of DīT-B/2 by 18.78 and DiT-L/2 by 17.93, underscoring its effectiveness for high-resolution image generation tasks.
4.5 Performance with Different Diffusion Architectures
We further apply our training method to other diffusion models beyond DiT, including Latent Diffusion Models [0) (using the UNet [41] architecture) and U-VIT (2]. Here, specifically refer to the use of the previously mentioned multi-step balance and velocity direction loss. The results, presented in Table indicate performance improvements for both U-ViT' and UNet with method. This suggests that our approach has the potential to generalize across broader range of diffusion model architectures.
5
Related Work
5.1 Generative Models and Diffusion Transformers
The denoising diffusion probabilistic model (22 has gradually replaced GANs 12 19
due to its stronger performance and more stable training, becoming the mainstream generative model. Among them, [4 first proves its effectiveness on ImageNet [13]. 40 38) extend the diffusion process from the diffusion space to the space VAE latent space, achieving high-performance, high-resolution generation. Meanwhile, some methods [32 46 [7] have also trained generative models using flow-matching (29 30 techniques to replace the diffusion path. These methods exhibit simpler mathematical properties and faster learning effects.
Among them, U-Net 41 is the most commonly used architecture for generative models. Recently, due to the scalability advantages of transformers [1]] 44, some transformer-base 2 37 generative

---

Training Samples
Method
Models
Resolution
FID-10k
Training Samples
Model
FID-10K
100k x 128
93.36
DiT
DFT-B/2
512x512
200k 128
77.11
U-ViT-L [2]
50.22
DisterDiT
DIT-B/2
512x512
77.85
U-ViT-L Ours
200k x 128
37.12
200k 128
58.33
LDM-UNet 40
66.73
100k 64
87.24
200k x 128
DIT
DIT-L/2
512x512
LDM-UNet Ours
60.07
200k 64
67.29
100k
71.58
FasterDiT
DIT-L/2
512x512
Table 3: Performance with Different Archi- tectures.
200k x 64
49,36
Table 2: Performance on ImageNet 512x
models have also emerged. However, similar to Vision Transformers ers 37 converge slowly.
[5] Diffusion Transform-
5.2 Fast Training of Generative Models
Work for accelerating Diffusion Transformers (DiT) could be simply divided into two categories. One is architecture modification. MaskDiT 45 and MDT [88 combine mask image modeling B 21
pre-training and diffusion training for speeding up. Similarly, SD-DiT [46] takes one step further
to combine DiT training with MoCo-like [I]] contrastive learning. CAN [5] proposes dynamic
weights for condition to speed up training of diffusion models 37 []
Another strategy involves changing non-model design approaches. For example, using different noise schedules can achieve better results on images of various resolutions [25]
35 10 27). Using different
prediction targets, such as noise, data, or velocity, can also directly impact the effectiveness of the training 34 Adjusting weights for the loss function and training sampling can also directly affect the outcomes of training (20 16]. However, they constitute a vast design space with high degree of direct interdependence, which makes the design of the model very complex.
Conclusion
Limitations We propose novel approach called FasterDiT for accelerating the training of dif- fusion transformers. The main limitation of this paper lies the lack of exploration of large-scale experiments, such 2K high-resolution images, toxt-to-image generation, and video generation. Among these, we particularly focus on the text-to-image generation aspect. Specifically, in the class-conditional generation described in this paper, the DiT block only needs to process image tokens. However, in some -ix-e architectures, such as SD3 [16], self-attention needs to handle sequences combining text and visual tokens. The features from different sources (such as T5 (39] and VAE (28)) may lead to potential instability. We plan to further investigate this issue in the future.
Societal Impacts Our work has the potential to significantly enhance the efficiency of creative professionals in completing their creative processes. However, also carries risk of misuse.
Conclusions In this paper, we discuss training strategies to accelerate Diffusion Transformers. Initially, slightly generalize the definition of SNR and conduct unified analysis of various training strategies through the examination of SNR probability density functions during the training process. We find that there is trade-off between robustness and performance in the training of diffusion models. Different noise schedules exhibit varying levels of robustness. Using weighting during the training can enhance the performance ceiling, but may also decrease the robustness of the process. Additionally, we discover that utilizing directional loss as an auxiliary loss function for velocity prediction significantly boosts training performance. Based these observations, we conduct experiments with - on ImageNet 256 256 and observe substantial acceleration in training. We refer to this straightforward training approach as FasterDiT. We hope our work inspires further exploration into training strategies for generative models.
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by the National Natural Science Foundation China (NSFC) under Grant No. 62276108. We also extend special thanks to Bencheng Liao for his valuable suggestions.
References
[1] Jhon Atchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261-772, 1980.
[2] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22669-22677 2023.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprin! urXiv:2106.0822 2021.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 2018.
[5] Han Cai, Muyang Li, Zhuoyang Zhang, Qinsheng Zhang, Ming- Yu Liu, and Song Han. Condition-awe aware neural network for controlled image generation. arXiv preprint arXiv:2404.01143, 2024.
[6] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedingy of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[7] Junsong Chen, Chongjian Ge, Enze Xie. Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang. Ping Luo, Huchuan Lu, and Zhenguo Li. Exxart>\sigmal Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403. 04692, 2024.
[8] Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-('deltta) Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.052.. 2024.
[9] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Fixart)\alpha Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
[10] Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint (XXi:23301.1092, 2023.
[11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020.
[12] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53-65, 2018.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and patlern recognition, pages
Icee, 2009.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint 2020.

---

[16] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. arXiv preprini 2xXiv:2433.33206, 2024.
[17] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arX5vv2405.05945 2024.
[18] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23164-23173, 2023.
[19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139 144, 2020.
[20] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 7441-7451, 2023.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[23] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[24] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
[25] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213- 13232. PMLR, 2023.
[26] Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212. 11972, 2022.
[27] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the clbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
[28] Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[29] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.022747, 2022.
[30] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
[31] Haoyu Lu, Guoxing Yang, Nanyi Fei, Yugi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: pereral-purpose video diffusion transformers via mask modeling. In The Twelfth International Conference on Learning Representations, 2023. 2023.
[32] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijndew and Saining Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers, arXiv preprint arXiv:2401 08740, 2024.
[33] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 2024.
12

---

[34] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[36] OpenAl. Sora. httpps//openai.c compeenaicco//oroo 2024.
[37] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[38] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307. 00952, 2023.
[39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 176 2020.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assise Assisted Intervention-MEL -MCCCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer, 2015.
[42] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[43] Yang Song. Jascha Sohl-Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09300 09330, 2023.
[46] Rui Zhu, Yingwei Pan, Yehao Li, Ting Yao. Zhenglong Sun, Tao Mei, and Chang Wen Chen. Sd-dit: Unleashing the power of self-supervised discrimination in diffusion transformer. arXiv preprint 170004 2024.
13

---

A Implementation Details
In this section, we offer a detailed overview of the experimental methodologies employed in our study. The investigation is delineated into two primary segments. Initially, we conduct thorough experiments to explore the potential relationship between training efficiency and effectiveness as detailed in Section D Subsequently, we extend our findings to larger training configurations of Diffusion Transformers to ascertain the efficacy of our methodologies, as discussed Section
A.1 Details of Training
The specific details of the training processes are delineated in Table and Table 5 A notable distinction arises in Section 3, where, to expedite training, we pre-computed and stored image features, abstaining from the use of data augmentation during the training phase. Training was conducted at resolution of 128, with each experiment running for 100,000 iterations. Conversely, in Section 4 our primary emphasis is on the outcomes of training. Consequently, we opted not to pre-compute image features and instead implemented various image enhancement techniques. Furthermore, the training duration was extended to ensure comprehensive model evaluation.
A.2 Details of Sampling
In this section, we detail the sampling procedures as discussed in Chapters 3 and 4. The specific outcomes are presented in Table and Table 7 A key distinction between the two sections is as follows: Section we assessed the FID 10,000 generated images without employing the cfş operation. Conversely, in Section 4, we evaluated the FID of 50,000 generated images and adjusted the cfg to 1.5, aligning with the methodologies of prior studies 37 32].
A.3 Details of Noise Scheduling
The focus of this paper is on the accelerated training of generative models, with a particular emphasis on Diffusion Transformers 337 The design of the generative process presented herein draws
inspiration from the pioneering work on DiT (37] and the subsequent advancements in SiT (32). The former primarily adopts the DDPM generation pipeline (22), while the latter enhances DiT training by integrating score-based diffusion [43 and flow matching.
In Section 3 we employ four distinct noise schedules, encompassing both diffusion and flow match- ing. Typically, these can be represented as xa o, + atc. Specifically, the initial two schedules (Figure &bbb) employ the widely utilized DDPM linear schedule (22] and the DDPM cosine sched- ule [35]. respectively. Owing to the complexity of these concepts, a detailed discussion is beyond the scope of this paper; readers are encouraged consult the original publications for more comprehen- sive information. Regarding the last two (Figure &d), we utilize the two most common flows, as delineated in Equations 9 and 10
Figure 2 :0: o, -1-t
(9)
Figure 2 :0; = cos( Tt), oe = sin( mT)
nTt),
1
(10)
B More Visualization Results
In this section, we present more images generated by FasterDiT, which underwent training on the ImageNet dataset a resolution of 256 for 1,000,000 iterations. The visualization results, depicted in Figure 10 10 feature samples from seven distinct ImageNet categories, emphasizing selected outputs. During the testing phase, the cfg value was consistently set at 4, in accordance with parameters established in previous research. Remarkably, despite a substantially shorter training period compared to DIT, Dis has successfully generated visually impressive results.
14

---

index:333 hamster
index:388 panda
index:99 goose
Index 207 golden retriever
index:282 tiger cat
index:367 baboon
index:387 lesser panda
Figure 9: Generation Results-1. We visualize generation results of FasterDiT, which trained on ImageNet at 256 resolution for 1000k iterations.
15

---

index::49 boathouse
indeex:833 castle
modex:644 four poster
index:822 stupa
index:42 beseball
indeec:360 otter
index:417 halloon
Figure 10: Generation Results-2. We visualize generation results FasterDiT, which trained on ImageNet at 256 resolution for 1000k iterations.
16

---

Number of Samples Table 4: Training Details Section
Parameters
Value
Optimizer
Adam W
81
0.9, 0.999
Learning Rate
lc-4
Weight Decay
0
Global Batchsize
256
Training Iterations
100,000
Dataset
ImageNet 13
Resolution
128
Number Workers
4
Loss Function
Lense yes
Precompute VAE Features
Timestep Sampling
none/ lognorm((,, 1/ lognorm(O, 0.5)
Data Augmentation
none
Table 5: Training Details Section
Parameters
Value
Optimizer
AdamW
$1. 32
0.9, 0.999
Learning Rate
le-4
Weight Decay
0
Global Batchsize
256
Training Iterations
1,000,000
Dataset
ImageNet [3
Resolution
256
Number Workers
Loss Function
Imses La (Eq
8
Precompute VAE Features
no lognorm((O, 1)
Timestep Sampling
Data Augmentation
Random Horizontal Flip
Table 6: Sampling Details of Section
Parameters
Value
Resolution
128
Batchsize per GPU
32
Number of Classes
1000
cfg
1.0
10,000
Number of Sampling Stpes
250/adaptive 0
Global Seed
Using tf32
yes
Table Sampling Details of Section
Parameters
Value
Resolution
256
Batchsize per GPU
128
Number of Classes
1000
cfg
1.5
Number of Samples
50,000
Number of Sampling Stpes
adaptive 0
Global Seed
Using tf32
yes
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our abstract and introduction accurately reflect the contribution and scope of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
18

---

Justification:
The limitations of the work are discussed in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide as many details as possible, such as algorithms and pseudo codes, ensure reproducibility. Besides, we also provide our source codes.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data (we use the public dataset in our experiments) and code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: Open access the data and code is provided in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public//uiids//odeSuttaue
sot.:eoubbiisiioooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/tpps.cc/public//uue
CodeeSumiiii wwabbbmssssiinii ssisssoiccyyiis for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All the training and test details are specified in the paper and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The paper does not report error bars due to limited resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sufficient information on the computer resources for each experiment are provided in the paper and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics atttps///neur/ htttps/s//arrrss..ppppllc//.tiit
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
22

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators of assets used in the paper are properly credited. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwpprswithhaeeooo thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets are introduced in the paper. The documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
23

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ekHajdKGOa", "reviewer": "Reviewer_fAgp", "summary": "This paper presents observations on training strategies for Diffusion Transformers (DiT) using Signal-to-Noise Ratio (SNR) Probability Density Function (PDF) analysis. Through extensive experiments, the authors derive insights into training performance and robustness. Based on these findings, they propose a method to accelerate the training process of DiT without modifying the model architecture.", "questions": "Why does adding cosine similarity loss for velocity direction supervision accelerate the convergence rate of the model?", "limitations": "The scalability of this method is not thoroughly evaluated, including its effectiveness for higher resolutions, other datasets, or different types of generative tasks (e.g., text-to-image, video generation).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem addressed is practical, focusing on the significant computational demands of training Diffusion Transformer models.\n2. The authors propose a simple yet effective technique to improve the training efficiency of Diffusion Transformer models without modifying the architecture\n3. The paper is well-structured with clear motivation, methodology, and results.", "weaknesses": "1. The paper lacks experiments with higher resolutions such as 512x512 or 1024x1024, which could provide insights into the method's scalability for more complex image generation tasks.\n2. The hyperparameter 'std' requires tuning to find the appropriate setting, which may introduce additional complexity in implementing the method across different datasets or tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\color{red}{Question1:}$ The paper lacks experiments with higher resolutions such as 512x512 or 1024x1024, which could provide insights into the method's scalability for more complex image generation tasks.\n\n$\\color{blue}{Response1:}$ Thanks for the suggestion. As requested, we have conducted experiments on a higher resolution (ImageNet 512x512) using two different model scales (DiT-B and DiT-L). The results show that FasterDiT still achieves faster convergence at higher resolutions. For more details, please refer to our **Response to All Reviewers**.\n\n$\\color{red}{Question2:}$ The hyperparameter 'std' requires tuning to find the appropriate setting, which may introduce additional complexity in implementing the method across different datasets or tasks.\n\n$\\color{blue}{Response2:}$ Thank you for the question. We will explain it from two perspectives. \n\nFirstly, we believe that an appropriate standard deviation (std) of the diffusion input can influence training performance in certain cases. This observation is similar to the concept of a 'magic number' or 'scale factor' in previous well-known studies [1, 2]. Using the CIFAR10 toy setting as an example, when we apply the most common image normalization, the standard deviation (std) of the data is approximately 0.4, and the training FID result is 51.40. However, when we adjust the std to 0.8, the result of the same training method improves significantly to 36.31. This demonstrates the importance of std adjustment. \n\n| Model | Std |Training Samples | FID-10k |\n|:--------|:--------:|:--------:|:--------|\n| DiT-S/2 | 0.4 | 50k x 128 | 51.40 |\n| DiT-S/2 | 0.8 | 50k x 128 | 36.31 (-15.09) |\n\nSecondly, the complexity brought by tuning the 'std' will decrease as research on diffusion deepens. For instance, [2] suggested that as resolution increases, training tends to favor a smaller std. In our work, the SNR PDF provides a new analytical tool and offers insights into this issue. We hope it can facilitate a deeper understanding of the diffusion process.\n\n$\\color{red}{Question3:}$ Why does adding cosine similarity loss for velocity direction supervision accelerate the convergence rate of the model?\n\n$\\color{blue}{Response3:}$ The direction loss of velocity serves as an auxiliary form of supervision that enables a more fully supervised training process. To provide further insights, we have conducted an ablation study. We train two DiT-B models: one with FasterDiT and the other with FasterDiT without direction loss. We then visualize the training loss at different timestep intervals, which can be seen in our **submitted PDF**. The results show that the model with directional loss demonstrates a significantly better training effect, particularly during the diffusion period of relative low-noise ($0.4 < t < 0.9$). The end-to-end performance comparison is shown in the table below.\n| Method | Models | Training Samples | Resolution | FID-10k (100k) | FID-10k (200k) |\n|:--------|:--------:|:--------:|:--------:|:--------:|:--------:|\n| FasterDiT (w/o direction loss)    | DiT-B/2 | 100k x 128 | 512x512 | 83.29 | 63.11 |\n| FasterDiT     | DiT-B/2 | 100k x 128 | 512x512 | **77.85** | **58.33** |\n\n$\\color{red}{Limitations}:$ The scalability of this method is not thoroughly evaluated, including its effectiveness for higher resolutions, other datasets, or different types of generative tasks (e.g., text-to-image, video generation).\n\n$\\color{blue}{Response:}$ Thanks for your suggestion. We have provided experiments with higher resolutions and different models of FasterDiT. We will explore more complex tasks you mentioned such as T2I generaiton and video generation in our following work. \n\n*Refs:*\n\n[1] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.\n\n[2] Chen T. On the importance of noise scheduling for diffusion models[J]. arXiv preprint arXiv:2301.10972, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ZHWWinzOe", "reviewer": "Reviewer_qPr3", "summary": "The paper focuses on accelerating the training process of Diffusion Transformers (DiT) without modifying their architecture. The authors identify two primary issues: inconsistent performance of certain training strategies across different datasets, and limited effectiveness of supervision at specific timesteps.\n\nKey contributions include:\n1. Extended Definition of Signal-to-Noise Ratio (SNR)\n2. Extensive experiments and empirical findings for SNR PDF\n3. A new supervision method", "questions": "see weakness", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- FasterDiT achieves competitive results (2.30 FID on ImageNet 256 resolution at 1000k iterations) while being seven times faster in training compared to traditional DiT (2.27 FID).\n- The paper presents a large number of experiments to empirically validate the proposed method. The paper presents various discussions and insights of SNR PDF.\n- By generalizing the definition of SNR and analyzing various training strategies through SNR PDFs, the paper shows promising outcomes of faster training.", "weaknesses": "- The experiments were conducted on 256-resolution ImageNet only. It would be interesting to validate the proposed method on larger resolutions (such as 512, 1024, etc.). Acceleration is more critical to those scenarios.\n- The figures could be improved. The rendered text is not easy to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\color{red}{Question1:}$ The experiments were conducted on 256-resolution ImageNet only. It would be interesting to validate the proposed method on larger resolutions (such as 512, 1024, etc.). Acceleration is more critical to those scenarios.\n\n$\\color{blue}{Response1:}$ Thanks for your suggestion. As requested, we have conducted experiments with higher resolution (ImageNet 512x512). It shows that FasterDiT could still achieve faster convergence speed compared to the original DiT. Please refer to our **Response to All Reviewers** for more details. Due to the limited time available for rebuttal, we will explore higher resolution and larger training scales in the future.\n\n$\\color{red}{Question2:}$ The figures could be improved. The rendered text is not easy to read.\n\n$\\color{blue}{Response2:}$ Thank you very much for your kind suggestions. We will improve our figures in the future version of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K2dwqrEtxA", "reviewer": "Reviewer_ttWn", "summary": "This paper propose FasterDiT, a diffusion model training method that considers the data distribution in the definition of signal-to-noise ratio. It formulates the SNR in a new framework, estimates the PDF of the SNR, and then employs it to improve the training efficiency of DiT. Experimental results show that FasterDiT achieves relative FID but with 1/7 training time.", "questions": "1. Does x_* in the paper indicate the x_0 (real images)? If so, I advise to replace it with x_0 to align with previous works.\n2. Section 4: Improving DiT Training. The \".\" should be removed.\n3. The space between line 140 and 141 are excessively reduced.\nIn summary, I think this paper may have good technical contribution. But it's really difficult for me to understand the details of this paper.", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The idea of re-formulating the SNR in the diffusion model is a very interesting and novel strategy for me. I believe this paper has a good technical contribution.\n2. The experimental results of DiT show both good acceleration and generation performance.", "weaknesses": "1. This paper has a bad quality in writing. There are some typos and the explanation to the SNR is not clear to me. I'm confused for many questions: (1) What does the std means in Line 109. Does it mean the std of the value of pixels in the images?  (2) Why can we assume std^2 approximate K(I)/std^2 as a constant C(I)?  (3) Authors mention \"robustness\" for many times in the paper for \"training robustness\", \"data robustness\". What does it mean in detail? (4) In Figure 6, the caption inside the figure is \"multi-step balance\", while the caption after the figure is multiple-step balance. Please use the same description. What does multi-step balance mean? Does that mean the new training strategy with SNR?  The \"single-step supervision\" is also not a good choice here. It should be something like \"directionality of velocity\" or \"single-step supervision (ours)\" since there has already been single-step supervision in the traditional training method.\n2. Does the proposed method generalize well to the diffusion models besides DiT, such as latent diffusion models? Please discuss on this. If so, experimental results should be provided for this since only experiments on DiT & ImageNet is not very convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\color{red}{Question1:}$ What does the $std$ means in Line 109. Does it mean the $std$ of the value of pixels in the images? \n\n$\\color{blue}{Response1:}$ The $std$ here refers to the standard deviation of the input. For traditional diffusion, it refers to the standard deviation of pixel values. For latent diffusion, it refers to the standard deviation of the latents after VAE encoding. In this paper, our experiments are primarily based on latent diffusion.\n\n$\\color{red}{Question2:}$ Why can we assume $std^2$ approximate $K(I)/std^2$ as a constant $C(I)$? \n\n$\\color{blue}{Response2:}$ The approximation maintains our definition as an extension of the previous one with minimal changes. Specifically, in previous definitions, the standard deviation ($std$) is the most crucial factor in quantifying the Signal-to-Noise Ratio ($SNR$). For ease of analysis, previous works [1, 4] treat both the input and noise as normal distributions, resulting in: \n\n$SNR_{pre} = \\frac{(\\alpha_t \\times std_{input})^2}{(\\sigma_t \\times std_{noise})^2}=\\frac{\\alpha_t^2}{\\sigma_t^2}, std_{input}=std_{noise}=1$\n\nIn FasterDiT, our definition extends this from the following two aspects: Firstly, we take the standard deviation of input back into consideration instead of assuming it to be one. Secondly, some work [5] has shown that $SNR$ relates to more properties of inputs besides $std_{input}$ such as resolution. Here, we denote these additional properties as $K(I)/std_{input}^2$. Thus, we obtain:\n\n$SNR = \\frac{(\\alpha_t \\times std_{input})^2}{\\sigma_t^2}\\times\\frac{K(I)}{std_{input}^2}$\n\nIn our work, we pay more attention to the former aspect and relax the definition. We assume the $std_{input}^2$ is a decoupled parameter in $K(I)$ for ease of analysis similar to the previous definition. It helps us simplify the modeling of the SNR PDF with minimal impact on the qualitative results. We plan to conduct a more in-depth exploration of this in our future work.\n\n$\\color{red}{Question3:}$ Authors mention \"robustness\" for many times in the paper for \"training robustness\", \"data robustness\". What does it mean in detail? \n\n$\\color{blue}{Response3:}$ Here, \"robustness\" refers to the stability of training performance despite variations in the standard deviation of the input data. We will standardize this terminology in future versions.\n\n$\\color{red}{Question4:}$ In Figure 6, the caption inside the figure is \"multi-step balance\", while the caption after the figure is multiple-step balance. Please use the same description. What does multi-step balance mean? Does that mean the new training strategy with SNR? The \"single-step supervision\" is also not a good choice here. It should be something like \"directionality of velocity\" or \"single-step supervision (ours)\" since there has already been single-step supervision in the traditional training method.\n\n$\\color{blue}{Response4:}$ The \"multi-step balance\" refers to modifying the SNR PDF by adjusting the proportion of different timesteps during training. We will standardize the terminology and improve the naming of the direction loss to enhance clarity. Thank you for your suggestions. \n\n$\\color{red}{Question5:}$ Does the proposed method generalize well to the diffusion models besides DiT, such as latent diffusion models? Please discuss on this. If so, experimental results should be provided for this since only experiments on DiT & ImageNet is not very convincing.\n\n$\\color{blue}{Response5:}$ Thanks for the suggestion. Yes, it does. The method proposed in FasterDiT is decoupled from the model architecture. It could be interesting to explore its performance alongside DiT. As requested, we employed the U-Net used in LDM [2] and another well-known architecture, U-ViT [3], to investigate the influence of our method. The results demonstrate that FasterDiT continues to enhance the convergence rate of these models. Please refer to **Response to All Reviewers** for more details.\n\n$\\color{red}{Question6:}$ (1) Does $x_*$ in the paper indicate the $x_0$ (real images)? If so, I advise to replace it with $x_0$ to align with previous works. (2) Section 4: Improving DiT Training. The \".\" should be removed. (3) The space between line 140 and 141 are excessively reduced. In summary, I think this paper may have good technical contribution. But it's really difficult for me to understand the details of this paper.\n\n$\\color{blue}{Response6:}$ Thank you very much for your careful review of our submission and your detailed suggestions. In this paper, we explore DiT training with both diffusion and flow matching. We didn't choose $x_0$ to denote the input because of the different timestep definitions. Different from DDPM, in Rectified Flow, $x (t=0)$ refers to the pure noise. We will consider clearer expressions in future versions of our paper. Besides, we will address all the other issues you raised and thoroughly review the entire manuscript to prevent similar issues from occurring in the future.\n\n*Refs:*\n\n[1] Salimans T, Ho J. Progressive Distillation for Fast Sampling of Diffusion Models[C]//International Conference on Learning Representations.\n\n[2] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.\n\n[3] Bao F, Nie S, Xue K, et al. All are worth words: A vit backbone for diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 22669-22679.\n\n[4] Hang T, Gu S, Li C, et al. Efficient diffusion training via min-snr weighting strategy[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 7441-7451.\n\n[5] Hoogeboom E, Heek J, Salimans T. simple diffusion: End-to-end diffusion for high resolution images[C]//International Conference on Machine Learning. PMLR, 2023: 13213-13232.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iOIEWc5nNE", "reviewer": "Reviewer_dNCD", "summary": "This work aims at solving the slow training convergence of Diffusion Transformer, from the perspective of Signal-to-Noise Ratio (SNR).\nDifferent from other works, the authors formulate the probability density function (PDF) of SNR during training, and then leverage such SNR PDF to analyze the association between training performance and robustness across some common pipelines of DiT.\nThe finding of the trade-off between method robustness and performance makes authors to propose Fast-DiT which significantly accelerate DiT's training convergence.", "questions": "Listed in weakness.", "limitations": "The authors has discussed the limations of the best performance of FastDiT without sufficient GPUs.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is very clear. DiT suffers from slow convergence and this work successfully solved this issue without any changes in model architecture.\n\n2. The formulation of SNR PDF is novel. With this analysis tool, authors can check the robustness and performance among various DiT training pipeline.\n\n3. The emipirical experiments are sufficient and the final results compared with SOTA are convincing for me.", "weaknesses": "1. Current Table 1 compares the CFG  (classifier-free guidance) results.\nPlease compare the Class-conditional results of Faster-DiT with DiT and SiT, under the same setting in Table 1 (or even more iterations of Faster DiT).\nUsually we need both CFG and Class-conditional results to check the training convergence. Currently Figure 1 right with small iterations is not convincing.\n\n2.  I am still unclear how you change the std of data without changing dataset. Please specify this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\color{red}{Question1:}$ Current Table 1 compares the CFG (classifier-free guidance) results. Please compare the Class-conditional results of Faster-DiT with DiT and SiT, under the same setting in Table 1 (or even more iterations of Faster DiT). Usually we need both CFG and Class-conditional results to check the training convergence. Currently Figure 1 right with small iterations is not convincing.\n\n$\\color{blue}{Response1:}$ Thank you for your suggestion. \n1. As requested, we have added the following content: (1) We continue training FasterDiT up to 2000k iterations. (2) We additionally report the generation results without CFG (classifier-free guidance). The results indicate that FasterDiT still demonstrates a significant advantage. Please refer to the **Response to All Reviewers** for details.\n\n2. All experiments in Figure 1 are conducted on ImageNet and trained for 100k iterations. This setting stems from our observation of the training process that the trend of the training FID at 100k is similar to the trend of the training FID over a longer period of time. For example, the table below shows one set of our experiments on ImageNet. It shows that the trend of FID-50k at 100k is similar to the trend of FID-50k at 200k. \n\n    | Type | Std |FID-50 (100k) | FID-50k (200k) |\n    |:------:|:------:|:------:|:------:|\n    |DDPM cosine schedule| 0.5 | 43.22 | 22.08 |\n    |  | 0.6 | 61.15 | 25.70 |\n    |  | 0.7 | 64.18 | 25.96 |\n    |  | 0.8 | 72.39 | 28.94 |\n    |  | 0.9 | 78.45 | 30.45 |\n    |  | 1.0 | 98.91 | 42.41 |\n    |  | 1.1 | 81.26 | 38.10 |\n    |  | 1.2 | 111.83 | 42.44 |\n\n$\\color{red}{Question2:}$ I am still unclear how you change the std of data without changing dataset. Please specify this point.\n\n$\\color{blue}{Response2:}$ The $std$ here refers to the standard deviation of the input. We can achieve the scaling of the standard deviation simply by multiplying a scale factor.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yjn3Fi6Wiq", "reviewer": "Reviewer_FhtE", "summary": "The paper presents FasterDiT, a method intended to speed up the training of Diffusion Transformers (DiT) without making changes to the architecture. By utilizing insights from the Probability Density Function (PDF) of Signal-to-Noise Ratio (SNR), FasterDiT improves training strategies and the effectiveness of supervision. The paper also includes a thorough set of experiments and a new supervision approach called FasterDiT.", "questions": "No questions.", "limitations": "A more extensive discussion regarding the drawbacks of the proposed approach is needed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces an innovative approach to training strategies by considering the Probability Density Function (PDF) of Signal-to-Noise Ratio (SNR), aiming to improve training efficiency. It includes a thorough empirical analysis with significative experimental results, providing robust evidence to support the findings. FasterDiT achieves significant acceleration in training Diffusion Transformers with a notable increase in training speed, making it a relevant and practical solution for enhancing the efficiency of training large-scale generative models.", "weaknesses": "The paper would benefit from additional experiments to demonstrate the generalizability of the proposed approach. A more comprehensive comparison with existing methods for accelerating training of Diffusion Transformers would provide a broader context for evaluating the effectiveness of FasterDiT. Additionally, the scalability of FasterDiT to larger datasets or more complex tasks is not sufficiently discussed, potentially limiting its applicability. The absence of theoretical results and proofs in the paper limits the depth of understanding of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\color{red}{Question1:}$ The paper would benefit from additional experiments to demonstrate the generalizability of the proposed approach. Additionally, the scalability of FasterDiT to larger datasets or more complex tasks is not sufficiently discussed, potentially limiting its applicability.\n\n$\\color{blue}{Response1:}$ Thanks for your suggestion. As requested, we have conducted three types of additional experiments to further explore the generalization of FasterDiT. Please refer to **Response to All Reviewers** for details.\n\n$\\color{red}{Question2:}$ A more comprehensive comparison with existing methods for accelerating training of Diffusion Transformers would provide a broader context for evaluating the effectiveness of FasterDiT.\n\n$\\color{blue}{Response2:}$ Previous well-known acceleration methods, such as MaskDiT [1], MDT [2], and MDTv2 [3], have achieved significant performance improvements by **modifying architectures** and redesigning the training pipelines of DiT. In contrast, our work approaches the problem from a different perspective. Our aim is to improve DiT training with minimal modifications and **without any changes to the architecture**. This approach is complementary to previous methods. In the future, we plan to explore how combining FasterDiT with more structured improvements can achieve even more efficient generation performance.\n\n$\\color{red}{Question3:}$ The absence of theoretical results and proofs in the paper limits the depth of understanding of the proposed method.\n\n$\\color{blue}{Response3:}$ Thank you for your suggestions. Our work is to analyze the relationship between SNR PDFs and generation performance through a large number of experiments. We try to derive empirical results and insights to accelerate diffusion training. We will further explore this issue from a theoretical perspective in the future.\n\n$\\color{red}{Limitations:}$ A more extensive discussion regarding the drawbacks of the proposed approach is needed.\n\n$\\color{blue}{Response:}$ The main limitation of this paper lies in the lack of exploration of large-scale experiments, such as 2K high-resolution images, text-to-image generation, and video generation. Among these, we particularly focus on the text-to-image generation aspect. Specifically, in the class-conditional generation described in this paper, the DiT block only needs to process image tokens. However, in some text-to-image architectures, such as SD3 [4], self-attention needs to handle sequences combining text and visual tokens. The features from different sources (such as T5 and VAE) may lead to potential instability. We plan to further investigate this issue in the future. Besides, we will update our section on 'Limitations' in the next version of our paper. Thanks for your suggestion.\n\n*Refs:*\n\n[1] Zheng H, Nie W, Vahdat A, et al. Fast training of diffusion models with masked transformers[J]. arXiv preprint arXiv:2306.09305, 2023.\n\n[2] Gao S, Zhou P, Cheng M M, et al. Masked diffusion transformer is a strong image synthesizer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 23164-23173.\n\n[3] Gao S, Zhou P, Cheng M M, et al. MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer[J]. arXiv preprint arXiv:2303.14389, 2023.\n\n[4] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]//Forty-first International Conference on Machine Learning. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Response to All Reviewers

We appreciate all of your valuable feedback. Your valuable suggestions have significantly improved our manuscript.

All reviewers have suggested that we conduct additional experiments to demonstrate the effectiveness and generalization ability of our method. As requested, we have added the following **three types of additional experiments**: 
1. Training FasterDiT with longer iterations and performance without CFG (R_dNCD). 
2. Training FasterDiT with higher resolution (R_fAgp, R_qPr3) 
3. Applying the FasterDiT strategy to different diffusion models (R_ttWn, R_FhtE). 

Due to the limited time available for the rebuttal, we have conducted experiments to the maximum extent that our resources allow. The larger scale experiments mentioned by the reviewers, such as 1024 resolution, text-to-image generation ,and video generation, cannot be completed within this short period. We plan to explore these in future work. Thank you for all the suggestions.

### 1. Longer Training of FasterDiT:

We further improve the performance of FasterDiT with longer training iterations and report more results of FID-50k on ImageNet 256 resolution. 

It shows that even without any structural modifications, FasterDiT could achieve comparable results to the original DiT at 1000k iterations. Furthermore, when the training period is extended to 2000k iterations, FasterDiT's performance further improves, achieving an FID-50k score of 2.03, demonstrating the effectiveness of our approach.

**Performance without Classifier-Free Guidance (CFG)**

| Method     | Models     | Training Samples     | FID-50k   |
|:--------|:--------:|:--------:|:--------:|
| DiT     | DiT-XL/2 | 7000k x 256 | 9.60 |
| SiT    | DiT-XL/2 | 7000k x 256 | 8.60 |
| FasterDiT    | DiT-XL/2 | 1000k x 256 | 8.72 |
|     |  | 1500k x 256 | 8.22 |
|     |  | **2000k x 256** | **7.91** |

**Performance with Classifier-Free Guidance (CFG)**

| Method     | Models     | Training Samples     | FID-50k   |
|:--------|:--------:|:--------:|:--------:|
| DiT (*cfg=1.5*)     | DiT-XL/2 | 7000k x 256 | 2.27 |
| SiT (*cfg=1.5*)     | DiT-XL/2 | 7000k x 256 | 2.06 |
| FasterDiT (*cfg=1.5*)    | DiT-XL/2 | 1000k x 256 | 2.30 |
|     |  | 1500k x 256 | 2.12 |
|     |  | **2000k x 256** | **2.03** |

### 2. Higher Resolution:

As requested, we explore FasterDiT on higher resolution generation experiments to demonstrate its generalization ability. Specifically, we apply our method to DiT-B/2 and DiT-L/2 for ImageNet 512x512 generation training. The results are shown in the table below. 

FasterDiT achieves faster convergence across all experiments. When trained with 200k iterations, we improve the FID-10k performance of DiT-B/2 by 18.78 and DiT-L/2 by 17.93. This demonstrates the effectiveness of our method in high-resolution tasks.

| Method | Models | Training Samples | Resolution | FID-10k   |
|:--------|:--------:|:--------:|:--------:|:--------|
| DiT     | DiT-B/2 | 100k x 128 | 512x512 | 93.36 |
|      |  | 200k x 128 | 512x512 | 77.11 |
| FasterDiT     | DiT-B/2 | 100k x 128 | 512x512 | 77.85 (-11.51)  |
|      |  | 200k x 128 | 512x512 | 58.33 **(-18.78)** |
| DiT     | DiT-L/2 | 100k x 64 | 512x512 | 87.24 |
|      |  | 200k x 64 | 512x512 | 67.29 |
| FasterDiT     | DiT-L/2 | 100k x 64 | 512x512 | 71.58 (-15.66)  |
|      |  | 200k x 64 | 512x512 | 49.36 **(-17.93)** |


### 3. Different Architectures:

As requested, we explore our training method with other diffusion models besides DiT, such as Latent Diffusion Models (UNet architecture) [1] and U-ViT [2]. The results are shown in the table below. 


It shows that with our training methods, the performance of U-ViT and UNet both improve. It demonstrates that our method might have opportunities to be generalized to more diffusion architectures. 


| Model | Training Samples | FID-10k |
|:--------|:--------:|:--------:|
| U-ViT-L |  200k x 128 | 50.22 |
| U-ViT-L + Ours |  200k x 128 | **37.12** |
| UNet | 200k x 128 | 66.73 |
| UNet + Ours | 200k x 128 | **60.07** 


*Refs:*

[1] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[2] Bao F, Nie S, Xue K, et al. All are worth words: A vit backbone for diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 22669-22679.

Meta-review: The paper introduces an innovative approach to accelerate the training of Diffusion Transformers (DiT). The authors present a thorough empirical analysis, validating their method across various experiments and demonstrating competitive performance on ImageNet, while being significantly faster than the traditional DiT. The reviewers were unanimous in the decision to accept the paper and the AC agrees with the decision. The AC recommends that the authors put extra effort in polishing the paper, as well as adding the additional experiments provided in rebuttal.
Decision: Accept (poster)