FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification

Authors: jingfeng  yao
cheng  wang
wenyu  liu
xinggang  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 10:46:04

Abstract: Diffusion Transformers (DiT) have attracted significant attention in research. However, they suffer from a slow convergence rate. In this paper, we aim to accelerate DiT training without any architectural modification. We identify the following issues in the training process: firstly, certain training strategies do not consistently perform well across different data. Secondly, the effectiveness of supervision at specific timesteps is limited. In response, we propose the following contributions: (1) We introduce a new perspective for interpreting the failure of the strategies. Specifically, we slightly extend the definition of Signal-to-Noise Ratio (SNR) and suggest observing the Probability Density Function (PDF) of SNR to understand the essence of the data robustness of the strategy. (2) We conduct numerous experiments and report over one hundred experimental results to empirically summarize a unified accelerating strategy from the perspective of PDF. (3) We develop a new supervision method that further accelerates the training process of DiT. Based on them, we propose FasterDiT, an exceedingly simple and practicable design strategy. With few lines of code modifications, it achieves 2.30 FID on ImageNet at 256x256 resolution with 1000 iterations, which is comparable to DiT (2.27 FID) but 7 times faster in training.

Main paper: 
 introduction: 
 With the advent of Sora [36] , its foundational model, Diffusion Transformers (DiT) [37] , has ignited extensive research interest. DiT is characterized by its remarkable flexibility and scalability, demonstrating exceptional capabilities in both image [37, 32, 16, 9, 7, 8] and video generation [31, 33, 17] . However, similar to the challenges faced with Vision Transformers [15] , DiT is associated with high training costs. Its convergence rate remains slow, necessitating over 4700 GPU hours training on ImageNet generation tasks at 256 resolution 2 . This significant computational demand highlights the need for enhancing training efficiency for large-scale training.One effective method for improving training is to modulate the Signal-to-Noise Ratio (SNR) distribution across different timesteps during training. Denoising generative models [22, 30] create a conversion from noise to data and methodically transfer noise into data as the timestep, denoted t, progresses. In this conversion process, the SNR gradually increases from zero to infinity. Given a generative process x t = α t x ⋆ + σ t ϵ. Assuming that the input data x ⋆ are ideally normally distributed with a variance of one, the SNR is typically defined as the ratio of variances [24, 14, 34, 40, 20] . During one training step, for a pair of input data and noise, we randomly select one t for training. Modulating noise scheduling [35, 25, 10] , loss weight [34, 20] , and timestep sampling strategy [16] Same strategy but opposite trend?α 2 t σ 2 tDiT [37] DiT-XL-2 400k 19.5 SiT [32] 400k 17.2 FasterDiT (Ours) 400k 11.9 (-5.3)SiT [32] However, these methods do not always work across different data. For example, Stable Diffusion 3 (SD3) [16] proposes a timestep sampling strategy by logit normal function (lognorm) [1] for better training. Lognorm sampling is an excellent pioneering strategy and has been proven effective in subsequent work [17] . But it also has some limitations. According to SD3 exploration, lognorm(-0.5, 1) substantially outperforms lognorm(0.5, 0.6) in terms of Frechet Inception Distance (FID) results on ImageNet [13] . Nevertheless, we discover that this conclusion is a particular solution specific to the training data used. As shown in Figure 1 , as we continuously reduce the signal strength, the FID results for lognorm(-0.5, 1) progressively worsen, eventually becoming the poorest outcome. This underscores the necessity of evaluating the robustness of such methods across varying data conditions.In this paper, we aim to provide a more comprehensive perspective for interpreting the problem. Our first contribution is to suggest interpreting the performance robustness with the perspective of the Probability Density Function (PDF) of SNR during training. Specifically, these methods typically modulate the distribution of SNR based on the timestep t. Although the same timestep has the same relative SNR asα 2 t σ 2 t, the absolute SNR at the step actually increases with the enhancement of the data signal [25, 10] . We believe that the original definition of SNR only reflects the relative signal-to-noise ratio in the same data distribution. However, the distribution of data-related absolute SNR during the training process is the key factor determining the effectiveness of the training. Hence, we slightly extend the definition of SNR and visualize its probability density function.Our second contribution is to conduct extensive experiments and report about one hundred experiment results in our paper to empirically analyze the association between training performance and robustness with PDF. We analyze the differences in data robustness among commonly used basic pipelines on DiT [37, 32] . We find there is a trade-off between method robustness and performance, and suggest that the training process can be designed more intuitively from a PDF perspective.The other method to improve training is to modify single-timestep prediction or supervision [34] . Our third contribution is the introduction of a new supervision method for velocity predictionbased approaches. With the same schedule, different prediction targets [34] or different supervised methods [22, 14] may also lead to different training results. Recently, velocity has been demonstrated to be a superior predictive target. Specifically, in addition to the traditional Mean Squared Error (MSE) loss, we have incorporated supervision of the velocity direction. We have found that this method is conceptually simple and significantly accelerates the training process.To sum up, in this paper, we propose a new perspective on training the SNR PDF to offer a straightforward interpretation of the efficiency in training generative models (Section 2). We conduct extensive experiments and report near one hundred experimental results, aiming to empirically derive insights into training performance, robustness, and their correlations with the SNR PDF (Section 3). Subsequently, we apply these observations to refine the DiT process and, together with introducing a novel supervision method, developed FasterDiT to significantly enhance training efficiency (Section 4). FasterDiT achieves an FID of 2.30 on ImageNet at a resolution of 256, comparable to the original DiT's FID of 2.27, yet achieves convergence seven times faster. We hope that our explorations contribute valuable insights to future research in generative model training. 
probability density function of snr during training: 
 We hypothesize that the distribution of attention to different Signal-to-Noise Ratios (SNRs) during training is a crucial determinant of training efficiency and effectiveness. Here, two issues need to be addressed. Firstly, the previous definition of SNR asα 2 t σ 2 tignore the influence of data signal intensity, which is proven to be important during training [25, 35] . Secondly, a unified and intuitive approach is required to analyze the training SNR distribution. Therefore, in this section, we propose a slight modification to the existing definition of SNR. Subsequently, we utilize the Probability Density Function (PDF) of SNR during training to integrate noise scheduling, loss weighting, and timestep sampling strategies into a cohesive framework.For ease of comprehension, we give a brief introduction to the formulation and training pipeline of generative models [22, 35, 29, 30] . Since our focus is on the SNR distribution, we universally consider flow matching and the diffusion model from a high-level perspective, similar to previous work [16] . Given data x ⋆ ∼ X and Gaussian noise ϵ ∼ N (0, 1), we define the transport between noise and data as equation 1. In a discrete diffusion process [22, 35] , t is an integer from 0 to 1000. In a continuous flow process [29, 30] , t is a continuous value between [0, 1]. α t and σ t is are coefficients related to t defined differently according to schedules.EQUATIONDuring the training process, noised data x t and timestep t are input to the generative model. The model is required to predict the specific target (noise, velocity, and so on). For example, in the most commonly used DDPM [22] pipeline, the prediction target is noise ϵ and the loss function is defined as equation 2. Only a single x t will be trained in each iteration for one image, instead of the whole timesteps. Therefore, some methods choose to change the sampling of t [16] or give different loss weights [20, 35] EQUATION2.2 Formulate Probability Density Function of SNR Data-dependent SNR Previous work [24, 14, 34, 40, 20] highlights the importance of SNR. Typically, they assume that the data is an ideal normalization distribution and define the SNR as the ratio of variances (Equation 3). This definition is independent of data and only associated with the coefficients in timestep t.SNR prev (t) = α 2 t σ 2 t (3)However, the distribution of actual data is indescribable. It is closely linked to the nature of the image itself. In this study, we introduce a coefficient, K(I), associated with the image I that directly scales the signal. The value of K(I) is influenced by various image characteristics such as the range of high and low frequencies, resolution, variance, and so on. It is known that the larger the variance of an image, the higher its SNR, illustrated as Equation 4. Further, for a specific dataset, we assume that the change in variance has little effect on the other nature of the image, i.e. std 2 is an independent variant in K(I), and approximate K(I) std 2 as a constant C(I). 5) is that we can use the same dataset with different std to simulate different C(I) variations. This simplifies the scheme of probing the robustness of the data and allows us to not need to change the training data repeatedly.EQUATIONEQUATION)Weightings Consider a training process with the timestep sampling function f s (t) and loss weight function f l (t). For most cases, f s (t) and f l (t) are non-negative. To unify these into a single distribution of t, we define a new probability density function f t (t) as:EQUATIONPDF of SNR Now, we try to get the probability density function of SNR during training. It is independent with timestep t. Assume distribution SN R(t) ∼ Y and t follows the distribution of f t (t).Solving for the distribution of Y can be transformed into a problem of probability transformation. Mathematically, it could be defined as Equation 7. 3 What can we learn from SNR PDF?EQUATIONBack to the problem we discuss in Section 1 Figrue 1. We argue that strategies that are solely based on t tend to focus only on relatively high or low noise conditions in the generation process but are limited to considering the absolute Signal-to-Noise Ratio (SNR) with different signal intensity. They provide awesome solutions to improve the training but lack a comprehensive evaluation. In this section, we will explore this issue from the perspective of the Probability Density Function (PDF) of SNR, aiming to identify commonalities in these strategies and to delineate a more comprehensive design space. 
experiment settings: 
 We introduce our experiment settings first. We mainly adopt the Diffusion Transformer (DiT) training pipeline from two previous important works, DiT [37] and SiT [32] . We choose four different noise scheduling from them, including DDPM with linear beta schedule [22] , DDPM with cosine beta schedule [35] , flow matching with linear schedule (Rectified Flow) [30] , and flow matching with cosine schedule [32] . In the subsequent experiments, we initially set the prediction target of each noise schedule to noise (epsilon) to ensure a fair comparison. Subsequently, we modify the input strength by scaling the data to various standard deviations (std). We then shift the prediction target of flow matching to velocity the same as its original setting, to compare the similarities and differences between speed prediction and noise prediction outcomes. Each experiment was conducted on ImageNet [13] at a resolution of 128. We train each model for 100,000 iterations and assess their performance using the FID-10k metric for comparative analysis. Each experiment has been conducted with 8 H800 GPUs.Our primary observational objectives are twofold: (1) Performance and robustness under different signal intensities. (2) Potential connections between the state of the PDF and performance and robustness.Different data signal intensities lead to different training effects. In Figure 2 , the prediction target for all schedules is uniformly set to noise to ensure a fair comparison. The curves demonstrate how different input intensities impact training outcomes. Notably, the performance of a single noise schedule fluctuates with changes in data intensity. For instance, the DDPM with a linear beta scheduler achieves an FID of 65.38 at std = 0.3 and improves significantly to 47.04 at std = 2.5 (Figure 2a ).Different schedules exhibit significant differences in data robustness. In Figure 2 , as the intensity of the data varies, distinct schedules demonstrate significantly different robustness profiles. Specifically, with increasing standard deviation, the range of the FID for the DDPM-linear [22] schedule is observed at 18.03 (Figure 2a ). In contrast, the DDPM-cosine [35] schedule exhibits a considerably larger FID range of 50.02(Figure 2c ), highlighting substantial variability in performance between schedules.Schedule with a broad SNR focus could have a robust performance. In Figure 3 , we present the SNR PDFs as outlined in Section 2. For each noise schedule, we display three PDF curves corresponding to different signal intensities: low (std = 0.3), high (std = 3.3), and the level that results in the optimal FID. These PDF curves visually represent the training process's focus Figure 5 : Flow Matching with v-prediction. We evaluate the robustness of commonly used flow matching [32] with v-prediction on ImageNet. ( 1) We find flow matching with v-prediction gets a more robust performance than noise-prediction. ( 2) There still exists a trade-off between performance and robustness.across various noise interpeaksvals. Empirical analysis of the curve shapes across different noise schedules reveals that greater variance in the PDF, indicating a broader consideration of noise intervals (Figure 3a&b ), is associated with improved stability in the schedule.The optimal SNR ranges for different schedules seem to be similar. Direct observation of the standard deviations across different schedules does not readily lead to consistent conclusions. For instance, in Figure 2 , while the DDPM-Linear schedule (Figure 2a ) achieves optimal generative performance at an std of approximately 2.5, the DDPM-Cosine schedule (Figure 2c ) peaks at an std of 0.9. However, insights might be gleaned from analyzing the PDF curves. In our experiments, as depicted in Figure 3 , with a unified prediction target, better FID performance correlates with the mean of the PDF falling within the designated gray area. This observation offers a novel perspective and enriches our understanding of the previously established results.There is a trade-off between performance and robustness. The primary purpose of employing weighting during the training process is to intensify the focus on specific SNR levels. We implement a logit normal function (lognorm) [1] to adjust the timestep sampling for two distinct schedules (Figure 2a&b ), with results displayed in Figure 4 . Our analysis reveals that while a well-defined sampling strategy can enhance performance, it may also introduce risks. For instance, the DDPM-Linear schedule achieves an optimal FID of 47.04, demonstrating robust performance across a range of 18.02. However, when we intensify the focus using lognorm (Figure 4a ), although the best FID improves to 36.9, it also results in more variable outcomes, with a performance range widening to 80 (Figure 4b ).Flow matching with v-prediction gets more robust performance. Recently, flow matching [29, 30] has been regarded as a more concise and efficient pipeline for generative models. In the previous discussion, to facilitate a fair comparison, we set the prediction target on noise prediction. In Figure 5 , We find flow matching with v-prediction achieves a more stable output performance, demonstrating enhanced robustness in data stability. For instance, Linear Flow exhibits a range in FID of 17.54 when using noise prediction, compared to 11.79 when using v-prediction (Figure 5a ). This characteristic is even more pronounced in Flow with cosine schedule, where the ranges are 34.76 and 12.39, respectively (Figure 5b ). Additionally, we find that the overall performance of v-prediction generally surpasses that of noise prediction. The trade-off still exists in flow matching with v-prediction. Furthermore, we observe that although v-prediction is generally more robust, there remains a trade-off between performance and stability. As shown in Figure 5c&d , when we used a lognorm to concentrate the focus of Linear-Flow, its performance resembled that of noise prediction, where the upper limit improved but stability decreased. We hypothesize that such trade-offs may be broadly prevalent.To sum up, from the perspective of the PDF, efficient training requires satisfying two conditions: (1) the PDF should have a concentrated focus; (2) the focus area of SNR needs to fall within the correct range. Based on the simple observation, we try to improve DiT training.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: The paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
improving dit training: 
 In this paper, our objective is not to devise novel model architectures for achieving state-of-the-art outcomes. Instead, we aim to explore a simpler, more interpretable, and universally applicable training approach for Diffusion Transformers (DiTs) [37] .In Section 3, we demonstrate a trade-off between performance and robustness in the DiT training process via signal-to-noise (SNR) probability density function (PDF) analysis. To expedite training, we should have a concentrated PDF that focuses on the right SNR during training. Initially, we select flow matching [30] with v-prediction as our noise schedule, owing to its robustness and superior performance. We then adjust the PDF to focus on the optimal SNR during training by modulating the standard deviation (std) of training data. Further, we employ a logit normal function [1] to sharpen the focus on the adjusted regions. Finally, we introduce a new, straightforward supervisory strategy that significantly enhances training outcomes.Here, we first modulate the standard deviation from 0.5 to 1.2 as illustrated in Figure 7 (left). During this sweep, we refrain from employing additional techniques such as weighting. It shows that when std is around 0.70, the DiT training gets a better generative performance. Empirically, we set the target std to 0.82, which is near the proper and stays consistent with previous work's setting [32, 37] . However, we argue that when the input data changes, e.g. different resolution, the choice of std will differ. The strategy is similar to previous work [10] , but we discuss it in a new perspective of training SNR PDF.Then we improve the performance by a human-made concentration for PDF. Specifically, we use logit normal function [1] for timestep sampling, while mu and sigma are set to 0 and 1 respectively. Notably, mu is set to 0 because the PDF is already properly shifted (see Figure 3 ). It could help avoid instability discussed in Section 1 and improve the training performance (see Figure 7 (left) (b)).Effective single-step supervision is essential in training generative models. For instance, employing different prediction targets can result in diverse predictive outcomes [34] . In the original DiT model, a strategy that simultaneously predicts noise and sigma is utilized to maximize performance [37] . Typically, the Mean Squared Error (MSE) loss function is employed for supervising these targets.EQUATIONRecently, in the context of flow matching [29, 30] , the prediction target of velocity acquires a more specific physical significance, representing the flow rate from noise to data. Based on this, we hypothesize that supervising the direction of velocity could serve as an effective supervision strategy. Consequently, in this paper, we not only use the Mean Squared Error (MSE) to supervise velocity predictions but also apply cosine similarity to further supervise the directionality of velocity. Here we present velocity direction loss as shown in Equation 8, while H, W denotes the height and width of predicted latent by DiT. We demonstrate that this combined supervisory approach significantly enhances the model's convergence rate (see Figure 7 (left) (b))Owing to resource limitations, we train FasterDiT on ImageNet at 256 resolution [13] for 2,000k iterations and benchmark it against current advanced generative models (see Table 1 ). Our experiment has been conducted with 8 H800 GPUs. Notably, the focus of our work is to explore training strategies beyond structural enhancements. Thus, FasterDiT utilizes the identical architecture as DiT [32] . As illustrated in Section 1 Figure 1 , FasterDiT achieves an FID-50k score of 11.9 at 400k iterations, markedly outperforming the original DiT model (FID-50K 19.5), and its enhanced counterpart, SiT (FID-50k 17.2) [32] . Utilizing the same architecture as DiT [37] , FasterDiT attains a similar performance level with an FID score of 2.30 but converges in just 1,000k iterations.We evaluate FasterDiT on higher resolution generation tasks to explore its resolution generalization capabilities. Specifically, we apply our approach to DiT-B/2 and DiT-L/2 models for ImageNet generation at a 512×512 resolution. The results, presented in Table 2 , indicate that FasterDiT consistently achieves faster convergence across all configurations. Notably, after 200k training iterations, our method improves the FID-10k performance of DiT-B/2 by 18.78 and DiT-L/2 by 17.93, underscoring its effectiveness for high-resolution image generation tasks.We further apply our training method to other diffusion models beyond DiT, including Latent Diffusion Models [40] (using the UNet [41] architecture) and U-ViT [2] . Here, we specifically refer to the use of the previously mentioned multi-step balance and velocity direction loss. The results, presented in Table 3 , indicate performance improvements for both U-ViT and UNet with our method. This suggests that our approach has the potential to generalize across a broader range of diffusion model architectures.5 Related WorkQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide as many details as possible, such as algorithms and pseudo codes, to ensure reproducibility. Besides, we also provide our source codes. 
generative models and diffusion transformers: 
 The denoising diffusion probabilistic model [22] has gradually replaced GANs [12, 19] due to its stronger performance and more stable training, becoming the mainstream generative model. Among them, [14] first proves its effectiveness on ImageNet [13] . [40, 38] extend the diffusion process from the diffusion space to the space of VAE latent space, achieving high-performance, high-resolution generation. Meanwhile, some methods [32, 46, 17] have also trained generative models using flow-matching [29, 30] techniques to replace the diffusion path. These methods exhibit simpler mathematical properties and faster learning effects.Among them, U-Net [41] is the most commonly used architecture for generative models. Recently, due to the scalability advantages of transformers [15, 44] , some transformer-based [2, 37] generative models have also emerged. However, similar to Vision Transformers [15] , Diffusion Transformers [37] converge slowly.Work for accelerating Diffusion Transformers [37] (DiT) could be simply divided into two categories.One is architecture modification. MaskDiT [45] and MDT [18] combine mask image modeling [3, 21] pre-training and diffusion training for speeding up. Similarly, SD-DiT [46] takes one step further to combine DiT training with MoCo-like [11] contrastive learning. CAN [5] proposes a dynamic weights for condition to speed up training of diffusion models [37, 2] .Another strategy involves changing non-model design approaches. For example, using different noise schedules can achieve better results on images of various resolutions [25, 35, 10, 27] . Using different prediction targets, such as noise, data, or velocity, can also directly impact the effectiveness of the training [34] . Adjusting weights for the loss function and training sampling can also directly affect the outcomes of training [20, 16] . However, they constitute a vast design space with a high degree of direct interdependence, which makes the design of the model very complex.Question: Does the paper provide open access to the data (we use the public dataset in our experiments) and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Open access to the data and code is provided in supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 Limitations We propose a novel approach called FasterDiT for accelerating the training of diffusion transformers. The main limitation of this paper lies in the lack of exploration of large-scale experiments, such as 2K high-resolution images, text-to-image generation, and video generation. Among these, we particularly focus on the text-to-image generation aspect. Specifically, in the class-conditional generation described in this paper, the DiT block only needs to process image tokens. However, in some text-to-image architectures, such as SD3 [16] , self-attention needs to handle sequences combining text and visual tokens. The features from different sources (such as T5 [39] and VAE [28] ) may lead to potential instability. We plan to further investigate this issue in the future.Societal Impacts Our work has the potential to significantly enhance the efficiency of creative professionals in completing their creative processes. However, it also carries a risk of misuse.Conclusions In this paper, we discuss training strategies to accelerate Diffusion Transformers. Initially, we slightly generalize the definition of SNR and conduct a unified analysis of various training strategies through the examination of SNR probability density functions during the training process. We find that there is a trade-off between robustness and performance in the training of diffusion models. Different noise schedules exhibit varying levels of robustness. Using weighting during the training can enhance the performance ceiling, but it may also decrease the robustness of the process. Additionally, we discover that utilizing a directional loss as an auxiliary loss function for velocity prediction significantly boosts training performance. Based on these observations, we conduct experiments with DiT-XL-2 on ImageNet 256×256 and observe substantial acceleration in training. We refer to this straightforward training approach as FasterDiT. We hope our work inspires further exploration into training strategies for generative models.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All the training and test details are specified in the paper and supplemental material.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not report error bars due to limited resources.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Sufficient information on the computer resources for each experiment are provided in the paper and supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses both potential positive societal impacts and negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we offer a detailed overview of the experimental methodologies employed in our study. The investigation is delineated into two primary segments. Initially, we conduct thorough experiments to explore the potential relationship between training efficiency and effectiveness as detailed in Section 3. Subsequently, we extend our findings to larger training configurations of Diffusion Transformers to ascertain the efficacy of our methodologies, as discussed in Section 4.The specific details of the training processes are delineated in Table 4 and Table 5 In this section, we detail the sampling procedures as discussed in Chapters 3 and 4. The specific outcomes are presented in Table 6 and Table 7 . A key distinction between the two sections is as follows: In Section 3, we assessed the FID of 10,000 generated images without employing the cfg operation. Conversely, in Section 4, we evaluated the FID of 50,000 generated images and adjusted the cfg to 1.5, aligning with the methodologies of prior studies [37, 32] .The focus of this paper is on the accelerated training of generative models, with a particular emphasis on Diffusion Transformers [37] . The design of the generative process presented herein draws inspiration from the pioneering work on DiT [37] and the subsequent advancements in SiT [32] . The former primarily adopts the DDPM generation pipeline [22] , while the latter enhances DiT training by integrating score-based diffusion [43] and flow matching.In Section 3, we employ four distinct noise schedules, encompassing both diffusion and flow matching. Typically, these can be represented as x t = α t x ⋆ + σ t ϵ. Specifically, the initial two schedules (Figure 2a&b ) employ the widely utilized DDPM linear schedule [22] and the DDPM cosine schedule [35] , respectively. Owing to the complexity of these concepts, a detailed discussion is beyond the scope of this paper; readers are encouraged to consult the original publications for more comprehensive information. Regarding the last two (Figure 2c&d ), we utilize the two most common flows, as delineated in Equations 9 and 10.EQUATIONIn this section, we present more images generated by FasterDiT, which underwent training on the ImageNet dataset at a resolution of 256 for 1,000,000 iterations. The visualization results, depicted in Figure 9&10 , feature samples from seven distinct ImageNet categories, emphasizing selected outputs.During the testing phase, the cf g value was consistently set at 4, in accordance with parameters established in previous research. Remarkably, despite a substantially shorter training period compared to DiT, FasterDiT has successfully generated visually impressive results. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduces an innovative approach to accelerate the training of Diffusion Transformers (DiT). The authors present a thorough empirical analysis, validating their method across various experiments and demonstrating competitive performance on ImageNet, while being significantly faster than the traditional DiT. The reviewers were unanimous in the decision to accept the paper and the AC agrees with the decision. The AC recommends that the authors put extra effort in polishing the paper, as well as adding the additional experiments provided in rebuttal.

Reviews: review: paper_summary: paper_summary: This paper presents observations on training strategies for Diffusion Transformers (DiT) using Signal-to-Noise Ratio (SNR) Probability Density Function (PDF) analysis. Through extensive experiments, the authors derive insights into training performance and robustness. Based on these findings, they propose a method to accelerate the training process of DiT without modifying the model architecture.
strength_weakness: strength_weakness: 1. The paper lacks experiments with higher resolutions such as 512x512 or 1024x1024, which could provide insights into the method's scalability for more complex image generation tasks.
2. The hyperparameter 'std' requires tuning to find the appropriate setting, which may introduce additional complexity in implementing the method across different datasets or tasks.
questions: questions: Why does adding cosine similarity loss for velocity direction supervision accelerate the convergence rate of the model?
limitations: limitations: The scalability of this method is not thoroughly evaluated, including its effectiveness for higher resolutions, other datasets, or different types of generative tasks (e.g., text-to-image, video generation).
paper_summary: paper_summary: The paper focuses on accelerating the training process of Diffusion Transformers (DiT) without modifying their architecture. The authors identify two primary issues: inconsistent performance of certain training strategies across different datasets, and limited effectiveness of supervision at specific timesteps.

Key contributions include:
1. Extended Definition of Signal-to-Noise Ratio (SNR)
2. Extensive experiments and empirical findings for SNR PDF
3. A new supervision method
strength_weakness: strength_weakness: - The experiments were conducted on 256-resolution ImageNet only. It would be interesting to validate the proposed method on larger resolutions (such as 512, 1024, etc.). Acceleration is more critical to those scenarios.
- The figures could be improved. The rendered text is not easy to read.
questions: questions: see weakness
limitations: limitations: n/a
paper_summary: paper_summary: This paper propose FasterDiT, a diffusion model training method that considers the data distribution in the definition of signal-to-noise ratio. It formulates the SNR in a new framework, estimates the PDF of the SNR, and then employs it to improve the training efficiency of DiT. Experimental results show that FasterDiT achieves relative FID but with 1/7 training time.
strength_weakness: strength_weakness: 1. This paper has a bad quality in writing. There are some typos and the explanation to the SNR is not clear to me. I'm confused for many questions: (1) What does the std means in Line 109. Does it mean the std of the value of pixels in the images?  (2) Why can we assume std^2 approximate K(I)/std^2 as a constant C(I)?  (3) Authors mention "robustness" for many times in the paper for "training robustness", "data robustness". What does it mean in detail? (4) In Figure 6, the caption inside the figure is "multi-step balance", while the caption after the figure is multiple-step balance. Please use the same description. What does multi-step balance mean? Does that mean the new training strategy with SNR?  The "single-step supervision" is also not a good choice here. It should be something like "directionality of velocity" or "single-step supervision (ours)" since there has already been single-step supervision in the traditional training method.
2. Does the proposed method generalize well to the diffusion models besides DiT, such as latent diffusion models? Please discuss on this. If so, experimental results should be provided for this since only experiments on DiT & ImageNet is not very convincing.
questions: questions: 1. Does x_* in the paper indicate the x_0 (real images)? If so, I advise to replace it with x_0 to align with previous works.
2. Section 4: Improving DiT Training. The "." should be removed.
3. The space between line 140 and 141 are excessively reduced.
In summary, I think this paper may have good technical contribution. But it's really difficult for me to understand the details of this paper.
limitations: limitations: n/a
paper_summary: paper_summary: This work aims at solving the slow training convergence of Diffusion Transformer, from the perspective of Signal-to-Noise Ratio (SNR).
Different from other works, the authors formulate the probability density function (PDF) of SNR during training, and then leverage such SNR PDF to analyze the association between training performance and robustness across some common pipelines of DiT.
The finding of the trade-off between method robustness and performance makes authors to propose Fast-DiT which significantly accelerate DiT's training convergence.
strength_weakness: strength_weakness: 1. Current Table 1 compares the CFG  (classifier-free guidance) results.
Please compare the Class-conditional results of Faster-DiT with DiT and SiT, under the same setting in Table 1 (or even more iterations of Faster DiT).
Usually we need both CFG and Class-conditional results to check the training convergence. Currently Figure 1 right with small iterations is not convincing.

2.  I am still unclear how you change the std of data without changing dataset. Please specify this point.
questions: questions: Listed in weakness.
limitations: limitations: The authors has discussed the limations of the best performance of FastDiT without sufficient GPUs.
paper_summary: paper_summary: The paper presents FasterDiT, a method intended to speed up the training of Diffusion Transformers (DiT) without making changes to the architecture. By utilizing insights from the Probability Density Function (PDF) of Signal-to-Noise Ratio (SNR), FasterDiT improves training strategies and the effectiveness of supervision. The paper also includes a thorough set of experiments and a new supervision approach called FasterDiT.
strength_weakness: strength_weakness: The paper would benefit from additional experiments to demonstrate the generalizability of the proposed approach. A more comprehensive comparison with existing methods for accelerating training of Diffusion Transformers would provide a broader context for evaluating the effectiveness of FasterDiT. Additionally, the scalability of FasterDiT to larger datasets or more complex tasks is not sufficiently discussed, potentially limiting its applicability. The absence of theoretical results and proofs in the paper limits the depth of understanding of the proposed method.
questions: questions: No questions.
limitations: limitations: A more extensive discussion regarding the drawbacks of the proposed approach is needed.

Comments: title: Response to author rebuttal
comment
comment: After reading the response from the authors, I decide to increase my rate from 5 to 6. I believe this paper has real novelty in the training of diffusion models and it still can be greatly improved by better writing. Please keep polishing it.
Thank you for your response. These additional experiments address most of my concerns. Based on this, I am willing to upgrade my score.
Thanks for your experiments and answers! These answers have solved my problems.
I will keep my ratings.
Thank you for the experiments. I hope the authors will add the high-res results to the final paper.