A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training

## Abstract

Abstract Training diffusion models is always a computation-intensive task. In this paper, we introduce a novel speed-up method for diffusion model training, called SpeeD , which is based on a closer look at time steps.
Our key findings are:
i) Time steps can be empirically divided into acceleration, deceleration, and convergence areas based on the process increment.
ii) These time steps are imbalanced, with many concentrated in the convergence area.
iii) The concentrated steps provide limited benefits for diffusion training.
To address this, we design an asymmetric sampling strategy that reduces the frequency of steps from the convergence area while increasing the sampling probability for steps from other areas. Additionally, we propose a weighting strategy to emphasize the importance of time steps with rapid-change process increments.
As a plug-and-play and architecture-agnostic approach, SpeeD consistently achieves 3 × \times × acceleration across various diffusion architectures, datasets, and tasks. Notably, due to its simple design, our approach significantly reduces the cost of diffusion model training with minimal overhead.
Our research enables more researchers to train diffusion models at a lower cost.

## 1 Introduction

Training diffusion models is not usually affordable for many researchers, especially for ones in academia. For example, DALL$\cdot$E 2 (OpenAI, ) needs 40K A100 GPU days and Sora (OpenAI, ) at least necessitates 126K H100 GPU days. Therefore, accelerating the training for diffusion models has become urgent for broader generative AI and other computer vision applications.

Recently, some acceleration methods for diffusion training focus on time steps, primarily using re-weighting and re-sampling
1) Re-weighting on the time steps based on heuristic rules.
P2 (Choi et al., ) and Min-SNR (Hang et al., ) use monotonous and single-peak weighting strategies according to sign-to-noise ratios (SNR) in different time steps.
2) Re-sampling the time steps. Log-Normal (Karras et al., ) assigns high sampling probabilities for the middle time steps of the diffusion process.
CLTS (Xu et al., ) proposes a curriculum learning based time step schedule, gradually tuning the sampling probability from uniform to Gaussian by interpolation for acceleration as shown in Fig. 1(b).

To investigate the essence of the above accelerations, we take a closer look at the time steps.
The diffusion models essentially learn and estimate the process increment noted as $\delta_{t}:=x_{t+1}-x_{t}$ at time step $t$ at training and inference phase. As illustrated in the left of Fig. 1(a), we visualize the changes of mean and variance of process increment through the time steps. The time steps are categorized into three areas: acceleration, deceleration, and convergence, based on observations and theoretical analyses (Sec. 2.2) that both the changes of mean and variance initially accelerate, subsequently decelerate, and ultimately converge in a narrow interval.

One can easily find that the proportions of the three areas are imbalanced: there are a large number of time steps at convergence area, and the others are small.
Another finding is that process increments at the convergence area are almost identical noise, e.g., in DDPM, the distribution are nearly $\mathcal{N}(0,2\mathbf{I})$, where $\mathbf{I}$ is the unit matrix.
To further explore the characteristics of these three areas, we visualize the training loss curve in the right of Fig. 1(a).
The loss values from the convergence area are much lower than those from the other areas, which indicates estimating the identical noise is easy.

Previous acceleration works have achieved promising results, but the analysis of time steps remains relatively under-explored. P2 (Choi et al., ) and Min-SNR (Hang et al., ) are two re-weighting methods, with their weighting curves across time steps as shown in Fig. 1(b). They employ uniform sampling of time steps, which include too many easy samples from the convergence area during diffusion model training.

On the other hand,
most re-sampling methods heuristically emphasize sampling the middle-time steps, but they do not dive into the difference between the acceleration and convergence areas.
For example, CLTS (Xu et al., ) gradually changes the sampling distribution from uniform to Gaussian by interpolation as shown in Fig. 1(b).
The sampling probability of the acceleration area drops faster than the one of the convergence area.
The acceleration area is still under-sampled and therefore not well-learned.

Figure: (a)
Refer to caption: x1.png

Figure: Figure 2: Core designs of SpeeD. Red and blue lines denote sampling and weighting curves.
Refer to caption: x3.png

Motivated by the
analyses from a closer look at time steps, we propose SpeeD , a novel approach that aims to
improve the training efficiency for diffusion models.
The core ideas are illustrated in Fig. 2.
To mitigate the redundant training cost, different from uniform sampling, we design an asymmetric sampling strategy that suppresses the attendance of the time steps from the convergence area in each iteration.
Meanwhile, we weight the time steps by the change rate of the process increment, emphasizing the importance of the rapid-change intervals.

Our approach has the following characteristics:
SpeeD is compatible with various diffusion model training methods, i.e., U-Net (Ronneberger et al., ) and DiT (Peebles & Xie, ) with minimal modifications.
For performance, SpeeD achieves non-trivial improvements than baseline and other methods at the same training iterations.
For efficiency, SpeeD consistently accelerates the diffusion training by 3$\times$ across various tasks and datasets. It helps mitigate the heavy computational cost for diffusion model training,
enabling more researchers to train a model at an acceptable expense.
The extra time complexity of SpeeD is $\mathcal{O}(1)$,
costing only seconds to reduce days of diffusion models training on datasets like FFHQ, MetFaces and ImageNet-1K. We hope this work can bring novel insights for efficient diffusion model training.

## 2 Speeding Up Training from Process Increments at Time Steps

In this section, we first introduce the preliminaries of diffusion models and then focus on a closer look at time steps and key designs of our proposed SpeeD .

### 2.1 Preliminaries of Diffusion Models

In conventional DDPM (Ho et al., ; Sohl-Dickstein et al., ), given data $x_{0}\sim p(x_{0})$, the forward process is a Markov-Gaussian process that gradually adds noise to obtain a perturbed sequence {$x_{1},x_{2},...,x_{T}$},

$$ $q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathrm{ \textbf{I}}),q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1}),$ $$

where $\mathbf{I}$ is the unit matrix, $T$ is the total number of time steps, $q$ and $\mathcal{N}$ represent forward process and Gaussian distribution parameterized by scheduled hyper-parameters $\{\beta_{t}\}_{t\in[T]}$. Perturbed samples are sampled by $x_{t}=\sqrt{\bar{\alpha}_{t}}\cdot x_{0}+\sqrt{1-\bar{\alpha}_{t}}\cdot
\epsilon,\epsilon\sim\mathcal{N}(0,\mathbf{I})$,
where $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$.

For diffusion model training, the forward process is divided into pairs of samples and targeted process increments by time steps $t$, defined as $\delta_{t}:=x_{t+1}-x_{t}$. The diffusion model is expected to predict the next step from the given time step.
The training loss (Ho et al., ) for diffusion models is to predict the normalized noise. The loss highlighted with weighting and sampling modules:

$$ $L=\mathbb{E}_{\mu_{t}}[w_{t}||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}]:=\int _{t}w_{t}||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}\mathbf{d}\mu_{t},\epsilon \sim\mathcal{N}(0,\mathbf{I}).$ (1) $$

Intuitively, a neural network $\epsilon_{\theta}$ is trained to predict the normalized noise $\epsilon$ added at given time-step $t$. The probability of a sample being sampled in the forward process is determined by the probability measure $\mu_{t}$, while the weight of the loss function is determined by $w_{t}$ at $t^{\text{th}}$ time-step.

### 2.2 Case Study on DDPM: A Closer Look at Time Steps

In DDPM, the diffusion model learns the noise added in the forward process at given $t^{\text{th}}$ time step. The noise is presented as $\epsilon$, the label in Eqn. 1, which is the normalized process increment at given time step. This label tells what the output of the diffusion model is aligning to. To take a closer look, we focus on the nature of the process increment $\delta_{t}$ itself to study the diffusion process $x_{t}\rightarrow x_{t+1}$, instead of $\epsilon$ the normalized one. According to Theorem 1 and Remark 1, based on the variation trends of process increments $\delta_{t}$, we can distinguish three distinct areas: acceleration, deceleration, and convergence. The detailed discussion is shown as follows:

###### Theorem 1 (Process increment in DDPM) .

In DDPM’s setting (Ho et al., ), the linear schedule hyper-parameters $\{\beta_{t}\}_{t\in[T]}$ is an equivariant series, the extreme deviation $\Delta_{\beta}:=\max_{t}\beta_{t}-\min_{t}\beta_{t}$, $T$ is the total number of time steps, and we have the bounds about the process increment $\delta_{t}\sim\mathcal{N}(\phi_{t},\Psi_{t})$, where $\phi_{t}:=(\sqrt{\alpha_{t+1}}-1)\sqrt{\bar{\alpha}_{t}}x_{0},\Psi_{t}:=[2-
\bar{\alpha}_{t}(1+\alpha_{t+1})]\mathbf{I}$, $\mathbf{I}$ is the unit matrix:

$$ Upper-bound: $\displaystyle||\phi_{t}||^{2}$ $\displaystyle\leq\hat{\phi}_{t}||\mathbb{E}x_{0}||^{2},$ $\displaystyle\hat{\phi}_{t}:=\beta_{\max}\exp\{-(\beta_{0}+{\Delta_{\beta}t}/{ 2T})t\}$ (2) Lower-bound: $\displaystyle\Psi_{t}$ $\displaystyle\succeq\hat{\Psi}_{t}\mathbf{I},$ $\displaystyle\hat{\Psi}_{t}:=2-2\exp\{-(\beta_{0}+{\Delta_{\beta}t}/{2T})t\}$ $$

###### Remark 1 .

The entire diffusion process can be approximated using the upper and lower bounds from Theorem 1, which we visualize as shown in Figure 3. We can observe that the diffusion process can be divided into three areas: acceleration, deceleration, and convergence. The two boundary points of these areas are denoted as $t_{\text{a-d}}$ and $t_{\text{d-c}}$ with their specific definitions and properties outlined below.

Figure: (a)
Refer to caption: x4.png

Definition of $t_{\text{a-d}}$. The boundary between the acceleration and deceleration areas is determined by the inflection point in the parameter variation curves, as illustrated in Figure 3. This inflection point represents the peak where the process increment changes most rapidly. The key time-step $t_{\text{a-d}}$ between acceleration and deceleration areas satisfies $t_{\text{a-d}}=\arg\max_{t}\partial_{t}\hat{\Psi}_{t}$ and $\beta_{t_{a-d}}=\sqrt{{\Delta_{\beta}}/{T}}$ in our setting, where $\partial_{t}\hat{\Psi}_{t}=2(\beta_{0}+{\Delta_{\beta}t}/{T})\exp\{-(\beta_{0}
+{\Delta_{\beta}t}/{2T})t\}$.

Definition of $t_{\text{d-c}}$. The process is considered to be in the convergence area where the increments’ variance is within a range. The convergence area is identified by a magnitude $r$, where $1-1/r$ is the ratio to the maximum variance.

According to Theorem 1, the convergence area is defined as one magnitude reduction of the scale factor (i.e., $1\times r$), and we have the lower-bound of the magnitude $\hat{r}_{t}:=\exp\{(\beta_{0}+\Delta_{\beta}t/2T)t\}$ employed as the threshold selection function in Section 2.5. The time step $t$ is guaranteed to be in convergence area when $\hat{r}_{t}>r$, .

###### Theorem 1 (Process increment in DDPM) .

###### Remark 1 .

##### Analyses.

In the convergence area, the variations of $\delta_{t}$ stabilize, signifying that the process is approaching a steady state.

This area corresponds to a very large proportion of the overall time steps. On top of that, the training loss in this area is empirically low, which leads to the redundant training cost on time steps with limited benefits for training.

In the acceleration area, the variations of $\delta_{t}$ increase, indicating a rapid change. Conversely, in the deceleration area, the variations of $\delta_{t}$ decrease, reflecting a slowing down of the process. Notably, near the peak between the acceleration and deceleration areas, the process exhibits the fastest changes.
These time steps only occupy a small proportion.
Beyond that, the training losses in this area are empirically high.
The issue is that a hard-to-learn area is even under-sampled, necessitating more sampling and training efforts.

##### Takeaways.

Based on the analyses and observations, we provide takeaways as follows:

- •
The samples from convergence area provide limited benefits for training. The sampling of the time step from this area should be suppressed.
- •
Pay more attention to the process increment’s rapid-change area which is hard to learn and the corresponding time steps are fewer than the other areas.

### 2.3 General Cases Beyond DDPM: Dive into The Commonality.

This section generalize the above findings on DDPM to broader settings. The findings are about the process increments $\delta_{t}:=x_{t+1}-x_{t}$, and the related differentiation of right limit $\mathbf{d}x$ in forward process.
The corresponding SDE (Karras et al., ) and the discretization are:

$$ $\mathbf{d}x=x\dot{s}/s\mathbf{d}t+s\sqrt{\dot{\sigma}\sigma}\mathbf{d}w,\qquad x _{t}=s_{t}x_{0}+s_{t}\sigma_{t}\epsilon,\epsilon\sim\mathcal{N}(0,\mathbf{I}),$ $$

where scale factor $s=s_{t}$ and noise standard deviation (std.) $\sigma=\sigma_{t}$ are the main designs related to the main factors $\hat{\phi}_{t}$ and $\hat{\Psi}_{t}$ about process increment $\delta_{t}$ in Theorem 1 across time steps $t$.

##### Generalize Theorem 1 : s 𝑠 s italic_s - σ 𝜎 \sigma italic_σ Scheduled Process Increments.

The generalized process increment is $\delta\sim\mathcal{N}(\Delta x_{0},\Sigma\mathbf{I})$, where $\Delta:=s_{+}-s$ and $\Sigma:=s_{+}^{2}\sigma_{+}^{2}+s^{2}\sigma^{2}$ across $t$. $\Delta$, $\Sigma$ are continuous on $t$ without discretization, where $s_{+}$ and $\sigma_{+}$ are the right outer limits, i.e., $s(t+\mathbf{d}t)$. In discretization, $\Delta_{t}$ and $\Sigma_{t}$, marked by $t$, are related to sample granularity of time step $t$. Like Theorem 1, we study the variation of process increments by $\dot{\Delta}=\dot{s}_{+}-\dot{s}$ and
$\dot{\Sigma}=\mathbf{m}^{\top}\dot{\mathbf{n}}$
where $\mathbf{m}=[s_{+}^{2},\sigma_{+}^{2},s^{2},\sigma^{2}]^{\top}$, $\mathbf{n}=[\sigma_{+}^{2},s_{+}^{2},\sigma^{2},s^{2}]^{\top}$. This formulation involves only terms about derivatives of given schedule functions, which brings computational convenience. Tab. 1 provides all ingredients needed to calculate curves of $\dot{\Delta}$ and $\dot{\Sigma}$ in schedules of VP, VE and EDM.

We also generalize the previous takeaways from the DDPM case study to $s$-$\sigma$ scheduled setting with the following analyses and discussion.

$\sigma$ is better for the design of sampling and weighting than $s$. It stands due to its direct reflection about SNR (signal-to-noise ratio), and additionally, because $s$ is usually adapted to heuristic motivations. In DDPM, corresponding to VP in Tab. 1, the SDE design is simply from data to normal noise. In VE, realistic diffusion processes inspire that the diffusion rate is limited to $\sigma=\sqrt{t}$. Further in EDM, motivation become more complex of training objective and concise of schedule and motivation at the same time, bringing benefits to training. Its key ideas and designs are 1) the std. of inputs and targeted outputs of a neural network $F_{\theta}$ in EDM is constrained to 1 with preconditioning; 2) the weights $w_{t}$ in Eqn. 1 are allocated according to $c_{\text{out}}w_{t}=1$, where $c_{\text{out}}$ is the scale factor of F-prediction neural network $F_{\theta}$’s outputs, and is related to $\sigma$ and the std. of data (sometimes normalized as 1).

Sampling deserves more attention. The SDE design goal of most diffusion models nowadays is to add much larger noise to the data so that the samples can cover larger space. However, in terms of the process increment, it always results in a low signal-to-noise ratio of the late data when $t$ is large. Either the standard deviation is too large or the $s$ used to suppress it is too small. For instance, EDM does not bias the data distribution from expectations due to the scale $\dot{\Delta}=0$, but $\dot{\Sigma}=2[(t+\mathbf{d}t)^{2}+t^{2}]$ is a quadratic increase as $t$ grows. In VP, $s\rightarrow 0$, as $t$ grows, leads that the model needs to recover expectations in approximation. For these samples, which are not very informative, a single weight adjustment is not as efficient as directly reducing the sampling rate.

**Table 1: The ingredients of generalized curves $\dot{\Delta}$ and $\dot{\Sigma}$ schedules about mainstream SDE designs, including VP, VE (Song et al., ), EDM (Karras et al., ).**
| Schedules | $s$ | $\sigma^{2}$ | $\dot{s}$ | $\dot{\sigma}$ |
| --- | --- | --- | --- | --- |
| VP | $\exp\{-\frac{1}{4}\Delta_{\beta}t^{2}-\frac{1}{2}\beta_{0}t\}$ | $\exp\{\frac{1}{2}\Delta_{\beta}t^{2}+\beta_{0}t\}-1$ | $-\frac{\sigma\dot{\sigma}}{(1+\sigma^{2})^{3/2}}$ | $\frac{(1+\sigma^{2})(\Delta_{\beta}t+\beta_{0})}{2\sigma}$ |
| VE | $1$ | $t$ | 0 0 | $1$ |
| EDM | $1$ | $t^{2}$ | 0 0 | $2t$ |

### 2.4 Overview of SpeeD

Based on the above observations and analyses, we propose SpeeD , a novel approach for achieving lossless training acceleration tailored for diffusion models.
As illustrated in Fig.
2,
SpeeD suppresses the trivial time steps from convergence area, and weight the rapid-change intervals between acceleration and deceleration areas.
Correspondingly, two main modules, asymmetric sampling and change-aware weighting, are proposed. Asymmetric sampling uses a two-step step function to respectively suppress and increase the sampling probability corresponding to trivial and beneficial time steps. Change-aware weighting is based on the change rate of process increment $\partial_{t}\Psi(t)$.

### 2.5 Asymmetric Sampling

SpeeD adopts the time steps sampling probability $\mathbf{P}(t)$ as the step function in Eqn. 3 to construct the loss in Eqn. 1. We first define $\tau$ as the step threshold in $\mathbf{P}(t)$.
The pre-defined boundary $\tau$ means the area where the time step are suppressed. The sampling probability is $k$ times from time-steps where $t<\tau$ than that from $t>\tau$ instead of the uniform sampling $\mathbf{U}(t)=1/T$.

$$ $\mathbf{P}(t)=\begin{cases}\frac{k}{T+\tau(k-1)},\quad&0<t\leq\tau\\ \frac{1}{T+\tau(k-1)},\quad&\tau<t\leq T\end{cases},\quad\text{where suppression intensity }k\geq 1\text{ and }\tau\in(0,T].$ (3) $$

##### Threshold Selection τ 𝜏 \tau italic_τ .

According to Theorem 1, given a magnitude $r$, $\tau$ should satisfy $\hat{r}(\tau)>r$ to make sure that ${\tau}>t_{\text{d-c}}$, where the time steps suppressed are all time steps in the convergence area. To maximum the number of suppressed time steps, we set $\tau\leftarrow\sqrt{2T\log r/\Delta_{\beta}+T^{2}\beta_{0}^{2}/\Delta_{\beta}^
{2}}-T\beta_{0}/\Delta_{\beta}$.

### 2.6 Change-Aware Weighting

According to Theorem 1, a faster change of process increment means fewer samples at the corresponding noise level.
This leads to under-sampling in acceleration and deceleration areas.
Change-aware weighting is adopted to mitigate the under-sampling issue.
The weights $\{w_{t}\}_{t\in[T]}$ are assigned based on the gradient of the variance over time, where we use the approximation $\partial_{t}\hat{\Psi}_{t}$ in Theorem 1.

The original gradient $\partial_{t}\hat{\Psi}_{t}$ is practically not suitable for weighting due to its small scale. Therefore, $\partial_{t}\hat{\Psi}_{t}$ is re-scaled into range $[1\text{-}\lambda,\lambda]$ that $\min\{1,\max_{t}{\partial_{t}\hat{\Psi}_{t}}\}\rightarrow\lambda$ and $\max\{0,\min_{t}{\partial_{t}\hat{\Psi}_{t}}\}\rightarrow 1\text{-}\lambda$, where symmetry ceiling $\lambda\in[0.5,1]$. $\lambda$ regulates the curvature of the weighting function. A higher $\lambda$ results in a more obvious distinction in weights between different time-steps.

## 3 Experiments

We first present visualizations in Sec. 3.1 and describe the main experiment setups in Sec. 3.2, including benchmark datasets, network architectures, training details, and evaluation protocols.
In Sec. 3.3, we present the main results regarding both performance and efficiency.
After that, in Sec. 3.6 and 3.4, we ablate the effectiveness of each designed module and evaluate the generalization of SpeeD in various diffusion tasks and settings. Finally, we investigate the compatibility of our approach with other recent methods in 3.5.

### 3.1 Visualization

Figure: (a) Visualization comparison on MetFaces dataset.
Refer to caption: x7.png

The comparison of visualizations between SpeeD and DiT-XL/2 models on the MetFaces and FFHQ datasets clearly demonstrates the superiority of SpeeD. As shown in Fig. 4, SpeeD achieves significantly better visual quality at just 20K or 30K training iterations, compared to DiT-XL/2. This highlights that SpeeD reaches high-quality results much faster than the baseline method, making it a more efficient and effective approach for training diffusion models.

### 3.2 Implementation Details

Datasets. We mainly investigate the effectiveness of our approach on the following datasets: MetFaces (Karras et al., ) and FFHQ (Karras et al., ) are used to train unconditional image generation, CIFAR-10 (Krizhevsky et al., ) and ImageNet-1K (Deng et al., ) are used to train conditional image generation, and MS-COCO (Lin et al., ) is used to evaluate the generalization of our method in the text to image task. More details of these datasets can be found in the Appendix A.

Network architectures. U-Net (Ronneberger et al., ) and DiT (Peebles & Xie, ) are two famous architectures in the diffusion model area. We implement our approach on these two architectures and their variants. We follow the same hyper parameters as the baseline by default.
More information about the details of the architectures can be found in Appendix A.3.

Training details.
We train all models using AdamW  (Kingma & Ba, ; Loshchilov & Hutter, ) with a constant learning rate 1e-4. We set the maximum step in training to 1000 and use the linear variance. All images are augmented with horizontal flip transformations if not stated otherwise. Following common practice in the generative modeling literature, the exponential moving average (EMA) (Gardner Jr, ) of network weights is used with a decay of 0.9999.
The results are reported using the EMA model.
Details can be found in Tab. 7.

Evaluation protocols.
In inference, we default to generating 10K images.
Fréchet Inception Distance (FID) is used to evaluate both the fidelity and coverage of generated images.

### 3.3 Comparisons with other strategies.

Table: Table 2: The FID$\downarrow$ comparison to the baseline: DiT-XL/2, re-weighting methods: P2 and Min-SNR, and re-sampling methods: Log-Normal and CLTS. All methods are trained with DiT-XL/2 for 50K iterations. We report the FID per 10K iterations. Our approach achieves the best results on MetFaces and FFHQ datasets. Bold entries are best results. Following previous work (Go et al., ), more results of 100K iterations and longer training phase with different schedules are in Appendix B.1.

##### Performance comparisons.

Before our comparison, we first introduce our baseline, i.e., DiT-XL/2, a strong image generation backbone as introduced in DiT (Peebles & Xie, ). We follow the hyperparameter settings from DiT and train DiT-XL/2 on MetFaces (Karras et al., ) and FFHQ (Karras et al., ), respectively.
We compare our approach with two re-weighting methods: P2 (Choi et al., ) and Min-SNR (Hang et al., ), and two re-sampling methods: Log-Normal (Karras et al., ) and CLTS (Xu et al., ).
In the evaluation, we use 10K generated images to calculate FID (Heusel et al., ) for comparison.
To make a detailed comparison, all the approaches are trained for 50K iterations and we report the FID scores per 10K iterations.

As shown in Tab LABEL:tab:comparison, compared to DiT-XL/2, re-weighting, and re-sampling methods, our approach obtains the best FID results. Specifically, at the 50K iteration, compared to other methods, we reduce 2.3 and 2.6 FID scores on MetFaces and FFHQ at least. Another interesting finding is that the re-weighting methods reduce the FID very slowly at the beginning of the training, i.e., from the 10K to 20K iterations. That aligns with our analysis well: re-weighting methods involve a lot of steps from the convergence area. Based on the experimental results, the time steps that come from the convergence area indeed contribute limited to the diffusion training.

##### Efficiency comparisons.

In addition to the performance comparison, we also present the acceleration results of our SpeeD .
This naturally raises a question: how to calculate the acceleration? Here, we follow the previous diffusion acceleration methods (Gao et al., ) and other efficient training papers (Qin et al., ; Xu et al., ):
visualizing the FID-Iteration curve and reporting the estimated highest acceleration ratio. We mainly compare with DiT-XL/2, one re-weighting method Min-SNR (Choi et al., ), and one re-sampling method CLTS (Xu et al., ) in Fig 5.
At the same training iterations, our approach achieves significantly better FID scores than other methods. Notably, SpeeD accelerates the Min-SNR, and CLTS by 2.7 and 2.6 times, respectively. More comparisons with other methods can be found in Appendix B.

For the comparison with the baseline, i.e., DiT-XL/2, considering the 50K iterations might be too short for converge, we extend the training iterations from 50K to 200K. In the long-term training, we speed up the DiT-XL/2 by 4 times without performance drops. That shows the strong efficiency of our proposed method. Most importantly, we can save 3$\sim$5 times the overall training cost with very minimal overhead. For instance, we save 48 hours (result obtained by training on 8 A6000 Nvidia GPUs) of training time for DiT-XL/2 with negligible seconds overhead.

Figure: Figure 5: We plot the FID-Iteration curves of our approach and other methods on the MetFaces dataset. SpeeD accelerates other methods obviously. The horizontal and vertical axes represent the training iterations and log(FID), respectively.
Refer to caption: x9.png

### 3.4 Generalization Evaluation

**Table 3: Cross-architecture robustness evaluation. ‘Baseline’ denotes training diffusion models without acceleration strategy. ’DiT’ refers to the DiT-XL/2 network. All FID scores are obtained by testing 10K generated images.**
|  | Metfaces | FFHQ | ImageNet |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| DiT | U-Net | DiT | U-Net | DiT | U-Net |  |
| baseline | 36.61 | 46.77 | 12.86 | 17.37 | 26.74 | 45.71 |
| SpeeD | 21.13 | 22.88 | 9.95 | 16.52 | 20.63 | 37.33 |
| improve | 15.48 | 23.89 | 2.91 | 0.85 | 6.11 | 7.38 |

Cross-architecture robustness evaluation.
There are mainly two architectures in the diffusion models: U-Net (Ronneberger et al., ) and DiT (Peebles & Xie, ).
SpeeD is not correlated to specific model architecture, thereby it is a model-agnostic approach. We implement our method with DiT-XL/2 and U-Net on MetFaces, FFHQ, and ImageNet-1K, respectively.
We default to training the models for 50K iterations on MetFaces and FFHQ, 400K on ImageNet-1K.To ensure a fair comparison, we keep all hyper-parameters the same and report the FID scores at 50K iterations.
As shown in Tab. 4, SpeeD consistently achieve significantly higher performance under all settings, which indicates the strong generality of our method for different architectures and datasets.

Cross-schedule robustness evaluation.
In the diffusion process, there are various time step schedules, including linear (Ho et al., ), quadratic and cosine (Nichol & Dhariwal, ) schedules.
We verify SpeeD’s robustness across these schedules.
We report FID and inception score (IS) (Salimans et al., ) scores as metrics for comparisons.
As shown in Tab. 4, SpeeD achieves significant improvement on linear, quadratic, and cosine schedules both in FID and IS. That shows the strong generality of SpeeD in various schedules.

Table: Table 5: Text to image.

Cross-task robustness evaluation.
We apply SpeeD to the text-to-image task for evaluating the generality of our method.
For text-to-image generation, we first introduce CLIP (Radford et al., ) to extract the text embedding for MS-COCO (Lin et al., ) dataset. Then, DiT-XL/2 is used to train a text-to-image model as our baseline.
Following prior work(Saharia et al., ), FID score and CLIP score on MS-COCO validation set are evaluation metrics for quantitative analyses.
As illustrated in Tab. 5, we obtain the better FID and CLIP score than our baseline.

### 3.5 Compatibility with other acceleration methods

Figure: (a) FID-Iteration curve comparisons on ImageNet-1K.
Refer to caption: x12.png

Until now, we evaluate the effectiveness and generalization of our proposed method: SpeeD is a task-agnostic and architecture-agnostic diffusion acceleration approach.
Is SpeeD compatible with other acceleration techniques? To investigate this, we apply our approach with two recent proposed algorithms: masked diffusion transformer (MDT) (Gao et al., ) and fast diffusion model (FDM) (Wu et al., ).

MDT + SpeeD.
MDT (Gao et al., ) proposes a masked diffusion transformer method, which applies a masking scheme in latent space to enhance the contextual learning ability of diffusion probabilistic models explicitly. MDT can speed up the diffusion training by 10 times.
They evaluate their MDT with DiT-S/2. We just inject our SpeeD on their MDT and report FID-Iteration curves for comparison in Fig. 6(a).
All the results are obtained on ImageNet-1K dataset.
Our approach can further accelerate MDT at least by 4 $\times$, which indicates the good compatibility of SpeeD.

FDM + SpeeD.
Fast Diffusion Model (Wu et al., ) is a diffusion process acceleration method inspired by the classic momentum approach to solve optimization problem in parameter space.
By integrating momentum into the diffusion process, it achieves similar performance as EDM (Karras et al., ) with less training cost.
Based on the official implementation, we compare EDM, FDM, and FDM + SpeeD on CIFAR-10 of $32\times 32$ images.
FDM accelerates the EDM by about 1.6 $\times$. Meanwhile, SpeeD can further reduce the overall training cost around 1.6 $\times$.

### 3.6 Ablation Experiments

We perform extensive ablation studies to illustrate the characteristics of SpeeD . The experiments in ablation studies are conducted on the FFHQ dataset and U-Net model by default. We ablate our designed components: asymmetric sampling (abbreviated as asymmetric) and change-aware weighting (abbreviated as CAW), suppression intensity $k$ in asymmetric sampling defined in Eqn. 3 and symmetry ceiling $\lambda$ for weighting in Sec. 2.6.

**Table 6: Ablation studies on FFHQ dataset. Default settings and baseline are in gray and cyan.**
| sampling curve | CAW | FID $\downarrow$ |
| --- | --- | --- |
| uniform |  | 17.37 |
| uniform | ✓ | 16.75 |
| asymmetric |  | 15.82 |
| asymmetric | ✓ | 15.07 |

##### Evaluating the components of SpeeD.

Our approach includes two strategies: asymmetric sampling and change-aware weighting. We note these two strategies using ‘asymmetric’ and ‘CAW’.
We ablate each component in SpeeD. As illustrated in Tab. 6(a), combining our proposed strategies achieves the best results. Using weighting and sampling strategies alone improves the baseline by 0.6 and 1.5 FID scores, respectively, indicating that filtering most samples from the convergence area is more beneficial for training.

##### Evaluating of suppression intensity k 𝑘 k italic_k .

To prioritize the training of critical time steps, asymmetric sampling focus on the time steps out of the convergence area. A higher probability is given for these time steps which is $k$ times than the time steps from the convergence area.
A larger suppression intensity $k$ means a larger gap in training between the different areas of time steps.
We evaluate different suppression intensity $k$ from 1 to 25 and report the FID score In Tab. 6(b).
We observe that k = 5 achieves the best performance. A huge suppression intensity decrease FID scores seriously, which means that it hurts the diversity a lot to modeling data.
This means that the samples in the convergence area, although very close to pure noise, still contains some useful information. Extreme complete discard of these samples results in a degradation of the acceleration performance.

##### Evaluating of symmetry ceiling λ 𝜆 \lambda italic_λ .

The symmetry ceiling $\lambda$ is a hyper-parameter that regulates the curvature of the weighting function. $\lambda$ is set in the interval $[0.5,1]$. The midpoint of the re-scaled weight interval is fixed at $0.5$. The symmetry ceiling $\lambda$ is the right boundary of the interval and the left boundary is $1$-$\lambda$. A higher $\lambda$ results in a larger weight interval and a more obvious distinction in weights between different time steps. In Tab. 6(c), settings $\lambda\leq 0.8$ obtain higher performance on FID scores than the baseline, which indicates SpeeD is relatively robust on symmetry ceiling $\lambda$. Further increase $\lambda$ leads to performance degradation. This means weighting should be in moderation.

## 4 Related Work

We discuss the related works about Diffusion Models and its Training Acceleration. The most related works are as follows, and more about cross-modality and video generation are in Appendix C.

##### Diffusion models

Diffusion models have emerged as the dominant approach in generative tasks (Saharia et al., ; Chen et al., ; Wang et al., ; Igashov et al., ), which outperform other generative methods including GANs (Zhu et al., ; Isola et al., ; Brock et al., ), VAEs (Kingma & Welling, ), flow-based models (Dinh et al., ). These methods (Ho et al., ; Song et al., ; Karras et al., ) are based on non-equilibrium thermodynamics (Jarzynski, ; Sohl-Dickstein et al., ), where the generative process is modeled as a reverse diffusion process that gradually constructs the sample from a noise distribution (Sohl-Dickstein et al., ).
Previous works focused on enhancing diffusion models’ generation quality and alignment with users in visual generation. To generate high-resolution images, Latent Diffusion Models (LDMs) (Saharia et al., ; Rombach et al., ) perform diffusion process in latent space instead of pixel space, which employ VAEs to be encoder and decoder for latent representations.

##### Acceleration in diffusion models

To reduce the computational costs, previous works accelerate the diffusion models in training and inference.
For training acceleration, the earliest works (Choi et al., ; Hang et al., ) assign different weights to each time step on Mean Square Error (MSE) loss to improve the learning efficiency. The other methods in training acceleration are proposed, e.g., network architecture (Ryu & Ye, ; Wang et al., ) and diffusion algorithm (Karras et al., ; Wu et al., ). Masking modeling (Gao et al., ; Zheng et al., ) are recently proposed to train diffusion models. Works
(Go et al., ; Park et al., ; ; Kim et al., ) provide observations for explanation from the perspective of multi-tasks learning.
SpeeD is of good compatibility with these methods, e.g., (Yu et al., ; Zheng et al., ; ).
In the field of sampling acceleration, a number of works tackle the slow inference speed of diffusion models by using fewer reverse steps while maintaining sample quality, including DDIM (Song et al., ), Analytic-DPM (Bao et al., ), and DPM-Solver (Lu et al., ).

##### Conditional generation.

To better control the generation process with various conditions, e.g., image style, text prompt and stroke, Classifier-Free Guidance (CFG) proposes a guidance strategy with diffusion models that balance the sample quality and prompt alignment. ControlNet (Zhang et al., ) reuses the large-scale pre-trained layers of source
models to build a deep and strong encoder to learn specific conditions.
Recently benefiting from diffusion models in the image generation field, video generation (Ma et al., ; Lab & etc., ) is getting trendy.

## 5 Conclusion

We present SpeeD , a novel approach for accelerating diffusion training by closely examining time steps. The core insights from this examination are: 1) suppressing the sampling probabilities of time steps that offer limited benefits to diffusion training (i.e., those with extremely small losses), and 2) emphasizing the importance of time steps with rapidly changing process increments.
SpeeD demonstrates strong robustness across various architectures and datasets, achieving significant acceleration on multiple diffusion-based image generation tasks. Additionally, SpeeD is easily compatible with other diffusion acceleration methods, highlighting its wide applicability.
We provide extensive theoretical analysis in this paper, aiming to support future research in both academia and industry.

##### Acknowledgments.

This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-008). This work is partially supported by the National Natural Science Foundation of China (62176165), the Stable Support Projects for Shenzhen Higher Education Institutions (20220718110918001), the Natural Science Foundation of Top Talent of SZTU(GDRC202131). Yang You’s research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant (WBS number: A-8001104-00-00), Alibaba grant, and Google grant for TPU usage. We thank Tianyi Li, Yuchen Zhang, Yuxin Li, Zhaoyang Zeng, and Yanqing Liu for the comments on this work.
Kai Wang (idea, writing, story, presentation), Yukun Zhou (implementation), and Mingjia Shi (theory, writing, presentation) contribute equally to this work. Xiaojiang Peng, Hanwang Zhang, and Yang You are equal advising. Xiaojiang Peng is the corresponding author.

## Appendix A More Detail of Experiments

In this section, we introduce detailed experiment settings, datasets, and architectures.

### A.1 Statement

Experimental design note: Referring to previous works (Peebles & Xie, ; Go et al., ; Park et al., ; Choi et al., ; Hang et al., ; Esser et al., ; Xu et al., ), our experiments train 50K, 100K, and 400K in MetFaces 256 × 256, FFHQ 256 × 256, and ImageNet-1K, respectively. we know that DiT used 7M iterations to train ImageNet in the original work Peebles & Xie (), which 1) is not the amount of resource usage that can be achieved in general research. Meanwhile, 2) the focus of this paper is on the acceleration effect of training, and a direct comparison of the ultimate convergence stage is very unnecessary for the research topic of this paper.

### A.2 Instructions for code

Code is provided in the link at the very beginning of the main paper.

### A.3 Architecture and Training Recipe.

We utilize Unet and DiT as our base architecture in the diffusion model. pre-trained VAE which loads checkpoints from [huggingface](https://huggingface.co/stabilityai/sd-vae-ft-mse) is employed to be latent encoder. Following Unet implementation from [LDM](https://github.com/CompVis/stable-diffusion) and DIT from official implementation, we provide the architecture detail in Tab. 7. We provide our basic training recipe and evaluation setting with specific details in Tab. 8.

**Table 7: Architecture detail of Unet and DiT on MetFaces, FFHQ, and ImageNet.**
| Architecture | input size | input channels | patch size | model depth | hidden size | attention heads |
| --- | --- | --- | --- | --- | --- | --- |
| U-Net | 32 $\times$ 32 | 3 | - | 8 | 128 | 1 |
| DiT-XL/2 | 32 $\times$ 32 | 4 | 2 | 28 | 1152 | 16 |
| DiT-S/2 | 32 $\times$ 32 | 4 | 2 | 12 | 384 | 6 |

**Table 8: Our basic training recipe based on MetFaces, FFHQ, ImageNet datasets**
|  | MetFaces 256 $\times$ 256 | FFHQ 256 $\times$ 256 | ImageNet-1K 256 $\times$ 256 |
| --- | --- | --- | --- |
| latent size | 32 $\times$ 32 $\times$ 3 | 32 $\times$ 32 $\times$ 3 | 32 $\times$ 32 $\times$ 3 |
| class-conditional |  |  | ✓ |
| diffusion steps | 1000 | 1000 | 1000 |
| noise schedule | linear | linear | linear |
| batch size | 256 | 256 | 256 |
| training iterations | 50K | 100K | 400K |
| optimizer | AdamW | AadamW | AdamW |
| learning rate | 1e-4 | 1e-4 | 1e-4 |
| weight decay | 0 | 0 | 0 |
| sample algorithm | DDPM | DDPM | DDPM |
| number steps in sample | 250 | 250 | 250 |
| number sample in evaluation | $10,000$ | $10,000$ | $10,000$ |

### A.4 Datasets

CIFAR-10. CIFAR-10 datasets consist of 32 $\times$ 32 size colored natural images divided into categories. It uses $50,000$ in images for training and [EDM evaluation suite](https://github.com/NVlabs/edm) in image generation.

MetFaces is an image dataset of human faces extracted from works of art. It consists of 1336 high-quality PNG images at 1024×1024 resolution. We download it at 256 resolution from [kaggle](https://www.kaggle.com/).

FFHQ is a high-quality image dataset of human faces, contains 70,000 images. We download it at 256x256 resolution from [kaggle](https://www.kaggle.com/).

ImageNet-1K is the subset of the ImageNet-21K dataset with $1,000$ categories. It contains $1,281,167$ training images and $50,000$ validation images.

MSCOCO is a large-scale text-image pair dataset. It contains 118K training text-image pairs and 5K validation images. We download it from [official website](https://cocodataset.org/#home).

FaceForensics is a video dataset consisting of more than 500,000 frames containing faces from 1004 videos that can be used to study image or video forgeries.

### A.5 Detail of MDT + SpeeD Experiment

MDT utilizes an asymmetric diffusion transformer architecture, which is composed of three main components: an encoder, a side interpolater, and a decoder. During training, a subset of the latent embedding patches is randomly masked using Gaussian noise with a masking ratio. Then, the remaining latent embedding, along with the full latent embedding is input into the diffusion model.

Following the official implementation of MDT, We utilize DiT-S/2 and MDT-S/2 as our base architecture, whose total block number both is 12 and the number of decoder layers in MDT is $2$. We employ the AdamW (Loshchilov & Hutter, ) optimizer with constant learning rate 1e-4 using $256$ batch size without weight decay on class-conditional ImageNet with an image resolution of $256^{2}$. We perform training on the class-conditional ImageNet dataset with images of resolution 256x256. The diffusion models are trained for a total of 1000K iterations, utilizing a mask ratio of 0.3.

### A.6 Detail of FDM + SpeeD Experiment

FDM add the momentum to the forward diffusion process with a scale that control the weight of momentum for faster convergence to the target distribution.
Following official implementation, we train diffusion models of EDM and FDM. We retrain these official network architecture which is U-Net with positional time embedding with dropout rate 0.13 in training. We adopt Adam optimizer with learning rate 1e-3 and batch size 512 to train each model by a total of 200 million images of $32^{2}$ CIFAR-10 dataset. During training, we adopt a learning rate ramp-up duration of 10 Mimgs and set the EMA half-life as 0.5 Mimgs. For evaluation, EMA models generate 50K images using EDM sampler based on Heun’s $2^{nd}$ order method (Süli & Mayers, ).

### A.7 Text-to-Image Experiment Detail

In text to image task, diffusion models synthesize images with textual prompts. For understanding textual prompts, text-to-image models need semantic text encoders to encode language text tokens into text embedding. We incorporate a pre-trained CLIP language encoder, which processes text with a maximum token length of 77. DiT-XL/2 is employed as our base diffusion architecture. We employ AdamW optimizer with a constant learning rate 1e-4 without weight decay. We train text-to-image diffusion models for 400K training iterations on MS-COCO training dataset and evaluate the FID and CLIP score on MS-COCO validation dataset. To enhance the quality of conditional image synthesis, we implement classifier-free guidance with 1.5 scale factor.

### A.8 Theoretical Analysis

#### A.8.1 Notations

In this section, we will introduce the main auxiliary notations and the quantities that need to be used. The range of schedule hyper-parameter group $\{\beta_{t}\}_{t\in[T]}$ turns out to be $t=1$ to $t=T$. For analytical convenience, we define $\beta_{0}$ as
$\beta_{0}:=\beta_{1}-\Delta_{\beta}/T$.

Another auxiliary notation is forward ratio $\rho_{t}$, which is defined as $\rho_{t}=t/T$. Forward ratio provide an total number free notation for general diffusion process descriptions.

Based on the two auxiliary notations $\beta_{0}$ and $\rho_{t}$, the expression of $\beta_{t}$ with respect to the forward process ratio is
$\beta_{t}=\beta_{0}+\Delta_{\beta}\rho_{t}$.

The relationship between $\alpha_{t}$ and $\beta_{t}$ is recalled and re-written as follows:
$\alpha_{t}=1-\beta_{t}=1-\beta_{0}-\Delta_{\beta}\rho_{t}$.
$\bar{\alpha}_{t}$ the multiplication of $\alpha_{t}$ is re-written as $\bar{\alpha}_{t}=\Pi_{s=1}^{t}(1-\beta_{0}-\Delta_{\beta}\rho_{s})$.

Perturbed samples’ distribution:
$x_{t}|x_{0}\sim\mathcal{N}(\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})
\mathbf{I})$

#### A.8.2 Auxiliary Lemma and Core Theorem

###### Lemma 1 (Bounded $\alpha$ by $\beta$ ) .

In DDPM (Ho et al., ), using a simple equivariant series $\{\beta_{t}\}_{t\in[T]}$ to simplify the complex cumulative products $\{\bar{\alpha}_{t}\}_{t\in[T]}$, we obtain the following auxiliary upper bound of $\bar{alpha}_{t}$.

$$ $\displaystyle\bar{\alpha}_{t}$ $\displaystyle\leq\exp\{-(\beta_{0}t+\frac{\Delta_{\beta}t^{2}}{2T})\}$ $$

###### Lemma 1 (Bounded α 𝛼 \alpha italic_α by β 𝛽 \beta italic_β ) .

#### A.8.3 Propositions

###### Proposition A.1 (Jensen’s inequality) .

If $f$ is convex, we have:

$$ $\displaystyle\mathbf{E}_{X}f(X)\geq f(\mathbf{E}_{X}X).$ $$

A variant of the general one shown above:

$$ $\displaystyle||\sum_{i\in[N]}x_{i}||^{2}\leq N\sum_{i\in[N]}||x_{i}||^{2}.$ $$

###### Proposition A.2 (triangle inequality) .

The triangle inequality is shown as follows, where $||\cdot||$ is a norm and $A,B$ is the quantity in the corresponding norm space:

$$ $||A+B||\leq||A||+||B||$ $$

.

###### Proposition A.3 (matrix norm compatibility) .

The matrix norm compatibility, $A\in\mathbb{R}^{a\times b},B\in\mathbb{R}^{b\times c},v\in\mathbb{R}^{b}$:

$$ $\displaystyle||AB||_{m}\leq||A||_{m}||B||_{m}$ $\displaystyle||Av||_{m}\leq||A||_{m}||v||.$ $$

###### Proposition A.4 (Peter Paul inequality) .

$$ $2\langle x,y\rangle\leq\frac{1}{\epsilon}||x||^{2}+\epsilon||y||^{2}$ $$

.

###### Proposition A.1 (Jensen’s inequality) .

###### Proposition A.2 (triangle inequality) .

###### Proposition A.3 (matrix norm compatibility) .

###### Proposition A.4 (Peter Paul inequality) .

#### A.8.4 Proof of Lemma 1

###### Proof.

To proof the auxiliary Lemma 1, we re-arrange the notation of $\bar{\alpha}_{t}$ as shown in Section A.8.1, and we have the following upper bound:

$$ $\displaystyle\log\bar{\alpha}_{t}$ $\displaystyle=\sum_{s=1}^{t}\log(1-\beta_{0}-\Delta_{\beta}\rho_{s})$ $\displaystyle\leq t\log(\frac{1}{t}\sum_{s=1}^{t}(1-\beta_{0}-\Delta_{\beta} \rho_{s}))$ $\displaystyle=t\log(1-\beta_{0}-\Delta_{\beta}\frac{1}{t}\sum_{s=1}^{t}\frac{s }{T})$ $\displaystyle=t\log(1-\beta_{0}-\Delta_{\beta}\frac{t+1}{2T})$ $\displaystyle\leq-(\beta_{0}t+\frac{\Delta_{\beta}(t+1)t}{2T}),$ $$

where the two inequalities are by the concavity of $\log$ function and the inequality: $\log(1+x)\leq x$. Taking exponents on both sides simultaneously, we have:

$$ $\displaystyle\bar{\alpha}_{t}$ $\displaystyle\leq\exp\{-(\beta_{0}t+\frac{\Delta_{\beta}t^{2}}{2T})\}.$ $$

∎

###### Proof.

#### A.8.5 Proof of Theorem 1

Before the proof of the theorem, we note that the samples $x_{t}|x_{0}\sim\mathcal{N}(\mu_{t},\mathbf{\sigma}_{t})$ have the following bounds with Lemma 1:

- •
Reformulate the expression of $\sqrt{\bar{\alpha}}$, we have the mean vector $\mu_{t}$’s components $\dot{\mu}_{t}$ bounded by $\dot{x}_{0}$ the corresponding components of data $x_{0}$ as follows:
$\dot{\mu}_{t}=\sqrt{\bar{\alpha}_{t}}\dot{x}_{0}\leq\exp\{-\frac{1}{2}(\beta_{
0}t+\frac{\Delta_{\beta}t^{2}}{2T})\}\dot{x}_{0},$
- •
Reformulate the expression of $\bar{\alpha}$, we have a partial order relation on the cone about covariance matrix of $x_{t}|x_{0}$ as follows:
$\mathbf{\sigma}_{t}=(1-\bar{\alpha}_{t})\mathbf{I}\succeq(1-\exp\{-(\beta_{0}t
+\frac{\Delta_{\beta}t^{2}}{2T})\})\mathbf{I}.$

###### Proof.

The process increment at given $t^{\text{th}}$ time step is $\delta_{t}=x_{t+1}-x_{t}$. $\delta_{t}$ is a Gaussian process as follows:

$$ $\delta_{t}\sim\mathcal{N}(\underbrace{(\sqrt{\alpha_{t+1}}-1)\sqrt{\bar{\alpha }_{t}}x_{0}}_{\phi_{t}},\underbrace{[2-\bar{\alpha}_{t}(1+\alpha_{t+1})] \mathbf{I}}_{\Psi_{t}})$ $$

The theorem’s key motivation is that the label is noisy, and noisy magnitude is measured by mean vector’s norm $||\phi_{t}||$ and covariance matrix $\Psi_{t}$.

The upper bounds of mean vectors’ norm and the partial order of covariance matrix at different time step $t$ are shown as follows:

$$ $\displaystyle||\phi_{t}||^{2}$ $\displaystyle\leq(\sqrt{\alpha_{t+1}}-1)^{2}\bar{\alpha}_{t}||\mathbb{E}x_{0}| |^{2}$ $\displaystyle\leq(1-\alpha_{t+1})\bar{\alpha}_{t}||\mathbb{E}x_{0}||^{2}$ $\displaystyle\leq(\underbrace{\beta_{0}+\Delta_{\beta}\rho_{t+1}}_{\beta_{t+1} })\exp\{-(\underbrace{\beta_{0}+\frac{\Delta_{\beta}t}{2T}}_{\beta_{t/2}})t\}| |\mathbb{E}x_{0}||^{2}$ $\displaystyle\leq\beta_{\max}\exp\{-(\underbrace{\beta_{0}+\frac{\Delta_{\beta }t}{2T}}_{\beta_{t/2}})t\}||\mathbb{E}x_{0}||^{2}$ $$

where the inequalities are by Lemma 1, $(1-x)^{2}\leq(1-x^{2})=(1-x)(1+x)$, when $x\in[0,1]$, and $\beta_{t+1}\leq\beta_{\max}$

$$ $\displaystyle\Psi_{t}$ $\displaystyle=[2(1-\bar{\alpha}_{t})+\bar{\alpha}_{t}(\beta_{0}+\Delta_{\beta} \rho_{t+1})]\mathbf{I}$ $\displaystyle\succeq 2(1-\exp\{-(\underbrace{\beta_{0}+\frac{\Delta_{\beta}t}{ 2T}}_{\beta_{t/2}})t\})\mathbf{I}+\bar{\alpha}_{t}\beta_{t+1}\mathbf{I}$ $\displaystyle\succeq 2(1-\exp\{-(\underbrace{\beta_{0}+\frac{\Delta_{\beta}t}{ 2T}}_{\beta_{t/2}})t\})\mathbf{I}$ $$

where the inequalities are by Lemma 1 and $\bar{\alpha}_{t}\beta_{t+1}\mathbf{I}\succeq\mathbf{0}$. The residual term is

$$ $\bar{\alpha}_{t}\beta_{t+1}=\beta_{t+1}\Pi_{s=1}^{t}(1-\beta_{s})\geq\exp\{ \log\beta_{t+1}+t\log(1-\beta_{t})\}$ $$

∎

###### Proof.

## Appendix B More Experiment Results

##### Efficiency comparisons.

In Fig. 7, besides the Min-SNR and CLTS, we show the efficiency comparison with P2 and Log-Normal methods.
One can find that our method consistently accelerates the diffusion training in large margins.

Figure: Figure 7: More efficiency comparison on MetFaces.
Refer to caption: x14.png

Table: Table 9: Super resolution.

Super resolution with SpeeD. We employ SpeeD to super-resolution image generation on 512 $\times$ 512 MetFaces compared with vanilla DiT. We train DiT-XL/2 for 100K training iterations and compare the FID score at 50K, 100K training iterations. The batch size is $32$ for saving the GPU memory. As shown in 9, SpeeD obtain better performance than vanilla DiT at same training iterations on $512^{2}$ MetFaces dataset. It indicates that SpeeD can achieve training acceleration on super-resolution tasks.

### B.1 Additional Experiments

#### B.1.1 100K iterations cut on FFHQ

A cut of training process comparison on FFHQ dataset are shown in Tab. 10.

**Table 10: 100K iterations cut on FFHQ.**
| Schedules | linear | quadratic | cosine |
| --- | --- | --- | --- |
| DiT-XL/2 (Peebles & Xie, ) | 7.8 | 7.6 | 8.0 |
| ANT-UW (Go et al., ) | 6.3 | 7.1 | 8.3 |
| TskDif-Cur (Park et al., ) | 6.7 | 6.7 | 7.2 |
| P2 (Choi et al., ) | 10.1 | 10.2 | 12.3 |
| MinSNR (Hang et al., ) | 7.6 | 7.6 | 11.2 |
| Log-Normal (Esser et al., ) | 15.6 | 14.9 | 19.2 |
| CLTS (Xu et al., ) | 7.9 | 7.7 | 9.1 |
| SpeeD(ours) | 5.8 | 5.8 | 6.4 |

#### B.1.2 Guidance Scale

Ablation experiments about guidance scale in related conditional generation task are shown in Tab. 11.

**Table 11: Text-to-image (t2i) task on MS-COCO at 400K iteration and EDM on CIFAR-10 with 200M training images for class-conditional generation (ccg).**
| Guidance | 0.75 | 1 | 1.5 | 2 | 3 |
| --- | --- | --- | --- | --- | --- |
| t2i DiT-XL/2 | 20.1 | 22.5 | 27.4 | 31.0 | 36.2 |
| t2i SpeeD | 19.4 | 20.8 | 25.3 | 28.8 | 34.8 |
| ccg EDM | 1.8 | 1.8 | 1.9 | 1.9 | 2.5 |
| ccg SpeeD on EDM | 1.7 | 1.7 | 1.8 | 1.8 | 2.3 |

#### B.1.3 DiT Architecture

Ablation experiments about DiTs of different architectures are shown in Tab. 12.

**Table 12: Different size of DiT with 100K iterations on FFHQ.**
|  | S/2 | B/2 | XL/2 |
| --- | --- | --- | --- |
| DiTs | 18.1 | 12.9 | 7.8 |
| SpeeD | 15.3 | 10.8 | 5.8 |

#### B.1.4 Detailed Training Process

The detailed training process on FFHQ through 100K iterations are shown in Tab. 13.

**Table 13: Details about training to 100K on FFHQ.**
| Iterations (K) | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DiT-XL/2 | 356.1 | 335.3 | 165.2 | 35.8 | 12.9 | 11.9 | 10.5 | 9.6 | 8.7 | 7.8 |
| SpeeD | 322.1 | 320.0 | 91.8 | 19.8 | 9.9 | 7.6 | 7.1 | 6.6 | 6.2 | 5.8 |

## Appendix C More Related Works

We discuss other works related to SpeeD, including Text to Image and Video generation. Another point to mention is that we learn from InfoBatch (Qin et al., ) in writing.

Text to image generation with diffusion models Text-to-image generation has emerged as a hotly contested and rapidly evolving field in recent years, with an explosion of related industrial products springing up (Saharia et al., ; Rombach et al., ; Betker et al., ; Chen et al., ; Esser et al., ). Convert textual descriptions into corresponding visual content, models not only learn to synthesize image content but also ensuring alignment with the accompanying textual descriptions. To better align images with textual prompt guidance, previous work has primarily focused on enhancements in several schemes including strengthening the capacity of text encoder (Raffel et al., ; Radford et al., ) improving the condition plugin module in diffusion model (Zhang et al., ), improving data quality (Betker et al., ).

Video generation with diffusion models.
As diffusion models achieve tremendous success in image generation, video generation has also experienced significant breakthroughs, marking the field’s evolution and growth.
Inspired by image diffusion, pioneering works such as RVD Yang et al. () and VDM Ho et al. () explore video generation using diffusion methods. Utilizing temporal attention and latent modeling mechanisms, video diffusion has advanced in terms of generation quality, controllability, and efficiency Ho et al. (); Singer et al. (); Zhou et al. (); He et al. (); Zhang et al. (); Guo et al. (); Wu et al. (); Wang et al. (). Notably, Stable Video Diffusion Blattmann et al. () and Sora Brooks et al. () achieve some of the most appealing results in the field.

Other diffusion acceleration works
To achieve better results with fewer NFE steps, Consistency Models Song et al. () and Consistency Trajectory Models Kim et al. () employ consistency loss and novel training methods. Rectified Flow Liu et al. (), followed by Instaflow Liu et al. (), introduces a new perspective to obtain straight ODE paths with enhanced noise schedule and improved prediction targets, together with the reflow operation.

## Appendix D Visualization

Figure: Figure 8: Comaprisons to the baseline and other acceleration methods.
Refer to caption: x17.png

##### Comparisons to the baseline and other methods.

The Fig. 8 compares our method (Ours) with the baseline and other acceleration methods (P2 and Min-SNR) in terms of FID scores at various training iterations (K). The baseline method starts with a high FID score of 335.2 and gradually decreases to 12.8, showing slow convergence and less sharp final images. P2 begins with a slightly higher FID score of 357.9 and reduces to 15.0, improving faster than the baseline but still exhibiting slower convergence compared to Min-SNR and our method. Min-SNR starts with 334.1 and achieves a final score of 12.2, producing clearer and higher quality images consistently compared to the baseline and P2. Our method stands out, starting with a significantly lower FID score of 289.3 and rapidly dropping to 9.9, demonstrating the fastest convergence and the best overall image quality. This rapid decrease highlights the efficiency of our approach, requiring fewer iterations to produce high-quality results. Overall, our method outperforms the baseline, P2, and Min-SNR in both speed and quality of image generation, proving its effectiveness in accelerating the training of diffusion models.

##### Visualizations of the generated images.

The figures above illustrate the quality of images generated by our method across various datasets, including CIFAR-10, FFHQ, MetFaces, and ImageNet-1K. In Fig. 9, the generated images from the CIFAR-10 dataset display distinct and recognizable objects, even for challenging categories. Fig. 10 presents generated images from the FFHQ dataset, showcasing diverse and realistic human faces with varying expressions and features. Fig. 11 exhibits images from the MetFaces dataset, depicting detailed and lifelike representations of artistic portraits. Finally, Fig. 12 includes images from the ImageNet-1K dataset, featuring a wide range of objects and scenes with excellent accuracy and visual fidelity. These results emphasize the superior performance of our method in generating high-quality images across different datasets, indicating its potential for broader applications in image synthesis and computer vision tasks.

Figure: Figure 9: Generated images of CIFAR-10.
Refer to caption: x18.png

Figure: Figure 10: Generated images of FFHQ.
Refer to caption: x19.png

Figure: Figure 11: Generated images of MetFaces.
Refer to caption: x20.png

Figure: Figure 12: Generated images of ImageNet-1K.
Refer to caption: x21.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. **How do the re-sampling and weighting strategies differ?**  
   Re-sampling and weighting strategies share a similar motivation and produce identical effects in the objective function. Existing studies typically focus on either re-sampling or weighting, while SpeeD combines both approaches. Could the authors elaborate on the rationale for using both strategies instead of just one, and provide experimental evaluations to highlight the differences?
soundness: 3
strengths: 1. The paper addresses a critical issue—accelerating diffusion model training—and includes comparisons with existing methods.
2. The proposed analysis of process increment bounds is both novel and theoretically grounded.
3. The experiments are thorough and persuasive, showcasing the effectiveness of the proposed method.
summary: This paper introduces SpeeD, a novel approach aimed at improving training efficiency for diffusion models. By analyzing the process increment between adjacent timesteps, the authors categorize timesteps into acceleration, deceleration, and convergence zones based on process increment bounds. Through asymmetric sampling and change-aware weighting, SpeeD effectively reduces the number of timesteps required for training, as demonstrated by comprehensive experimental results.
weaknesses: 1. **Unclear motivation:** The paper introduces the analysis of $ \delta_t = x_{t+1} - x_t $ as a representation of each timestep, yet the motivation behind this choice could be clearer. Furthermore, the authors use $ \partial_t \hat{\Psi}_t $ for timestep categorization. While $ \delta_t $, representing the amount of change, seems more intuitive, the motivation for using $ \partial_t \hat{\Psi}_t $ lacks explanation. Additional insights into this choice would enhance the understanding of the approach.

2. **Limited applicability to general schedules:** The method relies on a linear schedule for discrete-timestep diffusion models. Although general cases are mentioned, extending SpeeD to other scenarios, such as EDM, and supporting this with experiments would strengthen the paper’s applicability.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: please see above
soundness: 2
strengths: The experimental validation of the article seems thorough. The proposed method is compared against several acceleration methods, and an ablation study is performed.
summary: The article proposes a methodology to accelerate training in diffusion models. Their proposal is presented in the context of other acceleration methods and builds on an analysis of the way training is performed in DMs. The proposed method is validates empirically in a number of experiments.
weaknesses: I have to admit that I found this paper hard to follow. 

The paper is redundant and lacks proper organisation. For instance, on the last page of the article, just before the conclusions, the authors revisit what a diffusion model is (while the entire paper is about DMs) and mention classifier guidance (which is not referred to in the paper whatsoever). 

The diagrams in the first pages (Figs 1,2 & 3) do not really help to clarify the contribution. They are cluttered, and it is difficult to understand what they are trying to illustrate. It is unclear whether these are based on data or just illustrations. In any case, it is unclear what the authors mean when the paper refers to these figures stating that they help to "visualize a loss curve" 

The detailed presentation of the proposed method is also unclear. The first 4 pages of the paper introduce the motivation and context, and back up the observations leading to the proposed method. However, in Sec 2.4 (?), the method is finally presented but without the necessary clarity. Is eq (3) the main definition of the procedure? 

Sec 2 is 3.5 pages long and covers the basics of DM and the brief presentation (see above) of the method. Then, the paper jumps directly to the experiments. 

Overall, I think that there is certainly some practical value in this article, but there a fair amount of work needed in term of presentation and organisation to take this paper at the acceptance bar at ICLR.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: See the weakness section, and also the following.

1. In the current approach, the asymmetric sampling does not differentiate between the accelerate and decelerate region and put both of them to high higher probability compared to the convergence region. However, I believe these two regions are differentiated in the reweighting part. Could the authors clarify if this is the case and provide a more thorough discussion on the reweighting strategy? Please also refer to weakness (1).
2. From table 3 in paper, it seems that the proposed method's effectiveness is not as significant when evaluated on FFHQ compared to Metfaces, could the author provide intuition why?
soundness: 3
strengths: This paper made several interesting contributions. Although there are other works which studies the importance of timesteps in diffusion model training, the analysis in the division of timesteps in this paper seems novel and technically solid. This novel observation then leads to a simple asymmetric sampling and reweighting approach. The paper is overall well-organized and well-presented. Many experiments are conducted to demonstrate the effectiveness of the proposed method, and the results are quite convincing. Finally, as the proposed methodology is architecture-agnostic and can be used in a plug-and-play manner in addition to many other acceleration methods, it could potentially be very versatile and be widely used.
summary: The paper studies an important problem of accelerating diffusion model training. The authors proposed a division of timesteps into three categories of acceleration/deceleration/convergence. The author provides evidence that the convergence region is of no big importance to the training, and thus propose an asymmetric sampling strategy SpeeD to focus on the earlier timesteps. Multiple empirical results are shown to demonstrate the effectiveness of the proposed methodology.
weaknesses: 1. Although the paper is overall well-presented and well-motivated, the presentation of some sections can be made clearer. For example, could the authors elaborate on the rescaling scheme in section 2.6?
2. The authors provide ablation studies showcasing the transferability of SpeeD into other types of schedule. However, it seems that the improvement regarding training speed for the quadratic and cosine schedulers are not as strong compared to linear scheduler. As the theoretical analysis is solely based on linear scheduler, can similar results be provided in the quadratic or cosine scheduler case? 
3. Contrary to previous works, the authors argues one should put higher probability in sampling earlier timesteps (up to a threshold $\tau$) rather than the middle timesteps. Could the authors provide some empirical ablations on the choice of the threshold $\tau$?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the weakness.
soundness: 3
strengths: 1. The long training process of the diffusion model is a critical issue for computational cost.
2. A scheduling mechanism is proposed to dynamically to adjust the sampling strategy.
3. The proposed method is evaluated on two datasets by comparing it with several baselines.
summary: This paper proposes to speed up the training process of the diffusion model with an adaptive sampling strategy. The training process is
 empirically divided into 3 stages based on increment: acceleration, deceleration, and convergence. Then the sampling strategy will reduce the frequency of steps from the convergence area. The importance of time steps is also considered. Five baselines are introduced compared with the proposed method on 2 datasets. Overall, this paper studies an important issue of diffusion models. However, there are major flaws and the experiment quality does not allow the acceptance of this paper.
weaknesses: 1. The rationale behind the motivation is not clearly stated and verified. The author states that the time steps can be empirically divided into 3 states. However, there is no empirical result to support the claim. All figures in the method part (i.e., Figure 1 and Figure 2) are pseudo figures. The value in real experiments should be provided. 

2. It is not practical to decide the boundary of each state in real application. Also, it is not very clear how to decide the boundary of each state. If the depends on the convergent speed, the states could vary significantly based on the learning rate, model framework, and the quality of training data. Such a strategy is not practical. In fact, the analysis is not comprehensive. Is it possible that an adaptive learning rate will address the issue? The author should exclude other factors to verify it is the sampling quality, not another factor that results in the difference between the 3 stages.

3. The whole paper assumes that the diffusion model is DDPM. However, there are so many papers[1] that have already addressed the quality of sampling such as DDIM. The author should address the problem with a SOTA framework regarding efficiency.

4. The quality of the experiment is low. While the motivation is to speed up the training process, is it intuitive to report the real training time? The majority of the experiment is report FID with baselines. FID is not the metric to verify the efficiency. Figures 5 and 6 are used to report the convergent speed. However, it looks Log(FID) is not convergent yet. Also, what is the learning rate for each baseline in Figure 5? Why there are 3 sub-figures in Figure 5? Should it be a single figure including a comparison with all baselines?

5. Ablation study is missing. The author should remove the sampling strategy for each stage and vary the boundary. The presentation in the experiment could be improved.

6. Important baselines are missing including [2,3,4, 5] and many others. 

[1] Shivam Gupta, Ajil Jalal, Aditya Parulekar, Eric Price, Zhiyang Xun:
Diffusion Posterior Sampling is Computationally Intractable.
[2] Tae Hong Moon, Moonseok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, Jaewoong Cho, Juho Lee:
A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models. ICML 2024
[3] Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, Tsung-Hui Chang:
Accelerating Parallel Sampling of Diffusion Models. ICML 2024
[4] Towards Faster Training of Diffusion Models: An Inspiration of A Consistency
Phenomenon
[5] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, Anima Anandkumar:
Fast Sampling of Diffusion Models via Operator Learning. ICML 2023: 42390-42402
[6] 	Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, Nima Anari:
Parallel Sampling of Diffusion Models. NeurIPS 2023

Overall, there are major drawbacks in the proposed method and the quality of the experiment can be significantly improved.