Denoising Task Difficulty-based Curriculum for Training Diffusion Models

## Abstract

Abstract Diffusion-based generative models have emerged as powerful tools in the realm of generative modeling.
Despite extensive research on denoising across various timesteps and noise levels, a conflict persists regarding the relative difficulties of the denoising tasks.
While various studies argue that lower timesteps present more challenging tasks, others contend that higher timesteps are more difficult.
To address this conflict, our study undertakes a comprehensive examination of task difficulties, focusing on convergence behavior and changes in relative entropy between consecutive probability distributions across timesteps.
Our observational study reveals that denoising at earlier timesteps poses challenges characterized by slower convergence and higher relative entropy, indicating increased task difficulty at these lower timesteps.
Building on these observations, we introduce an easy-to-hard learning scheme, drawing from curriculum learning, to enhance the training process of diffusion models.
By organizing timesteps or noise levels into clusters and training models with ascending orders of difficulty, we facilitate an order-aware training regime, progressing from easier to harder denoising tasks, thereby deviating from the conventional approach of training diffusion models simultaneously across all timesteps.
Our approach leads to improved performance and faster convergence by leveraging benefits of curriculum learning, while maintaining orthogonality with existing improvements in diffusion training techniques.
We validate these advantages through comprehensive experiments in image generation tasks, including unconditional, class-conditional, and text-to-image generation.

## 1 Introduction

Diffusion-based generative models [, , ] have achieved significant advancements in the realm of generative tasks, demonstrating notable success across various fields such as image [], video [, ], and 3D [, ] generation.
Specifically, their exceptional adaptability and promising performance in diverse image generation contexts, such as unconditional [, ], class-conditional [], and text-conditional scenarios [, ], demonstrate their significant impact.
Such achievements have led to a growing interest in further deepening the analysis and enhancing diffusion models.

Diffusion models [, ] are designed to reverse the corruption of the data through the learning process at different noise levels and over multiple timesteps.
Recent works have delved into the learning of diffusion models across various noise levels and timesteps, revealing different stages of diffusion models.
For example, Choi et al. [] observe that when a diffusion model performs a denoising task from large to small timestep, it first generates coarse features, then gradually generates perceptually rich content, and later refines the details.
Similar observation is also identified in text-to-image diffusion models [].
Besides this aspect, various studies have further explored the learning of diffusion models across timesteps and noise levels, elucidating their transition from denoising to generative functionalities [], modular attributes [], frequency characteristics [, ], trajectories [], affinity [], and variations of targets [].

These observations have not only deepened understanding of diffusion models but have also directly contributed to improvement in diffusion models.
Specifically, these insights are incorporated into their method design in various works, including loss functions [, , ], architectures [, ], accelerated sampling [], representations [], and guidance [].
Given the tangible benefits already realized from such studies, further in-depth analysis of diffusion models across timesteps and noise levels is crucial for uncovering insights and achieving unprecedented advancements in their capabilities.

In this paper, to enrich the current understanding across various timesteps and noise levels, we investigate under-explored areas within diffusion models focusing on the task difficulties of denoising.
Regarding denoising task difficulties, previous works speculate that denoising tasks across timesteps have different difficulties [, ], yet a detailed exploration of these variances remains sparse.
Moreover, there exists a notable discrepancy among studies, with works identifying larger timesteps as more difficult [, ], while others argue that smaller timesteps pose greater difficulties [, , ].
The discrepancy in difficulty across timesteps not only impedes the accurate interpretation of previous studies but also hinders the development of sophisticated training methods that properly utilize the timestep-wise variation in difficulty.

In this regard, we first analyze task difficulties in two aspects to resolve these conflicts: 1) convergence properties in the learning of denoising tasks at each timestep, and 2) the change in relative entropy between consecutive probability distributions over timesteps.
In the first aspect, our analysis reveals distinct convergence behaviors across timesteps, demonstrating that models trained on larger timesteps exhibit faster convergence.
In the second aspect, we also observe a decrease in relative entropy as we progress to later timesteps.
By integrating these, we confirm that denoising tasks at earlier timesteps are more difficult, indicated by slower convergence speeds and greater changes in relative entropy.

Furthermore, building on these observations, we integrate an easy-to-hard learning scheme, a concept well-established in the curriculum learning literature [, , , , ], into the training process of diffusion models.
Specifically, we organize timesteps or noise levels into clusters and train the diffusion models with ascending levels of difficulty, moving from clusters categorized by higher to lower timesteps.
After this curriculum process, models simultaneously learn whole timesteps as standard diffusion training [, , ] to reach the convergence point.
Unlike conventional approaches where diffusion models are trained simultaneously across all timesteps, our method distinguishes itself by incorporating a sequential, order-aware training regime, reflecting an intended progression from easier to harder denoising tasks.

Building upon this foundation, our curricular approach offers several notable advantages: 1) Improved Performance and 2) Faster Convergence: By leveraging the inherent benefits of curriculum learning, our method significantly enhances the quality of generation and the speed of convergence.
3) Orthogonality with Existing Improvements: Our approach is inherently model-agnostic, ensuring broad applicability across various diffusion models. Additionally, it can be integrated with advanced diffusion training techniques, such as loss weighting [, , , ].

Finally, we empirically validate the advantages of our method by conducting comprehensive experiments across a variety of image-generation tasks. These include unconditional generation, class-conditional generation, and text-to-image generation, utilizing datasets such as FFHQ [], ImageNet [], and MS-COCO [].
By integrating our curriculum learning strategy into architectures— DiT [], EDM [], and SiT []—we demonstrate the efficacy of our approach in enhancing performance, accelerating convergence speed, and maintaining compatibility with existing techniques.

## 2 Related Works

### 2.1 Diffusion Models

Diffusion models [, , ] are a group of generative models that create samples by utilizing a learned denoising process to noise.
Several works have focused on improving diffusion models in various aspects, including model architectures [, , ], sampling speed [, , ], training objectives [, , , , ].
These endeavors often involve investigating what diffusion models learn by dividing its process, aiming to enhance the performance of diffusion models.
P2 [] under-weights training loss functions at the clean-up stage from their observation that diffusion models learn coarse, perceptual, and removing noises at large, medium, and small timesteps.
Ediff-I [] observes that earlier sampling parts rely on conditions for generation, whereas later parts ignore the conditions. They employ multiple denoisers to address the diverse characteristics of tasks associated with different parts of the sampling process.
Moreover, various works have also investigated these aspects related to timesteps [, , , , , , ] (detailed illustrations can be found in Appendix A).
While our study aligns with the above works, we analyze the under-explored aspect of denoising task difficulty.
Furthermore, we leverage these observations to propose a curriculum learning approach.

### 2.2 Denoising Difficulties on Diffusion Models

Difficulties in denoising tasks in diffusion have been referred to by various works, but this aspect is not deeply explored.
Several studies hypothesize that denoising tasks in diffusion encompass diverse difficulties [, ], and there have been conflicts regarding these difficulties between previous works.

Certain studies consider denoising at larger noise levels and timesteps to be more difficult [, ], the focus is on the challenges associated with reconstructing data from substantial noise.
For instance, Hang et al. [] articulate that while smaller timesteps (approaching zero) may require straightforward reconstructions, such strategies become less effective at higher noise levels or in larger timesteps.
Similarly, Ho et al. [] elucidate that their approach de-emphasizes loss terms at smaller timesteps to prioritize learning on the more challenging tasks at larger timesteps, thereby enhancing sample quality.
Conversely, other studies argue that earlier timesteps or lower noise levels also present significant challenges.
Karras et al. [] suggest that detecting noise at low levels is challenging due to its minimal presence.
Also, Kim et al. [] illustrate the increasing difficulty and high variance in score estimation as timesteps approach zero, disturbing stable training of models.
In line with these observations, Dockhorn et al. [] build upon insights of [], acknowledging the complexities at near zero timesteps, where score becomes highly complex and potentially unbounded.

In this work, we aim to resolve this conflict through an in-depth analysis of convergence properties and changes in relative entropy between consecutive probability distributions across timesteps.

### 2.3 Curriculum Learning

Curriculum learning [, , ], inspired by human learning patterns, is a method of training models in a structured order, starting with easier tasks [] or examples [] and gradually increasing difficulty.
As pointed out by [], curriculum learning formulation can be viewed as a continuation method [], which starts from a smoother objective and gradually transformed into a less smooth version until it reaches the original objective function.
Through this foundation, various works have achieved improved performance and faster convergence compared to standard training based on random mini-batches sampled uniformly [, , , ].

Curriculum learning primarily comprises two components: a curriculum scoring function, measuring the difficulty of tasks or examples, and a pacing function, modulating the speed of the curriculum progress.
Regarding a curriculum score function, early studies have utilized human intuition for measuring difficulty, such as the complexity of geometric shapes in images [] or the length of sequences [].
Recently, various works employ models to measure difficulty, including confidence of pre-trained models [] and the loss of the current models [].
For the pacing function, a predefined pacing function has been employed, which involves training using a predetermined curriculum progression [, ]. There are various forms of this and they can be generally represented as a function of training iteration [, ].
Contrary to this, there have been proposals for pacing techniques dynamically adjusting based on the loss or performance of the current model during training [, ].

In the diffusion model literature, curriculum learning has been utilized to organize the order of training data types based on prior knowledge of targeted generation tasks [, ].
Tang et al. [] sequentially train video diffusion models with lower resolution and FPS datasets before progressing to higher resolution and FPS datasets.
Similarly, Yang et al. [] order text-to-sound generation data based on the number of events in audio clips, training diffusion models from lower to higher events datasets.
In contrast, our method explores the nature of denoising task difficulty in diffusion models and proposes a curriculum learning approach that progresses from easy to hard timesteps, deviating from the standard simultaneous training of all timesteps.
Also, while consistency models [, ] adopt a curriculum approach to discretizing noise levels, progressively increasing the discretization steps of noise levels during training, we have distinct by exploring which noise level should be learned first and investigating the difficulty of denoising at each noise level.

## 3 Preliminaries

In this section, we provide the necessary background on diffusion models [, , ].
Let $\bm{x}_{0}\in\mathbb{R}^{d}$ be a sample from the data distribution $p_{0}(\bm{x})$.
The forward process of diffusion models transforms data $\bm{x}_{0}$ to latent $\bm{x}_{t\in[0,T]}$ by iteratively adding Gaussian noise.
This can be formulated as a stochastic differential equation (SDE) [] as $\text{d}\bm{x}_{t}=f(t)\bm{x}_{t}\text{d}t+g(t)\text{d}\bm{w}_{t}$,
where $f(t)$ and $g(t)$ are drift and diffusion coefficients, and $\bm{w}_{t}$ is the standard Wiener process.
The Gaussian transition kernel of this SDE is formulated as:

$$ $p_{0t}(\bm{x}_{t}|\bm{x}_{0})=\mathcal{N}(\bm{x}_{t};s_{t}\bm{x}_{0},s_{t}^{2} \sigma_{t}^{2}\mathbf{I}),\;\;s_{t}=\text{exp}\left(\int_{0}^{t}f(\xi)\text{d} \xi\right),\;\;\sigma_{t}=\sqrt{\int_{0}^{t}\frac{g(\xi)^{2}}{s_{\xi}^{2}} \text{d}\xi}.$ (1) $$

For generation, diffusion models aim to learn the corresponding reverse SDE represented as:

$$ $\text{d}\bm{x}_{t}=\left[f(t)\bm{x}_{t}-g^{2}(t)\nabla\log p_{t}(\bm{x}_{t}) \right]\text{d}\bar{t}+g(t)\text{d}\bar{\bm{w}}_{t},$ (2) $$

where $\bar{\bm{w}}_{t}$ and $\text{d}\bar{t}$ denote the reverse-time Wiener process and the infinitesimal reverse-time, respectively, with the actual data score $\nabla\log p_{t}(\bm{x}_{t})$.
In most cases, a neural network $\epsilon_{\theta}$ having parameter $\theta$ is utilized to approximate this score function by learning the denoising tasks for each timestep $t$ from score matching loss $\mathcal{L}$ []:

$$ $\mathcal{L}=\frac{1}{2}\int^{T}_{0}\mathcal{L}_{t}\text{d}t,\quad\mathcal{L}_{ t}=\omega(t)\mathbb{E}_{\bm{x}_{t}\sim p_{0t}(\bm{x}_{t}|\bm{x}_{0}),\bm{x_{0} }\sim p_{0}}\left[||\epsilon_{\theta}(\bm{x}_{t},t)-\nabla\log p_{0t}(\bm{x}_{ t}|\bm{x}0)||_{2}^{2}\right],$ (3) $$

where $\omega(t)$ is loss weights for $t$ and $p_{0t}(\bm{x}_{t}|\bm{x}_{0})$ is the transition density of $\bm{x}_{t}$ from the initial timestep 0 0 to $t$.
This object can be interpreted as a noise-matching loss in DDPM [], which predicts noise components in $\bm{x}_{t}$ and can be illustrated as $\int_{0}^{T}\mathbb{E}_{\bm{x}_{0}\sim p_{0},\epsilon\sim\mathcal{N}(0,\mathbf
{I})}[||\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{
\alpha}_{t}}\epsilon,t)-\epsilon||_{2}^{2}]\text{d}t$.
This is regularly denoted as $\epsilon$-prediction parameterization [, ], and several other parameterizations including $F$-prediction [, ], score-prediction [] and velocity-prediction [] have been proposed.

## 4 Observations

In this section, we examine the difficulties associated with learning denoising tasks across different timesteps, addressing inconsistencies in prior works regarding these difficulties.
Our analysis is structured around two key aspects: 1) the convergence of loss and denoising performance across timesteps, providing insights into learning dynamics at various timestep stages in Section 4.1; and 2) the relative entropy change from $p_{t}$ to $p_{t-1}$ as a function of $t$, offering a quantitative measure of task difficulty progression over $t$ in Section 4.2.
Upon integrating our findings, we establish a key conclusion: the learning difficulty for denoising tasks escalates as the timestep $t$ decreases.

### 4.1 Analysis on the Task Difficulty in terms of Convergence

In this study, we analyze the convergence speed of loss and denoising performance across timesteps.
To comprehensively cover various diffusion parameterizations, we utilized the notable frameworks DiT [] for $\epsilon$-prediction, EDM [] for $F$-prediction, SiT [] for velocity prediction.
Detailed descriptions of the experimental setups are provided in Appendix B.

#### Convegence speed on loss.

First, we analyze convergence characteristics of training loss across timesteps $t$.
We divided whole timesteps $[0,T]$ into 20 uniformly divided intervals and trained 20 models $\{\mathrm{M}_{i}\}_{i=1}^{20}$ where $i$-th model learns denoising tasks in $[\frac{i-1}{20}T,\frac{i}{20}T]$ for DiT and SiT, $[\mathrm{\Phi}^{-1}(\frac{i-1}{N}),\mathrm{\Phi}^{-1}(\frac{i}{N})]$ for EDM where $\mathrm{\Phi}^{-1}$ is the inverse cumulative distribution function of the Gaussian distribution.
During training, we tracked the loss values through iterations and plotted their convergence speed by normalizing their value in Figs. 1(a)-1(c).
As shown in the results, it is apparent that as $i$ increases towards $i=20$, the convergence accelerates in both DiT, EDM, and SiT, suggesting that models learning larger timesteps can reach convergence more swiftly and reinforcing the notion that denoising tasks with larger timesteps are less difficult.

#### Convegence speed on denoising performance.

We also delve deeper into a convergence of denoising performance according to timesteps with 20 distinct models $\{\mathrm{M}_{i}\}_{i=1}^{20}$.
To evaluate the performance of denoising tasks of each model, we generated samples where $\mathrm{M}_{i}$ was employed for denoising within the timesteps that it was trained on, while a diffusion model learned whole timesteps handled denoising for the remaining timesteps as in [].
Then, the performance of the denoising capability of $\mathrm{M}_{i}$ can be quantitatively measured using the FID score [], enabling us to observe the performance convergence of each model on denoising tasks throughout the training process.
Figures 1(d)-1(f) depict this convergence.
They illustrate that, similar to loss convergence experiments, denoising performance converges more swiftly for models $\mathrm{M}_{i}$ with larger $i$ values, as observed across the DiT, EDM, and SiT.
These results also suggest that models trained on later timesteps, indicated by larger $i$ values, achieve faster convergence, highlighting easier task difficulty at larger timesteps.

Figure: (a) DiT (Loss convergence)
Refer to caption: x1.png

### 4.2 Exploration on Difficulties of Denoising Tasks

Beyond empirical convergence metrics, we also delve into analyzing the relative entropy between $p_{t}$ and $p_{t-1}$ to better understand task difficulties from a distributional perspective.
The training of diffusion models implicitly involves learning the distribution of the reverse process of the corresponding SDE.
To be specific, the transition probability of the reverse process is expressed as a conditional normal distribution whose mean parameter is modeled by neural networks, and they are thereby trained to learn the dynamics of the reverse process [].
Furthermore, an unconditional distribution of $\bm{x}_{t}$ can be obtained by marginalizing transition densities over the prior distribution, indicating that information on the dynamics of the marginal distribution is fed to neural networks [].

Figure: Figure 2: The KLD of $p_{t-1}$ from $p_{t}$ against denoising timestep. As the timestep increases, the dynamics decrease.
Refer to caption: x8.png

To analyze the relationship between the dynamics of the unconditional distribution and the rate of loss convergence, we use the Kullback-Leibler (KL) divergence of $p_{t-1}$ to $p_{t}$, $D_{KL}(p_{t-1}||p_{t})$, as a quantitative measure. It is a pertinent divergence in that the training mechanism of diffusion models involves maximizing the likelihood of the reverse process.
The KL divergence $D_{KL}(p_{t-1}||p_{t})$ is given by $D_{KL}\left(p_{t-1}||p_{t}\right)=\mathbb{E}_{\bm{x}\sim p_{t-1}}\left[\log
\left(\frac{p_{t-1}(\bm{x})}{p_{t}(\bm{x})}\right)\right]$.
Moreover, the distribution $p_{t}$ of $\bm{x}$ at $t$ is expressed by $p_{t}(\bm{x}_{t})=\int p_{0t}(\bm{x}_{t}|\bm{x}_{0}=\bm{y})p_{0}(\bm{y})\text{
d}y=\mathbb{E}_{\bm{x}_{0}\sim p_{0}}\left[p_{0t}\left(\bm{x}_{t}|\bm{x}_{0}
\right)\right]$.
However, since the explicit density form of $p_{0}$ is unknown and it is computationally infeasible to estimate high-dimensional integrals, we approximate them through unbiased estimators (details in Appendix C).
The empirical results of $D_{KL}(p_{t-1}||p_{t})$ for $64\times 64$ image data are given in Fig. 2.
As seen, the relative entropy tends to decrease as $t$ increases (i.e., $D_{KL}\left(p_{s-1}||p_{s}\right)\leq D_{KL}\left(p_{t-1}||p_{t}\right)$ for $s\leq t$), which is consistent with the results in Fig. 1.

This observation may stem from the inherent low-dimensional manifold of image data.
As is well-known (e.g., []), the image data is distributed on a relatively low-dimensional manifold with a narrow support and a highly peaked multi-modal structure.
On the other hand, as Gaussian noise is iteratively added, the distribution of $\bm{x}_{t}$ approaches the independent Gaussian distribution in the ambient space.
Consequently, the support of the manifold broadens and the score function becomes regular over the ambient space with increasing $t$.
This nature of the unconditional distribution may cause the relative entropy from $p_{t}$ to $p_{t-1}$ to decrease with $t$, indicating that it is more difficult to accurately represent the dynamics of the reverse process at small $t$.

## 5 Methodology

In Section 4, we observe that denoising tasks at smaller $t$ are more difficult to learn by models.
From these order of difficulties in denoising tasks, we propose the incorporation strategy of an easy-to-hard training scheme, that has demonstrated its effectiveness in curriculum literature [, , ], for improving diffusion models’ training.

### 5.1 Design of Curriculum Learning in Diffusion Models

Figure: Figure 3: The overview of our curriculum learning approach for diffusion models. (Left) We divide the timesteps into $N$ clusters, ${C_{1},...,C_{N}}$, with the difficulty of denoising tasks increasing from $C_{N}$ (easiest) to $C_{1}$ (hardest). (Right) As the curriculum progresses, learning accumulates harder task clusters, gradually increasing task difficulties.
Refer to caption: x9.png

As we observed in Section 4, difficulties in denoising tasks increase as $t$ gets smaller.
To utilize an easy-to-hard curriculum learning approach, we first divide the entire range of timesteps into $N$ clusters, denoted as $\{C_{i}\}_{i=1}^{N}$, where each cluster $C_{i}$ spans an interval $[l_{i},l_{i+1}]$, ensuring $l_{i}<l_{i+1}$, with $l_{1}=0$ and $l_{N+1}=T$, as shown on the left side of Fig. 3.
The curriculum for training is constructed by regarding these task clusters as unit tasks, starting from the least challenging (the $N$-th cluster $C_{N}$) and advancing towards the most difficult (the first cluster $C_{1}$), through $N$ distinct stages.
Specifically, in the $n$-th curriculum stage, we jointly train the model with denoising tasks sampled from the clusters $\bigcup_{j=N-(n-1)}^{N}C_{j}$ as illustrated in the right side of Fig. 3.
The transition of the curriculum stages is determined by the pacing function, which will be discussed in the next section.
After completing these $N$-stages of curriculum learning, the model continues to learn across the entire range of timesteps, $\bigcup_{j=1}^{N}C_{j}$, same as standard diffusion training.

The next consideration involves determining the boundaries for each cluster $l_{i}$.
A straightforward approach is to uniformly divide the entire timestep interval $[0,T]$ as $C_{i}=[\frac{(i-1)\cdot T}{N},\frac{i\cdot T}{N}]$ for $i=1,2,\ldots,N$.
However, this method does not account for variations in noise levels across different timesteps.
Therefore, to address this issue more effectively, we adopt an SNR-based interval clustering technique as used in [], which aligns the clustering with the actual changes in noise levels, potentially enhancing curriculum learning adaptability to varying noise conditions.

For EDM [] which operates based on the noise level $\sigma$ rather than the timestep $t$, and where $\sigma$ is sampled from a log-normal distribution such that $\log(\sigma)\sim\mathcal{N}(P_{\text{mean}},P_{\text{std}}^{2})$ during training, our clustering strategy for timesteps cannot be directly transposed.
Given the log-normal distribution of $\sigma$, dividing it directly is impractical because $\sigma$ can extend over a wide range of values.
Instead, we adapt our clustering approach to suit the log-normal characteristics by defining noise level clusters $C_{i}$.
Specifically, we delineate $C_{i}=[\mathrm{\Phi}^{-1}(\frac{i-1}{N}),\mathrm{\Phi}^{-1}(\frac{i}{N})]$, where $\mathrm{\Phi}^{-1}$ is the inverse cumulative distribution function (quantile function) of the Gaussian distribution $\mathcal{N}(P_{\text{mean}},P_{\text{std}}^{2})$.
This method segments the noise levels into intervals by reflecting their probabilistic distribution.

### 5.2 Pacing Strategy of Curriculum

To effectively train the diffusion model according to the provided curriculum design, it is crucial to define a suitable pacing function for determining the transition of each $N$ distinct curriculum.
Training for a fixed number of iterations for each curriculum stage is the simplest implementation (We also contain this method in experiments as ‘NaiveCL’ in Section 6.2).
However, the convergence rate of each curriculum phase varies significantly, as demonstrated in Fig. 1.
Hence, we propose adopting an adaptive number of iterations for each curriculum, akin to the varied exponential pacing approach explored by Hacohen et al. [].
Our pacing function utilizes the training loss to determine transition moments and transitions to the next stage occur when the training loss converges at the current stage.
Specifically, we introduce the maximum patience iteration $\tau$, and if the loss does not improve consecutively for $\tau$, the current curriculum stage is terminated, and the subsequent curriculum stage is initiated.
Here, the maximum patience is a fixed hyper-parameter, and the detailed process and overall curriculum learning procedure are outlined in Algorithm 1 and 2 in Appendix D, respectively.

## 6 Experimental Results

In this section, we present experimental results to validate the effectiveness of our method.
The advantages of our curriculum method, 1) Improved Performance, 2) Faster Convergence, and 3) Orthogonality with Existing Improvements, are validated in this section.
To begin, we outline our experimental setups in Section 6.1.
Then, we provide the results of the comparative evaluation in Section 6.2, showing that our curriculum approach significantly improves the quality of generated samples compared to the baseline.
Finally, analyses of our method are illustrated in Section 6.3 to deeply understand the effectiveness of our method.

### 6.1 Experimental Setup

Here, we provide experimental setups concisely. Detailed setups are presented in Appendix E.

#### Evaluation protocols.

For our comprehensive evaluation of various methods, we employed three distinct image-generation tasks:
1) Unconditional generation with the FFHQ dataset [], 2) Class-conditional generation with CIFAR-10 [] and ImageNet [] datasets, and 3) Text-to-Image generation with MS-COCO dataset [].
In 2) and 3) setups, we applied classifier-free guidance [].

#### Target models.

We employed three exemplary diffusion architectures for experiments: DiT [], which integrates latent diffusion models [] with Transformer architectures [] parameterized as $\epsilon$-prediction, EDM [], which focuses on pixel-level diffusion utilizing UNet-based architectures [] parameterized as $F$-prediction, and SiT [] for score- and velocity-prediction.
For the text-to-image generation, we incorporated a CLIP text encoder [] as described in DTR [].

### 6.2 Comparative Results

In this section, we assess the effectiveness of our curriculum-based training approach. For a thorough comparison, we examine three distinct training variants, with further details provided in Appendix E:
To achieve this, we compare three variants of training:
1) Vanilla: This term refers to diffusion models trained using conventional methods without any curriculum learning strategies;
2) NaiveCL: In this variant, we incorporate a basic curriculum learning strategy, which simply repeats the same number of iterations for each stage across an $N$-stage process and does not employ SNR-based clustering;
3) Ours: This denotes our proposed curriculum approach, which is designed to enhance the training process of diffusion models by systematically structuring the learning stages.

#### Quantitative evaluation.

We quantitatively validate the effectiveness of our methods with various architectures-DiT [], EDM [], and SiT []- and tasks including unconditional, class-conditional, and text-to-image generation.
Table 1 shows the results, confirming two empirical observations: 1) NaiveCL fails to consistently achieve improved performance compared to Vanilla, and 2) our approach outperforms both NaiveCL and Vanilla.
Regarding the first observation, NaiveCL shows inconsistent improvements due to its lack of robust adaptation on incorporating task difficulties in various task conditions.
In contrast, our method demonstrates superior performance across all scenarios by improving the clustering and pacing of curriculums.
Consequently, our approach consistently achieves significant performance enhancements across all metrics on four datasets: FFHQ [], ImageNet [], CIFAR-10 [], and MS-COCO [], illustrating its effectiveness regardless of data or model used.

**Table 1: We evaluated unconditional image generation on FFHQ with DiT-B, EDM, and SiT-B, class-conditional image generation on ImageNet and CIFAR10 with DiT-L and EDM, respectively, and text-conditional image generation on MS-COCO with DiT-B. Note that our curriculum learning for diffusion models improves substantial performance without any additional parameters.**
| $\epsilon$-prediction |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | FFHQ 256$\times$256 | ImageNet 256$\times$256 | COCO 256$\times$256 |  |  |  |
| FID$\downarrow$ | FID$\downarrow$ | IS$\uparrow$ | Prec$\uparrow$ | Rec$\uparrow$ | FID$\downarrow$ |  |
| DiT (Vanilla) | 10.49 | 11.18 | 146.95 | 0.75 | 0.47 | 7.62 |
| DiT + NaiveCL | 7.95 | 11.90 | 151.66 | 0.75 | 0.47 | 7.71 |
| DiT + Ours | 7.55 | 8.18 | 186.37 | 0.79 | 0.47 | 7.51 |

#### Qualitative evaluation.

Due to space constraints, we illustrate a detailed collection of generated examples in Appendix F.
In summary, our curriculum methodology demonstrates a notable enhancement in the quality of the images produced, when compared to NaiveCL and Vanilla.

### 6.3 Analysis

To elucidate our curriculum approach’s effectiveness, we present a series of analytical studies.
All the analysis is conducted by using the DiT-B model on the ImageNet dataset.

Figure: Figure 4: Ablation study on $N$ and $\tau$. We use DiT-B on ImageNet 256$\times$256.
Refer to caption: x10.png

#### Effects of N 𝑁 N italic_N and τ 𝜏 \tau italic_τ .

We examined the robustness of the proposed curriculum training with respect to hyper-parameters: the number of clusters $N$ and the maximum patience $\tau$.
As shown in Fig. 4, our method consistently outperforms the vanilla model, and the best result is observed at $N=20,\tau=200$.
It shows that as $\tau$ increases, it may lead to overtraining due to excessive iterations for each task, whereas as $\tau$ decreases, curriculums may not be sufficiently trained.
Furthermore, when the entire range of timesteps is finely partitioned (i.e., with an increase in $N$), each cluster becomes excessively granular, resulting in suboptimal performance.
Conversely, with a decrease in $N$, tasks that should be in distinct clusters are learned together, forming a coarser cluster, which also leads to suboptimal outcomes.
Overall, our method outperforms vanilla training across a range of hyperparameters, demonstrating the robustness of our approach.

#### Effects of Curriculum Design.

**Table 2: Comparative results on various curriculum designs.**
| Class-Conditional ImageNet 256$\times$256. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Curriculum Design | FID$\downarrow$ | IS$\uparrow$ | Prec$\uparrow$ | Rec$\uparrow$ |  |
| (a) | Vanilla | 30.27 | 60.06 | 0.55 | 0.52 |
| (b1) | + anti-curriculum + uniform | 31.12 | 62.80 | 0.55 | 0.53 |
| (b2) | + anti-curriculum + SNR | 27.74 | 68.10 | 0.58 | 0.52 |
| (c1) | + curriculum + uniform | 25.01 | 71.99 | 0.58 | 0.53 |
| (c2) | + curriculum + SNR | 22.96 | 75.98 | 0.62 | 0.52 |

In our curriculum design, we initially partitioned the entire set of timesteps into $N$ clusters using SNR-based clustering, organizing the curriculum from easy to hard clusters.
To thoroughly assess the impact of each component, we conducted the ablation study as shown in Table 2.
Firstly, we investigated the effect of curriculum learning via comparison with an anti-curriculum approach [], which progresses from hard to easy clusters, unlike conventional curriculum learning.
While both training methods in (b2) appear to enhance performance compared to vanilla training (a), anti-curriculum training cannot consistently guarantee performance improvement concerning the curriculum design as shown in (b1).
In contrast, the proposed curriculum learning method (c1, c2) consistently exhibited performance improvement even with the uniformly partitioned clusters.
Besides, with findings that utilizing SNR-clustering was more effective, clustering with the actual changes in noise levels enhanced the curriculum learning adaptability.

#### Visualization of curriculum.

To gain deeper insights into the functioning of our curriculum pacing, we plotted loss metrics against curriculum phases, as illustrated in Fig. 6.
During the curriculum training, tasks progressively transition from the easiest to the most challenging, with varying amounts of iterations for each task based on the pacing function.
The training loss decreased during each curriculum phase but increased after curriculum changes via the pacing function due to the inclusion of a newly added task in the updated curriculum.
Additionally, as $\tau$ increases, the curriculum phases change more gradually, highlighting the role of $\tau$ in controlling the pace of curriculum transitions.

#### Analysis on convergence speed.

As demonstrated in previous works [, ], the adoption of curriculum learning can lead to faster convergence in model performance.
To illustrate the efficacy of our approach in this regard, we plotted the FID, IS, precision, and recall calculated over 10,000 samples across the training iterations, as depicted in Fig. 6.
We observed the models trained through the proposed curriculum learning method converge faster than vanilla models, regardless of evaluation metrics.
Notably, our approach achieves these improvements without requiring additional parameters or training iterations, thereby significantly saving time and computational resources.

Figure: Figure 5: We visualized the curriculum transition and the corresponding loss across iterations ($N=20$). To make the loss graph more easily readable, the y-axis was truncated to 1.0.
Refer to caption: x11.png

#### Effectiveness on various sizes of models

To verify the generalizability of our method across different model sizes, we evaluated the performance gains achieved using our curriculum learning approach on various scales of the DiT model: DiT-S (small), DiT-B (base), and DiT-L (large).
Table. 4 shows that the proposed curriculum learning for diffusion model improves the performance regardless of the model size.
Moreover, it is notable that larger models exhibit a more substantial performance enhancement: DiT-S improved by 8% in terms of FID, while DiT-B and DiT-L showed improvements of 24% and 27%, respectively.
These findings validate the efficacy of our curriculum approach across a diverse range of model sizes, underscoring its generalizability to various model parameters.

#### Orthogonality of Our Curriculum Approach

Lastly, we illustrate the seamless integration of our method with sophisticated training techniques such as DTR [] and MinSNR [].
Initially, we observed that each sophisticated method yields a superior performance compared to the vanilla method.
Meanwhile, as shown in Table. 4, the performance is significantly enhanced when we apply the proposed curriculum learning.
Consequently, the curriculum approach proves to be compatible with previous promising methods such as loss weighting (MinSNR) and architectural enhancements (DTR), demonstrating our orthogonality with recent diffusion techniques.

#### Additional experimental results

Due to limited space, we present additional experimental results in Appendix G.
These results also support the effectiveness of our method, emphasizing the importance of curriculum approaches in diffusion training.

**Table 3: Note that the curriculum learning achieves consistent improvements across the model sizes.**
| Class-Conditional ImageNet 256$\times$256. |  |  |  |  |
| --- | --- | --- | --- | --- |
| Model | FID$\downarrow$ | IS$\uparrow$ | Prec$\uparrow$ | Rec$\uparrow$ |
| DiT-S/2 | 43.30 | 33.63 | 0.42 | 0.54 |
| DiT-S/2 + Ours | 39.66 | 36.57 | 0.44 | 0.54 |
| DiT-B/2 | 30.27 | 60.06 | 0.55 | 0.52 |
| DiT-B/2 + Ours | 22.96 | 75.98 | 0.62 | 0.52 |
| DiT-L/2 | 11.18 | 146.95 | 0.75 | 0.47 |
| DiT-L/2 + Ours | 8.18 | 186.37 | 0.79 | 0.47 |

## 7 Conclusion

In this study, we tackle the challenge of denoising task difficulty within the diffusion model framework and introduce a novel task difficulty-based curriculum learning approach.
To the best of our knowledge, we are the first to define task difficulty by considering both the convergence rates of loss and performance metrics.
Moreover, in terms of data distribution analysis, we observe a reduction in relative entropy between consecutive probability distributions as timesteps progress.
We believe that these observations might help reorganize the conflicts of previous works regarding denoising task difficulties.
Building upon these insights, we propose a curriculum learning framework for diffusion models, comprising curriculum design and pacing strategies.
Our experimental results convincingly demonstrate the efficacy of our approach across diverse diffusion model designs, datasets, and tasks.
From these results, we emphasize that considering an order of learning denoising tasks is also a potential direction to improve training of diffusion models.
In future research, for further enhancements, more advanced curriculum learning strategies such as self-pacing can be elaborated.

## Appendix

## Appendix A Extended Related Work

In this section, we review works related to analyzing diffusion model behaviors in each timestep but not covered in detail in Section 2.1.
Deja et al. [] explore denoising during the backward diffusion process and observe that transition from denoising to generation exists in the backward process.
Go et al. [] investigate the affinity between denoising tasks, showing that temporal proximal denoising tasks exhibit higher task affinity.
Then, they also observe that simultaneously learning all denoising tasks by one model suffers from negative transfer.
They can achieve better performance than standard diffusion training by alleviating negative transfer.
Lee et al. [] analyze frequency characteristics according to timesteps and observe that high-frequency components are lost as timesteps increase.
From this observation, they propose a multi-architecture multi-experts diffusion model, which utilizes multiple denoiser models specialized in each timestep interval but utilizes a transformer-like model as the timestep increases.
From observations that smaller and larger models produce similar latent noise, Pan et al. [] propose T-Stitch, which leverages a pre-trained smaller model at the beginning of the backward process to accelerate the sampling speed.
Xu et al. [] investigate the average trace-of-covariance of training targets according to timesteps, showing that it peaks in the intermediate timesteps, causing unstable training targets.
For more stable training targets, they utilize weighted conditional scores with a reference batch.

## Appendix B Detailed Experimental Setups for Observation

In Section 4, we examined the difficulty of denoising tasks in terms of convergence with various models $\{\mathrm{M}\}_{i=1}^{20}$, which are trained within specific timesteps $[\frac{i-1}{20}T,\frac{i}{20}T]$ for DiT and SiT, and $[\mathrm{\Phi}^{-1}(\frac{i-1}{N}),\mathrm{\Phi}^{-1}(\frac{i}{N})]$ for EDM where $\mathrm{\Phi}^{-1}$ is the inverse cumulative distribution function of the Gaussian distribution.
For the DiT architecture, we employed the DiT-B/2, whereas for EDM, we used the DDPM++ architecture.
Both DiT and EDM models were trained on the FFHQ dataset, with a batch size of 256, for approximately 20,000 and 4,000 iterations (equivalent to processing 1 million images), respectively.
This training was conducted until both loss and performance converged.
Configuration of training optimizers and learning rates are the same as setups in Section 6.

To evaluate the performance of diffusion models through the FID score of the generated images, performing a recursive denoising task from $T$ to zero is necessary, complicating the assessment using only $\mathrm{M}_{i}$.
Following [], we generated samples where $\mathrm{M}_{i}$ was specifically utilized for denoising within its trained intervals. At the same time, the diffusion model is responsible for the denoising tasks across the entire range of timesteps.
For this evaluation, we sampled 10K images using a DDPM sampler over 250 steps for DiT and SiT, and an Euler solver over 40 steps for the other models.

Figure: Figure A: Qualitative comparison between vanilla, naive curriculum, and ours on the FFHQ dataset.

Figure: Figure B: Qualitative comparison between vanilla, naive curriculum, and ours on ImageNet dataset.

## Appendix C Approximation of KL Divergence of p t − 1 subscript 𝑝 𝑡 1 p_{t-1} italic_p start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT and p t subscript 𝑝 𝑡 p_{t} italic_p start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT .

Here, we supplement the approximation of KL Divergence of $p_{t-1}$ and $p_{t}$ omitted in Section 4.2.
To explore the difficulties of denoising tasks from the distributional viewpoint, we analyze the KL divergence of $p_{t-1}$ and $p_{t}$, $D_{KL}(p_{t-1}||p_{t})$.
However, due to the unknown explicit density form of $p_{0}$, it is approximated through unbiased estimators as follows:

$$ $\hat{D}_{KL}\left(p_{t-1}||p_{t}\right)=\frac{1}{M}\sum_{\begin{subarray}{c}i \in\{1,2,\cdots,M\}\\ \bm{x}_{i}\sim p_{t-1}\end{subarray}}\log\left(\frac{p_{t-1}(\bm{x}_{i})}{p_{t }(\bm{x}_{i})}\right),$ (4) $$

$$ $\hat{p}_{t}(\bm{x}_{t})=\frac{1}{L}\sum_{\begin{subarray}{c}j\in\{1,2,\cdots,L \}\\ \bm{y}_{j}\sim p_{0}\end{subarray}}p_{0t}(\bm{x}_{t}|\bm{x}_{0}=\bm{y}_{j}),$ (5) $$

where $\hat{D}_{KL}$ and $\hat{p}_{t}$ are unbiased estimators of $D_{KL}$ and $p_{t}$, respectively, and we choose Monte-Carlo estimators for them [, ].

## Appendix D Algorithm

Due to the limited space of the main manuscript, we hereby present the step-by-step process of our method to supplement the details of our approach.
The pacing function, which determines the moments to transit between curriculum stages is described in Algorithm 1.
By incorporating this pacing function, the detailed procedure of our proposed curriculum learning method for training diffusion is illustrated in Algorithm 2.

Figure: Algorithm 1 Pacing Function

## Appendix E Details on Experimental Setups

#### Evaluation metrics.

To evaluate the performance of models, we utilized three metrics: FID [], IS [], and Precision/Recall [].
Specifically, we applied FID and IS to measure sample quality, while Precision is used to assess quality further and Recall was utilized to evaluate the diversity of the generated samples in ImageNet setup.
In other datasets, we employed FID to evaluate sample quality.
Unless otherwise mentioned, we sampled 50K samples for evaluation.
In tasks involving conditional generation, including class-conditional image generation (e.g. CIFAR-10, ImageNet) and text-to-image conversion (e.g. MS-COCO), we adapted the classifier-free guidance [] with a guidance scale of 1.5.

#### Training details.

For training diffusion models, we utilized the AdamW optimizer [] with a constant learning rate of 0.0001, and weight decay was not applied.
The exponential moving average (EMA) to the model’s weights was used to stabilize the training and the decay ratio was set to 0.9999.
The batch size was set to 256, and we augmented the training data by a horizontal flip.
While the diffusion timestep $T$ was configured as 1,000 for all experiments, we trained for 100K iterations for the FFHQ dataset [], and 400K iterations for the ImageNet dataset [] and MS-COCO dataset [].
The number of clusters $N$ was 20 unless otherwise specified. The maximum patience $\tau$ was varied across model sizes: it was set at 200 for DiT-S/2, DiT-B/2, and EDM, and 400 for DiT-L/2. EDM was trained using fp16, while the other models were trained using fp32.
We used 8 A100 GPUs for all experiments.

Figure: Figure C: Qualitative comparison between vanilla, naive curriculum, and ours on MS-COCO dataset.

## Appendix F Qualitative Results

In this section, we present qualitative comparisons between three methods: 1) Vanilla, 2) NaiveCL, and 3) Ours, across the FFHQ, ImageNet, and MS-COCO datasets. All methods are evaluated using DiT-B models, and the final trained models generate all samples shown in the results.
As shown in the results in the following subsections, our approach can synthesize more accurate and realistic images compared to Vanilla and NaiveCL.

### F.1 Qualitative Evaluation on the FFHQ Dataset.

Figure A presents a qualitative analysis of the performance in unconditional facial image synthesis among the vanilla, the naive curriculum approach, and our method.
Our approach demonstrates superior ability in generating realistic images.

### F.2 Qualitative Analysis on the ImageNet Dataset.

In the conditional image synthesis, we exhibit the outcomes generated by the vanilla, the naive curriculum strategy, and our proposed method.
Figure B clearly shows that our methodology surpasses the competing approaches in terms of quality.

### F.3 Qualitative Assessment on the MS-COCO Dataset.

To further substantiate the effectiveness of our proposed technique, we conduct a qualitative comparison of the results in the text-to-image generation task among the vanilla, the naive curriculum method, and our own approach, as depicted in Fig. C.

## Appendix G Further Experimental Results

### G.1 Convergence Speed across Model Size

Figure: Figure D: We observed an increase in convergence speed across various model sizes when the proposed curriculum learning approach was applied.
Refer to caption: x13.png

By leveraging the advantages of curriculum learning in diffusion training, our method offers faster convergence than vanilla training.
To further investigate this aspect, we measured FID-10K through training iterations for DiT-S and DiT-L.
Figure D describes the results, showing that our curriculum approach achieves faster convergence in both models.
These results also support the effectiveness of our method.

### G.2 Robustness on Noise Schedule

**Table A: Ablation study on noise scheduler. Note that our approach improves performance consistently across each scheduler.**
| Class-Conditional ImageNet 256$\times$256. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Schedule | Method | FID$\downarrow$ | IS$\uparrow$ | Prec$\uparrow$ | Rec$\uparrow$ |
| cosine | Vanilla | 30.27 | 60.06 | 0.55 | 0.52 |
| Ours | 22.22 | 75.98 | 0.62 | 0.52 |  |
| linear | Vanilla | 16.99 | 83.62 | 0.68 | 0.53 |
| Ours | 16.03 | 87.66 | 0.69 | 0.53 |  |

For a more comprehensive ablation study, we also trained the diffusion model with different noise schedules.
In contrast to cosine scheduling, the $\beta_{t}$ is set by uniformly dividing the interval $[0.0001,0.02]$, and the $C_{i}$ are obtained corresponding to SNR on a linear schedule.
As shown in Table. A, our approach improves the performance with cosine and linear noise schedulers.

### G.3 Qualitative Results from DiT-L/2 with 2M iterations

In Figures E-I, we present images generated by DiT-L using our curriculum training method for 2M iterations. The results demonstrate that our method produces highly realistic images.

Figure: Figure E: Uncurated 256$\times$256 DiT-L/2 samples. Classifier-free guidanzce scale = 2.0. Class label = “golden retriever" (207)
Refer to caption: x14.png

Figure: Figure F: Uncurated 256$\times$256 DiT-L/2 samples. Classifier-free guidance scale = 2.0. Class label = “panda" (388)
Refer to caption: x15.png

Figure: Figure G: Uncurated 256$\times$256 DiT-L/2 samples. Classifier-free guidance scale = 2.0. Class label = “cliff drop-off" (972)
Refer to caption: x16.png

Figure: Figure H: Uncurated 256$\times$256 DiT-L/2 samples. Classifier-free guidance scale = 2.0. Class label = “lake shore" (975)
Refer to caption: x17.png

Figure: Figure I: Uncurated 256$\times$256 DiT-L/2 samples. Classifier-free guidance scale = 2.0. Class label = “lion" (291)
Refer to caption: x18.png

## Appendix H Broader Impacts

Generative models, such as diffusion models, have the potential to significantly impact society, particularly through DeepFake applications and the use of biased datasets.
One primary concern is the possibility for these models to amplify misinformation, which can erode trust in visual media.
Additionally, if these models are trained on biased or deliberately altered content, they may unintentionally perpetuate and intensify existing social biases.
This situation may result in the dissemination of incorrect information and the manipulation of public opinion.

## Appendix I Limitations

In this work, we demonstrated the varying difficulties of denoising tasks through empirical results on various diffusion frameworks and proposed a curriculum learning approach that effectively enhances diffusion model training.
While we have shown the robustness of our method’s hyperparameters in improving vanilla diffusion training, there is potential for further improvement.
Specifically, curriculum learning methods that utilize smaller hyperparameters and adjust dynamically based on the model itself could yield better results.
We acknowledge the validity of this direction and consider it a promising avenue for future work.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: **Q0 |** Please address the concerns and questions in Weaknesses.

**Q1 |** In Section 4.2, the relative entropy analysis indicates that marginal distributions become less similar as $t$ approaches 0. Could the authors elaborate on why this leads to the conclusion that training is more challenging at low noise levels? If I understand correctly, at small $t$, the tasks become more independent, which may limit the model to share knowledge between adjacent timesteps. Does this explanation seem reasonable?

**Q2 |** The method uses the SNR-based clustering. Did the authors consider using gradient-based clustering [1,2] in their approach? Perhaps, it may provide more accurate intervals for different datasets.
soundness: 3
strengths: **S1 |** The authors directly answer the question, "Which timestep intervals are more challenging during training?" and provide a convincing analysis for different diffusion parameterizations.
 
**S2 |** The experiments demonstrate that the proposed approach can improve the convergence and final performance of several popular DM methods.

**S3 |** The ablation study explores important questions, such as whether the performance gains persist with larger models and how the approach interacts with other training techniques, e.g., MinSNR loss weighting.
summary: The paper notices that the diffusion training at low noise levels is more challenging than at high ones. 
Based on this observation, the authors propose a curriculum learning approach for diffusion models (DMs) that facilitates faster convergence and improves overall performance.
weaknesses: **W1 |** The proposed method has limited scientific contribution: the clustering is adopted from [1], the pacing function, while reasonable, is rather trivial, and the idea behind curriculum learning is pretty general. This could be fine if accompanied by very insightful and comprehensive analysis and strong results. Currently, I feel that the overall contribution is not sufficient.

**W2 |** Most experiments are performed using DiT, which currently seems to be a relatively weak baseline. EDM may also be considered outdated. I believe it is important to apply the proposed approach to EDM2[2] and demonstrate the gains on top of it. EDM2 focuses on training techniques and outperforms DiT and EDM by a large margin. Also, it proposes the dynamic loss weighting, which strongly relates to the proposed approach. 

**W3 |** The analysis is performed only on FFHQ256 while the dataset and image resolution can be important factors as well. For example, [3] observed that larger models are more beneficial at high noise levels for CIFAR10 and ImageNet 64x64, and, in contrast, larger models are preferable at low noise levels for the LSUN dataset. [4] revealed different optimal timestep intervals for various datasets under the same noise schedule. Thus, it seems valuable to perform analyses across different datasets and discuss any observed trends. It would also be interesting to discuss whether pixel and latent spaces exhibit different behaviors.

---
[1] Go et al. Addressing Negative Transfer in Diffusion Models, 2023

[2] Karras et al. Analyzing and Improving the Training Dynamics of Diffusion Models, 2023

[3] Ganjdanesh et al. Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection, 2024

[4] Liu et al. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models, 2023

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I have already asked questions in the weakness ection
soundness: 3
strengths: * The authors show consistent improvement across a variety of architectures and datasets thus denoting the effectiveness of their approach.

* I like the anti-curriculum learning ablation, where instead of training on easier to harder tasks, the authors instead train on harder to easier tasks which doesn't perform any better than the baseline.

* I like the analysis done by the authors in determining the hardness of different timestamp schedules.
summary: The paper proposes a curriculum learning based training of diffusion models, where the model is progressively trained on easier to harder tasks. The authors identify the task hardness based on the timestamps used for the training. For this, they train multiple models, where each model is trained to denoise only a particular subset of timestamp ranges. The loss curve of these models with training steps, depict that model trained on earlier timestamp ranges converges slower than the models trained on later timestamp ranges. Apart from the loss curves, the authors also look at the FID score by sampling from these models during training(to denoise or timestamps other than what the model is trained on, a model trained on all timestamps is used for denoising). Again the FID of models trained on later timestamps is lower than those trained on earlier timestamps. 

Based on these analysis, the authors conclude that denoising for earlier timestamps is harder than denoising for later timestamps. Base on this, the authors then devise a curriculum learning-based training of diffusion models. The model is iteratively trained on later to earlier timestamps(easier to harder tasks) clusters. Instead of uniformly dividing the total timestamp into each cluster, the authors use an SNR-based interval clustering technique. Furthermore, since the hardness of different timestamp clusters varies, so the authors propose a better approach than training on each cluster for a fixed number of steps. They define a maximum threshold hyper-parameter(patience), which determines whether to switch to the next training cluster. If the loss in that cluster stays constant for more than patience steps, then the training proceeds to the next cluster. 

The authors conduct experiments across different diffusion training architectures, such as as DiT, SiT and also over different training datasets such as FFHQ, ImageNet. Across different architectures and datasets, the proposed approach consistently outperforms the vanilla approach of training without a curriculum learning schedule.
weaknesses: I have a few concerns regarding the paper - 

* It seems that the effectiveness of the approach is reduced when the model is trained for longer steps. The difference in performance between the baseline and test after 2M training steps is much smaller than the difference at 400k steps. In that way, the main effectivess of the approach is just faster convergence, instead of improved performance. How do the authors justify the improved performance then?

* Can the authors show comparison between the curriculum and anti-curriculm approach for unconditional image generation also instead of just class-conditional generation?

* The authors should explain more in the paper about the SNR-based interval clustering technique and why it is a better interval technique than uniformly clustering.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Regarding Weakness 1: Can authors add theoretical justification or explanation, perhaps by citing other curriculum learning literature?
2. Also related to Weakness 1: Diffusion models differ from networks in other curriculum learning literatures in that the time step conditioning changes at each training phase. This might cause behaviors at different time steps (which should ideally be learned independently) to influence each other through shared network parameters. Therefore, I guess that the traditional curriculum learning framework might not fully explain the effectiveness of this method. Can you provide any references or discussion to address this question?
3. Regarding Weakness 2: Can you discuss why the performance of the proposed method is significantly inferior to current SoTA methods? For instance, the latest results on the ImageNet 256x256 dataset had FID scores below 5 as of 2022. (While additional experiments are not expected, if you could demonstrate improvement on a very high-performing model, it would strongly support the claims and effectiveness of this paper's method.)
4. Also related to Weakness 2: From Table 2, it seems that the results in Table 1 are from models that haven't fully been converged. While improved convergence performance is promising, it would be beneficial to show that performance improvements are still observed with further training for other models as well.
5. Regarding Weakness 3: Can you discuss the relationship between the proposed method and other diffusion model learning improvement techniques? For example, while this method is compatible with noise weighting techniques, I feel they might not be completely orthogonal in theory.
6. Can the KL divergence analysis in Section 4.2 be explicitly calculated in the forward process of diffusion?

Additional Notes
1. In the supplementary material, references are not properly cited.
2. When discussing convergence speed, for example in Figure 6, I believe it's important to include data points that show full convergence. Reaching the performance ceiling quickly is an advantage, so demonstrating this would be valuable.
soundness: 2
strengths: The key features of this paper are as follows:

1. The motivation for the research is clearly explained in Section 4 (Observations). In particular, Figures 1 and 2 effectively illustrate the problems that need to be addressed.
2. The proposed method requires the design of a pacing function and scheduling, as described in Section 5.2. However, the core idea itself is simple and elegantly solves the identified problems.
3. In the experimental section, the method is tested on multiple baseline models, confirming that the proposed approach brings improvements in each case. Additionally, appropriate ablation studies have been conducted, suggesting the generalizability of the proposed method.
summary: This paper aims to improve the learning process of image generation using diffusion models through a technique called curriculum learning. The authors first observed the progress of diffusion model training. They found that denoising images with more noise (larger time steps) is easier than denoising images with less noise (smaller time steps). This was confirmed by examining the convergence of loss functions and FID scores at each noise level.

Furthermore, by analyzing the KL divergence between marginal probability distributions of consecutive time steps, they also demonstrated that the denoising task becomes more difficult with larger time steps.

Based on these observations, the authors adopted a common curriculum learning approach called "easy-to-hard training scheme." Specifically, they proposed a strategy that starts learning from time steps with more noise and gradually expands the range of time steps being learned. This strategy requires designing a "pacing function" to determine how to expand the learning range. The authors adopted a technique that transitions to the next phase based on the status of the training loss.

In experiments, they confirmed that the proposed method improves performance compared to vanilla learning strategies across multiple baseline models. Additionally, they demonstrated faster convergence and orthogonality (i.e., compatibility) with existing learning improvement methods.
weaknesses: 1. While the motivation in section 4 is clear, there is insufficient explanation as to why this method is expected to improve performance. Even though the "easy-to-hard training scheme" is well-known in the field of curriculum learning, the paper lacks discussion on why it works effectively and its connection to theoretical aspects.
2. It is appealing that the proposed method can generally improve upon baselines. However, from my understanding, the reported performance seems to be significantly different from the current state-of-the-art results. While achieving state-of-the-art performance is not mandatory for this type of paper, the lack of discussion about this performance gap raises concerns about the generalizability of the method.
3. This method presents a novel approach in applying curriculum learning to diffusion model training. However, there seems to be a lack of discussion regarding its relationship and comparison with other learning improvement techniques.

---

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: - Interesting that the anti-curriculum training (hard to easy) can also improve performance over vanilla, even if not consistently (Table 3). Do the authors have insight on why? This might come across as contradictory to the main claims of the paper and I suggest the authors explain it clearly to avoid confusing the reader.
- Might be small typo in line 294. As time step increases, the expression in parenthesis suggests KL increases which contradicts Fig 2.
- It seems like the diffusion community is moving more towards flow/ODE-based/consistency models, rather than DDPM-style models. Have the authors tried applying their method to flow-based methods?
soundness: 3
strengths: - Dividing training into difficulty phases determined by diffusion timestep clusters is novel to my knowledge. 
- The paper is well structured and the method is well motivated. The presentation of evidence that difficulty varies depending on diffusion timestep (Sec 4) into the proposed method (Sec 5) is well thought and makes the paper flow nicely.
- Comprehensive experiments show improved performance over baselines at same number of training iterations, i.e., with no additional training overhead, with sensible ablations justifying design choices made by the authors.
summary: The paper proposes a curriculum-based training schedule for diffusion models that trains the model on progressively increasing task difficulty, which corresponds to different bins of the diffusion noise schedule. The authors provide empirical evidence that task difficulty increases with decreasing time steps (as SNR increases), and provide a simple scheme of dividing the training into difficulty tasks. Results show that the scheme improves the convergence and quality of diffusion samples with no training overhead.
weaknesses: - Overall the main novelty of the work is on the proposed training schedule, which is simple and on the lower side. Sec 4 is more of an empirical confirmation that lower noise levels is more difficult, which I believe is already well-known. I think this is not a big negative point however, as there is merit to simple ideas that work.
- Some missing citations as the general idea of training diffusion models from easy to difficult tasks is not new. The earliest and most influential ones to my knowledge are progressive distillation [1] (many to few sampling steps) and cascaded diffusion [2] (low to high-res). 
- Sec 4.2, it is not clear to me why high KL between marginals of two time steps implies higher task difficulty. I assume it is because higher KL means the model has to make larger changes to the image between timesteps, thus making it more challenging. I think this can be more clearly stated.

[1] Salimans, Tim, and Jonathan Ho. "Progressive distillation for fast sampling of diffusion models." arXiv preprint arXiv:2202.00512 (2022).

[2] Ho, Jonathan, et al. "Cascaded diffusion models for high fidelity image generation." Journal of Machine Learning Research 23.47 (2022): 1-33.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - How splitting the process into separate models for separate parts affected the loss convergence? I can imagine that when training a single model on all of the steps, there is some positive/negative transfer between different tasks.
- Are all of the models in the evaluation section trained for the same number of steps? How is it achieved in the CL training with maximum patience iteration that introduces various number of training steps in each CL task?
- The results presented in Table 1 for some methods are relatively close to each other. How many examples were used to calculate FID, are those results statistically significant (as there are no confidence intervals)
- Were all of the models trained with a linear noise scheduler? How would the results be affected by changing it?
soundness: 3
strengths: - The idea of curriculum learning for faster and more efficient diffusion training is interesting and, to the best of my knowledge, novel. The experimental evaluation is convincing as the method seems to improve the performance of the models across several benchmarks and with different architectures 
- Submission provides interesting experiments in the preliminary observation. I particularly appreciate experiments where part of the sampling trajectory was generated using a separate model trained for a limited period of time. This experiment brings valid observations regarding how challenging the training of earlier/later diffusion steps is. However, there is still one question remaining - what is the influence of the training on the remaining steps on other steps - see the questions section.
- On top of the proposed method and main benchmarking, the authors present an extensive additional experiments section that provides an in-depth analysis of the solution’s strengths.
summary: In this work, a new analysis of the levels of difficulty of different parts of diffusion model is presented followed by the introduction of new curriculum-based approach for improved training. Authors first show that it is harder to learn how to denoise samples that are less noisy in the diffusion process. On top of this observation they introduce a new method for the training of diffusion models, where model is first trained with only simpler timesteps, followed by the harder ones and full training. The extensive evaluation validates that such approach leads to better performance of the final model and faster convergence.
weaknesses: - I’m left with a single concern. Is it really thanks to the curriculum learning, or is it just important to first learn how to do denoising in the initial steps of the diffusion process - which define the mapping between random Gaussian noise and training data so that later training is easier? Driven by the confusing results of the evaluation presented in Figure 4, I lack one last experiment where the model is first trained using only timesteps from the C_N cluster followed by random ordering or standard training. Would it be significantly worse than the presented approach?
- In Section 5.2 there is a statement that “the convergence rate of each curriculum phase varies significantly, as demonstrated in Fig. 1.” - - - This is true, but I hope that I understand correctly that in the CL scenario, the model is first trained with the easy task, but then the same model is further finetuned with harder tasks. This should affect the convergence speed.
- Figure 4 is very puzzling. It suggests that it actually doesn’t matter how much we split the process used by the curriculum training, the results are almost identical except for the magical 20 splits used throughout the rest of the submission.  
- Small errors/suggestions: Section 5 describes the Method rather than the Methodology. Table 2, Figure 4 are not within the specified margins

Meta-review: This paper provides a neat study of curriculum learning in diffusion models. The idea of training different diffusion models at different training regimes and analyzing the convergence to assess the task difficulty is quite interesting. This is an important contribution as there have been some debate on which noise region is easy to learn. The authors propose a way to cluster the noise regimes, show concretely with experiments that easy to hard curriculum learning helps. Experimental results are solid.

Decision: Accept (Poster)