Denoising Task Difficulty-based Curriculum for Training Diffusion Models
Authors: Jin-Young Kim
Hyojun Go
Soonwoo Kwon
Hyun-Gyoon Kim
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion models
Task difficulty
Curriculum learning
Primary area: generative models
TL;DR: Our study of diffusion models shows that denoising tasks have different levels of difficulty in terms of time step, leading us to develop a curriculum-based training approach and improve performance and efficiency.

Abstract: Diffusion-based generative models have emerged as powerful tools in the realm of generative modeling. Despite extensive research on denoising across various timesteps and noise levels, a conflict persists regarding the relative difficulties of the denoising tasks. While various studies argue that lower timesteps present more challenging tasks, others contend that higher timesteps are more difficult. To address this conflict, our study undertakes a comprehensive examination of task difficulties, focusing on convergence behavior and changes in relative entropy between consecutive probability distributions across timesteps. Our observational study reveals that denoising at earlier timesteps poses challenges characterized by slower convergence and higher relative entropy, indicating increased task difficulty at these lower timesteps. Building on these observations, we introduce an easy-to-hard learning scheme, drawing from curriculum learning, to enhance the training process of diffusion models. By organizing timesteps or noise levels into clusters and training models with ascending orders of difficulty, we facilitate an order-aware training regime, progressing from easier to harder denoising tasks, thereby deviating from the conventional approach of training diffusion models simultaneously across all timesteps. Our approach leads to improved performance and faster convergence by leveraging benefits of curriculum learning, while maintaining orthogonality with existing improvements in diffusion training techniques. We validate these advantages through comprehensive experiments in image generation tasks, including unconditional, class-conditional, and text-to-image generation.

Published as a conference paper at ICLR 2025
DENOISING TASK DIFFICULTY-BASED CURRICULUM FOR TRAINING DIFFUSION MODELS
LinYYoug Kim Ajou U (scago0828, gohyojun15, swkwen. John ) 8gmail 11. com, byungyoonk.mmajjouuac..
Hyojun Go*
Soonwoo Kwon*
Hyun-Gyoon kim
ABSTRACT
Diffusion-based generative models have emerged as powerful tools in the realm of generative modeling. Despite extensive research on denoising across various timesteps and noise levels, conflict persists regarding the relative difficulties of the denoising tasks. While various studies argue that lower timesteps present more difficult.
more challenging tasks, others contend that higher timesteps To address this conflict, our study undertakes a comprehensive examination task difficulties, focusing convergence behavior and changes in relative entropy between consecutive probability distributions across timesteps. Our observational study reveals that denoising earlier timesteps poses challenges characterized by slower convergence and higher relative entropy, indicating increased task difficulty at these lower timesteps. Building on these observations, we introduce an easy-to- hard learning scheme, drawing from curriculum learning, to enhance the training process of diffusion models. By organizing timesteps or noise levels into clusters and training models with ascending orders of difficulty, we facilitate an order- aware training regime, progressing from easier to harder denoising tasks, thereby deviating from the conventional approach of training diffusion models simulta- neously across all timesteps. Our approach leads to improved performance and faster convergence by leveraging benefits of curriculum learning, while maintain- ing orthogonality with existing improvements in diffusion training techniques. We validate these advantages through comprehensive experiments in image generation tasks, including unconditional, class-conditional, and text-to-image generation.
INTRODUCTION
Diffusion-based generative models (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song et al., 2021) have achieved significant advancements in the realm of generative tasks, demonstrating notable success across various fields such as image (Dhariwal & Nichol, 2021), video (Ho et al., 2022; Harvey et al., 2022), and 3D (Woo al., 2023; Liu et al., 2023b) generation. Specifically, their exceptional adaptability and promising performance in diverse image generation contexts, such as unconditional (Karras et al., 2022; Nichol & Dhariwal, 2021), clas-cconiiiinal (Dhariwal & Nichol, 2021), and text-conditional scenarios (Balaji et al., 2022; Ramesh et al., 2022), demonstrate their significant impact. Such achievements have led to growing interest further deepening the analysis and enhancing diffusion models.
Diffusion models (Ho et al., 2020; Song et al., 2021) are designed to reverse the corruption of the data through the learning process at different noise levels and over multiple timesteps. Recent works have delved into the learning of diffusion models across various noise levels and timesteps, revealing different stages of diffusion models. For example, Choi et al. (Choi et al., 2022) observe that when a diffusion model performs a denoising task from large to small timestep. it first generates coarse features, then gradually generates perceptually rich content, and later refines the details. Similar observation is also identified in text-to-image diffusion models (Balaji et al., 2022). Besides this aspect, various studies have further explored the learning of diffusion models across timesteps and noise levels, elucidating their transition from denoising to generative functionalities (Deja et al., 2022), modular attributes (Yue et al., 2024), frequency characteristics (Yang et al., 2023b; Lee et al., 2023), trajectories (Pan et al., 2024), affinity (Go et al., 2023a), and variations of targets (Xu et al., 2023).
"Co-first author Corresponding author

---

Published as a conference paper at ICLR 2025
These observations have not only deepened understanding of diffusion models but have also directly contributed to improvement in diffusion models. Specifically, these insights are incorporated into their method design in various works, including loss functions (Hang et al., 2023; Xu et al., 2023), architectures (Lee et al., 2023; Balaji et al., 2022), accelerated sampling (Pan et al., 2024), representa- tions (Yue al., 2024), and guidance (Go et al., 2023b). Given the tangible benefits already realized from such studies, further in-depth analysis of diffusion models across timesteps and noise levels is crucial for uncovering insights and achieving unprecedented advancements in their capabilities.
In this paper, to enrich the current understanding across various timesteps and noise levels, we investigate under-explored areas within diffusion models focusing on the task difficulties of denoising. Regarding denoising task difficulties, previous works speculate that denoising tasks across timesteps have different difficulties (Li et al., 2023; Balaji et al., 2022), yet a detailed exploration of these variances remains sparse. Moreover, there exists a notable discrepancy among studies, with works identifying larger timesteps as more difficult (Ho et al., 2020; Hang et al., 2023), while others argue that smaller timesteps pose greater difficulties (Karras et al., 2022; Dockhorn et al., 2021; Kim et al., 2022). The discrepancy in difficulty across timesteps not only impedes the accurate interpretation of previous studies but also hinders the development of sophisticated training methods that properly utilize the timestep-win wise variation in difficulty.
In this regard, we first analyze task difficulties in two aspects to resolve these conflicts: 1) convergence properties in the learning of denoising tasks at each timestep, and 2) the change in relative entropy between consecutive probability distributions over timesteps. In the first aspect, our analysis reveals distinct convergence behaviors across timesteps, demonstrating that models trained on larger timesteps exhibit faster convergence. In the second aspect, we also observe a decrease in relative entropy we progress to later timesteps. By integrating these, we confirm that denoising tasks at earlier timesteps are more difficult, indicated by slower convergence speeds and greater changes in relative entropy.
Furthermore, building on these observations, we integrate an casy-to-hard learning scheme, a concept well-established in the curriculum learning literature (Hacohen & Weinshall, 2019; Kong et al., 2021; Chang et al., 2021; Wang et al., 2020; Pentina et al., 2015), into the training process of diffusion models. Specifically, we organize timesteps or noise levels into clusters and train the diffusion models with ascending levels of difficulty, moving from clusters categorized by higher to lower timesteps. After this curriculum process, models simultaneously learn whole timesteps as standard diffusion training (Ho et al., 2020; Song et al., 2021; Ho & Salimans, 2022) to reach the convergence point. Unlike conventional approaches where diffusion models are trained simultaneously across all timesteps, our method distinguishes itself by incorporating sequential, order-aware training regime, reflecting an intended progression from easier to harder denoising tasks.
Building upon this foundation, currieular approach offers several notable advantages: 1) Im- proved Performance and 2) Faster Convergence: By leveraging the inherent benefits of curriculum learning, our method significantly enhances the quality of generation and the speed of convergence. 3) Orthogonality with Existing Improvements: Our approach is inherently model-agnostic, ensuring broad applicability across various diffusion models. Additionally, it can be integrated with advanced diffusion training techniques, such as loss weighting (Choi et al., 2022; Hang et al., 2023; Go et al., 2023a; Karras et al., 2023).
Finally, we empirically validate the advantages of our method by conducting comprehensive exper- iments across a variety of mmage-generation tasks. These include unconditional generation, class- conditional generation, and text-to-image generation, utilizing datasets such as FFHQ (Karras et al., 2019), ImageNet (Deng al., 2009), and MS-COCO (Lin et al., 2014). By integrating our curriculum learning strategy into architectures- DiT (Peebles & Xie, 2022), EDM (Karras et al., 2022), and SiT (Ma et al., 2024) we demonstrate the efficacy of our approach in enhancing performance, accelerating convergence speed, and maintaining compatibility with existing techniques.
2
RELATED WORKS
2.1 DIFFUSION MODELS
Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song et al., 2021) are a group of generative models that create samples by utilizing learned denoising process noise. Several works have focused on improving diffusion models in various aspects, including model architectures (Park et al., 2024b; Dhariwal & Nichol, 2021; Park al., 2024a), sampling speed (Song et al., 2020; Lu
2

---

Published as a conference paper at ICLR 2025
et al., 2022; Liu et al., 2023a), training objectives (Hang et al., 2023; Choi et al., 2022; Go et al., 2023a; Kingma & Gao, 2023; Ma et al., 2024). These endeavors often involve investigating what diffusion models learn by dividing its process, aiming to enhance the performance of diffusion models. P2 (Choi et al., 2022) under-weights training loss functions at the clean-up stage from their observation that diffusion models learn coarse, perceptual, and removing noises large, medium, and small timesteps. Ediff-I (Balaji et al., 2022) observes that earlier sampling parts rely on conditions for generation, whereas later parts ignore the conditions. They employ multiple denoisers to address the diverse characteristics of tasks associated with different parts of the sampling process. Moreover, various works have also investigated these aspects related to timesteps (Deja et al., 2022; Yue et al., 2024; Yang et al., 2023b; Lee et al., 2023; Pan et al., 2024; Go et al., 2023a; Xu et al., 2023) (detailed illustrations can be found in Appendix A). While our study aligns with the above works, we analyze the under-explored aspect of denoising task difficulty. Furthermore, we leverage these observations to propose a curriculum learning approach.
2.2 DENOISING DIFFICULTIES ON DIFFUSION MODELS
Difficulties in denoising tasks in diffusion have been referred to by various works, but this aspect is not deeply explored. Several studies hypothesize that denoising tasks in diffusion encompass diverse difficulties (Li et al., 2023; Balaji et al., 2022), and there have been conflicts regarding these difficulties between previous works.
Certain studies consider denoising at larger noise levels and timesteps to be more difficult (Ho et al., 2020; Hang et al., 2023), the focus is on the challenges associated with reconstructing data from substantial noise. For instance, Hang et al. (Hang et al., 2023) articulate that while smaller timesteps (approaching zero) may require straightforward reconstructions, such strategies become less effective at higher noise levels or in larger timesteps. Similarly, Ho et al. (Ho et al., 2020) elucidate that their approach de-emphasizes loss terms at smaller timesteps to prioritize learning on the more challenging tasks at larger timesteps, thereby enhancing sample quality. Conversely, other studies argue that earlier timesteps or lower noise levels also present significant challenges. Karras et al. Karras et al. (2022) suggest that detecting noise at low levels is challenging due to its minimal presence. Also, Kim et al. (Kim et al., 2022) illustrate the increasing difficulty and high variance in score estimation as timesteps approach zero, disturbing stable training of models. In line with these observations, Dockhorn el al. (Dockhom et al., 2021) build upon insights of (Kim et al., 2022), acknowledging the complexities near zero timesteps, where score becomes highly complex and potentially unbounded.
In this work, we aim to resolve this conflict through an in-depth analysis of convergence properties and changes in relative entropy between consecutive probability distributions across timesteps.
2.3 CURRICULUM LEARNING
Curriculum learning (Bengio et al., 2009; Hacohen & Weinshall, 2019; Kong et al., 2021), inspired by human learning patterns, is method of training models in a structured order, starting with easier tasks (Pentina et al., 2015) or examples (Bengio et al., 2009) and gradually increasing difficulty. As pointed out by (Bengio et al., 2009), curriculum learning formulation method (Allgower & Georg, 2003), which starts from a smoother objective and gradually transformed into a less smooth version until it reaches the original objective function. Through this foundation, various works have achieved improved performance and faster convergence compared to standard training based on random mini-batches sampled uniformly (Hacohen & Weinshall, 2019; Kong et al., 2021; Chang et al., 2021; Wang et al., 2020).
be viewed a continuation
Curriculum learning primarily comprises two components: curriculum scoring function, measuring the difficulty of tasks or examples, and a pacing function, modulating the speed of the curriculum progress. Regarding a curriculum score function, early studies have utilized human intuition for measuring difficulty, such as the complexity of geometric shapes in images (Bengio et al., 2009) or the length of sequences (Spitkovsky et al., 2010). Recently, various works employ models to measure difficulty, including confidence of pre-trained models (Hacohen & Weinshall, 2019) and the loss of the current models (Kong et al., 202 1). For the pacing function, a predefined pacing function has been employed, which involves training using predetermined curriculum progression (Hacohen & Weinshall, 2019; Wu et al., 2020). There are various forms of this and they can be generally represented as function of training iteration (Hacohen & Weinshall, 2019; Wu et al., 2020). Contrary to this, there have been proposals for pacing techniques dynamically adjusting based on the loss or performance of the current model during training (Kumar al., 2010; Jiang et al., 2014).
3

---

Published as a conference paper at ICLR 2025
In the diffusion model literature, curriculum learning has been utilized to organize the order of training data types based on prior knowledge targeted generation tasks (Tang et al., 2023; Yang et al., 2023a). Tang et al. (Tang et al., 2023) sequentially train video diffusion models with lower resolution and FPS datasets before progressing to higher resolution and FPS datasets. Similarly, Yang et al. (Yang al., 2023a) order text-to-sound generation data based on the number of events in audio clips, training diffusion models from lower to higher events datasets. In contrast, our method explores the nature of denoising task difficulty in diffusion models and proposes a curriculum learning approach that progresses from easy to hard timesteps, deviating from the standard simultaneous training of all timesteps. Also, while consistency models (Song al., 2023; Song & Dhariwal, 2023) adopt a curriculum approach to discretizing noise levels, progressively increasing the discretization steps of noise levels during training. we have distinet by exploring which noise level should be learned first and investigating the difficulty of denoising at each noise level.
3 PRELIMINARIES
In this section, we provide the necessary background on diffusion models (Ho et al., 2020; Sohl- Dickstein et al., 2015; Song et al., 2021). Let 2n R2 be sample from the data distribution (x). The forward process of diffusion models transforms data to latent ( by iteratively adding Gaussian noise. This can be formulated as a stochastic differential equation (SDE) (Song et al., 2021) as dæ, [(()æ,d! + g(()dw,, where [() and g(l) are drift and diffusion coefficients, and w, is the standard Wiener process. The Gaussian transition kernel of this SDE formulated as:
por(xs = N (ari 8, o) 8, - exp ( (((ME)) or J.
9(E)
de.
(I)
For generation, diffusion models aim to learn the corresponding reverse SDE represented as: dæ, = [f(I)= (1) (x:)] dī + g(I)du.,
where w, and df denote the reverse-time time Wiener process and the infinitesimal reverse-tima respec- tively, with the actual data score V log Pt (zzz). In most cases, a neural network eg having parameter 0 utilized to approximate this score function by learning the denoising tasks for each timestep from score matching loss L (Song & Ermon, 2019):
L 2-2/" Lidt, L. f== (t)lE por (a)) [1l,)(21,t) t) V log Pos Pos (20))
(3)
where w(t) is loss weights for and Por Por (x)) is the transition density of o; from the initial timestep 0 to f. This object can be interpreted as noise-matchin matching loss in DDPM (Ho al., 2020),
which predicts noise components in 2, and can be illustrated as Jo E
(e (Va,2 +
v t) - < dt. This is regularly denoted c-prediction parameterization (Ho et al., 2020; Jabri et al., 2022), and several other parameterizations including F-prediction (Karras al., 2022;
Kingma & Gao, 2023), score-prediction (Song et al., 2021) and velocity- (Ma et al., 2024) have been proposed.
OBSERVATIONS
In this section, we examine the difficulties associated with learning denoising tasks across different timesteps, addressing inconsistencies in prior works regarding these difficulties. Our analysis is structured around two key aspects: 1) the convergence of loss and denoising performance across timesteps, providing insights into learning dynamics at various timestep stages in Section 4.1; and 2) the relative entropy change from p. to Pt-1 as a function i, offering quantitative measure of task difficulty progression over / in Section 4.2. Upon integrating our findings, we establish key conclusion: the learning difficulty for denoising tasks escalates as the timestep decreases.
4.1 ANALYSIS ON THE TASK DIFFICULTY IN TERMS OF CONVERGENCE SPEED
In this study, analyze the convergence speed of loss and denoising performance across timesteps. To comprehensively cover various diffusion parameterizaaions, we utilized the notable frameworks DiT (Jabri et al., 2022) for e-prediction, EDM (Karras et al., 2022) for F r-prediction, SiT (Ma et al., 2024) for velocity prediction. Detailed descriptions of the experimental setups are provided in Appendix B.
4

---

Published as a conference paper at ICLR 2025
H5
M11
M13
M15
M19
M2
144
M6
MB
M10
M12
M14
M16
M18
M20
1.5
1.0
Nrmoliee
Homallge
Monnslged los
0.5
0.5
0.0
4000
2000
Itzration
12000
16000
20000
200
Rorattons (kimg)
400
500
300
1000
4000
8000
12000
16000
20203
(a) DiT (Loss convergence)
(b) EDM (Loss convergence)
(c) SiT (Loss convergence)
1.0
1.0
Nomaleed
Nometoerr
Bormared
0.5
0.0
4000
2000
Beration
12000
16000
20000
250
Itaration lkigt
400
620
RC0
1000
12000 1,000 20000
(d) DiT (Task convergence) (e) EDM (Task convergence)
(f) SiT (Task convergence)
Figure I: Loss and FID convergence plotted during training for each diffusion model M; in DIT, EDM, and SiT. Since the loss scale for each model different, we show the normalized value. We observe that as increases (i.e., corresponding to larger denoising timesteps), the loss converges more rapidly, and this convergence speed correlates with that of the FID scores.
Convegence speed on loss. First, we analyze convergence characteristics of training loss across timesteps 1. We divided whole timesteps [0, 7] into 20 uniformly divided intervals and trained 20 models (M where i-th model learns denoising tasks in [ T for DiT and SiT, (6-1( IN (+) for EDM where 4 the inverse cumulative distribution function of the Gaussian distribution. During training, we tracked the loss values through iterations and plotted their convergence speed by normalizing their value in Figs. la-le. As shown in the results. is apparent that as i increases towards = 20, the convergence accelerates in both DiT, EDM, and SiT, suggesting that models learning larger timesteps can reach convergence more swiftly and reinforcing the notion that denoising tasks with larger timesteps are less difficult.
Convegence speed denoising performance. We also delve deeper into convergence of denoising performance according to timesteps with 20 distinct models {M, ) To evaluate the performance of denoising tasks of each model, generated samples where M, employed for denoising within the timesteps that it was trained on, while a diffusion model learned whole timesteps handled denoising for the remaining timesteps in (Go et al., 2023a). Then, the performance of the denoising capability of M, can be quantitatively measured using the FID score (Heusel et al., 2017), enabling us observe the performance convergence of each model on denoising tasks throughout the training process. Figures 1d-1f depict this convergence. They illustrate that, similar to loss convergence experiments, denoising performance converges more swiftly for models M, with larger i values, observed across the DiT, EDM, and SiT. These results also suggest that models trained on later timesteps, indicated by larger values, achieve faster convergence, highlighting easier task difficulty at larger timesteps.
4.2 EXPLORATION ON DIFFICULTIES OF DENOISING TASKS
Beyond empirical convergence metrics, we also delve into analyz- ing the relative entropy between Pt and Pt-1 to better understand task difficulties from a distributional perspective. The training of diffusion models implicitly involves learning the distribution of the reverse process of the corresponding SDE. To be specific, the transition probability of the reverse process is expressed a conditional normal distribution whose mean parameter is mod- eled by neural networks, and they are thereby trained to learn the dynamics of the reverse process (Ho et al., 2020). Furthermore, an unconditional distribution of z; be obtained by marginalizing transition densities the prior distribution, indicating that information on the dynamics of the marginal distribution is fed to neural networks (Song al., 2021).
5
400
Figure 2: The KLD of P:-1 from Pt against denoising timestep. As the timestep increases, the dy- namics decrease.
5

---

Published as a conference paper at ICLR 2025
Harder task
Easier task
Task Clusters
Task Accumulaatinn
Hasy-to-Hard
G
Ca
Timestep
Cy-1
1x-1
Cμ
CN
C2
Clustered denoising tasks
Curriculum stages
Figure 3: The overview of our curriculum learning approach for diffusion models. (Left) We divide the timesteps into N clusters, Ci,..., CN. with the difficulty of denoising tasks increasing from Cw (easiest) to C₁ (hardest). (Right) As the curriculum progresses, learning accumulates harder task clusters, gradually increasing task difficulties.
To analyze the relationship between the dynamics of the unconditional distribution and the rate of loss convergence, we use the Kullback-Leibler (KL) divergence of P:-1 Pt. DKz (p:-1 lpr), as quantitative measure. It is a pertinent divergence in that the training mechanism of diffusion models involves maximizing the likelihood of the reverse process. The KL divergence D (Pt-1 (p:)) is given
by DE (pt-1) (pr-1l/p:) 1p- log
Morcover, the distribution pt of æ at is expressed (t-1(w)
Pc(3))
by (e(2))) = Por (32 2 y (o)))o(y)) (y) Emarpp [Por (2)/2 However, since the explicit density form of is unknown and is computationally infeasible to estimate high-dimensional integrals, we approximate them through unbiased estimators (details
Appendix C). The empirical results of DxL (Pt-1lIPd) for 64 x 64 image data are given in Fig. 2. As seen, the relative entropy tends to decrease increases (i.e., DEL. (Pe-)llpx) S DRL (P:-1 (1p.) for S (), which is consistent with the results in Fig.
This observation may stem from the inherent low-dimensional manifold of image data. As is well- known (e.g., (Ruderman, 1994)), the image data is distributed on a relatively low-dimensions manifold with narrow support and a highly peaked multi-modal structure. On the other hand, as Gaussian noise is iteratively added, the distribution of æ, approaches the independent Gaussian distribution in the ambient space. Consequently, the support of the manifold broadens and the score function becomes regular over the ambient space with increasing This nature of the unconditional distribution may cause the relative entropy from Pt to Pt-1 to decrease with i, indicating that is more difficult to accurately represent the dynamics of the reverse process small t. More discussion is in Appendix C.
5 METHODOLOGY
In Section 4, we observe that denoising tasks at smaller are more difficult to learn by models. From these order of difficulties in denoising tasks, we propose the incorporation strategy of an easy-to-hard training scheme, that has demonstrated its effectiveness in curriculum literature (Bengio et al., 2009; Hacohen & Weinshall, 2019; Kong al., 2021), for improving diffusion models' training.
5.1 DESIGN OF CURRICULUM LEARNING IN DIFFUSION MODELS
As we observed in Section 4, difficulties in denoising tasks increase gets smaller. To utilize an easy-to-hare curriculum learning approach, we first divide the entire range of timesteps into N clusters. denoted as {C.), where each cluster C, spans interval [4, 441]. ensuring 4 4+1. with 1 = 0 and IN+1 = T, as shown on the left side of Fig. 3. The curriculum for training is constructed by regarding these task clusters unit tasks, starting from the least challenging (the N-th cluster Cn) and advancing towards the most difficult (the first cluster C1), through N distinct stages. Specifically, the n-th curriculum stage, we jointly train the model with denoising tasks sampled from the clusters Uj (n-1) Cj as illustrated the right side of Fig. 3. The transition of the curriculum stages is determined by the pacing function, which will be discussed in the next section. After completing these N N-stages of curriculum learning, the model continues to learn across the entire range of timesteps, U Cj. same standard diffusion training.

---

Published as a conference paper at ICLR 2025
The next consideration involves determining the boundaries for each cluster 1, A straightforward approach to uniformly divide the entire timestep interval [0, 7] as C ['=1) N for i 1, .. N. However, this method does not account for variations in noise levels across different timesteps. Therefore, to address this issue more effectively, we adopt an SNR-based interval clustering technique as used in (Go et al., 2023a), which aligns the clustering with the actual changes in noise levels, potentially enhancing curriculum learning adaptability to varying noise conditions.
For EDM (Karras et al., 2019) which operates based on the noise level rather than the timestep t, and where a is sampled from log-normal distribution such that log(o) ~ (N P during training, clustering strategy for timesteps cannot be directly transposed. Given the log-normal distribution of a, dividing it directly is impractical because a can extend over a wide range of values. Instead, adapt clustering approach to suit the log-normal characteristics by defining noise ((x)) )], where Φ- is the inverse cumulative distribution function (quantile function) of the Gaussian distribution (N Pld) This method segments the noise levels into intervals by reflecting their probabilistic distribution.
level clusters C,. Specifically, we delineate C, = [ ( ((xx') (
5.2 PACING STRATEGY OF CURRICULUM
To effectively train the diffusion model according to the provided curriculum design, is crucial to define a suitable pacing function for determining the transition of each N distinct curriculum. Training for a fixed number of iterations for each curriculum stage is the simplest implementation (We also contain this method in experiments as 'NaiveCI.' in Section 6.2). However, the convergence rate of each curriculum phase varies significantly, as demonstrated in Fig. 1. Hence, we propose adopting an adaptive number of iterations for each curriculum, akin to the varied exponential pacing approach explored by Hacohen et al. (Hacohen & Weinshall, 2019). Our pacing function utilizes the training loss to determine transition moments and transitions to the next stage occur when the training introduce the maximum patience iteration and
loss converges at the current stage. Specifically, the loss does not improve consecutively for T. the current curriculum stage is terminated, and the subsequent curriculum stage is initiated. Here, the maximum patience is fixed hyper-parameter, and the detailed process and overall curriculum learning procedure are outlined Algorithm ?? and ?? in Appendix D, respectively.
EXPERIMENTAL RESULTS
In this section, we present experimental results to validate the effectiveness of method. The advantages of our curriculum method, 1) Improved Performance, 2) Faster Convergence, and 3) Orthogonality with Existing Improvements, are validated in this section. To begin, we outline our experimental setups in Section 6.1. Then, we provide the results of the comparative evaluation in Section 6.2, showing that our curriculum approach significantly improves the quality of generated samples compared to the baseline. Finally, analyses of our method are illustrated in Section 6.3 to deeply understand the effectiveness of our method.
6.1 EXPERIMENTAL SETUP
Here, we provide experimental setups concisely. Detailed setups are presented in Appendix E.
Evaluation protocols. For our comprehensive evaluation of various methods, we employed three distinct image-generin generation tasks: 1) Unconditional generation with the FFHQ dataset (Karras et al., 2019), 2) Class-conditional generation with CIFAR-10 (Krizhevsky et al., 2009) and Ima- geNet (Deng et al., 2009) datasets, and 3) Text-to-lmage generation with MS-COCO dataset (Lin et al., 2014). In 2) and 3) setups, we applied classifier-free guidance (Ho Salimans, 2022).
Target models. We employed three exemplary diffusion architectures for experiments: DiT (Pee- bles & Xie, 2022), which integrates latent diffusion models (Rombach et al., 2022) with Transformer architectures (Vaswani et al., 2017) parameterized e-prediction, EDM (Karras et al., 2022), which focuses on lixel-level diffusion utilizing UNet- based architectures (Ronneberger et al., 2015) pa- rameterized as F F-prediction, and SiT (Ma et al., 2024) for score- and -rreiity-neeiiction. For the text-to-image generation, we incorporated a CLIP text encoder (Radford et al., 2021) as described in DTR (Park al., 2024b).
7

---

Published as a conference paper at ICLR 2025
Table I: We evaluated unconditional image generation on FFHQ with DiT-B, EDM, and SiT-B, class-conditional image generation on ImageNet and CIFARIO with D and EDM, respectively, and cext-conditional image generation on MS-COCO with DiT-B, Note that our curriculum learning for diffusion models improves substantial performance without any additional parameters.
e-prediction
FFHQ 256x256
ImageNet 256
COCO 256
Model
FIDI
FIDI
1St
Prect
Reet
FIDL
DiT (Vanilla)
10.49
11.18
146.95
0.75
0.47
7.62
DiT' NaiveCL
7.95
11.90
151.66
0.75
0.47
7.71
DiT Ours
7.55
8.18
186.37
0.79
0.47
7.51
F-prediction
Velocity-reddictin prediction
Score preeepreeiction
Model
ITIQ 64x64
32x
Model
ITIQ 256
Model
MTH
FID]
FID]
FIDJ
FID
EDM (Vanilla)
2,93
2.67
SIT (Vanilla)
7.44
SIT (Vanilla)
9.64
NaiveCL
3.13
NaiveCL
7.69
NaiveCL
EDM Ours
2.71
2.44
SIT Ours
6.95
SIT Ours
9.15
6.2 COMPARATIVE RESULTS
In this section, we assess the effectiveness of our curriculum- -based training approach. For thorough comparison, we examine three distinct training variants, with further details provided in Appendix E: To achieve this, we compare three variants of training: /) Vanilla: This term refers to diffusion models trained using conventional methods without any curriculum learning strategies; 2) NaiveCL: In this variant, we incorporate basic curriculum learning strategy, which simply repeats the same number of iterations for each stage across N N-stage process and does not employ SNR- clustering; 3) Ours: This denotes our proposed curriculum approach, which is designed to enhance the training process of diffusion models by systematically structuring the learning stages.
Quantitative evaluation. We quantitatively validate the effectiveness of our methods with various architectures- DiT (Peebles & Xie, 2022), EDM (Karras al., 2022), and SiT (Ma et al., 2024)- and tasks including unconditional, class-conditional, and text-to-image generation. Table
Table 2: Evaluating the effectiveness of curriculum learning with extended train- ing iterations the ImageNet 256x256 dataset using the DiT-L architecture
FID
Iteration
shows the results, confirming two empirical observations: 1) NaiveCL fails to consistently achieve improved perfor-
400k
2M
Method
mance compared to Vanilla, and 2) our approach outper-
DiT (Vanilla)
11.18
7.84
forms both NaiveCL and Vanilla. Regarding the first obser- vation, NaiveCL shows inconsistent improvements due to its lack of robust adaptation on incorporating task difficulties in various task conditions. In contrast, method demonstrates superior performance across all scenarios by improving the clustering and pacing of curriculums. Consequently, our approach consistently achieves significant performance enhancements across all metrics on four datasets: FFHQ (Karras al., 2019), ImageNet (Deng et al., 2009), CIFAR-10 (Krizhevsky et al., 2009), and MS-COCO (Lin et al., 2014), illustrating its effectiveness regardless of data or model used.
DiT + Ours
8.18
6.24
Showing the results of longer training might demonstrate the robustness of our method in more extended training scenarios. We trained DiT-L/2 with 2M iterations and reported the results in Table 2. Our model consistently outperformed the baseline, demonstrating its effectiveness even in prolonged training. Therefore, our method proves be robust and effective for longer training durations.
Qualitative evaluation. Due to space constraints, we illustrate a detailed collection of generated ex- amples in Appendix F. In summary, our curriculum methodology demonstrates notable enhancement in the quality the images produced, when compared to NaiveCL and Vanilla.
6.3 ANALYSIS
8

---

Published as a conference paper at ICLR 2025
Vanill
Ours
30
To elucidate our curriculum approach's effectiveness, we present series of analytical studies. All the analysis conducted by using the DiT-B model on the ImageNet dataset.
25
100
200
400
800
10
Effects of N and т. We examined the robustness of the proposed curriculum training with respect to super-parameters: the number of clusters N and the maximum patience As shown in Fig. 4, method consistently outperforms the vanilla model, and the best result is ob- served at N 20, 200. It shows that as 7 increases. it may lead overtraining due to excessive iterations for each task, whereas as de- creases, curriculums may not be sufficiently trained. Furthermore, when the entire range of timesteps is finely partitioned (i.e., with an increase in N), each cluster becomes excessively granular, resulting in suboptimal performance. Conversely, with a decrease in N, tasks that should be in distinct clusters are learned together, forming coarser cluster, which also leads to suboptimal outcomes. Overall, our method outperforms vanilla training across a range of hyperparameters, demonstrating the robustness of our approach.
25
20
Figure 4: Ablation study on N and т. We use DiT- B on ImageNet x56x
Effects of Curriculum Design. In our cur- riculum design, we initially partitioned the entire set of timesteps into N clusters using SNR-based clustering, organizing the curricu- lum from easy hard clusters. To thoroughly assess the impact of each component, we con-
Table 3: Comparative results on various curriculum designs.
Class-Conditioall ImageNet 256x256.
Curiculum Design
FID.
IS*
Prect
Rect
(4)
Vanilla
30.27
60.06
0.55
0.52
ducted the ablation study as shown in Table 3. Firstly, we investigated the effect of curricu- lum learning via comparison with an anti-
(b1)
cuti-currculumm uniform
31.12
62.80
0.55
0.53
(h2)
anti curriculum + SNR
27.74
68.
0,58
0.52
(cl) curriculum + uniform
25.01
71.99
0.58
0.53
curriculum SNR
22.96
75.98
0.62
0.52
curriculum approach (Hacohen & Weinshall, 2019), which progresses from hard easy clusters, unlike conventional curriculum learning. While both training methods in (b2) appear to enhance performance compared to vanilla training (a), anti-curriculum training cannot consistently guarantee performance improvement concerning the curriculum design as shown in (bl). In contrast, the proposed curriculum learning method (cl, c2) consistently exhibited performance improvement even with the uniformly partitioned clusters. Besides, with findings that utilizing SNR-clustering more effective, clustering with the actual changes in noise levels enhanced the curriculum learning adaptability.
Visualization of curriculum. To gain deeper insights into the functioning of our curriculum pacing, plotted loss metrics against curriculum phases, illustrated in Fig. 5. During the curriculum training, tasks progressively transition from the easiest to the most challenging, with varying amounts of iterations for each task based the pacing function. The training loss decreased during each curriculum phase but increased after curriculum changes via the pacing function due to the inclusion of newly added task in the updated curriculum. Additionally, as increases, the curriculum phases change more gradually, highlighting the role of r in controlling the pace of curriculum transitions.
Analysis on convergence speed. As demonstrated in previous works (Bengio et al., 2009; Hacohen & Weinshall, 2019), the adoption of curriculum learning can lead to faster convergence in model performance. To illustrate the efficacy of our approach in this regard, we plotted the FID, IS, precision, and recall calculated over 10,000 samples across the training iterations, as depicted in Fig. 6. We observed the models trained through the proposed curriculum learning method converge faster than vanilla models, regardless of evaluation metrics. Notably, our approach achieves these improvements without requiring additional parameters or training iterations, thereby significantly saving time and computational resources.
Effectiveness on various sizes of models To verify the generalizability of our method across different model sizes, we evaluated the performance gains achieved using our curriculum learning approach on various scales of the DiT model: DiT-S (small), DiT-B (base), and DiT-L (large). Table. shows that the proposed curriculum learning for diffusion model improves the performance regardless of the model size. Moreover, it is notable that larger models exhibit a more substantial performance enhancement: DÎT-S improved by 8% in terms of FID, while DīT-B and DīT--. showed improvements

---

Published as a conference paper at ICLR 2025
T-100
01882
OTT82 OJfS
Curtculum
Te 400
15-10k
105
12.0
10091
12300
1480
1/0
Training Stepsek)
Traiting
LO33
Precetion
D1-412
DE-82
DTT-22
Figure 5: We visualized the curriculum transition and the corresponding loss across iterations (N 20). To make the loss graph more easily readable, the y-axis was truncated to 1.0.
Training SiepscK]
Traiiing Stepstk)
Figure 6: The models trained using the proposed curricu- lum learning approach demonstrate faster convergence compared to vanilla models, irrespective of evaluation metrics.
Table 5: Note that the curriculum learning is compatible with the previous works such as the loss weighting (MinSNR) and architecture (DTR) study which, specified the multi-task learning for diffusion model.
Table 4: Note that the curriculum learning achieves consistent im- provements across the model sizes.
ImageNei
Model
FIDL
1S"
Preet
Rec
DIL-2
43,30
32.63
0.42
0.54
Conditionul ImugeNet
Ous
39.86
36.57
0.44
0.54
DIT B/2 Ours
DIT82 Ous
DITLAC
11:18
146.55
0.75
0,47
Vanilla
30.27
611.00
0.52
75.98
Ous
MinSNR 2003)
58
DIE-LQ
9.40
196.83
0.77
0.49
20240)
DVI-XL-2 Oues
7.87
234.03
0.52
0.48
of 24% and 27%, respectively. These findings validate the efficacy of our curriculum approach across a diverse range of model sizes, underscoring its generalizability to various model parameters.
Orthogonality of Our Curriculum Approach Lastly, illustrate the seamless integration of our method with sophisticated training techniques such as DTR (Park et al., 2024b) and MinSNR (Hang al., 2023). Initially, we observed that each sophisticated method yields superior performance compared to the vanilla method. Meanwhile, as shown in Table. 5, the performance is significantly enhanced when we apply the proposed curriculum learning. Consequently, the currieulum approach proves to be compatible with previous promising methods such as loss weighting (MinSNR) and architectural enhancements (DTR), demonstrating our orthogonality with recent diffusion techniques.
Additional experimental results Due to limited space, present additional experimental results in Appendix G. These results also support the effectiveness of our method, emphasizing the importance of curriculum approaches in diffusion training.
CONCLUSION
In this study, we tackle the challenge of denoising task difficulty within the diffusion model framework and introduce a novel task difficulty-basse curriculum learning approach. To the best of our knowledge, we are the first define task difficulty by considering both the convergence rates of loss and performance metrics. Moreover, in terms of data distribution analysis, we observe a reduction in relative entropy between consecutive probability distributions as timesteps progress. We believe that these observations might help reorganize the conflicts of previous works regarding denoising task difficulties. Building upon these insights, we propose a curriculum learning framework for diffusion models, comprising curriculum design and pacing strategies. Our experimental results convincingly demonstrate the efficacy of our approach across diverse diffusion model designs, datasets, and tasks. From these results, emphasize that considering order of learning denoising tasks is also potential direction to improve training of diffusion models. future research, for further enhancements, advanced curriculum learning strategies such self-pacing be elaborated.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Eugene L. Allgower and Kurt Georg. Introduction to numerical continuation methods. SIAM, 2003.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: rext-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0133 2022. 2,3
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009, 3, 6, 9
Ernie Chang, Hui-Syuan Yeh, and Vera Demberg. Does the order of training samples matter? improv- ing neural data-to-text generation with curriculum learning. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Associa- lion for Computational Linguistics: Main Volume, pp. 727-733. Association for Computational Linguistics, 2021. 2,
Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11472-1148,, 2022. 1.2.3
Kamil Deja, Anna Kuzina, Tomasz Trzcinski, and Jakub Tomczak. On analyzing generative and denoising capabilities of diffusion-based deep generative models. Advances in Neural Information Processing Systems, 35:26218-26223 2022. 1,3
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009. 2, 7.8
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 11:8780-8794, 2021. 1,2
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. arXiv preprint arXiv:21 2021. 2,3
Hyojun Go, Jin Young Kim, Yunsung Lee, Seunghyun Lee, Shinhyeok Oh, Hyeongdon Moon, and Se- ungtaek Choi. Addressing negative transfer in diffusion models. arXiv preprint 4rx15:2306.00354, 2023a. 2,5,7
Hyojun Go, Yunsung Lee, Jin- Young Kim, Seunghyun Lee, Mycongho Jeong, Hyun Seung Lee, and Seungtack Choi. Towards practical plug-and-play diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1962-1971 2023b. 2
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International conference on machine learning, pp. 2535 2544. PMLR, 2019. 2, 3, 6, 7,
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. arXiv preprint arXiv:2303.0m 095566 2023. 2, 3, 10
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35: 27953-27965, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 5
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12595 2022. 2,
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020. 3,
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 2022.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint 2rXiv:2212.11972 2022.
Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, and Alexander Hauptmann. Self-paced learning with diversity. Advances in neural information processing systems, 27, 2014.1 3
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019. 8
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26677 2022. 1, 3, 4, 7.8
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:23/2.026666 2023.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Soft truncation: A universal training technique of boore diffusion model for high precision International Conference on Machine Learning, pp. 11201-1 1228. PMLR, 2022. 2,
estimation. In
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances Neural Information Processing Systems, 36, 2023. 3, 3.4
Yajing Kong. Liu Liu, Jun Wang, and Dacheng Tao. Adaptive curriculum learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5067-5076, 2021. 2, 3, 6
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, 2009. 7,8
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in neural information processing systems, 23, 2010. 3
Yunsung Lee, Jin-Young Kim, Hyojun Go, Myeongho Jeong, Shinhyeok Oh, and Seungtack Choi. Mult-archiiecture multi-expert diffusion models. arXiv preprint arXiv:2306. 2023. 1, 2, 3
Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-fres free optimization of time steps and architectures for automated diffusion model acceleration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7105-7114, 2023. 2, 3
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014; 13th European Conference, Zurich, Switzerland, September 62, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014. 2, 7.8
Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang. and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. arXiv preprint 22ii:22300088860 2023a. 3
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisten images from a single-view image. arXiv preprint arXiv:2309.03455 2023b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver; A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022. 2
12

---

Published as a conference paper at ICLR 2025
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- and Saining Xie. Sit: Exploring flow and diffusion- generative models with scalable interpolant transformers. arXiv preprint 1.XYx:240100870000 2024. 2, 3,4, 4, 7,
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar. T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching. arXiv preprini arXiv:2402. 2024. 1, 3,3
Byeongjun Park, Hyojun Go, YounY Kim, Sangmin Woo, Seokil Ham, and Changick Kim. Switch diffusion transformer: Synergizing denoising tasks with sparse mixture-of-eepertt. arXiv preprint arXiv:2403.09176, 2024a.
Byeongjun Park, Sangmin Woo, Hyojun Go, Jin- Young Kim, and Changick Kim. Denoising task routing for diffusion models. In The Twelfth International Conference on Learning Representations. 2024b. 2, 7, 10
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09778, 2022. 7,7.8
Anastasia Pentina, Viktoriia Sharmanska, and Christoph H Lampert. Curriculum learning of multiple tasks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5492-5500, 2015. 3,3
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8766 PMLR, 2021. 7
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 22Xiv:2204.06/25, 1(2):, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022. 7
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention-MICCAl 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234 241. Springer, 2015. 7
Daniel L Ruderman. The statistics of natural images. Network: computation in neural systems, 5(4): 517, 1994. 6
Jascha Sohl- Sohl-Dicksttin, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2256-2265. PMLR, 2015. 1, 2,
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02521 2020. 2
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint 2rXiv:2310.14166 2023. 4
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.
Advances in neural information processing systems, 32, 2019. 4
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021. 4,
13

---

Published as a conference paper at ICLR 2025
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 4
Valentin Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. From baby steps to leapfrog: How "less is more" in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp.
751-759 2010.
Zineng Tang, Ziyi Yang. Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In Thirty-seventt Conference on Neural Information Processing Systems, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 7
Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. Curriculum pre-training for end-to-end speech translation. Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3728-3738. Association for Computational Linguistics, 2020. 2,
Sangmin Woo, Byeongjun Park, Hyojun Go, Jin- Young Kim, and Changick Kim. Harmonyview: Harmonizing consistency and diversity in one-image-to-3d. arXiv preprint 11.1v:2312.15980. 2023.
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? arXiv preprint arXiv:2012.03107, 2020. 3
Yilun Xu, Shangyuan Tong, and Tommi Jaakkola. Stable target field for reduced variance score estimation in diffusion models. arXiv preprint arXiv:2302.00670, 2023. 2,
Dongehao Yang. Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation, 2023a. 4
Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22552-252222 22562, 2023b. 1,3
Zhongqi Yue, Jiankun Wang, Qianru Sun, Lei Ji, Eric I-Chao Chang, and Hanwang Zhang. Explor- ing diffusion time-steps for unsupervised representation learning. The Twelfth International Conference on Learning Representations, 2024. 1, 2,
14


Reviews:
[{"review_id": "IL5g252KpU", "reviewer": "Reviewer_PdyQ", "summary": "The paper notices that the diffusion training at low noise levels is more challenging than at high ones. \nBased on this observation, the authors propose a curriculum learning approach for diffusion models (DMs) that facilitates faster convergence and improves overall performance.", "questions": "**Q0 |** Please address the concerns and questions in Weaknesses.\n\n**Q1 |** In Section 4.2, the relative entropy analysis indicates that marginal distributions become less similar as $t$ approaches 0. Could the authors elaborate on why this leads to the conclusion that training is more challenging at low noise levels? If I understand correctly, at small $t$, the tasks become more independent, which may limit the model to share knowledge between adjacent timesteps. Does this explanation seem reasonable?\n\n**Q2 |** The method uses the SNR-based clustering. Did the authors consider using gradient-based clustering [1,2] in their approach? Perhaps, it may provide more accurate intervals for different datasets.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**S1 |** The authors directly answer the question, \"Which timestep intervals are more challenging during training?\" and provide a convincing analysis for different diffusion parameterizations.\n \n**S2 |** The experiments demonstrate that the proposed approach can improve the convergence and final performance of several popular DM methods.\n\n**S3 |** The ablation study explores important questions, such as whether the performance gains persist with larger models and how the approach interacts with other training techniques, e.g., MinSNR loss weighting.", "weaknesses": "**W1 |** The proposed method has limited scientific contribution: the clustering is adopted from [1], the pacing function, while reasonable, is rather trivial, and the idea behind curriculum learning is pretty general. This could be fine if accompanied by very insightful and comprehensive analysis and strong results. Currently, I feel that the overall contribution is not sufficient.\n\n**W2 |** Most experiments are performed using DiT, which currently seems to be a relatively weak baseline. EDM may also be considered outdated. I believe it is important to apply the proposed approach to EDM2[2] and demonstrate the gains on top of it. EDM2 focuses on training techniques and outperforms DiT and EDM by a large margin. Also, it proposes the dynamic loss weighting, which strongly relates to the proposed approach. \n\n**W3 |** The analysis is performed only on FFHQ256 while the dataset and image resolution can be important factors as well. For example, [3] observed that larger models are more beneficial at high noise levels for CIFAR10 and ImageNet 64x64, and, in contrast, larger models are preferable at low noise levels for the LSUN dataset. [4] revealed different optimal timestep intervals for various datasets under the same noise schedule. Thus, it seems valuable to perform analyses across different datasets and discuss any observed trends. It would also be interesting to discuss whether pixel and latent spaces exhibit different behaviors.\n\n---\n[1] Go et al. Addressing Negative Transfer in Diffusion Models, 2023\n\n[2] Karras et al. Analyzing and Improving the Training Dynamics of Diffusion Models, 2023\n\n[3] Ganjdanesh et al. Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection, 2024\n\n[4] Liu et al. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UbM3F2z6gm", "reviewer": "Reviewer_3pvQ", "summary": "The paper proposes a curriculum learning based training of diffusion models, where the model is progressively trained on easier to harder tasks. The authors identify the task hardness based on the timestamps used for the training. For this, they train multiple models, where each model is trained to denoise only a particular subset of timestamp ranges. The loss curve of these models with training steps, depict that model trained on earlier timestamp ranges converges slower than the models trained on later timestamp ranges. Apart from the loss curves, the authors also look at the FID score by sampling from these models during training(to denoise or timestamps other than what the model is trained on, a model trained on all timestamps is used for denoising). Again the FID of models trained on later timestamps is lower than those trained on earlier timestamps. \n\nBased on these analysis, the authors conclude that denoising for earlier timestamps is harder than denoising for later timestamps. Base on this, the authors then devise a curriculum learning-based training of diffusion models. The model is iteratively trained on later to earlier timestamps(easier to harder tasks) clusters. Instead of uniformly dividing the total timestamp into each cluster, the authors use an SNR-based interval clustering technique. Furthermore, since the hardness of different timestamp clusters varies, so the authors propose a better approach than training on each cluster for a fixed number of steps. They define a maximum threshold hyper-parameter(patience), which determines whether to switch to the next training cluster. If the loss in that cluster stays constant for more than patience steps, then the training proceeds to the next cluster. \n\nThe authors conduct experiments across different diffusion training architectures, such as as DiT, SiT and also over different training datasets such as FFHQ, ImageNet. Across different architectures and datasets, the proposed approach consistently outperforms the vanilla approach of training without a curriculum learning schedule.", "questions": "I have already asked questions in the weakness ection", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors show consistent improvement across a variety of architectures and datasets thus denoting the effectiveness of their approach.\n\n* I like the anti-curriculum learning ablation, where instead of training on easier to harder tasks, the authors instead train on harder to easier tasks which doesn't perform any better than the baseline.\n\n* I like the analysis done by the authors in determining the hardness of different timestamp schedules.", "weaknesses": "I have a few concerns regarding the paper - \n\n* It seems that the effectiveness of the approach is reduced when the model is trained for longer steps. The difference in performance between the baseline and test after 2M training steps is much smaller than the difference at 400k steps. In that way, the main effectivess of the approach is just faster convergence, instead of improved performance. How do the authors justify the improved performance then?\n\n* Can the authors show comparison between the curriculum and anti-curriculm approach for unconditional image generation also instead of just class-conditional generation?\n\n* The authors should explain more in the paper about the SNR-based interval clustering technique and why it is a better interval technique than uniformly clustering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tkq26guQM9", "reviewer": "Reviewer_Ujmn", "summary": "This paper aims to improve the learning process of image generation using diffusion models through a technique called curriculum learning. The authors first observed the progress of diffusion model training. They found that denoising images with more noise (larger time steps) is easier than denoising images with less noise (smaller time steps). This was confirmed by examining the convergence of loss functions and FID scores at each noise level.\n\nFurthermore, by analyzing the KL divergence between marginal probability distributions of consecutive time steps, they also demonstrated that the denoising task becomes more difficult with larger time steps.\n\nBased on these observations, the authors adopted a common curriculum learning approach called \"easy-to-hard training scheme.\" Specifically, they proposed a strategy that starts learning from time steps with more noise and gradually expands the range of time steps being learned. This strategy requires designing a \"pacing function\" to determine how to expand the learning range. The authors adopted a technique that transitions to the next phase based on the status of the training loss.\n\nIn experiments, they confirmed that the proposed method improves performance compared to vanilla learning strategies across multiple baseline models. Additionally, they demonstrated faster convergence and orthogonality (i.e., compatibility) with existing learning improvement methods.", "questions": "1. Regarding Weakness 1: Can authors add theoretical justification or explanation, perhaps by citing other curriculum learning literature?\n2. Also related to Weakness 1: Diffusion models differ from networks in other curriculum learning literatures in that the time step conditioning changes at each training phase. This might cause behaviors at different time steps (which should ideally be learned independently) to influence each other through shared network parameters. Therefore, I guess that the traditional curriculum learning framework might not fully explain the effectiveness of this method. Can you provide any references or discussion to address this question?\n3. Regarding Weakness 2: Can you discuss why the performance of the proposed method is significantly inferior to current SoTA methods? For instance, the latest results on the ImageNet 256x256 dataset had FID scores below 5 as of 2022. (While additional experiments are not expected, if you could demonstrate improvement on a very high-performing model, it would strongly support the claims and effectiveness of this paper's method.)\n4. Also related to Weakness 2: From Table 2, it seems that the results in Table 1 are from models that haven't fully been converged. While improved convergence performance is promising, it would be beneficial to show that performance improvements are still observed with further training for other models as well.\n5. Regarding Weakness 3: Can you discuss the relationship between the proposed method and other diffusion model learning improvement techniques? For example, while this method is compatible with noise weighting techniques, I feel they might not be completely orthogonal in theory.\n6. Can the KL divergence analysis in Section 4.2 be explicitly calculated in the forward process of diffusion?\n\nAdditional Notes\n1. In the supplementary material, references are not properly cited.\n2. When discussing convergence speed, for example in Figure 6, I believe it's important to include data points that show full convergence. Reaching the performance ceiling quickly is an advantage, so demonstrating this would be valuable.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The key features of this paper are as follows:\n\n1. The motivation for the research is clearly explained in Section 4 (Observations). In particular, Figures 1 and 2 effectively illustrate the problems that need to be addressed.\n2. The proposed method requires the design of a pacing function and scheduling, as described in Section 5.2. However, the core idea itself is simple and elegantly solves the identified problems.\n3. In the experimental section, the method is tested on multiple baseline models, confirming that the proposed approach brings improvements in each case. Additionally, appropriate ablation studies have been conducted, suggesting the generalizability of the proposed method.", "weaknesses": "1. While the motivation in section 4 is clear, there is insufficient explanation as to why this method is expected to improve performance. Even though the \"easy-to-hard training scheme\" is well-known in the field of curriculum learning, the paper lacks discussion on why it works effectively and its connection to theoretical aspects.\n2. It is appealing that the proposed method can generally improve upon baselines. However, from my understanding, the reported performance seems to be significantly different from the current state-of-the-art results. While achieving state-of-the-art performance is not mandatory for this type of paper, the lack of discussion about this performance gap raises concerns about the generalizability of the method.\n3. This method presents a novel approach in applying curriculum learning to diffusion model training. However, there seems to be a lack of discussion regarding its relationship and comparison with other learning improvement techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksYDhdqsfE", "reviewer": "Reviewer_xMw1", "summary": "The paper proposes a curriculum-based training schedule for diffusion models that trains the model on progressively increasing task difficulty, which corresponds to different bins of the diffusion noise schedule. The authors provide empirical evidence that task difficulty increases with decreasing time steps (as SNR increases), and provide a simple scheme of dividing the training into difficulty tasks. Results show that the scheme improves the convergence and quality of diffusion samples with no training overhead.", "questions": "- Interesting that the anti-curriculum training (hard to easy) can also improve performance over vanilla, even if not consistently (Table 3). Do the authors have insight on why? This might come across as contradictory to the main claims of the paper and I suggest the authors explain it clearly to avoid confusing the reader.\n- Might be small typo in line 294. As time step increases, the expression in parenthesis suggests KL increases which contradicts Fig 2.\n- It seems like the diffusion community is moving more towards flow/ODE-based/consistency models, rather than DDPM-style models. Have the authors tried applying their method to flow-based methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Dividing training into difficulty phases determined by diffusion timestep clusters is novel to my knowledge. \n- The paper is well structured and the method is well motivated. The presentation of evidence that difficulty varies depending on diffusion timestep (Sec 4) into the proposed method (Sec 5) is well thought and makes the paper flow nicely.\n- Comprehensive experiments show improved performance over baselines at same number of training iterations, i.e., with no additional training overhead, with sensible ablations justifying design choices made by the authors.", "weaknesses": "- Overall the main novelty of the work is on the proposed training schedule, which is simple and on the lower side. Sec 4 is more of an empirical confirmation that lower noise levels is more difficult, which I believe is already well-known. I think this is not a big negative point however, as there is merit to simple ideas that work.\n- Some missing citations as the general idea of training diffusion models from easy to difficult tasks is not new. The earliest and most influential ones to my knowledge are progressive distillation [1] (many to few sampling steps) and cascaded diffusion [2] (low to high-res). \n- Sec 4.2, it is not clear to me why high KL between marginals of two time steps implies higher task difficulty. I assume it is because higher KL means the model has to make larger changes to the image between timesteps, thus making it more challenging. I think this can be more clearly stated.\n\n[1] Salimans, Tim, and Jonathan Ho. \"Progressive distillation for fast sampling of diffusion models.\" arXiv preprint arXiv:2202.00512 (2022).\n\n[2] Ho, Jonathan, et al. \"Cascaded diffusion models for high fidelity image generation.\" Journal of Machine Learning Research 23.47 (2022): 1-33.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rzjkL4FRE7", "reviewer": "Reviewer_Sh9g", "summary": "In this work, a new analysis of the levels of difficulty of different parts of diffusion model is presented followed by the introduction of new curriculum-based approach for improved training. Authors first show that it is harder to learn how to denoise samples that are less noisy in the diffusion process. On top of this observation they introduce a new method for the training of diffusion models, where model is first trained with only simpler timesteps, followed by the harder ones and full training. The extensive evaluation validates that such approach leads to better performance of the final model and faster convergence.", "questions": "- How splitting the process into separate models for separate parts affected the loss convergence? I can imagine that when training a single model on all of the steps, there is some positive/negative transfer between different tasks.\n- Are all of the models in the evaluation section trained for the same number of steps? How is it achieved in the CL training with maximum patience iteration that introduces various number of training steps in each CL task?\n- The results presented in Table 1 for some methods are relatively close to each other. How many examples were used to calculate FID, are those results statistically significant (as there are no confidence intervals)\n- Were all of the models trained with a linear noise scheduler? How would the results be affected by changing it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of curriculum learning for faster and more efficient diffusion training is interesting and, to the best of my knowledge, novel. The experimental evaluation is convincing as the method seems to improve the performance of the models across several benchmarks and with different architectures \n- Submission provides interesting experiments in the preliminary observation. I particularly appreciate experiments where part of the sampling trajectory was generated using a separate model trained for a limited period of time. This experiment brings valid observations regarding how challenging the training of earlier/later diffusion steps is. However, there is still one question remaining - what is the influence of the training on the remaining steps on other steps - see the questions section.\n- On top of the proposed method and main benchmarking, the authors present an extensive additional experiments section that provides an in-depth analysis of the solution’s strengths.", "weaknesses": "- I’m left with a single concern. Is it really thanks to the curriculum learning, or is it just important to first learn how to do denoising in the initial steps of the diffusion process - which define the mapping between random Gaussian noise and training data so that later training is easier? Driven by the confusing results of the evaluation presented in Figure 4, I lack one last experiment where the model is first trained using only timesteps from the C_N cluster followed by random ordering or standard training. Would it be significantly worse than the presented approach?\n- In Section 5.2 there is a statement that “the convergence rate of each curriculum phase varies significantly, as demonstrated in Fig. 1.” - - - This is true, but I hope that I understand correctly that in the CL scenario, the model is first trained with the easy task, but then the same model is further finetuned with harder tasks. This should affect the convergence speed.\n- Figure 4 is very puzzling. It suggests that it actually doesn’t matter how much we split the process used by the curriculum training, the results are almost identical except for the magical 20 splits used throughout the rest of the submission.  \n- Small errors/suggestions: Section 5 describes the Method rather than the Methodology. Table 2, Figure 4 are not within the specified margins", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)