Consistency Models Made Easy
Authors: Zhengyang Geng
Ashwini Pokle
Weijian Luo
Justin Lin
J Zico Kolter
Venue: ICLR 2025 Poster (2025)
Keywords: Consistency Models
Efficient Generative Models
Diffusion Models
Primary area: generative models
TL;DR: Consistency Models Made Easy

Abstract: Consistency models (CMs) offer faster sampling than traditional diffusion models, but their training is resource-intensive. For example, as of 2024, training a state-of-the-art CM on CIFAR-10 takes one week on 8 GPUs. In this work, we propose an effective scheme for training CMs that largely improves the efficiency of building such models. Specifically, by expressing CM trajectories via a particular differential equation, we argue that diffusion models can be viewed as a special case of CMs. We can thus fine-tune a consistency model starting from a pretrained diffusion model and progressively approximate the full consistency condition to stronger degrees over the training process. Our resulting method, which we term Easy Consistency Tuning (ECT), achieves vastly reduced training times while improving upon the quality of previous methods: for example, ECT achieves a 2-step FID of 2.73 on CIFAR10 within 1 hour on a single A100 GPU, matching Consistency Distillation trained for hundreds of GPU hours. Owing to this computational efficiency, we investigate the scaling laws of CMs under ECT, showing that they obey the classic power law scaling, hinting at their ability to improve efficiency and performance at larger scales. Our [code](https://github.com/locuslab/ect) is available.

Published as a conference paper at ICLR 2025
CONSISTENCY MODELS MADE EASY
Zhengyang Geng'
Ashwini Pokle'
Weijian Luo?
Justin Lin'
J. Zico Kolter'
CMU
Peking University
ABSTRACT
Consistency models (CMs) offer faster sampling than traditional diffusion models, but their training is resource-intensive. For example, of 2024, training state-of- the-art CM on CIFAR-10 -- takes one week on 8 GPUs. In this work, we identify the "curse of consistency" for training such models and propose an effective training scheme that largely mitigates this issue and improves the efficiency of building such models. Specifically, by expressing CM trajectories via the differential equation, we argue that diffusion models can be viewed as special case of CMs. We can thus fine-tune a consistency model starting from pretrained diffusion model and progressively approximate the full consistency condition to stronger degrees over the training process. Our resulting method, which we term Easy Consistency Tuning (ECT), achieves vastly reduced training times while improving upon the quality of previous methods: for example, ECT achieves a 2-step FID of 2.73 on CIFARIO within I hour on single A100 GPU, matching Consistency Distillation trained for hundreds of GPU hours. Owing to this computational efficiency, we investigate the scaling laws of CMs under ECT, showing that they obey the classic power law scaling, hinting at their ability to improve efficiency and performance at larger scales. Our code available.
INTRODUCTION
Diffusion Models (DMs) (Ho et al., 2020; Song et al., 2021a), or Score-based Generative Models (SGMs) (Song al., 2020; 2021b), have vastly changed the landscape of visual content generation with applications in images (Rombach et al., 2021; Saharia et al., 2022; Ho et al., 2022a; Dhariwal and Nichol, 2021; Hatamizadeh et al., 2023; Ramesh et al., 2021), videos (Brooks al., 2024; Blattmann et al., 2023; Bar-Tal et al., 2024; Ho et al., 2022b; Gupta et al., 2023), and 3D objects (Poole et al., 2022: Wang et al., 2024a; Lee et al., 2024; Chen al., 2024; Babu et al., 2023). DMs progressively transform a data distribution to a known prior distribution (e.g. Gaussian noise) according stochastic differential equation (SDE) (Song et al., 2021b) and train a model to denoise noisy observations. Samples can be generated via reverse-time SDE that starts from noise and uses the trained model to progressively denoise it. However, sampling from a DM naively requires hundreds to thousands of model evaluations due the curvature of the diffusion sampling trajectory (Karras et al., 2022), making the entire generative process slow. Many approaches have been proposed to address this issue, including training- based techniques such as distillation (Luhman and Luhman, 2021; Salimans and Ho, 2022; Luo et al., 2024; Gu et al., 2023; Sauer et al., 2023; Geng et al., 2024; Yin al., 2023; Nguyen and Tran, 2023), adaptive compute architectures for the backbone model (Moon et al., 2023; Tang et al., 2023), as well as training-frre free methods such as fast samplers (Kong and Ping, 2021; Lu al., 2022a; Zhang and Chen, 2022; Zhou al., 2023; Xue et al., 2024) interleaving small and large backbone models during sampling (Pan et al., 2024). However, sample quality and mode coverage achieved by these speedup techniques still struggles when the number of model evaluations is reduced to below 5.
Consistency Models (CMs) (Song et al., 2023) are a new family of generative models, closely related to diffusion models, that have demonstrated promising results as faster generative models. These models learn a mapping between noise and data, and all the points of the sampling trajectory map to the same initial data point. Owing to this condition, consistency models high-quality samples in 1-2 model evaluations. The best such models so far, built using improved Consistency Training (iCT) (Song and Dhariwal, 2023), have pushed the quality of images generated
capable of generating

---

Published as a conference paper at ICLR 2025
by 1-step CMs trained from scratch level comparable with SoTA DMs using thousands of steps for sampling. Unfortunately, CMs remain consuminn and practically challenging to train: the best practice takes many times longer than similar-quality DMs while involving complex hyperparameter choices in the training process. In total, this has substantially limited the uptake of CMs within the community.
In this work, we introduce a differential perspective on consistency models, leading to the formulation of the differential consistency condition in continuous time. This insight reveals the link between diffusion models and consistency models, viewing DMs as a special case of CMs with loose dis- cretization. This observation motivates to smoothly interpolate from DM CM by progressively tightening the consistency condition, bootstrapping pretrained DMs to 1-step CMs w/o using extra frozen teachers. We term this strategy as Easy Consistency Tuning (ECT), which includes diffusion pretraining as special stage of the continuous time training schedule.
ECT significantly improves both training efficiency and performance. On ImageNet 64x64 (Deng et al., 2009), ECT achieves superior 1-step and 2-step sample quality compared to the prior art. Similarly, on CIFAR-IO (Krizhevsky, 2009) 2-step sample quality of ECT surpasses previous methods. The total cost of the pretraining and tuning scheme requires only 1/4 ~ 1/3 of the computational resources (FLOPs) used by the current statc-of-the-at method, iCT (Song and Dhariwal, 2023), while the tuning stage can be remarkably lightweight, typically accounting for 10% or less of the overall cost and further benefiting from scaling.
Leveraging ECT's computational efficiency, conduct the first study into the scaling behaviors of CMs, revealing the classic power law scaling for model size, FLOPs, and training compute. The scaling also suggests a sweet spot of using smaller few-step CMs over larger I-step CMs in certain scenarios. This computational efficiency enables us to explore the design space of CMs by using the tuning stage as a proxy. Notably, tuning findings, such as weighting functions. can improve the pretraining stage and, in turn, enhance the overall pretraining + tuning pipeline for CMs.
In short, we summarize our contributions follows:
We develop Easy Consistency Tuming (ECT), a pretraining + tuning scheme for training CMs in continuous time, demonstrating significant efficiency and performance gains compared to the current best practices for training CMs.
We investigate CMs' scaling behaviors for the first time and reveal the classic power law.
We explore the design space of CMs through ECT, introducing the contimuous-time schedule and better weighting functions for CMs.
PRELIMINARIES
Diffusion Models. Let (xo) denote the data distribution. Diffusion models (DMs) perturb this distribution by adding monotonically increasing i.i.d. Gaussian noise with standard deviation (1)
from 0 to T such that p. px(x (x, xxo) -N(xo, (t) and (l)) is chosen such that (0) Omin
and o (T) = maax This process is described by the following SDE (Song et al., 2021b)
dx = g())dw
(1)
where w is the standard Wiener process, and 9(-) generated by solving the reverse-time SDE starting from = T to 0 and sampling X ~ A A((0, o T). (Song et al., 2021b) show that this SDE has a corresponding ODE, called the probability flow ODE (PF-ODE), whose trajectories share the same marginal probability densities as the SDE, We follow the notation in (Karras al., 2022) to describe the ODE as
R is the diffusion coefficient. Samples can be
dx - (1) log pi (x:)dt,
where Vx, log p. (x²) denotes the score function. Prior works (Karras et al., 2022; Song et al., 2023) set (1) which yields
dx; -V log (x;;)
dt
We will follow this parameterization in the rest of this paper. Note that time is the same as noise level with this parametrization, and will use these two terms interchangeably.
2

---

Published as a conference paper at ICLR 2025
Consistency Models. CMs are built upon the PF-ODE in Eq. (3), which establishes bijective mapping between data distribution and noise distribution. CMs learn a consistency function f(x,, t) that maps the noisy image x, back to the clean image Xo
f(x,, t) Xo-
By taking the time derivative of both sides, given by continuous-time consistency models (Song et al., 2023).
df di
O.
d/ 0 alone is not sufficient to guarantee that the model output However, this differential form ar will match the clean image, as there exist trivial solutions where the model maps all the inputs to constant value, such as fxx, t) O. To eliminate these collapsed solutions, (Song et al., 2023; Song and Dhariwal, 2023) impose the boundary condition for f(x,, t) Xg via model parameterization:
df = 0, (xu, 0) = Xu- dt
f(xi.t) = xx +
(6)
This boundary condition fxx, 0) Xo ensures that the model output matches the clean image when the noise level is zero. Together, the differential form in Eq. (5) and the boundary condition define the consistency condition.
Prior works (Karras et al., 2022; Song et al., 2023; Song and Dhariwal, 2023) impose this boundary condition by parametrizing the CM as
fe t) = Cskp (t) x + (tur t)
where is the model parameter, Fg is the network to train, and Cskip (1) and com (() are time-dependent scaling factors such that Cskip( 1, Cout (0) = O. This parameterization guarantees the boundary condition by design. We discuss specific choices of cakip (t) and Cour(t) in Appendix D.
Training Techniques for Consistency Models. During training. CMs first discretize the PF-ODE into N 1 subintervals with boundaries given by tmin = ti < t2 < < tx = T. The model is trained on the following CM loss, which minimizes a metric between adjacent points on the sampling trajectory
fo- aig E (wo(t) (ffo (Xt.u1s (441) (Xx,,
(8)
Here, d(.).) is metric function, the fo indicates the consistency function, fo- indicates an exponential moving average (EMA) of the past values of fe. and &t, = Xt.+1 (t: 1++1)4+1 V Vx.,,, logPt.. (Xt,.)). Further, the discretization curriculum N should be adjusted during training achieve strong performance.
In the seminal work, Song et al. (2023) use Learned Perceptual Similarity Score (LPIPS) (Zhang al., 2018) as a metric function, set wt,,)
for all 4,, and sample f, according to the sampling
scheduler by Karras al. (2022): f, = (/// + ~ (/// (1/)) for E (11 N 1] and Further, the score function V x, ((x,) can either be estimated from pretrained diffusion model, which results in Consistency Distillation (CD), or can be estimated with an unbiased score estimator Eq. (9), corresponding to consistency training (CT).
Vx, log pl (x (x)) X = E x,
X0
Vx, logp(x;))
X4
The follow-up work, iCT (Song and Dhariwal, 2023), introduces techniques that significantly improve the performance of CMs. First, the LPIPS metric, which introduces undesirable bias in generative modeling, is replaced with Pseudo-Huber metric. Second, the network fe does not maintain an EMA of the past values of fo. Third, iCT replaces the uniform weighting scheme w(t,) 1 with w(tx)= = - Further, the scaling factors of noise embeddings and dropout are carefully selected. Fourth, iCT introduces discretization curriculum during training:
M
(mm) min (2,22 .1) + 1,
M'=
(10)
loga
S1
+
3

---

Published as a conference paper at ICLR 2025
where is the current number of iterations, M is the total number of iterations, max and min is the largest and smallest noise level for training. So = 10 and 81 1280 are hyperparameters. Finally,
during training, iCT samples p()) x erf log 1) Paeun
erf log(t,)- -Paa from discrete V2P
y2P
Lognormal distribution, where m = 1.1 and Paud 2.0.
3 PROBING CONSISTENCY MODELS
We first discuss the finite difference approximation on continuous-time consistency models. Next, we will analyze this loss objective and highlight the challenges of training CMs with it. Based on this analysis, we present our method, Easy Consistency Tuning (ECT). ECT is simple, principled approach to efficiently train CMs to meet the consistency condition. The resulting CMs can generate high-quality samples in or 2 sampling steps.
3.1 THE "CURSE OF CONSISTENCY" AND ITS IMPLICATIONS
To learn the consistency condition, continuous timee consistency models (Song et al., 2023) could be discretized using finite-difference approximation:
df fo(x)) fx(x)) di
0
(11)
1-r
where di R A = ----r,,>> > O, and sx(X)) denotes So (xx, l). For a given clean image Xo. produce two perturbed images x, and x, using shared noise direction ~ p(e) at two noise levels and r, i.e., x + l. and x, = Xo r.
Given the discretization, the objective function for CMs in Eq. (8) can be written as
arg arg min Ex,,e,t [w[t,, fagto
(12)
0
while removing the discrete training schedules. The consistency condition in Eq. (6) holds by eliminating discretization errors, i.e., At rt 0.
However, the consistency loss can be challenging to opti-
mize when At + 0. This is because the prediction errors from each discretization interval accumulate, leading to
200
slow training convergence in the worst case, divergence. To further elaborate, consider a large noise level T. We first split the interval from T' into N smaller consecutive subintervals.
100
maximum noise level
40
20
10
[[o(())) - xoll < [)(()),)) So(xx;)ll S N emax:
4
5
10
15
20
25
30
35
40
(13)
(10k)
= -. = and Cmax = max; - for 1, N. Ideally, want both N and emax be small that this upper bound is small. But in practice, there is a trade- off between these two terms. As At; (4, T2) + 0, Cmax decreases because X1, and X,, will be close, and it easier predict both fo (x,,) and fo However, N will increase as A/; 0. In contrast, for large Al, vice-versa holds true. It is difficult to theoretically estimate the rate which 6max decreases when At, = 4 - r; - 0, as it depends on the optimization sprresssspeccficallll how effectively we train the model to minimize the consistency error in each interval. If emax decreases more slowly than N, the product of the two can increase instead, resulting in worse prediction error (x(o Xull-
where rı =
Figure 1: The "Curse of Consistency": The consistency condition holds At dt. However, the training dynamics con- verges more slowly and is less stable as At + 0 (i.e., N - oo).
The insight from the above observation that when training from scratch by strictly following the differential form df/a+ with tiny At; z 0, the resulting model might converge slowly due to accumulated consistency errors from each interval At, To investigate this hypothesis, we conducted experiments training series of CMs using different fixed numbers of intervals N, and corresponding
4

---

Published as a conference paper at ICLR 2025
At values for the consistency condition, as illustrated in Fig. 1. Our observations align with Song and Dhariwal (2023). with the key distinction that we use a fixed N for this analysis, as this approach provides better insight into how much the precise approximation of the consistency condition matters and isolates the effect of discretization errors.
3.2 EASY CONSISTENCY TUNING (ECT)
The discussion in Sec. 3.1 highlighted the training instability issue that can arise when we naively optimize for the differential consistency condition with the loss objective in Eq. (12) while directly following At 0. In this section. we propose several strategies that largely alleviate the aforemen- tioned issues and improve the efficiency of CMs. We term this approach Easy Consistency Tuning (ECT), as effectively balances training stability and model performance while simplifying the CM training process. ECT follows a two-stage approach: diffusion pretraining, followed by consistency tuning, which we will detail the following subsections with a summary in Alg. 1.
Diffusion Pretraining + Consistency Tuning. Drawing inspiration from iCT's adaptive discrete- time schedule (Song and Dhariwal, 2023), we start ECT with large At, and gradually shrink Al + In our problem setup, since > r 2 O, we have the largest possible At I with r 0, which yields
arg min (x.) Sagio) (x,) )l = ())(x.) - (x)(o) 1[fo(x.) -xol-
(14)
Training model with this loss is identical to diffusion model/Score SDE (Ho et al., 2020; Song ct al., 2021b). This observation suggests learning scheme that smoothly interpolates from DMs At = t to CMs At dt by gradually shrinking At - 0 during training by gradually tightening the consistency condition. With this reasoning, diffusion pretraining can be considered as a special case of consistency training with a loose discretization of the consistency condition. Therefore, in practice, we start ECT with a pretrained diffusion model, resulting in a training scheme of pretraining+tunigg Another benefit of this initialization is that during training, especially in the initial stages, it ensures good targets (g) (x,) in the loss objective, avoiding trivial solutions.
We highlight two advantages of this learning scheme: 1. outperforms iCT's training-from-ccralee from-scratch approach with lower overall computational cost (see Sec. 4.1). 2. Tuning could serve as an efficient proxy for exploring the CM design space. Given a pretrained diffusion model, insights gained during tuning can be applied to improve the pretraining stage, resulting in more refined overall CM training pipeline. Please refer to Appendix for details.
The pretraining+tuning scheme of ECT
Continuous-time Training Schedule. We investigate the design principles of a continuous-time schedule whose "boundary" condition yields standard diffusion pretraining, i.e., constructing training pairs of r 0 for all t the beginning. Note that this is unlike the discrete-time schedule used in iCT (See Sec. and Appendix A). We consider overlapped intervals for consistency models, which allows for factoring rtt, p(r)p( and continuous sampling of infinite ł from noise distribution p(t), for instance, LogNormal( (Paean: Pau), and r ~ p(rlt).
p(rlt)
Since need to shrink At - 0 the training progresses, augment the p(rlt) to depend on training iterations, p(rlt, iters), to control At (t r)) + O. We parametrize p(r iters) as
1
1
n(1) = gittersdjj n(!)
(15)
where we take n(t)) - 1
bt) = + 1tels with (-)) as the sigmoid function, iters refers to training iterations. In general, we set > 1, k = 8 and b = 1. Since r 2 O, we also clamp to satisfy this constraint. At the beginning of training, this mapping function produces r/2 = 0, which recovers the diffusion pretraining. We discuss design choices of this function in Appendix A.
k
Weighting function. Weighting functions usually lead to substantial difference in performance in DMs, and the same holds for CMs. When considering iCT's weighting function, w(t, r) = l/t-r, derived from the finite difference approximation, it couples the timestep weighting with p(r(t). Instead, we consider more flexible timestep weighting independent from the finite difference approximation. We formulate the weighting function
(t) = w(t) w(A) vi())
(16)
A
+ 2)P
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Easy Consistency Tuning (ECT)
Input:
Dataset D, pretrained diffusion model Φ, mapping function p(r f, lttrs), weighting function w(t).
Init: 8 + o, Iters O.
repeat
Sample Xg ~ e p(c), p(e))t=ph p(t,,, r p(r t, Iters)
Compute x, X0 I. 6,,,,,,, 6, X0 +r 6, Al
[(0) = w(t) (x)))(x); (xg(o)
sg is stop-gradient operator
4 8 ((0)
Iters Iters 1
until Al + 0 return 0
D ECM
where A = f(x)) f(x)). We define w(A) as adaptive weighting since it operates on a per data-sample basis and adapts to different magnitudes of error. In particular, p = 1/2 corresponds to the Pseudo-Huber loss introduced in the prior art (See Appendix A for more details).
In general, we notice that CMs' generative capability greatly benefits from weighting functions that control the variance of the gradients across different noise levels. We direct the reader to Appendix B for a detailed overview of various choices of (t) and ()) considered in this work.
Dropout. In line with (Song and Dhariwal, 2023), we find that CMs benefit significantly from dropout (Hinton et al., 2012). On ImageNet 64 x6, we note that ECT benefits from surprisingly high dropout rate. When increasing the dropout rate from 0.10 to 0.40, the 2-step FID decreases from 4.53 to 3.24. Finally, we note that the dropout rate tuned at given weighting function w(t) transfers well to the other weighting functions, thereby reducing the overall cost of hyperparameter tuning.
4 EXPERIMENTS
This section compares different learning schemes and investigates scaling laws of ECT, while more experiments design choices and scaling shown in Appendix B. We evaluate the efficiency and scalability of ECT on two datasets: CIFAR 10 (Krizhevsky. 2009) and ImageNet 64 x 64 (Deng al., 2009). We measure the sample quality using Fréchet Inception Distance (FID) (Heusel et al., 2017) and Fréchet Distance under the DINÓv2 model (Oquab et al., 2023) (FDDINowz) (Stein et al., 2024) and sampling efficiency using the number function evaluations (NFEs). We also indicate the relative training costs of each of these methods. Implementation details can be found in Appendix D.
4.1 COMPARISON OF TRAINING SCHEMES
We compare CMs trained with ECT (denoted as ECM) against state-of-the-art diffusion models, ad- vanced samplers for diffusion models, distillation methods such as consistency distillation (CD) (Song et al., 2023), and improved Consistency Training (iCT) (Song and Dhariwal, 2023). We show the training FLOPs, inference cost, and generative performance the four training schemes.
Score SDE/Diffusion Models. We compare ECMs against Score SDE (Song et al., 2021b), EDM (Karras et al., 2022), and EDM with DPM-Solver-v3 (Zheng et al., 2024). 2-step ECM, which has been fine-tuned for only 100k iterations, matches Score SDE-deep, with 2x model depth and 2000 NFEs, in terms of FID. As noted in Fig. 2, ECM only requires a fraction of its inference cost and latency achieve the same sample quality. 2-step ECM fine-tuned for 100k iterations outperforms EDM (Karras et al., 2022) with advanced DPM-Solver-v3 (NFE=10).
Diffusion Distillation. We compare ECT against Consistency Distillation (CD) (Song et al., 2023), SoTA approach that distills pretrained DM into CM. As shown in Tab. 1, ECM significantly outperforms CD on both CIFAR-10 10 and ImageNet 64x64, We note that ECT is free from the errors of teacher DM and does not ineur any additional cost of running teacher DM. 2-step ECM outperforms 2-step CD (with LPIPS (Zhang et al., 2018)) in terms of FID (2.20 vs 2.93) on CIFAR-10 while using around 1/3 of training compute of CD.

---

Published as a conference paper at ICLR 2025
Comparison Learning Schemes
4.5
4.0
FID: 2.24
Diffusion Pretraining
Inference Cost (Log Scale)
Distillation
10
Training
3.0
Tuning
2.5
Inference Cost
10
2.04
Training
1.5
10k
FID: 2.15
0.5
109
0.0
Pretraining SDE/DM
Pretraining Distillation
Training from Scratch
Pretraining+ Tuning
Figure Comparison of training schemes for the diffusion-consisteneyy family on CIFAR-IO. Without relying on distillation from frozen diffusion teachers or extra adversarial supervision, ECT surpasses Consistency Distillation (CD) (Song al., 2023) and Consistency Models trained from scratch (iCT) (Song and Dhariwal, 2023) using 1/4 of the total training cost. ECT significantly reduces the inference cost compared to Score SDE/DMs (Diffusion Pretraining) while maintaining comparable sample quality.
Figure Scaling up training compute and model sizes results in improved sample quality on ImageNet 64x64 Each triplet (left-to-right) shows 2-step samples from ECM-S trained with 12.8M images, ECM-S trained with 102.4M images, and XL trained with 102.4M images.
Consistency training from scratch. Improved Consistency Training (ICT) (Song and Dhariwal, 2023) is the SoTA recipe for training consistency model from scratch without inferring the diffusion teacher. Compared to training from scratch, ECT rivals iCT-deep using 1/4 of the overall training compute (1/s in the tuning stage) and 1/2 of the model size shown in Fig. 2 and Tab. 1.
4.2 SCALING LAWS OF ECT
We leverage the efficiency of ECT examine the scaling behavior of CMs, including training compute, model size, and model FLOPs. We find that when computational resources are not bottleneck, ECT scales well and follows the classic power law.
Training Compute. Initializing from the weights of EDM (Karras al., 2022), we fine-tune ECMs across six compute scales on CIFAR-I0 (Krizhevsky, 2009) and plot the trend of FD against training compute Fig. (Left). The largest compute reaches 2x the diffusion pretraining budget. As we scale up the training budget, we observe classic power-law decay in FDpINOv2, indicating that increased computational investment in ECT leads to substantial improvements generative performance. Intriguingly, the gap between 1-step and 2-step generation becomes narrower when scaling up training compute, even while using the same At - schedule. We further fit the pawer-law FDDINOv2 K C where C is the normalized training FLOPs. The Pearson correlation coefficient between log (Training Compute) and log(F (FDpINwa)) for 1-step and 2-step generation is -0.9940 and -0.9996, respectively, both with statistical significance (p-values 10 10-1). 1).
7

---

Published as a conference paper at ICLR 2025
Table I:
Generative performance unconditional CIFAR-10 and class-conditional ImageNet 64x64. We use a budget of 12.8M training images (batch size 128 and 100k iterations) for ECMs. stands for budget of 102.4M training images (batch size 1024 and 100k iterations) on ImageNet 64x64. Results for prior methods are reported from Song and Dhariwal (2023); Karras et al. (2024).
CIFAR-10
ImageNel 64x64
Method
FIDJ
NFE
Method
FIDI
NFE,
Diffusion Models
Diffusion Models
Score SDE (Song et al., 2020)
2,38
2000
ADM (Dhariwal and Nichol, 2021)
2.07
250
Score SDE-deep (Song 2020)
2.20
2000
EDM (Karrus et al., 2022)
2.22
EDM (Karras et al., 2022)
2,01
35
EDM2-XL (Karras al., 2023)
1.33
63
EDM (DPM-Solver-v3) (Zheng et al., 2024)
2,51
10
Diffusion Distillation
Diffusion Distillation
BOOT (Gu al.. 2023)
16.3
1
PD (Salimans and Ho, 2022)
8.34
DENO (LPIPS) (Zheng ct al., 2023)
7.83
GET (Geng et al., 2024)
5.49
Diff-Instruct (Luo al., 2024)
5.57
Dift- Instruct (Luo ct al., 2024)
4.53
TRACT (Berthelot al., 2023)
4.97
TRACT (Berthelot et al., 2023)
3.32
PD (LPIPS) (Salimans and Ho, 2022)
5.74
CD (I.PPSS) (Song et al., 2023)
3,55
CD (LPIPS) (Song al., 2023)
4.70
2
CD (LPIPS) (Song al., 2023)
2,93
Consistency Models
Consistency Models
iCT (Song and Dhariwal, 2023)
3.20
CT (Song and Dhariwal, 2023)
2,83
iCT-deep (Song and Dhariwal, 2023)
2.77
iCT-deep (Song and Dhariwal, 2023)
2.51
ECT
2.24
ECM-S (100k iters)
3.18
2
ECT
ECM-M (100k iters)
2.35
2
ECM-L (100k iters)
2.14
2
ECM (100k iters)
4.54
ECM-XL (100k iters)
1.96
ECM (200k
3.86
BCM (400k iters)
3.60
ECM-S'
4.05
1
ECM (100k iters)
2,20
2
ECM-S"
2.79
2
ECM (200k iters)
2.15
ECM-XL"
2.49
RCM (400k iters)
2.11
ECM-XI."
1.67
Table 2: Ablation study of design choices on ImageNet 64 x4
Methods
I 1-step FID
2-step FID
iCT EDM2 Pretraining
21.09
4.39
+ Continuous time schedule
14.34
4.33
+ Dropout 0.40
9.28
3.22
+ (() 1/22 + 1/0
5.51
3.18
Model Size & FLOPs. Initialized from EDM2 (Karras al., 2024) pretraining, we train ECM- S/M/L/XLL models with parameters from 280M to 1.1B and model FLOPs from 102G to 406G. As demonstrated Fig. both 1-step and 2-step generation capabilities exhibit log-linear scaling for model FLOPs and parameters. This scaling behavior confirms that ECT effectively leverages increased model sizes and computational power improve 1-step and 2-step generative capabilities.
Notably, ECT achieves better 2-step generation performance than offe-of-the-arr CMs, while uti- lizing only 33% of the overall computational budget compared to iCT (Song and Dhariwal, 2023) (batch size 4096 800k). This significant efficiency is achieved through two-stage process: pre- training and tuning. While the pretraining stage utilizes the EDM2 pipeline, the tuning stage of ECT requires remarkably modest budget of 12.8M training images (batch size 128 x 100k), ranging from 0.60% to 1.91% % of the pretraining budget, depending on the model sizes.
Inference. Our scaling study also indicates a sweet spot for the inference of CMs. On both CIFAR- 10 and ImageNet 64x64, there are 2-step inferences of smaller models surpassing 1-step inferences larger models, e.g., 498M ECM-M against 1.1B ECM-XL. This calls for further studies of inference-optimal scaling and test-time compute scaling for visual generation.
8

---

Published as a conference paper at ICLR 2025
260
step 2-step ECM
step
ECM-S
2-step RID
240
220
ECM-M
FDoND12
ECM
200
180
160
2
2 2 2+
2 100
200 300
400
500
Training Compute
Model GFLOPs
Figure 4: (Left): Scaling up training compute yields the classic power-law between FD $ and training compute, with K = 263, a = -0.060 for 1-step inference, and K = 164, a = -0.028 for 2-step inference, (Right): Given the same batch size and iterations, scaling up model sizes and model FLOPs strongly correlates with FID 4 improvements on ImageNet 64 x 64. The diameter is proportional the model size.
4.3 SCALING TO RICH-RRSOLUTION DATASETS
We further extend our method to latent space and conducted experiments on ImageNet 512 x 512. Our ECM-M (498M parameters) achieves FID of 6.88 for 2-step generation using a modest budget of 12.8M training images, surpassing ADM-G (Dhariwal and Nichol, 2021)(559)) parameters, FID of 7.72 NFE=250x2) on the ImageNet 512 x 512 dataset. The strong performance on ImageNet 512x512 also indicates that the insights from our scaling analysis generalize beyond the low-resolution regime.
4.4 ABLATION STUDY
We perform ablation studies to analyze the impact of various tuning design choices. Using iCT's design choices as the baseline, we tune EDM2-S and present the results in Tab. 2. These results highlight the benefits of the continuous-ting training schedule, increased dropout, and weighting functions in enhancing ECMs' efficient generative performance.
RELATED WORK
Consistency Models. Consistency models (Song et al., 2023; Song and Dhariwal, 2023) are a new family of generative models designed for efficient generation with few model steps without the need for adversarial training. CMs do not rely on pretrained diffusion model (DM) to generate training targets but instead leverage unbiased score estimator. CMs have been extended to multi-step sampling (Kim et al., 2024: Wang et al., 2024b; Heek et al., 2024), latent space models (Luo et al., 2023), ControlNet (Xiao al., 2023). video (Wang et al., 2024c), and combined with additional adversarial losses (Kim et al., 2024; Kong et al., 2023). Despite their sampling efficiency, CMs are typically more challenging train and require significantly more compute resources compared to their diffusion counterparts. Our work substantially improves the training efficiency of CMs, reducing the cost future research and deployment on CMs.
While initializing CMs with pretrained diffusion models has also been considered in Song et al. (2023); Song and Dhariwal (2023), their training schedules for discrete-ting time models neither correspond the standard diffusion pretraining phase initially nor demonstrate the experimental advantages of this two- stage approach. While Continuous- CT considers initializing from pretrained diffusion models, it is schedule-fres jumping from the initialization to the continuous-time training. ECT, starting from the pretraining stage, progressively reduces the discretization error using the continuous -time schedule.
ECT is considered a training method rather than a distillation approach since distillation needs rumning inference of a frozen teacher model to generate training signals online offline. ECT requires neither

---

Published as a conference paper at ICLR 2025
of these. It utilizes single network (and maintains its Exponential Moving Average (EMA) copy) throughout the pretraining and tuning stage. The division between pretraining and tuning under the continuous-ting schedule is more conceptual than technical since they approximations at different degrees of consistency models.
Diffusion Distillation. Drawing inspiration from knowledge distillation (Hinton et al., 2015), distillation is the most widespread toining-based approach to accelerate the diffusion sampling procedure. diffusion distillation, a pretrained diffusion model (DM), which requires hundreds to thousands of model evaluations to generate samples, acts as teacher. A student model is trained to match the teacher model's s sample quality, enabling it to generate high-quality samples in a few steps.
There are two main lines of work in this area. The first category involves trajectory matching, where the student learns to match points on the teacher's sampling trajectory. Methods in this category include offline distillation (Luhman and Luhman, 2021; Geng et al., 2024; Zheng et al., 2023), which require an offline synthetic dataset generated by sampling from a pretrained DM to distill a teacher model into a stw-step student model; progressive distillation (Salimans and Ho, 2022; Meng et al., 2023), and TRACT (Berthelot et al., 2023), which require multiple training passes or offline datasets to achieve the same goal; and BOOT (Gu et al., 2023), Consistency Distillation (CD)(Song et al., 2023), and Imagine-Flash (Kohler et al., 2024), which minimize the difference between the student predictions at carefully selected points on the sampling trajectory.
CD is closely related to our method, as it leverages a teacher model to generate pairs of adjacent points and enforces the student predictions these points to map the initial data point. However, employs a fixed schedule derived from a specific sampler, which may introduce non-negligible dis- cretization errors in approximating the consistency condition. also limits the quality of consistency models to that of the pretrained diffusion model.
The second category minimizes the probabilistic divergence between data and model distributions, i.e, distribution matching (Poole et al., 2022; Wang et al., 2024a; Luo et al., 2024; Yin et al., 2023; Zhou et al., 2024a). These methods (Luo et al., 2024; Sauer et al., 2023; Yin et al., 2023; Nguyen and Tran, 2023; Kohler et al., 2024: Xu et al., 2023a; Lin et al., 2024; Zhou et al., 2024b) use score distillation or adversarial loss, to distill an expensive teacher model into an efficient student model. However, they can be challenging to train in a stable manner due to the alternating updating schemes from either adversarial or score distillation. Some of these methods such as DreamFusion (Poole et al., 2022) and ProlificDreamee (Wang et al., 2024a) used for 3D object generation.
A drawback of training- based approaches is that they need additional training procedures after pretraining to distill efficient student, which can be computationally intensive. For detailed discussion on the recent progress of diffusion distillation, we direct the readers to Dieleman (2024).
Fast Samplers for Diffusion Models. Fast samplers are usually training- and use advanced solvers to simulate the diffusion stochastic differential equation (SDE) or ordinary differential equation (ODE) to reduce the number of sampling steps. These methods reduce the discretization eтror during sampling by analytically solving a part of SDE or ODE (Lu et al., 2022a; Xue et al., 2024; Lu et al., 2022b), by using exponential integrators and higher order polynomials for better approximation of the solution (Zhang and Chen, 2022), using higher order numerical methods (Karras et al., 2022), using better approximation of noise levels during sampling (Kong and Ping, 2021), correcting predictions at each step of sampling (Zhao et al., 2024) and ensuring that the solution of the ODE lies on a desired manifold (Liu et al., 2022a). Another orthogonal strategy is the parallel sampling process (Pokle et al., 2022; Shih et al., 2024), solving fixed points of the entire trajectory. A drawback of these fast samplers is that the quality of samples drastically reduces as the number of sampling steps goes below a threshold such as 10 steps.
6 CONCLUSION
We present Easy Consistency Tuning (ECT), a simple yet efficient scheme for training consistency models. The resulting models, ECMs, unlock state-of-the-art few-step generative capabilities at minimal tuning cost and are able to benefit from scaling. Our code is available for future prototyping. studying. and deploying consistency models within the community.
10

---

Published as a conference paper at ICLR 2025
BROADER IMPACTS AND ETHICS STATEMENT
We propose Easy Consistency Tuning (ECT) that can efficiently train consistency models as state-of- the-art few-step generators, using only a small fraction of the computational requirements compared to current CMs training and diffusion distillation methods. We hope that ECT will democratize the creation of high-quality generative models, enabling artists and creators to produce content more efficiently. While this advancement can aid creative industries by reducing computational costs and speeding up workflows, also raises concerns about the potential misuse of generative models to produce misleading, fake, or biased content. We conduct experiments on academic benchmarks, whose resulting models are less likely to be misused. Further experiments are needed to better understand these consisteney model limitations and propose solutions to address them.
REPRODUCIBILITY STATEMENT
We provide extensive details experimental settings and hyperparameters to reproduce our experi- mental results in Appendix D. We have released our code ensure transparency and reproducibility of the results.
REFERENCES
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Information Processing Systems (NeurIPS), 33, 2020. 5,5
Neural
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations (ICLR), 2021a.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 1, 8
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021b. 1, 2, 6, 24
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. 2022 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2021.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713-4726, 2022.
Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 2022a.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Neural Information Processing Systems (NeurIPS), 34, 2021. 1, 8,
Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. Diffit: Diffusion vision transformers for image generation. arXiv preprint arXiv:2312.02139, 2023. 1
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning (ICML), 2021.
Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024.
11

---

Published as a conference paper at ICLR 2025
Andreas Blattmann, Tim Dockhom, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512) 2023. 1
Omer Bar-Tal Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401. 12945, 2024.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi. David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022b.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorcalistic video generation with diffusion models. arXiv preprint 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXi /4988, 2022. 1, 10
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu, Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024a. 1, 10
Kyungmin Lee, Kihyuk Sohn, and Jinwoo Shin. Dreamflow: High-quality text-to-3d generation by approximating probability flow. arXiv preprint arXiv:2403. 2024. 1
Yang Chen, Yingwei Pan, Haibo Yang. Ting Yao, and Tao Mei. Vp3d: Unleashing 2d visual prompt for text-to-3d generation. arXiv preprint arXiv:2403. 17001, 2024.
Sudarshan Babu, Richard Liu, Avery Zhou, Michael Maire, Greg Shakhnarovich, and Rana Hanocka. Hyperfields: Towards zero-shot generation nerfs from text. arXiv preprint arXiv:2310.17075 2023. 1
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurIPS, 2022. 2, 6, 7, 8, 10, 18, 20, 21, 22, 24
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. ArXiv, abs/2101.02388, 2021.
10, 24
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005/2, 2022. 1, 8, 10, 18, 19
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Neural Information Processing Systems (NeurIPS), 2024. 1, 8, 10, 23, 26
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Boot: Data-free distil-
lation of denoising diffusion models with bootstrapping. In ICML 2023 Workshop Structured Probabilistic Inference &{&} Generative Modeling, 2023. 1, 8, 10, 23, 26
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311. 2023. 1, 10
Zhengyang Geng, Ashwini Pokle, and Zico Kolter. One-step diffusion distillation via deep equilibrium models. Neural Information Processing Systems (NeurIPS), 36, 2024. 1,8, 10, 24
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman. and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311. 188288, 2023. 10, 23, 26
Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step -onaaee diffusion model with variational score distillation. arXiv preprint arXiv:2312.05239, 2023. 1, 10
12

---

Published as a conference paper at ICLR 2025
Tachong Moon, Moonseok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, and Juho Lee. Early exiting for accelerated inference in diffusion models. ICML 2023 Workshop on Structured Probabilistic Inference { Generative Modeling, 2023.
Shengkun Tang, Yaqing Wang, Caiwen Ding. Yi Liang, Yao Li, and Dongkuan Xu. Deediff: Dynamic uncertainty-awent early exiting for accelerating diffusion model generation. arXiv preprint arXiv:2309. (tt::2309./70774 77074, 2023.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprini arXiv:2106.00/33 2021.
10
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint 000::22006.00927, 2022a. 1, 10
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator.
arXiv preprint arXiv:2204. /3902, 2022. 1, 10
Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around steps. arXiv preprint arXiv:2312.00094, 2023. 1
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling diffusion models. Neural Information Processing Systems (NeurIPS), 2024. 1, 10
Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar. T-stitch: Accelcrating sampling in pre-trained diffussin models with trajectory stitching. arXiv preprint 4rXlv:2402.14167, 2024.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01 01...01469, 469, 2023. 2, 3, 4, 6, 8, 9, 10, 23, 24, 26
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. /4189, 2023. 2, 3, 7, 8, 9, 18, 20, 22, 24, 25, 26
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009. 2,
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https: // //www. cs toronto. edw/eduurrrreeee ci far html. 2, 6,7
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018. 3, 6, 18, 21
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. 2012. 6, 20
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Neural Information Processing Systems (NeurIPS), 30, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXi::2304.07193. 2023. 6
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze. Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza- Ganiia Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 6, 22, 25, 26
13

---

Published as a conference paper at ICLR 2025
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm--olver-v3: Improved diffusion ode solver with empirical model statistics. Neural Information Processing Systems (NeurIPS), 2024. 6, 8
Tero Karras, Miik Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2024. 8, 19, 22, 23, 24, 25, 26
David Berthelot, Amaud Autef, Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.04245 2023. 8, 10
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.026666 2023.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar, Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning (ICML). PMLR, 2023. 8, 10, 24
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probabil- ity flow ODE trajectory of diffusion. In International Conference on Learning Representations (ICLR), 2024.
Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency model. arXiv preprint ar 18847 2024b. 9
Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint 2rXiv:2403.06807, 2024. 9
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023. 9
Jie Xiao, Kai Zhu, Han Zhang, Zhiheng Liu, Yujun Shen, Yu Liu, Xueyang Fu, and Zheng-Jun Zha. Cem: Adding conditional controls to text-to-image consistency models. arXiv preprint arXiv:2312.06997, 2023. 9
Fu-Yur Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelem: Accelerating the animation of personalized diffusion models and adapters with decoupled consistency learning. arXiv preprint arXiv:2402.08 00769, 2024c. 9
Fei Kong, Jinhao Duan, Lichao Sun, Hao Cheng, Renjing Xu, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Act: Adversarial consistency models. arXiv preprint arXiv:2311. 1. 14097, 2023. 9
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint Ttl:::503.025.11 2015. 10
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023. 10
Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Vajda, and Ali Thabet. Imagine flash: Accelerating emu diffusion models with backward distillation. arXiv preprint arXhv:2405.0522. 2024. 10
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024a. 10, 23, 26
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:2311.09255 2023a. 10
14

---

Published as a conference paper at ICLR 2025
Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.13992) 2024. 10
Mingyuan Zhou, Zhendong Wang. Huangjie Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step text-to-image generation. ArXiv 2406.01561, 2024b. URL https: //axiv. org/ alss 2406 6. 01561. 10
Sander Dieleman. The paradox of diffusion distillation, 2024. URL https:t/tts/aaaae ai/ 2002/02/28/par//
html. 10
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:2211.01095 2022b. 10
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. Neural Information Processing Systems (NeurIPS), 2024. 10
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. arXiv preprint arXiv:2202.09775. 2022a. 10
Ashwini Pokle, Zhengyang Geng, and Zico Kolter. Deep equilibrium approaches to diffusion models. Neural Information Processing Systems (NeurIPS), 2022. 10
Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Neural Information Processing Systems (NeurIPS), 2024. 10
Pierre Charbonnier, Laure Blanc-Féraud, Gilles Aubert, and Michel Barlaud. Deterministic edge- preserving regularization in computed imaging. IEEE Transactions image processing. 6(2): 298-311, 1997. 18
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Neural Information Processing Systems (NeurlPS), 2024. 18
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-sur weighting strategy. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7441-7451, 2023. 18
Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Shippole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. arXiv preprint arXiv:2401. /605, 2024. 18
Yaron Lipman, Ricky TQ Chen, Heli Ben Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 270.022210 2022. 19, 20
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03m4 2000200090000, 2022b. 19, 20
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser. and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403. /2015, 2024. 20
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 18Xiv:1809.11096 2018. 21
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Neural Information Processing Systems (NeurIPS), 2020. 21
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, 2022. 21
Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi Jaakkola. Pfgm++: Unlocking the potential of sonysics-isspinee generative models. In International Confer- ence on Machine Learning, pages (8566-3859) PMLR, 2023b. 21
15

---

Published as a conference paper at ICLR 2025
Junlong Lyu, Zhitang Chen, and Shoubo Feng. Convergence guarantee for consistency models. arXiv preprint arXiv:2308.11444 11449, 2023. 22
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning and beyond. arXiv preprint 2019. 24
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /ee...122.... 69800, 2014. 24, 26
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021. 25
16

---

Published as a conference paper at ICLR 2025
A MOTIVATIONS BEHIND DESIGN CHOICES IN ECT
In this section, expand upon motivation behind the design decisions for the mapping function, metric, and weighting function used for ECT.
Mapping Function. We first assume that At is approximately proportional to t. Let 0 < c be this constant of proportionality, then write:
-
As training progresses, the mapping function should gradually shrink At + 0. However, the above parameterization does not achieve this. An alternative parameterization is to decrease At exponentially. We assume the ratio between and & can be written as:
1
(17)
q
where q > 1. a= liters/d]. and d is hyperparameter controlling how quickly At - dt. At the 0, which falls back DMs. Since we can initialize
beginning of training. - 1/2 0 from the diffusion pretraining, this stage can be skipped by setting a = stter/d]. As training progresses (iters 1). + leads to At - dt.
Finally, we adjust the mapping function to balance the prediction difficulties across different noise levels:
1 9r
n(t)
9 itors d (t)).
(18)
t
For nt)), we choose n(t) + k b(b,,)) 1 14 using the sigmoid function a. Since > we also clamp to satisfy this constraint after the adjustment.
Plot for
Plot a-2
1.0
a=1
0.8
2.0
0.5
3.6
0.1
3.4
0.2
a=2
0.8
Plot for a
Plot for a=4
am3
a
10
t
Figure 5: Visualization of r/2 during training. At > dt when r/a + 1.
The intuition behind this mapping function is that the relative difficulty of predicting f(x,) from X; vary significantly across different noise levels t when using linear mapping between and
Consider r/- = 0.9. At small values of t, x; and x, are close, making the alignment of fx,) with fxx,) relatively easy. In contrast, larger t, where x; and x, relatively far apart, the distance between the predictions f(x,) and f fxx,) can be substantial. This leads to imbalanced gradient flows across different noise levels, impeding the training dynamics.
17

---

Published as a conference paper at ICLR 2025
Therefore, downscale r/6 when t is near 0 through the mapping function, balancing the gradient flow across varying noise levels. This prevents the gradient at any noise level from being too small or too large relative to other noise levels, thereby controlling the variance of the gradients.
We direct the reader to Appendix B for details of how to set q [ites/dd.
Choice of Metric. iCT uses pseudo-Huber metric (Charbonnier et al., 1997) to mitigate the perceptual bias caused by the LPIPS metric (Zhang al., 2018),
L(x,y) - VII yli E>
(19)
This metric indeed improves the performance of CMs over the classic squared Le loss. When taking careful look as this metric, we reveal that one of the reasons for this improvement is that this metric is more robust to the outliers compared to the L2 metric due to its adaptive per-sample scaling the gradients. Let A x y, then the differential of the pseudo-Hube Huber metrie can be written
(
dL
(20)
VIATE 2
weighting term
differential of squared L. loss
where we have decomposed the differential of pseudo-Huber loss into adaptive weighting term and the differential of the squared L2 loss. Therefore, we retain the squared L2 metric used in DMs, and explore varying adaptive weighting terms which we explore in detail in Appendix B.
Distinction between the training schedules of ECT and iCT. As noted in Sec. 2, iCT (Song and Dhariwal, 2023) employs a discrete- time curriculum given by Eq. (10). This curriculum divides the noise horizon [0, 7 into N smaller consecutive subintervals to apply the consistency loss, characterized by non-overlapping segments [4,, 1), and gradually increases the number intervals N = 10 1280. However, the "boundary" condition of this schedule is start with the number of intervals to 1, learning a model solely mapping samples at noise levels Tmax to the clean data X largely distinet from the classic diffusion models training. We instead investigate a continuous-time schedule whose "boundary" condition yields diffusion pretraining, i.e., constructing training pairs of 0 for all t the beginning.
B EXPLORING DESIGN SPACE & SCALING OF CONSISTENCY MODELS
Due to ECT's efficiency, examine the weighting function, training schedule, and regularization for CMs.
explore the design space of CMs a minimal cost. We specifically
Our most significant finding is that controlling gradient variances and balancing the gradients across different noise levels are fundamental to CMs' training dynamics. Leveraging the deep connection between CMs and DMs, we also improve the diffusion pretraining and the full pretraining+tuning pipeline using our findings.
Weighting Function. Forward processes with different noise schedules and model parameteriza- tions can be translated into each other at the cost of varying weighting functions (Kingma and Gao, 2024). From our experiments on a wide range of weighting schemes, we learn three key lessons.
(1) There is no free lunch for weighting function, i.e., there is likely no universal timestep weighting (t) that can outperform all other candidates on different datasets, models, and target metrics for both 1-step and 2-step generation.
We refer these results to Tab. 3, including SNR(1) = 1/2, SNR(I) + +1= 1/2² + (Salimans and Ho, 2022), EDM weighting SNR(() + 1/a3.ta = 1/22 + 1/o2... (Karras et al., 2022), and Soft-Min- SNR weighting 3/(1++ 2/m))) (Hang et al., 2023; Crowson et al., 2024), where SNR(!) = 1/22 the signal-to-noise ratio in our setup.
On CIFAR-1O, the weighting (t) = 1/(t-)) from the discretization of consistency condition in Eq. (12) achieves the best 1-step FID, while the square root of SNR(t), w(t) = //SN/E /SNR(t) = 1/t, produces the best 2D On ImageNet 64 64,, considering that have already had the adaptive
18

---

Published as a conference paper at ICLR 2025
Table 3: Performance of ECMs trained with various weighting functions on ImageNet 64x64. We enable the adaptive weighting w(A) +/(
(tt)
1-step FID|
2-step FID
1
5.39
3,48
1/2
17.79
3.24
(t-r)
9.28
3.22
//++ 1/aauso
5.68
3.44
1/22
190.80
20.65
1/2 +
6.78
3.12
1/22 + 1/0txe 5.51
3.18
163.01
13.33
Table 4: Performance of HCMs trained with varying adaptive weightings on ImageNet 64 64.
((t)
w A)
1-step FID
2-step FID
1/22
1/0aeaa
6.51
3.28
1/12
Youna
1/(1A/1+c))
6,29
3.25
1/22 daan
1/111
5.51
3.18
3/(1111++0")5
5.39
3.48
weighting w ()), the uniform weighting (it) = 1 can demonstrate the best 1-step FID when tuning from EDM2 (Karras al., 2024). In contrast to 1-step FIDs, wider range of timestep weighting (t) produces close 2-step FIDs for HCMs.
When starting on new dataset with no prior information, ((t) SNR(t) + a generally strong choice as the default timestep weighting of data prediction models (x-pred). In this situation, this weighting function corresponds to using v-pred (Salimans and Ho, 2022) or flow matching (Lipman et al., 2022; Liu et al., 2022b) as model parameterization when π =
(2) The adaptive weighting w(A) achieves better results by controlling gradient variance. The adaptive weighting w(A) on per-sample basis shows uniform improvements on both CIFAR-10 and ImageNet 64x64. See Tab. for the ablation study.
Beyond ECT, we further investigate the role of adaptive weighting w A) in pretraining on toy Swiss roll dataset using the parameterization and forward process of flow matching (Lipman et al., 2022) and an MIPP network.
Consider the objective function w(A) (x:) (x1 x,) Il.. where x; (1 -t) Xo + X1. t &~ Uniform x1 ~ ~00, I), and the adaptive weighting
w A
[
where p 0 corresponds to no adaptive weighting. We set c² 10 and control the strength of gradient normalization by varying p from to 1.
As we increase the strength of adaptive weighting, flow models become easier to sample from in few steps. Surprisingly, even p = demonstrates strong few-step sampling results when pretraining the flow model. See Fig. for visualization.
Mapping Function. We compare the constant mapping function with ntt) = 1 in Eq. (17) and mapping function equipped with the sigmoid ntt) in Eq. (18). We use k: = 8 and b = 1 for all the experiments, which transfers well from CIFAR-10 10 to ImageNet 64x64 64 and serves our experiments. Though b = 2 can further improve the 1-step FIDs on ImageNet 64 x 64, noticed post hoc, don't rerun experiments.
baseline in
19

---

Published as a conference paper at ICLR 2025
4-step Sample
10-Step Sample
20-Step Sample
100-Step Semple
Real Data
Figure 6: Influence of adaptive weighting w(A) 1/(1|A/ 3+) on pretraining using varying p.
On CIFAR-10, the constant mapping function with n1(1 = achieves 1-step FID of 4.06 at 200k iterations, worse than the 1-step FID of 3.86 by nt)) = + It Under our forward process (X; = Xu + t. c) and model parameterization (EDM (Karras et al., 2022)), the constant mapping function incurs training instability ImageNet 64x64, likely due to the imbalanced gradient flow.
&
The role of the mapping function, regarding training, is to balance the difficulty of learning consistency condition across different noise levels, avoiding trivial consistency loss near f O. For model parameterizations and forward processes different from ours, for example, flow matching (Lipman al., 2022; Liu al., 2022b; Sauer al., 2024), advise readers start from the constant mapping function due to its simplicity.
Dropout. line with (Song and Dhariwal, 2023), we find that CMs benefit significantly from dropout (Hinton et al., 2012). On CIFAR-10, we apply dropout of 0.20 for models evaluated on FID and dropout 0.30 for models evaluated on FDDINOV2.
On ImageNet 64x64, we note that ECT benefits from surprisingly high dropout rate. When increasing the dropout rate from 0.10 to 0.40, the 2-step FID decreases from 4.53 3.24. Increasing the dropout rate further the 2-step FID starts to deteriorate. In general, we optimize our model configurations for 2-step generation and choose the dropout rate of 0.40 for ECM-S.
be helpful for 1-step FID under certain timestep weighting w(t). but
Finally, we note that the dropout rate tuned at a given weighting function w(t) transfers well to the other weighting functions, thereby reducing the overall cost of hyperparameter tuning. On ImageNet 64x64 64, the dropout rate can even transfer to different model sizes. We apply a dropout rate of 0.50 for all the model sizes of ECM-M/L/XI_.
20

---

Published as a conference paper at ICLR 2025
Table 5: Generative performance
class-conditional CIFAR-10.
Method
FDuiNov2k
NFE4
GANs
BigGAN (Brock et al., 2018)
326.66
I
StyleGAN2-ADA (Karras et al., 2020)
305.92
I
StyleGAN-XL (Sauer al., 2022)
204.60
Diffusion Models
EDM (Karras et al., 2022)
145.20
35
PFGM++ (Xu al., 2023b)
141.65
35
ECT
ECM (ECT Pretrained)
121.05
35
ECM (Tuned)
198.51
1
ECM (Tuned)
128.63 2
Shrinking At - O. In the mapping function discussed in Sec. 3.2 and Appendix A, we use the hyperparameter d to control the magnitude of q, thereby determining the overall rate of shrinking At - 0, given by (1 - 1/9 fiters/eee d In practice, we set q = 2 and total total_iters// /8 for CIFAR- 10 experiments, and q 4 and d total potal_iters//4 /4 for ImageNet 64 experiments, achieving r/c = 0.99 at the end of training.
Compared with no shrinkage of At, where At dt throughout, we find that shrinking At results in improved performance for ECMs. For example, on CIFAR-10, starting ECT directly with At x dt by setting q = 256 (corresponding 7/c 0.99) leads to quick improvements in sample quality initially but slower convergence later on. The 1-step FID drops from 3.60 to 3.86 using the same 400k training iterations compared to gradually shrinking At - O. On ImageNet 64 64, At Rs dł with y 256 from the beginning results in training divergence, as the gradient flow is highly imbalanced across noise levels, even when initializing from pretrained diffusion models.
This observation suggests that ECT's schedule should be adjusted according to the compute budget. At small compute budgets, as long as training stability permits, directly approximating the differential consistency condition through a small At x dt leads to fast sample quality improvements. For normal to rich compute budgets, shrinking At > generally improves the final sample quality, which is the recommended practice.
Using this feature of ECT, we demonstrate its efficiency by training ECMs to surpass previous Consistency Distillation, which took hundreds of GPU hours, using hour single A100 GPU.
Training Generative Models in GPU Hour. Deep generative models are typically computation- ally expensive to train. Unlike training classifier on CIFAR-10, which usually completes within one GPU hour, leading generative models on CIFAR-10 as of 2024 require days to week to train on GPUs. Even distillation from pretrained diffusion models can take over day on 8 GPUs or even more, cquivalently hundreds of GPU hours.
To demonstrate the efficiency of ECT and facilitate future studies, we implemented a fast prototyping setting designed to yield results within one hour on a single GPU. This configuration uses a fixed mapping function (corresponding to 0.99), which allows for
At z dt by setting 9 256 in S quick approximation of the differential consistency condition. Through $000 gradient descent steps batch size of 128, within hour single A 100 40GB GPU, ECT achieves 2-step FID of 2.73, outperforming Consistency Distillation (2-step FID of 2.93) trained with 800k iters at batch size 512 and LPIPS (Zhang al., 2018) metric.
Importance of Pretraining We conducted a controlled experiment, training CMs from scratch following iCT best practices and tuning EDM using both iCT and ECT schedules. The combined pretraining and fine-tuning cost for ECT is 50% + 6.25% of iCT trained from scratch. Results are presented in Tab. 6.
21

---

Published as a conference paper at ICLR 2025
This performance gap is particularly noticeable when evaluating models using modern metries like DDD (Stein et al., 2024). FDDINOv2 uses the representation space of the DINOv2-L model to compute distributional distance, which has been shown to better align with human evaluations.
Table 6: Impact of pretraining on model performance
Model
FDDNN
iCT from scratch Pretraining iCT tuning
242.30
200.31
Pretraining + ECT tuning EDM
190.13
168.16
When scaling up the pretraining+tuning cost to match the overall cost of iCT in daas-cooniitinal settings, ECM achieves a FDDINNvz of 152.21, significantly outperforming the iCT model trained from scratch 1205.11). For context, StyleGAN-XL achieves an FDDINO22 204.60.
Improving Pretraining using Findings in Tuning. The exploration of the design space through the tuning stage proxy led to a question: Can the insights gained during tuning be applied to improve the pretraining stage and, consequently, the entire pretraining+tuning pipeline for CMs? The results of our experiments confirmed this hypothesis.
For the largest At = t, ECT falls back to diffusion pretraining with r = 0 and thus fa(x.) = Xg. We pretrain EDM (Karras al., 2022) the CIFAR- 10 dataset using the findings in ECT. Instead of using EDM weighting, SNR(() + 1/raacn we enable the adaptive weighting w/ (A)) with = 1/2 and smoothing factor с and timestep weighting (() 1/c.
Compared with the EDM baseline, the recipe from ECT brings a convergence acceleration over 2x regarding FDpINo22, matching EDM's final performance using less than half of the pretraining budget and largely outperforming it at the full pretraining budget.
EDM pretrained by ECT achieves FDDINO,2 of 150.39 for unconditional generation and 121.05 for class-conditional generation, considerably better than the EDM baseline's 's FDDINOV2 of 168. 17 for unconditional generation and 145.20 for classcooniiiional generation, when using the same pretraining budget and inference steps (NFE=35).
Influence of Pretraining Quality. Using ECT pretrained models (FDpINov2 of 121.05) and original EDM (Karras et al., 2022) (FDDINoV2 of 145.20), we investigate the influence of pretraining quality on consistency tuming and resulting ECMs. Our experiments confirm that better pretraining leads to easier consistency tuning and faster convergence. In the same budget of 204.8M images, tuning from ECT pretrained models achieves FDDINOv2 of 128.63, better than FDDINOV2 of 152.21 from EDM.
ECM from the ECT pretraining surpasses SoTA GANs in 1 sampling step and advanced DMs in sampling steps, only slightly falling behind our pretrained models and setting up SoTA for the modern metric FDDINOV2. See Tab. for details.
On ImageNet 64x64, ECM-M, initialized from EDM2-M (Karras al., 2024), deviates from the power law scaling and achieves better generative performance than the log-linear trend. (See Fig. 4, Right). We speculate that is due to higher pretraining budget, which EDM2-M was pretrained by 2x training images compared with other model sizes (S/L/XL).
Differences between 1-step and 2-step Generation. Our empirical results suggest that the training recipe for the best 1-step generative models can differ from the best few-step generative models in many aspects, including weighting function, dropout rate, and EMA rate/length. Fig. 7 (Left) shows an example of how FIDs from different numbers function evaluations (NFEs) at inference vary with dropout rates.
In our setups, starting from a proper model size, the improvements from 2-step sampling seem larger than doubling the model size but keeping 1-step sampling. In the prior works, iCT (Song and Dhariwal, 2023) employs 2x deeper model, but the 1 1-step generative performance can be inferior to the 2-step results from ECT. This finding is consistent with recent theoretical analysis (Lyyu et al.,
22

---

Published as a conference paper at ICLR 2025
10
500
StStep 2 Step
8
2-step
200
6
F10 (1ag Scalel
200
4
20
3
0.05
0.1
0.15
0.2
0.25
0.3
0.35
25
50
75
100
125
175
200
Dropout
Iterations (X 180
Figure 7: (Left): Relationship between the dropout and FIDs for models trained on CIFAR- 10 with varying numbers of function evaluations (NFE) inference. (Right): Evolution of model performance w.r.t. training iterations.
2023), which indicates tighter bound on the sample quality for the 2-step generation compared to the 1-step generation.
Pareto Frontier & Scaling Law. The Pareto Frontier reveals seemingly power law scaling behavior. Training configurations not optimized for the current compute budget, i.e, not on the Pareto Frontier, deviate from this scaling. Simply scaling up the training compute without adjusting other parameters may result suboptimal performance. In our compute scaling experiments, we increased the batch size and enabled the smoothing factor in the adaptive weighting to maintain this trend.
Performance Evolution along Training. We visualize the performance evaluation along the training process in Fig. (Right). When the approximation errors of the consistency condition are reduced, the sample quality improves correspondingly.
C EXTENSION TO CONSISTENCY DISTILLATION.
Easy Consistency Distillation Consistency Distillation (CD) (Song et al., 2023) uses a fixed discrete schedule derived from specific sampler, such as the EDM sampler. This approach inherently limits At + dl, therefore causing non-trivial discretization error approximating the differential consistency condition. Building upon ECT, we extend our continuous schedule to Consistency Distillation, which we term Easy Consistency Distillation (ECD).
On ImageNet 64 x4, we implement Consistency Distillation as the baseline using pretrained EDM2- S (Karras al., 2024) with weighting functions, noise distribution, and dropout. The results, shown in Tab. 7, demonstrate that continuous time distillation improves upon the standard CD approach.
Given pretrained DMs, ECD typically demonstrates performance advantage over ECT at smaller batch sizes (e.g., 128), primarily due to the variance reduction in approximating the score function:
Xx, log (xx:) E[V logp(x;/xo)|x,
This advantage stems from the teacher models, which provide reduced variance estimate compared to the Monte Carlo estimation used ECT. This gap tends to narrow as we scale up to larger model sizes and training budget. Scaling compensates for the limited training budget and variances, closing the gap between the two.
Consequently, while ECD may outperform ECT in resource-constrained scenarios or with smaller models, the distinction becomes less pronounced as we move to larger scales. This observation underscores the scaling factor (computational resources and model size) when choosing between ECT and ECD for given application.
Data-Free ECD. Inspired by the recent progress on data-free distillation (Luo et al., 2024; Gu al., 2023; Yin et al., 2023; Zhou al., 2024a), explore data- variant of Easy Consistency Distillation (ECD). Rather than sampling data points xn from a given dataset D, we generate synthetic data points directly from the CM itself the fly, Xo (sg(o)(E', T), where e' p(e), T is the
23

---

Published as a conference paper at ICLR 2025
Table 7: Easy Consistency Distillation (ECD) ImageNet 64x64 using the same budget of 12.8M training images (batch size 128 and 100k iterations) as ECT in Tab. 1.
Methods
1-step FID
2-step FID
CD-S
8.18
3.71
ECD-S
3.33
2.10
ECD-M
2.78
1.92
ECD-XL
2.54
1.77
maximum noise level for sampling, followed by the same ECD training loss over the self synthetic data. This eliminates the need for distillation dataset the construction of synthetic data from the teacher model (Luhman and Luhman, 2021; Geng et al., 2024; Zheng et al., 2023).
In ImageNet 64x64, this data-free ECD achieves 1-step FID of 4.38 and 2-step FID of 2.77, comparable to the data-dependent ECD and ECT schemes in the same budget. It suggests that data-free ECD can be competitive alternative scenarios where access to large datasets limited or unavailable. We summarize both ECD and its data-free variant in Alg. 2.
Algorithm 2 Easy Consistency Distillation (ECD)
Dataset D (for ECD), pretrained diffusion model o, mapping function p(r Iters). weighting function wt).. Init: 0 <- o, Iters = 0.
Input:
repeat
if ECD then
Sample Xo ~
else if Data-Firee ECD then
Sample ~ p(c)
Compute Xg fag(0) (e',
sg is stop- stop-gradient operator
end
Sample p(c), p(t), r p(r t, Iters)
=x Compute x; = Xo + E, At =t=
Compute x, Solver (x,,
fo)
L(0) = w(t) dd(o(x))) l
- 0 - (0))
Tters Iters +
until At + 0 return 0
> ECD
D EXPERIMENTAL DETAILS
Model Setup. For both unconditional and class-conditional CIFAR-IO experiments, we initial
ECMs from the pretrained EDM (Karras et al., 2022) of DDPM++ architecture (Song al., 2021b).
For classcconditiona ImageNet 64 x4x6e 64 experiments, we initial ECM-S/M/L/XL. ranging from 280M to 1.1B, from the pretrained EDM2 (Karras et al., 2024). Detailed model configurations presented in Tab. 8.
We follow (Karras al., 2022; Song al., 2023) and set Caip (t) (i +o and Comr()) te /o+43 an where oi is the variance of (normalized) data, and set to 0.5 for both CIFAR-10 and ImageNet 64x64.
Computational Cost. ECT is computationally efficient. On ImageNet 64x64, the tuning stage of ECT requiring only 0.39% of the iCT (Song and Dhariwal, 2023) training budget, and 0.60% to 1.91% of the EDM2 (Karras al., 2024) pretraining budget depending on the model sizes. The exact computational resources required to train each individual model are shown in Tab. 8.
Training Details. We use RAdam (Liu al.. 2019) optimizer for experiments CIFAR-10 and Adam (Kingma and Ba, 2014) optimizer for experiments on ImageNet 64x64. We set the 3 to (0.9, 0.999) for CIFAR-10 and (0.9, 0.99) for ImageNet 64 x44. All the hyperparameters indicated in
24

---

Published as a conference paper at ICLR 2025
Table 8: Model Configurations and Training Details for unconditional and class-conditional ECMs on CIFAR-10, and ECM-S/M///XL on ImageNet 64 x6.
Model Setups
Uncond CIFAR-10
Cls-Cond CIFAR-10
ImageNet 64x64
RCM-S
ECM-M
FCM-L.
BCM-XT.
Model Channels
128
128
192
256
320
381
Model capacity (Mparams)
66.7
55.7
250.2
497.8
777.5
1119.3
Model complexity (GFLLOPs)
21.3
21.3
101.9
180.8
252.2
405.9
Training Details
Training Duration (Mimg)
12.8
12.8
12.8
12.8
12.8
12.8
Minibatch
Iterations
100k
100k
100k
100k
100k
100k
Dropout probability
20%
211%
400%
50%%
500%
Dropout feature resolution
S 16 x 16
16 x 16
16 16
S 16 x 16
Optimizer
RAdam
RAdam
Adam
Adam
Adam
Adam
Learning rate max (Ovef)
0.0001
0.0001
0.0010
0.0003
0.0008
0.0007
Learning rate decay
2000
2000
2000
2000
EMA beta
0.9890
0.9800
Training Cost
Number of GPUs
1 A6000
4
8 H100
8
GPU types Training time (hours)
H100
21
21
8.5
8.5
12
15
Generative Performance
1-step HID
4.54
3.81
5,51
3.67
3.55
3.35
2-step FID
2.20
2.02
3.18
2.35
1.96
ECT Details
Regular Weighting (i(())
4/c-+)
V/c-r
1/2 + 1/22..
1/2 1/22.
1/e 1/42.
4/2 + 1/22.
Adaptive Weighting (v((a))
Adaptive Weighting Smoothing (c)
0.0
0.0
0.06
0.06
0.06
0.06
Noise distibution mean (Paca))
-1.1
-111
-0.8
-0.8
-0.8
-0.8
Noise distribution sid (Pa))
2.0
2.0
1.6
1.6
1.6
Tab. 8. We do any learning rate decay, weight decay, warmup CIFAR-10. We follow EDM2 (Karras et al., 2024) to apply an inverse square root learning rate decay schedule on ImageNet 64x64.
On CIFAR-10, we employ the traditional Exponential Moving Average (EMA). To better understand the influence of the EMA rate, we track three Power function EMA (Karras et al., 2024) models on ImageNet 64 64, using EMA lengths of 0.01, 0.05, and 0.10. The multiple EMA models introduce no visible cost to the training speed. Considering our training budget much smaller than the diffusion pretraining stage, we didn't perform Post-Hoc EMA search as in EDM2 (Karras et al., 2024).
Experiments for ECT are organized in non-adversarial setup to better focus and understand CMs and avoid inflated FID (Stein et al., 2024). We conducted ECT using full parameter tuning this work, even for models over 1B parameters. Investigating the potential of Parameter Efficient Fine Tuning (PEFT) (Hu et al., 2021) can further reduce the cost ECT to democratize efficient generative models, which is left for future research.
We train multiple ECMs with different choices batch sizes and training iterations. By default, ECT utilizes batch size of 1 128 and 100k iterations, leading training budget of 12.8M ImageNet 64x64. We have individually indicated other training budgets alongside the relevant experiments, wherever applicable.
Sampling Details. We apply stochastic sampling for 2-step generation. For 2-step sampling, follow (Song and Dhariwal, 2023) and set the intermediate = 0.821 for CIFAR- 10, and = 1.526 for ImageNet 64.
Intriguingly, these sampling schedules, originally developed for iCT, also perform well with our ECMs. This effectiveness across different CMs and training methods likely links to the inherent characteristics of the datasets and the forward process. Developing scientific approach to determine optimal intermediate sampling schedules for CMs remains an open research problem.
25

---

Published as a conference paper at ICLR 2025
Evaluation Metrics. For both CIFAR-10 10 and ImageNet 64 64x64, FID and FDDINOV2 are computed using 50k images sampled from ECMs. As suggested by recent works (Stein et al., 2024; Karras et al., 2024), FDDNNWw2 aligns better with human evaluation. We use dgm-eval' to calculate FDDINOv2 (Stein et al., 2024) to ensure align with previous practice. Performance results for prior methods in Tab. I are reported from previous works (Song et al., 2023; Song and Dhariwal, 2023; Karras et al., 2024).
Visualization Setups. Image samples in Fig. 3 are from class b (971), class flamings (130), class golden retriever (207), class space shuttle (812), classs Siberi abberian husky (250), classs ice cream (928), class oscilloscope (688), class 1lama (355), class tiger shark (3).
Each triplet (left-to-right) includes from 2-step samples from ECM-S trained with 12.8M images, ECM-S trained with 102.4M images, and ECM-XL trained with 102.4M images.
GPU Hour Prototyping Settings. This configuration uses fixed At dt by setting q 256 in our mapping function (corresponding to _ 0.99) and an EMA rate of 0.9993 for the model parameters. Using these settings, we run 8000 gradient descent steps with a batch size of 128 on a single A100 40GB GPU.
Scaling of Training Compute. For the results on scaling laws for training compute on CIFAR-10 shown in Fig. 4 (Left), we train 6 class-conditional ECMs, each with varying batch size and number of training iterations. All ECMs in this experiment are initialized from the pretrained class-conditional EDM.
The minimal training compute at 2 scale corresponds to a total budget of 12.8M training images. The largest training compute at 2 scale utilizes total budget of 409.6M training images, at 2x EDM pretraining budget.
The first two points 20 and 2 on Fig. 4 (Left) batch size of 128 for 100k and 200k iterations, respectively. The third point of 2 corresponds to ECM trained with batch sizes of 256 for 200k iterations. The final three points of 23, 2 and 2 correspond to ECM trained with batch size of 512 for 200k, 400k, and 800k iterations, respectively, with the smoothing factor = 0.03 enabled in the adaptive weighting w(A), We use w(t) 1/1 as the timestep weighting function to train all these models as this wt) achieves good performance on FDDINOV2-
Scaling of Model Size and Model FLOPs. We include details of model capacity as well as FLOPs
in Tab. 8 to replicate this plot on ImageNet 64x64.
On ImageNet 64 64, we scale up the training budgets of ECM-S and ECM-XL from 12.8M (batch size of 128 and 100k iterations) to 102.4M (batch size of 1024 and 100k iterations). We empirically find that scaling the base learning rate by vn works well when scaling the batch size by a factor when using Adam (Kingma and Ba, 2014) optimizer.
E LIMITATIONS
One of the major limitations of ECT is that it requires dataset to tune DMs to CMs. Recent works developed data-free approaches (Luo et al., 2024; Gu et al., 2023; Yin et al., 2023; Zhou et al., 2024a) for diffusion distillation. The distinction between ECT and data-free methods is that ECT learns the consistency condition on a given dataset through the self teacher, while data-free methods transfer knowledge from frozen diffusion teacher. This feature of ECT can be potential limitation since the training data of bespoke models are unavailable to the public. However, we hold an optimistic view on tuning CMs using datasets different from pretraining. Synthetic data, data composition, and data scaling for consistency models will be valuable research directions.
26

---

Published as a conference paper at ICLR 2025
F QUALITATIVE RESULTS
We provide some randomly generated 2-step samples from ECMs trained on CIFAR-10 and ImageNet- 64 64 in Fig. and Fig. 9, respectively.
Figure 8: 2-step samples from class-conditional ECM trained on CIFAR-10. Each row corresponds to different class.
27

---

Published as a conference paper at ICLR 2025
Figure 9: 2-step samples from class-conditional ECM-XL trained on ImageNet 64x64, Each row corresponds to different class.
28


Reviews:
[{"review_id": "BRvRQiKQwZ", "reviewer": "Reviewer_FML7", "summary": "In this paper, the athors made the point that diffusion models can be viewed as a special case of CMs. Based on it, they fine-tuned a consistency model starting from a pretrained diffusion model and progressively approximate the full consistency condition to stronger\ndegrees over the training process. The experiments verified its efficiency.", "questions": "1. Pseudo-Huber metric is adopted in the model. Is there any other alternative?\n2. In the experiments, it is better to provide more metrics other than FID only.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The approach is very efficent as they showed. It could be used to greatly mprove efficiency and\nperformance of CMs at a large scale.", "weaknesses": "The main motivation of the paper is straightforward. It is hard for the reader to fullly trust their obverstaion that  diffusion models can be viewed as a special case of CMs in practice. The data sets and metircs on the images generation are limited. More extensive experiments or analysis should be conducted to justify their claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RcsWQOIIwV", "reviewer": "Reviewer_LAzX", "summary": "The authors propose easy consistency tuning (ECT), a method to train consistency models (CMs) using less compute than competing methods by initializing them from pre-trained diffusion models (DMs) and then fine-tuning them for few-step generation. The fine-tuning step enables faster convergence compared to training CMs from scratch (consistency training, CT) and avoids accumulating errors from a frozen teacher DM compared to consistency distillation (CD). Training CMs with ECT achieves state-of-the-art performance on both CIFAR-10 and ImageNet 64×64. The authors additionally explore scaling laws in the fine-tuning step, demonstrating performance gains with increased training compute.", "questions": "In addition to my questions already mentioned in the paper’s weaknesses, I have a few other minor questions and clarifications:\n- I do not count this as a weakness since CIFAR-10 and ImageNet 64×64 are standard benchmarks, but have the authors also considered evaluating their method on higher dimensional datasets (e.g., ImageNet 512x512, LAION) and/or latent diffusion models (e.g., Stable Diffusion)? It would certainly be interesting and further strengthen results to position ECT as the de facto standard training regime for CMs more generally.\n- Why do the authors use FD-DINOv2 for Figure 4 (left) and use FID for Figure 4 (right)? It seems inconsistent to use different metrics here.\n- In Figure 2, why is the diffusion pre-training compute for both CD and ECT less than that for SDE/DM? Are the authors using a different DM?\n\nOverall, should the authors adequately address my concerns and questions, I will consider increasing my score.\n\n---\nAs per my comment below:\n\nThank you to the authors for continuing to engage in discussion and for acting in good faith throughout this rebuttal process by incorporating my feedback into the updated manuscript. The novelty point has been clarified, and the improved separation of pedagogical and methodological content between Sections 2 and 3 helps better focus Section 3 on the paper's contributions. I believe the authors have addressed most of my concerns/questions, so I will increase my score to 8. With the method's simplicity, combined with its strong empirical results, my hope is that the community will build on this framework for training consistency models.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Results.**\nThe primary strength of this work lies in its empirical results, achieving state-of-the-art performance on standard image generation benchmarks. Overall, the experimental analysis is quite comprehensive by comparing against (and outperforming) recent and strong baseline methods in Table 1, and ablating some key design choices in the appendix. The ECT scaling laws in Section 4.2 are an interesting and underexplored direction in CMs, and the authors provide evidence of improved sample quality with increased training compute. If this were to also hold true for both higher dimensional datasets (e.g., ImageNet 512x512, LAION) and latent DMs (e.g., Stable Diffusion), it would have important implications for few-step generation at scale.\n\n**Insights.**\nFurthermore, some of the insights provided in the methodology of this paper are quite interesting. Specifically, Section 3.2 provides some interesting discussion on the limitations of training CMs from scratch, dubbed the “curse of CMs”, by bounding the single-step prediction error as a function of the granularity of discretization.", "weaknesses": "**Novelty.**\nDespite the strong results demonstrated in this work, I have questions about its novelty. The main methodological contribution is to initialize CM training (CT) with a pre-trained DM to enable faster convergence. In the setting of consistency distillation (CD) and DM distillation in general, it is already common practice to initialize the student from the weights of the teacher DM, effectively reducing distillation to a fine-tuning task to reduce computational requirements and facilitate convergence, similar to what’s being motivated here. The difference in this work compared to CD is to instead initialize a CM from a DM in the setting of CT, which alleviates DM error accumulation as in the case of CD. However, this seems like a rather simple extension of CMs and the connection between CMs and DMs does not seem particularly novel, so I would ask the authors to clarify their proposed novelty here.\n\nMoreover, the authors formulate CMs in continuous-time which was already done in the seminal CM work by Song et al. in Appendix B.2, so it’s not clear what the difference is here if any. Along this vein, I would also ask the authors to clarify the purpose of Section 3.1 and why it’s necessary to formulate CMs in terms of the differential condition since it currently reads as a seemingly disconnected derivation and it’s not clear from reading the paper why this formulation is needed for the rest of their method instead of using the standard discrete/continuous CM formulation by Song et al. In addition, the particular weighting function derived in Section 3.1, $w(t,r)=\\frac{1}{t-r}$, does not seem to be what the authors actually end up using in practice since, in Section 3.3, the authors go on to say “Instead, we consider decoupled weighting functions without relying on $p(r|t)$” in L311-L312. Can the authors please clarify these points and also update the manuscript accordingly.\n\n**Ablations.**\nGiven that the central contribution of ECT is initializing CMs from a DM, it would be important to see the performance of training ECT from scratch (i.e., w/o initializing from a DM) while keeping all other ECT design choices fixed, namely those outlined in Section 3.3 in terms of training schedule, loss function, and weighting function. Can the authors please provide this result in their rebuttal as it is a relevant ablation?\n\n**Easiness.**\nThe authors claim that their method makes CMs easy to train but, from a design perspective, various choices (e.g., Equation 15) do not necessarily seem more intuitive or “easier” compared to those in the original CT or in the follow-up iCT. I think it would be fair to say that ECT makes training CMs more accessible and more computationally efficient but it’s not clear that “easy” is the right characterization. Can the authors please clarify their position?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLzFD7pSUR", "reviewer": "Reviewer_QQZ4", "summary": "This paper aims to improve the training efficiency of CMs by proposing three techniques: \"continuous-time schedule,\" \"dropout,\" and \"weighting function.\" The resulting method demonstrates good performance. Additionally, the paper explores a phenomenon called the \"curse of consistency\" as well as the scaling laws of ECT.", "questions": "I think all of my questions are presented in the “weaknesses” part. If you can solve most of them well, I will raise my score.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper explores the concept of the \"curse of consistency,\" which presents an intriguing perspective.\n2. The method proposed in this paper achieves good performance.\n3. This paper discusses the “scaling laws”.", "weaknesses": "1. While the \"curse of consistency\" is indeed fascinating, discussing only the upper bound fails to capture the true nature of errors. An increase in the upper bound does not necessarily indicate a corresponding increase in error.\n2. Since the primary advantage of your method lies in its training speed, I believe you may have overlooked an important scenario: \"pretraining + iCT tuning,\" as illustrated in Figure 2.\n3. The relationship between your primary observation and your main method is not clear. Specifically, I find it difficult to understand the necessity of proposing a \"continuous-time training schedule\" and a \"weighting function\" to address the \"curse of consistency\" problem. If none of your techniques are linked to the \"curse of consistency,\" what is the rationale for mentioning it, and how does this discussion relate to the overall objectives of your paper? In fact, the \"curse of consistency\" is not even addressed in your abstract.\n4. You should place the ablation studies of your techniques (Table 6) in your main paper instead of the appendix.\n5. The \"dropout\" technique appears to be quite significant; could you clarify why it is placed in the appendix?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3skELU0A5Q", "reviewer": "Reviewer_EoPV", "summary": "This paper describes a new recipe for consistency model training. By noticing that consistency training at high discretization levels amounts to the training of a diffusion model, the authors propose to initialize the consistency model with the weights of a pre-trained diffusion model. The actual consistency model training phase is then adapted to interpolate between diffusion and continuous-time consistency training. Supplemented by additional changes to the metric and its weighting, this new recipe is shown to present favorable experimental properties: outperforming consistency distillation and training, as well as diffusion models, with a fraction of the training cost. This efficiency is highlighted in a study of the scaling laws of the proposed method.", "questions": "Given the above weaknesses but also the potential impact of its contribution, I think that this paper barely misses the acceptance threshold of ICLR, hence my recommendation of \"5: marginally below the acceptance threshold\". My questions are detailed in the \"Weaknesses\" section. I am willing to update my score following the authors' response as I believe most of them can be answered with a revision of the paper during the discussion period.\n\nI would have two additional questions.\n- Consistency distillation is tested following the original setup of Song et al. (2023). How would it benefit from later improvements (even though presented only in the setting of consistency training) of iCT (Song & Dhariwal, 2023)?\n- I would be interested in hearing the authors' opinion on whether the presented model is a distillation model or not. This is not discussed explicitly in the paper.\n\n----\n### Post-rebuttal update\n\nThe authors have addressed many of my concerns, so I choose to now recommend an \"accept\". **Nonetheless**, in case the paper is accepted, **I would like the authors to further improve the paper as suggested in the thread below**.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's strengths are immediate: by presenting a simple, easily actionable method to improve the performance of consistency models while significantly reducing their training cost, **the presented method has the potential to be used as a strong baseline for future research** in consistency models.\n\nThe paper is overall **well written**. The method is sufficiently motivated with the described insights on consistency training discretization, making diffusion pre-training an **organic improvement**. The **clarity** of its exposition, its available codebase -- that I could not assess in detail -- and, perhaps more importantly, its **simplicity** are important factors in its reusability.\n\nThe experimental results confirm the appeal of the new consistency training recipe with not only noticeable performance boosts, but also significant **efficiency gains**. The efficiency advantage is not only significant compared to standard consistency training, but even stronger compared to consistency distillation. Finally, even though no comparison with another method is provided, the presented scaling laws are an interesting addition.", "weaknesses": "While I believe this paper provides a valuable and actionable contribution, a few weaknesses, listed below in decreasing order of importance, prevent me from providing a positive recommendation.\n\n## Clarification of the novelty of some contributions\n\nThe idea of **initializing consistency models with a pre-trained diffusion model has already been considered** in the initial paper of Song et al. (2023, Appendix B.3), albeit in a restricted setting and without the experimental value highlighted in the present paper. Still, this should be discussed and may have consequences on the empirical study (see next weakness).\n\nSection 3.1 remains **vague whether the presented derivations are contributions of the present paper** or reformulated background from consistency models. My understanding is that these derivations, including the final loss function, are all already included in the original paper of Song et al. (2023): the differential equation appears in another form in their Appendix Remark 4, and the loss function closely resembles the original loss function (by removing scheduling information). This should be clarified.\n\n**The choice of weighting function appears as a reformulation** of Song & Dhariwal (2023), unless I misunderstood its description in the paper. The insight of the pseudo-Huber metric providing an adaptive weighting term is interesting, but I do not see how the proposed alternative (using the $\\ell_2$ as metric and modulate it with the weighting that the pseudo-Huber metric would have brought) is any different. Could the authors clarify this?\n\n## Lacking experimental insights\n\nWhile appealing, the presented experimental results lack additional insights and ablations to fully support the claims of the paper and highlight its added value w.r.t. prior work. Two main components are missing.\n\nThe first missing component is a **full fledged ablation study**. One is already included in Appendix Section B (Table 5). However, given this preliminary result showing that the simple diffusion pre-training (already considered in prior work, cf. previous weakness) brings most of the improvements, I think the paper would benefit from including in its main part an augmented ablation study in the same experimental setting as Section 4. This ablation study should then be discussed to assess the significance of each contribution in the new training recipe.\n\nThe second missing component is a **study of the evolution of the methods' performance w.r.t. training time/iterations**. As is, baseline and ablation results are presented at a fixed number of iterations. Plotting their performance w.r.t. the number of iterations would enable a more comprehensive efficiency comparison, as well as justify the choice of stopping time for baselines in e.g. Table 1.\n\n## Lack of moderation and rigor in some assertions\n\nThis is a less important issue that still deserves to be addressed. The following statements lack moderation and/or rigor.\n- In Section 1, it is stated that \"the speedup achieved by these sampling techniques usually comes at the expense of the quality of generated samples\". This is partially incorrect as some works like the one of Karras et al. (2022) significantly reduced the number of required model evaluations while maintaining performance.\n- It is incorrect that the proposed loss function of Eq. (12) \"generalizes Song et al. (2023)’s discrete training schedule to continuous time\". The proposed loss indeed relies on discretizing the aforementioned differential equation. Instead, this loss is exactly the consistency training loss of Song et al. (2023) untied from its specific discretization grid.\n- Writing $\\Delta t \\to \\mathrm{d}t$ instead of $0$ is confusing.\n- In Section 4.1, the proposed model is said to \"only [require] 1/1000 of [Score SDE's] inference cost\". This is correct but misleading as state-of-the-art diffusion models no longer require thousands of model evaluations. This statement should be toned down.\n\n## Minor issues\n\n- The diffusion ODE of Eq. (2), from Karras et al. (2022), requires that $\\mathbf{f} = 0$ in Eq. (1).\n- $f(\\mathbf{x}_t, t)$ is said to be \"a denoising function\" after Eq. (3). This statement should be more detailed.\n- To my knowledge, in Karras et al. (2022)'s timestep schedule, $\\rho = 7$ instead of $0.7$ on line 134.\n- To my understanding, a part of the numbers in Table 1 were not obtained by the authors but are reported from other papers. This should be clarified.\n- $\\epsilon$ in Eq. (19) should be $c^2$ to be consistent with the rest of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)