Consistency Models Made Easy

## Abstract

Abstract Consistency models (CMs) offer faster sampling than traditional diffusion models, but their training is resource-intensive. For example, as of 2024, training a state-of-the-art CM on CIFAR-10 takes one week on 8 GPUs. In this work, we propose an effective scheme for training CMs that largely improves the efficiency of building such models. Specifically, by expressing CM trajectories via a particular differential equation, we argue that diffusion models can be viewed as a special case of CMs. We can thus fine-tune a consistency model starting from a pretrained diffusion model and progressively approximate the full consistency condition to stronger degrees over the training process. Our resulting method, which we term Easy Consistency Tuning (ECT), achieves vastly reduced training times while improving upon the quality of previous methods: for example, ECT achieves a 2-step FID of 2.73 on CIFAR10 within 1 hour on a single A100 GPU, matching Consistency Distillation trained for hundreds of GPU hours. Owing to this computational efficiency, we investigate the scaling laws of CMs under ECT, showing that they obey the classic power law scaling, hinting at their ability to improve efficiency and performance at larger scales. Our code is publicly available, making CMs more accessible to the broader community.

## 1 Introduction

Diffusion Models (DMs) (Ho et al., , Song et al., ), or Score-based Generative Models (SGMs) (Song et al., ; ), have vastly changed the landscape of visual content generation with applications in images (Rombach et al., , Saharia et al., , Ho et al., , Dhariwal and Nichol, , Hatamizadeh et al., , Ramesh et al., ), videos (Brooks et al., , Blattmann et al., , Bar-Tal et al., , Ho et al., , Gupta et al., ), and 3D objects (Poole et al., , Wang et al., , Lee et al., , Chen et al., , Babu et al., ).
DMs progressively transform a data distribution to a known prior distribution (e.g. Gaussian noise) according to a stochastic differential equation (SDE) (Song et al., ) and train a model to denoise noisy observations.
Samples can be generated via a reverse-time SDE that starts from noise and uses the trained model to progressively denoise it.
However, sampling from a DM naively requires hundreds to thousands of model evaluations due to the curvature of the diffusion sampling trajectory (Karras et al., ), making the entire generative process slow.
Many approaches have been proposed to address this issue,
including training-based techniques such as distillation (Luhman and Luhman, , Salimans and Ho, , Luo et al., , Gu et al., , Sauer et al., , Geng et al., , Yin et al., , Nguyen and Tran, ), adaptive compute architectures for the backbone model (Moon et al., , Tang et al., ), as well as training-free methods such as fast samplers (Kong and Ping, , Lu et al., , Zhang and Chen, , Zhou et al., , Xue et al., ) or interleaving small and large backbone models during sampling (Pan et al., ).
However, the speedup achieved by these sampling techniques usually comes at the expense of the quality of generated samples.

Consistency Models (CMs) (Song et al., ) are a new family of generative models, closely related to diffusion models, that have demonstrated promising results as faster generative models.
These models learn a mapping between noise and data, and all the points of the sampling trajectory map to the same initial data point.
Owing to this condition, consistency models are capable of generating high-quality samples in 1-2 model evaluations.
The best such models so far, built using improved Consistency Training (iCT) (Song and Dhariwal, ), have pushed the quality of images generated by 1-step CMs trained from scratch to a level comparable with SoTA DMs using thousands of steps for sampling.
Unfortunately, CMs remain time-consuming and practically challenging to train: the best practice takes many times longer than similar-quality DMs while involving complex hyperparameter choices in the training process.
In total, this has substantially limited the uptake of CMs within the community.

In this work, we introduce a differential perspective on consistency models, leading to the formulation of the differential consistency condition in continuous time. This insight reveals the link between diffusion models and consistency models, viewing DMs as a special case of CMs with loose discretization.
This observation motivates us to smoothly interpolate from DM to CM by progressively tightening the consistency condition, bootstrapping pretrained DMs to 1-step CMs w/o using extra frozen teachers.
We term this strategy as *Easy Consistency Tuning (ECT)*, which includes diffusion pretraining as a special stage of the continuous time training schedule.

ECT significantly improves both training efficiency and performance. On ImageNet 64$\times$64 (Deng et al., ), ECT achieves superior 1-step and 2-step sample quality compared to the prior art. Similarly, on CIFAR-10 (Krizhevsky, ) 2-step sample quality of ECT surpasses previous methods. The total cost of the pretraining and tuning scheme requires only $\nicefrac{{1}}{{4}}\sim\nicefrac{{1}}{{3}}$ of the computational resources (FLOPs) used by the current state-of-the-art method, iCT (Song and Dhariwal, ), while the tuning stage can be remarkably lightweight, typically accounting for $10\%$ or less of the overall cost and further benefiting from scaling.

Leveraging ECT’s computational efficiency, we conduct the first study into the scaling behaviors of CMs, revealing the classic power law scaling for model size, FLOPs, and training compute.
The scaling also suggests a sweet spot of using smaller few-step CMs over larger 1-step CMs in certain scenarios.
This computational efficiency enables us to explore the design space of CMs by using the tuning stage as a proxy. Notably, tuning findings, such as weighting functions, can improve the pretraining stage and, in turn, enhance the overall pretraining + tuning pipeline for CMs.

In short, we summarize our contributions as follows:

- •
We develop Easy Consistency Tuning (ECT), a pretraining + tuning scheme for training CMs in continuous time, demonstrating significant efficiency and performance gains compared to the current best practices for training CMs.
- •
We investigate CMs’ scaling behaviors for the first time and reveal the classic power law.
- •
We explore the design space of CMs through ECT, introducing the continuous-time schedule and better weighting functions for CMs.

## 2 Preliminaries

#### Diffusion Models.

Let $p_{\text{data}}({\mathbf{x}}_{0})$ denote the data distribution. Diffusion models (DMs) perturb this distribution by adding monotonically increasing i.i.d. Gaussian noise with standard deviation $\sigma(t)$ from $t=0$ to $T$ such that $p_{t}({\mathbf{x}}_{t}|{\mathbf{x}}_{0})={\mathcal{N}}({\mathbf{x}}_{0},\sigma
^{2}(t){\bm{I}})$, and $\sigma(t)$ is chosen such that $\sigma(0)=\sigma_{\text{min}}$ and $\sigma(T)=\sigma_{\text{max}}$.
This process is described by the following SDE (Song et al., )

$$ ${\mathrm{d}}{\mathbf{x}}_{t}=\mathbf{f}({\mathbf{x}}_{t},t){\mathrm{d}}t+g(t){ \mathrm{d}}\mathbf{w}_{t},$ (1) $$

where $\mathbf{w}$ is the standard Wiener process, $\mathbf{f}(\cdot,t):\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ is the drift coefficient, and $g(\cdot):\mathbb{R}\rightarrow\mathbb{R}$ is the diffusion coefficient.
Samples can be generated by solving the reverse-time SDE starting from $t=T$ to 0 0 and sampling ${\mathbf{x}}_{T}\sim{\mathcal{N}}(0,\sigma^{2}_{\text{max}}{\bm{I}})$.
(Song et al., ) show that this SDE has a corresponding ODE, called the probability flow ODE (PF-ODE), whose trajectories share the same marginal probability densities as the SDE. We follow the notation in (Karras et al., ) to describe the ODE as

$$ ${\mathrm{d}}{\mathbf{x}}_{t}=-\dot{\sigma}(t)\sigma(t)\nabla_{{\mathbf{x}}_{t} }\log p_{t}({\mathbf{x}}_{t}){\mathrm{d}}t,$ (2) $$

where $\nabla_{{\mathbf{x}}_{t}}\log p_{t}({\mathbf{x}}_{t})$ denotes the score function. Prior works (Karras et al., , Song et al., ) set $\sigma(t)=t$ which yields

$$ $\dfrac{{\mathrm{d}}{\mathbf{x}}_{t}}{{\mathrm{d}}t}=-t\nabla_{{\mathbf{x}}_{t} }\log p_{t}({\mathbf{x}}_{t})=\dfrac{({\mathbf{x}}_{t}-f({\mathbf{x}}_{t},t))} {t},$ (3) $$

where $f({\mathbf{x}}_{t},t)$ denotes a denoising function that predicts clean image ${\mathbf{x}}_{0}$ given noisy image ${\mathbf{x}}_{t}$. We will follow this parametrization in the rest of this paper. Note that time is same as noise level with this parametrization, and we will use these two terms interchangeably.

#### Consistency Models.

CMs are built upon the PF-ODE in Eq. 3, which establishes a bijective mapping between data distribution and noise distribution.
CMs learn a consistency function $f({\mathbf{x}}_{t},t)$ that maps the noisy image ${\mathbf{x}}_{t}$ back to the clean image ${\mathbf{x}}_{0}$

$$ $f({\mathbf{x}}_{t},t)={\mathbf{x}}_{0}.$ (4) $$

Note that the consistency function needs to satisfy the boundary condition at $t=0$. Prior works (Song et al., , Song and Dhariwal, , Karras et al., ) impose this boundary condition by parametrizing the CM as

$$ $f_{\theta}({\mathbf{x}}_{t},t)=c_{\text{skip}}(t)\,{\mathbf{x}}_{t}+c_{\text{ out}}(t)\,F_{\theta}({\mathbf{x}}_{t},t),$ (5) $$

where $\theta$ is the model parameter, $F_{\theta}$ is the network to train, and $c_{\text{skip}}(t)$ and $c_{\text{out}}(t)$ are time-dependent scaling factors such that $c_{\text{skip}}(0)=1$, $c_{\text{out}}(0)=0$.
This parameterization guarantees the boundary condition by design.
We discuss specific choices of $c_{\text{skip}}(t)$ and $c_{\text{out}}(t)$ in Appendix D.

During training, CMs first discretize the PF-ODE into $N-1$ subintervals with boundaries given by $t_{\text{min}}=t_{1}<t_{2}<\ldots<t_{N}=T$.
The model is trained on the following CM loss, which minimizes a metric between adjacent points on the sampling trajectory

$$ $\operatorname*{arg\,min}_{\theta}\mathbb{E}\left[w(t_{i})d(f_{\theta}({\mathbf {x}}_{t_{i+1}},t_{i+1}),f_{\theta^{-}}(\tilde{{\mathbf{x}}}_{t_{i}},t_{i})) \right].$ (6) $$

Here, $d(\cdot,\cdot)$ is a metric function, the $f_{\theta}$ indicates a trainable neural network that is used to learn the consistency function, $f_{\theta^{-}}$ indicates an exponential moving average (EMA) of the past values of $f_{\theta}$, and $\tilde{{\mathbf{x}}}_{t_{i}}={\mathbf{x}}_{t_{i+1}}-(t_{i}-t_{i+1})t_{i+1}
\nabla_{{\mathbf{x}}_{t_{i+1}}}\log p_{t_{i+1}}({\mathbf{x}}_{t_{i+1}})$.
Further, the discretization curriculum $N$ should be adaptive and tuned during training to achieve good performance.

In the seminal work, (Song et al., ) use Learned Perceptual Similarity Score (LPIPS) (Zhang et al., ) as a metric function, set $w(t_{i})=1$ for all $t_{i}$, and sample $t_{i}$ according to the sampling scheduler by  (Karras et al., ):
$t_{i}=\left(t_{\text{max}}^{1/\rho}+\frac{i}{N-1}(t_{\text{min}}^{1/\rho}-t_{
\text{max}}^{1/\rho})\right)^{\rho}$ for $i\in{\mathcal{U}}[1,N-1]$ and $\rho=0.7$. Further, the score function
$\nabla_{{\mathbf{x}}_{t}}\log p({\mathbf{x}}_{t})$
can either be estimated from a pretrained diffusion model,
which results in Consistency Distillation (CD),
or can be estimated with an unbiased score estimator

$$ $\nabla_{{\mathbf{x}}_{t}}\log p({\mathbf{x}}_{t})=\mathbb{E}\left[\nabla_{{ \mathbf{x}}_{t}}\log p({\mathbf{x}}_{t}|{\mathbf{x}}_{0})\bigg{|}{\mathbf{x}}_ {t}\right]=\mathbb{E}\left[-\frac{{\mathbf{x}}_{t}-{\mathbf{x}}_{0}}{t^{2}} \bigg{|}{\mathbf{x}}_{t}\right],$ (7) $$

which results in consistency training (CT).

The follow-up work, iCT (Song and Dhariwal, ), introduces several improvements that significantly improve training efficiency as well as the performance of CMs.
First, the LPIPS metric, which introduces undesirable bias in generative modeling, is replaced with a Pseudo-Huber metric.
Second, the network $f_{\theta^{-}}$ does not maintain an EMA of the past values of $f_{\theta}$. Third, iCT replaces the uniform weighting scheme $w(t_{i})=1$ with $w(t_{i})=\frac{1}{t_{i+1}-t_{i}}$.
Further, the scaling factors of noise embeddings and dropout are carefully selected.
Fourth, iCT introduces a complex discretization curriculum during training:

$$ $\displaystyle N(m)=\min(s_{0}2^{\left\lfloor\frac{m}{M^{\prime}}\right\rfloor} ,s_{1})+1,\quad M^{\prime}=\left\lfloor\frac{M}{\log_{2}\left\lfloor\frac{s_{1 }}{s_{0}}\right\rfloor+1}\right\rfloor,$ (8) $$

where $m$ is the current number of iterations, $M$ is the total number of iterations, $\sigma_{\max}$ and $\sigma_{\min}$ is the largest and smallest noise level for training, $s_{0}=10$ and $s_{1}=1280$ are hyperparameters.
Finally, during training, iCT samples $i\sim p(i)\propto\text{erf}\left(\frac{\log(t_{i+1})-P_{\text{mean}}}{\sqrt{2}
P_{\text{std}}}\right)-\text{erf}\left(\frac{\log(t_{i})-P_{\text{mean}}}{
\sqrt{2}P_{\text{std}}}\right)$ from a discrete Lognormal distribution, where $P_{\text{mean}}=-1.1$ and $P_{\text{std}}=2.0$.

## 3 Probing consistency models

We will first introduce the differential consistency condition and the loss objective based on this condition.
Next, we will analyze this loss objective and highlight the challenges of training CMs with it.
Based on this analysis, we present our method, Easy Consistency Tuning (ECT).
ECT is a simple, principled approach to efficiently train CMs to meet the (differential) consistency condition.
The resulting CMs can generate high-quality samples in 1 or 2 sampling steps.

### 3.1 Differential Consistency Condition

As stated in Sec. 2, CMs learn a consistency function $f({\mathbf{x}}_{t},t)$ that maps the noisy image ${\mathbf{x}}_{t}$ back to the clean image ${\mathbf{x}}_{0}$: $f({\mathbf{x}}_{t},t)={\mathbf{x}}_{0}$.
Instead, by taking the time derivative of both sides, given by the differential form

$$ $\frac{\mathrm{d}f}{\mathrm{d}t}=\frac{\mathrm{d}}{\mathrm{d}t}{\mathbf{x}}_{0} =0.$ (9) $$

However, the differential form $\frac{\mathrm{d}f}{\mathrm{d}t}=0$ alone is not sufficient to guarantee that the model output will match the clean image,
as there exist trivial solutions where the model maps all the inputs to a constant value, such as
$f({\mathbf{x}}_{t},t)\equiv 0$.
To eliminate these collapsed solutions, (Song et al., , Song and Dhariwal, ) impose the boundary condition for $f({\mathbf{x}}_{t},t)={\mathbf{x}}_{0}$ via model parameterization:

$$ $\displaystyle f({\mathbf{x}}_{t},t)={\mathbf{x}}_{0}\Leftrightarrow\frac{ \mathrm{d}f}{\mathrm{d}t}=0,f({\mathbf{x}}_{0},0)={\mathbf{x}}_{0}.$ (10) $$

This boundary condition $f({\mathbf{x}}_{0},0)={\mathbf{x}}_{0}$ ensures that the model output matches the clean image when the noise level is zero. Together, the differential form in Eq. 9 and the boundary condition define the differential consistency condition in Eq. 10, or consistency condition in short.

#### Finite Difference Approximation.

To learn the consistency condition, we discretize the differential form $\frac{\mathrm{d}f}{\mathrm{d}t}=0$ using a finite-difference approximation:

$$ $0=\frac{\mathrm{d}f}{\mathrm{d}t}\approx\frac{f_{\theta}({\mathbf{x}}_{t})-f_{ \theta}({\mathbf{x}}_{r})}{t-r}$ (11) $$

where $\mathrm{d}t\approx\Delta t=t-r$, $t>r>=0$, and $f_{\theta}({\mathbf{x}}_{t})$ denotes $f_{\theta}({\mathbf{x}}_{t},t)$.
For a given clean image ${\mathbf{x}}_{0}$, we produce two perturbed images ${\mathbf{x}}_{t}$ and ${\mathbf{x}}_{r}$ using the same perturbation direction $\bm{\epsilon}\sim p(\bm{\epsilon})$ at two noise levels $t$ and $r$.
Specifically,
we compute $t$ and $r$ according to the forward process of CMs i.e., ${\mathbf{x}}_{t}={\mathbf{x}}_{0}+t\cdot\bm{\epsilon}$ and ${\mathbf{x}}_{r}={\mathbf{x}}_{0}+r\cdot\bm{\epsilon}$.

To satisfy the consistency condition, we minimize the distance between $f_{\theta}({\mathbf{x}}_{t})$ and $f_{\theta}({\mathbf{x}}_{r})$. For the boundary condition, we optimize $f_{\theta}({\mathbf{x}}_{t})$ to align with the clean image ${\mathbf{x}}_{0}$. For noise levels $t>r>0$, we freeze $f_{\operatorname{sg}(\theta)}({\mathbf{x}}_{r})$ using the stop-gradient operator $\operatorname{sg}$ and optimize $f_{\theta}({\mathbf{x}}_{t})$ on higher variance inputs to align with self teacher’s prediction $f_{\operatorname{sg}(\theta)}({\mathbf{x}}_{r})$ on lower variance inputs (Chen and He, ).

#### Loss Function.

Given the discretization, the training objective for CMs can be formulated as:

$$ $\underset{\theta}{\arg\min}\;\mathbb{E}_{{\mathbf{x}}_{0},\bm{\epsilon},t} \left[w(t,r){\mathbf{d}}(f_{\theta}({\mathbf{x}}_{t}),f_{\operatorname{sg}( \theta)}({\mathbf{x}}_{r}))\right],$ (12) $$

where we can extract a weighting function $w(t,r)=\frac{1}{t-r}$, ${\mathbf{x}}_{t}={\mathbf{x}}_{0}+t\cdot\bm{\epsilon}$, ${\mathbf{x}}_{r}={\mathbf{x}}_{0}+r\cdot\bm{\epsilon}$ using a shared noise direction $\bm{\epsilon}\sim p(\bm{\epsilon})$, and ${\mathbf{d}}(\cdot,\cdot)$ is a metric function.
This formulation generalizes Song et al. ()’s discrete training schedule to continuous time.
To regularize the model, dropout masks should also be consistent across $f_{\theta}({\mathbf{x}}_{t})$ and $f_{\theta}({\mathbf{x}}_{r})$ by setting a shared random seed, ensuring that the noise levels $t$ and $r$ are the only varying factors.

### 3.2 The "Curse of Consistency" and its Implications

The objective in Eq. 12 can be challenging to optimize when $\Delta t\to 0$. This is because the prediction errors from each discretization interval accumulate, leading to slow training convergence or, in the worst case, divergence.
To further elaborate, consider a large noise level $T$. We first follow the notation of CT (Song et al., ) and split the noise horizon $[0,T]$ into $N$ smaller consecutive subintervals. We can error bound the error of $1$-step prediction as

$$ $\|f_{\theta}({\mathbf{x}}_{T})-{\mathbf{x}}_{0}\|\leq\sum_{i}^{N}\|f_{\theta}( {\mathbf{x}}_{t_{i}})-f_{\theta}({\mathbf{x}}_{r_{i}})\|\leq Ne_{\max},$ (13) $$

Figure: Figure 1: The "Curse of Consistency": The consistency condition holds at $\Delta t=\mathrm{d}t$. However, the training dynamics converges more slowly and is less stable as $\Delta t\to{\mathrm{d}}t$ (i.e., $N\to\infty$).
Refer to caption: extracted/5917567/fig/delta_t.png

where $r_{1}=0<t_{1}=r_{2}<\cdots<t_{N-2}=r_{N}<t_{N}=T$, and $e_{\max}=\max_{i}\|f_{\theta}({\mathbf{x}}_{t_{i}})-f_{\theta}({\mathbf{x}}_{r
_{i}})\|$, for $i=1,\cdots,N$.
Ideally, we want both $N$ and $e_{\max}$ to be small so that this upper bound is small. But in practice, there is a trade-off between these two terms.
As $\Delta t_{i}=(t_{i}-r_{i})\to 0$, $e_{\max}$ decreases because ${\mathbf{x}}_{t_{i}}$ and ${\mathbf{x}}_{r_{i}}$ will be close, and it is easier to predict both $f_{\theta}({\mathbf{x}}_{r_{i}})$ and $f_{\theta}({\mathbf{x}}_{t_{i}})$.
However, $N$ will increase as $\Delta t_{i}\to 0$.
In contrast, for a large $\Delta t_{i}$, vice-versa holds true.
It is difficult to theoretically estimate the rate at which $e_{\max}$ decreases when $\Delta t_{i}=t_{i}-r_{i}\to 0$, as it depends on the optimization process—specifically, how effectively we train the model to minimize the consistency error in each interval.
If $e_{\max}$ decreases more slowly than $N$, the product of the two can increase instead, resulting in a worse prediction error $\|f_{\theta}({\mathbf{x}}_{T})-{\mathbf{x}}_{0}\|$.

The insight from the above observation is that when training from scratch by strictly following the differential form $\nicefrac{{\mathrm{d}f}}{{\mathrm{d}t}}=0$ with a tiny $\Delta t_{i}\approx 0$, the resulting model might converge slowly due to accumulated consistency errors from each interval $\Delta t_{i}$.
To investigate this hypothesis, we conducted experiments training a series of CMs using different fixed numbers of intervals $N$, and corresponding $\Delta t$ values for the consistency condition, as illustrated in Fig. 1.
Our observations align with Song and Dhariwal (), with the key distinction that we use a fixed $N$ for this analysis, as this approach provides better insight on how much the precise approximation of the differential consistency condition matters and isolates the effect of discretization errors.

### 3.3 Easy Consistency Tuning (ECT)

The discussion in Sec. 3.2 highlighted the training instability issue that can arise when we naively train CMs from scratch and directly optimize for the differential consistency condition with the loss objective in Eq. 12 while directly following $\Delta t\approx 0$.
In this section, we propose several strategies that largely alleviate the aforementioned training issues and improve the efficiency of CMs. We term this approach Easy Consistency Tuning (ECT), as it effectively balances training stability and model performance while simplifying the CM training process. ECT follows a two-stage approach: diffusion pretraining, followed by consistency tuning, which we will detail in the following subsections.

#### Diffusion Pretraining + Consistency Tuning.

Drawing inspiration from iCT’s adaptive discrete-time schedule (Song and Dhariwal, ), we start ECT with a large $\Delta t$, and gradually shrink $\Delta t\to 0$. In our problem setup, Since $t>r\geq 0$, we have the largest possible $\Delta t=t$ with $r=0$, which yields

$$ $\underset{\theta}{\arg\min}\;\|f_{\theta}({\mathbf{x}}_{t})-f_{\operatorname{ sg}(\theta)}({\mathbf{x}}_{r})\|=\|f_{\theta}({\mathbf{x}}_{t})-f_{ \operatorname{sg}(\theta)}({\mathbf{x}}_{0})\|=\|f_{\theta}({\mathbf{x}}_{t})- {\mathbf{x}}_{0}\|.$ (14) $$

Training a model with this loss is *identical* to diffusion model/Score SDE (Ho et al., , Song et al., ). This observation suggests a learning scheme that smoothly interpolates from DMs $\Delta t=t$ to CMs $\Delta t=\mathrm{d}t$ by gradually shrinking $\Delta t\to\mathrm{d}t$ during training by gradually tightening the consistency condition.
With this reasoning, diffusion pretraining can be considered as a special case of consistency training with a loose discretization of the consistency condition.
Therefore, in practice, we start ECT with a pretrained diffusion model, resulting in a training scheme of pretraining+tuning.
Another benefit of this initialization is that during training, especially in the initial stages, it ensures good targets $f_{\operatorname{sg}(\theta)}({\mathbf{x}}_{r})$ in the loss objective, avoiding trivial solutions.

We highlight two advantages of this learning scheme:
1. The pretraining+tuning scheme of ECT outperforms iCT’s training-from-scratch approach with lower overall computational cost (see Sec. 4.1).
2. Tuning serves as an efficient proxy for exploring the CM design space. Given a pretrained diffusion model, insights gained during tuning can be applied to improve the pretraining stage, resulting in a more refined overall CM training pipeline.
Please refer to Appendix B for more details.

#### Continuous-time Training Schedule.

Figure: Algorithm 1 Easy Consistency Tuning (ECT)

We investigate the design principles of a continuous-time schedule whose "boundary" condition yields standard diffusion pretraining, i.e., constructing training pairs of $r=0$ for all $t$ at the beginning. Note that this is unlike the discrete-time schedule used in iCT (See Sec. 2 and Appendix A).
We consider overlapped intervals for consistency models, which allows for factoring $p(t,r)=p(t)\,p(r|t)$ and continuous sampling of infinite $t$ from noise distribution $p(t)$, for instance, $\mathrm{LogNormal}(P_{\text{mean}},P_{\text{std}})$, and $r\sim p(r|t)$.

We refer to $p(r|t)$ as the mapping function.
Since we need to shrink $\Delta t\to{\mathrm{d}}t$ as the training progresses, we augment the mapping function to depend on training iterations, $p(r|t,\text{iters})$, to control $\Delta t=(t-r)\to{\mathrm{d}}t$.
We parametrize the mapping function $p(r|t,\text{iters})$ as

$$ $\frac{r}{t}=1-\frac{1}{q^{a}}n(t)=1-\frac{1}{q^{\lfloor\mathrm{iters}/d\rfloor }}n(t),$ (15) $$

where we take $n(t)=1+k\,\sigma(-bt)=1+\frac{k}{1+e^{bt}}$ with $\sigma(\cdot)$ as the sigmoid function, $\mathrm{iters}$ refers to training iterations. In general, we set $q>1$, $k=8$ and $b=1$.
Since $r\geq 0$, we also clamp $r$ to satisfy this constraint.
At the beginning of training, this mapping function produces $\nicefrac{{r}}{{t}}=0$, which recovers the diffusion pretraining.
We discuss design choices of this function in Appendix A.

#### Choice of metric.

iCT uses the Pseudo-Huber metric (Charbonnier et al., ) to mitigate the perceptual bias caused by the LPIPS metric (Zhang et al., ).
When taking a careful look at this metric, we reveal that this metric offers adaptive per-sample scaling of the gradients, which reduces the variance of gradients during training.
Specifically, the differential of the Pseudo-Huber loss can be decomposed into two terms: an adaptive scaling factor $w(\Delta)$ and the differential of the squared $L_{2}$ loss. We discuss this in more detail in Appendix A.
Thus, we disentangle the weighting function from the loss metric as it provides greater flexibility. In our experiments, we retain the squared $L_{2}$ metric used in DMs and explore alternate choices of adaptive weighting terms that reduce the variance of gradients during training. Please refer to Appendix B for further discussions.

#### Weighting function.

Weighting functions usually lead to a substantial difference in performance in DMs, and the same holds true for CMs.
Substituting the mapping function $p(r|t)$ in Eq. 15 into the weighting function in Eq. 12, we get $w(t,r)=\nicefrac{{1}}{{(t-r)}}=q^{a}\cdot\nicefrac{{1}}{{tn(t)}}$, where $q^{a}$ is the scaling factor to maintain the loss scales when $\Delta t\to 0$, and $\nicefrac{{1}}{{tn(t)}}$ is the weighting akin to DMs.
This couples the weighting function with the mapping function $p(r|t)$. Instead, we consider decoupled weighting functions without relying on $p(r|t)$.
Motivated by the adaptive scaling factor that appears in Pseudo-Huber loss (See Appendix A for more details), we rewrite the weighting function as

$$ $w(t)=\bar{w}(t)\cdot w(\Delta)=\bar{w}(t)\cdot\frac{1}{(\|\Delta\|_{2}^{2}+c^{ 2})^{\nicefrac{{1}}{{2}}}},$ (16) $$

where $\Delta=f({\mathbf{x}}_{t})-f({\mathbf{x}}_{r})$. We define $\bar{w}(t)$ as timestep weighting, while $w(\Delta)$ as adaptive weighting.
The adaptive weighting improves training efficiency with the $L_{2}$ metric because, as $\Delta\to 0$ (usually happens when $t\to 0$), this weighting $w(\Delta)$ upscales gradients to avoid vanishing gradient during learning of fine-grained features, while $c$ mitigates potential numerical issues when $\Delta\approx 0$.
We direct the reader to Tab. 2 in Appendix B for a detailed overview of various choices of $\bar{w}(t)$ and $w(\Delta)$ considered in this work.
In general, we notice that CMs’ generative capability greatly benefits from weighting functions that control the variance of the gradients across different noise levels.
We provide a brief summary of the core steps of ECT in Alg. 1.

## 4 Experiments

This section compares different learning schemes and investigates scaling laws of ECT, while more experiments on design choices and scaling are shown in Appendix B. We evaluate the efficiency and scalability of ECT on two datasets: CIFAR-10 (Krizhevsky, ) and ImageNet $64\times 64$ (Deng et al., ).
We measure the sample quality using Fréchet Inception Distance (FID) (Heusel et al., ) and Fréchet Distance under the DINOv2 model (Oquab et al., ) ($\text{FD}_{\text{DINOv2}}$) (Stein et al., ) and sampling efficiency using the number of function evaluations (NFEs). We also indicate the relative training costs of each of these methods. Implementation details can be found in Appendix D.

### 4.1 Comparison of Training Schemes

We compare CMs trained with ECT (denoted as ECM) against state-of-the-art diffusion models, advanced samplers for diffusion models, distillation methods such as consistency distillation (CD) (Song et al., ), and improved Consistency Training (iCT) (Song and Dhariwal, ).
The key results are summarized in Fig. 2. We show the training FLOPs, inference cost, and generative performance of the four training schemes.

Figure: Figure 2: Comparison of training schemes for the diffusion-consistency family on CIFAR-10. Without relying on distillation from frozen diffusion teachers or extra adversarial supervision, ECT surpasses Consistency Distillation (CD) (Song et al., ) and Consistency Models trained from scratch (iCT) (Song and Dhariwal, ) using $\nicefrac{{1}}{{4}}$ of the total training cost. ECT also significantly reduces the inference cost to $\nicefrac{{1}}{{1000}}$ compared to Diffusion Pretraining (Score SDE/DMs) while maintaining comparable sample quality.
Refer to caption: x1.png

Figure: Figure 3: Scaling up training compute and model sizes results in improved sample quality on ImageNet 64$\times$64. Each triplet (left-to-right) has $2$-step samples from ECM-S trained with $12.8\mathrm{M}$ images, ECM-S trained with $102.4\mathrm{M}$ images, and ECM-XL trained with $102.4\mathrm{M}$ images.
Refer to caption: extracted/5917567/fig/scaling.png

#### Score SDE/Diffusion Models.

We compare ECMs against Score SDE (Song et al., ), EDM  (Karras et al., ), and EDM with DPM-Solver-v3 (Zheng et al., ).
$2$-step ECM, which has been fine-tuned for only $100$k iterations, matches Score SDE-deep, with $2\times$ model depth and $2000$ NFEs, in terms of FID.
As noted in Fig. 2, ECM only requires $\nicefrac{{1}}{{1000}}$ of its inference cost and latency to achieve the same sample quality.
$2$-step ECM fine-tuned for $100$k iterations outperforms EDM with advanced DPM-Solver-v3 (NFE=$10$).

#### Diffusion Distillation.

We compare ECT against Consistency Distillation (CD) (Song et al., ), a SoTA approach that distills a pretrained DM into a CM. As shown in Tab. 1, ECM significantly outperforms CD on both CIFAR-10 and ImageNet 64$\times$64. We note that ECT is free from the errors of teacher DM and does not incur any additional cost of running teacher DM.
$2$-step ECM outperforms $2$-step CD (with LPIPS (Zhang et al., )) in terms of FID ($2.20$ vs $2.93$) on CIFAR-10 while using around $\nicefrac{{1}}{{3}}$ of training compute of CD.

#### Consistency training from scratch.

Improved Consistency Training (iCT) (Song and Dhariwal, ) is the SoTA recipe for training a consistency model from scratch without inferring the diffusion teacher.
Compared to training from scratch, ECT rivals iCT-deep using $\nicefrac{{1}}{{4}}$ of the overall training compute ($\nicefrac{{1}}{{8}}$ in the tuning stage) and $\nicefrac{{1}}{{2}}$ of the model size as shown in Fig. 2 and Tab. 1.

Table: Table 1: Generative performance on unconditional CIFAR-10 and class-conditional ImageNet 64$\times$64. We use a budget of $12.8\mathrm{M}$ training images (batch size 128 and 100k iterations) for ECMs. ^⋆ stands for a budget of $102.4\mathrm{M}$ training images (batch size 1024 and 100k iterations) on ImageNet 64$\times$64.

### 4.2 Scaling Laws of ECT

We leverage efficiency of ECT to examine the scaling behavior of CMs, including training compute, model size, and model FLOPs.
We find that when computational resources are not a bottleneck, ECT scales well and follows the classic power law.

#### Training Compute.

Initializing from the weights of EDM (Karras et al., ), we fine-tune ECMs across six compute scales on CIFAR-10 (Krizhevsky, ) and plot the trend of $\text{FD}_{\text{DINOv2}}$ against training compute in Fig. 4 (Left). The largest compute reaches $2\times$ the diffusion pretraining budget. As we scale up the training budget, we observe a classic power-law decay in $\text{FD}_{\text{DINOv2}}$, indicating that increased computational investment in ECT leads to substantial improvements in generative performance. Intriguingly, the gap between 1-step and 2-step generation becomes narrower when scaling up training compute, even while using the same $\Delta t\to\mathrm{d}t$ schedule. We further fit the power-law $\text{FD}_{\text{DINOv2}}=K\cdot C^{\alpha}$, where $C$ is the normalized training FLOPs. The Pearson correlation coefficient between $\log(\text{Training Compute})$ and $\log(\text{FD}_{\text{DINOv2}})$ for 1-step and 2-step generation is $-0.9940$ and $-0.9996$, respectively, both with statistical significance ($p$-values $<10^{-4}$).

#### Model Size & FLOPs.

Initialized from EDM2 (Karras et al., ) pretraining, we train ECM-S/M/L/XL models with parameters from $280\mathrm{M}$ to $1.1\mathrm{B}$ and model FLOPs from $102\mathrm{G}$ to $406\mathrm{G}$. As demonstrated in Fig. 4, both 1-step and 2-step generation capabilities exhibit log-linear scaling for model FLOPs and parameters. This scaling behavior confirms that ECT effectively leverages increased model sizes and computational power to improve 1-step and 2-step generative capabilities.

Notably, ECT achieves better 2-step generation performance than state-of-the-art CMs, while utilizing only $\mathbf{33\%}$ of the overall computational budget compared to iCT (Song and Dhariwal, ) ($\text{batch size }4096\times 800\text{k}$). This significant efficiency is achieved through a two-stage process: pretraining and tuning. While the pretraining stage utilizes the EDM2 pipeline, the tuning stage of ECT requires a remarkably modest budget of $12.8\mathrm{M}$ training images ($\text{batch size }128\times 100\text{k}$), ranging from $\mathbf{0.60\%}$ to $\mathbf{1.91\%}$ of the pretraining budget, depending on the model sizes.

#### Inference.

Our scaling study also indicates a sweet spot for the inference of CMs. On both CIFAR-10 and ImageNet 64$\times$64, there are 2-step inferences of smaller models surpassing 1-step inferences of larger models, e.g., $498\mathrm{M}$ ECM-M against $1.1\mathrm{B}$ ECM-XL. This calls for further studies of inference-optimal scaling and test-time compute scaling for visual generation.

Figure: Figure 4: (Left): Scaling up training compute yields the classic power-law between $\mathrm{FD}_{\text{DINOv2}}\downarrow$ and training compute, with $K=263$, $\alpha=-0.060$ for 1-step inference, and $K=164$, $\alpha=-0.028$ for 2-step inference, (Right): Given the same batch size and iterations, scaling up model sizes and model FLOPs strongly correlates with $\mathrm{FID}\downarrow$ improvements on ImageNet $64\times 64$. The diameter is proportional to the model size.
Refer to caption: x2.png

## 5 Related Work

#### Consistency Models.

Consistency models (Song et al., , Song and Dhariwal, ) are a new family of generative models designed for efficient generation with few model steps without the need for adversarial training. CMs do not rely on a pretrained diffusion model (DM) to generate training targets but instead leverage an unbiased score estimator.
Training CMs have been extended to multi-step sampling (Kim et al., , Wang et al., , Heek et al., ), latent space models (Luo et al., ), ControlNet (Xiao et al., ), video (Wang et al., ), and combined with an additional adversarial loss (Kim et al., , Kong et al., ).
Despite their sampling efficiency, CMs are typically more challenging to train and require significantly more compute resources than their diffusion counterparts. Our work substantially improves the training efficiency of CMs, reducing the cost of future research and deployment on CMs.
Parallel to our work, Heek et al. (), Wang et al. () relax the single-step constraint of CMs and generalize them to multi-step variants, drawing inspiration from TRACT (Berthelot et al., ).
Under the piecewise linear design, increasing the number of segments will also lead to diffusion pretraining, showing a different approach to connect CMs to the standard diffusion models.

#### Diffusion Distillation.

Drawing inspiration from knowledge distillation (Hinton et al., ), distillation is the most widespread training-based approach to accelerate the diffusion sampling procedure. In diffusion distillation, a pretrained diffusion model (DM), which requires hundreds to thousands of model evaluations to generate samples, acts as a teacher. A student model is trained to match the teacher model’s sample quality, enabling it to generate high-quality samples in a few steps.

There are two main lines of work in this area. The first category involves trajectory matching, where the student learns to match points on the teacher’s sampling trajectory. Methods in this category include
offline distillation (Luhman and Luhman, , Zheng et al., , Geng et al., ), which require an offline synthetic dataset generated by sampling from a pretrained DM to distill a teacher model into a few-step student model;
progressive distillation (Salimans and Ho, , Meng et al., ), and TRACT (Berthelot et al., ), which require multiple training passes or offline datasets to achieve the same goal;
and BOOT (Gu et al., ), Consistency Distillation (CD)(Song et al., ), and Imagine-Flash (Kohler et al., ), which minimize the difference between the student predictions at carefully selected points on the sampling trajectory.

CD is closely related to our method, as it leverages a teacher model to generate pairs of adjacent points and enforces the student predictions at these points to map to the initial data point. However, it employs a fixed schedule derived from a specific sampler, which may introduce non-negligible discretization errors in approximating the consistency condition. It also limits the quality of consistency models to that of the pretrained diffusion model.

The second category minimizes the probabilistic divergence between data and model distributions, i.e., distribution matching (Poole et al., , Wang et al., , Luo et al., , Yin et al., , Zhou et al., ). These methods (Luo et al., , Yin et al., , Nguyen and Tran, , Sauer et al., , Kohler et al., , Xu et al., , Lin et al., , Zhou et al., ) use score distillation or adversarial loss, to distill an expensive teacher model into an efficient student model. However, they can be challenging to train in a stable manner due to the alternating updating schemes from either adversarial or score distillation. Some of these methods such as DreamFusion (Poole et al., ) and ProlificDreamer (Wang et al., ) are used for 3D object generation.

A drawback of training-based approaches is that they need additional training procedures after pretraining to distill an efficient student, which can be computationally intensive. For a detailed discussion on the recent progress of diffusion distillation, we direct the readers to Dieleman ().

#### Fast Samplers for Diffusion Models.

Fast samplers are usually training-free and use advanced solvers to simulate the diffusion stochastic differential equation (SDE) or ordinary differential equation (ODE) to reduce the number of sampling steps. These methods reduce the discretization error during sampling by analytically solving a part of SDE or ODE (Lu et al., ; , Xue et al., ), by using exponential integrators and higher order polynomials for better approximation of the solution (Zhang and Chen, ), using higher order numerical methods (Karras et al., ), using better approximation of noise levels during sampling (Kong and Ping, ), correcting predictions at each step of sampling (Zhao et al., ) and ensuring that the solution of the ODE lies on a desired manifold (Liu et al., ).
Another orthogonal strategy is the parallel sampling process (Pokle et al., , Shih et al., ), solving fixed points of the entire ODE/SDE trajectories. A drawback of these fast samplers is that the quality of samples drastically reduces as the number of sampling steps falls below a threshold such as 10 steps.

## 6 Conclusion

We propose Easy Consistency Tuning (ECT), a simple yet efficient scheme for training consistency models.
The resulting models, ECMs, unlock state-of-the-art few-step generative capabilities at a minimal tuning cost and are able to benefit from scaling.
We have made our [code](https://github.com/locuslab/ect) available to ease future prototyping, studying, and deploying consistency models within the community.

## 7 Limitations

One of the major limitations of ECT is that it requires a dataset to tune DMs to CMs. Recent works developed data-free approaches (Gu et al., , Luo et al., , Yin et al., , Zhou et al., ) for diffusion distillation.
The distinction between ECT and data-free methods is that ECT learns the consistency condition on a given dataset through the self teacher, while data-free methods transfer knowledge from a frozen diffusion teacher.
This feature of ECT can be a potential limitation since the training data of bespoke models are unavailable to the public.
However, we hold an optimistic view on tuning CMs using datasets different from pretraining.

In Appendix C, we discuss a data-free variant of Easy Consistency Distillation (ECD), in which we extend the continuous time formulation to Consistency Distillation, by generating self-synthetic data from CM on-the-fly. However, it only applies to the distillation settings.
Further data-centric research is needed regarding
synthetic data, data composition, and data scaling for consistency models.

## Broader Impacts and Ethics Statement

We propose Easy Consistency Tuning (ECT) that can efficiently train consistency models as state-of-the-art few-step generators, using only a small fraction of the computational requirements compared to current CMs training and diffusion distillation methods.
We hope that ECT will democratize the creation of high-quality generative models, enabling artists and creators to produce content more efficiently.
While this advancement can aid creative industries by reducing computational costs and speeding up workflows, it also raises concerns about the potential misuse of generative models to produce misleading, fake, or biased content.
We conduct experiments on academic benchmarks, whose resulting models are less likely to be misused. Further experiments are needed to better understand these consistency model limitations and propose solutions to address them.

## Reproducibility Statement

We provide extensive details of experimental settings and hyperparameters to reproduce our experimental results in Appendix D. We have provided a zip file of our source code in this submission. We plan to release our code to ensure transparency and reproducibility of the results.

## Acknowledgments

Zhengyang Geng and Ashwini Pokle are supported by funding from the Bosch Center for AI. Zico Kolter gratefully acknowledges Bosch’s funding for the lab. The FLAME Center at CMU generously supports this research.

We appreciate Yang Song for discussions and feedback. We acknowledge the discussions and comments provided by Tianle Cai, Antony Jia, Diyang Xue, Runtian Zhai, and Yonghao Zhuang during the preparation of the [blog post](https://gsunshine.notion.site/Consistency-Models-Made-Easy-954205c0b4a24c009f78719f43b419cc).

## Appendix A Motivations behind Design Choices in ECT

In this section, we expand upon our motivation behind the design decisions for the mapping function, metric, and weighting function used for ECT.

#### Mapping Function.

We first assume that $\Delta t$ is approximately proportional to $t$.
Let $0<c\leq 1$ be this constant of proportionality, then we can write:

$$ $\displaystyle c\approx\frac{\Delta t}{t}=\frac{t-r}{t}=1-\frac{r}{t} \Rightarrow\frac{r}{t}\approx 1-c.$ $$

As training progresses, the mapping function should gradually shrink $\Delta t\to 0$. However, the above parameterization does not achieve this. An alternative parameterization is to decrease $\Delta t$ exponentially.
We assume the ratio between $r$ and $t$ can be written as:

$$ $\frac{r}{t}=1-\frac{1}{q^{a}},$ (17) $$

where $q>1$, $a=\lfloor\text{iters}/d\rfloor$, and $d$ is a hyperparameter controlling how quickly $\Delta t\to{\mathrm{d}}t$. At the beginning of training, $\frac{r}{t}=1-\frac{1}{q^{0}}=0\Rightarrow r=0$, which falls back to DMs. Since we can initialize from the diffusion pretraining, this stage can be skipped by setting $a=\lceil\text{iters}/d\rceil$. As training progresses ($\text{iters}\uparrow$), $\frac{r}{t}\to 1$ leads to $\Delta t\to{\mathrm{d}}t$.

Finally, we adjust the mapping function to balance the prediction difficulties across different noise levels:

$$ $\frac{r}{t}=1-\frac{1}{q^{a}}n(t)=1-\frac{1}{q^{\lceil\text{iters}/d\rceil}}n( t).$ (18) $$

For $n(t)$, we choose $n(t)=1+k,\sigma(-b,t)=1+\frac{k}{1+e^{bt}}$, using the sigmoid function $\sigma$. Since $r\geq 0$, we also clamp $r$ to satisfy this constraint after the adjustment.

Figure: Figure 5: Visualization of $\nicefrac{{r}}{{t}}$ during training. $\Delta t\to{\mathrm{d}}t$ when $\nicefrac{{r}}{{t}}\to 1$.
Refer to caption: x4.png

The intuition behind this mapping function is that the relative difficulty of predicting $f({\mathbf{x}}_{r})$ from ${\mathbf{x}}_{t}$ can vary significantly across different noise levels $t$ when using a linear mapping between $t$ and $r$.

Consider $\nicefrac{{r}}{{t}}=0.9$. At small values of $t$, ${\mathbf{x}}_{t}$ and ${\mathbf{x}}_{r}$ are close, making the alignment of $f({\mathbf{x}}_{t})$ with $f({\mathbf{x}}_{r})$ relatively easy. In contrast, at larger $t$, where ${\mathbf{x}}_{t}$ and ${\mathbf{x}}_{r}$ are relatively far apart, the distance between the predictions $f({\mathbf{x}}_{t})$ and $f({\mathbf{x}}_{r})$ can be substantial. This leads to imbalanced gradient flows across different noise levels, impeding the training dynamics.

Therefore, we downscale $\nicefrac{{r}}{{t}}$ when $t$ is near 0 0 through the mapping function, balancing the gradient flow across varying noise levels.
This prevents the gradient at any noise level from being too small or too large relative to other noise levels, thereby controlling the variance of the gradients.

We direct the reader to Appendix B for details of how to set $q^{\lceil\text{iters}/d\rceil}$.

#### Choice of Metric.

As discussed in Sec. 2, iCT uses pseudo-Huber metric (Charbonnier et al., ) to mitigate the perceptual bias caused by the LPIPS metric (Zhang et al., ),

$$ $\displaystyle L({\mathbf{x}},{\mathbf{y}})=\sqrt{\|{\mathbf{x}}-{\mathbf{y}}\| _{2}^{2}+\epsilon}-\epsilon,\quad\epsilon>0.$ (19) $$

This metric indeed improves the performance of CMs over the classic squared $L_{2}$ loss. When taking a careful look as this metric, we reveal that one of the reasons for this improvement is that this metric is more robust to the outliers compared to the $L_{2}$ metric due to its adaptive per-sample scaling of the gradients.
Let $\Delta={\mathbf{x}}-{\mathbf{y}}$, then the differential of the pseudo-Huber metric can be written as

$$ $\mathrm{d}L=\underbrace{\frac{1}{\sqrt{\|\Delta\|_{2}^{2}+c^{2}}}\vphantom{ \mathrm{d}\left(\frac{1}{2}\|\Delta\|_{2}^{2}\right)}}_{\text{weighting term}} \underbrace{\mathrm{d}\left(\frac{1}{2}\|\Delta\|_{2}^{2}\vphantom{\frac{1}{ \sqrt{\|\Delta\|_{2}^{2}+\epsilon}}}\right)}_{\text{differential of squared }L_{2}\text{\,loss}},$ (20) $$

where we have decomposed the differential of pseudo-Huber loss into an adaptive weighting term and the differential of the squared $L_{2}$ loss.
Therefore, we retain the squared $L_{2}$ metric used in DMs, and explore varying adaptive weighting terms which we explore in detail in Appendix B.

#### Distinction between the training schedules of ECT and iCT.

As noted in Sec. 2, iCT (Song and Dhariwal, ) employs a discrete-time curriculum given by  Eq. 8.
This curriculum divides the noise horizon $[0,T]$ into $N$ smaller consecutive subintervals to apply the consistency loss, characterized by non-overlapping segments $[t_{i},t_{i+1}]$, and gradually increases the number of intervals $N=10\to 1280$.
However, the "boundary" condition of this schedule is to start with the number of intervals to $N=1$, learning a model solely mapping samples at noise levels $T_{\text{max}}$ to the clean data ${\mathbf{x}}_{0}$, largely distinct from the classic diffusion models training.
We instead investigate a continuous-time schedule whose "boundary" condition yields diffusion pretraining, i.e., constructing training pairs of $r=0$ for all $t$ at the beginning.

## Appendix B Exploring Design Space & Scaling of Consistency Models

Due to ECT’s efficiency, we can explore the design space of CMs at a minimal cost. We specifically examine the weighting function, training schedule, and regularization for CMs.

Our most significant finding is that controlling gradient variances and balancing the gradients across different noise levels are fundamental to CMs’ training dynamics. Leveraging the deep connection between CMs and DMs, we also improve the diffusion pretraining and the full pretraining+tuning pipeline using our findings.

#### Weighting Function.

Forward processes with different noise schedules and model parameterizations can be translated into each other at the cost of varying weighting functions (Kingma and Gao, ).
From our experiments on a wide range of weighting schemes, we learn three key lessons.

**Table 2: Performance of ECMs trained with various weighting functions on ImageNet 64$\times$64. We enable the adaptive weighting $w(\Delta)=\nicefrac{{1}}{{(\|\Delta\|_{2}^{2}+c^{2})^{\frac{1}{2}}}}$.**
| $\bar{w}(t)$ | 1-step FID$\downarrow$ | 2-step FID$\downarrow$ |
| --- | --- | --- |
| $1$ | 5.39 | 3.48 |
| $\nicefrac{{1}}{{t}}$ | 17.79 | 3.24 |
| $\nicefrac{{1}}{{(t-r)}}$ | 9.28 | 3.22 |
| $\nicefrac{{1}}{{t}}+\nicefrac{{1}}{{\sigma_{\mathrm{data}}}}$ | 5.68 | 3.44 |
| $\nicefrac{{1}}{{t^{2}}}$ | 190.80 | 20.65 |
| $\nicefrac{{1}}{{t^{2}}}+1$ | 6.78 | 3.12 |
| $\nicefrac{{1}}{{t^{2}}}+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}$ | 5.51 | 3.18 |
| $\nicefrac{{1}}{{(t^{2}+\sigma_{\mathrm{data}}^{2})}}$ | 163.01 | 13.33 |

$(1)$ There is no free lunch for weighting function, i.e., there is likely no universal timestep weighting $\bar{w}(t)$ that can outperform all other candidates on different datasets, models, and target metrics for both 1-step and 2-step generation.

We refer these results to Tab. 2, including $\mathrm{SNR}(t)=\nicefrac{{1}}{{t^{2}}}$, $\mathrm{SNR}(t)+1=\nicefrac{{1}}{{t^{2}}}+1$ (Salimans and Ho, ), EDM weighting $\mathrm{SNR}(t)+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}=\nicefrac{{1}}{{t
^{2}}}+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}$ (Karras et al., ), and Soft-Min-SNR weighting $\nicefrac{{1}}{{(t^{2}+\sigma_{\mathrm{data}}^{2})}}$ (Hang et al., , Crowson et al., ), where $\mathrm{SNR}(t)=\nicefrac{{1}}{{t^{2}}}$ is the signal-to-noise ratio in our setup.

On CIFAR-10, the weighting $\bar{w}(t)=\nicefrac{{1}}{{(t-r)}}$ from the discretization of consistency condition in Eq. 12 achieves the best 1-step FID, while the square root of $\mathrm{SNR}(t)$, $\bar{w}(t)=\sqrt{\mathrm{SNR}(t)}=\nicefrac{{1}}{{t}}$, produces the best $\text{FD}_{\text{DINOv2}}$.
On ImageNet 64$\times$64, considering that we have already had the adaptive weighting $w(\Delta)$, the uniform weighting $\bar{w}(t)\equiv 1$ can demonstrate the best 1-step FID when tuning from EDM2 (Karras et al., ).
In contrast to 1-step FIDs, a wider range of timestep weighting $\bar{w}(t)$ produces close 2-step FIDs for ECMs.

When starting on a new dataset with no prior information, $\bar{w}(t)=\mathrm{SNR}(t)+n$ is a generally strong choice as the default timestep weighting of data prediction models (x-pred). In this situation, this weighting function corresponds to using v-pred (Salimans and Ho, ) or flow matching (Lipman et al., , Liu et al., ) as model parameterization when $n=1$.

**Table 3: Performance of ECMs trained with varying adaptive weightings on ImageNet 64$\times$64.**
| $\bar{w}(t)$ | $w(\Delta)$ | 1-step FID$\downarrow$ | 2-step FID$\downarrow$ |
| --- | --- | --- | --- |
| $\nicefrac{{1}}{{t^{2}}}+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}$ | $1$ | 6.51 | 3.28 |
| $\nicefrac{{1}}{{t^{2}}}+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}$ | $\nicefrac{{1}}{{(\|\Delta\|_{1}+c}})$ | 6.29 | 3.25 |
| $\nicefrac{{1}}{{t^{2}}}+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}$ | $\nicefrac{{1}}{{(\|\Delta\|_{2}^{2}+c^{2})^{\frac{1}{2}}}}$ | 5.51 | 3.18 |
| $1$ | $\nicefrac{{1}}{{(\|\Delta\|_{2}^{2}+c^{2})^{\frac{1}{2}}}}$ | 5.39 | 3.48 |

$(2)$ The adaptive weighting $w(\Delta)$ achieves better results by controlling gradient variance. The adaptive weighting $w(\Delta)$ on a per-sample basis shows uniform improvements on both CIFAR-10 and ImageNet 64$\times$64. See Tab. 3 for the ablation study.

Beyond ECT, we further investigate the role of adaptive weighting $w(\Delta)$ in pretraining on a toy Swiss roll dataset using the parameterization and forward process of flow matching (Lipman et al., ) and an MLP network.

Consider the objective function $w(\Delta)\|v_{\theta}({\mathbf{x}}_{t})-({\mathbf{x}}_{1}-{\mathbf{x}}_{0})\|_
{2}^{2}$, where ${\mathbf{x}}_{t}=(1-t)\cdot{\mathbf{x}}_{0}+t\cdot{\mathbf{x}}_{1}$, $t\sim\mathrm{Uniform}(0,1)$, ${\mathbf{x}}_{1}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, and the adaptive weighting

$$ $w(\Delta)=\frac{1}{(\|\Delta\|_{2}^{2}+\epsilon)^{p}},$ $$

where $p=0$ corresponds to no adaptive weighting. We set $\epsilon=10^{-6}$ and control the strength of gradient normalization by varying $p$ from 0 0 to $1$.

As we increase the strength of adaptive weighting, flow models become easier to sample from in a few steps. Surprisingly, even $p=1$ demonstrates strong few-step sampling results when pretraining the flow model. See Fig. 6 for visualization.

#### Mapping Function.

We compare the constant mapping function with $n(t)\equiv 1$ in Eq. 17 and mapping function equipped with the sigmoid $n(t)$ in Eq. 18. We use $k=8$ and $b=1$ for all the experiments, which transfers well from CIFAR-10 to ImageNet 64$\times$64 and serves as a baseline in our experiments. Though $b=2$ can further improve the 1-step FIDs on ImageNet 64$\times$64, noticed post hoc, we don’t rerun our experiments.

Figure: Figure 6: Influence of adaptive weighting $w(\Delta)=\nicefrac{{1}}{{(\|\Delta\|_{2}^{2}+\epsilon)^{p}}}$ on pretraining using varying $p$.
Refer to caption: x5.png

On CIFAR-10, the constant mapping function with $n(t)\equiv 1$ achieves 1-step FID of 4.06 at $200$k iterations, worse than the 1-step FID of 3.86 by $n(t)=1+\frac{k}{1+e^{bt}}$.
Under our forward process (${\mathbf{x}}_{t}={\mathbf{x}}_{0}+t\cdot\epsilon$) and model parameterization (EDM (Karras et al., )), the constant mapping function incurs training instability on ImageNet 64$\times$64, likely due to the imbalanced gradient flow.

The role of the mapping function, regarding training, is to balance the difficulty of learning consistency condition across different noise levels, avoiding trivial consistency loss near $t\to 0$.
For model parameterizations and forward processes different from ours, for example, flow matching (Lipman et al., , Liu et al., , Sauer et al., ), we advise readers to start from the constant mapping function due to its simplicity.

#### Dropout.

In line with (Song and Dhariwal, ), we find that CMs benefit significantly from dropout (Hinton et al., ).
On CIFAR-10, we apply a dropout of $0.20$ for models evaluated on FID and a dropout of $0.30$ for models evaluated on $\text{FD}_{\text{DINOv2}}$.

On ImageNet 64$\times$64, we note that ECT benefits from a surprisingly high dropout rate. When increasing the dropout rate from $0.10$ to $0.40$, the 2-step FID decreases from $4.53$ to $3.24$.
Increasing the dropout rate further can be helpful for 1-step FID under certain timestep weighting $\bar{w}(t)$, but the 2-step FID starts to deteriorate.
In general, we optimize our model configurations for 2-step generation and choose the dropout rate of $0.40$ for ECM-S.

Finally, we note that the dropout rate tuned at a given weighting function $w(t)$ transfers well to the other weighting functions, thereby reducing the overall cost of hyperparameter tuning.
On ImageNet 64$\times$64, the dropout rate can even transfer to different model sizes. We apply a dropout rate of $0.50$ for all the model sizes of ECM-M/L/XL.

#### Shrinking Δ ⁢ t → d ⁢ t → Δ 𝑡 d 𝑡 \Delta t\to\mathrm{d}t roman_Δ italic_t → roman_d italic_t .

In the mapping function discussed in Sec. 3.3 and Appendix A, we use the hyperparameter $d$ to control the magnitude of $q$, thereby determining the overall rate of shrinking $\Delta t\to\mathrm{d}t$, given by $\left(1-\nicefrac{{1}}{{q}}^{\lceil\text{iters}/d\rceil}\right)$.
In practice, we set $q=2$ and $d=\text{total_iters}//8$ for CIFAR-10 experiments, and $q=4$ and $d=\text{total_iters}//4$ for ImageNet 64$\times$64 experiments, achieving $\nicefrac{{r}}{{t}}\approx 0.99$ at the end of training.

Compared with no shrinkage of $\Delta t$, where $\Delta t\approx\mathrm{d}t$ throughout, we find that shrinking $\Delta t\to\mathrm{d}t$ results in improved performance for ECMs.
For example, on CIFAR-10, starting ECT directly with $\Delta t\approx\mathrm{d}t$ by setting $q=256$ (corresponding to $\nicefrac{{r}}{{t}}\approx 0.99$) leads to quick improvements in sample quality initially but slower convergence later on. The 1-step FID drops from 3.60 to 3.86 using the same 400k training iterations compared to gradually shrinking $\Delta t\to\mathrm{d}t$.
On ImageNet 64$\times$64, $\Delta t\approx\mathrm{d}t$ with $q=256$ from the beginning results in training divergence, as the gradient flow is highly imbalanced across noise levels, even when initializing from pretrained diffusion models.

This observation suggests that *ECT’s schedule should be adjusted according to the compute budget*. At small compute budgets, as long as training stability permits, directly approximating the differential consistency condition through a small $\Delta t\approx\mathrm{d}t$ leads to fast sample quality improvements. For normal to rich compute budgets, shrinking $\Delta t\to\mathrm{d}t$ generally improves the final sample quality, which is the recommended practice.

Using this feature of ECT, we demonstrate its efficiency by training ECMs to surpass previous Consistency Distillation, which took hundreds of GPU hours, using one hour on a single A100 GPU.

#### Training Generative Models in 1 GPU Hour.

Deep generative models are typically computationally expensive to train. Unlike training a classifier on CIFAR-10, which usually completes within one GPU hour, leading generative models on CIFAR-10 as of 2024 require days to a week to train on 8 GPUs. Even distillation from pretrained diffusion models can take over a day on 8 GPUs or even more, equivalently hundreds of GPU hours.

To demonstrate the efficiency of ECT and facilitate future studies, we implemented a fast prototyping setting designed to yield results within one hour on a single GPU. This configuration uses a fixed $\Delta t\approx\mathrm{d}t$ by setting $q=256$ in our mapping function (corresponding to $\frac{r}{t}\approx 0.99$), which allows for quick approximation of the differential consistency condition. Through $8000$ gradient descent steps at batch size of $128$, within 1 hour on a single A100 40GB GPU, ECT achieves a 2-step FID of $2.73$, outperforming Consistency Distillation (2-step FID of $2.93$) trained with 800k iters at batch size 512 and LPIPS (Zhang et al., ) metric.

#### Ablation on Pretraining

We conducted a controlled experiment, training CMs from scratch following iCT best practices and tuning EDM using both iCT and ECT schedules. The combined pretraining and fine-tuning cost for ECT is $50\%+6.25\%$ of iCT trained from scratch. Results are presented in Tab. 4.

There is a noticeable performance gap between iCT’s training and ECT’s pretraining+tuning scheme, particularly when evaluating models using modern metrics like $\mathrm{FD}_{\text{DINOv2}}$ (Stein et al., ). $\mathrm{FD}_{\text{DINOv2}}$ employs the representation space of the DINOv2-L model (Oquab et al., ) to compute distributional distance, which has been shown to better align with human evaluations.

**Table 4: Impact of pretraining on model performance**
| Model | $\mathrm{FD}_{\text{DINOv2}}$ |
| --- | --- |
| iCT | 242.30 |
| Pretraining + iCT tuning | 200.31 |
| Pretraining + ECT tuning | 190.13 |
| EDM | 168.16 |

When scaling up the pretraining+tuning cost to match the overall cost of iCT in class-conditional settings, ECM achieves a $\mathrm{FD}_{\text{DINOv2}}$ of 152.21, significantly outperforming the iCT model trained from scratch (205.11). For context, StyleGAN-XL achieves an $\mathrm{FD}_{\text{DINOv2}}$ of 204.60.

**Table 5: Generative performance on class-conditional CIFAR-10.**
| Method | $\text{FD}_{\text{DINOv2}}$$\downarrow$ | NFE$\downarrow$ |
| --- | --- | --- |
| GANs |  |  |
| BigGAN (Brock et al., ) | 326.66 | 1 |
| StyleGAN2-ADA (Karras et al., ) | 305.92 | 1 |
| StyleGAN-XL (Sauer et al., ) | 204.60 | 1 |
| Diffusion Models |  |  |
| EDM (Karras et al., ) | 145.20 | 35 |
| PFGM++ (Xu et al., ) | 141.65 | 35 |
| ECT |  |  |
| ECM (ECT Pretrained) | 121.05 | 35 |
| ECM (Tuned) | 198.51 | 1 |
| ECM (Tuned) | 128.63 | 2 |

#### Ablation on Tuning Design Choices

Besides quantitatively evaluating the pretraining impact, we conduct further ablation studies regarding fine-tuning design choices. Due to the computational constraints, we tune EDM2-S using iCT’s design choices as the baseline. Results are presented in Tab. 6. These results demonstrate the improvements of the continuous-time training schedule, increased dropout, and weighting functions on ECMs’ generative performance.

**Table 6: Ablation study of continuous-time CMs on ImageNet 64$\times$64**
| Methods | 1-step FID | 2-step FID |
| --- | --- | --- |
| iCT + EDM2 Pretraining | 21.09 | 4.39 |
| + Continuous-time schedule | 14.34 | 4.33 |
| + Dropout = 0.40 | 9.28 | 3.22 |
| + $\bar{w}(t)=\nicefrac{{1}}{{t^{2}}}+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}$ | 5.51 | 3.18 |

#### Improving Pretraining using Findings in Tuning.

The exploration of the design space through the tuning stage as a proxy led to a question: Can the insights gained during tuning be applied to improve the pretraining stage and, consequently, the entire pretraining+tuning pipeline for CMs? The results of our experiments confirmed this hypothesis.

For the largest $\Delta t=t$, ECT falls back to diffusion pretraining with $r=0$ and thus $f_{\theta}({\mathbf{x}}_{r})={\mathbf{x}}_{0}$.
We pretrain EDM (Karras et al., ) on the CIFAR-10 dataset using the findings in ECT. Instead of using EDM weighting, $\mathrm{SNR}(t)+\nicefrac{{1}}{{\sigma_{\mathrm{data}}^{2}}}$, we enable the adaptive weighting $w(\Delta)$ with $p=\nicefrac{{1}}{{2}}$ and smoothing factor $c=0$ and a timestep weighting $\bar{w}(t)=\nicefrac{{1}}{{t}}$.

Compared with the EDM baseline, the recipe from ECT brings a convergence acceleration over $2\times$ regarding $\text{FD}_{\text{DINOv2}}$, matching EDM’s final performance using less than half of the pretraining budget and largely outperforming it at the full pretraining budget.

EDM pretrained by ECT achieves $\text{FD}_{\text{DINOv2}}$ of $150.39$ for unconditional generation and $121.05$ for class-conditional generation, considerably better than the EDM baseline’s $\text{FD}_{\text{DINOv2}}$ of $168.17$ for unconditional generation and $145.20$ for class-conditional generation, when using the same pretraining budget and inference steps (NFE=35).

#### Influence of Pretraining Quality.

Using ECT pretrained models ($\text{FD}_{\text{DINOv2}}$ of $121.05$) and original EDM (Karras et al., ) ($\text{FD}_{\text{DINOv2}}$ of $145.20$), we investigate the influence of pretraining quality on consistency tuning and resulting ECMs.
Our experiments confirm that better pretraining leads to easier consistency tuning and faster convergence.
At the same budget of $204.8\mathrm{M}$ images, tuning from ECT pretrained models achieves $\text{FD}_{\text{DINOv2}}$ of $128.63$, better than $\text{FD}_{\text{DINOv2}}$ of $152.21$ from EDM.

ECM from the ECT pretraining surpasses SoTA GANs in 1 sampling step and advanced DMs in 2 sampling steps, only slightly falling behind our pretrained models and setting up a new SoTA for the modern metric $\text{FD}_{\text{DINOv2}}$. Results can be found in Tab. 5.

On ImageNet 64$\times$64, ECM-M, initialized from EDM2-M (Karras et al., ), deviates from the power law scaling and achieves better generative performance than the log-linear trend. (See Fig. 4, Right). We speculate that it is due to a higher pretraining budget, in which EDM2-M was pretrained by $2\times$ training images compared with other model sizes (S/L/XL).

#### Differences between 1-step and 2-step Generation.

Our empirical results suggest that the training recipe for the best 1-step generative models can differ from the best few-step generative models in many aspects, including weighting function, dropout rate, and EMA rate/length.
Fig. 7 shows an example of how FIDs from different numbers of function evaluations (NFEs) at inference vary with dropout rates.

In our setups, starting from a proper model size, the improvements from 2-step sampling seem larger than doubling the model size but keeping 1-step sampling.
In the prior works, iCT (Song and Dhariwal, ) employs $2\times$ deeper model, but the 1-step generative performance can be inferior to the 2-step results from ECT.
This finding is consistent with recent theoretical analysis (Lyu et al., ), which indicates a tighter bound on the sample quality for the 2-step generation compared to the 1-step generation.

Figure: Figure 7: Relationship between the dropout and FIDs for models trained on CIFAR-10 with varying numbers of function evaluations (NFE) at inference.
Refer to caption: x6.png

#### Pareto Frontier & Scaling Law.

The Pareto Frontier reveals a seemingly power law scaling behavior. Training configurations not optimized for the current compute budget, i.e., not on the Pareto Frontier, deviate from this scaling. Simply scaling up the training compute without adjusting other parameters may result in suboptimal performance. In our compute scaling experiments, we increased the batch size and enabled the smoothing factor $c$ in the adaptive weighting to maintain this trend.

## Appendix C Extension to Consistency Distillation.

#### Continuous-time Consistency Distillation

Consistency Distillation (CD) (Song et al., ) uses a fixed discrete schedule derived from a specific sampler, such as the EDM sampler. This approach inherently limits $\Delta t\to{\mathrm{d}}t$, therefore causing a non-trivial discretization error on approximating the differential consistency condition. Building upon ECT, we extend our continuous-time schedule to Consistency Distillation, which we term Easy Consistency Distillation (ECD).

On ImageNet 64$\times$64, we implement Consistency Distillation as the baseline using pretrained EDM2-S (Karras et al., ) with weighting functions, noise distribution, and dropout. The results, shown in Tab. 7, demonstrate that continuous-time distillation improves upon the standard CD approach.

Given well-pretrained DMs, ECD typically demonstrates a performance advantage over ECT at smaller batch sizes (e.g., 128), primarily due to the variance reduction in approximating the score function:

$$ $\nabla_{{\mathbf{x}}_{t}}\log p({\mathbf{x}}_{t})=\mathbb{E}[\nabla_{{\mathbf{ x}}_{t}}\log p({\mathbf{x}}_{t}|{\mathbf{x}}_{0})|{\mathbf{x}}_{t}]$ $$

This advantage stems from the teacher models, which provide a variance-reduced estimate of the score function compared to the Monte Carlo estimation used in ECT.
However, this gap tends to narrow as we scale up to larger model sizes.
Scaling compensates for the limited training budget and variances, closing the gap between the two.

**Table 7: Easy Consistency Distillation (ECD) on ImageNet 64$\times$64 using the same budget of $12.8\mathrm{M}$ training images (batch size 128 and 100k iterations) as ECT in Tab. 1.**
| Methods | 1-step FID | 2-step FID |
| --- | --- | --- |
| CD-S | 8.18 | 3.71 |
| ECD-S | 3.33 | 2.10 |
| ECD-M | 2.78 | 1.92 |
| ECD-XL | 2.54 | 1.77 |

Consequently, while ECD may outperform ECT in resource-constrained scenarios or with smaller models, the distinction becomes less pronounced as we move to larger scales. This observation underscores the scaling factor (computational resources and model size) when choosing between ECT and ECD for a given application.

#### Data-Free ECD.

Inspired by the recent progress on data-free distillation (Gu et al., , Luo et al., , Yin et al., , Zhou et al., ), we explore a data-free variant of Easy Consistency Distillation (ECD).
Instead of sampling data points ${\mathbf{x}}_{0}$ from a given dataset $\mathcal{D}$, we generate synthetic data points directly from the CM itself on the fly, ${\mathbf{x}}_{0}=f_{\operatorname{sg}(\theta)}(\bm{\epsilon}^{\prime},T)$, where $\bm{\epsilon}\textquoteright\sim p(\bm{\epsilon})$, $T$ is the maximum noise level for sampling, followed by the same ECD training step over the self synthetic data. This eliminates the need for a distillation dataset or the construction of synthetic data from the teacher model (Luhman and Luhman, , Geng et al., , Zheng et al., ).

On ImageNet 64$\times$64, this data-free ECD achieves 1-step FID of 4.38 and 2-step FID of 2.77, comparable to the data-dependent ECD and ECT schemes at the same budget. It suggests that data-free ECD can be a competitive alternative in scenarios where access to large datasets is limited or unavailable.
We summarize both ECD and its data-free variant in Alg. 2.

Figure: Algorithm 2 Easy Consistency Distillation (ECD) / Data-Free ECD

## Appendix D Experimental Details

#### Model Setup.

For both unconditional and class-conditional CIFAR-10 experiments, we initial ECMs from the pretrained EDM (Karras et al., ) of DDPM++ architecture (Song et al., ).
For class-conditional ImageNet 64$\times$64 experiments, we initial ECM-S/M/L/XL, ranging from $280\mathrm{M}$ to $1.1\mathrm{B}$, from the pretrained EDM2 (Karras et al., ).
Detailed model configurations are presented in Tab. 8.

We follow (Karras et al., , Song et al., ) and set $c_{\text{skip}}(t)=\nicefrac{{\sigma_{\text{data}}^{2}}}{{(t^{2}+\sigma_{\text
{data}}^{2})}}$ and $c_{\text{out}}(t)=\nicefrac{{t\sigma_{\text{data}}}}{{\sqrt{t^{2}+\sigma_{
\text{data}}^{2}}}}$, where $\sigma_{\text{data}}^{2}$ is the variance of (normalized) data, and set to $0.5$ for both CIFAR-10 and ImageNet 64$\times$64.

#### Computational Cost.

ECT is computationally efficient. On ImageNet 64$\times$64, the tuning stage of ECT requiring only $0.39\%$ of the iCT (Song and Dhariwal, ) training budget, and $0.60\%$ to $1.91\%$ of the EDM2 (Karras et al., ) pretraining budget depending on the model sizes.
The exact computational resources required to train each individual model are shown in Tab. 8.

Table: Table 8: Model Configurations and Training Details for unconditional and class-conditional ECMs on CIFAR-10, and ECM-S/M/L/XL on ImageNet 64$\times$64.

#### Training Details.

We use RAdam (Liu et al., ) optimizer for experiments on CIFAR-10 and Adam (Kingma and Ba, ) optimizer for experiments on ImageNet 64$\times$64.
We set the $\beta$ to ($0.9$, $0.999$) for CIFAR-10 and ($0.9$, $0.99$) for ImageNet 64$\times$64. All the hyperparameters are indicated in Tab. 8.
We do not use any learning rate decay, weight decay, or warmup on CIFAR-10. We follow EDM2 (Karras et al., ) to apply an inverse square root learning rate decay schedule on ImageNet 64$\times$64.

On CIFAR-10, we employ the traditional Exponential Moving Average (EMA). To better understand the influence of the EMA rate, we track three Power function EMA (Karras et al., ) models on ImageNet 64$\times$64, using EMA lengths of $0.01$, $0.05$, and $0.10$. The multiple EMA models introduce no visible cost to the training speed. Considering our training budget is much smaller than the diffusion pretraining stage, we didn’t perform Post-Hoc EMA search as in EDM2 (Karras et al., ).

Experiments for ECT are organized in a non-adversarial setup to better focus and understand CMs and avoid inflated FID (Stein et al., ).
We conducted ECT using full parameter tuning in this work, even for models over $1\mathrm{B}$ parameters.
Investigating the potential of Parameter Efficient Fine Tuning (PEFT) (Hu et al., ) can further reduce the cost of ECT to democratize efficient generative models, which is left for future research.

We train multiple ECMs with different choices of batch sizes and training iterations.
By default, experiments on ImageNet 64$\times$64 utilize a batch size of $128$ and $100$k iterations, leading to a training budget of $12.8\mathrm{M}$.
We have individually indicated other training budgets alongside the relevant experiments, wherever applicable.

#### Sampling Details.

We apply stochastic sampling for 2-step generation.
For 2-step sampling, we follow (Song and Dhariwal, ) and set the intermediate $t=0.821$ for CIFAR-10, and $t=1.526$ for ImageNet 64$\times$64.

Intriguingly, these sampling schedules, originally developed for iCT, also perform well with our ECMs. This effectiveness across different CMs and training methods likely links to the inherent characteristics of the datasets and the forward process. Developing a scientific approach to determine optimal intermediate sampling schedules for CMs remains an open research problem.

#### Evaluation Metrics.

For both CIFAR-10 and ImageNet 64$\times$64, FID and $\text{FD}_{\text{DINOv2}}$ are computed using $50$k images sampled from ECMs. As suggested by recent works (Stein et al., , Karras et al., ), $\text{FD}_{\text{DINOv2}}$ aligns better with human evaluation. We use dgm-eval(^1^11[https://github.com/layer6ai-labs/dgm-eval](https://github.com/layer6ai-labs/dgm-eval/tree/master)) to calculate $\text{FD}_{\text{DINOv2}}$ (Stein et al., ) to ensure align with previous practice.

#### Visualization Setups.

Image samples in Fig. 3 are from class bubble (971), class flamingo (130), class golden retriever (207), class space shuttle (812), classs Siberian husky (250), classs ice cream (928), class oscilloscope (688), class llama (355), class tiger shark (3).

Each triplet (left-to-right) includes from $2$-step samples from ECM-S trained with $12.8\mathrm{M}$ images, ECM-S trained with $102.4\mathrm{M}$ images, and ECM-XL trained with $102.4\mathrm{M}$ images.

#### 1 GPU Hour Prototyping Settings.

This configuration uses a fixed $\Delta t\approx\mathrm{d}t$ by setting $q=256$ in our mapping function (corresponding to $\frac{r}{t}\approx 0.99$) and an EMA rate of 0.9993 for the model parameters. Using these settings, we run $8000$ gradient descent steps with a batch size of 128 on a single A100 40GB GPU.

#### Scaling of Training Compute.

For the results on scaling laws for training compute on CIFAR-10 shown in Fig. 4 (Left), we train 6 class-conditional ECMs, each with varying batch size and number of training iterations.
All ECMs in this experiment are initialized from the pretrained class-conditional EDM.

The minimal training compute at $2^{0}$ scale corresponds to a total budget of $12.8\mathrm{M}$ training images. The largest training compute at $2^{5}$ scale utilizes a total budget of $409.6\mathrm{M}$ training images, at $2\times$ EDM pretraining budget.

The first two points of $2^{0}$ and $2^{1}$ on Fig. 4 (Left) use a batch size of $128$ for $100$k and $200$k iterations, respectively.
The third point of $2^{2}$ corresponds to ECM trained with batch sizes of $256$ for $200$k iterations.
The final three points of $2^{3}$, $2^{4}$, and $2^{5}$ correspond to ECM trained with a batch size of $512$ for $200$k, $400$k, and $800$k iterations, respectively, with the smoothing factor $c=0.03$ enabled in the adaptive weighting $w(\Delta)$.
We use $\bar{w}(t)=\nicefrac{{1}}{{t}}$ as the timestep weighting function to train all these models as this $\bar{w}(t)$ achieves good performance on $\text{FD}_{\text{DINOv2}}$.

#### Scaling of Model Size and Model FLOPs.

We include details of model capacity as well as FLOPs in Tab. 8 to replicate this plot on ImageNet 64$\times$64.

On ImageNet 64$\times$64, we scale up the training budgets of ECM-S and ECM-XL from $12.8\mathrm{M}$ (batch size of $128$ and $100$k iterations) to $102.4\mathrm{M}$ (batch size of $1024$ and $100$k iterations). We empirically find that scaling the base learning rate by $\sqrt{n}$ works well when scaling the batch size by a factor of $n$ when using Adam (Kingma and Ba, ) optimizer.

## Appendix E Qualitative Results

We provide some randomly generated 2-step samples from ECMs trained on CIFAR-10 and ImageNet-$64\times 64$ in Fig. 8 and Fig. 9, respectively.

Figure: Figure 8: 2-step samples from class-conditional ECM trained on CIFAR-10. Each row corresponds to a different class.
Refer to caption: extracted/5917567/imgs/cond_CIFAR_FD.png

Figure: Figure 9: 2-step samples from class-conditional ECM-XL trained on ImageNet 64$\times$64. Each row corresponds to a different class.
Refer to caption: extracted/5917567/imgs/ECM-XL.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. Pseudo-Huber metric is adopted in the model. Is there any other alternative?
2. In the experiments, it is better to provide more metrics other than FID only.
soundness: 2
strengths: The approach is very efficent as they showed. It could be used to greatly mprove efficiency and
performance of CMs at a large scale.
summary: In this paper, the athors made the point that diffusion models can be viewed as a special case of CMs. Based on it, they fine-tuned a consistency model starting from a pretrained diffusion model and progressively approximate the full consistency condition to stronger
degrees over the training process. The experiments verified its efficiency.
weaknesses: The main motivation of the paper is straightforward. It is hard for the reader to fullly trust their obverstaion that  diffusion models can be viewed as a special case of CMs in practice. The data sets and metircs on the images generation are limited. More extensive experiments or analysis should be conducted to justify their claims.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: In addition to my questions already mentioned in the paper’s weaknesses, I have a few other minor questions and clarifications:
- I do not count this as a weakness since CIFAR-10 and ImageNet 64×64 are standard benchmarks, but have the authors also considered evaluating their method on higher dimensional datasets (e.g., ImageNet 512x512, LAION) and/or latent diffusion models (e.g., Stable Diffusion)? It would certainly be interesting and further strengthen results to position ECT as the de facto standard training regime for CMs more generally.
- Why do the authors use FD-DINOv2 for Figure 4 (left) and use FID for Figure 4 (right)? It seems inconsistent to use different metrics here.
- In Figure 2, why is the diffusion pre-training compute for both CD and ECT less than that for SDE/DM? Are the authors using a different DM?

Overall, should the authors adequately address my concerns and questions, I will consider increasing my score.

---
As per my comment below:

Thank you to the authors for continuing to engage in discussion and for acting in good faith throughout this rebuttal process by incorporating my feedback into the updated manuscript. The novelty point has been clarified, and the improved separation of pedagogical and methodological content between Sections 2 and 3 helps better focus Section 3 on the paper's contributions. I believe the authors have addressed most of my concerns/questions, so I will increase my score to 8. With the method's simplicity, combined with its strong empirical results, my hope is that the community will build on this framework for training consistency models.
soundness: 3
strengths: **Results.**
The primary strength of this work lies in its empirical results, achieving state-of-the-art performance on standard image generation benchmarks. Overall, the experimental analysis is quite comprehensive by comparing against (and outperforming) recent and strong baseline methods in Table 1, and ablating some key design choices in the appendix. The ECT scaling laws in Section 4.2 are an interesting and underexplored direction in CMs, and the authors provide evidence of improved sample quality with increased training compute. If this were to also hold true for both higher dimensional datasets (e.g., ImageNet 512x512, LAION) and latent DMs (e.g., Stable Diffusion), it would have important implications for few-step generation at scale.

**Insights.**
Furthermore, some of the insights provided in the methodology of this paper are quite interesting. Specifically, Section 3.2 provides some interesting discussion on the limitations of training CMs from scratch, dubbed the “curse of CMs”, by bounding the single-step prediction error as a function of the granularity of discretization.
summary: The authors propose easy consistency tuning (ECT), a method to train consistency models (CMs) using less compute than competing methods by initializing them from pre-trained diffusion models (DMs) and then fine-tuning them for few-step generation. The fine-tuning step enables faster convergence compared to training CMs from scratch (consistency training, CT) and avoids accumulating errors from a frozen teacher DM compared to consistency distillation (CD). Training CMs with ECT achieves state-of-the-art performance on both CIFAR-10 and ImageNet 64×64. The authors additionally explore scaling laws in the fine-tuning step, demonstrating performance gains with increased training compute.
weaknesses: **Novelty.**
Despite the strong results demonstrated in this work, I have questions about its novelty. The main methodological contribution is to initialize CM training (CT) with a pre-trained DM to enable faster convergence. In the setting of consistency distillation (CD) and DM distillation in general, it is already common practice to initialize the student from the weights of the teacher DM, effectively reducing distillation to a fine-tuning task to reduce computational requirements and facilitate convergence, similar to what’s being motivated here. The difference in this work compared to CD is to instead initialize a CM from a DM in the setting of CT, which alleviates DM error accumulation as in the case of CD. However, this seems like a rather simple extension of CMs and the connection between CMs and DMs does not seem particularly novel, so I would ask the authors to clarify their proposed novelty here.

Moreover, the authors formulate CMs in continuous-time which was already done in the seminal CM work by Song et al. in Appendix B.2, so it’s not clear what the difference is here if any. Along this vein, I would also ask the authors to clarify the purpose of Section 3.1 and why it’s necessary to formulate CMs in terms of the differential condition since it currently reads as a seemingly disconnected derivation and it’s not clear from reading the paper why this formulation is needed for the rest of their method instead of using the standard discrete/continuous CM formulation by Song et al. In addition, the particular weighting function derived in Section 3.1, $w(t,r)=\frac{1}{t-r}$, does not seem to be what the authors actually end up using in practice since, in Section 3.3, the authors go on to say “Instead, we consider decoupled weighting functions without relying on $p(r|t)$” in L311-L312. Can the authors please clarify these points and also update the manuscript accordingly.

**Ablations.**
Given that the central contribution of ECT is initializing CMs from a DM, it would be important to see the performance of training ECT from scratch (i.e., w/o initializing from a DM) while keeping all other ECT design choices fixed, namely those outlined in Section 3.3 in terms of training schedule, loss function, and weighting function. Can the authors please provide this result in their rebuttal as it is a relevant ablation?

**Easiness.**
The authors claim that their method makes CMs easy to train but, from a design perspective, various choices (e.g., Equation 15) do not necessarily seem more intuitive or “easier” compared to those in the original CT or in the follow-up iCT. I think it would be fair to say that ECT makes training CMs more accessible and more computationally efficient but it’s not clear that “easy” is the right characterization. Can the authors please clarify their position?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: I think all of my questions are presented in the “weaknesses” part. If you can solve most of them well, I will raise my score.
soundness: 3
strengths: 1. This paper explores the concept of the "curse of consistency," which presents an intriguing perspective.
2. The method proposed in this paper achieves good performance.
3. This paper discusses the “scaling laws”.
summary: This paper aims to improve the training efficiency of CMs by proposing three techniques: "continuous-time schedule," "dropout," and "weighting function." The resulting method demonstrates good performance. Additionally, the paper explores a phenomenon called the "curse of consistency" as well as the scaling laws of ECT.
weaknesses: 1. While the "curse of consistency" is indeed fascinating, discussing only the upper bound fails to capture the true nature of errors. An increase in the upper bound does not necessarily indicate a corresponding increase in error.
2. Since the primary advantage of your method lies in its training speed, I believe you may have overlooked an important scenario: "pretraining + iCT tuning," as illustrated in Figure 2.
3. The relationship between your primary observation and your main method is not clear. Specifically, I find it difficult to understand the necessity of proposing a "continuous-time training schedule" and a "weighting function" to address the "curse of consistency" problem. If none of your techniques are linked to the "curse of consistency," what is the rationale for mentioning it, and how does this discussion relate to the overall objectives of your paper? In fact, the "curse of consistency" is not even addressed in your abstract.
4. You should place the ablation studies of your techniques (Table 6) in your main paper instead of the appendix.
5. The "dropout" technique appears to be quite significant; could you clarify why it is placed in the appendix?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: Given the above weaknesses but also the potential impact of its contribution, I think that this paper barely misses the acceptance threshold of ICLR, hence my recommendation of "5: marginally below the acceptance threshold". My questions are detailed in the "Weaknesses" section. I am willing to update my score following the authors' response as I believe most of them can be answered with a revision of the paper during the discussion period.

I would have two additional questions.
- Consistency distillation is tested following the original setup of Song et al. (2023). How would it benefit from later improvements (even though presented only in the setting of consistency training) of iCT (Song & Dhariwal, 2023)?
- I would be interested in hearing the authors' opinion on whether the presented model is a distillation model or not. This is not discussed explicitly in the paper.

----
### Post-rebuttal update

The authors have addressed many of my concerns, so I choose to now recommend an "accept". **Nonetheless**, in case the paper is accepted, **I would like the authors to further improve the paper as suggested in the thread below**.
soundness: 3
strengths: The paper's strengths are immediate: by presenting a simple, easily actionable method to improve the performance of consistency models while significantly reducing their training cost, **the presented method has the potential to be used as a strong baseline for future research** in consistency models.

The paper is overall **well written**. The method is sufficiently motivated with the described insights on consistency training discretization, making diffusion pre-training an **organic improvement**. The **clarity** of its exposition, its available codebase -- that I could not assess in detail -- and, perhaps more importantly, its **simplicity** are important factors in its reusability.

The experimental results confirm the appeal of the new consistency training recipe with not only noticeable performance boosts, but also significant **efficiency gains**. The efficiency advantage is not only significant compared to standard consistency training, but even stronger compared to consistency distillation. Finally, even though no comparison with another method is provided, the presented scaling laws are an interesting addition.
summary: This paper describes a new recipe for consistency model training. By noticing that consistency training at high discretization levels amounts to the training of a diffusion model, the authors propose to initialize the consistency model with the weights of a pre-trained diffusion model. The actual consistency model training phase is then adapted to interpolate between diffusion and continuous-time consistency training. Supplemented by additional changes to the metric and its weighting, this new recipe is shown to present favorable experimental properties: outperforming consistency distillation and training, as well as diffusion models, with a fraction of the training cost. This efficiency is highlighted in a study of the scaling laws of the proposed method.
weaknesses: While I believe this paper provides a valuable and actionable contribution, a few weaknesses, listed below in decreasing order of importance, prevent me from providing a positive recommendation.

## Clarification of the novelty of some contributions

The idea of **initializing consistency models with a pre-trained diffusion model has already been considered** in the initial paper of Song et al. (2023, Appendix B.3), albeit in a restricted setting and without the experimental value highlighted in the present paper. Still, this should be discussed and may have consequences on the empirical study (see next weakness).

Section 3.1 remains **vague whether the presented derivations are contributions of the present paper** or reformulated background from consistency models. My understanding is that these derivations, including the final loss function, are all already included in the original paper of Song et al. (2023): the differential equation appears in another form in their Appendix Remark 4, and the loss function closely resembles the original loss function (by removing scheduling information). This should be clarified.

**The choice of weighting function appears as a reformulation** of Song & Dhariwal (2023), unless I misunderstood its description in the paper. The insight of the pseudo-Huber metric providing an adaptive weighting term is interesting, but I do not see how the proposed alternative (using the $\ell_2$ as metric and modulate it with the weighting that the pseudo-Huber metric would have brought) is any different. Could the authors clarify this?

## Lacking experimental insights

While appealing, the presented experimental results lack additional insights and ablations to fully support the claims of the paper and highlight its added value w.r.t. prior work. Two main components are missing.

The first missing component is a **full fledged ablation study**. One is already included in Appendix Section B (Table 5). However, given this preliminary result showing that the simple diffusion pre-training (already considered in prior work, cf. previous weakness) brings most of the improvements, I think the paper would benefit from including in its main part an augmented ablation study in the same experimental setting as Section 4. This ablation study should then be discussed to assess the significance of each contribution in the new training recipe.

The second missing component is a **study of the evolution of the methods' performance w.r.t. training time/iterations**. As is, baseline and ablation results are presented at a fixed number of iterations. Plotting their performance w.r.t. the number of iterations would enable a more comprehensive efficiency comparison, as well as justify the choice of stopping time for baselines in e.g. Table 1.

## Lack of moderation and rigor in some assertions

This is a less important issue that still deserves to be addressed. The following statements lack moderation and/or rigor.
- In Section 1, it is stated that "the speedup achieved by these sampling techniques usually comes at the expense of the quality of generated samples". This is partially incorrect as some works like the one of Karras et al. (2022) significantly reduced the number of required model evaluations while maintaining performance.
- It is incorrect that the proposed loss function of Eq. (12) "generalizes Song et al. (2023)’s discrete training schedule to continuous time". The proposed loss indeed relies on discretizing the aforementioned differential equation. Instead, this loss is exactly the consistency training loss of Song et al. (2023) untied from its specific discretization grid.
- Writing $\Delta t \to \mathrm{d}t$ instead of $0$ is confusing.
- In Section 4.1, the proposed model is said to "only [require] 1/1000 of [Score SDE's] inference cost". This is correct but misleading as state-of-the-art diffusion models no longer require thousands of model evaluations. This statement should be toned down.

## Minor issues

- The diffusion ODE of Eq. (2), from Karras et al. (2022), requires that $\mathbf{f} = 0$ in Eq. (1).
- $f(\mathbf{x}_t, t)$ is said to be "a denoising function" after Eq. (3). This statement should be more detailed.
- To my knowledge, in Karras et al. (2022)'s timestep schedule, $\rho = 7$ instead of $0.7$ on line 134.
- To my understanding, a part of the numbers in Table 1 were not obtained by the authors but are reported from other papers. This should be clarified.
- $\epsilon$ in Eq. (19) should be $c^2$ to be consistent with the rest of the paper.

Meta-review: This paper explores a method to train consistency models by initializing them from pre-trained diffusion models and then fine-tuning. Various tricks are employed in the training to improve convergence and efficiency of the training, resulting in SOTA performance on small-scale image generation baselines with low cost. Scaling laws are also discussed.

Reviewers were initially critical of some of the presentation choices and felt that prior work had not been sufficiently acknowledged. They wanted the novelty of the proposed approach clarified in comparison to past work. Reviewers were also concerned about the lack of experimental insights and ablations, along with several other more minor issues. Largely these concerns were addressed in the rebuttal phase, with better referencing and reorganization of the paper’s structure, as well as new experiments that filled in existing gaps in the narrative. 

I am recommending acceptance as a Poster, but ask the authors to ensure that the reviewer’s concerns on the writing is taken into account for the final version (especially the results of discussion with LAzX and EoPV on clarifying prior work).

Decision: Accept (Poster)