Multistep Consistency Models

## Abstract

Abstract Diffusion models are relatively easy to train but require many steps to generate samples. Consistency models are far more difficult to train, but generate samples in a single step. In this paper we propose Multistep Consistency Models: A unification between Consistency Models (Song et al., 2023 ) and TRACT (Berthelot et al., 2023 ) that can interpolate between a consistency model and a diffusion model: a trade-off between sampling speed and sampling quality. Specifically, a 1-step consistency model is a conventional consistency model whereas a ∞ \infty ∞ -step consistency model is a diffusion model. Multistep Consistency Models work really well in practice. By increasing the sample budget from a single step to 2-8 steps, we can train models more easily that generate higher quality samples, while retaining much of the sampling speed benefits. Notable results are 1.4 FID on Imagenet 64 in 8 step and 2.1 FID on Imagenet128 in 8 steps with consistency distillation, using simple losses without adversarial training. We also show that our method scales to a text-to-image diffusion model, generating samples that are close to the quality of the original model.

## 1 Introduction

Diffusion models have rapidly become one of the dominant generative models for image, video and audio generation (Ho et al., ; Kong et al., ; Saharia et al., ). The biggest downside to diffusion models is their relatively expensive sampling procedure: whereas training uses a single function evaluation per datapoint, it requires many (sometimes hundreds) of evaluations to generate a sample.

Figure: Figure 1: This figure shows that Multistep Consistency Models interpolate between (single step) Consistency Models and standard diffusion. Top at $t=0$: the data distribution which is a mixture of two normal distributions. Bottom at $t=1$: standard normal distribution. Left to right: the sampling trajectories of (1, 2, 4, $\infty$)-step Consistency Models (the latter is in fact a standard diffusion with DDIM) are shown. The visualized trajectories are real from trained Multistep Consistency Models. The 4-step path has a smoother path and will likely be easier to learn than the 1-step path.
Refer to caption: x1.png

Recently, Consistency Models (Song et al., ) have reduced sampling time significantly, but at the expense of image quality. Consistency models come in two variants: Consistency Training (CT) and Consistency Distillation (CD) and both have considerably improved performance compared to earlier works. TRACT (Berthelot et al., ) focuses solely on distillation with an approach similar to consistency distillation, and shows that dividing the diffusion trajectory in stages can improve performance. Despite their successes, neither of these works attain performance close to a standard diffusion baseline.

Here, we propose a unification of Consistency Models and TRACT, that closes the performance gap between standard diffusion performance and low-step variants. We relax the single-step constraint from consistency models to allow ourselves as much as 4, 8 or 16 function evaluations for certain settings. Further, we generalize TRACT to consistency training and adapt step schedule annealing and synchronized dropout from consistency modelling. We also show that as steps increase, Multistep CT becomes a diffusion model. We introduce a unifying training algorithm to train what we call Multistep Consistency Models, which splits the diffusion process from data to noise into predefined segments. For each segment a separate consistency model is trained, while sharing the same parameters. For both CT and CD, this turns out to be easier to model and leads to significantly improved performance with fewer steps. Surprisingly, we can perfectly match baseline diffusion model performance with only eight steps, on both Imagenet64 and Imagenet128.

Another important contribution of this paper that makes the previous result possible, is a deterministic sampler for diffusion models that can obtain competitive performance on more complicated datasets such as ImageNet128 in terms of FID score. We name this sampler Adjusted DDIM (aDDIM), which essentially inflates the noise prediction to correct for the integration error that produces blurrier samples.

In terms of numbers, we achieve performance rivalling standard diffusion approaches with as little as 8 and sometimes 4 sampling steps. These impressive results are both for consistency training and distillation. A remarkable result is that with only 4 sampling steps, multistep consistency models obtain performances of 1.6 FID on ImageNet64 and 2.3 FID on Imagenet128.

## 2 Background: Diffusion Models

Diffusion models are specified by a destruction process that adds noise to destroy data: ${\bm{z}}_{t}=\alpha_{t}{\bm{x}}+\sigma_{t}{\bm{{\epsilon}}}_{t}$ where ${\bm{{\epsilon}}}_{t}\sim\mathcal{N}(0,1)$. Typically for $t\to 1$, ${\bm{z}}_{t}$ is approximately distributed as a standard normal and for $t\to 0$ it is approximately ${\bm{x}}$. In terms of distributions one can write the diffusion process as: $q({\bm{z}}_{t}|{\bm{x}})=\mathcal{N}({\bm{z}}_{t}|\alpha_{t}{\bm{x}},\sigma_{t})$.

Following (Sohl-Dickstein et al., ; Ho et al., ) we will let $\sigma_{t}^{2}=1-\alpha_{t}^{2}$ (variance preserving). As shown in Kingma et al. (), the specific values of $\sigma_{t}$ and $\alpha_{t}$ do not really matter. Whether the process is variance preserving or exploding or something else, they can always be re-parameterized into the other form. Instead, it is their ratio that matters and thus it can be helpful to define the signal-to-noise ratio, i.e. $\mathrm{SNR}(t)=\alpha_{t}^{2}/\sigma_{t}^{2}$. To sample from these models, one uses the denoising equation:

$$ $\small q({\bm{z}}_{s}|{\bm{z}}_{t},{\bm{x}})=\mathcal{N}({\bm{z}}_{s}|\mu_{t \to s}({\bm{z}}_{t},{\bm{x}}),\sigma_{t\to s})$ (1) $$

where ${\bm{x}}$ is approximated via a learned function that predicts $\hat{{\bm{x}}}=f({\bm{z}}_{t},t)$. Note here that $\sigma_{t\to s}^{2}=\big{(}\frac{1}{\sigma_{s}^{2}}+\frac{\alpha_{t|s}^{2}}{
\sigma_{t|s}^{2}}\big{)}^{-1}$ and ${\bm{\mu}}_{t\to s}=\sigma_{t\to s}^{2}\big{(}\frac{\alpha_{t|s}}{\sigma_{t|s}
^{2}}{\bm{z}}_{t}+\frac{\alpha_{s}}{\sigma_{s}^{2}}{\bm{x}}\big{)}$ as given by (Kingma et al., ). In (Song et al., ) it was shown that the optimal solution under a diffusion objective is to learn $\mathbb{E}[{\bm{x}}|{\bm{z}}_{t}]$, i.e. the expectation over all data given the noisy observation ${\bm{z}}_{t}$. One than iteratively samples for $t=1,1-1/N,\ldots,1/N$ and $s=t-1/N$ starting from ${\bm{z}}_{1}\sim\mathcal{N}(0,1)$.
Although the amount of steps required for sampling depends on the data distribution, empirically generative processes for problems such as image generation use hundreds of iterations making diffusion models one of the most resource consuming models to use (Luccioni et al., ).

#### Consistency Models

In contrast, consistency models (Song et al., ; Song & Dhariwal, ) aim to learn a direct mapping from noise to data. Consistency models are constrained to predict ${\bm{x}}=f({\bm{z}}_{0},0)$, and are further trained by learning to be consistent, minimizing:

$$ $\small||f({\bm{z}}_{t},t)-\operatorname{nograd}(f({\bm{z}}_{s},s))||,$ (2) $$

where ${\bm{z}}_{s}=\alpha_{s}{\bm{x}}+\sigma_{s}{\bm{{\epsilon}}}$ and ${\bm{z}}_{t}=\alpha_{t}{\bm{x}}+\sigma_{t}{\bm{{\epsilon}}}$, (note both use the same ${\bm{{\epsilon}}}$) and $s$ is closer to the data meaning $s<t$. When (or if) a consistency model succeeds, the trained model solves for the probability ODE path along time. When successful, the resulting model predicts the same ${\bm{x}}$ along the entire trajectory. At initialization it will be easiest for the model to learn $f$ near zero, because $f$ is defined as an identity function at $t=0$. Throughout training, the model will propagate the end-point of the trajectory further and further to $t=1$. In our own experience, training consistency models is much more difficult than diffusion models.

Figure: Figure 2: Qualititative comparison between a multistep consistency and diffusion model. Top: ours, samples from aDDIM distilled 16-step concistency model (3.2 secs). Bottom: generated samples usign a 100-step DDIM diffusion model (39 secs). Both models use the same initial noise.
Refer to caption: extracted/5639888/images/tti/mscm.png

#### Consistency Training and Distillation

Consistency Models come in two flavours: Consistency Training (CT) and Consistency Distillation (CD). In the paragraph before, ${\bm{z}}_{s}$ was given by the data which would be the case for CT. Alternatively, one might use a pretrained diffusion model to take a probability flow ODE step (for instance with DDIM). Calling this pretrained model the teacher, the objective for CD can be described by:

$$ $\small||f({\bm{z}}_{t},t)-\operatorname{nograd}(f(\operatorname{DDIM}_{t\to s} ({\bm{x}}_{\mathrm{teacher}},{\bm{z}}_{t}),s))||,$ (3) $$

where DDIM now defines ${\bm{z}}_{s}$ given the current ${\bm{z}}_{t}$ and (possibly an estimate of) ${\bm{x}}$.

An important hyperparameter in consistency models is the gap between the model evaluations at $t$ and $s$. For CT large gaps incurs a bias, but the solutions are propagated through diffusion time more quickly. On the other hand, when $s\to t$ the bias tends to zero but it takes much longer to propagate information through diffusion time. In practice a step schedule $N(\cdot)$ is used to anneal the step size $t-s=1/N(\cdot)$ over the course of training.

#### DDIM Sampler

The DDIM sampler is a linearization of the probability flow ODE that is often used in diffusion models. In a variance preserving setting, it is given by:

$$ $\small{\bm{z}}_{s}=\operatorname{DDIM}_{t\to s}({\bm{x}},{\bm{z}}_{t})=\alpha_ {s}{\bm{x}}+(\sigma_{s}/\sigma_{t})({\bm{z}}_{t}-\alpha_{t}{\bm{x}})$ (4) $$

In addition to being a sampling method, the $\operatorname{DDIM}$ equation will also prove to be a useful tool to construct an algorithm for our multistep diffusion models.

Another helpful equations is the inverse of DDIM (Salimans & Ho, ), originally proposed to find a natural way parameterize a student diffusion model when a teacher defines the sampling procedure in terms of ${\bm{z}}_{t}$ to ${\bm{z}}_{s}$. The equation takes in ${\bm{z}}_{t}$ and ${\bm{z}}_{s}$, and produces ${\bm{x}}$ for which $\operatorname{DDIM}_{t\to s}({\bm{x}},{\bm{z}}_{t})={\bm{z}}_{s}$. It can be derived by rearranging terms from the $\operatorname{DDIM}$ equation:

$$ $\small{\bm{x}}=\operatorname{invDDIM}_{t\to s}({\bm{z}}_{s},{\bm{z}}_{t})= \frac{{\bm{z}}_{s}-\frac{\sigma_{s}}{\sigma_{t}}{\bm{z}}_{t}}{\alpha_{s}- \alpha_{t}\frac{\sigma_{s}}{\sigma_{t}}}.$ (5) $$

## 3 Multistep Consistency Models

In this section we describe multi-step consistency models. First we explain the main algorithm, for both consistency training and distillation. Furthermore, we show that multi-step consistency converges to a standard diffusion training in the limit. Finally, we develop a deterministic sampler named aDDIM that corrects for the missing variance problem in DDIM.

Figure: Algorithm 1 Training Multistep CMs

### 3.1 General description

Multistep consistency split up diffusion time into equal segments to simplify the modelling task. Recall that a consistency model must learn to integrate the full ODE integral. This mapping can become very sharp and difficult to learn when it jumps between modes of the target distribution as can be seen in Figure 1. A consistency loss can be seen as an objective that aims to approximate a path integral by minimizing pairwise discrepancies. Multistep consistency generalizes this approach by breaking up the integral into multiple segments. Originally, consistency runs until time-step 0 0, evaluated at some time $t>0$. A consistency model should now learn to integrate the DDIM path until 0 0 and predict the corresponding ${\bm{x}}$. Instead, we can generalize the consistency loss to targets $z_{t_{\mathrm{step}}}$ instead of ${\bm{x}}$ ($\approx{\bm{z}}_{0}$). It turns out that the DDIM equation can be used to operate on ${\bm{z}}_{t_{\mathrm{step}}}$ for different times $t_{\mathrm{step}}$, which allows us to express the multi-step consistency loss as:

$$ $\small||\operatorname{DDIM}_{t\to t_{\mathrm{step}}}(f({\bm{z}}_{t},t),{\bm{z} }_{t})-\hat{{\bm{z}}}_{\mathrm{ref},t_{\mathrm{step}}}||,$ (6) $$

where $\hat{{\bm{z}}}_{\mathrm{ref},t_{\mathrm{step}}}=\operatorname{DDIM}_{s\to t_{
\mathrm{step}}}(\operatorname{nograd}f({\bm{z}}_{s},s))$ and where the teaching step ${\bm{z}}_{s}=\operatorname{aDDIM}_{t\to s}(x,{\bm{z}}_{t})$. For now it suffices to think $\operatorname{aDDIM}$ as $\operatorname{DDIM}$. It will be described in detail in section 3.2. In fact, one can drop-in any deterministic sampler (or integrator) in place of $\operatorname{aDDIM}$ in the case of distillation.

A model can be trained on directly on this loss in $z$ space, however make the loss more interpretable and relate it more closely to standard diffusion, we re-parametrize the loss to $x$-space using:

$$ $\small||\hat{{\bm{x}}}_{\mathrm{diff}}||=||f({\bm{z}}_{t},t)-\operatorname{ invDDIM}_{t\to t_{\mathrm{step}}}(\hat{{\bm{z}}}_{\mathrm{ref},t_{\mathrm{step }}},{\bm{z}}_{t})||.$ (7) $$

This allows the usage of existing losses from diffusion literature, where we have opted for $v$-loss (equivalent to $\mathrm{SNR}+1$ weighting) because of its prior success in distillation (Salimans & Ho, ).

As noted in (Song et al., ), consistency in itself is not sufficient to distill a path (always predicting 0 0 is consistent) and one needs to ensure that the model cannot collapse to these degenerate solutions. Indeed, in our specification observe that that $\operatorname{DDIM}_{t_{\mathrm{step}}\to t_{\mathrm{step}}}({\bm{z}}_{t_{
\mathrm{step}}},\hat{\bm{x}})={\bm{z}}_{t_{{\mathrm{step}}}}$ and that for the final step in each segment $s=t_{\mathrm{step}}$. As such, the loss of the final step cannot be degenerate and is equal to:

$$ $\small||f({\bm{z}}_{t},t)-\operatorname{invDDIM}_{t_{\mathrm{step}}\to t}({\bm {z}}_{t_{\mathrm{step}}})||.$ (8) $$

#### Many-step CT is equivalent to Diffusion training

Consistency training learns to integrate the probability flow through time, whereas standard diffusion models learn a path guided by an expectation $\hat{{\bm{x}}}=\mathbb{E}[{\bm{x}}|{\bm{z}}_{t}]$ that necessarily has to change over time for non-trivial distributions. There are two simple reasons that for many steps, Multistep CT converges to a diffusion model. 1) At the beginning of a step (specifically $t={t_{\mathrm{step}}}+\frac{1}{T}$) the objectives are identical. Secondly, 2) when the number of steps equals the total number of steps $T$, then every step is equal to the diffusion objective. This can be observed by studying Algorithm 1: let $t=t_{\mathrm{step}}+1/T$. For consistency training, aDDIM reduces to DDIM and observe that in this case $s=t_{\mathrm{step}}$. Hence, under a well-defined model $f$ (such as a $v$-prediction one) $\operatorname{DDIM}_{s\to t_{\mathrm{step}}}$ does nothing and simply produces $\hat{{\bm{z}}}_{\mathrm{ref},t_{\mathrm{step}}}={\bm{z}}_{s}$. Also observe that $\hat{{\bm{z}}}_{t_{\mathrm{step}}}=\hat{{\bm{z}}}_{s}$. Further simplification yields:

$$ $w(t)||{\bm{x}}_{\mathrm{diff}}||=\operatorname{invDDIM}_{t\to s}(\hat{{\bm{z}} }_{s},{\bm{z}}_{t})-\hat{{\bm{x}}}=w(t)||{\bm{x}}-\hat{{\bm{x}}}||$ (9) $$

Where $||{\bm{x}}-\hat{{\bm{x}}}||$ is the distance between the true datapoint and the model prediction weighted by $w(t)$, which is typical for standard diffusion. Interestingly, in (Song & Dhariwal, ) it was found that Euclidean ($\ell_{2}$) distances typically work better than for consistency models than the more usual squared Euclidean distances ($\ell_{2}$ squared). We follow their approach because it tended to work better especially for smaller steps, which is a deviation from standard diffusion. Because multistep consistency models tend towards diffusion models, we can state two important hypotheses:

- 1.
Finetuning Multistep CMs from a pretrained diffusion checkpoint will lead to quicker and more stable convergence.
- 2.
As the number of steps increases, Multistep CMs will rival diffusion model performance, giving a direct trade-off between sample quality and duration.

Figure: Algorithm 2 Sampling from Multistep CMs

#### What about training in continuous time?

Diffusion models can be easily trained in continuous time by sampling $t\sim\mathcal{U}(0,1)$, but in Algorithm 1 we have taken the trouble to define $t$ as a discrete grid on $[0,1]$. One might ask, why not let $t$ be continuously valued. This is certainly possible, if the model $f$ would take in an additional conditioning signal to denote in which step it is. This is important because its prediction has to discontinuously change between $t\geq t_{\mathrm{step}}$ (this step) and $t<t_{\mathrm{step}}$ (the next step). In practice, we often train Multistep Consistency Models starting from pre-trained with standard diffusion models, and so having the same interface to the model is simpler. In early experiments we did find this approach to work comparably.

Figure: Algorithm 3 Generating Samples with aDDIM

### 3.2 The Adjusted DDIM (aDDIM) sampler.

Popular methods for distilling diffusion models, including the method we propose here, rely on deterministic sampling through numerical integration of the probability flow ODE. In practice, numerical integration of this ODE in a finite number of steps incurs error. For the DDIM integrator (Song et al., ) used for distilling diffusion models in both consistency distillation (Song et al., ) and progressive distillation (Salimans & Ho, ; Meng et al., ) this integration error causes samples to become blurry. To see this quantitatively, consider a hypothetical perfect sampler that first samples ${\bm{x}}^{*}\sim p({\bm{x}}|{\bm{z}}_{t})$, and then samples ${\bm{z}}_{s}$ using

$$ $\small{\bm{z}}^{*}_{s}=\alpha_{s}{\bm{x}}^{*}+\sigma_{s}\frac{{\bm{z}}_{t}- \alpha_{t}{\bm{x}}^{*}}{\sigma_{t}}=(\alpha_{s}-\frac{\alpha_{t}\sigma_{s}}{ \sigma_{t}}){\bm{x}}^{*}+\frac{\sigma_{s}}{\sigma_{t}}{\bm{z}}_{t}.$ (10) $$

If the initial ${\bm{z}}_{t}$ is from the correct distribution $p({\bm{z}}_{t})$, the sampled ${\bm{z}}^{*}_{s}$ would then also be exactly correct. Instead, the DDIM integrator uses

$$ $\small{\bm{z}}^{\text{DDIM}}_{s}=(\alpha_{s}-\alpha_{t}\sigma_{s}/\sigma_{t}) \hat{{\bm{x}}}+(\sigma_{s}/\sigma_{t}){\bm{z}}_{t},$ (11) $$

with model prediction $\hat{{\bm{x}}}$. If $\hat{{\bm{x}}}=\mathbb{E}[{\bm{x}}|{\bm{z}}_{t}]$, we then have that

$$ $\small||{\bm{z}}^{*}_{s}||^{2}-||{\bm{z}}^{\text{DDIM}}_{s}||^{2}=\text{trace} (\mathrm{Var}[{\bm{z}}_{s}|{\bm{z}}_{t}]),$ (12) $$

where $\mathrm{Var}[{\bm{z}}_{s}|{\bm{z}}_{t}]$ is the conditional variance of ${\bm{z}}_{s}$ given by

$$ $\small\mathrm{Var}[{\bm{z}}_{s}|{\bm{z}}_{t}]=(\alpha_{s}-\alpha_{t}\sigma_{s} /\sigma_{t})^{2}\cdot\mathrm{Var}[{\bm{x}}|{\bm{z}}_{t}],$ (13) $$

and where $\mathrm{Var}[{\bm{x}}|{\bm{z}}_{t}]$ in turn is the variance of $p({\bm{x}}|{\bm{z}}_{t})$.

The norm of the DDIM iterates is thus too small, reflecting the lack of noise addition in the sampling algorithm. Alternatively, we could say that the model prediction $\hat{{\bm{x}}}\approx\mathbb{E}[{\bm{x}}|{\bm{z}}_{t}]$ is too smooth. Second-order integrators like the Heun sampler proposed by Karras et al. () aim to remove this bias by using the prediction at $s$ in addition to that at $t$, but in practice these methods did not significantly improve sample quality in our experiments.

Figure: Figure 3: Comparison of the vanilla and adjusted DDIM sampler for the small ImageNet 128 diffusion model without distillation.
Refer to caption: x2.png

Currently, the best sample quality is achieved with stochastic samplers, which can be tuned to add exactly enough noise to undo the oversmoothing caused by numerical integration. However, current distillation methods are not well suited to distilling these stochastic samplers directly. Here we therefore propose a new deterministic sampler that aims to achieve the norm increasing effect of noise addition in a deterministic way. It turns out we can do this by making a simple adjustment to the DDIM sampler, and we therefore call our new method Adjusted DDIM (aDDIM). Our modification is heuristic and is not more theoretically justified than the original DDIM sampler. However, empirically we find aDDIM to work very well leading to improved FID scores (Fig. 3) and thus a stronger deterministic teacher.

Instead of adding noise to our sampled ${\bm{z}}_{s}$, we simply increase the contribution of our deterministic estimate of the noise $\hat{{\bm{{\epsilon}}}}=({\bm{z}}_{t}-\alpha_{t}\hat{{\bm{x}}})/\sigma_{t}$. Assuming that $\hat{{\bm{x}}}$ and $\hat{{\bm{{\epsilon}}}}$ are orthogonal, we achieve the correct norm for our sampling iterates using:

$$ $\small{\bm{z}}^{\text{aDDIM}}_{s}=\alpha_{s}\hat{{\bm{x}}}+\sqrt{\sigma_{s}^{2 }+\text{tr}(\mathrm{Var}[{\bm{z}}_{s}|{\bm{z}}_{t}])/||\hat{{\bm{{\epsilon}}}} ||^{2}}\cdot\hat{{\bm{{\epsilon}}}}.$ (14) $$

In practice, we can estimate $\text{tr}(\mathrm{Var}[{\bm{z}}_{s}|{\bm{z}}_{t}])=(\alpha_{s}-\alpha_{t}
\sigma_{s}/\sigma_{t})^{2}\cdot\text{tr}(\mathrm{Var}[{\bm{x}}|{\bm{z}}_{t}])$ empirically on the data by computing beforehand $\text{tr}(\mathrm{Var}[{\bm{x}}|{\bm{z}}_{t}])=\eta||\hat{{\bm{x}}}({\bm{z}}_{
t})-{\bm{x}}||^{2}$ for all relevant timesteps $t$. Here $\eta$ is a hyperparameter which we set to $0.75$. Alternatively, we obtain equally good results by approximating the posterior variance analytically with $\text{tr}(\mathrm{Var}[{\bm{x}}|{\bm{z}}_{t}])/d=0.1/(2+\alpha^{2}_{t}/\sigma^
{2}_{t})$, for data dimension $d$, which can be interpreted as $10\%$ of the posterior variance of ${\bm{x}}$ if its prior was factorized Gaussian with variance of $0.5$. In either case, note that $\mathrm{Var}[{\bm{z}}_{s}|{\bm{z}}_{t}]$ vanishes as $s\rightarrow t$: in the many-step limit the aDDIM update thus becomes identical to the original DDIM update. For a complete description see Algorithm 3.

Figure: Figure 4: Another qualititative comparison between a multistep consistency and teacher, using the same prompt. Top: ours, a distilled 16-step concistency model (3.2 secs). Bottom: generated samples using a 100-step DDIM diffusion model (39 secs). Both models use the same initial noise.
Refer to caption: extracted/5639888/images/tti/bird/mscm_0.png

## 4 Related Work

Multistep Consistency Models are a direct combination of (Song et al., ; Song & Dhariwal, ) and TRACT (Berthelot et al., ). Compared to consistency models, we propose to operate on multiple stages, which simplifies the modelling task and improves performance significantly. On the other hand, TRACT limits itself to distillation and uses the self-evaluation from consistency models to distill models over multiple stages. The stages are progressively reduced to either one or two stages and thus steps. The end-goal of TRACT is again to sample in either one or two steps, whereas we believe better results can be obtained by optimizing for a slightly larger number of steps. We show that this more conservative target, in combination with our improved sampler and annealed schedule, leads to significant improvements in terms of image quality that closes the gap between sample quality of standard diffusion and low-step diffusion-inspired approaches.

Earlier, DDIM (Song et al., ) showed that deterministic samplers degrade more gracefully than the stochastic sampler used by Ho et al. () when limiting the number of sampling steps. Karras et al. () proposed a second order Heun sampler to reduce the number of steps (and function evaluations), while Jolicoeur-Martineau et al. () studied different SDE integrators to reduce function evaluations. Progressive Distillation (Salimans & Ho, ; Meng et al., ) distills diffusion models in stages, which limits the number of model evaluations during training while exponentially reducing the required number of sampling steps with the number stages.

Other methods inspired by diffusion such as Rectified Flows (Liu et al., ) and Flow Matching (Lipman et al., ) have also tried to reduce sampling times. In practice however, flow matching and rectified flows are generally used to map to a standard normal distribution and reduce to standard diffusion. As a consequence, on its own they still require many evaluation steps. In Rectified Flows, a distillation approach is proposed that does reduce sampling steps more significantly, but this comes at the expense of sample quality.

#### Adversarial distillation

Distillation to a few steps was very difficult to do using only simple distance metrics. Therefore, many works resort to a form of adversarial training. For example Luo et al. () distill the knowledge from the diffusion model into a single-step model and Zheng et al. () use specialized architectures to distill the ODE trajectory from a pre-created noise-sample pair dataset.
A very similar approach to ours is Consistency Trajectory Models (CTMs) (Kim et al., ), which are trained to arbitrarily integrate to a given timestep. This is implemented by modifying the inputs of the denoising network to include an endpoint of the integration. Although CTMs produce very high quality image samples in few steps, their performance relies on adversarial training: Without it, CTMs cannot produce great samples and have a considerable gap in FID score. In contrast, our Multistep CMs can be trained with simple distance metrics and still achieve very good FID scores under few sampling steps. A possible explanation is that it is much easier to learn a handful of fixed integration trajectories (Multistep CMs) instead of every possible integration with arbitrary endpoints (CTMs). Another advantage of Multistep CMs is that the inputs to the denoising network are not changed, making fine-tuning of existing diffusion models to Multistep CMs very straightforward.

**Table 1: Imagenet performance with multistep consistency training (CT) and consistency distillation (CD), started from a pretrained diffusion model. A baseline with the aDDIM sampler on the base model is included.**
|  |  |  | Small | Large |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | ImageNet64 | ImageNet128 | ImageNet64 | ImageNet128 |  |  |  |  |
|  |  | Steps | Train | Distill | Train | Distill | Train | Distill |  |  |
| Base | Consistency Model | 1 | 7.2 | 4.3 | 16.0 | 8.5 | 6.4 | 3.2 | 14.5 | 7.0 |
|  | MultiStep CM (ours) | 2 | 2.7 | 2.0 | 6.0 | 3.1 | 2.3 | 1.9 | 4.2 | 3.1 |
|  | MultiStep CM (ours) | 4 | 1.8 | 1.7 | 4.0 | 2.4 | 1.6 | 1.6 | 2.7 | 2.3 |
|  | MultiStep CM (ours) | 8 | 1.5 | 1.6 | 3.3 | 2.1 | 1.5 | 1.4 | 2.2 | 2.1 |
|  | MultiStep CM (ours) | 16 | 1.5 | 1.5 | 3.4 | 2.0 | 1.6 | 1.4 | 2.3 | 2.0 |
|  | Diffusion (aDDIM) | 512 | 1.5 | 2.2 | 1.4 | 2.2 |  |  |  |  |

**Table 2: Ablation of CD on Image128 with and without annealing the teacher steps on ImageNet128. Annealing the teacher stepsize improves the performance.**
| Steps | (64 $\to$ 1280) | (step = 128) | (step = 256) | (step = 1024) |
| --- | --- | --- | --- | --- |
| 1 | 7.0 | 8.8 | 7.6 | 10.8 |
| 2 | 3.1 | 5.3 | 3.6 | 3.8 |
| 4 | 2.3 | 5.0 | 3.5 | 2.6 |
| 8 | 2.1 | 4.9 | 3.2 | 2.2 |

## 5 Experiments

Our experiments focus on a quantitative comparison using the FID score on ImageNet as well as a qualitative assessment on large scale Text-to-Image models. These experiments should make our approach comparable to existing academic work while also giving insight in how multi-step distillation works at scale.

### 5.1 Quantitative Evaluation on ImageNet

Figure: Table 4: Literature Comparison on ImageNet.

For our ImageNet experiments we trained diffusion models on ImageNet64 and ImageNet128 in a base and large variant. We initialize the consistency models from the pre-trained diffusion model weights which we found to greatly increase robustness and convergence. Both consistency training and distillation are used. Classifier Free Guidance (Ho & Salimans, ) was used only on the base ImageNet128 experiments. For all other experiments we did not use guidance because it did not significantly improve the FID scores of the diffusion model. All consistency models are trained for $200,000$ steps with a batch size of $2048$ and an step schedule that anneals from $64$ to $1280$ in $100.000$ steps with an exponential schedule.

In Table 1 it can be seen that as the multistep count increases from a single consistency step, the performance considerably improves. For instance, on the ImageNet64 Base model it improves from 7.2 for one-step, to 2.7 and further to 1.8 for two and four-steps respectively. In there are generally two patterns we observe: As the steps increase, performance improves. This validates our hypothesis that more steps give a helpful trade-off between sample quality and speed. It is very pleasant that this happens very early: even on a complicated dataset such as ImageNet128, our base model variant is able achieve 2.1 FID in 8 steps, when consistency distilling.

To draw a direct comparison between Progressive Distillation (PD) (Salimans & Ho, ) and our approaches, we reimplement PD using aDDIM and we use same base architecture, as reported in Table 3. With our improvements, PD can attain better performance than previously reported in literature. However, compared to MultiStep CT and CD it starts to degrade in sample quality at low step counts. For instance, a 4-step PD model attains an FID of 2.4 whereas CD achieves 1.7.

Further we ablate whether annealing the step schedule is important to attain good performance. As can be seen in Table 2, it is especially important for low multistep models to anneal the schedule. In these experiments, annealing always achieves better performance than tests with constant steps at $128,256,1024$. As more multisteps are taken, the importance of the annealing schedule decreases.

#### Literature Comparison

Compared to existing works in literature, we achieve SOTA FID scores in both ImageNet64 on 2-step, 4-step and 8-step generation. Interestingly, we achieve approximately the same performance using single step CD compared to iCT-deep (Song & Dhariwal, ), which achieves this result using direct consistency training. Since direct training has been empirically shown to be a more difficult task, one could conclude that some of our hyperparameter choices may still be suboptimal in the extreme low-step regime. Conversely, this may also mean that multistep consistency is less sensitive to hyperparameter choices.

In addition, we compare on ImageNet128 to our reimplementation of Progressive Distillation. Unfortunately, ImageNet128 has not been widely adopted as a few-step benchmark, possibly because a working deterministic sampler has been missing until this point. For reference we also provide the recent result from (Kingma & Gao, ). Further, with these results we hope to put ImageNet128 on the map for few-step diffusion model evaluation.

### 5.2 Qualitative Evaluation on Text to Image modelling

In addition to the analysis on ImageNet, we study the effects on a text-to-image models. We distill a 16-step consistency model a base teacher model. In Table 5 one can see that Multistep CD is able to distill its teacher almost perfectly in terms of FID. The loss of clip score can be attributed to the guidance distillation, which a baseline 256-step student model also has trouble distilling. Compared to the guidance-distilled baseline, the 16-CD model has no loss in performance measured in CLIP and FID on the low guidance setting (and for the high guidance setting only a minor degradation in FID). Even the 8-step CD model attains an impressive FID score of 8.1, which is well below the existing literature.

In Figure 2 and 5 we compare samples from our 16-step CD aDDIM distilled model to the original 100-step DDIM sampler. Because the random seed is shared we can easily compare the samples between these models, and we can see that there are generally minor differences. In our own experience, we often find certain details more precise, at a slight cost of overall construction. Another comparison in Figure 4 shows the difference between a DDIM distilled model (equivalent to $\eta=0$ in aDDIM) and the standard DDIM sampler. Again we see many similarities when sharing the same initial random seed.

**Table 5: Text to Image performance. Note that when 8/16-step Consistency is compared to a teacher model that is only guidance distilled at 256 steps, there is practically no performance loss.**
| Method | NFE | FID${}_{\text{30k}}$ | FID${}_{\text{5k}}$ | CLIP | non-adv |
| --- | --- | --- | --- | --- | --- |
| SDv1.5 (Rombach et al., ) low g (from DMD) | 512 | 8.8 |  | - | ✓ |
| high g (from DMD) | 512 | 13.5 |  | 0.322 | ✓ |
| DMD (low guidance) (Yin et al., ) | 1 | 11.5 |  | - |  |
| (high guidance) | 1 | 14.9 |  | 0.32 |  |
| UFOGen (Xu et al., ) | 1 | 12.8 | 22.5 | 0.311 |  |
|  | 4 |  | 22.1 | 0.307 |  |
| InstaFlow-1.7B (Liu et al., ) | 1 | 11.8 | 22.4 | 0.309 | ✓ |
| PeRFlow (Yan et al., ) | 4 | 11.3 |  |  | ✓ |
| Teacher Diffusion Model g=0.5 (ddpm) | 256 | 7.9 | 13.6 | 0.305 | ✓ |
| guidance distilled (ddim) | 256 | 8.2 | 13.8 | 0.300 | ✓ |
| Multistep-CD (teacher g=0.5) | 4 | 8.7 | 14.4 | 0.298 | ✓ |
|  | 8 | 8.1 | 13.8 | 0.300 | ✓ |
|  | 16 | 7.9 | 13.9 | 0.300 | ✓ |
| Teacher Diffusion Model g=3 (ddpm) | 256 | 12.7 | 18.1 | 0.315 | ✓ |
| guidance distilled (ddim) | 256 | 13.9 | 19.0 | 0.312 | ✓ |
| Multistep-CD (teacher g=3) | 4 | 12.4 | 18.1 | 0.311 | ✓ |
|  | 8 | 13.9 | 19.6 | 0.311 | ✓ |
|  | 16 | 14.4 | 20.0 | 0.312 | ✓ |

## 6 Conclusions

In conclusion, this paper presents Multistep Consistency Models, a simple unification between Consistency Models (Song et al., ) and TRACT (Berthelot et al., ) that closes the performance gap between standard diffusion and few-step sampling. Multistep Consistency gives a direct trade-off between sample quality and speed, achieving performance comparable to standard diffusion in as little as eight steps. The main limitation of multistep consistency is that one pays the price of several function evaluations to generate a sample. Here, adversarial approaches generally perform better when only one or two evaluations are permitted, but they come the cost of more difficult training dynamics.

#### Broader Impacts

This paper proposes a method to speed up sampling from diffusion models. Although generative models may be used for positive applications such as enhancing human creativity or drug discovery, they may also be used to create deepfakes or misinformation. Hence, enabling faster sampling may amplify both the positive and the negative impacts of generative modelling.

## Appendix A Experimental Details

### A.1 Setup

In this paper we follow the setup from simple diffusion (Hoogeboom et al., ). Following their approach, we use a standard UViTs. These are UNets with MLP blocks instead of convolutional layers when a block has self-attention, making the entire block a transformer block. This contains the details for the architecture and how to define diffusion process. There are some minor specifics which we share per experiment below.

#### Multistep Constinency Hyperparameters

For all ImageNet runs (small/large, 1 through 16 step) we use a log-linear interpolated schedule from 64 teacher steps to 1280 teacher steps, annealed over 100000 training iterations at a batch size of 2048. We use a xvar_frac of 0.75 for aDDIM. And we use a huber epsilon of 1e-4. The model is trained for 200000 steps. The interpolation starts quite low and takes a long time, and these settings are somewhat excessive for the larger student step models such as the 8- or 16-step model. However, fixing these settings for the model allowed for clean comparisons.

For the text-to-image model, we ran consistency distillation where we kept the teacher steps fixed at 256 and used an xvar_frac of 0.75. Note that the xvar_frac should always be computed on the conditional output, not the guided output (so guidance zero). We used a huber epsilon of 1, which is essentially a scalar-scaled l2 squared loss for the normalized [-1, 1] domain of interest. We train these models for 30000 steps at a batch size of 2048.

#### ImageNet64

For the ImageNet64 experiments, the levels of the UViT small are as follows. Down: 3 ResNet blocks with 256 channels, 3 Transformer Blocks with 512 channels both stages ending with an average pool. Middle: 16 transformer blocks with 1024 channels, mlp expansion factor is 4. Up, matching the down blocks, starting a stage with a nearest neighbour upsampling and obviously no pooling. Dropout is applied to the middle with a factor of 0.2. For the large variant, all channels are multiplied by 2, and dropout is applied to all transformers albeit with a lower factor of 0.1. The network is trained with an interpolated cosine schedule from noise resolution 32 to 64 at a resolution of 64 (this is practically identical to a normal cosine schedule).

#### ImageNet128

For the ImageNet128 experiments, the UViT is the same as the UViT for ImageNet64, but with an extra 3 ResNet Blocks at the resolution 128x128 with 128 channels at both the start and the end of the UViT. For completeness, down: 3 ResNet blocks with 128 channels, 3 ResNet blocks with 256 channels, 3 Transformer Blocks with 512 channels both stages ending with an average pool. Middle: 16 transformer blocks with 1024 channels, mlp expansion factor is 4. Up, matching the down blocks, starting a stage with a nearest neighbour upsampling and no pooling.

Different from before, dropout is applied to the middle with only a factor of 0.1. For the large variant, all channels are multiplied by 2, and dropout is applied to all blocks (both convolutional and transformer) except for the ones at the resolution of 128, also at a factor of 0.1. The network is trained with an interpolated cosine schedule from noise resolution 32 to 128 at a resolution of 128, with a multiscale loss (Hoogeboom et al., ) that $2\times 2$ average pools once.

#### Text-to-Image

The text-to-image model is directly trained on $512\times 512$, with a multiscale loss and an interpolated cosine schedule starting at noise resolution 32 and ending at 512. The UViT has the following stages, down: 3 ResNet blocks at 128 channels, 3 ResNet blocks at 256 channels, 3 ResNet blocks at 1024 channels, 3 transformer blocks at 2048 channels, average pool at the end of each stage. Mid: 16 transformer blocks with 4096 channels and dropout ratio 0.1. Up: identical to reversed down with nearest neighbour instead of average pooling.

### A.2 Compute resources

All small model variants are run on 64 TPUv5e chips. For ImageNet64 CT takes 2.7 training steps per second and CD takes 2.5 steps per sec. For ImageNet128 CT takes 2.2 training steps per second and CD takes 1.7 steps per sec.

The large variants are trained on 256 TPUv5e chips. For ImageNet64 CT takes 2.9 training steps per second and CD takes 2.5 steps per sec. For ImageNet128 CT it takes 2.2 training steps per second and CD takes 1.8 steps per second. The text to image experiment is also run on 256 TPUve chips and takes 0.71 steps per second to train, and is only trained for 30000 iterations.

### A.3 Datasets

The models in this paper are trained on ImageNet dataset (Russakovsky et al., ). The text to image model is trained on a privately licensed text-to-image dataset, comparable with public text-to-image datasets but filtered for content.

### A.4 Additional results

In Figure 5, some additional results are shown for the same prompt. Again, the distilled model is very similar to the original teacher model with minor variations.

Figure: Figure 5: Another qualititative comparison between a multistep consistency and diffusion model. Top: ours, samples from aDDIM distilled 16-step concistency model (3.2 secs). Bottom: generated samples using a 100-step DDIM diffusion model (39 secs). Both models use the same initial noise.
Refer to caption: extracted/5639888/images/tti/car/mscm_car1.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Are there any insights into the choice of hyperparameters in aDDIM, such as $\eta$?

2. The time axis is divided into equal segments. Are there alternative segmentation strategies that could improve performance?

3. The experiments are conducted using consistency tuning/distillation, where the model is initialized directly from a pretrained Diffusion Model, providing a good starting point. Could the proposed method also work with consistency training, i.e., random initialization?
soundness: 3
strengths: The aDDIM sampler addresses the oversmoothing problem in deterministic sampling mechanisms, achieving performance comparable to that of a second-order sampler. The paper also provides comprehensive experiments on class-conditioned ImageNet and text-to-image generation.
summary: The paper introduces Multistep Consistency Models (MCMs) as an interpolation between Consistency Models (CMs) and Diffusion Models (DMs), dividing the time axis into multiple segments and applying a consistency tuning/distillation objective within each segment. Additionally, the Adjusted DDIM (aDDIM) is presented as an efficient deterministic sampling method that not only enhances synthetic sample quality but also improves MCM results when incorporated into the training algorithm.
weaknesses: See questions below.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness section.
soundness: 2
strengths: This work extends the Consistency Model by leveraging the inversion of DDIM to generate a clean regression target based on the DDIM solution at a middle timestep, starting from a noisier timestep. The proposed method demonstrates strong empirical performance in diffusion distillation without the need for adversarial training.
summary: This work extends the Consistency Model by leveraging the inversion of DDIM to generate a clean regression target based on the DDIM solution at a middle timestep, starting from a noisier timestep. The proposed method demonstrates strong empirical performance in diffusion distillation without the need for adversarial training.
weaknesses: 1. There are some typos, such as in line 101 where "*One than...*" should be "*One then...*". Additionally, some characters should be in bold but are not. Some parts are not consistent, such as using both *Table XX* and *Tbl. XX*.

2. Regarding the flow in Section 2, would it improve clarity to switch the paragraphs *Consistency Training and Distillation* and *DDIM Sampler*? Currently, the notation $\mathrm{DDIM}_{t\rightarrow s}$ appears in Eq. (3) without prior introduction. Additionally, certain terms need definition or mention, such as $\mathbf{x}$-teacher. I appreciate the strong performance of the method, as highlighted in the abstract and introduction. However, further elaboration on the proposed method and the sources of its performance in these sections would be greatly appreciated.

3. Could you clarify the fundamental difference between the proposed method and [1], particularly regarding the design of the distillation loss and model parameterizations? [1] also unifies the distillation-based model (which involves a few long jump steps for generation) and the diffusion model (which uses many steps for generation). The proposal seems to be a minor modification of this framework.


4. In Table 1, is the Consistency Model's result also from UViTs backbones? In Table 5, is the proposed method using standard UNet structures that baselines were using? I'd like to know if all the comparisons are fair in terms of architectures, model sizes. Additionally, what is the teacher model used in the distillation training? If it is based on EDM [3], that would be fine; otherwise, it may not be an apples-to-apples comparison with respect to the baselines.


5. Could you explain how one can ensure that the learned model, using objective (8) with the adjusted DDIM for teacher steps, converges to either the teacher model (for distillation) or the true model (for training from scratch)—similar to Theorem 1 and Theorem 2 in [2]? The adjusted DDIM seems to alter the learning target. 

6. The DDIM-related solutions (e.g., $\mathrm{DDIM}$ or $\mathrm{invDDIM}$, which seem to solve in a single step from $t$ to $s$ even when the gap $|t - s|$ is large) appear to take large step jumps directly, rather than solving in a time-stepping manner—this approach could introduce significant discretization errors during training.

7. Is there evidence supporting the claim that the proposed method is more stable and converges faster (presumably compared to the consistency model)? Additionally, how many function evaluations are required in the proposed method compared to those needed for training the Consistency Model? Overall, does the proposed method still converge faster regardless of model sizes?


[1] Kim, D., Lai, C.H., Liao, W.H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y. and Ermon, S., 2023. Consistency trajectory models: Learning probability flow ode trajectory of diffusion.

[2] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency models. arXiv preprint arXiv:2303.01469.

[3] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: -
soundness: 2
strengths: This paper proposes a method that can be directly applied to the large-scale models like SDXL.
summary: This paper introduces a segmented distillation method, along with aDDIM.
weaknesses: Major Comments

- In line 135, what does the term "difficult" refer to? Is it about the instability during training or the model’s overall performance?
- In the paragraph starting at line 145, my understanding is that Consistency Models did not originally introduce a gap between model evaluations at t and s, although extending in that direction seems intuitive. Additionally, even as s->t, would longer propagation during training significantly affect the author's setup? To my knowledge, 200k iterations with an EMA rate of 0.9999 are sufficient for convergence, regardless of propagation length.
- In line 176, the phrase "difficult to learn when it jumps between modes" feels inappropriate. Simply stating that "jumping between modes represents a more complex function" seems sufficient. If this is why we need MCM, wouldn’t increasing the model’s capacity address the issue?
- In line 182, there are already several papers that handle multiple segments. What makes this approach novel?
- In line 188, would it be better to set $z_{t_{step}}$ as the target instead of x? What’s the intuition here? Moreover, transitioning from Eq. (6) to Eq. (7) with differences in the x-space appears contradictory.
- Could the model be trained using LPIPS? What would happen if LPIPS were used instead? Would it lead to better performance with smaller steps? Since using l2 instead of squared l2 no longer ensures the denoiser function as the optimal solution, I suspect LPIPS could be beneficial if denoiser optimality is not the goal.
- In the paragraph starting at line 240, I’m confused. In Consistency Models, performance in continuous time was poor. How does the proposed algorithm work better in continuous time? Could you provide more intuition?
- I found Section 3.2 on aDDIM unclear. Given that there are already several papers covering segmented timestep distillation (the claimed contribution), I only see aDDIM as the novel part. However, since I don’t fully understand this concept, I can't recommend acceptance. Please make it clearer.
- Even if aDDIM is clear, Table 4 shows only marginal effects of aDDIM. Can the authors provide further arguments for why aDDIM deserves top-tier conference recognition?
Regarding the loss function, given the existing GAN-based distillation approaches like ADD [1] or LADD [2], wouldn’t it be better to adopt an adversarial loss? Why insist on l2?
- In line 265, what does "this integration error causes samples to become blurry" mean? Which samples? Multi-step or one-step samples? What exactly is the integration error? If DDIM works, how does integration error affect sample quality?
- A more comprehensive related work section on distillation models is needed. For instance, [3] is the first work to distill diffusion models. [4] is also closely related, particularly in the DDPM setup.
- Could the proposed distillation method works on DiT?
- I’m curious how the proposed algorithm fits into both the VE and FM settings. Could the authors discuss this in the main paper?

Minor Comments

- In line 99, Song's Score SDE paper didn’t describe the object as a denoiser. The concept of denoising was introduced later. Song's derivation of score matching was combined with the Tweedie formula to reach that conclusion. Please cite either the Tweedie formula or other papers that clearly mention this. I recommend [5] if you prefer to stay within the diffusion community.
- Make $z_{t_{step}}$ in line 188 bold.
- Change ADDIM in the heading of Section 3.2 to aDDIM.

[1] Sauer, Axel, et al. "Adversarial diffusion distillation." arXiv preprint arXiv:2311.17042 (2023).
[2] Sauer, Axel, et al. "Fast high-resolution image synthesis with latent adversarial diffusion distillation." arXiv preprint arXiv:2403.12015 (2024).
[3] Luhman, Eric, and Troy Luhman. "Knowledge distillation in iterative generative models for improved sampling speed." arXiv preprint arXiv:2101.02388 (2021).
[4] Zheng, Jianbin, et al. "Trajectory consistency distillation." arXiv preprint arXiv:2402.19159 (2024).
[5] Jolicoeur-Martineau, Alexia, et al. "Adversarial score matching and improved sampling for image generation." arXiv preprint arXiv:2009.05475 (2020).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. I have one question regarding the objective of the network.
Imagine $t$ is one of the boundaries for the network to switch targets. Assume $t_1 = t-\delta$ and $t_2=t+\delta$. 
We expect the network to predict two different targets for $t_1$ and $t_2$. Is this right?
So how does the network handle this sudden change? Will this introduce higher variance or instability to the training process?
Or will these two targets coincide under some assumption?
soundness: 3
strengths: 1. MCM achieves a trade-off between DM and CM. Unlike CTM, MCM does not rely on GAN loss to achieve high-quality samples, which yields more stable training and more flexible application.
2. The author discussed that the DDIM step tends to underestimate the variance. The proposed aDDIM can address this mismatch and improve the results.
3. The author provides both lower-resolution and higher-resolution results using MCM, with and without distillation. These comprehensive experiments demonstrate the efficiency of MCM.
summary: This paper extends CMs to multiple steps, achieving a balanced trade-off between CM and DM. Additionally, it introduces an adjusted DDIM step designed to reduce variance mismatch, which arises from using an expected value in the DDIM step instead of sampling from the true distribution. The authors evaluate their approach on several datasets, demonstrating its efficiency and effectiveness.
weaknesses: 1. **(major) the trade-off seems to be limited.** This paper achieves a trade-off between CMs and DMs. However, this trade-off is achieved by set the number of student steps in advance. While this approach demonstrates a clear improvement over standard CMs and DMs in terms of generation quality and speed, respectively, it significantly limits flexibility due to the fixed design choice. The authors also discuss CTMs, arguing that adversarial training is required to ensure high-quality samples. However, I am not entirely convinced that this is a major drawback, as using GANs is a mostly free lunch, and when combined with DSM and CTM loss, the training process is not as unstable compared to standalone GAN training.
Do you think it is possible to train a single model using the MCM objective to achieve a real post-training trade-off? 

2. **(major) some design choices are unmotivated.** 

(a) the $\eta$ in aDDIM is set to 0.75. How is this chosen?

(b) Alternatively, aDDIM can be set so that the variance can be interpreted as 10% of the posterior variance of x if its prior was factorized Gaussian with variance of 0.5. Why this choice?

(c) "We initialize the consistency models from the pre-trained diffusion model weights which we found to greatly increase robustness and convergence." It is not surprising this will help. But what if there is no such initialization? Considering that CM and CTM can be trained without this initialization, it would be helpful to see the results.



3. **Some results are not convincing.**

(a) "Empirically, we observe that the adjusted sampler is important when more student steps are used. In contrast, vanilla DDIM works better when few steps are taken and the student does not get close to the teacher as measured in FID." This seems to be weird to me.
As far as I understand, aDDIM and DDIM will converge when the number of steps becomes larger. Therefore, I will expect the gain to become smaller when using more steps. ADDIM will be more accurate (in terms of variance) compared to DDIM when the step number is small. However, it seems that the experimental results are the opposite. Any explanation?

(b) This may be a little too much. But I am asking just out of curiosity. In the experiments, you use UViT as the network. However, both CTM and CM use the standard UNet. How does this architectural difference contribute to the performance? Will MCM still perform well when using UNet?


4. **Missing related works**. The motivation behind aDDIM is to address the underestimation of variance in the DDIM step. To remedy this, aDDIM increases the predicted noise contribution to match the empirical variance. Similar ideas have been proposed in [1][2] (and also [2] gives a good summary of other approaches in its related work section), though a key distinction is that aDDIM remains deterministic, whereas these methods still involve adding noise. However, given the shared rationale between aDDIM and these approaches, I suggest discussing them in related works.


5. **(minor) writing and formatting is sometimes confusing and contains typos.** Here are some examples:

(a) Algorithm 1. There should be punctuation after the first line. I thought $N_{per segment}$ was part of the first line for a while.

(b)  line 215: I do not think $T$ is clearly defined somewhere?

(c) line 225-226, typo: ...Euclidean distances typically work better than for consistency models than the more usual squared Euclidean distances...

(d) references for tables are not consistent. Line 454/463 use *Tbl*.XXX, but line 446/437 use *Table* XXX.



[1] Bao, Fan, et al. "Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models." ICML 2022.
[2] Ou, Zijing, et al. "Diffusion Model With Optimal Covariance Matching." arXiv.