Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Authors: Yiding Chen
Yiyi Zhang
Owen Oertell
Wen Sun
Venue: ICML 2025 poster (2025)
Keywords: Consistency models
diffusion models
learning theory
Primary area: theory

Abstract: Diffusion models accomplish remarkable success in data generation tasks across various domains. However, the iterative sampling process is computationally expensive. Consistency models are proposed to learn consistency functions to map from noise to data directly, which allows one-step fast data generation and multistep sampling to improve sample quality. In this paper, we study the convergence of consistency models when the self-consistency property holds approximately under the training distribution. Our analysis requires only mild data assumption and applies to a family of forward processes. When the target data distribution has bounded support or has tails that decay sufficiently fast, we show that the samples generated by the consistency model are close to the target distribution in Wasserstein distance; when the target distribution satisfies some smoothness assumption, we show that with an additional perturbation step for smoothing, the generated samples are close to the target distribution in total variation distance. We provide two case studies with commonly chosen forward processes to demonstrate the benefit of multistep sampling.

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Yiding Chen
Yiyi Zhang'
Owen Oertell
Wen Sun
Abstract
ally expensive especially when generating high high-resolution images (Ho et al., 2020; Song et al., 2021; Lu et al., 2022; Zhang & Chen, 2023; Song & Dhariwal, 2024).
Diffusion models accomplish remarkable success in data generation tasks across various domains. However, the iterative sampling process is com- putationally expensive. Consistency models are proposed to learn consistency functions to map from noise to data directly, which allows one-step fast data generation and multistep sampling to im- prove sample quality. In this paper, we study the convergence of consistency models when the self- consistency property holds approximately under the training distribution. Our analysis requires only mild data assumption and applies to a family of forward processes. When the target data distri- bution has bounded support or has tails that decay sufficiently fast, we show that the samples gen- erated by the consistency model are close to the target distribution in Wasserstein distance; when the target distribution satisfies some smoothness assumption, we show that with an additional per- turbation step for smoothing, the generated sam- ples are close to the target distribution in total variation distance. We provide two case stud- ies with commonly chosen forward processes to demonstrate the benefit of multistep sampling.
Consistency model (CM) (Song et al., 2023) is proposed to accelerate sample generation by learning a consistency function that maps from noise data directly. It allows both one-step fast data generation and multistep sampling to trade computation for sample quality. Consistency model can be trained with consistency distillation or consistency training (Song et al., 2023), which enforce that any points on the same trajectory specified by the probability- flow ODE are mapped to the same origin, i.e. the self-comsistencyy prop- erty. Despite the empirical success of consistency models, their theoretical foundations remain inadequately under- stood. In particular, recent studies (Luo et al., 2023; Song & Dhariwal, 2024; Kim et al., 2024) observe diminishing improvements in sample quality when increasing the num- ber of steps in multistep sampling. They find that two-step generation enhances the sample quality considerably while additional sampling steps provide minimal improvements. Such phenomenon motivates the theoretical understanding consistency models, especially multistep sampling.
The analysis of consistency models can be challenging for the following reasons:
Mismatch on the initial starting distributions: Consis- tency models generate samples from Gaussian noise (Song et al., 2023) while the ground truth reverse processes (i.e.. the denoising process) start from the marginal distribution of the forward process, which is unknown in practice. As consequence, we need to analyze the eгror caused by the mismatch in starting distributions. This difficulty shows up even if we have access to the ground truth consistency function: the consistency function is not Lipschitz even for distributions as simple as Bernoulli, which makes challenging analyze this error pointwise. Because the consistency function is the solution to the probability flow ODE, it is natural to consider the stability of the initial value problem. However, without a strong assumption on the con- sistency function, this approach results in an upper bound with exponential dependency in problem parameters (see Appendix A).
1. Introduction
Diffusion models have been widely acknowledged for their high performance across various domains, such material and drug design (Xu et al., 2022; Yang et al., 2023; Xu et al., 2023), control (Janner et al., 2022), and -ox-tto-image gener- ation (Black et al., 2023; Oertell et al., 2024). The key idea of diffusion models is to transform noise into approximate samples from the target data distribution by iterative de- noising. This iterative sampling process typically involves numerical solutions of SDE or ODE, which is computation-
'Cornell University.
Correspondence to: Yiding Chen
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Approximate self-consisteney: While the training pro- cess enforces the self-consistency property, it is impractical to obtain a consistency function estimate with the point-wise exact selficonnistency due to various error sources during training (e.g.. optimization errors, statistical errors from fi- nite training examples). It is thus natural to focus on the case where the consistency estimator only has approximate self- consistency under the training distribution. The key chal- lenge is how transfer the approximate self-consistency measured under the training distribution to the quality of the generated samples (e.g., Wasserstein distance between the learned distribution and the ground truth distribution).
is (elog Rd -close to Paata in W2 distance; two-step sam- e²)
H² pling returns a distribution that is (elog + clooe to Paata in W2 distance.
Our error guarantee scales peacefully in problem parame- ters and is dimension-free. It shows that two-step sampling reduces the error by half in the ideal case (e < 1). Our analysis also suggests that further improvements with mul- tistep sampling is unclear even with an increased number of sampling steps. This observation is consistent with the findings from empirical studies. Additionally, the bounded support assumption can be further relaxed to a light-tail condition.
Complexity of multistep sampling: We still have limited understanding of the theoretical advantages of performing multistep sampling during inference steps of CM. When performing multistep inference, we need to apply the consis- tency estimator repeatly to the distributions that are different from its original training distribution. Since we can only guarantee approximate self-consistence under the training distribution, analyzing the benefit of multistep sampling requires us to carefully bound the divergence between the training distributions and the test distributions where consis- tency estimator will be applied during inference time.
Guarantees in Total Variation distance. Standard con- sistency model cannot guarantee closeness in Total Variation (TV) distance, as the consistency loss enforces pointwise distance, which differs fundamentally from the structure of TV distance. However, we show that incorporating an addi- tional smoothing step provides a guarantee in TV when the target distribution meets certain smoothness assumptions.
1.2. Related Work
The theory of diffusion models has been widely studied. Chen et al. (2023b). Lee et al. (2023), and Chen et al. (2023a) study the convergence of score-based generative model and provide polynomial guarantees without assuming log-concavity or functional inequality on the data. Our data assumption is similar to that of Lee et al. (2023), which is quite minimal. Recently, deterministic samplers with probability-foo flow ODE have been explored from the theoreti- cal perspective (Li et al., 2023; Chen et al., 2024; Li et al.. 2024a).
1.1. Our Contributions
In this paper, we analyze the convergence of consistency models under minimal assumptions and provide guarantees in both Wasserstein distance and Total Variation distance.
Guarantees in Wasserstein distance. Given an arbitrary distribution with bounded support, our main theorem es- tablishes guarantees in Wasserstein distance for multistep sampling with general set of forward processes and approximate self-consistent consistency function estimator. The assumption in our result is much weaker than those in previous works. Previous works make an implicit assump- tion on both the target distribution and the forward process. They assume the ground truth consistency function to be Lipschitz. In contrast, our result only assumes that the target distribution has bounded support. In addition, our analysis applies to broad class of forward processes that captures both Variance Preserving and Variance Exploding SDEs as forward processes, whereas previous work has focused only on the former.
Consistency model, which learns a direct mapping from noise to data via trajectory of probability-flow ODE, is pro- posed to accelerate the sampling step (Song et al., 2023). Song et al. (2023) provides asymptotic theoretical results on consistency models. At high level, they show that if the consistency distillation objective is minimized, then the consistency function estimate is close the ground truth. However, they assume the consistency function esti- mator achieves exact self-consistency in point-wise man- ner. Such a point-wise accurate assumption is not realistic and cannot even be achieved in a standard supervised learn- ing setting.
For illustration purposes, we summarize the instantiation of our main result applied to the Omnstenn-Ullenbeck (OU) process:
Lyu et al. (2023), ei al. (2024b), and Dou et al. (2024) pro- vide the first set of theoretical results towards understanding consistency models. Lyu et al. (2023) shows that with small consistency loss, consistency model generates samples that are close to the target data distribution in Wasserstein dis- tance or in total variation distance after modification. Li et al. (2024b) focuses on consistency training. Dou et al. (2024)
Theorem 1 (informal, see Theorem 2 and Corollary 1). Suppose the consistency function estimate is è accurate and the support of the target distribution Pama is bounded by R. then one-step sampling returns a distribution that
2

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
provides the first set of statistical theory for consistency models. However, notice that all of these works require strong assumption on the data distribution. Specifically, they assume that the ground truth consistency function is Lipschitz. While the Lipschitz condition allows a direct ap- proach to control the error of mismatch on the initial starting distribution, it's unclear how large the Lipschitz coefficient is. Intuitively, when the target distribution has two modes, the derivative of the ground truth consistency function can become large near the intersection of the pre-images of the two modes. This occurs because small perturbation in the input can cause the output of the consistency func- tion to shift from one mode to the other. This intuition is formalized Appendix A. To overcome this, we use the data-processing inequality, which only requires approxi- mate self-consistency and minor assumptions on target data distribution. Moreover, our upper bound is polynomial in problem parameters. Additionally, all of these works focus only on variance preserving SDEs while our results apply to general family of forward processes.
g()) (Bakry al., 2014; Song et al., 2021) (interested read- ers may refer to Lemma 3 for an explicit dependency on the noise schedule). The convergence of the forward process facilitates a procedure to generate samples from Pana. ap- proximately: generate a sample from (0, of 7) and feed it to an approximate reversal of (I), However, the reverse-time SDE of (1) is usually computationally expensive.
It is known that the following probability flow ordinary differential equation (PF-ODE) generates the same distribu- tions as the marginal distribution of (1) (Song et al., 2021):
dx dt
1 h(t)x; 20 (t) Xo Paata. (2)
The time-reversal of (2) defines a deterministic map- ping from noise to data, which facilitates consistency model (Song et al., 2023) as computationally efficient one-step sample generation.
Consistency models: A consistency model (Song et al., 2023) is an alternative approach to generate samples from Pdam: instead of solving the reversal of the SDE in (1), one could directly learn a consistency function that maps point on a trajectory of (2) to its origin. For any x and ty 2 0. let {<[(; x, 40) 1o) bejo, be the trajectory specified by (2) and initial condition Xto = x.' The (ground truth) consistency function of (2) is defined as:?
2. Preliminaries
Score-based generative models (Song et al., 2021) and con- sistency models (Song et al., 2023) aim to sample from unknown data distribution Pasaa in Rdd We review some ba- sic concepts and introduce relevant notations in this section.
Score-based generative model: A score-based generative model, or diffusion model (Ho et al., 2020: Song et al., 2021) defines a forward process (x)) by injecting Gaussian noise into the data distribution Pista in d-dimensional space Rd, where X Pamta and T >>000 0. In this paper, we focus on a general family of forward processes characterized by stochastic differential equations (SDEs) with the following form:
(x, = :0;x)) x,0;;,t),, t), EXER t22 0.
(3)
A consistency function enjoys the self-consistency property: if (x, t) and (x', , ,t) are the same trajectory of (2), they are mapped to the same origin, i.e. fx''xx
The self-consistency property of the ground truth consis- tency function (:,) enlightens the training for consis- tency function via enforeing the celf-consistency property instead of learning the mapping from noise to data directly. At high level, in the training stage, we first discretize the interval [o, 2 with the following partition:
dx:- = h())x,dl+ + g())dw,, X ~ Paata
(1)
where w. is the standard Wiener process. It is known that the marginal distribution of x, in (1) is Gaussian condition- ing on Xy (Kingma et al., 2021; Lu et al., 2022):
T = To < T1 < T2 < ... < TM = T.
xr|x0 ~ N of1). V [0,7],
'Specifically, (p(:; x, to) is the solution to the ODE initial value problem specified by (2) and Xt, = x
where 0t, E R+ is specified by A(t)) dogaa (1) = do? - 2 of with proper initial ot
2Song et al., 2023) stops time t = 8 for some small 8 >> 0 and accepts f(x,t) = x(,;x, f). an estimate for yo (,6;,,1 t) as the approximate samples to avoid numerical instability. In this paper, we ignore this numerical issue to obtain cleaner theoretical analysis.
dr conditions. a and of specifiy the noise schedule of the forward process. The noise schedule { (f)),))) xe (o, and initial data distribution determine the marginal distribution of the forward process {P,} E []. where X4 ~ Pt and
'At a high level, this can be shown by contradiction: suppose (x', t') lies on the trajectory of (x, f), meaning tt) the tra- jectory of (x, t) and yp x', t'), the trajectory of (x', (2) intersect at (x',t). Then both trajectories satisfy the initial condition that takes value x' at time f'. By Picard's 's existence and uniqueness theorem, the trajectories of gp:: x, t) and pp: x f') are identical and have the same origin.
Po = Pama- We use (pt) hee [o. to denote the probability density functions (PDFs) of (Pr)ee0.
The forward process specified by (1) converges to Gaussian distribution N0, of for some properly chosen h(-) and
3

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
For simplicity, we assume the partition is equal, i.e. there exists Ar > 0, s.t. 7; Ar.i, -i, for i ,.....: M. (We defer the adaption to non-uniform distization to Appendix G) We then enforce the self-consistency property on each partition point by finding some f(: ), s.t.
and the multistep sampling procedure introduced above. To quantify the sample quality, we establish upper bounds on 2- Wasserstein distance (W2) in Euclidean norm, and upper bounds on Total Variation (TV) distance. The 2- Wasseerstei distance between two distributions P and Q is defined as:
Ex [| v3) - X,: R.), T441 |
Wa(P,Q)= == -in (Er(P.Q) V |Bixesss [|x - y1la]
(4)
where [(P, Q) is the set of all joint distributions such that the marginal distribution over the first random variable is P and the marginal distribution over the second random variable is Q.
is small for all 0. M 1. This strategy is jus- tified by our theoretical results in Section 3: even if the self-consistency property is violated slightly, the consistency function estimation will produce high-quality samples. In practice, the trajectories of the PF-ODE (2) are unknown, so the self-consistency objective cannot be optimized directly. With this regard, consistency distillation, which utilizes a pre pre-trained score function estimate, and consistency train- ing, which builds an unbiased estimate for the score function, proposed to approximate the transition on the trajecto- ries of the PF-ODE. Interested readers can find the details in (Song et al., 2023).
Total Variation distance between two distributions P and Q is defined as:
1 TV(P,Q) :- -1ll(x1xxx-xxxx)(())) -V((-))=5lp(x> g(x)1l2
where p(.) is the PDF of P Pnd q(.) is the PDF of Q-
3. Main Results
Multistep sampling: Given a consistency model estimate ((s)), we could generate approximate samples by feeding Gaussian noise into -(:--) using single-step or multistep sampling. Given &r ~ I)), one can generate sam- ple in single step by calculating f (T),,T). Furthermore, one can also design a sequence of time steps by selecting N 21 steps in the training partition T:
In this section, we present theoretical guarantees on sample quality for consistency models with multistep sampling. We first present two sets of results for the general forward process in (1) with arbitrary sampling time schedule: in Section 3.1, we demonstrate that the generated samples close to the target data distribution Pasm in W2 when Paata has bounded support or satisfies some tail condition; with an additional smoothing step, we show guarantee in TV distance for Pama with smoothness condition in Section 3.2. To illustrate the general results and gain better understanding on the multistep sampling, we choose two special SDEs as forward processes and design sampling time schedules in Section 3.3.
T t > 12 ... > IN > 0,
(5)
We refer to the sequence {t,)i=i. TTV \ {0} as sampling time schedule. Given this sampling time schedule, one can
alternatingly denoise by calculating x6 -(x (44 and
(6) where x Xr ~ ((0,071) and i = N. The
((+1) ~ N(a Xo , oi !), inject noise by drawing X441
The natural central assumption in our theoretical results is a good consistency function estimate:
(NN) in the last step is the output of the sampling pro- cess. When N = 1, this degenerates to single-step sam- pling. For completeness, we summarize this process in Algorithm 1 in Section B. For a concise presentation, we defines { [ to be the sequence of marginal distri- i=1:N
Assumption 1 (A proper consistency model). Suppose f(x, 0) x for all x E Rdd and there exists eam 0, s.t. (4) < 2 for all = 0, M 1.
The condition related to the accuracy of the consisteney function estimate is necessary: we cannot generate good samples with an arbitrary function. Instead of assuming the output of f(..) and (---)) to be close directly, we only require the self-consistency property to hold approx- imately under its training distribution, which aligns with the objective function when training for f((,-). Note that our assumption does not imply f will be self-consistent in point-wise manner.
(1) (2)0)))) Jimi:N and define { (P())
butions of {x(
to be the f 1:1:
sequence of marginal distributions of ==liN" In the following, we may reuse f(-,-) for operation on distribu- tions. Specifically, for any distribution P and t > 0,we use ((P.t) t) to denote the distribution of f(x, t) when x ~ P. In Section 3, we study how multistep sampling influences the sample quality from the theoretical perspective.
The self-consistency objective (4) can be approximated via consistency distillation or consistency training (Song et al., 2023). Consistency distillation uses a pre-trained score func- tion (an estimation for V Voo Pe (-)) to approximate (p(:::))
Performance metric: In this paper, we study the sample quality generated by a consistency function estimate f(:--)
4

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
and train for ((:-) with target network and online network. In Section F, we incorporate consistency distillation with minor modifications into our framework without additional data assumptions. On the other hand, consistency train- ing constructs an unbiased estimator for V log (x{) to approximate (4). Theorem of Song et al. (2023) shows that the self-consistency loss (4) can be approximated by consistency training under proper conditions when AT is small.
j=m 407 1 an upper bound on KL(Pex ||
Pxx).
accumulates; on the other hand, iy it, the error from an in- accurate consistency function decreases due to shorter ty. The design of sampling time schedule {4:)==:: which depends on the noise schedule { {(a,, of) is crucial in achieving good sample quality. We defer design choices for some specific forward processes and simplified upper bounds to Section 3.3. Indeed, both our simulation in Ap- pendix I and the results in Luo et al. (2023) provide empiri- cal evidence that increasing the number of sampling steps can harm the sample equality.
In (4), we use 11-112 as an error metric, which agrees with the choice in practice (Luo et al., 2023; Song et al., 2023). The metric 1.112 aligns better with the theoretical analysis: on the one hand. Lemma 2 demonstrates that this metric translates naturally to the 2- Wasserstein metric W2; on the other hand. I1-1/2 more suitable for the multi-step sampling because the squared error contracts nicely in the forward process with Gaussian noise as shown by Lemma 1 and 3.
When AT decreases, on the one hand, there would be more intermediate steps in the error decomposition of the consis- tency function estimate given a fix t (see Lemma 2): on the other hand, using a smaller AT allows a smaller fny and may potentially decrease 6.m as well.
The technique in Theorem 2 can be extended to distributions without finite support but with proper tail conditions. The detailed discussion is presented in Appendix D.
3.1. Guarantees in Wasserstein Metric
We now provide upper bounds on the sampling error in W2 distance.
3.2. Guarantee in Total Variation Distance
Theorem 2 (W2 error for distributions with bounded sup- port). Suppose Assumption 1 holds. Suppose there exists R O, s.I. SuPxesupp(Pam) |xx||= S R and [(xx)) 1)1, S R for all tx, E Rd 2 TI Let P(N) be the output of N- step sampling. Then W2 (2(P(,) Pdaaaa) the error in W2 is upper bounded by:
In the sampling process of consistency models, it is non- trivial to control the error in TV distance This diffi- culty arises even when we sample with a single step and have access to the exact marginal distribution Pr. As- sumption 1 ensures that f(Pr. T) is close to f* T) in W2- However, W2 and TV have very different structures: W2 controls the pointwise distance between distributions while TV only focuses on the density of the distribution. Even if (2())) Paaaa) is small, the densities of f(Pr. T) and f T) may not overlap well (see Figure 1a) and (V,,(P, f (T), can be as large as if f(Pr. T) is nearly deterministic while f (Pr, TT) has large variance. As result, it's not possible to control TV distance only with conditions on W2 distance in general.
N
1/4
2R
R²_
cm Ar
+ IN
Eam
(6)
4
40
j-1
AT
j=2
(ii))
(i)
where cem is the consistency loss. R is the diameter of distri- bution. dg are the drift and variance factor in the forward diffusion process. AT the time step size according to the partition.
One solution is to perturb P((N) slightly with Gaussian noise (0,, o²).. With this perturbation, İ(N) * (0 and Paata * (O, could have better overlap and be closer in TV (see Figure 1b), where we use P Q to denote the convolution of distribution P and Q. When Pasta satisfies smoothness assumption, the perturbation will not change Paata too much so TV *N(0 Pasa) is small (See Figure Ic).
Compared to Paata (Pty the sampling error of p(() f(Pry: comes from: (i). the error of starting
from Gaussian distribution (0, IT instead of Pr; (ii)
the error accumulated in the previous sampling steps; (ii). using an inaccurate consistency function estimate f(;,)) in- stead of f* (:,-). .). The term of the forward process as demonstrated by Lemma 3. It con- verges to 0 quickly for reasonable forward SDE (1). Asymp- totically, the right hand side of (6) goes to 0 as 4 + do and 6em O.
of of characterizes the convergence
Theorem 3 (TV error for distributions under smoothness assumption). Suppose Assumption 1 holds. Let (.iau(.) be the PDF of Paata- log (.aaa-)) is I--smooth, then for all
o. > 0, the error in TV distance of the smoothed output, i.e.
One implication from (6) is: increasing the number of sam- pling steps does not necessarily improve the sample quality. of
*We use KTT( || Q) to denote the Kullback-Leine (KI.) divergence of distribution P from distribution Q. which is defined as: KT.(P || Q) == Jxexd p(x) log p(x) dx.
When using more sampling steps: on one hand,
NR2 +
9(x)
5

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
1 of
Punx of0,
Peas o(0.o))
Pam
(a) TV distance between P and Pam.
(b) TV distance between P(N) *N(00001 and Paua * (0, I))).
(c) TV distance between Pasna NNO0, o2 and Pamn-
Figure I: Smoothing by additional perturbation
TV(P) * *N(00 T), Pa is bounded by:
guarantees that W2 (PEN- Pdata) is bounded by
2-2 22 R² 4(1 - e
22, 5-1
1/4
0 40
Exx
21 |
Σ
N
40%
j-1
)-1
22
+
20.
In
cem
AT
2R
4(1 21,
cm A
+NN
Cam
AT
j=2
In this case study, we focus on the design of the sampling time schedule based on upper bound (8). To develop reasonable multistep sampling procedure, we make the fol- lowing two practical assumptions: ÂT << 1 and Ar < R. The condition AT << 1 allows for selecting a small final sampling step ÊN. However, as we demonstrate below, an ultra-small ty not beneficial. The assumption fem < R ensures that the consistency model yields meaningful self- consistency loss. These assumptions are used solely for deriving the sampling time schedule; our theoretical results do not depend on them.
+ -dLo..
Compared to Theorem 2, the upper bound in Theorem 3 has an additional term 2dLo,, This the "bias" induced by the additional perturbation (0, o² I). To get a tighter bound, we may choose a. = V
IN and the upper bound AdlAr
becomes: Ex Pau ||x12] 2- act P2- A7
22
Cum 2 2/ ty
Ar'
One strategy for designing (4)),=ıN is to minimize the upper bound (8). We first establish a lower bound on (8) as baseline. Without loss of generality, we assume t 2 2. (8) can be lower bounded as:
3.3. Case Studies on Multistep Sampling
To illustrate the theoretical guarantee and understand the benefits of multistep sampling, we conduct case studies with two common forward processes. For simplicity, we assume Panta have bounded support and ignore the rounding issues when selecting sampling time schedule [t:]= from the training time partition T.
1/4
N
Cem
t
Cem AT
(8)
>
R
(t/-1
tz)
+ tN
AT
2t;
1/4
we consider the Variance Preserving SDE in Song et al. (2021) with ((t) = 2 as the forward process:
Case study 1:
Eem
rdæ
2 R
+ IN AT
AT
2x
where the first step is because 0 < l S l--1 and the sec- ond step is because -- monotonically decreases. Let
dx; = -x,dt + V2dw, Xy ~ Panta-
C1 > 0 be absolute constants, s.t. (J²
zdz 2
1/4
C2.
This is also known as the CeatteinUUhlenbekk (OU) pro- cess and is studied by Chen et al. (2023b) in the con- text of score-based generative models. The forward pro- cess defined by (7) has noise schedule a, = e and and its marginal distribution is X2 ~ N( -xo (1 e I) conditioning on Xg- Theorem 2
Then if tN > C1. (8) > c1 fcm
Q2( 5um ); if ty C1. In either case, (8)
M og / a Q(R
'When Spm > R. (8) S(R), which is uninformative be- cause the support of Passs is already bounded by R. This trivial scenario is not the focus of this case study.
of = =1 1- -
6

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Q( { fem, R 2+ implies (8) =
Eu Fam AT
The condition fam < R further Ar
lower bound of equation (12). For arbitrary sampling strategy (1 12 2 .... IN. the summation inside term (i)
Given this lower bound, one heuris-
tic is to set every term in (8) to Ô (()) )) to match this
can be lower bounded by:
baseline approximately, which requires:
N
N 4 - 41)
N
11114
A cm
R
tz
4 2 log
=; t 2 log
O.W.
(9)
2
jm
Ccm
(1 dt
log t1 in
With this heuristic, a two-step sampling procedure shows an improvement in sample quality:
bx t
A schedule defined by a geometric series approximates this lower bound well. Letting t tN with p > we have مز= C 27
Corollary 1 (Two-step sampling with OU process). Sup- pose the conditions in Theorem are satisfied. Suppose af = -e-,, of = 1 - -- Then for t = log R²
in o² log f p² logp iN logo EN N 2e log IN
f1.
where equality holds when p = Ve. Substituting this into equation (12), we obtain:
t2- -log AÉAI, we have:
Feal
W2 (Para)) S fcm
Ar
log
A²
+ 0(1).:
2R
41
R
4A72
2ee
log + log 1N
1/4
+ IN
6cm AT
(2( , S
Sam 3r
log
+0(
ReAr
Cem
To minimize this expression, we choose 1 = V2 RAT As tN - 0, the second term decreases linearly, while the first term increases slowly. Therefore, a reasonable choice is IN = AT, a small constant. To summarize, the 4 are specified by: (ignore the rounding issue)
(10)
a fau
Because fum < R, the leading term is strictly reduced Ar in the second sampling step. Furthermore, if Cem R Ar, (2( Pasaa) R 3 W Pana): if 6em < Ar, 0 Paanaa= R (P Pa Due to the constraint in (9), further improvement with this heuristic challeng- ing. This intuition aligns with the empirical result in Luo et al. (2023). Our simulation in Section I demonstrates that the sampling strategy in Corollary 1 achieves accuracy com- parable to baseline strategies while requiring significantly fewer sampling steps.
(2(())
t=ttee
2
i= ...,.
N.
(13)
where 4 = -v? RAT and N = 1 + 2 2log AT-
With this
choice, we obtain the following error guarantee:
Corollary 2 (Multistep sampling with the variance explod-
ing SDE). Suppose the conditions in Theorem 2 are satisfied.
Suppose a = 1, of = t2. For {tr)i= we have: (2(P( Pama)) S Ö 5cm
EaJi=i:N 1:N defined in (13),
Case study 2: In the second case study, we consider the following Variance Exploding SDE (Song et al., 2021; Kar- ras et al., 2022) as the forward process:
For comparison, the best error rate of (12) is O [R fum 1/3) when using a single sampling step AF =N By choosing the sampling schedule strategically, we see clear improvement the error rate when using multiple sampling steps.
dx,= = V2idwe:
(11)
which is used in Song et al. (2023) and Song& & Dhariwal (2024) as the forward process for consistency models. The noise schedule is (ar, of) (1 and the marginal distri- bution of x, conditioning Xy is: X2 ~ NNxx, I..).. upper bound in (6) is simplified to:
To summarize, the convergence of forward process in (1) is characterized by ajo, (according to Lemma 3). The forward process (11) has a polynomial convergence rate afa? =t= while (7) enjoys much faster exponential rate ai0 2 ~ e-2 The exponential convergence results in a shorter training step T. fewer sampling steps N, and better sample quality, provided that Assumption holds with the same eem in both cases.
N
1/4
2R
fem +tN AT
(12)
4
2
This implies trade-off in multi-step sampling with this particular forward process (11) when increasing the number of steps. Roughly speaking, (i) in (12) increases due to more terms with more steps while iN becomes smaller and (ii) will decrease. To design the sampling time schedule strategically, similar to Case study 1, we first estimate the
4. Technical Overview
In this section, we present the high-level ideas in the proof for our main result Theorem 2 since proof for Theorem 3 shares the same main building blocks. The proof for Theo- rem consists of three main components:
7

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Error decomposition:
intuitively, the error comes from:
stepwise decomposition. Conditioning on X7x
Pre. the
(i) inaccurate consistency function f(;;" .) and (ii) sampling from Gaussian distribution (10 o instead of perturbed data distribution Pt. (i) is controlled by the consistency loss Assumption and (ii) is controlled by the convergence of the forward process Lemma 3. However, the error (i) and (ii) interact with each other in the multi-step sampling. We handle this complication progressively, starting with the error decomposition in the final sampling step:
PF-ODE (2) defines deterministic trajectory:
((7k-11-TT))
XTy
XTx
X
Assumption
guarantees
that
- 7-1) is small in the sense
of L error for each intermediate step j. We could make the following decomposition:
(2 Pana) SWa(f(Pes AN), f(PN [N))
f(x,) TK) (x r2) =
1 f(xT (w) Xu
W2lf(Pi
[(xx)))T)) f( (xx)-11
Since the output of f(.-)) -) is bounded, we could simplify the first term with the TV distance, which is further up- per bounded by (Tt(PtN Ptx) by Pinsker's inequality and data processing inequality. The second term is solely controlled by the consistency loss 6cm-
The right-hand side is, roughly speaking S Th f.m We for- malize this idea with Minkowski inequality in Lemma 2.
5. Conclusion
Recursion on [I(P: II Pt)): we analyze KL(Ptx Ptx). via induction. First of all, the base case (Lt, I P..) is upper bounded using the conver- gence of the forward process; the induction step connects KL(Pr, P(.) and (L(P || Pu.1). According to the multi-step sampling, Pr, and Prtza is connected by f(s t.) and the forward SDE as
In this paper, we study the convergence of the consistency model multistep sampling procedure. We establish guaran- tees on the distance between the sample distribution and data distribution in terms of both Wasserstein distance and total variation distance. Our upper bound requires only mild assumptions on the data distribution.
Future research directions include providing lower bounds on multistep sampling and establishing end-to-end results on consistency models.
Pr. Rirtt, ((i) SDE Pre.
In this process, -(.-)) induced additional error while the forward SDE reduces it with convergence o?..,, -... This intuition is formalized by the error decomposition via chain rule of KL divergence:
Acknowledgements
Wen Sun is supported by NSF IIS-2154711, NSF CAREER 2339395, DARPA LANCER: LeAring Network CybERa- gents, and Sloan research fellowship.
KL(Pra 11)11)
af. E
< <KI(P, || P..) +
f(x, +(x
20 P-
Impact Statement
Another possibility is to construct the recursive formula for (2( Po). However, recursion on W2 requires the translation from KL to W2 that induces an R factor in each induction step. When {t,}, is not carefully designed, the R in each induction step results in an exploding upper bound casily. Meanwhile, this translation requires the data distri- bution to be bounded and hampers the application to more general data distributions.
This paper presents work whose goal is to advance the theo- retical understanding of the consistency model. Due to the theoretical nature of this paper, we do not feel the need to highlight the societal consequences.
References
Bakry, D., Gentil, I., Ledoux, M., et al. Analysis and geome- try of Markov diffiusion operators, volume 103. Springer, 2014.
Error of consistency function evaluation: another im- portant building block in our proof the evaluation error of consistency function,, i.c. [f(x, f (x, for
Black, K. Janner, M., Du, Y., Kostrikov, I., and Levine, S. Training diffuusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
Th E T. Assumption 1 controls the difference in f(...) and (*(---) indirectly by enforcing the consistency property. We connect the evaluation error and consistency loss via a
Chen, H., Lee, H., and Lu, J. Improved analysis of score- based generative modeling: User-friendly bounds under
8

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
minimal smoothness assumptions. In International Con- ference on Machine Learning, pp. 4735-4763. PMLR, 2023a.
Li, G., Huang, Y., Efimov, T., Wei, Y., Chi, Y., and Chen, Y. Accelerating convergence of score-based diffusion mod- els, provably. arXiv preprint arXiv:2403.03852 2024a.
Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. R. Sampling easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations, 2023b.
Li, G., Huang, Z., and Wei, Y. Towards a mathematical theory for consistency training diffusion models. arXiv preprint arXiv:2402.07802 2024b.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C.,, and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances Neural Information P Processing Systems, 35:5775-5787, 2022.
Chen, S., Chewi, S., Lee, H., Li, Y., Lu, J., and Salim, A. The probability flow ode is provably fast. Advances in Neural Information Processing Systems, 36, 2024.
Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Dou, Z., Chen, M., Wang, M., and Yang. Z. Theory of consistency diffiusion models: Distribution estimation meets fast sampling. In Forty-first International Con- ference Machine Learning, 2024. URL https: //openreview net. foree//ooummiida -iPpdddyuuuu......
Lyu, J., Chen, Z., and Feng, S. Convergence guarantee for consistency models. arXiv preprint urXiv:2308. 11449, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-666 6851, 2020.
Oertell, O., Chang, J. D., Zhang, Y., Brantley, K., and Sun, W. RI for consistency models: Faster reward guided text-to- to--to--iggee generation, 2024.
Huang, D. Z.,, Huang. J., and Lin, Z. Convergence analysis of probability flow ode for score-based generative models. arXiv preprint arXiv:2404.09730, 2024.
Rolland, P. T. Y. Predicting in uncertain environments:
methods for robust machine Iearning. Technical report, EPFL, 2022.
Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. arXiv preprint Xiv:2205500991, 2022.
Song, Y. and Dhariwal, P. Improved techniques for training consistency models. In The Twelfth International Confer- ence on Learning Representations 2024. URL https: / /oopeneeveww.co net forum? -ind-iddddiia- WNZyybRDVG,
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35: 26565-26577, 2022.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. In In- ternational Conference on Learning Representations, 2021. URL htt.h:ttt////o.ttm http::/:///peeeieee.... hwtpeeiwwwwwwweeeeeeeeeeeeeeeeee forum? id=PxTIG12RRES.
Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Ue- saka, T., He, Y., Mitsufuji, Y., and Ermon, Consistency trajectory models: Learning probability flow ODE trajec- tory of diffusion. In The Twelfth International Confer- on Learning Representations, 2024. URL https: //openreview net. forum 1 FeDTD.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. arXiv preprint 2xXiv:2303.01469 2023.
Xu, M., Yu, L., Song, Y., Shi, C., Ermon, S., and Tang. J. Geodiff: A geometric diffusion model for molecular con- formation generation. arXiv preprint arXiv:2203.02923; 2022.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Lee, H., Lu, J., and Tan, Y. Convergence of score-based generative modeling for general data distributions. In In- ternational Conference on Algorithmic Learning Theory, pp. 946-985. PMLR, 2023.
Xu, M., Powers, A. S., Dror, R. O., Ermon, S., and Leskovec, J. Geometric latent diffusion models for 3d molecule gen- eration. In International Conference on Machine Learn- ing. pp. 38592 38610. PMLR, 2023.
Li, G., Wei, Y., Chen, Y., and Chi, Y. Towards faster non- asymptotic convergence for diffusion- generative models. arXiv preprint arXiv:2306.0 09251, 2023.
Yang, M., Cho, K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., and Cubuk, E. D. Scalable diffusion for materials discovery. arXiv e-prints, 2023.

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Zhang, Q. and Chen, Y. Fast sampling of diffusion mod- els with exponential integrator. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL https::/spen.evieer net/ Forum? dd=iook7hfb445
10

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
A. The Lipschitz Condition of Consistency Function
this section, we consider a simple example, showing that the Lipschitz coefficeint the ground truth consistency function can be exponential problem parameters. As a result, the Lipschitz- based assumptions can be limited.
We consider 1-D example where the target distribution Paata Po is Gaussian mixture distribution (5,((1,, + (.,, where 0 < < 1. Let f(-) be the consistency function that maps samples from Pr samples from Po- where 7 is sufficiently large. We consider OU-process as the forward process so Pr N(,,1) 1). We will show that L, the Lipschitz constant of f is at least exponential in 2.
We first list few supporting claims with brief explanations:
C1: f(-) is continuous. This is because f is the solution the PF-ODE, which is continuous under proper conditions;
C2: f(0) = 0. This is by symmetry;
C3: Pix=N(0.0) 2 & exp ( (-2Ø7)
C4: for 0 < 1 Prxwx,, (X E [0, x]] 2 0.1x for I" properly large. This is becuse Pr is close to Gaussian for large T.
Here the proof: Let E (0,1) be small enough s.t. f(x) E (0, 1). Then, by C1-4:
0.1r PI X E [0, æ|] = Pr [f(x E [f(0), f())]] = = IY e [0,
50.5PI I E [0, f(x)] + 0.5 (1 [Z E E0 f())]
s PrZz~N(0,02 72 - f()) S exp (- (1- (xx)
20
In particular, choose z exp (- - we have:
(1pexp
1
2
80
By definition,
[s() s(y)
/ @10exp
(-
f(0)
1
L sup
exp
yl
10exp
(-
80
This confirms that the Lipschitz constant L must grow at least exponentially in o in this example.
B. Multistep Sampling
We present the multistep sampling procedure Algorithm Compared to Algorithm of (Song et al., 2023), we allow different choices of noise schedule in Algorithm
Algorithm 1 Multistep Consistency Sampling
1: Input: a trained consistency model f(.,) )) noise schedule {(as, } re sampling time schedule {ts):= where
iy T.
2: x, -=, of, I)
3: for :rri=1 to
do
4: x0 + f(x()) 4))
5:
(1+1)
t141 N( 8 2
6: end for
7: Output: x
(N)

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
C. Proof of Theorem 2
At a high level, we could decompose the W2 error W2(PO Panaa) into:
W2 Paam) SW2(PoM) f(Pt (N)) + (P(f(Ptx: Pama) -Wlff(Pe (N) f(PNN tNl) + Wo(f(PtN: (N). (Pe,,,
(14)
=:dda
In the enror decomposition (14): the first term on is caused by inaccurate noise distribution PtN and is controlled by the KL divergence of N. from Pix. We use the chain rule of KL divergence to derive recursive formula for (L(Pt, || Pt).. where the initial term (L(Pa, Pt.) is bounded by the convergence of the forward diffusion process.
Lemma (Decomposition of KL). Suppose f(..-) satisfies Assumption I, then for all ..... N, we have:
(I,(P, || Pt.) S of Exx Pare ||x|2
2
com
207
+
20
j-1 A
5=2
We defer the proof of Lemma to Section C.I. Given this result, can bound of as:
o1 52R TV(P(PeN, ((Pxs
(By Section 2.2.4 of (Rolland, 2022) and [ sn)
<2R
kL.[fPtw. (N) (w))
(By Pinsker's inequality)
KL(Pex || (
<2R
(By data processing inequality)
1/4
<2R
a
Exx Pan
||x
fmm Ar
(By Lemma with = N)
40
40 l
1-1
j=2
1/4
2
<2R
ii
R²
2
cm
Because SUPxesunp(Pau) ||x|| S R).
(15)
40
4
Ar
1j
The second term de is caused by the difference between the pre- mre-trained consistency function f(:,.) and the ground truth ("),,), which is controlled by the consistency loss t.m-
Lemma 2. Suppose f((:.) satisfies Assumption holds, then for all = 0, 1,..., M, we have:
(i) Ex-Pr, [| (1) ffx T3) 10)
S 5T A72
(ii) Ws(f(P,,, TT). (P T()) TT ST.
We defer the proof of Lemma to Section C.I. Part (ii) of Lemma shows that:
Gan
A S tN
(16)
AT'
We finish the proof of Theorem by combining (15) and (16).
C.1. Proof of Auxiliary lemmas
Proof of Lemma J. We prove this statement via induction. At a high level, the base is proved by the convergence of the forward process Lemma 3. We show the induction step by the chain rule of KI..
12

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
We use D (:; Oley of) to denote the operator on distributions defined by a noise schedule (at, af) Specifically, given any distribution P. D (P: ot of) is the marginal distribution of x', where x'|x ~ N(axx, of) and P. When it is clear from the context, we use D (t, () as shorthand. When i = 1, we can write Pr NN(0, oZ,) with the diffusion operator and a the dirac distribution:
P.. - D (60; Oht, ; ) where 6g is the delta distribution at O. By definition,
Pt, D (Po: 010, of2)) By Lemma 3,
ai ::2 : oi.) D (do: on : o;,)) KL(P, || Pt,) KL((D) ae1
an Exe Pare ||x|] 2
(2 50) 20
20
Thus the statement holds for i = 1. Suppose the statement holds for i = k, i..
k
2
2
KL(Pie I1 Pi.) S E
20
Pam
||x|
201,
12
j-1
(17)
j=2
We first explicitly write the sequence of random variables in the multistep inference:
x x
(N)
where x ~ NN of 1). x = f(x))) 4.), X ~ N(r,,, (1-1) of l). Similarly, we also define the following
(6+1)
process that starts at the ground truth noise distribution Pt, and evolves using the ground truth consistency function f (::))
X12
(2)
+ (2)
+x1
(N) Xix
x0
(N)
where x
Pc, 10) = (x(] 41), x (atttti x , l)-
By the chain rule of KL divergence, we have:
KL(P (1tt1) || P (x((+t)))
+ XwP (xx2xx)))) (
KL(P
Xii Xiizi
(k) (1++1)
= =x) P
(x (x1)) (((ttt) = x))
20
KLL(P
Xtx x
(k) (k+1))
P
(p(x(()
X x 2((t1))
-KI(P
x
(k) P
(
(K)
+ E
x~ (x())) )
[LL
(tpp(((tt))) (R+11) x (x
x) P
( ((++1)) |x
where we use P(x) to denote the distribution of random variable x. Because KL is non-negative, we have: (x((ttt))
KL(P x
(k++))
||
P
<KL(P Xix
(1)) P
X
(k)
+ B x
[KL(P(x (((t2)) (8)
=x) P
((xxtttt)
(k+1) (xx1tt2) pe
(K)
]
By definition, this means:
KL || Preni)
< <KL(Po, PPxt) + ExmPi (a ( 1) 11 || t oi 11)
arrm
[ - f(x, tx)) |
27
= =KL(Pt, || (Pa)t + 20
Ex-PIr Psp
20t
< <KI(Pb (Pt)) + a
2
2
(By part (i) of Lemma 2)
20
202
A7
< ai Exx Pan 200
k+1
0
2 (By (17))
||x
2
2
20
j-1
Ar?
1=2
13

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Proof of Lemma 2. We first prove part (i) with induction l. By the definition of (*(++) in (3),
(x) (000 0) =x, Vx ER..
By Assumption 1. f( for all x. Thus
Px [1((x,0) (xxx00))xxx00)
x|3 Px - 0,
which means (i) holds for i = 0.
Suppose (i) holds for i = s, i.e.
[1xx),75) - (x (x(xxT=s) T0)
ExnPr
S Trfcam/ AT.
(18)
By the property of the PF-ODE (2),
((T) ~ Prote
i Pr.
(19)
When i = s 1, we have:
( (x', ro+1)
Ext Poot
[ - ( T.), Ts+ [
ExmP,,
(By (19))
[[(10tta41xx)))744444))) fxx To
ExmPr,
(By the definition of (:)
ExmPr, [ 1.), f(xx Ta) 13)
ExwPr, [[ fx
2
(By Lemma 5)
Seem + T.Cemu/ AT (By Assumption and (18))
=Cem (1 + Ta/ AT) = Ts+16em/ AT.
We complete the proof for part (i). f(;tt) and (*(tt) induce joint distribution I xx
Pr(xxx))-T,, 1(x6s xo) E A] Prxp x, E fx: (xx, e a]]
for any event A. With this joint distribution I xx the marginal distribution of x is f(Pr, and the marginal distribution of xo is (P,,t). t). This means:
V Ex (((e. (x(,, 1) ( 26 [|xx) -xo13 2 ((.
By applying part (i). have
W) S Ttfem/AT.
We complete the proof for part (ii).
14

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
D. Generalization to Distributions with Tail Conditions
When Pata satisfies some tail condition, it sufficient to sample only from bounded region:
Theorem 4 (W2 error for distributions with tail condition). Suppose there exists C. > and R 2 C, s.t.
P [| 2 t) & ce- for all t 2 R. Let PammnBB0,, be the distribution truncated from Pdata i.e. the con- ditional distribution of x given ||x||2 & R where x ~ Paata. Let pp((;-.)) be the solution to the corresponding PF-ODE and :) be the corresponding consistency function. Les RR) TEEOTTT] be the marginal distribution of the forward process starting from B(0,)) If ((:-)) satisfies: (a) ||i(x,
R. for all (x,1) E [0, (b) f(x,0) x. for all
x; (c) Exp-r [| - (1 T.) T(+1)
S 2am for all O....,M. M for some cem > Then
1/4
(M Pdnna)= S 22
oi 40f
62 R² + Dرز sett 12
+IN + + 26 )
Ar
By restricting the output of ((: )) be B(0, R), the Fuclidean ball with radius R. focus learning the portion of Paata inside the Euclidean ball. This truncation step reduces the problem of sampling from unbounded distribution sampling from a distribution with finite support, the cost of introducing the additional term O 20
Proof. The error term can be decomposed
W2(P)ttx) Pana) S W2(P) PaaariB(o,n)) +
Panta)
(20)
By Theorem
N
2
1/4
(2(t(t)), Pa S 2R
an 40
R²
4
aig
2
1-1
j-1
+ Ee
j=2
For the second term, we first note that
(V(Pa PataamB(0, Pana)
Prxxran |||x||2 R s O(e 8).
By Lemma (Rolland, 2022),
(PaaaannBBB.R Paita) S O(Re 2²
We finish the proof by combining these bounds.
E. Proof of Theorem 3
At a high level, we can decompose the TV distance as follows:
Patta) STV(P!N) (N0 o2)), Pasta * xN((0,22)) + TV(Pamua NO oT)), Paaaa)
TV(P(N)
(21)
The first term can bounded by Lemma and Pinsker's inequality, which shows that the TV distance between ((N) and Pana is controlled after the Gaussian perturbation. While the second term is bounded when Paata satisfies the smoothness assumption, which shows that the perturbation will change Pasaa only slightly. We now illustrate these ideas in detail. We
15

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
first define Ctxin Itii == de. then by Pinsker's inequality and Lemma 1:
TV(P6 *N(O0, I) Panaa N(0,o21)
KL(Paata * *N((0,021) || P(N) (N((0,01))
2
KT.(Ptxt1 PeNts)
2
N+1
2
4
Pan
[|x11]]
+ Σ
40%;
0ig
12
3-1
1
cm
j=2
0g
E
Pan
||x|12
+
N
22
2 +
12 2
4
40
J-1
40
N
j=2
4
ai
Ex
Exu Pau
||x||
21
+
N >
40
17
J-1
2.m +
20.
j=2
On the other hand, by Lemma 4,
TV * (0, o² Panaa) S 2dL.o.-
We complete the proof by combining these two bounds into the decomposition in (21).
F. Connection to Consistency Distillation
Our Assumption1 assumes that the self-consistency property is satisfied approximately, which aligns with both consistency distillation (Song et al., 2023). For simplicity, we consider an OU process to be the forward process:
dx: = -x,dt + 2dw,, Xo
Pauta-
Given the pre-trained score function sxx, t), we train a consistency model from the following ODE:
dx;
dt -x, - sxx,,),, XT ~ NO0, e (1- (T)
(22)
We assume access to an ODE solver, which can calculate sp" the solution to (22), exactly. Even though this solver can be computationally expensive during the training procedure, the consistency model will still be computationally efficient during the inference time.
To avoid distribution shift, we optimize the consistency loss objective (4) using the data generated from (22), instead of that from Pr, the marginal distribution of the forward process. When optimized properly, we can find f. s.t.
[| T:)) (
E Ex,,
f(p(TTH15 T(), T++1)
(23)
is small for all i. Using the same argument in Lemma we can show that (0, el-e-21)1 2') and (O;NN (1 e-27 e-2T))) I)), T) are close in W2, this can be translated into bound in TV using the argument in Section 3.2.
When the pre-trained score function s(x, t) has small Le error, (Huang et al., 2024) show that 4² (0; (1 c I).
is close to Panaa in TV. To conclude, f(V (1 e -2)))), T) is close to Pasta in TV.
G. Adaptation to Non-uniform Discretization
In this paper, we adopt uniform discretization for clarity and ease of presentation. However, our results can be extended to the non-uniform discretization setting as well. Suppose To.M is an arbitrary discretization of the interval [o, T). this scenario, it reasonable to assume that the consistency loss scales with the length of the discretization interval:
Eay ((T,, - (a000-(((((ttaa) f(yp(TH11: m.), Te))) ||]
S (TH1 - 22
16

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Using the same argument in the proof of Lemma derive:
Ba., Pre [1( / (xT:) Ti)
Ex,,nP, NPe [|((00t) [1 Ta)s (o)TT++))
which is upper bounded by: E ( Ta)e- = T2e. The rest of the proof remains unchanged.
H. Technical Lemmas
We first present the result the convergence of SDE, which also connects KL-divergence and W2:
Lemma 3. Let P and Q be two distributions in Rd then
a²
KL(D (P;a, (D((Qi, S
20 (P.Q))
where we use D (P: to denote the marginal distribution of x,, with x'|x ~ (aax of) and x ~ P.
This result is comparable to Lemma C.4 of (Chen et al., 2023a). However, our results is self-contained and tighter.
Proof of Lemma 3. Let U and V be two random variables with joint distribution I, s.t. the marginal distributions of U and V are P and Q respectively. Let X ~ D (P; o²) and Y ~ (Q)a,o²). We use P(-) to denote the distribution of a random variable. By the chain rule KL-divergence, we have:
KL(P(X) || P(Y)) S {L((((X) || P()))
(LWPP)/ =x) (U) x)]
(By the non-negativity of KL)
= -KL(P(U.V) V) P)),
+ [(xx))-P(UU)) (KTTPP
= (I) (IV (u, ub)))]
(By the chain rule of KL)
= (Lux))-P(,.,))LLLPPPPP
u)) P(YV
(24)
(X is independent of V given U and similar holds for Y)
By the definition of D (, u~ (N(uu, and YIV = ~ N(avvo21), Thus,
1 Pllu-vl
KL((PXXUU u)) P(YIV =v) 20 2000²
By (24), we further have:
a²
KL(D (P:a())), || D (Q:a, -
E
(25)
20
By taking inf over I' on both sides of (25), we get:
a² (P..QQ)
KL(D (L: D |I 20
Lemma 4 (Gaussian perturbation a smooth distribution, a variant of Lemma 6.4 of (Lee et al., 2023)). Let P be
distribution in I&d with PDF p(x), if p(x) is L-smooth, then
TV(P, P < 2dLo,
where we use P Q to denote the convolution of distribution P and Q.
Proof. The results follows directly from Lemma 6.4 of (Lee et al., 2023) with o, and o, = a.
17

---

Convergence of Consistency Model with Multistep Sampling under General Data Assumptions
Lemma (Triangle inequality with both Ly norm and Lg norm). Let be random variable in Rd, and J g be mappings from Rd to Rd, then
Exllf((x) + g(x) (xx]]f(x Vp +Ex1lo(x)112)*
Proof.
[x]l]((x)
+ 1la(x))122)) (4ll
(Triangle inequality for L2 norm)
<xxlIf 1/2 Ex (Minkowski inequality).
Simulation
Motivations: Consistency model has already demonstrated its power on large-scale image generation tasks (Luo et al., 2023; Song et al., 2023; Song & Dhariwal, 2024). To verify our theoretical findings, we focus on a toy example that is easier to interpret.
We first refine our upper bound in Theorem 2, where we relax our result for cleaner presentation. We make adjustment to (15) and get:
at 200
N
1/4
sup (Pya
||x - yll2
Ex Pam ||x|
ccm IN
(26)
4
Ar
AT
Simulation setting: We consider OU process as the forward process, which is our setup in Case study 1. For simplicity, we consider Bernoulli data distribution: Pra-Pan. 0] Prew [x 100] 0.5. This data distribution ensures close-form for the ground truth consistency function:
if 50
fx = 100
o.w.
We construct a perturbed f(-.-) accordingly:
0
if < ar
f(x,1)
1000
O.W.
where the sequence a satisfies: Prewp, (r arl -0.5 + 0.000102, Vt. This choice of f(.--) makes sure:
Ex [| - (xx =
This means f(.--) satisfies the first statement Lemma 2 with Sin 1.
We simulate three instantiations of (t,) defined in (5), i.e. the sequence of time steps for multi-step sampling defined in (5):
schedule: the two-step schedule suggested by Case study We also calculate the upper bound in (26) for comparison;
baseline 1: design the sequence of sampling time steps by evenly dividing an interval;
baseline 2: start with some T' and reduce it by half every step until reaching small value.
In Figure 2, we plot the W2 error in multi-step sampling. We present the revolution of W2 error in sampling time schedule on a single curve. Specifically, we plot each curve by:
(1. (2( Paan)) i= 1,.,..., N.
Because the sampling time step t; decreases in the multi-step sampling by definition. We reverse the z-axis of the plot for presentation purposes.
18

---

Convergence of Consisteney Model with Multistep Sampling under General Data Assumptions
30
our schedule
our upper bound
25
baseline
baseline
20
15
10
0
20.0
17.5
15.0
12.5
10.0
7.5
5.0
2.5
0.0
smaple time step
Figure 2: W2 error in multi-step sampling.
Observations:
This simulation result demonstrates that:
Our upper bound is reasonable characterization of the performance for the designed sampling time schedule.
The two-step sampling time schedule suggested by Case study achieves comparable performance the best result in the baseline methods but with much smaller number of function evaluations;
Running too many sampling time steps may degrade the sampling quality. The error increases for both baseline methods in the last few sampling steps.
19


Reviews:
[{"review_id": "jeYMbYGkPw", "reviewer": "Reviewer_eZRV", "summary": "This paper analyzes consistency models—a recently introduced approach for accelerating sampling in diffusion-based generative models. Unlike classical diffusion models that rely on multiple iterative score-based updates, consistency models learn a direct mapping (“consistency function”) from noise to data while preserving the so-called self-consistency property. This enables both one-step sampling and optional multi-step refinement.\n\nIn this paper, the authors showed that if the self-consistency property holds approximately (quantified by a small “consistency loss”), then one-step or multi-step sampling from a CM can approximate the target data distribution in the Wasserstein-2 distance. Different from other relevant works, the analysis requires only mild assumptions on the data distribution (e.g., bounded support or sufficiently fast-decaying tails for the Wasserstein results, and log-smoothness for the TV results). The authors proved that two-step sampling can yield appreciable improvement over single-step sampling, but adding more than two steps leads to diminishing gains—mirroring empirical observations in prior work.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Please see our detailed responses below:\n1. **Regarding high-dimension issue:** even when accounting for the implicit dependency on the dimension, our upper bound remains at most polynomial in dimension and thus does not suffer from the curse of dimensionality. For example, in Corollary 1, our result shows that given an $\\epsilon$-accurate consistency function estimate, the Wasserstein distance $W_2(\\hat P, P_{\\text{data}})$ depends only on consistency error $\\epsilon$ (assuming $\\Delta \\tau = 1$ for simplicity) and the diameter of the distribution $R$. Considering the implicit dependency on $d$:\n    - **the diameter $R$ grows at most polynomially with $d$:** if the ground-truth distribution $P_{\\text{data}}$ has support on the $d$-dimensional hyper-cube $[0,1]^d$, then $R=\\sqrt{d}$; if the ground truth distribution $P_{\\text{data}}$ has support only on a low-dimensional manifold, e.g. $[0,1]\\times \\{0\\} \\times \\cdots \\times \\{0\\}$, then $R$ can be some constant.\n    - **the consistency error $\\epsilon$ scales at most polynomially with $d$:** by definition, $\\epsilon$ arises from a $d$-dimensional prediction problem and is expected to scale at most polynomially in $d$.\nGiven these factors, we believe our bounds remain both reasonable and meaningful in high-dimensional real-world scenarios.\n2. **Regarding experiments:** Please refer to our Appendix G for simulations related to Case study 1. We show that for two heuristic sampling strategies, increasing the number of sampling steps yields diminishing improvements or even degraded performance. In contrast, our sampling strategy nearly matches the best performance of the heuristic strategies with only two sampling steps. Furthermore, our theoretical upper bound closely approximates the Wasserstein distance observed in practice during sampling. Additionally, Tables 1 and 2 in Luo et al. (2023) show that in LCM, 2-step sampling significantly outperforms 1-step sampling in terms of FID score, while 2-, 4-, and 8-step sampling exhibit comparable performance. Notably, 8-step sampling even yields worse FID scores than 4-step sampling. These empirical findings further support our theoretical results.\n3. **Regarding adaptive schedules:**\n    - In **Case Study 1**, we optimize the general upper bound in Theorem 2 by choosing $t_j$'s strategically and propose a two-step sampling strategy that adapts to both the problem parameters (the distribution diameter $R$) and the learned $\\hat f$ (the consistency error $\\epsilon_{\\text{cm}}$);\n    - For **Case Study 2**, here is a strategical derivation for a sampling schedule. In order to adjust the $t_j$'s to minimize the error upper bound, we first estimate the lower bound of equation (12). For an arbitrary sampling strategy $t_1 \\ge t_2 \\ge \\ldots \\ge t_N$, the summation inside term (i) can be lower bounded by:\n  $$\n  \\sum_{j=2}^N \\frac{t_{j-1}^2}{t_j^2} \\ge \\sum_{j=2}^N \\frac{t_j(t_{j-1}-t_j)}{t_j^2} \n  = \\sum_{j=2}^N \\frac{t_{j-1}-t_j}{t_j} \\ge \\int_{t_N}^{t_1} \\frac{d t}{t} = \\log \\frac{t_1}{t_N}.\n  $$\n  A schedule defined by a geometric series approximates this lower bound well. Letting $t_j = t_N \\rho^{N-j}$ with $\\rho > 1$, we have $\\sum_{j=2}^N \\frac{t_{j-1}^2}{t_j^2} = \\rho^2 \\log_\\rho\\frac{t_1}{t_N} = \\frac{\\rho^2}{\\log \\rho}\\log\\frac{t_1}{t_N} \\ge 2e\\log\\frac{t_1}{t_N}$, where equality holds when $\\rho = \\sqrt{e}$. Substituting this into equation (12), we obtain: \n  $$\n  2R \\left(\\frac{R^2}{4t_1^2} + \\frac{2e\\epsilon_{\\text{cm}}^2}{4\\Delta\\tau^2}\\left(\\log t_1 + \\log\\frac{1}{t_N}\\right)\\right)^{1/4} + t_N\\frac{\\epsilon_{\\text{cm}}}{\\Delta\\tau}.\n  $$\n  To minimize this expression, we choose $t_1 = \\sqrt{\\frac{1}{e}}\\frac{R\\Delta\\tau}{\\epsilon_{\\text{cm}}}$.\n  As $t_N \\to 0$, the second term decreases linearly, while the first term increases slowly. Therefore, a reasonable choice is $t_N = \\Delta\\tau$, a small constant.\n   - In addition, for any new sampling strategy, one only needs to specify the noise schedule $\\{\\alpha_t,\\sigma_t^2\\}$ and the sampling steps $(t_1, t_2, \\ldots, t_N)$ in Theorem 2. The majority of the proof remains unchanged.", "claims_and_evidence": "The theoretical claims follow from some widely used lemmas (e.g., chain rule of KL, Minkowski’s inequality for L2 errors) and do not appear to be overstated or overclaimed. There are no large leaps of logic, and each main theorem is accompanied by a clear proof outline. \n\nClaim 1: Wasserstein Guarantees: The derivations rely on standard techniques—Pinsker’s inequality, data-processing inequalities, and mild assumptions (such as bounded or suitably light-tailed distributions). The authors' arguments are mathematically sound and are aligned with recognized results in diffusion modeling theory.\nClaim 2: Total Variation Bounds: The authors show that smoothing with a small Gaussian convolution can produce a valid bound in TV distance. This theoretical technique is consistent with prior approaches in generative modeling when bridging the gap between pointwise errors and distributional overlap.", "methods_and_evaluation_criteria": "This paper is a completely theoretical work and there is no benchmark or datasets involved or required.", "theoretical_claims": "Yes, I checked the correctness of the proofs.", "experimental_designs_or_analyses": "This paper is a theory work and there is no experimental designs or analyses needed.", "relation_to_broader_scientific_literature": "Prior theoretical works (e.g., Lyu et al. 2023; Dou et al. 2024) examine consistency models but often under stronger Lipschitz assumptions or only variance-preserving SDEs. This paper’s novelty is in requiring fewer assumptions on the data distribution (“light tails” or “bounded support” rather than strict Lipschitzness) and analyzing multi-step sampling for general forward processes.\n\nThey also draw connections to analyses of probability flow ODE solutions in diffusion modeling (Chen et al. 2023, Li et al. 2024, etc.), but adapt these arguments to the direct one-step or few-step sampling approach.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "1. Some proofs rely on dimension-free arguments (via Wasserstein and pinned distributions), but real-world data can be extremely high-dimensional. Whether the derived rates remain meaningful in large-scale real tasks is still an open question. I would like to ask the authors about the high-dimension issue, whether curse-of-dimension will make the derived rate in the high-dimensional case unreasonable in the real world scenario. \n\n2. It would be very helpful if the authors can do some toy experiments to add some simple empirical ideas. Even a small synthetic or 2D test could give a sense of how the theoretical results manifest empirically. For example, the paper’s theory suggests that after two steps, gains diminish significantly. Could the authors share real-data experiments or references to confirm that this theoretical phenomenon fully matches practice?\n\n3. You propose different strategies for picking sample times (e.g., geometric halving). Have you considered adaptive schedules dependent on learned \\hat{f} or data statistics during sampling? Will it make any difference to the proof? \n\nTo sum up, this paper offers a valuable theoretical understanding of consistency models under mild data assumptions, particularly clarifying the trade-offs with multi-step sampling. The main theorems are carefully argued, the bounding steps are standard but precise, and the conclusion that “two-step sampling yields a notable boost while further steps add smaller improvements” is both practically and theoretically important.", "other_comments_or_suggestions": "Please refer to the \"strength and weaknesses\" section.", "questions_for_authors": "Please refer to the \"strength and weaknesses\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "There is no ethical review concerns since it is a theoretical work."}}, {"review_id": "7IZga5fUd2", "reviewer": "Reviewer_kTQM", "summary": "The paper studies the convergence of consistency models with assumptions on the consistency property. It further assumes that the target data distribution has bounded support. In this case, it shows the convergence result in Wasserstein distance and total variation distance. The theoretical results indicate the benefit of multistep sampling with consistency models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We address your points in detail below:\n1. **Interpretation of Theorem 2:** trade-off means increasing the number of sampling steps does not necessarily lead to improved performance due to the influence of term (ii). This is in contrast to standard diffusion models, where, according to Theorem 2 of Chen et al. 2023b, the discretization error diminishes as the number of sampling steps increases, guaranteeing performance improvement with more steps.\\\nFor Case Study 2, we now show that a **uniform sampling** schedule $t_j = (N-j+1) \\Delta\\tau$ with $N = \\frac{t_1}{\\Delta\\tau}$ (**more** sampling steps) can yield a **larger** upper bound than the **halving schedule** defined in equation (13). Ignoring absolute constants, the upper bound in equation (12) becomes:\n  $$\n  R\\left(\\frac{R^2}{t_1^2} + \\frac{t_1}{\\Delta\\tau}\\frac{\\epsilon_{\\text{cm}}^2}{\\Delta\\tau^2}\\right)^{1/4} + \\epsilon_{\\text{cm}},\n  $$\n  which is minimized when $t_1 = \\frac{2^{1/3}R^{2/3}\\Delta\\tau}{\\epsilon^{2/3}}$, yielding a minimum value of:\n  $$\n  R^{7/6}\\frac{\\epsilon_{\\text{cm}}^{1/3}}{\\sqrt{\\Delta\\tau}} + \\epsilon_{\\text{cm}}.\n  $$\n  In contrast, the halving schedule achieves an upper bound of $\\tilde O\\left(R\\sqrt{\\frac{\\epsilon_{\\text{cm}}}{\\Delta\\tau}}\\right)$, which is **strictly smaller** than that of the uniform schedule.\\\n  Practical evidence further supports this trade-off:\n    - Our simulation results (Appendix G) show that both baseline methods experience degraded performance in the final sampling steps. \n    - In Table 1 and 2 of (Luo et al. 2023), LCM with 4-step sampling achieves better FID scores than with 8 steps.\n- Both theoretical analysis and empirical observations highlight the importance of **stratigically designing** the sampling schedule. Thus, we believe it is fair to emphasize the trade-off in choosing the number of sampling steps.\n\n2.\n  - **clarification on the result in Case Study 1:** Corollary 1 provides a **universial upper bound** on the $W_2$ distance for the VP process, without imposing constraints on $R$, $\\Delta\\tau$, and $\\epsilon_{\\text{cm}}$. Naturally, when the consistency model $\\hat f$ is poorly estimated, the generated sample quality deteriorates. We believe the case you mentioned belongs to this category. If $\\frac{\\epsilon_{\\text{cm}}}{\\Delta\\tau}\\approx R$, it indicates that $\\hat f$ is a poor approximation. Even when using the true marginal $P_T$ as input, Lemma 2 shows that $\\hat f$ incurs an error of $O(R)$. In this scenario, the resulting upper bound is inevitably loose and uninformative.\n  - **the significance of constant reduction in Case Study 1:** while many theoretical results focus on asymptotic rates and ignore constants, **constant reductions** can have significant practical implications. For example, latent diffusion models (Rombach et al.) demonstrate improved performance (e.g., lower FID scores) even with fractional gains, enabling high-resolution image synthesis in practice.\n  - **the rate improvement in Case Study 2:** our analysis in Case Study 2 shows a **clear rate improvement** for the VE process when sampling with multiple steps. If we use only a single step ($N=1$), equation (12) simplifies to:\n    $$\n    \\frac{R\\sqrt{R}}{\\sqrt{t_1}} + t_1 \\frac{\\epsilon_{\\text{cm}}}{\\Delta\\tau},\n    $$\n    with the minimum value being $R\\left( \\frac{\\epsilon_{\\text{cm}}}{\\Delta\\tau} \\right)^{1/3}$.\n    In contrast, using the specialized schedule from equation (13), we obtain a faster rate of $\\tilde O(R\\sqrt{\\frac{\\epsilon_{\\text{cm}}}{\\Delta\\tau}})$.\n3. **Clarification on $\\Delta\\tau$:** fix a $T$, decreasing $\\Delta\\tau$ results in a finer discretization $\\mathcal{T}$ (line 147), thereby increasing the number of discretization points $M$. This adds more terms to the error decomposition (line 419-423) when evaluated at $\\tau_M = T$.\n   - **Training steps vs sampling steps:** for simplicity, our current formulation assumes sampling steps are chosen from the training steps. Under this assumption, a smaller $\\Delta\\tau$ allows a smaller $t_N$. However, our theoretical framework remains valid even when sampling steps are chosen arbitrarily. In that case, we need to refine Lemma 2 to upper bound $||\\hat f(\\cdot,t)-f(\\cdot,t)||_2^2$ for all $t$. Because self-consistency is only enforced at the $\\tau_i$'s during training, $\\hat f(x,t)$ and $f(x,t)$ need to be Lipschitz in $t$.\n   - **Effect of $\\Delta\\tau$ on $\\epsilon$:** using a smaller $\\Delta\\tau$ may reduce the consistency error $\\epsilon$ for two reasons: (1). by continuity of $\\hat f$ and $\\varphi$, equation (4) decreases as $|\\tau_{i+1}-\\tau_{i}|$ decreases; (2). As shown in Theorem 1 and 2 of Song et al. 2023, smaller $\\Delta\\tau$ improves the approximation for the consistency loss in both consistency distillation and consistency training. Hence, $\\epsilon$ can be smaller.", "claims_and_evidence": "I think the presentation of this paper should be further improved. The main results are divided into two parts. In the first part (Section 3), the paper proves a error bound of the Wasserstein (TV) distance in the general form of time schedules. The meaning of the theorem, which contains very complex formulas, is not very clear to me. For explanation, the paper claims there exists a trade-off about the sampling steps. However, the result is still very hard to understand, because the parameter $\\alpha_t$, $\\sigma_t$ and $t_j$ has some intrinsic constraints. I find it too easy to draw the conclusion that there is a tradeoff. I’m also unclear about what this tradeoff definitely means in practice.\n\nWith these uncertainties in mind, I turned to the second part, which addresses the case studies. However, reading this section only deepened my confusion. Take the first case (VP) as an example: the paper presents a result concerning the Wasserstein distance between $\\hat P_0^1$ ($\\hat P_0^2$) and the data distribution. It argues that the leading term from the second sampling step is strictly reduced. While correct, this reduction merely adds at most another constant term—why, then, is this significant? Moreover, the paper considers the case $\\epsilon \\approx \\Delta \\tau$. In this case, the total right-hand side is $O(1)$. Yet, considering $R$ as a constant, any Wasserstein distance trivially has an upper bound of $O(1)$. Thus, I find myself questioning the significance and relevance of the stated result in this specific scenario.\n\nFinally, in Lines 226-231, the paper discusses the influence of $\\Delta \\tau$. I don’t quite get what is “more intermediate steps” even after reading Lemma 2. I also doubt the claim that “smaller $\\Delta \\tau$ allows a smaller $t_N$”, and “may decrease $\\epsilon$”. What’s the support of this argument? $\\tau_i$ for training is independent from $t_i$ for sampling, is that right?", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "I don’t find flaws in the theoretical proof. My concerns mainly lie in the meaning of the results, as written in the “Claims and evidence” part.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "See above", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hn4XjL2w0p", "reviewer": "Reviewer_VSKv", "summary": "This paper analyzes the convergence of consistency models under approximate self-consistency. With mild data assumptions, it proves sample closeness to the target distribution in Wasserstein or total variation distance. The study applies to various forward processes and highlights the benefits of multistep sampling through case studies.\n\n## Update after rebuttal\nI initially had no significant issues with the paper, but after reading Reviewer eZRV's comments, I believe the novelty of the work is somewhat limited, and my original score of 3 may have been overly generous. To ensure fairness in the review process, I have adjusted my score to 2.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We address your points in detail below:\n1. **non-uniform discretization:** in this paper, we adopt a uniform discretization for clarity and ease of presentation. However, our results can be extended to the non-uniform discretization setting as well. Suppose $\\tau_{0:M}$ is an arbitrary discretization of the interval $[0,T]$. In this scenario, it is reasonable to assume that the consistency loss scales with the length of the discretization interval: \n  $$\n  E_{x_{\\tau_i}\\sim P_{\\tau_i}}\\left[||\\hat f(x_{\\tau_i},\\tau_i)-\\hat f(\\varphi(\\tau_{i+1};x_{\\tau_i},\\tau_i),\\tau_{i+1})||\\_2^2\\right]\n  \\le (\\tau_{i+1}-\\tau_{i})^2\\epsilon^2.\n  $$\nUsing the same argument as in the proof of Lemma 2, we can derive:\n  $$\n  \\sqrt{E_{x_{\\tau_i}\\sim P_{\\tau_i}}\\left[||\\hat f(x_{\\tau_i},\\tau_i)- f^{\\star}(x_{\\tau_i},\\tau_{i})||\\_2^2\\right]}\n  \\le \\sum_{s=0}^{i-1}\\sqrt{\n    E_{x_{\\tau_s}\\sim P_{\\tau_s}}\\left[||\\hat f(x_{\\tau_s},\\tau_s)-\\hat f(\\varphi(\\tau_{s+1};x_{\\tau_s},\\tau_s),\\tau_{s+1})||\\_2^2\\right]\n  },\n  $$\n  which is upper bounded by:\n  $\n  \\sum_{s=0}^{i-1} (\\tau_{s+1}-\\tau_s)\\epsilon = \\tau_i \\epsilon\n  $. The rest of the proof remains unchanged.\n2. **regarding experiments on real-world data:** in designing the experiments, we aimed to achieve two goals:\n   - **evaluate the tightness of our upper bound:** this requires access to the true data distribution, which is not available for real-world datasets, making such evaluation infeasible in those settings;\n   - **demonstrate diminishing performance with multiple sampling steps:** we already observe this phenomenon in real-world datasets, as shown in Table 1 and 2 of Luo et al. (2023), where the performance of the consistency model deteriorates with more sampling steps.\n- Given these considerations, we prioritized simulation-based experiments over real-world datasets.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. I’ve checked the correctness of proofs for theorems 2 and 3.", "experimental_designs_or_analyses": "Yes. I checked the simulation in appendix G.", "relation_to_broader_scientific_literature": "The article presents the theoretical performance of consistency models with multi-step sampling under more general assumptions. The findings of the article can contribute to a deeper theoretical understanding of consistency models and can assist in the design of time steps for multi-step sampling in consistency models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper provides theoretical guarantees for multi-step generation in consistency models, focusing on Wasserstein distance and total variation distance. The results seem solid.\n\n2. Compared to previous work, the paper conducts its analysis under more general assumptions, including the removal of the Lipschitz assumption for the consistency function.\n\n3. The results are not limited to the form of SDEs and provide a more detailed analysis under two common types of SDEs.\n\nWeaknesses:\n\n1. The assumptions in the paper still differ from practical scenarios. For instance, the paper assumes that time discretization is uniform, which does not align with practical applications. In fact,  carefully designed  time discretization strageties are crucial for successful consistency training.\n\n2. The analysis and experiments in the paper lack connection with real-world data, which limits the further extension of the theoretical results.", "other_comments_or_suggestions": "NO.", "questions_for_authors": "Can the experiments in Appendix G be replicated on real-world datasets, such as CIFAR-10? This would significantly enhance the persuasiveness of the theoretical results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper establishes convergence guarantees of consistency diffusion models. Compared to existing works, this study extends to variance exploding forward process and the convergence guarantees include both the Wasserstein and TV distances.

The results are theoretical and reviewers appreciated the soundness and correctness of the results, despite concerns on interpretation and presentation. Understanding consistency model is also recognized as an important direction, given existing results deviates from practical implementations.

Major concerns are 1) whether the assumptions are realistic and restrictive, 2) interpretation of the upper bounds, and 3) numerical validations. For the first concern, authors' response provides discussions on the dimension dependence of the consistency error. Viewing the problem as a prediction problem not necessarily leads to a polynomial dependence on dimension---for example, a nonparametric regression inherently gives rise to an exponential dependence, known as the curse of dimensionality. While this is relatively a minor point, the assumption that the consistency error is relatively small, mimics the setup in Lyu et al. In Dou et al., only a Lipschitz assumption on the consistency function is required. Several discussions are favorable. For the second concern, the generality of the theory is appreciated, therefore, it is possible that the upper bound is large if the consistency training failed, or discretization step size is not appropriate. Therefore, it is suggested to revise the discussion around the theory as promised by the authors. Third, Appendix G provides a toy numerical example aiming to validate the theory. As suggested by several reviewers, since the theory implies relationship between consistency loss $\epsilon_{cm}$ and $\Delta \tau$, authors may consider devise new consistency model implementation and demonstrate their performance. This can largely strengthen the paper's contribution.
Decision: Accept (poster)