Convergence Of Consistency Model With Multistep Sampling Under General Data Assumptions

Authors: Yiding Chen, Yiyi Zhang, Owen Oertell, Wen Sun
Primary area: learning theory
Keywords: Consistency models, diffusion models, learning theory

Abstract:
Diffusion models accomplish remarkable success in data generation tasks across various domains. However, the iterative sampling process is computationally expensive. Consistency models are proposed to learn consistency functions to map from noise to data directly, which allows one-step fast data generation and multistep sampling to improve sample quality. In this paper, we study the convergence of consistency models when the self-consistency property holds approximately under the training distribution. Our analysis requires only mild data assumption and applies to a family of forward processes. When the target data distribution has bounded support or has tails that decay sufficiently fast, we show that the samples generated by the consistency model are close to the target distribution in Wasserstein distance; when the target distribution satisfies some smoothness assumption, we show that with an additional perturbation step for smoothing, the generated samples are close to the target distribution in total variation distance. We provide two case studies with commonly chosen forward processes to demonstrate the benefit of multistep sampling.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730648606808

thread_type:
official_review

<thread_object>
creator: reviewer_rxyv

<content>
summary:
The paper studies the convergence of consistency models theoretically. They provide bounds on the Wasserstein distance between target and estimated data distribution using such models for as most general assumptions as they can (specifically, for bounded density support and light tail density). They transfer this to total variation distance under smoothness assumption, giving convergence guarantee. Moreover, they also derive theoretically the benefit of an additional sampling step in the specific case of Ornstein-Uhlenbeck process. They provide advance in the theoretical understanding of empirical observations about consistency models which are (i) an additional step can significantly improve the quality of the sample and (ii) more steps only bring limited gains.

strengths:
The will of bringing theoretical claim about convergence of a method + analytical explanation of empirical observations is highly welcomed. It allows better understanding of flaws and successes of a method. Especially in the field of diffusion models, and specifically here consistency models which are SOTA methods for generative modeling. Consistency models aim to improve over classical diffusion, tackling the sampling efficiency. Bringing theory and explainability in convergence of such method is valuable.

- The paper base their claims on previous observations in the literature and try to improve in several ways. 

- It is welcomed to see work that tries to provide theoretical claim with light assumptions, making them as most general as possible.

- Interpretability (in the case of multi-step sampling) is thrilling.

weaknesses:
As a general comment, I found the goal laudable. However, the paper is really hard to follow and lacks clarity in its presentation and writing. I will try to pack similar remarks in thematic groups here below.

**General confusion**

The introduced notations are overly complex and sometimes confused. 
- Line 50-71 : The inline paragraphs bold titles are weird. Why not using clear separated paragraphs? (I know that you may lack space, I propose getting rid of useless repetition below).
 
- Score-based generative model notations are hard to follow.

    - Is the switch between $P_t$ and $p_t$ needed?

    - Operator $\mathcal{D}$ is not useful and can be simplified by just using the noise schedule kernel and the distribution defined line 143.

    - Why $x_{t_i}$ and not just $x_i$? e.g. : line 165 where we have $x_{t_0} = x$, usually denoted $x_0$ in the literature. But on the other hand, you define a $\Delta \tau$ accounting for step between consecutive time, ... all of that can be simplified. I understood the paper but found it hard due to intertwined overly complex notations (at least understandable but making it hard to follow without making tons of back and forth in the text to check each piece). The time subscript is sometimes omitted, confusing also the reader concerning what we are talking about.

- The paper lacks structure and the reader is too often referred to other part of the paper. The authors introduce a lot of preliminaries 'useful for later', 'used in theorem x'... leading to a lot of repetition in the main text (in the abstract, the contributions, the theorem in themselves,...). It would be better to have more structured blocks. E.g. : Contributions can be shortened, more summarized, leaving the reader to the proofs in the main text.

- The consistency function introduced line 167-168 is used on distribution line 211. Of course the author precise **after** that the operation is now on distributions, but this further confuse the notations and highlight again a general confusion.

- It would be nice to introduce Wasserstein distance and total variation somewhere. Even in Appendix, it's ok. Indeed, In theorem 2 you use $W_2$ and then make a sort of connection with the KL line 187 that pops from nowhere. It lacks proper explanation.

- Lines 246-255 repeats a lot with preceding text.

**Results**

Analytical results are interesting. However they stick to theory and it's hard to identify their contribution in practice. Could you add, even for small toy problem, experiments that illustrate your findings. Indeed, you derive bounds given several assumptions. Use those for a toy example. It is crucial to link theory to practice to motivate the worth of your work. Experiments might also help identify potential yet to study problems/questions. Your results are sometimes specific (e.g. you stick to VP and VE SDE in case study 2) which is normal for theoretical derivations, please extend the scope of your work a bit. Even if you show that your conclusions do not hold at all in other setup, it might be valuable as later study can try to understand why? Or in the opposite, if your claims extend in other setup, it might raise the question of generalization of your work.

**Found typos**

I list below typos I detected, just to correct them (I might have missed some).

- Uppercase in contributions (lines 76-92).

- Space after a point line 169 **. At a high level ...**.

- Line 262: ~insteading~ -> instead of.

- Where is Theorem 1? At the beginning of page 6 you directly have Theorem 2 but there are no Theorem 1. Keep separated counting between Assumptions, Lemmas and Theorems.

- Line 367: ~forawrd~ -> forward.

- Line 368: ~nosie~ -> noise.

- Line 377: Without loss of generality preferred to not clear *W.l.o.g.*.

- Line 450: ~samling~ -> sampling.

**Suggestions**

- Remove the references to theorems in your contributions. Focus only to the key message of each of them and the rest will follow in the main text.

- line 165: remove the 'as a function of t'.

- Remove parenthesis around 'ground truth' line 166.

- Putting ':' inside a sentence often make it long and hard to follow (can be more structured). E.g. : lines 170-172

- Combine Theorem 2 and 3. Can't we see theorem 2 as a specific case of 3 when $C \to 0$?

For me, the paper is not ready yet. The main causes being the presentation and lack of illustrations.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665701786

thread_type:
official_review

<thread_object>
creator: reviewer_eess

<content>
summary:
This paper studies the convergence of consistency models when the training process is approximately self consistent under the training distribution. This analysis involves mild data assumptions and applies to multi-step sampling for a family of forward processes. Sample quality guarantees are provided in terms of Wasserstein distance and total variation distance for target distributions that have particular properties. Finally, two multi-step sampling case studies are presented to demonstrate the implications of the results in this paper for two common forward processes.

strengths:
- The paper provides a good overview of background and related work on score-based diffusion models and consistency models.
- The contributions in this paper regarding sample quality guarantees in Wasserstein distance and total variation distance appear to be novel and technically sound.
- The high-level ideas for the proofs of the main results in this paper are presented reasonably well in Section 4.

weaknesses:
- While some of the implications and benefits of the results in this paper are presented as case studies from a theoretical standpoint in Section 3.3, there are no empirical/experimental results to accompany this discussion. The lack of such empirical results detracts somewhat from this paper, and adding these results would help to further validate the benefits of the authors’ results.
- To connect the results in this paper to real-world scenarios, it would be help for the authors to connect the data distribution properties used in Theorems 2 (bounded support) and 3 (light tail) and the data distribution smoothness assumption in Theorem 4 with some examples of real-world datasets commonly used to train consistency models that satisfy these properties/assumptions.

questions:
It would be helpful for the authors to respond to the weaknesses pointed out above. If the authors don’t currently have experimental results that they can share, I suggest that they add such results to a future draft of this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730883380355

thread_type:
official_review

<thread_object>
creator: reviewer_leny

<content>
summary:
This paper analyzes the distribution generated from the consistency model. It studies the upper bound of the total variation distance and Wasserstein distance between the generated distribution to true data distribution when the consistency model is approximately self-consistent. It then provides two examples that illustrate the main results.

strengths:
The paper provides the upper bounds for the consistency model and discusses two examples to give the exact bounds. It studies the analytical aspect of the diffusion models and uses the results to provide insights into optimal choices for sampling. It contributes to the development of diffusion models.

weaknesses:
The major concern is the tightness of the upper bounds discussed in the paper. It seems to me that given $R$ is never zero, the right-hand side of Equation (7) will be bounded away from zero. This might be because either $\hat{P}^{(N)}_0$ will not be asymptotically close to the true date distributions or the bound can be further tightened. Or is there a non-trivial condition that the right-hand side of Equation (7) would be small?

questions:
a. It would be nice to also discuss the bounds when self-consistency holds. 

b. The authors have provided two examples of case studies with specific upper bounds. It would be great to run the simulation on these two examples and see if the upper bound of calculated distance $W_2(\hat P_{t_{N}}, P_{\text{data}})$ matches the theories.

c. Is there a general rule of choosing optimal $N$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730940278638

thread_type:
official_review

<thread_object>
creator: reviewer_ruor

<content>
summary:
This paper gives the convergence of the consistency model multistep sampling procedure. The authors establish guarantees on the distance between the sample distribution and data distribution in terms of both Wasserstein distance and total variation distribution. The established upper bound requires only mild assumptions on the data distribution.

strengths:
1. The theoretical results seem solid
2. This manuscript is well-written and easy to follow

weaknesses:
1. The primary contribution stems from the utilization of a modest assumption. Nevertheless, this mild assumption still appears to have limitations in practical applications.
2. There is no empirical evaluation to verify the theoretical findings.

questions:
1. It would be beneficial if the authors could illustrate the main contribution through a table.
2. Assumption 1 also relies on the distribution $P_{\tau_i}$ . It still seems to have limitations in practical applications. What advantages does it offer over other theoretical assumptions?
3.  It would be better if the authors could offer empirical evaluations to validate these theoretical findings.
4. The primary challenge lies in how to translate the approximate self-consistency measured under the training distribution into the quality of the generated samples. What technical breakthrough has been achieved to address this challenge?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733083237538

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response regarding experiments

comment:
We thank all reviewers for your efforts in reviewing our paper. Several reviewers noted the lack of experiments in our original submission,  and we would like to address these concerns as follows:
1. **Moving simulation to the main text**: We added simulation results with a toy example in Section G in a recent revision. We hope this helps justify our theoretical results. However, we believe moving this section to the main text is unnecessary because it is purely for illustration purposes. Our primary contribution lies in providing a theoretical understanding of consistency models, not empirical validation. Including this simulation in the main text might shift focus away from our theoretical contributions.
2. **Absence of empirical result in a theory paper**: experiments are not always essential in theory-focused papers. Despite the tremendous amount of practical applications of score-based generative models, it is uncommon for a theoretical paper in this field to include experiments (See [1-6] below). The practical efficacy of diffusion and consistency models is already well-established, so we chose to focus on advancing the theoretical framework.
3. **Theoretical contributions**: we highlight our theoretical contribution here: (a) we relax the data assumptions in previous work by removing the Lipschitz condition on the consistency function; (b) we study a more general set of forward processes, which captures the widely-used variance exploding SDE; (c) we develop novel techniques to analyze multi-step sampling. All of these contribute to a more interpretable theoretical result. We respectfully ask the reviewers to evaluate our paper from this perspective. 




[1] Chen et al. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions.

[2] Chen et al. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

[3] Dou et al. Theory of consistency diffusion models: Distribution estimation meets fast sampling

[4] Lee et al. Convergence of score-based generative modeling for general data distributions

[5] Li et al. Towards a mathematical theory for consistency training in diffusion models

[6] Lyu et al. Convergence guarantee for consistency models
</content>
</thread_object>

</thread>


Metareview:
metareview:
A consistency model is proposed to converge to the true distribution of a diffusion model, so that data can be generated more efficiently. Although the approximation error is shown to be bounded, reviewers raised concern about the tightness of the bound. There are also criticism about lack of numerical experiments to support the theoretical claims.