DualFast: Dual-Speedup Framework for Fast Sampling of Diffusion Models

Authors: Hu Yu, Hao Luo, Fan Wang, Feng Zhao
Primary area: generative models
Keywords: Generative models, diffusion models, sampling acceleration

Abstract:
Diffusion probabilistic models (DPMs) have achieved impressive success in visual generation. While, they suffer from slow inference speed due to iterative sampling. 
   Employing fewer sampling steps is an intuitive solution, but this will also introduces discretization error. Existing fast samplers make inspiring efforts to reduce discretization error through the adoption of high-order solvers, potentially reaching a plateau in terms of optimization. This raises the question: can the sampling process be expedited further?
   In this paper, we re-examine the nature of sampling errors, discerning that they comprise two distinct elements: the widely recognized discretization error and the less acknowledged approximation error.
   Our research elucidates the dynamics between these errors and the step by implementing a dual-error disentanglement strategy.
   Building on these foundations, we introduce an unified and training-free acceleration framework, DualFast, designed to enhance the speed of DPM sampling by concurrently accounting for both error types, thereby minimizing the total sampling error.
   DualFast is seamlessly compatible with existing samplers and significantly boost their sampling quality and speed, particularly in extremely few sampling steps. 
   We substantiate the effectiveness of our framework through comprehensive experiments, spanning both unconditional and conditional sampling domains, across both pixel-space and latent-space DPMs.

Decision: reject
Reviewer scores: 6, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729518213238

thread_type:
official_review

<thread_object>
creator: reviewer_u2p2

<content>
summary:
This paper introduces a novel approach aimed at accelerating the sampling process in diffusion models. The proposed method seeks to minimize approximation errors by leveraging noise predictions from previous steps. When applied to DPM and DPM++ solvers, this approach effectively enhances the FID and human preference scores.

strengths:
1. This paper primarily concentrates on reducing the approximation error, a factor overlooked by previous studies. 
2. The experiments conducted in this paper are thorough. 
3. This paper is well composed.

weaknesses:
1. The paper do not illustrate the computation of the MSE. The analysis of the Dualfast model is based on the probability flow ODE, yet, as per Figure 2, the "exact form" is derived from the forward SDE. The "exact form" should be the analytical solution of the probability ODE, not the outcomes generated by the forward process of the diffusion model.

2. Line 263-269. The explanation of diffusion model is incorrect. According to [1], the expression for the optimal \epsilon_\theta can be computed explicitly. Given the training data $\{ y_1, …, y_N\}$, $\epsilon(x_t, t) = \sum_i \frac{\mathcal{N}(x_t; y_i, \sigma_t I)}{\sum_j \mathcal{N}(x_t; y_j, \sigma_t I)} \frac{x_t - \alpha_t y_i}{\sigma_t}$. This demonstrates that the optimal $\epsilon(x_t, t)$ is the weighted  average of the vectors pointing towards all the data points. The claim $\epsilon(x_\tau, \tau)$ is a better approximation of $\epsilon(x_t, t)$ can be contradicted with a simple 2D example with the situation where the training data only has two samples $\{(0,1), (0,-1)\}$.

3. Line 279. $x_\theta$ and $\epsilon_\theta$ are equivalent by equation (17), yet only the explicit $\epsilon_\theta(x_t, t)$ is replaced with $\epsilon(x_\tau, \tau)$. If $\epsilon(x_\tau, \tau)$ is indeed more accurate, the implicit part in $x_\theta$ should also be replaced. 

4. Line 286. According to figure 8, the coefficient $c$ is non-negative.  However, the coefficient of $\epsilon_\theta(x_\tau, \tau)$ in equation (12) is non-positive, implying that $x_t$ is pushed further away from $\epsilon_\theta(x_\tau, \tau)$.  This contradicts the claim that $\epsilon_\theta(x_\tau, \tau)$ is closer to the ground truth.

5. The paper attributes the observed phenomena to a reduction in approximation error without sufficient evidence. It would be more beneficial to explore other potential causes, such as the possibility that the term $-c \epsilon_\theta(x_\tau, \tau)$  pushes $x_t$ away from 0, leading to increased brightness and saturation.


[1] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in neural information processing systems 35 (2022): 26565-26577.

questions:
1. Could you please elaborate on the methodology used to calculate the MSE?
2. As per [1], the optimal $\epsilon_\theta(x_t, t)$ can be computed for the 2D example with training data $\{(0,1), (0,-1)\}$. Could you compute and numerically compare $\epsilon_\theta(x_t, t)$ and $\epsilon(x_\tau, \tau)$?
3. Would you be able to substitute the implicit $\epsilon_\theta$ in $x_\theta$ and conduct experiments to demonstrate its improvements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730133024309

thread_type:
official_review

<thread_object>
creator: reviewer_hemq

<content>
summary:
The authors consider the problem of accelerating DPM sampling using training-free methods. They highlight the fact that prior works in this area focus only on discretization error, neglecting the demonstrated substantial impact of approximation error in the total error. The main motivation, demonstrated in Figure 2, is that the approximation error of DPM networks is worst at small noise levels (i.e., close to x_0 along the probability path). This motivates the use of higher-noise-level outputs in the update steps of ODE solvers for DPM generation, where the goal is to reduce approximation error in addition to discretization error (where the latter is provided by existing efficient DPM solvers).

strengths:
The method addresses a highly important issue (approximation error) in DPM sampling which they note is neglecting in prior works. Moreover, they demonstrate that this issue may have a greater impact than discretization error. Therefore, this is a highly relevant problem.

The training-free approach to mitigate approximation error is a good contribution, especially given the fact that this approach can be used in a plug-and-play manner with many DPM sampling techniques addressing discretization error. 

Finally, the authors provide comprehensive experiments in a wide variety of settings and sampling schemes. This demonstrates the robustness of their method.

weaknesses:
Some aspects of the background and method are not very clear. For instance, in equations 8 and 10, are both x_\theta and \epsilon_\theta outputs of the neural network? It is not immediately clear to me how this improves the approximation error of the neural network. I think the main contribution should be made more clear, or at least, it should be more clear in the main text how the modified sampling scheme leads to decreased approximation error.

I think the order of Sections 2.1 and 2.2 is a bit confusing. Equation 6 seems to be the “base” equation from which all solvers can be specified - maybe it should be defined earlier. To be more clear, maybe the “ODE-based sampling process” can be included in Section 2.2, which focuses on sampling.

Formatting Issues:

*The citation style makes it difficult to read the paper, especially the introduction. In-text citations should be surrounded by parentheses.

*Line 12: “While, they suffer from…” -> “However, they suffer from…” (while does not make sense in this context)

*Line 14: “introduces” -> “introduce”

*Line 24: “boost” -> “boosts”

*Lines 116-119: it could be necessary to comment on how \sigma_t and \alpha_t are defined

*Line 166: “while ignore” -> “while ignoring”

*Line 271: “general from applicable” I’m not sure what this means

*Line 280: “of equation 11” -> “of equation 9”

*Line 284: “between equation 9 and equation 9” -> “between equation 9 and equation 11”

*Line 289-291: The two uses of “Besides” in this paragraph are confusing. I think they can be removed in both sentences.

*Line 299/300: Missing period after “Ours-DPM-Solver”

*Line 311/312: Missing period after “Ours-DPM-Solver++”

*Line 311/312: “DPM-Solver++ is the sota and default samplers in stable diffusion model” -> “DPM-Solver++ is the SOTA and the default sampler for the stable diffusion model.” Also need a citation here.

*Line 430/431: “bwtween” -> “between”

*Line 514/515: “Besides” doesn’t make sense here

questions:
Figure 2: It is mentioned that the time interval is divided into 9 time periods of size 111 (suggesting NFE=9), but also that NFE=111 and NFE=1. Also, if NFE=1 for operation 3, how do you obtain multiple data points for MSE in the right-side figure?

Figure 2 (right): Is T=1000 the initial time (x_t’ = x_s) and time T=0 the final time (x_t’ = x_t)? Maybe the x-axis should be inverted to demonstrate that error increases as the sampling procedure is run for a larger number of steps. Maybe the x-axis label should be something other than t, since it is suggested to be a fixed time point in the left side figure.

It is not clear to me how noise prediction/data predictions models can be derived from each other (the end of section 3). Is it possible to convert a noise prediction DPM to a data prediction DPM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730377081280

thread_type:
official_review

<thread_object>
creator: reviewer_p7sa

<content>
summary:
The paper proposes a modification to the sampling procedure in diffusion models to further reduce discretization error beyond that achieved with high-order solvers. The authors re-examine the sampling error at each time step and observe that the discretization error decreases over time. Building on this observation, they propose leveraging the denoiser at larger time steps to substitute for those at current steps, leading to reported improvements in performance.

strengths:
The paper’s idea is interesting, and the presentation is clear and easy to follow. Addressing discretization error is a important topic, and this work proposes a novel approach in that direction.

weaknesses:
Several aspects need further clarification: 
-  **Main Observation on Discretization Error:** The central observation, illustrated in Figure 2, is that discretization error decreases over time. The y-axis is labeled with MSE,  could the authors clarify what quantity the MSE is measuring and how it directly relates to the discretization error? 

- **Interpretation of Figure 2:**  The paper infers from Figure 2 that "the noise pattern is more recognizable as t increases,". But I feel there are other possible interpretations. For example, wouldn't it be possible that as sampling progresses from time T to 1, the discretization error accumulates, leading to an increase in error as we go backward in time?  I suggest the authors conduct an ablation study or provide theoretical justification to rule out this possibility.

- **Algorithm Procedure:** In equation (8), both terms on the right-hand side involve ϵθ ​ (with xθ also defined through ϵθ​). It’s unclear why only the ϵθ​ in the second term is evaluated at τ instead of t. Would it be beneficial if both terms were modified similarly? It would be valuable to discuss whether the authors explored modifying both terms and what the results were if so.

- **Similarity to Existing Methods:** In my opinion, the formulation of D_{t−1}^ours​ bears a strong similarity to equation (3) of [1].  Could the authors elaborate on the differences between the current method and that in [1] and provide a numerical comparison? 

**Minor Issues:**  
- The paper could benefit from enhanced self-containment. For instance, xθ is used before it is defined, and λt​ is not explicitly defined anywhere in the paper.
- The numerical results show a very mild improvement over baseline results. In Figures 4 and 5, the proposed method, on average, reduces NFE by no more than one for the same error level.
- Minor typos: Line 271, "general from" should be "general form"; Line 284, "between equation (9) and equation (9)" needs correction.


[1] Karras, Tero, et al. "Guiding a Diffusion Model with a Bad Version of Itself." arXiv preprint arXiv:2406.02507 (2024).

questions:
Please see the weakness.

I would be open to raising the score if these concerns are adequately addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690937027

thread_type:
official_review

<thread_object>
creator: reviewer_skpz

<content>
summary:
The paper studied the sampling process for diffusion models. The paper categorized sampling errors into discretization error and approximation error, and pointed out that existing (high-order) solvers aim to reduce the discretization error. The paper on the other hand focused on the approximation error, and proposed a speedup framework to reduce the error. The method introduced a new noise estimation term in the sampling process, with a tune-able coefficient to reduce the sampling error. The method can be combined with existing samplers, and the effectiveness is demonstrated in various experiment settings.

strengths:
1. The paper proposed an effective technique to improve samplers for diffusion models without introducing extra NFEs, especially in small NFE cases. The method can be readily combined with a wide range of existing samplers without post-training the diffusion models.

2. Extensive experiments were conducted to show the performance of the proposed method. Experiment settings vary across different condition types, sampler orders, guidance strategies, sampling space and datasets.

weaknesses:
1. While the paper performed motivational experiments to explain discretization error and approximation error, the two types of error are not discriminated in a formal formulation. It is also not clear why the new noise estimation term in eq 10-12 can help reduce the approximation error. Here \tau is a variable time step and is set to T in experiments. The connection between the solver and the choice of \tau is not specified.

2. The writing of the paper may need improving. The notations in formal formulation of the sampling process can be more consistent. The citation format makes parts of the paper hard to read.

questions:
1. Is there a principled way to set the values of c and \tau? Are they dependent on the order of solvers or specific solvers?

2. What is the performance of the proposed method for large NFEs? Is it able to improve the generation quality as NFE increases?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a unified acceleration framework for Diffusion Probabilistic Models that addresses both discretization and approximation errors, enabling faster and more accurate sampling. Key contributions include a dual-error disentanglement strategy and compatibility with existing solvers. 

The reviewers noted several positive aspects of the paper, but raised several doubts about the mathematical definitions and the method and accuracy of the calculations. These issues were not resolved through discussions between the authors and reviewers. Improvements will still be needed for this paper to be seen as sound.