Morse: Fast Sampling for Accelerating Diffusion Models Universally

Authors: Chao Li, Jiawei Fan, Anbang Yao
Primary area: generative models
Keywords: Diffusion models, image generation, text-to-image generation, model acceleration

Abstract:
In this paper, we present Morse, a simple and universal framework for accelerating diffusion models. The key insight of Morse is to reformulate the iterative generation (from noise to data) process via taking advantage of fast jump sampling and adaptive residual feedback strategies. Specifically, Morse involves two models called Dash and Dot that interact with each other. The Dash model is just the pre-trained diffusion model of any type, but operates in a jump sampling regime, creating sufficient space for sampling efficiency improvement. The Dot model is significantly faster than the Dash model, which is learnt to generate residual feedback conditioned on the observations at the current jump sampling point on the trajectory of the Dash model, lifting the noise estimate to easily match the next-step estimate of the Dash model without jump sampling. By chaining the outputs of the Dash and Dot models run in a time-interleaved fashion, Morse exhibits the merit of flexibly attaining desired image generation performance while improving overall runtime efficiency. With our proposed weight sharing strategy between the Dash and Dot models, Morse is efficient for training and inference. We validate the efficacy of our method under a variety of experimental setups. Our method shows an average speedup of 1.78× to 3.31× over a wide range of sampling step budgets relative to baseline diffusion models. Furthermore, we show that our method can be also generalized to improve the Latent Consistency Model (LCM-SDXL, which is already accelerated with consistency distillation technique) tailored for few-step text-to-image synthesis. The code will be made publicly available.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730280742298

thread_type:
official_review

<thread_object>
creator: reviewer_fndt

<content>
summary:
The paper proposes Morse, a framework to accelerate Diffusion model. Morse leverages fast jumping sampling and adaptive residual feedback strategies to reformulate the iterative generation models. Two models called Dash model and Dot model are involved in Morse. The proposed method shows significant speed-ups compared with some baseline methods.

strengths:
The paper is clearly written. The experiments are extensive, and demonstrate the practical efficiency and effectiveness of Morse. Besides, some theoretical properties are also discussed. The proposed framework is novel and interesting.

weaknesses:
The main concern is that the Morse needs an extra model called Dot model, which needs extra and non-negligible computing resources.

questions:
1, On page 6, you mention that Morse gets speed-ups around 1.0x compared with DPM solver on both DDPM and SDE with 100 LSDs, and they are not considered in the calculation of the average speed-ups. However, these are cases in practical scenarios. I wonder what the average speed-up is if they are considered.

2, In practice, how should we set the batch size and training iterations of the Dot Model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730421812907

thread_type:
official_review

<thread_object>
creator: reviewer_yvzb

<content>
summary:
This paper introduces "Morse," a novel framework designed to enhance the efficiency of diffusion models, which are essential in generating high-quality images. The authors tackle the challenge of maintaining high sampling efficiency while minimizing information loss, a common issue with jump sampling techniques. By leveraging fast jump sampling and adaptive residual feedback, Morse offers a promising solution.



**Experimental Evaluation**

The authors conducted extensive experiments using datasets such as CIFAR-10 and LSUN-Church, and tested Morse with various samplers. The results are impressive, showing that Morse can accelerate diffusion models by an average of 1.78x to 3.31x compared to baseline models. This demonstrates Morse's capability to enhance runtime efficiency significantly.



In conclusion, Morse represents an advancement in the field of diffusion models, providing a balanced solution to the trade-off between speed and quality. Future research could explore optimizing the Dot model's resource demands and expanding Morse's applicability to a wider range of diffusion scenarios.

strengths:
**Key Contributions and Methods**

The standout feature of Morse is its dual-model approach: the Dash and Dot models. The Dash model operates within a jump sampling regime, while the Dot model, which is significantly faster, compensates for any information loss through residual feedback. This combination allows for flexible and efficient image generation without compromising performance.

The paper situates Morse within the broader context of diffusion models, referencing established samplers like DDPM, DDIM, DPM-Solver, and SDE. This contextualization not only highlights the relevance of Morse but also underscores its potential to improve upon existing methods.

weaknesses:
**Limitations and Future Directions**

While Morse offers substantial improvements, it does come with some limitations. The introduction of the Dot model requires additional training and computational resources. Furthermore, Morse's effectiveness is limited to scenarios where increasing the number of sampling steps enhances sample quality. The authors also touch on the ethical considerations, acknowledging the potential misuse of such generative models in creating deceptive content.

questions:
1. Could the authors provide a more detailed analysis of the trade-offs and limitations of Morse, as highlighted in the Discussion section? The paper mentions that Morse "introduces an extra lightweight Dot model, which requires additional training and computational memory." A deeper exploration of these drawbacks would help readers better understand the practical implications of using Morse. Specifically, a quantitative comparison of the additional training time and memory requirements introduced by the Dot model versus the runtime speedups achieved would be particularly beneficial for readers.


2. Are there any plans to release Morse as an open-source project? Open-sourcing it could enhance its impact by allowing the diffusion community to build upon its contributions, fostering collaboration, and broader adoption.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484110819

thread_type:
official_review

<thread_object>
creator: reviewer_gsn7

<content>
summary:
This paper presents a new methodology for sampling in diffusion models called Morse. It has two components, the first one is the "Jump Sampling" which uses different existing samplers to reduce the number of diffusion steps to converge to the generated image. The second component is the use of a smaller and much more efficient "dot" model to effectively interpolate between samples, providing residual feedback, leading to faster convergence. The dot model utilizes weight sharing with the main model (dubbed dash) to decrease overall training time. The paper demonstrates the compatibility of their system with multiple existing samplers including DDIM, DDPM, SDE and DPM, showing a speedup, especially in the low FID regime with few steps.

strengths:
This paper provides a novel methodology to improve the sampling speed of diffusion models and is compatible with many samplers. The paper provides many details and nuances to make this system practical, especially when it comes to training the smaller model and how it is used in conjunction with the larger model. The evaluation is fairly comprehensive, although I do have more comments on that in the weaknesses section. I thought it was great that the authors also showed their results on multiple model sizes.

weaknesses:
I view this paper's contributions as an efficiency technique for diffusion models. As such, I expected to see (1) and end-to-end implementation of the system showing measured speedup, and (2) some comparison/compatibility with other efficiency techniques. Both of these things are missing from the paper. I realize that the latter (2) may be hard to do comprehenesively, but at least a quantitative comparison to distilled/quantized diffusion models would make sense here. For the first weakness (1) the use of LDM is very confusing to me. Is this measured speed per step? Why not show the end-to-end time for generating an image? I imagine having two models has more room for system-level optimization which might have not been done in this work? The final weakness is that presentation of results which is shown on a log-log scale. I don't think an FID of 320 matters, so a bigger focus on the low FID regime would make more sense. It seems that speedup generally decreases as more time is given to the diffusion process. Is this method useful for high-quality image generation? or is only useful in lower-quality and faster latency regime? I think both are fine but it would be good to clarify the numbers at both extremes.

questions:
Please see weaknesses. Most of my questions are there.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717918331

thread_type:
official_review

<thread_object>
creator: reviewer_e8na

<content>
summary:
The paper presents a framework for the acceleration of any diffusion model. The kay idea is the use of a dual-model design, that consists of a heavy and a light model. The heavy model in this case is the original diffusion model, where the light one is a similar model but operating in a lower resolution. The approach has been seen in many other fields, but the claim here is that this is the fist time that is applied to diffusion models, and the main challenge was the design of the light model and the decision on the information that it had to consider. This is a heavy experimental paper, where the authors demonstrate in a number of settings the efficacy of their approach.

strengths:
The approach is quite simple and easy to apply. It is orthogonal to many other approaches in the area.
A good set of experiments under varying settings that demonstrate the efficacy of the method

weaknesses:
There is limited discussion on how the scheduling of Morse is decided. A5 provides some insight on the performance of various configurations, but no actual approach is given.

Fig 4 and 5 show the performance gains for a number of scenarios. However the reported overall gains take into account non meaningful FID scores (which is also the case where the proposed method provides the largest gains). I would suggest to focus the reported of the gain to a narrower FID band.

A simple (and faster rectifier) would be a combination of the original model but with a down sampled input followed by an up sampling block. Assign this comparison would add more information about the value that is actually added by the tuning of the new proposed model, given that the demonstrated gains are mainly due to the down sampling of the input. Table 4 and the relevant discussion addresses part of it but a)what about other samplers, and b) how statistical significant are the differences in FID scores?

questions:
There is a trade-off between the relative computational cost of the Dot compared to Dash and the similarity of the two models. A2 provides some insight of the construction of the Dot model, but gives little insight on the actual space cost-quality. Have other design approaches attempted?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732475148389

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The Summary of Our Responses to All Official Reviews

comment:
Dear Reviewers, Area Chairs and Program Chairs,

We sincerely thank all four reviewers for their thorough and constructive comments. We are excited that the novelty, the proposed method, the experiments and the performance of our work have been mostly recognized by all four reviewers.

In the past week, we carefully improved the experiments, the clarifications and the discussions of our work to address the concerns, the questions and the requests by all four reviewers. **Summarily, we made the following improvements**:

**(1)** Following the constructive suggestions from all four reviewers, we provide the results of several additional experiments to further study our method, including **a)** a set of experiments to demonstrate the importance of the proposed strategy to construct a Dot model for Morse, including adding the extra up-sampling and down-sampling blocks and tuning with LoRA; **b)** an ablation study to demonstrate the compatibility of our Morse with the quantized diffusion model; **c)** a set of experiments to show the importance of the trajectory information provided to the Dot model with DDPM and DPM-Solver samplers; **d)** a set of experiments to study variant designs about the architecture of the Dot model, including Dot model with different number of extra blocks and independent Dot model without weight sharing. **e)** some other results and clarifications for better ablation studies.

**(2)** Regarding the additional cost introduced by Morse (concerned by Reviewer yvZB and Reviewer fndt), we provide related details and discussions about it.

**(3)** Regarding the acceleration ability of Morse with the high-quality image generation (comments by Reviewer e8na, Reviewer gSN7 and Reviewer fndt), we provide related discussions.

**(4)** We also provide detailed responses to the other concerns/questions/requests raised by each reviewer one by one.

**Finally**, with the constructive comments by all four reviewers and our responses, we carefully updated the manuscript of our work. We hope our detailed responses, the updated manuscript and the example code of Morse for Stable Diffusion are helpful to address the concerns, the questions and the requests of all reviewers.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**
This paper proposes what the authors call Morse which allows acceleration of the generation process of a pretrained diffusion model (DM). The proposal consists of a pretrained DM and an additional lightweight model which in a sense distills the pretrained DM model, and use these two model in an interleaved manner in the generation stage.

**Strengths:**
The results of extensive experiments demonstrate usefulness of the proposal across various existing DMs in some scenarios. 

**Weaknesses:**
- As Reviewer e8na mentioned, how one should decide the scheduling given a computational budget is not well discussed.
- The efficiency of the proposal was questioned by some reviewers.
- Although the basic architecture of the Dot model is shown in Figure 3, it has the extra input variables $x_{t_s},z_{t_s},t_s$ compared with the base Dash model, whose input variables are $x_{t_i},t_i$, and how it deals with these extra variables is not described.

**Reasons:**
Although all the reviewers evaluated the proposal as novel and interesting, some concerns on the efficiency remain. Also, this paper would benefit from extensive revision reflecting the review comments. The review scores were scattered around the acceptance threshold, with 2 just above and 2 just below, which would suggest that this paper is not strong enough to merit acceptance.

Minor points:
- Page 2, line 92; page 4, lines 177 and 189: What $N$ means is not explicitly stated at all.
- Page 2, equation (1): One needs integration with respect to $x_1\ldots,x_{t-1}$ for the equality to hold.
- Page 2, line 124: equivalent (with → to)
- Page 2, line 134: Again, one needs integration with respect to $x_1\ldots,x_{T-1}$ for the equality to hold.
- Page 2, lines 136-139: What is the main clause of this sentence ("While $T$ is large, which is essential, [what?]").
- Page 2, line 154: performance (degeneration → degradation?); The claim here "performance degradation issue becomes more serious" seems to be in conflict with the claim in line 152 "without doing much harm to the sample quality".
- Page 5, line 268: similar (with → to)
- Relation to distillation: As opposed to the authors' discussion at the end of Section 2, I feel that the proposal can still be regarded as one form of distillation. Indeed, distillation in the context of deep learning in general refers to the idea of transferring "knowledge" that has already been acquired in a large model to a smaller one. In the proposal, although the smaller model is not supposed to mimick the large one, it certainly distills some knowledge in the large model.
- I would like to see an argument that may justify the design choice of the proposal, especially regarding the weight sharing between the Dash and the Dot models, where the shared weights are to be used in different resolutions because of the added upsampling/downsampling operations in the latter, which may cause problems (although the LoRA would be expected to alleviate them to some extent).