Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Authors: Chao Li
Jiawei Fan
Anbang Yao
Venue: ICML 2025 poster (2025)
Keywords: Diffusion models
image generation
text-to-image generation
model acceleration
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: In this paper, we present a new dual-sampling framework, called Morse, to accelerate diffusion models losslessly.

Abstract: In this paper, we present $Morse$, a simple dual-sampling framework for accelerating diffusion models losslessly. The key insight of Morse is to reformulate the iterative generation (from noise to data) process via taking advantage of fast jump sampling and adaptive residual feedback strategies. Specifically, Morse involves two models called $Dash$ and $Dot$ that interact with each other. The Dash model is just the pre-trained diffusion model of any type, but operates in a jump sampling regime, creating sufficient space for sampling efficiency improvement. The Dot model is significantly faster than the Dash model, which is learnt to generate residual feedback conditioned on the observations at the current jump sampling point on the trajectory of the Dash model, lifting the noise estimate to easily match the next-step estimate of the Dash model without jump sampling. By chaining the outputs of the Dash and Dot models run in a time-interleaved fashion, Morse exhibits the merit of flexibly attaining desired image generation performance while improving overall runtime efficiency. With our proposed weight sharing strategy between the Dash and Dot models, Morse is efficient for training and inference. Our method shows a lossless speedup of 1.78$\times$ to 3.31$\times$ on average over a wide range of sampling step budgets relative to 9 baseline diffusion models on 6 image generation tasks. Furthermore, we show that our method can be also generalized to improve the Latent Consistency Model (LCM-SDXL, which is already accelerated with consistency distillation technique) tailored for few-step text-to-image synthesis. The code and models are available at https://github.com/deep-optimization/Morse.

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Chao Li Jiawei Fan'
Anbang Yao
Abstract
1. Introduction
Diffusion models (DMs), a class of likelihood-based gen- erative models, have achieved remarkable performance on variety of generative modeling tasks such as image gen- eration (Ho et al., 2022), text-to-image generation (Zhang et al., 2023), video creation (Blattmann et al., 2023), text-to- 3D synthesis (Poole et al., 2023) and audio synthesis (Liu et al., 2022). The powerful generalization ability of DMs comes from a forward-backwart diffusion framework: the forward process gradually degenerates the data into random noise with T'step noise schedule (typically, T 1000 as default), while the backward process learns neural net- work to iteratively estimate and remove the noise added to the data. However, to generate high quality samples, DMs usually require hundreds of sampling steps (i.e., function evaluations of the trained model). The slow sampling effi- ciency incurs heavy computational overhead at inference, especially large-scale DMs such as DALL-E (Ramesh et al., 2022), Imagen (Saharia et al., 2022) and Stable Dif- fusion (Rombach et al., 2022; Podell et al., 2024), posing great challenge for the deployment of DMs.
In this paper, we present Morse, a simple dual- sampling framework for accelerating diffusion models losslessly. The key insight of Morse is to reformulate the iterative generation (from noise to data) process via taking advantage of fast jump sampling and adaptive residual feedback strate- gies. Specifically, Morse involves two models called Dash and Dot that interact with each other. The Dash model is just the pre-trained diffusion model of any type, but operates in a jump sam- pling regime, creating sufficient space for sam- pling efficiency improvement. The Dot model is significantly faster than the Dash model, which is learnt generate residual feedback conditioned the observations at the current jump sampling point on the trajectory of the Dash model, lifting the noise estimate to easily match the next-step es- timate of the Dash model without jump sampling. By chaining the outputs of the Dash and Dot mod- els run in a imme-nterleeveed fashion, Morse ex- hibits the merit of flexibly attaining desired image generation performance while improving overall runtime efficiency. With our proposed weight sharing strategy between the Dash and Dot mod- els, Morse efficient for training and inference. Our method shows lossless speedup of 1.78x to 3.31 on average over a wide range of sam- pling step budgets relative 9 baseline diffusion models 6 image generation tasks. Furthermore, we show that our method can be also general- ized to improve the Latent Consistency Model (LCM-SDXL, which is already accelerated with consistency distillation technique) tailored for few-step text-to-image synthesis. The code and models are available at https://gituub.com/dee optimization/Moorse
Recently, there have been lots of research efforts aiming to design fast samplers for DMs, which can be grouped into two major categories. The first category focuses on evolving more advanced formulations for the sampling pro- cess that enjoy faster convergence. Denoising diffusion implicit models (DDIM) (Mohamed & Lakshminarayana 2016; Song et al., 2021a), stochastic differential equations (SDE) (Song et al., 2021b) and ordinary differential equa- tions (ODE) based solvers (Zhang & Chen, 2023; Lu et al., 2022) are representative ones. worth noting that the ODE samplers allow generate high quality samples tens of sampling steps. The second category relies knowledge distillation schemes, such as progressive distillation (Sali- mans & Ho, 2022), two-stage progressive distillation (Meng et al., 2023) and consistency distillation (Song et al., 2023; Luo et al., 2023), by which the few-step samples generated by a student DM using the distilled sampler can match to the many-step outputs of its corresponding teacher DM.
'Intel Labs China, Correspondence to: Anbang Yao <an- vaog. intel.com>>.
In this work, we attempt to improve the sampling efficiency of DMs in more generalized perspective. Specifically, we ask: given a pre-trained DM (with either U-Net or self- attention based backbone), no matter what kind of existing
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
10LSDs
LSDS
50 LSDs
10LSDs
LSDS
50 LSDs
Stable Diffusion
10LSDs
LSDS
50
10LSDs
LSDS
50
+Moss
A Liverwith and brick bridge overit.
A cargo truck driving down a highway with red text
LLSD
2 LSDs
LSDS
LSD
2 LSDs
3 LSDs
LCM-SDXL
L.33 LSDs
2 LSDs
3SSDs
1.33 LSDs
2LSDs
3LSDs
Morse
Closeup of a brown bearsitting in grassy area.
Birds perch on bunch of twigs in the winter.
Figure 1. Generated samples from Stable Diffusion (Rombach et al., 2022) and Stable Diffusion XL fine-tuned with Latent Consistency Models (LCM-SDXLL (Luo al., 2023) with and without Morse text-to-image generation. For simplicity. we use the Latency per Sampling step of the baseline DM (LSD) as the time unit to calculate the total latency of diffusion process.
samplers is used, is it possible reformulate the iterative denoising generation (from noise to data) process towards tradeoff's under wide range of sampling step budgets (including hundreds-step, tens- step and few-step sampling)? To address this problem, our method inspired by common property of prevailing DMs. We notice that they typically support jump sampling (JS) in function evaluation, especially when using the fast samplers discussed above. This observation inspires us to explore the use of IS for formulating our method. Not surprisingly, with JS, prevailing DMs can generate samples in a faster speed, yet inevitably leads to worse sample quality due to the information loss over unvisited steps between every two adjacent JS points on the diffusion trajectory. The performance degradation issue becomes more serious as the IS step length increases. Therefore, the double-edged nature of IS prohibits its use for performance lossless acceleration.
sion model that needs to be accelerated, but operates in JS regime, creating sufficient space for sampling efficiency improvement; (2) The Dot model is significantly faster (e.g.. multiple times faster in latency) than the Dash model, which is learnt to generate residual feedback conditioned on the ob- servations (including input and output samples, time steps and noise estimate) at the current point on the trajec- tory of the Dash model, lifting the noise estimate closely match the next-step estimate of the Dash model without JS; (3) Morse chains the outputs of the Dash and Dot models run in a time-interleaved fashion, allowing us to easily choose a proper JS step length to attain performance -ffaieceeff tradeoffs under a wide range of sampling step budgets. In- triguingly, the Dot model is significantly faster than the Dash model, it enables the Dot model to several times more sampling steps than the Dash model within the in- terval of two adjacent JS points while enjoying the same speed. Benefiting from this appealing merit, our method can perform more sampling steps under the same sampling step budget relative ing strong base achieve the desired acceleration goal. Besides the strong ability accelerate DMs, Morse is also efficient for training and inference, thanks weight sharing strategy between the Dash and Dot mod- els. In the strategy, we construct the Dot model by adding
better cefffommance-efficiinne
We overcome this barrier by presenting Morse, simple diffusion acceleration framework consisting of two models called Dash and Dot which tactfully couple IS with novel residual feedback learning strategy, compensating for the information loss and attaining the desired lossless accelera- tion in terms of image generation quality. In the formulation of Morse: (1) The Dash model is just the treitriine diffu-
pre-trained target DMs, establish-
proposed
2

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
extra light-weight blocks to the pre-trained DM and adopt lightweight Low-Rank Adaptation (LoRA) (Hu al., 2022) for fast training. During the training of the Dot model, the shared weights from the Dash model remain fixed, while the newly added layers and LoRA modules are trained jointly. On six public image generation benchmarks, our method achieves promising results under lots of experimental se- tups. In Fig. 1, we show illustrative text-to-image generation results using Stable Diffusion and LCM-SDXL with and without Morse under different sampling-step budgets.
to evaluate for one time. While the number of total steps T is mostly very large (e.g., T 1000 for DDPM (Ho et al., 2020)). It is essential for the generation process to well approximate the reverse of the forward process (Sohl- Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a). l)).
Jump Sampling, For a better sampling efficiency, most pre- vailing DMs adopt the jump sampling (JS) strategy, in which not all the time steps T, ,.... 0 but only decreasing sub- sequence of them are visited. We denote the sub- sequence as 4n > ... > lo(4 E [o, TT), mostly sampled uniformly from T to O. Therefore, the number of visited steps n can be much smaller than the total number of time steps T, leading to a faster speed for the generation process. With JS, each sampling step can be represented as:
2. Method
2.1. Background and Motivation
Basic Concept. A diffusion model (DM) can generate high quality images. It consists of a forward process for converting image to noise and a generation process (i.c., reverse process) for converting noise to image, both of which are typically formulated as Markov chains with T time steps in total. In the forward process, an image Xo E Raxuxx is first sampled from a data distribution D. At the t-th time step, the sample x, is added with a random noise 6 ~ (O, I) having the same dimension, which produces Xt+1 for the next time step + 1. The distribution for x; conditioned on Xo can be represented as:
_ = o(x,, Zt, te, 14-1),
(4)
where & is the schedule function used to update the sample from Xt, to X which is defined according to different samplers (e.g., DDPM (Ho et al., 2020), DDIM (Song et al., 2021a), SDE (Song et al., 2021b), DPM-Solver (Lu et al.. 2022), CM (Song et al., 2023)). Intuitively, for generation process, the neighboring steps tend to have similar sample X and close time step t as inputs for 0, leading to similar es- timate 2,. So that with the estimate Ze;- the sample can jump over multiple steps toward the same estimate from 4; to 44-1, without doing much harm to the sample quality, As more steps are jumped over, the step length between two adjacent JS points becomes longer and the performance degradation issue becomes more serious. Therefore, the double-edged nature of JS prohibits its use for performance lossless accel- eration, while it also leaves room for us to further improve it. If we can efficiently reduce the information loss caused by JS while maintaining its high sampling efficiency, then we can achieve better -ffferienneeffccinnncy tradeoff. This is the key motivation of our work.
p(x||x0)) = (11(x) II (D(x)(xx)))ddAs.
where p(x)) DD and p(xx/xx-1) corresponds to the pa- rameterized function for adding noise. As increases, X2 gets noisier, where XT conforms to the distribution (O, I). With the forward process. a neural network 0 is trained to estimate the original image Xo (equivalent to estimate noise <) from any time step t:
Z t(x,,t),
2.2. Morse
where z, denotes the estimate generated by the trained net- work for approximating Xy. Now, we can use 9 to reverse the forward process from noising to denoising for image generation. Specifically, in the generation process, a noise e ~ (O0 I) is firstly sampled as Xy. With the estimate zt from 0, we can approximate the distribution of p(x(-1|xx) by (Xx,, Xy = 2.) using Bayes' rule and Eq. 1:
As we discussed above, our key motivation is to efficiently reduce the information loss caused by JS while maintain- ing the high sampling efficiency. To achieve this goal, we present Morse, a simple diffusion acceleration framework with dual-sampling, as illustrated in Fig. 2. With Morse, the generation process is reformulated from iteration with a single model to interaction between two models, which are called Dash and Dot.
p(x p(xx|x4-1)((xx-- Xxn nt) pxx||X00 = 21)
(3)
Formulation of Morse. The Dash model is just the pre- trained DM, but operates in JS regime, creating sufficient space for sampling efficiency improvement. The Dot model is newly introduced by us for accelerating the Dash model, which is N times faster than the Dash model. During the generation process, each sampling step is either with noise estimate from the Dash model or the Dot model, while the
Therefore, we can iteratively convert a noise Xy to an im- age Xo along the time step from T to with p(xp) x7) = So far, we can gener- ate high quality images with T sampling steps following Eq. 3. However, such a generation process is very time- consuming. At each of T steps, the trained network needs
(p(xT) I[ p(x
3

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Network
Roverse Process
Latency
Quality
(a) Diffusion
Steps
High
High
Dash
0
Ze
End
x5
1000
800
600
400
200
5 LSDs
xtz
Xts
Dot
(b) Diffusion
3 Steps
Low
Low
f
C3
1
AZz.
End
x3
Ztz
1000
800
600
400
200
LSDs
(c) Diffusion with Morse (N=4)
6 Steps
Low
High
Latency per Step
θ
N
n
End
1000
800
600
400
200
3 LSDs
Time Step
Figure 2. Illustration of diffusion with Morse. Morse consists of two models named Dash and Dot, which interact with each other during the generation process. Dash is the pre-trained model of any type be accelerated, which operates in a jump sampling regime. Dot is the model newly introduced by us to accelerate Dash, which is N times faster than Dash in latency. We provide examples to show how our Morse works. For simplicity, we use the Lateney per Step of the baseline DM (LSD) as the time unit to calculate the total latency of a diffusion process. (i.e., the latency per step of the Dot model is mapped to that of the baseline Dash model) (a) Standard generation process, which performs steps under LSDs; (b) Standard generation process, which performs steps under LSDs: (c) Generation process with Morse, which performs 6 steps under 3 LSDs. Under the same latency, a generation process with Morse can perform more steps and achieve better sample quality.
two models play different roles. As we described in Sec. 2.1, the Dash model can estimate noise independently. The Dot model learns to generate residual feedback conditioned on the observations (including input and output samples, time steps, and noise estimate) at the current sampling point the trajectory of Dash, lifting the noise estimate to closely match the next-step estimate of the Dash model without JS. Morse chains the outputs of the Dash and Dot models run in a time- -nterintereeaee fashion. For a generation process with Morse, we reformulate how to estimate noise as:
trained blocks, the resolution of its input is reduced by 4t times. Therefore, the Dot model can be significantly faster than the Dash model. When training the Dot model, we fix the shared pre-trained layers and adopt lightweight Low- Rank Adaptation (LoRA) (Hu et al., 2022) for quickly adapt- ing to the new training objective and resolutions. With this simple and low-cost design, our Dot model can be derived from the pre-trained DM very efficiently, since it reserves nearly all the knowledge learned by the Dash model.
2.3. A Deep Understanding of Morse
Zt:- {{ [8(x...t) tt) Zt. + nT(x,,, Xt, tt, ti)
ESS
(5)
To have a deep understanding of how Morse can improve the sampling efficiency of DMs, we give detailed explanations in two perspectives.
t, $ S
where 9 denotes the Dash model; 17 denotes the Dot model: ls denotes the current sampling point on the trajectory of the Dash model when the Dot model produces noise estimation at the step t;; = ft.................... . tx, ) denotes the set of sampling steps with the noise estimates from the Dash model, which is sub-sequence of fa.... ....., to- The above formulation of Morse is simple and easy to implement, and has the great capability to accelerate diffusion models generally as tested with various experimental settings.
How can Morse Accelerate Different Diffussin Models? With JS, DMs can generate samples in a faster speed, yet in- evitably lead to worse sample quality due to the information loss over unvisited steps between two adjacent JS points on the diffusion trajectory. To compensate for the infor- mation loss, we insert extra multiple sampling points with Dot between every two adjacent JS points, which efficiently reduces the JS step length. Since Dot is N times faster than Dash, the inserted sampling steps can be completed by Dot with only 1 1/N time budget compared with Dash. In other words, Morse can perform more sampling steps under the same sampling step budget relative to the pre-trained DMs. We assume a standard generation process has n sampling steps. Under the same latency (for n sampling steps of base- line DMs), there could be n (0 < l < nı) sampling steps with Dash and Nkk: sampling steps with Dot in our Morse, which introduces (N 1)kk extra sampling steps. Given
Weight Sharing between Dash and Dot. To reduce the training and computational costs of the Dot model, we intro- duce a weight sharing strategy between Dash and Dot. As shown in Fig. 3, we construct the Dot model by adding mn trainable lightweight down-sampling blocks and up- sampling blocks on the top and under the bottom of the pre-trained Dash model respectively. The extra blocks have the significantly reduced numbers of channels and layers compared with the pre-trained blocks. For each of the pre-
4

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Input
Output
Input
kw
Xw
Output
The Dash Model
The Dat Model
Down sampling Dlock
Xx
It X
Extra Down-sampling Block
Bix W
Pt x
W
11 x
Middle Block
1 X
!1 W
Weight Sharing
Pi W
Up-sempling Black
Fixed
Fixed
16
16
+LoRA
Extra Up-sampling Block
Figure 3. Illustration of weight sharing between Dash and Dot. The Dot model is constructed by adding m (m for the illustrated example) trainable lightweight down-sampling and up-sampling blocks on the top and under the bottom of the pre-trained Dash model respectively. h x denotes the resolution of input feature maps. When training the Dot model, fix the shared pre-trained layers and add lightweight Low-Rank Adaptation A) to help the Dot model for fast convergence.
specific sampling step budget, Morse can flexibly change the JS step length by controlling k. Under ideal conditions where Dot and Dash perform exactly the same for noise es- timation, this leads to a speedup of (n - k + Nk)/n, which is the upper bound speedup for our Morse.
is reformulated interaction between the Dash and Dot models, rather than iteration with student DM; (2) Morse adopts an adaptive residual feedback strategy with trajectory information; (3) The aim of Morse is to efficiently reduce the information loss caused by jump sampling to attain the lossless acceleration goal. In distillation- student DM trained to match the outputs of its correspond- ing teacher DM in sampling process using much fewer steps, but always with performance degradation issue; (4) Morse is complementary to the distillation- -aaeed methods, which can be used to further accelerate a DM trained with distillation, as we show in the experiments.
methods, a
How can Dot Behave as Dash on Noise Estimation? To an- swer the question, we reiterate our design of the Dot model: (1) Dot cooperates with Dash by learning generate resid- ual feedback utilizing the trajectory information; (2) Dot inherits most of pre-trained weights from Dash. When train- ing the Dot model, we fix the shared pre-trained layers and add LoRA to help the Dot model for fast convergence. Ben- efiting from the first design, Dot does not need to estimate noise independently but generates residual feedback condi- tioned on the observations at the current sampling point the trajectory of Dash. By using the input sample, output sample, time steps and noise estimate as inputs. Dot gets the information about how the sample is updated between the two sampling steps, which largely helps Dot on adjusting the noise estimate of Dash. In the second design, we adopt a weight sharing mechanism between Dash and Dot. al- lows Dot to inherit most of the knowledge learned by Dash, which guarantees the consistency between Dash and Dot in the residual learning process. Additionally, the weight shar- ing mechanism also improves the parameter efficiency and training efficiency of Morse. By adding extra lightweight trainable blocks to a pre-trained DM, the Dot model can be trained very efficiently with LoRA. Thanks to the adap- tive residual feedback strategy with trajectory information and weight sharing mechanism, Dot is able easily lift the noise estimate at the current JS point to closely match the next-step estimate of the Dash model. Since IS strategy is adopted by most popular DMs, our Morse can widely accel- crate various DMs with different samplers, benchmarks, and network architectures under diverse sampling step budgets, as we show in what follows.
3. Experiments
3.1. Metric to Evaluate Speedup
Speedup. Before showing the experimental results, we first describe how we evaluate the speedup of Morse. For pre- trained DM, we assume two generation processes with and without Morse. The total latency of the process without Morse is n and the total latency of the process with Morse is I(n 2 l). The two processes get the same evaluation metrie. Then, the speedup of Morse under the latency of can be calculated as n/lx.
For a diffusion model (DM), we first measure its sample quality with and without Morse under different latencies, mainly using the mostly adopted metric Fréchet inception distance (FID, lower is better) (Heusel et al., 2017). The sampling steps are selected following the official settings. Then, we use linear interpolation to fit the curves between latency and evaluation metries for approximating the evalu- ation metric under any available latency. Note that it's too cime-consuming to evaluate the metrics with all the laten- cies. To be intuitive, we calculate an average speedup of Morse over the selected latencies. We fit a curve between a set of latencies and speedups to approximate speedups across all the latencies. All the speeds for different models are tested using an NVIDIA GeForce RTX 3090. Recall that Dot is N times faster than Dash. The speeds of the models may vary on different GPUs, leading to the change of N and speedup. While we find that Dash model and
Difference with the distillation-based methods. From the perspective of learning the knowledge from a pre-trained model, Morse is somehow similar with the distillation- based methods for diffusion. While they are different both in for- mulation and focus: (1) With Morse, the generation process
5

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
FID
DDPM
Speedup
FID
DDIM
Speedup
FID
(32-3)10
Spechup
FID
CelebA (64-64)
Speedup
320
Arg Spredap
All
2.51
4.0
160
Aug. Speelup
All
E.30
3.0
160
Aug. Speedupeppp
All
2.20
3.0
5.0
160
+00 LSD
3.40
80-
L50
2.45
2.5
90-
L201SD
2.45
20
4.0
80
3.0
40-
40
2201SD
40
20-
2.0
20-
2.0
10-
3.0
2.0
10-
1.5
10
1.5
10
-20
18D
1.24
1.0
5
1.0
1.0
1.0
150
150
3.42
5
10
1520
30
200
10 1520
100
200
10
1520
50 100
200
1520
100
200
Runtime Latency (LSD)
Runtime Lateney (LSD)
Runtime Latency (LSD)
Runtime Latency (LSD)
FID
DPM Solver (DDPM)
Speedup
FID
SDE
Speedup
FID
ImageNet (64x64)
Specdup
FID
CelehA-IIQ (256x256) Specdup
320
160
Aog Spoedap
All
2.50
2.50
4.0
320
640
Asg Scoodag
All
E.99
1.61
4.0
Aug Speedup
All
L&1
160
Avg Speedup
All
LS
L22
2.0
80
LSD
160
1.95
L30
80
2:00 LSD
40
80-
3.0
80
20
40-
40
1.5
10-
2.0
20
40-
10
20-
1.0
1.0
10 1520
30
1d0
200
10 1520
50
ión
200
10
10 1520
$0
100
200
10
10 1520
30
100
200
Runtime Latency (LSD)
Runtime Latency (LSDD
Runtime Latency (LSD)
Runtime Latency (LSD)
Solver (SDE)
Spesdlup
FID
L.UUN-Church (256x256)
320
Aug Speedup
Any Speodup L.7
2
3.0
160
80
2.0
LSD
2.10
2,5
Speedup
40
L20
162
Speedup
2.0
Baseline
20-
Baseline
10-
1.5
Morse
10-
+ Morse
5-
1.0
3-
1.0
3
5
10 1520
50 160
200
(0 1520
50 100
200
Runtime Lasency (LSD)
Runtime Latency (LSD)
Figure 4. Results of Morse with different samplers on CIFAR-10 henchmark. A speedup of n under the latency of means that the DM with Morse under I and the DM without Morse under nl achieve the same FID. We calculate the average speedups over the latencies, latencies from 3 to 15 LSDs and 20 to 100 LSDs.
Figure 5. Results of Morse with DDIM sampler on different image generation benchmarks.
Different Samplers. In the experiments, we evaluate our Morse with the mainstream samplers, including DDPM (Ho et al., 2020), DDIM (Song et al., 2021 a), DPM-Solver (Lu et al., 2022) for discrete samplers and SDE (Song et al., 2021b). DPM-Solver on SDE for continuous samplers. We conduct the experiments with CIFAR- 10 (Krizhevsky, 2009) benchmark, which adopted by all the above samplers for experiments. As shown in Fig. 4, our Morse can acceler- ate DMs consistently with all the samplers under different LSDs ranging from 3 to 100, achieving average speedups ranging from 2.01x to 2.94x. The results also show that our Morse can work with both discrete-time and continuous- time methods. Morse can even significantly accelerate the frate-of-the-ee sampler DPM-Solver, which can generate high quality images with very few steps by also utilizing the trajectory information from previous steps. Note that calculate the speedups of Morse N/A for DPM-Solver on both DDPM and SDE with 100 LSDs, which are not used for calculating the average speedups. The reason is that there is no room to accelerate, since the FIDs constantly keep the same (even worse) value with latencies larger than 100 LSDs for the baseline DMs. The results with different samplers other benchmarks are provided in the Appendix.
its Dot model mostly have little change in N on different GPUs, our Morse demonstrates good acceleration ability consistently. Details are provided in the Appendix.
LSD. For simplicity and generalization, we normalize the total latency of diffusion process with the time unit Latency per Step of the baseline DM (LSD), namely the time cost of the Dash model for one sampling step (i.e., the latency per step of the Dot model is mapped to that of the baseline Dash model). It takes 1 LSD for Dash and 1/N LSD for Dot to perform one step. For diffusion process without Morse under n sampling steps, its latency (namely the end-to-end time for generation images) can be represented as n LSDs.
3.2. Accelerate Image Generation
Experimental Setup. For each DM evaluated in experi- ments, we collect its official pre-trained model as the Dash model, of which the weights are fixed. With the weight shar- ing strategy, all the Dot models are trained following the official training settings, while typically with reduced batch size and training iterations. We typically set the number of extra down wown-sampling blocks and up-sampling blocks m to 2, leading to N in the range of 5 to 10. All the experiments are performed on the servers having 8 NVIDIA GeForce RTX 3090 GPUs. More details are described in the Appendix.
Different Benchmarks. In the experiments, we further evaluate our Morse with 5 popular image generation bench- marks, including CIFAR-10 (32x32) (Krizhevsky. 2009), ImageNet (64x64) (Russakovsky et al., 2015), CelebA (64x64) (Liu et al., 2015), CelebA-HQ (256x256) and
6

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
FID
Class-Conditioaal
Speedup
FID
Classifier Guidance
Speedup
Table 1. FIDs of Stable Diffusion with and without Morse on MS-
320
160
80
0
-30 1:0
Avg Speedap
All
1.78
1:57
1.50
2,5
3.0
520
160
80
LS0
Aug Sceelug
All
8.00
1.00
4.0
COCO. We calculate FIDs under different classifier- guidance
scales and select the best FID among all the scales and FID under
20-
40
0.
2.0
40-
20
2.0
default scale 7.5 for comparison.
I Latency (LSD)
10-
5
1.5
10-
1.0
Method
FID
10
15
20
50
1.0
Stable Diffusion
scale of 7.5
11.75
11.92
12.35
13.53
5 10 1.20
30
200
10 1520
100
200
best scale
10.65
9.47
8.70
8.22
Runtime Lateney (LSD)
Runtime Lateney (LSD)
scale of 7.5
9.29
10.07
10.93
13.22
+ MMore
Speedup
Baseline
Morse
best scale
8.60
8.55
8.29
8.15
Figure 6. Results of Morse with different conditional generation strategies ImageNet benehmark.
FID
18-
DM +Morse
LSUN-Chureh (256x256) (Yu et al., 2015). Since we have evaluated Morse with different samplers, we keep the sam- pler as the most widely used DDIM in the following ex- periments unless otherwise stated, to exclude the impact of differences in samplers. The results are shown in Fig. 5. Our Morse can be generalized well to all the benchmarks, which have different image resolutions (from 256 x 256 for LSUN- Church and CelebA-HQ -H to 32 for CIFAR-10) 1-), different dataset sizes (from 1.2 million for ImageNet to 30 thousand for CelebA-HQ) and different semantic information. For all the benchmarks under most LSDs, our Morse gets speedups around 2x. On the CelebA, it can even achieve specdups more than 4x under some LSDs.
16
10 LSDs
14
15 LSDs
12
20 LSDs
10
88
50 LSDs
8
2
3
4
5
6 Scale
7
8
9
10
Figure 7. Stable Diffusion with and without Morse under different latencies and scales.
with only about 2M text-image pairs at resolution 512x512 sampled from the LAION-5B dataset. We use DDIM as the sampler. Following the popular evaluation protocol, we adopt the FID (lower is better) and CLIP score (Radford et al., 2021) (higher is better) as the evaluation metrics and use the 30000 generated samples with the prompts from the MS-COCO (Lin al., 2014) validation set for evaluation. The CLIP scores are calculated using ViT-g/14, All the ex- periments are performed on server having 8 NVIDIA Tesla V100 GPUs. More details are described in the Appendix.
Different Conditional Generation Strategies. After show- ing the effectiveness of Morse under unconditional genera- tion, we next evaluate our Morse under conditional genera- tion with different strategies, including condiisondiii and classifier-guided image generation (Ho & Salimans, 2021) on ImageNet benchmark at resolution 64. For the clas- sifier guidance, we consider the classifier as part of Dash and train the Dot to approximate the estimate guided by a classifier. As shown in Fig. 6, Morse can well generalize to conditional generation with different strategies.
Results Comparison. Following the default settings, we first evaluate the FIDs of Stable Diffusion with and without Morse, using the classifier-free guidance scale of 7.5. While we find that increasing the number of steps does not always lead consistently better FID scores for standard Stable Diffusion, as shown in Table 1. Therefore, we find another two schemes to evaluate speedups. In the first scheme, we evaluate the FID with different scales and select the best FID score for comparison. From the results shown in Fig. 7, we can find that the best FID consistently gets better when the latency increases. Under this scheme, we can calcu- late an average speedup of 2.29x. In the other evaluation scheme, we fit the curves between FID and CLIP scores un- der different scales using the linear interpolation following Stable Diffusion. The results are shown in Fig. 8. When the scale is larger than 4, we can observe a tradeoff between the two metrics. The curve with our Morse is below the curve without Morse. For example. the curve with Morse under 10 LSDs is below the curve without Morse under 20 LSDs in most scales, indicating an average speedup of approxi- mately 2x. Some generated samples for comparison are provided in Fig. and Appendix. The results further demon- strate the generalization ability of Morse. On the popular rext-to-image generation task with a large DM 522M), our Morse still shows significant acceleration ability. As
Different Network Architectures. In the above experi- ments, there are different network architectures collected from 6 research works with model sizes ranging from 35.75M to 421.53M for the Dash models (Rombach et al., 2022; Ho et al., 2020; Nichol & Dhariwal, 2021; Song et al.,, 202 Dhariwal & Nichol, 2021). It can be seen that our Morse achieves good generalization ability under all diffusion architectures with different capacities and model sizes.
3.3. Accelerate Text-to-Image Generation
Next, we evaluate our Morse under the highly popular text- to-image generation task with the latent-space Stable Diffu- sion model (Rombach et al., 2022).
Experimental Setup. We select the Stable Diffusion v v1.4 as our Dash model, which is pre-trained with around bil- lion text-image pairs from LAION-SB dataset (Schuhmann et al., 2022). In our experiments, the Dot model is trained
7

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Table 2. Training details of Stable Diffusion and the corresponding Dot model in Morse. The training memory cost tested with the batch size of 8 per GPU.
Table 3. Ablation of Morse with different trajectory information.
Table 4. CLIP scores of LCM-
SDXL with and without Morse
Meibod
X,
Rr.
FID
on MS COCCC.CO
Model
Params
(M)
Training Samples
(M)
Training Cost
(A hours)
Training Memory
(MB)
DDIM
13.67
13.56
Mothod
LSD
CLIP score
25.39 15.39
Stable Diffusion
859,52
2,000
150,000
23,185
8.11
2
29.40
Dol model
97.84
2
190
18,841
8.06
LCM-SDXL
3
30.34
(+11.4%)
(+0.15)
(+0.1%)
(-19.8%
Morse
7.60
4
30,80
7,50
1,33
28.70
Stable Diffissio
Stable Diffusion with Morse
6.83
LCM-SDXL
1.67
29.84
FID FID vs CLIP Scores on MS-COCO
FID FID vs CLIP Scores on MS-COCO
7.27
with Monse
2
30.30
10LSDs
LSUS
6.60
3
30.83
15155s
18-
16
20 LSDs
16-
20 LSDs
LCM-SDXL, with Morse. In the above experiments we have demonstrated the great capability of Morse to acceler- ate 9 baseline diffusion models 6 image generation tasks, Here, we further show that Morse can be also generalized to improve Latent Consistency Models (LCM) (Luo et al., 2023), which is a popular distillation- based method tailored for few-step text-to-image synthesis. We use LCM-SDXIL as the baseline, which denotes a Stable Diffusion XL model fine-tuned with LCM. The resolution is x024 1024. We evaluate LCM-SDXL with Morse on MS-COCO benchmark as described in See 3.3. Same with Stable Diffusion, Stable Diffusion XL also adopts the classifier-free guidance, while LCM fixes the scale to 7.5 during the distillation. Under the fixed scale of 7.5, for standard LCM-SDXL, we notice that its FID score does not consistently get better as the number of steps increases, while the CLIP score does. Therefore, we select the CLIP score as the metric for evaluating LCM- SDXL with Morse. The results are shown in Table 4. Since the Dot model is 3 times faster than the Dash model, we can evaluate the CLIP scores for LCM-SDXL with Morse under LSDs of 1.33 and 1.67. Over a sampling step number from 1 to 4, we can calculate an average speedup of x.43x on the CLIP score for our Morse. Experimental details and some generated samples for comparison are provided in the Appendix.
50 LSD.
50 LSDs
10-
25
26
26
Score
CLIP (b)
FID
Best FID on MS-COCO
Specdup
11.0
10.5
Speedup
9.5
Averago Speedup: 2.29
Haseline
2.0
9.0
Morse
81
8.0
10
15
20
Runtime Latency (LSD)
(c)
Figure 8. Results of Morse with Stable Diffusion. (a) and (b) are curves between FIDs and CLIP scores for Stable Diffusion with and without Morse on different LSDs under guidance scales of 3, 5,5, 6, 7, 7.5, 8, 9, 10, which correspond to the points in the curves from left to right. We paint the background using the curve of standard Stable Diffusion for better illustration; (c) Curves between FIDs and LSDs using the hest FIDs among different scales.
shown in Table 2, while the Dash model is trained with heavy computational resources and large datasets, our Dot can be trained very efficiently with less than 0.1% text- image pairs and 0.1 % training cost compared with it. With the trajectory information, the Dot model can be easily close to the Dash model on noise estimation. The results also show that our Morse works well with the classifier-free guidance and the latent-space diffusion models.
Stable Diffusion FID COCO
CIFAR 10 (32x32)
128
3.4. Ablation Study and More Comparisons
64
18
32-
16-
Effect of Trajectory Information. Recall that our core insight is that the trajectory information can help the Dot model to perform as well as the Dash model without JS noise estimation. In Morse, we use the sample x,,, the time step to and the noise estimate Zt, at the current sampling point on the trajectory of Dash as the extra inputs for Dot. In the experiments, we evaluate Morse with different combina- tions of them with DDIM sampler on CIFAR- 10 benchmark under 10 LSDs. From the results shown in Table 3, we can see that each of the inputs is helpful for Dot on residual esti- mation. Without the trajectory information, the introduction of the Dot model can not accelerate DMs anymore, because of its inferior estimation. These ablative results prove that the trajectory information plays a key role in our design, which also validate our key insight to some extent.
16-
10
10
20
100
alo
400
Throughuu (Images Second) (a)
Throughput (Images Second)
(b)
Baseline
Morse
DeepCache
Figure 9. Comparison between Morse and DeepCache with Stable Diffusion v1.4 using DDIM sampler. To give a clear comparison, we report the FIDs of two methods under different throughputs, which are evaluated on an NVIDIA GeForce RTX 4090 GPU with the batch size of 20. The results are tested with the public code of DeepCache. Following the official settings of DeepCache, we evaluate it with the number of steps 20, 20,50, 100 under N 2, 3, 5, 10 on CIFAR-10 10 and the number of steps 20, 50 under N 2, 3,5 5 on MS-COCO.
8

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Table 5. FIDs of Morse and PFDiff on MS-COCO benehmark with Stable Diffusson. The results of PFDiff are collected from the paper. Following the default settings of PFDiff, we evaluate Morse with 10000 generated samples under the guidance scale of 7.5.
4. Related Work
Besides the fast samplers discussed in the Introduction sec- tion, there are other emerging efforts to speed up the infer- ence of DMs. Some recent works use quantization (Li et al., 2023b; Chen al., 2023b), pruning (Li et al., 2022; Wang et al., 2024), reuse of parameters and feature maps (Agarwal et al., 2024; Wimbauer et al., 2024; Ma et al., 2024), time step schedule optimization (Watson et al., 2021; Xue et al., 2024; Sabour et al., 2024), and GPU-specialized optimiza- tion (Chen et al., 2023e; Li et al., 2024) to reduce runtime model latency. Another line of research (Li et al., 2023c; Xu al., 2024; Li et al., 2023a) seeks to design lightweight network architectures for DMs, enabling to deploy them on mobile devices, In design, our method is orthogonal to these methods, and thus it should be able to combine with them for improved performance.
Latency
(LSDD
Method
6
10
15
20
Stable Diffusion
20.33
16.78
16.08
15.95
+ PFDDff
15.47
13.06
13.57
13.97
+ -Morse
13.88
11.67
12.22
13.63
Table 6. FIDs of Morse and AYS on ImageNet (64x64) bench- mark. For fair comparisons, collect the results of AYS from the paper and evaluate Morse under the same settings.
Latency (L.SD)
Method
5
10
15
DDIM
147.44
39.40
28.68
+ AYS
50.38
29.23
24.21
+ Morse
46.63
25.76
21.50
The idea of using dual-model designs to strike a better accuracy-efficieng tradeoff is popular in both computer vi- sion and natural language processing. SlowFast network (Fe- ichtenhofer et al., 2019), powerful and efficient architec- ture for video action recognition, uses a slow pathway op- erating at a low frame rate with low resolution to encode spatial semantics, and a parallel fast pathway operating at a higher frame rate with higher resolution to encode motion cues, Speculative decoding (Stern et al., 2018). fast de- coding mechanism for accelerating the inference of autore- gressive language models, predicts candidate tokens with a small approximation model, and verifies the acceptability of these candidate tokens by a larger and powerful target model with a single forward pass, significantly reducing the computation for accepted tokens. Many variants (Li et al., 2020; Leviathan et al., 2023; Chen et al., 2023a; Zhang et al., 2024) of them have been proposed. Although our method is also dual-model design, it focuses accelerating diffu- sion models with a simple framework, and its key insight is to reformulate the iterative generation (from noise to data) process via taking advantage of fast jump sampling and adaptive residual feedback strategies. Clearly, our method differs from them focus, motivation and formulation.
To further demonstrate the effectiveness of Morse, we compare Morse with the state- th-the-art feature reuse methods, including DeepCache (Ma et al., 2024) and PFDiff (Wang et al., 2025). Same to Morse, the methods also explore the temporal step redundancies for accelerating diffusion models, while in different ways. Specifically, DeepCache reuses the features at step for N 7--1 1 following steps, and PFDiff utilizes the states of past steps stored in a buffer to update the states of future steps. As shown in Figure 9, we can find that Morse consistently gets better throughput and FID than DeepCache on both benchmarks. For acceleration, DeepCache is lossy in gen- eration quality due to reusing most of the features at step for N 1 following denoising steps, yet Morse is lossless. From the results shown in Table 5, we can find that Morse is also superior to PFDiff.
Comparison with Feature Reuse.
Comparison with Time Step Schedule Optimization. To acceeerate diffusion models, time step schedule optimiza- tion methods design different strategies to choose optimal time steps when given a small number of sampling steps. The line of work is also closely related to Morse. In Ta- ble 6, we provide the comparison results between Morse and AYS (Sabour et al., 2024), which is a state-of-the-art method in this domain. As shown, Morse performs better than AYS over different sampling step budgets.
5. Discussion and Conclusion
We present a simple dual-sampling framework called Morse to accelerate diffusion models losslessly. Morse reformu- lates the itcrative generation process by involving two mod- els called Dash and Dot that interact with each other, which exhibits the merit of flexibly attaining high-fidelity image generation while improving overall sampling efficiency. Ex- perimental results show that Morse can generally accelerate diffusion models under various settings. While Morse shows the general acceleration ability, it introduces an extra Dot model with a small number of trainable parameters.
More Ablations and Discussions. In the Appendix, we provide more ablative experiments and discussions about Morse for a better understanding, including: (1) The runtime latencies of Dash models and Dot models on different GPUs; (2) The principles to determine the scheduling of Morse; (3) The performance of Morse under a specific latency using different numbers of steps with Dash; (4) Comparison be-
tween different architectural designs for the Dot model; (5)
Other results and more generated samples.
9

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Impact Statement
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In ICML., 2023.
As an acceleration method for diffusion models, Morse has broader impacts similar to most generative AI models. For example, it may be misused to help creating realistic fake news and videos to spread false information.
Li, M., Lin, J., Meng, C., Ermon, S., Han, S., and Zhu, J. Efficient spatially sparse inference for conditional gans and diffusion models. In NeurIPS, 2022.
References
Li, M., Cai, T., Cao, J., Zhang, Q. Cai, H., Bai, J., Jia, Y., Li, K., and Han, S. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024.
Agarwal, S., Mitra, S., Chakraborty, S., Karanam, S., Mukherjee, K., and Saini, S. K. Approximate caching for efficiently serving text-to-image diffusion models. In NSDI, 2024.
Li, S., Hu, T., Khan, F. Li, L.., Yang. S., Wang, Y., Cheng, M.-M., and Yang. J. Faster diffusion: Rethinking the role of unet encoder in diffusion models. arXiv preprint arXiv:2312. 09608, 2023a.
Blatmann, A., Rombach, R., Ling, H., Dockhor, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In CVPR, 2023.
Li, X., Wang, Y., Zhou, Z., and Qiao, Y. Smallbignet: Inte- grating core and contextual views for video classification. CVPR, 2020.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint 2rXiv:2302.013/6 2023a.
Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., and Keutzer, K. Q-diffusion: Quantizing diffusion models. In ICCV, 2023b.
Chen, T., Zhang. R., and Hinton, G. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. In ICLR, 2023b.
Li, Y., Wang, H., Jin, Q., Hu, J., Chemerys, P., Fu, Y., Wang, Y., Tulyakov, S., and Ren, J. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. NeurIPS, 2023c.
Chen, Y., Sarokin, R., Lee, J., Tang, J., Chang, C.,, Kulik, A., and Grundmann, M. Speed is all you need: On- device acceleration of large diffusion models via gpu- aware optimizations. In CVPR Workshops, 2023c.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. In NewrIPS, 2021.
Feichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast networks for video recognition. In ICCV, 2019.
Liu, J., Li, C., Ren, Y., Chen, F. and Zhao, Z. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In AAAI, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, Gans trained by a two time-scale update rule converge to local nash equilibrium. In NIPS, 2017.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In ICCV, 2015.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NewrlPS Workshop, 2021.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurlPS, 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, 2020.
Luo, S., Tan. Y., Huang, L., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Ho, J.. Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. JMLR, 2022.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Ma, X., Fang, G., and Wang, X. Deepeache: Accelerating diffusion models for free. In CVPR, 2024.
Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffusion models, In CVPR, 2023.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical Report, 2009.
10

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Mohamed, S. and Lakshninarayanan, B. Learn-
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er-
ing in implicit generative models. arXiv preprint arXiv:1610.03483, 2016.
mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In ICML, 2021.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. In ICML, 2023.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. In NeurIPS, 2018.
Wang, G., Cai, Y., Peng, W., Su, S.-7., et al. Pfdiff: Training- free acceleration of diffusion models combining past and future scores. In /CLR, 2025.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. ICLR, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In ICML, 2021.
Wang, H., Liu, D., Kang, Y., Li, Y., Lin, Z., Jha, N. K., and Liu, Y. Attention-driven training- free efficiency enhance- ment of diffusion models. In CVPR. 2024.
Watson, D., Ho, J., Norouzi, M., and Chan, W. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 2012002...... 2022.
2021.
Wimbauer, F., Wu, B., Schoenfeld, E., Dai, X., Hou, J., He, Z., Sanakoyeu, A., Zhang, P., Tsai, S., Kohler, J., et al. Cache me if you can: Accelerating diffusion models through block caching. In CVPR, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Xu, Y., Zhao, Y., Xiao, Z.,, and Hou, Ufogen: You forward once large scale text-to-image generation via diffusion gans. In CVPR, 2024.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Li, F.-.. Imagenet large scale visual recognition challenge. IJCV, 2015.
Xue, S., Liu, Z., Chen, F., Zhang, S., Hu, T., Xie, E., and Li, Z. Accelerating diffusion sampling with optimized time steps. In CVPR, 2024.
Sabour, A., Fidler, S., and Kreis, K. Align your steps: Optimizing sampling schedules in diffusion models. In ICML, 2024.
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint 15XX::150060366. 2015.
Xiao, Lsun:
Saharia, C., Chan, W., Saxena, S., Li, L.., Whang. J., Denton,
E. L.. Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T. et al. Photorealistic text-to-image dif- fusion models with deep language understanding. In NeurlPS, 2022.
Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In ACL, 2024.
Salimans, T. and Ho, Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5h: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. In ICL.R 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In ICML., 2015.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2021a.

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
A. Appendix
A.1. Benchmarks and Evaluation Details
Image Generation. In the experiments described in Sec. 3.2, we consider 5 mainstream image generation benchmarks
with various resolutions for evaluating the generalization ability of our Morse, including CIFAR-10 x32x32, 50 thousand images) (Krizhevsky, 2009), CelebA (64x64, 0.2 million images) (Liu et al., 2015), ImageNet (64x64, 1.2 million images) (Russakovsky et al., 2015), CelebA-HQ (256x256, 30 thousand images) (Liu al., 2015), LSUN-Church (256x256, 0.1 million images) (Yu et al., 2015). Following the popular evaluation protocol, for each DM, we generate 50000 samples and calculate the FID score between the generated images and the images of the corresponding benchmark. For fair comparisons, we adopt the settings including data processing pipeline and hyperparameters following the corresponding DMs.
Text-to-Image Generation. In the experiments for Stable Diffusion v1.4 and CCM-SDXL, we use million text-image pairs sampled from the LAION-SB (Schuhmann et al., 2022) dataset. Following the popular evaluation protocol, we evaluate the text-to-image diffusion models under zero-shot text-to-image generation on the MS- IS-COCO 2014 validation set (Lin et al., 2014) (256x22 256). All the generated images are down-sampled from 512 to 256x x 256 for evaluation. For each DM, we generate 30000 samples with the prompts from the validation set. The CLIP scores are calculated using ViT-g/14.
A.2. Implementation Details for Stable Diffusion
Implementation Details. For text-to-image generation, we evaluate our Morse with Stable Diffusion v1.4 (Rombach et al., 2022). In the experiments, we use the Dot model with the extra parameters of 97.8 97.84M to accelerate the Dash model with the size of 859.52M. The latencies of the Dash model and the Dot model are 0.709 second and 0.082 second per hatch respectively (NV = 8.6), which is tested using a single NVIDIA GeForce RTX 3090 under a batch size of 20. With the official settings, Stable Diffusion v1.4 is pre-trained with around 2 billion text-image pairs at resolution 256x256 and fine-tuned with around 600M text-image pairs at resolution 512x512 from LAION-5B dataset (Schuhmann et al., 2022). We add two trainable down-sampling blocks and up- up-sampling blocks, with the numbers of channels 96 and 160, on the top and under the bottom of the pre-trained Stable Diffusion to construct the Dot model respectively. We set the rank of LoRA to 64. In our experiments, the Dot model is trained with only about 2M text-image pairs at resolution 5 512x512 sampled from the LAION-5B dataset for 100,000 iterations. We use DDIM as the sampler.
For conditional image generation, Stable Diffusion v1.4 adopts the classifier free guidance, which has a parameter called guidance scale to control the influence of the text prompts on the generation process. To ensure that our Morse can also work well with different guidance scales besides the different numbers of steps, we also randomly sample the guidance scales between 2 and 10 during the training procedure. The Dot models are trained on server with 8 NVIDIA Tesla V100 GPUs. Considering the risk for misuse of the generative models, we use the safety checker module which is adopted by Stable Diffusion project for the released models.
Latency of Each Block. It may be not intuitive that we can construct Dot model with faster speed by adding extra blocks to DM. Here, we provide the latency of each block for the Stable Diffiusion with and without extra blocks in Fig. 10. The ofate-of-the-art DMs mostly adopt the U-Net architecture with self-attention layers. With the extra blocks on the top and under the bottom of the pre-trained Stable Diffusion, the resolution of the input for each pre-trained block is reduced by 16 times, which significantly reduces the latencies of the pre-trained blocks. Additionally, the extra blocks have the same architecture with the pre-trained blocks while removing the self-attention layers. Since the computational complexity of self-attention layer grows quadratically with the number of tokens, the pre-trained blocks with high-resolution feature maps have relatively slow inference speeds. By removing the self-attention layers and reducing the number of channels, the latencies of the extra blocks with the high-resolution feature maps are still relatively low. Therefore, the Dot model can be significantly faster than the Dash model.
A.3. Implementation Details for LCM-SDXL
In the main experiments, we also evaluate our Morse on the Latent Consistency Models (Luo et al., 2023) LDXL with 1024x x 1024 resolution, which is already accelerated with consistency distillation technique). LCM-SDXL can be used for high quality text-to-image generation with very few steps, which trained with heavy computational resources and large dataset. We add a trainable down samplsan block and up-sampling block on the top and under the bottom of the pre-trained LCM-SDXL respectively. For each of the original pre-trained blocks, the resolution of its input is reduced by
12

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Input
Output
Input
0.005
0.011
Output
The Dash Model
The Dot Model
0.003
0.008
0.123
0.209
0.005
0.011
0.056
0.126
0.005
0.011
0.044
0.095
0.005
0.010
0.006
0.018
0.001
0.003
0.009
0.002
Down- Down-sampling Block
Up sampling Block
Middle Block
Extra Down-sampling Block
Extra Up- sampling Block
Figure 10. Latency (second) of cach block for Stable Diffusion with and without adding extra down- lown-sampling and up-sampling blocks. The speeds are lested with the batch size 20 on a single NVIDIA RTX 3090 GPU.
4 times. The latencies of the Dash model and the Dot model are 0.646 second and 0.211 second per batch respectively (N = 3.1), which is tested using single NVIDIA Tesla V100 under a batch size of 5. We fix the shared pre-trained layers except some mismatched layers and add lightweight Low-Rank Adaptation (LoRA) (Hu et al., 2022) to help the Dot model for fast convergence. Compared to the LCM-SDXL with the model size of 2567.55M, the Dot model only has 229.19M trainable parameters, which can be efficiently injected to the Dash model. In our experiments, the Dot model is trained with about 2M text-image pairs resolution 1024x 1024 from the LAION-5B dataset for 100,000 iterations. The Dot model is trained on the servers with 8 NVIDIA Tesla V100 GPUs.
A.4. Implementation Details for Image Generation
In this section, we provide the implementation details for all the DMs adopted in our experiments for image generation.
Table 7. Lateney (sccond) per sampling step of the Dash models and the Dot models on different GPUs N denotes that the Dot model is N times faster than the Dash model. x w denotes the resolution of input feature maps.
RTX 3090
RTX 4090
Tesla V100
Model Source
Benchmark
Dash
Dol
N
Dush
Dol
N
Dush
Dol
N
CIFAR- 10 (32x32)
0.072
0.012
6.0
0.035
0,006
5.8
0.082
0.015
5.5
DDPM
CelebA Baebb--Q((u (256x256)
0.539
0.112
4.8
0.346
0.073
4.7
0.680
0.135
5.0
DDIM
CelebA (64x64)
0.244
0.042
5.8
0.143
0.021
6.8
0.292
0.054
5.4
Improved DDPM
ImageNet (64x64)
0.367
0.065
5.6
0.226
0.034
6.6
0.458
0.092
5.0
SDE
CIFAR- 10 (32x32)
0.120
0.020
6.0
0.113
0.018
6.3
0.139
0.025
5.6
LSUN-Church (256x256)
0,288
0.060)
4.8
0.185
0.022
8.4
0.360
0.057
6.3
LDM
MS-COCO (512x512)
0.709
0.082
8.6
0.344
0.042
8.2
0.771
0.088
8.8
ADM
0.956
0.149
6.5
0.760
0..885
8.9
1.105
0.186
5.9
ImageNet (64x64)
ADM-G
1.547
0.149
10.5
0.956
0.085
11.3
1.889
0.186
10.2
Training and Sampling. In the main experiments, we adopt multiple DMs to evaluate the effectiveness of our Morse, including the models from DDPM (Ho et al., 2020), DDIM (Song et al., 2021a), Improved DDPM (Nichol & Dhariwal, 2021). SDE (Song et al., 2021b), LDM (Rombach et al., 2022) and ADM (Dhariwal & Nichol, 2021). For a DM, we collect its official preitrdi model as the Dash model. To construct the corresponding Dot model, we add two lightweight down-sampling blocks and up-sampling blocks on the top and under the bottom of each pre-trained Dash model respectively. With the weight sharing strategy, all the Dot models are trained following the official training settings. As shown in Table 7, we provide the detailed information, including the source models, the speeds of the Dash models and Dot models and N. Recall that the Dot model is N times faster than the Dash model. All the speeds for different models are tested using single NVIDIA GeForce RTX 3090. When testing the speeds, we set the batch size to 100 for most benchmarks except 20 for CelebA-HQ and MS-COCO benchmarks. During the training procedures, a Dot model is trained to estimate the difference between the outputs from the Dash model at two randomly sampled steps. Here, we give an example of the training procedure and sampling procedure for DDIM sampler, as shown in Algorithm I and Algorithm 2. The procedures can be easily extended to other samplers with simple modification. The Dot models are trained on the servers with 8 NVIDIA
13

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Tesla V100 GPUs or 8 NVIDIA GeForce RTX 4090 GPUs.
N on Different GPUs. Recall that Dot is N times faster than Dash. For a Dash model and its trained Dot model, the speedup of Morse gets larger when N gets larger. While the speeds of the models may vary on different GPUs, leading to the change of N and speedup. In our design, we construct a Dot model by adding several extra blocks on the top and under the bottom of the pre-trained Dash model. Therefore, a Dot model has the architecture which is very similar with its corresponding Dash model. As shown Table 7, we can find that a pair of Dash and Dot mostly has little change in N on different GPUs. The results indicate that our Morse performs well on different GPUs.
Algorithm 2:
DDIM Sampling with Morse
Algorithm I: Training of Dot with DDIM
Require : Trained network Dash ((:-)
Require: : Trained Dash model ((:-)
Require: : Trained network Dot 7l 55))))))
Require: Dot model " :.))
to be trained
Require: Schedule function @(t, 's'
Require: Schedule function (0' (5)))
Require: : Sequence of time points t, mn-1 >
to
Require: : Dataset 2
Require : Number of dash steps d
Require: : Learning rate y
sample xt., ~ ~N(00,)
repeat
uniformly sample Sds.... So from tn to to
sample D
3 for i 4 n to do
3
sample E ~ (0,I)
if' 4 E .8.....,81) then
sample t,,to ~ U[0, T] (t, > to)
2i, ((xi,; (4)
x = =at,x 1 ot.-
6
Ze, = 8(x,, te)
6
t, t
else
xt. o(xt, Ze, tayto) fo)
| Z1, u, + n(xx,, X Zi, ber4)) end
Zto = 0(xto: to)
9
2 = 21, + [Xi,, ; x Bl Bi,, to)
10
1 1 11 - VV, Iizi, 2,011 11
10
Xt.-1 = (xx,, Zeo t1-)
11 end
until convergence
Return :X
How to Determine the Scheduling of Morse. Generally, how the scheduling of Morse is determined relies on three key factors: (1) The selection of the jump-sampll sampp--amliing-assed diffusion sub-sequence of the Dash model. With jump sampling, only a sub-sequence of the available time steps (e.g., T.....) are visited. Clearly, there are different strategies to select the sub-sequence. In Morse, we typically use a simple selection strategy following the standard jump sampling settings (which are supported with most existing diffusion methods) of any pre-trained diffusion model (i.e. the Dash model), for easy implementation; (2) Given the numbers of sampling steps for Dash and Dot, how to decide their orders. For a diffusion process with Morse, we denote the number of sampling steps for Dash as kdash and the number of sampling steps for Dot as kdot, leading to kdash + kidot sampling steps in total. Since the Dot model needs the trajectory information of the Dash model for predicting its residual feedback, the first sampling step can only be with the Dash model, and then we can flexibly decide orders of the remaining steps. For simplicity, we uniformly sample the steps with the Dash and Dot models. For example, if kdash = 2 and Kaot = 4, the order of the steps can be denoted as Dash, Dot, Dot, Dash, Dot, Dot; (3) Given the desired latency budget, how to decide the number of sampling steps for Dash and Dot. Recall that Dot is N x faster than Dash. Under the desired latency of LSDs (LSD is the defined time metrie, namely the time for the baseline diffusion model perform one step). there could be n k (0 I for Morse. The problem how to decide the k. Under ideal conditions where Dot and Dash perform exactly the same for noise estimation, Morse achieves a speedup of (n k- + Nk)/n. While Morse can accelerate a pre-trained diffusion model with a wide range of & (as shown in Fig. 13). Based on our experiments with 10 diffusion models on 6 benchmarks, we suggest setting (n- k + Nk)/n / between 2.0 and 3.0, which leads mostly to the best results. With the above three simple principles, it's easy and flexible to decide the scheduling of Morse under different latencies.
n)) sampling steps with Dash and Nk sampling steps with Dot
A.5. More Experiments for Studying Morse
Morse with Different Samplers. In the experiments described in Sec. 3.2, we evaluate our Morse on CIFAR-I0 (32x32) benchmark with different samplers. Here, we perform experiments to further validate the effectiveness of Morse on other datasets with different samplers. As shown in Fig. 11, our Morse achieves good generalization ability on CelebA-HO -HQ (256x256) dataset with different samplers.
14

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
FID
DDPM
Speedup
FID
DDIM
Speedup
DPM-Solver (DDPM) Speedup
160
Speeduu
2.5
160
Speedup
2.5
320
Speedup
Baseline
Baseline
160
Baseline
Morse
Mome
2.0
Morse
2.0
40
40-
20
1.5
20
1.5
20-
1.5
10
Average Speedup: 1.71
1.0
10
Average Speedup: 1.82
-10
10-
Average Speedup: 1.98
1.0
%
100
200
&
100
200
V
200
Runtime Lalency (LSD)
Runtime Latency (LSD)
Runtime Lalency (LSD)
Figure 11. Results of Morse with different samplers CelebA-HO x266x266) benchmark.
FID
Trajectory of Dash Model Speedup
FID
Trajectory of Both Models Speedup
160
3.0
160
3.0
80
80
40
Speedup
20
2.0
20
2.0
Baseline
10-
10-
5-
0i:::o
Morse
Average Speedup: 2.26
1.0
Average Speedup: 2.00
1.0
&
100
200
&
100
200
Runtime Latency (LSD)
Runtime Latency (LSD)
Figure 12, Results of Dot with trajectory information from the Dash model and the both two models.
Where Trajectory Information Comes from? Recall that Morse redefines how to estimate noise during the generation process
0 ((xt,,(1) (2) Zt. + n(x,, Xt, t4, Zi,) $ 5
4 4ES
Zt:
(6)
In the design, Dot model generates residual feedback conditioned on the observations at the current JS point on the trajectory of the Dash model. For another reasonable design, the observations can also come from the trajectory of the two models, which can be represented as:
[(xx,, te) 21: + nx(x, Xt, 1: bi 11-1,
t, E S
Z;
.) 4 $ S
In the experiments, we compare the two designs which utilize the different trajectory information on CIFAR-10 dataset with DDIM sampler. The results are shown in Fig. 12. It can be seen that our design (using trajectory information from the Dash model) performs better, which achieves an average speedup of 2.26x against to 2.00 In which case the number of steps extremely small (e.g., 3), using trajectory information from 4,-1 is better than that from t.. This is probably because the distance between t, and t. becomes relatively large when the number of steps very small, which makes the trajectory information less helpful for the Dot model. We also find that the Dot model also works well with the trajectory information from itself, though it is trained with the trajectory information from the Dash model during the training procedure.
Comparison under the Same Number of Steps. For evaluating the speedups of Morse under different sampling step budgets, we mostly compare the DMs with and without Morse under the selected latencies in the previous experiments. In Fig. 14, we provide the results of DMs under the selected number of steps, establishing a set of different time-interleaved configurations of Dot and Dash under a given LSD budget. It can be seen that the curves of DM with Morse are always below the curve without Morse, indicating the consistent acceleration ability of Morse under different numbers of sampling steps and proportion between the total numbers of sampling steps and the numbers of sampling steps with noise estimation from Dot.
Effect of Exchanged Steps Ratio. Recall that under a specific latency of n LSDs, there could be n (0 < sampling steps with Dash and Nk sampling steps with Dot for Morse. Morse can flexibly change the JS step length by controlling k. Here, we define the ratio of exchanged steps as k/n. In the experiments, we explore the effect of different
15

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Effect of Exchanged Steps Ratio
3.0
SLSDs
2.5
10 LSDs
Speedup
LSDs
20 LSDs
2.0
50 LSDs
100 LSDs
1.5
1.0
0.0
0.2
0.4 Ratio of Exchanged Steps
0.6
0.8
1.0
Figure Speedups Morse with DDIM sampler on CIFAR-10 (32x32) under different LSDs and exchanged steps ratios. The exchanged steps ratio denotes the ratio of the latency of steps with Dot to the total latency in generation process,
FID
CIFAR-10 (32x32)
FID
CelebA-HQ (256x256)
128
128
64
Baseline
64
32
+ Morse (10 Steps)
16
8
32
Morse (20 Steps)
OOOD
Morse (50 Steps)
8
D
16
+ Morse (100 Steps)
4
3
5
10 20
50
100
3 5
10
20
50 100
Runtime Latency (LSD)
Runtime Latency (LSD)
Figure 14. Results of Morse with DDIM sampler under different numbers of sampling steps. For diffusion process with Morse, we set 50%, 60%, 70%, 80% and 90% of the sampling steps for using the Dot model and the other steps using the Dash model.
ratios of the exchanged steps. We conduct the experiments on CIFAR-10 dataset with DDIM sampler. The results are shown in Fig. 13. Under most LSDs, Morse can achieve a speedup around 2x with most ratios. Under the extreme condition when we exchange most of the sampling steps with Dash for the sampling steps with Dot (e.g., more than 70%), the speedups sharply decrease below 1.0x. In our opinion, the reason that the trajectory information becomes less helpful for the Dot models residual estimation when the distance between the sampling steps is very large, since there much fewer sampling steps with Dash.
Different Designs for the Extra Down-Sampling and Up-Sampling Blocks of Dot. Recall that we construct the Dot model for Stable Diffusion by adding 2 trainable lightweight down-sampling blocks and up-sampling blocks to the Dash model. When training the Dot model, we fix the shared pre-trained layers and adopt lightweight LoRA. the experiments, we study the construction of Dot model with different designs for down- down-sampii sampling and up-sampling. We evaluate several variant designs for the Dot model including: (1) Down-sampling and up-sampling with proposed trainable blocks bilinear sampling; (2) Training the Dot model with or without LoRA. The shared pre-trained layers fixed. From the results shown significantly improve the performance of Morse. Without fine-tuning, the
in the Table 8, we can see that both designs original DM cannot adapt well to a lower resolution directly. While adding the trainable sampling blocks and adopting LoRA for training the Dot model can enhance learnable and soft resolution transformation and help the pre-trained blocks adapt to the new resolutions, respectively.
Different Architectures of the Dot Model. In the experiments, we evaluate the performance of Morse with the independent Dot model without sharing the pre-trained blocks with Dash model. We conduct the experiments on MS-COCO dataset with Stable Diffusion. In the variant design, the Dot model has similar architecture with the Dash model while with the reduced numbers of channels and blocks. The results are shown in Table 9. Even without fine-tuning with the pre-trained weights, our Morse can still accelerate the Stable Diffusson. While it needs more training iterations and trainable parameters to achieve similar performance with our proposed design. The results demonstrate the effectiveness and efficiency of our proposed weight sharing strategy for training a Dot model. While also shows that we can flexibly construct a Dot model with different architectures.
16

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
Table FIDs of Stable Diffusion with different variants of Dot. We calculate FIDs under different classifier-frce guidance scales and select the best FID among all the scales.
Method
Trainable Sampling Blocks
LoRA
LSDs
LSDs
20 LSDs
50 LSDs
Stable Diffusion
10.65
9,47
8.70
370.79 79
397.82
392.64
389.
9.23
8.94
8.60
8,36
Morse
9.79
9.21
8.89
8,51
8.60
8.55
8.29
8.15
Table Different architectures of the Dot model for Stable Diffusion.
Method
Training Iterations
Params
Average Speedup
Independent Dot model
0.4 million
324,93M
2.07x
Dul model with weight sharing stralegy
0.1 million
97.84M
2.29
A.6. More Generated Samples
We provide some generated samples from the diffusion models with and without Morse under different LSDs for better com- parisons, including image generation for CelebA-HQ x56x256) and LSUN- ChhhChh (256x256), text-to-image generation on MS-COCO with Stable Diffusion v1.4 and LCM-SDXL.
17

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
3LSDs
5LSDs
10LSDs
20
50 LSDS
HE
(a) DDIM
LSDs
LSDs
10 LSDs
LSDS
50 LSDs
Re
NE
(bb) DDIM with Morse
Figure 15. Generated samples resolution 256x 256 for CelebA-HQ dataset using DDIM sampler with and without Morse.
18

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
3LSDs
5LSDs
10
L0
LSDS
(a) DDIM
LSDs
LSSDs
10 LSDs
20 LSDs
50 LSDs
(bb) DDIM with Morse
Figure 16. Generated samples resolution 256x 256 for L-UN-Churd dataset using DDIM sampler with and without Morse.
19

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
LSDs
20 LSDs
50 LSDs
LSDS
20 LSDs
50 LSDs
boat that is floating a body of water.
dog laying on a pillow next to an open luggage bag
Artistic photograph of architecture and frost-laden trees at dawn
A green and yellow train riding down the train tracks.
A white covered in spray paint next to buildings.
Young girl combing the hair behind her head.
a brown teddy bear is sitting on green bed
(a) Stable Diffusion
(b) Stable Diffusion with Morse
Figure 17. Generated samples at resolution 512x512 with prompts from MS-COCO validation set from Stable Diffusion v1.4 using DDIM sampler with and without Morse. The classifier-free guidance scale set 7.5 following the official settings.
20

---

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models
L.SD
L.SDs
3 LSDs
1.33 LSDs
LSDs
3 LSDs
Closcup of'a brown bear sitting in grassy arca.
Birds perch bunch of twigs in the winter.
little girl standing by a table with some cakes on it.
A white cake topped with berries and plate of fruit and cheeses.
a dog holding a yellow disc its mouth
A young wearing white shirt and tie.
man riding board over the top of wave.
LCM-SDXL
(b) LCM-SDXL with Morse
Figure 18. Generated samples at resolution x024x 1024 with prompts from MS-COCO validation set from LCM-SDXI. with and without Morse.
21


Reviews:
[{"review_id": "XNRdmtAQny", "reviewer": "Reviewer_vMF4", "summary": "The paper introduces Morse, a framework for accelerating diffusion model sampling by training a lightweight model (Dot) to emulate a slower pretrained diffusion model (Dash). Dot leverages additional inputs, such as sampling history from earlier timesteps, to improve efficiency. Its architecture mirrors the original diffusion model but incorporates LoRA and additional downsampling/upsampling layers at the beginning and end. Only the newly added layers are fine-tuned, while the original layers operate at a reduced resolution, significantly lowering latency. By combining fast Dot models with slower Dash models during inference, Morse achieves a better quality-speed tradeoff, with 1.5–3.5× speedups depending on the setting. The authors evaluate Morse across various image generation benchmarks, demonstrating considerable improvements in efficiency without little degradation to output quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work and constructive comments.\n\n**1. To your main concern** about the comparison of Morse with existing methods for accelerating diffusion inference, **our responses include 3 parts**:\n\n**Part 1: Comparison with Feature Reuse.** Following the suggestion by you and Reviewer cEcd, we compare Morse with DeepCache (its code is public): **1)** Table A shows their results on Stable Diffusion tested with the same settings as in our paper. We can see **a)** for acceleration, DeepCache is lossy in generation quality due to reusing most of the features at step $t$ for N-1 following denoising steps, yet Morse is lossless; **b)** Morse consistently gets better throughput (e.g., 4.67 vs 2.33 images/second) and FID (8.60 vs 10.62) than DeepCache; **2)** At [this linker](https://anonymous.4open.science/r/Morse-Reviewer-vMF4), we provide more results to show the superiority of Morse.\n\n**Table A**: Results on MS-COCO(512×512) with Stable Diffusion (SD) v1.4 and DDIM solver\n|Method|FID↓|Throughput↑(images/second on a 4090 GPU)|\n|-|-|-|\n|SD (50 steps)|8.22|1.07|\n|+DeepCache (N=2, N: the cache updating at every N steps)|9.31|1.62|\n|+DeepCache (N=3)|9.55|1.94|\n|+DeepCache (N=5)|10.62|2.33|\n|+Morse (LSD=50, LSD: the Latency per Step of the baseline SD)|**8.15**|1.12|\n|SD (20 steps)|8.70|2.45|\n|+Morse (LSD=20)|8.29|2.43|\n|SD (10 steps)|10.65|4.62|\n|+Morse (LSD=10)|8.60|**4.67**|\n\n**Part 2: Comparison with Timestep Schedule Optimization.** Thanks for pointing out [1-4] that are closely related to our work. Generally, [2] presents a fast solver GGDM (a contemporary work is DPM-Solver, 1 of 5 samplers tested in our paper), and [1,3,4]  design different strategies to choose optimal time steps when giving a small number of sampling steps (e.g., 10) and a solver, and only the code of [3] is public. As [1] uses a log likelihoods metric instead of popular metrics like FID for evaluation, comparing Morse with it is not applicable. For fair comparisons with [2-4], we collect their results and ours under the same settings: **1)** Table B&C compare Morse with [2]. We can see, under the same steps-budget, **a)** [2] is mostly worse than DPM-Solver, but Morse can losslessly accelerate DPM-Solver; **b)** [2] is better than DDIM, but DDIM+Morse outperforms [2]; **2)** Table D&E compare Morse with [3-4]. We can see, when using the same solver and steps-budget, **a)** Morse is superior to [3-4]; **b)** as jump sampling is our basic component, Morse can readily accelerate this line of works (as validated on [3] in Table D).         \n\n**Table B**: Results on CIFAR-10(32x32)\n|Method\\LSDs|5|10|15|20|25|\n|-|-|-|-|-|-|\n|[2]|13.77|8.23|6.12|4.72|4.25|\n|DPM-Solver|268.14|6.26|4.13|3.66|3.50|\n|DPM-Solver+Morse|**13.39**|**3.93**|**3.45**|**3.45**|**3.44**|\n\n**Table C**: Results on ImageNet(64x64)\n|Method\\LSDs|5|10|15|20|25|\n|-|-|-|-|-|-|\n|[2]|55.1|37.3|24.7|20.7|18.4|\n|DDIM|147.4|39.4|28.7|22.2|20.0|\n|DDIM+Morse|**46.6**|**25.8**|**21.5**|**19.2**|**18.1**|\n\n**Table D**: Results on CIFAR-10(32x32)\n|Method\\LSDs|5|10|12|15|\n|-|-|-|-|-|\n|DPM-Solver++(SDE)|29.22|4.03|3.45|3.17|\n|+[3]|12.91|3.51|3.24|3.15|\n|+Morse|5.73|2.91|2.80|2.75|\n|+[3] and Morse|**4.77**|**2.89**|**2.79**|**2.74**|\n|DPM-Solver(SDE)|191.46|4.72|3.83|3.77|\n|+Morse|8.65|3.41|3.22|3.06|\n\n**Table E**: Results on ImageNet(64x64)\n|Method\\LSDs|5|10|15|20|25|\n|-|-|-|-|-|-|\n|DDIM|145.0|42.5|30.3|26.6|24.8|\n|+[4]|50.4|29.2|24.2|22.3|21.4|\n|+Morse|**46.6**|**25.8**|**21.5**|**19.2**|**18.1**|\n\n**Part 3: Comparison with Quantization.** In Table F, we compare Morse with Q-Diffusion (a leading open-source diffusion quantization method). We can see: **a)** Q-Diffusion is lossy due to the low-precision representation, but our Morse is lossless; **b)** Morse can also accelerate the quantized diffusion model, showing an average speedup of 1.9×.\n\n**Table F**: Results on CIFAR-10 with DDIM\n|Method\\LSDs|5|10|20|50|100|\n|-|-|-|-|-|-|\n|Baseline|41.9|13.7|7.0|4.8|4.3|\n|+Morse|**13.6**|**6.6**|**5.1**|**4.4**|**4.1**|\n|Q-Diffusion (4-bit weight, 8-bit activation)|43.7|14.2|7.6|5.6|5.1|\n|+Morse|14.6|8.0|6.5|5.3|4.8|\n\n**2. To your concern** about a discussion of tradeoffs in terms of method efficiency and usability,\n\nW.r.t. the above experiments, it can be concluded that: **1)** Morse is lossless thanks to its simple dual-model design with a small number of learnable parameters. Training-free methods such as feature reuse and quantization are lossy, but Morse can improve them; **2)** As jump sampling (JS) is our basic component (note Morse cannot speedup 1-step model as JS is not applicable), Morse can accelerate timestep schedule optimization methods[1-4], similar to the solvers/distillation-based methods tested in our paper; **3)** So, in practice, combing Morse with them can provide a promising way to attain more aggressive speedup ratios under bearable degeneration of image quality.\n\n**3. Regarding more experiments and discussions**, please see our responses to Reviewer cEcd/dKeM.", "claims_and_evidence": "Yes, the claims made in the paper are well substantiated. And sufficient ablation studies are performed that showcase the impact of each design decision.", "methods_and_evaluation_criteria": "The method is very simple and easy to understand. The benchmarking suite includes a good amount of baseline models and datasets, and the evaluation criteria make sense.", "theoretical_claims": "No major theoretical claims were made. This papers contributions is primarily an efficiency technique applicable to diffusion models.", "experimental_designs_or_analyses": "The experiments are generally sound. However, since this paper proposes an efficiency technique for accelerating diffusion inference, it would have been preferable to compare against other established acceleration methods, such as feature reuse, timestep schedule optimization, and quantization. Some of these approaches are training-free and do not require an additional model, making a comparison particularly relevant. A discussion of how the proposed method compares in terms of tradeoffs—such as speed, memory efficiency, and ease of adoption—would strengthen the evaluation.", "relation_to_broader_scientific_literature": "The slow sampling speed of diffusion- and flow-based generative models is a well-known limitation, and a significant amount of research has focused on addressing this issue.\n\nFor few-step sampling (1–4 steps), distillation-based approaches are essential for achieving acceptable quality. However, these methods require training an additional student model, which comes with substantial computational overhead.\n\nFor slightly higher NFE regimes (10–30 steps), a variety of techniques have been proposed, including improved ODE/SDE solvers, optimized sampling schedules, feature reuse across timesteps and model layers, model quantization, etc. The approach in this paper falls within this setting—while it improves sampling in the few-step regime, the sample quality still degrades significantly at very low NFE.\nAmong these methods, training-free approaches tend to be more impactful due to their ease of adoption, as they do not require additional model training. Comparing this method against such alternatives would help clarify its tradeoffs in terms of efficiency and practical usability.", "essential_references_not_discussed": "Among the relevant prior works related to accelerating diffusion model sampling, the line of work regarding optimizing timestep schedules seems to be missing, e.g. [1, 2, 3, 4].  \n\n[1] Watson, Daniel, et al. \"Learning to efficiently sample from diffusion probabilistic models.\" arXiv preprint arXiv:2106.03802 (2021).\n[2] Watson, Daniel, et al. \"Learning fast samplers for diffusion models by differentiating through sample quality.\" International Conference on Learning Representations. 2021.\n[3] Xue, Shuchen, et al. \"Accelerating diffusion sampling with optimized time steps.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[4] Sabour, Amirmojtaba, Sanja Fidler, and Karsten Kreis. \"Align your steps: Optimizing sampling schedules in diffusion models.\" arXiv preprint arXiv:2404.14507 (2024).", "other_strengths_and_weaknesses": "**Strengths**:\n* The paper is well written and nicely structured. \n* The proposed method is simple and easy to understand. \n\n**Weaknesses**:\n* No additional efficient diffusion model baselines were compared against. Adding comparisons to prominent work in the field, such as DeepCache, and discussing the pros/cons of such approaches compared to Morse would be helpful. (Addressed in the rebuttal)", "other_comments_or_suggestions": "No comments.", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AUjBPxf2EF", "reviewer": "Reviewer_dKeM", "summary": "This paper proposes a new faster sampling method for diffusion models. The goal of the paper is to provide a faster sampling method, not sacrificing performance (unlike many recent distillation-based methods). The main idea is to use another (the \"dot\") model in the sampling process. This additional sampler \"corrects\" the jump sampling of the main model, and is relatively quick and requires less training effort (downsizing a pretrained model and adding minimal extra layers, trained by LoRA). This enables more runs in the same time period, which greatly enhances the sampling efficiency. Accordingly, the focus boils down to reducing the entire time cost rather than minimizing the number of steps. Experiments show that the proposed method can enhance the performance of existing models and provide a two- to four-times speed up.\n\n## update after rebuttal\n\nSome reviewers pointed out that using an additional model can be a burden, and it might not be as favorable as other competing approaches. However, I still think that the two-denoiser approach is interesting and novel, and the additional results in the rebuttal sufficiently eliminate the doubts. I maintain my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the constructive comments, and the recognition of our work including the proposed method, the experiments and the performance. Please see our below responses to your concerns and questions one by one.\n\n**1 To your concern about** “However, one complaint I have is that there is no direct comparison to distillation-based methods…comparing the performance/speed to these will provide a better picture.”\n\n**Our responses are:** Thanks for your valuable suggestion. We agree that a direct comparison of Morse with distillation-based methods would improve our work further. \nIn Section 3.4 of our paper, we have shown that Morse can be combined with consistency distillation, which achieves an average speedup 1.43× for Latent Consistency Model on MS-COCO (see Table 3).\n\nHere, we further provide a direct comparison with **BK-SDM** [1], a state-of-the-art distillation method that constructs lightweight diffusion models through architectural compression and feature distillation. As shown in Table A, when comparing Stable Diffusion+Morse (10 LSDs) with BK-SDM-base|-small|-tiny models (25 steps), Morse achieves better FID scores (8.60 vs 13.35|14.17|15.60) while maintaining higher throughput (4.67 vs 2.79|2.84|2.92), demonstrating the superiority of Morse to BK-SDM.\n\nAdditionally, following the suggestions from Reviewer vMF4/cEcd, we provide further comparisons with other acceleration paradigms (e.g., feature reuse, sampling schedule optimization, quantization). For instance, our Morse outperforms DeepCache [2], a state-of-the-art feature reuse method, on MS-COCO with Stable Diffusion. The results are presented in Table B below. More results and discussions can be found in our responses to Reviewer vMF4/cEcd.\n\n**Table A**: Comparison of Morse and BK-SDM on MS-COCO with DDIM solver. The results are measured with the same settings as in our paper. Throughput is measured on a single RTX 4090 GPU. BK-SDM-base, BK-SDM-small and BK-SDM-tiny are student networks with different architectures distilled by Stable Diffusion as the teacher network. LSD denotes the Latency per Step of the baseline Stable Diffusion.\n\n|Method|FID↓|Throughput↑ (images/second)|\n|-|:-:|:-:|\n|BK-SDM-base (25 steps)|13.35|2.79|\n|BK-SDM-small (25 steps)|14.17|2.84|\n|BK-SDM-tiny (25 steps)|15.60|2.92|\n|Stable Diffusion (25 LSDs) |8.41|1.95|\n|+ Morse (25 LSDs)|**8.16**|1.97|\n|Stable Diffusion (20 LSDs)|8.70|2.45|\n|+ Morse (20 LSDs)|8.29|2.43|\n|Stable Diffusion (10 LSDs)|10.65|4.62|\n|+ Morse (10 LSDs)|8.60|**4.67**|\n\n**Table B**: Results of Morse and DeepCache on MS-COCO with Stable Diffusion v1.4 and DDIM solver. N denotes the cached feature maps of DeepCache is updated at every N steps.\n\n|Method|FID↓|Throughput↑ (images/second)|\n|-|:-:|:-:|\n|Baseline (50 steps)|8.22|1.07|\n|Baseline (20 steps)|8.70|2.45|\n|Baseline (10 steps)|10.65|4.62|\n|DeepCache (50 steps, N=2)|9.31|1.62|\n|DeepCache (50 steps, N=3)|9.55|1.94|\n|DeepCache (50 steps, N=5)|10.62|2.33|\n|Morse (50 LSDs)|**8.15**|1.12|\n|Morse (20 LSDs)|8.29|2.43|\n|Morse (10 LSDs)|8.60|**4.67**|\n\n[1] Kim, Bo-Kyeong, et al. \"Bk-sdm: A lightweight, fast, and cheap version of stable diffusion.\" ECCV, 2024.\n\n[2] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. \"Deepcache: Accelerating diffusion models for free.\" In CVPR, 2024.\n\n**2. To your question about** “In (5), the \"dot\" model receives five inputs, three of which are images… Are they concatenated, and then is the resulting channel reduced by the first (extra) layer?…”\n\n**Our responses are:**  As shown in Figure 2 of our paper, the Dot model receives five inputs, including three image features ($x_{t_{i}}$, $x_{t_{s}}$ and $z_{t_{s}}$) and two scalars ($t_{i}$ and $t_{s}$). As you noted, the three image features are concatenated before being fed into the first extra layer. The two scalars ($t_{s}$ and $t_{i}$), corresponding to the time steps of Dash and Dot respectively, are individually embedded and then concatenated to form the time embedding vectors. Thanks for pointing this out, we will add more descriptive details to clarify it in the revised manuscript.\n\n**3. To your question about** “Moreover, the explanation about the structure of the dot model itself is somewhat confusing. It says that the pretrained model is fixed, two extra layers are added, and then LoRA is applied? This was somewhat confusing, and after seeing Fig. 3, I guessed that LoRA is applied to the pretrained part. Articulating the description would benefit the readers”.\n\n**Our responses are:** Thanks a lot for your suggestion. Yes, for the Dot model, LoRA is only applied to the pre-trained layers shared from the Dash model (fixed). The newly added two layers and LoRA are trained jointly. We will revise the corresponding description about the structure to improve its clarity.", "claims_and_evidence": "Using another (lightweight) sampler to improve the sampling process in diffusion models is quite new and interesting. Recent one-step generators are somewhat limited in performance, but this method can be a viable alternative without sacrificing performance. The experimental results support this claim.", "methods_and_evaluation_criteria": "The design of the \"dot\" model is quite simple and efficient. The proposed method is evaluated on Stable Diffusion and LCM-SDXL, which is good enough in my opinion.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The paper provides extensive ablation (and hyperparameter tuning) experiments. The results are generally convincing.\n\nHowever, one complaint I have is that there is no direct comparison to distillation-based methods. I do understand that this paper's focus is somewhat different from these, but as the paper criticizes their shortcomings, comparing the performance/speed to these will provide a better picture.", "relation_to_broader_scientific_literature": "Diffusion models have become a significant part of generative models, so providing a faster method without sacrificing performance can benefit many related areas. Moreover, the main concept (using another auxiliary sampler to correct jump sampling) is quite novel and interesting.", "essential_references_not_discussed": "The bibliography is thorough enough.", "other_strengths_and_weaknesses": "Please see the above points.", "other_comments_or_suggestions": "As mentioned above, comparing the performance/speed to the distillation-based method will benefit the readers.", "questions_for_authors": "- In (5), the \"dot\" model receives five inputs, three of which are images. This model is a modified version of a pretrained diffusion model, so how are these inputs combined? Are they concatenated, and then is the resulting channel reduced by the first (extra) layer? This is not really explained in the paper or the supplementary material.\n\n- Moreover, the explanation about the structure of the dot model itself is somewhat confusing. It says that the pretrained model is fixed, two extra layers are added, and then LoRA is applied? This was somewhat confusing, and after seeing Fig. 3, I guessed that LoRA is applied to the pretrained part. Articulating the description would benefit the readers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbkTFGQ2wi", "reviewer": "Reviewer_cEcd", "summary": "This work proposes a method to speed up diffusion models by skipping steps (Dash) and by compensating errors for the Dash (Dot). The Dash model uses a pre-trained model with mere skipping steps (i.e., existing diffusion models such as Stable Diffusion, but using larger, less steps) while the Dot model must be trained to provide adaptive residual feedback. LoRA structure was used for the Dot model along with the weight sharing with the Dash model for efficient tuning. This work was tested with Stable Diffusion and LCM-SDXL, showing 1.78x to 3.31x speed ups with comparable performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your detailed comments and the recognition of our method.\n\n**1. To your main concern** about comparisons of Morse with recent related methods, **our responses include 4 parts**:\n\n**Part 1: Comparison with DeepCache and PFDiff.** In formulation, they and our Morse explore the temporal step redundancies for accelerating diffusion models in different ways: **1)** DeepCache (DC, for brevity) reuses the features at step t for N-1 following steps, and PFDiff utilizes the states of past steps stored in a buffer to update the states of future steps, and they are training-free; **2)** Morse is a simple dual-model design relying on step skipping and residual feedback with a small number of learnable parameters. In performance: **1)** Table A compares Morse with open-source DC on CIFAR-10 under the same settings. We can see: **a)** DC is lossy in output quality while Morse is lossless; **b)** Morse performs better in terms of both FID and Throughput; **2)** In Part A of our responses to Reviewer vMF4, we provide more results on Stable Diffusion (SD), further showing the superiority of Morse to DC; **3)** Table B compares Morse with PFDiff (code is public) on SD, showing Morse is better than PFDiff.\n\n**Table A**: Results on CIFAR-10. **DC denotes DeepCache**\n|Method|FID↓|Throughput↑(img/second on a 4090 GPU)|\n|-|-|-|\n|DDIM(100 steps)|4.3|14.4|\n|+DC(N=2, N: the cache updating at every N steps)|4.4|19.1|\n|+DC(N=3)|4.8|22.2|\n|+DC(N=5)|5.7|23.7|\n|+DC(N=10)|10.0|27.2|\n|+Morse(LSD=100)|**4.1**|14.4|\n|DDIM(50 steps)|4.8|29.1|\n|+Morse(LSD=50)|4.4|28.8|\n|DDIM(20 steps)|7.0|71.7|\n|+Morse(LSD=20)|5.1|**73.9**|\n\n**Table B**: Results on MS-COCO. Evaluated on 10k samples.\n\n|Method\\LSDs|5|10|15|20|\n|-|-|-|-|-|\n|SD|23.9|16.8|16.1|16.0|\n|+PFDiff|18.3|13.1|13.6|14.0|\n|+Morse|**13.9**|**11.7**|**12.2**|**13.6**|\n\n**Part 2: Comparison with Step Skipping Methods.** Thanks for pointing out three works in this line: **1)** EMDM is a GAN-based diffusion learning method tailored for video-based human motion generation, and AdaDiff is an RL-based diffusion learning method tailored for text-to-image generation in a prompt-difficulty-aware manner. Our Morse addresses a variety of image generation tasks (see Sec.3), thus it differs with them in focus and formulation (see Part 1); **2)** AMED (Zhou et al., CVPR 2024) is an improved solver using a 2-MLP network for each steps-budget to learn the mean direction for fast sampling. Table C compares Morse with AMED on CIFAR-10, showing Morse is better; **3)** In Part B of our responses to Reviewer vMF4, we provide more experiments, showing Morse is also better than other step skipping methods.\n  \n**Table C**: Results on CIFAR-10\n|Method\\LSDs|3|5|7|9|\n|-|-|-|-|-|\n|DPM-Solver++|110.0|24.97|6.74|3.42|\n|+AMED|25.95|7.68|4.51|3.03|\n|+Morse|**14.36**|**5.73**|**3.31**|**2.98**|\n\n**Part 3: Comparison with One-step Diffusion Models.** They aim to get the extreme diffusion speedup, yet lead to serious degeneration of output quality compared to full-step teachers. Methods based on step skipping (e.g., Morse & AMED) and past-step feature reuse (e.g., DeepCache & PFDiff) cannot accelerate one-step diffusion models that have no temporal redundancy, which is a common limitation. To avoid an over-claim, \"universally\" will be removed in our paper.\n\n**Part 4:** Morse is a leading lossless method that needs to train only once given a pre-trained diffusion model. \"190 A100 hours\" is for a billion-level SD model (its pre-training cost is 1000x longer, see Table 2). For more comparisons, please check [this link](https://anonymous.4open.science/r/Morse-Reviewer-cEcd) and our responses to Reviewer vMF4/dKeM.\n\n**2. To Q1** regarding Fig.8, the FID improvements are decent to the highest CLIP score cases, but are really small to the lowest cases, as illustrated in Table D below. This is due to the random sampling strategy of guidance scales (GS) from 2 to 10 when training the Dot model (see Line580-585). The lowest CLIP scores appear at GS=2 occupying a small portion of samples, making it hard to improve FIDs.\n\n**Table D:** Results (FID|CLIP score) on MS-COCO with SD\n|Method\\LSDs|10|20|50|\n|-|-|-|-|\n|SD(GS=2)|18.8\\|25.4|11.1\\|26.5|9.0\\|27.0|\n|+Morse|18.7\\|25.4|11.1\\|26.5|9.0\\|27.0|\n|SD(GS=7.5)|11.8\\|29.7|12.4\\|30.1|13.5\\|30.0|\n|+Morse|9.3\\|29.8|11.3\\|30.1|13.3\\|30.0|\n\n**3. To  Q2** regarding what is the unit of LSD, it is the Latency per Step of the baseline Diffusion model, i.e., the one-step time cost of the Dash model (see Line244-253). Recall that, in our Morse, Dot is N times faster than Dash, so a diffusion process with n-k Dash steps (skipping k steps totally) and Nk Dot steps has the same latency (n LSDs) to the baseline running n steps. This is why, given n LSDs, the runtime latencies of the diffusion model with and w/o Morse are at the same point in Fig.4-5. The runtime speed under different settings of n LSDs is shown in Table 5 of our paper.\n\n**4. To  Q3**, the answer is no, as clarified in Part 3.", "claims_and_evidence": "While there were some unclear aspects, this manuscript seems to provide partial evidence for their claims such as speed ups (1.78x - 3.31x), thus demonstrating the effectiveness of the proposed method. However, these claims were not well-compared with other recent works, so it is not straightforward to see if this proposed method is indeed a state-of-the-art. In fact, it is unclear if the proposed method is indeed \"fast\" as claimed over the entire manuscript since there are a number of missing comparisons and a number of new works on one-step diffusion models. Moreover, \"Universally\" in the title seems an over-claim since the proposed architecture of the Dot model was not tested universally (enough) for other recent diffusion models.", "methods_and_evaluation_criteria": "The methods make sense to me. However, Table 2 looks like somewhat unfair comparisons since the Stable Diffusion is the pre-trained model and this work is the tuning model. There are a number of recent works that do not even require tuning such as Ma et al. CVPR 2024 (cited in this manuscript) and Wang et al. PFDIff, ICLR 2025 (available online since last year), so this table seems to deliver a biased message without properly discussing with other related works.", "theoretical_claims": "There is no theoretical claim in this manuscript as far as I know.", "experimental_designs_or_analyses": "Experimental designs or analyses were good except for the lack of comparisons with other similar baselines such as DeepCache (or PFDiff). While they look different from the proposed method, they were also using the temporal redundancies of diffusion steps in different ways (and without tuning unlike the proposed method, which seems important), so it is important to clearly show the advantage of the proposed method over related prior arts using the same underlying principles.", "relation_to_broader_scientific_literature": "Step skipping strategies while exploiting temporal redundancies in diffusion steps have been discussed and developed such as DeepCache or PFDiff, but this work unfortunately did not properly address and compare with them in my view. Moreover, there are a number of recent works on one step diffusion models, so the insight in this work may become no so useful anymore quickly as the major diffusion models vendors will adopt this one step approaches. Full discussion on these other works and potential limitation seem important to justify the key contributions of this work.\n\nHere are some examples for step skipping methods in diffusion models:\n- Wenyang Zhou et al., EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generation, ECCV 2024.\n- Hui Zhang et al., AdaDiff: Adaptive Step Selection for Fast Diffusion Models, AAAI 2025.\n- Zhenyu Zhou et al., Fast ODE-based Sampling for Diffusion Models in Around 5 Steps, CVPR 2024.\n\nIt is especially important to see Zhou et al. CVPR 2024 since this work is about using large step sizes with the aid of corrections.", "essential_references_not_discussed": "As mentioned above, the following papers and other recent works should be properly discussed and compared:\n- X Ma et al., Deepcache: Accelerating Diffusion Models for Free, CVPR 2024 (cited, but not discussed)\n- G Wang et al., PFDiff: Training-free Acceleration of Diffusion Models Combining Past and Future Scores, ICLR 2025 (not cited)", "other_strengths_and_weaknesses": "While this work seems to propose a clear method for accelerating diffusion models, there are a number of weaknesses as below:\n- Requiring additional training for speed up does not look attractive over other training-free methods (PFDiff, Deepcache) or one-step diffusion models. For a new model, this work requires an additional training (190 A100 hours may not be much as compared to the original model training, but can be cumbersome when a new model is given multiple times). It is also unclear how this work can work with personalization of diffusion models or concept erasing of diffusion models. \n- As mentioned above, this work seems to demonstrate its effectiveness in Stable Diffusion Models only, but there are a number of other diffusion models that are available online. It seems important to demonstrate for some of them, especially to justify the Dot method.", "other_comments_or_suggestions": "In Figure 1 caption, \"Latency Consistency Models\" should be \"Latent Consistency Models\"", "questions_for_authors": "In Figure 8, it was interesting to see that the major FID improvements were done around the cases with CLIP scores near 28-29, while the improvements were almost none for the lowest CLIP score cases (around 25) and the highest CLIP score cases (around 30). Any insight for them?\n\nWhat was the unit of LSD? In Figures 4-5, there were large FID gaps around 3-5 LSDs for the baseline and the proposed method, but their runtime latencies were at the same point - in terms of absolute computational time scales, they should not be exactly matching due to additional computation of the Dot model, right? Any explanation for them? Actually, these results were also my major concerns since even 3-5 LSD with Morse did not achieve great results as compared to consistency models with around 4 steps (not compared in here) or more recent one-step models.\n\nWill this work (and other recent works such as Deepcache) be useful for one-step diffusion models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a fast sampling approach for diffusion models. The idea is to train a lightweight model, called Dot, that is finetuned to take the sample history as input and improve the slow model's output. Reviewers acknowledge that the idea is novel, and results are sound (after rebuttal). Although there are more to be desired in terms of its simplicity, the AC believes that it is a reasonable contribution to the diffusion modeling field and would lean towards accepting.
Decision: Accept (poster)