Linear Multistep Solver Distillation for Fast Sampling of Diffusion Models
Authors: Yuchen Liang
Xiangzhong Fang
Hanting Chen
Yunhe Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Probabilistic Model
Diffusion Sampler
Solver Schedule
Primary area: generative models
TL;DR: We provide a solver distillation  framework for diffusion models and search for solver schedules based on it.

Abstract: Sampling from diffusion models can be seen as solving the corresponding 
   probability flow ordinary differential equation (ODE). 
   The solving process requires a significant number of function 
   evaluations (NFE), making it time-consuming. 
   Recently, several solver search frameworks have attempted to find 
   better-performing model-specific solvers. However, predicting the impact of 
   intermediate solving strategies on final sample quality remains challenging, 
   rendering the search process inefficient.
   In this paper, we propose a novel method for designing 
   solving strategies. We first introduce a unified prediction formula 
   for linear multistep solvers. Subsequently, we present a solver distillation 
   framework, which enables a student solver to mimic the sampling trajectory 
   generated by a teacher solver with more steps. We utilize the mean Euclidean 
   distance between the student and teacher sampling trajectories as a metric, 
   facilitating rapid adjustment and optimization of intermediate solving strategies.
   The design space of our framework encompasses multiple aspects, 
   including prediction coefficients, time step schedules, and time scaling 
   factors. 
   Our framework has the ability to complete a solver search 
   for Stable-Diffusion in under 12 total GPU hours.
   Compared to previous reinforcement learning-based 
   search frameworks, 
   our approach achieves over a 10$\times$ increase in search efficiency. 
   With just 5 NFE, we achieve FID scores of 3.23 on CIFAR10, 7.16 on ImageNet-64, 
   5.44 on LSUN-Bedroom, and 12.52 on MS-COCO, resulting in a 2$\times$ sampling acceleration ratio 
   compared to handcrafted solvers.

Published as a conference paper at ICLR 2025
LINEAR MULTISTEP SOLVER DISTILLATION FOR FAST SAMPLING OF DIFFUSION MODELS
Yuchen Liang, Xiangzhong Fang School Mathematical Sciences Peking University
Hanting Chen, Yunhe Wang
Huawei Noah's Ark Lab
ireehhanting@huaaee com
ycliangepku.edie edw....eedu.oom
ABSTRACT
Sampling from diffusion models can be seen as solving the corresponding prob- ability flow ordinary differential equation (ODE). The solving process requires a significant number of function evaluations (NFE), making it time-consuming. Re- cently, several solver search frameworks have attempted to find better-performing model-specific solvers. However, predicting the impact of intermediate solving strategies final sample quality remains challenging, rendering the search pro- cess inefficient. In this paper, we propose a novel method for designing solving strategies. We first introduce unified prediction formula for linear multistep solvers. Subsequently, we present a solver distillation framework, which enables student solver to mimic the sampling trajectory generated by a teacher solver with more steps. We utilize the mean Euclidean distance between the student and teacher sampling trajectories as metric, facilitating rapid adjustment and opti- mization of intermediate solving strategies. The design space of our framework encompasses multiple aspects, including prediction coefficients, time step sched- ules, and time scaling factors. Our framework has the ability to complete solver search for Stable-Diffusion under 12 total GPU hours. Compared previous reinforcement lcarning-bei -baed search frameworks, 10x increase in search efficiency. With just 5 NFE, we achieve FID scores 3.23 CIFAR1O, 7.16 on ImageNet-64, 5.44 on LSUN-Bedroom, and 12.52 MS-COCO, resulting in a 2x sampling acceleration ratio compared to handcrafted solvers.
approach achieves over
INTRODUCTION
Diffusion models -Sckt-Dickstein et al.) 2015 Ho et al. 2020; Song et al. 2021b) have gained
widespread success in various applications including image generation (Dhariwal & Nichol 2021 Rombach et al.] (2022), audio synthesis (Kong et al. 2021 Chen et al. 2021). video generation (Ho)
al. 2022h a lattmann al. 2023), and text-to-image synthesis (Shariia et al. 2022 Ruiz al.;
2023 Podell et al.) 2024 Esser et al. 2024). When generating samples, diffusion models perform reverse solving of predefined Stochastic Differential Equation (SDE) or its corresponding Proba- bility Flow Ordinary Differential Equation (ODE) (Song et al. 2021bb. This solving process often required hundreds of function evaluations (NFE), making it extremely time-consuming compared to classical generative models like Generative Adversarial Networks (GANs) (Goodfellow et al.,
2014).
Fortunately. significant advancements have been made in accelerating the sampling process of dif- fusion models. Existing acceleration methods can be broadly categorized into two classes. The first class of methods involves an additional distillation training phase (Luhman & Luhman [021]: Sali- mans & Ho 2022, Song et al. 2023 Sauer et al., 2023 Luo et al. 2024). Distilled models require only 1-4 NFE to generate high-quality samples. However, the distillation phase typically requires thousands GPU hours, posing a significant training cost, especially for large scale models. Addi- tionally, many of distilled models lack the ability to perform image processing tasks, such image editing and restoration Yuu et al.) 2023 He et al.) 2023) Wang et al. 2022] Chung et al. 2023 Zhu
et al. 2023
We will provide
detailed discussion in Sec. 2.2

---

Published as a conference paper at ICLR 2025
NFE=S
NFE=10
DEIS
+ Lu et al. 20226
UniPC
DLMS Ours
Zhang Chen 2022
Zhao et al. 2023
(Rombach et al. 2022) with default classifier- free guidance scale 7.5 and text prompt "A large brown bear walking through a forest". Our proposed DLMS can generate more realistic and visually detailed images compared with previous handcrafted samplers (Zhang & Chen 2022 Lu et al. 2022b: Zhao et al. 2023).
Figure I: Synthesized images of
And the second class, which is also widely regarded, focuses designing efficient solvers without further training the model. Techniques such as parameterization, exponential integrators, and higher- order solvers have successfully reduced the NFE to 15-20 for the sampling process (Lu et al. 2022b
Zhao et al. 2023 Zhang & Chen 2022 Liu et al. 2022 Lu et al., 2022a Song et al. 202 la Karras
al. 2022 However, when the NFE iS less than 10, the sample quality deteriorates significantly.
The challenges have given rise to a series of search frameworks seeking more efficient solvers. However, due the iterative nature of solving processes, it is difficult to predict the impact of intermediate solving strategies on the final sample quality, making the search process difficult and inefficient. These frameworks still often require tens of GPU hours and were limited to specific design space such as time steps (Watson et al. 2022; Li et al. [2023] Liu et al. [023a Sabour et al., 2024 Chen et al., 2024), parameterization mode (Zheng al. (023). ) solver combination (Liu et al. 2023b). Recently, some work (Shaul et al. 2024 Zhang et al. 2024) has attempted to use solver distillation to design solvers, but the performance improvements have been relatively limited.
In this paper, we propose novel solver distillation framework that greatly accelerates the search efficiency. We begin by presenting unified prediction formula for linear multistep solvers. Next, we introduce our solver distillation algorithm, which allows student solver to replicate the sam- pling trajectory of a teacher solver that uses more steps. We employ the mean Euclidean distance between the student and teacher sampling trajectories optimization of intermediate solving strategies. Compared to previous reinforcement learning-bast based search frameworks, approach achieves
metrie, enabling quick adjustments and
10x increase in search efficiency.
Our framework has the ability to complete a solver distillation for Mtale-DDiffusiee (Rombach et al.) 2022 in less than 1.5h 8 NVIDIA V100 GPUs. The design scope includes the time steps, the time scaling factors and the prediction coefficients. We extensively evaluate our approach on various resolution datasets in both pixel space and latent space. The Distilled Linear Multistep Solver (DLMS) significantly surpasses previous handcrafted and search-based solvers. Compared handcrafted solvers, DLMS achieves 2x sampling acceleration ratio.
BACKGROUND AND RELATED WORK
2.1 DIFFUSION MODELS
Given the data distribution Po. diffusion models employ forward process 0x:00 + ozE, [0, 7] with marginal distribution {pe) gradually degenerate the data Xy Pn with Gaussian
2

---

Published as a conference paper at ICLR 2025
noise I). The noise schedule at, di 0 is designed to make pr approximately a pure Gaussian distribution (O, I). Notably, there exists corresponding Probability Flow ordinary differential equation (PF-ODE) (Song et al. 2021b):
dx, = [(1)x 20 2 (iVlogpp(x)) dl,
where f(t) diog 01 (²) do? 2 dog of Kingma al. 2021). The PF-ODE shares the
dt
same marginal distribution as the forward process. And the score function V log (x,,) can be expressed using Tweedic's s formula (Robbins
1992):
0, [o) x1] X
E
V
(2)
of
To estimate the score function, neural network (x(x,, () is trained predict [[([Xx]] via the least square estimation by minizing the L2 loss
Ex e. ((0.r) (x., () - <
(3)
for each E [0, Consequently, we can sample from diffusion models by solving the empirical PF-ODE:
dx, [10)++
g²(()
ea (x, 1)
20,
from time 7 to time O. Additionally, the conditional sampling can carried out by guided sampling (Dhariwal & Nichol 2021; Ho & Salimans [202]). Remarkably, classifier-free guidance (Ho &
Salimans 2021) defines a guided noise predictor:
Eo(x,, () = co (x,,,,,,) (1) cx(x,, 1,
where c is the condition, stand for the unconditional sampling, and >> 0 is the guidance scale,
In practice, except for the noise predictor eø (x,, t), diffusion models can also be parameterized as data predictor an. (x,, f) predict Xo velocity predictor (x,,,,t) predict 0&e 0xXo, where the parameterizations are theoretically equivalent, but have impact in practice performance (Karras)
al. 2022 Hang al. 2023).
2.2
IMAGE PROCESSING WITH PRE-TRAINED DIFFUSION MODELS
Pre-trained diffusion models are versatile tools a variety of image processing tasks. When aiming to manipulate or restore reference image y, can simply adjust the sampling score function using Bayes' Rule:
log Pt (x:ly) V log Pt (x+) V log (x)
(6)
conditional
guidunce
The guidance term be derived from the gradient of a differentiable loss function. For example, FreeDoM (Yu et al., 2023) uses Gram matrices (Johnson et al.) 2016) of the intermediate layers of the CLIP image encoder as the loss function.
Another image processing technique is post-processing. These methods achieve the goal of image processing tasks by projecting the estimated value 2x, i) each time step onto predefined manifold M, (He et al. 2023 Wang et al. 2022. Chung et al. 2023 Zhu et al. 2023).
While there are efforts towards distillation work for the guided diffusion model (Meng al. 2023 the high distillation costs make impractical to distill for every kind of guidance. In contrast, efficient ODE solvers are much more flexible.
2.3 FAST SAMPLING WITH EXPONENTIAL INTEGRATORS
Samplers based on exponential integrator have been found to be more efficient than directly solving the ODE (4). Given initial value x, at time the ODE solution x, can be analytically computed
3

---

Published as a conference paper at ICLR 2025
Designer Network
Channel-wise mean pooling
Fully connected layers
Embedding layers
Bottleneck Feuture
Prediction stop
Ground truth trajectory
Inputs
Sampling trajectory
Optimization direction
Time Step
Step Index
Prodiction formula
Outputs
Designer network architecture. We concatenate the embeddings of the bottleneck feature, time step, and step index, and pass them through fully connected layer to obtain the parameters required for the prediction formula 88 9).
Figure 2:
with the following exponentially weighted integral by changing the variable from to half' log-SNR A log((;//00 (Lu al., 2022a
X1 _x, ai //m E'Eo(8,, = - _x,, + 01 //m -~> (8x, A)dA 0,
Ar
02
where ég(,, A) = c ((A))
and &2 == X(())- To predict the integral part, approximates co (x,,0 1) with polynomial interpolation w.r.t similarly DPM-Solver (Lu et al. 2022a approximate 60 (xx A) w.r.t À (equation 7 left) and DPM-Solvert+
(o( =
DEIS (Zhang& Chen 2022
right). UniPC (Zhao et al. 2023 introduces correcting strategy and various scale functions. And AMED-Solver (Zhou et al. adopts an approximate mean value method instead polynomial interpolation.
(Lu et al. 2022b) approximate Te (xx, w.r.t À (equation
2024)
2.4 EFFICIENT SAMPLER SEARCH AND DISTILLATION
Recently, several frameworks have incorporated a search phase to enhance the efficiency of sam- plers. The time step schedule is commonly explored aspect (Chen al., [0224] (Saoorr al. 2024 Li et al. 2023 Watson et al.] 2022 Liu et al. (023a). Moreover, DPM-Solver-v3 (Zheng et al. 2023) aims optimize the parameterization mode beyond noise prediction and data prediction.
USF
(Luu et al. 2023b) strives to identify the most suitable solver for each time step.
Bespoke solver (Shaul al. 2024) is also a type of solver distillation method, but only uses the final output to adjust hyperparameters, with a deep and complex computation graph that hinders effective hyperparameter tuning. Zhang et al.) (2024) proposed a method that utilizes local mean squared error (MSE) to refine existing solvers, which incurs minimal cost but yields limited effec- tiveness. We actively use intermediate values the trajectory and judiciously stop unnecessary gradient backpropagation, greatly enhancing the effectiveness of solver distillation method.
METHODOLOGY
In this section, we present our method to solve the PF-ODE 4 We start with the discussion about the optimal linear multistep prediction strategy. And then we introduce how to achieve the optimal solver with an adaptive time schedule and time scaling factors. Finally, we introduce some practical techniques to further improve performance. A comparison with related work is provided in Appendix A
3.1 TOWARDS OPTIMAL LINEAR MULTISTEP SOLVER
Given a time schedule (t, decreasing from to = to tx = 0. the solvers are trying to predict the ground truth trajectory {x. 1.V with numerical simulation trajectory (x. IN with
4

---

Published as a conference paper at ICLR 2025
x =x Linear multistep solvers aim to predict each local intermediate xf, 2 by leveraging the linear combination of p previous outputs from the denoising network. Take the data prediction type (equation 7 right) for example, the prediction formula can be expressed as:
P
D2
ag ag (xin
8n kn-kkn-k).
ak
(8)
Ohn x at,, (e nen 1)D
Oh-1
where {ax ]=== are the prediction coefficients and the time scaling factors (8n ) n= are usually 1. This implies that the candidate x5_ lies in p-dimensional hyperplane and hence the theoretically optimal x the linear projection of xf. onto the hyperplane.
worth noting that for each trajectory {x{., In-o and for each step n the optimal coefficients {at]== are different. Nevertheless, some studies have shown that the ODE trajectories from diffu- sive models have similar properties CChnn et al. 2024), so it is possible to design unified {ax ) that works well on all trajectories just as most previous works, but we consider this be suboptimal choice.
As mentioned in the previous paragraph, we aim to assign distinet coefficients for each trajectory and step. To achieve this, we establish a mapping g in 1) - {as J Draw- ing inspiration from Zhou et al. (202)) we utilize the bottleneck feature he.,,, obtained from the pretrained denoising model instead of x to circumvent additional computational costs. Subse- quently, we employ an extremely lightweight designer network with approximately 9k parameters, denoted as 9o h 1 12-1::10 1), to generate the coefficients {ax) The network architecture
is shown in Fig.
Given the availability of the ground truth trajectory {x(, L.W we can establish the optimal solver by minimizing the square distance dl (xx; xx.) where the prediction xx_ is computed using equations and with {au)k= = I (hi 1n-1, 1).
In practical terms, we utilize a numerical ground truth trajectory xx_,, generated by a teacher solver T such DPM-Solver++ with M interpolation time steps between 4n-1 and l,, which we denote as xf = &, (x]_ ( In-1,
3.2 ADAPTIVE TIME SCHEDULE
Adaptive time schedule is a class of approaches that can significantly improve the efficiency of various solvers. Some previous works (Xia et al.] [2024] Zhou et al.] (024)) also show that adjusting the time scaling factors (s,, // properly can improve the accuracy of the solution. In Sec. 3.1 we described how to get the distilled solver for fixed time schedule {t,) J In this section, we show how to use our distillation framework to get adaptive time schedules.
When adjusting the prediction coefficients {a=) in Sec. 3.1 we chose xS__, time in- as the starting point to predict x Now, we aim to incorporate fn-1 and 8n-1 into the design space. Therefore, we modify the starting point to be x", at time t,-2. To ensure each trajectory has an independent adaptive time schedule rather than a unified schedule, we include the time schedules {t.)) 1N and { in the output of Ig. This is achieved by setting {a tn,1, Io (hi... fn- , 2) for n > Let Q denote the buffer list that collects outputs x xx sntn)- We write the prediction formula as:
e tn En. {ax Ja Q)-
(10)
Thus, the distillation phase be summarized by Algorithm 1 After the distillation phase, Algo- rithm sisows the sampling process with trained designer network 9ø-
It is worth noting that we have employed the stop gradient operation multiple times in Algorithm which is crucial for the proper functioning of the algorithm. When using the state x at time
2 predict x!_ at time ln, retaining the gradients of I,,, 2, buy x² 27 x will lead to two issues:
5

---

Published as a conference paper at ICLR 2025
Algorithm Linear Multistep Solver Distillation
Require: Designer network ge. teacher solver 2,, number of time steps N, number of interpolation time steps M, initial timestep I', max order p.
repeat
2: Sample initial value x = x ~ (N, 12)
3: tn 4- T,,00 80 + 1. Initialize an empty buffer Q.
4:
Set the parameter gradient of & to
5:
Q _
builer xx(xx, solo)
6:
Extract the bottleneck feature li10
7:
for = to N do
8:
{ax]==1 m 1p)
tn- Sn-1 t Io Co (2 n--2 max(n- -1,p)) .Q)
9:
(xx
2 fak k==
Generate prediction x
Q buffer
No (x fn- 8n- 8n-14n-1)
Extract the bottleneck feature h4.-1 max(n.p)
12:
{ax )=1 bba 8n go 00(hen-11 In 1)
13:
in t sg(tn)
D Stop the gradient t,-
14:
S $(x 1n {ak sk=1
maxin.p) ,Q)
Generate prediction x
15:
n=2 then
x sg( (1 (x]
tn- 1,
end if
18:
x t sg( p (X in- Ln M))
b Solve the numerical ground truth x Calculate the prediction error at time t.-
19:
(Ø) d(x; x
20:
Perform backpropagation for L. (6).
21:
Q Sg Q), n- + sg((n-1)
D. Stop the gradient of buffers and -
22:
end for
Update the parameter
24: until convergence
Algorithm 2 Distilled Solver Sampling
Require: Designer network Je, number of time steps N, initial timestep max order p.
1: Sample initial value x N(0,ā")
2: in < T, + 1. Initialize an empty buffer Q.
3: for
do
4: Q Qquiler xo(x, 10 sn-1tn-1)
5: Extract the bottleneck feature hin-1
6: {a*)max(m=)) 8n go (ooca-ai In- 1)
mmax(n.p)
7:
xf, + -((xx In tn {o {a
max(n.p) Q))
8: end for
9: return x
Firstly, the gradients from past time steps will not be removed from the computation graph, resulting in a linear increase in memory usage with each step n.
Secondly, i, will move towards l,-2 to ease the prediction difficulty, ultimately resulting in the collapse of the entire time schedule. This is not ideal outcome.
In addition, we find it necessary to adopt data prediction type (equation 7 right). We believe this is because the data prediction type can automatically align the noise intensity with the time step t, thus stabilizing the prediction error for the solvers.
3.3 PRACTICAL. TECHNIQUES
In this section, we introduce several practical techniques to further improve the performance of DL.S.

---

Published as a conference paper at ICLR 2025
High-order initialization. Since the prediction formula { 9 covers almost all linear multi-step solvers with exponential integrators such as DDIM (Song et al. 2021a), PLMS (iPNDM) (Zhagg & Chen 2022; Liu et al.] 2022), DPM-Solver++ (Lu et al. 2002b). UniP-p (Zhao et al., 2023) and DEIS (Zhang & Chen 2022). We can initialize with these pre-designed solvers by setting the output biases of the designer network Jé- Therefore, DLMS has theoretical accuracy that is at least on par with the aforementioned methods. Thus, we still refer to the number of history outputs p used in each step as the "order". Although our framework still works with DDIM (Song et al. 202 initialization, using higher-order solvers allows the distillation phase to be completed more quickly with fewer generated trajectories. In our experiments, initializing with any higher-order methods did not show significant differences; for simplicity, we recommend PLMS (Zhang & Chen 2022 Liu et al. 2022) as the initial solving strategy.
Analytical First Step (AFS). In Algorithm [ the initial time step to = T and time scaling factor 80 = 1 are excluded from the design space. Instead, we can treat io T as a virtual initial step. Specifically, we set æz (xy, soto) 0 and he, O, without employing the denoising network. This approach ensures that l is the actual first time step at which the diffusion model is engaged, while t and $1 remain in the design space. This strategy is fundamentally equivalent to the analytical first step (AFS) proposed by Dockhorn et al. (2022).
Exponential Moving Average (EMA). Due to NFE limitations, the student solver cannot fully replicate the sampling trajectory generated by the teacher solver. This leads to a non-zero stochastic gradient even convergence. To reduce the impact of parameter oscillations solver performance, we introduce EMA updates inspired by diffusion models. During solver distillation, we apply EMA updates with half-lives of 1, 2. and 3 kimg, selecting the best-performing configuration from four parameter sets.
Inception distance at the final step. Some search-based frameworks directly use Fréchet Inception Distance (FID) (Heusel et al. 2017) as the optimization objective (Liu et al. 2023b Watson et al., 2022 Li et al. 2023). In contrast, our method employs square distance as the optimization target. Since square distance is less sensitive to high- fighffrequency information, we observe that optimizing square distance does not always lead better FID scores. Therefore, in experiments conducted in pixel space, we replace the final step's objective from pixel square distance to Inception Distance (the square distance of features from the Inception network). We only make this replacement in the final step because find that Inception Distance does not effectively capture the differences between noisy images. In latent space tasks, the issue of nigh-frequency information is not present. To avoid unnecessary calls to the network modules, we do not perform this replacement in those cases.
4 EXPERIMENTS
In this section, we demonstrate that DLMS exhibits significant advantages in both unconditional and conditional sampling with pixel-space and latent-space diffusion models. We conducted experiments across multiple datasets with resolutions ranging from 32 to 512 and compared our approach with current of-teoff-the-rt both handcrafted and search-based solvers. Then, showcase the benefits from adaptive time steps and time scales, as well as ablation studies of the practical techniques we provided. Finally, we visualize the adaptive time schedules and the samples generated by DLMS.
4.1 SETTINGS
In our experiments, we uniformly use the noise schedule a; 1, = f from Karras et al. (2022) We initialized the prediction coefficients with PLMS (Zhang Chen 2022 Liu et al. 2022). using uniform time schedule (Ho et al.) 2020) and time scaling factors of 1. We use DPM- SPM-Solver++ (Lu et al.. 2022b) to generate ground truth trajectories. The designer network g& consists of a two-layer MLP with total parameter count of only 9k. We use Adam as the optimizer with learning rate of 5 x 10-3 To ensure fairness, we use the same random seed to evaluate the FID score. For details,
see Appendix B
7

---

Published as a conference paper at ICLR 2025
4.2 MAIN RESULTS
We select the current state-of-the-an artificially designed solvers as baseline methods, including DEIS Zhang & Chen (202)) DPM-Solver++ (Lu et al. 2022b), and UniPC (Zhao et al. 2023) All results are obtained from an open-source toolbox uti utilizing the recommended settings from the original papers. Detailed results can be found in the Appendix C
DESS
DES Drvv--
25
C&PCC 25-
UsPP
DEMS
TIEES
DEMS
20-
155
16
10-
16
10
(a) CIFARIO 32x32 (Pixel space, unconditional)
(b) FFHQ 64 x464
(c) ImageNet-64 64x64
(Pixel space, unconditional)
(Pixel space, conditional)
85
D115
IPM Salven
Deitt
IPM
30-
DEMS
DLMS
DUNS
15-
03
16
16
NFB
NEE
(d) LSUN-Bedroom 256 (Latent space, unconditional)
(e) ImageNet 256 256
(l) MS-COCO 512x512 (Latent space, guidance scale 7.5)
(Pixel space, guidance scale 2.0)
Figure 3: Comparison of FIDI, scores between DLMS and handcrafted solvers. DLMS demonstrates
a significant advantage over handcrafted methods.
Fig. 3 presents a comparison of FID scores between DLMS and previous handcrafted methods. As shown in Fig. 3 handerafted solvers based polynomial interpolation exhibit sharp decline in performance as NFE decreases. In contrast, our proposed DLMS maintains high-quality sam- pling, demonstrating significant advantage over baseline methods. In text-to-image generation with Stable-Diffusion (Rombach et al., (2022). DLMS achieves an FID of 12.52 with only NFEs, while the baseline methods require more than 10 NFEs for comparable performance, resulting in a 2 acceleration ratio.
We further compare our method with other search frameworks. The methods include in the compar- ison are the reinforcement learning- -baedd framework USF (Liu et al. 2023b). the parameterization- focused DPM-Solver-v3 (Zheng al., (023). Additionally, we include GITS (Chen et al., 2024). an outstanding adaptive time scheduling method, and AMED-Plugin (Zhou et al., 20244) closely related work. As shown in Tab. [ DLMS still stands out among various search frameworks.
Table I: Comparison of FIDI on CIFARIO between DLMS and search-based solvers.
NFE
Solver
4
5
7
8
9
10
DPM-Solver-vs USF
12.21
8.56
3.50
2.51
11.50
6.86
5.18
3.81
3.41
3.02
2.69
AMED-Plugin
6.61
3.65
2.63
GITS
10.11
6.77
4.29
3.43
2.70
2.42
2.28
DLMS
4.52
3.23
2.81
2.53
2.43
2.37
2.24
8

---

Published as a conference paper at ICLR 2025
In Tab. compare the total GPU hours required for 7 NEE solver with USF and DPM-Solver- v3. Due to limitations in code availability, the reported time costs are sourced from original papers and measured on different devices. Nevertheless, method demonstrates significant order-of- magnitude advantage, achieving 10x increase in search efficiency.
Table 2: Comparison of total GPU hours for search phase. DLMS demonstrates a significant order- of-magnitude advantage.
Solver
CIFARI0
MS-COCO
Device
DPM-Solver-v3 28
88
NVIDIA A40
USF
12.15
106.64
NVIDIA 3090/A100
DLMS
0.7
10
NVIDIA V100
4.3 ABLATION STUDY
Tab. 3 demonstrates the ablation effects of each component in the DLMS framework. As shown in the results, the time step-related time schedule and time scaling contribute the most significant improvements.
Table 3: FID results of ablation study CIFARIO. The time step-related time schedule and time scaling contribute the most significant improvements.
NFE
10
DLMS
4.52
2.81
2.43
2.24
w/o AFS
6.48
3.30
2.42
2.30
w/o bottleneck feature
4.71
3.40
2.46
2.25
w/o high- initialization
4.92
3.25
2.94
2.44
w/o Inception distance
6.67
3.77
3.10
2.80
w/o time scaling w/o adaptive time schedule
7.75
3.86
3.07
2.41
10.41
6.18
3.17
3,03
Handcrafted (best)
25.66
9.40
3.99
2.89
Another finding is that DLMS performance does not always improve as the number of interpolation time steps M increases. In fact, the fitting capacity of student solvers limited. When the teacher solver is significantly superior to the student, side effects can arise due fitting difficulties. Espe- cially for models the latent space, setting = 1, mcaning the teacher has about twice NFEs of the student, yields the best results. The table below illustrates the effects of increasing M on the SSUN-Bedroom dataset.
Table The sensitivity M diffusion models.
Setting M yields the best results on latent
NFE
4
6
8
10
M=1
8.20
4.44
3.89
3.50
M=2
8.86
4.56
4.51
3.69
M=3
11.07
4.88
4.54
3.81
4.4 VISUALIZATIONS
Visual Quality. We present qualitative comparisons in Fig. 1 Handcrafted solvers struggle gen- erate vegetation and accurately colored bears with 5 NFEs. In contrast, DLMS effectively learns the generation results of the teacher solver with double NFEs. With 10 NFEs, DLMS is the only method that successfully produces the correct lighting and head pose. Additional samples are provided in Appendix

---

Published as a conference paper at ICLR 2025
Visualization of adaptive time schedule. Fig. shows the adaptive time schedules (average of different trajectories) obtained from DLMS with 10 NFE using AFS. By comparing these schedules with handcrafted time schedules such as logSNR (Zheng et al. 2023 [u et al.) 2022b), polynomial (Karras et al., 2022), time uniform (Ho et al.) 2020) and time square (Zhang & Chen 2022). we uncover some intriguing findings. The adaptive time schedules exhibit striking differences across various datasets. For the CIFARIO and FFHQ datasets, the adaptive time schedules are similar to the time square schedule. In contrast, for ImageNet-644 and uniform schedule. Notably, the schedule learned for the Stable-Diffusii model aligns closely with the uniform logSNR schedule, except for the final step.
they resemble the time
The observed differences may arise from several factors, including the solving strategy, image res- olution, solving space, and guidance. For instance, the EDM context, DPM-Solver++ (Lu al. 2022b) and UniPC (Zhao et al. 2023) perform better with logSNR, while DEIS (Zhang & Chen, 2022] demonstrates superior performance with the time square schedule. However, on other models, all three solvers tend to prefer the time uniform schedule. This highlights the importance of simul- taneously searching for both time schedules and solving strategies within our framework across different datasets.
CIOARD
LSUN-Boloon
NS-EXCO
TrageMeedd TTT3
L-h Peguerill
Pobsunil
Trae anilion
Time anffars
Transgrac
Tine oegars
upare
tades
9op Iade
(a) EDM
(b) Dreent-Diffusion
(c)
Figure 4: Visualization of adaptive time schedule. We use dotted lines to represent handcrated time schedules, and use solid lines represent adaptive time schedules learned each data set. DLMS learns different adaptive schedules on different data sets.
CONCLUSION
We propose a linear multistep solver distillation framework. Our framework enables the student solver to replicate the sampling trajectory of a teacher solver that utilizes more steps, facilitating rapid adjustments and optimization of prediction coefficients, time step schedules, and time scal- ing factors. Experiments demonstrate the effectiveness of our framework across various resolution datasets, using both pixel-space and latent-space pre-trained diffusion models, and reveal signifi- cant improvement in sample quality with 4~~0 NFEs.
Limitations and Future Work. Our framework is currently limited to ODE solvers, while in prac- tice, stochastic samplers (Xue et al., 2024) often outperform deterministic samplers. Therefore, extending method to stochastic samplers is promising direction. Additionally, integrating work with approaches such as Deepcache (Ma et al., (024) and FreeU (Si eti al. 2024) is also worth exploring.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -2567,, 2023.
Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of ODE-based diffusion sampling. In Forty-first International Conference on Machine Learning, 2024.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021.
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Decomposed diffusion sampler for accelerating large-scale inverse problems. arXiv preprint arXiv:2303.05754 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248 25-.2555.22 IEEE, 2009.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 35:30150-3011
2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominík Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXivv2403.032066 2024.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pp. 2672-2680, 2014.
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao. Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 7441-7451, October 2023.
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei- Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. arXiv preprint arXiv:2311.1642. 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300. 2022a.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, pp. 8633- 8646, 2022b.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part II 14, pp. 694-711, Springer, 2016.

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A byye-based generator architecture for generative adversarial networks. In CVPR, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 2022.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-free optimization of time steps and architec- tures for automated diffusion model acceleration. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pp. 7105-7114, 2023.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, pp. 740-755. Springer, 2014.
Enshu Liu, Xuefei Ning. Zinan Lin, Huazhong Yang, and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. In International Conference on Machine Learning, pp. 21915-21936. PMLR, 2023a.
Enshu Liu, Xuefei Ning, Huazhong Yang, and Yu Wang. A unified sampling framework for solver searching of diffusion probabilistic models. In The Twelfth International Conference on Learning Representations, 2023b.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:2211.01095, 2022b.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint
2021.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Xinyin Ma, Gongfan Fang. and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15762-15777 2024.
Chenlin Meng. Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14297-14306, June 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2025
Herbert E. Robbins. An Empirical Bayes Approach to Statistics, pp. 388-394. Springer New York, New York, NY, 1992. ISBN 978- 978-1-4612-0919-5. doi: 10.1007/978 1117777----612-019--22222 URL https://doi
26
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pallern Recognition (CVPR), pp. 10684-10695, June 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in diffusion models. arXiv preprint arXiv:2404. 14507, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Infor- mation Processing Systems, pp. 36479-36494 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprini 1rXXiv2331.17044 2023.
Neta Shaul, Uriel Singer, Ricky T. Q Chen, Matthew Le, Ali Thabet, Albert Pumarola, and Yaron Lipman. Bespoke non-stationary solvers for fast sampling of diffusion and flow models. In Forty- first International Conference on Machine Learning, 2024.
Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4733 4743, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In International Conference on Machine learning, pp. 3221 1-32252, 2023.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion 2022.
null-space model. arXiv preprint
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2022.
Mengfei Xia, Yujun Shen, Changsong Lei, Yu Zhou, Deli Zhao, Ran Yi, Wenping Wang, and Yong- Jin Liu. Towards more accurate diffusion model acceleration with a timestep tuner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5736- 5745, June 2024.
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li. and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506 550.003356600355 2015.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVE International Con- ference on Computer Vision, pp. 23174-23184, 2023.
Guoqiang Zhang, Kenta Niwa, and W. Bastiaan Kleijn. On accelerating diffusion-based sampling processes via improved integration approximation. In The Twelfth International Conference Learning Reprexentations, 2024.
Qinsheng Zhang and Yongxin Chen. Fast sampling diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2022.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. In Advances in Neural Information Processing Systems, pp. 49842 49869, 2023.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Processing Systems, 36: 55502-55542, 2023.
Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7777-7786, June 2024.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1219-1229, 2023.
14

---

Published as a conference paper at ICLR 2025
A COMPARING WITH RELATED METHODS
Comparing with Progressive Distillation. Progressive distillation (Salimans & Ho 2022) is a distillation method for diffusion models that gradually distills the results of multi-step solvers into fewer steps. In our framework, we also employ a multi-step teacher solver and aim to distill the results into our student solver with fewer steps. However, our approach does not require training the parameters of the diffusion model, while the parameter requirements and training duration for progressive distillation (PD) are thousands of times greater than those of our framework.
Comparing with AMED--luggi. AMED-Plugin (Zhou et al. 2024) is a method for selecting intermediate time steps for existing solvers and time schedules. The designer network Io used in our work is modified from AMED-Plugin. In contrast, the DLMS in this paper do not rely on existing solvers or time schedules. AMED-Plugi can be seen as adjusting half of the time schedule, while our proposed adaptive time schedule method is for full time schedule adjustment.
Comparing with USF. USF (Liu et al. 2023b) is a search framework based on reinforcement learn- ing. The authors train a predictor network to estimate the FID performance of a solver with specific hyperparameters, using this to guide an evolutionary search process for hyperparameter optimiza- tion. The relationship between hyperparameters and performance is complex, making the training of a predictor network both challenging and conssonns...g.
In contrast, framework leverages local prediction enror to design an optimal solution strategy, which aligns more closely with the iterative nature of the sampling process and the criteria for manually designing solvers. Furthermore, while USF simply combines existing handcrafted solvers, our approach utilizes prediction formulas (8 9 to achieve a true unification of multiple solving strategies. This enables us to develop new solving strategies by learning the prediction coefficients {ax 1. k==1
Comparing with Bespoke solver. Bespoke solver (Shaul al. 2024) also a type of solver distil- lation method, but it only uses the final output to adjust hyperparameters, with a deep and complex computation graph that hinders effective hyperparameter tuning. We actively use intermediate val- ues in the trajectory and judiciously stop unnecessary gradient backpropagation, greatly enhancing the effectiveness of solver distillation method. For EDM trained on CIFAR10, our method achieved an FID of 2.53 with NFEs, outperforming their result of 2.75 with 20 NFEs.
B DETAILED SETTINGS
EDM on CIFARIO, FFHQ, ImageNet-64. The sampling on CIFARIO (Krizhevsky et al. 2009) 32x32, FFHQ Karras et al. 2019) 64 x64, ImageNet-64 (Deng et al. 2009) 64x64 is based on the pretrained pixel-space diffusion model provided by EDM (Karras et al. 2022). Among these, ImageNet-64 is for conditional sampling and CIFAR1O 10, FFHQ are for unconditional Sampling. The order p for student solver DLMS is set to 4. The number of interpolation time steps M is set to 4. For each dataset, we conduct solver distillation on 20k trajectories. The distillation times are approximately NOX seconds, 80XNFE seconds, and 150xNFE seconds, respectively, on 8 NVIDIA V100 GPUs We measure sample quality using the FID score calculated on 50k generated images.
Latent-Diffusion on LSUN-Bedroom. 2015) 256x256, is based on the pretrained latent-space diffusion model provided by Latent- Diffusion (Rombach et al. 2022). The order p for student solver DLMS is set to The number of interpolation time steps M is set to 1. We conduct solver distillation on 10k trajectories. The distillation times are approximately 3 NFF mins on NVIDIA V100 GPUs. We measure sample quality using the FID score calculated on 50k generated images.
The unconditional sampling on LSUN-Bedroom (Yu al.,
on ImageNet. The conditional sampling on ImageNet (Deng et al., 2009) 256, is based on the pretrained pixel-space diffusion model provided by Duided-Diffusion (Dhariwal & Nichol, 2021). The order p for student solver DLMS is set to 3. The number of in- terpolation time steps M is set to 3. We conduct solver distillation on 5k trajectories with default guidance scale 2.0. The distillation times are approximately 6xN mins on NVIDIA V100
GPUs We measure sample quality using the FID score calculated on 10k generated images.
15

---

Published as a conference paper at ICLR 2025
Stable-Diffussie MS-COCO prompts. The text-to-image sampling MS-COCO (2014) Lin et al. 2014) 512x512, is based on the pretrained latent-space diffusion model provided by Stable- Diffusion v1.5 (Rombach et al. 2022). The order p for student solver DLMS is set to 2. The number of interpolation time steps M is set to 1. We conduct solver distillation on 5k trajectories with default guidance scale 7.5. The distillation times approximately NxNFE mins 8 NVIDIA V100 GPUs. We measure sample quality using the FID score calculated on 30k generated images generated by 30k prompts from the MS-COCO validation set.
C FID RESULTS FOR DLMS AND HANDCRAFTED SOLVERS.
Table 5: Comparison of FID scores between DLMS and handcrafted solvers.
NFE
Dataset
Solver
4
5
6
10
DEIS
25.66
14.39
9.40
6.94
5.55
4.68
4.09
DPM-Solver++
46.52
24.97
11.99
6.74
4.54
3.42
3,00
CIFAR1O
UniPC
45.20
23.98
11.14
5.83
3.99
3.21
2.89
DLMS
4.52
3.23
2.81
2.53
2.43
2.37
2.24
DEIS
28.31
17.36
12.25
9.37
7.59
6.39
5.56
DPM-Solver++
45.95
22.51
13.74
8.44
6.04
4.77
4.12
FFHQ
UniPC
44.78
21.40
12.85
7.44
5.50
4.47
3.84
DLMS
9.63
6.85
5.82
5.16
4.81
4.23
4.12
DEIS 23.53
14.75
12.57
8.20
6.84
5.97
5.34
DPM-Solver++
56.63
25.49
15.06
10.14
7.84
6.48
5.67
ImageNet-64
UniPC
55.63
24.36
14.30
9.57
7.52
6.34
5.53
DLMS
10.07
7.16
7.08
6.31
5.93
4.57
4.30
DEIS
12.89
6.83
5.35
4.78
4.43
4.25
4.05
DPM-Solver++
48.49
18.44
8.39
5.18
4.12
3.77
3.60
LSUN-Bedroom
UniPC
39.66
13.76
6.46
4.52
3.96
3.72
3.56
DLMS
8.20
5.44
4.44
3.99
3.89
3.70
3.50
26.30
17.08
13.06
11.08
9.95
9.25
8.72
ImageNet
UniPC
24.99
15.71
11.95
10.24
9.36
8.82
8.45
DLMS
19.74
13.83
10.90
9.66
9.07
8.89
8.22
DEIS
15.16
13.95
13.61
13.58
13.46
13.38
DPM-Solver++
17.40
15.78
15.08
14.77
14.43
14.18
MS-COCO
UniPC
17.33
15.70
15.06
14.74
14.44
14.28
DLMS
12.52
11.87
11.65
11.63
11.52
11.45
D MORE QUALITATIVE RESULTS
16

---

Published as a conference paper at ICLR 2025
(a) DEIS. FID=25.66
(b) DLMS, FID=4.52
Figure 5: Uncurated samples on CIFARIO 32 32 with 4 NFE.
(a) DEIS, FID=28.31
(b) DLMS, FID=9.63
Figure 6: Uncurated samples on FFHQ 64 64 with 4 NFE.
17

---

Published as a conference paper at ICLR 2025
(a) DEIS, FID=14.75
(b) DLMS, FID=7.16
Figure 7: Uncurated samples ImageNet-64 64 x 64 with NFE.
(a) DEIS, FID=12.89
(b) DLMS, FID=8.20
Figure Uncurated samples I.SUN-Bedroom 256 256 with 4 NFE.
18

---

Published as a conference paper at ICLR 2025
Table 6: Additional samples of Stable-Diffusion (Rombach et al. ance scale 7.5, using only 10 NFE and selected text prompts.
2022
with a classifier guid-
Text Prompts
+ (FID=14.18)
DEIS (FID=13.38)
DLMS (FID=11.45)
"a sandwich on wheat bread sits on a plate"
"three big elephants walking across wide river"
"Air force jet a take off position above the tree line."
"An empty bench next to a potted tree up against brick wall."
"A star hangs upon a canopied bed in a bed- room."
19


Reviews:
[{"review_id": "iXo6VuNx3k", "reviewer": "Reviewer_V3hM", "summary": "This paper proposes a Distilled Linear Multistep Solver (DLMS) to learn a faster sampler for diffusion models, requiring fewer function evaluations. The distillation approach is to train a solver that minimizes the Euclidean distance between its trajectory and a teacher solver’s trajectory. DMLS can be trained faster than previous reinforcement learning-based approaches to solver distillation. Experimental results in image generation using unconditional, conditional, latent space, and pixel space diffusion show improved FID scores compared to existing methods, especially in low NFE settings.", "questions": "- On line 398 it is reported that the distillation time of DMLS is approximately 1.5 * 8 = 12 hours for stable diffusion, while the abstract claims that the framework “has the ability to complete a solver search for Stable-Diffusion in less than 10 total GPU hours”. Is this difference due to a rounding error or do these claims refer to different times? Clarify this discrepancy and ensure consistency between the abstract and results.\n\n**Minor suggestions that do not individually affect the score**\n- Line 129: “can be carry out” -> “can be carried out”.\n- Line 189: Remove “precious”.\n- Line 263: Introduce the strop gradient operation.\n- Line 284: Reformulate.\n- Line 292: “PLMS(iPNDM)” -> “PLMS (iPNDM)”.\n- Line 340: “AMED-Plugin(Zhou et al., 2024)” -> “AMED-Plugin (Zhou et al., 2024)”.\n- Line 363: Specify “various aspects” and “as well as the ablation…” -> “as well as ablation…”.\n- Line 394: “MS-COCO(2014)” -> “MS-COCO (2014)”.\n- Line 485: “Handcrafted(best)” -> “Handcrafted (best)”.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are its well-motivated method and a wide range of experiments.\n- The approach can be initialized using existing solvers.\n- Unlike model distillation, solver distillation like DLMS can be used for downstream tasks like image restoration.\n- The proposed method is simpler than using a reinforcement learning-based approach.\n- The method is evaluated in multiple contexts (unconditional, conditional, latent space, and pixel space diffusion) and on multiple datasets with convincing results.\n- The paper is overall well-written.", "weaknesses": "The weaknesses of the paper include its figure and table captions and reliance on FID as the sole quantitative metric.\n- In general, the figures and table are not self-contained. Figure 4, for example, could be made more interpretable by describing the significance of dashed lines.\n- FID scores are the only quantitative metric used to evaluate the method. Quantifying the quality of generated images is challenging, so adding multiple metrics like IS or CMMD [1] would increase confidence in the results. \n- In the introduction (line 047) it is argued that distillation is expensive, requiring multiple GPU days of training. The reported training times for DLMS are still more than ten hours, so these methods could still be compared, if not to better understand their respective strengths and weaknesses.\n- The time comparisons in Table 2 are hard to draw conclusions from considering the reported times are compared to those from previous papers that were run on different systems with different GPUs and software environments. It is stated that this is due to limited code availability. If they are possible to obtain, FLOP counts (or an estimate of them) would be more comparable.\n\n**References**\n\n[1] Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A. and Kumar, S. “Rethinking FID: Towards a Better Evaluation Metric for Image Generation”, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2kZGEjwUol", "reviewer": "Reviewer_AsEn", "summary": "The paper proposes a method for learning a trajectory-specific solver for diffusion models. The method suggest to use a small network to predict the best time step size and coefficients of a linear multistep at each step of the solver. The method is tested on a number of image generation tasks.", "questions": "1. Does a different designer network needs to be learned for each choice of NFE, multistep order?\n2. Is the designer network is somehow constrained such that always $t_N=0$?\n3. Could the authors provide the size of the designer network?\n4. The designer network is dependent only on $h_{t_{n-1}}$ or previous times as well?\n\n\n[1] Zheng, Kaiwen, et al. \"Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics.\" Advances in Neural Information Processing Systems 36 (2023): 55502-55542.\n\n[2] Shaul, Neta, et al. \"Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models.\" arXiv preprint arXiv:2403.01329 (2024).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea to learn a trajectory-specific solver is novel and interesting.\n2. The method shows good results on number of benchmark datasets.\n3. The method is compared to a number of diffusion dedicated solvers.", "weaknesses": "1. The authors make use of what they call \"bottleneck feature\" without any explanation what are those features, only referencing the relevant paper. The paper should be self contained and the authors should make an effort to give even brief explanation about these features.\n2. The method is not compared to any other solver distillation methods such as [1], [2].\n3. Discussion and comparison to model distillation is too minimal.\n4. The size of the designer network is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NUUA6QHKop", "reviewer": "Reviewer_23f3", "summary": "This paper proposes DLMS, a flexible solver framework that incorporates the combination of previous model outputs, timestep schedule and timestep scaling factor. The authors further introduce a light weight designer network to dynamically decide the solver strategies for each single trajectory. Experimental results demonstrate DLMS achieves notable improvements over existing solvers and offers faster optimization than search-based methods.", "questions": "1. What is the relationship between the coefficients predicted by the designer network and those derived from Taylor expansion in previous methods? Could you provide a comparison of these coefficients with those from previous methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. DLMS offers a flexible framework for diffusion solver, unifying existing methods.\n2. DLMS uses dynamic solving strategies for different ODE trajectories, enhacing the potential for diffusion solver.\n3. Experimental results demonstrate significant performance improvements compared to existing solvers, and the designer network's training cost is more efficient than that of search-based solvers.", "weaknesses": "1. Algorithm 1 implies that each NFE configuration may require independent designer networks, which could limit flexibility in the NFE-performance trade-off. If this is not the case, how does the designer network ensure that $t_N$ becomes a reasonably small when the step count reaches N?\n2. The time scaling factor may introduce input distribution misalignment, so further discussion on the motivation and explanation of this would be beneficial.\n3. The designer network currently relies on U-Net intermediate feature. As transformers gain popularity in diffusion models, it is uncertain if this approach is adaptable to such architectures.\n4. It would be helpful to illustrate differences in solver design choices provided by the designer network across various ODE trajectories to support the claim that a unifed choice for all trajectories is suboptimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fWdtR3juZy", "reviewer": "Reviewer_Dd8U", "summary": "This paper considers diffusion distillation for  a student linear multistep solver from a teach linear multistep solver with more timesteps.  In particular,  the authors propose to train a lightweight neural network to predict the coefficients, time step schedules, and time scaling factors of the student linear multistep solver. The cost function when training the lightweight neural network is taken as the mean squared distance of the difference of diffusion states produced by the student and the teacher linear multipstep solver, respectively. Experiments on FID shows the effectiveness of the new method.", "questions": "please check my responses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strength of the paper is that the authors propose to train a lightweight neural network that produce not only the coefficients of the student linear multistep solver but also the timesteps and the scaling factors.  This is based on the assumption that for different ODE trajectories, the optimal coefficients, timesteps and scaling factors are different.", "weaknesses": "(0) One weakness is that a small neural network is required to be trained for each particular pre-trained model. Note that not every university or research institute has 8 A00 or H100 GPUs for conducting the training process.  \n\n(1) The literature is not thorough. This work is closely related to a recent paper [1], which is not mentioned at all. The work of [1] considers computing the optimal coefficients of a student linear multistep solver per timestep by solving a quadratic optimization problem. The computational complexity of [1] is negligible as the quadratic optimization problem takes a closed-form solution. The authors should include the performance of [1] in their work. \n\n(2) One thing that is not clear to me is results for the two experiments of Latent-Diffusion on LSUN-Bedroom and Stable-Diffusion on MS-COCO prompts, where the number of interpolation timesteps M=1. I would think that the teacher ODE solver with two times of the number of times perform betters than the student ODE solver.  Is it the case? If not, explain why. \n\n(3) I would think that in general, the higher the M value, the better FID score of the student ODE solver. So why in different experimental setups, M were chosen differently? Would higher M value in some cases lead to poor performance? If so, explain why and include the results in the revision. \n\n(4) Typo:  \"can be carry out\"\n\n[1] Guoqiang Zhang, Kenta Niwa, W. Bastiaan Kleijn, \"On Accelerating Diffusion-Based Sampling Processes via Improved Integration Approximation, ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)