S4S: Solving for a Fast Diffusion Model Solver
Authors: Eric Frankel
Sitan Chen
Jerry Li
Pang Wei Koh
Lillian J. Ratliff
Sewoong Oh
Venue: ICML 2025 poster (2025)
Keywords: Diffusion models
efficient sampling
ODE solvers
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We learn the coefficients and time steps of diffusion model solvers to minimize global approximation error.

Abstract: Diffusion models (DMs) create samples from a data distribution by starting from random noise and iteratively solving a reverse-time ordinary differential equation (ODE).
Because each step in the iterative solution requires an expensive neural function evaluation (NFE), there has been significant interest in approximately solving these diffusion ODEs with only a few NFEs without modifying the underlying model.
However, in the few NFE regime, we observe that tracking the true ODE evolution is fundamentally impossible using traditional ODE solvers.
In this work, we propose a new method that learns a good solver for the DM, which we call **S**olving **for** the **S**olver (**S4S**).
S4S directly optimizes a solver to obtain good generation quality by learning to match the output of a strong teacher solver.
We evaluate S4S on six different pre-trained DMs, including pixel-space and latent-space DMs for both conditional and unconditional sampling.
In all settings, S4S uniformly improves the sample quality relative to traditional ODE solvers. 
Moreover, our method is lightweight, data-free, and can be plugged in black-box on top of any discretization schedule or architecture to improve performance.
Building on top of this, we also propose **S4S-Alt**, which optimizes both the solver and the discretization schedule.
By exploiting the full design space of DM solvers, with 5 NFEs, we achieve an FID of 3.73 on CIFAR10 and 13.26 on MS-COCO, representing a $1.5\times$ improvement over previous training-free ODE methods.

S4S: Solving for a Fast Diffusion Model Solver
Eric Frankel
Sitan Chen?
Jerry Li
Pang Wei Koh
Lillian J. Ratliff'
Sewoong Oh
Abstract
May gigu i ()
Diffusion models (DMs) create samples from a data distribution by starting from random noise and iteratively solving reverse-time ordinary dif- ferential equation (ODE). Because each step in the iterative solution requires an expensive neural function evaluation (NFE), there has been sig- nificant interest in approximately solving these diffusion ODEs with only few NFEs without modifying the underlying model. However, in the few NFE regime, observe that tracking the true ODE evolution is fundamentally impossible using traditional ODE solvers. In this work, we propose a new method that learns a good solver for the DM, which we call Solving for the Solver (S4S). S4S directly optimizes solver to obtain good generation quality by learning to match the output of strong teacher solver. We evaluate S4S six different pre-trained DMs, including pixel- space and latent-space DMs for both conditional and unconditional sampling. In all settings, S4S uniformly improves the sample quality relative to traditional ODE solvers. Moreover, our method is lightweight, data-free, and can be plugged in black-box top of any discretization schedule architecture to improve performance. Building top of this, we also propose S4S-Alt, which optimizes both the solver and the discretization schedule. By exploiting the full design space of DM solvers, with NFEs, we achieve an FID of 3.73 on CIFAR10 and 13.26 on MS-COCO. representing 1.5x improvement previous training-free free ODE methods.
latho
Figure I. Abstraction of the S4S approach. low-NFE environ- ments, common off off-the-shellf ODE solvers unable to approx- imate the true diff'usion ODE trajectory and produce low-quality samples. In S4S, we learn optimal combination of solver coeffi- cients discretization steps that more closely match output of the true ODE trajectory.
1. Introduction
Diffusion models (DMs) are a class of powerful models that have revolutionized generative modeling and achieve state- of-the-art performance in a wide number of domains (Sohl- Dickstein al., 2015; Ho al., 2020; Song et al., 2021b). Abstractly, DMs learn score network that approximates the time-dependent score function of diffusion process (Song et al., 2021b; Chen et al., 2023). Sampling from them often involves solving an ordinary differential equation (ODE) referred to as the diffusion ODE, where the dynamics are determined by the score network (Song et al., 202a,;a). This ODE typically requires large number of neural func- tion evaluations (NFEs) to numerically solve, and conse- quently is quite slow (Ho et al., 2020; Karras et al., 2022). This is directly at odds with many exciting applications of DMs for which low-latency inference is essential, such as robotics (Chi et al., 2024) or game engines (Valevski et al., 2024). Therefore, there is tremendous amount of interest in understanding how the number of NFEs may be reduced without sacrificing performance.
'University of Washington HHrvard University AAlenn Institute for AI. Correspondence to: Eric Frankel <er- iesf@es.wand wat.sesswssnnnoon.d......
Methods for enabling DMs to fewer NFEs generally fall under one of two categories: learning an entirely new model that distills multiple score network evaluations into sin- gle step (training-based), or designing efficient diffusion ODE samplers while keeping the score network unchanged (training-free). From practical standpoint, training-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

S4S: Solving for Fast Diffusion Model Solver
methods, such as progressive distillation (Salimans & Ho, 2022; Meng et al., 2023) and consistency models (Song et al., 2023) require access to original data samples and sub- stantial computational resources, which may not be avail- able or feasible. Additionally, training-basee basedd methods often optimize objectives that fundamentally alter the model's interpretation as a score function, making them unsuitable for tasks that rely on score-based modeling, such as guided generation (Ho& & Salimans, 2021), composition (Du al., 2023), and inverse problem solving (Xu et al., 2024).
1.1. Contributions
Below we summarize the contributions, breaking them into description of our efforts developing a novel method for learning the solver, and novel method for alternating between learning solver coefficients and the time discretiza- tion.
1.1.1. SOLVING FOR THE SOLVER (S4S)
The first contribution a new method for finding numerical solvers for DMs in the low NFE regime. Rather than using any fixed set of pre-existing methods, we instead take the approach of learning good solver for the diffusion model. We call our approach Solving For the Solver, or S4S.
For these reasons, we focus on training- raining-free approaches, which requires selecting a discretization of the diffusion ODE and determining both the optimal evaluation time- steps and synthesis strategy to accurately approximate the continuous trajectory. The majority of the literature has focused on choosing good time-step schedule in this low NFE regime-iee., choosing when to spend our budget of NFEs (Watson et al., 2021; Sabour et al., 2024; Tong et al., 2024; Xue et al., 2024; Chen et al., 2024). Yet, in prac- tice, it is equally important to choose a good solver- this corresponds roughly to choosing how to synthesize these different function evaluations. Most works still rely on tradi- tional "textbook" ODE solvers such as single-step (SS) (Lu et al., 2022a;b) linear multi-step (LMS) methods (Lu et al., 2022b; Zhang& & Chen, 2023). While there is some literature that explores going beyond these solvers (Zheng et al., 2023; Zhang et al., 2023; Zhou et al., 2024a), these approaches only explore narrow components of the sampler design space.
Crucially, we seek to find a solver that is good at approxi- mating the overall diffusion process, rather than attempting to discretize any ODE. Indeed, as we demonstrate in Ap- pendix H.2.4, any attempts at maintaining the "standard" invariants that guarantee that traditional solvers track the continuous-time ODE trajectory actively hurt performance. This reinforces our intuition that we must break from this standard approach to obtain the best results.
In somewhat more detail, S4S uses a distillation-styy objec- tive for learning solver coefficients. Here, base "teacher" ODE solver that takes small step sizes-and thus requires many NFEs provides trajectories that give high sample quality. In turn a "student" solver with learnable coeffi- cients, given the same noise latent, learns to produce equiva- lent images with a smaller number of steps. We explain our method in more detail in Section 3.1. Our method has the following advantageous properties.
At their heart, off-the-shelf solvers (and much of the prior work on optimizing samplers) seek to approximate the path of the true ODE in discrete time, which can be done given a sufficiently fine discretization (i.e. many NFEs). These methods are carefully crafted so that each step yields an accurate low-degree Taylor approximation of the ODE solu- tion over small time window. Our key observation is that in the low NFE regime, this is the wrong thing to target!, as analytic tools such as low-degree approximation simply do not make sense in the setting where the step-size is gigantic.
Universal improved performance. In our experiments, we demonstrate that in every setting we tried, our method uni- versally improves the FID achieved compared to previous state-of-the-ae th-thr-art solvers.
Plug-in, black-box improvement. Relatedly, our method can easily be plugged-in in black-box manner on top of any discretization schedule, and for any architecture. Notably, the gains we achieve from optimizing the solver are orthog- onal to the gains from optimizing these other axes, e.g. even with a carefully optimized discretization schedule, plugging in S4S will achieve a noticeable improvement in FID. Thus, our method offers a simple way for any practitioner to in- stantly improve the performance of their generative model.
We propose to abandon this formalism, and rather to di- rectly optimize a solver to improve performance of the diffu- sion model. similar observation was made independently by Shaul et al. (2024b); however, among other issues, the method they derived seeks to completely generalize all pre- viously known solvers. As result, their solver incorporates large amounts of irrelevant information and optimizes very complex objective, and is thus unable to match state-of-the- art performance in many settings. In contrast, we give a cleaner, more direct approach for obtaining an optimized solver and demonstrate that our method uniformly improves upon traditional solver performance in virtually all settings we tested.
Lightweight and data-free. Our method is lightweight, with minimal computational expense which is comparable to (and often less than) alternative methods for optimizing aspects of the solver, often taking less than an hour on a single A A100. Our method is also completely data-free, thus coming at no additional statistical cost to the user.
2

---

S4S: Solving for Fast Diffusion Model Solver
1.1.2. SOLVING FOR THE FULL SAMPLER: S4S-ALT
where Xo ~ p(Xo), ~ (O, u T). w(() is time-dependent weighting function, and x, 0x + 0 is a noisy sample at time t (Ho et al., 2020; Lu et al., 2022a). By Tweedie's formula, c0(x;, learns to approxi- mate - og Vx log (r(),, thereby defining the diffussio ODE
While S4S by itself already presents uniform and substantial improvements across the board, we find that much of the power of S4S is truly revealed when it is effectively com- bined with methods for choosing good discretization. By doing so, we are able to fully exploit the design space of the ODE sampler, something which appears to have been poorly explored in the literature previously. We propose an alternating minimization- based approach that iteratively updates either the coefficients or the discretization schedule one at a time. We call this approach S4S-Alt.
[(0)xx++ + g²(t)
dx:
ca co 8) dt.
(2)
20,
with initial condition XT ~ (T(TT). To exactly solve the diffusion ODE at x2 given an initial value Xs, where t < 8, Lu et al. (2022a) reparametrizes Eq. (2) terms of the log signal-to-noise ratio A == log(as/or), yielding
While S4S already improves upon previous baselines, by using S4S-Alt to jointly optimize the discretization schedule as well as the solver, we are able to dramatically improve upon state-of-the methods across the board, often by a factor of 1.5 - 2 or more (see e.g., Table 3 and the tables in the appendix). For example, with only five NFEs, achieve FID scores of 3.89 on AFHQ-v2, 3.73 on CIFAR-10, 6.25 on FFHQ, 4.39 on condiccon ImageNet, and 13.26 on MS-COCO with Stable Diffusion. Notably, these numbers are substantially better than what can be achieved by just optimizing the discretization schedule or S4S, separately.
Au
01 Xti
Xx: 0tt
Oity
e E A)dA,
where Xy and ca (xx,, A) denote the reparametrized forms of x2 and EØ (xe, 1) in the A domain.
2.2. Background: Solving the Diffusion ODE
Sampling from a DM requires numerically solving the dif- fusion ODE in Eq. (2). Given a decreasing sequence of N discretization steps [ from to = T to ty 0, we iteratively compute sequence of estimates {&,, { start- ing from X. XT N(x;; 0, a²1) such that the global truncation error between Rty and the true solution X,y is low. The standard approach of controlling this error is to bound the local truncation error between Xt, and xt, at each t,. Since Eq. (3) gives the exact solution of the diffusion ODE given an initial value X1, 1 an accurate approximation of the integral in turn provides an accurate approximation X for the true solution at time f One can take Taylor expansion of &o
2. Background and Related Work
We review background on diffusion models and ODEs, solvers for diffusion ODEs, and learning-based samplers.
2.1. Background: Diffusion Models
Let xo E Rdd be a random variable from an unknown data distribution po DMs define a forward process [xe)rejo.T with T > O that starts from Xo and progres- sively adds Gaussian noise to converge to a marginal distri- bution, px(x)), that approximates an isotropic Gaussian, i.e. pT(xT) N(xT; 0, a) time T' for some > (Ho et al., 2020; Song et al., 2021b). Given Xu, we can characterize the process of adding Gaussian noise by the transition kernel Poe(x, (xo) = A/ (xx; axXo, of)), for all I E [0, where Oiz, oe > 0 are selected such that the signal-to-ne to-iise ratio (SNR), af /of. decays as increases. Remarkably, Song et al. (2021b) demonstrated that this for- ward process shares the same marginal distribution pa as the probability flow ODE, reverse-time ODE starting at XT (xT) given by
(Xx, A) about AG. in Eq, (3), yielding
St. Ctt, Nr.
Ct2
0
(n)
Aiz 1 du 100)
Ott-1
n=
O(b+t))
(4)
for some v/,, (h) depending on n, Xe. and A see Ap- pendix B. for further details. Computing such k-th order approximation requires accurate estimates of the derivatives o up to order n = k- - 1. Existing methods use two main approaches from ODE literature: single-step methods (Lu et al., 2022a;b; Zheng et al., 2023; Zhao et al., 2023; Zhang & Chen, 2023; Karras et al., 2022), which use A - interme- diate points in (t4, and linear multi-step methods (Lu et al., 2022b; Zheng et al., 2023; Zhao et al., 2023; Zhang & Chen, 2023; Liu et al., 2022), which use information from A: - previous steps. For low order methods (k: < 4), under appropriate regularity conditions (see Appendix B.2) and when Aman == max h; is bounded by O1/N), these methods achieve local truncation error of (hh+1) and therefore global error of (lh
dx. = f(1)x; 1
dl, dloogaz/ ot/ and g(t) = (do7 2(d log au /d) (Kingma et al., 2021). Since the score function Vx log (x.) in Eq. (1) is unknown, DMs leam it using a noise prediction neural network to minimize
where f(t)
(()))
E [w(t)] (x t) =| X..
3

---

S4S: Solving for Fast Diffusion Model Solver
When the number of NFES is large and thus fmoz is small, local truncation error control yields high quality samples (Lu et al., 2022a;b; Zhang & Chen, 2023). However, with few NFEs and large hmaz. the higher-order Taylor errors dominate, leading to large global error. In contrast, our approach in Eq. (6) directly minimizes the global error.
we describe our method for learning both the solver coeffi- cients and the discretization steps.
3.1. S4S: Learning Solver Coefficients
For a learned score network and initial noise latent xT NO, a9I), one can sample from diffusion ODE using an appropriate sequence of pre-determined discretization steps [ and an ODE solver Y determined by its coefficients & and the number of steps k it uses. For SS and LMS solvers, we write their estimate of the next step as
2.3. Related Work: Learned Samplers
In practice, no single pair of ODE solver and a time dis- cretization generates high quality samples universally across various datasets and model architectures, e.g. Appendix H.4 and Tong et al. (2024). This inspired learning-baid based meth- ods for deriving ODE solvers and time discretizations adapted to the given task and architecture. We give brief survey here and discuss in detail in Appendix A. One pop- ular approach exclusively learns the discretization steps (Watson et al., 2021; Sabour et al., 2024; Xue et al., 2024; Tong et al., 2024; Chen et al., 2024). Our approach S4S learns the solver coefficients, complementing the gains of such methods and universally improving the performance in all scenarios, as seen in Table 2 and comprehensively in Appendix H.4. Another line of research focuses on op- timizing only the solver coefficients (Zheng et al., 2023; Zhang et al., 2023), or jointly optimizing both solver coeffi cients and time discretizations (Zhou et al., 2024a; Zheng et al., 2023; Liu et al., 2023; Shaul et al., 2024a). However, these methods are designed to minimize the local approxi- mation error through the same methods as in Eq. (4) or by closely matching the entire trajectory of the teacher solver. Instead, by minimizing the global error by matching the end of the teacher trajectory, as in Eq. (6), S4S significantly improves over these approaches. Closest to our approach BNS (Shaul et al., 2024b), which learns both the solver coefficients and time discretizations to minimize global er- ror. We provide comparisons in Table 4 and explain our improvements over BNS in Appendix A.3.
04 _ - -0, 1)),(a),
X2
(5)
Cle
where (,()) represents the increment of the solver as a function of the coefficients &. We explicitly define (:))) in Table 1. A PC solver further refines this initial prediction, by subsequently applying Eq. (5) again with new coeffi- cients. We provide the intuition behind this formulation in Appendix B.1 and equivalent examples for data predic- tion model in Appendix C. To denote the fact that learned solver uses & steps of information, we abuse notation and refer to it as having order k.
We propose Solving for the Solver (S4S) in Algorithm I to learn these coefficients to adapt to the problem instance of the given score network. Consider the outputs from "teacher" solver, V (xT). which accurately solves the dif- fusion ODE, We aim to minimize the global error between the sample v (xT) generated by sequentially applying Vs from In = T to iN- = 0 and the sample from the teacher:
((Ø) = min
E Xy (2)
[d( L (xT), V (x+)))]
(6)
where d(...) is an appropriate distance function that is dif- ferentiable, non-negative, and reflexive. For now, {4) is pre-determined discretization schedule, though we also propose learning the discretizations in Section 3.2. We em- phasize the importance of learning a solver with respect to the global error: although some existing works try to match the teacher solver's trajectory, many teacher trajec- tories contain pathologies that subsequently distilled into the student; see Appendix B.3 for further discussion. While this method, as stated, already improves performance out-of-the-box, we now detail two optimizations that further improve our performance.
3. Learning Diffusion Model Samplers
We detail our strategy for creating DM samplers that pro- duce high-quality samples using small number of NFEs. We exploit the full design space of diffusion model solvers by learning both the coefficients and discretization steps of the sampler, as both necessarily interact with one an- other. We first characterize this design space by providing a general formulation for three general types of diffusion ODE solvers: single-step (SS), linear multi-step (LMS), and predictor-correcto methods (PC). We then describe the objective we minimize directly control the global error. Next, given pre-specified set of discretization steps, we introduce our algorithm for learning only the solver coef- ficients; this uniformly improves performance over hand- crafted solvers for an equivalent number of NFEs. Finally,
1.1.1. TIME-DEPENDENT COEFFICIENTS
Traditional methods for solving ODEs (e.g. Adams- Bashforth or Runge- Kutta) are often defined by constant set of coefficients, regardless of what time step along the ODE they are estimating. While this is not uniformly the case for diffusion ODE solvers, many keep coefficients fixed across steps of solving the reverse- pendix C.2. This fails to fully capture the complexity of
see Ap-
4

---

S4S: Solving for Fast Diffusion Model Solver
Solver Type
($)))
Φ
NFEs per Step
Params.
LMS
Σ
Djirc0 (xt, jt (14
[b,,z}
A(2N + +1 +1-1)/2
j=l
U3atkyi Kj = ca
X +
Σ 0s,, 4:-1 t
(6 agatts Gat)
(k² k: 1)N
k Σ cxis; و ,
LMS PC
(b,,)) + {afa]
k(2N + k)
Table 1. We apply S4S to three types of diffusion ODE solvers; we show their increment (A,), learnable parameters, number of NFEs per step, and total parameter over N 1 steps. By default, use linear multi-step predictor for the PC method, so {af,] refer coefficients during the correction step, and the total set of learnable parameters accounts for the underlying multi-step predictor.
Algorithm S4S
Require: Coefficient parameters Φ, student solver Vs. teacher solver Y distance metric d, and 1: D < XT, V (x(x)) XT NV(0 (2),, X xT) } // Generate data D
2: while not converged do
3: Xy, (x())) ~ 'D
4: L(Ø, x = o (x))), v (xr)) subject to xyy E B(xT, roT)
5: Update & and XG using the corresponding gradients V L(Φ, x))
6: x + +
xx
1|121/ xTl12 r] xy xx // Projected SGD
Ix xxrl2
7: Update D with the new X/7
8: end while
diffusion ODEs: the score network increasingly suffers from prediction eтror as the marginal distribution Ps (Xx) resem- bles Gaussian noise less and less, while estimation error that occurs at a noisy time step propagates through the estimated trajectory differently than a "cleaner" step. Accordingly, as an additional optimization, S4S learns time-dependent coefficients, exemplified by the dependence on the cur- rent iteration i in Table 1. We ablate the design decision to use time-dependent coefficients in Appendix H.2; time- dependent coefficients significantly outperform the use of fixed coefficients.
nonetheless underfit the objective and fail to fully achieve the expected performance improvements.
Instead, similar to Tong et al. (2024), we propose a relax- ation of our training objective that is easier to optimize with limited number of parameters. In particular, rather than forcing the student solver to exactly reproduce the teacher's output for xT, instead only require the existence of an input X7 sufficiently close to Xy (i.e. within bounded radius) such that (x(x)) V (xT). As result, so long as x/ is appropriately close to xy, the average global error of the learned student model can still be quite low, while mitigating the difficulty of the objective. Concretely, our relaxed objective is expressed
3.1.2. RELAXED OBJECTIVE
For each student solver Va the number of both NFEs and learnable parameters is determined by the type of solver, the number of discretization steps, and the step parameter k of the solver, as displayed in Table 1. Accordingly, when the target solver uses few NFEs, the number of learnable parameters may be very low, e.g. 6 parameters for LMS when N = k = 3. This can make optimizing Eq. (6) dif- ficult: indeed, given an initial condition XT, our objective tries to ensure that (x(x)) number of learnable parameters, however, the student solver will almost always produce an output with non-trivial trun- cation error. As a result, though our learned coefficients may be successful at reducing the global error, they might
Lrelax (() = min &
E xm.0(000331
((xT,x())
(7)
J(XT, X4) x,EB,(xT)
min
d(T (x/), V (xx))
where (x) = (x' ||x x 5 rō}
is the L2 ball of radius rō about x. This objective has several appealing prop- erties. First, in Appendix D.2, we empirically verify, similar to Tong et al. (2024), that this objective is easier to solve than our original objective, which we recover when Moreover, under appropriate assumptions on the solver, we can ensure that distribution generated by the learned solver, Po (Xa), and that of the teacher solver, p* (xx). are sufficient close; see Appendix D.I for details. Finally, although we
" (xx). Given the small
O.
5

---

S4S: Solving for Fast Diffusion Model Solver
minimize this objective during training, at inference time, we only use the initial condition Xr ~ Pr (xr) rather than finding and using X ~ (XT) as an initial condition.
we solve the objectives
k arg min
E (0,
[d( V @V V v(xx))))
Φ = arg min
E [d(V te (XT), Wi (xT))]
3.2. S4S-Alt: Coefficients and Time Steps
(0,
(8)
While learning the solver coefficients alone improves the quality of samples, the choice of discretization steps remains crucial for achieving optimal performance. In that vein, we present S4S-Alt, which learns both solver coefficients and discretization steps by using alternating minimization over objectives for the coefficients or the discretization steps.
In the first objective, we learn only Ee using the LD3 ob- jective (Tong et al., 2024) from a student solver with coeffi- cients and time steps initialized Φx-1 and - 1. respec- tively. In the second, we learn Φb from solver initialized at the newly learned time steps 5 and coefficients Φh-11
A natural alternative to this approach would be to optimize the coefficients and time steps simultaneously. However, in our experiments, we found that optimizing both simul- taneously presents several challenges, namely that the op- timization landscape becomes significantly more complex due to the interaction between the solver coefficients and time steps. Additionally, we found that learning both jointly has a greater risk of over-fitting. We found that S4S-Alt performed significantly better in practice, as seen in Table 6.
3.2.1. DISCRETIZATION STEP PARAMETRIZATION
When sampling from DM, the choice of discretization steps determines (1) the expected amount of signal-to-nokes present in an estimated sample, (2) the error present in the score network's 's prediction, and (3) the amount of error propagated by using estimated trajectory points as input to the score network. We take these consequences into account when parametrizing a learned set of discretization steps by separating the learned steps into two parts. First, we use set of time steps, [+ that is parametrized by learnable vector & E IR N+1 used for determining the step size and SNR parameters, thereby accounting for (1). We explicitly parameterize es such that it is a monotonically decreasing sequence of parameters between O and T' i.c
3.3. Implementation Details
Below, we discuss the practical details used for S4S. For ease of notation, we first ground our explanation in the version of S4S that only learns coefficients before discussing details specific to our S4S-Alt. We direct explicit queries about hyperparameters, etc. to Appendix G.2.
: - > tf explicit description of this parametrization. Second, we use modified set of time steps imput to the score network to mitigate (2) and (3). Specifically, we use set of decoupled steps {tg = t + J. as input to the score network, where go E R N-, ; we describe the construction of go in Appendix E.2. Under this parametrization, the update step
> Eyy = 0; see Appendix E.1 for an
Practical Objective. Despite formulating our relaxed ob- jective in Eq. (7), optimizing it in practice is still unclear. To do so, we treat our optimization problem as jointly opti- mizing both & and Xy,, using projected SGD to enforce the constraint that xyp remain close to XT Concretely, this is
of the ststep LMS in Eq. (5) and Table 1 is:
Lrelax (0b, xx)
E
[d V ( (x Ve" (XT))),
(0,670
subj. Xy (x1).
age
0 Σ 10 (x,, 2n -it kinks) N1,
In practice, we use LPIPS as our distance metric, com- mon loss for distillation-base 2022; Song et al., 2023); for other modalities, alternativly appropriate distance metrics should be used. We ablate the decision to use LPIPS in Section 4.2.
0,5_1
methods (Salimans & Ho,
1=0
where h4 = t 1- For simplicity, we denote the col- lection of learnable time parameters as == )). Con- sequently we represent solver with learnable coefficients and time steps as V and its outputs V Vo.=(xT).
Algorithm Details. The algorithm for S4S learning coeffi- cients is displayed in Algorithm 1. First, we collect a dataset from sequence of noise latents used create samples from the teacher solver v (XT). Initially, we use the same ini- tial condition for both the student and teacher solver, i.e. Xy = XT. At each iteration, for a given batch, we compute the loss between the output of our learned solver V (x²) and Th (x7), and use backpropagation to get the gradients of this loss with respect to 6 and x/y. To enforce our con- straint on x/y, we use projected SGD to ensure remains
3.2.2. ALTERNATING OPTIMIZATION
We next consider how to optimize both the solver as well as the discretization schedule. We propose an iterative ap- proach, S4S-Alt, that alternates between optimizing the time steps and the solver coefficients. Formally, at iteration k;
6

---

S4S: Solving for Fast Diffusion Model Solver
inside of (xT); method for applying the gradients, although momentum- based methods work best empirically, Notably, after we update XT. we keep it with its original (xr, V (xx)) pair, and update the dataset with the new noise latent. We also op- timize our computation of the gradient computation graph; see Appendix F.2 for more details.
): for coefficients, we can use arbitrary
Schedule
Method
NFF=4
NFE=6
NFE=8
CIFAR-10
UniPC
50.63
19.47
9.68
UniPC-S4S
44.30
17.80
9.05
EDM
iPNDM
29.50
9.75
5.24
iPNDM-S4S
25.74
8.81
4.98
DPM-v3
34.39
18.44
7.39
UniPC
15.83
3.55
2.87
Initialization. A natural question to consider is how the student ODE solver coefficients may be initialized. Since our approach generally subsumes common diffusion ODE solvers, including the best-performing methods like DPM- Solver++ (Lu al., 2022b), iPNDM (Zhang & Chen, 2023), and UniPC (Zhao al., 2023), can initialize Φ with the same coefficients as these methods. This can be in- terpreted as wrapping one of these traditional solvers in our lightweight approach; in this setting where just coef- ficients are learned, we refer to this as c.g. iPNDM-S4S, Alternatively, we could consider initializing the coefficients according to a Gaussian.
UniPC-S4S
13.46
3.17
2.67
LD3
iPNDM
10.93
5.40
2.75
iPNDM-S4S
9.30
4.76
2.61
DPM-v3
29.86
10.69
3.59
ImageNet
UniPC
53.22
10.97
5.53
UniPC-S4S
45.53
10.09
5.19
L-Unif
iPNDM
36.23
16.15
7.93
iPNDM-S4S
31.81
14.85
7.53
UniPC
11.33
4.74
4.87
We ablate this decision in Ap- pendix 1, I, finding that solver initialization outperforms Gaussian initialization.
UniPC-S4S
10.56
4.54
4.58
LD3
iPNDM
6.45
4.70
4.91
iPNDM-S4S
6.05
4.57
4.68
Algorithms for Learning Coefficients and Time Steps. In practice, when learning both time steps and solver coef- ficients for student solver V d. S4S optimizes an equiv- alent, alternating version of Eq. (9) (and equivalently for jointly learning coefficients); likewise, the pseudocode for doing so is quite similar, which we detail in Appendix F.1. Nonetheless, in practice, learning V generally requires a larger dataset compared to just learning the coefficients, largely attributable larger number of parameters. We ablate performance with dataset size in Appendix H.2.3.
Table 2. FIDs on CIFAR-10 and ImageNet show that common diffusion ODE solvers can be improved by S4S initialized that solver for almost all NFEs and schedules.
per around learnable third-order multi-step versions of the best current ODE solvers: UniPC (Zhao et al., 2023) and iPNDM (Zhang & Chen, 2023). Here, we initialize our stu- dent solver to have the same coefficients as their unlearned counterparts before optimizing our relaxed objective. When possible, we also compare with DPM-Solver-v3 (Zheng et al., 2023), which learns coefficients, but only to attain a guarantee local truncation error. We evaluate our learned solvers seven discretization schedule methods, ranging from common heuristics to modern step-selection methods, with further details in Appendix G. We also characterize the performance of S4S on learnable single-step methods, which can be found in Appendix H.I.
4. Experiments
We evaluate S4S on a number of pre-trained diffusion mod- els trained on common image datasets. We use pixel-space diffusion models for CIFAR-10 (32x32), FFHQ (64x64), and AFHQv2 (64x64), each having an EDM-style back- bone (Karras al., 2022). We also latent diffusion models, including LSUN-Bedroom (256x256) and class- conditional ImageNet (256x256) with guidance scale of 2.0. Finally, we present both qualitative and quantitative results for Stable Diffusion v1.4 at 512x512 pixels with a variety of guidance scales. We provide precise experimental details in Appendix for all sets of experiments, includ- ing choice of teacher solver, dataset size, and selection of noise radius We use the Frechet Inception Distance score (FID) as metric for image quality on all datasets using 30k samples generated from MS-COCO captions for evaluating Stable Diffusion and 50k samples for all other datasets.
Next, we evaluate S4S-Alt against several methods of learn- ing sampler attributes, including AMED-Plugin (Zhou et al., 2024a) and BNS (Shaul et al., 2024b), in sample quality and computational efficiency. We instantiate S4S-Alt as LMS method initialized with iPNDM coefficients and LD3 discretization; this limits the amount of overfitting to the training data due to fewer parameters relative to SS and PC methods. Finally, we ablate key design decisions in S4S.
4.1. Main Results
When used as a wrapper for learning solver coefficients, S4S almost uniformly improves image generation quality
First, we show the benefits of S4S as a standalone wrap-
7

---

S4S: Solving for Fast Diffusion Model Solver
Method
NFE=4
NFF=6
NFE=8
CIFAR-10
Best DPM-v3
17.88
7.32
3.59
2-2
Best Trad. (LD3)
10.93
3.55
2.75
Best S4S
8.25
3.17
2.61
S4S Alt
6.35
2.67
2.39
MS-COCO
(a) PCA Iearned S4S coef- ficients at each discretization step.
DPM-v3
23.90
15,22
12.10
(b) PCA of learned S4S coeffi- cients at each epoch of training,
Best Trad. (LD3)
20.22
12.33
11.30
Best S4S
19.14
11.97
10.82
S4S Alt
16.05
11.17
10.68
Figure 2. PCA of learned S4S coefficients at a each point of the reverse process
b each training epoch; darker points refer to earlier values in the reverse process or training. We initialize S4S coefficients at iPNDM and learn solver with 5 NFEs and order 3. In a, we take the PCA of the combined set of final Iearned coefficients {(b1,s; b2, R= across three training random seeds used. We also include the iPNDM coefficients in the PCA, using total of vectors R b, we concatenate the learned coefficient vectors at the end of each epoch, resulting in vector of dimension 15 for each epoch. again perform PCA on a collection of 16 of these vectors, again including iPNDM as reference point.
Table 3.
S4S-Alt consistently offers significant improvements in FID over best-performing alternatives at given number of NFEs.
CIFAR
MS COCO
Method
NFE
FID
NFE
FID
S4S-Alt
7
2.52
6
11.17
S4S
10
2.18
8
10.84
LD3
10
2.32
8
12.28
DPM-v3
10
2.32
8
12.10
BNS
8
2.73
12
20.67
PDt
8
2.57
ECM
2
2.20
also provide qualitative comparisons in Appendix H.5.
iCT-deep'
1
2.51
Finally, we provide a detailed comparison of S4S-Alt to methods that learn aspects of the solver, as well as training- based distillation methods, in Appendix H.3 and briefly in Table 4. S4S-Alt outperforms the vast majority of learnable solver methods and achieves competitive performance to training- based methods for a fraction of the compute.
Table Number of NFBs required to match/bcat S4S-Al perfor- mance on CIFAR and MS-COCO. i denotes that results were taken from original papers. PD refers Progressive Distillation (Sali- mans & Ho, 2022), ECM to Easy Consistency Models (Geng et al., 2025). iCT-deep to Improved Consistency Training (Song& Dhari- wal, 2024). Red cells are methods that cannot match S4S-Alt in our experiments w/ our NFE settings or in recorded experiments.
4.2. Ablations
Table 5 shows ablation on the solver order in learned LMS models. In both versions of S4S, excessively large order tends to decrease performance; despite setting r proportion- ally to the larger number of parameters, using information from distant time steps hurts output sample quality. We also characterize the importance of our alternating minimization objective for S4S-Alt. As an alternative, we consider learn- ing both the solver coefficients and discretization steps si- multaneously using the same objective; see Appendix H.2.2 for an explicit description. We present our results in Table 6. Finally, we characterize many more design decisions made over our solver; see Appendix H.2.
across datasets, solver types, and discretization methods in the few-NFE regime. Our full results are available in Appendix H.4, while we present a selection of results on CIFAR-10 and ImageNet in Table 2. We observe that the size of the improvement that S4S provides is dependent on the underlying discretization schedule and solver type, and while S4S always improves performance for any dis- cretization schedule, the amount of the improvement varies across different choices of schedule. When we both opti- mize the solver and the schedule, i.e., with S4S-Alt, we obtain significantly greater improvements compared to prior state-of-the-ar We display some of these results in Table 3, where we compare against methods that learn single di- mension of the sampler: the best "traditional" ODE solver using the learned LD3 discretization schedule, the best FID DPM-Solver-v3 across all schedules, and the best S4S solver across all schedules; see Appendix H.4 for the full set of FID values across our experiments. S4S-Alt achieves extremely strong performance relative to simple learned methods. We
5. Conclusion
We introduce $4S, a method for learning DM solvers in the low NFE regime. Our approach matches the output of a teacher solver while simplifying the optimization landscape for learning a student solver. While we achieve superior results, there are nonetheless limitations and opportunities
8

---

S4S: Solving for Fast Diffusion Model Solver
Method
Order
NFF=4
NFF=6
NFF=8
EF is also supported in part by the NSF Graduate Research Fellowship Program. PWK is supported by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and In- formation under the AI Visiting Professorship Programme (award number AIVP-2024-001), and by the A12050 pro- gram at Schmidt Sciences. SC was supported in part by NSF IIS 133133,,,,, NSF CCF-2441635, and the Harvard Dean's Competitive Fund for Promising Scholarship.
14.24
5.45
3,55
S4S
13.94
5.68
3.61
6
6.11
3.89
10.63
4.62
3.15
S4S-Alt 4
10.21
4,40
3.24
4.83
3.42
Baseline
16.68
6.19
3.75
Table Effeect of solver order FID for FFH Both S4S meth- ods are LMS initialized with iPNDM, and standalone S4S uses LD3 schedule. Cells that have worse performance than traditional iPNDM with LD3 are highlighted in red. Excessively high order degrades quality both versions of S4S.
References
Chen, D., Zhou, Z.,, Wang. C.,, Shen, C., and Lyu, S. On the trajectory regularity of ODE-based diffusion sam- pling. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F, (eds.), Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 7905-7934. PMLR, 21-27 Jul 2024.
Method
Order
NFE=4
NFE=6
NFE=8
S4S-AIt
3
6.35
2.67
2.39
Joint Obj.
3
6.81
3.28
2.91
Joint Obj.
Eq NFE
6.42
3.37
3.76
iPNDM-SAN S4S
3
9.30
4.76
2.61
iPNDM
3
10.93
5,40
2.75
Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. R. Sampling is as easy as learning the score: the- ory for diffusion models with minimal data assumptions. International Conference on Learning Representation, 2023.
Table 6. Using joint objective for learning both coefficients and time steps, and the interaction of the joint objective with the order of the underlying LMS method vs. S4S-Alt CIFAR-IO. Eq- NFE denotes having an order equal to the number of NFEs used, e.g. order 6 at 6 NFEs. Orange indicates worse performance than S4S on iPDNM; red indicates worse than traditional iPNDM.
Chi, C., Xu, Z., Feng. S., Cousineau, E.,, Du Y., Burchfiel, B., Tedrake, R., and Song. S. Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robotics Research, 2024.
for future work: 1) we only experimented on ODE solvers, 2) coefficients must be learned for each number of NFEs and cannot be re-used, and 3) we learn dataset-level coeffi cients rather than sample-level coefficients. Our low-NFE performance also lags behind the very best training- based approaches.
Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Diele- man, S., Fergus, R., Sohl-DDccstteinn J., Doucet, A., and Grathwohl, W. S. Reduce, reuse, recycle: Composi- tional generation with energy- based diffusion models and meme. In International conference on machine learning, pp. 8489-8510. PMLR, 2023.
Impact Statement
Geng, Z., Pokle, A., Luo, W., Lin, J., and Kolter, J. Z. Consistency models made easy. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
In terms of broader impact, the techniques in this work may be used to improve samples produced for beneficial pur- poses, such as private data synthesis; moreover, the efficient generation that S4S provides helps reduce the energy us- age in this era of modern machine learning. Nonetheless, these models could also be used to generate highly realistic images, audio, or videos that could be used for harmful purposes. We believe technical innovations such as ours should be balanced with safety measures.
Ho, and Salimans, T. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems,
2020.
Hochbruck, M. and Ostermann, A. Exponential integrators. Acta Numerica, 19:209-286, 2010.
Acknowledgments
SO and EF were supported in part by NSF Awards 2019844, 21 12471, and 2229876, and Microsoft Grant for Customer Experience Innovation. LJR and EF were supported in part by NSF CPS- 184472,, and ONR YIP N00014-20-1-2571.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of biffusion-based generative models. Advances in neural information processing systems, 35: 26565-26577 2022.

---

S4S: Solving for Fast Diffusion Model Solver
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Song, J., Meng. C., and Ermon, S. Denoising diffusion implicit models. International Conference on Learning Representation, 2021a.
Liu, E., Ning, X., Yang, H., and Wang, Y. A unified sam- pling framework for solver searching of diffusion proba- bilistic models. In The Twelfth International Conference on Learning Representations, 2023.
Song, Y. and Dhariwal, P. Improved techniques for train- ing consistency models. International Conference on Learning Representation, 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representation, 2021b.
Liu, L., Ren, Y., Lin, Z., and Zhao, Z. Pseudo numerical methods for diffusion models on manifolds. In Interna- tional Conference on Learning Representations, 2022.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5788 2022a.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. In International Conference on Machine Learning, pp. 3221222211222 1-2225.2 PMLR, 2023.
Tong, V., Liu, A., Hoang, T.-D., Broeck, G. V. d., and Niepert, M. Learning to discretize denoising diffusion odes. arXiv preprint arXiv:2405. 15506, 2024.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01099, 2022b.
Valevski, D.,
Leviathan, Y., Arar, M., and Fruchter, S. Dif- fusion models are real-time game engines, arXiv preprint arXiv:2408. 148-377788.487,, 14837, 2024.
Meng, C.,, Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Watson, D., Ho, J., Norouzi, M., and Chan, W. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint arXiv:2106.1 03802, 2021.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spec- tral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Xu, T., Zhu, Z., Li, J., He, D., Wang, Y., Sun, M., Li, L.., Qin, H., Wang, Y., Liu, J., and Zhang, Y.Q. Con- sisteney model is an effective posterior sample approx- imation for diffusion inverse solvers. arxiv preprint arXiv:2403. 12063, 2024.
Sabour, A., Fidler, S., and Kreis, K. Align your steps: Optimizing sampling schedules in diffusion models. In Proceedings of the st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research. PMLR, 2024.
Xue, S., Liu, Z., Chen, F., Zhang, S., Hu, T., Xie, E., and Li, Z. Accelerating diffusion sampling with optimized time steps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8292- 8301, 2024.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. International Conference on Learning Representation, 2022.
Zhang, G., Kenta, N., and Kleijn, W. B. On accelerating diffusion based sampling process via improved integra- tion approximation. arXiv preprint arXiv:2304. /1328, 2023.
Shaul, N., Perez, J., Chen, R. T., Thabet, A., Pumarola, A., and Lipman, Y. Bespoke solvers for generative flow models. In 12th International Conference on Learning Representations, ICLR 2024, 2024a.
Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. In International Conference on Learning Representations, 2023.
Shaul, N., Singer, U., Chen, R. T., Le, M.. Thabet, A., Pumarola, A., and Lipman, Y. Bespoke non-stationary solvers for fast sampling of diffusion and flow models. In Proceedings of the 41st International Conference on Machine Learning, pp. 44603-44627, 2024b.
Zhao, W., Bai, L., Rao, Y., Zhou, J., and Lu, J. Unipe: A unified predictor predictor-correcto corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36:49842 00049942449869,, 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256 2265.. PMLR, 2015.
Zheng, K., Lu, C., Chen, J., and Zhu, J. Dpm-solver-v3: Im- proved diffusion ode solver with empirical model statis- tics. Advances in Neural Information Processing Systems, 36:55502-555 555422, 2023.
10

---

S4S: Solving for Fast Diffusion Model Solver
Zhou, Z.,, Chen, D., Wang, C., and Chen, C. Fast ode- based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7777-7786, 2024a.
Zhou, Z., Chen, D., Wang, C., Chen, C., and Lyu, S. Simple and fast distillation of diffusion models. Advances in Neu- ral Information Processing Systems, 77:40831-40866 2024b.

---

S4S: Solving for Fast Diffusion Model Solver
A. Comparisons with Existing Works
Here, we provide a detailed discussion of similar works to our method, accentuating limitations existing methods and noting how our approach improves upon them.
A.1. Upper Bounds: Comparison with AYS and DMN
First, we discuss our relationship with Align Your Steps (AYS) (Sabour et al., 2024) and DMN (Xue et al., 2024), two methods for learning optimized discretization schedules for DMs by minimizing upper bounds of various forms of error; however, minimizing these upper bounds provides no guarantee of actually minimizing the true global error. Additionally, because these methods only focus on selecting discretization schedules, they fail to fully explore the full design space of the DM sampler.
DMN In DMN, Xue et al. (2024) minimizes an upper bound for the global error by optimizing only over the discretization schedules without considering the influence of the ODE solver method or the neural network; this bound is constructed solely by the chosen schedules for o and a, that govern the SNR. Moreover, it makes strong assumption that the prediction error of the score network is uniformly bounded by a small constant, which often fails to be the case (Zhang & Chen, 2023).
AYS In AYS, Sabour et al. (2024) constructs an upper bound on the KI. divergence between the true diffusion SDE solution distribution and the observed sampling distribution. They minimize this bound through an expensive Monte Carlo procedure and require bespoke numerical solutions, such as early stopping and a large batch size, to ensure stable optimization. More generally, both methods optimize an upper bound their specific notions of error, which fails to guarantee minimization of the actual global error.
A.2. Local Truncation Error: Comparison with DPM-Solver-v3, GITS, AMED-Plugin, IIA, and Bespoke Solvers
Here, we provide discussion of variety of works, which learn discretization schedules (Chen et al., 2024), solver coefficients (Zheng et al., 2023; Zhang et al., 2023), combination of both (Zhou et al., 2024a; Shaul et al., 2024a) by minimizing various forms of local truncation error. As previously discussed, we emphasize that such an optimization pattern is insufficient in ensuring that the global error is minimized, as well as method-specifie differences or pathologies.
DPM-Solver-v3 DPM-Solver-v3 (Zheng et al., 2023) is descended from a remarkable family of exponential integrator- based work (Lu et al., 2022a: Zheng et al., 2023). Notably, DPM-Solver-v3 computes empirical model statistics, or EMS, that define coefficients that minimize the first-order discretization error produced from a Taylor expansion of their solver formulation. Interestingly, while these methods only minimize the first-order error, they are also used in higher-order versions of DPM-Solver-v3. Crucially, however, the EMS are calculated to ensure local truncation error control and ultimately provide global error control of the form O(h*) given k-th order predictor and maximum step size h. As result, DPM-Sovler-v3 suffers from the same pathologies other traditional solvers that aim to control the local truncation error when the step size becomes large. Additionally, Zheng et al. (2023) only learns the solver coefficients, leaving half of the sampler design space on the table.
GITS Similarly, GITS (Chen et al., 2024), method that uses DP-based search to select and optimized sequence of discretization steps for DM, seeks to minimize the local truncation error of a student sampler. However, as discussed in Section 2.2, minimizing the local truncation error provides no guarantees for bound on the global error, particularly in the low NFE regime; their algorithm reflects as much, as it assumes scaling of the local truncation error in order to obtain an estimate of the global error. Additionally, their method of selecting the discretization steps agnostic to the specific choice of ODE solver used by the student sampler.
AMED-Plugin AMED-Plugin (Zhou et al., 2024a) is a recently proposed approach that learns both coefficients and time step for existing solvers by selecting intermediate time steps within an existing discretization schedule and applying learned scaling factor when using the intermediate point in an ODE solver; they do so by learning an additional "designer" neural network on top of the bottleneck feature extracted from a UNet-based score network. A reasonable interpretation of AMED-Plugin is that it learns half of the time steps used in a sampling procedure that can be used on top of many common solvers; accordingly, it does not take full advantage of the sampler design space, e.g. selecting all solver coefficients and time steps. Moreover, the neural network used in AMED-Plugii is also trained to minimize truncation error by matching
12

---

S4S: Solving for Fast Diffusion Model Solver
teacher trajectories along intermediate points, resulting in the same limitations as in Section 2.2. It also requires longer training time, which is likely attributable to the more expressive number of parameters being learned.
II IA (Zhang et al., 2023) is an approach that learns specific solver coefficients of different traditional solvers by minimizing the MSE between student trajectory, requiring relatively minimal optimization costs. Similar to earlier critiques, matching the teacher trajectory can still learn pathologies along the teacher trajectory that are corrected with the benefit of additional NFEs but are ill-suited for the sutdent solver. Moreover, this approach only learns coefficients, failing to exploit the full design space; as a result, their quantitative performance is not as good as S4S.
Bespoke Solvers Bespoke solver (Shaul et al., 2024a) is a solver distillation method that effectively learns both time steps and coefficients by constructing and minimizing an upper bound for the global error; in practice, this bound essentially just results in minimizing the sum the local truncation error from a teacher solver. As a result, though it makes use of the full sampler design space, also seeks to minimize a sub-optimal objective.
A.3. Minimizing Global Error: Comparison with BNS and LD3
Finally, we discuss two approaches that seek to directly minimize the global error, either by learning discretization steps (Tong et al., 2024) or by learning both time steps and solver coefficients (Shaul et al., 2024b). While both of these objectives are aligned with our approach, they fail achieve optimal performance in particular ways.
BNS Bespoke Non- Non-stationary Solvers (BNS) (Shaul al., 2024b) directly minimizes the global error, in this case PSNR, based solely on the outputs of the student and teacher DM sampler. While this is aligned with our approach, they have three key limitations. First, their solvers, which are essentially learned versions of linear multi-step methods, have maximal order; that is, they allow the earliest predictions of the diffusion model to serve as gradient information even at very late time steps. Essentially, these solvers are -tstep methods that leverage information from the full trajectory. Past work (Zheng et al., 2023) and our own ablations demonstrate that attempting to use methods with too much influence from past steps can result in instability in the ODE trajectories. Second, in the low NFE regime, BNS still has a relatively small number of parameters, which makes their objective difficult to optimize and results in solvers that likely are underfitted; we rectify such issues with our relaxed objective. Third, BNS optimizes all parameters simultancously, which results in a complex optimization landscape irrespective of the whether the student model is adequately parametrized. In contrast, our approach uses alternating minimization to improve the stability of our overall optimization and iteratively solve optimization problems with easier loss landscape.
LD3 LD3 (Tong et al., 2024) uses a gradient-based method for learning discretization schedule that minimizes the global error. Moreover, they also make use of relaxed objective that makes their optimization problem easier when using relatively small number of parameters. However, LD3 similarly fails to make use of the second half or the DM sampler design space, which yields a significant improvement in performance.
B. Local Error Control in ODE Solvers
For completeness, provide some details truncation error control for traditional ODE solver methods; significantly more details can be found in Lu et al. (2022a).
B.1. Taylor Series Derivation
Here, provide brief details of the derivation of the Taylor series and its low-order derivative terms, referenced in Section 2.2. For further details and the most informative description of the relationship of diffusion ODE solvers to the low-order Taylor approximation, see Lu et al. (2022a;b); our explanation is essentially derived from their analysis. Recall that an exact solution for the diffusion ODE in its À parametrization can be given by
dre
X1:
Xt:
Oit:
e E A)dd,
(10)
Ott,1
13

---

S4S: Solving for Fast Diffusion Model Solver
where Xn and (x A) denote the reparametrized forms of x, and (x () in the À domain. To compute X approximate the integral in Eq- (10); do so, consider Taylor expansion of ég (8xx 2)
(A A " (Xxe, )m_1) O((A )m.
Eo (xx, A)
n!
Additionally, define the functions
(x((2) = //' cl1-8)2 8k-1
dó, po(z)=
(k -
which are common terms in exponential integrator methods (Hochbruck & Ostermann, 2010). Note that we have that (0) 1/k! with recurrence relation ( (px(2) 4px (0)) Substituting the Taylor expansion into Eq. (10) and defining h := Ad,- A gives:
Ari
X1, = at Xti-
E co(x,,
att-11
04 041-1-11
Xt--1
atti
e-d
Σ n==
(A A4
)
e0a (Xt,__,) (r...) 00 da
0tt-11
24111
n!
k-1
0t. Xt1-1
Ut:
Pntt (h)e) 1n
- ( +
+ (hht1)
Ctt,
0tt-11
at
n=0
k-1
at
X
ha+1
pnti (h)e² ( + 1 (h+t2)
at:-1
n
Taking (h) h2+ pn+1 (h) yields the expression in Eq. (4). Moreover, note that
en -1
ch h h-1
(Ps(h) e - 122 /2
10 (h)
(2((1)
h
h2
h
and accordingly we factor out an e - to receive
1-1
(6 1)
Σ
(x )t + 01(+1)
ca (h))
Ot111 Xt:-1
where e(h) captures the appropriate coefficient of each e/r) This essentially captures the desired formulation we provide: given ODE solver method approximates the e(m) terms, we capture this approximation using A and ignore the higher-order Taylor terms.
B.2. Regularity Conditions for Local Truncation Error Control
In general. three regularity conditions (Lu et al., 2022a;b; Zheng et al., 2023) are required for ensuring that the local truncation error can be bounded common diffusion ODE solvers:
1. The derivatives e(n) in Eq. (4) exist and are continuous for all 0 2 S k.
2. The score network Eo is Lipschitz in its first parameter
3. The maximum step size hmaz is O(1/N), where N is the number of discretization steps.
These assumptions break down in the following ways:
1. The derivatives of the noise prediction model e(() cannot be guaranteed to exist or be continuous, since neural networks trained with standard optimizers like SGD or Adam do not enforce smoothness constraints on the learned function. While techniques like spectral normalization (Miyato et al., 2018) can help control Lipschitz constants, they do not ensure differentiability.
14

---

S4S: Solving for Fast Diffusion Model Solver
Solver Type
Φ
NFEs per Step
# Params.
LMS
Σ
1i xo (Rt ()
(bg,e)
1
k(2N + 12/2
j=l
Uzattii Kj = X0
x1
a3 + Gss
(bsa agad (33e
k
(k² + 1)N
k Σ xo (e-s)
LMS PC
{byo} + {af.)
4(2N -1-1) k))
Table 7. We apply S4S to three types of diffusion ODE solvers; we show their increment (A,), learnable parameters, number of NFEs per step, and total parameter over N 1 steps. By default, use linear multi-step predictor for the PC method, so {af,) refer coefficients during the correction step, and the total set of learnable parameters accounts for the underlying multi-step predictor.
2. The Lipschitz condition on eg is typically violated in practice, as modern score networks use architectures like U-Nets that can have very large Lipschitz constants. Even with normalization techniques, these constants often scale poorly with network depth and width.
3. The step size restriction hmaz = (1/N) forces a trade-off between computational cost and numerical accuracy that may be unnecessarily conservative many regions of the trajectory where the ODE is well-behaved.
These theoretical limitations help explain why practical implementations often deviate from the idealized analysis. In particular, alternative methods for local truncation error control (Zhang Chen, 2023; Chen al., 2024) can achieve good empirical performance despite violating these assumptions, suggesting that weaker conditions may be sufficient in practice.
B.3. Local Error Control
number of related works (Chen et al., 2024; Zhang al., 2023; Shaul et al., 2024a) recommend matching the trajectory of the teacher solver. In our setting, given an intermediate point x from the teacher solver, this would require optimizing an objective of the form:
min |d(x x()|
for all in [N], either simultaneously iteratively for each Nonetheless, across many teacher trajectories, many solvers have pathological behavior that is corrected in regimes with large numbers of NFEs. For example, Figure 9 in Zhou et al. (2024b) demonstrates such example: as the guidance scale increases, the teacher trajectories become increasingly pathological, but benefit from correcting errors made in early steps. However, by training a student solver with few NFEs to match such a trajectory on overlapping points with the teacher solver, it can learn these same pathologies that are resolved in the teacher by a larger number of NFEs.
C. Generalized Formulation of Diffusion ODE Solvers
C.1. Data Prediction Solver Instantiation
While we focus in the main paper generalized versions of ODE solvers in terms of noise prediction, also provide general expression in terms of the data prediction model. Note that the general form of the exact solution to the diffusion ODE under parametrization by the data prediction model is
I A e (xxs A)dA Tt X. oe o.
X
Therefore, we just need take Taylor approximation of the integral, did in Appendix B.I. This results in general expression for diffusion ODE as
Tt. Xr. ar, -1)A((0) )
X
We display the equivalent definitions for xx in Table
15

---

S4S: Solving for Fast Diffusion Model Solver
C.2. Constant Coefficients in Diffusion ODE Solvers
Coefficients diffusion model solvers are not "inherently" constant; whether they are constant or not depends on the choice of discretization schedule and design decisions in the solver. For example, the iPNDM solver (Zhang & Chen, 2023) demonstrates this principle clearly after its initial warmup period, it settles into using constant coefficients for subsequent steps. This design choice provides computational efficiency while maintaining numerical stability. The solver achieves this by carefully transitioning from variable coefficients during the warmup phase to fixed values that work well across the remaining time steps.
Similarly, DPM-Solverrs (Lu et al., 2022b) multi-step methods can be viewed through the lens of constant coefficients, particularly in their higher-order variants. This perspective helps explain their computational efficiency, the coefficients don't need to be recalculated each step, while still maintaining high-order accuracy in solving the diffusion ODE.
D. Relaxed Objective
D.1. Theoretical Guarantee
Here, we briefly restate the theoretical guarantee for the relaxed objective presented in Eq. (7); this guarantee was provided by Tong et al. (2024).
Theorem D.1. Let V, and V be a teacher and student ODE solver each with noise distribution N' of E Rd, and with, respectively, distributions q and pœ- Assume both D and V & are invertible. Let > if the objective from Equation (6) has an optimal solution o² for r with objective value O, we have
x
DKL(g(x) |1 Ps- (x))
+ +rvd +I E xx
((V)(())) (x C( (x))Il:
(11)
where CVV (x)) log det Ju ( (x))]-
Below, we provide a provide a brief overview of the proof; see Tong et (2024)[A.1] for further details.
Proof. We proceed in several steps:
Step 1: Initial Setup. By assuming the invertibility of the solvers and the loss of Equation (6) having optimal (zero loss and satisfying all roy-ball constraints) solution d", we have for every g(x) exactly one b with Vx- b and exactly one corresponding with v d. (x) a. Moreover, since a is an optimal and therefore feasible solution, we have a E B(b, ror) and thus ||a - bl22 S ror.
Step KI. Divergence Expansion. Using the density function of the normal distribution, we can write:
((b) det
db_((b)
Ex-g(x) log (x(x))
q(x)
]
E [x)g(x) log
db
((aa) det
di (a)
[xx)(x) log(N(()) + log (lact dV, dV)((b)
log((V()) log (dcee
d dV (-)((a) (a)
da [
det
db
Step 3: Normal Distribution Terms. The normal distribution terms can written explicitly:
d
69 200
exp
E Ex-g(x)
log
[Td or v
1 2
exp
Step Clever Substitution. We rewrite a; = + e; for 6; E This gives:
Exwg(x)
(2e,bs + +
1
Exng(x)
Egbs
E Exng(x)
20
o7
207
16

---

S4S: Solving for Fast Diffusion Model Solver
Step 5: Final Bound on First Term. Since ||a b S ror, we have that Σ Σi= (a; b,)2 s r² and with az et
we have Ei c o
Therefore:
1
[r ,²
(()
Exxg(x)
20
201j
Step 6: Independence and Cauchy-Schwarz. The last equality follows from the independence of random variables in the multivariate distribution. Applying the Cauchy-Schwarz Schwarz inequality:
1/27
1
(xx
Σ bbb
I
1
Exx
Σ
2
b
oj
i=1
1/27
1
E [x)
FoT
Σ
b
0f
1/2
Ebw.((0.off)
b
OT
Step 7: Chi-squared Distribution. Since by iid. of:
~ N(O, o the sum of squares follows Chi-squared distribution scaled by
b of x²
1=1
This allows to write:
Σ
d
b
1/2
T
OT
X2 = BoT
Vx3
= =
r
[+1
OT
Step Gautschi's Inequality. Applying Gautschi's inequality:
r
d+11
d+11
<
r
This gives us:
I 1+1
d+1
rv
2
2 = rvdd+1
(
Step 9: Final Bound. Combining all terms from Equation (8), we obtain our final bound:
(x((g(x) p px)(x)) < 22 + +rdd+ 14 gxx (I((((((())))))))))))))) C( (x))
where C(V (x)) log det Jeg. (v (v=-((x) (x))).
Evaluating whether the solver is invertible is difficult to characterize in practice. We note, however, that LMS solvers can at least be represented matrix form, as they scale a linear combination of previous evaluations of the model. Accordingly, only the coefficients are learned, then the LMS solver can made invertible by the transform A + + el for a sufficiently small, non-zero le].
17

---

S4S: Solving for Fast Diffusion Model Solver
Relazed Objective
0.23
0.22
0.21
Cidaa (0)
0.16
0.15
1-00
p-0.1
z-0.2
x-0.5
t-10
Figure Values Creix as we expand As r increases, objective becomes easier to optimize, thereby validating utility of the relaxed objective in making easier optimization problem for learning solver coefficients.
D.2. Easier Objective
We also hope verify that the relaxed objective is indeed easier optimize. We characterize this by running an experiment CIFAR-10: optimize the S4S coefficients initialized iPNDM with logSNR discretization and characterize the empirical loss of Eq. (7) as increases. We affirmatively verify this in Figure
E. Parametrization of Solver Discretization Steps
We parameterize the two versions of our time steps, e and tú, in two distinct stages described below.
E.1. General Time Steps
Given learnable vector RR N+1 we construct each time step of through two-stage process. First, we apply cumulative softmax operation to ensure strict monotonicity:
N
T<
softmax
We then apply a linear rescaling to map these values to the interval [tmins T':
Te( E.ii
(T (min) + I'in
E E.max
Emmin
This construction ensures that t = T of sizes and signal-ton signal-to-noise ratio parameters, as described in the main text.
1 mmnn and ultimately provides the foundation for determining step
E.2. Decoupled Time Steps
Following the parameterization of (t. Ti=0 we now construct the decoupled time steps (te) that are used as input to the score network. Specifically, we define each decoupled time step i5 as
N+1
E + c
where go E N+1 is a learnable offset vector. For numerical stability, we constrain the magnitude the decoupled offsets E Let At, 1 Eς be the gap between consecutive time steps. We define the maximum allowed offset as j min, At, where a > is hyperparameter. The final decoupled time steps are then given by:
if E N0 otherwise
clip(Ef, (-8,8)
18

---

S4S: Solving for Fast Diffusion Model Solver
where clip(ar, [a,b)) b)) clamps the value of to the interval (a, bj. This ensures that the endpoints remain fixed while intermediate steps only shift fraction of the smallest step size.
F. Additional Implementation Details
F.1. Pseudocode for S4S-Alt
Here, we describe the pseudocode for S4S-Alt, which strongly resembles that of S4S. However, we emphasize that we use the same value of r that bounds the allowed deviation of the initial noise condition both optimization objectives. We do this because both objectives must share the same allowable distribution of the noise; otherwise, starting from different initial conditions in different parts of the overall optimization makes learning the effective parameters much more difficult. Additionally, using S4S-Alt generally requires significantly more examples relative to S4S, as we hope to ensure that both sets of parameters do not begin to overfit.
Algorithm 2 S4S-Alt
Require: Coefficient parameters Φ, discretization step parameters E, student solver d. teacher solver w distance metric d, number of alternating steps K, and r.
1: D < {(xx. XT, V0" (xT)) XT NN0, (0, Xy- = xT // Generate data Di.
2: k 1-1
3: for = I,..., K do
4: while converged do
5:
XT, V (xT)) D
6:
((,, = d( (x))) V v((xT) subject to X E B(xT, roT) Update @ and X4 using the corresponding gradients V (()xx))
7:
8:
X/ + Xy + 1[122 xr > r| x1 -x-x
xrlla // Projected SGD
9: Update D with the new x/- end while
10:
11:
while not converged do
12:
(x,. xy, !r* (xT)) D
13:
C(E, x = d( &' (x(x)) subject to X/. E B(x,, roT) Update and xty using the corresponding gradients V (L(
14:
X + XT 1[1x4 l2 2 r]
x Xx
// Projected SGD
[x xrl22
16:
Update D with the new x/7
17: end while
18:
end for
F.2. Efficient Computational Techniques
To optimize memory usage during training, we employ gradient rematerialization when computing V & Rather than storing all intermediate neural network activations, which would incur O(N) memory overhead with respect to the number of parameters, we recompute them on the fly during backpropagation. This approach follows Tong et al. (2024) and Watson al. (2021), trading increased computation time for reduced memory requirements. Specifically, we rematerialize calls to the pretrained score network eg while maintaining the chain of denoised states in memory, allowing our method to scale large diffusion architectures while maintaining reasonable batch sizes.
G. Experiment Details
G.1. Discretization Heuristies and Methods
We use four time discretization heuristics and three methods for adaptively selecting the discretization steps. Here, we consider time interval from T to e over which the ODE is solved with + total time steps; here, solving the ODE to rather than 0 helps with numerical stability.
19

---

S4S: Solving for Fast Diffusion Model Solver
G.1.1. DISCRETIZATION HEURISTICS
Time Uniform and Time Quadratic Discretization In the Time Uniform discretization schedule, we split the interval [T] uniformly; this gives discretization schedule:
n (c- () T) =
f ri E [ Alternatively, the Time Quadratic schedule assigns each time step as
12
t = T+
+
(c c--18 T).
N
These schedules are popular for variance preserving-style DMs (Ho al., 2020; Song et al., 2021a; Lu et al., 2022a).
dt and creating discretization Time EDM Discretization Karras al. (2022) propose a change of variables to K2 az schedule according to
tn = te ((w)/
1/0
(N! (N N1. )²
N
where tx is the inverse of + ni, which exists as Ki is strictly monotone by the construction of ot, at.
Time log-SNR Discretization Alternatively, Lu al. (2022a;b) propose a change of variables d log(a+//:))
log-SNR domain and discretizing uniformly over the interval, i.e.
n In = (x +
(Ae (Ac- AT))
N
where xx is the inverse mapping of A,, which again exists because of striet monotonicity.
G.1.2. DISCRETIZATION SCHEDULE SELECTION METHODS
DMN DMN (Xue et al., 2024) constructs optimization problem that creates upper bound on the global error. Concretely, they model sequentially solving the diffusion ODE in terms of Lagrange approximations, construct an upper bound of the error on the assumption that the score network prediction error is uniformly upper bounded by a constant, and select a sequence of X; that minimizes the derived upper bound.
GITS GITS (Chen et al., 2024) a method that uses DP-based search to select an optimized sequence of discretization steps for DM that minimizes the deviation the diffusion ODE. They do so by calculating the local error incurred from estimating the next time step from the current step 4-1 cost matrix of all pair-wise costs is calculated, they then use DP algorithm to select the lowest-cost sequence of steps given number of NFEs. Intuitively, this approach seeks to take steps that are relatively large in regions of low curvature and smaller steps in regions with high curvature where the discretization error might be high.
finely discretized search space of possible time steps. Once
LD3 LD3(Tong et al., 2024) seeks to learn a sequence of coefficients using the same parameterization as in Appendix E. They similarly try to minimize an objective over the global discretization error, often LPIPS.
G.2. Practical Implementation
Here, we discuss important practical details that we use for both S4S and S4S-Alt. Most crucial is our choice of r when optimizing our relaxed objective both S4S and S4S-Alt. Let m denote the total number of parameters learned in the student solver. Then both S4S and S4S-Alt, we set r x -,-7-- This helps balance the solver's ability to learn the relaxed objective with the number of parameters that it has available.
In practice, for CIFAR-10, 10, FFHQ, and AFHQv2, 20; when learning coefficients and time steps in S4S-Alt, we generally use 1400 samples as training data with batch size of 40. We use 200 samples and 400 samples as validation data set, respectively. For latent DMs, we use 600 samples for learning $4S with a batch size of 20 using gradient accumulation, and use a dataset of 1000 samples with batch size of 40 for A4t-Alt. We again use 200 samples and 400 samples as a validation data set, respectively. both settings, we run S4S for 10 epochs, and S4 for K=8 alternating steps.
700 samples for learning coefficients in S4S with batch size of
20

---

S4S: Solving for Fast Diffusion Model Solver
Method
Order
NFE=3
NFF=4
NFF=6
NFE=8
DPM-Solver (2S)
2
239.41
65.24
28,06
3PM--olver-S4. (2S)
2
66.82
34.91
24.73
DPM (3S)
3
89.75
42.02
iPNDM-S4S (3M)
3
48.19
21.58
8.91
4.33
Table 8. FID of SS methods for S4S initialized at DPM-Solver. Although DPM-Solver-S4s achieves significant gains in FID, especially relative to its unlcarned counterpart, it lags hehind the simpler and much easier to optimize LMM methods.
For teacher solvers, in general we follow Tong et al. (2024) and select the best-performing solver at 20 NFE. This is UniPC with 20 NFE and logSNR discretization for CIFAR-10, FFHQ, and AFHQv2; UniPC with 20 NFE and time uniform discretization for LSUN Bedroom, UniPC with 10 NFE and time uniform discretization for Imagenet, and UniPC with GITS discretization at 10 steps for MS-COCO.
H. Additional Results
H.1. Single-Step Solvers
While in the main text mainly focus LMS methods, we also consider SS solver methods, in particular focusing on DPM-Solver (Lu et al., 2022a). In particular, we consider learnable equivalents of DPM-Solver (2S), second-order method which uses single intermediate step as well as _,_, to estimate x and DPM-Solver (3S), which uses two intermediate steps ún and U2 and therefore a third-order method. Note that while the practical algorithmic approach for learning the SS coefficients is the same as that in the LMS setting, there are significantly more parameters that can be learned as compared to LMS or even PC methods. Consequently, the allowable radius r of our relaxed objective is much smaller than its LMS counterparts.
Table 8 demonstrates results FFHQ using the logSNR discretization schedule. We compare against iPNDM-S44 as a baseline for LMS methods as well as to traditional DPM-Solver (2S). Here, we find that S4S similarly leads to sigmificant gains for SS solvers, in fact even larger than the gains seen for LMS solvers. Nonetheless, despite the significant improvements attained by learning the solver coefficients, SS methods still lag behind their LMS counterparts. Intuitively, this is because SS methods have significantly more parameters to optimize. If r is not chosen properly, then there is a significant chance that S4S overfits to the training dataset but fails to generalize well to the original noise distribution. Moreover, SS methods suffer from the fact that their effective step size is larger than that of LMS methods, i.e. for equal number of NFEs, the step size of k-step LMS method is 1/k the step size of the k-step SS method. As result, for the core remaining parts of our experiments, we focus on LMS methods.
H.2. Additional Ablations
We ablate several of the design decisions in approach in Table 9, Specifically, we characterize the importance of time-dependent coefficients, the choice of LPIPS as our distance metric, and the use of the relaxed objective. We find that time-dependent coefficients significantly improves the performance of S4S and S4S-Alt; this is somewhat expected, since using a fixed set of coefficients for several iterations significantly decreases the number of learnable parameters. Additionally, we find that we still attain strong performance when using the Le loss in lieu of LPIPS. Finally, using our relaxed objective greatly improves performance, particularly in S4S with few NFEs, though with more NFEs the benefit decays as the optimization problem becomes less underparametrized.
1.2.1. $4S INITIALIZATION
A natural question to consider is the importance of the initialization heuristic used for $4S. Here, we consider the results of initializing an LMS method according to a standard Gaussian. We evaluate this initialization on CIFAR-10 and FFHQ with the loRSNR discretization schedule; Table 10 contains our results for this evaluation. Although S4S initialized with standard Gaussian coefficients achieves meaningful improvements, it is nonetheless outperformed by initializing at existing solver methods.
21

---

S4S: Solving for Fast Diffusion Model Solver
Setting
NFF=4
NFF=5
Full
19.39
Glob. Ø
==
19.62
10.17
L2
33.25
15.13
Full 33.34
19.39
Time Dep. Φ
r ==
33.75
19.50
L2
56.49
21.65
Full 27.99
13.32
Glob. Φ
30.52
14.20
1 26.01
16.33
S4S-Alt
Full
30.94
16.76
Time Dep.
31.04
17.64
L
38.73
21.97
Table Ablations of model design decisions on CIFAR-10. To show the largest range of effects as a result of S4S, we use the EDM discretization schedule.
Dataset
Method
NFE=3
NFE=4
NFE=5
NFE=6
Gaussian-SA4 (3M)
91.84
42.17
25.61
11.93
CIFAR-I0
iPNDM-S4S (3M)
75.88
30.12
17.97
10.61
SSM- (3M)
93.58
40.18
22.21
11.04
Gaussian-Sal (3M)
81.44
44.91
24.83
15.01
FFHQ
iPNDM-S4S (3M)
76.81
36.23
24.16
16.15
DPM-Solver-+4-S4S (3M)
86.39
45.89
22.52
13.78
Table 10. FID of LMS methods initialized with standard Gaussian coefficients and optimized using S4S compared against initialization iPNDM or DPM-Solver++. We use the logSNR discretization heuristic for all samples. ODE solvers, but nonetheless improves less than its solver-initialized counterparts.
S4S outperforms traditional
H.2.2. JOINT OPTIMIZATION OBJECTIVE AND DETAILS
Below, we describe the optimization objective and implementation details for learning the joint optimization objective, which learns both the solver coefficients and the time steps simultancously. The pseudocode is essentially restatement of that of S4S, but propagating the gradients to both sets of learnable coefficients. We use the same batch size
H.2.3. TRAINING DATASET SIZE
We also ablate the significance of the training dataset size in S4S-Alt. We display these results for CIFAR-10 with NFEs in Figure 4.
H.2.4. ENSURING CONSISTENCY IN SINGLE- STEP SOLVERS
Although in general we abandon the notion of maintaining notions of local error control in our diffusion solvers, we consider an additional ablation for enforcing consistency in single-step solvers. That is, we ablate requiring the b in single-step solvers sum to for every i. We display these results in Table 11 rather than consistency resulting in better global error, it in fact worsens our gobal error performance.
H.3. Comparisons with Existing Learning-Based Methods
We continue our comparison of S4S and S4S-Alt against alternative learning-basee methods. Table 12, we characterize the computational requirements needed for each method for learning the solver, as well as its corresponding FID and number of NFEs needed beat S4S-Alt.
22

---

S4S: Solving for Fast Diffusion Model Solver
Algorithm 3 Joint Optimization Algorithm
Require: Coefficient parameters Φ. discretization step parameters &, student solver Va,z, teacher solver v distance
metric d, and r.
1: D < {(x/:xx; Xr, (xx))) xx N((0, = =xr // Generate data 'D
2: while not converged do
3: Xx, (xr)) ~
4: L( E, x = a( (xy),, (xT)) subject to X4 rox),
5: Update Φ B, and x using the corresponding gradients V B,x4)
6: x
x +
xT
111x7 xrl12 r|
x xr
Xy Xr // Projected SGD
7: Update with the new x
8: end while
186
12:00
1750
2006
Figure 4. FID vs. Training Dataset Size in S4S-Alt.
CIFAR
MS-COCO
Method
NFE
FID
GPU Type
Time
NFE
FID
GPU Type
Time
S4S-Alt
7
2.52
A100
hour
6
11.17
A100
4.2 hours
S4S
10
2.18
A100
hour
10.84
A100
1.4 hours
LD3
10
2.32
A100
hour
8
12.28
A100
hour
DPM-v3
10
2.32
A40
28 hours
8
12.10
A40
88 hours
BNS
2.73
12
20.67
PD
2.57
TPU
192 hours
ECM
A100
hours
iCT-ddepp
2.51
Table 12. Number of NFEs required match/beat S4S-Alt performance CIFAR and MS-COCO. denotes that results taken from original papers. PD refers to Progressive Distillation (Salimans & Ho, 2022), ECM to Easy Consistency Models (Geng et al., 2025), iCT-deep Improved Consisteney Training (Song & Dhariwal, 2024). Red cells are methods that cannot match S4S-Alt in experiments w/ our NEE settings or in recorded experiments.
23

---

S4S: Solving for Fast Diffusion Model Solver
Method
Order
NFE=4
NFE=6
NFF=8
DPM- (2S)
66.82
34.91
24.73
Consistent DPM-Solver-S4S (2S)
75.82
39.14
31.69
Table 11. FID of SS methods initialized at IPM--olver-Se on FFHQ with logSNR discretization. Enforcing consistency in the single-step model decreases performance rather than achieving better global error.
24

---

S4S: Solving for Fast Diffusion Model Solver
H.4. Full FID Tables
Schedule
Solver
4
10
DPM-Solver++ (3M)
82.45
37.52
30,08
18.40
12.31
8.95
7.40
3.69
DoMe-Sblvee+-4S (3M)
75.43
34.48
28.24
17.55
11.75
8.66
7.06
3.51
DMN
IPNDM (3M)
76.99
33.13
26.
16.00
10.20
10.19
8.84
3.56
iPDDM (3M)
30.58
UniPC (3M)
70.52
30.32
23.04
14.46
8.55
6.78
5.15
3.12
UniPC-S4S (3M)
63.84
28.43
21.66
13,88
8.24
6.53
4.84
2.98
DPM-Solvar++ (3M)
43,47
19.52
13.36
9.67
5.08
4.20
OPM-Solver++-445 (3M)
39.90
18.32
12.55
9.11
7.61
6.37
4.86
3.96
Time EDM
iPNDM (3M)
38.33
15.30
8.80
6.24
4.52
3.85
3.33
3.04
iPNDM-S4S (3M)
35.56
14.23
8.32
5.97
4.37
3.77
3.12
2.88
UniPC (3M)
44.77
23.55
15.83
10.30
8.46
7.83
6.78
6.38
UniPC-S4S (3M)
41.48
21.82
14.73
9.68
8.12
7.52
6.47
6.06
DPM-Solvert+ (3M)
30.74
13.57
SPM-Solva*+--44
28.20
16.41
12.74
9.34
3.42
GITS
iPNDM (3M)
26.55
13.88
9.60
6.
4.85
3.72
3.43
3.02
iPNDM-SS S44 (3M)
24.36
12.75
9.12
5.83
4.66
3.58
3.26
2.90
UniPC (3M)
25.14
12.63
9.64
7.27
4.75
4.25
3.27
3.04
UniPC-S4S (3M)
23.36
11.56
9.13
6.85
4.55
4.08
3.13
2.95
DPM-Solver++ (3M)
24.11
13.95
7.46
3.04
DPM-Solvert+-S5 (3M)
12.58
LD3
IPNDM (3M)
23.64
9.06
5.00
3.44
2.78
2.87
2.85
2.62
iPNDM-S45 (3M)
20.65
8.25
4.61
3.21
2.61
2.76
2.71
2.51
UniPC (3M)
22.02
10.84
6.10
3.65
3.44
3.32
2.44
2.87
UnIPC-S4S (3M)
19.38
9.69
5.61
3.40
3.27
3.19
2.32
2.69
DPM-Solver++ (3M)
60.83
27.58
17.92
10.72
6.14
4.31
3.63
3.15
SPM-Soleer++54S (3M)
55.88
2.99
Time LogSNR
iPNDM (3M)
$2.63
22.99
15.58
9.45
5.92
4,51
3.71
3.14
IPNDM-S4S (3M)
48.19
21.58
14.57
8.91
5.64
4,33
3.48
2.98
UniPC (3M)
94.93
33.70
12.95
8.30
5.12
4.47
3.80
UniPC: $4S (3M)
88.13
31.23
12.18
7.91
4.85
4.47
4.26
3.62
DPM DPM-Solvert+ Solvert+ (3M)
113.09
68.88
42.36
30.99
24.82
21.04
18.66
16.93
DPM-SSbees++S45 (3M)
103.66
63.86
39.78
29.43
23.66
20.54
17.70
16.07
Time Quadratic
iPNDM (3M)
102.48
53.71
32.09
23.86
20.36
18.22
16,62
15.23
iPNDM S4S (3M)
94.08
49.39
29.95
22.56
19.65
17.71
15.57
14.32
UniPC
66.50
41.62
30.69
18.20
16.54
UniPC-845 (3M)
101.64
62.03
39.30
29.17
23.63
19.89
17.05
15.77
DPM-Solver++ (3M)
169.39
153.47
143.52
134,39
125. 18
115.83
106.83
98.
DPM- So5 (3M)
155.10
143.47
134.98
125.98
120.75
111.54
101.13
92.01
Time Uniform
iNDMM (3M)
178.95
159.28
139.32
124.94
113.44
102.81
92.46
82.91
iPNDM-S45 (3M)
163.79
146.77
129.81
117.56
107.45
99.27
87.04
77.95
UniPC (3M)
153.52
143.45
UniPC-S4S (3M)
156.96
142.90
S4S-Alt
14.71
6.52
3.89
2.70
2.56
2.29
2.18
2.18
Table 13. FID scores on AFHQ-v2 64x64. Numbers in column headers indicate NFE counts. Bold: best within schedule; shaded: best overall.
25

---

S4S: Solving for Fast Diffusion Model Solver
Schedule
Solver
8
9
10
DPM-Solver++ (3M)
83.73
39.32
22.89
12.38
7.23
7,00
5.20
2.09
DPM Solvert+-545 (3M)
70.89
34.00
20.53
11,30
6.63
6.71
4.98
2.53
(3M)
5.77
7.59
5.85
3.17
DMN
IPNDM-S48 (3M)
50.05
24.21
14.99
8.35
5.37
7.20
5.57
3.02
UniPC: (3M)
66.45
26.33
12.95
8.11
4.96
5.79
4.01
2.38
UnIPC-S4S (3M)
56.44
23.07
11.63
7.44
4.66
5.53
3.79
2,26
DPM-Solvve--3 (3M)
58.48
17.88
12.31
7.32
4.86
4.72
3.49
2.32
DPM-Solver++ (3M)
70.06
50.40
32.01
18.41
11.58
8.39
6.48
5.18
DPM-SSalver+--555 (3M)
59.61
10.87
6.07
4.96
iNDDM (3M)
48.02
29.50
16.57
9.75
6.93
5.24
4.34
3.70
Time
IPNDM-54S (3M)
41.27
25.74
14.72
8.81
6.35
4.98
4.07
3.47
UniPC
57.85
50.63
34.27
12.65
9.68
7.84
6.16
UniPC-SAS (3M)
48.40
44.30
30.60
17.80
11.62
9.05
7.46
5.86
17MM--Slver-3311 (3M)
44.64
34.39
33.20
18.44
10.50
7.39
5.91
4.72
DPM-Solver++ (3M)
70.47
31.23
17.19
10.76
7.79
5.63
3.97
3.52
Sole-olver+ (3M)
60.14
26.75
15.41
9.70
7.20
5.29
3.72
3.37
iPNDM (3M)
43.91
10.83
6.97
5.80
4.30
3.10
2.78
GITS
(3M)
37.75
4.04
2.62
UniPC (3M)
53.43
21.93
15.40
10:47
7.88
5.69
4.41
3.70
UniPC-S4S (3M)
45.12
18.94
13.71
9.53
7.39
5.42
4.22
3.50
DPM Solver (3M)
60.14
24.46
16.15
11.06
8.20
5,90
3.88
2.99
DPM Solver++ (3M)
33.38
27.08
12.42
9.24
4.40
4.00
3.87
3.33
UPM-Solver+++-4S (3M)
27.73
22.85
10.81
8.14
4.08
3.74
3.65
3.16
iPNDM (3M)
32.64
10.93
5.64
5.40
5.36
2.75
3.79
2.32
iPNDM S4S (3M)
26.39
9.30
4.84
4.76
4.90
2.61
3.59
2.18
UniPC
32.62
4.67
2.87
3.30
2.73
UniPC-S4S (3M)
26.63
13.46
11.35
3.17
4.22
2.67
3.09
2.56
DPM Solver (3M)
84.42
29.86
14.83
10.69
5.51
3.59
2.78
2.56
DPM- (3M)
110.06
46.49
24.98
12.06
6.79
4.56
3.43
3.00
DPM Sole--Slver++-AS (3M)
93.58
40.18
22.21
11.04
6.34
4.34
3.25
2.85
iPDDM (3M)
88.39
34.88
20.49
11.61
7.50
5.53
4.24
3.58
Time LogSNR
iPNDM (3M)
5.24
3.43
UniPC (3M)
155.31
43.93
23.90
12.98
6.54
4.38
3.48
3.07
UniPC-S4S (3M)
133.37
38.05
21.42
11.69
6.02
4.10
3.33
2.94
DPM DPM-Sollve-v3 (3M)
84.49
29.87
14.85
10.71
5.52
3.59
2.78
2.56
DPM-Solver++ (3M)
223.06
170.85
124.72
91.51
69.90
54.84
44.38
37.00
DPM Salvert+ (3M)
188.15
149.60
110.20
82,92
64.23
51.47
41.63
34.91
iPNDM (3M)
199.73
139.72
96.56
68.68
52.22
37.64
27.37
23.28
Time Quadratic
iPNDM- (3M)
167.85
121.49
85.83
62.41
48.65
35.96
25.74
22.26
UniPC (3M)
220.29
117.49
85.38
51.80
42.29
35.34
UnIPC-S4S (3M)
187.89
141.66
78.11
61.21
49.21
39.65
33.29
DPM-Solver-r3 (3M)
299.55
249.40
188.77
129.51
90.93
65.13
50.30
41.07
DPM-Salvert+ (3M)
249.52
237.94
227.53
217.62
208.19
DPM Sulvert+ S4S (3M)
259.33
244.55
234.70
225.57
219.83
218.06
205.34
197.85
iPNDM (3M)
287.80
266.13
242.76
229 10
216.95
205.06
194.64
185.30
Time Uniform
PNDM-SAS (3M)
243.96
227.78
215.84
208.27
199.97
192.03
186.11
175.26
UniPC (3M)
304.86
282.77
263.43
249 18
237.56
226.95
216.85
207.23
UniPC S4S (3M)
255.32
246.58
235.16
226.52
222.79
214.66
203.80
194.92
DPM-Salver-3 (3M)
313.89
312.46
304.90
294.57
285.39
S4S-Alt
16.95
6.35
3.73
2.67
2.52
2.39
2.31
2.18
Table 14. FID scores CIFAR-I0 10 32x32. Numbers in column headers indicate NFE counts. Bold: best within schedule; shaded: best overall.
26

---

S4S: Solving for Fast Diffusion Model Solver
Schedule
Solver
8
9
10
DPM-Solver++ (3M)
82.21
40.23
26.30
14.74
9.78
10.10
8.63
4,63
DPM Solvert+-545 (3M)
68.64
34.69
23.55
13.23
9.05
9.52
8.09
4.44
10.95
5.29
iPNDM-S4S (MM)
52.45
27.51
18.83
11:03
8.07
10.29
9.20
5.06
UniPC: (3M)
65.07
25.80
13.32
9.48
7.27
6.78
5.57
3,66
UnIPC-S4S (3M)
54.58
22.32
11.79
8.57
6.69
6.39
5.30
3,50
DPM-Sulvert+ (3M)
62.58
39.52
23.66
15.16
11.10
9.61
9.06
6,99
DPM-Solvor++-S45 (3M)
53.21
34.47
20.93
13.75
10.20
9.07
8.52
6,62
(3M)
5.72
iPNDM-S4S (3M)
5.42
4.75
UniPC (3M)
59.88
47.73
26.54
15.07
11.20
11.63
10.91
8.89
UniPC S4S (3M)
50.37
41.83
23.43
13.56
10.36
11.04
10.28
8.48
SPM--elleere+ (3M)
53,42
29.07
17.54
12.74
9.74
7.70
6.30
4.99
DPM Sulve-Stvee+++ASS (3M)
45.10
25.00
15.72
11.41
9.00
7.30
5.98
4.77
GITS
IPNDM (3M)
33.09
18.04
12.91
9.38
7.57
5.76
4.76
3.97
iPNDM-545 (3M)
3.76
UniPC (3M)
43.63
21.38
14.34
12:22
9.95
8.02
6.20
4.46
UniPC-S4S (3M)
36.59
18.30
12.68
11.19
9.20
7.56
5.83
4.24
DPM-Solver++
28.67
7.70
5.01
4.21
3.56
3.41
DPM- (3M)
41.43
23.92
12.32
6.95
4.57
3.94
3.42
3.24
LD3
iPNDM (3M)
43.05
16.68
9.41
6.19
4.62
3.75
3.41
3.13
iPNDM-S4S (3M)
35.12
14.24
8.14
5.45
4.20
3.55
3.23
2.97
UniPC (3M)
40.27
18.04
10.85
8.04
4.33
3.46
3.53
3.30
UniPC-S4S S4S (3M)
33.33
15.09
9.34
7.09
3.91
3.23
3.34
3.12
DPM Solvert+ (3M)
4.77
4.12
DPM- (3M)
74.18
12.66
5.72
4.56
3.88
Time LogSNR
iPNDM (3M)
76.81
36.23
24.16
16.
11.07
7.93
6.27
5.30
iPNDM S4S (3M)
65.27
31.81
21.33
14.85
10.38
7.53
5.89
5.07
UniPC (3M)
126.00
53.22
20.02
10.97
6.97
5.53
4.53
3.89
UniPC-S4S (3M)
105.40
45.53
17.73
10.09
6.48
5.19
4.28
3.72
DPM-Solver++ (3M)
131.14
94.28
44.74
37.45
32.26
28.
Solvertt (3M)
Time Quadratic
1NDM (3M)
105.90
71.59
51.72
39.21
31.40
26.52
23.42
21.30
iPNDM-S4S (GM)
90.71
62.94
45.63
35.77
28.99
25.12
22.34
20.32
UniPC (3M)
128.38
89.94
66.09
51.36
41.54
34.76
29.98
26.55
UnIPC-S4S (3M)
107.65
77.61
$8.90
46.69
38.75
32.51
28.48
25.04
DPM--Sllvert (3M)
195.55
179.13
165.48
153.52
142.81
133.112
124.36
116.46
DPM-Solver++-S45 (3M)
167
157.05
147.30
141.23
134.10
127.40
117.91
Time Uniform
iPNDM (3M)
177.99
160.85
146.31
133.60
122.28
112.25
103.46
95.78
PNDM-S4S (3M)
152.72
140.92
129.90
119.79
114.77
105.73
98.98
90.75
UniPC (3M)
195.24
178.73
165.03
152.95
142.09
132.28
123,39
115.35
UniPC: S4S (3M)
164.34
154.02
146.30
138.21
133.36
125.43
117.83
109.96
S4S-Alt
19.86
10.63
6.25
4.62
3.45
3.15
3,00
2.91
Table 15. FID scores on FFHQ 64x64, Numbers in column headers indicate NFE counts. Bold: best within schedule; shaded: best overall.
27

---

S4S: Solving for Fast Diffusion Model Solver
Schedule
Solver
4
5
6
DPM-Sober++ (3M)
58.17
20.03
7.14
5.04
4.69
4.53
DPM Solvertt (3M)
55.18
19.40
6.87
4,88
4.63
4.28
(3M)
4.74
4.38
IPNDM-S4S (3M)
23.72
7.84
4.63
4.25
4.18
UniPC (3M)
48.95
15.05
5,46
4,60
4.83
4.52
UnIPC-S4S (3M)
46.99
14.36
5.33
4.43
4.80
4.31
DPM-Solvert+ (3M)
123.72
60.70
18.40
7.71
5.42
5.29
DPM-Sober++-S45 (5M)
117.96
57.69
17.77
7.42
5.36
5.04
Time
(3M)
33.84
7.25
IPNDM-SAS (3M)
85.46
32.50
UniPC (3M)
121.64
57.32
16.05
6.92
5.45
5.47
UniPC. (3M)
117.82
55.51
15.53
6.72
5.29
5.11
DPM-Solvers+ (3M)
99.24
39.63
28.21
15.79
5.20
DPM SPRM-Solver++-oe S4S (3M)
95.26
37.86
27.17
15.35
6.84
4,96
GITS
iPNDM (3M)
69.12
22.22
20.73
11.79
4.51
IPNDM-S4S (3M)
20.07
UniPC (3M)
85.37
24.59
16.08
8.68
4.93
4.33
UniPC-S4S (3M)
82.13
23.61
15.42
8.53
4.79
4.11
DPM-Solvar+
52.28
6.81
4.89
4.91
DPM-Solver++S48 (3M)
48.04
16.69
6.43
4.76
4.67
4.61
LD3
iPNDM (3M)
17.93
6.45
4.86
4.70
4.73
4,91
IPNDM-S4S (3M)
16.48
6.05
4.68
4.57
4.68
UniPC (3M)
43.25
11.33
5.25
4.74
4.79
4.87
UniPC-S4 S4S (3M)
40.24
10.56
5.05
4.54
4.71
4.58
Solverot (3M)
6.32
DPM-Solvar++-4S (3M)
14.00
6.18
Time LogSNR
iPNDM (3M)
93.77
38.81
14.79
7.70
5.61
4.85
iPNDM S4S (3M)
89.31
36.99
14.43
7.53
5.55
4.62
UniPC (3M)
109.14
50.60
12.29
6.40
5.78
5.11
UnIPC-S4S (3M)
103.63
48.95
11.94
6,18
5.67
4.89
DPM-Solver++ (3M)
91.57
40.27
17.77
8.51
4.86
SPPM-Solvert--Be (3M)
Time Quadratic
iNDMM (3M)
63.67
22.65
11.32
6.60
5.02
4.48
iPNDM-545 (3M)
60.36
21.87
10.98
6.39
4.91
4.28
UniPC: (3M)
82.66
28.84
11.06
5.73
4.65
4.37
UnIPC-S4S (3M)
78.47
27.66
10.77
5.61
4.62
4.09
DPM-Solvertt (3M)
68.92
26.34
9.95
6.12
5.27
5.06
DPM-Solver++S45 (3M)
65.90
25.21
9.71
5.99
4.81
Time Uniform
iPNDM (3M)
32.79
8.63
5.23
4.67
4.66
4.69
PNDMMSSS (3M)
31.58
8.29
5.13
4.52
4.61
4.50
UniPC (3M)
63.78
20.14
7.58
5.34
5.06
5.02
UniPC: S4S (3M)
61.41
19.50
7.27
5.20
4.91
4.80
S4S-Alt
13.26
5.13
4.30
4.09
4.06
4.06
Table 16. FID scores on ImageNet 256x256. Numbers in column headers indicate NFE counts. Bold: best within schedule; shaded: best overall.
28

---

S4S: Solving for Fast Diffusion Model Solver
Schedule
Solver
DPM-Soher++ (3M)
136.53
79.29
39,67
22.66
17.18
15.19
DPM-Solver++-S55 (IM)
116.46
71.36
35.82
21.42
16.56
14.48
IPNDM (3M)
DMN
IPNDM-84S (3M)
66.77
40.05
27.40
19.24
17.17
17.70
UniPC (3M)
126.55
68.04
31.75
19.46
15.27
14.81
UnIPC-S4S (3M)
110.11
60.68
29.18
18.40
14.57
13.90
DPM-Solver-v3 (3M)
96.83
42.95
19.82
12.81
11,10
12.59
DPM-Solver++ (3M)
213.95
141.37
80.75
46.62
30.78
22.73
DPM-Solver++-SS (MM)
iPNDM (3M)
126.55
88.04
52.94
37.81
31.41
26.39
EDM
IPNDM-S4S (3M)
109.90
77.86
48.79
35.27
30.26
24.78
UniPC (3M)
135.41
75.79
44.78
UniPC-S4S (3M)
183.16
121.59
70.02
42.48
28.73
21.10
DPM-Solver-v3 (3M)
159.55
114.26
43.56
23.64
19.59
15.68
DPM-Solver++ (3M)
169.26
121.24
90.64
72.89
64.37
61.41
DPM:Soler+++S45 (7M)
148.05
108.86
83.25
67.94
61.69
58.83
iPNDM (3M)
126.10
109.43
94.83
75.52
64.55
GITS
S4S (3M)
UniPC (3M)
150.08
103.39
86.62
71.78
61,22
56.08
UniPC-S4S (3M)
129.69
91.15
79.82
67.24
59.50
53.15
DPM.Solver-vs (3M)
166.97
129.41
92.66
64.21
50.95
47.19
DPM-Solveré+ (3M)
144.81
77.15
42,05
23.09
15.62
12.45
DPM-Solvar++-S0 (3M)
121.66
66.97
38.19
21.57
15.11
11.72
iPNDM (3M)
73.94
35.58
20.55
14.99
12.37
11.45
LD3
iPNDM (3M)
61.83
30.82
18.69
13.84
11.76
10.85
UniPC (3M)
130.65
31.89
UniPC-S4S (3M)
110.02
52.03
28.69
16.29
12.75
DPM-Solver (3M)
110.47
52.81
23.85
15.08
11.11
9.73
DPM-Solvere+ (3M)
227.26
113.01
63.69
41.60
32.96
27.31
So5 (3M)
198.27
101.50
57.33
39.04
31.62
25.61
iPNDM (3M)
192.47
96.55
62.97
45.45
35.94
29.28
Time LogSNR
iPNDM-54S (3M)
UniPC (3M)
223.84
106.68
62.32
43.15
34.15
26.99
UniPC-S4S (3M)
195.09
95,94
57.50
40.87
33.21
25.52
DPM-Solver-v3 (3M)
193.57
86.05
36.58
22.02
19.16
17.16
DPM-Solveree (3M)
159.24
93.01
56. 11
39.58
30.28
26.43
DPM-Solvertt-S0 (3M)
139.02
82.74
51.65
37.37
29.10
25.35
iPDDM (3M)
110.97
71.14
52.42
42.89
34.06
28.76
Time Quadratic
iPNDM-54S (3M)
95.17
63.24
48.46
40.36
33.04
27.18
UniPC (5M)
147.85
78.28
36.08
28.10
UnIPC-S4S
127.10
70.17
43.36
27.28
22.69
DPM-Solvar-v3 (3M)
136.22
64.17
38.96
25.96
23.31
20.50
DPM-Solvart+ (3M)
S45 (3M)
134.61
75.20
36.64
21.28
14.86
11.55
iPNDM (3M)
86.76
34.61
19.56
15.85
13.52
12.17
Time Unifom
IPNDM-S4S (3M)
75.52
30.95
17.69
14.99
12.95
11.40
UniPC (5M)
150.76
73.74
31,62
18.22
12.66
10.31
UniPC-S4S (3M)
131.51
65.59
28.62
17.05
12,20
9.69
DPM-Solvar-v3
S4S-Alt
37.65
20.89
13.03
10.49
10.03
9.64
Table 17. FID scores on LSSUN-Bedroon 256x256. Numbers in column headers indicate NFE counts. Bold: best within schedule; shaded: best overall. Curiously, despite using essentially the same replication code Zheng al. (2023) Tong et al. (2024) for LSUN-Beddoom generation, we were persistently unable to achieve the FID stated in many papers; accordingly, present this mainly as demonstrating overall trend for S4S on LSUN-Bedroome
29

---

S4S: Solving for Fast Diffusion Model Solver
Schedule
Solver
5
8
DPM-Solvvr-+ (3M)
65.43
26.54
20.40
15.36
14,42
13.14
DPM S44 (3M)
60.26
24.81
19.50
14.92
13.96
12.44
IPNDM
19.42
DMN
iPNDM-S48 (3M)
61.37
25.45
18.68
13.88
11.40
11.13
UniPC: (3M)
60.75
24.18
18.21
14.59
14.26
14.03
UnIPC-S4S (3M)
56.42
22.47
17.51
14.12
13.97
13.24
DPM-Sollver-33 (3M)
107.07
59.43
49.97
30.55
23.10
18.98
DPM-Solver++ (3M)
61.90
33.61
40.91
28.52
16.72
12.35
HPM--Sllver++-455 (3M)
57.01
16.46
iPNDM (3M)
46.66
28.48
19.66
14.77
12.28
11.45
Time
iPNDM-S4S (3M)
43.48
26.59
18.90
14.05
11.87
10.72
UniPC
62.90
36.93
53.93
19.59
UniPC-S4S (5M)
57.67
34.68
51.46
46.84
32.40
18.54
DPM Solver-v3 (3M)
97.31
64.57
77.71
65.88
34.62
17.82
DPM-Solvare+ (3M)
39.62
20.57
19.18
13.64
12.52
11.72
SoM (3M)
36.87
19.53
18.04
13.03
12.23
11.00
iPNDM (3M)
43.06
23.29
16.40
12.33
11.56
11.36
GITS
iPNDM (3M)
UniPC (3M)
39.42
20.22
22.25
14.63
12:40
11.30
UniPC-S4S (3M)
36.94
19.14
21.28
13.96
12.08
11.14
DPM Solver- (3M)
70.07
35.45
27.86
15.31
13.31
12.10
DPM Solvere+ (3M)
34.32
20.64
15.47
14.26
14.07
13.67
DPM-Solvere++-Se (3M)
31.77
19.21
14.83
13.89
13.75
13.01
iPNDM (3M)
43.73
26.14
17.33
13.19
12.31
12.28
iPNDM $4S (3M)
40.75
24.60
16.27
12.63
11.89
11.69
UniPC
UniPC-S48 (3M)
31.03
19:70
15.30
13.95
14.23
12.26
DPM Solver (3M)
49.07
23.92
17.35
15.22
14.43
14.11
DPM-Solvvrr+ (3M)
61.46
36.02
27.02
19.31
13.86
11.76
S0V (3M)
56.23
33.93
25.49
18.56
13.48
11.16
Time LogSNR
iPDDM (3M)
52.27
30.47
20.11
15.18
12.65
11.60
iPNDM-545 (3M)
UniPC (3M)
61.08
37.31
32.95
27.20
19.12
14.30
UniPC-S4S (3M)
57.23
34.56
31.41
26.11
18.58
13.73
DPM DPM-Solvve-v3 (3M)
99.16
64.39
44.02
33.51
24.23
15.99
DPM-Solvere+ (3M)
63.33
28.33
17.00
13.57
12.34
11.82
DPM Solver++ (3M)
58.16
26.61
16.25
13.21
11.89
11.35
iPNDM (3M)
59.94
27.93
16.65
13.03
11.84
11.48
Time Quadratic
iPNDM-S4S (3M)
54.75
26.19
15.76
12.42
11.67
10.84
UniPC (3M)
60.51
26.47
16.49
13.41
12.26
11.74
UniPC-S48 (3M)
56.50
24.95
15.67
13.05
11.87
11.07
DPM-Solver-y3 (3M)
103.62
60.78
34.99
21.88
16.17
13.42
DPM-Solver=+
14.82
DPM-Solvar-4-S45 (3M)
31.67
20.05
16.03
14.85
14.57
13.68
iPNDM (3M)
48.29
28.75
18.52
14.40
13.00
12.78
Time Uniform
iPNDM-S4S (3M)
44.89
27.27
17:45
14.00
12.77
12.11
UniPC (3M)
35.33
21 14
17.31
15.39
14.65
14.36
UniPC (3M)
20.32
16.57
14.83
14.30
13.78
DPM-Solver-v3 (3M)
S4S-Alt
25.44
16.05
13.26
11.17
10.83
10.68
Table 18. FID scores on MS-COCO 512x512. Numbers in column headers indicate NFE counts. Bold: best within schedule; shaded: best overall.
30

---

S4S: Solving for Fast Diffusion Model Solver
H.S. Qualitative Model Samples We provide qualitative samples below.
(a) Teacher Solver UniPC, 2ONFE, logSNR, 2.03 FID
(b) iPNDM, 5 NFE, logSNR, 20.49 FID
(c) iPNDM-S4S, NEE, LD3, 4.84 FID
(d) S4S-Alt, 5 NFE, 3.73 NFE
Figure S. Examples from CIFAR-10 32
31

---

S4S: Solving for Fast Diffusion Model Solver
25
T
(a) Teacher Solver UniPC, 20NFE, logSNR, 2.62 FID
(b) iPNDM, 6 NFE, logSNR, 16.15 FID
a
(c) $4S, 6 NFE, LD3, 5.45 FID
(d) S4S-Alt, 6 NFE, 4.62 FID
Figure 6. Examples from FFHQ 64
32

---

S4S: Solving for Fast Diffusion Model Solver
(a) Teacher Solver UniPC, 20NFE, logSNR, 2.24 FID
(b) iPNDM, 7 NFE, logSNR, 5.92 FID
S4S, 7 NFE, LD3, 2.61 FID
(d) S4S-Alt, 7 NFE, 2.56 FID
Figure 7. Examples from AFHQv2 64x64
33


Reviews:
[{"review_id": "fW1PEyYxOy", "reviewer": "Reviewer_Ngqf", "summary": "The paper proposes the S4S method for optimizing diffusion model solvers. The optimization space includes the solver coefficients, time discretization schedule, and time correction terms. The optimization objective is a relaxed version of the global error with LPIPS as the distance metric, which only requires the existence of an input xT' sufficiently close to the original xT. Experiments demonstrate that S4S can outperform previous learning-free/learning-based solvers and fixed/learned timestep discretizations over diverse datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reviewing our work and for your recommendation of acceptance. Below, we hope to address the weaknesses you identified and the questions you raised.\n\n> With a larger parameter space, the method is more data-driven and less theory-grounded than traditional solvers. The coefficients require separate optimization under different NFE.\n\nWe agree that our approach is less grounded in theory relative to other diffusion model solvers. However, we think that this is more a feature of the particular problem setting -- very low-NFE sampling -- than of our specific approach. For instance, in many impressive theory-grounded diffusion model solvers, e.g. DPM-Solver++, the underlying assumptions that guarantee convergence in these solvers begin to break down in the low-NFE regime (~3-5 steps) as the step size for each solver step increases. Accordingly, we see significant degredation in performance in many of these theory-grounded solvers as the number of NFEs decrease; this decrease in performance can be seen our tables in Appendix H. S4S's data-centric approach avoids trying to make these strong assumptions in the low-NFE regime and as a result achieves stronger performance than its theoretically-grounded alternatives. Nonetheless, from a theoretical perspective, it's still **not intuitive** why S4S even works; in fact, it seems shocking that solvers like iPNDM or DPM-Solver++ get remotely reasonable performance in the low-NFE regime in the first place. We think that trying to characterize why these approaches even achieve a modicum of success with few NFEs is a very exciting direction of future work that we hope to provide answers to.\n\nWe also recognize that our data-centric approach necessitates S4S to learn a new solver for each number of NFEs. In practice, this is a similar limitation to alternative methods for learning diffusion model solvers (e.g. LD3 [1], BNS [2], $\\Pi$A [3]). Creating a reusable method for crafting diffusion model sovlers without needing to retrain each time is similarly a direction for future work that we aim to address.\n\n> There are no intuitive visualizations of the learned coefficients and timestep discretizations.\n\nThank you for raising this point! Here, we provide some visualizations of the learned time-step discretizations for [LSUN bedroom](https://imgur.com/a/uGAP88Q) and for [FFHQ](https://imgur.com/a/rFKizhf). These visualizations indicate that the learned time-step discretizations are similar to the \"best\" heuristic time discretizations, i.e. the learned time discretization for latent diffusion models are similar to the uniform time discretization, while the learned time discretizations are similar to EDM / logSNR for pixel-space models.\n\n> Does the sampling trajectory of the learned solver still follow the teacher's, or is only the final sampler closer?\n\nThank you for asking this question! The answer to this question depends on the difficulty of the underlying task. Interestingly, in relatively simple domains (e.g. CIFAR-10), the trajectory of the learned solver still closely follows that of the teacher's, despite being trained to explicitly only match the output of the teacher solver. In contrast, however, on more complex domains (e.g. conditional generation in ImageNet-256 or MS-COCO text-to-image), the trajectories of the student solver can have notable differences from that of the teacher solver.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper applies a practical objective to optimizing the solver in a free space. Unlike more restricted papers like DPM-Solver-v3, there are no strict theorems.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper can optimize both the solver coefficients and timestep discretization with a novel alternating objective. The optimization cost and final performance are better than those of previous works.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The authors did a nice job in appendix A comparing to previous works, including (1) optimizing time steps (2) optimizing local truncation error (3) optimizing global error.\n- Ablations are comprehensive, including (1) the benefits of alternating optimization (2) the benefits of relaxed objective (3) the disadvantage of large order, as in BNS (4) the benefits of LPIPS as distance metric (5) the initialization method (6) the training dataset size.\n\nWeakness:\n- With a larger parameter space, the method is more data-driven and less theory-grounded than traditional solvers. The coefficients require separate optimization under different NFE.\n- There are no intuitive visualizations of the learned coefficients and timestep discretizations.", "other_comments_or_suggestions": "I suggest the authors visualize the learned coefficients and timestep discretizations in comparison to previous solvers.", "questions_for_authors": "- Does the sampling trajectory of the learned solver still follow the teacher's, or is only the final sampler closer?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bDwEKNqbFp", "reviewer": "Reviewer_qWi4", "summary": "The paper proposes S4S and S4S-Alt, methods to optimize diffusion ODE solvers for fast, high-quality sampling with minimal neural function evaluations (NFEs). S4S learns solver coefficients via a distillation objective, matching the output of a high-NFE \"teacher\" solver while minimizing global error (not local truncation error). Additionally, S4S-Alt jointly optimizes solver coefficients and discretization steps via alternating minimization, further improving performance. The authors achieve state-of-the-art FID scores across datasets (e.g., 3.73 on CIFAR-10, 13.26 on MS-COCO with 5-8 NFEs), outperforming prior solvers like DPM-Solver++ and iPNDM. The method is lightweight (<1 A100 hour), data-free, and compatible with existing schedules.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reviewing our work, and for recommending acceptance! We hope we answer your outstanding questions below.\n\n> How sensitive is performance to the radius $r$? Does the $r\\propto m^{-5/2}$ heuristic hold across varying $m$?\n\nIn practice, the heuristic $r \\propto m^{-5/2}$ works reasonably well \"out of the box\" across the experimental settings we looked at. Sensitivity emerges in two different directions though on both sides of the parameter scale. With very few number of parameters, choosing the radius to be too small makes the optimization problem more difficult. On the other hand, with a larger number of parameters, allowing for too large of a radius enables more overfitting. While $m^{-5/2}$ is a good heuristic that achieves baseline strong performance in both settings, we would expect improvements to come by carefully tuning this parameter based on the experimental settings.\n\nTo help clarify this, we ran some additional experiments on CIFAR-10 and ImageNet-256 to characterize this dependence; our results are in the table below.\n\n| Dataset | Model | Parameters $(m)$ | NFEs | $r=0.1m^{-5/2}$ | $r=0.5m^{-5/2}$ | $r=1.0m^{-5/2}$ | $r=2.0m^{-5/2}$ |\n|---------|-------|-----------------|------|-----------------|-----------------|-----------------|-----------------|\n| CIFAR-10 | iPNDM-S4S | 6 | 4 | 32.15 | 30.58 | 28.91 | 31.24 |\n| CIFAR-10 | S4S-Alt | 16 | 4 | 17.23 | 16.05 | 16.95 | 18.39 |\n| ImageNet | iPNDM-S4S | 6 | 4 | 8.12 | 7.84 | 8.06 | 8.53 |\n| ImageNet | S4S-Alt | 16 | 4 | 5.28 | 5.13 | 5.37 | 5.72 |\n\nAs shown in the table, the performance is generally robust within a reasonable range around our heuristic, with the optimal value typically falling between $0.5m^{-5/2}$ and $1.0m^{-5/2}$. The sensitivity increases with larger model sizes, supporting our conclusion that a well-calibrated radius is more important as parameter count increases to prevent overfitting.\n\n> Can S4S/S4S-Alt achieve competitive performance with 1-2 NFEs, or does it require a minimum step count?\n\nThank you for the interesting question! In practice, we found that going below 3 NFEs requires solving a very difficult problem that likely requires directly modifying the underlying score network. Concretely, 1 NFE generation entails directly generating an image from a noise latent; as such we completely lose any \"degrees of freedom\" over the time discretization; similarly, we may now only control ~2-4 coefficients, which is generally insufficient for producing high-quality outputs. While having 2 NFEs yields better performance, on traditional score network architectures, S4S and S4S-Alt still fall short of the performance seen in 1-2 step generation in training-based distillation methods. Nonetheless, this challenge persists for all other methods that don't modify the underlying score network.\n\nTo further characterize these results, we conducted a few more experiments. First, for CIFAR-10, we characterized S4S and S4S-Alt on 2 NFE generation for LD3 discretization.\n\n| Dataset | Method | NFE=2 | NFE=3 | NFE=4 |\n|---------|--------|-------|--------|-------|\n| CIFAR-10 | iPNDM | 155.37 | 23.64 | 9.06 |\n| CIFAR-10 | iPNDM-S4S | 142.45 | 20.65 | 8.25 |\n| CIFAR-10 | S4S-Alt | 104.62 | 14.71 | 6.52 |\n\nAs shown in the table, while S4S-Alt achieves significant improvements over traditional solvers at 2 NFEs, the FID scores are still substantially higher than at 3-4 NFEs. This supports our observation that there's a minimum effective step count (~3 NFEs) for maintaining reasonable image quality with training-free methods that don't modify the score network architecture.\n\nOn the other hand, we decided to replicate an experiment from the LD3 paper that we didn't have time to include in our results. Here, we examined results on InstaFlow, a flow network that is explicitly trained for high-quality few-step sampling. When the underlying model is explicitly trained to produce high-quality few-step samples, S4S correspondingly improves the image quality as well, on a scale competitive with the teacher solver (8 NFEs, Uniform Time Disc. = 14.16 FID).\n\n| Method | NFE=2 |\n|---------|--------|\n| InstaFlow | 24.13 |\n| InstaFlow [LD3] | 16.74|\n| InstaFlow [LD3 + S4S] | 15.22 |\n| InstaFlow [LD3 + S4S-Alt] | 14.31 |", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A. There are no theoretical results in the main document (and in the supplementary material, the authors only restate the theoretical guarantee for the relaxed objective presented in Eq. (7); this guarantee was provided by Tong et al. (2024)).", "experimental_designs_or_analyses": "Strengths: Extensive evaluation across various datasets (CIFAR-10, FFHQ, ImageNet, etc.) and multiple baselines (DPM-Solver++, iPNDM, UniPC). Ablations validate design choices (time-dependent coefficients, relaxed objective).", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: The proposed approaches are novel, and demonstrate universal improvement across datasets, architectures, and schedules. Moreover, they require no data/retraining, are computationally efficient, and can be plugged in black-box on top of any discretization schedule or architecture. The authors have performed extensive and impressive experiments to demonstrate the effectiveness of the proposed approaches.\n\nI have not found major weaknesses in this submission.", "other_comments_or_suggestions": "No.", "questions_for_authors": "- How sensitive is performance to the radius $r$? Does the $r \\propto m^{-5/2}$ heuristic (mentioned in Appendix G.2) hold across varying $m$?\n\n- Can S4S/S4S-Alt achieve competitive performance with 1-2 NFEs, or does it require a minimum step count?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bcbhtierz", "reviewer": "Reviewer_xsUN", "summary": "The sampling process of diffusion models heavily depends on numerical solvers. This paper provides a comprehensive overview of existing works, including \n(1) Vanilla solver-based fast samplers, such as single-step, multi-step, predictor-corrector methods;\n(2) Data-driven solver-based fast samplers, which involve learning the discretization schedule, learning solver coefficients, or a combination of both, and highlights their differences,\nThen, this paper introduces Solving for the Solver (S4S), a method that optimizes numerical solver coefficients by distilling from a teacher solver with a relaxed objective function. An improved version, S4S-Alt, alternatively optimizes both the solver coefficients and the time schedule, achieving state-of-the-art results for few-step sampling of diffusion models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback, we address your questions and concerns, starting with two of the weaknesses you mentioned.\n\n> Lack of direct experimental comparisons in the main context\n\n[In the linked image](https://imgur.com/a/jkFi6tJ), we provide direct comparisons against the recent works [2,3] that focus on *coefficient* distillation.\nCompared to these works, on the same discretization schedule (Time EDM), S4S achieves superior performance in the low-NFE regime across several key datasets.\nThis helps reinforce the strength of S4S, even when only optimizing over the solver coefficients.\nHere, * refers to FID values that were estimated from Figures if no results table was present, and the \"Rel. Fig.\" column details which Figure the value was taken from. \n\nRegarding BNS, it is difficult to provide a wholesale, fair comparison to the original paper for two main reasons: (1) the authors do not release their code for learning BNS, and (2) the model checkpoints used for BNS are not publicly released. \nAs a result, we cannot (1) port an exact implementation of BNS over onto the pretrained DMs that we used, or (2) port over our own implementation onto the pretrained (flow) models that BNS used.\n\nTo try to compare against BNS, we note that it has two significant differences from our work:\n- In BNS, the order of the learned solver is equal to the number of NFEs used; that is, it assigns a coefficient to all of the previous denoising steps.\n- BNS jointly learns the solver coefficients and the time discretization steps.\n- BNS uses PSNR as the global objective function.\n\nIn our submission, we tried to explore the first two components of this tradeoff in the third row of Table 6, although we should have made the connection to BNS more clear. \nTo further extend this analysis, we **explicitly re-implemented the approach from BNS into our own setup** to directly compute this comparison.\nWe include this in the table below for CIFAR-10 and FFHQ.\n\n> Pathologies/errors from the teacher trajectory distilled into the student\n\nWe agree that more experiments are needed in the revision. We reference a visualization from recent work [4] that visualizes trajectories from few-step sampling at [this link](https://imgur.com/a/EgcOKEY). At the beginning of the boomerang shape, there is greater variance in the trajectories. Therefore, training a model to explicitly match these trajectories can result in distilling these pathological behaviors directly into the student solver. To further assess this claim, we conducted several experiments to understand the consequences of using this as an objective. In the attached image, we see that **training on a dataset that includes these \"pathological\" trajectories teaches the student solver to mimic them**. In particular, conducting a full FID evaluation of a solver trained in this manner reveals worse FID performance.\n\n> Concerns about novelty\n\nTo best characterize our novelty, we compare with two types of learning diffusion solvers.\nFirst, unlike $\\Pi$A or D-DDIM, which mimic the ODE trajectory, our approach directly matches the output of the teacher solver.\nWhile this view is shared at a high-level by LD3 in how discretization points should be selected, at its core, LD3 still views sampling from a diffusion model as solving an ODE and instead focuses on selecting discretization points that minimize the global error.\nIn contrast, S4S argues that in the very few-NFE regime, an ODE-centric view is limiting.\n\nSecond, S4S and S4S-Alt are similar in spirit to BNS but differ in key methodological choices. Through a careful exploration of the diffusion solver design space, S4S-Alt greatly outperforms BNS. This improvement stems from four key decisions:\n- Using local rather than global information—learning coefficients for only the three most recent denoising points.\n- Employing an alternating objective to learn coefficients and discretization points.\n- Relaxing the objective for more effective learning.\n- Exploring a wider array of distance functions.\n\nWhile these choices may seem minor, they significantly enhance performance. BNS, in contrast, attempts to generalize across solvers, optimizing a complex objective over a large coefficient space, which makes it vastly slower (requiring orders of magnitude more compute) and leads to a more difficult optimization landscape. Our ablation study confirms that jointly optimizing solver coefficients and schedules underperforms compared to S4S-Alt.\n\n> LPIPS losses\n\nThank you for this question! Although our results achieve stronger FID scores when used with the LPIPS loss, our overall results still hold when using an alternative distance function in our objective. Below, we include a table that characterizes our performance using non-LPIPS losses for both S4S and S4S-Alt. Broadly speaking, using an alternative loss decreases FID scores, though not to such an extent that it weakens our overall approach. \n\n[4] https://arxiv.org/pdf/2409.19681 SFD", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes. I've checked the soundness and validity of all the experiments in the main content.", "relation_to_broader_scientific_literature": "The proposed method sets a new record for few-step sampling of diffusion models, with potential applications in broader fields such as video and 3D generation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The proposed method achieves state-of-the-art results on few-step sampling of diffusion models, while maintaining a low additional training overhead.\n\n2. The authors provide a thorough review of existing works in Appendix A, clearly distinguishing their proposed method from previous methods. This helps readers better understand the literature and the unique contributions of this paper. \n\n3. Extensive quantitative results are presented in Appendix H.4, which supports the effectiveness of the proposed method.\n\nWeaknesses\n\n1. Concerns about novelty:\n(i) The proposed S4S closely resembles LD3 [1] in terms of algorithm, with the primary difference being that this paper optimizes solver coefficients, whereas LD3 focuses on optimizing the time schedule. \n(ii) Meanwhile, the idea of learning solver coefficients has already been explored before [2][3], as acknowledged in Appendix A (I appreciate the authors' efforts in maintaining academic integrity). \n(iii) Also, the proposed S4S-Alt appears to be an iterative combination of two LD3-like algorithms, i.e., alternating between optimizing solver coefficients and time schedule. \n\nGiven these points, the novelty of the contributions seems somewhat limited.\n\n2. Lack of direct experimental comparisons in the main context:\nSec 4 does not provide sufficient direct comparison with existing works, making it difficult to assess the superiority of the proposed S4S and S4S-Alt. To be more specific, there is no comparison between S4S and other distillation-based methods for learning solver coefficients [2][3], despite these being the most relevant works. Besides, S4S-Alt is not fully and fairly compared against BNS [4], which also optimize solver coefficients and the time schedule. Only a single value  is directly taken from the original BNS paper in Tab 4, rather than conducting a fair head-to-head comparison.\n\n3. Section 3.1 claims that the pathologies/errors contained in the teacher trajectory could be distilled into the student, making it crucial to optimize global error. However, no experiments are provided to support this claim. It would be beneficial to conduct an experiment to further discuss this point.\n\nReferences\n[1] 2025ICLR Learning to Discretize Denoising Diffusion ODEs https://arxiv.org/abs/2405.15506\n[2] 2024ICLR On accelerating diffusion-based sampling process via improved integration approximation https://arxiv.org/abs/2304.11328\n[3] 2024CVPR Distilling ODE Solvers of Diffusion Models into Smaller Steps https://arxiv.org/abs/2309.16421\n[4] 2024ICML Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models https://arxiv.org/abs/2403.01329", "other_comments_or_suggestions": "I don't have other comments.", "questions_for_authors": "The proposed method incorporates LPIPS loss, which has been criticized for leaking ImageNet features that may cause inflated FID scores [1]. How does the proposed method perform without LPIPS loss? Have you evaluated its impact on these results?\n\nReferences\n[1] Song Y, Dhariwal P. Improved Techniques for Training Consistency Models[C]//The Twelfth International Conference on Learning Representations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nRJj6bfFFO", "reviewer": "Reviewer_ZUmA", "summary": "Solving ODEs in diffusion models using traditional ODE solvers is expensive due to the iterative Neural Function Evaluations. If only use a few NFEs, the evolution trajectory is broken because of large step sizes. Targeting this problem, the authors propose S4S, a method that learns optimized, few-NFE solvers from conventional, many-NFE teacher solvers. Additionally, the authors propose S4S-Alt that also optimizes the discretization schedule.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the time taken to review our work and your helpful feedback! Please find our responses below:\n\n> However, the focus is largely on improvements in the few-NFE regime for training-free methods, and it remains somewhat unclear how the student solver matches the output of a teacher solver (which is supposed to run for many NFEs) as claimed in the conclusion.\n\nYou raise an important point about our objective of matching teacher outputs. I'd like to clarify this aspect of our approach.\nWhile our training objective involves matching teacher solver outputs, this is primarily a means to an end rather than the ultimate goal. Our core innovation is recognizing that in the few-NFE regime, traditional error control approaches break down, so we instead directly optimize for what matters most: high-quality final samples.\nWe use teacher matching as a proxy for quality for several reasons, including:\n\n - It provides a clear optimization target that avoids the need for dataset access, unlike distillation methods.\n - It learns to reproduce the outcomes of many-NFE solvers without being constrained to follow their intermediate steps.\n - The relaxed objective (Equation 7) further enhances this by allowing flexibility in finding solutions that prioritize final quality over exact trajectory matching.\n\nEmpirically, our results demonstrate that this approach succeeds not just at matching teacher outputs but often exceeding them at lower NFE counts. For instance, on CIFAR-10, S4S-Alt with 7 NFEs (FID 2.52) outperforms the teacher with 20 NFEs (FID 2.87). This indicates our method is learning a fundamentally more efficient sampling strategy, not merely approximating the teacher's steps.\nThe effectiveness of our global error minimization approach suggests that, while traditional solvers accumulate errors across many small steps, our learned solvers take optimal large steps that directly target high-quality outputs. This represents a paradigm shift in diffusion model sampling that could inspire further research into learning-based ODE solvers beyond diffusion models.\nWe'll emphasize this distinction more clearly in the final version of the paper to ensure the conceptual contribution is properly understood.\n\n> Since S4S is designed for few-NFE, while the teacher solvers are specifically for many-NFE. It would be insightful to examine how far are the student solvers away from ideal performance (many-NFE for teacher solvers).\n\nYou raise an excellent point about comparing student solvers to teacher performance. We did analyze this gap in our experiments, though we may not have emphasized it enough in the main paper due to space constraints.\nIn our full results tables (H.4), we provide FID scores for a range of NFEs up to 10, while in H.5 we mention the FID scores of the various teacher solvers. For example, on CIFAR-10, our S4S-Alt achieves 2.18 FID at 10 NFEs, while the teacher solver (UniPC with logSNR) achieves 2.03 at 20 NFEs. \nFor FFHQ, we see a similar pattern: S4S-Alt with 10 NFEs achieves 2.91 FID, while the teacher at 20 NFEs achieves 2.62. \n\n> The convergence of quality vs. NFE (at how many NFEs do the student solvers cease to improve) may be a useful reference of quality-speed tradeoff\n\nThis is an insightful suggestion. While we didn't include full convergence curves in the paper, our results in Tables 13-18 show the quality-speed tradeoff across 3-10 NFEs. We observe that:\n\n- For most datasets, improvements diminish beyond 8 NFEs, with minimal gains between 8-10 NFEs.\n- S4S-Alt shows more rapid convergence than traditional solvers, achieving near-optimal quality at just 5-6 NFEs on most datasets.\n- Different datasets show varying convergence patterns: CIFAR-10 and AFHQ-v2 converge more quickly (plateau around 6-7 NFEs), while more complex datasets like MS-COCO and LSUN-Bedroom continue improving up to 10 NFEs.", "claims_and_evidence": "More evidence may be needed to prove the authors' approach matches the output of a teacher solver.", "methods_and_evaluation_criteria": "The evaluations are generally good and the dataset choices (cifar-10 and imagenet) are good.", "theoretical_claims": "No new theorem is proposed or proven in this work. I do not see any mistakes in math.", "experimental_designs_or_analyses": "See questions", "relation_to_broader_scientific_literature": "It is closely related to other few-NFE approaches such as LD3 and BNS, which are thoroughly discussed in this work.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The paper is generally enjoyable to read and clearly written, with an interesting plug-and-play approach. It clearly points out the key differences and improvements over previous works, and the detailed discussions are likely to be valuable for researchers interested in the topic. However, the focus is largely on improvements in the few-NFE regime for training-free methods, and it remains somewhat unclear how the student solver matches the output of a teacher solver (which is supposed to run for many NFEs) as claimed in the conclusion.", "other_comments_or_suggestions": "None", "questions_for_authors": "(1) Since S4S is designed for few-NFE, while the teacher solvers are specifically for many-NFE. It would be insightful to examine how far are the student solvers away from ideal performance (many-NFE for teacher solvers). \n\n(2) The convergence of quality vs. NFE (at how many NFEs do the student solvers cease to improve) may be a useful reference of quality-speed tradeoff.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to optimize diffusion ODE solvers for high-quality sampling in the low-NFE regime. The method is lightweight, training-free, and compatible with existing schedules. All reviewers gave positive scores. They commended the paper for its clear presentation, a well-articulated discussion of its differences and novelty compared to previous works, and a comprehensive, promising, and persuasive experimental evaluation. The AC concurs, noting that the proposed method establishes a new benchmark for few-step sampling in diffusion models, holds great potential for wide-ranging applications in AIGC, and thus recommends acceptance.
Decision: Accept (poster)