Differentiable Solver Search for Fast Diffusion Sampling
Authors: Shuai Wang
Zexian Li
Qipeng zhang
Tianhui Song
Xubin Li
Tiezheng Ge
Bo Zheng
Limin Wang
Venue: ICML 2025 poster (2025)
Keywords: solver
diffusion sampling
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Differentiable Solver Search for Fast Diffusion Sampling

Abstract: Diffusion models have demonstrated remarkable generation quality but at the cost of numerous function evaluations. Recently, advanced ODE-based solvers have been developed to mitigate the substantial computational demands of reverse-diffusion solving under limited sampling steps. However, these solvers, heavily inspired by Adams-like multistep methods, rely solely on t-related Lagrange interpolation. We show that t-related Lagrange interpolation is suboptimal for diffusion model and reveal a compact search space comprised of time steps and solver coefficients. Building on our analysis, we propose a novel differentiable solver search algorithm to identify more optimal solver. Equipped with the searched solver, rectified-flow models, e.g., SiT-XL/2 and FlowDCN-XL/2, achieve FID scores of 2.40 and 2.35, respectively, on ImageNet-$256\times256$ with only 10 steps. Meanwhile, DDPM model, DiT-XL/2, reaches a FID score of 2.33 with only 10 steps. Notably, our searched solver outperforms traditional solvers by a significant margin. Moreover, our searched solver demonstrates generality across various model architectures, resolutions, and model sizes.

Differentiable Solver Search for Fast Diffusion Sampling
Shuai Wang'
Zexian Li
Qipeng Zhang? 2 Tianhui Song' Bo Zheng?
Xubin Li²
Tiezheng Ge
Limin Wang
Sampling Steps
Searched TimeSteps
DDPM/VP
RectifiedFlow
DDPM
Figure I: Visualization of the searched solver parameters of DDPM/VP and Rectified Flow. We limit the order of solver coefficients of the two steps for 5/6 NFE, The left images show absolute value of searched coefficients {c]}- The right image shows the searched timesteps different NFE and the fitted curves.
Abstract
with only 10 steps. Notably, our searched solver significantly outperforms the traditional solvers, even for some distillation-based methods. More- over, our searched solver demonstrates generality across various model architectures, resolutions, and model sizes.
Diffusion models have demonstrated remarkable generation quality, but at the cost of numer- ous function evaluations. Advanced ODE-based solvers have recently been developed to miti- gate the substantial computational demands of reverse-diffusion solving under limited sampling steps. However, these solvers, heavily inspired by Adams-like multi-step methods, rely solely t-related Lagrange interpolation. We show that t-related Lagrange interpolation is subopti- mal for diffusion models and define a compact search space comprised of time steps and solver coefficients. Based on our analysis, we propose new differentiable solver search algorithm find more optimal solver. Equipped with the searched solver, rectified-flor flow models, e.g., SiT- XL/2 and FlowDCN-XL/2, achieve FID scores 2.40 and 2.35, respectively, on ImageNet- 256 x 256 with only 10 steps. Mcanwhile, DDPM model of DIT-XL//2 reaches a FID score of 2.33
1. Introduction
Image generation is fundamental task in computer vision, which aims at capturing the inherent data distribution of orig- inal image datasets and generating high-quality synthetic im- ages through distribution sampling. Diffusion models (Ho et al., 2020; Song et al., 2020b; Karras et al., 2022: Liu et al., 2022; Lipman al., 2022; Wang et al., 2025) have recently emerged highly promising solutions learn the underlying data distribution in image generation, outper- forming GAN- models (Brock et al., 2018; Sauer et al., 2022) and Auto-Regressive models (Chang et al., 2022) by a significant margin.
However, diffusion models necessitate numerous denoising steps during inference, which incur substantial computa- tional cost, thereby limiting their widespread deployment. The existing studies have explored two distinct approaches to achieve fast diffusion sampling. maining-basee tech- niques by distilling the fast ODE trajectory into the model parameters, thereby circumventing redundant refinement steps. In addition, colver-based methods (Lu et al., 2023;
'State Key Lab of Novel Software Technology, Nanjing Uni- versity, Nanjing, China. "Taobao Tmall Group of Alibaba, Hangzhou, China. Shannhai Al Lab, Shanghai, China.. Corre- spondence to: Limin Wang <Imwang@ eevraag@@ju.cduuu edu.e..
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Differentiable Solver Search for Fast Diffusion Sampling
Zhang& & Chen, 2023; Song et al., 2020a) tackle the fast sampling problem by designing high-order numerical ODE solvers.
corresponding noise schedulers. In this paper, we present new solver-based method for fast diffusion sampling by using data-driven approaches without destroying the pre- trained internality in diffusion models. Inspired by (Xue et al., 2024), we investigate the sources of error in the dif- fusion ODE and discover that the form of the interpolation function is inconsequential. which can be reduced to coef- ficients. Furthermore, we define a compact search space related to the timesteps and solver coefficients. Therefore, we propose a differentiable solver search method to identify the optimal parameters in this compact search space.
For training- based acceleration, (Salimans & Ho, 2022) aligns the single-step student denoiser with the multi-step teacher output, thereby reducing inference burdens. The con- sistency model concept. introduced by (Song et al., 2023), directly teaches the model to produce consistent predic- tions at any arbitrary timesteps. Building upon (Song al., 2023), subsequent works (Zheng et al., 2024; Kim et al., 2023; Wang et al., 2024a: Xu et al., 2024; Song et al., 2025) propose improved techniques to mitigate discrete errors in LCM training. Furthermore, (Lin et al., 2024; Kang et al., 2024; Yin et al., 2024; Zhou et al., 2024; Wang et al., 2023) leverage adversarial training and distribution matching to en- hance the quality of generated samples. However, training- based methods introduce changes to the model parameters, resulting in an inability to exploit the power of pre-trained models fully.
Based on our novel differentiable solver search algorithm, we investigate the upper bound performance of pre-trained diffusion models under limited steps. Our searched solver significantly improves the performance of pre-trained dif- fusion models, and outperforms traditional solvers with large gap. On ImageNet-256 x 256, armed with our solver, rectified-flow models of SiT-XL/2 and FlowDCN- XL/2 achieve 2.40 and 2.35 FID, respectively, under 10 steps, while DDPM model of DiT-XL/2 achieves 2.33 FID. Surprisingly, our findings reveal that when equipped with an optimized high-order solver, the DDPM can perform com- parably or even surpass the performance of rectified flow models under similar NFE constraints.
Solver-based methods rely heavily on the ODE formula- tion in the reverse-diffusion dynamics and hand-crafted multi-step solvers. (Lu al., 2023; 2022) and (Zhang & Chen, 2023) point out the semi-linear structure of the diffu- sion ODE and propose an exponential integrator to tackle faster sampling in diffusion models. (Zhao et al., 2023) further enhances the sampling quality by introducing the predictor-corrector structure. Thanks to the multistep- based ODE solver methods, high-quality samples can be gener- ated within few as 10 steps. To further improve efficiency, (Gao al., 2023) tracks the backward error and determines the adaptive step. Moreover, (Karras et al., 2022; Lu al., 2022) propose handcrafted timesteps scheduler to sample respaced timesteps. (Xue et al., 2024) argues that timesteps sampled in (Karras et al., 2022; Lu et al., 2022) are subop- timal, thus proposing an online optimization algorithm to find the optimal sampling timesteps for generation. Apart from timesteps optimization, (Shaul et al., 2023) learns specific path transition to improve the sampling efficiency.
To summarize, our contributions are
We reveal that the interpolation function choice is not important and can be reduced to coefficients through the pre-integral technique. We demonstrate that the upper bound of discretization error in reverse-diffusion ODE is related to both timesteps and solver coefficients. Thus, we define compact solver search space.
Based on our analysis, we propose a novel differen- tiable solver search algorithm find the optimal solver parameter for given pre-trained diffusion models.
In contrast to training- based acceleration methods. solver- based approaches do not necessitate parameter adjustments and are able to preserve the original performance of the pre-trained models. Moreover, solvers can be seamlessly applied to any arbitrary diffusion model trained with simi- lar noise scheduler, offering high degree of flexibility and generality. This property enables us to investigate the gen- erative capabilities of pre-trained diffusion models within limited steps from a diffusion solver perspective.
For DDPM/VP time scheduling, armed with our searched solver, DiT-XL/2 achieves 2.33 FID under 10 steps, beating DPMSSolver+//UiiPC by a signifi- cant margin.
For Rectified-flow models, armed with our searched solver, SiT-XL/2 and FlowDCN-XL/2 achieve 2.40 and 2.35 FID respectively under 10 steps on ImageNet- 256 x 256.
Current sfate-of-the art diffusion solvers (Lu et al., 2023; Zhao et al., 2023) adopt Adams-like multi-step methods that use the Lagrange interpolation function minimize integral errors. We argue that an optimal solver should be tailored to specific pre-trained denoising functions and their
For Text-to-Image diffusion models like FLUX, SD3, PixArt-E, our solver searched on ImageNet-256 x 256 consistently yields better images compared to tradi- tional solvers with the same CFG scale.
2

---

Differentiable Solver Search for Fast Diffusion Sampling
2. Related Work
2023) discovered the semi-linear structure in DDPM/VP re- verse ODEs. Furthermore, (Zhao et al., 2023) enhanced the sampling quality by borrowing the predictor-corrector struc- ture. Thanks to the multi-step ODE solvers, high-quality samples can be generated within as few as 10 steps. To further improve efficiency, (Gao et al., 2023) tracks the backward error and determines the adaptive step. Moreover, (Karras et al., 2022; Lu et al., 2022) proposed a handcrafted timestep scheduler to sample respaced timesteps. However, (Xue et al., 2024; Sabour et al., 2024; Chen et al., 2024a) ar- gued that the handcrafted timesteps are suboptimal, and thus proposed an online optimization algorithm to find the opti- mal sampling timestep for generation. Apart from timestep optimization, (Shaul et al., 2023) learned a specific path transition to improve the sampling efficiency.
Diffusion Model gradually adds zo with Gaussian noise to perturb the corresponding known data distribution p(z))
into a simple Gaussian distribution. The discrete perturba-
tion function at each step & satisfies follows:
of as
Et Oral + OiE,
(1)
where 0zg, of >> O. Moreover, as shown in Equation (2),
Equation (1) has a forward continuous-SDE: description, where [(L) = d aloga
do? logg di of. (Ander- di and g(() = di de
son, 1982) establishes a pivotal theorem that the forward SDE has an equivalent reverse-tee time diffusion process as in Equation (3), so the generating process is equivalent to solv- ing the diffusion SDE. Typically, diffusion models employ neural networks and distinct prediction parametrization to estimate the score function V log, Pa, (mx) along the sam- pling trajectory (Song et al., 2020b; Karras et al., 2022; Ho et al., 2020).
3. Problem Definition
As rectified-flow has a simple yet elegant formulation within the diffusion family, we choose it as the primary subject of discussion in this paper to enhance readability. Importantly, our proposed algorithm is not constrained to rectified flow models. We explore its applicability to other diffusion mod- els such DDPM/VP in Section 6.
dæ; = [(1)x,, + g())dw.
(2)
dæ,- = ((1))a--g()))
- -g(() p(,)( + g())dw.
(3)
(Song al., 2020b) also shows that there exists a corre- sponding deterministic process Equation (4) whose trajecto- ries share the same marginal probability densities of Equa- tion (3) as follows:
Recall the continuous integration of reverse-diffusion in Equation (7) has the pre-defined interval (to, ti...tth J. Given the pre-trained diffusion models and their corre- sponding ODE defined in Equation (4), before we tackle the integration of interval [t,, t441), we have already ob- tained the sampled velocity field of previous timestep {(x), ty: U2 vg (y) Here, we directly denote Dt; as æ; for presentation clarity:
dx; = [f()); 1 50( log dlt.
(4)
Rectified Flow Model simplifies diffusion model under the framework of Equation (2) and Equation (3). Different from (Ho et al., 2020) introduces non-linear transition scheduling, the flow model adopts linear function to transform data to standard Gaussian noise:
1 22 1.¹ vo (1,,, t)dt.
As shown in Equation (8), we strive to develop more optimal solver that minimizes the integral error while en- hancing image quality under limited sampling steps (NFE) without requiring any parameter adjustments for the pre- trained model:
T2 = tareall (1 (1e
(5)
Instead of estimating the score function V., log Pt (x:), rectified-flow models directly learn neural network t) predict the velocity field v. dx,- = (Ireal e).
+ = arg min BII( - (E 1/" no( ()d( (8)
((0) = /
l [100( - -) adt)
(6)
4. Analysis of reverse-diffusion ODE Sampling
Solver-based Fast Sampling Method does not necessitate parameter adjustments and preserves the optimal perfor- mance of the pre-trained model. It can be seamlessly ap- plied to an arbitrary diffusion model trained with similar noise scheduler, offering a high degree of flexibility and adaptability. Solvers heavily rely on the reverse diffusion ODE in Equation (4). Current solvers are mainly focused on DDPM/VP noise schedules. (Lu et al., 2022; Zhang & Chen,
Initially, we revisit the multi-step methods commonly used by (Zhao et al., 2023; Zhang & Chen, 2023; Lu et al., 2023) and identify potential limitations. Specifically, we argue that the Lagrange interpolation function used in Adams- Bashforth methods is suboptimal for diffusion models. Moreover, we show that the specific form of the interpo- lation function inconsequential, as pre-integration and
3

---

Differentiable Solver Search for Fast Diffusion Sampling
Sampling Steps
Euler-Shitt
DS: 3o-vor
a silver vase and some silver leaves and green plant
Threequarters of meatlovers pizza with mush mushrooms with
Euler Shier-Shiftt
SS--Slver
Co
GUSTH
CLOOA
A marsh area with egrets and shrimp boats
A street sign pointing the way to Jordan
Figure 2: Generated images from Flux.1-dev with Guidance=2.0 and our solver (searched on SiT-XL/2). Euler-Shift3 is the default solver provided by diffusers and Flux community. Our Solver(DS-Solvee) achieves better visual quality from 5 to 10 steps(NFE).
expectation estimation ultimately reduce it set of efficients. Inspired by (Xue al., 2024), prove that timesteps and these coefficients effectively constitute our search space.
solvers employ the Lagrange interpolation function dif- ference formula to estimate the value in the given interval.
However, the Lagrange interpolation function and other similar methods only take into account while the v(x; I) also needs a as inputs. Using first-order Taylor expansion of æ at æ; and higher-order expansion of t,, we can readily derive the error bound the estimation.
4.1. Revisit the multi-step methods
As shown in Equation (9), the Fuler method employs v; an estimate of Equation (9) throughout the interval [t,, 1++ Higher-order multistep solvers further improve the estima- tion quality of the integral by incorporating interpolation functions and leveraging previously sampled values.
4.2. Focus on solver coefficients instead the interpolation function
In contrast to typical problems of solving ordinary differ- ential equations, when considering reverse-diffusion ODEs along with pretrained models, a compact searching space is present. We define a universal interpolation function P. which has no explicit form. P measures the distance of (x,, t) between previous sampled points ((z,:tz)); to determine the interpolation weight for (v, (r)))_o-
T 1 = + (t++1 - to),o(a,,,,) t.).
(9)
The most classic multi-step solver Adams Bashhooth method (Bashforth & Adams, 1883)ddeemed Adams for brevity) incorporates the Lagrange polynomial to improve the estimation accuracy within a given interval.
0:+1
to Du,dt.
(10)
0i4
20;
P( L, Lj),dd.
(12)
1=0
0:41
C;
U
t-
t- th ddt
(11)
Uj
(13)
1=0
k=0,kej
lj bx
3=0
Assumption 4.1. We assume that the remainder term of the universal interpolation function Σ; P(x,, t, 2y, v, for v(x, t) is bound as (da=") + O(dt") where O(dxm is the mtt-orrdr infinitesimal for dx, Odtt") ) is the nth-order infinitesimal for dl.
As Equation (11) states, 1: (I t- dt of the 1--b2
I Lagrange polynomial can be pre-integrated into con- stant coefficient, resulting in only naive summation be- ing required for ODE solving. Current SoTA multi-step solvers (Lu et al., 2023; Zhao et al., 2023) are heavily in- spired by Adams Bashforth-liis multi-step solvers. These
Equation (13) has recurrent dependency, as 2; also relies

---

Differentiable Solver Search for Fast Diffusion Sampling
on Σj= Pla L, By dy), To eliminate the recurrent dependency, shown in Equation (14), we simply use the first order Taylor expansion of ætt) at to replace the original form. Recall that o; is already determined by and t,, thus the partial integral of Equation (14) can be formulated as Equation (15). Unlike naive Lagrange interpolation, (x,,) a function of the current æ; instead of a constant scalar.
equation. We can employ triangle inequalities obtain the error-bound([_]) of [ Ex|l. the proof can be found in the Appendix I.
I0N uNll Sn
Σ ==
(1 12))
Learning a (a,)) function will cause the generalization to be lost. This limits the actual usage in diffusion model sampling.
Based Theorem 4.4, since the error bound is related to timesteps and solver coefficients, we can define much more compact search space consisting of {cl 1 and [
fe11 P(z; + vi(t- t,),,, t dt).
Σ
2++1 2 04
Uj
J::
1=0
Theorem 4.5. Based Theorem 4.4 and Theorem 4.2. We can derive the total upper error bound( of our solver search method and other counterparts.
(14)
0i41 W. Σ (xs)((z)((enr -01) 1.).
(15)
The total upper ervor bound of our solver search is:
Theorem 4.2. Given sampling time interval [tt,t t+l and suppose Cy(zx) = 9,(2;) + b/. Adams-like lin- ear multi-step methods have an error expectation of (f++1 - tz)Bx.Il 11 Ej-o Uj9; (xx)]l. replacing (x( with Ex,, Ec,,I,, (x.)) ]]]]]]] is the optimal choice and owns an error expec- tation of (fi+1 - t.) Lz, - (z())(w;) - Bx.g;(a;)) We place the proof in Appendix H.
N-1
(f++1
tt)
11 9x ) + b/1.
5=0
+ Rez
Σ Ugg 1g (2:)
1=0
Compared to Adams-like linear multi-ster step methods. Our searched solver has a small upper error bound. The proof can be found in the Appendix I.
According to Theorem 4.2, we opt replace (a,) with its expectation [C, (c::)) thus we obtain diffusion- scheduler-related coefficients while maintaining generaliza- tion ability.
Through Theorem 4.5, our searched solvers own relatively small upper error bound. Thus, we can theoretically guaran- tee optimal compared to Adams-like methods.
Finally, given the predefined time intervals, we obtain the optimization target Equation (16), where ef = Ex,, [C, (a;)) The expectation can be deemed as optimized through mas- sive data and gradient descent.
5. Differentiable Solver Search
Through previous discussion and analysis, we identify [c] [c] and [ as the target search items. To this end, we propose differentiable data-driven solver search approach to determine these scarchable items.
x;+ R D3 +
Σ U (t t2).
(16)
1=0
4.3. Search optimal solver
Timestep Parametrization. As shown in Algorithm 1, we employ unbounded parameters (r&, 1 as the opti- mization subject, as the integral interval is from 0 to 1, we convert r; into time-space deltas Al; with softmax normal- ization function force their summation to 1. We can access timestep f4+1 through 4,+1 t, + At, We initialize (r, J.N- with 1.0 to obtain uniform timestep distribution.
Assumption 4.3. As shown in Equation (17), the pre-trained velocity model is not perfect and the error between and ideal velocity field è is Ll-bounded, where 17 is a constant scalar.
|| - v Sn < ||
(17)
Previous discussions assume we have perfect velocity func- tion. However, the ideal velocity is hard to obtain; we only have pre-trained velocity models. Following Equation (16), we can expand Equation (16) from = to t==N to obtain the error bound caused by non-ideal velocity estimation.
Coefficients Parametrization. Inspired by (Xue et al., 2024). Given Equation (16) and Equation (7), when the velocity field uo(z, t) yields constant value, an implicit constraint Ei=o cj = 1 emerges. This observation mo- tivates us to re- parameterizze the diagonal value of M as [1 [1-3 - ci 1- We initialize {c/, } with zeros to mimic the behavior of the Fuler solver.
Theorem 4.4. The error caused by the non-ided veloc- ity estimation model can be formulated in the following
5

---

Differentiable Solver Search for Fast Diffusion Sampling
Algorithm 1 Solver Parametrization
Algorithm 2 Differentiable Solver Search
Requires: }rr. and (c/-}
Require: vy model, {At, Jio' .M, buffer Q. Compute { ! Euler(e, 28).
TimeDeltas: Ala. Al...., A1--1-
SolverCoefficietss:: M E NNNN
for 0 to 1
(At,) )=Softmax((
Q butfer (i)
Compute v = Ej=o Mza
1- c-
f+11 = t, + At:
M=
L1; 22; + vDLi
end for
Cp-1
Cn-1
1 Lk=o 1
return: Lt, ( t-os (art
Mono-alignment Supervision. We take the L-step Euler solver's ODE trajectory { {/) as reference. We minimize the gap between the target and source trajectories with the MSE loss. We also adopt Huber loss as auxiliary supervision for Tix-
40-
Suhu Deforett Souch Mocelie S1oU2
Schoo with slep rofereeee oe tuke StaU2
18-
37/40
34-
38-
43
4.45
2 4.4
33-
6. Extending to DDPM/VP framework
Hembrr stolo
Applying our differentiable solver search to DDPM is infea- sible. However, (Song et al., 2020b) suggests that there ex- ists a continuous SDE process with {f(t) = 3:: g(t vBr corresponding to discrete DDPM. This motivates us to transform the search space from the infeasible dis- crete space to its continuous SDE counterpart. (Lu al., 2022) and (Zhang & Chen, 2023) discover the semi-linear structure of diffusion and propose exponential integral with parametrization to tackle the fast sampling prob- lem of DDPM models, where 0g = elá 10,ds de
(a) FID of Search Model
(b) FID of RefTraj Steps
Figure 3: Ablations studies of Differentiable Solver
Search, We evaluate the searched solver SiT-XL/2, and report the FID performance curve of searched solvers.
& Xie, 2023) trained with DDPM scheduling and rectified- flow models SiT-XL/2 (Ma et al., 2024) and FlowDCN. XI.2 (Wang et al., 2024b). Our default training setting employs the Lion optimizer (Chen et al., 2024c) with constant learning rate of 0.01 and no weight decay. We sample 50,000 images for the entire search process. No- tably, searching with 50,000 samples using FlowDCN-B/2 requires approximately 30 minutes on x H20 computa- tion cards. During the search, we deliberately avoid using CFG to construct reference and source trajectories, thereby preventing misalignment.
V V1-el elo -3,ds and A = log d: (Lu et al., 2023) fur- ther discovers that x parametrization is more powerful for diffusion sampling under limited steps, where 21-06
Ø: xx Fs + oe. 1"' e' (()))a) (18)
22 =
We opt to follow the 7 parametrization as DPM-Solver++. However, we find directly interpolating (x d) as a whole part is hard for searching, and yields worse results. To avoid conflating the interpolation coefficients with expo- nential integral, we employ Wa ai and transform Equa- tion (18) into Equation (19) with similar interpolation format as Equation (15), where t(w) maps to timestep.
7.1. Rectified flow models
We search solvers with FlowDCN-B/2, FlowDCN-S/2 and SIT-XLL/2. We compare the search solver's performance with the second-order and fourth-order Adam multistep method on SiT-XL/2, Flow DCW-XL/X trained on ImageNet 256 x 256, and FlowDCN-XX./2 trained on ImageNet 512 x 512.
a,
Search Model. We tried different search models among different sizes and architectures. We report the FID per- formance of SiT-XL/2 in Figure 3a. Surprisingly, we find that the FID performance of SiT-XL/2 equipped with the solver searched using FlowDCN-B/2 outperforms the solver searched on SiT-XL/2 itself. The reconstruction error(in the Appendix) between the sampled result produced by Euler- 250 steps is as expected. These findings suggest that there exists a minor discrepancy between FID and the pursuit of
R L. + orlwu- w.) 0a
Σ
cas 1=1
(19)
7. Experiment
We demonstrate the efficiency of our differentiable solver search by conducting experiments on publicly available dif- fusion models. Specifically, we utilize DIT-XL/2 (Peebles
6

---

Differentiable Solver Search for Fast Diffusion Sampling
Deformance STAL2
Parformance achvers on Row.CCH-A
Perfrrance Flaw2Cha.221122555
10
20
Adams Sahar Searc Salvar
calaar acched Solver
Adam
Soarthed So
18-
Namber seoppo
Rumber stap
Hamber of whip
(a) SIT-XL/2-R256
(b) FlowDCN-XX12-R2256
(c) FlowDCN-XI/22R55
Figure 4: The same searched solver on different Rectified- Flow Models. R256 R512 indicate the generation resolution of given model. We search solver with FlowDCN-B/2 ImageNet-256 x 256 and evaluate it with SiT-XL/2 and FlowDCN-XL/2 on different resolution datasets. Our searched solver outperforms traditional solvers by significant margin. More metrics(sFID, IS, Precision, Recall) are places at Appendix
minimal error in the current solver design.
SiT-XL-R223
NFE-CFG
Params
FID
IS
Step of Reference Trajectory. We provide reference trajec- tory {=)== of different sampling step L for differentiable solver search. We take FlowDCN-BB as the search model and report the FID measured on SiT-XL/2 in Figure 3b. As the sampling step of the reference trajectory increases, the FID SiT-XL/2 further improves and becomes better. How- ever, the performance improvement is not significant when the number of steps is 5 which suggests that there is limit to the improvement achievable with extremely small number of steps.
Heun
16x2
3.68
Heun
22x2
2.79
Heun
30x2
2.42
Adam2
15x2
/
2.49
236
Adam2
16x2
0
2.42
237
Adam4
15x2
/
2.33
242
Adam4
16x2
2.27
243
FlowTurbo
(743)x2
2.9 107
3.93
224
Flow
(8+2)x2
2.9 107
3.63
FlowTurbo
(12+2)x2
2.9 x 107
2.69
ImageNet 256 x 256. We validate the searched solver on SiT-X1/2 and FlowDCN-XI/2. We arm the pre-trained model with CFG of 1.375. As shown in Figure 4a, our searched solver improves FID performance significantly and achieves 2.40 FID under 10 steps. As shown in Fig- ure 4b, searched solver achieves 2.35 FID under 10 steps, beating traditional solvers by large margins.
FlowTurbo
(1743)x2
2.9 x 107
2.22
248
ours
6x2
21
3.57
214
ours
7x2
28
2.78
229
ours
8x2
36
2.65
234
ours
10x2
55
2.40
238
ours
15x2
55
2.24
244
We also provide the comparison with recent solver-based distillation (Zhao et al., 2024) to demonstrate the efficiency of our searched solver in Table Our searched solver achieves better metric performance under similar NFE with fewer parameters.
Table 1: Comparison with the distillation- methods. Our searched solver achieves much better result under the same NFE with much fewer parameters.
generation. alleviate the grid pattern, we decouple the velocity field into mean and direction, only apply our solver to the direction, and replace the mean with an exponential decayed mean. The details can be found in the appendix.
ImageNet 512 x 512. Since (Ma et al., 2024) has not re- leased SiT-XL/2 trained on 512 x 512 resolution, we directly report the performance collected from FlowDCN-XL./2 We arm FlowDCN-XL.. with CFG of 1.375 and four chan- nels. Our searched solver achieves 2.77 FID under 10 steps, beating traditional solver by large margin, even slightly outperforming the Euler solver with 50 steps(2.81FID).
We also provide the result of distillation on SDI.5 and SDXL with solver search in Appendix F.
7.2. DDPM/VP models
Text to Image. Shown in Figure 2, we apply our solver search on FlowDCN-B/ and SiT-XL/2 to the most advanced Rectified-Flow model Flux. 1-dev and SD3 (Esser et al., 2024). We find Flux.1-Dey would produce grid points in
We choose the open-source DiT-XL/2(Peebles & Xie, 2023) model trained on ImageNet 256 x 256 as the search model to carry out the experiments. We compare the performance of the searched solver with Dol-Steler++ and UniPC on

---

Differentiable Solver Search for Fast Diffusion Sampling
Sampling Steps
DPMSolver++
UniPC
DS-Solver
boat sailing of body of water
bus that sitting the street
DPMSolver++
UniPC
DS 05-Solver
A close of a plate of food containing broccoli
A chicken sandwich in wrapper near cell phone
Figure 5: The images generated from PixArt-E with CFG=2.0 equipped with Our DS-Solver ( searched on DiT-XI/2--R25
In comparison to DPM- + and UniPC, our results consistently exhibit greater clarity and possess more details. Our solver achieves better quality from 5 to 10 steps(NFE).
Methods \NFEs
5
7
9
10
DPM-Solver++ with uniform-X (Lu et al., 2023)
38.04
20.96
14.69
11.09
8.32
6.47
DPM-Solvent+ with uniform-f (Lu al., 2023)
31.32
14.36
7.62
4.93
3.77
3.23
DPM-Solvert+ with uniform-A-oot (Xue et al., 2024)
12.53
5.44
3.58
7.54
5.97
4.12
DPM-Solvert+ with uniform-f-opt (Xue et al., 2024)
12.53
5.44
3.89
3.81
3.13
2.79
UniPC with uniform-A (Zhao 2023)
41.89
30.51
19.72
12.94
8.49
6.13
UniPC with uniform-t (Zhao al., 2023)
23.48
10.31
5.73
4.06
3.39
3.04
UniPC with uniform-A-opt (Xue al., 2024)
8.66
4.46
3.57
3.72
3.40
3.01
UniPC with uniform-t-opt (Xue al., 2024)
8.66
4.46
3.74
3.29
3.01
2.74
Searched-Solver
7.40
3.94
2.79
2.51
2.37
2.33
Table 2: FID (L) of difffrent NFES on DIT-XL/2-R256
-pp indicates online optimization of the timesteps scheduler.
Methods \NFEs
5
UniPC with uniform-A (Zhao al., 2023)
41.14
19.81
13.01
9.83
8.31
7.01
UniPC with uniform-I (Zhao et al., 2023)
20.28
10.47
6.57
5.13
4.46
4.14
UniPC with uniform-X-opt (Xue et al., 2024)
11.40
5.95
4.82
4.68
6.93
UniPC with uniform-f-opt (Xue al., 2024)
11.40
5.95
4.64
4.36
4.05
3.81
Searchedd-oolver((sarrheee on DYT-X1/2-R255)
10.28
6.02
4.31
3.74
3.54
3.64
Table 3: FID (L) of different NFEs DiT-XI/22-R512. -opt indicates online optimization of the timesteps scheduler.
ImageNet 256 x 256 and ImageNet 512 x 512.
ImageNet 256 x 256. Following (Peebles & Xie, 2023) and (Xue et al., 2024), We arm pre-trained DiT-XL/22 with
8

---

Differentiable Solver Search for Fast Diffusion Sampling
GenEval Metrics
7.3. Visualization of solver parameters
Solver
Steps CFG
Overall
Color.Attr Two.Obj Pos
Searched Coeflicients are visualized in Figure 1. The absolute value of searched coefficients corresponding to DDPM/VP shares different pattern, coefficients in DDPM/VP are more concentrated on the diagonal while rectified-flow demonstrates a more flattened distribution. This indicates that there exists a more curved sampling path in DDPM/VP compared to rectified-flow
2.0
6.50
33.08
4.75 0.40519 19
DPM++
8
2.0
5.25
39.65
5.75 0.43074
2.0
6.50
34.85
5.25 0.41387
UniPC
2.0
6.72
40.66
6.00 0.44134
5
2.0
5.25
37.37
4.75 0.41933
Ours
8
2,0
7.25
42.68
7.50 0.45064
Searched Timesteps are visualized in Figure 1. Compared to DDPM/VP, rectified-flow models focus more on the more noisy region, exhibiting small time deltas at the beginning. We fit the searched timestep of different NFE with polyno- mials and provide the respacing curves as follows:
Table 4: Results on GenEval Benchmark for PixArt at 512 Resolution.Ou searched solver achieves better performance compared with UniPC/DPM++ on PixArt-512 x 512.
Steps
FID
sFID
IS
PR
Recall
ReFlow : - 1.96/* + 3.51 0.97/ + 0.431,
(20)
DDPM/VP: - 2.73/ + 6.300 - 4.7446 + 2.171, (21)
DPM++
5
60.0
209
25.59
0.36
0.20
+
8
38.4
116.9
33.0
0.50
0.36
E [0, 1]. and t indicates the most noisy timestep.
DPM++
10
35.6
114.7
33.7
0.53
0.37
UniPC
5
57.9
206.4
25.88
0.38
0.20
8. Conclusion
UniPC
8
37.6
115.3
33.3
0.51
0.36
UniPC
10
35.3
113.3
33.6
0.54
0.36
We have found compact solver search space and proposed novel differentiable solver search algorithm to identify the optimal solver. Our searched solver outperforms traditional solvers by a significant margin. Equipped with the searched solver, DDPM/VP and Rectified Flow models significantly improve under limited sampling steps. However, our pro- posed solver still has several limitations which we plan to address in future work.
Ours
5
46.4
204
28.0
0.46
0.23
Ours
8 33.6
115.2
32.6
0.54
0.39
Ours
10
33.4
114.7
32.5
0.55
0.39
Table 5: Metrics of different NFEs PixArt-a (Our Solver are searched on ImageNet 256x256)
CFG of 1.5 and apply CFG only on the first three channels. As shown in Table 2, our searched solver improves FID performance significantly and achieves 2.33 FID under 10 steps.
9. Limitations
We have demonstrated text-to-image visualization with a small CFG value. However, it intuitive that using a larger CFG would result in superior image quality. We attribute the inferior performance of large CFGs in our solver the limitations of current naive solver structures and searching techniques. We hypothesize that incorporating predictor- corrector solver structures would enhance numerical sta- bility and yield better images. Additionally, training with CFGs may also be beneficial.
Net 512 x 512. We directly apply the solver searched on 256 x 256 resolution to ImageNet 512 x 512. The result is also great to some extent, 22112 x 512) achieves 3.64 FID under 10 steps, outperforming DPM-Solver++ and UniPC with large gap.
Text to Image. As we search solver with DIT and its corresponding noise scheduler, so it is infeasible to apply our solver to other DDPM models with different 8min and 3max. Fortunately, we find (Chen et al., 2024b) and (Chen et al., 2023) also employ the same Smin and 3mux as DIT. So we can provide the visualization results of our searched solver on PixArt-E and PixArt-a. Our visualization result produced with CFG of 2. We take PixArt-alpha as the text-to-image model. We follow the evaluation pipeline of ADM and take COCO17- 7-Va Val as the reference batch. We generate 5k images using DPM-Solver++, UniPC and our solver searched on DiT-XL/2-R256. Also, we provided the performance results on GenEval Benchmark (Ghosh et al., 2023) in Section 7.2.
Impact Statement
This paper proposes a search-based solver for fast diffusion sampling. We acknowledge that it could lower the barrier for creating diffusion-based AIGC contents.
This work is supported by the National Key R&D Program of China (No. 0022ZD0160900) Jiangsu Frontier Technology Research and Development Program (No. BF2024076). and the Collaborative Innovation Center of Novel Software Technology and Industrialization, Alibaha Group through the Alibaba Innovative Research Program.
Acknowledgments.

---

Differentiable Solver Search for Fast Diffusion Sampling
References
Lin, S., Wang. A., and Yang. X. Sdxl-lighming: Progressive adver- sarial diffusion distillation. arXiv preprint 2iti::2402.1332222 2024.
Anderson, B. D. Reverse-time diffusion equation models. Stochas- tic Processes and their Applications, 12(3):313-326, 1982.
Lipman. Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02777, 2022.
Bashforth, and Adams, J. C. An attempt test the theories of copillary action by comparing the theoretical and measured forms of drops of fluid. University Press, 1883.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 3XXv:2209003003 2022.
Brock, A., Donahue, J., and Simonyan, Large scale gan train- ing for high fidelity natural image synthesis. arXiv preprint arXiv:1809) 1096, 2018.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315 2022.
Lu, C., Zhou, Y., Bao, F., Chen, J., LI, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances Neural Information Processing Systems, volume 35, pp. 5775-5787, 2022.
Chen, D., Zhou, Z., Wang, C.. Shen, C., and Lyu, S. On the trajectory regularity of ode-based diffusion sampling. arXiv preprint arXiv:2405.11322 2024a.
Lu, C., Zhou, Y., Bao, F. Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion proba- bilistic models, 2023.
Chen, J., Yu, J., Ge, C., Yao, I.,, Xie, E., Wa, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-)alpha Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.000
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint ar 240100000 2024.
2023.
Chen, J., Ge, C.,, Xie, E., W1, Y. Yao, I.., Ren, X., Wang. 7., Luo, P., Lu, H., and Li, Z. Pixart- Pixart-\sigmar Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXivv2403.000 04692, 2024b.
Peehles, W. and Xie, S. Scalable diffusion models with transform- ers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4195 4205, 2023.
Chen, X., Liang. C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., I Luong. T., Hsich, C.-J., Lu, Y,, Symbolic discov- ery of optimization algorithms. Advances in neural information processing systems, 36, 2024c.
Sabour, A., Fidler, S., and Kreis. K. Align your steps: Optimiz- ing sampling schedules in diffusion models. arXiv preprint arXiv:2404.1 14507, 2024.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J.. Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint 2rXiv:2433.03206 2024.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:200.000512, 2022.
Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1 1-10, 2022.
Gao, Y., Pan, Z., Zhou, X., Kang, L., and Chaudhari, P. Fast diffu- sion probabilistic model sampling through the lens of backward error analysis. arXiv preprint arXiv:2304.1444 2023.
Shaul, N., Perez, J., Chen, R. T., Thabet, A., Pumarola, A., and Lipman, Y. Bespoke solvers for generative flow models. arXiv 2023.
Ghosh, D.,, Hajishirzi, H. and Schmidt, L. Geneval: An object- focused framework for evaluating text-to-image alignment. Ad- vances in Neural Information Processing Systems, 36:52132- 52152, 2023.
preprint 2310
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv:2010.02502, October 2020a. URL https: //arxiv. wecivivorrggabae absaaaas//100 C2502.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusson probabilistic models. Advances in neural information processing xy syxtems, 15:6840-651,, 2020.
Song, T., Feng, W., Wang. S., Li, X., Ge, T., Zheng, B., and Wang, L. Dmm: Building a versatile image generation model via distillation- model merging. arXiv preprint arXiv:2504.11 123644............. 2025.
Kang, M., Zhang, R., Barnes, C., Paris, S., Kwak, S., Park, J., Shechtman, F.., Zhu, J.-Y., and Park, T. Distilling diffusion models into conditional gans. arXiv preprint arXiv:2405.05961 2024.
Song, Y., Sohl- Sick-sDikst,,n, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score- generative modeling through stochas- tic differential equations. arXiv preprint arXiv:2011.1345. 2020b.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577. 2022.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consistency models. arXiv preprint arXiv:2303.01 01130.0000499...0 2023.
Kim, D.,. Lai, C.HH., Liao, W.H.. Murata, N., Takida, Y., Uesaka, T. He, Y., Mitsufuji, Y., and Ermon, S. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXi:22100022. 2023.
Wang, F.-Y., Huang, Z., Bergman, A. W., Shen, D., Gao, P., Lin- gelbach, M., Sun, K., Bian, W., Song, G., Liu, Y., al. Phased consistency model. arXiv preprint 2405 200, 2024a.
10

---

Differentiable Solver Search for Fast Diffusion Sampling
Wang, S., Teng, Y., and Wang, L.. Deep equilibrium object detec- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6296-6306, 2023.
Wang. S., Li, Z., Song. T., Li, X., Ge. T., Zheng, B., and Wang. L. Flowden: Exploring den-like architectures for fast image genera- tion with arbitrary resolution. arXiv preprint arXiv:2410.22633 2024h.
Wang. S., Tian, Z., Huang, W., and Wang, L. Ddt: Decoupled diffusion transformer. arXiv preprint arXiv:2504.0574) 2025.
Xu, C., Song, T., Feng, W. Li, X., Ge, T., Zheng, B., and Wang. L. Accolerating image generation with sub-path linear approxima- tion model. In ECCV, pp. 323-339, 2024.
Xue, S., Liu, Z.,, Chen, F., Zhang, S., Hu, T., Xic, F.,, and Li, 7., Accelcrating diffusion sampling with optimized time steps. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8292 8301, 2024.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Free- man, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pp. 6613- 6623, 2024.
Zhang, and Chen, Y. Fast sampling of diffusion models with ex- ponential integrator. In The Eleventh International Conference on Learning Representations, 2023.
Zhao, W. Bai, L., Rao, Y., Zhou, J., and Lu, J. Unipe: A unified predictor-correctoo framework for fast sampling of diffusion models. arXiv preprint arXiv:2332..044 048677 2023.
Zhao, W. Shi, M.. Yu. X., Zhou. J.. and Lu, J. Flowturbo: Towards real-time flow-based image generation with velocity refiner. arXiv preprint arX9v:2409. 18/28, 2024.
Zheng, J. Hu, M., Fan, Z., Wang, C., Ding, C., Tao, D., and Cham, T.J.. Trajectory consistency distillation, arXiv preprint arXiv:2402. 19159, 2024.
Zhou, M., Zheng, H., Wang, Z., Yin, M., and Huang, Score identity distillation: Exponentially fast distillation pretrained diffusion models for one-step generation. In Forty-first Interna- tional Conference on Machine Learning, 2024.

---

Differentiable Solver Search for Fast Diffusion Sampling
A. More Metrics of Searched Solver
We adhere to the evaluation guidelines provided by ADM and DM-nonuniform, reporting only the FID as the standard metric in Figure 4a. To clarify, we do not report selective results rectified flow models; present sFID, IS, PR. and Recall metrics for SiT-X SR25XL(R256) HowDDCN-XL2(RR2)
and FlowDCN- . Our solver searched on FlowDCN- consistently outperforms the handcrafted solvers across FID, sFID, IS, and Recall metrics.
B. Computational complexity compared to other methods.
For sampling. When performing sampling over y time steps, our solver caches pre-sampled predictions. resulting memory complexity of (n). The model function evaluation also has a complexity of ((n) (O(2 x n) for CFG enabled). It is important to note that the memory required for caching predictions is negligible compared to that used by model weights and activations. Besides classic methods, have also included comparison with the latest Flowturbo published NeurIPS24.
Steps
NFE
NFE-CFG
Cache Pred
Order
search samples
Adam2
2n
2
2
Adam4
2n
4
4
/
heun
2n
4n
2
2
1
DPM-Solver++
2n
2
2
/
UniPC
2n
3
3
Flow Turbo
>n
>2n
2
2
540000(Real)
our
2n
n
n
50000(Generated)
For searching. Solver-based algorithms, limited by their searchable parameter sizes, demonstrate significantly lower performance few-step settings compared to distillation based (oorithms(556sttpp)) making direct comparisons inappropriate. Consequently, we selected algorithms that are both acceleratable on ImageNet and comparable in performance, including popular methods such as DPM-Sole Sole--..rrr+... UniPC, and classic Adams-like linear multi-step methods, Since experiments primarily utilize SiT, DIT, and FlowDCN that trained on the ImageNet dataset. We also provide fair comparisons by incorporating the latest acceleration method, FlowTurbo. Additionally, we have included results from the heun method as reported FlowTurbo
C. Ablation on Search Samples
We ablate the number of search samples on the 10-step and 8-step solver settings. Samples means the total training samples the searched solver has seen. Unique Samples means total distinct samples the searched solver has seen. Our searched solver converges fast and gets saturated near 30000 samples.
iters( 10---t---solverrr
samples
unique samples
FID
IS
PR
Recall
313
10000
10000
2.54
239
0.79
0.59
626
20000
10000
2.38
239
0.79
0.60
939
30000
10000
2.49
240
0.79
0.59
1252
40000
10000
2.29
239
0.80
0.60
1565
50000
10000
2.41
240
0.80
0.59
626
20000
20000
2.47
237
0.78
0.60
939
30000
30000
2.40
238
0.79
0.60
1252
40000
40000
2.48
237
0.80
0.59
1565
50000
50000
2.41
239
0.80
0.59
D. Solver Across different variance schedules
Since our solvers are searched on a specific noise scheduler and its corresponding pre-trained models, applying the searched coefficients and timesteps to other noise schedulers yiclds meaningless results. We have tried applied searched solver on SiT(Rectified flow) and DiT(DDPM with 8min 0.1, 3maz 20) to SD1.5 with 8min 0.085, 8max = 12), but the results were inconclusive. Notably, despite sharing the DDPM name, DiT and SD1.5 employ distinct 8min: 8maz values, thereby featuring different noise schedulers. more in-depth discussion these experiments can be found Section(Extent) to DDPM/VP).
12

---

Differentiable Solver Search for Fast Diffusion Sampling
Performence solvers ST3U/
Performance sclvers 20
Performance solvers FRowCCN..P1S122551
Alan Silee
Alert Sulee
Narrt Seivet
18-
Akrrs Sohkr
22
12
15-
10-
Dotatt 48
(a) SIT-XL/2-R256
(b) FlowDCN-XL/--R22
(c) NRWDDCN-XL/2-BR
Performance of solvers on STTu22
Performanca of solvers on Fo0D0C--822222662205
Performancc of solkers on Flow 20.2151245120
290
200
240
228
117
Adami
Adams
130
Adamn-Sc
Searche Gelver
Sourched Sol
Searched Sol
132
(d) SiT-XL/2-R256
(e) Flow.DCN-XLL--R2S
(I) FLoWDCN-XL/2-R512
Performance of solvers on STAU
Pe formance of solvers on How www.
Perfo-mance of scheess on 0U00000000221
Acano-cooer Suecched Salver
064
Salvce
0422
volvor
Scariied Sobot
3.74
E.lr130lippo 3.35
Sumber
Akndar
Murbber
(g) S/T-XIL/2-R256
(h)
(i) FlowDDCN-XI/2-R511
Performance solve ST3222
formance selvers FlowOC 25220550200
Performane RlweOO XL 2151220122
0.49
moptvrr Sdwe diee deacced
Cdla- 200lap00-p---- 3.51
0355
0.52
0.5
0353
3.50
0.6
3.5
3.59
0.54
854
0402
652
Ademt S0lnd
Adanz Silee
Adars4
650
0.00
04435
Nenber steps
Sumber statoo
(i) SIT-XL/2-R256
(k) Fllo-DCN--LL2--222
(1) FlowDCN-XL/2-R112
Figure 6: The same searched solver on different Rectified-Flow Models. R256 and R512 indicate the generation resolution of given model. We search solver with FlowDCN-B/2 on ImageNet-256 x 256 and evaluate it with SiT-XT/2 and FlowDCN-XI/2 on different resolution datascts. Our scarched solver outperforms traditional solvers by significant margin.
13

---

Differentiable Solver Search for Fast Diffusion Sampling
iters(8-step-solve)
samples
unique samples
FID
IS
PR
Recall
313
10000
10000
2.99
228
0.78
0.59
626
20000
10000
2.78
229
0.79
0.60
939
30000
10000
2.72
235
0.79
0.60
1252
40000
10000
2.67
228
0.79
0.60
1565
50000
10000
2.69
235
0.79
0.59
626
20000
20000
2.70
231
0.79
0.59
939
30000
30000
2.82
232
0.79
0.59
1252
40000
40000
2.79
231
0.79
0.60
1565
50000
50000
2.65
234
0.79
0.60
E. Solver for different variance schedules
As every DDPM has corresponding continuous VP scheduler, we successfully searched better solver compared to DPM-Solvers The details can be found Section put simply, under the empowerment our high-order solver, the performance DDPM and FM does not differ significantly (8, 9, 10 steps), which contradicts the common belief that FM is stronger at limited sampling steps.
transform the discreet DDPM into continuous VP, thus
F. Text to image Distillation Experiments
We unify distillation and solver search to obtain high-quality multi-step generative models. We adopt adversarial training and trajectory supervision. We will open source the training code of unified training techniques.
Table 6: Performance comparison on validation set of COCO-2017.
Method
Res.
Time (4)
Steps
Param.
FID (L)
DDvMSseerr (Upper-Bound) (Lu et al., 2022)
512
0.88s
25
20.1
Rectified Flow
512
0.88s
25
0.9B
21.65
Rectified Diffusion
512
0.88s
25
0.9B
21.28
Rectified Flow
512
0.21s
4
0.9B
103.48
PcRFlow Rectified Diffusion
512
0.21s
0.9B
22.97
512
0.21s
0.9B
20.64
Durs(Distillaiion-sotee search)
512
0.21s
4
0.9B
18.99
PcRFlow-SDDX
1024
0.71s
4
3B
27.06
Rectified Diffusion-SDXL.
1024
0.71s
4
3B
25.81
OurxLLLORA--Diiillliinn----oeo search)
1024
0.71s
21.3
Table 7: Performance comparison COCO-2014.
Method
Res.
Time (L)
# Steps
# Param.
FID (L)
Stable Diffusion XL (3B) and its accelerated distilled versions
SDXL--Turbo
512
0.34s
4
3B
23.19
SDXL-Lightning
1024
0.71s
4
3B
24.56
DMDv2
1024
0.71s
4
3B
19.32
LCM
1024
0.71s
4
3B
22.16
Phased Consistency Model
1024
0.71s
4
3B
21.04
PeRFlow-XL
1024
0.71s
4
3B
20.99
Rectified Diffusion-XL (Phased)
1024
0.71s
4
3B
19.71
OuusLLORA-DDstilaafon---ove search)
1024
0.71s
4
3B
11.4
14

---

Differentiable Solver Search for Fast Diffusion Sampling
G. Limitations.
We place the limitation at the appendix, in order to provide more discussion space and obtain more insights from reviews. We copy the original limitation content and add more.
Misalignd Reconstrucion loss and Performance. Our proposed methods are specifically designed minimize integral error within a limited numher of steps. However, ablation studies reveal a mismatch hetween FID performance and Reconstruction error. To address this issue, we plan to enhance our searched solver by incorporating distribution matching supervision, thereby hetter aligning sampling quality.
Larger CFG Inference. In the main paper. we demonstrate text- to image visualization with small CFG value. However, it is intuitive solver
that utilizing larger CFG would result in superior image quality. We attribute the inferior performance of large CFGs the limitations of current naive solver structures and searching techniques. We hypothesize that incorporating wedictor-correctu solver structures would enhance numerical stability and yield better images. Additionally, training with CFGs may also be beneficial.
Resource Consumption We can hard code the searched coefficients and timesteps into the program files. However, Compared to hand-crafted solvers. our solver still needs a searching process.
H. Proof of pre-integral error expectation
Theorem H.I. Given sampling time interval [ts, te+1] and suppose C5(x) = 95(2)) + b2, Adams-like linear multi-step methods will
introduce upper error bound of (tu41 - -x,)Be, 2i (2,)(
Our solver search(replacing (x(x) with 12,,, [C, (x:))) owns an upper error bound of (1++1-1:) f.) 1l Ei= و5 (ax)) F2,, Dj (a:))l
Proof. Suppose (ar,) = g, + Adams-like linear multi-step methods would not consider z-related interpolation. thus pre-integral coefficients of Adams-like linear multi-step methods will only reduce into b.
We obtain the error expectation of the pre-integral of Adams-like like linear multi-step step methods:
Bz.
Σ
(o) - -t) Σ (ti+1 t,)1l
(22)
-E2,1 Σ
(1 - (x)CC((x))) - b/ll
(23)
==
=(1e41 Em,
(x:))
(24)
f=0
We obtain the error expectation of the pre-integral of our solver search methods:
Ex
u (,:)(())) - 4.) Σ
Un (2:))(((44 4)ll
(25)
1=0
1=
=E,,1
Σ
u,(ti+1 - (a:)) FT, (=;))l
(26)
1=0
= [ta+1 t:) Ear
Σ
vsl2)(2x.) Em, (x:))l
(27)
1=0
Next, define the optimization problem:
F. E,n Σ n La (a:) a,11.3
We suppose different vy are orthogonal and optimal.
||0y||3 = 1. As we leave of as the expectation of C3 (2xs), we will demonstrate this choice
aE 2E,, (11s)15(0s(2z)) - a;))
(28)
0a,
ag = 0, we obtain: oj Ex, 9. (2,) I1, Let Bay
Ee 9, (xx.) - Em, (x.)) 6/.
So our searched solver has lower and optimal enror expectation:
(ti=1 Be, Σ vy (x))) - Ez, on (2x))l < (tit1 - t,) Σ (x.))
(29)
15

---

Differentiable Solver Search for Fast Diffusion Sampling
Recall Assumption 4.1, the integral upper error bound of universal interpolation P will be:
vlare, t)dt
Dj
(30)
5=0
||
v(art, 1)dt
P L, 2,,,,
(31)
G11
(0(22,,))
E
Plax, t, n. |d
(32)
J-U
4+ [|u(2rr))
<
Σ Plan, 1, 4,
(33)
(0411-4400 4)(0(de")) (dd)))
(34)
Combining Equation (34) and the error expectation of the pre-inter pre-integral part, we will get the total error bound of the solver search.
oisee
t)dl
EC,
4)]l.
(35)
1=0
144
(
-11
v( l)dt - Σ
Us
2(0,,1,25,15,)
(36)
5=0
Σ 1
t
P, tj)dt
Σ oy (ox 4.]1
(37)
1=0
4:11
<|1
o( t)dt Σ Us
2,,,
(38)
3-0
Σ ==0
Uj
Plxs, L, 2,, 1y)ct
Σ
En (() - 4.)1.
(39)
1=0
M++1
v( t)dt Σ Us
Pla,,t,,,,,,
(40)
3=0
- t:)
Σ Uソ Em, [C)(a:))(tt+1 t:)ll-
(41)
1=
<(t++1 - -1---+)) (0)((d + (dtt)) (te+1 Ea; Σ (x.) Fm, (x.))l
(42)
1=0
<(t - 4)((0(e=")) + O(tt") Em;
(x.)) - Em, Es
(43)
Since (dOn"=")) + Odt")) is much smaller than Ez, 11) uy (a))
E,, E,, (ax,)]l We can omit the ((O(d=") Odtt)) term.
I. Proof of total upper error bound
Theorem L.1 Compared to Adams-like linear multi-step methods. Our Solver search has small upper error bound.
The total upper error bound of Adams-like linear multi-step methods is:
Σ
nbb
Σ
2y (x..)))
DD
1-0
16

---

Differentiable Solver Search for Fast Diffusion Sampling
The total upper ervor bound of Our solver search
Σ (L+11 4)) Σ n|Ea + bgl + Ea, Σ oago (a:) - 9a (as)]l)
Proof. We donate the continuous integral result of the ideal velocity field as æ. the solved integral result the ideal velocity field as v. the continuous integral result of the pre-trained velocity model Uh as è, the solved integral result of the pre-trained velocity model vp as a.
N-1 N -+ Σ Σ e,cc(tur11 -4)
(44)
The caused by the non-ideal velocity estimation model can he formulated in the following equation. we employ triangular incqualities to obtain the error-hound | xx ll, which related to solver coefficients and timestep choices.
N N-1
| xx
Σ v (,)c t.)l
<=0
f=
(wy - 05)c(tt11 t:)l
N-1
Uy ا,)ة (c/44 4))
0= ==
N-1
1 (1i+1 1:)l
1=0 1=
The total error of our searched solver is:
k Ø||
IN &x + ΦN - =11
IN .N
+ ||ā: +
N-1
cdd (fi+1 4:)[+
1=0 1=
N-1
(4+1 - (0)(0(d O(dl") + Em; Σ 1 [gs (zx.) - En,9 (x:)]]
(1 -2) 4i) (tit1 Ro, Σ
my (x.) (a;)((m;))l
1=0
==
1=
Σ (tum1 f.) Σ 1e + b/1 Bz,
Σ
1 (xx.) Bz,g, (x:))l III)
j=0
The total error of Adams-like linear multi-step method is:
nb
+ F.r,
Σ my (x::))]])))
1=0
==
==0
Obviously, as (Ej=o n/6/1 Ex, 11 vy (x;)) [I) is not equal between different timestep intervals, Optimized timesteps owns smaller upper error bound than uniform timesteps.
(x:))
Recall that 77 & ||ssl, the error is mainly determined by E,,, ES u 13 (#:)]ll-
Recall that Ex, || E) Vy (x.)) - 9, (x,))) < EB,, 1| Us (x.)) because search coefficients and timesteps simultaneously.
||. thus our solver search has minimal upper error bound
17

---

Differentiable Solver Search for Fast Diffusion Sampling
J. Searched Parameters
We provide the searched parameters A and c. Note ei needs to be converted into M follwing Algorithm 1.
J.1. Solver Searched on SiT-XL/2
NFE
TimeDeltas At
Coeffciintss
T
0.0
0.0
0.0
0.0
0.0
0.1225
-1177
0.0
0.0
0.0
0.0
5
0.2144
1.07
-188
0.0
0.3073
0.0
0.0
-0.33
0.0
0.0
00.3135
0.0
0.0
0.0
0.71
0.0
T0.03897
0.0
0.0
0.0
0.0
0.0
0.07
0.0976
-1.04
0.0
0.0
0.0
0.0
0.0
0.161
1.62
2.98
0.0
0.0
0.0
0.0
6
0,2046
1.32
2.52
-2.4
0.0
0.2762
0.0
0.0
0.0
-0.76
0.0
0.0
[0.2217
0.0
0.0
0.0
0.0
0.66
0.0
T0.0299
0.0
0.0
0.0
0.0
0.0
0.0
0.07
0.0735
0.93
0.0
0.0
0.0
0.0
0.0
0.0
0.1119
1.23
2.31
0.0
0.0
0.0
0.0
0.0
0.1451
0.59
1.53
-2.09
0.0
0.0
0.1959
-0.09
-0.07
0.99
1.91
0.0
0.0
0.0
0.2698
0.05
00.1
0.09
0.55
-147
0.0
0.0
0.1738
0.05
0.19
0.31
0.37
0.67
1.79
0.0
00.0303
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0702
-0.92
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0716
0.78
0.0
0.0
0.0
0.1112
0.06
0.52
-1.76
0.0
0.0
0.0
0.0
0.0
8
0.1501
-0.02
-0.16
0.98
-1.8
0.0
0.0
0.0
0.0
0.1833
-0.02
-0.12
0.22
0.24
-1.36
0.0
0.0
0.0
0.2475
0.1
0.06
0.02
0.18
0.12
-1.1
0.0
0.0
L0.1358
0.14
0.02
0.02
0.38
0.32
1.72
0.0_
0.028
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.07
0.0624
0.93
0.0
0.0
0.0
0.0
0.0
0.0
0.0717
0.63
1.29
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0894
0.39
0.11
-141
0.0
0.0
0.0
0.0
0.0
0.0
0.1092
-0.07
-0.05
0.83
-.59
0.0
0.0
0.0
0.0
0.0
0.1307
0.07
-0.11
0.27
0.27
-1.33
0.0
0.0
0,1729
-0.05
0.03
0.01
0.15
0.17
-1.5
0.0
0.0
0.0
0.2198
0.21
0.27
-0.07
-0.03
0.19
0.09
0.99
0.0
0.0
0..1159
0.15
0.15
0.03
0..9
0.25
0.25
0.21
-1.1
0.0
T0.02797
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.07
0.0479
-0.95
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0646
0.59
-117
0.0
0.0
0.0
0.0
0.0
0.0
0.0659
0.35
00.1
-1.45
0.0
0.0
0.0
0.0
0.0
0.0
0.1045
-0.3
0.01
0.75
-1.99
0.0
0.0
0.0
0.0
0.0
0.0
10
0.1066
0.05
0.05
0.31
0.29
1.59
0.0
0.0
0.0
0.0
0.0
0.1355
0.05
0.03
-0.09
0.23
0.17
1.19
0.0
0.0
0.0
0.0
0.1622
-0.03
-0.03
0.27
-0.03
-0.91
0.1942
-0.15
0.17
0.03
-0.09
0.05
0.09
0.05
-0.79
0.0
0.0
00.0908
-0.17
0.11
0.15
0.03
0.05
0.25
0.05
-0.07
-1.49
0.0,
18

---

Differentiable Solver Search for Fast Diffusion Sampling
J.2. Solver Searched FlowDCN-B/2
NFE
TimeDeltas A
Coeffcients ej
T0.0521
0.0
0.0
0.07
0.1475
-1.26
0.0
0.0
5
0.2114
1.38
2.26
0.0
0.0
0.0
0.2797
0.0
0.0
-0.92
0.0
0.0
0..0092
0.0
0.0
0.0
0.7
0.0
T0.0391
0.0
0.0
0.0
0.0
0.0
0.0
0.0924
-1.22
0.0
0.0
0.0
0.0
0.0
0.165
1.12
0.0
0.0
0.2015
0.3
0.9
1.56
0.0
0.0
0.0
0.2511
0.0
0.0
0.0
0.74
0.0
0.0
0.2551
0.0
0.0
0.0
0.0
00.2
0.0
T0.0387
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0748
1.11
0.0
0.0
0.0
0.0
0.0
0.0
0.103
1.03
-1.9
0.0
0.0
0.1537
0.07
0.43
-157
0.0
0.0
0.0
0.0
0,184
0.221
-0.15
1.53
2.29
0.0
0.0
0.0
0.234
0.05
0.07
-0,23
0.61
11.3
0.0
0.0
0.2117
0.17
0.31
0.41
0.17
0.59
1..1
0.0
10.0071
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0613
2.43
0.0
0.0
0.078
0.61
-155
0.0
0.0
0.0
0.0
0.0
0.0
0.1163
0.99
-0.11
-2.77
0.0
0.0
0.0
0.0
0.0
0.1421
0.05
-0.49
1.33
-1.93
0.0
0.0
0.0
0.0
0.188
0.05
-0.3
0.23
0.73
-1.71
0.0
0.2077
0.09
0.05
0.61
-1.45
00.1996
0.23
0.21
0.01
0.25
0.25
0.41
1.25
0.0,
70.0017
0.0
0.0
0.0
0.0
0.0
0.0
0.051
-6.19
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0636
-0.11
0.81
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0911
0.73
0.17
-1.37
0.0
0.0
0.0
0.0
0.0
0.0
0.1007
0.31
-0.05
0.19
-..4
0.0
0.0
0.1443
0.03
0.23
0.29
0.35
-1335
0.0
0.0
0.0
0.0
0.1694
0.19
0.05
0.01
0.21
0.25
-123
0.0
0.0
0.0
0.191
0.23
0.21
-0.13
0.17
0.09
0.09
-1.09
0.0
0.0
.0.1872
0.17
0.15
0.11
-0.19
0.03
0.23
0.17
11.21
0.0]
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0538
7.8801
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0347
-0.4
-0.74
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0853
0.48
0.18
0.86
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0853
0.26
0.04
00.0
-1.28
0.0
0.0
0.0
0.0
0.0
0.0
0.1198
0.0
0.06
0.26
0.26
1..2
0.0
0.0
0.0
0.0
0.0
0.1351
-0.1
0.06
0.08
0.2
-1.24
0.0
0.0
0.0
0.165
-0.18
0.14
-0.08
0.1
0.08
0.14
-1.6
0.0
0.0
0.0
0.1788
0.12
0.16
-0.1
0.04
0.08
0.06
0.08
-1.02
0.0
0.0
00.1466
0.16
0.02
0.14
0.0
-0.14
0.08
0.14
0.34
-138
0.0
19

---

Differentiable Solver Search for Fast Diffusion Sampling
J.3. Solver Searched DiT-X11/2
NFE TimeDeltas Al
Coeffcients cj
00,2582
0.0
0.0
0.0
0.0
0.1766
-143
0.0
0.0
5
0.1766
0.93
-1.55
0.0
0.0
0.0
0.2156
0.0
0.0
-0.69
0.0
0.0
0.1731
0.0
0.0
0.0
0.59
0.0
T0.24837
0.0
0.0
0.0
0.0
0.0
0.07
0.1506
-1.36
0.0
0.0
0.0
0.0
0.0
0.1476
1.84
0.0
0.0
6
0.1568
0..8
0.5
1.08
0.0
0.0
0.0
0.1733
0.0
0.0
0.0
0.56
0.0
0.0
0.1233
0.0
0.0
0.0
0.0
-0.56
0.0
T0.2241
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1415
-1.38
0.0
0.0
0.0
0.0
0.0
0.0
0.1205
2.02
0.0
0.0
0.0
0.0
7
0.1158
-0.28
0.78
- 1.52
0.0
0.0
0.0
0.0
0.1443
1.4901e 08
-0.1
0.64
-15
0.0
0.0
0.0
0.1627
0.06
0.06
-0.06
0.26
1.0
0.0
0.0
0.0911
0.0
0.1
0.02
0.2
0.26
1.12
0.0
00,2033
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1476
-1.14
0.0
0.0
0.0
0.1094
0.8
-177
0.0
0.0
0.0
0.0
0.0
0.0
0.099
0.02
0.48
-1.62
0.0
0.0
0.0
0.0
0.0
8
0.1116
-0.12
0.06
0.62
-1.2
0.0
0.0
0.0
0.0
0.1233
0.04
-0.1
0.12
0.16
-104
0.0
0.0
0.0
0.131
0.06
0.04
0.06
0.08
0.08
0.56
0.0
00.0748
0.02
0.04
0.04
0.12
0.14
0.04
0.9
0.0,
0.19957
0.0
0.0
0.0
0.0
0.0
0.0
0.1313
-1.28
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1142
0.78
-1.62
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0863
0.02
0.44
-1.48
0.0
0.0
0.0
0.0
0.0
0.0
0.0898
0.16
-13
0.0
0.0
0.0916
-0.06
-0.04
0,22
0.12
-1.08
0.0
0.0
0.0
0.0
0.1119
0.08
-0.1
-0.04
0.24
0.06
0.86
0.0
0.0
0.0
0.1054
0.04
-0.04
-0.44
0.0
0.06
0.08
00.5
0.0
0.0
.0.0755
0.04
0.0
0.0
00.02
0.14
0.02
0.0
0.74
0.0
00.21747
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.07
0.1123
-1.17
0.0
0.0
0.0
0.0
0.0
0.0
0.1037
0.35
-0.99
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0724
0.25
0.11
0.99
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0681
0.03
0.05
-0.07
0.85
0.0
0.0
0.0
0.0
0.0
0.0
10
0.0816
-0.03
0.03
0.25
-0.09
0.93
0.0
0.0
0.0
0.0
0.0
0.0938
-0.01
-0.03
0.21
-0.11
-0.67
0.0977
0.01
-0.03
-0.03
0.07
0.09
-0.03
-0.81
0.0
0.0
0.0
0.0849
0.03
-0.03
-0.03
-0.03
0.05
0.01
-0.11
-0.27
0.0
0.0
00.0611
-0.01
-0.01
-0.01
-0.01
0.03
0.07
-0.01
-0.05
00.57
0.0
20


Reviews:
[{"review_id": "n8BdB65spP", "reviewer": "Reviewer_mznZ", "summary": "This paper proposes a differentiable solver search algorithm to find an optimal ODE solver for reverse-diffusion solving of pre-trained diffusion models. The authors use gradient-based optimization to identify solver parameters that lead to improved sample quality with very few function evaluations. The approach is evaluated on both rectified flow models and DDPM/VP frameworks, showing improvements in FID scores on ImageNet benchmarks under 10 sampling steps.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback on our manuscript. \n\n**Q.1 Total burden of searching**\n\nSearching one solver step with 50,000 samples using FlowDCN-B/2 requires approximately 30 minutes on 8 × H20 computation cards.\n\n**Q.2 More Quality comparison**\n\nWe plan to expand the quality comparison by including more models, such as SD3, Pixart-$\\alpha$-R512, and Pixart-$\\alpha$-1024. This will provide a more comprehensive evaluation of the performance and quality across a wider range of relevant models, enhancing the depth and validity of our analysis. \n\nThe anonymous visualization link: https://anonymous.4open.science/r/NeuralSolver-ICML25/README.md", "claims_and_evidence": "The authors claim that their differentiable search method significantly reduces discretization error compared to traditional solvers. This claim is supported by extensive experiments, including comparisons to state-of-the-art methods such as DPM-Solver++ and UniPC, as well as ablation studies examining the impact of search sample size and solver parameterization. The theoretical analysis, detailed in the appendix, provides error bounds that reinforce the empirical findings.​", "methods_and_evaluation_criteria": "The methodology addresses limitations of t-related Lagrange interpolation in existing fast sampling solvers by reparameterizing solver coefficients and timesteps into a differentiable framework. The evaluation is comprehensive, utilizing FID and other metrics across multiple model architectures and resolutions. The choice of benchmarks, including ImageNet-256 and ImageNet-512, and the inclusion of both rectified flow and DDPM-based models, provide a strong basis for assessing the method’s generality and effectiveness.​", "theoretical_claims": "The paper provides theoretical support for its solver-search method by deriving explicit bounds on discretization error. Key results include Theorem 4.4, showing that solver error depends explicitly on solver coefficients and timesteps, and Theorem 4.2, establishing the optimality of expectation-based solver coefficients over traditional Adams-like interpolation. And also, Theorem 4.5 which argues analytically that the proposed solver achieves tighter error bounds than conventional multi-step methods. These results justify the approach theoretically, I also checked the proof of these claims but not very carefully.", "experimental_designs_or_analyses": "The experimental evaluation seems to bde quite comprehensive, with detailed comparisons to recent solver-based methods. The ablation studies are informative, demonstrating how the performance of the searched solver varies with different numbers of search samples and parameter settings.", "relation_to_broader_scientific_literature": "The authors situate their work within the context of recent advances in fast diffusion sampling and solver-based methods. The paper builds on insights from prior works on DDPM/VP solvers and rectified flow models, providing relevant comparisons to state-of-the-art techniques like DPM-Solver++ and UniPC. This discussion clarifies how the proposed approach advances the current understanding of efficient diffusion sampling.", "essential_references_not_discussed": "Authors reference most relevant works.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and well-structured\n2. The experiments are quite comprehensive to evaluate the method\n\nWeakness:\n1. The improvements, while consistent, are incremental compared to existing solvers. \n2. The paper would benefit from a more detailed discussion on the computational overhead of the search process.​", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ld5EzAmTeE", "reviewer": "Reviewer_Gbsr", "summary": "The paper aims to accelerate reverse diffusion by integrating a novel differentiable solver search algorithm for better diffusion solvers.\nThe paper demonstrates that a data-driven approach in the post-training scenario can also enable fast sampling.\nUsing a compact search space related to the timesteps and solver coefficients, the proposed method can find the optimal solver parameter for each diffusion model. \nThe experiment shows the effectiveness of the proposed method on multiple models compared to the current solver-based fast sampling method.\n\n## update after rebuttal\nThe extended visualization and evaluation shows the improvement of the Solver Search. Although the proposed method can not be generalized to multi-resolution scenario. It still offers a good solution for optimal timestep determination. Thus, my recommendation for this paper is weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback on our manuscript. Your insights are extremely helpful and have provided us with clear directions for improvement. \n\n**Q.1 Quality comparison**\n\nWe plan to expand the quality comparison by including more models, such as SD3, Pixart-$\\alpha$-R512, and Pixart-$\\alpha$-1024. This will provide a more comprehensive evaluation of the performance and quality across a wider range of relevant models, enhancing the depth and validity of our analysis. \n\nThe anonymous visualization link: https://anonymous.4open.science/r/NeuralSolver-ICML25/README.md\n\n**Q.2 More Comparison with Flow-Turbo**\n\nWe presented the performance comparison in Tab. 4. Additionally, we have summarized the sampling and searching complexity in the table below, in relation to FlowTurbo. It should be noted that the value of $n$ will not exceed 15 steps. .\n\n|              | Steps | NFE  | NFE-CFG | Cache Pred | Order | search samples   | Params|\n|--------------|-------|------|---------|------------|-------|------------------|----|\n| Adam2        | n     | n    | 2n      | 2          | 2     | /                | n |\n| Adam4        | n     | n    | 2n      | 4          | 4     | /                | n |\n| heun         | n     | 2n   | 4n      | 2          | 2     | /                | n |\n| FlowTurbo    | n     | $>$n | $>$2n   | 2          | 2     | 540000(Real)     | $2.9\\times10^7$|\n| our          | n     | n    | 2n      | n          | n     | 50000(Generated) | n + 0.5(n$\\times$(n-1))|\n\n**Q.3 Pixart on GenEval**\nWe provided the solver(searched on DiT-XL/2-R256) with PixArt-$\\alpha$ on GenEval benchmark\n\n**Resolution 512 for PixArt-$\\alpha$ on GenEval benchmark**\n|          | steps | cfg | colors | counting | color_attr | two_object | single_object | position | all     |\n|----------|-------|-----|--------|----------|------------|------------|---------------|----------|---------|\n| dpm++    | 5     | 1.5 | 72.07  | 27.19    | 5.75       | 26.26      | 91.25         | 3        | 0.37587 |\n|          | 8     | 1.5 | 77.66  | 32.19    | 6.75       | 36.36      | 94.06         | 4.5      | 0.41921 |\n| unipc    | 5     | 1.5 | 73.14  | 25.94    | 6.25       | 26.26      | 90.94         | 3        | 0.37588 |\n|          | 8     | 1.5 | 78.72  | 32.50    | 6.5        | 40.15      | 93.75         | 5.5      | 0.42875 |\n| ours     | 5     | 1.5 | 72.87  | 31.56    | 6          | 33.08      | 91.88         | 5        | 0.40065 |\n|          | 8     | 1.5 | 76.86  | 33.44    | 7          | 40.40      | 94.06         | 5.5      | 0.42878 |\n\n**Resolution 512 for PixArt-$\\alpha$ on GenEval benchmark**\n|          | steps | cfg | colors | counting | color_attr | two_object | single_object | position | all     |\n|----------|-------|-----|--------|----------|------------|------------|---------------|----------|---------|\n| dpm++    | 5     | 2.0 | 76.60  | 30.94    | 6.50       | 33.08      | 91.25         | 4.75     | 0.40519 |\n|          | 8     | 2.0 | 76.86  | 37.19    | 5.25       | 39.65      | 93.75         | 5.75     | 0.43074 |\n| unipc    | 5     | 2.0 | 77.66  | 31.87    | 6.50       | 34.85      | 92.19         | 5.25     | 0.41387 |\n|          | 8     | 2.0 | 79.52  | 36.56    | 6.72       | 40.66      | 95.31         | 6        | 0.44134 |\n| ours     | 5     | 2.0 | 77.62  | 33.75    | 5.25       | 37.37      | 92.81         | 4.75     | 0.41933 |\n|          | 8     | 2.0 | 79.52  | 38.44    | 7.25       | 42.68      | 95.00         | 7.50     | 0.45064 |\n\n**Q.4 Total burden of searching**\n\nSearching one solver step with 50,000 samples using FlowDCN-B/2 requires approximately 30 minutes on 8 × H20 computation cards.", "claims_and_evidence": "The paper claims that the error caused by the non-ideal velocity estimation model can be estimated by a function related to the timesteps and coefficients. The claim is verified in the appendix.", "methods_and_evaluation_criteria": "The proposed method is evaluated on text-to-image generation using multiple metrics.\nHowever, these metrics are limited. For example, CLIP-score, GenEval, aesthetic score, etc, are not included.", "theoretical_claims": "I checked correctness of Theorem 4.4.", "experimental_designs_or_analyses": "- The experiments only provide a quantitative comparison for DDPM/VP for text-to-image models but not rectified flow models. A similar evaluation should also be conducted.\n- Solver-based methods are also included in the comparison with distillation in table 1.\n- The comparison between the proposed method and FlowTurbo is limited, more results should be exhibited such as on different models and metrics, other than FID and IS.", "relation_to_broader_scientific_literature": "The proposed method might help reveal the error of each timestep and identify the importance of each diffusion timestep.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "- The quality comparison is limited. Only Figure 2 provides a few examples. \n- More quality results would be helpful to demonstrate the effectiveness of the proposed method across different prompts and diffusion models.\n- More comparison should be focused on FlowTurbo since they both are parameterized velocity refiners.", "other_comments_or_suggestions": "- using $\\times$ instead of x in table 1.\n- There are replicated parts in the supplementary materials, G and L.", "questions_for_authors": "- What is the CLIP-score, the aesthetic score, and GenEval score for PixArt-$\\alpha$? It would be helpful if the method could be evaluated on these metrics on large diffusion models, such as SD3.\n- What is the overhead to derive the optimal coefficients?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gV6FavBXIp", "reviewer": "Reviewer_U5Bt", "summary": "This paper proposes a novel solver search algorithm for fast sampling of diffusion models, which optimizes both timesteps and solver coefficients. The key idea is to treat the solver design as a learning problem, optimizing solver parameters to minimize the numerical error and improve image quality. Experiments on rectified-flow models (SiT-XL/2, FlowDCN-XL/2) and DDPM (DiT-XL/2) demonstrate that the searched solvers can achieve improved FID with 5-10 steps. The learned time steps and coefficients can generalize to different model architectures and resolutions empirically.\n\n## Update after rebuttal\n- Most initial concerns have been addressed or clarified during the rebuttal. I'm supportive of acceptance as it's effective and well-supported by empirical evidence. \n- That said, I would not advocate for a higher rating due to the limited broader impact and significance of the contribution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our heartfelt gratitude for the valuable feedback you've provided on our manuscript. Your in-depth analysis and suggestions are of great significance to us, and we are committed to using them to enhance the quality of our work. \n\n**Q.1 Writing typos and inconsistent presentations**\n\nThank you for pointing out the detailed writing typos and inconsistent presentations. We sincerely apologize for these inadvertent errors. We will meticulously review and revise every detail to enhance the readability of the text. \n\n**Q.2 Comparison with DM-Nonuniform[1]**\n\nDM-Nonuniform[1] primarily centers on the theoretical optimal timesteps, yet it fails to take into account the solver coefficients and model statistics. In contrast, our method conducts a statistical search for both the coefficients and timesteps concurrently. Through theoretical analysis, we have demonstrated that our method has a smaller error bound compared to those that neglect coefficients. This shows the superiority of our approach in more comprehensively handling the relevant factors in this context.\n\nWe compared the performance with DM-Nonuniform[1] in tab.2 and tab.3. We copy the result here.\n| Methods \\ NFEs                            | 5       | 6       | 7       | 8       | 9       | 10      |\n|--------------------------------------------------------|---------|---------|---------|---------|---------|---------|\n| DPM-Solver++ with uniform-$\\lambda$-opt[1]    | 12.53   | 5.44    | 3.58    | 7.54    | 5.97    | 4.12    |\n| DPM-Solver++ with uniform-$t$-opt[1]         | 12.53   | 5.44    | 3.89    | 3.81    | 3.13    | 2.79    |\n| DPM-Solver++ with EDM-opt[1]                            | 12.53   | 5.44    | 3.95    | 3.79    | 3.30    | 3.14    |\n| UniPC with uniform-$\\lambda$-opt[1]            | 8.66    | 4.46    | 3.57    | 3.72    | 3.40    | 3.01    |\n| UniPC with uniform-$t$-opt [1]                | 8.66    | 4.46    | 3.74    | 3.29    | 3.01    | 2.74    |\n| UniPC with EDM-opt  [1]                                 | 8.66 | 4.46 | 3.78    | 3.34    | 3.14    | 3.22    |\n| Searched-Solver                               | 7.40 | 3.94 | 2.79 | 2.51 | 2.37 | 2.33 |\n\nFor DiT-XL/2-R512\n\n| Methods \\ NFEs                           | 5        | 6       | 7       | 8       | 9       | 10      |\n|-------------------------------------------------------|----------|---------|---------|---------|---------|---------|\n| UniPC with uniform-$\\lambda$-opt[1]          | 11.40    | 5.95 | 4.82    | 4.68    | 6.93    | 6.01    |\n| UniPC with uniform-$t$-opt[1]                 | 11.40    | 5.95 | 4.64    | 4.36    | 4.05    | 3.81    |\n| Searched-solver(searched on DiT-XL/2-R256) | 10.28 | 6.02    | 4.31 | 3.74 | 3.54 | 3.64 |\n\n\n\n\n**Q.3 Reduce the search burden**\n\nFirst, a significant amount of computational resources is wasted on constructing the target trajectory. Since this target trajectory can be reused for each step in the search for solvers, we can cache it to prevent redundant recomputation. \n\nFurthermore, we have observed that the solver optimized on base-sized or even small-sized models exhibits a high degree of generalization when applied to XL-sized models. Thus, using a small model as a proxy is a viable and practical choice. This approach not only reduces computational overhead but also provides a more efficient way to achieve good performance across different model scales. \n\n**Q.4 Alignment between two trajectories**\n\nSince the learned trajectory of length $N$ has its corresponding timesteps, we select a subset of length $N$ from the reference trajectory based on the timesteps of the learned trajectory. \n\n**Q.5 Total burden of searching**\n\nSearching one solver step with 50,000 samples using FlowDCN-B/2 requires approximately 30 minutes on 8 × H20 computation cards. \n\n[1]: Xue, Shuchen, et al. \"Accelerating diffusion sampling with optimized time steps.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024", "claims_and_evidence": "Yes, most claims are well-supported.", "methods_and_evaluation_criteria": "Yes, the approach and evaluation criteria make sense.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes, the experimental designs make sense.", "relation_to_broader_scientific_literature": "This paper is most related to [1] where only time steps are learned. This paper extends the search space to both time steps and coefficients and employs a different optimization objective and strategy. \n\n[1]: Xue, Shuchen, et al. \"Accelerating diffusion sampling with optimized time steps.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "essential_references_not_discussed": "A few prior works ([1,2]) have also explored the idea of accelerating the solver via learning. They should also be cited and discussed in the paper. \n\n[1]: Watson, Daniel, et al. \"Learning fast samplers for diffusion models by differentiating through sample quality.\" *International Conference on Learning Representations*. 2021.\n\n[2]: Dockhorn, Tim, Arash Vahdat, and Karsten Kreis. \"Genie: Higher-order denoising diffusion solvers.\" *Advances in Neural Information Processing Systems* 35 (2022): 30150-30166.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The proposed method consistently achieves improved FID score within few-step regime compared to previous methods like DPM-solver++ and UniPC.\n- The authors provide theoretical justification for the approach, including error bound analysis and theorems supporting.\n- The learned time steps and coefficients can generalize to different model architectures and resolutions empirically.\n\n**Weaknesses**\n\n- Clarity: The paper currently contains numerous typographical errors, inconsistencies, and formatting issues, which affect readability and clarity. See details in “Other Comments or Suggestions”.\n- The proposed approach requires generating tens of thousands of reference trajectories to learn the time steps and coefficients, which could be computationally expensive both in terms of time and space.\n- The learned solver might become suboptimal for different guidance scales.", "other_comments_or_suggestions": "Below are examples of specific errors noted during my review. There are more in other parts of the paper. I strongly recommend that the authors thoroughly proofread and revise the manuscript to address these issues comprehensively. \n\n- Line 191: “with prerained model” should be “with pretrained model”\n- Line 229: “of Our solver” → “of our solver”\n- Line 230: the equation is out of page, $b_i^j$ is used without definition.\n- Line 260: remove comma from “{1-\\sum_{j=0}^{i-1}c_i^j,}_{i=0}^{N-1}” and “{c_i^k, }”.\n- Line 314: “reconstruction error(in Appendix)” → “reconstruction error[need a space](in Appendix~[need to add reference])”\n- Line 315: “Euler-250 steps” → “250-step-Euler”?\n- Line 409: “Of Solver Parameters” → “of Solver Parameters”\n- Line 381: “Comparison with Distillation methods” → “Comparison with distillation methods”\n- The use of capitalization and periods in section headings and table captions is inconsistent, confusing, and distractive:\n    - Some sections only capitalize the first word like Section 4 “Optimal search space for a solver” while the other sections capitalize all the words like Section 2 “Related Works”\n    - Section 4.2: “Focus on Solver coefficients instead of the interpolation function” capitalize the first character of “focus” and “solver”, which is even more confusing.\n    - Section 5: why is there a additional period?\n    - Table 1: “Comparsion with Distillation methods”  why is the first character of “Distillation” capitalized? Also, “Comparsion” → “Comparison”", "questions_for_authors": "- How does the proposed method compare to the prior work [1] which optimizes the time steps?\n- What is the total computation cost to train the time steps and coefficients? How does it compare to [1]? Could you elaborate on the computational cost and efficiency of the solver search process in more detail, particularly in relation to the performance gains achieved?\n- Are there any techniques or optimization strategies that could be explored to reduce the computational burden of the search process?\n- Since the discretization schemes of the reference trajectory (L steps) and learned trajectory (N steps) are different, how do you compute the MSE loss between these two trajectories?\n\n[1]: Xue, Shuchen, et al. \"Accelerating diffusion sampling with optimized time steps.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance (1 accept, 2 weak accepts). After reading the paper, reviews, and discussion, I agree with the reviewers and recommend acceptance.
Decision: Accept (poster)