A Unified Sampling Framework for Solver Searching of Diffusion Probabilistic Models
Authors: Enshu Liu
Xuefei Ning
Huazhong Yang
Yu Wang
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Probabilistic Model
Diffusion Sampler
Solver Schedule
Primary area: generative models
TL;DR: We provide a unified sampling framework for diffusion model and search for solver schedules based on it.

Abstract: Recent years have witnessed the rapid progress and broad application of diffusion probabilistic models (DPMs). Sampling from DPMs can be viewed as solving an ordinary differential equation (ODE). Despite the promising performance, the generation of DPMs usually consumes much time due to the large number of function evaluations (NFE). Though recent works have accelerated the sampling to around 20 steps with high-order solvers, the sample quality with less than 10 NFE can still be improved. In this paper, we propose a unified sampling framework (USF) to study the optional strategies for solver. Under this framework, we further reveal that taking different solving strategies at different timesteps may help further decrease the truncation error, and a carefully designed \emph{solver schedule} has the potential to improve the sample quality by a large margin. Therefore, we propose a new sampling framework based on the exponential integral formulation that allows free choices of solver strategy at each step and design specific decisions for the framework. Moreover, we propose $S^3$, a predictor-based search method that automatically optimizes the solver schedule to get a better time-quality trade-off of sampling. We demonstrate that $S^3$ can find outstanding solver schedules which outperform the state-of-the-art sampling methods on CIFAR-10, CelebA, ImageNet-64, and LSUN-Bedroom datasets. Specifically, we achieve 2.69 FID with 9 NFE and 6.86 FID with 5 NFE on CIFAR-10 dataset, outperforming the SOTA method significantly. We further apply $S^3$ to Stable-Diffusion model and get an acceleration ratio of 2$\times$, showing the feasibility of sampling in very few steps without retraining of the neural network.

Published as a conference paper at ICLR 2024
A UNIFIED SAMPLING FRAMEWORK FOR SOLVER SEARCHING OF DIFFUSION PROBABILISTIC MODELS
Enshu Liu
Xuefei Ning', Huazhong Yang, Yu Wang" Department of EE, Tsinghua University foxdoraame@gman .com berighz@tsinghhne edu. cn
Department of EE, Tsinghua University & Infinigence-Al
les230ma i 1s. ts weiinngaa.ddeee
yu yu-wangemmill zsinghua..... edu. on
ABSTRACT
Recent years have witnessed the rapid progress and broad application of diffusion probabilistic models (DPMs). Sampling from DPMs can viewed as solving ordinary differential equation (ODE). Despite the promising performance, the generation of DPMs usually consumes much time due to the large number of function evaluations (NFE). Though recent works have accelerated the sampling to around 20 steps with high-order solvers, the sample quality with less than 10 NFE can still be improved. In this paper, we propose unified sampling frame- work (USF) to study the optional strategies for solver. Under this framework, we further reveal that taking different solving strategies different timesteps may help further decrease the truncation error, and a carefully designed solver sched- ale has the potential improve the sample quality by large margin. There- fore. we propose new sampling framework based on the exponential integral formulation that allows free choices of solver strategy each step and design spe- cific decisions for the framework. Moreover, we propose 83 a predictor- predictor-based search method that automatically optimizes the solver schedule to get better time-quality trade-off of sampling. We demonstrate that S can find outstand- ing solver schedules which outperform the state-of-the-art sampling methods on CIFAR-I0. CelebA, ImageNet, and LSUN-Bedroom datasets. Specifically, we achieve 2.69 FID with 10 NFE and 6.86 FID with NFE on CIFAR-10 dataset, outperforming the SOTA method significantly. We further apply $ to Stable- Diffusion model and get an acceleration ratio of 2x, showing the feasibility sampling in very few steps without retraining the neural network. Our code is available at hnttps://githhe http://twwb.co//ttttttttt///////
INTRODUCTION
Diffusion probabilistic models (DPMs) (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al. 2020b) have emerged new generative modeling paradigm in recent years. DPMs model the probability distribution through training a neural network to estimate the score function or other equivalent form along pre-defined forward diffusion stochastic differential equations (SDEs) (Song lt al., 20205). Sampling from DPMs can be viewed as solving the corresponding reverse diffu- sion SDEs (Song al.) 20206] Ho et al. 2020) the diffusion ODEs (Song al. 2020b by discretizing the continuous timesteps.
Despite the high generation ability, the main drawback of DPMs is the slow sampling speed due to the large number of discretized timesteps for the numerical accuracy of the DE solver (Song et al.) 2020b Ho al.] 2020) with neural inference on each step. Therefore, accelerating the sampling process of DPMs is very meaningful.
Current work focusing on decreasing the DPM sampling steps can be divided into two categories. The first of them needs retraining of the neural network (Luhman & Luhman 2021 Salimans & Ho 2022 Meng et al., [023] [Song et al.) (202)), which takes significant extra costs, especially for large models like |Rombach et al. (2022). The other branch of works attempts design efficient solvers of differential equations (DEs) to accelerate DPMs without retraining the existing off-the-shelf mod-
els (Song et al. 2020a Watson et al. 202T Bao et al. 2022
Liu et al.) 2022
Lu et al.) 2022a
"Corresponding Authors

---

Published as a conference paper at ICLR 2024
Zhang & Chen 2022 Zhang et al. 2022 Zhao al. 2023 State-of-the-au methods utilize the equivalent exponential integral form of the reverse diffusion ODE, accurately calculating the linear term while estimating the integral term through various high-order numerical methods (Zhang & Chen 2022 Lu et al., 2022a b. Zhao et al. 2023). These methods can reduce the number of steps to 155~20 steps, but the performance decreases rapidly with lower than 10 steps.
We notice that many solving strategies of these solvers are empirically set and kept constant along the sampling process except for a few special timesteps, leading suboptimal performance with inadequate timesteps. In this paper, to systematically study the impact of these strategies, we propose unified sampling framework based on the exponential integral form of the diffusion ODE, named USF, which splits the solving process of one step into independent decisions of several components, including the choice of 1. timestep, neural network, 4. order of Taylor expansion, 5. derivative estimation method, and 6. usage of ODE correctors. Based on this framework, we reveal that the quality and efficiency of training-free free samplers can be further improved by designing appropriate solver schedules, motivated by the key observation that the suitable solving strategies vary among different timesteps. Therefore, solver schedules, which means the different solving strategies assigned to each timestep, is very important for the sample quality. Our proposed framework can incorporate the existing diffusion solvers by assigning corresponding decisions those components and allows the ensemble of different solving strategies in the timestep dimension, enabling sufficient potential to outperform existing sampling methods. In addition, we also design new strategies different from existing diffusion solvers for the derivative estimation component, making the proposed sampling framework more promising.
starting point of the current step, 3. prediction type of the
However, designing solver schedules is difficult due the vast decision space. To address this problem, we propose S to search for optimal solver schedules automatically. We construct a per- formance predictor to enable the fast evaluation of one solver schedule and use it to guide a multi- stage search process to find well- well-performing solver schedules under certain budget of number of function evaluation (NFE). Our contributions are summarized as follows:
We propose a new sampling framework for DPMs, called USF, which unifies existing diffu- sion ODE solvers based on exponential integral, to systematically and conveniently study the strategies chosen for diffusion samplers. Based on this framework, we design some new solver strategies, including using different solver strategies across timesteps, low-order estimation for derivatives, more types of scaling methods and searchable timesteps.
We propose predictor-based multi-stage search algorithm, S to search for the well- performing solver schedule under a certain NFE budget. Our method can directly utilize off- the-shelf DPMs without any retraining of the diffusion neural network and find outstanding solver schedules with a moderate search cost, demonstrating its practical applicability.
We experimentally validate our method on plenty of unconditional datasets, including CIFAR- 10 (Krizhevsky et al. 2009), CelebA (Liu et al. (015)), ImageNet-64 (Deng et al. 2009) and LSUN-Bedroom (Yu et al. 2015). Our searched solver schedules outperform the SOTA diffu- sion sampler (Lu et al. [0026)] Zhao al. 2023) by large margin with very few NFE, e.g., 6.86
v.s. 23.44 on CIFAR- 10 with only 5 NFE. We further apply 5 to Stable-Diffusion (Rombach) et al., 2022) models, achieve 2x (from 10 NFE to 5 NFE) acceleration without sacrificing the performance on text-to-image generation task on MS-COCO 256 x 256 dataset (Lin et al.) 2014). Based on the experimental results, we summarize some knowledge to guide future schedule de- sign.
RELATED WORK
2.1 DIFFUSION PRORABILISTIC MODEL.
Diffusion Probabilistic Models (DPMs) -icks-Dic et al. 2015 Ho et al. 2020; Song et al.
2020b) are used to construct the probability distribution g(wo) of a D-dimension random variable
Ep E RD DPMs define forward diffusion process to add noise the random variable Zo (Song
et al. 2020b) progressively:
dr, = f(z, t)dt + g())dwe.
(1)
where x, stands for the marginal distribution at time for E [o, T]. and un obeys the target distri- bution (2x)). we is the standard Wiener process. When (xx, is affine form of 24, the marginal
2

---

Published as a conference paper at ICLR 2024
distribution of ra given ze can be written (Song al. 2020b
(<1)/20)) A (n of
where a oo is called signal-to to-iose ratio (SNR), which strictly decreasing function and approximately equals when t = T. The forward diffusion process Eq. ① has a reverse probability flow ODE (Song et al. 2020b a):
dx; (f(t)z; 3 - (t)V
(3)
where æy ~ (2x 001). The ODE has two equivalent forms w.r.t noise prediction network and data prediction network ze, which are more commonly used for fast sampling:
g²(t)
dæ (f(1)=+ +
(x ())dt,
(4)
20.
dr, ((f(t)+ (() 20 2
ang)(1)
xo(
20
can be proved that the marginal distribution De in Eq. [T] equals the distribution of 2, in
Eq. 3 (Anderson 1982). Thus, sampling from DPMs can be viewed as solving the above ODEs.
2.2 TRAINING-FREE SAMPLERS
One major drawback of DPMs is the large number of discretized timesteps needed for numerical solvers ensure the sample quality. To handle such issue, training-free samplers for the ODE Eq. are proposed to achieve quicker convergence (Song et al. 2020b a Liu et al.) 2022 Zhang
& Chen 2022:
Lu et al. 2022a
Zhao al. 2023).
2.2.
Low ORDER SAMPLERS
The early samplers can be viewed low-order solvers of the diffusion ODE (Song et al. 2020a Liu
ct al. 2022). DDIM (Song et al. 2020a) proposes the following formula for the update one step:
Ft. Vaa -V-- ( + /I (2,)(2.,5) s). PNDM (Liu et al. 2022 proposes to replace the Va. neural term (2, 8) in the DDIM formula with new term 54 (55c, 59c, 37E,-28 9c,-36) inspired by Adams method. The DDIM formula is 1-st order discretization the diffusion ODE Eq. (3) and thus is not efficient enough due to the inadequate utilization of high-order information.
2.2.2 EXPONENTIAL-IITTERR.CCC BASED SAMPLERS
The solution Ia of the diffusion ODE Eq. (3) at timestep t from timestep s can be analytically computed using the following exponentially weighted integral (Lu et al. 2022a)
r. a 1/a e co)
The linear term G1 xr can be accurately computed and the high order numerical methods can be o,
applied to the integral term ag A^ e A)dA, improving the sample speed dramatically than low-order formulas. The following methods apply different strategies to estimate the integral term.
DEIS iZhang & Chen 2022) uses Lagrange outer interpolation to estimate the integrand. Then, the exponential integral can computed approximately since the integral a polynomial can easily computed. Additionally, DEIS computes this integral in domain rather than A.
DPM-Solver (Lu et al. 2022a) utilizes the Taylor expansion of the exponential integral as follows to estimate this term
(n) (M); As) 1 -x((-->)" Aa + O((Ar - Ar) n!
Additionally, Lu al. (202aa) proposes discretize the timesteps with uniform logSNR. which has empirically better performance. DPM-Solver++ (Lu et al.] (0026) introduces different strategies under the same Taylor expansion framework, e.g., multi-step solver and data prediction neural term.
3

---

Published as a conference paper at ICLR 2024
Timestep Schedule
4,2
Starting Point
Corrector
eatto clbs))
aettor Roc)
Di Derivative Estimation
+ Ceante Teal
nerg (o
Ep
Exppansion Order
by Prediction Type
Figure 1: The sampling process of USF and all searchable strategies.
UniPC Zhaoo et al. 2023) introduces correcting strategy to the Taylor expansion framework. After computing 2, through Eq. (7. UniPC uses the newly computed function evaluation eg (Te, As) to correct the value of 2,, which does not consume extra NFE as cØ(2s, Aq) is calculated only once.
2.3 AUTOML
AutoML methods aim to automatically decide the optimal configurations for machine learning systems. The decidable elements in AutoML works include training hyperparameters (Jaderberg et al. 2017), model selection (Yang et al.] 2019; Liu et al.)
2023), neural architecture (Zoph & Le 2016), etc. Predictor-based search methods (Snoek al. 2012) widely used in the AutoML
search, predictor is trained with evaluated field to accelerate the optimization. In predictor-
iinigurrationobjecttv pairs, and then used to predict the objectives for new configurations.
A UNIFIED SAMPLING FRAMEWORK
In this section, we propose unified sampling framework called USF based on exponential integral in the À domain. We introduce our framework by describing the steps of solving the diffusion ODE Eq. (3) and listing all optional strategies and tunable hyperparameters of the sampling process.
3.1 SOLVING STRATEGY
To solve the diffusion ODE Eq. without closed-form solutions, should discretize the tinuous time into timesteps and calculate the trajectory values at each timestep. Therefore, the first step of solving the diffusion ODE is to determine a discretization scheme (4,,4,, ....... .Nv, where 8 == 40 < 4 < ... < IN = T. Then consider a timestep / E [4...... _IN, exponential integral has been validated to be an effective method of computing 2, due the semi-linear property (Zhang & Chen 2022 Lu et al. 2022a bj Zhao et al. 2023). especially in the À domain. We follow the Tay- lor expansion of Eq. in À domain (Lu et al. 2022a) estimate the integral term, and summarize the strategies that can be applied to numerically compute the expansion as follows.
Prediction type of neural network. Typically, the neural network is trained with denoising objec- tive (Ho et al.) (2020) and the corresponding diffusion ODE writes as Eq. (4). It can be proved that Eq. (6) is the accurate solution of Eq. (4) (see App. H). Denote that a another timestep before
and h = Xx- >x., the Taylor expansion of Eq. (6) can be written as (Lu et al. 2022a):
Dt
Ct
T.
Ti
A+1 p (h)e (8) (x,, (x.,))
0(h)t2)
(8)
ag
k=
where sp satisfies vh+1 (h) = 2i (k)- and 4 (h) = ch. Alternatively, the neural network can
"
be a data prediction model xo (2,, t), whose relationship with the noise prediction model co(2,, t) is given by ze (2),,) diffusion ODE w.r.t. data prediction model writes Eq. whose Tay larlor expansion is given below:
pappeol (Kingma et al. 2021 1 Lu et al. 2022b). The corresponding
T1
0t x, + 0i Σ
x+1 Pk
(h))) +0
k=
4

---

Published as a conference paper at ICLR 2024
where si satisfies (k) (k)) and (b) = = See App. H for detailed derivations.
Since the linear term and the integral to estimate are different between the two prediction types, the
numerical solutions of Eq. (8) and Eq. ! are essentially different except for 1.
Starting point s. s can be any timestep larger than t. Multistep solvers (Lu et al. 2022b Zhao
ct al., (023) always choose the previous timestep of & while single-step solvers (Lu et al. 2022a)
choose more distant timesteps.
Order of the Taylor expansion. To compute the Taylor expansion Eq. 18 or Eq. 9 the number of order retained should be decided. Low retained order may result in higher truncation error. However, the estimation of high-order derivatives need extra information from other timesteps other than the starting point which may not be accurate enough for calculation. Therefore, the choice of Taylor expansion order may have a strong influence on the sample quality.
Derivative estimation method. After determining the prediction type and the retained order n. the last thing is numerically calculate all unknown derivative terms 4 (Iz..) s) (take noise prediction model as an example), where k S n, to obtain the final solution .;,- The strategies of this component are very flexible since many off-the-shelf numerical differential methods can be used. To simplify the workflow, we utilize the system of linear equations consisting of 2 A: Taylor expansions from the starting point 8 to other timesteps, the k-order derivative can be approximately calculated by eliminating derivatives of other orders (Zhao et al. 2023), which we call m-th Taylor-difeernce method (see Def. [.I for its formal definition). Inspired by the 2-nd solver ou et al. (2022a b), we -=()) note that for a derivative estimation result Ey (x,, s), rescaling the estimated value by multiplying a coefficient 1 + R(hh) has the potential to further correct this numerical estimation. To sum up, we formulate the estimation of k-th order derivative as the following two steps: (1). choose at least k other timesteps & * S, i E [1, [1...... .k] with pre-computed function evaluation 6g (T tx) to calculate the estimation value through Taylor-different
method, and (2). rescale the estimation value with a coefficient 1 + R()). In the search space used in our experiments, we allow low-order Taylor- difference estimation to be taken for the 1-st derivative in high-order Taylor expansions (i.e., m can be smaller than n) and design five types of scale coefficients R4(h), i 0,..., 4 for the 1-st derivative. See App. B for details.
After calculating Ie, correctors can be used to improve the accuracy of xy through involving the newly computed ea (2,,t) the re-computation of z, (Griffiths & Higham 2010)) Specifically, the Taylor expansion Eq. (8) or Eq. (9) is estimated again with t as one other timestep for derivative es- timation. Typically, en (2,,t) also have to be evaluated again after recomputing 2, and the correcting process can be iteratively conducted. However, in practice, re-evaluations and iterative correctors are not used to save NFE budgets. Since our goal is to improve the sample quality with very few NFE, we follow Zhao et al. (2023) to use additional neural inference for correctors.
In summary, all decidable components in our framework are: (I). timestep discretization scheme, (2). prediction type of the neural term, (3). starting point of the current step, (4). retained order of Taylor expansion, (5). derivative estimation methods, and (6). whether to use corrector. Our sampling framework is summarized in Alg. and demonstrated in Fig.
3.2 SOLVER SCHEDULLE: PER-STEP SOLVING STRATEGY SCIIMM
In this section, we demonstrate the phenomenon that the proper solving strategies appropriate for different timesteps are different. We construct a "ground-truth' trajectory e [Ito "ty," "'" Ftx] using 1000-step DDIM (Song al. 2020a) sampler (i.e., N 1000). We choose serials of target timesteps [tzero ... htrss where tear, E [to,-. ... tw]. and estimate the value Sterr, with previous timesteps [t....... .t.,. ttx. where s, E N] and te, > tear,. Then we calculate the distance between the ground- truth trajectory value and the estimated value, given by L(trar;) | Rtturpl |- We compare different solving strategies by showing their ((Itar;)-ttar, curve in Fig. 2 The key observation is that the suitable strategies for most components varies among timesteps. For example, the loss of the 3-rd solver with noise prediction network is smaller than the 2-nd solver at 400-700 timesteps but larger at the rest timesteps (see the green and red curves of "Orders and Prediction Types').
This phenomenon reveals the potential of using different solving strategies in different timesteps. We call the decision sequence of solving strategies for every timestep solver schedule.
5

---

Published as a conference paper at ICLR 2024
Orders and Predi ction Types
Low Order Derivative Estim ation
Derivative Scale Methods
Date
Scas Tr
sofee, aredic sas vediciinn
TuBl enter mate prodiciion
4820 eosking
RAN: Son ind
loe eater mute deta prod thke
X320
Nomoiizod loze
4329
4470
Nomms
300 20
304 20 60
Timmntap
Correcting Methods
Starting points
Der Feclctin
Nebe Prediction
Data Prrtt etee
Latter mnnor your comeree
wervec connere
Namma lznd los
Norme LoNd COAN
Narma
200 al
2ss <ie sio 68c 150
550 250
300
353
160
100
300
300
600
TImestrps
Timestoge
Timesteps
Figure 2: Losses with ground trajectory using different strategies. (i). For the losses in 'Orders and Prediction Types', choose different expansion orders and prediction types for every update. (ii). For the losses in 'Low Order Derivative Estimation', we use the 3-rd Taylor expansion with both prediction types and choose whether apply the 1-st Taylor-difference method to estimate the 1-st order derivative rather than the 2-nd Taylor- Taylor-difference method. (iii). For the losses in 'Correcting Methods', we choose whether to use correctors after the update under different expansion orders and prediction types. (iv). For the losses in 'Starting points', we choose different indices of the starting points, including 'Single step' (all additional timesteps are between the starting point and the target point) and 'Multi step' (all additional timesteps are larger than the starting point). The ranks of all strategies for all components are not constant at different timesteps.
3.3 ADVANTAGES OF THE FRAMEWORK
The contribution of USF lies in two ways. First, USF gives unified perspective for all sampling methods based on exponential integral. Second, USF allows different solving strategies in each timestep, making the sampler more flexible. To demonstrate the universality and reliability of USF, we show in Tab. [ that almost all existing SOTA samplers based on exponential integral can be incorporated by through assigning corresponding strategies (more derivations
be found in App. H) Note that since DEIS (Zhang & Chen, (022) computes the integral in the domain, is essentially different from other methods. However, when applying the core numerical method in DEIS to the À domain, we show that it can still be incorporated in USF (the last row in Tab. I.
SEARCH FOR SOLVER SCHEDULE THROUGH $
4.1 PROBLEM DEFINITION
Denote the 8; as the numerical solver to compute Ft,, we formulate the problem of deciding the optimal solver schedule as below:
arg min 1 * F(((s1,t)); (82, t2), (SM, (M). S.MM SC,
AMM
where the given NFE budget, (()1,41, (#2, 12).... (SAr, Jas)] is the solver schedule s, and F is an arbitrary evaluation metric of diffusson models, and IF is an indicator coefficient which equals to when smaller F indicates better performance and equals to -1 otherwise.
As discussed in Sec. 3.2 the best solving strategies for different timesteps are different, which leads to a search space that grows exponentially with the number of steps. Additionally, the decision of one component may be coupled with the decision of other components. For example, demonstrated in Fig. 2 low-order estimation of derivatives almost consistently outperforms full-order estimation method with data prediction network (see the orange and blue curves of 'Low Order Derivative

---

Published as a conference paper at ICLR 2024
Table I: Relationship between USF and existing SOTA methods. The value of "starting point' represents the index difference between the target point and the starting point. All methods use Taylor-difference to estimate derivatives or all orders. The formulas in 'Scale' are the coefficients for scaling the 1-st derivative. h' equals h if prediction type is noise and equals if data.
Method
Prediction
Taylor order
Starting point
Scale
Corrector
4=(e" -h-h-1
DPM-Solver-25
Noise
1,2
-1,-2
None
DPM-Solver-35
Noise
1,2,2
-1,-2,-3
None
None
c1+e
DPM-Solver++(2M0
Data
-1
None
UniPC-2-B31A4
Noise/Data
2
-1
UniC-2
UniPC-2-B2(h)
Noise/Data
2
-1
UniC-2
-h-1
UniPC-p (p>> 2)
Noise/Data
-1 None
UniC-p
DEIS-p (in the À domain)
Noise
-1 None
None
Estimation') ) while worse than it with noise prediction network at 500-700 timesteps (see the red and green curves). Therefore, well-performing solver schedules are hard design manually.
Inspired by AutoML works (Real ct al. 2019 Ning et al., 2020 Liu et al. 2023), we attempt to auto- matically search for solver schedules with good performance under certain NFE budgets. However, the search space is extremely large, and evaluating a solver schedule is also time-consuming since the standard calculation caleulatiooo processes most metries need generate large number of images (Heusel ee al., 2017). Thus, random search with regular metric evaluation method is inefficient in this case. 4.2
PREDICTOR-BASE MULTI-STAGE SEARCH FOR SOLVER SCHEDULE
To surmount the aforementioned challenges and accelerate the search process, we propose $3, predictor-based multi-stage evolutionary method. Specifically, train a light predictor to predict the performance of a solver schedule and use to guide an evolutionary search process to sample solver schedules. We can evaluate solver schedules in the evolutionary search with negligible cost with the help of the predictor. making the search process much more efficient.
But training the predictor still needs a bunch of evaluated schedules, which is time consuming. To decrease this part cost, we propose the multi-stage workflow 53. In the early stage of the evo- lutionary search, only the sub-space adjacent to the current population the predictor is not required to generalize to the whole search space and can be trained with a small amount of data. As the population expands, generalization ability to new sub-spaces is needed, and thus the predictor should be trained with more sampled schedules. Moreover, the training data of the predictor should also be selected carefully and efficiently.
be explored. Therefore,
Based on the above intuition, we propose the multi-stage workflow. Our workflow is summarized in Alg. and demonstrated in App. D.2] which contains three steps in every loop. (1). A set of solver schedules are sampled from the search space through evolutionary search. The performances needed to guide the evolutionary search are calculated by the predictor except for the first loop in which all performances are truly evaluated. (2). The true metrics of all sampled solver schedules are calculated. (3). All evaluated perfuulee data pairs are used to update the weights of the predictor. Compared to the single-stage method, which evaluates the true performance of all schedules in one go and trains a predictor to search the whole space, S accelerates the exploration of schedules which have to be truly evaluated.
5 EXPERIMENTS
We choose several SOTA diffusion samplers, including DPM-Solver (Lu et al.) (022a). DPM- Solver++ (Lu et al. 2022b). and UniPC (Zhao et al.) 2023) as baseline methods. We use FID (Heusel et al. 2017) as the evaluation metric. Our method is validated on CIFAR-10, CelebA, ImageNet-64, ImageNet- 288 (guided), ImageNet-2556 (guided), and beduoom datasets and outperforms baseline methods by large margin all of them. We further apply S text-image generation task with Stable Diffusion pre-tree pre-trained models (Rombach et al. 2022) on MS CSSCC002001
validation set. The models use for all experiments are listed in App. E.

---

Published as a conference paper at ICLR 2024
Table 2: FIDs of the searched solver schedules on unconditional and label-guided generation tasks.
NFE
Dataset
Method
4
5
6
7
8
9
10
Baseline- ((()
255.21
288.12 128.12
32.15
14.79
22.99
6.41
5.97
Baseline- W(M
61.13
33.85
20.84
13.89
10.34
7.98
6.76
CIFAR-10
Baseline-B
57.52
23.44
10.33
6.47
5.16
4.30
3.90
Ours
11.50
6.86
5.18
3.81
3.41
3.02
2.69
Baseline-W W(S)
321.39
330.10
52.04
17.28
16.99
10.39
6.91
Baseline-W W(M)
31.27
20.37
14.18
11.16
9.28
8.00
7.11
CelebA
Baseline-B
26.32
8.38
6.72
6.72
5.17
4.21
4.02
Ours
12.31
5.17
3.65
3.80
3.62
3.16
2.73
Baseline- W(S)
364.60
366.66
72.47
47.84
54.21
28.22
27.99
Baseline-W ((M)
93.98
69.08
50.35
40.99
34.80
30.56
27.96
ImageNet-64
Baseline-B
76.69
61.73
42.81
31.76
26.99
23.89
24.23
Ours
33.84
24.95
22.31
19.55
19.19
19.09
16.68
Baseline- W(M)
44.29
24.33
15.96
12.41
10.87
9.99
8.89
LSUN-Bedroom
Baseline-B B
22.02
17.99
12,43
10.79
9.92
9.11
8.52
Ours
16.45
12.98
8.97
6.90
5.55
3.86
3.76
Baseline- W(M)
32.08
15.39
10.08
8.37
7.50
7.06
6.80
ImageNet-128
Baseline-B
25.77
13.16
8.89
7.13
6.28
6.06
6.03
Ours
18.61
8.93
6.68
5.71
5.28
4.81
4.69
Baseline- Baseline-W(M)
80.46
54,00
38.67
29.35
22.06
16.74
13.66
ImageNet-256
Baseline-B
51.09
27.71
17.62
13.19
10.91
9.85
9.31
Ours
33.84
19.06
13.00
10.31
9.72
9.06
9.06
We evaluate 9 common settings in total of these SOTA methods. We list the best (Baseline-B in all tables) and the worst (Baseline-W in all tables) results of all baseline methods we evaluate under the same NFE budget. For the worst baseline, we distinguish singlestep methods (S) and multistep methods (M) since the former is usually significantly worse with meager budgets. Detailed settings and full results can be found in App. and App. G.I correspondingly.
5.1 MAIN RESULTS
We list the results on unconditional datasets and with classifier-guidance in Tab. and the results with Stable Diffusion models in Tab. 3 The key takeaways (1) S achieve much higher sample quality than baselines, especially with very few steps. Our searched schedules outperform all baselines across all budgets and datasets by large margin. On text-to-image generation with Stable Diffusion models, S achieves comparable FIDs with only 5 NFE compared to baselines with 10 NFE, bringing a 2x acceleration ratio. Remarkably, our method achieves a significant boost under very tight budgets (like 11.50 v.s. 57.52 on CIFAR-10, 33.84 v.s. 76.69 on ImageNet- 64), making it feasible to sample with very low NFEs. (2) Choosing proper settings for existing methods is important. We can see that in many cases, the performances of the best baseline and the worst baseline are significantly different from each other, even both of which are under the SOTA frameworks. Additionally, the best baselines different datasets even under different NFE budgets are not consistent. It indicates the necessity of a careful selection solver settings and the significance of our USF since it proposed to present all adjustable settings systematically.
5.2 ABLATION STUDY: REDUCE SEARCH OVERIEAD
The main overhead of our method comes from evaluating the true performance of solver schedules since the predictor is much lighter than diffusion U-Nets and has negligible overheads. The time consumed is proportional to the number of generated images used for metric calculation. Fewer evaluation images lead to higher variance and lower generalization ability to other texts and initial noises, thus may cause worse performance. In this section, ablate this factor to give relationship
8

---

Published as a conference paper at ICLR 2024
Table 3: FID results on MS-COCO 256x256. Ours-500/Ours-2555 stand for using 500/250 images to calculate FID when searching.
NFE
Method
4
5
10
Baseline- (S))
161.03
156.72
106.15
75.28
58.54
39.26
29.54
Baseline- W(M)
30.77
22.71
19.66
18.45
18.00
17.65
17.54
Baseline-B
24.95
20.59
18.80
17.83
17.54
17,42
17.22
Ours
22.76
16.84
15.76
14.77
14.23
13.99
14.01
5urs-500
24.47
17.72
15.71
14.60
14.47
14.15
14.27
Ours-250
23.84
18.27
17.29
14.90
15.50
14.12
14.31
between overhead and performance. We choose the text-to-image task because the text condition is far more flexible than label condition no condition and thus more difficult to generalize from a small number of samples to other samples. We validate S with 250 and 500 generated images rather than the default setting 1000, and our results are shown in the last two rows in Tab. 3 The performance under almost all NFE budgets becomes worse as the number of generated samples decreases. Moreover, due to the high variance caused by lower sample numbers. Remarkably, the search results can still outperform baseline methods, indicating the potential of S
FIDs under higher budgets even higher compared to lower budgets
5.3 ANALYSIS AND INSIGHTS
We give some empirical insights in this section based on the pattern of our searched schedules. Our general observation is that most components of searched schedules show different patterns on different datasets. We analyze the patterns of different components as follows.
Timesteps. For low resolution datasets, more time points should be placed when is small. Unlike the default schedule "logSNR uniform", we suggest putting more points at 0.2 < t < 0.5 rather than putting too much at / < 0.05. For high resolution datasets, we recommend slightly smaller step size at 0.35 < t < 0.75 on top of the uniform timestep schedule.
Prediction Types. We find that data prediction can outperform noise prediction by large margin the following cases: 1. the NEE budget is very low (i.c., 4 or 5), 2. the Taylor order of predictor corrector is high and 3. on low resolution datasets. But for the rest cases, their performance has no significant difference. Noise prediction can even outperform data prediction sometimes.
Derivative Estimation Methods. We find that low order derivative estimation is preferred for high Taylor order expansion when step size is very high. Oppositely, full-order estimation is more likely to outperform low-order estimation when step size is not large. For derivative scaling functions, we find they are often used in all searched schedules. But the pattern varies among datasets.
Correctors.
Though previous work validates the effectiveness of using correctors Zhao al.) (203)),
we find that not all timesteps are suitable for correctors. For large guidance scale sampling, correc- tor is not usually used. But in general, corrector more likely bring positive impact on the performance.
6 CONCLUSION AND FUTURE WORK
In this paper, we propose a new sampling framework, USF, for systematical studying of solving strategies of DPMs. We further reveal that suitable strategies at different timesteps are different, and thus propose to search solver schedules with the proposed framework in the predictor- multi- stage manner. Experiments show that proposed method can boost the sample quality under very tight budget by a large margin, making feasible to generate samples with very few NFE.
Although propose general framework for sampling based exponential integral, we prune the search space empirically to avoid excessive search overheads. Exploring the strategies not used in this work could be a valuable direction for future research. Besides, method has additional overhead related to the evaluation speed and the number of sampled schedules. Therefore, fast evaluation methods and efficient search methods worth studying.

---

Published as a conference paper at ICLR 2024
ACKONWLEDGEMENT
This work was supported by National Natural Science Foundation of China (No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164), Tsinghua EE Xilinx Al Research Fund, and Beijing National Research Center for Information Science and Technology (BNRist). We thank Cheng Lu and Prof. Jianfei Chen for valuable suggestions and discussion, Yizhen Liao for provid- ing some experimental results and reviewing the revision in the rebuttal phase. We thank for all the support from Infinigence-Al.
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytie-dpm an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint vrXiv:2201.06500 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-878 8794, 2021.
David Francis Griffiths and Desmond J Higham. Numerical methods for ordinary differential equa- fions: initial value problems, volume 5. Springer, 2010.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based train- ing of neural networks. arXiv preprint arXiv:1711.0984 2017.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances neural information processing systems, 34:21696-21707, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. arXiv preprint arXiv:2305.0m 08891, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Enshu Liu, Xuefei Ning. Zinan Lin, Huazhong Yang, and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. arXiv preprint arXiv:2306. 08860, 2023.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint vrXiv:2202.09778 2022.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
10

---

Published as a conference paper at ICLR 2024
Cheng Lu. Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint (xXiv:2206.009271 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 2022b.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14297-14306, 2023.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm lan- guage models. arXiv preprint arXiv: 100::1700.00 0700000182, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162 PMLR, 2021.
Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor-based nas. In European Conference on Com- puter Vision, pp. 189-204. Springer, 2020.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780 4789, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202. 00512, 2022.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXivv2303.01446, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning efficiently sam- ple from diffusion probabilistic models. arXiv preprint 10Xi::21066038020 2021.
Chengrun Yang. Yuji Akimoto, Dae Won Kim, and Madeleine Udell. Oboe: Collaborative filtering for automl model selection. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 1173-1183, 2019.

---

Published as a conference paper at ICLR 2024
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506. 03365, 2015.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprinl arXiv:2204. 2022.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05540 2022.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. arXiv preprint arX2v:2302.04867, 2023.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statisties. arXiv preprint arXiv22310.13268 2023.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1610v1/61.01157, 2016.
12

---

Published as a conference paper at ICLR 2024
In the appendix, we first state several assumptions in App. A for the later discussion. Then, provide the details of our search space and methods in App. B and App. D In App. we elaborate on how existing methods we discuss before. In App. F and App. G.I] we list our settings for the experiments and provide all experimental results. Finally, give the derivation of how get Taylor expansion Eq. (8) and Eq. (9) through the diffusion ODE Eq. (4) and Eq. (5) and discuss the convergence of our framework.
be incorporated by frameworks based on the detailed algorithm
A DEFINITIONS AND ASSUMPTIONS
In this section we list some necessary definitions and assumptions for the discussion follows. We first define the convergence order of an arbitrary estimation T.
Definition A.1. Suppose is given value, i{h) is an estimation of the ground truth value z. If Ee, such that 4h, (ah 2| S ch", then we call ((h) is p-order converge h.
The defined p-order convergence can also be given as i(h) Ohh)).
Then we list several regularity assumptions for the neural network c0. Te. and the hyperparameters of DPMs.
Assumption A.1. (z, and (e(x,t) t) are Lipschitz w.rt. with Lipschitz constant L
Assumption A.2. The i-th derivatives deea(atdde) and d'za(1))ds) exist and are continuous for all 1 < i M, where M is an large enough value.
dx
Assumption A.3. 3B such that Vt, s E (te, and t < s. a 2 S.
B SEARCH SPACE DETAILS
In this section, we detail the searchable strategies for all components of the search space we use for experiments in Sec. 5 We also try several different designs and list the results in A.p.G..4
Timestep Schedule For all experiments, we set the start timestep IN = T following the conclusion in Lin al. (023) that setting IN # T can be harmful to the sample quality. For the last timestep lo- previous works usually set it to 0.001 or 0.0001 empirically (Lu et al. 2022a b). We find that the value to has large impact on the sample quality since low to can cause numerical instability and high to contains more noise. Therefore, we set to to be searchable in the interval [0.0001, 0.0015]. For other 4 where E [1,
1). all values between tg and iN are optional.
Prediction Type. For the experiments in Sec. we simply set data prediction and noise prediction as the two only strategies that can be applied for one-step update. We further try interpolation between the two forms: given xf calculated by Eq. (8) and æf calculated by Eq. (9), we set the final solution He to be ar; + - a)x. The results of this adjusted search space are shown in Tab. 19
Starting Point. We observe that in most cases, singlestep solvers perform worse than multistep solvers. Therefore, we simply set the starting point to be the previous timestep Zt,., of the target point Zt2 for all experiments in Sec. 5 To have better understanding of the singlestep approach, we further set the starting point searchable with two candidates Tt,4 and Tt for every It,, and list the search results with this new search space in Tab. 18
Taylor expansion order. As discussed in the previous work Zhao Zhao et al. (2023), too high solver order detrimental to the solution, possibly because of the numerical instability of high-order derivatives. So, we set the candidates of Taylor expansion order as 1, 2, 3, and 4.
Derivative Estimation Method. As discussed in Sec. 3.1 we divide the estimation of derivatives into two steps: Taylor-differene and scaling operation. For the Taylor-difference method, the main idea utilize Taylor expansions from other points the target points and eliminate all derivatives of other orders to preserve the derivative needed to be computed. We give the formal definition here.
Definition B.1. Given the function value of the target point f(r,, t), and m additional points f( t1), f(rz t2),... "f (Zt,,, tm) where t; = + r;h, then f(k) (2,, t) can be estimated by Dd a/ h where D is m-dimension vector given by D [f(xe.) [ (1), ), [ t)
13

---

Published as a conference paper at ICLR 2024
(2 12).... (x,, - f( f-)] And [a1, ag..... aml is the solution vector of the linear equation system Ca = b, where C is a square matrix with m rows and columns satisfying
Ciy = . for all integers i.j E [1] and is a m-dimension vector satisfying b, = -=== for all integers m].
The convergence order of Taylor-differenee estimation is given by m ++ 1. The proof is simple by substituting the Taylor expansion of f(r,,,)) to
to [Ia, 1) into the estimation formula.
Proof. Write the estimated value
f(() (12, 1) through Def. B.1
810 to)
1
Σ
m
a (f(s,tt - t()
(r,hs (1 10) + (rih)) (12))
1
2
(10)
i=1
(rih) ((m) (Irta to) + (1h
n!
(Eto 40) + Ohh mmhkk)
The last equation holds because a; is constant and independent of h. If we consider the error of all involved points Σt. and f satisfy the regularity assumptions in App. A then the convergence order should be = min(m,, 1.) k + 1, where Asm. A. the convergence order of [ ti) equals the convergence order of ãt.. By substituting the convergence order of f into Eq. (T)). we can get a similar proof.
the convergence order of ãt,, According to
The derivation of the last equation utilizes Ca b that Σ ai TŞ= All existing expo- nential integral-based methods set the number of additional timesteps to be the same as the order of Taylor expansion to keep the convergence order (see App. D for details). In our framework, we al- low low-order estimation for derivatives as an optional strategy, i.e., the number of additional points could be smaller than Taylor expansion order, since low-order difference more stable and our experiments in Fig. 2 have revealed the potential of this method. For the scaling operation, we use + R(h) to rescale the derivatives. We choose five scaling coefficients R(h) as follows.
(o) 0 (i.e., no scaling operation),
(10
* (h) oh wW- 1, W2
R2(h)= e²' -1-
Ra(h) = 0.5 - -1),
R4(h) - 0.5(e - -1).
where h1 / when the prediction type is noise prediction, and H' -h if the prediction is data prediction. The choices of R2 and R2 are inspired by existing methods (Lu al. 2022a b Zhao
et al. 2023) (see App. for details). R3 and R4 are chosen for larger scale coefficients. We demonstrate the R(h)-h curve in Fig. 3 All R(h)s are small quantities of ((h) order(except Ro (h)) and thus convergence is guaranteed for sufficiently small h. Note that though we only try to apply low-order Taylor-difference and scaling operation to the 1-st derivative out consideration for the potential instability of high-order derivatives and the difficulty in search phase, these methods can be applied to high-order derivatives. Furthermore, in addition to the proposed two-step method numerical differentiation methods also be
with Taylor-difference and scaling, other applied. These extensions are left for future exploration.
Corrector. As discussed in Sec. 3.1 the correcting process be viewed an independent step using the function evaluation of the target timestep. Therefore, the searchable strategies listed above can also be applied here, causing too large search space. To avoid this problem, we simply increase
14

---

Published as a conference paper at ICLR 2024
Noise Prediction
Data Prediction
0.20
0.20
kpm:
0.15
0.15
0307
0.05
0.05
0.00
0.00
-0.10
0.15
-0.15
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Step
Step
Figure The scales of all R
Table 4: FID results of independent search of every component in $ on CIFAR-10. 10. All results are calculated on 50k generated samples.
NFE
5
7
10
Baseline-B
23.44
6.47
3,90
Timestep
10.06
5.36
2.88
Order
17.76
5.86
3.90
Prediction Type
23.44
6.47
3.90
Corrector
22.58
6.23
3,90
Scaling
21.67
5.56
3.90
Low Order Estimation
23.44
5.91
3.84
the Taylor expansion order and the Taylor-difference order for all derivatives of the corrector by compared the corresponding predictor. The prediction type kept consistent with the predictor. No derivative scaling operations are used for the corrector. In this way, the searchable part only includes whether to use corrector, making the search problem easier to handle. Further exploration of independent strategies for the corrector can be a valuable direction for future research.
C ADDITIONAL INVESTIGATE THE POTENTIAL OF EVERY COMPONENT
In this section, we first discuss the improvement potential of each component and offer suggestions on which component should be paid more attention in the future designing of solver schedules. We find from search results that most datasets, the timestep schedule and order schedule the most different with baseline methods among all components. To further validate this observations, independently search for every single component and fix others. We only evolutionary sample a small amount of solver schedules and truly evaluate them to do the search. The results are listed conclude that timestep is the component with the
in Tab. and Tab. From the results most improvement space for baseline and order comes as the second. Moreover, all searchable components can be improved enhance the sampling quality. We suggest that for future design of solver schedule, the design of timestep and order schedule should be treated with high priority and the strategies other components could be tuned for better performance.
D ALGORITHM DETAILS
In this section we detail the search process 53
D.I DETAILED ALGORITHM OF THE UNIFIED SAMPLING FRAMEWORK
We present the detailed general process of our sampling framework S in Alg. We mark all searchable components of the framework in italies.
15

---

Published as a conference paper at ICLR 2024
Table 5: FID results of independent search of every component in 5 on ASUN-Bedroom All results are calculated on Ik generated samples.
NFE
5
7
10
Baseline-B
33.92
25.94
23.65
Timestep
27.23
22.99
18.64
Order
31.89
25.41
22.38
Prediction Type
32.43
24.84
23.25
Corrector
33.40
25.60
23.27
Scaling
32.84
25.60
23.25
Low Order Estimation
33.92
25.62
23.52
Algorithm Sample through USF
1: Get timestep discretization scheme (to, ,t]].
2: Sample Ety from (0,
1, do 4: Get the prediction type from noise prediction or data prediction of neural network.
3: for
5: Get the starting point lss, where 8; 2 i.
6: Get the order Taylor expansion R4, where n4 S N - i.
7:
h = A1; At.,
8:
if noise prediction then ctec
9:
IT,
ate Its;
10:
for k = 0, ... n4 do
Eo (It,, to.) t- Estimate 6g (k) (Es, ts.) 2t, 24, h h ( (I to. end for
(())
11:
12:
13:
Correct Ft, using ca(
15:
else if data prediction then
diIt, 31, ai.s
16:
for 1o 0,... 44 do
(1h) (10) (It,, 1.,) t- Estimate (0) (st bei) 6
18:
3t, = st, + +o, h+ paktligHi ph+1 (1) (Dt, lar) 20: end for
19:
21: Correct 2t, using (o)
22:
end if
23:
end for
24:
return Vito
D.2 DETAILED ALGORITHM OF MULTI-STAGE SEARCH
As discussed in Sec. schedules as described in Alg. 2 and demonstrated in App. D.2 We first initialize the population with baseline solver schedules mentioned in App. F3 to avoid the 'cold-start' dilemma since most solver schedules the search space are not good enough. In the first iteration of the search process, conduct the evolutionary search based on the performance of all sampled schedules since no predictor can be utilized. At the end of each iteration, we use all evaluated schedules to update the weights of the predictor, which will be used to guide the evolutionary search in the next iteration.
multi-stage search process search for well-performed solver
D.3 PREDICTOR DESIGN
Architecture. Our performance predictor P takes parameterized solver schedules input and out- puts the performance. The predictor contains three modules: timestep encoder -+ ,fM] + [Emb]T ... Em which maps the sequential timesteps to sequential embeddings using a po- sitional embedder followed by ML.P (Vaswani al. 2017 Ho et al., 2020) [Song al.; 2020b);
16

---

Published as a conference paper at ICLR 2024
Dataset of Evaluated schedules
Score: F
2. Evaluate true performance sampled schedules
Train the predictor with evaluated schedules
Score: F2
Predictor
Sampled Schedules
4. Use the trained predictor to guide evolutionary search
1. Sample subset of schedules through evolutionary search
Solver Schedules
Search Space
Figure 4: The iterative workflow of predictor- multi-stage search.
Algorithm 2 Predictor-based Multi-stage Search
Require:
S: search space of solver schedule
P S + IR: predictor which takes a parameterized solver schedule as input and outputs its performance
S + N+: get the NFE of solver schedule
Hyperparameter:
N(0); number of solver schedules sampled in the k-th iteration
Input:
C2, Cr.]: series of timestep budgets
Search Process:
Initialize P randomly
2: Initialize S with baseline solver schedules
3: for k = 1.... MMAXIITE__NT M do
4: if 1 = then
5: Sample set of solver schedules S(k) = {sj} (s.))=,,...,,W (00) from S using evolutionary search guided by the ground-truth performance.
6: else
7:
Sample set of solver schedules ((k) {s, {= Nay N00 from S using evolutionary search
guided by the predictor P
8: end if
9:
Evaluate the ground- truth performance P, of all solver schedules sy S(R), get 5(h) (s,, .N(0)
.N(6)
10:
S and Š is used to train the predictor P 11: end for
8
arg min Pj. S.L. N(s)) < <;
return S
2. encoder of other solving strategies, which uses one-hot embedders followed by MLPs to embed the decisions of component (e.g., order of Taylor-expansion) Cg and get [Emb; Emb ] Embeddings of decisions from all components are concatenated to obtain the final embedding of solving strategies [Embç.... ... Emb;, [Emb/]-. ... |mmb,' ..... Emb,y ... |mmb K] where K the number of searchable components in the our framework of solving Eq. (6); 3. sequence pre-
17

---

Published as a conference paper at ICLR 2024
Emb |mmbyyl as imput and outputs the final predicted score, An LSTM (Merity et al. 2017) is used to process the sequential embedding, followed by an MLP to regress the final score.
dictor, which takes the concatenated embedding [Emby] Emb; k
Training. Since ranking information of different schedules is far more important than their absolute metrics, pair-wise ranking loss to train the predictor, which is validated to be more effective in extracting relative quality information and preserving the ranking information (Ning et al. 2020
Liu et al. 2023).
loss Σ
max (P(s,) P(s:))),
(11)
where $ is the dataset of schedule- pereoulaeppr pair data. Lower output value of the predictor trained with ranking loss indicates higher performance of the input solver schedule.
E RELATIONSHIP BETWEEN USF AND EXISTING EXPONENTIAL INTEGRAL BASED METHODS
In this section, we discuss the relationship between existing exponential integral methods (Zhang
& Chen 2022 Lu al., 2022a b. Zhao al., (023) and USF, We show that all existing methods in the À domain (Lu et al. 2002a by Zhao et al. (002)) are included in our framework by assigning corresponding decisions to every component. The method in / domain (Zhang & Chen 2022) also shares strong relevance with USF.
E. DPM-SOLVER
DPM-Solver (Lu et al. 2022a) is singlestep solver based on the noise prediction type Taylor expansion exponential integral. et (2022a) gives two examples DPM-Solver-2 and DPM- Solver-3 in Algorithm-1 and Algorithm-2 of its paper, correspondingly. We discuss the relationship between these two examples and USF below.
DPM-Solver-2. In DPM-Solver-2, to calculate the trajectory value Ty. of the target point 4 from
the starting point 1; midpoint s; E (+-)) is given by = tt( - and its trajectory value u; is calculated through the 1-st expansion of Eq. 18 (i.e., DDIM (Song et al. 2020al) from the starting point 1 Then 2,, is given by: Ir ae It.-, (e I)ep [; s,) (see the
2
Algorithm-1 in the original paper Lu et al. (002a) for details). We further write this formula:
0t It, (e - 1)ea(ue, 8i)
Tt, 0t:11
0t. -ti-1 -0 (e - tr-1) - (e, (asein1))((((seeee 8:) - (o(
01-1
04 -i, (eh h - 1)Ea(Trr 11b-1)
0t
Ot: (e 1))(,, )
chi - h4 1 1 p2pp(hh)) co(us, 81) 6B cā( Ag N
ēt, _0,
0i
or, (e/e 4 1)ca(Tr, or, + Ri(hs) colui, 8i) - co( tr-1)
0tt-1
As; At:-1
We can see that the 2-nd solver expands Eq. X to the 2-nd order and uses 1 1+ Ri (h+) to scale the 1-st derivative estimated by Taylor-difference (degrade to direct two-points difference approximation in this case). Therefore, DPM-Solver-2 can be viewed as two updates with Taylor expansion orders and 2. The second update uses second point 1 before the target timestep as the starting point and 1 + (h) to scale the 1-st derivative.
DPM-Solver-3. In DPM-Solver-3, two midpoints $2--1 and 82 selected to calculate the final solution ,,,- Firstly, the trajectory value of 8 is calculated by the 1-st solver DDIM (Song et al. 2020a). Then, the trajectory value of 82 is calculated from the starting point f;-1 and the previous
18

---

Published as a conference paper at ICLR 2024
midpoint --
0'g2i Tt,-1 Szg (6 - ()eg(Et))
U2i
at
Osz (Moz) - A A -de-s -1 1) (1 5o 10 41-1))
Asgi-1 A81-1
Azge Xen-
aszi Dii-1 Jozi (EArzs-des - 1)eg((e)).11tt11
a1,-1
10(u2i-1, 821-1) - co (2 11-11)
Jezi (dozi Ar M No,_.)
Auz - )1,
This process also uses the 2-nd order expansion of Eq. 8 like DPM-Solver-2, but uses R+oo de 1) rather than 1 + R in DPM- 2. The final calculation of कG, is similar to the above process but uses $2; as the midpoint of this 2-nd singlestep solver. In conclusion, DPM-Solver-3 can be viewed as three updates with Taylor expansion orders 1, 2, and 2. The last two solvers both use 1--1 as the starting point.
E.2 DPM-SOLVER++
DPM-Solver++ (Lu et al. 2022b) propose several new solving strategies based on the framework of DPM-Solver (Lu et al.
2022a). The first is to switch the noise prediction model to the data prediction model, both of which are optional strategies in USF. The second is to apply multistep solvers. The original paper provides the process DPM-Solver+++(2)) and (PM-Solver++*2M) in Algorithm and Algorithm 2. We discuss the relationship between these examples and USF as follows.
OPM-Solver+++2S) DPM-Solver++ is similarly designed to DPM-Solver-2 Lu al. (002aa) except for switch to data prediction. See the Algorithm in Lu et al. (2022b) for details. We can draw an analogous conclusion DPM-Solver-2 by rewriting the formula of the 2-nd update.
he 1)h; 26 (..: 81) xo (
Tt: T1,-1 - (e - () (Rt, 14-1) - Ott;
ft. =
0t-1
2
As. dtm
0n -1,-1 ar, (e-h, 1)x0 (2 ti -1)
Ot.1
(e 1)
xo 5o(00s. (8s,,,))) 8e)
xo 20( ti-1)
0tt: c-h - + h
11 5p (11s)
)o; Ati-
Th: Z1.
(c - 1)xx (2+, 1.t 1;-1)
Ot.-1
(Z:; si) (Ttz 1. t.
an, (1 (hh
do; di
We can see that DPM-Solver+++(2) also uses (h) to scale the 1-st derivative. Therefore, predic- tion type is the only difference between the two methods.
(2M)) (2M no longer uses singlestep update. Despite the first step. which needs cold start with the 1-st solver, the rest of the steps use 2-nd expansion of Eq. (9) (See
19

---

Published as a conference paper at ICLR 2024
the Algorithm 2 in Lu et al. (202bb) for details). We reformulate the 2-nd update formula below.
(e 1)h, 2 (T t1-1) 20 20 2--2
0n Bi.-1 as, (c, (2 :t-11) au,
Ft. 1
2
Xn-- A6,-
01 Tt,-1 - (0 ()x 1:ti-1
01,
h (e- hi - 1)
Ie x0( t(-1) - 20(0,, 222)
h h2 (h:)
at: e - 1 h
Au: Av;
T1. 04- 81--1
011, (e 1) (8 1i-1)
Oh--1
Fg F (i-1) Fzll (#eizz 22-2)
- -04, (1 (1b.))
A1-1 X4-2
We find that the only difference between DPM-Solvert+(2MM) and the 2-nd update of DPM- (olver++(2S) the position of starting point. In the 2-nd update of DPM-Solver+++(2S) the starting point is the second point before the target point. In DPM-Solver++(MM), the starting point is the first point before the target point.
E.3 UNIPC
proposes two update methods, UniP and UniC. In fact, UniC can be viewed as special version of UniP with the involvement of the function value c0 t) or (x at the target point t. Therefore, we mainly discuss the relationship between UniP and USF.
UniPC (Zaoo et al. 2023)
UniP. Take noise prediction model
example, the update formula of Uni-P is given by It. o:; "4 - oz, (E 1)eal (āt, t4-1)
o4, ((h,) Emsl am. where m eg (
Tm
co co( lim), ap-1 = [a1, 02 ... ap-1) satisfies ap-1 = B B-1(h, p (h. f (hi), and
Ry- (h) and 1 (l) are given below (see the Algorithm 2 in Zhao et al. 2023
1
1
rih
rzh
Tp-1h
1h-1(1)=
(r1h)m-2
(r2h)a-2
(rp-1h)
Φ ( = [1:h455(1) (l), 22h22hh22p (p (p - 1)!! 1)11P((3(0 (p))
Then suppose the solution given by our USF with p-th Taylor expansion and pure Taylor Taylor-difference estimation for each derivative is 2// We state that a!. ã. if the starting point and all additional points in the two solvers are the same. The proof is as follows.
Proof.
_1 Ct I4,-1 D4,
01, k+t (h)e (xmnnti-))
(12)
ar,11
p-1
T1.-1 0
Otta
Olt
Ti, (e,( - (::((t;;1::-11 (1111) - 0t:
Σ
n k+1
ph+1 Dan
(13)
(tt--1
k=1
where DU = [Ea 1, eo dii,,...
col 1: (1-1) eo( , 4i } and ag sat-
isfies ay = Cū bb. As discussed in Def. B.1 Cy and b/ are written as:
T1 //2
r2
Tp-1
r 22
7: 1
C
rp
1(p - 1)! 1h (pp 1)!
/ / 1))!
b/
[Di, b2. bp- b, = T
20

---

Published as a conference paper at ICLR 2024
substitute Du, Cy and by to Eq. 13)
p-1
z an F4--1 - on, - (--1)
(1-1) - ot. Σ
hoph (h) ( bj
(14)
att-1
an; Iti-1 on () 11--1) - ot -0
pk 1 (h)
(15)
04.1
p-1
at -4i-1 (e -
01,
Dm
(16)
0t:-11
m==
where qu is the m th element of vector q h
fk (N)CT = CT-by- where byy
(h), hops)))))p))))... (h))())
Then we rewrite this equation by left multiply Cy at both sides: ( (h),
P2
Tp-
qi
r9/21
r2/21
Cuq=
42
(17)
rf /(p - 1)!
r
1pp-1) 1)!
/ / 1))!
9p-1
hup((h)
(h)
= b/y
(18)
hupp(h)
This equation is equivalent to:
T1q1
hup5
rih
r2h
Tp-1h 1h
T242
21h 21h2+gg(1)
(19)
...
(rih)--2
(r2h)x-2
(rp- (1)-1))-22
pp-19p-
(p - 1)1h1-1
Compare Eq. 19 with ap-1 B (h;) R (h;) pp is obvious to obtain that 7m
am B(,). Substitute this equation to Eq.
I6 get:
Oct Dz.-1 (c,
(4-1) oo, B(ha)
Σ m=1
O. Dm Dt.
(20)
a1
Tm
We can that UniP is equivalent to using full-order Baylor-difference to estimate all derivatives.
The coefficient Bhh) is eliminated here and thus has no impact on the sample quality. However, the implementation (see https: / /ggtttb........g com w1 zhao / UniPC for details) of UniP applies special case for the 2-nd solver:
Rr.
Ott Dti-1
(6 (co ate 2
B(h)) D1
att
Iti- (E - 1)ep(Temr)
Olt,-1
h B1B(hi
6 ti-1) Ep( ttp)
Ti, 2(e - h4 - h2pg((h)
Xi.n - Ae.
This is equivalent to scaling 1-st derivative with 1+ R(h)- = B.B(h)) Stchi-h;-1 UniPC Zhao et al. 2023
empirically provides two choices for B(h): (h) = h and B2 = h 1. The corresponding
12 R(h)) for (h) is - - 1 (2) and the corresponding R(h) for B2(h) is ((o0-1)
1
ch-
Rí (h).
21

---

Published as a conference paper at ICLR 2024
In conclusion, 2-nd order UniP is equivalent to using R₃ R2 to scale the 1-st derivative, while UniP with other orders is equivalent to applying full-order faylor-difference method to all deriva- tives without scaling.
UniC. UniC has no fundamental difference with UniP. Inspired by it, our framework USF set the corrector to be searchable component without re-evaluation of the function at the target timestep.
E.4 DEIS
As discussed in Sec. 2.2.2 DEIS uses Lagrange interpolation to calculate the integral domain directly without Taylor expansion, given below.
4--1
01:-1
St.
04-1
X (F) T)dT
((F
(21)
at.
01:-1 St.
04
(T)
EI tits ti+k
P
Lith ( lits dd,
(22)
at.
Note that other methods (Lu al. in the Ả domain, so DEIS has fundamental differences with these methods. However, we show that the numerical integral estimation method of DEIS (i.e., Lagrange interpolation), which one of the
2022a Zhao et al. 2023 including USF estimate the integral
core designs of this work, can still be incorporated in USF in the À domain. We rewrite Eq. (2)
w.r.t. A:
0H1-1 St.
Tt.-1
Ct.-1
E r)dA
(23)
O't
JAn
0tti11 It. -
P
A Aei )eg(Ie M
(24)
0t.
Aei
Atsts Atoik
j=0 kijj
Lagrange interpolation with Ea(x;, At) Ap + ap-12p-1 +... + aiλ + do, which pass through all points. We rewrite this
points ep 17
(j .,.., ,)) constructs a polynomial
polynomial to make the starting point d be the 'center": EØ (2,, Ae) = Se (A >r,) + Sp-1 G-1y (A
p
Xx,) )-- +... + a1(A de;)) c0(3t,,)),)) and construct linear equation system solve all coefficients.
Aumi - Ads (Dect 2/2
(Deu dr. /ppp (Deus de, P/pl
a1
A du (deuz Ne.)
a2
A do (Nesup 2/,
(Neeo Ae, \p/p!
Op
c c deaes
c (o(stes dca) ED(It, )t.)
ep (It,421 desuz)
col desup) co (Be,s de,)
Suppose h4 At_ de,, decaa
d rahy, and cp( drne) colIt; de;)
DR. Then
rewrite the above equation:
T2
Tp-1
rf/21
r 21
rセ-1
Up
r (pp 1)!
r--
1p-1/(p-1))
p-1 / 1))!
=
D1 D2
Dp
Extract the k-thh column of the both sides.
F1 r
T2
Tp-1
r
//-1/21
as = Dr D2
Dp
1
1 p- -
p-
=DC-Lb
22

---

Published as a conference paper at ICLR 2024
The value of D, C, and b can be found in Def. B.1 by replacing f with co and with A. We can see that ag is equal to the k-th derivative at timestep t, calculated by full order Taylor-differenee Therefore, the update formula using USF with expansion order p + and full order Taylor- difference estimation for all derivatives is given by:
Ott--1 St.
Σ k=
Pkt (1;)an.
h k+1
Ott
And the update formula of DEIS in X domain is written
A
:i-- 0f,-1 a1
Σ a dda.
Zt. Oit11
It is proved in App. H that oz, 1 ++1 + (h:) on, Jx e Xr. dd, so that कV.
-t.-.. In summary, estumating the integral through Lagrange interpolation in À domain is equivalent to using full order Taylor- dieferrdiifeeence to estimate all derivatives. Thus, DEIS in À domain can also be incorporated in USF.
F EXPERIMENTAL SETTINGS
In this section, we list the settings of our experiments.
F.1 MODELS
All models we use in the experiments are open-source pre-trained models. We list the information of them below.
CIFAR-10 We use the model the paper (Song al.] (2020b). This model applies linear noise schedule with VP-SDE and is trained with continuous-time. It can be got by download- ing the checkpoint ZZhLBbuusT--parrruuuiIIAAAEa
wet.:pppeeeecoonuuuuuusssu. continuousssseepp provided by
/ttps::/drive.eooo wttes/le...cmeeooooo drive ftw.ee//foodrr////ooooo
CelebA We use the model provided by the paper Song et al. (2020a). This model applies linear noise schedule with VP-SDE and is trained with discrete- time. It can be downloaded at SaK.
www..oowwoo.oooooooooooooooooooo wtt.cccmm..lleee file/d/1R
ImageNett64 We use the unconditional model trained with L.hybrid provided by the paper Nichol Dhariwal (202T). This model applies cosine noise schedule with VP-SDE and is trained with discrete-time. It can be down- loaded at Maachh-2221/mmagene
blob core windows. net/diffusior.
uncond 100M 1500K.
LSUN-Bedroom We use the model provided by the paper Dhariwal & Nichol (022 This model applies linear noise schedule with VP-SDE and is trained with discrete-ting time. It can be downloaded at https: http//ppaaapbli.../o//p/o httpss///penaaipblllic wtaliiccbbbbcooo blob. windows, net diffusion/Jun 200 /lsu bedroom.pt.
ImageNet-128 (classifier guidance) We use the model provided by the paper Dhari- wal & Nichol (002T) This model applies linear noise schedule with VP-SDE and is trained with discrete-time. It can be downloaded at wttpphps/s///ppppaappbbb........
bbob. The
core.winde net
classifier is also provided by this paper at http:: pttppss///oerrappbbb......bbb...
ImageNet-256 (classifier guidance) We use the model provided by the paper Dhari- wal & Nichol 2022 This model applies linear noise schedule with VP-SDE and is
trained with discrete-timme It can be downloaded at http:h/wwwetaacaepppppopp/ppaa.a
core windows. net /ii htt.://.....aaa....aaaaaa/////// www.ssiinfffffiiiic.o The
http::::ttttttaatattaaaat windows. net. ftt//ifffustt prtprpa....aaaaaaaa.....o..oonna
classifier is also provided this bb
23

---

Published as a conference paper at ICLR 2024
MS-COCO 2 (classifier-free guidance (Ho & Salimans 2022) We use the model
provided by the paper Rombach et al. (2022). This model applies linear noise sched- ule with VP-SDE and is trained with discrete-time. Additionally, this model is trained in the latent space. can be downloaded at httpt:://///.. htattp::///ugggnggaee co/ runwayml stable-diffi on
F.2 EVALUATION SETTINGS
We sample images to evaluate the FID as proxy the final performance in the search phase. For the final evaluation, we sample 50k images for evaluation on all unconditional datasets and ImageNet- 128 (classifier guidance) dataset and 10k images on ImageNet-256 (classifier guidance) and MS-COCO x56x256, For CIFAR-10 and CelebA, we get the target statistics by calculat- ing the activations of all images in the training set using the pytorch-fid repository [htpps: //github. com/ commmmeei mss t wtter//ytorch-feee fid. For ImageNet-64, ImageNet-129, ImageNet- 256, and LSSUN-Bedrom, we get the target statistics /htpps: thub com compappaaa/// guided di
ions For MS-COCO 256 256, we calculate the activations of all corresponding images with their caption used for generation using the pytorch- fid repository. To align the settings of text-to-image evaluation with Rombach al. (2022), resize the short edge of the image to 256 and center crop 256 x56 patch before feeding them into the Inception V3. We keep these settings for all samplers in experiments.
F.3 BASELINE SETTINGS
We mainly choose the three SOTA methods DPM-Solver (Lu et al. 2022a), DPM-Solver++ iet al. 2022b) and UniPC (Zhao et al. al. (023) as our baselines. For the evaluation of DPM-Solver and
DPM-Solver++. we use the code at httt:::: /ttp::////tthiie www/LoCChegglaaaaoooccccac /dp-HHHUdpppaae solver and solve DPM-Solver-35 DPM- and DPM-Solver+++3MM as base-
choose
com/ww-2hacc/niPP and choose UniPC-2-Br (h), UniPC-2-BB(A), UniPC-3-B1 UniPC-3-B,(h) and UniPC as baselines. We follow Lu et (2022b) choose uniform t as the time schedule for nigh-resolution datasets (i.e., ImageNet-128. ImageNet-256 and LSUN-Bedroom) and text-image generation. We use lower or- ders at the final few steps for DPM-Solver++ and UniPC following their implementation. We keep other settings default in the official implementations of these samplers.
lines. For UniPC, we use the code https:
F.4 SEARCH ALGORITHM SETTINGS
We sample and evaluated 4000 solver schedules CIFAR- 3000 solver schedules on CelebA and ImageNet-64, 2000 solver schedules ImageNet-128 and ImageNet-256, and 1500 solver schedules on L.SUN-Bedroom and for text-to-image generation. We sample 1000 images to evaluate the true performance of all solver schedules. We use stages the multi-stage search process, with the first stage evaluating half of all schedules and the other two stages each evaluating a quarter of all schedules.
F.5 DISCUSSION OF SEARCH OVERHEADS
Since the we need to generate images for solver schedule evaluation, calculate the metric and train predictors, our method introduces additional overhead. The network used to calculate FID is much lighter than diffusion U-Net and need not to inference iteratively. The predictor is even lighter and its training costs only tens of minutes. Therefore, the total cost is mainly on the generation of images. This overhead is equal to the time cost of N N times neural inference of the diffusion U-Net (together with the classifier if it is glassifier-guidanee sampling), where N is the number of evaluated solver schedules, M is the number of images generated to calculate the metric, and S is the mean NFE of all solver schedules. N and M for each dataset can be found at App. F4 Since we conduct the search under NFE budget 4- 10 and we uniformly sample budget in this range, value 7 for S. We then test the runtime for diffusion U-Net to forward one batch of images on one single GPU and estimate the overall GPU time cost. We list the results in Tab. 6
the mean
24

---

Published as a conference paper at ICLR 2024
Table 6: Computational Cost of S stands for the total number of evaluated solver schedules. M stands for the number of generated images calculate the metric. S stands for the mean NFE of solve schedules. Time/Batch the GPU time consumed for the diffusion U-Net to complete one time of inference with the corresponding batch size. Total GPU hours is the overall overhead of the search, which is estimated by NxMxSS (Time/aatch//600, "Ours-250% stands for our ablated setting of using only 250 images calculate the metric, whose performance be found at Tab.
Dataset
N
M
Time/Batch
Total GPU hours
Device
CIFAR-I0
4000
1000
400ms/256
12.15
NVIDIA 3090
CelebA
3000
1000
500ms/256
11.39
NVIDIA 3090
ImageNet-64
3000
1000
788ms/256
17.96
NVIDIA 3090
ImageNet-128
2000
1000
2465ms/2256
37.45
NVIDIA A100
ImageNet-2256
2000
1000
2148ms/64
130.33
NVIDIA A100
LSUUN-Bedrom
1500
1000
7
3332ms/128
68.36
NVIDIA A100
MS-COCO
1500
1000
585ms/16
106.64
NVIDIA A100
MS-COCO (Ours-250)
1500
250
585ms/16
26.66
NVIDIA A100
Comparison with other methods We compare our overhead with a popular training-basee based method Consistency Model (Song et al. 2023) and concurrent work DPM-Solver-v3 (Zheng et al. 2023) of improving training-free sampler to demonstrate the efficiency of our search method $
Training-base based Method. While distillation-ban based methods have astonishing performance with very low NFE, their overheads are far larger than ours. Consistency model (Song et al., (203) is a popular and effective work which achieves very impressive performance with I- sampling steps. However, the training cost of this work heavy. According to Tab.3 in its paper, consistency distillation/training on CIFAR-10 needs 800kx512 samples, which is nearly 15 times more than 4000 x000x in our methods. Moreover, one iteration of training consistency models needs more than one forward pass (3 for CD and 2 for CT) and an additional backward process (nearly 3 x time cost than forward pass). Therefore, our search process is nearly 90 times faster than training consistency models. For large resolu- tion datasets, this ratio even larger. On LSUN-Bedroom. consistency distillation needs 600kx2044 samples, which is 117 times more than 1500 1000 in our methods, and the overall cost is almost 700 times larger. Besides, our method does not have GPU memory limitations, while training based methods suffer from this problem (e.g., consistency mod- els are trained on 8 GPUs on CIFARIO and 64 GPUs on LSUN). In conclusion. our S offers a effective trade-off between sample quality and computation overhead compared to training-basse based methods.
Training-free Sampler. DPM-Solvers (Zheng et al. 2023) is a concurrent work that aims to improve the training-free diffusion samplers. It replaces traditional model predic- tion types (i.e., data prediction and noise prediction) with a novel new prediction type. To calculate such prediction type, DPM-Solver-v3 has calculate empirical model statistics (EMS) related to the data distribution before being used for sampling. Therefore, similar to 3 this method also needs additional offline overhead for the EMS calculation. Accord- ing to Appendix D in this paper, we compare its cost with our 5 search in Tab. 7 We that the overhead of these two methods is within the same order of magnitude. It is worth mentioning that 5 can incorporate DPM-Solver-v3 by introducing its novel pre- diction type as another searchable strategy without changes to other components and be further boosted.
G ADDITIONAL RESULTS
G.1 COMPARISON WITH MORE BASELINE METHODS
We provide the full results of all baseline methods at Tabs. performance of these baselines in Sec.
to 14 We report best and worst
25

---

Published as a conference paper at ICLR 2024
Table 7: Comparison of the additional cost (GPU hours) between DPM-Solver-v3 (Zheng et al., 2023) and our method. We use the setting of 250 generated images for "Ours-250" and the results
can be found at Tab. and Tab. 16
Dataset
CIFARIO
MS-COCO
DPM-Solver-v3
56
88
Ours
12.15
106.64
Ours-250
3.04
26.66
Table 8: Full baseline results on CIFAR-10
NFE
Method
9
10
DPM-Solver-2S
247.23
32.16
32.15
13.22
10.80
6.41
5.97
DPM-Solvve-33S
255.21
288.12
23.87
14.79
22.99
5.72
4.70
DPM-Solver++(2MM
61.13
33.85
20.84
13.89
10.34
7.98
6.76
DPM-Solver+++(MM
61.13
29.39
13.18
7.18
5.27
4.41
3.99
UniPC-2-B1
62.67
33.24
16.78
9.96
6.87
5.19
4.41
UniPC-2-B2(h,
60.20
31.10
17.77
11.23
8.03
6.27
5.40
UniPC-3-B, (h)
62.67
23.44
10.33
6.47
5.16
4.30
3.91
UniPC-3-Ba(h)
60.20
26.30
11.64
6.83
5.18
4.31
3.90
UniPC,-
57.52
24.93
11.12
6.67
5.15
4.30
3.90
Ours
11.50
6,86
5.18
3.81
3.41
3.02
2.69
G.2 SEARCH RESULTS UNDER LARGER NFE BUDGETS
To further show the effectiveness of our method, we report the FID result under larger NFE budgets in Tab. [5] Noting that when the NFE budget is adequate, the negative impact of sub-optimal empir- ical strategies diminishes. So, it is nothing surprising that the gap between our method and baselines decreases. But we can see that our method still outperforms all baselines by large margin at larger NFEs like 12, 15, and 20. Our method completely converges at NFE=15, much faster than existing solvers.
G.3 MORE RESULTS WITH LOWER SEARCH COST
Since 3 has additional search cost, meaningful to investigate its performance if the search cost is limited. As discussed in Sec. 5.2 and App. F.5 the search overhead of 5 is proportional to the number of evaluated schedules and the number of images for FID calculation. In Sec. [5.2] demonstrate the results of using less generated images to calculate the metric. In this section, we conduct more systematic experiments on CIFAR-10 to investigate the impact of both the number of sampled images and the number of evaluated solver schedules. Our results are listed in Tab. 16 From these results, we can see that though most cases, lower search cost leads to worse search results, our method consistently outperforms the best baseline method significantly. These results show the large space of optimization for baseline solver schedules, requiring only small search cost to enhance.
G.4 SEARCH RESULTS WITH OTHER SEARCH METHODS AND SPACES
In addition to the search space we introduce in App. B and the multi-stage method we introduce in Sec. 4.2, try other search space designs and search approaches. We introduce these methods in this section.
G.4.1 SEARCH METHOD
Cascade Search The search method introduced Sec. 4.2 searches for all components simultane- ously, which may lead huge search space and thus is inefficient. We try another cascade search
26

---

Published as a conference paper at ICLR 2024
Table Full bascline results on CelebA
NFE
Method
4
8
9
10
DPM-Solver-2S
313.62
31.14
52.04
7.61
7.40
4.98
4.37
DPM-Solver-33
321.39
330.10
25.14
17.28
16.99
10.39
6.91
(2M)
31.27
20.37
14.18
11.16
9.28
8.00
7.11
DPM-Solver++(3M)
31.27
14.31
8.21
7.56
6.43
5.17
4.47
(niPC-2-B1,h)
28.51
13.90
8.95
7.44
5.96
4.84
4.20
UniPC-2-Ba(h))
29.95
18.57
12.29
9.83
7.74
6.81
5.97
UniPC-3-Bi(h)
28.51
8.38
6.72
6.72
5.17
4.21
4.02
UniPC-3-By(h)
29.95
12.44
7.84
7.38
5.92
4.71
4.25
-
26.32
10.61
7.31
7.13
5.68
4.55
4.15
Ours
12.31
5.17
3.65
3.80
3.62
3.16
2.73
Table 10: Full baseline results ImageNet-64
NFE
Method
4
5
7
8
9
10
DPM-Solver-2S
321.67
65. 16
72.47
34.08
35.00
28.22
27.99
DPM-Solver-3S
364.60
366.66
51.27
47.84
54.21
23.89
24.76
(PM)
82.72
63.48
50.35
40.99
34.80
30.56
27.96
DPM-Solver++(3M)
82.72
69.06
45.32
33.25
27.65
25.47
24.56
UniPC-2-B;(h)
93.98
64.38
49.12
36.12
29.42
26.04
24.23
UniPC-2-Ba(h)
80.70
61.73
47.61
38.03
31.96
28.08
25.79
UniPC-3-B:
93.87
67.32
41.73
31.45
27.21
25.67
24.76
UniPC-3-B2(h)
80.73
69.08
43.80
32.05
26.98
25.26
24.35
U
76.69
67.05
42.81
31.76
26.99
25.34
24.44
Ours
33.84
24.95
22.31
19.55
19.19
19.09
16.68
method. We start by searching for only some of the components and keep other components con- sistent with the setting of baselines. After obtaining the search results with this sub-search space, we search for other components based on these searched schedules and keep the rest of the compo- nents unsearchable. In our experiment, we search for timesteps, Taylor expansion orders, prediction types, and correctors at the first stage, then search for the derivative estimation method in the second stage. The results of this search method are shown in Tab. 17 The number of sampled schedules is kept consistent for both search methods. Although the cascade method more likely fall into local optimal, our results show that under some NFE budgets, the cascade method finds better solver schedules, indicating that it can also be used for searching.
G.4.2 SEARCH SPACES
Starting Point (SP) Since singlestep methods usually perform worse than multistep methods, don't include other choices of starting points than the point before the target timestep in the original search space used by experiments Sec. To explore the potential of more distant starting point, we add the second point before the target timestep into the original search space and obtain the results shown in Tab. 18 (see the row 'Ours-SP'). We keep the search method and settings constant. We find that though this new search space provides more potential to find better solver schedules under some NFE budgets, the search results under most budgets are worse than the original search space. This is because the newly added decision does not bring improvements in most cases, making the search process more inefficient. Therefore, we recommend using multistep solver when the NEE budget is very tight.
Interpolated Prediction Type (IP). There are only two choices for prediction type in our original search space. We further try interpolation between these two types given below:
St. ax, a)
27

---

Published as a conference paper at ICLR 2024
Table 11: Full bascline results on LSUN-Bedroom
NFE
Method
7
8
10
(PM-Solver+++2M)
44.29
24.33
15.96
12.41
10.41
9.28
8.49
DPM-Solver+++3M)
44.29
20.79
13.27
10.85
9.62
8.95
8.38
UniPC-2-B1(4)
38.66
19.09
13.88
12.06
10.87
9.99
9.22
UniPC-2-Bz(h (h)
22.02
20.60
13.79
11.27
9.92
9.11
8.52
UniPC-3-BI(A)
38.66
17.98
13.09
11.49
10.47
9.66
8.89
UniPC-3-Ba(A)
22.02
18.44
12.44
10.69
9.76
9.14
8.53
-
39.14
17.99
12.43
10.79
9.97
9.26
8.60
Ours
16.45
12.98
8.97
6.90
5.55
3.86
3.76
Table 12: Full baseline results on ImageNet- 128 (classifier-guidane) generation, 4.0)
NFE
Method
4
5
8
10
(2M)
26.59
14.42
10.08
8.37
7.50
7.06
6.80
DPM-Solvcr++<3M)
26.58
15.02
9.30
7.24
6.54
6.44
6.37
UniPC-2-B, (h)
32.08
15.39
9.01
7.13
6.62
6.49
6.36
UniPC-2-By
25.77
13.16
8.89
7.49
6.88
6.68
6.50
UniPC-3-B1
32.08
16.14
9.81
7.20
6.28
6.03
6.03
UniPC-3-Ba
25.77
14.68
9.57
7.24
6.28
6.06
6.04
U
26.60
14.82
9.55
7.31
6.35
6.12
6.03
Ours
18.61
8.93
6.68
5.71
5.28
4.81
4.69
where 2. and 2 given by:
n
Ot
0k+1 (h)eg (0)
(I
2
T
0t
0.
k=0
h k++
Fk+1 (h)aa) (Ma,
11 0
8g
04
Note that interpolating the neural term by fe( (2, + (1 - a)ce (2,, will lead to an integral term which can not be analytically computed as the integral in Eq. (36). causing a more complicate situation, for the interpolation ratio a only and evaluate another 500 schedules based on the search results of the cascade search method (see App. G.4.I) on CIFAR-10. The results shown in the 'Ours- IP' row of Tab. 19 We can see that the continuous search space of prediction type may have the potential to boost the search results with low NFE budget like 4, 5,
simply interpolate the solution of these prediction types. We search
Guidance Scale (GS). The choice of guidance scale for classifier-free guidance has a significant impact on the content and quality of the generated image. Usually, guidance scale is given by the users to adjust the control strength of provided conditions. But in some cases, users only care about the image quality and have to try different values of guidance scale to get high-quality samples. Therefore, we propose to include this hyper-parameter in our search space. Like other strategies, we allow the guidance scale of each timestep be searched independently. Our search results are shown at Tab. 20 From the results of baseline methods with different guidance scales, validate the fact that a suitable guidance scale is very important for sample quality, and our 5 can help find proper guidance scales to achieve higher quality generation. Noting that sample quality evaluated by different metrics may have different preferences on guidance scale, but using S can help to automatically find the optimal setting given arbitrary metries.
28

---

Published as a conference paper at ICLR 2024
Table 13: Full baseline results on ImageNet-256 (classifier-guidance generation, 8.0)
NFE
Method
4
5
6
7
10
(2M)
51.09
27.71
17.62
13.19
10.91
9.85
9.31
(PM-Solver+++33M)
51.09
38.08
26.30
18.33
13.52
11.35
9.97
UniPC-2-B11h)
80.46
49.10
29.91
19.70
14.38
11.69
10.20
(niPC-2-Bagh)
56.06
31.79
20.16
14.66
11.71
10.28
9.51
UniPC-3-B1(h)
80.46
54.00
38.67
29.35
22.06
16.74
13.66
UniPC-3-Ba(h)
56.06
43.13
34.37
27.45
20.42
16.00
13.01
Ours
33.84
19.06
13.00
10.31
9.72
9.06
9.06
Table 14: Full baseline results of text-to-image generation on MS-COC02014 validation set, 8.0)
NFE
Method
5
8
9
10
DPM-Solver-25
149.55
88.53
79.46
41.50
37.07
23.22
20.74
DPM-Solver-3S
161.03
156.72
106.15
75.28
58.54
39.26
29.54
(PM-Solver+++(MM)
24.95
20.59
19.07
18.17
17.84
17.53
17.31
(3M))))ver+++3M))
24.95
20.59
18.80
17.89
17.54
17.42
17.22
UniPC-2-B (h)
30.77
22.71
19.66
18.45
18.00
17.65
17.54
UniPC-2-B2(h)
25.49
20.65
19.16
18.13
17.84
17.65
17.52
-
26.39
21.30
19.05
17.83
17.58
17.50
17.53
Ours
22.76
16.84
15.76
14.77
14.23
13.99
14.01
G.5 TRANSFER ABILITY VALIDATION OF SOLVER SCHEDULES SEARCHED ON TEXT-TO-IMAGE TASK
To verify practicality 5 we use a different version model, SD-v1-3, to evaluate the solver sched- ules searched with SD-v1-5 model. Other settings are kept consistent with previous experiments in Sec. 55) We list the results in Tab. 21 We can see that the searched results can directly transfer to SD-vI-3 and achieve similar acceleration ratio with SD-v1-5 model, making more convenient to apply our searched results.
G.6 SEARCHED SOLVER SCHEDULES
We give some of our searched solver schedules on CIFAR-I0 here as examples. We list these
schedules in Tab. 22 and Tab. 23
H ADDITIONAL DERIVATIONS
In this section, we write the detailed derivation of Eq. 8 and Eq. 9 from Eq. and Eq. (5) correspondingly.
H.1 NOISE PREDICTION MODEL
Firstly, we use the variation of constant method to derive Eq. (6) Suppose the solution z(t) of Eq. can be written the product of two independent variables: 2xt) = u(t)v(t). Substitute it to Eq.
we get:
(2)
u(t)du+ u (F())(())
co d))
(25)
20,
29

---

Published as a conference paper at ICLR 2024
Table 15: FIDs of searched solver schedules and baseline methods under larger NFE budgets.
NFE
Dataset
Method
12
15
20
Baseline-W W
5.31
4.52
3.54
CIFAR-10
Baseline-B
3.67
3.03
2.80
Ours
2.65
2.41
2.41
Baseline-W W
6.12
4.20
3.56
CelebA
Baseline-B
3.82
2.72
2.70
Ours
2.32
2.06
2.06
Table 16: FIDs of the searched solver schedules with lower search cost.
NFE
Number of Schedule
Number of Image
4
5
8
10
1000
14.94
6,86
5.10
4.18
4.14
3.13
2.67
2000
500
11.50
6.94
5.11
4.40
3.95
3.20
2.90
250
12.55
7.84
6.26
4.77
3.46
3.13
3.13
1000
15,41
7.32
5.58
4.21
4.14
3.17
3.12
500
11.97
7.33
5.11
4.40
4.12
3.20
3.20
1500
250
13.74
8.04
6.11
4.85
3.46
3.46
3.33
1000
15.43
7.78
5.78
4.78
4.12
3.48
3.48
500
11.97
8.01
5.31
4.40
4.41
3.20
3,20
1000
250
15.42
8.11
6.11
4.84
3.46
3.46
3.33
Baseline-B
57.52
23.44
10.33
6.47
5.16
4.30
3.90
We let u()dv = (())((()((() to solve v(() and obtain v(() = vs))ed. s(r)dr s(r)ar where is an arbitrary
value larger than I. Then substitute the v(t) into Eq. 25
g²(()
vndu = 20, Eo (Ie, t)dt.
(26)
Then solve un through Eq. 26
ull) u(s) + J. (²(() (o( dr
(27)
20, u(T)
=u(s) + l (2()) co(or,7)
20, u(s)el 1(rddr dr
(28)
(29)
Finally, we write z(t) = u(t)u(t):
z(t) =(t)u(t)=v(s)) (ordar (u(s) + [ (2(T) color,T)
20, v(s)edi f(rjar dr)
(30)
=u(s)el f(r)dr u(s) + L' (2)
ca( T)el f(r)dr dr
(31)
20,
do? - -2 dlogga o²
=I(s)el; alogo,
dr
co( T)el dingar
(32)
20,
a(s) [s)) - -a(t)
a(t)
J.
or E dldo
ar
(33)
ar
a
a(l)
Jhv (2),
d
x(s) -a(1)
(34)
a(s)
30

---

Published as a conference paper at ICLR 2024
Table 17: FIDs of the searched solver schedules using the cascade search method.
NFE
Dataset
Method
4
8
9
10
Baseline-Ve (S))
255.21
288.12
32.15
14.79
22.99
6.41
5.97
Baseline-Wor ((M))
61.13
33,85
20.84
13.89
10.34
7.98
6.76
CIFAR-10
Baseline-B
57.52
23.44
10.33
6.47
5.16
4.30
3.90
Ours
11.50
6,86
5.18
3.81
3,41
3.02
2.69
Ours-Cascade
12.60
7.17
5.81
3.76
3.76
3.23
3.23
Table FIDs of the searched solver schedules in the search space containing more choice starting point.
NFE
Dataset
Method
4
5
6
7
10
Baseline-WGG
321.39
330.10
52.04
17.28
16.99
10.39
6.91
Baseline-Ve (M)) W(M)
31.27
20.37
14.18
11.16
9.28
8.00
7.11
CelebA
Baseline-B
26.32
8.38
6.72
6.72
5.17
4.21
4.02
Ours
12.31
5.17
3.65
3.80
3.62
3.16
2.73
Ours-SP
10.76
7.30
4.26
4.09
3.81
3.48
2.39
Miseline-W(SS
364.60
366.66
72.47
47.84
54.21
28.22
27.99
Baseline-W ((M)
93.98
69.08
50.35
40.99
34,80
30.56
27.96
ImageNet-64
Baseline-B
76.69
61.73
42.81
31.76
26.99
23.89
24.23
Ours
33.84
24.95
22.31
19.55
19.19
19.09
16.68
Ours-SP
34.31
28.03
23.31
19.92
18.34
18.34
17.75
Then we get the exponential integral in Eq. <x(r,, A) with Taylor expansion 60 A)
16 According to Asm.
A.2
we further write the
5D (Ix,, Xs) x-) (n M) and substitute
(k)
into Eq. 6
Ae - (2 \a)*
Ft E. at
(n)
(xx,)da)
AA + OO(A 22))
(35)
a
k!
k=0
01
T
(n)
(I),
de (A--ee)
() - >a)*
dd + O((): - -1,)tt2.
(36)
0s
hd
k=0
Wto Sane e (2- (*-2,)* dd (A: 2 0). Use partial integral
Let h = >; Am. and suppose pk++1 to calculate i
h
(2-2 (1 (k)
+ Sx.
(k: 1 dl)
(37)
M++
h!
vt (h) 1/k!
(38)
h
The initial value sof (h) can be easily computed: (i(h) S e e ch
And
further give (h) d through ip 11/0 By substituting Pi Eq. Eq. 8
36 we obtain
H.2
DATA PREDICTION MODEL
The derivation with data prediction model is very similar to that with noise prediction model. There- fore, omit some derivation in this section.
31

---

Published as a conference paper at ICLR 2024
Table 19: FIDs of the searched solver schedules in the search space containing continuous interpo- lation ratio of prediction types.
NFE
Dataset
Method
4
5
6
7
8
10
Baseline-W(C
255.21
288.12
32.15
14.79
22.99
6.41
5.97
Bas)line-W(M)
61.13
33.85
20.84
13.89
10.34
7.98
6.76
CIFAR-10
Baseline-B
57.52
23.44
10.33
6.47
5.16
4.30
3.90
Ours
11.50
6.86
5.18
3.81
3.41
3.02
2.69
Ours-Cascade
12.60
7.17
5.81
3.76
3.76
3.23
3.23
Ours-IP
12.47
6.92
4.90
3.77
3.77
3.27
3.27
Table 20: FIDs of the searched solver schedules in the search space containing guidance scale.
NFE
Guidance Scale
Method
4
5
6
8
9
10
Baseline-W(S ((())
161.03
156.72
106.15
75.28
58.54
39.26
29.54
Baseline- W(M)
30.77
22.71
19.66
18.45
18.00
17.65
17.54
7.5
Baseline-B
24.95
20.59
18.80
17.83
17.54
17.42
17.22
Ours
22.76
16.84
15.76
14.77
14.23
13.99
14.01
DPM-Solver+++3M)
24.64
17.59
15.21
14.24
13.77
13.37
13.13
3.0
-
22.01
16.60
14.88
14.14
13.73
13.42
13.20
Searchable
Ours-GS 15.63
13.94
13.90
13.69
12.81
12.56
12.34
Firstly, by using the variation of constant method in App. with data prediction model:
H.1
we can get the exponential integral
Tt 0.
) + oo
Lr e (eo(zx,))dl
(39)
Then replace the (x A) with its Taylor expansion:
(1 Ao
Ft
R.
(IX,, 2o)
ed
k! dd
(40)
k=0
de
Ad Aa) dx.
0t
Tn
(k)
(I), 2o)
(41)
0x
k=0
Let = d Ao. and suppose sh+t1 calculate vh+1i
Sal
ed-
(1-2,1* k!
dx (k 2 0). Use partial integral to
hETl
A- () (k-1) dl
(42)
MK+1 1/k! (h)
K!
(k - 1)!
22
(43)
h
The initial value upf (h) can be easily computed: ( = 1rv cd 1-e-h
And we further
give (h) = - through yp (b)) By substituting o to Eq. (41 we obtain Eq.
I DISCUSSION OF CONVERGENCE
In this section, we discuss the convergence order USF. We give the convergence bound at Thmm [.
32

---

Published as a conference paper at ICLR 2024
Table 21: Results on MS-COCO 256x256 with SD-VI-3 model.
NFF
Method
4
5
7
10
U
24.50
21.09
19.04
18.06
17.71
17.64
17.64
(3M))
24.50
20.58
18.83
18.25
17.75
17.56
17.43
Ours
22.49
16.74
15.81
14.84
14.42
14.29
14.40
Table 22: Searched schedule under 4 and 7 NFE budget on CIFAR-10. "D" and "N" in the row of "Prediction Type" stand for "data prediction" and "noise prediction". "T" and "F" in the row of "Corrector" stand for using and not using corrector. The same for "T" and "F" in the row of "Low Order Estimation".
NFE
Timestep
1.000.7,0022,00100....33
1.0079..6.0-20005500000000......
Order Prediction Type
1,1,3,2
1,2.3.2.3.2.2
D.D.N,D
D.D.D.N.NDDDD T.T.TFFFFFF.... -, Rg Ro R -.F.FFF.
Corrector
T.T.F.F
Scaling
R.,R,
Low Order Estimation
--
Theorem I.1. Suppose Ft, is the ground truth trajectory value, and the estimated value 3t, (h) is given by the step update in Alg. 7 where h = X1, X,,, is the step length w.r.t. the logSNR. Then the convergence order of &z, (h) h is pi min(n; where Is, is the convergence order of the starting point and dę is the convergence order of the k-th derivative estimation E (It., tx.) or 21hh (It,, tt.) h.
lag d1 + 2, d2 + 3..... dn--1 Tt)),
The proof is very simple by substituting the starting point It., It., O(hl.)) and the estimation
of all derivatives =(k) (It,,. (1,)) to (1) (It., (x2) + 01142) into Eq. (8) (the same for data prediction model). As result, the convergence order of USF decided by The Taylor expansion order nis accuracy of the starting point and the convergence orders of all derivative estimations.
As discussed in App. B when no low-order derivative estimation is used and the additional involved points have convergence orders not smaller than ni, the convergence order of the k-th derivative estimation n k, and the scaling operation doesn't 't change the convergence order of the derivative estimation. Therefore, the convergence order p, of 2,, (h) equals n. + 1. When low-order estimation used, the convergence order p; will decreases. It worth noting that the there is no strong relation with the convergence order and the actual truncation error. As shown in Fig. ? using low-order derivative estimation certain timesteps can achieve lower error with the ground truth value. This because the actual truncation error is not smooth and using high-order information for low-order derivative may introduce additional bias. Therefore, we apply low-order Taylor Taylor-difference method for derivative estimation in our experiments.
QUALITATIVE RESULTS
We provide some samples generated by our searched solver schedules and baseline methods in this section.
33

---

Published as a conference paper at ICLR 2024
Table 23: Searched schedule under NFE budget on CIFAR-IO. "D" and "N" in the row of "redic-- tion Type" stand for "data prediction" and "noise prediction". "T" and "F" in the row of "Corrector" stand for using and not using corrector. The same for "T" and "F" in the row "Low Order Esti-
mation".
NFE
Timestep Order Prediction Type
10.7605774402203001160770...2..3
1,2.3,2.33333,133
D.N.D.D.D.D.D.D..D
Corrector
I.T.TTTTFFT.
Scaling
Low Order Estimation
T.T.T.T,,,T.FFFFFFF
(a) Samples generated by the searched schedule Figure Samples of CIFAR-I0 dataset with NFE=4.
(b) Samples generated by baseline-B method
(a) Samples generated by searched schedule
(b) Samples generated by baseline-B method
Figure 6: Samples of CIFAR-10 dataset with NFE=5.
34

---

Published as a conference paper at ICLR 2024
(a) Samples generated by the searched schedulc
(b) Samples generated by baseline-B method
Figure 7: Samples of CelebA dataset with NFE=4.
(a) Samples generated by the searched schedule
(b) Samples generated by baseline-B method
Figure Samples of -SUN-Bedroom dataset with NFE=4.
(a) Samples generated by the searched schedule
(b) Samples generated by baseline-B - method
Figure 9: Samples of -SUN-Bedroom dataset with NFE=5.
35

---

Published as a conference paper at ICLR 2024
(a) Samples generated by the searched schedulc
(b) Samples generated by baseline-B method
Figure 10: Samples of ImageNet-66 64 dataset with NFE=4.
(a) Samples generated by the searched schedule
(b) Samples generated by baseline-B method
Figure 11: Samples of ImageNet-66 64 dataset with NFE=5.
(a) Samples generated by searched schedule
(b) Samples generated by baseline-B method
Figure 12: Samples of ImageNet- 128 dataset with NFE=4.
36

---

Published as a conference paper at ICLR 2024
(a) Samples generated by the searched schedulc
(b) Samples generated by baseline-B method
Figure 13: Samples of ImageNet- 128 dataset with NFE=5.
(a) Samples generated by the searched schedule
(b) Samples generated by baseline-B method
Figure 14: Samples of ImageNet- 128 dataset with NFE=7.
(a) Samples generated by the searched schedule
(b) Samples generated by baseline-B method
Figure 15: Samples of ImageNet-256 dataset with NFE=4.
37

---

Published as a conference paper at ICLR 2024
(a) Samples generated by searched schedule
(b) Samples generated by baseline-B method
Figure 16: Samples of ImageNet-256 dataset with NFE=5.
(a) Samples generated by the searched schedule
(b) Samples generated by baseline-B method
Figure 17: Samples guided by the prompt 'A person on a motorcycle high the air' with NFE=5.
(a) Samples generated by searched schedule
(b) Samples generated by baseline-B B method
Figure 18: Samples guided by the prompt 'A fire hydrant sitting on the side of a city street" with NFE=5.
38

---

Published as a conference paper at ICLR 2024
(a) Samples generated by the searched schedule
(b) Samples generated by baseline-B B method
Figure 19: Samples guided by the prompt "A bus yard filled with yellow school buses parked side by side' with NFE=5.
39


Reviews:
[{"review_id": "lM5Hr1QuXX", "reviewer": "Reviewer_HHFa", "summary": "This paper proposes a framework to unify the existing Diffusion Probalisitc Model (DPM) solvers such as DPM-Solver, DPM-Solver++, UniPC and DESI. Under the framework, these methods can be represented with different options on some strategies and hyperparameters.\nThis paper shows the phenomenon that different solver strategies at different timesteps may yield different performances. This paper is inspired to design a schedule between different solving strategies during the sampling based on the unified framework. In this paper, the predicator-based method is proposed to schedule the solvers, which outperforms the SOTA solvers on several conditional datasets.", "questions": "1. Can you provide an intuitive explanation of the phenomenon observed in the paper? \n2. Can you add the experiment results on larger NFEs? Since the search method also has the time consumption, can you provide some experiment results on it?\n3. Can you give a brief explanation of the meaning and effect of multi-stage in Algorithm 2?\n4. Since the USF can include the method DEIS in the domain $t$, why do you not include this method as a baseline in experiments?\n\nHere are some typos in App. D. In analyzing the current solvers, the index of $\\tilde{x}_{i-1}$ may be incorrect.\n\nThey should be $t_{i-1}$ in DPM-Solver-2, DPM-Solver++(2S) and DPM-Solver++(2M). \n\nWhen using the Assumption in the Appendix, it may not be Thm A.1, A.2 etc.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. USF: This paper unifies the existing DPM solvers under a framework with detailed derivations and analysis, which provides a new perspective to review the DPM solvers and improve their performance in real applications.\n2. $S^3$: This paper proposes the $S^3$ method which uses light predictors to search the solver schedule which demonstrates good performance on unconditional datasets with limited NEF.\n3. Empirical Results: This paper has plentiful empirical studies. It demonstrates the situation that there is no single best solver under different timesteps. It also shows good performance compared with baselines and ablation studies on the consumption of generated images.", "weaknesses": "1. This paper illustrates the phenomenon that the suitable strategies vary among timesteps but has no further explanation on why this phenomenon happens. The proposed method paper may lack motivation.\n2. This paper may lack novelty and contributions in the sense that it does not provide a further understanding of why suitable solvers are different on different timesteps. \n3. On empirical results, this paper may lack the results when the NFE is larger than 10. The gap between the baselines and the proposed method may decrease with larger NFE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bnh16p5FRN", "reviewer": "Reviewer_VK39", "summary": "In this paper, the authors propose a sampling framework for diffusion models for the systematic study of solving strategies for diffusion models. Specifically, the authors explore the solver schedule in the following aspects: timestep discretization, prediction type, starting point, solver order, derivative estimation, and corrector usage. Experiment results show that the proposed method boosts the performance, especially in the regime of very few NFEs.", "questions": "Does the predictor model need to be trained separately on different NFEs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "$\\cdot$ This paper combines the AutoML methods into the diffusion sampling procedure. The authors explore the search space over timestep discretization, prediction type, starting point, solver order, derivative estimation, and corrector usage.\n\n$\\cdot$ The experiment results are appealing, especially in a few NFEs regime. The proposed method overperforms all baseline methods.\n\n$\\cdot$ The paper is overall well-written and the structure is clear.", "weaknesses": "$\\cdot$ The experiment part seems to lack exact data on the computational cost of the overall pipeline of proposed methods. The overall pipeline needs iterative sampling images under a given sampling configuration, evaluating the performance and training of the proposed predictor models.\n\n$\\cdot$ The content of section 4.2 is not well-organized and needs further explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1hM3WsoQNQ", "reviewer": "Reviewer_7SgP", "summary": "In this paper, the authors proposed a predictor-based search method that optimizes the solver schedule to get a better time-quality trade-off of sampling. It is based on a unified sampling framework (USF) to study the optimal strategies for solver. Therefore, it enables us to take different solving strategies at different time steps, and according to the authors' study, it has a considerable potential to improve the sample quality and model efficiency. Finally, the authors used extensive experiments to validate the method of a large number of datasets such as CelebA, Cifar-10, LSUN, ImageNet and so on. Also, the authors applied S^3 to Stable-Diffusion models, and achieve 2x acceleration without losing the performance on text-to-image generation task on MS-COCO 256x256 dataset.", "questions": "1. Can you briefly tell me the differences between this paper and Lu et al's DPM-solver++, since both of them seem to use different strategies as multi-step solver? What kind of improvements have you done in your paper compared with DPM Solver++?\n2. If time is available for you, maybe you can follow my suggestions on writing in the previous section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The reference list of this paper is very complete and the authors introduced their differences and connections quite clearly.\n2. The biggest strength of this paper is that, the experiments are really solid for me.", "weaknesses": "1. The only weakness for me is on the writing style:\n(a) In Equation (1), dt and dw_t should be \\mathrm{d} t and \\mathrm{d}w_t. Similarly, in other parts of this paper, the authors sometimes use dt and sometimes use \\mathrm{d} t. The notation should be unified. \n(b) In Equation (5), the \\mathrm{d} t is missing in the right hand side of the equation. \n(c) For the Table 1, I suggest you using \\bigstrut before each line to make the text shown in the middle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHVhkn5Ce8", "reviewer": "Reviewer_HfC4", "summary": "To further improve the sample quality with less than 10 NFE, this paper proposed a unified sampling framework (USF) to study the optional strategies for solver. Specifically, USF splits the solving process of one step into independent decisions of several components, which also reveal that an appropriate solver schedules will benefit the quality and efficiency of diffusion models. Moreover, for each component, $\\mathcal{S} ^{3}$ is proposed to search for optimal solver schedules in each time step automatically. Based on this framework, USF enables to achieve significant performance on various benchmark datasets when implemented to various diffusion models.", "questions": "The same as Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.USF improves the sample quality with less than 10 NFE, which greatly benefits diffusion models in practical application. It unifies previous fast training-free samplers in every independent sampling process, which is very flexible and extensible.\n\n2.The generated images are comparable to previous SOTA fast sampling method in various time step, which demonstrates the superior of USF.\n\n3.In my humble opinion, the mathematical analysis is rigorous and can support the improvements on the experiment results.", "weaknesses": "1.The caption of Table 3 should be improved, it’s hard to figure our the meaning.\n\n2.The search budget should be analysed more since it is very important to implement to previous diffusion models.\n\n3.Why not demonstrated more categories of generated images when NFE is less? A fuller qualitative analysis would add strength on the USF.\n\n4.How about the reproducibility of USF? Can you share the code? The open source code will help researchers continue to improve slow sampling speed since USF has great performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HMcW6fIXPz", "reviewer": "Reviewer_qhfh", "summary": "The paper presents an AutoML-style experimental setup for searching optimal Diffusion Model sampler (solver). The authors, first noted the various “moving parts” of modern reverse process solver and then applied an evolution-based search technique to explore these while keeping track of FID under an NFE budget. Authors argued and validated that a non-uniform combination of these design choices in different timesteps is the key to reach optimal “solver schedule”. A lot of experiments are conducted to search through this space of solver schedules. In fact, they also proposes a special evolutionary search method S3 that does it more efficiently.", "questions": "Here are some concrete questions I’d like the authors to answer.\n\n- In high-level, it is not clear to me which part of the paper is being claimed to be the primary contribution — the search algorithm S3 itself, or the optimal solver schedules and corresponding FID numbers ? Is the paper supposed to be about the “search algorithm S3” or an empirical AutoML report ?\n- If your focus is S3, there is surprisingly little written about it in the main paper. Almost all details are in supplementary. So that indicates it to be just an AutoML experimental report — is it ?\n- Outright confession: I am not at all experienced in AutoML. So, is this S3 something novel or it is a usual practice in AutoML ? I don’t see any proper citation in case this has been adopted from existing literature/libraries.\n- What exactly the paper “concludes” at the end the day — did you find any “pattern” in the optimal solver schedules that may motivate further work? I don’t see the exact configuration of the optimal solver schedules (found by S3) — are they written anywhere? You can certainly report the ones with low NFE. This is important for verifying them later on by others.\n\nCouple of high-level points are unclear to me; so I am now going with a “below threshold” rating. I am open to increasing it depending on authors response, if satisfactory.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "As far as I can tell, there hasn’t been such an effort to use AutoML setup for searching for better solver schedule — so it is indeed novel in some sense. Also, the large amount of experiments have been conducted, which can also be quite helpful for the community in general. The presentation/writing quality is also pretty good.", "weaknesses": "Here are some concerns I’d like to emphasize:\n\n- The major weakness I see is that the paper can be called “just a search” without any solid conclusion. While I agree that figuring out good FID numbers for existing benchmarks is pretty good, it is not usually preffered to lock the contribution to very specific models/dataset/solver. If a new dataset/model/solver comes out in future, one need to perform the entire search again.\n- The USF isn’t really something new. It is just a “dumb” aggregation of a bunch of design options. It is only based on empirical evidence — but I wonder if it can be theoretically justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)