GENIE: Higher-Order Denoising Diffusion Solvers
Authors: Tim Dockhorn
Arash Vahdat
Karsten Kreis
Venue: NeurIPS 2022 Accept (2022)
Keywords: Diffusion Models
Score-based Generative Models
Generative Learning
ODE Solvers
Higher-Order Solvers

Abstract: Denoising diffusion models (DDMs) have emerged as a powerful class of generative models. A forward diffusion process slowly perturbs the data, while a deep model learns to gradually denoise. Synthesis amounts to solving a differential equation (DE) defined by the learnt model. Solving the DE requires slow iterative solvers for high-quality generation. In this work, we propose Higher-Order Denoising Diffusion Solvers (GENIE): Based on truncated Taylor methods, we derive a novel higher-order solver that significantly accelerates synthesis. Our solver relies on higher-order gradients of the perturbed data distribution, that is, higher-order score functions. In practice, only Jacobian-vector products (JVPs) are required and we propose to extract them from the first-order score network via automatic differentiation. We then distill the JVPs into a separate neural network that allows us to efficiently compute the necessary higher-order terms for our novel sampler during synthesis. We only need to train a small additional head on top of the first-order score network. We validate GENIE on multiple image generation benchmarks and demonstrate that GENIE outperforms all previous solvers. Unlike recent methods that fundamentally alter the generation process in DDMs, our GENIE solves the true generative DE and still enables applications such as encoding and guided sampling. Project page and code: https://nv-tlabs.github.io/GENIE.

GENIE: Higher-Order Denoising Diffusion Solvers
Tim Dockhorn 1.2.3,
Arash Vahdat
Karsten Kreis!
INVIDIA University of Waterloo
Vector Institute
wwb.boochorrsuuaeeroooo... Kkveid
Abstract
Denoising diffusion models (DDMs) have emerged powerful class of generative models. forward diffusion process slowly perturbs the data, while a deep model learns to gradually denoise. Synthesis amounts solving differential equation (DE) defined by the learnt model. Solving the DE requires slow iterative solvers for high-quality generation. In this work, we propose Higher-Order Denois- ing Diffiusion Solvers (GENIE): Based on truncated Taylor methods, we derive a novel higher-order solver that significantly accelerates synthesis. Our solver relies on higher-order gradients of the perturbed data distribution, that is, higher-order score functions. In practice, only Jacobian-vector products (JVPs) are required and we propose to extract them from the first-order score network via automatic differ- entiation. We then distill the JVVs into a separate neural network that allows us efficiently compute the necessary higher-order terms for our novel sampler during synthesis. We only need to train a small additional head on top of the first-order score network. We validate GENIE on multiple image generation benchmarks and demonstrate that GENIE outperforms all previous solvers. Unlike recent methods that fundamentally alter the generation process in DDMs, our GENIE solves the true generative DE and still enables applications such as encoding and guided sampling. Project page and code: https: http///vttllleee tlabs. github. io/GENIE.
Introduction
Denoising diffusion models (DDMs) offer both state-of-the-art synthesis quality and sample diversity in combination with robust and scalable learning objective. DDMs have been used for image [1-5]
and video |6, 7] synthesis, super-resolution [8, 9], deblurring [10, 11], image editing and inpainting [5,
12-14], text-to-image synthesis [15-17], conditional and semantic image generation [18-22], image- to-image translation [14, 23, 24] and for inverse problems in medical imaging [25-31]. They also enable high-quality speech synthesis [32-37], 3D shape generation [38-42], molecular modeling [43- 46), maximum likelihood training [47-50], and more (51-56) In DDMs, a diffusion process gradually perturbs the data towards random noise, while deep neural network learns to denoise. Formally, the problem reduces to learning the score funcctio,, i.e., the gradient of the log density of the perturbed data. The (approximate) inverse of the forward diffusion can be described by an ordinary or stochastic differential equation (ODE or SDE, respectively). defined by the learned score function, and can therefore be used for generation when starting from random noise [47, 57].
A crucial drawback of DDMs is that the generative ODE or SDE is typically difficult to solve, due the complex score function. Therefore, efficient and tailored samplers are required for fast synthesis. In this work, building on the generative ODE [47, 57, 58], we rigorously derive novel second-order ODE solver using truncated Taylor methods [59]. These higher-order methods require higher-order gradients of the ODE in our case this includes higher-order gradients of the
Work done during internship at NVIDIA.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

First truncated Taylor method (lineur extrapolation)
Second truncuted Taylor method (quadratic extrapolation)
Po(X5)
Probubility Flow ODE trajecturies
(1(x))
Figure 1: Our novel Higher- Higherr-Ordrr Order Denoising Diffusion Solver (GENIE) relies on the second truncated Taylor method (TTM) to simulate (re-parae p Probability Flow ODE for sampling from denoising diffusion mod- els. The second TTM captures the local curvature the ODE's gradient field and enables more accurate extrapola- tion and larger step sizes than the first TTM (Euler's method), which previous methods such as DDIM [58] utilize.
log-density of the perturbed data, i.e., higher-order score functions. Because such higher-order scores usually not available, existing works typically use simple first-order solvers samplers with low accuracy [1, 57, 58, 60], higher-order methods that rely on suboptimal finite difference or other approximations [61-63], or alternative approaches [64-66] for accelerated sampling. Here, we fundamentally avoid such approximations and directly model the higher-order gradient terms: Importantly, our novel Higher-Order Denoising Diffusion Solver (GENIE) relies Jacobian-vector products (JVPs) involving second-order scores. We propose to calculate these JVPs by automatic differentiation of the regular learnt first-order scores. For computational efficiency, we then distill the entire higher-order gradient of the ODE, including the JVPs, into a separate neural network. In practice, we only need to add small head the first-order score network predict the components of the higher-order ODE gradient. By directly modeling the JVPs we avoid explicitly forming nigh-dimensional higher-order scores. Intuitively, the higher-order terms in GENIE capture the local curvature of the ODE and enable larger steps when iteratively solving the generative ODE (Fig. I).
Experimentally, we validate GENIE multiple image modeling benchmarks and achieve state-of- the-art performance in solving the generative ODE of DDMs with few synthesis steps. In contrast to recent methods that fundamentally modify the generation process of DDMs by training conditional GANS [67] or by distilling the full sampling trajectory [68, 69], GENIE solves the true generative ODE. Therefore, also show that can still encode images in the DDM's latent space, as required for instance for image interpolation, and use techniques such as guided sampling [4, 57, 70].
We make the following contributions: (i) We introduce GENIE, novel second-order ODE solver for fast DDM sampling. (ii) We propose to extract the required higher-order terms from the first-order score model by automatic diffcrentiation. In contrast to existing works, explicitly work with higher-order scores without finite difference approximations. To the best of our knowledge, GENIE is the first method that explicitly uses higher-order scores for generative modeling with DDMs. (iii) We propose to directly model the necessary JVPs and distill them into a small neural network. (iv) We outperform all previous solvers and samplers for the generative differential equations of DDMs.
2 Background
We consider continuous-time DDMs [1, 57, 71] whose forward process can be described by (xo) (xx; 00xX0,
where Xy (x()) is drawn from the empirical data distribution and x; refers to diffused data samples at time t E [0, 1] along the diffusion process. The functions a and a, are generally chosen such that the logarithmic signal-to-noise ratio [48] log of decreases monotonically with t and the data of diffuses towards random noise, i.e., P1 (i(xi) (x Nx O, I). We use variance-preserving [57] diffusion processes for which of = 1 - af (however, all methods introduced in this work are applicable to more general DDMs). The diffusion process can then be expressed by the (variance-preservvng)) SDE
dxt = Bix, dt + / B, dw.,
(2)
where 8, d log of Xg Po (xo) and w. is standard Wiener process. corresponding reverse diffusion process that effectively inverts the forward diffusion is given by [57, 72, 73]
di
dx, f [x; + 2V. x, log (x.)) dt + dB,
(3)
and this reverse-time generative SDE is marginally equivalent to the generative ODE [47, 57]
dx, 3t [x, + Vx, logpp(x;)]
(4)

---

where Vx, Po Pa (x+) is the score fimetion. Eq. (4) is referred to as the Probability Flow ODE [57], an instance of continuous Normalizing flows [74, 75]. To generate samples from the DDM, one can sample ~ (x1 0, 7) and numerically simulate either the Probability Flow ODE or the generative SDE, replacing the unknown score function by a learned score model s0 8x(x,, 1) R Vx, log (x+).
The DDIM solver [58] has been particularly popular to simulate DDMs due to its speed and simplicity. has been shown that DDIM is Euler's 's method applied to ODE based on re-parameterization of
1-aa and X₁ x
the Probability Flow ODE [58, 69]: Defining "t-
+ 7p,
we have
dx
V
dxp dt di d' + X,
7
Xx, log (xx))(
(5)
dy
VI+ + 7
VI+ 7
where we inserted Eq. (4) for dx and used B(t) a parameterization of the score model, the approximate generative DDIM ODE is then given by
di 27d
Letting sø(x,, t) = co(x..)) denote
de 7/+1'
dx. = Eo (x t) d'y.
(6)
(see App. A for more detailed derivation of Eq. (6)). The where we used o V
model Eø (xx t) can be learned by minimizing the score matching objective [1, 76]
min Ex-di(teeutos xxxxxx
[o(t)]le cx,,,, t)1[3] x 0x + oeC,
for small 0 < feutoff < 1. As is standard practice, set g(t) 1. Other weighting functions g(t) are possible; for example, setting g(t)
8e
recovers maximum likelihood learning [47-50].
20
3 Higher-Order Denoising Diffusion Solver
As discussed Sec. 2, the so-known DDIM solver [58] simply Euler's method applied the DDIM ODE (cf. Eq. (6)). In this work, we apply higher-order method to the DDIM ODE, building on the truncated Taylor method (TTM) [59]. The p-th TTM is simply the p-th order Taylor polynomial applied to an ODE, For example, for the general dy
f(y, t), the p-th TTM reads as
de
dy
h d'y p!
1
Vt.11 Yt.. + fnn dt
+
dtp (n))
(8)
where ft, = fn+1 bn (see App. B.1 for a truncation error analysis with respect to the exact ODE solution). Note that the first TTM is simply Buler's method. Applying the second TTM to the DDIM ODE results in the following scheme:
2 h deg
Xin+1 = -in + E tn) +
dri (x),)
(9)
where hn = Maass Ye. Recall that 7/- = V'-aa" where the function is a time-dependent
dea can be decomposed as follows hyperparameter of the DDM. The total derivative Ex d%
co Deo(Xi, 1) dx: 0x, dr Øt dj1²
deo(xi, 1) dt
+
(10)
where (xe(x..t) tx;
denotes the Jacobian of co t) and
dx: Ox, dx:
Ix.
7t. 1 1+ X
+
(x,,
(11)
dy Ox drys Oy
V 7
If not explicitly stated otherwise, we refer to the second TTM applied to the DDIM ODE, i.c., the scheme in Egq. (9), Higher-Order Denoising Diffusion Solver (GENIE). Intuitively, the higher-order gradient terms used the second TMM model the local curvature of the ODE. This translates into a Taylor formula-based extrapolation that is quadratic in time (cf. Eqs. (8) and (9)) and more accurate than linear extrapolation, as in Euler's method, thereby enabling larger time steps (see Fig. for visualization). In App. B, also discuss the application of the third TTM the DDIM ODE. We emphasize that TTMs are not restricted to the DDIM ODE and could just as well be applied to the Probability Flow ODE [57] (also see App. B) or neural ODEs [74, 75] more generally.
3

---

(a) Ground truth
(h) DDIM
(c) GENTE
Figure 2: Modeling complex 2D toy distribution: Samples in (b) and (c) are generated via DDIM and GENIE,
respectively, with 25 solver steps using the analytical score function of the ground truth distribution.
The Benefit of Higher-Order Methods: We showcase the benefit of higher-order methods on 2D toy distribution (Fig. 2a) for which we know the score function as well as all higher-order derivatives necessary for GENIE analytically. We generate Ik different accu- rate "ground truth" trajectories using DDIM with 10k steps. We compare these "ground truth" trajectories to single steps of DDIM and GENIE for varying step sizes At. We then measure the mean Ly--distance of the single steps & (Att) to the "ground truth" trajec- tories Xg, and repeat this experiment for three starting points t E {0.1, 0.2, 0.5]. We see (Fig. (top)) that GENIR can use larger step sizes to stay within certain error tolerance for all starting points f. We further show samples for DDIM and GENIE, using 25 solver steps, in Fig. DDIM has the undesired behavior of sampling low-density regions between modes, whereas GENIE looks like a slightly noisy version of the ground truth distribution (Fig. 2a).
GENIK
CEXIR 2000
DCOL
0.312
0.0
0.01
0.00
6.01
Bry
Figure 3: Top: Single step using analytical score function. Bottom: Norm of difference (A(tt) between analytical and approximate derivative computed via finite difference method.
Comparison to Multistep Methods: Linear multistep methods are an alternative higher-order method to solve ODEs. Liu et al. [63] applied the well-established Adams-Bashforth [AB, 77] method to the DDIM ODE. AB methods can be derived from TTMs by ap-
proximating higher-order derivatives d'y dEP using the finite difference method [78]. For example, the second AB method is obtained from the second TTM by replacing dy de with the first-order forward difference approximation (f(y,,, (n)) f(y fn-1)) 11) h/ In Fig. 3 (bottom), we visualize the mean L2-norm of the difference Er(tt) between the analytical derivative dyy Eg and its first-order forward difference approximation for varying step sizes At for the 2D toy distribution. The approximation is especially poor at small & for which the score function becomes complex (App. for details on all toy experiments).
3.1 Learning Higher-Orde Order Derivatives
The above observations inspire to apply GENIE to DDMs of more complex and high-dimensional data such as images. Regular DDMs learn a model eg for the first-order score; however, the higher-order gradient terms required for GENIE (cf. Eq. (10)) are not immediately available to us, unlike in the
toy example above. Let us insert Eq. (11) into Eq. (10) and analyze the required terms more closely:
1
Oco (x,, t)
Yt
(co(
Oco (x,,)),)) t) dt
dy tx), = VTi
cØ (X1, t)
X. +
(12)
Ox:
1
Ox.
Ōt
dyc
JVP,
JVP,
We see that the full derivative decomposes into two JVP terms and one simpler time derivative term. Deo (x..) plays erucial role in Eq. (12). It can be expressed
The term
dco(x;,t) t) dx, -0+ 0x.
(x;,)) se - Vx. log (x:),
(13)
which means that GENIE relies second-order score functions V Vx. log under the hood.
Given a DDM, that is, given 6g, we could compute the derivative d,, eg for the GENIE scheme in Eq. (9) using automatic differentiation (AD). This would, however, make a single step of GENIE at least twice as costly as DDIM, because we would need a forward pass through the eg network com- pute Eo (x t) itself, and another pass to compute the JVPs and the time derivative in Eq. (12). These

---

forward passes cannot be parallelized, since the vector part of 1VP; in Eq. (12) involves itself, and needs to be known before computing the JVP. To accelerate sampling, this overhead is too expensive.
Gradient Distillation: To avoid this overhead, we propose to first distill d,, Eg into a separate neural network. During distillation training, we can use the slow AD-based calculation of d,.ce, Eg. but during synthesis we call the trained neural network. We build on the observation that the internal representations of the neural network modeling (in our case a U-Net [79] architecture) can be used for downstream tasks [80, 81]: specifically, we provide the last feature layer from the Eg network together with its time embedding as well as x, and the output Eo (x(x,,) () to a small prediction head ky (xx,,) t) that models the different terms in Eq. (12) (see Fig. 4). The overhead generated by by is small, for instance less than 2% for our CIFAR-10 10 model (also see Sec. 5), and we found this approach to provide excellent performance. Note that in principle we could also train an independent deep neural network, which does not make use of the internal representations of eg and could therefore theoretically be run in parallel to the Eg model. We justify using small prediction heads over independent neural networks because AD-based distillation training is slow: in each training iteration we first need to call the ég network, then calculate the JVP terms, and only then can we call the distillation model. By modeling d Eg via small prediction heads, while reusing the internal representation of the score model, we can make training relatively fast: we only need to train kyp for up to 50k iterations. In contrast, training score models from scratch takes roughly an order of magnitude more iterations. We leave training of independent networks to predict dyy Eg to future work.
+ colo,, f)-
(x,,
Figure 4: Our distilled model ky that predicts the gradient dyy Eg is implemented as small additional output head on top of the first-order score model co. Purple layers are used both in ce and kw; green layers are specific for cæ and kyp.
Mixed Network Parameterization: We found that learning d.,, Eg directly as single output of neural network can be challenging. Assuming a single data point distribution po (x() = 8(x0- = O), for which we know the diffused score function and all higher-order derivatives analytically,
found that the terms in Eq. (12) all behave very differently within the t E [0, 1] interval (for instance, the prefactor of JVP, in Eq. (12) approaches 1 as + 0, while JVP2's prefactor vanishes). As outlined in detail in App. C.2.3, this simple single data point assumption implies an effective mixed network parameterization, an approach inspired by the "mixed score parametrizations" in Vahdat et al. [49] and Dockhorn et al. [60]. In particular, we model
k
7t
1 (1 + 72)
(3) +
ss dr. EB.
(14)
Y'
1
where k (xe, t), E {1, 3). are different output channels of the neural network (i.e. the additional head on top of the cg network). The three terms in Eq. (14) exactly correspond to the three terms of Eq. (12), in the same order. We show the superior performance of this parametrization in Sec. 5.3. Learning Objective: Ideally, we would like our model Ky to match d Eo exactly, for all E [o. T] and x, in the diffused data distribution, which the generative ODE trajectories traverse. This suggests a simple (weighted) Lz-llos,, similar to regular score matching losses for DDMs [1, 57]:
min Bx-2((tonoot ex. [9a(t)] (arxo + ore, () dry c0 + ote,, 1)22 (15) +
for diffused data points axXo + a,E and ga(t) 72 to counteract the 1/7; in the first and third terms of Eq. (14). This leads to roughly constant loss over different time values t. During training we compute dg via AD; however, inference time we use the learned prediction head by to approximate d', 6g. In App. C.2.4, we provide pseudo code for training and sampling with heads kg. Note that our distillation objective is consistent and principled: if k. matches d,, en exactly, the resulting GENIE algorithm recovers the second TTM exactly (extended discussion in App. B.4).
Alternative Learning Approaches: As shown in Eq. (13), GENIE relies on second-orde -ooorr score functions. Recently, Meng et al. [82] directly learnt such higher-order scores with higher-order score matching objectives. Directly applying these techniques has the downside that we would need to ex- plicitly form the higher-order score terms V (x,,, l). which are very high-dimensional for data such as images. Low-rank approximations are possible, but potentially insufficient for high performance. In approach, we avoiding this complication by directly modeling the nower-dimensions JVPs. We
5

---

found that the methods from Meng et al. [82] can be modified to provide higher-order score matching objectives for the JVP terms required for GENIE and we briefly explored this (see App. D). However, our distillation approach with AlD-aseed higher-order gradients worked much better. Nevertheless, this is an interesting direction for future research. To the best of our knowledge, GENTE is the first solver for the generative differential equations of DDMs that directly uses higher-order scores (in the form of the distilled JVPs) for generative modeling without finite difference or other approximations.
4 Related Work
Accelerated Sampling from DDMs. Several previous works address the slow sampling of DDMs:
One line of work reduces and readjusts the timesteps [3, 64] used in dime-discretized DDMs [1. 71]. This can be done systematically by grid search [32] or dynamic programming [83]. Bao et al. [65] speed up sampling by defining a new DDM with optimal reverse variances. DDIM [58], discussed in Sec. 2, was also introduced as method to accelerate DDM synthesis. Further works leverage modern ODE and SDE solvers for fast synthesis from (continuous-time) DDMs: For instance, higher-order Runge-Kutta methods [57, 84] and adaptive step size SDE solvers [62] have been used. These methods are not optimally suited for the few-step synthesis regime, in which GENIE shines; see also Sec. 5. Most closely related to our work is Liu al. [63]. which simulates the DDIM ODE [58] using a higher-order linear multistep method [77]. As shown in Sec. 3, linear multistep methods can be considered approximation of the TTMs used in GENIE. Furthermore, Tachibana et al. [61] solve the generative SDE via a higher-order Itô-Taylor method [59] and in contrast to our work, they propose to use "ideal derivative trick" to approximate higher-order score functions. In App. B.2, we show that applying this ideal derivative approximation to the DDIM ODE does not have any effect: the "ideal derivatives" are zero by construction. Note that in GENIE, we in fact use the DDIM ODE, rather than, for example, the regular Probability Flow ODE [57], as the base ODE for GENIE
Alternatively, sampling from DDMs can also be accelerated via learning: For instance, Watson et al. [66] learn parameters of a generalized family of DDMs by optimizing for perceptual output quality, Luhman and Luhman [68] and Salimans and Ho [69] distill DDIM sampler into a student model, which enables sampling in as few as a single step. Xiao et al. [67] replace DDMs' Gaussian samplers with expressive generative adversarial networks, similarly allowing for few-step synthesis. GENIE can also be considered a learning- based approach, as we distill a derivative of the generative ODE into separate neural network. However, in contrast to the mentioned methods, GENIE still solves the true underlying generative ODE, which has major advantages: for instance, it can still be used easily for classifier-guided sampling [4, 57, 70] and efficiently encode data into latent space-a prerequisite for likelihood calculation [47, 57] and editing applications [17]. Note that the learnt sampler [66] defines a proper probabilistic generalized DDM; however, it isn't clear how it relates to the generative SDE or ODE and therefore how compatible the method is with applications such as classifier guidance.
Other approaches to accelerate DDM sampling change the diffusion itself [60, 85, 86] or train DDMs in the latent space of a Variational Autoencoder [49]. GENIE is complementary to these methods.
Higher-Order ODE Gradients beyond DDMs. TTMs [78] and other methods that leverage higher- order gradients are also applied outside the scope of DDMs. For instance, higher-order derivatives can play crucial role when developing solvers [87] and regularization techniques [88, 89] for neural ODEs [74, 75). Outside the field of machine learning. higher-order TTMs have been widely studied, for example, to develop solvers for stiff [90] and non-stiff [90, 91] systems.
Concurrent Works. Zhang and Chen [92] motivate the DDIM ODE from an exponential integrator perspective applied to the Probability Flow ODE and propose to apply existing solvers from the numerical ODE literature, namely, Runge-Kutta and linear multistepping, to the DDIM ODE directly. Lu et al. [93] similarly recognize the semi-linear structure of the Probability Flow ODE, derive dedicated solvers, and introduce new step size schedulers to accelerate DDM sampling. Karras et al. [94] propose new fast solvers, both deterministic and stochastic, specifically designed for the differential equations arising in DDMs. Both Zhang et al. [95] and Karras et al. [94] realize that the DDIM ODE has "straight line solution trajectories" for spherical normal data and single data points this exactly corresponds to our derivation that the higher-order terms in the DDIM ODE are zero in such setting (see App. B.2). Bao et al. [96] learn covariance matrices for DDM sampling using prediction heads somewhat similar to the ones in GENIE; in App. G.I, we thoroughly discuss the differences between GENIE and the method proposed in Bao et al. [96].

---

5
Experiments
Datasets:
We run experiments on five datasets: CIFAR-10 [97] (resolution 32), LSUN Bedrooms [98] (128), LSUN Church-Outdoon [98] (128), (conditional) ImageNet [99] (64), and AFHQv2 [100] (512). On AFHQv2 we only consider the subset of cats; referred to "Cats" in the remainder of this work.
Except for CIFAR- 10 (we use a checkpoint by Song et al. [57]), we train our own score models using architectures introduced by previous works [1, 4]. The architecture of our prediction heads is based on (modified) BigGAN residual blocks [57, 101]. To minimize computational overhead, we only use single residual block. See App. C for training and architecture details.
Architectures:
Evaluation: We measure sample quality via Fréchet Inception Distance [FID, 102] (see App. F.I).
Synthesis Strategy: We simulate the DDIM ODE from t=1 up to t=10 using evaluation times following quadratic function (quadratic striding [58]). For parierie- DDMs, it can be beneficial to denoise the ODE solver output the cutoff t=10 3 i.e., Xn xx-0,c0(x..)) [57, 103]. Note that the denoising step involves score model evaluation, and therefore "loses" a function evaluation that could otherwise be used as an additional step in the ODE solver. To this end, denoising the output of the ODE solver is left as a hyperparameter of our synthesis strategy.
Cte
Analytical First Step (AFS): Every additional neural network call becomes crucial in the low number of function evaluations (NFEs) regime. We found that can improve the performance of GENIE and all other methods evaluated on our checkpoints by replacing the learned score with the (analytical) score of (0, I) Pt=1 (x₂) in the first step of the ODE solver. The "gained" function evaluation can then be used as an additional step in the ODE solver. Similarly to the denoising step mentioned above, AFS is treated as a hyperparameter of our Synthesis Strategy. AFS details App. F.2.
Accounting for Computational Overhead: GENIE has a slightly increased computational overhead compared other solvers due the prediction head kyg. The computational overhead is increased by 1.47% 2.83%, 14.0%, and 14.4% on CIFAR-IO, 10, ImageNet, LSUN Bedrooms, and LSUN Church- Outdoor, respectively (see also App. C.2.5). This additional overhead is always accounted for implicitly: we divide the NFEs by the computational overhead and round to the nearest integer. For example, on LSUN Bedrooms, we compare baselines with 10/15 NFEs to GENIE with 9/13 NFEs.
5.1 Image Generation
In Fig. 5 we compare our method to the most competitive baselines. In particular, on the same score model checkpoints, we compare GENIE with DDIM [58]. S-PNDM [63], and F-PNDM [63]. For these four methods, we only include the best result over the two hyperparameters discussed above, namely, the denoising step and AFS (see App. F.6 for tables with all results). We also include three competitive results from the literature [64-66] that use different checkpoints and sampling strategies: for each method, we include the best result for their respective set of hyperparameters. We do not compare in this figure with Knowledge Distillation [KD, 68]. Progressive Distillation [PG, 69] and Denoising Diffusion GANs IDDGAN, 67] as they do not solve the generative ODE/SDE and use fundamentally different sampling approaches with trawbacks discussed in Sec. 4.
For NFEs E {10, 15, 20, 25), GENIE outperforms all baselines (on the same checkpoint) all four datasets (see detailed results in App. F.6 and GENIE image samples in App. F.7). On CIFAR-10 10 and (conditional) ImageNet, GENIE also outperforms these baselines for NFEs=5, whereas DDIM outperforms GENIE slightly on the LSUN datasets (see tables in App. F.6). GENIE also performs better than the three additional baselines from the literature (which use different checkpoints and sampling strategies) with the exception of the Learned Sampler [L.S, 66) on LSUN Bedrooms for NFEs=20. Though LS uses learned striding schedule on LSUN Bedrooms (whereas GENIE simply uses quadratic striding), the LS's advantage is most likely due to the different checkpoint. In Tab. 1, we investigate the effect of optimizing the striding schedule, via learning (LS) or grid search (DDIM & GENIE), on CIFAR-10 and find that its significance decreases rapidly with increased NFEs (also see App. F.6 for details). In Tab. 1, we also show additional baseline results; however, we do not include commonly-used adaptive step size solvers in Fig. 5, as they are arguably not well-suited for this low NFE regime: for example, on the same CIFAR- 10 checkpoint we use for GENIE, the adaptive SDE solver introduced in Volicoeur-Martineee on the same checkpoint, the adaptive Lange-KKutaa 4(5) [84] method applied to the ProbabilityFlow ODE achieves an FID of 13.1 at 38 NFEs (solver tolerances set to 10 10-2).
et al. [62] obtains an FID of 82.4 at 48 NFEs. Also
7

---

CIFAR-10
InageNes
GENIE (ours)
DDIM [58]
S-PNDM [63]
F-PNDM [63]
Learned Sampler [66] (f)
FastDDIM [64] (f)
Analytic DDIM [65] (f)
30
Figure 5: Unconditional perfor- mance on four popular bench- mark datasets. The first four meth- ods use the same score model checkpoints, whereas the last three methods all different checkpoints. (f): numbers are taken from literature.
14
10
20
10
20
25
Method
NFEs=5
NFEs=10
NFEs=15
NFEs=20
NFEs=25
Table Unconditional CIFAR-10 generative performance (mcasured in FID). Methods above the mid- dle line the same model checkpoint; methods below different ones. (f): numbers are taken from literature. (+): meth- ods cither learn optimal striding schedule (Learned Sampler) do small grid search over striding schedules (DDIM & GENTE); also App. F.6
GENIE (ours) (+)
11.2
5.28
4.49
3.94
3.64
GENIE (ours)
13.9
5.97
4.49
3.67
DDIM [58] (+)
27.6
11.2
7.35
5.87
5.16
DDIM [58]
29.7
11.2
7.35
5.16
S-PNDM [63]
35.9
10.3
6.61
5.20
4.51
F.PNDM [63]
N/A
N/A
10.3
5.96
4.73
Buler Maruyyan
325
230
164
112
80.3
FasDDIM (64]
Leamed Sampler [66] (f/**)
12.4
7,86
5,90
4.72
4.25
Leamed Sampler [66]
14.3
8.15
5.94
4.89
4.47
Analytic DDIM [65] (?)
14.0
5.71
CLD-SGM [60]
334
306
236
162
106
VESDE PC [57]
461
461
461
461
462
The results Fig. 5 suggest that higher-order gradient information, as used in GENIE, can be efficiently leveraged for image synthesis. Despite using small prediction heads our distillation seems to be sufficiently accurate: for reference, replacing the distillation heads with the derivatives computed via AD, we obtain FIDs of 9.22, 4.11, 3.54, 3.46 using 10, 20, 30, and 40 NFEs, respectively (NFEs adjusted assuming an additional computational overhead of 100%). As discussed in Sec. 3, linear multistep methods such as S-PNDM [63] and F-PNDM [63] can be considered (finite difference) approximations to TTMs as used GENIE. These approximations can be inaccurate for large timesteps, which potentially explains their inferior performance when compared to GENIE. When compared to DDIM, the superior performance of GENIE seems to become less significant for large NFE: this is in line with the theory, higher-order gradients contribute less for smaller step sizes (see the GENIE scheme in Eq. (9)). Approaches such as FastDDIM [64] and AnalyticDDIM [65], which adapt variances and discretizations of discrete-time DDMs, are useful; however, GENIE suggests that rigorous higher-order ODE solvers leveraging the continuous-time DDM formalism are still more powerful. To the best of our knowledge, the only methods that outperform GENIE abandon this ODE or SDE formulation entirely and train NFE-specific models [67, 69] which are optimized for the single use-case of image synthesis.
5.2 Guidance and Encoding
GENIE: NHtssed
GENIE, NFEs-10
As discussed in Sec. 4, one major drawback of approaches such as KD [68], PG [69] and DDGAN [67] is that they abandon the ODE/SDE formalism, and cannot easily use methods such as classifier(-free) guidance [57, 70] or perform image encoding. However, these techniques can play an important role in synthesizing photorealistic images from DDMs [3, 4, 15, 17). as well as for image editing tasks [12, 17].
25
NFEs-15
DDIM, NFF
20
NHE--10
DDIM, NFEe:1
Classifier-Frre Guidance [70]: We replace the unconditional
1.5
model 6 (x,,, () with (x,,, L, c, w) = (1 + wx,,(x,,<,c)
Guidance scale
Figure 6: Sample quality func- tion of guidance scale ImageNet.
weo( (Xe, !) in the DDIM ODE (cf Eq. (6)), where (x(x,, c)
conditional model and w > 1.0 the "guidance scale".
GENIE then requires the derivative
8

---

NFE
NFBa-10
N5
NFBas5
NF
NFEs-15
NFE
NFE=-10
NFE
Figure Classifier-free guidance for the ImageNet classes Pem- broke Welsh Corgi (263) and Streetcar (829).
Guädance stale 0.0
Guidkmoe stalle 1.0
Guidance scale 2.0
Reference
NFB
NFB 25
NPE
NFB
20
GRNII DDIM
15
Mest
10
GENTE
60
NFP
Figure 8: Encoding and subsequent decoding on LSUN Church-Outdoor Left: Visual reconstruction. Right: Ly-distance to reference in Inception feature space [104], averaged over 100 images.
dr, (x c, w) +1+ + w ddy, Eo t, wdd, Ex(Xe, t).
(16)
for guidance. Hence, we need to distill d:y (x,, t, c) and dy, (X,, t), for which we could also share parameters [70]. We compare GENIE with DDIM on ImageNet Fig. 6. GENIE clearly outperforms DDIM, in particular for few NFEs, and GENIE also synthesizes high-quality images (see Fig. 7). Image Encoding:
We can use GENIE also to solve the generative ODE in reverse to encode given images. Therefore, we compare GENIE to DDIM on the "encode-decode task, analyzing reconstructions for different NFEs (used twice for encoding and decoding): We find that GENIE reconstructs images much more accurately (see Fig. 8). For more details on this experiment as well as the guidance experiment above, see App. F.4 and App. F.3, respectively. We also show latent space interpolations for both GENIE and DDIM in App. F.5.
5.3 Ablation Studies
Table 2: CIFAR-10 10 ablation studics (measured in FID).
Ablation
NFE==5
NFEs=10
NFEs=15
NFE:=20
NFEs=25
We perform ablation studies over architec- ture and training objective for the prediction heads used in GENIE: In Tab. 2, "No mixed" single
Standard
3.67
mixed
14.7
6.32
4.82
4,31
4.10
No weighting
14.8
7.45
5.89
5.17
4,80
refers to learning d co directly network output without mixed network parameterization; "No weighting" refers to setting ga(t) in Eq. (15); "Standard" uses both the
Bigger model
13.7
5.58
4.66
4.05
3.77
mixed network parameterization and the weighting function ga (t) = 72 We can see that having both the mixed network parametrization and the weighting function is clearly beneficial. We also tested deeper networks in the prediction heads: for "Bigger model" we increased the number of residual blocks from one to two. The performance roughly par with "Standard", and therefore opted for the smaller head due to the lower computational overhead.
Table 3: Cats (upsampler) generative perfor- mance (measured in FID).
5.4 Upsampling
Cascaded diffusion model pipelines [2] and DDM-based super-resolution [8] have become crucial ingredients in
Method
NFEs=5
NFEs=10
NFEs=15
DDMs for large-scale image generation [105]. Hence, we also explore the applicability of GENIE in this setting. We train a 128 x 128 base model well as 128 x 128 + 512 x 512 diffusion upsampler [2, 8] on Cats. In Tab. 3. we compare the generative performance of GENIE other fast samplers for the upsampler (in isolation). We find that GENIE performs very well on this task: with only five NFEs GENIE outperforms all other methods at NFEs=15. We show upsampled samples for GENIE with NFFs=5 in Fig. 9. For more quantitative and qualitative results, we refer to App. F-6 and App. F.7, respectively. Training and inference details for the score model and the GENTE prediction head, for both base model and upsampler, can be found in App. C.
GENIE (ouns)
DDIM [58]
9.47
6.64
5.85
[63]
F-PNMM [63]
N/A
N/A
11.7

---

Figure 9: High-resolution images generated with the 128 128 - 512 512 GENIE upsampler using only five neural network calls. For the two images at the top, the upsampler is conditioned on test images from the Cats dataset. For the two images the bottom, the upsampler conditioned on samples from the 128 x 128 GENIE base model (generated using 25 NFEs); upsampler neural network evaluation is roughly four times as expensive as a base model evaluation.
6 Conclusions
We introduced GENIE, higher-order ODE solver for DDMs. GENIE improves upon the commonly used DDIM solver by capturing the local curvature of its ODE's gradient field, which allows for larger step sizes when solving the ODE. We further propose to distill the required higher-order derivatives into small prediction heac-which we can efficiently call during inference-on top of the first-order score network. A limitation of GENIE is that it is still slightly slower than approaches that abandon the differential equation framework of DDMs altogether, which, however, comes the considerable cost of preventing applications such guided sampling. To overcome this limitation, future work could leverage even higher-order gradients accelerate sampling from DDMs even further (also see App. G.2).
Broader Impact. Fast synthesis from DDMs, the goal of GENIE, can potentially make DDMs an attractive method for promising interactive generative modeling applications, such as digital content creation or real-time audio synthesis, and also reduce DDMs environmental footprint by decreasing the computational load during inference. Although we validate GENIE on image synthesis, it could also be utilized for other tasks, which makes its broader societal impact that context, it is important that practitioners apply abundance of caution to mitigate impacts given generative modeling can also used for malicious purposes, discussed for instance in Vaccari and Chadwick [106], Nguyen et al. [107], Mirsky and Lee [108].
detiicatioondeeeedene In
Acknowledgements
This work was funded by NVIDIA. Tim Dockhorn acknowledges additional funding from the Vector Institute Research Grant, not in direct support of this work. We thank Yaoliang Yu for discussions.
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, 2020.
[2] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. arXiv:2106. /5282, 2021.
[3] Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diftusion Probabilistic Models, In International Conference Machine Learning, 2021.
[4] Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. In Neural Information Processing Systems, 2021.
[5] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10522, 2021.
[6] Ruihan Yang. Prakhar Srivastava, and Stephan Mandt. Diffusion Probabilistic Modeling for Video Generation. arXiv:2203.0948) 2022.
[7] Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models. 2rX1v:2204.03458, 2022.
[8] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image Super-Resolution via Iterative Refinement. arXiv:2 2021.
[9] Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models. arXiv:2104. 2021.
[10] Jay Whang, Mauricio Delhracio, Hossein Talehi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via Stochastic Refinement. arXiv:2112.02475, 2021.
[11] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising Diffusion Restoration Models. arXiv:2201.1179 2022.
[12] Chenlin Meng, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Image Synthesis and Editing with Stochastic Differential Equations. 1eXiv:2108 2021.
[13] Andreas Lugmayr. Martin Danelljan, Andres Romero, Fisher Yu. Radu Timofte, and Luc Van Gool. RePaint: Inpainting using Denoising Diffusion Prohabilistic Models. wxXiv:221..09866 2022.
[14] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lec, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-Image Diffusion Models. (rXiv:2111.05822 2021.
[15] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text- Guided Diffusion Models. wr.iv:2112..077 2021.
[16] Omri Avrahami, Dani Lischinski, and Ohad Fried, lended Diffussin for Text-driven Editing of Natural Images. arXiv:2111. 14818, 2021.
[17) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical Text- Conditional Image Generation with CLIP Latents. arXiv:2204.06125 2022.
[18] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Condition- ing Method for Denoising Diffusion Probabilistic Models. In Proceedings of the JEEE/CVF International Conference Computer Vision (ICCV), 2021.
[19] Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffuse VAE: Efficient, Control- lable and High-Fidelity Generation from now-Dimensional Latents. arXiv:2201.0M0) 003308, 2022.
[20] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakorn. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. arXiv:2111. 2021.
[21] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More Control for Free! Image Synthesis with Semantic Diffusion Guidance. ar arXiv:21 2021.
[22] Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional Image Generation with Score- Based Diffusion Models. arXiv:2111.3306, 2021.
11

---

[23] Hiroshi Sasaki, Chris G. Willcocks, and Toby P. Breckon. UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models. arXiv:2104.05358 2021.
[24] Xuan Su, Jiaming Song. Chenlin Meng, and Stefano Ermon. Dual Diffusion Implicit Bridges for Image-to-Image Translation, arXiv:2233.08382, 2022.
[25] Yang Song. Liyue Shen, Lei Xing, and Stefano Ermon. Solving Inverse Problems in Medical Imaging with Score-Based Generative Models. In International Conference on Learning Representations, 2022.
[26] Cheng Peng, Pengfei Guo, S. Kevin Zhou, Vishal Patel, and Rama Chellappa. Towards performant and
reliable undersampled MR reconstruction via diffusion model sampling. arXiv:2203.04292 2022.
[27] Yutong Xie and Quanzheng Li. Meusuremen--comddtiione Denoising Diffusion Probabilistic Model for Under-sampled Medical Image Reconstruction, arXiv:2033.0366 2022.
[28] Guanxiong Luo, Martin Heide, and Martin Uecker. MRI Reconstruction via Data Driven Markov Chain with Joint Uncertainty Estimation. arXiv:2202.01479 2022.
[29] Hyungjin Chung and Jong Chul Ye. Score-hased diffusion models for accelerated MRI. arX1::2110.05243; 2021.
[30] Dewei Hu, Yuankui K. Tao, and lpek Oguz. Unsupervised Denoising of Retinal OCT with Diffusion Probabilistic Model, arXiv:2201. 11760, 2022.
[31] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come- Saaee-Coooe--fffu--aasttte Accelerating Con- ditional Diffusion Models for Inverse Problems through Stochastic Contraction. arXiv:2112.05146, 2021.
[32] Nanxin Chen, Yu Zhang. Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating Gradients for Waveform Generation. International Conference on Learning Representations. 2021.
[33] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In International Conference on Learning Representations, 2021.
[34] Myeonghun Jeong. Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-TTS: A Denoising Diffusion Model for Text-to- Speeehh arXiv preprint arXiv:2104.011 00440.40099 2021.
[35] Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis. arXiv:2 10l.:200,,,9600, 2021.
[36] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In International Conference on Machine Learning. 2021.
[37] Songxiang Liu, Dan Su. and Dong Yu. DiSIGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs. arXiv:2201. 1172, 2022.
[38] Shitong Luo and Wei Hu. Diffusion Probabilistic Models for 3D Point Cloud Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[39] Ruojin Cai, Guandao Yang, Hadar Berbucch-Flor, 7ekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning Gradient Fields for Shape Generation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[40] Linqi Zhou, Yilun Du, and Jiajun Wu. 3D Shape Generation and Completion through Point- Voxel Diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[41] Zhaoyang Lyu, Zhifeng Kong, Xudong XU, Liang Pan, and Dahua Lin. A Conditional Point Diffusion- Refinement Paradigm for 3D Point Cloud Completion. In International Conference Learning Repre- sentations. 2022.
[42] Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you should treat it like one. arXiv:2001.12204, 2022.
[43] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning, 2021.
12

---

[44] Minkai Xu, Lantao Yu, Yang Song, Chence Shi. Stefano Ermon, and Jian Tang. GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation. In International Conference on Learning Representations, 2022.
[45] Tian Xie, Xiang Fu, Octavian -uggen Ganea, Regina Barzilay, and Tommi S. Jaakkola. Crystal Diffusion Variational Autoencoder for Periodic Material Generation, In International Conference on Learning Representations, 2022.
[46] Jachyeong Jo, Seul Lee, and Sung Ju Hwang. Score- Generative Modeling of Graphs via the System of Stochastic Differential Equations. 0.xiv:2202.0251144 2022.
[47] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum Likelihood Training of Score- Based Diffusion Models. In Neural Information Processing Systems (NeurIPS), 2021.
[48] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models, In Advances in Neural Information Processing Systems, 2021.
[49] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-hased Generative Modeling in Latent Space. In Neural Information Processing Systems (NeurlPS), 2021.
[50] Chin- Huang, Jae Hyun Lim, and Aaron Courville, A Variational Perspective on Diffusion- Baased Generative Models and Score Matching. In Newral Information Processing Systems (NeurIPS), 2021.
[51] Chenhao Niu, Yang Song, Jiaming Song. Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score- based generative modeling. In International Conference on Artificial Intelligence and Storistics, 2020.
[52] Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial Purification with Score- basee Generative Models. In International Conference on Machine Learning, 2021.
[53] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. Advances in Neural Information Processing Systems, 2021.
[54) Valentin De Bortoli, Emile Mathicu, Michael Hutchinson, James Thomton, Yee Whye Tch, and Arnaud Doucet, Riemannian Score-Based Generative Modeling. arXiv:2202.02765 2022.
[55] Pedro Sanchez and Sotirios A. Tsaftaris. Diffusion Causal Models for Counterfactual Estimation. In First Conference Causal Learning and Reasoning, 2022.
[56] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar, Diffusion Models for Adversarial Purification. In International Conference on Machine Learning (ICML), 2022.
[57] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations, 2021.
[58] Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference on Learning Representations, 2021.
[59] Peter E. Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Springer, Berlin, 1992.
[60] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-Base Generative Modeling with Critically- Damped Langevin Diffusion. In International Conference Learning Representations, 2022.
[61] Hideyuki Tachibana, Mocho Go, Muncyoshi Inahara, Yotaro Katayama, and Yotaro Watanabe. Itô- Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives. arXiv:2/12.1333 2021.
[62] Alexia Aalccour-Martineal Ke Li, Rémi Tiilé-Tail,efer, Tal Kachman, and loannis Mitliagkas. Gotta Go Fast When Generating Data with Score- Models. arXiv:2 14080, 2021.
[63] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo Numerical Methods for Diffusion Models on Manifolds. In International Conference Learning Representations, 2022.
[64] Zhifeng Kong and Wei Ping. On Fast Sampling of Diffusion Probabilistic Models, arXiv:2106.00132. 2021.
13

---

[65] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytie-DPM: Analytic Estimate of the Op- timal Reverse Variance in Diffusion Probabilistic Models. In International Conference on Learning Representations, 2022.
[66] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality. International Conference on Learning Representations, 2022.
[67] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In International Conference Learning Representations, 2022.
[68] Eric Luhman and Troy Luhman. Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed. arXiv:2101.02388 2021.
[69] Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models. In International Conference on Learning Representations, 2022.
[70] Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications,, 2021.
[71] Jascha Sohl-Dicksted Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In International Conference on Machine Learning, 2015.
[72] Brian DO Anderson. Reverse- time diffusion equation models. Stochastic Processes and their Applications. 12(3)2313-326, 1982.
[73] Ulrich G Haussmann and Etienne Pardoux. Time Reversal of Diffusions. The Annals of Probability, pages 188-1200, 1986.
[74] Ricky T. Q. Chen, Yulia Ruhanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. Advances in Neural Information Prucessing Systems, 2018
[75] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free- form Continuous Dynamics for Scalable Reversible Generative Models. International Conference on Learning Representations, 2019.
[76] Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computa- tion, 23(7):1661-167) 2011.
[77] John Charles Butcher. Numerical Methods for Ordinary Differential Equations. John Wiley & Sons, 2016.
[78] Peter E Kloeden and Eckhard Platen. Numerical Solution Stochastic Differential Equations. Springer, 1992.
[79] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisteee Intervention, pages 234-241. Springer, 2015.
[80] Jooyoung Choi, Jungheom Lec, Chachun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception Prioritized Training of Diffusion Models. arXiv:2204.00227, 2022.
[81] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachew, Valentin Khrulkov, and Artem Babenko. Label- Efficient Semantic Segmentation with Diffusion Models. In International Conference on Learning Representations, 2022.
[82] Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon. Estimating High Order Gradients of the Data Distribution by Denoising, Advances in Neural Information Processing Systems, 34, 2021.
[83] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to Efficiently Sample from Diffusion Prohabilistic Models. arXiv:2/06.03300 2021.
[84] J. R. Dormand and P. J. Prince. A family of embedded Rutte-Kutta formulae. Journal of Computational and Applied Mathematics, 6(1):19-26, 1980.
[85] Eliya Nachmani, Robin San Roman, and Lior Wolf. Non Gaussian Denoising Diffusion Models. 07X55 2021.
14

---

[86] Max WY Lam. Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral Denoising Diffusion Models. arXiv:2108. 1 11514, 2021.
[87] Franck Djeumou, Cyrus Neary, Eric Goubault, Sylvie Putot, and Ufuk Topeu. Taylor-Lagrange Neural Ordinary Differential Equations: Toward Fast Training and Evaluation of Neural ODEs, arXiv: arXi::2001.05715, 2022.
[88] Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learning differential equations that are easy to solve. Advances in Neural Information Processing Systems, 33:4370-4380, 2020.
[89] Chris Finlay, Förn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ODE: the world of Jacobian and kinetic regularization. In International conference on machine learning. pages 164. PMLR, 2020.
[90] YF Chang and George Corliss ATOMFT: solving ODEs and DAEs using Taylor series. Computers & Mathematics with Applications, 28(10-
1994.
[91] George Corliss and YF Chang. Solving ordinary differential equations using Taylor series. ACM Transactions on Mathematicul Software (TOMS), 14-1444 1982.
[92] Qinsheng Zhang and Yongxin Chen. Fast Sampling of Diffusion Models with Exponential Integrator. arXiv:2204.1399 13992, 2022.
[93] Cheng Lu, Yuhao Zhou, Fan Bao, Jíanfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. arXiv:2206.000) 2022.
[94] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion- Based Generative Models. vrXiv:2206.00366 2022.
[95] Qinsheng Zhang. Molei Tao, and Yongxin Chen. gDDIM: Generalized denoising diffusion implicit models. arXiv:2206.0 055664 2022.
[96] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models. arXi::2266.03009 2022.
[97] Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009.
[98] Fisher Yu, Ari Seff. Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of Large-scale Image Dataset using Deep Learning with Humans in the Loop. arXiv preprint arXiv: 1506.0366603.3 2015.
[99] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li. and Fei-Fei. ImageNet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[100] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo
Woo Ha. StarGAN v2: Diverse Image Synthesis for Multiple Domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8188-8197, 2020.
[101] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations, 2019.
[102] Martin Hcusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time Scale Update Rule Converge to Local Nash Equilibrium. In Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[103] Alexia Jolicocur-Martincaan Rémi Piché-Taillefer, Joannis Mitliagkas, and Remi Tachet des Combes. Adversarial score matching and improved sampling for image generation. In International Conference on Learning Representations, 2021.
[104] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2818 2818-2826, 2016.
[105] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic Text-to- Texaae Diffusion Models with Deep Language Understanding. arXiv:2205. 11487, 2022.
15

---

[106] Cristian Vaccari and Andrew Chadwick. Deepfakes and Disinformation: Exploring the Impact of Synthetic Political Video on Deception, Uncertainty, and Trust in News. Social Media Society, 6(1): 2056305120903408, 2020.
[107] Thanh Thi Nguyen. Quoc Viet Hung Nguyen, Cuong M. Nguyen, Dung Nguyen, Duc Thanh Nguyen, and Saeid Nahavandi. Deep Learning Deepfakes Creation and Detection: A Survey. arXiv:990.115773 2021.
[108] Yisroel Mirsky and Wenke Lee. The Creation and Detection Deepfakes: Survey, ACM Comput. Surv. 54(1), 2021.
[109] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations (ICLR), 2018.
[110] Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
[111] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. Advances in Neural Information Processing Systems, 31, 2018.
[112] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved Precision and Recall Metric for Assessing Generative Models. Advonces in Neural Information Processing Systems, 32, 2019.
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We did not derive novel theoretical results. We rather propose a novel higher-order solver for sampling from denoising diffusion generative models.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] We will provide code and instructions to reproduce all experiments in the near future.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We provide all model training and evaluation details in the Appendix, including all hyperparameters.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] Following standard conventions in the related generative modeling literature, we do not report error bars. Furthermore, we avoid running similar setups repeatedly to save computational resources.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see App. F.8.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] On CIFAR-
10, we use a checkpoint provided by Song et al. [57]. For baseline comparisons, we run publicly available code: in particular, we make use of the repositories https://githen.
com/yang- song/score sde_pytorch and https://githur CC/D/CLDCCC SGM from previous, publicly available papers, which we cite (Song et al. [57] and Dockhorn et al. [60], respectively). We also cite all used datasets: CIFAR- 10 [97], LSUN Bedrooms [98], LSUN Church-Outdoor [98], ImageNet [99], and AFHQv2 [100].
con/nv-tlabss tlttta/laaaaaaaae
(b) Did you mention the license of the assets? [Yes] In the Appendix, mention the licenses of all codes we are running for baseline comparisons and evaluation. See App. F.I.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We are only using publicly available datasets, which have been widely used in the generative modeling literature standard bench- marks CIFAR-10, 10, ImageNet, LSUN Church-Qutdoor, LSUN Bedrooms, AFHQv2) and avoid using face image datasets.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "xA_ilZcYmTM", "reviewer": "Reviewer_bcTa", "summary": "In this work, the authors proposed Higher-Order Denoising Diffusion Solvers (GENIE) using truncated Taylor methods,  which significantly accelerates image synthesis. The solver relies on higher-order score functions. To obtain the Jacobian-vector products (JVPs) for the higher-order score function, the authors proposed to extract them from the first-order score network via automatic differentiation and using a small additional head on top of the first-order score network.  The authors provide empirical  results to verify the advantages of the GENIE.", "questions": "It would be better to quantify the discretization error of the GENIE compared to the existing higher order methods.", "limitations": "Still, the slow convergence is considered a limitation.  Aside from the higher-order score function, it would be better to consider other type of acceleration method for the reverse sampling procedure. I", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The derivation of the higher-order score function using automatic differentiation and small additional head is novel. The empirical results are persuading.\n\nWeakness:  The training scheme for the higher-order score function appears heuristic, which directly minimizes the differences. Given that the score training comes from more mathematical reasoning from the variational lower bound, the suggested heuristic derivation may need more theoretical justification from that aspect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "lfZlkMnVOal", "reviewer": "Reviewer_LSQr", "summary": "This paper proposes a high order method to solve the DDM ODE. To overcome the expansive computation of the JVP appeared in the high order term, this paper uses NN to distill the high order term. This paper also uses parameter sharing to avoid to much computation of NN. Experiments show GEINE has a good speed up performance.", "questions": " ", "limitations": " ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nAlthough the idea of high order solver already exists in diffusion models, this work further improves the performance of the high order solver. Therefore, the significance is sufficient in my opinion. Besides, this paper is also clearly written, and sufficient experiments are provided.\n\n\nWeakness \\& Questions:\n\n1. How are the coefficients in Eq.(14) constructed? It does not directly match the coefficients in Eq.(12).\n\n2. The $x_t$ in $d_{\\gamma_t} \\epsilon_\\theta(x_t, t)$ follows the ODE. However, $x_t$ from $d_{\\gamma_t} \\epsilon_\\theta(x_t, t)$ in Eq.(15) comes from $x_t = \\alpha_t x_0 + \\sigma_t \\epsilon$. Does $d_{\\gamma_t} \\epsilon_\\theta(x_t, t)$ in Eq.(15) actually denotes the function in the r.h.s. of Eq.(12), where $x_t$ can be arbitrary. If does, perhaps it would be better to use a new notation for the function in the r.h.s. of Eq.(12).\n\n3. The author has mentioned prior high order solvers are suboptimal. The key difference that makes GENIE perform better can be discussed more. For example, both aiming to solve Eq.(8), why the NN approximation is better than other approximations in prior solvers?\n\n4. I suggest to cite and discuss [1], which also learns an extra head at the top of the UNet to speed up sampling of diffusion models.\n\n\n[1] Bao et al., Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ug_Qv0TQ6vI", "reviewer": "Reviewer_6WXQ", "summary": "The paper proposes a novel higher-order denoising diffusion solver, called GENIE, which is applicable particularly for denoising diffusion models with the variational preserving forward noising process. The GENIE solver is established from two novel approaches proposed in the paper.\n\nFirst, the paper identifies that p-th order Taylor approximation (truncated Taylor Method, TTM) to DDIM ODE results in much simpler terms than the TTM of the equivalent probability flow ODE.\n- Note that an equivalent probability flow ODE of a denoising SDE, aka denoising diffusions, is an ordinary differential equation whose marginals equal (in distribution) to the denoising SDE's marginals at every time step. Moreover, DDIM ODE is a scaled stochastic process (time-inhomogeneous manner) of the equivalent probability flow ODE.  \n\nMore specifically, the authors show that the 2nd-order term in the DDIM ODE's Taylor approximation reduces to two Jacobian-vector products (JVP) plus one time-derivative; One JVP is a product of the Jacobian of the score function (wrt input) and the score function, and the Jacobian and input.\n\nSecond, instead of using the finite difference approximation of the Jacobian (of the score wrt input) during inference, it proposes to distill the JVPs (or a sum of two) by another network. The training of the JVP network is via minimizing the mean squared error to the finite difference approximation.\n- Note that JVP will have the same dimensionality of the data. This is much lower than some previous approaches to learning higher order scores, which is $d^2$ for $d$-dimensional data.\nTo reduce the complexity of learning additional networks, the paper proposes a model both predicts score and JVP by sharing feature extractors.\n\nIn addition, the paper further introduces higher-order (>2) GENIE solvers.\n\nFor the experiments, the paper demonstrates the effectiveness of GENIE on multiple image generation benchmarks.", "questions": "N/A", "limitations": "N/A", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\nIn my understanding, the paper's contributions are clear, and I also consider that the results are essential for several reasons:\n\n1. The paper proposes a novel higher-order denoising diffusion solver applicable to pre-trained score-based generative models.\n2. The paper motivates each part of the proposed method well so that readers can understand how each step contributes to the merits of GENIE.\n3. I found that the paper has a well-organized structure that makes it clear to understand the proposed method. Significantly, the paper shares sufficient information to understand the new solver's behaviors, which will help readers understand relevant backgrounds and the proposed method.\n    - For example, the first-order method to DDIM ODE is equivalent to the second-order with the \"ideal derivative trick\". Therefore, the first-order method to DDIM ODE performs better than the first-order method to the original probability flow ODE. \n\nIn general, the paper's contributions wrt the novelty are clear, and the proposed methods are well-defined. In addition, I found that the paper has a well-organized structure that makes it clear to understand the proposed methods. Thus, I'm inclined to accept the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GXWBhG8Y3uk", "reviewer": "Reviewer_kzgD", "summary": "This paper proposes a higher-order denoising diffusion solver called GENIE to speed up the sampling process for DPM. \n\nIn particular, the idea of the paper is inspired by the fact that DDIM is a first-order discretization of the probability ODE. Naturally, the paper attempts to use higher-order methods to solve the ODE with smaller errors in the same step or equivalently, obtain a good result with much fewer steps.\n\nThis paper considers the second-order method in particular. The resulting sampling process requires the second-order score functions under the hood, which is also natural. \n\nThe paper tackles the problem by distilling the terms related to the second-order score functions (approximated by the jacobian of a score network). Namely, learning neural networks to predict the jacobian vector product terms appeared in the sampling process.\n\nThe proposed method is compared to training-free methods under FID on several standard benchmarks.", "questions": "See weakness.", "limitations": "I did not see a direct negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. At a very high level, I agree that introducing higher-order methods into DPMs can be helpful. \n\n2. The presentation is clear and the related work is properly discussed.\n\nWeakness:\n\n1. The way to deal with the second-order score functions is not that principled.\n\n2. It would be much better if the authors could provide some theoretical benefits of the higher-order methods.\n\n3. Likelihood results are missing. It is well known that the likelihood and FID are complementary metrics for generative modeling. It is necessary to evaluate all models under both metrics.\n\n4. The baselines in Table 1 are mainly training-free methods. Why not include [69] in Table 1? [69] is more comparable to GENIE since it also trains additional models for distillation.\n\n5. Why GENIE is not that good with NFE=5 in Table 1? I did not see an analysis in depth.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept