Efficient Integrators for Diffusion Generative Models
Authors: Kushagra Pandey
Maja Rudolph
Stephan Mandt
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion models
Deep Generative Models
Efficient Sampling
Primary area: generative models
TL;DR: We propose generic frameworks for efficient sampling from diffusion models

Abstract: Diffusion models suffer from slow sample generation at inference time. Therefore, developing a principled framework for fast deterministic/stochastic sampling for a broader class of diffusion models is a promising direction. We propose two complementary frameworks for accelerating sample generation in pre-trained models: Conjugate Integrators and Splitting Integrators. Conjugate integrators generalize DDIM, mapping the reverse diffusion dynamics to a more amenable space for sampling. In contrast, splitting-based integrators, commonly used in molecular dynamics, reduce the numerical simulation error by cleverly alternating between numerical updates involving the data and auxiliary variables. After extensively studying these methods empirically and theoretically, we present a hybrid method that leads to the best-reported performance for diffusion models in augmented spaces. Applied to Phase Space Langevin Diffusion [Pandey \& Mandt, 2023] on CIFAR-10, our deterministic and stochastic samplers achieve FID scores of 2.11 and 2.36 in only 100 network function evaluations (NFE) as compared to 2.57 and 2.63 for the best-performing baselines, respectively. Our code and model checkpoints will be made publicly available at https://github.com/mandt-lab/PSLD

Published as a conference paper at ICLR 2024
EFFICIENT INTEGRATORS FOR DIFFUSION GENERATIVE MODELS
Kushagra Pandey
Maja Rudolph
Department of Computer Science University of California, Irvine pandeyk1 (eebeyk@@uii.. edd
Bosch Center for Artificial Intelligence
Ma ja Rudolph@us bosch. com
Stephan Mandt
Department of Computer Science University of California, Irvine mandt@uci. edu
ABSTRACT
Diffusion models suffer from slow sample generation at inference time. Therefore, developing a principled framework for fast determinissti/ssoohaaste sampling for a broader class of diffusion models is promising direction. We propose two com- plementary frameworks for accelerating sample generation pre-trained models: Conjugate Integrators and Splitting Integrators. Conjugate integrators general- ize DDIM, mapping the reverse diffusion dynamics to more amenable space for sampling. In contrast, splitting-based integrators, commonly used in molec- ular dynamics, reduce the numerical simulation error by cleverly alternating be- tween numerical updates involving different partitions of the drift (and diffusion) components. After extensively studying these methods empirically and theoreti- cally, we present hybrid method that leads to the best-reported performance for diffusion models in augmented spaces. Applied to Phase Space Langevin Dif- fusion [Pandey & Mandt, 2023] CIFAR-10, our deterministic and stochastic samplers achieve FID scores of 2. and 2.36 in only 100 network function eval- uations (NFE) as compared to 2.57 and 2.63 for the best-performing baselines, respectively. Our code and model checkpoints will be made publicly available at https: //gi thub.
Lab/SLD...
INTRODUCTION
Score-based Generative models (or Diffusion models) (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho al., 2020; Song et al., 2020) have demonstrated impressive performance on various tasks, such as image and video synthesis (Dhariwal & Nichol, 2021; Ho et al., 2022a; Rombach al., 2022: Ramesh et al., 2022; Saharia et al., 2022a; Yang et al., 2022; Ho al., 2022b; Harvey et al., 2022), image super-resolution (Saharia et al., 2022b), and audio and speech synthesis (Chen
et al., 2021; Lam et al., 2021).
However, high-quality sample generation in standard diffusion models requires hundreds to thou- sands of expensive score function evaluations. While there have been recent advances in improving the sampling efficiency (Song et al., 2021; Lu et al., 2022; Zhang & Chen, 2023), most of these ef- forts have been focused towards a specific family of models that perform diffusion in the data space (Song et al., 2020; Karras et al., 2022). Interestingly, recent work (Dockhorn et al., 2022b; Pandey & Mandt, 2023; Singhal et al., 2023) indicates that performing diffusion in a joint space, where the data space is augmented with auxiliary variables, can improve sample quality and likelihood over data-space-only diffusion models. However, with a few exceptions focusing on specific score parameterizations (Zhang al., 2022). improving the sampling efficiency for augmented diffusion models is still underexplored but a promising avenue for further improvements.
Work partially done during an internship Bosch Center for Artificial Intelligence

---

Published as a conference paper at ICLR 2024
NFE (FID@ 50k L)
Method
Description
Diffusion
50
100
(Ours) CSPS-D
Conjugate Splittigg PSLD Sampler (CSPS)
3.21
(Ours) CSPSDD (+Pre.)
CSPS-D + Score Network precenditooing
FSLD
2.65
2.24
Deterministic
DDIM (Song et al., 2021)
Denoising Dillission Implicit Model
DDPM
4.67
4.16
DEIS (hhagg & Chen, 2023)
Exponemtial Integratee with polynomial extrapolation
VP
2.59
2.57
DPM-Salver-3 (Lu ot al., 2022)
Exponential Integrator (ordor=3)
VP
2.59
2.59
PNDM (Liu
2022)
Solver diffcrential equations manifolds
DDPM
3.68
3.53
EDM (Karras et al., 2022)
Heun's 's methed upplied to re scaled diflusion ODE
VP
3.08
3,06
gDDIM (Zhang et al., 2022)
Generalized form DDIM (g
3.31
A-DDIM (Bao al., 2022)
Analytic variance estimation reverse diflusion
DDPM
4.04
3.55
(Ours) SPS-S
Splitting-based PSLD Sampler (SDE)
FSLD
2.76
2.36
(Ours) SPS-S (+Pre.)
SPS-D + Score Network Precenditioning
PSLD
2.74
2.47
Siochaticc
Solver (Xue al., 2023)
Stochustic Adamss Solver applied to reverse SDEs
2.92
SEEDS-2 (Gonzalez et al. 2023)
Exponential Integrators for SDEs (onderw2)
DDPM
11.10
3.19
EDM (Karras et at, 2022)
Custom stochastic sampler with churn
3.19
2.71
A-DDPM (Bao al., 2022)
Analytic variance estimation reverse diffusion
DDPM
5.50
4.45
SSCS (Dockhorn el al., 2022b)
Symmetric Spliting CLD Sampler
PSLD
18.83
4.83
EM (Kloeden Platen, 1992)
Baker Maruyana aaryaama SDE sampler
PSLD
30.81
Table 1: Our proposed samplers perform comparably or outperform prior methods for CIFAR-IO.
Diffusion: (VP.VE) (Song et al., 2020), CLD (Dockhorn et al., 2022b), DDPM (Ho et al., 2020), PSLD (Pandey & Mandt, 2023). To ensure fair comparison, methods indicated with * were evaluated without incorporating additional training tricks. (Extended Results: Fig. 5)
Efficient Sampling during Inference. Our goal is to develop efficient de- terministic and stochastic integration schemes that are applicable to sampling from a broader class of diffusion models (for instance, where the data space is augmented with auxiliary variables) and achieve high-fidelity samples, even when the NFE budget is greatly reduced, e.g., from 1000 to 100 even 50. We evaluate the effectiveness of the proposed samplers in the context of the Phase Space Langevin Diffusion (PSLD) (PSLD) (Pandey & Mandt, 2023) due to its strong empirical perfor- mance. However, the presented techniques also apply other diffusion models, some of which are special cases of PSLD (e.g. Dockhorn et al. (2022b)). We make the following contributions,
Problem Statement:
Conjugate Deterministic Integrators. These numerical integrators leverage invertible transformations to map the reverse process' deterministic dynamics to space more suit- able for fast sampling. We show that several existing deterministic sampling frameworks like DDIM (Song et al., 2021) and exponential integrators (Lu et al., 2022; Zhang & Chen, 2023; Zhang et al., 2022) are special cases of our framework, allowing us to generalize these methods to generic diffusion models in principled manner. Moreover, analyze the proposed framework from the lens of stability analysis and provide a theoretical justifi- cation for its effectiveness.
Reduced Splitting Integrators. Taking inspiration from molecular dynamics (Leimkuhler, 2015), we present Splitting Integrators for efficient sampling in diffusion models. However, we show that their naive application can be sub-optimal for sampling efficiency. Therefore, based on local error analysis for numerical solvers (Hairer et al., 1993), we present several improvements to our naive schemes to achieve improved sample efficiency. We denote the resulting samplers as Reduced Splitting Integrators.
Conjugate Splitting Integrators. We combine conjugate integrators with reduced splitting integrators for improved sampling efficiency and denote the resulting samplers as Con- jugate Splitting Integrators. Our proposed samplers significantly improve sampling effi ciency in PSLD. For instance, our best deterministic sampler achieves FID scores of 2.65 and 2. 1, while our best stochastic sampler achieves FID scores of 2.74 and 2.36 in 50 and 100 NFEs, respectively, for CIFAR-10 (Krizhevsky, 2009) (See Table I for comparisons).
2 BACKGROUND
As follows, we provide relevant background on diffusion models and their augmented versions. Diffusion models assume that continuous time forward process (usually with an affine drift).
dz, F,z, dt G2 dwe, 1 E [0, T),
2

---

Published as a conference paper at ICLR 2024
with standard Wiener process Wi, time-dependent matrix F: [0, Rdxd ficient G: [0, + Rd converts data Zu E R into noise. A reverse SDE specifies how data is generated from noise (Anderson, 1982; Song et al.. 2020),
! and diffusion coef-
dz: = Fiz - G,G. Vx. log))(z:) dt + G.dWe,
(2)
which involves the score Vz, logpr(z:) (Z:) of the marginal distribution over 2, at time l. Alternatively, data can be generated from the Probubility-Fiee ODE (Song et al., 2020),
F.z: 1 G.G. Vz, log
dz
log pt (z+) dt.
(3)
The score is intractable to compute and is approximated using parametric estimator 8g(2,, l), trained using denoising score matching (Vincent, 2011; Song & Ermon, 2019; Song et al., 2020). Once the score has been learned, generating new data samples involves sampling noise from the stationary distribution of Eqn. 1 (typically an isotropic Gaussian) and numerically integrating Eqn. 2, resulting in stochastic sampler, on efficient sample generation in diffusion models has focused on limited class of non-augmented diffusion models (Song et al., 2020; Karras al., 2022), our work is also applicable to broader class of diffusion models. These two classes of diffusion models are presented next.
Eqn. resulting in deterministic sampler. While most work
Non-Augmented Diffusions. Many existing diffusion models are formulated purely in data space, i.c. Z, = X₁ E Rd One popular example is the Variance Preserving (VP)-SDE (Song et al., 2020) with F,- - - G. v Recently, Karras et al. (2022) instead propose re-scaled process, with F, - 02, G. = / 26 .which allows for faster sampling during generation. Here 3 E R define the noise schedule in their respective diffusion processes.
Augmented Diffusions. For augmented diffusions, the data (or position) space, Xt, is coupled with auxiliary (a.k.a momentum) variables, m,, and diffusion is performed in the joint space. For instance, Pandey & Mandt (2023) propose PSLD, where Z, = [X,, m,!" g2d, Moreover,
F. 1 -1 - )))= 0
M- ((=))
(VL3
G.
Ia).
(4)
VMB
where (,,T, M-1) R are the SDE hyperparameters. Augmented diffusions have been shown to exhibit better sample quality with a faster generation process (Dockhorn et al., 2022b; Pandey & Mandt, 2023), and better likelihood estimation (Singhal et al., 2023) over their non-augmented counterparts. In this work, we focus on sample quality and, therefore, study the efficient samplers develop in the PSLD setting.
DESIGNING EFFICIENT SAMPLERS FOR GENERATIVE DIFFUSIONS
We present two complementary frameworks for efficient diffusion sampling. We start by discussing Conjugate Integrators, generic framework that maps reverse diffusion dynamics into more suit- able space for efficient deterministic sampling. Next, we discuss Splitting Integrators, which alter- nate between numerical updates for separate components to simulate the reverse diffusion dynamics. Lastly, we unify the benefits of both frameworks and discuss Conjugate Splitting Integrators, which enable the generation of high-quality samples, even with a low NFE budget.
3.1 CONJUGATE INTEGRATORS FOR EFFICIENT DETERMINISTIC SAMPLING
Given dynamical system (e.g., the ODE Eqn. 3), the primary intuition behind conjugate integra- tors is to use invertible transformations to project the current state at time t into another space which more amenable for numerical integration. The transformation is chosen such that integration can be performed with relatively larger step size and therefore reaches solution faster. The result- ing dynamics in the projected space can then be inverted to obtain the final solution in the original space. We first define conjugate integrators before deriving a mapping that allows us to use them in diffusion model sampling.
Definition 3.1 (Conjugate Integrators). Given ODE: dz; f((1, dt, let Gn z, Zeth de- note a numerical integrator map for this ODE with step-size > 0. Furthermore, given a continuous- invertible mapping Φ [0, T) x IRd Rd such that 24 (2.).. let H.. 24th denote
3

---

Published as a conference paper at ICLR 2024
numerical integrator map for the transformed ODE in the projected space. Then the maps Gn and H4, are conjugate under Φ if,
G, = H,, di.
We provide an intuitive illustration of conjugate integrators in Fig. 1. Consequently, the iterated maps 9% and H% (where denotes the number of iterations) are also conjugate under c. Next, we design conjugate integrators for efficient deterministic sampling from diffusion models.
Conjugate Integrators for Diffusion ODEs. We develop conjugate inte- grators for solving the probability flow ODE defined in Eqn. 3. In practice, we approximate the actual score by its parametric approximation so s0 t).
9A
Following prior work (Karras et al., 2022; Salimans & Ho, 2022; Dockhorn et al., 2022b), we assume the following score network parameterization:
Pith
so (z t) Cskip (t)z: + ee) (C (t)ze) Caoise (t)).
HK
Z
We restrict the mapping o; in this work to invertible affine transformations such that 2, = A. To derive the probability flow ODE in the projected space, we reparameterize A, in terms of another mapping B [0, Ti and introduce 4, for notational convenience as follows.
Figure 1: Conjugate
Integrators (Def. 3.1)
Rd
A, exp Bf F.+.F.++,..0 Caspsssssssss Casp(()ds)
1 8. $-=- - A.A.G.G, (onns))ds (6)
where exp (-) denotes the matrix-expooentin exponential, and F; and G. are the drift and diffusion coefficients of the underlying forward process (Eqn. 1). The probability flow ODE in the projected space 2, = A, can be written in terms of these quantities.
Theorem 1. Let ze evolve according to the probubility-low ODE in Eqn. 3 with the score function purameterization given in Eqn. 5. For any mapping B [0, T x Rdd + Rd and A 4. given by Eqn. 6, the probability flow ODE in the projected space 2, AiZ; is given by
dù: = A, 2,dt + ddR (t) 22, Onoise (t))
(7)
We present the proof in Appendix B. Applying Euler update to the transformed ODE in Eqn. 7 with a step-size h > 0 yields the update rule for our proposed conjugate integrator:
2 - 2: AA 2, + (Ø,-n D, (Cin (t) A
24, Cuoise (t))
(8)
For given timestep schedule (1,) and a user-specified matrix B,, we present a complete algorithm for the proposed conjugate integrator and some practical considcrations, such as computing the coefficients in Eqn. 6 and the invertibility of A, in Appendix B.6.
Intuitively, projecting the probability- ODE dynamics into a different space introduces the ma- trix B, as additional degree of freedom that can be tuned during inference to improve sampling efficiency. In the rest of this section, we demonstrate how certain choices of B, connect to previous work and how B, be chosen further improve upon prior work.
Choice of B, and connections with other integrators. There has been a lot of recent work in accelerating diffusion models using ODE-based methods like DDIM (Song al., 2021) and ex- ponential integrators (Zhang & Chen, 2023; Zhang et al., 2022; Lu et al., 2022). We find several theoretical connections between the proposed conjugate integrator in Eqn. 8 and existing determin- istic samplers. More specifically, the following theoretical results hold for the choice of B 3
Proposition For the VP-SDE (Song et al., 2020), the transformed ODE in Eqn. 7 is equivalent the DDIM ODE proposed in Song al. (2021) (See Appendix B.2 for proof).
Proposition 2. For the diffusion model formulation considered in Lu et al. (2022), the exponential integrator proposed in DPM- Solver (Lu et al., 2022) analogous to the numerical integrator in Eqn. 8 (See Appendix B.3 for proof). More generally, for the forward process in Eqn. 1, the conjugate integrator update Eqn. is equivalent to applying the exponential integrator proposed in Zhang& & Chen (2023) in the original space 2, (See Appendix B.4 for a prooff).
These theoretical connections allow us to extend methods like DDIM and exponential integrators to novel diffusion models in principled manner. For empirical evaluation, implement DDIM
4

---

Published as a conference paper at ICLR 2024
Euler
(0urs)2.2000 220M (8: -0
( DOM (9. -)-0)
(BI (8)
DDIM
FID@55kk
FID
10g [i
10urs. (CL]) DOIM
1Ours. (C2)) DOIK
NFE
NFE
NFE
(b)
Figure 2: (Ablation) Conjugate Integrators can significantly improve deterministic sampling effi ciency in PSLD for CIFAR-10. a) The Conjugate Integrator proposed in Eqn. 8 (B, = 0) outper- forms Euler applied directly to the Prob. Flow ODE. b) Comparison between different choices of B, c) Impact of the number diffusion steps on the optimal value in X-DDIM.
for a PSLD model pre-trained on CIFAR-10. We measure sampling efficiency via network function evaluations (network function evaluations (NFE)) and measure sample quality using FID (Heusel et al., 2017). See Appendix E. for all implementation details. In Fig. 2a, we find that even with a straightforward choice of B, O (which corresponds DDIM), the conjugate integrator in Eqn. 8 significantly outperforms Euler applied to the PSLD ODE in the original space, confirming the efficacy DDIM. We next discuss other choices of B,, which help generalize beyond exponential integrators and further improve sampling efficiency.
Beyond Exponential Integrators. To derive more efficient samplers, we study conjugate integra- tors with B, = XI and B, A1 where 1 is matrix of all ones, and À is scalar hyperparameter. For fixed compute budget, we tune À during sampling to optimize for sample quality. We denote the resulting conjugate integrators as X-DDIM-I and X-DDIM-II, respectively. Empirically, in the context of PSLDD, tuning during sampling can lead to significant improvements in sampling eff- ciency (see Fig. 2b) over setting À 0 (which corresponds to DDIM exponential integrators). Moreover, we find that the optimal values of for both our choices of B; decrease in magnitude as the sampling budget increases (see Fig. 2c), suggesting that all three schemes are likely to perform similarly for a larger sampling budget. Next, we provide a theoretical justification for improved sample quality for non-zero À values using stability analysis for numerical methods.
Stability of Conjugate Integrators. Despite impressive empirical performance, is unclear why non-zero À values in X-DDIM improve sample quality, particularly large step sizes h (i.e. for a small number of reverse diffusion steps). To this end, we analyze the stability of the conjugate integrator proposed Eqn. and present the following result:
Theorem 2. Let UAU-1 denote the eigendecomposition of the matrix G.G.G (1) Bep (e)(C.(t))...)) Under invertibility of A, and certain regularity conditions (as Oz.
stated in Appendix B.5), the conjugate integrator defined in Eqn. 8 is stable if the eigenvalues À of the matrix A = - -U-BBU satisfy [1 + hX] < 1. (See Appendix B.5 for proof)
Corollary 1. X-DDIM-I is stable if h(ā A) where X A.
In the context of X-DDIM-I, the result in Corollary implies that tuning the hyperparameter conditions the eigenvalues of A during sampling. This results in more stable integrator, likely leading to good sample quality even for large step size h. In contrast, setting À = 0 disables this conditioning, leading to worse sample quality if the eigenvalues Ả are not already cell-conditioned.
Discussion. In this section, we introduced Conjugate Integrators for constructing efficient deter- ministic samplers for diffusion models. In addition to establishing connections with prior work on deterministic sampling, we propose novel conjugate integrator, X-DDIM, that generalizes samplers based on exponential integrators. Lastly, we provide theoretical results that justify the effectiveness of the proposed sampler. However, while we apply the Euler method to the transformed ODE in Eqn. 7, other numerical schemes can also be used. Consequently, our result in Theorem 2 is specific to this case, and leave deriving similar results for other integrators applied to the transformed ODE in Eqn. 7 as future work. Lastly, while X-DDIM-II (Fig. 2b) performs the best, further exploration of better choices of B, also remains interesting direction for future work.
5

---

Published as a conference paper at ICLR 2024
(Ours, [51] N51
EM
OOuss (52))
(Curs.
(Ours.
FID@
(Ours, (54J) RV
IOurs,
rOurs, 520 NVV
NFE
NF E
NFE
(a)
(h)
(c)
Figure 3: (Ablation) Splitting Integrators significantly improve deterministiicstochaste sampling efficiency PSLD for C1F a) Naive ODE splitting samplers outperform Euler by large margin. b) Reduced ODE splitting samplers outperform naive schemes. c) Reduced SDE splitring samplers outperform other baselines.
3.2 SPLITTING INTEGRATORS FOR FAST ODE AND SDE SAMPLING
We bring another innovation for faster sampling to generative diffusion models. The methods de- scribed here are complementary to conjugate integrators, and in Section 3.3, we will study their combined strength. Splitting integrators are commonly used design symplectic numerical solvers for molecular dynamics systems (Leimkuhler, 2015) which preserve a certain geometric property of the underlying physical system. However, their application for fast diffusion sampling still underexplored (Dockhorn al., 2022b). The main intuition behind splitting integrators is to split ODE/SDE into subcomponents which are then independently solved numerically (or analytically). The resulting independent updates then composed specific order obtain the final solution. Splitting integrators are particularly suited for augmented diffusion models since they can leverage the split into position and momentum variables for faster sampling. We provide brief introduc- tion to splitting integrators in Appendix 1 and refer interested readers to Leimkuhler (2015) for detailed discussion.
Setup: We use the same setup and experimental protocol from Section 3.1 and develop splitting integrators for the PSLD Prob. Flow ODE and Reverse SDE. Though our discussion is primarily focused on PSLD, the idea of splitting is general and can also be applied to other types of diffusion models (Wizadwongsa Suwajanakorn, 2023; Dockhorn et al., 2022b).
Deterministic Splitting Integrators. We choose the following splitting scheme for the PSLD ODE,
dx; dm:
3
Txx - M-1m,
m, + TTj((, T - 1)) dt + 2 x, vm, + Mus (z. T - dt,
B
where XT- m. mr-t. 8 and s denote the score components in the data and momentum space, respectively. Given step size h, we denote the Fuler updates for the components A and B as Ct. and CR respectively. Consequently,
propose composition schemes namely, [BBA) CA C and c[BAB] 2h o C Ch where h/2 denotes an update with half-step. We denote the samplers corresponding to these schemes as Naive Symplectic Euler (NSE) and Naive Velocity Verlet (NVV), respectively (see Appendix C.2.1 for exact numerical updates). While the motivation behind the notation "naive" will become clear later, even direct application of our naive splitting samplers can lead to substantial improvements in sample efficiency over Euler (see Fig. 3a). This is intuitive since, unlike Euler, the proposed naive samplers alternaee between updates in the momentum and the position space, thus exploiting the coupling between the data and the momentum variables. We formalize this intuition the following result.
Theorem 3. Given step size h, the NVV sampler has local truncation errors with orders O(Th?) and O(ui)) in the position and momentum space, respectively (See Appendix C.2.4 for proof).
Since the choice in PSLD is usually comparable to the step size (Pandey & Mandt, 2023), the local truncation error for the NVV sampler in the position space is usually O(h3). However, Fig.

---

Published as a conference paper at ICLR 2024
3a also suggests that naive splitting schemes exhibit poor sample quality at low NFE budgets. This suggests the need for a deeper insight into the error analysis for the naive schemes.
Therefore, based on local error analysis for ODEs, propose the following improvements to naive samplers.
We reuse the score function evaluation between the first consecutive position and the momentum updates in both the NSE and the NVV samplers.
Next, for NVV, we use the score function evaluation sø *x(xth m4+h/2, T (t + h)) in the last update step instead.
Consequently, we denote the resulting samplers as Reduced Symplectic Fuler (RSE) and Reduced Velocity Verlet (RVV), respectively (see Appendix C.2.2 for exact numerical updates). Though both the naive and the reduced schemes have the same convergence order (see Appendix C.2.5), the reduced schemes significantly improve PSLD sampling efficiency over their naive counterparts (Fig. 3b). This is because our proposed adjustments serve two benefits: Firstly, the number of NFEs per update step is reduced by one, enabling smaller step sizes for the same sampling budget. This reduces numerical error during sampling. Secondly, our proposed adjustments lead to the cancellation of certain error terms, which is especially helpful for large step sizes during sampling (see Appendix C.2.4 for a theoretical analysis).
Stochastic Splitting Integrators. Analogously, we can also apply splitting integrators to the PSLD Reverse SDE. Based initial experimental results, we use the following splitting scheme.
dx: dm,
M- m + 2T (21,1) dt +0 2Tx,
B 2
0
xx + 2/m + -2Mvs (Z., dt.
2
0
(
ST X, + dBldw,
where 0 =
Bu m, + /Muwwd represents the Ornstein-Uhlenheck process in the joint
space. Among several possible composition schemes, we found the schemes OBA, BAO, and OBAB to work particularly well. We discuss - [(OBA] C o C C. which we denote as Naive OBA (NOBA), in more details here and defer all discussion related to other schemes to Appendix C.3. Analogous to the deterministic setting, we propose several adjustments over the naive scheme.
We reuse the score function evaluation between the position and the momentum updates, which leads to improved sampling efficiency over the naive scheme (Fig. 3c).
Next, similar to Karras et al. (2022), we introduce a parameter X₂ in the position space update for Lo to control the amount of noise injected in the position space. However, adding a similar pa- rameter in the momentum space led to unstable behavior and, therefore, restricted this adjustment to the position space.
With these adjustments, denote the resulting sampler as Reduced OBA (ROBA) (see Appendix C.3.3 for full numerical updates). Empirically, the RÓBA sampler with a tuned X, outperforms other baselines by significant margin (see Fig. 3c).
Discussion. In this section, we presented Splitting Integrators for constructing efficient deterministic and stochastic samplers for diffusion models. We construct splitting integrators with alternating updates in the position and momentum variables, leading to higher-order integrators. However, naive application of splitting integrators can be sub-optimal. Consequently, we propose principled adjustments for naive splitting samplers, which lead to significant improvements. However, a more principled theoretical investigation in the role of X remains an interesting direction for future work.
3.3 COMBINING SPLITTING AND CONJUGATE INTEGRATORS
In the context of Splitting Integrators, so far, we have used Euler for numerically solving each splitting component. However, in principle, each splitting component can also be solved using more efficient numerical schemes like Conjugate Integrators discussed in Section 3.1. We refer to the latter as Conjugate Splitring Integrators. For subsequent discussions, we combine the X- DDIM-II conjugate integrator proposed in Section 3.1 and the reduced splitting samplers discussed

---

Published as a conference paper at ICLR 2024
ESGE
[bus]] Sis
[SSH] COBA.
(Cumi SPS4 (=Reccod))
[50g ASE
FIN 50k
10uns ICSED ose
Ould Sul dro 101
NFE
NFE
NFE
NFE
(b)
(c)
(d)
Figure 4: (Ablation) a) Conjugate-spliting samplers outperform their reduced counterparts for de- terministic sampling. b) For stochastic sampling, however, using conjugate-spliting samplers incur slight degradation in sample quality over the reduced scheme. (c, d) Impact of preconditioning on sample quality for the proposed ODE (Left) and SDE (Right) samplers at low sampling budgets.
in Section 3.2. Consequently, we denote the resulting deterministic samplers as Conjugate Velocity Verlet (CVV) and Conjugate Symplectic Euler (CSE) corresponding their reduced counterparts. Similarly, we denote the resulting stochastic sampler as Conjugate OBA (COBA).
Conjugacy in the position vs. momentum space. Our initial empirical results indicated that applying conjugacy in the position space yields the most significant gains in sample quality. This might be intuitive since, during reverse diffusion sampling, the dynamics in the position space might be more complex due to a more complex equilibrium distribution. Therefore in this work, we apply conjugacy only in the position space updates (see Appendix for full update steps).
Empirical Evaluation. Fig. 4a illustrates the benefits of using the proposed conjugate-splitingg samplers, CVV and CSE, over their corresponding reduced schemes for deterministic sampling on the unconditional CIFAR- 10 dataset. Notably, the proposed CVV sampler achieves an FID score of 2.11 within a sampling budget of 100 NFEs which is comparable to the FID score of 2.10 reported in PSLD (Pandey & Mandt, 2023), which requires 242 NFE, For stochastic sampling, we find that applying conjugate integrators to the ROBA sampler slightly degrades sample quality (see Fig. 4b) for CIFAR-10. However, the benefits of the latter are more prominent for higher-resolution datasets (see Section 4), indicating the scalability of the proposed sampler.
ADDITIONAL EXPERIMENTAL RESULTS
Ablation Summary. We summarize our ablation samplers presented in Section 3 in Table 14. In short, we presented Conjugate Integrators in Section 3.1, which enable efficient deterministic sampling in PSLD (Fig. 2). Next, we presented Reduced Splitting Integrators for faster deterministic and stochastic sampling in PSLD (Fig. 3). Lastly, we combined the two frameworks for further gains in sampling efficiency (Fig. 4). We now present additional quantitative results and comparisons with prior methods for faster deterministic and stochastic sampling.
Notation. For simplicity, we denote our best-performing Reduced Splitting and Conjugate Splitting PSLD Sampler (SPS) and Conjugate Splitting-basee PSLD Sampler (CSPS), respectively. Consequently, we refer to the Deterministic RVV and Stochastic ROBA sam- plers as SPS-D and SPS-S, and their conjugate variants as CSPS-D and CSPS-S, respectively.
integrators Splitring-bar
Datasets and Evaluation Metrics. We use the CIFAR-10, 664 (Liu et al., 2015) and the AFHQ-v2 (Choi et al., 2020) datasets for comparisons. Unless specified otherwise, we report FID for 50k generated samples for all datasets and quantify sampling efficiency using NFE. We include full experimental details in Appendix E.
Baselines and setup. In addition to samplers based on exponential integrators like DDIM (Song et al., 2021), DEIS (Zhang & Chen, 2023) and DPM-Solver (Lu et al., 2022), we compare our best ODE and SDE samplers with PNDM (Liu et al., 2022), EDM (Karras et al., 2022), SA-Solver (Xue et al., 2023) and Analytic DPM (Bao et al., 2022). We provide a brief description of these baselines in Table 1. While the techniques presented in this work generally apply other types of diffusion models, we compare the empirical performance of our proposed samplers for PSLD with the highlighted baselines for completeness. Lastly, we find that, similar to prior works (Dockhorn
8

---

Published as a conference paper at ICLR 2024
CIFAR-10 (ODE)
CIFAR-10 (SDE)
SA-Solver
S-PNDM
(VE)
F-PNON
DDIN
4G2
[Duns] 575-5
FID&50K
DPM Solver
[Ouns] SPS Pro))
EDM (V9)
(Ouns)
NFE
NFE
CefebA-664
ARHQQ2-64
S-PND
EDM (VP, ODE)
EDM IVE ODE)
(Ours) SPS-S (+Pre.)
[Durs]
[Duno CSPS-5 (+Pre)
(Ouns)
(0uns) CSPS (1P).))
NFE
NFE
Figure 5: Extended results for Table 1. Our proposed samplers perform comparably or outperform other baselines for similar NFE budgets for the CIFAR-10, CelebA-64, and AFHQv2 datasets.
et al., 2022b; Karras et al., 2022), score network preconditioning leads to better sample quality at low sampling budgets for both deterministic (Fig. 4c) and stochastic sampling (Fig. 4d). For in- stance, CSPS-D achieves an FID score of 2.65 in NFE=50 with preconditioning as compared to 3.21 without. We provide full technical details for our preconditioning setup Appendix E.3. Conse- quently, we report empirical results for our ODE/SDE samplers with and without preconditioning for CIFAR-I0 and with preconditioning for other datasets.
Empirical Observations: For CIFAR-10, our ODE sampler performs comparably outperforms all other baselines for NFE 2 50 (Fig. 5, Top Left). Similarly, our SDE sampler outperforms all other basclines for NFE 2 40 (Fig. 5, Top Right). We make similar observations for the CelebA-64 and AFHQv2-64 datasets, primarily in the high NFE regime (See Fig. 5, Bottom Left). Therefore, our proposed samplers for PSLD are competitive with recent work. Interestingly, for all datasets, our stochastic sampler achieves better sample quality for low sampling budgets (NFE < 50) as compared to our deterministic sampler. Lastly, in contrast to CIFAR-10, find that the CSPS-S sampler works better than the SPS-S sampler for the CelebA-64 and AFHQy2-64 datasets, indicating its effectiveness for higher-resolution sampling.
DISCUSSION
Contributions. We have presented complementary frameworks, Conjugate and Splitting Inte- grators, for efficient deterministic and stochastic sampling from broader class of diffusion models. Furthermore, we combine the two frameworks and propose Conjugate Splitting Integrators for fur- ther improvements in sampling efficiency in the context of PSLD. The resulting samplers perform comparably with several recent approaches for fast diffusion sampling (see Table 1, Fig. 5).
Future Directions. While the framework presented in this work can serve as a good starting point for designing efficient samplers for diffusion models, there are several promising future directions. Firstly, extending our framework of conjugate integrators to design efficient stochastic sampling methods would interesting. Secondly, our current choice the core design parameters in conju- gate integrators (B,) is mostly heuristical and, therefore, requires further theoretical investigation. In the context stochastic sampling, firstly, we find that empirically controlling the amount of stochasticity injected during sampling largely affect sample quality. Therefore, further inves- tigation into the theoretical aspects of optimal noise injection in diffusion model sampling can be an interesting direction for future work. Lastly, hope that presentation of reduced splitting integrators for fast deterministic and stochastic sampling in diffusion models can serve as a good initial starting point for further research in exploring more efficient splitting techniques.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
KP acknowledges support from the Bosch Center for Artificial Intelligence and the HPI Research Center in Machine Learning and Data Science UC Irvine. SM acknowledges support from the National Science Foundation (NSF) under an NSF CAREER Award, award numbers 2003237 and 2007719, by the Department of Energy under grant DE-SC0022331, the IARPA WRIVA program, and by gifts from Qualcomm and Disney.
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models.
Stochastic Processes and doi: https:/doi.org/10
their Applications, 12(3):313332, 1982. ISSN 0304-4149.
10160030--411906290001--1 URL https:/ http:://wwwwwccineee.m wec.. http///eeeeree.....cccoeeceeeeee wtt.:/oc/cccccceee atticcle/ppt//00041144999933
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an analytic estimate of the opti- mal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations, 2022. URL https: http:///pppeeee...emmm net forum? 1d-OXiLKK uuf.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021. URL https:
Ricky T. Q. Chen. torchdiffeq, 2018. torchdiffed ffeq.
URL http::////gttuu..co//trete
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8188-8197, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 35:30150-30166. 2022a.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In International Conference on Learning Representations, 2022b.
URL httph:tt//// / wet.anerrwweeeeeeoooorieeeeeeeee /o
net
J.R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Compu- tational and Applied Mathematics, 6(1):19-25 26, 1980. ISSN 0377-0427. doi: https://doi.oom org/ 10.1016/0711-650X888890003333333 URL https: //ww://wwwwwww www.ssceiieeeerr......... com/ science/ atticcle/ssA/071100000099000
Martin Gonzalez, Nelson Fernandez, Thuy Tran, Elies Gherbi, Hatem Hajri, and Nader Masmoudi. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models, 2023.
E. Hairer, P. Nørsett, and G. Wanner. Solving Ordinary Differential Equations I (2nd Revised. Ed.): Nonstiff Problems. Springer- Verlag, Berlin, Heidelberg, 1993. ISBN 0387566708.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35: 27953-27965, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68511 2020.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23 (47):1-33, 2022a.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In ICLR Workshop on Deep Generative Models for Highly Struc- tured Data, 2022b. URL https: openreview net forum? d=BBe R2NdD25.
Alexia Jolicoeur-Martineee Ke Li, Rémi Piché- Taileefa,,,,eer Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080 2021a.
Alexia Dolicoeur-Martinen. Rémi Piché-Taillefer Ioannis Mitliagkas, and Remi Tachet des Combes, Adversarial score matching and improved sampling for image generation. In Interna- tional Conference on Learning Representations, 2021b. URL https: /ttp:://ppeeeeeeeeeeeeee net./
forum? id=eefgM12eLLL33
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-2661 265777, 2022.
Peter Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Springer Berlin Heidelberg, 1992. doi: 10.1007/978-3-662-1200 12611-6 URL /ttps://doi..
org/10..107/9772 0-199978--6622-26666-- 122212616--262
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL
hitpps:/www... www.ss.trooooo.ooooooooo edu/- serrii/eaannnngeeeee features wew.eeeeeee00------- 2000-TRR.... pdÉ.
Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. In International Conference on Learning Representations, 2021.
B. Leimkuhler. Molecular dynamics: with deterministic and stochastic numerical methods / Ben Leimkuhler, Charles Matthews. Interdisciplinary applied mathematics, 39. Springer, Cham, 2015. ISBN 3319163744.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In International Conference Learning Representations, 2022. URL https: / ///penreview. wete/forumiideeeeeeeeee PLWVdddyy.KKV.y....
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2 22////00.0238 2021.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Anton Obukhov, Maximilian Seitzer, Po-Wei Wu, Semen Zhydenko, Jonathan Kyl, and Elvis Yu- Jing Lin. High-fidelity performance metrics for generative models in pytorch, 2020. URL ht https: / thub. com/ odo.4957735
lit.. Version: 0.3.0. DOI: 10.5281/zen-
Kushagra Pandey and Stephan Mandt. Generative diffusions in augmented spaces: A complete recipe, 2023.
11

---

Published as a conference paper at ICLR 2024
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents, 2022. URL https: pttxps://axxiv.oo/ org/ abs/ 2204. 06125.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. volume 35, pp. 36479-36494, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL https: openreview..eeeeeee fortt hawprummddddIddIIi-aaaaa-
Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where diffuse, how diffuse, and how to get back: Automated learning for multivariate diffusions. In The Eleventh International Conference on Learning Reprexentations, 2023. URL pattts://opeereeeee id=osei3I2Uia,
net forum?
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn ing. pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffiusion implicit models. In Interna- tional Conference on Learning Representations, 2021. URL Forum?id= Serrumid==Stgiaaan www.ddssstgiarrcH.....
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
H. F. Trotter. On the product of semi-groups of operators. Proceedings of the American Math- ematical Society, 10(4):545-55, 1959. doi: 10.10900/:0002-99330 1910000--999911111-1000833333 1959-0108732-6. URL https: //dos: doi. /1c. 1090/ s0002 9939 00009090000----1 5.
Loup Verlet. Computer "experiments" on classical fluids. i. thermodynamical properties of lennard-
jones molecules. Phys. Rev. 159:98-103, Jul 1967. doi: 10. 00.110//hysRee.159..88 URL https: ///innk.a... aps http:////ooa/o/e/eee
Bat000//hhyySeee htt.:///eeeeeeee..9...... 15..98.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, 23(7): 166--167, 201 doi: 1162/NECO...001..
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis I Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero,
12

---

Published as a conference paper at ICLR 2024
Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt. and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 10.10388/441522-09--0060
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning efficiently sam- ple from diffusion probabilistic models. arXiv preprint arXiv:2106.0388 03800, 2021.
Suttisak Wizadwongsa and Supasorn Suwajanakorn. Accelerating guided diffusion sampling with splitting numerical methods. In The Eleventh International Conference on Learning Representa- lions, 2023. URL
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang. Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models, 2023.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation, 2022. URL https::t///..... https::/arxxi..ottttt abs /2003. 09481.
Haruo Yoshida. Construction of higher order symplectic integrators. Physics Letters A, 150(5):262- 268, 1990. ISSN 0375-9601. doi: http:///dbooggg/////ooo/////1119 /ww. sc endeidediiii rect. com
URL https:
90900923.
Qinsheng Zhang and Yongxin Chen. Fast sampling diffusion models with exponential integrator. In The Eleventh International Conference Learning Representations, 2023. URL https:
/
fhet:/oorumiddi...m d=Loek fhfb466 6P.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.055:5 2022.
Richard Zhang. Making convolutional networks shift-invariant again. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7324-7334, PMLR, 09-15 Jun 2019. URL https: http://tttceeee.n.eeeee .mlr www..prres.....ee.ommm.m.mmme.ee www.ppresss.... /v..ss/v9//0 zhangl 9a html.
13

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
2
Background
3
Designing efficient Samplers for Generative Diffuusions
3.1
Conjugate Integrators for efficient deterministic Sampling
3.2 Splitting Integrators for Fast ODE and SDE Sampling
6
3.3
Combining Splitting and Conjugate Integrators
4
Additional Experimental Results
5
Discussion
Related Work
15
Conjugate Integrators for Faster ODE Sampling
15
B.1 Proof of Theorem
15
B.2 Proof of Proposition 1: Connection with DDIM
16
B.3 Proof of Proposition 2: Connections with DPM-Solver
17
B.4 Proof of Proposition 2: Connections with DEIS
18
B.5 Proof of Theorem
20
B.6 Conjugate Integrators in the Wild
22
Splitting Integrators for Fast ODE/SDE Sampling
23
C.1 Introduction to Splitting Integrators
23
C.2 Deterministic Splitting Integrators
................................
C.2.1 Naive Splitting Samplers
24
C.2.2 Reduced Splitting Samplers
24
C.2.3
Local Error Analysis for Deterministic Splitting Integrators
25
C.2.4 Error Analysis: Naive Velocity Verlet (NVV)
26
C.2.5 Error Analysis: Reduced Velocity Verlet (RVV)
30
C.3
Stochastic Splitting Integrators
33
C.3.1
Naive Splitting Samplers
C.3.2 Effects of controlling stochasticity
34
C.3.3 Reduced Splitting Schemes
34
D Conjugate Splitting Integrators
35
D.1 Deterministic Conjugate Splitting Samplers
35
D.2 Stochastic Conjugate Splitting Samplers
37
14

---

Published as conference paper at ICLR 2024
E
Implementation Details
38
E.1 Datasets and Preprocessing
38
E.2 Pre-trained Models
38
E.3 Score Network Preconditioning
................................
E,4
Evaluation
38
Extended Results
40
F.1
Extended Results for Section 3.1: Conjugate Integrators
40
F2
Extended Results for Section 3.2: Splitting Integrators
40
F33
Extended Results for Section 3.3: Conjugate Splitting Integrators
41
F.4
Extended Results: Impact of Preconditioning
F55
Extended Results for Section 4: State-of-the-art Results
42
A RELATED WORK
In addition to the recent work based on exponential integrators (Zhang Chen, 2023; Lu et al., 2022; Zhang et al., 2022; Song et al., 2021), PNDM (Liu et al., 2022) re-casts the sampling pro- cess in DDPM (Ho et al., 2020) as numerically solving differential equations on manifolds. Ad- ditionally, Karras et al. (2022) highlight and optimize several design choices in diffusion model training (including score network preconditioning. improved network architectures, and improved data augmentation) and sampling (including improved disrretietii schedules), which leads to significant improvements in sample quality during inference. While this is not our primary focus, exploring these choices in the context of other diffusions like PSLD (Pandey & Mandt, 2023) could be an interesting direction for future work. Other works for faster sampling have also focused on using adaptive solvers Uolicoeur-Martingg et al., 2022), and optimizing timestep schedules (Watson et al., 2021). Though prior works have fo- cused mostly on speeding up deterministic sampling, there have also been some recent advances in speeding up stochastic sampling in diffusion models (Karras et al., 2022; Xue et al., 2023; Gonzalez al., 2023).
al., 2021a). optimal variance during sampling (Bao
Splitting integrators extensively used in the design of symplectic integrators in molecular dy- namics (Leimkuhler, 2015; Yoshida, 1990; Verlet, 1967; Trotter, 1959). However, their application for efficient sampling in diffusion models is only explored by few works (Dockhorn et al., 2022b; Wizadwongsa Suwajanakorn, 2023). In this work, in the context of PSLD, we show the structure in the diffusion model ODE/SDE can be used to design efficient splitting-basse samplers. However, as shown in this work, naive application of splitting integrators can be sub-optimal for sample quality, and careful analysis might be required to design splitting integrators for diffusion models.
Lastly, another line of research for fast diffusion model sampling involves additional training (Song al., 2023; Dockhorn al., 2022a: Salimans & Ho, 2022: Meng et al., 2023; Luhman & Luhman, 2021). In contrast, our proposed framework does not require additional training during inference.
B CONJUGATE INTEGRATORS FOR FASTER ODE SAMPLING
B. PROOF OF THEOREM 1
We restate the full theorem for completeness.
Theorem. Let z, evolve according to the probability-foow ODE Eqn. with the score function parameterization given in Eqn. 5. For any mapping B [0. Rd Rdd and A,, D given by Eqn. 6, the probability flow ODE in the projected space 2, = A,Z, is given by
dà, = A, A.B.A 12,l + ddb.5n (Cin(1) A 14,, Caoise (1)
(9)
15

---

Published as a conference paper at ICLR 2024
The forward process for diffusion with affine drift can be specified as:
dzp Fiz; dt + G. dw.-
(10)
Consequently, the probability flow ODE corresponding to the process in Eqn. 10 is given by:
1 G.G. so(z..t) dt. dz: Fize -00--2G0
(11)
Furthermore, the score network is parameterized as follows:
80 (24, 1) (()zp(()2: + Comn())e(CCin()))) Cnoise
(12)
Substituting the score network parameterization in Eqn. 11, we have the following form of the probability flow ODE:
dzr Fiz. 2 G.G.T Caip(t)z; Com(t))e(Cinn(t)aa Croise (t))
(13)
F. G.G. Csii(()) 2t G
Gnoise(t)
(14)
2
Given an affine transformation which projects the state z, to 2,,
2, A,2;
(15)
Therefore, by the Chain Rule of calculus,
dà,
dA,
A dzi
24 +
(16)
dt
dt
dt
Substituting the ODE in Eqn. 14 in Eqn. 16,
dē.
dA.
[ Ft G.G.T Cupp(t)
1
21 + A₁
21 2
G.G.G
Cnoise (t))
dt
dt
2
(17)
dAi
+ Ar
F;
G.G.T Caip(t))
21
A.G.G Com(t)) Cnoise (t))
dl
(18)
d.A,
1 G.G.G )
+
Ai
A A - Coul)eCCC '2,, Cnoise (t))
dt
(19)
We further define the matrix coefficients B, and 4, such that,
dA,
G.G. Cuap(t)) A,
Ar
F, 2
(20)
dt
dΦ
1
A,G.G, Coui(t)
(21)
dt 2
which yields the required diffusion ODE in the projected space:
dž,
dΦ; Eo (Ci 2r, Cuvise (1))
A,B,A, + dt
(22)
dt
B.2 PROOF OF PROPOSITION 1: CONNECTION WITH DDIM
Proposition. For the VP-SDE (Song et al., 2020), for the choice of B, = O. the transformed ODE in Eqn. 7 corresponds to the DDIM ODE proposed in Song et al. (2021)
16

---

Published as a conference paper at ICLR 2024
Proof. The forward process for the VP-SDE (Song al., 2020) is given by:
dz: = - 2 d2 + VB,dw,
(23)
where Bg determines the noise schedule. This implies F; - - B and G. v BLI. Further-
more, the score network in the VP-SDE is often parameterized as so s0(2,, 1) -60 1)/a, where of is the variance of the perturbation kernel p(z/ It follows that for VP-SDE,
Ssip (t) = O, (t oi
(t) Id: Caoise (f) = t.
(24)
Setting B, O. we can determine the coefficients A, and Dr as follows:
dA. dt
1 G.G. Casip(t)
A,Br dA, 1
+ A.
F 2
3eA
(25)
dt 2
0 Bads Id
A: exp
(26)
Similarly,
dΦ; 2
1
% Bdds
Be Ia
dt
G,G,G Coult)) = exp
2 5
(27)
Since the variance of the perturbation kernel p(x(|xo) is given by of we can reformulate the above ODE as:
[] exp
(- B.ds)
d4:
Bt
1a
(28)
dt
20
/ of
Consequently, the ODE in the transformed space can be specified as:
dē. dt
dΦ;
A,B,A, 2,
dt co (Cin (t) 22,, Cnoise (t))
(29)
Be
(V1- (VI-ottan ofbert)
Eo
(30)
20, 1 oi
Defining %t = 01/ /I o7. it can be shown that, d%
9.
dt. Therefore, reformulating the
20, / 1-
ODE in Eqn. 30 in terms of %t.
dž, d:
co
V1
(31)
7
which is the DDIM ODE proposed Song al. (2021). Therefore for the VP-SDE and the choice of B, = 0, the proposed conjugate integrator is equivalent to the DDIM integrator.
B.3 PROOF OF PROPOSITION 2: CONNECTIONS WITH DPM-SOLVER
Proposition. For the diffusion model formulation considered in Lu et al. (2022), the exponential integrator proposed in DPM-Solver (Iu et al., 2022) is numerical integrator for the transformed ODE in Eqn. 8 (See Appendix B.3 for proof)
Proof. For simplicity, we restrict the parameterization of the score estimator to so (Rt, L) = --, where L, the Cholesky decomposition of the variance E,, of the perturbation kernel. This implies,
Cakip (1) = O, Coun (1) = LL (t) = Ids Cnoise (1) = L.
(32)
Furthermore, for the choice of B, = 0, the transformed ODE simplifies to,
dž, ceo (A.)2
(33)
da, 2 A.G.G. L, L,TT,,oo TT (o(AAt2n))
(34)
17

---

Published as a conference paper at ICLR 2024
It follows that for any two timepoints t and 8, have,
1 2: = 2. A,G.G, EO E, 2,, T)dT
(35)
A,2: A,2. 1 L L,G.G, Eo T
(36)
Ai
2 [ L, ( 27, T)
2 A A +
(37)
Moreover, since B, = 0. we have,
dA.
A. Fr 1 G.G. Cskip(t) Ar
(38)
dl
dAc
dAa,
+ A,F: = 0 F. AA,-
(39)
dt
dt
Next, we consider diffusions of the form,
dz: = (11)z,dl + g(()dw:
(40)
where,
f(1) dlog at. dt
g²(1) dop d dlog at dt dl oi.
(41)
Here atf) and (t) are differentiable functions defining the diffusion process's noise schedule. Moreover, for this process, the score is usually parameterized as s0(2,,
cz(2.)) implying
de
L, in our parameterization corresponds to o, Moreover, comparing the drift scaling factors in Eqns. 39 and 41, it follows that a, 1/a, where A, a,Ls- Therefore, the integrator in Equ. 37 can be re-written as,
at 2
a4
gh a co(2,, T)dT
2,
(42)
a2
Lastly, defining A = log dt it can be shown that of = 20, Eqn. 42, we get the following integrator,
dd. di
Substituting this result for of in
at 2o 04
di or co(z,, r)
Z
(43)
a
dr a,
Applying change of variables in Eqn. 43 from to X, get the exponential integrator in DPM- Solver (Lu et al., 2022) as follows:
Gt 2, - aq / e (2)
de
2,
(44)
0g
which concludes our proof.
B.4 PROOF OF PROPOSITION 2: CONNECTIONS WITH DEIS
Proposition. More generally, for any diffusion model specified in Eqn. 1, the conjugate integrator update Eqn. is equivalent to applying the exponential integrator proposed Zhang Chen (2023) in the original space 2₁. Moreover, using polynomial extrapolation in Zhang & Chen (2023) corresponds to using the explicit Adams- Adams-Bashfone Bashforth solver for the transformed ODE in Eqn. 7.
Proof. For simplicity, we restrict the parameterization of the score estimator to sø (zr, 1) = L where L, is the Cholesky decomposition of the variance E, of the perturbation kernel. This implies,
Cakip (t) 0, Cou (t) L, (tn()) Id, Cnuise (t)
(45)
Furthermore, for the choice of B, = 0, the simplified transformed ODE can be specified as:
d². d (A, 1),
(46)
18

---

Published as a conference paper at ICLR 2024
Subsequently, the update rule for the proposed conjugate integrator reduces to the following form:
- 24 + ( D1)E0 (A, 21: t)
(47)
where,
dA.
+ A
(48)
dt
4 2 /
A.G.G.L,, ds
(49)
Transforming the update rule in Eqn. 47 back to the original space.
22. - = 2, + (+i-h D.)co (A, (A,'2 t)
(50)
A - = A,z + (1- -eicco (A, 2..t)
(51)
Pre-multiplying with A both sides and substituting the value of Db from Eqn. 49
h = A, + A ( - D,)E0 (A,'2,,t)
(52)
(
1 A
A.G.G...L Lo T ds 1 A.G.G., L, Tds 60 (A,"2r) 1)
- A.
2
(53)
( A.G.G.,,, ds co (Ai'2,,t)
Zi-h = A, + 1
A
(54)
1 A. 11..A11AAAA.GGGG.G T ds Eo (A,'
A Aiz
(55)
2
Defining (), = AT A., we can rewrite the update rule in Eqn. 55 as follows:
(f.- stt ((t-hh s)t-h,)),.GI1, ds co (A 2),
1
Zi-h = bbt h, f); + 2
(56)
The update rule in Eqn. 56 is the same as the exponential integrator proposed in Zhang & Chen (2023); Zhang et al. (2022). Furthermore, Zhang & Chen (2023) proposes using polynomial extrap- olation speed up the diffusion process further. We next show that using polynomial extrapolation is equivalent to applying the explicit
method to the transformed ODE in Eqn. 46.
Explicit adams-Bashfone applied to the transformed ODE: Given the transformed ODE Eqn. 46, it follows that,
2. =1, + 1.¹ dd,50
(57)
As done in the explicit Adams-Bashforte method, we can approximate the integrand eo (A, 2,, 8) by a polynomial (s(ss) with degree r. As an illustration, for = 1, we have (s = Co + ci (s -;), where the coefficients Co and e₁ specified as,
Co = Eg tt))
ci = t t EE 1 21
t--1) - Eo (AT 2,, t (58)
Therefore we have the polynomial approximation (s(s) for eo (2, as,
(s( E0
AA 12)
8 -
E0
A 2eg 1 1)lj11) ED (Ar)2e
(59)
1/-1 - 1j
1 1 Eo
s s- t
(1, to) +
s s-tz tj
Eo (Aij) 21
(60)
1--1 tú
19

---

Published as a conference paper at ICLR 2024
In the general case, the polynomial (s(s) can be compactly represented as,
Pr(s) - Σ Cu(s)Eo (Ai beg ( kn)) C2(s) II
(61)
l 1-
Therefore, replacing the integrand 60 (,, 8) by its polynomial approximation P. we have:
{" d
2, = 21, + 11
Pr(s)
(62)
21. 21, + /
dΦ 9
CR(S)Eo (
(63)
k=
21, 21," +
Σ
d Cx(s)
Eo (Ai) (1 -ey=
(64)
k=0
21: 00-0,, 80,
Σ
L. ED (Ai, (0 tj
(65)
Ar, A Σ
L, Eo (Z()_entt)x)
(66)
Z1: A Ati Rtg
+
Σ
2 A A
L,G.G,
L, Cu(s)ds 60 (2
(67)
k=0
Z. (t,,,t)))z
2 G.G. T. cØ (z
(68)
2
which is the required exponential integrator with polynomial extrapolation proposed in Zhang & Chen (2023). Therefore, applying Adams-Bar Bashf-Bashfotth in the transformed ODE in Eqn. 46 corresponds to polynomial extrapolation in Zhang & Chen (2023).
B.5 PROOF OF THEOREM 2
We restate the full statement of Theorem here (with regularity conditions) as follows.
Theorem. Let Fr and G. be the flow maps induced by the transformed ODE d²,
dΦ, co (Cin(t)A, 22, Cnoise (t))
A,B,A, 2, +
(69)
dt
dt
and by the conjugate integrator defined as 2 = 2, - b,,,BA,, 2, + (# - Di)co (Cin(tA, 2,, Cnoise (t))
(70)
respectively. We define two points, ²(t) and 2., sampled from and g respectively at time t such that (]²(t) 2.1 < 5 for some 5 > O. Furthermore, let UAU denote the eigendecomposition
of the matrix G C.ar")tttt)) @ea(C.z..)) The conjugate integrator defined in Eqn. 70 stable 8z.
+ h < 1, where Ā denotes the eigenvalues of the matrix À
B.U.
Proof. We denote the conjugate integrator numerical update defined in Eqn. 70 by G.. Therefore, for this integrator to be stable, we need to show that,
G (2())) - Gr (2.) s A
A>0
(71)
i.e., two nearby solution trajectories should not diverge under the application of the numerical update in each step. Next, we compute Gr (ε())) as follows:
Gn (2(()) 2(t) hA, A, ²tt) + (i.-A D.)co (Cin(t),, 2(t), Cnoise (t))
(72)
db,
2(() - KA, ((()
Ea (C,n 2()), Caoise (1)) + 0
(73)
di
20

---

Published as a conference paper at ICLR 2024
where we have used the first-order taylor series approximation of Φi-h in the above equation. Sub-
d4, stituting di - AAG,G, Cou(t) in the above equation and ignoring the higher order terms Ohh)), we get,
+ h A.G.G. Com Co (t)co (t)), Cuoise (t))
(x())) - (,00)))).,47
(74)
2
h
= =2(t) - -A, 2(t) A.G.G. (t)ee) (Cin (t) (t))s
(75)
Similarly, (2,) can be computed follows:
(2.) = 2, h, 24 2
A.G.G. Com (Cin (f)ze, Cnoise (t))
(76)
Therefore, (,())))) - (2.) = [(t(t) 2.] LA, (ā(t) 2c]
(77)
h A.G.G. Coult) (oo (Cin(t)z(t) Cnoise (t)) co (Cin (t)ze, Cnoise (t))
(78)
Approximating the term Eg (Cin())(((,, Cnoiso (1)) using a first-order taylor series approximation around the point Eg (Cin(t)zz z nooss (t)) as, Ep (Cin(t)z(() nooise (t)) = Eg iC (t) Cnoise (t)) + Vz, Ep (Cin (t)z,, noise (t)) [z(t) z,] (79)
Eg (Cin (t)zz, noose (t)) + Vz, Eo (C (t) noisee (t)) A, [ž(t) 2.]
(80)
Substituting the first order approximation of (Cin(1)z((), Cnoise (1)) in Eqn. 78,
(x()(()) - (2x) = + R: ((1) 21
(81)
where we have defined,
R [ Conl() co (Cin(()as, Cnoive (1)) A, A.B.A,"
(82)
Therefore,
(g())) (2))) = ||[ (Z()) 2,)1
(83)
S || hRR [(((t) 2.1l
(84)
Since ||²(t) 2:)11 < 6, we need the growth factor | + h RR to be bounded, which implies,
p(I + hR)) < 1 where p denotes the spectral radius of diagonalizable matrix. Furthermore, let,
(85)
1/ G. Cou (1) (I), Cuoise (1)) UAU
(86)
2
Therefore, simplify R2 as,
1 [ A.G.G.T Coun (1)V (Cin Gnoise (1)) A, A,B,A,"
R²
(87)
A 12 G.G. Coul (t) (Cin(t) Cnoise (t)) Bi A,
(88)
AI(UAUU-1 Bi A,
(89)
(A.U) A A- U-U (A,U)-1
(90)
-ViV-1
= (A.UV)((A,VV)
(91)
Substituting this simplified expression for R² in Eqn. 85, it follows that,
|| + hWl < 1
(92)
where À is an eigenvalue of the matrix A - U-B which concludes the proof.
As a special case, for B, = Ala, we have R; (A.U){A (A))A. In this case the
condition for stability reduces to 11 h(s A) < which concludes the proof for Corollary
21

---

Published as a conference paper at ICLR 2024
Algorithm 1 Conjugate Integrators (defined in Eqn. 8)
Trajectory length T, Network function eø (CCnz,, f), number of sampling steps N, a monotonically decreasing timestep discretization {4 spanning the interval (e. T) and choice B,
Input:
Output: z, = (x., m.)
Compute {Ai Him and (1t, Yam in Eqn. Zio p(zr)
Pre- compute coefficients Draw initial samples from the generative prior
2ig
Ato Bio
>> Transform
for
to N 1 do
h (tn+1 tn)
D Time step differential
d (#ot+1 *tn))
Phi differential
2i.,11 2in + hAc, Bi, Ant 2en dd):co(Cu((t)) Den Cuoise (tx.)
Update
end for
ZeN
Aix nN
Project to original space
B.6 CONJUGATE INTEGRATORS IN THE WILD
Here, we highlight some practical considerations when implementing Conjugate Integrators. We present high-level algorithmic implementation for the conjugate integrator defined in Eqn. 8 in Algorithm 1. Next, we discuss several practical aspects, including the invertibility of the transfor- mation A, and computing the coefficients A, and ④ as specified in Eqn. 6.
Invertibility of the transformation A,: Since we need to transform back the diffusion ODE dy- namics from the projected space é, to the original space 2,, ensuring the invertibility of the transfor- mation A, is a crucial requirement of conjugate integrators. However, since the expression for A, is composed of integral over multiple terms in the matrix exponential, it is non-trivial to guarantee matrix inversion since the matrices B, and Cski are user-specifie.. An alternate choice could to update the mapping A, to A, 5I where 5 > is small constant to ensure non- non-zero eigenvalues at any time t, thus ensuring invertibility. In this work, we set 5 = 0 for all experiments since we do not encounter any such instabilities during sampling.
Computing the Coefficients A, and 4,: The coefficients A, and b are defined as:
A, exp (B'B,- 1 G.GG Cssip())ds)
p = J A.G.G. Cou (93)
ct 1
where exp(.) denotes the matrix exponential. For the score parameterization in PSLD (Eqn. 273), these coefficients can be simplified as,
A, exp (B..B-F)) F.)
D - -.=."AA,G.CCLU.
(94)
For X-DDIM, the matrix B, is time-independent. Similarly, for PSL.D, the matrix F, is also time- independent. Therefore, the coefficient A, further simplifies to,
A cxp (BB ())F)t)
(95)
The above matrix exponential can be computed using standard scientific librarics like PyTorch (Paszke et al., 2019) or SciPy (Virtanen et al., 2020). Consequently, the coefficient o, reduces the following form,
p - f (B-
(96)
Therefore, at any time t, we estimate the coefficient Φ, using numerical integration. For given timestep schedule [t,} during sampling, we precompute the coefficient o,, which can be shared between all generated samples. For numerical integration, we use the odeint method from the torchdi fidfiffee package (Chen, 2018) with parameters atol=le-5, rtol=le-5 and the RK45 solver (Dormand & Prince, 1980). We set p = 0 as an initial condition. This is because, for the VP-SDE, &, corresponds to the noise-to-siee signal ratio at time t. Since recover the data time t= 0, the noise-to-siee to-sgnall ratio drops to zero. We extend this intuition to multivariate diffusions like PSLD and find this initial condition work well in practice.
22

---

Published as a conference paper at ICLR 2024
Time Required for Computing coefficients Dy Given set of sampling timepoints {t,), since Pt, is shared between all samples, we only need to compute { once the start sampling. Empirically, for our largest budget of NFE=100 in this work, numerical integration for computing coefficients 4; takes around 20 seconds on our setup, which very cheap when amortized over a large number of generated samples.
C SPLITTING INTEGRATORY FOR FAST ODE/SDE SAMPLING
C.1 INTRODUCTION TO SPLITTING INTEGRATORS
Here we provide a brief introduction to splitting integrators. For a detailed account of splitting integrators for designing symplectic numerical methods, refer interested readers Leimkuh- ler (2015). As discussed in the main text, the main idea behind splitting integrators is to split the vector field of an ODE the drift and the diffusion components of an SDE into independent sub- components, which are then solved independently using numerical scheme (or analytically). The solutions independent sub-components solution. Thus, three key steps in designing a splitting integrator are split, solve, and compase. We illustrate these steps with an example of deterministic dynamical system. However, the concept generic and can be applied to systems with stochastic dynamics as well.
then composed in a specific order obtain the final
Consider dynamical system specified by the following ODE:
dxe dm,
[xx,, mi) [(,, m,)
=
dt
(97)
We start by choosing a scheme to split the vector field for the ODE in Eqn. 97. While different types of splitting schemes can be possible, choose the following scheme for this example,
dx: dm:
[(x,, me)
dt + gxx, mi)
(98)
where we denote the individual components by and B. Next, we solve each of these components independently. i.e., we compute solutions for the following ODEs independently.
dx;
[xx m.)
dxe dmm;
dt.
g(x,, m,) dt
(99)
dm,
While any numerical scheme can be used to approximate the solution for the splitting components, we use Euler throughout this work. Therefore, applying an Euler approximation, with step size h, each of these splitting components yields the solutions C% and L as follows,
Xith X,
hx,(,, m,)
C
Xi+h X4
(100)
m m,
mm+h m, + hg(x,, m.)
In the final step, we compose the solutions to the independent components in a specific order. For instance, for the composition scheme AB, the final solution c\, C LA. Therefore,
[AB]
Xt+h x, lif(x, m.)
(101)
m++h m, hg hg(x++h, m,)
the required solution. worth noting that the final solution depends on the chosen composition scheme, and often it is not clear beforehand which composition scheme might work best.
C.2 DETERMINISTIC SPLITTING INTEGRATORS
We split the Probability Flow ODE for PSLD using the following splitting scheme
d&c dm:
3
[x
M m,
1s Z.,T T
dt +
3
2 X; + vm, + Mus (z,, T t) dt
0
2
0
(102)
23

---

Published as a conference paper at ICLR 2024
where x, Xy- my my--t, sg and sy denote the components in the data and momentum space, respectively. this work, we approximate the numerical update for each split using a simple Euler-based update. Formally, we denote the Euler approximation for the splits A and B by La and LB, respectively. The corresponding numerical updates for La and LB can be specified as:
Xith
=x,
hg rx: M- + (x, m., T 7-11
LA:
(103)
mi+h
m.
X1th X₁
CB
m. + h8 X, + vm, + Musy (x,, m,, T b))
(104)
Mi+h
Next, we summarize the exact update equations for all deterministic splitting samplers proposed in this work.
C.2.1 NAIVE SPLITTING SAMPLERS
We propose the following naive splitting samplers:
Naive Symplectic Euler (NSE): this scheme, for given step size h, the solutions to the splitting pieces L/ and f are composed as f/2 [BAl Ln CR Consequently, one numerical update step for this integrator can be defined as,
133
m m,
+
[2 + vm, + Mus (Xt, m., T t)]
(105)
2
h
R x, +
Tx, M m h + rs (R,, mc+h, T t)]
(106)
2
Therefore, one update step for the NVV sampler requires two NFEs.
Naive Velocity Verlet (NVV): In this scheme, for a given step size h, the solutions to the splitting pieces C₁ and 4/ are composed [BAB] L2/2 Cg 2/22- Consequently, numerical update step for this integrator can be defined
h3
m m.
& vm, Mus (Xe, m; --()]
(107)
h
8ith 3,
Tx: M M m,44/2 + Ts (X,, h1/2 T t)]
(108)
hB
mith m
Ni-h + +mm+4/2 Mus (Zi-h. mi h// T -
(109)
Therefore, one update step for the NVV sampler requires three NFEs.
C.2.2 REDUCED SPLITTING SAMPLERS
Analogous to the NSE and NVV samplers, propose the Reduced Symplectic Euler (RSE) and the Reduced Velocity Verlet (RVV) samplers, respectively.
Reduced Symplectic Euler (RSE): The numerical updates for this scheme are follows (the terms in red denote the changes from the NSE scheme),
h
m = m, +
X; + rm; + Mvs (Xi, m, t)]
(110)
2
h8
8eth x, +
Tx, M- - Pag (x, m,; t)]
(111)
2
is worth noting that the RSE sampler requires only one NFE per update step since single score evaluation re-used in both the momentum and the position updates.
24

---

Published as a conference paper at ICLR 2024
Reduced Velocity Verlet (RVV): The numerical updates for this scheme follows (the terms in blue denote the changes from the NVV scheme).
h
m44/22 m, +
X,
m. T t)] vm Mus [x,
(112)
hB
8eth 8, +
1x, M M
me + 1' (Xe, m,, T t)]
(113)
hB
m = m
4 [8tth vm2+ Mus (K++h. mc+h/2. 7 (t A)]]
(114)
In contrast to the NVV sampler, the RVV sampler requires two NFEs per update step. It is worth noting that the reduced schemes require fewer NFEs per update step than their naive counterparts. This implies that for the same compute budget. the reduced schemes use smaller step sizes as com- pared to the naive schemes. This is one of the reasons for the empirical effectiveness of the reduced schemes compared to their naive counterparts. Next, we discuss the effectiveness of the reduced samplers from the lens of local error analysis.
C.2.3 LOCAL ERROR ANALYSIS FOR DETERMINISTIC SPLITTING INTEGRATORS
We analyze the naive and reduced splitting samplers proposed in this work from the lens of local error analysis for ODE solvers. The probability flow ODE for PSLD defined as,
dx
B
Tx, - M- m, + X, vm, Muss" (Ze. - dt, I [0,T]
(115)
dm
We denote the proposed numerical schemes by Gn and the underlying ground- truth flow map for the probability flow ODE as Fr where h > 0 is the step-size for numerical integration. Formally, we analyze the growth of a A = ET (1++)) || h) Zet where Z++A (r-((-A) = 9h (2.) and z(t + h) (t (Z))) are the approximated and ground- truth solutions at time T (t ( h). Furthermore,
ēcth (F)()(t)) (z.))l
(116)
9r = Cx(((() (,,
(117)
S (Z)(t) Gx(2(()) + (I)) (z.)
(118)
The first term on the right-hand side of the above error bound is referred to the local truncation error. Intuitively, gives an estimate of how much error is introduced by our numerical scheme given the ground truth solution till the previous time step f. The second term in the error bound is referred to as the stability of the numerical scheme. Intuitively, it gives an estimate how much divergence is introduced by numerical scheme given two nearby solution trajectories such that ||z(t) - zell < 6. Here, we only deal with the local truncation error in the position and the momentum space. To this end, we first compute the term Fú (z())) using the Taylor-series expansion.
Computation of (z())))): Using the Taylor-series expansion in the position space, we have, x(t + h) = x(t) + h
dx(()
h2
(x(())
+
+ +
(119)
dt
2
dt
m + h)) m (t) + h dm (t)
h22
d
+ 0(h²)
(120)
dt
2
dt
Substituting the values of dx(t) de
and
dm(() dt
from the PSLD Prob. Flow ODE, it follows that,
(x(((1) = x(1) + h
Tx((1)
M
m( +
(1) (2)))
(121)
2
2 d
Tx(()= M M
m(t (()(t)))),.))))) t) 0(h3)
(122)
4 dt
h33
(.(((t)) = m(t) + x(t) + +m + Mus ((t),TT M
(123)
2
h²
d
xxt)) + vm + Mvs
(z(t), ( + Ohh3)
(124)
Next, analyze the local error for the Naive and Redueed Velocity Verlet samplers while high- lighting the justification for the difference in the update rules between the naive and the reduced schemes.
25

---

Published as a conference paper at ICLR 2024
C.2.4 ERROR ANALYSIS: NAIVE VELOCITY VERLET (NVV)
The NVV sampler has the following update rules:
h3
m++h/2 m,
[x
ym, Mus (x,, m. T t)]
(125)
4
h/
Xith = xx +
[x;
M M- m2441/2 + T m2/2 T f)]
(126)
2
h
m m4+h/2 +
Xe-h + Pm + Mvss (Ke-h: m - t)]
(127)
4
We first compute the local truncation error for the NVV sampler in both the position and the mo-
mentum space.
NVV local truncation error in the position space: From the update equations,
h3 [ix(t) M 1-1-Mmm'- + h/2) +1's) mi h/2), -t)]
x(t + 1)) x(t)
(128)
2
h
[x(()= M ((t)) h (x(t) vm(() Mus MvsT((((t),.... (t), (),T T t)]
x(())
(129)
+ + (x()) mit + h/2), -t) t)
(130)
(x(t)) = x(t) - 1:3
[x(() - M- (t) ('s mtt h/2), -t]
(131)
2
1:
+(t)+um() + Mus (x)), m(),
T-)) -1]
(132)
(x(())) x(1) + h3
Tx(() - M M- (x)))))>)(x()))))))) ml h/22) - -b] 1:2 1233 dm(()
dt
(133)
Therefore, the local truncation error in the position space is given by,
h.T 2
Fx(((()) - (x(t))
$ (x(,), m()), - - (x((()) m(t + m(22 -
(134)
1:2 2 x(t) + (()) -t)
(135)
di
We can approximate the term (x((), mit + h/2),T - t) using the Taylor-series expansion as follows,
+ (x m(t),
(x(1), (t +h22), - -)) =x(((()) m(), T-0 -
(136)
dm(()
m(t + h/2) m(t)) - 0
(137)
= =x((((), (t), 7 - ash (x(t), m(), - -t)
(138)
dm(t)
h
(x(t) vm()) + Mus (x()). mtt), T t)) + 0
4
(139)
h as (x() m(), T 8) dm,
(x(()) m(t) T t) (x()))) h/2), - 2 0m(t) dt
+ O(h))
(140)
26

---

Published as a conference paper at ICLR 2024
Substituting the above approximation (while ignoring the higher-order terms ((h))) in Eqn. 135,
3T
0s (x(t), (t), - din
Fr,((()) (x(()(
(141)
4
(m)
dt
h? PT d xxt) + st m(t). -t)
(142)
4
dt
12 31
d
x(()) + (x((()) m -0t) t)) (x)((()) mt), T' - t) dm,
4
dt
dm(()
dt
(143)
122
dx((1) + ( (x(() T -1) (x)(((1), m ml), T 1) dm. dt
4
dt
dt
@m(()
(144)
From the Chain rule, we have the following result, (x((x m t) (x((), (e)x(()),(t,TT)
0s (x(t,, (t), T - t) dx;;
(145)
01
0x.
dl
as (x(t), m()), T - dm,
(146)
0m:
dl
Substituting the above result in Eqn. 144,
(x((())))((((()) 3 dx(()
0s7 (x(1,, m(),, -1) 11) m(), dR.
+
dt
0x(()
dt.
Öt
(147)
The above equation implies that,
(xFx(x(tt) - (x(x)))) $ Car12
(148)
4
Since we choose 3 8 throughout this work, 3/4 2 can be absorbed in the constant C. Therefore, the local truncation error for the Naive Velocity Verlet (NVV) is of the order of (T'h²). Since is usually small in PSLD (Pandey & Mandt, 2023) (for instance, 0.01 for CIFAR-10 and 0.005 for CelebA-64), its magnitude is comparable or less than (particularly in the low NFE regime). Therefore, the effective local truncation order for the NVV scheme of the order of (
Next, we analyze the local truncation error for NVV in the momentum space.
NVV local truncation error in the momentum space: From the update equations,
h [x(t ( + vm h/2) + Mus (x(t + h), mlt + h/2), f)]
m(t + h) m(t + h/2) +
(149)
h
(x(t) + (t) + Mus, (x(s) m() t)] + h
m(t + --=h))=m (t) +
|x(t + h)] + (150)
4
h,3v
h5Mv
jm + h/2)l
sp (x(t + h), m(t + h /)) T - t)
(151)
4
h
m( + It) ml) + [x(1) + vm( + Mus (x(1), ((),
(152)
h
hB
x(t)
+
Tx((t) M - m( Tss m h/2), t)
4
(153)
h,3v
hB [x(t) + vm(() + Mus (x(t), m(), T +
mmt)
+
(154)
4
h,9Mv
8%
(x(t + f). m(t + h/2), T
(155)
27

---

Published as a conference paper at ICLR 2024
hB3 (x(t) + (
mtt + h)
mt) +
+ (0), -]]+
(156)
12
3
[xx(t) - M (t+ -00 (|) (157)
2
h.P ,B
3
(x(t) + (t) + M (t), +
(158)
8
2
dma(t)
h3MV
80 (x(1+ 11,
(8()))(()))))))))))))))))))(())) T 09]
(159)
4
hB
m(t+M) h) mt) +
(x(t) + mm()) + MM. (x(t). m(t), T t) +
(160)
2
12:
3
(x((1) M (m(() + h3
(x() vml()) + MuSy (x(1) (x(), T ()]
((),,
2
4
(161)
+ +xs)(((t),, ((((x(())(tttt))) - +))] + h² 3v din(t)
dt +
(162)
hBMv
80 h1), - ((),
(163)
hB
m(t+ h)- mt)
+ (
+ (M))))
(164)
2
B
(xx(t) M (0(00)00M)))))0(((((0000)0)))0))
+
(165)
4
2
h RT
(x((x() m[t + h/2),T (x((()), t)] h2 dm(()
8
8 dt
(166)
h,3Mv
(x(t + m + h/2), -) T")))
(x()), t)] + Ohh)))
4
(167)
Approximating *x((x(( + h), m(t+4/2))) T - t) around s/ (x(1), ml), T 1) using a first-order Taylor series,
(x(( + (x))) m (t), T - t) + 3s7(((1) (x(t). m(), T 1)
(168)
0x(t)
x((+h)) h) (((+))=((1] x(0] + 0sH (x)), m(),, - t)
ml + 1122) - m()
(169)
dm(()
*8 *x(( + +1)mm( + h/2), T- ()- = 28 (x(1,, (1), + 08h'' (x(1), m(), 1)
(170)
(x(1)
h.3 (xx(t) M -m + I's (x m(t h/2),
+) -t))
(171)
h 0 (x(1), m(I, 1) dm,
(172)
2
(m)
dt
(x h). + h/2), - t) = si (x(1), m(t, + (x(1,,((()) T 1) dx;
0x(1)
di
(173)
h Osg (x(t), mt).t(())) T t)
dm,
h
+
8 (x(t), m(t + h/22,T - t)-
(174)
2
dm(t)
dt
2
(z()(t) m mtt,
(175)
28

---

Published as a conference paper at ICLR 2024
Substituting the above results in Eqn. 167, get the following result,
h33
xx(t) + vm)(t) + Muss (x(t), - +)] 12, dx;; dxx;
m(t + h) = (t())
(176)
2
dt
h Bv dm
1² Osp (x(t), -t) dx;
1 (x(t). mt), T' t) dm
8
dt
xx(1) di
dm(1)
di
(177)
122 (1 + Mv)
88 (x(t), mt (t h/2), T (x(() (L))) -) + +0(1)
(178)
Using the multivariate Taylor-series expansion, we approximate 8 (2(+/22))) around (x(x(t), m(t), using first-order approximation as follows,
<*((),, m(L+ T- ()((()) ((),)))))) 0s m(), T mLL+h/2) mm
dm((
(179)
*8(x((), (())) (x(1), ml), - 1) + 0sy (x(),, m(), 1)
(180)
@m(t)
h
[x(1) mm(() + Mus; Muss((x(() (1), T [)]
(181)
Substituting the above result in Eqn. 178 and ignoring the higher order terms O(h)), we get,
h
1:2,3 dx;
m + h) = -t())= +
xx(t) + vm + Mus (x(t), m +)]
+ (182)
2
dt
1/2 dm(()
he2 BM
m(),
1) dx; dt
0 (x(1). m ((), 1)
dm,
+
+
8 dt
0x(()
3m()
dt
(183)
h (x() + vm(() + Muss (x()),m(t), T +)] 2
123
dxi
dmn
m(t + 11) = =m(t)= +
+
+ Mv
4
dt
dt
(184)
Os8' (x(t), m(t), dx,
dt + 0sh m(),, - dm, @m(t)
0sh (x(t), m(),, T t)
0x(t)
dt
Öt
(K),
(185)
dt
My (x((((()) m(t, -) My Os (x(t), mt), t)
(186)
dt
2
dm(()
Ōt
mit + (t) + h
12
dx
dm,
[x( vm(() + Mush (x(1), m(), T t)]
+ Mv
dt
dt
(187)
dsy (x(1, (t), T 11] h2 dd
+ M ash (x(1), ((()),m())) (l), 1) (188) 0m(t)
dt
2M 0sh (x()) mt), T -t)
(189)
We can now use the above result analyze the local truncation error in the momentum space as follows.
= 1233v dii(t)
asm (x(t), m(),, -t. +2M
dsh (x(t), (t), T t)
Fr (
8 dt +M
8m
Øt
(190)
29

---

Published as a conference paper at ICLR 2024
The above equation implies that,
C3vh2
||Fa (m)()) 9n (mn()))
(191)
Since we choose 3 8 throughout this work, 3/8 = 1 can be absorbed in the constant C. Therefore, the local truncation error for the Naive Velocity Verlet (NVV) in the momentum space is of the order of Ovvh²).
While the NVV sampler has nice theoretical properties, the local truncation error analysis can be misleading for large step sizes. This is because at low NFE regimes (or with high step sizes h), the assumption to ignore error contribution from higher-order terms like O(h²) might not be reasonable. In the NVV scheme, we make similar assumption in Eqns. 135,167 and 178 (when approximat- ing the term blue). This the primary motivation for re-using the score function evaluation so (x, m,, T t) between consecutive position and momentum updates in the RVV scheme. This design choice has the following advantages:
1. Firstly, re-using the score function evaluation s& 88(X,, m,, T - t) between consecutive po- sition and momentum updates exactly cancels out the term in blue in Eqn. 135 eliminat- ing error contribution from additional terms introduced by approximating sp (x(),, mit + h/2), T - t). This is especially significant for larger step sizes during sampling.
2. Secondly, re- using score function evaluation also reduces the number of NFEs per update step from three in NVV to two in RVV. This allows the use of smaller step sizes during inference for the same compute budget.
Next, we analyze the local truncation error for the RVV sampler.
C.2.5 ERROR ANALYSIS: REDUCED VELOCITY VERLET (RVV)
The NVV sampler has the following update rules:
h
mz+h/2 = m. +
X1 1/m; + Mus (x,, m. T - t)]
(192)
4
hB
8eth = 3, +
Tx, M M + 1's (2,, m, T - t)]
(193)
2
h
m = ma+1/2
8eth + um,+4/2 + Muss (Xeth, m T (1 + h))]
(194)
4
Similar to our analysis for the NVV sampler, we first compute the local truncation error in both the position and the momentum space.
RVV local truncation error in the position space: From the update equations,
x(1 + 1)) x() - h8
[x((1= - M (l + h/2) + [s (x(1), m (L), - o)
(195)
x((1)= - hB
[x(()= - -M-1 ((()) + h [x(() + vm(() + Mus (x(1), ((), T ()
2
4
(196)
+ I's (xx)(((()), m(()), T t)
(197)
h
(x((t)) x(t) +
[x(() M -m ())())))))) T (--)) t)]
(198)
3² M 1
x(t)+ + vm + Mus." (x(t), T
(199)
8
(x(()) x(1) - h3
[x(() - +Ts m()) T T 1)]
h? 6223M- dmi(t
2
4 dt
(200)
Therefore, the local truncation error in the position space is given by,
Fx(x(()) - (x())) 1:2 31 d
4 di x(1) + (x()) (()
(201)
30

---

Published as a conference paper at ICLR 2024
The above equation implies that,
C [[Fx(x(t))- - (x())) - 4
(202)
Similar to the NVV case, the local truncation error for RVV is of the order O(Th²). Since T is usually small in PSLD (Pandey & Mandt, 2023) (for instance, 0.01 for CIFAR-10 and 0.005 for CelebA-64), its magnitude is comparable or less than h (particularly in the low NFE regime). Therefore, the effective local truncation order for the NVV scheme is of the order of (
Next, we analyze the local truncation error for RVV in the momentum space.
RVV local truncation error in the momentum space: From the update equations,
x(t + h) + vmi( + h/22++ hB 4
m(t + h) m( + h/2)
(203)
Mvs (x(t + h), mmttmmt++/// h/2), T (t ht)
(204)
h8 (x(1) + vm(l) + Muss (x(), + h3
m(t + h) ml)) +
4 [x(1 + h)] + (205)
h,3vv
h.3vv
jm h/2)] +
8/ (x(t h), m(t h/2), T (t+ h))
(206)
4
h.8
m(t + h) m (t) +
(x(t) + vm)()
M M (x m() +]
(207)
1
x())
h
[Tx M m h/2) + T (x)) m(t,, T - t)] [)] |+
(208)
4
h,3v
hB (x(t) + vm(() Mys (x(). m(t), T +) }
m(t)
+
(209)
4
h,MF
86
(x(1
+ h). m( + - h1)
(210)
4
h,9
m(t + h) = m() +
(x()) + vm(() + Mus; (x(t), (xs) m(), -t]]+ +)]+ +
(211)
2
1 3
8
[Tx(t)= - M- 1 h (x(1). m (t), T []] +
(212)
4
2
h 3v
[x(t) + (m) + (xt) t)] +
(213)
8
2
d
h,3
(x(1 + h), ml + h/2), T (1. hi) - (x((), (()),,(())) -
()] (214)
89
4
h
m(t + h) = mff +
(x(t) + vt + Mus,"' (x(t), m(), - -t)]+ +
(215)
2
1:2
1
(Tx(t) - MM (m()))
h
[x(t) + vm(t) + Mus (x(t), m(), T - t)]
4
2
4
(216)
+ +Tsg(x(t). m(). T - +))]
h dm(t)
dt +
(217)
h.3v
8 (x(( h), mlt + h/2), T + h)) s0
1)
(218)
4
(219)
31

---

Published as a conference paper at ICLR 2024
h33
(t + h) mt)
[tt) + vm + Mwws m(), +]]+
(220)
2
h
3
(x((t) - M (x)) (00 -1)]
h2 dm((t
8 dt +
2
(221)
h3MV
(x(( + h), mit + h/2), - (x), (t+ +1)) + +1(1)
80
4
(222)
Approximating (x(1 + h), m(t+ h/2)) - +
around (x(())) m(), T - () using a
first-order Taylor series approximation (Ignoring higher order terms in O(h)))
$6 (x(( h). + - (t) 8 (x(t). mt), t) + 0sh (x(((x(),, m()), (())) T t)
Ōt
(223)
2 (x()., (t). (t). t)
x(1 + h) x(1]
(224)
(x((1)
08h (8(1), m(),T 1)
(m( + h/2) m())
(225)
dm(()
28 (x(1 + mm, + h/2),T T 1) = =x (1), T 1) + h 08h (x(t). m(t), T -t)
+ (226)
Øt
h 0s." (x(t). m(). T t) dm, Ösh (x(t), m mf), ( t)
(227)
2
dt
(x(t)
h/
Tx((t) M m + h/2) + 1s (t(), m(t + h/2).
(228)
$7 (x(1 + h), (t h/2), T sg (x(1), ((), T-) 1) h 3sh (x(t). m(), t) dx,
(x(t)
dt
(229)
h 0sh (x),, (t),T 1) dm,
Osh (x(1,, T
+ h
(230)
2
(m)
dt
Øt.
Substituting the above results in Eqn. 222, we get the following result,
h3 [x(1) + vm(() + MusT ((),)) -
12. dx
m(t + h) = m() +
+
(231)
2
dl
122 dm())
B²M Ds (x(t),m(())) t) dx, 18 (x(t), mt), T t) dm,
8 dt.
0x (1)
0m
dl
(232)
2sm (x(t),
m(t), T T.,.T-t -t) + +1h3)
+ /
(233)
01
32

---

Published as a conference paper at ICLR 2024
m(t a h) = mt())= + h (x(t) - vmn(t) + Muss (x(t), (t), T t)]
12
rdx;
dm,
+ v
+ Mv
2
4
dt
dt
(234)
0s (t))), t) dx. dt
Osy (x(t). (t), T t) dm,
0s (s" (x(t),
(t), T t)
+
(x(()
@m(
dt
Ōt
# ((()))
(235)
1/ dm
Mv 0s
(x(t), (t),
(236)
2
dt
2
dm(()
h
123
ddx;
dm:
m(t+ h) mt)
(x(t) vtm(t) Mus (x(t), (t), t)]
+ dt (237)
Mv
2
4 dt
(x(t), m(t), -
t)1
h2 dmm(t) M
3sh (x((), m (t), T
(238)
dl
dt
0m(t)
We can now use the above result to analyze the local truncation error in the momentum space as follows,
h2 dim(t)
Osy (x(t), m((), t)
(m) Gn (m)
+ M
(239)
8 dl
0m(()
The above equation implies that,
Cßvh2
- (m(()))
(240)
8
Similar to the NVV sampler, the scaling factor 3/8 1 can be absorbed in the constant C. There- fore, the local truncation error for the Reduced Velocity Verlet (RVV) the momentum space is of the order of (vvh²).
C.3 STOCHASTIC SPLITTING INTEGRATORS
We split the Reverse Diffusion SDE for PSLD using the following splitting scheme.
dxx; dm;
3
2Ixx
M m, + 21 (Z,,)) t) dt +0+ 3
0
x; + 2vm + 2M 2M ( t) dt
2
(241)
31 dt + /BTdw: is the Ornstein-Ullenbeck component which injects (_ B dt + / VMBddw:
where 0
stochasticity during sampling. Similar to the deterministic case, X; XT-1, m, = my-t. s and sg denote the score components in the data and momentum space, respectively. We approxi- mate the solution for splits A and B using simple suler-based numerical approximation. Formally, we denote the Euler approximation for the splits A and B by LA and LB. respectively, with their corresponding numerical updates specified as:
h9
Ix: M-1m, + 187 m. T T 1)
8
LA
(242)
mi+h
m,
Xith
=X,
LB
-m; + h8 x, + vm, + Muss (Xe, m, T -t))
(243)
mi+h
It is worth noting that the solution to the OU component can be computed analytically:
8rth
exxp
h.r
X, / exp (- EF ~ No Is)
Lo
(244)
mith
exp
h,39
mi
V /M /
exp w-p(phai (-h3v)em: em N'(O,, Ia)
33

---

Published as a conference paper at ICLR 2024
Next, highlight the numerical update equations for the Naive-OBA sampler and the Reduced OBA, BAO, and OBAB samplers.
C.3.1 NAIVE SPLITTING SAMPLERS
Naive OBA: In this scheme, for a given step size h, the solutions to the splitting pieces Cg. C and C3 are composed as c// c|OBA]= L o C c Consequently, one numerical update step for this integrator can be defined as,
1
84th = exp
8, + V - exp ex (-h3T)).
(245)
2
h13v
mith = exp
m + VM cxp (-hBv)em
(246)
2
h
m4+h mc+h
8 + 2 + 2Mvs (Kethh meth. t)]
(247)
2
hB
84th-
84th
22 M M- 21 2% (Reth mh T
(248)
where Euy Em ~ (Oad Ia). Therefore, one update step for Naive OBA requires two NFEs.
C.3.2 EFFECTS OF CONTROLLING STOCHASTICITY
Similar to Karras et al. (2022), we introduce a parameter X, the position space update for Lo to control the amount of noise injected in the position space. More specifically, we modify the numerical update equations for the Ornstein-Uhlenbeck process the position space as follows:
- h19r
8ith exp
& /1 exp (-tx C, ~~ Ia)
(249)
(T- i.c., the mid-point for two consecutive time steps during sampling. where
Adding similar parameter in the momentum space leads to unstable sampling. We therefore restrict this adjustment to only the position space.
C.3.3 REDUCED SPLITTING SCHEMES
We obtain the Reduced Splitting schemes by sharing the score function evaluation between the first
consecutive position and momentum updates for all samplers. Additionally, for half-step updates (as in the OBAB scheme), condition the score function with the timestep embedding of T (t + h) instead of T t. Moreover, we make the adjustments as described in Appendix C.3.2.
Reduced OBA: The numerical updates for this scheme follows (the terms in red denote the changes from the Naive OBA scheme),
h
84th exp
V
exp (-iX,
(250)
2
hh3v
M44h exp
m. VM exp (-hBu)em
(251)
h3
moth -meth
[x + 2wmTt + 2M (Reth, meth, T ())
(252)
2
h3
8ith = Nith +
2T - M m + 2T S (Xtth mith, T t)]
(253)
where Ex, cm ~ NO,, I2). is worth noting that Reduced OBA requires only one NFE per update step since single score evaluation is re-used in both the momentum and the position updates.
34

---

Published as a conference paper at ICLR 2024
Reduced BAO: The numerical updates for this scheme are follows,
h,3
M++h = m, +
[ +
+ 2M (x,, m, T -t]]
(254)
2
h
X(+h = x +
2
M m + 2T s/ [ m, , T - 1)]
(255)
2
84th exp
X++h + V exp (-1x,3T)e 6xr
(256)
2
-h,8v
mith exp
m + M V exp (-h,vv)em
(257)
2
where Ex, Em (O,, La). Similar to the Reduced OBA scheme, Reduced BAO also requires only one NFE per update step since a single score evaluation is re-used both the momentum and the position updates.
Reduced OBAB: The numerical updates for this scheme are as follows,
h
Xith exp
X; + /1 - -1-cxpp-tx,pp (-t,p
(258)
2
(h1Bv
mith exp
m + VM / exp (-h$v)em
(259)
2
h3
1m441/2 mi+h +
Nith + 2wm + 2Mvs (tth, mith, t)]
(260)
h
Xith + X(th
x M- m, h12 2['s (x mi+h T t)]
(261)
h33
mu+h mc+h//2 +
[Re+h + 20tm444/2 2M (Reth, m244/22 (t + h))]
(262)
4
where 62, 6m 6m
AO,,,, Ia). worth noting that, in contrast to the Reduced OBA and BAO schemes, Reduced OBAB requires two NFE per update step. This is similar to the Reduced Velocity Verlet (RVV) sampler.
D CONJUGATE SPLITTING INTEGRATORS
Here, we highlight relevant update equations for the Conjugate Splitting Samplers discussed in Section. 3.3.
D.1 DETERMINISTIC CONJUGATE SPLITTING SAMPLERS
The splitting scheme for deterministic splitting samplers discussed in Section 3.2 is specified as follows,
dxz dm:
3
184
M
m, + 1s [z,, -t)
-t)
t)) dt + 3
0 -t) dt
Xc + vm, + Mus (z,, T t)
2
0
B
(263)
Conjugate Integrators applied Splitting components. The Splitting component A in the posi- tion space can be simplified as follows,
dx. dm:
1xx
M
1'87(Zo, T-t) 1)
di
(264)
2
0
3
M-1
X
T3
TT,
T
df
(265)
2
m;
2
where / - - Moreover, for any time-dependent matrix Cr, we denote C
C,, where,
1
denotes the Hadamard product of the mask m =
with the matrix C, Therefore, Eqn. 265
0
35

---

Published as a conference paper at ICLR 2024
Algorithm 2 Conjugate Symplectic Euler
Trajectory length T, Network function co(..), .), number of sampling steps N, monotonically de- creasing timestep discretization {4:]_ spanning the interval (c. T) choice of B
Input:
Output: z, = (x., m.)
Compute {Aa Him and ( Yam in Eqn. 267 Zio ~ p(zr) for 0 to N do
Pre compute coefficients Draw initial samples from the generative prior
Compute (xt,,,, m (n) and s0 m fn)
b Compute score b Time step differential
(ln+1 (n)
M6,11
min
hg Xtn ummn + Muss (Xt,, :entta)
Momentum Update
Construct Ztn
(Xt,, M
d+b (#0t db..)
Phi differential Transform
21., A,, thin
Br,11 + Zin + hddee IA 2c, (xt, mC,, ta)
Update
Xin+11 A Mnat1
b Project original space and discard momentum
Construct Ztnt1 Xtn41 m
Algorithm 3 Conjugate Velocity Verlet
Trajectory length T. Network function co(...), number of sampling steps N, monotonically de- creasing timestep discretization {ta spanning interval (c, T) choice of B,. Output: 2, = (x., m.)
Input:
Compute (Ac, J and (d:;; ) in Eqn. 267 Zea p(zr) for to do
Pre-compute coefficients Draw initial samples from the generative prior
Compute e c0(E mc,, tn)) and (x(x,,, me,,
Compute score D Time step differential
(bn+1 bn)
Mc+11 Men
hd Xt, vmnn Musj (xt.. menita)
Momentum Update
Construct Zin (xt,, mines)
dé, = (4:n+1 :b,))
Phi differential Transform
2i. Aen
21,111 <- 21n + l.d.i. 1A
(L.)),,, 2i,, Cmise (bx))
Update
Xtn+11- A intt 2tat1
b Project original space and discard momentum
Construct Zin+1 Xints "i.
Compute (o(Xi,t1
1it bnt1) and so so(Xi, mu, bu+1)
> Compute score
M Min+1
Xtn+i + vmitnts MMs"" (xt+t Meatis tn+1)
Momentum Update
end for
can be simplified as follows,
dxc dmn,
F. G
Tm (G. G. pm
Cmp (t) 2; + C (t) (t)z,, Cnoise (t))] de (266)
where F; and G; are the drift scaling matrix and the diffusion coefficients, respectively. We can then determine the transformed ODE corresponding the masked ODE in Eqn. 266 and perform numerical integration in the projected space. We use the X-DDIM-II as our choice of the conjugate integrator and, therefore, set B, A1. The coefficients A, and 4, defined as,
Å, = exp (f.AL
cxp
Fm
(G.) C. (s)ds)
$ $. $.-- 2
"Cm dsss,
2
(267)
36

---

Published as a conference paper at ICLR 2024
Algorithm 4 Conjugate OBA
Trajectory length T, Network function co(.,), .), number of sampling steps N, monotonically de- creasing timestep discretization {4:]_ spanning the interval (c. T) choice of B
Input:
Output: z, = (x., m.)
Compute {Au, Him and ( yam in Eqn. 272 Zio ~ p(zr) for n to N 1 do
Pre compute coefficients
Draw initial samples from the generative prior
= (ln+1 - (n)
b Time step differential
t' (tn + tn+1)/2
Sr. = exp
het
2
Xtn + V1 exp((t'X.3T))
P OU-Update (Position)
m..
h13
m VM exp (h.32) Em
b OU-Update (Momentum)
Compute (x( mi, in) and se .,.., m.n. tin)
Compute score
M6 = m hi I x,, 2vmi. + 2Mvs' (St,, mu, kn)
b Momentum Update
Construct 2t,, = [xt,, m
dΦ. = (i::41 .
Phi differential D Transform Update
2tn A., Bta
2tat1 + 2tn hrdr, AA 2e, dé:cco( (Xt, m,,,,t)) tn)
Xt,111-- Alati 2en11
Project original space and discard momentum
Construct Bin+1 Xt,+1 M
end for
Based on this analysis, we provide the numerical update rules for the CSE and CVV samplers in Algorithms and 3, respectively.
D.2 STOCHASTIC CONJUGATE SPLITTING SAMPLERS
We split the Reverse Diffusion SDE for PSLD using the following splitting scheme.
dxt dm:
8 21x, M- m. + 2T 8 (Zz, dt +0+
0
dt
2
Xx + 2um, + 2M us (2,)
(268)
Therefore, the splitting component corresponding to the position space is,
dx. dm,
3
-21x,
+ M - m
(z,,
T
df
(269)
2
3
-11 M
X;
8 (g) T t) )) TB
-t)
df
(270)
2
m
where /. = T T t. Eqn.
270 can be further simplified as follows,
(f.z. GGG Caip(()); + Coun(L) (Cin((CC Croice (1))) di
dz,
(271)
9) We use the X-DDIM-II our choice of the conjugate integrator and, therefore, set B, = A1. The coefficients A, and & are defined as,
-21 M-1) and G. VTB 0
where F, 7/4 0
A. exp (L/ A1 F. &.G Cmi (s)ds) 4; -- -S- A.G.GT C. Cun (s)ds,
(272)
1 1 Based on this
where, skip (t) = m Caui (t) and C. (t) = m Cou(t)) and m
0 0
analysis, present complete analysis for the Conjugate OBA sampler in Algorithm
37

---

Published as a conference paper at ICLR 2024
E IMPLEMENTATION DETAILS
Here, we present complete implementation details for all the samplers presented in this work.
E.1 DATASETS AND PREPROCESSING
We the CIFAR-I0 (Krizhevsky, 2009) (50k images), CelebA-64 (downsampled 64 64 reso- lution, RJ 200k images) (Liu et al., 2015) and the AFHQv2-64 (Choi et al., 2020) (downsampled to 64 64 resolution, 15k images) datasets for both quantitative and qualitative analysis. We use the AFHQv2 dataset (downsampled to the 128 x 128 resolution) only for qualitative analysis. During training, all datasets are preprocessed use random horizontal flips to train all new models across datasets as data augmentation strategy. During inference, we scale all generated samples between the range [0, 1].
numerical range of [-1, 1]. Following prior work, we
E.2 PRE-TRAINED MODELS
For all ablation results in Section 3 in the main text, we use pre-trained PSLD (Pandey & Mandt, 2023) models for CIFAR-I0 with SDE hyperparameters I' = 0.01, v/ 4.01 and = 8.0. The resulting model consists of approximately 97M parameters. For more details the score network architecture, refer to Pandey & Mandt (2023). Moreover, pre-trained models from PSLD correspond to the following choices of the design parameters in the score parameterization defined in Equ. 5,
Cski (t) = O, Coa(t) = -, (tn(1)= = Croise (t) = =t.
(273)
where L, is the transposed-inverrs of the perturbation kernel in PSLD. Most comparison baselines in Section 5 (like DEIS (Zhang & Chen, 2023) and DPM-Solver (Lu et al., 2022)) use the VP-SDE (deep) model, which is around 108M parameters in size. Therefore, our model sizes comparable with other baselines, making our comparisons fair.
the Cholesky decomposition of the covariance matrix Σ,
E.3 SCORE NETWORK PRECONDITIONING
For the score network parameterization discussed in Eqn. 5, choose,
(tsip()) = (tag((,)) Coa(l) = -, L, Ctn( = I, Cnoise(t) = t.
where Le is the Cholesky factorization of the variance E, of the perturbation kernel in PSLD. Similarly, Σ, is the variance of the perturbation kernel in PSLD with initial variance ΣΦ I 0 smn = 0, De = MyI. For optimal sample quality, we set the weighting scheme A(t) ||Caul] We set of 0.25 for all experimental analysis. Since this requires newly trained PSLD models, we highlight our score network architectures and training configuration next.
Score-Network architecture. Table illustrates our score model architectures for different datasets. We use the NCSN++ architecture (Song et al., 2020) for all newly trained models.
SDE Hyperparameters: Similar to Pandey Mandt (2023), we set 3 8.0. M 1 and = 0.04 for all datasets. For -I0, we set r = 0.01 and v = 4.01, corresponding to the best settings in PSLD. Similarly, for CelebA-64 and AFHQv2-64 datasets, we set | 0.005 and 4.005. Similar to Pandey & Mandt (2023), we add a stabilizing numerical epsilon value of -9 in the diagonal entries of the Cholesky decomposition of E, when sampling from the perturbation
kernel (2) during training.
Training Table summarizes the different training hyperparameters across datasets. We use the Hybrid Score Matching (HSM) objective during training.
E.4 EVALUATION
We report FID (Heusel et al., 2017) scores on 50k samples for to assess sample quality. We use the Number of Function Evaluations (NFEs) for assessing sampling efficiency.
Timestep Selection during Sampling: We use quadratic striding for timestep discretization pro- posed in Dockhorn al. (2022b) during sampling, which ensures more number of score function
38

---

Published as a conference paper at ICLR 2024
Hyperparameter
CIFAR-10
CelebA-64
AFHQV2-64
Base channels
128
128
128
Channel multiplier
[2,2,2]
[1,2,2,2]
[1,2,2,2]
# Residual blocks
8
Non-Linearity
Swish
Swish
Swish
Attention resolution
[16]
[16]
[16]
Attention heads
1
Dropout
0.15
0.1
0.25
FIR (Zhang, 2019)
True
True
True
FIR kernel
[1,3,3,1]
[1,3,3,1]
[1,3,3,1]
Progressive Input
Residual
Residual
Residual
Progressive Combine
Sum
Sum
Sum
Embedding type
Fourier
Fourier
Fourier
Sigma scaling
False
False
False
Model size
97M
62M
62M
Table 2: Score Network hyperparameters for training Preconditioned PSLDD models. of is set to 0.25 for all datasets
CIFAR-10
CelebA-64
AFHQv2
Random Seed
0
0
# iterations
1.2M
1.2M
400k
Optimizer
Adam
Adam
Adam
Grad Clip. cutoff
1.0
1.0
1.0
Learning rate (LR)
2e-4
2e-4
2e-4
LR Warmup steps
5000
5000
5000
FP16
False
False
False
EMA Rate
0.9998
0.9998
0.9998
Effective Batch size
128
128
128
# GPUs
8
8
Train eps cutoff
le-5
le-5
le-5
Table 3: Training hyperparameters
evaluations in the lower timestep regime (i.e., t, which is close to the data). This kind of timestep selection is particularly useful when the NFE budget limited. We also explored the timestep discretization proposed in Karras et al. (2022) but noticed degradation in sample quality.
Last-Step Denoising: It is common to add an Fuler-based denoising step from a cutoff zero to optimize for sample quality (Song et al., 2020; Dockhorn et al., 2022b; 2021b) at the expense of another sampling step. For deterministic samplers presented in this work, omit this heuristic due to observed degradation in sample quality. However, for stochastic sam- plers, we find that using last-step denoising leads to improvements in sample quality (especially when adjusting the amount of stochasticity discussed in Appendix C.3.2). Formally, perform the following update as a last denoising step for stochastic samplers:
Martinnna et al.,
X
X m
B,E
Tx, - M- mm se (ze, e) x, vm, 2M (z. e) l)laa))
(274)
mo
2
Similar to PSLD, set 1e 3 during sampling for all experiments. Though recent works (Lu et al., 2022; Zhang & Chen, 2023) have found lower cutoffs to work better for a certain NFE budget, leave this exploration in the context of PSLD to future work.
Evaluation Metrics: Unless specified otherwise, we report the FID (Heusel et al., 2017) score on 50k samples for assessing sample quality. Similarly, use the network function evaluations (NFE) to assess sampling efficiency. In practice, we use the torch- wiweeiiileeebbuttoooooooooooo et al., 2020) package for computing all FID reported in this work.
39

---

Published as a conference paper at ICLR 2024
NFE (FID@50k
Method 50
70
100
150
200
250
500
1000
Buler
431.74
397.51
330.18
18
233.28
163.13
110.68
33.93
11.54
A-DDIM (B,-
48.55
11.49
4.81
3.53
3.31
3.19
3.04
3.01
Table 4: Extended results for Fig. 2a. X-DDIM outperforms baseline Euler when applied to the
PSLD Prob. Flow ODE. The choice of B, 0 corresponds the exponential integrators proposed in Zhang & Chen (2023); Zhang et al. (2022). In this case, Euler fails to generate high-quality samples even with high compute budget of 1000 NFEs. Values in bold indicate the best FID scores for that column.
NFE
X-DDIM (B,
X-DDIM-I AT)
A-DDIM-II (B, A1)
FID@ 50k (1)
FID@50k ())
FID 50k (1)
À
30
311.08
23.53
+0.0038
13.6
0.59
50
48.55
5.54
-0.0016
5.04
0,46
70
11.49
4.41
+0.0009
4.26
0.35
100
3.76
-0.0004
3.71
0.21
150
3.53
3.49
-0.0002
3.46
0.12
200
3.31
3.32
-0.00008
3.28
0.06
250
3.19
3.21
-0.00004
3.19
0.02
Table 5: Extended results for Figs. 2b,2c. Comparison between different choices of B, for the proposed X-DDIM sampler. X-DDIM with non-zero choices of B, outperforms baseline choice with B, = 0 which suggests that the latter choice can be sub-optimal in certain scenarios. Most gains sample quality using a non-zero B, observed low sampling budgets (NFE < 70). Values in bold indicate the best among the three methods for a particular sampling budget.
0.7
0.6
0.5
0.46
0.4
0.3
0.2
0.1
FID@ 50k
21.51
10.68
5.53
5.04
5.77
10.5
19.54
32.61
48.55
Table Impact of the magnitude of on CIFAR-10 sample quality for a fixed NFE=50 steps for X- DDIM-II. Entries in bold indicate the best FID scores and the corresponding À value. Interestingly, increasing improves sample quality significantly compared to O. However, too much increase in À leads to significant degradation in sample quality.
F EXTENDED RESULTS
EXTENDED RESULTS FOR SECTION 3.1: CONJUGATE INTEGRATORS
We include extended results corresponding to Figs. 2a in Table 4 and for Figs. 2b, 2c in Table 5, respectively.
Impact of varying À sample quality. Additionally, illustrate the impact of varying À sample quality for a fixed NFF=50 for X-DDIM-II Table 6. Increasing the value of leads to significant improvements in sample quality. However, excessively inereasing leads degraded sample quality. This observation empirically supports our theoretical results in Theorem 2.
F.2 EXTENDED RESULTS FOR SECTION 3.2: SPLITTING INTEGRATORS
We include extended results corresponding to Figs. 3a, 3b in Table 7 and for Fig. 3c Table 8, respectively.
Comparison between different Stochastic Reduced Splitting schemes. Table 9 compares the performance of different reduced splitting schemes for stochastic sampling.
Impact of varying X, on stochastic sampling. Additionally, we illustrate the impact of varying the parameter Xa sample quality in the context of the Reduced OBA sampler (See Table 10).
40

---

Published as a conference paper at ICLR 2024
NFE
Prob Flow ODE
NSE
RSE
NVV
RVV
50
431.74
132.45
23.5
69.06
14.19
397.51
63.69
10.03
31.54
5.72
100
330.18
23.47
5.31
14.49
3,41
150
233.28
8.85
3,54
7.51
2.8
200
163.13
5.44
3.1
5.53
2.7
250
110.68
4.16
2.98
4.68
2.71
33.93
3.05
2.88
3.56
2.79
1000
11.54
2.92
2.89
3.2
2.86
Table 7: Extended Results for Figs. 3a, 3b. Comparison between Euler, Naive, and Reduced Split- ting samplers applied to the PSLD ODE. Naive schemes improve significantly over Euler, indicating the benefits of splitting. Adjusted schemes improve significantly over naive splitting samplers, high- lighting the benefit of our proposed modifications to naive schemes. Values bold highlight the best-performing sampler among all comparison baselines. FID reported 50k samples.
NFE
EM SDE
Naive OBA
Reduced OBA
Reduced OBA (+).)
50
30.81
36.87
19.96
2.76 (1.16)
70
15.63
24.23
12.71
2.51 (0,66)
100
7.83
15.18
7.68
2.42 (0.37)
150
4.26
9.68
5.21
2.40 (0.2)
200
3.27
7.09
4.06
2.38 (0.13)
250
2.75
5.56
3.63
2.40 (0.1)
500
2.3
3.41
2.74
1000
2.27
2.76
2.45
Table 8: Extended Results for Figs. 3c. Comparison between EM, Naive, and Reduced OBA sam- plers applied to the PSLD Reverse SDE. Adjusted schemes combined with the tuned parameter X, improve stochastic sampling performance significantly. Values in bold highlight the best-performing sampler among all comparison baselines. FID 4 reported on 50k samples.
NFE
RBAO
ROBAB
ROBA
(+^a)
(-Aa)
(+Aa)
(-A.)
(+^a)
(-A.)
30
7.83 (1.18)
26.88
21.60 (0.24)
22.09
4.03 (2.72)
39.51
50
3.33 (0.7)
12.96
6.86 (0.2)
6.01
2.76 (1.16)
19.96
70
2.59 (0.44)
8.2
4.66 (0.16)
3.6
2.51 (0.66)
12.71
100
2.65 (0.3)
5.31
3.54 (0.14)
2.73
2.36 (0.37)
7.68
150
(0.18)
2.96 (0.12)
2.44
2.40
5.21
200
2.43 (0.1)
3.26
2.67 (0.1)
2.27
2.38 (0.13)
4.06
Table 9: Comparison between Reduced OBA, BAO, and OBAB schemes. Reduced OBA (with A
performs the best among all schemes. Values in bold highlight the best-performing sampler among all comparison baselines for given NFE budget. FID J reported 50k samples. Values in (.)
indicate the corresponding A for a sampler at a given compute budget.
Increasing the value A leads to significant improvements in sample quality. However, a large X degrades sample quality significantly.
F.3 EXTENDED RESULTS FOR SECTION 3.3: CONJUGATE SPLITTING INTEGRATORS
We include extended results corresponding to Fig. 4a in Table and for Fig. 4b in Table 12.
F.4
EXTENDED RESULTS: IMPACT OF PRECONDITIONING
We include extended results corresponding Figs. 4c, 4d in Table 13.
41

---

Published as a conference paper at ICLR 2024
A
0.1
0.4
0.8
1.0
1.16
1.2
1.4
1.6
5OD
24.68
14.34
5.57
3.29
2.76
2.82
4.21
7.07
Table 10: Impact of the magnitude on CIFAR- 10 sample quality for fixed NFE=50 steps for X- DDIM-II. Entries in bold indicate the best FID scores and the corresponding À value. Interestingly, increasing improves sample quality significantly compared to 0. However, too much increase in À leads to significant degradation in sample quality.
RVV
RSE
CVV
CSE
NFE
FID@ 50K
FID@ 50K,
FID@
A
FID@ 4
30
89.86
94.21
7.23
-0.41
7.38
1.38
40
31.78
44.3
4.21
-0.3
4.95
1.35
50
14.19
23.5
3.21
-0.25
3.92
1.33
60
8.22
14.46
2.73
-0.21
3.38
1.33
70
5.72
10.03
2.44
-0.2
3,07
1.31
80
4.44
7.64
2.27
-0.17
2.87
1.3
90
3.78
2.18
-0.16
2.76
1.27
100
3.41
5.31
2.11
-0.14
2.68
1.25
Table 11: Extended Results for Fig. 4a. Comparison between Reduced Splitting samplers and Conjugate Splitting samplers applied to the PSLD Prob. flow ODE. Conjugate Splitting samplers largely outperform their reduced counterparts by significant margin.
Reduced OBA
Conjugate OBA
NFE
FID@
A
F
A
30
4.03
2.72
4.4
2.72
40
3.11
1.7
3.34
-0.2
1.7
50
2.76
1.16
2.94
-0.1
1.16
60
2.62
0.84
2.8
-0.1
0.84
70
2.51
0.66
2.64
-0.1
0.66
80
2.47
0.53
0.53
90
2.44
0.43
2.55
-0.1
0.43
100
2.36
0.37
2.49
-0.1
0.37
Comparison between Reduced OBA and Conjugate OBA stochastic samplers. Both samplers share churn values for given sampling budget. For CIFAR-10, Conjugate OBA slightly degrades sample quality over Reduced OBA.
Table 12: Extended Results for Fig. 4b.
F.5 EXTENDED RESULTS FOR SECTION 4: STATE-OF-THE-AR RESULTS
We include extended results corresponding to Fig. 5 in Tables 13, 15, 16 for CIFAR-10, 10, CelebA-64 and the AFHQv2-64 datasets, respectively. We include qualitative samples from samplers used for state-of-the state-of-the-ar art comparisons in Figs. 6-11
42

---

Published as a conference paper at ICLR 2024
CSPS-D (+Pre.)
CSPS-D
SPS-S (+Pre.)
SPS-S
NFE
FID@ 50k 4
50K
À
FID@50K 4
A
FID 50k 4
A
30
7.23
-0.41
3,89
2.65
2.72
40
3.26
-0.31
4.21
-0.3
3.05
1.7
3.11
1.7
50
2.65
-0.25
3.21
-0.25
2.74
1.15
2.76
1.16
60
2.42
-0.22
2.73
-0.21
2.58
0.86
2.62
0.84
70
2.34
-0.2
2.44
-0.2
2.54
0.66
2.51
0.66
80
2.27
-0.17
2.52
0.53
0.53
90
2.23
-0.16
2.18
-0.16
2.5
0.45
2.44
0.43
100
2.24
-0.14
2.11
-0.14
2.47
0.38
2.36
0.37
Impact of score network preconditioning on sam- pler performance. Preconditioning improves sample quality for a low sampling budget but slightly degrades sample quality for higher budget. Values in bold indicate the best-performing sampler with/without preconditioning.
Table 13: Extended Results for Figs. 4c, 4d.
Ablation
Description
Type
NPU
HID & (NFE=50)
FID J (NFE=100)
Conjugate (See. 3.1)
[CI] A-DDIM-I
Conjugate Integratoe choice
5.54
3.76
[C2] X-DDIM-II
Conjugate Integrator with choice B,
D
5.04
3,71
[SI] NSE
Naive Symplectic Faler
D
132.45
23.47
[S2] NVV
Naive Velocity Verket
D
69.
14.49
Splitting (Sec 3.2)
Reduced Symplectic Ealer (IS1) adjustments)
23.5
[S4] RVV
Reduced Velucity Verlet (IS2) + adjustments)
D
14.19
3.41
[86] ROBA
Reduced OBA [[85) adjustments)
2,76
2.36
[CSI] CSE
Conjugate Symplectic Faler ([S3) [C2])
D
3,92
2.68
Conjugate Splitting (Sec 3.3)
[CS2] CVV
Conjugate Velocity Verlet (IS4) (C2)
D
3.21
2.11
[CS3]
Conjugate ([S6) + (C2)
Table 14: Overview of our ablation samplers on the CIFAR-10 dataset for PSLD diffusion. NPU: NFE per numerical update, D: Deterministic, S: Stochastic
CSPS-D D (+Pre.)
CSPS-S (+Pre.)
SPS-S (+Pre.)
NFE
FID@ 50k 4
A
FID@50k +
A
FID@ +
A
30
25.75
-0.5
6.16
0.85
6.32
3.92
40
7.97
-0.4
3.94
0.8
4.56
2.6
50
4.41
-0.33
3.32
0.7
3.88
1.8
70
2.75
-0.23
2.81
0.5
3.06
1
100
2.25
-0.13
2.6
2.64
0.55
Table 15: State-of-the-ar results for CelebA-64,
CSPS-D (+Pre.)
CSPS-S (+Pre.)
SPS-S (+Pre.)
NFE
FID@50K
A
FID@50k 1
A
FID@50K A
9.83
-0.15
3.59
2.8
6.17
40
5.22
-0.1
3.38
2.8
5.37
1.35
50
3.63
-0.07
3.1
2
4.36
1.15
70
2.7
-0.04
2.83
3.31
0.8
100
2.39
-0.02
2.61
0.2
2.73
0.5
Table 16: State-of-the-art results for AFHQv2-64.
43

---

Published as a conference paper at ICLR 2024
FID 2.65 (NFE=50), Sampler: CSPS-D (+Pre.)
FID 2.11 (NFE=100). Sampler: CSPS-D
Figure 6: Random CIFAR-10 samples generated using our deterministic samplers
44

---

Published as a conference paper at ICLR 2024
FID 2.74 (NFE=50), Sampler: SPS-S (+Pre.)
2.36 (NFE=100). Sampler: SPS-S
Figure 7: Random CIFAR-10 samples generated using stochastie samplers
45

---

Published as a conference paper at ICLR 2024
12
FID = 4.41 (NFE=50),
Sampler: CSPS-D (+Pre.)
FID 2.25 (NFE=100), Sampler: CSPS-D (+Pre.)
Figure 8: Random CelebA-64 samples generated using deterministic samplers
46

---

Published as a conference paper at ICLR 2024
3.32 (NFE=50), Sampler: CSPS-S (+Pre.)
BUL
FID 2.6 (NFE=100), Sampler: CSPS-S (+Pre.)
Figure 9: Random CelebA-64 -44 samples generated using our stochastic samplers
47

---

Published as a conference paper at ICLR 2024
FID 3.63 (NFE=50), Sampler: CSPS-D (+Pre.)
FID = 2.39 (NFE=100). Sampler: CSPS-D (+Pre.)
Figure 10: Random AFHQv2-64 samples generated using our deterministic samplers
48

---

Published as a conference paper at ICLR 2024
FID 3.1 (NFE=50), Sampler: CSPS-S (+Pre.)
FID 2.61 (NFE=100), Sampler: CSPS-S (+Pre.)
Figure 11: Random AFHQv2-64 samples generated using our stochastic samplers
49


Reviews:
[{"review_id": "MiZPR94vyf", "reviewer": "Reviewer_9LJY", "summary": "Very sorry for the late review! I hope the authors will find my advice helpful! As the review is late, I think this is more like a piece of advice for the authors about how to improve this idea and submission. Please feel free to tell me if you have concerns about some of my suggestions!\n\nThis paper is based on a novel and fundamental idea, which has not been explored previously. I am supervised by the authors when they present their ideas, as it is a so fundamental and elegant formulation while I have not thought of it previously.\n\nThe authors propose to study the integral in a projected space, where the new variable has a stable homeomorphism with the original variable. Under this setting the original integral is equivalent to the new integral, while under careful design, the new integral could be much more easy and fast to compute. Almost all accelerating methods, as I can recall, can fit in this formulation, which means it can generally define the theory insights of all acceleration methods. It could be a good idea to simplify and extend the definition in this paper to a more generalized form. While this paper only considers linear projections, I think it is enough to cover most cases.\n\nThe authors then split the new integrals into multiple components, which earns further accelerations. The authors give enough discussions about the error analysis. Extensive studies on small datasets are used to fairly evaluate the proposed methods.", "questions": "how the dpm-solver will formulate under your settings and when will your method outperform it? \n\nSorry for the late review, hope you will find those points useful.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The formulation of the problem is fundamental and sound. Very promising direction of future research\n\n2. Theory analysis about the stability and error is solid.\n\n3. The method can extend to many familiar sampling techniques, providing a unified and novel insight for all of them.", "weaknesses": "1. I don't know whether I deviate too much from the author's experiences. In my experience, in most cases, 25-step DPM-solver sampling could produce good enough sampling results for large models like StableDiffusions on large diverse image distribution; and less than 15-step DPM-solver sampling could produce good enough results for small diffusion models in small data domain. Please tell me if you observe a different phenomenon, and I will be happy to test it. Getting back to the subject, are the 100 and 50 NFEs considered by this paper a bit redundant?\n\n2. In eq 1, the authors actually simplify the original diffusion process to linear ones, and replace the f(z_t) function to linear map F_t z_t, so do please kindly mention that in the corresponding place. Otherwise, it may cause confusion. I care about the scope of this simplification, could it be applied to generation diffusion processes? Especially those in large-scale datasets and models? For example, \n\n3. The stability analysis seems to avoid the key points? I would like to know a two-phase conclusion: first, when will matrix A_t stability have an inverse, this is the key to the stable conjuncture; second, under the condition that A_t is inversible and stable, when will eq 7 and 8 as ODE solvers be stable? Theorem 2 seems like a naive application of ODE stability conditions, and gets rid of the key part. A_t is an integral of multiple components, so we may not directly assume it to be stably invertible. But you also do not assume it stable anyway (minimum eigenvalue larger than some positive constant). It would also be helpful if the authors could explain in what reality settings those conditions can be satisfied.\n\n4. Theorem 3 gives error analysis based on h, I have concerns about its error with respect to the original t. After the projection is to solve the original problem, measuring the error in a projected space could be less meaningful. This is the same problem in the DPM-Solver paper, they only analyze errors when lambda is stably transformed from t, but in reality is often not the case. But DPM-Solver performs great generally so I am not criticizing it. Just point out that error analysis with respect to t variable is preferred for typical ode numerical analysis.\n\n5. This work, also reminds me of the DPM-Solvers. DPM-Solver could be viewed as a special case, it also projects the integral to another space, the lambda space in fact. It uses a prediction-correction (PC) method to achieve much higher accuracy in the projected space. So I care about two things: first how the dpm-solver will formulate under your settings and when will your method outperform it? Second, can your method benefit from PC? While this work does not compare with DPM-Solver, considering the huge influences of and similarity with DPM-Solver, it could be better if the authors could compare with it.\n\n6. Improvements seem not to be good enough, especially considering speed. Large datasets and models are also not considered.\n\n\nSome minor things and suggestions include:\n1. I recommend using the ICLR official math notations, including in the original tex file, to express math concepts. For example, $dt$ should be $\\mathrm{d}t$, $\\bm{z}_t$ is better than $\\rv{z}_t$ in an ODE equation (and I also think is better in SDEs), and set A should be $\\mathcal{A}$.\n2. Remember to add , or . after equations, they are components of your sentences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4X5E8gHLsL", "reviewer": "Reviewer_eNRC", "summary": "The authors propose two complementary frameworks for accelerating sample generation in pre-trained models: 'Conjugate Integrators' and 'Splitting Integrators'. Conjugate integrators generalize DDIM, mapping the reverse diffusion dynamics to a more amenable space for\nsampling. In contrast, splitting-based integrators reduce the numerical simulation error by alternating between numerical updates involving the data and auxiliary variables. The authors test these approaches as well as combinations of these methods on different benchmark datasets.", "questions": "- Do the authors have an explanation why some methods outperform CSPS-D for very small numbers of NFE? (Figure 5, up-left plot). \n- Have the authors measured the total integration time? If the generation speed is our main objective, then such results should be provided as well.\n- How is $m_0$ defined during the training of 'Conjugate Symplectic Euler' and 'Conjugate Velocity Verlet'? That is, what is the generated $m_{\\epsilon}$. In my opinion, the paper would be improved if corresponding training algorithms to Algorithms 2 and 3 would be added. On a related note, does the transformation along the position dimensions $x_t$ remain a diffeomorphism in the augmented setting? If not, this could have negative implications about density estimation, and mode coverage in generation.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The theory is interesting and opens up many potential paths for future investigations. \n- For some datasets and low-medium number of total integration steps the aforementioned hybrid model shows excellent generative capabilites and outperform many well-known state of the art methods. \n- The writing is clear and the paper is well structured.", "weaknesses": "- Combinations between datasets and methods are not consistent. For example in Figure 5, EDM is not tested on CelebA-64. Furthermore, these methods must be tested in more complex distributions such as ImageNet. \n- The training of each pretrained model and the size of the model is not specified in the main paper. Are these models identical and trained for the same amount of time?\n- In Collorary 1, the authors show a connection between stability and the parameter $\\lambda$ (which should evolve with time?), however it is not clear how this hyperparrameter is chosen. Furthermore, Corollary 1 does not explain the behaviour of $\\lambda$-DDIM-II, which performs best.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VUHqquz4Gp", "reviewer": "Reviewer_dkXq", "summary": "The paper proposes two frameworks for improving integrators in generative diffusion models, conjugate integrators and splitting integrators, and it proposes a hybrid method combining the two. The integrators are developed from previous use in physics simulation, e.g. molecular dynamics, and they are applicable to augmented diffusion models, for example when including momentum. The authors provide intution behind the integrators and investigate theoretical properties. Finally, they demonstrate experimentally the power of the integrators.", "questions": "no questions", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- very clear and well-written paper\n- very clever application of numerical integration methodology for generative diffusion models\n- potentially high impact in improving results for fixed computational budgets", "weaknesses": "no apparent weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qof4KXhjSk", "reviewer": "Reviewer_8Lyn", "summary": "This paper...\n- proposes conjugate integrators and splitting integrators for accelerating diffusion sampling,\n- introduces practical changes to splitting integrators for efficient sampling (\"reduced\" samplers),\n- combines conjugate and splitting integrators to achieve competitive performance in CIFAR-10, CelebA-64, and AFHQ-64 sampling.", "questions": "- The relative behavior of stochastic (SPS-S, CSPS-S) and deterministic samplers (CSPS-D) is un-intuitive. Common knowledge is that deterministic samplers outperform stochastic samplers in the low-NFE regime, and vice versa in the high-NFE regime. But, we observe a reverse trend in Figure 5 bottom. Can the authors clarify why this happens?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Studies conjugate and splitting integrators, which were relatively unexplored in diffusion sampling.\n- Proposed samplers generalize previous diffusion integrators, such as DDIM.\n- Clearly explains the advantages of conjugate and splitting integrators, and how they contribute in orthogonal ways.", "weaknesses": "While the paper has clear strengths, I think the paper needs a major revision. Specifically, I am inclined to give \"reject\" for the following reasons:\n\n**Weakness 1 : Difficult to figure out the \"main\" contribution**\n- There are too many proposed integrators, and it is difficult to figure out which one is / ones are the \"main\" contribution of the paper. More specifically, it is difficult to see in which situation we should prefer one integrator over the other. To me, it seems like the authors are taking a very unclear stance on their \"main\" sampler : the authors are proposing a whole arsenal of samplers, and picking results which happened to beat some baselines results under very specific dataset+sampler+hyper-parameter combinations.\n- For instance, what is the rationale behind choosing $B_t = \\lambda I$ and $B_t = \\lambda 1$? When should we prefer DDIM-I over DDIM-II and vice versa? While there is a theorem on DDIM-I, I don't see any theorem on DDIM-II, which could clarify differences between DDIM-I and DDIM-II.\n- Also, readers could expect conjugate+splitting integrators to out-perform conjugate/splitting integrators, as they combine the best of both worlds. However, we don't see this trend in Table 2, where we see Reduced OBA out-performing Conjguate OBA. The authors hypothesize \"this might be due to a sub-optimal choice of $B_t$\" -- I think this kind of explanation only confuses the readers, as it does not provide any guide on when we should and should not use conjugate splitting.\n- I think one factor that makes this paper confusing is the lack of theoretical results -- the authors rely mostly on numerical results to judge the performance of integrators. A theorem comparing, e.g., truncation error of proposed samplers would greatly improve the strength of the paper.\n\n**Weakness 2 : Incomplete / questionable baseline results**\n- The authors claim if $B_t = 0$, conjugate integrator is equivalent to DDIM. But, if we see Figure 2 (a), FID for DDIM ($\\lambda$-DDIM with $B_t = 0$) is too poor, compared to results in the original DDIM paper. For instance, in the original DDIM paper, DDIM achives FID 13.36 with NFE=10 while in Figure 2 (a), we see FID > 50.\n- Result for EDM + pre-conditioning is missing in Figure 5. Why do the authors add pre-conditioning for their methods, but not for EDM? Is it because EDM + pre-conditioning out-performs the proposed integrators? For instance, EDM + pre-conditioning achieves 1.97 FID with NFE=35, which beats the best result in this paper, 2.11 FID with NFE=100.\n\n**Weakness 3 : Incomplete evaluation**\n- Results on higher-dimensional data is missing. I would like to see additional results on $\\geq 512$ resolution images.\n- Results on conditional generation is missing. I would like to see additional results on e.g., class-conditional and text-conditional generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)