SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Iterative procedure	Algorithm 2 SEEDS-1(Fθ, x̂s, s, t)
Input: initial value xT, steps {ti}M=0, model Fθ
Initialize x̂t0← xT
for i = 1 to M - 1 do
(t, s) ← (ti, ti-1), h ← λt - λs
x̂t = SEEDS-k(Fθ, x̂s, s, t)
end for
Return x̂tM ← last-step(xtM-1, tM-1, tM)	z ← N(0,1)
x̂t ← αt/αs x̂s - 2σt(eh-1) Fθ(x̂s, s) - σt√e2h-1z
Algorithm 3 SEEDS-2(Fθ, x̂s, s, t)
s1 ← tλ(λs + h/2), (z^1, z^2) ← N(0, Id) ⊗ N(0, Id)
u ← αs1/αs x̂s - 2σs1(eh/2 - 1) Fθ(x̂s, s) - A, A := σs1√eh-1z^1
x̂t ← αt/αs x̂s - 2σt(eh-1) Fθ(u, s1) - B, B := σt(√e2h-eh^2z^1 + √eh^2 - 1z^2)
Algorithm 4 SEEDS-3(Fθ, x̂s, s, t) with 0 < r1 < r2 < 1
s1 ← tλ(λs + r1h), s2 ← tλ(λs + r2h), (z^1, z^2, z^3) ← N(0, Id)^⊗3
u1 ← αs1/αs x̂s - 2σs1(er1h - 1) Fθ(x̂s, s) - σs1√e2r1h-1z^1
A ← σs2(√e2r2h - e2r1hz^1 + √e2r1h - 1z^2)
u2 ← αs2/αs x̂s - 2σs2(er2h - 1) Fθ(x̂s, s) - 2σs2r2/r1(er2h-1/r2h - 1)(Fθ(u1, s1) - Fθ(x̂s, s))) - A
B ← σt(√e2h - e2r2hz^1 + √e2r2h - e2r1hz^2 + √e2r1h - 1z^3)
x̂t ← αt/αs x̂s - 2σt(eh-1) Fθ(x̂s, s) - 2σt/r2(eh-1/h - 1)(Fθ(u2, s2) - Fθ(x̂s, s))) - B
Proposition 4.2. Assume that  $\hat{F}_{\theta}$  is a  $\mathcal{C}^{2n + 1}$ -function with respect to  $\lambda$ . Then the truncated Itô-Taylor expansion of (8) reads, for  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I}_d)$ ,
$$
\mathbf {x} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \mathbf {x} _ {s} - 2 \bar {\sigma} _ {t} \sum_ {k = 0} ^ {n} h ^ {k + 1} \varphi_ {k + 1} (h) \hat {F} _ {\theta} ^ {(k)} \left(\hat {\mathbf {x}} _ {\lambda_ {s}}, \lambda_ {s}\right) - \bar {\sigma} _ {t} \sqrt {e ^ {2 h} - 1} \epsilon + \mathcal {R} _ {n + 1}, \tag {16}
$$
with  $\hat{F}_{\theta}^{(k)}(\widehat{\mathbf{x}}_{\lambda},\lambda) = L_{\lambda}^{k}\hat{F}_{\theta}(\widehat{\mathbf{x}}_{\lambda},\lambda)$ , with  $L_{\lambda}$  is an infinitesimal operator defined in Appendix E.2.2 and  $\mathcal{R}_{n + 1}$  consists on the usual deterministic residue and all iterated integrals of length at greater or equal to 2 in which there is at least one stochastic component among them.
Our approach for constructing derivative-free 2-stage and 3-stage SEEDS schemes consists on exploiting the analytic computation of the Itô-Taylor coefficients in Proposition 4.2 and replace the  $\hat{F}_{\theta}^{(k)}(\widehat{\mathbf{x}}_{\lambda},\lambda)$  terms by well-adapted correction terms which do not need any derivative evaluation and dropping the  $\mathcal{R}_{n + 1}$  contribution as in the Runge-Kutta approach.
Markov-preserving noise decomposition. We use collocation methods for constructing higher-stage derivative-free solvers. Although the chosen truncated Itô-Taylor expansion produces approximations for the deterministic integral similar to [22], adding the corresponding noise contribution found by the SETD method at each step does not yield Markov chains in general. The reason is that stochastic integrals on overlapping time intervals need to be dependent, a phenomenon that has no deterministic counterpart. As such, our last and key element to construct SEEDS consists on a novel decomposition of stochastic integrals which enforces the Markov property for multi-stage SEEDS.
Algorithms 1 to 4 prescribe all SEEDS schemes obtained by this procedure in the VP case. We now show (see App. C for the proofs) that all methods yield Markov chains and are weakly convergent.
Proposition 4.3. The sequences  $\{\tilde{\mathbf{x}}_t\}_t$  induced by the choice of stochastic noise contributions presented in Algorithms.3 and 4 satisfy the Markov property.
Corollary 4.4. Under Assumption C.2, the numerical solutions  $\tilde{\mathbf{x}}_t$  produced by the SEEDS methods (3) and (4) converge to the exact solution  $\mathbf{x}_t$  of (15) with coefficients (7) in weak sense with global order 1 in both cases: there is a constant  $C > 0$  such that, for any continuous bounded function  $G$ :
$$
\left| \mathbb {E} \left[ G \left(\tilde {\mathbf {x}} _ {t _ {M}}\right) \right] - \mathbb {E} \left[ G \left(\mathbf {x} _ {t _ {M}}\right) \right] \right| \leqslant C h.
$$
Comparison with existing sampling methods. Let us now examine the connection between SEEDS and existing sampling techniques used for DPMs, emphasizing the contrasts between them.
The main distinctive feature of SEEDS is that they are off-the-shelf solvers. This means that, not only they are training-free, contrary to [9], but they do not require any kind of optimization procedure to achieve their optimal results. This is in contrast to methods such as: gDDIM, which is training-free but not off-the-shelf as one needs to make preliminary optimization procedures such as simulating the transition matrix of their method in the CLD case; Heun-Like method from EDM (for all baseline models and the EDM-optimized models for ImageNet) since they need preliminary optimization procedures on 4 parameters which actually break the convergence criteria. Moreover, neither gDDIM, EDM nor the SSCS method in [10] present full proofs of convergence for their solvers. Also, both DEIS and gDDIM identify their methods with stochastic DDIM theoretically, but the poor results obtained by their stochastic solvers do not yield to further experimentation in their works. In a way, SEEDS can be thought as improved and generalized DPM-Solver to SDEs. Nevertheless, such generalization is not incremental as the tools for proving convergence in our methods involve concepts which are exclusive to SDEs. We now make rigorous statements of the above discussion.
Proposition 4.5. Consider the SEEDS approximation of (15) with coefficients (7). Then
1. If we set  $g = 0$  in (15), the resulting SEEDS do not yield DPM-Solver.
2. If we parameterize (15) in terms of the data prediction model  $D_{\theta}$ , the resulting SEEDS are not equivalent to their noise prediction counterparts defined in Alg. 1 to 4.
3. The gDDIM solver [41, Th. 1] equals to SEEDS-1 in the data prediction mode, for  $\ell = 1$ .
The first point makes it explicit that SEEDS are not incremental based on DPM-Solver. The second point in Prop. 4.5 is analogous to the result in Appendix B of [23], where the authors compare DPM-Solver2 and DPM-Solver++(2S), that is the noise and data prediction approaches, and find that they do not equate. The last point exhibits gDDIM as a special case of SEEDS-1 for isotropic DPMs.
Remark 4.6. Building solvers from the representation of the exact solution in (5) requires computing the transition matrix  $\Phi_A(t,s)$ , which cannot be analytically computed for non-isotropic DPMs such as CLD [10]. Nevertheless, the SEEDS approach can be applied in this scenario in at least two different ways. On the one hand, the SSCS method from [10] resides splitting  $\Phi_A(t,s)$  into two separate terms. The first can be analytically computed. The second describes the evolution of a semi-linear differential equation [10, Eq. 92]. While [10] approximates the latter by the Euler method, crafting exponential integrators for approximating such DE may yield an acceleration of the SSCS method. On the other hand, gDDIM [41] proposes an extension of DEIS sampling [40] to CLD by setting a pre-sampling phase [41, App. C.4] in which they compute an approximation of  $\Phi_A(t,s)$  in order to apply their method, and the latter was shown in Prop. 4.5. to be a special case of our method. Unfortunately, the authors did not release pre-trained models in [41], and the latter are not the same as those in [10]. Sampling in this scenario may also benefit from our approach.
We compare SEEDS with several previous methods on discretely and continuously pre-trained DPMs. We report results of many available sources, such as DDPM [12], Analytic DDPM [2], PNDM [20], GGF [15], DDIM [33], gDDIM [41], DEIS [40] and DPM-Solver [22]. Although we do not include training-based schemes here, we still included GENIE [9], which trains a small additional network but still solves the correct generative ODE at higher-order. For each experiment, we compute the FID score for 50K sampled images on multiple runs and report the minimum along different solvers. Details on model specifications and experiment illustrations are shown in Appendix F.
Practical considerations. For continuously trained models, SEEDS use the EDM discretization [16, Eq. 5] with default parameters and does not use the last-step iteration trick, meaning that the last iteration of SEEDS is trivial. For discretely trained models, SEEDS use the linear step schedule in the interval  $\left[\lambda_{t_0},\lambda_{t_N}\right]$  interval following [22, Sec. 3.3, 3.4]. All the reported SEEDS results were obtained using the noise prediction mode. We conducted comparative experiments on SEEDS for both the data and noise prediction modes and found better results with the latter (see Tab. 3 for details). EDM solvers [16, Alg. 2] depend on four parameters controlling the amount of noise to
Table 1: Sample quality measured by FID  $\downarrow$  on pre-trained DPMs. We report the minimum FID obtained by each model and the NFE at which it was obtained. For CIFAR, CelebA and FFHQ, we use baseline pre-trained models [34, 16]. For ImageNet, we use the optimized pre-trained model from [16]. *discrete-time model, *continuous-time model, †recomputed FID for the non-deep model.
SAMPLING METHOD	FID↓	NFE
CIFAR-10* VP-UNCOND.
DDIM [33]	3.95	1000
ANALYTIC-DDPM [2]	3.84	1000
GENIE [9]	3.64	25
ANALYTIC-DDIM [2]	3.60	200
F-PNDM (LINEAR) [20]	3.60	250
DPM-SOLVER† [22]	3.48	44
F-PNDM (COSINE) [20]	3.26	1000
DDPM [12]	3.16	1000
SEEDS-3 (OURS)	3.08	201
CIFAR-10* VP-COND.
DPM-SOLVER† [22]	3.57	195
EDM (Schum = 0) [16]	2.48	35
SEEDS-3 (OURS)	2.08	129
CIFAR-10* VP-UNCOND.
DPM-SOLVER† [22]	2.59	51
GGF [15]	2.59	180
GDDIM [41]	2.56	100
DEIS ρ3KUTTA [40]	2.55	50
EULER-MARUYAMA [34]	2.54	1024
STOCHASTIC EDM [16]	2.54	1534
SEEDS-3 (OURS)	2.39	165
EDM (OPTIMIZED) [16]	2.27	511
SAMPLING METHOD	FID↓	NFE
CELEBA-64* VP-UNCOND.
ANALYTIC-DDPM [2]	5.21	1000
DDIM [33]	4.78	200
DDPM [12]	3.50	1000
GDDIM [41]	3.85	50
ANALYTIC-DDIM [2]	3.13	1000
3-DEIS [40]	2.95	50
F-PNDM (LINEAR) [20]	2.71	250
DPM-SOLVER [22]	2.71	36
SEEDS-3 (OURS)	1.88	90
FFHQ-64* VP-UNCOND.
DPM-SOLVER† [22]	3.52	90
SEEDS-3 (OURS)	3.40	150
EDM (Schum = 0) [16]	3.39	79
IMAGENET-64 EDM-COND.
DPM-SOLVER† [22]	3.01	270
EDM (Schum = 0) [16]	2.22	511
SEEDS-3 (OURS)	1.38	270
EDM (OPTIMIZED) [16]	1.36	511
(a) Uncond. CIFAR, VP
(b) Uncond. CIFAR, VP
(c) Stochasticity on ImageNet-64
Figure 1: (a-b) Comparison of sample quality measured by FID  $\downarrow$  of SEEDS, DPM-Solver and other methods for discretely trained DPMs on CIFAR-10 with varying number of function evaluations. (c) Effect of  $S_{\mathrm{churn}}$  on SEEDS-3 (at NFE = 270) and EDM method (at NFE = 511) on class-conditional ImageNet-64.  $\dagger$ baseline ADM model.  $\star$  EDM preconditioned model.
be injected in a specific sub-interval of the iterative procedure. We consider three scenarios: when stochasticity is injected along all the iterative procedure, we denote it stochastic EDM, when no stochasticity is injected we denote it EDM ( $S_{\mathrm{churn}} = 0$ ) and we denote EDM (Optimized) the case where such parameters were subject to an optimization procedure. To better evaluate sampling quality along the sample pre-trained DPM, we recalculate DPM-Solver for sampling from the non-deep VP continuous model on the CIFAR-10 dataset. All implementation details can be found in Appendix D.
Comparison with previous works. In Table 1 we compare SEEDS with other sampling methods for pre-trained DPMs, and report the minimum FID obtained and their respective NFE. For each of the reported pre-trained models in CIFAR-10, CelebA-64 and ImageNet-64, SEEDS outperform all off-the-shelf methods in terms of quality with relatively low NFEs. For the discrete pre-trained DPM on CIFAR-10 (VP Uncond.) it is  $\sim 5\times$  faster than the second best performant solver. Additionally, SEEDS remain competitive with the optimized EDM sampler. For ImageNet-64, it is nearly as good
Figure 2: Trajectories of 10 pixels (R channel) sampled from SEEDS (1st line) and Stochastic EDM (2nd line) on the optimized pre-trained model [16] on ImageNet64. Schedule=scaling=vp corresponds to the VP coefficients in (7) and schedule=linear, scaling=none to the EDM coefficients (11). We use the time discretizations disc=vp (linear) and disc=edm given in [16, Tab.1].
as the optimized EDM sampler while being almost twice faster than the latter. Figure 1 (a) compares the FID score of SEEDS and DPM-Solver with varying NFEs. While DPM-Solver methods stabilize faster in a very low NFE regime, our methods eventually surpass them. Interestingly, after reaching their minimum, SEEDS methods tend to become worse at higher NFEs, a fact that is also visible in Figure 1 (b), where we notice that such phenomenon is also present on other SDE methods. We report in Appendix F, the results of our SEEDS methods in the low NFE regime and connect their behavior with their proven convergence rate.
Combining SEEDS with other methods. While being an off-the-shelf method, SEEDS can be combined with the Churn-like method used in EDM incurring into SDE solvers with an additional source of stochasticity. As done in [16], we evaluate the effect of this second kind of stochasticity, measured by a parameter denoted  $S_{\mathrm{churn}}$ . Figure 1 (c) shows that SEEDS and EDM show similar behavior, although SEEDS-3 is twice faster, more sensitive to  $S_{\mathrm{churn}}$ , and quickly achieves comparable performance to EDM. This indicates that SEEDS could possibly outperform EDM after a proper parameter optimization that will be left for future works. Nevertheless, we highlight the fact that obtaining such optimal parameters is costly and might scale poorly.
Stiffness reduction with SEEDS. In Fig. 2, we illustrate the impact of different choices of discretization steps, noise schedule and dynamic scaling on SEEDS and stochastic EDM. We see that choosing the EDM discretization over the linear one has the effect of flattening the pixel trajectories at latest stages of the simulation procedure. Also, choosing the parameters (11) over those in (7) has the effect of greatly changing the distribution variances as the trajectories evolve. Notice that all the SEEDS trajectories seem perceptually more stable than those from EDM. It would be interesting to relate this to the stiffness of the semi-linear DE describing these trajectories, and to the magnitude of the parameters involved in the noise injection for EDM solver amplifying this phenomenon.
Ablation studies. As said earlier, our principled use of the Chasles rule to enforce independence only on non-overlapping paths for SEEDS-2/3 ensures that the set of resulting iterations of our solvers satisfy the Markov property, is new and is the central key of success of SEEDS. To highlight this,
(a) 2-stage SDE methods
(b) 3-stage SDE methods
Figure 3: Quality (measured by FID at increasing NFEs) comparison of SEEDS-2/3 with the enumerated ablation versions of it on CIFAR-10 in the (baseline) VP conditional framework.
we conduct an ablation study on how 4 different combinations of the noise components  $A$  and  $B$  in Algorithms 3 and 4 have an impact on the sampling quality of SEEDS.
For simplicity, we explain this for SEEDS-2 (Alg. 3). Set  $(z^1,z^2,z^3)$  three independent standard Gaussian random variables. Denote  $A = \bar{\sigma}_{s_1}\sqrt{e^h - 1} z^1$  for the noise contribution in the  $\mathbf{u}$  term. We have the following choices for the noise contribution  $B$  in  $\tilde{\mathbf{x}}_t$ :
- SEEDS-2-Correct: our noise combination  $\mathbf{B} = {\bar{\sigma }}_{t}\left( {\sqrt{{e}^{2h} - {e}^{h}}{z}^{1} + \sqrt{{e}^{h} - 1}{z}^{2}}\right)$
- SEEDS-2-Naive-1: one noise per stage  $\mathbf{B} = {\bar{\sigma }}_{t}\left( {\sqrt{{e}^{2h} - {e}^{h}} + \sqrt{{e}^{h} - 1}}\right) {z}^{2}$
- SEEDS-2-Naive-2: one noise per step  $\pmb{B} = \bar{\sigma}_{t}\left(\sqrt{e^{2h} - e^{h}} +\sqrt{e^{h} - 1}\right)z^{1}$
- SEEDS-2-Naive-3: one noise per integral evaluation  $\mathbf{B} = {\bar{\sigma }}_{t}\left( {\sqrt{{e}^{2h} - {e}^{h}}{z}^{3} + \sqrt{{e}^{h} - 1}{z}^{2}}\right)$
- SEEDS-2-Naive-4: noises in inverse position  $\mathbf{B} = \bar{\sigma}_{t}\left(\sqrt{e^{2h} - e^{h}} z^{2} + \sqrt{e^{h} - 1} z^{1}\right)$ .
Figure 3 experimentally shows that all naive combinations of noises for SEEDS-2 and SEEDS-3 lead to FID/NFE curves with different behavior and a sharp drop of performance both in quality and speed in all of them, a phenomenon having no deterministic parallel.
Our focus is on addressing the challenge of training-free sampling from DPMs without compromising sampling quality. To achieve this, we introduce SEEDS, an off-the-shelf solution for solving diffusion SDEs. SEEDS capitalize on the semi-linearity of diffusion SDEs by approximating a simplified formulation of their exact solutions. Inspired by numerical methods for stochastic exponential integrators, we propose three SEEDS schemes with proven convergence order. They transform the integrals involved in the exact solution into exponentially weighted integrals, and estimate the deterministic one while analytically computing the variance of the stochastic integral. We extend our approach to handle other isotropic DPMs, and evaluate its performance on various benchmark tests. Our experiments demonstrate that SEEDS can generate images of optimal quality, outperforming existing SDE solvers while being  $3 \sim 5 \times$  faster. Limitations and broader impact. While SEEDS prioritize optimal quality sampling, they may require substantial computational resources and energy consumption, making them less suitable for scenarios where speed is the primary concern. In such cases, alternative ODE methods may be more appropriate. Additionally, as with other generative models, DPMs can be employed to create misleading or harmful content, and our proposed solver could inadvertently amplify the negative impact of generative AI for malicious purposes.
Acknowledgments. This work has been supported by the French government under the "France 2030" program, as part of the SystemX Technological Research Institute.
1 Introduction 1
2 Background on Diffusion Probabilistic Models 2
3 Accelerating Optimal Quality Solvers for Diffusion SDEs 3
4 Higher Stage SEEDS for DPMs 5
5 Experiments 7
6 Conclusions 10
A Discussion 14
B Detailed Derivation of the SEEDS Design Space 16
B.1 The Isotropic General SDE framework 16
B.2 Re-framing and Generalizing Previous Exponential Solvers 18
B.2.1 The VP case 18
B.2.2 Proof of Proposition 3.1 18
B.2.3 Proof of Proposition 4.2 19
B.2.4 Generalization to the remaining data prediction and deterministic modes 20
B.2.5 Proof of Proposition 4.5 21
B.2.6 The VE, DDIM and iDDPM cases 23
B.2.7 The EDM case 23
B.2.8 Proof of Proposition 3.2 24
B.2.9 The SEEDS algorithms in the EDM Noise Prediction case 25
C Convergence Proofs 26
C.1 Preliminaries 27
C.2 Convergence of SEEDS-1 28
C.2.1 Strong Itô-Taylor approximation 28
C.2.2 Continuous approximation of SEEDS-1 29
C.2.3 Proof of Theorem 4.1 31
C.2.4 Discrete-time approximation 33
C.2.5 Useful Lemmas 34
C.3 Proof of Proposition 4.3 35
C.4 Proof of Corollary 4.4 36
C.4.1 Convergence of SEEDS-2: 36
C.4.2 Convergence of SEEDS-3: 37
D Implementation Details 38
D.1 Stabilization of the exponential terms 38
D.2 Noise schedules parameterizations 39
D.3 EDM discretization 39
D.4 Final sampling step 40
E Reminders on Stochastic Exponential Integrators 40
E.1 Derivative-free exponential ODE schemes 40
E.1.1 Runge-Kutta approach 40
E.1.2 Exponential Runge-Kutta approach 41
E.2 Derivative-free exponential SDE schemes 44
E.2.1 Strong and Weak Stochastic Runge-Kutta approach 45
E.2.2 Truncated Itô-Taylor expansions 45
E.2.3 Integral approximations 47
FExperiment Details 49
F.1 Pre-trained model specifications 49
F.2 Noise vs. Data Prediction approaches 50
F.3 Low vs. High stage Solvers 50
F.4 Deterministic vs. Stochastic Solvers 51
F.5 Run-time Comparison 51
F.6 Hardware configuration 52
F.7 Licences 52
F.8 Supporting samples 52
Why do SEEDS exhibit high FID scores in the low NFE regime? In Appendix F.4, we provide all tables with FID values at increasing NFEs corresponding to Fig. 1. Notice that SEEDS exhibit high FID scores in the low NFE regime. Let us discuss some theoretical facts that might explain this phenomenon. In [38, Theorem 1.], there is a formal explanation on why ODE samplers outperform SDE samplers in the small NFE regime and fall short in the large NFE regime. In particular, it is theoretically shown that, at large step sizes, it is the discretization error that dominates sampling errors while, at small step sizes, it is the approximation error that dominates it. One can infer that in the large NFE regime, SDE methods (with proven convergence orders) will outperform ODE methods in terms of sampling quality. It would be interesting to see if combining SEEDS with the new ideas in [38] might improve SEEDS in the low NFE regime.
Additionally, SDE solvers and ODE solvers may have different sample quality, even if the score models are optimal. Indeed, as proven in [21, Appendix B.], even if the score model has been trained to the optimal score function, the distributions between SDEs and ODEs are still different. This is because the distribution at time  $T$  in the forward process is always not exactly a standard normal distribution as in the reverse SDE/ODE process, and thus the distributions at time 0 are different.
Finally, [38, 5] show that the overall generalization error divides into discretisation and approximation errors, indicating the theoretical role of the sampling error as a contribution to a DPM's performance. This is also consistent with the suggestion in [16, Section 5], stating that more diverse datasets continue to benefit from stochastic sampling rather than deterministic sampling.
Do SEEDS maintain good performances for higher resolution image generation? As training-free and optimization-free SDE solvers, SEEDS naturally maintain good performances in higher
resolution images in the realm of unguided image generation (unconditional and conditional). For unconditional generation using Latent Diffusion Model, SEEDS are able to generate good quality images already at 100 NFEs. Now, for guided image generation, higher-stage SEEDS will, as expected, see their performance sharply drop as the guidance scale grows for the same reasons DPM-solver do (see [23] for details). Yet, SEEDS-1 still maintains high quality sampling in this scenario. In Appendix F.8, Fig. 15 exhibits a  $512^{2}$  image generated with SEEDS-1 at 90 NFEs on Stable Diffusion with default guidance scale.
Do SEEDS exhibit other distinguishing features beyond generation optimal quality? As SDE solvers, SEEDS have a distinctive (although indirect) capability in the realm of adversarial robustness compared to DPM-Solver:
1. For more than 2 years, the leader-board of RobustBench has been dominated by Diffusion-based data augmentation techniques on top of Adversarial Training. The current SOTA [28, 37] uses EDM-preconditioned DPMs and need to generate as many as 50M images to achieve SOTA robustness results. For ImageNet-64, [37] use the EDM pretrained model and optimization-based stochastic EDM sampler for data augmentation, leading to a  $5\%$  robust accuracy improvement compared to doing so for the baseline ADM pretrained model. Since SEEDS reach same FID quality as EDM, but twice faster, we believe it will have a positive impact in this domain, making diffusion-based data augmentation schemes more affordable with limited computational capacity.
2. The work [27] uses DPM-based adversarial purification as a test-time adversarial defense. The idea is to use off-the-shelf DPMs to annihilate the adversarial content in inputs in test-time before feeding it to a pretrained classifier. In [27, Table 6], one can see that SDE solvers show substantial robustness capabilities compared to ODE solvers.
We'd like to stress out that, although indirect, such capabilities are on the side of the sampling methods rather than on the side of intrinsic robustness properties of score-based learning method. More broadly, robustness properties of ML models determined as neural differential equations (DPMs being in this scope) has been studied in [39, 11].