SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} R _ {1} \\ = \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} L _ {t} f (\mathbf {x} (\tau_ {2}), \tau_ {2}) \mathrm {d} \tau_ {2} \mathrm {d} \tau_ {1} + \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} G _ {t} f (\mathbf {x} (\tau_ {2}), \tau_ {2}) \mathrm {d} \omega (\tau_ {2}) \mathrm {d} \tau_ {1} \\ = \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \left[ L _ {t} f (\mathbf {x} (t), t) + \int_ {s} ^ {\tau_ {2}} L _ {t} ^ {2} f (\mathbf {x} (\tau_ {3}), \tau_ {3}) \mathrm {d} \tau_ {3} + \int_ {s} ^ {\tau_ {2}} G _ {t} L _ {t} f (\mathbf {x} (\tau_ {3}), \tau_ {3}) \mathrm {d} \omega (\tau_ {3}) \right] \mathrm {d} \tau_ {2} \tau_ {1} \\ { + } { \int _ { s } ^ { t } \int _ { s } ^ { \tau _ { 1 } } \left[ G _ { t } f ( \mathbf { x } ( t ) , t ) + \int _ { s } ^ { \tau _ { 2 } } L _ { t } G _ { t } f ( \mathbf { x } ( \tau _ { 3 } ) , \tau _ { 3 } ) \mathrm { d } \tau + \int _ { s } ^ { \tau _ { 2 } } G _ { t } ^ { 2 } f ( \mathbf { x } ( \tau _ { 3 } ) , \tau _ { 3 } ) \mathrm { d } \omega ( \tau _ { 3 } ) \right] \mathrm { d } \omega ( \tau _ { 2 } ) \mathrm { d } \tau _ { 1 } } \\ = L _ {t} f (\mathbf {x} (t), t) \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \mathrm {d} \tau_ {2} \mathrm {d} \tau_ {1} + G _ {t} f (\mathbf {x} (t), t) \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \mathrm {d} \omega (\tau_ {2}) \mathrm {d} \tau_ {1} \\ + \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \int_ {s} ^ {\tau_ {2}} L _ {t} ^ {2} f (\mathbf {x} (\tau_ {3}), \tau_ {3}) d \tau_ {3} d \tau_ {2} d \tau_ {1} + \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \int_ {s} ^ {\tau_ {2}} G _ {t} L _ {t} f (\mathbf {x} (\tau_ {3}), \tau_ {3}) d \omega (\tau_ {3}) d \tau_ {2} d \tau_ {1} \\ + \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \int_ {s} ^ {\tau_ {2}} L _ {t} G _ {t} f (\mathbf {x} (\tau_ {3}), \tau_ {3}) \mathrm {d} \tau_ {3} \mathrm {d} \omega (\tau_ {2}) \mathrm {d} \tau_ {1} \\ + \int_ {s} ^ {t} \int_ {s} ^ {\tau_ {1}} \int_ {s} ^ {\tau_ {2}} G _ {t} ^ {2} f (\mathbf {x} (\tau_ {3}), \tau_ {3}) d \omega (\tau_ {3}) d \omega (\tau_ {2}) d \tau_ {1} \\ = L _ {t} f (\mathbf {x} (s), s) I _ {(0, 0)} + G _ {t} f (\mathbf {x} (s), s) I _ {(0, 1)} + L _ {t} ^ {2} f (\mathbf {x} (s), s) I _ {(0, 0, 0)} + G _ {t} L _ {t} f (\mathbf {x} (s), s) I _ {(0, 0, 1)} \\ + L _ {t} G _ {t} f (\mathbf {x} (s), s) I _ {(0, 1, 0)} + G _ {t} ^ {2} f (\mathbf {x} (s), s) I _ {(1, 1, 0)} + R _ {2}, \\ \end{array}
$$
and with  $R_{2}$  consisting on quadruple integrals. As such, our solution now reads
$$
\begin{array}{l} \mathbf {x} (t) = \mathbf {x} (s) + f (\mathbf {x} (s), s) h + g (s) (\omega (t) - \omega (s)) + L _ {t} f (\mathbf {x} (s), s) I _ {(0, 0)} + G _ {t} f (\mathbf {x} (s), s) I _ {(0, 1)} \\ + L _ {t} ^ {2} f (\mathbf {x} (s), s) I _ {(0, 0, 0)} + G _ {t} L _ {t} f (\mathbf {x} (s), s) I _ {(0, 0, 1)} \\ + L _ {t} G _ {t} f (\mathbf {x} (s), s) I _ {(0, 1, 0)} + G _ {t} ^ {2} f (\mathbf {x} (s), s) I _ {(1, 1, 0)} + R _ {2}. \\ \end{array}
$$
Now, in the SDE regime, one cannot get rid of the iterated Itô integral and so stochastic RK methods cannot be derived as simple extensions of their deterministic counterparts. In order to continue we now take into account the fact that the diffusion SDE has additive and diagonal noise. In this case, both the Itô and the Stratonovich SDE coincide.
Iterated integrals
Now, for simplicity, set  $t = 0$ . We then have  $I_{(0)} = h$ ,  $I_{(0,0)} = \int_0^h\int_0^{\tau_1}\mathrm{d}\tau_2\mathrm{d}\tau_1 = \frac{h^2}{2}$ ,  $I_{(0,0,0)} = \frac{h^3}{6}$ . Now notice that
$$
I _ {(1)} := \hat {w} _ {h} \sim \mathcal {N} (0, h)
$$
$$
{I _ {(0, 1)}} {:=} {\hat {z} _ {h} := \int_ {0} ^ {h} \int_ {0} ^ {\tau_ {1}} \mathrm {d} \omega (\tau_ {2}) \mathrm {d} \tau_ {1} = \lim _ {n \to \infty} \frac {h}{n} \sum_ {i = 0} ^ {n - 1} \sum_ {j = 1} ^ {i} \epsilon_ {j}, \qquad \epsilon_ {j} \sim \mathcal {N} \left(0, \frac {h}{n}\right).}
$$
Additionally,  $\hat{w}_h$  and  $\hat{z}_h$  satisfy  $\mathbb{E}[\hat{w}_h^2 ] = h$
$$
\mathbb {E} \left[ \left(\hat {w} _ {h} h - \hat {z} _ {h}\right) ^ {2} \right] = \mathbb {E} \left[ \left(\int_ {0} ^ {h} \tau \mathrm {d} \omega (\tau)\right) ^ {2} \right] = \frac {1}{3} h ^ {3}
$$
$$
\mathbb {E} [ \hat {w} _ {h} \hat {z} _ {h} ] = \mathbb {E} \left[ \hat {w} _ {h} \int_ {0} ^ {h} \tau \mathrm {d} \omega (\tau) \right] = \mathbb {E} \left[ \int_ {0} ^ {h} \tau \mathrm {d} \tau \right] = \frac {1}{2} h ^ {2}
$$
$$
\mathbb {E} [ \hat {z} _ {h} ^ {2} ] = \mathbb {E} [ (\hat {w} _ {h} h - \hat {z} _ {h}) ^ {2} - h ^ {2} \hat {w} _ {h} ^ {2} + 2 h \hat {w} _ {h} \hat {z} _ {h} ] = \frac {1}{3} h ^ {3}.
$$
E.2.3 Integral approximations
Weak Approximations
When crafting weak stochastic approximations to SDEs one may replace multiple Itô integrals by other random variables satisfying the corresponding moment conditions. We will denote  $\hat{I}_{\alpha}$  the approximation of  $I_{\alpha}$  for  $\alpha$  a multi-index following [17, Corollary 5.12.1]. Of course, the deterministic integrals  $I_{(0,\dots,0)}$  need not be approximated.
First order approximations
The random variable  $\hat{I}_{(1)}$  must satisfy for some constant  $K$ :
$$
\left| \mathbb {E} \left[ \hat {I} _ {(1)} \right] \right| + \left| \mathbb {E} \left[ \left(\hat {I} _ {(1)}\right) ^ {3} \right] \right| + \left| \mathbb {E} \left[ \left(\hat {I} _ {(1)}\right) ^ {2} - h \right] \right| \leqslant K h ^ {2}
$$
Two possible choices for  $\hat{I}_{(1)}$  are either  $\hat{I}_{(1)}\sim \mathcal{N}(0,h)$  or  $\hat{I}_{(1)}$  is a two-pointed distributed discrete random variable with
$$
\mathbb {P} \left[ \hat {I} _ {(1)} = \pm \sqrt {h} \right] = \frac {1}{2}.
$$
Second order approximations
$$
| \mathbb {E} [ \hat {I} _ {(1)} ] | + | \mathbb {E} [ (\hat {I} _ {(1)}) ^ {3} ] | + | \mathbb {E} [ (\hat {I} _ {(1)}) ^ {5} ] | + | \mathbb {E} [ (\hat {I} _ {(1)}) ^ {2} - h ] | + | \mathbb {E} [ (\hat {I} _ {(1)}) ^ {4} - 3 h ^ {2} ] | \leqslant K h ^ {2}.
$$
Two possible choices for  $\hat{I}_{(1)}$  are either  $\hat{I}_{(1)}\sim \mathcal{N}(0,h)$  or  $\hat{I}_{(1)}$  is a three-pointed distributed random variable with
$$
\mathbb {P} \left[ \hat {I} _ {(1)} = \pm \sqrt {3 h} \right] = \frac {1}{6}, \qquad \mathbb {P} [ \hat {I} _ {(1)} = 0 ] = \frac {2}{3}.
$$
The rest follows from the above calculations:
$$
\hat {I} _ {(0, 1)} = \frac {1}{2} h \hat {I} _ {(1)}, \qquad i = 0, 1.
$$
Third order approximations
One can choose  $\hat{I}_{(1)}\sim \mathcal{N}(0,h),\hat{I}_{(0,1)}\sim \mathcal{N}\left(0,\frac{1}{3} h^3\right)$  satisfying  $\mathbb{E}[\hat{I}_{(1)}\hat{I}_{(0,1)}] = \frac{1}{2} h^2$
Then, one can deduce the following:
$$
\hat {I} _ {(1, 0, 0)} = \hat {I} _ {(0, 1, 0)} = \hat {I} _ {(0, 0, 1)} = \frac {1}{6} h ^ {2} \hat {I} _ {(1)}
$$
$$
\hat {I} _ {(0, 1, 1)} = \hat {I} _ {(1, 0, 1)} = \hat {I} _ {(1, 1, 0)} = \frac {1}{6} h (\hat {I} _ {(1)} ^ {2} - h).
$$
Thus, we can write the solution weak approximation as
$$
\begin{array}{l} {\bf x} (t) = {\bf x} (s) + f ({\bf x} (s), s) h + g (s) \hat {I} _ {(1)} + L _ {t} f ({\bf x} (s), s) \frac {h ^ {2}}{2} + R _ {2} \\ + G _ {t} f (\mathbf {x} (s), s) \hat {I} _ {(0, 1)} + L _ {t} ^ {2} f (\mathbf {x} (s), s) \frac {h ^ {3}}{6} + G _ {t} L _ {t} f (\mathbf {x} (s), s) \frac {1}{6} h ^ {2} \hat {I} _ {(1)} \\ + L _ {t} G _ {t} f (\mathbf {x} (s), s) \frac {1}{6} h ^ {2} \hat {I} _ {(1)} + G _ {t} ^ {2} f (\mathbf {x} (s), s) \frac {1}{6} h \left(\hat {I} _ {(1)} ^ {2} - h\right). \tag {60} \\ \end{array}
$$
An example of such a pair  $(\hat{I}_{(1)},\hat{I}_{(0,1)}) = (\hat{w}_h,\hat{z}_h)$  can be easily obtained as follows
$$
\left[ \begin{array}{c} \hat {w} _ {h} \\ \hat {z} _ {h} \end{array} \right] = \left[ \begin{array}{c c} \sqrt {h} & 0 \\ \frac {h \sqrt {h}}{2} & \frac {h \sqrt {h}}{2 \sqrt {3}} \end{array} \right] \left[ \begin{array}{c} u _ {1} \\ u _ {2} \end{array} \right], \qquad u _ {1}, u _ {2} \stackrel {{\text {i . i . d .}}} {{\sim}} \mathcal {N} (0, 1).
$$
Indeed, for such a pair we have
$$
\mathbb {E} \left[ \left[ \begin{array}{c} \hat {w} _ {h} \\ \hat {z} _ {h} \end{array} \right] \left[ \begin{array}{c c} \hat {w} _ {h} & \hat {z} _ {h} \end{array} \right] \right] = \left[ \begin{array}{c c} h & \frac {h ^ {2}}{2} \\ \frac {h ^ {2}}{2} & \frac {h ^ {3}}{3} \end{array} \right].
$$
In light of the above expression, the truncated Taylor expansions we refer in the main part of the paper consists on the consideration of only the coefficients in  $L_{t}^{k}$ . The only noise noise contribution we will consider corresponds to  $g(s)\hat{I}_{(1)}$ .
We evaluate the Fréchet inception distance (FID) after generating 50K samples with each solver, and compare with the statistics of real-data. In our experiments we make use of the code from [16] for continuously trained models as well as their reference FID stats<sup>2</sup> and that of [22] for discretely trained models.
All the experiments of SEEDS for continuous-time models are parameterized within the EDM framework with the discretization of type EDM, linear schedule, and scaling none as described on [16] in noise prediction mode unless explicitly stated. We use the SEEDS-3 method that has 3 NFEs per step and fixed step-size and report FID scores at NFEs divisible by 3.
We leverage the explicit Langevin-like "churn" trick using in [16] to add or remove noise in the sampling phase. Specifically, [16] uses 4 hyper-parameters  $S_{\mathrm{churn}}$ ,  $S_{\mathrm{min}}$ ,  $S_{\mathrm{max}}$  and  $S_{\mathrm{noise}}$  in which  $S_{\mathrm{churn}}$  controls the overall amount of stochasticity added before giving the input to the SEEDS-3 method when the noise level (or time step in EDM configuration)  $t_i$  is contained in the noise interval  $[S_{\mathrm{min}}, S_{\mathrm{max}}]$ . It means that the EDM proposed sampler is stochastic under some conditions of those hyper-parameters and deterministic otherwise, while our method is completely stochastic. In our experiments, we set  $S_{\mathrm{churn}} = 0$  except for ImageNet-64 EDM optimized model. We noticed that using the additional stochasticity indeed helps to improve the image quality as in Fig. 1 (c). Moreover, setting  $S_{\mathrm{noise}}$  slightly above 1 might correct the errors in earlier steps more effectively as indicated in [16].
F.1 Pre-trained model specifications
For producing the CIFAR-10 time-continuous results in Table 1, we use the VP DDPM++ continuous architecture. These models are publicly available in conditional $^3$  and unconditional $^4$  versions and were directly derived from [34] under the Apache 2.0 license. On the unconditional mode (Figure 1 (a-b)), we first generate the FID curves of 3 types of DPM-Solver (with orders 1, 2 and 3) using the updates from their official implementation $^5$  in noise prediction mode. Taking profit of the tuning advancements proposed by [16], we used a linear noise schedule with  $\beta_{\mathrm{d}} = 19.1$  and  $\beta_{\mathrm{min}} = 0.1$  that slightly differs from the original parameters from [34] but were proven beneficial. We set the end time of sampling  $\varepsilon$  to 1e-4 as recommended by [22, Appendix D.2]. The values of all benchmark models for Figure 1 (b) were taken directly from tables provided by [22].
In our FFHQ-64 experiments, we employ the unconditional VP pretrained $^{6}$  model provided by [16].
For the CelebA-64 experiments, we use the pre-trained VP unconditional model whose checkpoint<sup>7</sup> is provided by [33]. We use the Type-1 discretization proposed in [22] to ensure compatibility of our method with the prescribed trained steps of such model.
For ImageNet-64, we both use the baseline and the optimized pre-trained models given in [16]. We note that the baseline is trained on the iDDPM class of model [26], which actually uses different preconditioning and thus the change of variables compared to the optimized model. The Figure 1 (c) was obtained using the EDM-preconditioned checkpoint<sup>8</sup>. The added noise settings of SEEDS-3 solver were not subject to a grid-search optimization procedure. The chosen hyper-parameters were  $S_{\mathrm{churn}} = 11$ ,  $S_{\mathrm{noise}} = 1.003$ ,  $S_{\mathrm{min}} = 0.05$ , and  $S_{\mathrm{max}} = 15$  but we are confident that this configuration can be optimized to further improve our results.
$^{2}$ https://nvlabs-fi-cdn.nvidia.com/edm/fid-refs/  
 $^{3}$ https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/baseline/  
baseline-cifar10-32x32-cond-vp.pkl  
 $^{4}$ https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/baseline/  
baseline-cifar10-32x32-uncond-vp.pkl  
 $^{5}$ https://github.com/LuChengTHU/dpm-solver  
 $^{6}$ https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/baseline/  
baseline-ffhq-64x64-uncond-vp.pkl  
 $^{7}$ https://drive.google.com/file/d/1R_H-fJYXSH79wfSKs9D-fuKQVan5L-GR/view?usp=sharing  
 $^{8}$ https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/edm-imagenet-64x64-cond-adm.pkl
F.2 Noise vs. Data Prediction approaches
In Appendix B of [23], the authors compare DPM-Solver2 and DPM-Solver++(2S), which amounts on comparing in our framework the difference between the obtained exponential integrators for the PFO in the noise and data prediction regimes to detect exactly a coefficient on the non-linear term that is absent in the noise prediction regime. The term they find corresponds exactly to the difference between applying the variation of constants formula before (instead of after) replacing the score function with the desired neural network. In Tab. 3 we report both data and noise prediction SEEDS-3. At low NFEs the DP approach gives better results but stabilizes in high NFEs at a FID score that is worse than the one the NP approach reaches.
Table 3: Comparison between noise prediction  $F_{\theta,t}$  and data prediction  $D_{\theta,t}$  modes of SEEDS-3 on CIFAR-10 (VP uncond. cont.).
METHOD \ NFE	9	30	60	90	150	165	180
SEEDS-3 DATA PREDICTION	60.75	22.42	12.47	2.95	2.51	2.54	2.55
SEEDS-3 NOISE PREDICTION	471.29	288.20	33.92	3.76	2.40	2.39	2.47
F.3 Low vs. High stage Solvers
Similar to DPM-Solver [22], the FID scores in Tab. 4 and 5 show that at low NFEs, higher stage methods performs more poorly while at higher NFEs, DPM-Solver-3 and SEEDS-3 are better than their counterparts with 1 and 2 stages.
Table 4: FID comparison between SEEDS (Ours) and DPM-Solver for low NFEs on CIFAR-10 VP uncond. discrete. We recomputed the DPM-Solver score using the "non-deep" model while [22] reports results for the "deep" architecture. The symbol  $\star$  is used when using 1 NFE more and  $\dagger$  when using 1 NFE less because the given NFE cannot be divided by 2 or 3. This corresponds to Figure 1 (a).
METHOD \ NFE	10	12	15	20	30	40	50	100
DPM-SOLVER-1	22.90	17.73	13.36	9.78	6.87	5.77	5.17	4.22
DPM-SOLVER-2	12.22	6.52	*4.55	-	3.75	3.68	3.64	3.60
DPM-SOLVER-3	†66.92	9.72	5.32	*3.83	3.66	-	*3.61	†3.58
SEEDS-1	303.48	239.79	279.84	192.68	84.78	45.26	28.18	8.24
SEEDS-2	481.09	473.48	*430.98	305.88	223.01	51.41	11.10	3.19
SEEDS-3	†483.04	482.19	479.63	*462.61	280.48	†247.44	*62.62	†3.53
Table 5: FID comparison between SEEDS (Ours) and DPM-Solver for high NFEs on CIFAR-10 VP uncond. discrete. We recomputed the DPM-Solver score using the "non-deep" model while [22] reports results for the "deep" architecture. The symbol  $\star$  is used when using 1 NFE more because the given NFE cannot be divided by 2 or 3. This corresponds to Figure 1 (a).
METHOD \ NFE	150	200	300	510
DPM-SOLVER-3	3.59	*3.58	-	3.58
SEEDS-1	-	4.07	3.40	-
SEEDS-3	3.12	*3.08	3.14	3.24
Table 6: FID comparison between SEEDS-3 (Ours) and other solvers on CIFAR-10 VP unconditional discrete. The symbol  $\star$  is used when using 1 NFE more and  $\dagger$  when using 1 NFE less because the given NFE cannot be divided by 2 or 3. This corresponds to Figure 1 (b).
METHOD \ NFE	10	12	15	20	50	200	1000
EULER-MARUYAMA	278.67	246.29	197.63	137.34	32.63	4.03	3.16
ANALYTIC DDPM	35.03	27.69	20.82	15.35	7.34	4.11	3.84
ANALYTIC DDIM	14.74	11.68	9.16	7.20	4.28	3.60	3.86
DDIM	13.58	11.02	8.92	6.94	4.73	4.07	3.95
DPM-SOLVER-3	†6.92	9.72	5.32	*3.83	*3.61	*3.58	-
SEEDS-3	†483.04	482.19	479.63	*462.61	*62.62	*3.08	-
F.4 Deterministic vs. Stochastic Solvers
Deterministic solvers as DPM-Solver [22] are fast and well-adapted to applications in which speed is the most concern. As shown in [22], Table 4 and 5, DPM-Solver converges to a local minimum at early steps and cannot be improved in large NFEs. Moreover, preconditioned deterministic solver in EDM gives optimal quality on unconditional CIFAR-10 [16, Fig. 5 (b)]. However, for more complicated datasets as ImageNet64, the stochasticity indeed helps improve the samples quality [16, Fig. 5 (c)]. We can consider the random noise as a corrector that approaches a better local or even global minimum. At  $S_{\mathrm{churn}} = 0$ , our SEEDS-3 with stochasticity gives lower FID score than EDM deterministic Heun (see Fig. 1 (c)). SEEDS-3 also reaches the best quality prior to the number steps needed in Euler Maruyama and other solvers as in Table 6. Table 7 completes the analysis in Table 1 for CIFAR-10 showing FID scores for varying NFEs and multiple versions of the EDM solver, depending on the chosen optimization hyper-parameters. Tables 8 to 10 complete the findings in Table 1 for the remaining used pretrained models. In particular, we report the exact FID values of SEEDS in the low NFE regime.
Table 7: FID comparison of different solvers on CIFAR-10-uncond-vp-continuous. The symbol  $\star$  is used when using 1 NFE more and  $\dagger$  when using 1 NFE less because the given NFE cannot be divided by 2 or 3. This corresponds to Figure 1 (b). All values for EDM are retrieved from the Latex source file SdePlotNfe.tex in the arXiv version of [16].
METHOD\NFE	30	48	63	126	165	180	511
GOTTAGOFAST	-	82.42	-	-	2.75	2.44	-
EDM (\( S_{CHURN} \)= 0)	*3.10	†2.99	2.94	*2.98	-	-	2.93
EDM (\( S_{TMIN,TMAX} \)+\( S_{NOISE} \)= 1)	*3.44	†2.89	2.77	2.72	-	-	2.69
EDM (\( S_{NOISE} \)= 1)	*3.99	†3.13	2.90	2.60	-	-	2.55
EDM (\( S_{TMIN,TMAX} \)=[0,∞])	*3.43	†2.87	2.71	2.52	-	-	2.54
EDM (OPTIMIZED)	*3.77	†3.08	2.84	2.47	-	-	2.27
DPM-SOLVER (NON-DEEP)	2.95	2.90	2.89	-	-	-	-
SEEDS (\( S_{CHURN} \)= 0)	*288.20	†90.25	33.91	2.45	2.39	2.47	-
F.5 Run-time Comparison
As a sanity check, we provide a run-time comparison table for our experiments on different datasets (see Tab. 11). One can see that the run-time is linear with respect to the NFE also for SEEDS, as the main advantage of the SETD method is to analytically compute the stochastic components in our solver, making their computation cost negligible. Some mild improvements about repetitive terms allowed our implementation of SEEDS to be slightly more effective than EDM and even DPM-Solver at same NFE.
Table 8: FID comparison of different solvers on CelebA 64x64 discrete. The symbol  $\star$  is used when using 1 NFE more and  $\dagger$  when using 1 NFE less because the given NFE cannot be divided by 2 or 3.
METHOD\ NFE	9	12	15	21	51	60	90	102	200
E-M	*310.22	227.16	207.97	†120.44	†29.25	-	-	-	3.90
AN.-DDPM	*28.99	25.27	21.80	†18.14	†11.23	-	-	-	6.51
AN.-DDIM	*15.62	13.90	12.29	†10.45	†6.13	-	-	-	3.46
DDIM	*10.85	9.99	7.78	†6.64	†5.23	-	-	-	4.78
DPM-SOLVER	6.92	4.20	3.05	2.82	2.72	-	-	-	-
SEEDS	460.87	374.48	301.66	261.87	3.84	6.58	1.88	1.97	-
Table 9: SEEDS-3 on CIFAR-10-cond-vp-continuous (using the baseline model in [16]).
METHOD \ NFE	15	21	30	60	90	120	129	150	165	180
SEEDS	239.2	167.5	131.1	25.06	3.19	2.17	2.08	2.15	2.16	2.19
F.6 Hardware configuration
During the experiments, we used three Linux-based servers with 60GB memory each and 4 GPUs NVIDIA V100 32GB, 4 GPUs NVIDIA V100 16GB, and 2 GPUs NVIDIA V100 32GB, respectively. Table 12 shows the detail of the configuration utilized for each experiment. We noted that when using the 4 GPUs configuration, the FID results were slightly lower (around  $2\%$ ), even after using a stacked fixed random seed. We run the experiments multiple times and reported the minimum FID value each time.
F.7 Licences
Pre-trained models:
CIFAR-10 models by [34]: Apache V2.0 license
- FFHQ-64 model by [16]: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
CelebA-64 model by [33]: Apache V2.0 license
- ImageNet-64 model by [16]: Apache V2.0 license
- Inception-v3 model by [35]: Apache V2.0 license
F.8 Supporting samples
In this subsection, we report the image grids supporting our claims in section 5.
Table 10: Tailored SEEDS-3 for the EDM-preconditioned pretrained model [16] on ImageNet 64x64.
METHOD \ NFE	12	15	21	51	102	201	270
SEEDS	209.12	197.79	153.72	63.75	16.35	1.56	1.38
Table 11: Run-time comparison (second/batch + std) on a single NVIDIA V100 of EDM, DPM-Solver and SEEDS. Discretely-trained models are run with the implementation based on the code from [22]. Continuously-trained models are run using the code from [16].
METHOD\NFE	9	21	51	90
CIFAR-10 32×32 CONTINUOUS (BATCH SIZE = 128)
EDM	2.096(±0.003)	4.891(±0.004)	11.592(±0.019)	21.213(±0.009)
DPM-SOLVER	2.099(±0.002)	4.871(±0.004)	11.841(±0.014)	20.888(±0.020)
SEEDS	2.086(±0.002)	4.867(±0.003)	11.817(±0.006)	20.896(±0.028)
FFHQ 64×64 CONTINUOUS (BATCH SIZE = 128)
EDM	4.361(±0.005)	10.179(±0.005)	24.738(±0.018)	44.145(±0.016)
DPM-SOLVER	4.344(±0.005)	10.148(±0.009)	24.637(±0.007)	43.526(±0.029)
SEEDS	4.353(±0.003)	10.157(±0.004)	24.661(±0.011)	43.537(±0.013)
IMAGENET 64×64 CONTINUOUS (BATCH SIZE = 128)
EDM	7.525(±0.007)	17.579(±0.004)	42.696(±0.010)	76.175(±0.024)
DPM-SOLVER	7.535(±0.048)	17.556(±0.009)	42.629(±0.018)	75.222(±0.026)
SEEDS	7.429(±0.006)	17.572(±0.013)	42.654(±0.014)	75.245(±0.034)
CIFAR-10 32×32 DISCRETE (BATCH SIZE = 128)
DPM-SOLVER	0.272(±0.004)	0.529(±0.007)	1.324(±0.007)	2.632(±0.004)
SEEDS	0.261(±0.002)	0.523(±0.002)	1.299(±0.003)	2.598(±0.003)
CELEBA 64×64 DISCRETE (BATCH SIZE = 128)
DPM-SOLVER	0.936(±0.004)	1.812(±0.003)	4.558(±0.008)	9.108(±0.008)
SEEDS	0.912(±0.002)	1.808(±0.004)	4.526(±0.002)	9.033(±0.004)
LSUN-BEDROOM 256×256 DISCRETE (BATCH SIZE = 32)
DPM-SOLVER	5.566(±0.019)	11.124(±0.018)	27.815(±0.031)	55.648(±0.021)
SEEDS	5.498(±0.008)	11.001(±0.022)	27.503(±0.029)	54.842(±0.020)
LDM-CELEBAHQ 256×256 (BATCH SIZE = 64)
DPM-SOLVER	8.648(±0.013)	17.492(±0.013)	39.569(±0.015)	68.205(±0.017)
SEEDS	8.652(±0.005)	17.469(±0.010)	39.524(±0.010)	68.240(±0.026)
STABLE DIFFUSION 512×512 (BATCH SIZE = 16)
DPM-SOLVER	19.598(±0.070)	40.679(±0.107)	92.914(±0.079)	163.902(±0.116)
SEEDS	19.656(±0.106)	40.781(±0.136)	93.409(±0.123)	161.451(±0.518)
Table 12: Details of GPUs utilized during the experiments.
EXPERIMENT	MODEL	NUMBER	GPU SIZE
CIFAR-10 CONTINUOUS	NVIDIA V100	4	16 GB
CIFAR-10 DISCRETE	NVIDIA V100	2	32 GB
FFHQ64	NVIDIA V100	4	16 GB
CELEBA64	NVIDIA A100	2	16 GB
IMAGENET64	NVIDIA V100	4	32 GB
Figure 4: Samples on CIFAR-10 from low to high NFEs by SEEDS-3 in Noise Prediction (NP) and Data Prediction (DP) modes, using conditional VP continuous baseline model [16].
Figure 5: Samples on CIFAR-10 from low to high NFEs by SEEDS-3 in Noise Prediction (NP) and Data Prediction (DP) modes, using unconditional VP continuous baseline model [16].
3- 3
Pm-3
Figure 6: Samples on CelebA-64 from low to high NFEs by SEEDS-3 and DPM-Solver-3, using pre-trained model from [33].
Figure 7: Samples on FFHQ-64 from low to high NFEs by SEEDS-3 in Noise Prediction (NP) and Data Prediction (DP) modes, using unconditional VP continuous baseline model [16].
Figure 8: Samples on ImageNet-64 from low to high NFEs in Noise Prediction (NP) and Data Prediction (DP) modes, using conditional EDM optimized model [16].
Figure 9: Samples on ImageNet-64 from low to high NFEs in Noise Prediction (NP) and Data Prediction (DP) modes, using conditional EDM baseline model [16].
Figure 10: Example of samples on CelebA-64 generated by SEEDS-3 in 90 NFEs, using pre-trained model from [33].
(a) DPM-Solver-3 at 44 NFE.
Figure 11: Visual sample quality comparison between DPM-Solver-3 and SEEDS-3 using their optimal settings and unconditional CIFAR-10 discrete model [34].
(b) SEEDS-3 at 201 NFE.
(a) DPM-Solver-3 at 80 NFE.
(b) EDM at 80 NFE.
(c) SEEDS-3 at 150 NFE.
Figure 12: Visual sample quality comparison between DPM-Solver-3, EDM and SEEDS-3 using their optimal settings and unconditional VP FFHQ-64 continuous model [16].
(a) DPM-Solver-3 at 30 NFE.
Figure 13: Visual sample quality comparison between DPM-Solver-3, EDM and SEEDS-3 using their optimal settings and conditional VP CIFAR-10 baseline continuous model [16].