SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $p_t(\boldsymbol{x})$  denotes the marginal distribution of  $\boldsymbol{x}_t$ , by Proposition 4.1, we know that for any bounded measurable function  $\tau(t): [0, T] \to \mathbb{R}$ , the following Reverse SDEs
$$
\mathrm {d} \boldsymbol {x} _ {t} = \left[ f (t) \boldsymbol {x} _ {t} - \left(\frac {1 + \tau^ {2} (t)}{2}\right) g ^ {2} (t) \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x} _ {t}) \right] \mathrm {d} t + \tau (t) g (t) \mathrm {d} \bar {\boldsymbol {w}} _ {t}, \quad \boldsymbol {x} _ {T} \sim p _ {T} (\boldsymbol {x} _ {T}), \tag {95}
$$
have the same marginal probability distributions. In practice, we substitute  $\nabla_{\pmb{x}}\log p_t(\pmb{x}_t)$  with  $s_\theta (\pmb {x}_t,t)$  and substitute  $p_T(\pmb {x}_T)$  wht  $\pi$  to sample the reverse SDE.
$$
\mathrm {d} \boldsymbol {x} _ {t} ^ {\boldsymbol {\theta}} = \left[ f (t) \boldsymbol {x} _ {t} ^ {\boldsymbol {\theta}} - \left(\frac {1 + \tau^ {2} (t)}{2}\right) g ^ {2} (t) \boldsymbol {s} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t} ^ {\boldsymbol {\theta}}, t) \right] \mathrm {d} t + \tau (t) g (t) \mathrm {d} \bar {\boldsymbol {w}} _ {t}, \quad \boldsymbol {x} _ {T} ^ {\boldsymbol {\theta}} \sim \pi , \tag {96}
$$
where  $\pi$  is a known distribution, specifically here a Gaussian. We have the following theorem under the Assumption in Appendix A in [3].
Theorem C.1. Let  $p = p_0$  be the data distribution, which is the distribution if we sample from the ground truth reverse SDE (54) at time 0. Let  $p_{\theta}^{\tau(t)}$  be the distribution if we sample from the practical reverse SDE (96) at time 0. Under the assumptions above, we have
$$
\begin{array}{l} D _ {K L} \left(p \| p _ {\boldsymbol {\theta}} ^ {\tau (t)}\right) \\ \leq D _ {K L} \left(p _ {T} \| \pi\right) + \frac {1}{8} \int_ {0} ^ {T} \mathbb {E} _ {p _ {t} (\boldsymbol {x})} \left[ \left(\tau (t) + \frac {1}{\tau (t)}\right) ^ {2} g ^ {2} (t) \| \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) - \boldsymbol {s} _ {\boldsymbol {\theta}} (\boldsymbol {x}, t) \| ^ {2} \right] d t. \tag {97} \\ \end{array}
$$
This evidence lower bound (ELBO) is minimized when  $\tau(t) \equiv 1$ .
Proof. Denote the path measure of Eq. (95) and Eq. (96) as  $\pmb{\mu}$  and  $\pmb{\nu}$  respectively. Both  $\pmb{\mu}$  and  $\pmb{\nu}$  are uniquely determined by the corresponding SDEs due to assumptions. Consider a Markov kernel  $K\left(\{z_{t}\}_{t\in [0,T]},y\right) = \delta (z_{0} = y)$ . Thus we have the following result
$$
\int K \left(\left\{\boldsymbol {x} _ {t} \right\} _ {t \in [ 0, T ]}, \boldsymbol {x}\right) \mathrm {d} \boldsymbol {\mu} \left(\left\{\boldsymbol {x} _ {t} \right\} _ {t \in [ 0, T ]}\right) = p _ {0} (\boldsymbol {x}), \tag {98}
$$
$$
\int K \left(\left\{\boldsymbol {x} _ {t} ^ {\boldsymbol {\theta}} \right\} _ {t \in [ 0, T ]}, \boldsymbol {x}\right) \mathrm {d} \boldsymbol {\nu} \left(\left\{\boldsymbol {x} _ {t} ^ {\boldsymbol {\theta}} \right\} _ {t \in [ 0, T ]}\right) = p _ {\boldsymbol {\theta}} ^ {\tau (t)} (\boldsymbol {x}). \tag {99}
$$
By data processing inequality for KL divergence
$$
\begin{array}{l} D _ {K L} \left(p \| p _ {\boldsymbol {\theta}} ^ {\tau (t)}\right) = D _ {K L} \left(p _ {0} \| p _ {\boldsymbol {\theta}} ^ {\tau (t)}\right) \\ = D _ {K L} \left(\int K \left(\left\{\boldsymbol {x} _ {t} \right\} _ {t \in [ 0, T ]}, \boldsymbol {x}\right) \mathrm {d} \mu \left(\left\{\boldsymbol {x} _ {t} \right\} _ {t \in [ 0, T ]}\right) \| \int K \left(\left\{\boldsymbol {x} _ {t} ^ {\boldsymbol {\theta}} \right\} _ {t \in [ 0, T ]}, \boldsymbol {x}\right) \mathrm {d} \nu \left(\left\{\boldsymbol {x} _ {t} ^ {\boldsymbol {\theta}} \right\} _ {t \in [ 0, T ]}\right)\right) \\ \leq D _ {K L} (\boldsymbol {\mu} \| \boldsymbol {\nu}). \tag {100} \\ \end{array}
$$
By the chain rule of KL divergence, we have
$$
\left. D _ {K L} \left(\boldsymbol {\mu} \| \boldsymbol {\nu}\right) = D _ {K L} \left(p _ {T} \| \pi\right) + \mathbb {E} _ {\boldsymbol {z} \sim p _ {T}} \left[ D _ {K L} \left(\boldsymbol {\mu} (\cdot | \boldsymbol {x} _ {T} = \boldsymbol {z}) \| \boldsymbol {\nu} (\cdot | \boldsymbol {x} _ {T} ^ {\theta} = \boldsymbol {z})\right) \right]. \right. \tag {101}
$$
By Girsanov Theorem,  $D_{KL}\left(\pmb {\mu}(\cdot |\pmb{x}_T = \pmb {z})\| \pmb {\nu}(\cdot |\pmb{x}_T^\theta = \pmb {z})\right)$  can be computed as
$$
\begin{array}{l} D _ {K L} \left(\boldsymbol {\mu} (\cdot | \boldsymbol {x} _ {T} = \boldsymbol {z}) \| \boldsymbol {\nu} (\cdot | \boldsymbol {x} _ {T} ^ {\theta} = \boldsymbol {z})\right) \\ = \mathbb {E} _ {\mu} \left[ \int_ {0} ^ {T} \frac {1}{2} \left(\tau (t) + \frac {1}{\tau (t)}\right) g (t) \left(\nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) - \boldsymbol {s} _ {\boldsymbol {\theta}} (\boldsymbol {x}, t)\right) d \bar {\boldsymbol {w}} _ {t} \right] \\ + \mathbb {E} _ {\mu} \left[ \frac {1}{2} \int_ {0} ^ {T} \frac {1}{4} \left(\tau (t) + \frac {1}{\tau (t)}\right) ^ {2} g ^ {2} (t) \| \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) - \boldsymbol {s} _ {\boldsymbol {\theta}} (\boldsymbol {x}, t) \| ^ {2} d t \right] \tag {102} \\ = \frac {1}{8} \int_ {0} ^ {T} \mathbb {E} _ {p _ {t} (\boldsymbol {x})} \left[ \left(\tau (t) + \frac {1}{\tau (t)}\right) ^ {2} g ^ {2} (t) \| \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) - \boldsymbol {s} _ {\boldsymbol {\theta}} (\boldsymbol {x}, t) \| ^ {2} \right] \mathrm {d} t \\ \end{array}
$$
For our 2-step SA-Predictor and 1-step SA-Corrector, we find that the coefficient will degenerate to a simple case.
For 2-step SA-Predictor, assume on  $[t_i,t_{i + 1}]$ ,  $\tau (t) = \tau$  is a constant,
$$
b _ {i} = e ^ {- \lambda_ {t _ {i + 1}} \tau^ {2}} \sigma_ {t _ {i + 1}} (1 + \tau^ {2}) \int_ {\lambda_ {t _ {i}}} ^ {\lambda_ {t _ {i + 1}}} e ^ {(1 + \tau^ {2}) \lambda} \frac {\lambda - \lambda_ {t _ {i - 1}}}{\lambda_ {t _ {i}} - \lambda_ {t _ {i - 1}}} d \lambda , \tag {103}
$$
$$
b _ {i - 1} = e ^ {- \lambda_ {t _ {i + 1}} \tau^ {2}} \sigma_ {t _ {i + 1}} (1 + \tau^ {2}) \int_ {\lambda_ {t _ {i}}} ^ {\lambda_ {t _ {i + 1}}} e ^ {(1 + \tau^ {2}) \lambda} \frac {\lambda - \lambda_ {t _ {i}}}{\lambda_ {t _ {i - 1}} - \lambda_ {t _ {i}}} d \lambda , \tag {104}
$$
$$
\begin{array}{l} \boldsymbol {x} _ {t _ {i + 1}} = \frac {\sigma_ {t _ {i + 1}}}{\sigma_ {t _ {i}}} e ^ {- \int_ {\lambda_ {t _ {i}}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\tilde {\lambda}) d \tilde {\lambda}} \boldsymbol {x} _ {t _ {i}} + b _ {i} \boldsymbol {x} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t _ {i}}, t _ {i}) + b _ {i - 1} \boldsymbol {x} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}) + \tilde {\sigma} _ {i} \boldsymbol {\xi} \\ = \frac {\sigma_ {t _ {i + 1}}}{\sigma_ {t _ {i}}} e ^ {- \int_ {\lambda_ {t _ {i}}} ^ {\lambda_ {t _ {i + 1}}} \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}} \boldsymbol {x} _ {t _ {i}} + (b _ {i} + b _ {i - 1}) \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) \tag {105} \\ - b _ {i - 1} \left(\boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i}}, t _ {i}\right) - \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t _ {i - 1}}, t _ {i - 1}\right)\right) + \tilde {\sigma} _ {i} \boldsymbol {\xi}. \\ \end{array}
$$
Let  $h = \lambda_{t_i + 1} - \lambda_{t_i}$ , we have
$$
\begin{array}{l} b _ {i} + b _ {i - 1} = e ^ {- \lambda_ {t _ {i + 1}} \tau^ {2}} \sigma_ {t _ {i + 1}} (1 + \tau^ {2}) \int_ {\lambda_ {t _ {i}}} ^ {\lambda_ {t _ {i + 1}}} e ^ {(1 + \tau^ {2}) \lambda} \mathrm {d} \lambda \\ = \alpha_ {t _ {i + 1}} \left(1 - e ^ {- h \left(1 + \tau^ {2}\right)}\right) \\ \end{array}
$$
$$
\begin{array}{l} b _ {i - 1} = \alpha_ {t _ {i + 1}} \frac {e ^ {- (1 + \tau^ {2}) h} + (1 + \tau^ {2}) h - 1}{(1 + \tau^ {2}) \left(\lambda_ {t _ {i}} - \lambda_ {t _ {i - 1}}\right)} \tag {106} \\ = \frac {\alpha_ {t _ {i + 1}}}{\lambda_ {t _ {i}} - \lambda_ {t _ {i - 1}}} \frac {1 - (1 + \tau^ {2}) h + \frac {1}{2} (1 + \tau^ {2}) ^ {2} h ^ {2} + \mathcal {O} (h ^ {3}) + (1 + \tau^ {2}) h - 1}{1 + \tau^ {2}} \\ = \frac {\alpha_ {t _ {i + 1}}}{\lambda_ {t _ {i}} - \lambda_ {t _ {i - 1}}} \frac {1}{2} (1 + \tau^ {2}) h ^ {2} + \mathcal {O} \left(h ^ {3}\right) \\ \end{array}
$$
Thus we implement  $\hat{b}_{i-1}$  as  $\frac{\alpha_{t_{i+1}}}{\lambda_{t_i} - \lambda_{t_{i-1}}} \frac{1}{2}(1 + \tau^2)h^2$  and  $\hat{b}_i$  as  $\alpha_{t_{i+1}}(1 - e^{-h(1 + \tau^2)}) - \hat{b}_{i-1}$ . Note that substituting  $b_i, b_{i-1}$  as  $\hat{b}_i, \hat{b}_{i-1}$  will maintain the convergence order result of 2-step SA-Predictor since the modified term is  $\mathcal{O}(h^3)$ . The implementation detail for 1-step SA-Corrector is technically the same.
E.1 Details on  $\tau(t)$ , Predictor Steps and Corrector Steps
CIFAR10 32x32 For the CIFAR10 experiment in Section 6.4, we use the pretrained baseline-unconditional-VE model from [27]. It's an unconditional model with VE noise schedule. To fairly compare with results in [27], we use a piecewise constant function  $\tau(t)$  inspired by [27]. Concretely, denoting  $\sigma_t^{EDM} = \frac{\sigma_t}{\alpha_t}$ , our  $\tau(t)$  is set to be a constant  $\tau$  in the interval  $[( \sigma^{EDM})^{-1}(0.05), (\sigma^{EDM})^{-1}(1)]$  and to be zero outside the interval. We find empirically that this piecewise constant function setting makes our SA-Solver converge better, especially in large noise scale cases. We use a 3-step SA-Predictor and a 3-step SA-Corrector. For the CIFAR10 experiment in Section 6.2 and 6.5, we also use the piecewise constant function  $\tau(t)$  as above. The predictor steps and corrector steps vary to verify the effectiveness of our proposed method in Section 6.2, while they are both set to be 3-steps in Section 6.5.
ImageNet 64x64 For the ImageNet 64x64 experiment in Section 6.4, we use the pretrained model from [4]. It's a conditional model with VP cosine noise schedule. To fairly compare with results in [27], we use a piecewise constant function  $\tau(t)$  inspired by [27]. Concretely, denoting  $\sigma_t^{EDM} = \frac{\sigma_t}{\alpha_t}$ , our  $\tau(t)$  is set to be a constant  $\tau$  in the interval  $[( \sigma^{EDM})^{-1}(0.05), (\sigma^{EDM})^{-1}(50)]$  and to be zero outside the interval. We find empirically that this piecewise constant function setting makes our SA-Solver converge better, especially in large noise scale cases. We use a 3-step SA-Predictor and a 3-step SA-Corrector.
Other experiments For other experiments, we use a constant function  $\tau(t) \equiv \tau$ . It's generally not the optimal choice for each individual task, thus further fine-grained tuning has the potential to improve the results. We aim to report the result of our SA-Solver without extra hyperparameter tuning. We use a 3-step SA-Predictor and a 3-step SA-Corrector under 20 NFEs and 2-step SA-Predictor and a 1-step SA-Corrector beyond 20 NFEs.
E.2 Details on Pretrained Models and Settings
CIFAR10 32x32 For the CIFAR10 experiment, we use the pretrained baseline-unconditional-VE model from [27]. It's an unconditional model with VE noise schedule. To fairly compare with results in [27], we follow the time step schedule in it. Specifically, we set  $\sigma_{min} = 0.02$  and  $\sigma_{max} = 80$  and select the step by  $\sigma_{i} = (\sigma_{\text{max}}^{\frac{1}{7}} + \frac{i}{N - 1} (\sigma_{\text{min}}^{\frac{1}{7}} - \sigma_{\text{max}}^{\frac{1}{7}}))^7$  for SA-Solver and UniPC. We directly report the results of the deterministic sampler and stochastic sampler of EDM. To make it a strong baseline, we report the results of the optimal setting for 4 hyper-parameters  $\{S_{churn}, S_{tmin}, S_{tmax}, S_{noise}\}$  and report its lowest observed FID. While for SA-Solver and UniPC, we report the averaged observed FID.
ImageNet 64x64 For the ImageNet 64x64 experiment, we use the pretrained model<sup>7</sup> from [4]. It's a conditional model with VP cosine noise schedule. To fairly compare with results in [27], we follow the time step schedule in it and use conditional sampling. Specifically, we set  $\sigma_{min} = 0.0064$  and  $\sigma_{max} = 80$  and select the step by  $\sigma_i = (\sigma_{max}^{\frac{1}{7}} + \frac{i}{N - 1} (\sigma_{min}^{\frac{1}{7}} - \sigma_{max}^{\frac{1}{7}}))^7$  for SA-Solver, UniPC, DPM-Solver and DDIM. We directly report the results of the deterministic sampler and stochastic sampler of EDM. To make it a strong baseline, we report the results of the optimal setting for 4 hyper-parameters  $\{S_{churn}, S_{tmin}, S_{tmax}, S_{noise}\}$  and report its lowest observed FID. While for SA-Solver, UniPC, DPM-Solver and DDIM, we report the averaged observed FID.
$^{4}$ https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/baseline/  
baseline-cifar10-32x32-uncond-ve.pkl  
 $^{5}$ https://openaipublic.blob.core.windows.net/diffusion/jul-2021/64x64_diffusion.pt  
 $^{6}$ https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/baseline/  
baseline-cifar10-32x32-uncond-ve.pkl  
 $^{7}$ https://openaipublic.blob.core.windows.net/diffusion/jul-2021/64x64_diffusion.pt
ImageNet 256x256 For the ImageNet 256x256 experiment, we use three different pretrained models: LDM $^8$  (VP, handcrafted noise schedule) from [5], DiT-XL/2 $^9$  (VP, linear noise schedule) from [41], Min-SNR $^{10}$  (VP, cosine noise schedule) from [42]. We use classifier-free guidance of scale  $s = 1.5$  and a uniform time step schedule because it's the most common setting for guided sampling for ImageNet 256x256.
ImageNet 512x512 For the ImageNet 256x256 experiment, we use the pre-trained model: DiT-XL/2 $^{11}$  from [41]. We use classifier-free guidance of scale  $s = 1.5$  and a uniform time step schedule following the settings of DiT [41].
LSUN Bedroom 256x256 For the LSUN Bedroom 256x256 experiment, we use the pretrained model<sup>12</sup> from [4]. We use unconditional sampling and a uniform lambda step schedule from [23].
We include the detailed FID results in Figure 1, Figure 2 and Figure 4 in the tables 4 to 14. The ablation study shows that stochasticity indeed helps improve sample quality. We find that for small NFEs, the magnitude of stochasticity should be small while for large NFEs, large magnitude of stochasticity helps improve sample quality. It can also be observed that in latent space, SDE converges faster as in Table 13. With only 10 NFEs,  $\tau = 0.6$  is better than  $\tau = 0$ . With 20 NFEs, our SA-Solver can achieve 3.87 FID, which outperforms all ODE samplers even with far more steps.
Table 4: Sample quality measured by FID  $\downarrow$  on CIFAR10 32x32 dataset (VE-baseline model from [27]) varying the number of function evaluations (NFE). For the results from  ${\mathrm{{EDM}}}^{ \dagger  }$  ,we reported its lowest observed FID.
Method \ NFE	11	15	23	31	47	63	95
DDIM(η = 0)	18.28	12.23	7.93	6.45	5.27	4.83	4.42
DPM-Solver	9.26	5.13	4.52	4.30	4.02	3.97	3.94
UniPC	6.42	5.02	4.19	4.00	3.91	3.90	3.89
EDM(ODE)†	13.46	5.62	4.04	3.82	3.79	3.80	3.79
EDM(SDE)†	23.94	8.94	4.73	3.95	3.59	3.36	3.06
SA-Solver	6.46	4.91	3.77	3.40	2.92	2.74	2.63
Table 5: Sample quality measured by FID  $\downarrow$  on CIFAR10 32x32 dataset (VE-baseline model from [27]) varying the number of function evaluations (NFE) and the magnitude of stochasticity  $\left( \tau \right)$  .
SA-Solver \ NFE	11	15	23	31	47	63	95
τ = 0.0	6.46	5.06	4.22	4.02	3.93	3.92	3.91
τ = 0.2	6.54	5.01	4.14	3.95	3.89	3.84	3.83
τ = 0.4	6.79	4.91	4.03	3.81	3.76	3.74	3.67
τ = 0.6	7.34	4.91	3.85	3.65	3.60	3.56	3.57
τ = 0.8	8.61	5.28	3.77	3.48	3.45	3.43	3.50
τ = 1.0	10.89	6.52	3.98	3.40	3.21	3.25	3.29
τ = 1.2	14.49	9.33	5.19	3.69	3.00	3.03	3.07
τ = 1.4	20.19	13.76	7.60	4.91	2.92	2.86	2.93
τ = 1.6	27.90	20.51	11.89	8.07	3.25	2.74	2.80
τ = 1.8	36.26	29.43	18.13	14.00	4.60	2.83	2.63
$^{8}$ https://ommmer-lab.com/files/latent-diffusion/nitro/cin/model.ckpt  
 $^{9}$ https://dl.fbaipublicfiles.com/DiT/models/DiT-XL-2-256x256.pt  
 $^{10}$ https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/releases/download/v0.0.0/ema_0.9999_x1.pt  
 $^{11}$ https://dl.fbaipublicfiles.com/DiT/models/DiT-XL-2-512x512.pt  
 $^{12}$ https://openaipublic.blob.core.windows.net/diffusion/jul-2021/lsun-bedroom.pt
Table 6: Sample quality measured by FID  $\downarrow$  on ImageNet 64x64 dataset (model from [4]) varying the number of function evaluations (NFE). For the results from  ${\mathrm{{EDM}}}^{ \dagger  }$  ,we reported its lowest observed FID.
Method \ NFE	15	23	31	47	63	95
DDIM(η = 0)	8.48	5.39	4.27	3.46	3.17	2.95
DPM-Solver	3.49	3.04	2.88	2.80	2.76	2.74
UniPC	3.51	2.84	2.75	2.72	2.71	2.72
EDM(ODE)†	4.78	3.12	2.84	2.73	2.73	2.67
EDM(SDE)†	8.94	4.30	3.40	2.72	2.44	2.22
SA-Solver	3.41	2.61	2.23	1.95	1.88	1.81
Table 7: Sample quality measured by FID  $\downarrow$  on ImageNet 64x64 dataset (model from [4]) varying the number of function evaluations (NFE) and the magnitude of stochasticity  $\left( \tau \right)$  .
SA-Solver \ NFE	15	23	31	47	63	95
τ = 0.0	3.48	2.72	2.72	2.66	2.64	2.71
τ = 0.2	3.41	2.80	2.63	2.63	2.64	2.60
τ = 0.4	3.52	2.70	2.51	2.51	2.49	2.49
τ = 0.6	3.98	2.61	2.44	2.39	2.34	2.35
τ = 0.8	5.80	2.68	2.32	2.24	2.19	2.21
τ = 1.0	10.06	3.38	2.23	2.09	2.08	2.08
τ = 1.2	18.39	5.52	2.52	1.95	1.97	2.00
τ = 1.4	32.42	10.37	3.83	2.05	1.89	1.89
τ = 1.6	52.31	19.64	7.10	2.60	1.88	1.81
G Additional Samples
We include additional samples in this section. In Figure 5 and Figure 6 we compare samples of our proposed SA-Solver with other diffusion samplers. In Figure 7 and Figure 8, we compare samples of our proposed SA-Solver under different NFEs and  $\tau$ . In Figure 9 and Figure 10, we compare samples of our proposed SA-Solver with other diffusion samplers on text-to-image tasks. Our SA-Solver can generate more diverse samples with more details.
Table 8: Sample quality measured by FID  $\downarrow$  on CIFAR10 32x32 dataset (model trained by ourselves; see Section 6.5) varying the sampling method and the training epoch.
method (NFE = 31) \ epoch	1250	1300	1350	1400	1450	1500
DDIM	39.32	29.79	19.59	12.98	8.63	6.64
DPM-Solver	30.57	22.11	13.85	8.85	5.68	4.55
EDM(ODE)	27.51	19.82	12.37	8.03	5.33	4.32
SA-Solver(τ = 0.6)	20.55	14.89	9.71	6.55	4.61	4.08
SA-Solver(τ = 1.0)	13.62	10.01	6.79	4.81	3.70	3.47
Table 9: Sample quality measured by FID ↓ on ImageNet 256x256 dataset (model trained by ourselves; see Section 6.5) varying the sampling method and the training epoch.
method (NFE = 40) \ epoch	50	100	150	200	250
DDIM	19.40	9.61	6.75	5.86	5.12
DPM-Solver	18.75	8.96	6.15	5.28	4.62
SA-Solver(τ = 0.4)	17.93	8.39	5.69	4.84	4.24
SA-Solver(τ = 0.8)	16.57	7.54	5.15	4.48	3.99
Table 10: Sample quality measured by FID  $\downarrow$  on ImageNet 256x256 dataset(model from [5]) varying the number of function evaluations (NFE).
Method \ NFE	5	10	20	40	60	80	100
DDIM(η = 0)	58.68	16.32	6.82	4.71	4.45	4.28	4.23
DPM-Solver	166.88	6.19	5.51	4.17	4.18	4.21	4.15
UniPC	12.79	4.96	4.21	4.14	4.12	4.09	4.10
DDIM(η = 1)	138.91	50.05	14.60	6.09	4.56	4.12	3.87
SA-Solver	11.46	4.82	3.88	3.47	3.37	3.37	3.33
Table 11: Ablation study on the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID  $\downarrow$  on CIFAR10 32x32 dataset(model from [27]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(τ).
τ\NFE	15	23	31	47	63	95	127
0	4.84	4.11	3.94	3.86	3.88	3.87	3.87
0.2	4.96	4.04	3.84	3.75	3.74	3.79	3.75
0.4	5.27	4.00	3.87	3.64	3.71	3.70	3.62
0.6	6.05	3.95	3.61	3.49	3.46	3.53	3.43
0.8	7.40	4.12	3.53	3.28	3.37	3.32	3.30
1.0	10.00	4.49	3.41	3.18	3.24	3.17	3.15
1.2	13.58	5.14	3.59	3.10	3.02	2.97	3.05
1.4	17.88	6.55	3.94	3.04	3.01	2.89	2.95
1.6	22.42	8.44	4.69	3.20	3.02	2.94	2.89
Table 12: Ablation study on the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID  $\downarrow$  on ImageNet 64x64 dataset(model from [4]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(τ).
τ\ NFE	20	40	60	80	100
0	3.30	2.83	2.78	2.79	2.82
0.2	3.32	2.77	2.72	2.74	2.79
0.4	3.37	2.68	2.63	2.62	2.59
0.6	3.61	2.57	2.49	2.49	2.47
0.8	4.19	2.51	2.40	2.34	2.30
1.0	5.55	2.54	2.32	2.21	2.20
1.2	7.93	2.77	2.29	2.14	2.14
1.4	11.55	3.20	2.40	2.14	2.08
1.6	16.15	3.97	2.60	2.20	2.09
Table 13: Ablation study on the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID  $\downarrow$  on ImageNet 256x256 dataset(model from [5]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(τ).
τ\ NFE	5	10	20	40	60	80	100
0	11.46	5.04	4.30	4.16	4.12	4.10	4.16
0.2	11.88	4.89	4.29	4.05	4.02	4.01	4.03
0.4	12.69	4.84	4.14	3.86	3.84	3.83	3.84
0.6	14.84	4.82	3.99	3.63	3.62	3.63	3.61
0.8	18.82	5.09	3.87	3.55	3.50	3.47	3.47
1.0	25.96	6.06	3.88	3.47	3.41	3.39	3.38
1.2	37.20	8.23	3.92	3.47	3.37	3.37	3.33
1.4	53.03	12.93	4.08	3.53	3.40	3.38	3.36
1.6	71.30	24.08	4.43	3.56	3.44	3.45	3.33
Table 14: Ablation study on the effect of the magnitude of stochasticity using SA-Solver. Sample quality measured by FID  $\downarrow$  on LSUN Bedroom 256x256 dataset(model from [4]) varying the number of function evaluations (NFE) and the magnitude of stochasticity(τ).
τ\ NFE	20	40	60	80	100
0	3.60	3.14	3.06	3.09	3.07
0.2	3.51	3.12	3.00	2.99	2.99
0.4	3.70	3.09	2.97	3.03	3.16
0.6	4.10	3.08	2.95	2.99	3.03
0.8	4.75	3.11	2.97	2.89	2.99
1.0	6.18	3.28	2.98	2.90	2.91
1.2	8.54	3.53	3.12	2.86	3.00
1.4	12.14	4.25	3.24	2.98	2.93
1.6	16.63	5.50	3.75	3.18	3.10
Figure 5: Samples by DDIM, DPM-Solver, UniPC, EDM(ODE), EDM(SDE) and our SA-Solver with 15, 23, 47, 95 NFEs with the same random seed from CIFAR10  $32 \times 32$  VE baseline model [27]
Figure 6: Samples by DDIM, DPM-Solver, UniPC, and our SA-Solver with 15, 23, 47, 95 NFEs with the same random seed from ImageNet 64x64 model [27](conditional sampling)
Figure 7: Samples by SA-Solver with 20, 40, 60, 100 NFEs varying stochasticity  $(\tau)$  with the same random seed from LSUN-Bedroom  $256\times 256$  model [4](unconditional sampling).
Figure 8: Samples by SA-Solver with 60 NFEs varying stochasticity  $(\tau)$  with the same random seed from ImageNet 512x512 DiT model [41] with classifier-free guidance scale  $s = 4.0$  (default setting to show image).
Figure 9: Samples using Stable-Diffusion v1.5 [5] with a classifier-free guidance scale 7.5 with different solvers and NFEs. Prompt:The Legend of Zelda landscape atmospheric, hyper realistic, 8k, epic composition, cinematic, octane render, artstation landscape vista photography by Carr Clifton Galen Rowell, 16K resolution, Landscape veduta photo by Dustin Lefevre tdraw, 8k resolution, detailed landscape painting by Ivan Shishkin, DeviantArt, Flickr, rendered in Enscape, Miyazaki, Nausicaa Ghibli, Breath of The Wild, 4k detailed post processing, artstation, rendering by octane, unreal engine.
Figure 10: Samples using Stable-Diffusion v1.5 [5] with a classifier-free guidance scale 7.5 with different solvers and NFEs. Prompt: glowwave portrait of curly orange-haired mad scientist man from borderlands 3, au naturel, hyper detailed, digital art, trending in artstation, cinematic lighting, studio quality, smooth render, unreal engine 5 rendered, octane rendered, art style by pixar dreamworks warner bros disney riot games and overwatch.