SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We generate the underlying discrete dataset  $S$  with  $|S| = 2000$  as follows. Firstly, we sample 2000 points, denoted as  $S_{1}$ , from a mixture of two Gaussians in  $\mathbb{R}^4$ . Next, we project these points onto  $\mathbb{R}^{20}$ . To ensure a variance of 1 on each dimension, we scale the coordinates accordingly. This setup aims to simulate data points that primarily reside on a lower-dimensional manifold with multiple modes.
The specific details are as follows:  $S_{1} \sim 0.3N(a, s^{2}I) + 0.7(-a, s^{2}I)$ , where  $a = (3, 3, 3, 3) \subset \mathbb{R}^4$  and  $s = 1$ . Then, we randomly select a projection matrix  $P \in \mathbb{R}^{20 \times 4}$ , where each entry is drawn from  $N(0, 1)$ , and compute  $S_{2} = PS_{1}$ . Finally, we scale each coordinate by a constant factor to ensure a variance of 1.
4https://github.com/NVlabs/edm
5https://github.com/Newbeeer/pfgmpp
(b)
Figure 8: Comparison of additional sampling error versus (a) contracted error (plotting the Pareto frontier) and (b) total error (using a scatter plot). (c) Pareto frontier of NFE versus total error.
C.5.2 Model Architecture
We employ a common MLP architecture with a latent size of 64 to learn the score function. The training method is adapted from [13], which includes the preconditioning technique and denoising score-matching objective [25].
C.5.3 Varying Hyperparameters
To achieve the best trade-off between contracted error and additional sampling error, and optimize the NFE versus FID (Frechet Inception Distance) performance, we explore various hyperparameters. [13] shows that the Vanilla SDE can be endowed with additional flexibility by varying the coefficient  $\beta(t)$  (Eq.(6) in [13]). Hence, regarding SDE, we consider NFE values from  $\{20, 40, 80, 160, 320\}$ , and multiply the original  $\beta(t) = \dot{\sigma}(t) / \sigma(t)$  [13] with values from  $\{0, 0.25, 0.5, 1, 1.5, 2, 4, 8\}$ . It is important to note that larger NFE values do not lead to further performance improvements. For restarts, we tried the following two settings: first we set the number of steps in Restart backward process to 40 and vary the number of Restart iterations  $K$  in the range  $\{0, 5, 10, 15, 20, 25, 30, 35\}$ . We also conduct a grid search with the number of Restart iterations  $K$  ranging from 5 to 25 and the number of steps in Restart backward process varying from 2 to 7. For ODE, we experiment with the number of steps set to  $\{20, 40, 80, 160, 320, 640\}$ .
Additionally, we conduct an experiment for Improved SDE in EDM. We try different values of  $S_{\mathrm{churn}}$  in the range of  $\{0,1,2,4,8,16,32,48,64\}$ . We also perform a grid search where the number of steps ranged from 20 to 320 and  $S_{\mathrm{churn}}$  takes values of  $[0.2\times \mathrm{steps},0.5\times \mathrm{steps},20,60]$ . The plot combines the results from SDE and is displayed in Figure 8.
To mitigate the impact of randomness, we collect the data by averaging the results from five runs with the same hyperparameters. To compute the Wasserstein distance between two discrete distributions, we use minimum weight matching.
C.5.4 Plotting the Pareto frontier
We generate the Pareto frontier plots as follows. For the additional sampling error versus contracted error plot, we first sort all the data points based on their additional sampling error and then connect the data points that represent prefix minimums of the contracted error. Similarly, for the NFE versus FID plot, we sort the data points based on their NFE values and connect the points where the FID is a prefix minimum.
D Extra Experimental Results
D.1 Numerical Results
In this section, we provide the corresponding numerical results of Fig. 3(a) and Fig. 3(b), in Table 2, 3 (CIFAR-10 VP, EDM, PFGM++) and Table 4, 5 (ImageNet  $64 \times 64$  EDM), respectively. We also include the performance of Vanilla SDE in those tables. For the evaluation, we compute the Fréchet distance between 50000 generated samples and the pre-computed statistics of CIFAR-10 and ImageNet  $64 \times 64$ . We follow the evaluation protocol in EDM [13] that calculates each FID scores three times with different seeds and report the minimum.
We also provide the numerical results on the Stable Diffusion model [19], with a classifier guidance weight  $w = 2,3,5,8$  in Table 6, 7, 8, 9. As in [17], we report the zero-shot FID score on 5K random prompts sampled from the COCO validation set. We evaluate CLIP score [6] with the open-sourced ViT-g/14 [11], Aesthetic score by the more recent LAION-Aesthetics Predictor V2<sup>6</sup>. We average the CLIP and Aesthetic scores over 5K generated samples. The number of function evaluations is two times the sampling steps in Stable Diffusion model, since each sampling step involves the evaluation of the conditional and unconditional model.
Table 2: CIFAR-10 sample quality (FID score) and number of function evaluations (NFE) on VP [23] for baselines
NFE	FID
ODE (Heun) [13]	1023	2.90
511	2.90
255	2.90
127	2.90
63	2.89
35	2.97
Vanilla SDE [23]	1024	2.79
512	4.01
256	4.79
128	12.57
Gonna Go Fast [12]	1000	2.55
329	2.70
274	2.74
179	2.59
147	2.95
49	72.29
Improved SDE [13]	1023	2.35
511	2.37
255	2.40
127	2.58
63	2.88
35	3.45
Table 3: CIFAR-10 sample quality (FID score), number of function evaluations (NFE) and Restart configurations on VP [23], EDM [13] and PFGM++ [28]
Method	NFE	FID	Configuration (Nmain, NRestart,i, Ki, tmin,i, tmax,i)
VP
519	2.11	(20,9,30,0.06,0.20)
115	2.21	(18,3,20,0.06,0.30)
75	2.27	(18,3,10,0.06,0.30)
55	2.45	(18,3,5,0.06,0.30)
43	2.70	(18,3,2,0.06,0.30)
EDM
43	1.90	(18,3,2,0.14,0.30)
PFGM++
43	1.88	(18,3,2,0.14,0.30)
<sup>6</sup>https://github.com/christophschuhmann/improved-aesthetic-predictor
Table 4: ImageNet  ${64} \times  {64}$  sample quality (FID score) and number of function evaluations (NFE) on EDM [13] for baselines
NFE	FID (50k)
ODE (Heun) [13]	1023	2.24
511	2.24
255	2.24
127	2.25
63	2.30
35	2.46
Vanilla SDE [23]	1024	1.89
512	3.38
256	11.91
128	59.71
Improved SDE [13]	1023	1.40
511	1.45
255	1.50
127	1.75
63	2.24
35	2.97
Table 5: ImageNet  ${64} \times  {64}$  sample quality (FID score),number of function evaluations (NFE) and Restart configurations on EDM [13]
NFE	FID (50k)	Configuration
Nmain, {(NRestart,i, Ki, tmin,i, Tmax,i)}l i=1
623	1.36	36, {(10, 3, 19.35, 40.79), (10, 3, 1.09, 1.92), (7, 6, 0.59, 1.09), (7, 6, 0.30, 0.59), (7, 25, 0.06, 0.30)}
535	1.39	36, {(6, 1, 19.35, 40.79), (6, 1, 1.09, 1.92), (7, 6, 0.59, 1.09), (7, 6, 0.30, 0.59), (7, 25, 0.06, 0.30)}
385	1.41	36, {(3, 1, 19.35, 40.79), (6, 1, 1.09, 1.92), (6, 5, 0.59, 1.09), (6, 5, 0.30, 0.59), (6, 20, 0.06, 0.30)}
203	1.46	36, {(4, 1, 19.35, 40.79), (4, 1, 1.09, 1.92), (4, 5, 0.59, 1.09), (4, 5, 0.30, 0.59), (6, 6, 0.06, 0.30)}
165	1.51	18, {(3, 1, 19.35, 40.79), (4, 1, 1.09, 1.92), (4, 5, 0.59, 1.09), (4, 5, 0.30, 0.59), (4, 10, 0.06, 0.30)}
99	1.71	18, {(3, 1, 19.35, 40.79), (4, 1, 1.09, 1.92), (4, 4, 0.59, 1.09), (4, 1, 0.30, 0.59), (4, 4, 0.06, 0.30)}
67	1.95	18, {(5, 1, 19.35, 40.79), (5, 1, 1.09, 1.92), (5, 1, 0.59, 1.09), (5, 1, 0.06, 0.30)}
39	2.38	14, {(3, 1, 19.35, 40.79), (3, 1, 1.09, 1.92), (3, 1, 0.06, 0.30)}
Table 6: Numerical results on Stable Diffusion v1.5 with a classifier-free guidance weight  $w = 2$
Steps	FID (5k) ↓	CLIP score ↑	Aesthetic score ↑
DDIM [22]	50	16.08	0.2905	5.13
100	15.35	0.2920	5.15
Heun	51	18.80	0.2865	5.14
101	18.21	0.2871	5.15
DDPM [9]	100	13.53	0.3012	5.20
200	13.22	0.2999	5.19
Restart	66	13.16	0.2987	5.19
Table 7: Numerical results on Stable Diffusion v1.5 with a classifier-free guidance weight  $w = 3$
Steps	FID (5k) ↓	CLIP score ↑	Aesthetic score ↑
DDIM [22]	50	14.28	0.3056	5.22
100	14.30	0.3056	5.22
Heun	51	15.63	0.3022	5.20
101	15.40	0.3026	5.21
DDPM [9]	100	15.72	0.3129	5.28
200	15.13	0.3131	5.28
Restart	66	14.48	0.3079	5.25
Table 8: Numerical results on Stable Diffusion v1.5 with a classifier-free guidance weight  $w = 5$
Steps	FID (5k) ↓	CLIP score ↑	Aesthetic score ↑
DDIM [22]	50	16.60	0.3154	5.31
100	16.80	0.3157	5.31
Heun	51	16.26	0.3135	5.28
101	16.38	0.3136	5.29
DDPM [9]	100	19.62	0.3197	5.36
200	18.88	0.3200	5.35
Restart	66	16.21	0.3179	5.33
Table 9: Numerical results on Stable Diffusion v1.5 with a classifier-free guidance weight  $w = 8$
Steps	FID (5k) ↓	CLIP score ↑	Aesthetic score ↑
DDIM [22]	50	19.83	0.3206	5.37
100	19.82	0.3200	5.37
Heun	51	18.44	0.3186	5.35
101	18.72	0.3185	5.36
DDPM [9]	100	22.58	0.3223	5.39
200	21.67	0.3212	5.38
Restart	47	18.40	0.3228	5.41
D.2 Sensitivity Analysis of Hyper-parameters
We also investigate the impact of varying  $t_{\mathrm{min}}$  when  $t_{\mathrm{max}} = t_{\mathrm{min}} + 0.3$ , and the length the restart interval when  $t_{\mathrm{min}} = 0.06$ . Fig. 10(a) reveals that FID scores achieve a minimum at a  $t_{\mathrm{min}}$  close to 0
Table 10: Restart (Steps=66) configurations on Stable Diffusion v1.5
w	Configuration
Nmain, {(NRestart,i, Ki, tmin,i, Tmax,i)}l i=1
2	30, {(5, 2, 1, 9), (5, 2, 5, 10)}
3	30, {(10, 2, 0.1, 3)}
5	30, {(10, 2 0.1, 2)}
8	30, {(10, 2, 0.1, 2)}
(b) FID versus Aesthetic score
Figure 9: FID score versus (a) CLIP ViT-g/14 score and (b) Aesthetic score for text-to-image generation at  $512 \times 512$  resolution, using Stable Diffusion v1.5 with varying classifier-free guidance weight  $w = 2,3,5,8$ .
on VP, indicating higher accumulated errors at the end of sampling and poor neural estimations at small  $t$ . Note that the Restart interval 0.3 is about twice the length of the one in Table 1 and Restart does not outperform the ODE baseline on EDM. This suggests that, as a rule of thumb, we should apply greater Restart strength (e.g., larger  $K$ ,  $t_{\mathrm{max}} - t_{\mathrm{min}}$ ) for weaker or smaller architectures and vice versa.
In theory, a longer interval enhances contraction but may add more additional sampling errors. Again, the balance between these factors results in a V-shaped trend in our plots (Fig. 10(b)). In practice, selecting  $t_{\mathrm{max}}$  close to the dataset's radius usually ensures effective mixing when  $t_{\mathrm{min}}$  is small.
(a)
Figure 10: (a): Adjusting  $t_{\mathrm{min}}$  in Restart on VP/EDM; (b): Adjusting the Restart interval length when  $t_{\mathrm{min}} = 0.06$ .
E Extended Generated Images
In this section, we provide extended generated images by Restart, DDIM, Heun and DDPM on text-to-image Stable Diffusion v1.5 model [19]. We showcase the samples of four sets of text prompts in Fig. 11, Fig. 12, Fig. 13, Fig. 14, with a classifier-guidance weight  $w = 8$ .
(c) Heun (Steps=101)
Figure 11: Generated images with text prompt="A photo of an astronaut riding a horse on mars" and  $w = 8$ .
(d) DDPM (Steps=100)
F Heun's method is DPM-Solver-2 (with  $r_2 = 1$ )
The first order ODE in DPM-Solver [16] (DPM-Solver-1) is in the form of:
$$
\hat {x} _ {t _ {i - 1}} = \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} \hat {x} _ {t _ {i - 1}} - \left(\hat {\sigma} _ {t _ {i - 1}} \frac {\alpha_ {t _ {i}}}{\alpha_ {t _ {i - 1}}} - \hat {\sigma} _ {t _ {i}}\right) \hat {\sigma} _ {t _ {i}} \nabla_ {x} \log p _ {\hat {\sigma} _ {t _ {i}}} \left(\hat {x} _ {t _ {i}}\right) \tag {19}
$$
The first order ODE in EDM is in the form of
$$
x _ {t _ {i - 1}} = x _ {t _ {i}} - \left(\sigma_ {t _ {i - 1}} - \sigma_ {t _ {i}}\right) \sigma_ {t _ {i}} \nabla_ {x} \log p _ {\sigma_ {t _ {i}}} \left(x _ {t _ {i}}\right) \tag {20}
$$
(c) Heun (Steps=101)
Figure 12: Generated images with text prompt="A raccoon playing table tennis" and  $w = 8$ .
When  $x_{t} = \frac{\hat{x}_{t}}{\alpha_{t}}$ ,  $\hat{\sigma}_{t} = \sigma_{t}\alpha_{t}$ , we can rewrite the DPM-Solver-1 (Eq. (19)) as:
$$
\begin{array}{l} x _ {t _ {i - 1}} = x _ {t _ {i}} - \left(\sigma_ {t _ {i - 1}} - \sigma_ {t _ {i}}\right) \hat {\sigma} _ {t _ {i}} \nabla_ {x} \log p _ {\hat {\sigma} _ {t _ {i}}} \left(\hat {x} _ {t _ {i}}\right) \\ = x _ {t _ {i}} - \left(\sigma_ {t _ {i - 1}} - \sigma_ {t _ {i}}\right) \hat {\sigma} _ {t _ {i}} \nabla_ {x} \log p _ {\sigma_ {t _ {i}}} \left(x _ {t _ {i}}\right) \frac {1}{\alpha_ {t _ {i}}} \quad (\text {c h a n g e - o f - v a r i a b l e}) \\ \mathbf {\sigma} = x _ {t _ {i}} - (\sigma_ {t _ {i - 1}} - \sigma_ {t _ {i}}) \sigma_ {t _ {i}} \nabla_ {x} \log p _ {\sigma_ {t _ {i}}} (x _ {t _ {i}}) \\ \end{array}
$$
where the expression is exact the same as the ODE in EDM [13]. It indicates that the sampling trajectory in DPM-Solver-1 is equivalent to the one in EDM, up to a time-dependent scaling  $(\alpha_{t})$ . As  $\lim_{t\to 0}\alpha_t = 1$ , the two solvers will leads to the same final points when using the same time discretization. Note that the DPM-Solver-1 is also equivalent to DDIM (c.f. Section 4.1 in [16]), as also used in this paper.
With that, we can further verify that the Heun's method used in this paper corresponds to the DPM-Solver-2 when setting  $r_1 = 1$ .
(c) Heun (Steps=101)
Figure 13: Generated images with text prompt="Intricate origami of a fox in a snowy forest" and  $w = 8$ .
(c) Heun (Steps=101)
Figure 14: Generated images with text prompt="A transparent sculpture of a duck made out of glass" and  $w = 8$ .
The field of deep generative models incorporating differential equations is rapidly evolving and holds significant potential to shape our society. Nowadays, a multitude of photo-realistic images generated by text-to-image Stable Diffusion models populate the internet. Our work introduces Restart, a novel sampling algorithm that outperforms previous samplers for diffusion models and PFGM++. With applications extending across diverse areas, the Restart sampling algorithm is especially suitable for generation tasks demanding high quality and rapid speed. Yet, it is crucial to recognize that the utilization of such algorithms can yield both positive and negative repercussions, contingent on their specific applications. On the one hand, Restart sampling can facilitate the generation of highly realistic images and audio samples, potentially advancing sectors such as entertainment, advertising, and education. On the other hand, it could also be misused in deepfake technology, potentially leading to social scams and misinformation. In light of these potential risks, further research is required to develop robustness guarantees for generative models, ensuring their use aligns with ethical guidelines and societal interests.