SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {d}{d t} \mathbb {E} \left[\left\| x _ {t} ^ {\leftarrow} - \bar {x} _ {t} ^ {\leftarrow} \right\|\right]\left. \right. \\ \leq 2 \left(\mathbb {E} \left[ \| t v \left(x _ {t} ^ {\leftarrow}, t\right) - t \bar {v} \left(x _ {t} ^ {\leftarrow}, t\right) \| \right] + \mathbb {E} \left[ \| t \bar {v} \left(x _ {t} ^ {\leftarrow}, t\right) - t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, t\right) \| \right]\right) \\ + 2 \left(\mathbb {E} \left[ \| t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, t\right) - t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, k \delta\right) \| \right] + \mathbb {E} \left[ \| t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, k \delta\right) - t \bar {v} \left(\bar {x} _ {k \delta} ^ {\leftarrow}, k \delta\right) \| \right]\right) \\ \leq 2 \left(\epsilon_ {a p p r o x} + L _ {2} \mathbb {E} \left[ \| x _ {t} ^ {\leftarrow} - \bar {x} _ {t} ^ {\leftarrow} \| \right] + \delta L _ {0} + L _ {2} \mathbb {E} \left[ \| \bar {x} _ {t} ^ {\leftarrow} - \bar {x} _ {k \delta} ^ {\leftarrow} \| \right]\right)\left. \right. \\ \leq 2 \left(\epsilon_ {a p p r o x} + L _ {2} \mathbb {E} \left[ \| x _ {t} ^ {\leftarrow} - \bar {x} _ {t} ^ {\leftarrow} \| \right] + \delta L _ {0} + L _ {2} \left(\delta L _ {1} + \sqrt {2 \delta d t _ {\max}}\right)\right). \\ \end{array}
$$
By Gronwall's Lemma,
$$
\begin{array}{l} \mathbb {E} \left[ \| x _ {t _ {\min }} - \bar {x} _ {t _ {\min }} \| \right] \\ = \mathbb {E} \left[ \left| \left| x _ {- t _ {\min }} ^ {-} - \bar {x} _ {- t _ {\min }} ^ {-} \right| \right| \right] \\ \leq e ^ {2 L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\mathbb {E} \left[ \left\| x _ {- t _ {\max }} ^ {\leftarrow} - \bar {x} _ {- t _ {\max }} ^ {\leftarrow} \right\| \right] + \left(\epsilon_ {\text {a p p r o x}} + \delta L _ {0} + L _ {2} \left(\delta L _ {1} + \sqrt {2 \delta d t _ {\max }}\right)\right) \left(t _ {\max } - t _ {\min }\right)\right) \\ = e ^ {2 L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\mathbb {E} \left[ \| x _ {t _ {\max }} - \bar {x} _ {t _ {\max }} \| \right] + \left(\epsilon_ {\text {a p p r o x}} + \delta L _ {0} + L _ {2} \left(\delta L _ {1} + \sqrt {2 \delta d t _ {\max }}\right)\right) \left(t _ {\max } - t _ {\min }\right)\right) \\ \end{array}
$$
A.4 Mixing Bounds
Lemma 5. Consider the same setup as Theorem 3. Assume that  $\delta \leq t_{\min}$ . Let
$$
x _ {t _ {\min}} = S D E _ {\theta} \left(x _ {t _ {\max}}, t _ {\max} \rightarrow t _ {\min}\right)
$$
$$
y _ {t _ {\text {m i n}}} = S D E _ {\theta} \left(y _ {t _ {\text {m a x}}}, t _ {\text {m a x}} \rightarrow t _ {\text {m i n}}\right).
$$
Then there exists a coupling between  $x_{s}$  and  $y_{s}$  such that
$$
T V \left(x _ {t _ {\min }}, y _ {t _ {\min }}\right) \leq \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\max } ^ {2} - t _ {\min } ^ {2}}}\right) \cdot e ^ {- B L _ {1} / t _ {\min } - L _ {1} ^ {2} t _ {\max } ^ {2} / t _ {\min } ^ {2}}\right) T V \left(x _ {t _ {\max }}, y _ {t _ {\max }}\right)
$$
Proof. We will construct a coupling between  $x_{t}$  and  $y_{t}$ . First, let  $(x_{t_{\mathrm{max}}},y_{t_{\mathrm{max}}})$  be sampled from the optimal TV coupling, i.e.,  $Pr(x_{t_{\mathrm{max}}} \neq y_{t_{\mathrm{max}}}) = TV(x_{t_{\mathrm{max}}},y_{t_{\mathrm{max}}})$ . Recall that by definition of SDE, for  $t \in ((k - 1)\delta ,k\delta ]$ ,
$$
d x _ {t} = - 2 t s _ {\theta} \left(x _ {k \delta}, k \delta\right) d t + \sqrt {2 t} d B _ {t}.
$$
Let us define a time-rescaled version of  $x_{t}$ :  $\overline{x}_t \coloneqq x_{t^2}$ . We verify that
$$
d \bar {x} _ {t} = - s _ {\theta} (\bar {x} _ {(k \delta) ^ {2}}, k \delta) d t + d B _ {t},
$$
where  $k$  is the unique integer satisfying  $t \in [((k - 1)\delta)^2, k^2\delta^2)$ . Next, we define the time-reversed process  $\overline{x}_t^{\leftarrow} \coloneqq \overline{x}_{-t}$ , and let  $v(x,t) \coloneqq s_{\theta}(x, - t)$ . We verify that there exists a Brownian motion  $B_t^x$  such that, for  $t \in [-t_{\max}^2, - t_{\min}^2]$ ,
$$
d \bar {x} _ {t} ^ {\leftarrow} = v _ {t} ^ {x} d t + d B _ {t} ^ {x},
$$
where  $v_{t}^{x} = s_{\theta}(\overline{x}_{-(k\delta)^{2}}^{-}, - k\delta)$ , where  $k$  is the unique positive integer satisfying  $-t \in (((k - 1)\delta)^{2}, (k\delta)^{2}]$ . Let  $d\overline{y}_t^{\leftarrow} = v_t^y dt + dB_t^y$ , be defined analogously. For any positive integer  $k$  and for any  $t \in [-(k\delta)^2, - ((k - 1)\delta)^2)$ , let us define
$$
z _ {t} = \overline {{x}} _ {- k ^ {2} \delta^ {2}} ^ {\leftarrow} - \overline {{y}} _ {- k ^ {2} \delta^ {2}} ^ {\leftarrow} + (2 k - 1) \delta^ {2} \left(v _ {- (k \delta) ^ {2}} ^ {x} - v _ {- (k \delta) ^ {2}} ^ {y}\right) + \left(B _ {t} ^ {x} - B _ {- (k \delta) ^ {2}} ^ {x}\right) - \left(B _ {t} ^ {y} - B _ {- (k \delta) ^ {2}} ^ {y}\right).
$$
Let  $\gamma_t \coloneqq \frac{z_t}{\|z_t\|}$ . We will now define a coupling between  $dB_t^x$  and  $dB_t^y$  as
$$
d B _ {t} ^ {y} = \left(I - 2 \mathbb {1} \left\{t \leq \tau \right\} \gamma_ {t} \gamma_ {t} ^ {T}\right) d B _ {t} ^ {x},
$$
where  $\mathbb{1}\{\}$  denotes the indicator function, i.e.  $\mathbb{1}\left\{t\leq \tau \right\} = 1$  if  $t\leq \tau$ , and  $\tau$  is a stopping time given by the first hitting time of  $z_{t} = 0$ . Let  $r_t\coloneqq \| z_t\|$ . Consider some  $t\in \left(-i^2\delta^2, - (i - 1)^2\delta^2\right)$ , and Let  $j\coloneqq \frac{t_{\max}}{\delta}$  (assume w.l.o.g that this is an integer), then
$$
\begin{array}{l} r _ {t} - r _ {- t _ {\max } ^ {2}} \leq \sum_ {k = i} ^ {j} (2 k - 1) \delta^ {2} \left\| \left(v _ {- (k \delta) ^ {2}} ^ {x} - v _ {- (k \delta) ^ {2}} ^ {y}\right) \right\| + \int_ {- t _ {\max } ^ {2}} ^ {t} \mathbb {1} \{t \leq \tau \} 2 d B _ {s} ^ {1} \\ \leq \sum_ {k = i} ^ {j} \left(k ^ {2} - (k - 1) ^ {2}\right) \delta^ {2} 2 L _ {1} / \left(t _ {\min }\right) + \int_ {- t _ {\max } ^ {2}} ^ {t} \mathbb {1} \{t \leq \tau \} 2 d B _ {t} ^ {1} \\ = \int_ {- t _ {\max } ^ {2}} ^ {- (i - 1) \delta^ {2}} \frac {2 L _ {1}}{t _ {\min }} d s + \int_ {- t _ {\max } ^ {2}} ^ {t} \mathbb {1} \left\{t \leq \tau \right\} 2 d B _ {s} ^ {1}, \\ \end{array}
$$
where  $dB_{s}^{1} = \langle \gamma_{t}, dB_{s}^{x} - dB_{s}^{y} \rangle$  is a 1-dimensional Brownian motion. We also verify that
$$
\begin{array}{l} r _ {- t _ {\max} ^ {2}} = \left\| z _ {- t _ {\max} ^ {2}} \right\| \\ = \left\| \bar {x} _ {- t _ {\max } ^ {2}} ^ {\leftarrow} - \bar {y} _ {- t _ {\max } ^ {2}} ^ {\leftarrow} + (2 j - 1) \delta^ {2} \left(v _ {- t _ {\max } ^ {2}} ^ {x} - v _ {- t _ {\max } ^ {2}} ^ {y}\right) + \left(B _ {t} ^ {x} - B _ {- t _ {\max } ^ {2}} ^ {x}\right) - \left(B _ {t} ^ {y} - B _ {- t _ {\max } ^ {2}} ^ {y}\right) \right\| \\ \leq \left\| \bar {x} _ {- t _ {\max } ^ {2}} ^ {\leftarrow} + (2 j - 1) \delta^ {2} v _ {- t _ {\max } ^ {2}} ^ {x} + \left(B _ {- (j - 1) ^ {2} \delta^ {2}} ^ {x} - B _ {- t _ {\max } ^ {2}} ^ {x}\right) \right\| \\ + \left\| \bar {y} _ {- t _ {\max } ^ {2}} ^ {\leftarrow} + (2 j - 1) \delta^ {2} v _ {- t _ {\max } ^ {2}} ^ {y} + \left(B _ {- (j - 1) ^ {2} \delta^ {2}} ^ {x} - B _ {t} ^ {x} + B _ {t} ^ {y} - B _ {- t _ {\max } ^ {2}} ^ {y}\right) \right\| \leq B \\ \end{array}
$$
where the third relation is by adding and subtracting  $B_{-(j-1)^2\delta^2}^x - B_t^x$  and using triangle inequality. The fourth relation is by noticing that  $\overline{x}_{-t_{\max}^2}^{\leftarrow} + (2j-1)\delta^2 v_{-t_{\max}^2}^x + \left(B_{-(j-1)^2\delta^2}^x - B_{-t_{\max}^2}^x\right) = \overline{x}_{-(j-1)^2\delta^2}^{\leftarrow}$  and that  $\overline{y}_{-t_{\max}^2}^{\leftarrow} + (2j-1)\delta^2 v_{-t_{\max}^2}^y + \left(B_{-(j-1)^2\delta^2}^x - B_t^x + B_t^y - B_{-t_{\max}^2}^y\right) \stackrel{d}{=} \overline{y}_{-(j-1)^2\delta^2}^{\leftarrow}$ , and then using our assumption in the theorem statement that all processes are supported on a ball of radius  $B/2$ .
We now define a process  $s_t$  defined by  $ds_t = 2L_1 / t_{\min} dt + 2dB_t^1$ , initialized at  $s_{-t_{\max}^2} = B \geq r_{-t_{\max}^2}$ . We can verify that, up to time  $\tau$ ,  $r_t \leq s_t$  with probability 1. Let  $\tau'$  denote the first-hitting time of  $s_t$  to 0, then  $\tau \leq \tau'$  with probability 1. Thus
$$
P r (\tau \leq - t _ {\min } ^ {2}) \geq P r (\tau^ {\prime} \leq - t _ {\min } ^ {2}) \geq 2 Q \left(\frac {B}{2 \sqrt {t _ {\max } ^ {2} - t _ {\min } ^ {2}}}\right) \cdot e ^ {- B L _ {1} / t _ {\min } - L _ {1} ^ {2} t _ {\max } ^ {2} / t _ {\min } ^ {2}}
$$
where we apply Lemma 6. The proof follows by noticing that, if  $\tau \leq -t_{\mathrm{min}}^2$ , then  $x_{t_{\mathrm{min}}} = y_{t_{\mathrm{min}}}$ . This is because if  $\tau \in [-k^2\delta^2, -(k - 1)^2\delta^2]$ , then  $\overline{x}_{-(k - 1)^2\delta^2}^{\leftarrow} = \overline{y}_{-(k - 1)^2\delta^2}^{\leftarrow}$ , and thus  $\overline{x}_t^\leftarrow = \overline{y}_t^\leftarrow$  for all  $t \geq -(k - 1)^2\delta^2$ , in particular, at  $t = -t_{\mathrm{min}}^2$ .
Lemma 6. Consider the stochastic process
$$
d r _ {t} = d B _ {t} ^ {1} + c d t.
$$
Assume that  $r_0 \leq B/2$ . Let  $\tau$  denote the hitting time for  $r_t = 0$ . Then for any  $T \in \mathbb{R}^+$ ,
$$
P r (\tau \leq T) \geq 2 Q \left(\frac {B}{2 \sqrt {T}}\right) \cdot e ^ {- a c - \frac {c ^ {2} T}{2}},
$$
where  $Q$  is the tail probability of a standard Gaussian defined in Definition 1.
Proof. We will use the following facts in our proof:
1. For  $x \sim \mathcal{N}(0, \sigma^2)$ ,  $Pr(x > r) = \frac{1}{2}\left(1 - \operatorname{erf}\left(\frac{r}{\sqrt{2}\sigma}\right)\right) = \frac{1}{2}\operatorname{erfc}\left(\frac{r}{\sqrt{2}\sigma}\right)$ .
2.  $\int_0^T\frac{a\exp\left(-\frac{a^2}{2T}\right)}{\sqrt{2\pi t^3}} dt = erf c\left(\frac{a}{\sqrt{2T}}\right) = 2Pr\left(\mathcal{N}(0,T) > a\right) = 2Q\left(\frac{a}{\sqrt{T}}\right)$  by definition of  $Q$
Let  $dr_{t} = dB_{t}^{1} + cdt$ , with  $r_0 = a$ . The density of the hitting time  $\tau$  is given by
$$
p (\tau = t) = f (a, c, t) = \frac {a \exp \left(- \frac {(a + c t) ^ {2}}{2 t}\right)}{\sqrt {2 \pi t ^ {3}}}. \tag {18}
$$
(see e.g. [3]). From item 2 above,
$$
\int_ {0} ^ {T} f (a, 0, t) d t = 2 Q \left(\frac {a}{\sqrt {T}}\right).
$$
In the case of a general  $c \neq 0$ , we can bound  $\frac{(a + ct)^2}{2t} = \frac{a^2}{2t} + ac + \frac{c^2t}{2}$ . Consequently,
$$
f (a, c, t) \geq f (a, 0, t) \cdot e ^ {- a c - \frac {c ^ {2} t}{2}}.
$$
$$
P r (\tau \leq T) = \int_ {0} ^ {T} f (a, c, t) d t \geq \int_ {0} ^ {T} f (a, 0, t) d t e ^ {- c} = 2 Q \left(\frac {B}{2 \sqrt {T}}\right) \cdot e ^ {- a c - \frac {c ^ {2} T}{2}}.
$$
A.5 TV Overlap
Definition 1. Let  $x$  be sampled from standard normal distribution  $\mathcal{N}(0,1)$ . We define the Gaussian tail probability  $Q(a) \coloneqq Pr(x \geq a)$ .
Lemma 7. We verify that for any two random vectors  $\xi_x \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$  and  $\xi_y \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$ , each belonging to  $\mathbb{R}^d$ , the total variation distance between  $x' = x + \xi_x$  and  $y' = y + \xi_y$  is given by
$$
T V (x ^ {\prime}, y ^ {\prime}) = 1 - 2 Q (r) \leq 1 - \frac {2 r}{r ^ {2} + 1} \frac {1}{\sqrt {2 \pi}} e ^ {- r ^ {2} / 2},
$$
where  $r = \frac{\|x - y\|}{2\sigma}$ , and  $Q(r) = Pr(\xi \geq r)$ , when  $\xi \sim \mathcal{N}(0,1)$ .
Proof. Let  $\gamma := \frac{x - y}{\|x - y\|}$ . We decompose  $x', y'$  into the subspace/orthogonal space defined by  $\gamma$ :
$$
x ^ {\prime} = x ^ {\perp} + \xi_ {x} ^ {\perp} + x ^ {\parallel} + \xi_ {x} ^ {\parallel}
$$
$$
y ^ {\prime} = y ^ {\perp} + \xi_ {y} ^ {\perp} + y ^ {\parallel} + \xi_ {y} ^ {\parallel}
$$
$$
x ^ {\parallel} := \gamma \gamma^ {T} x \qquad x ^ {\perp} := x - x ^ {\parallel}
$$
$$
y ^ {\parallel} := \gamma \gamma^ {T} y \qquad y ^ {\perp} := y - y ^ {\parallel}
$$
$$
\xi_ {x} ^ {\parallel} := \gamma \gamma^ {T} \xi_ {x} \qquad \xi_ {x} ^ {\perp} := \xi_ {x} - \xi_ {x} ^ {\parallel}
$$
$$
\xi_ {y} ^ {\parallel} := \gamma \gamma^ {T} \xi_ {y} \qquad \xi_ {y} ^ {\perp} := \xi_ {y} - \xi_ {y} ^ {\parallel}
$$
We verify the independence  $\xi_x^\perp \perp \xi_x^\parallel$  and  $\xi_y^\perp \perp \xi_y^\parallel$  as they are orthogonal decompositions of the standard Gaussian. We will define a coupling between  $x'$  and  $y'$  by setting  $\xi_x^\perp = \xi_y^\perp$ . Under this coupling, we verify that
$$
\left(x ^ {\perp} + \xi_ {x} ^ {\perp}\right) - \left(y ^ {\perp} + \xi_ {y} ^ {\perp}\right) = x - y - \gamma \gamma^ {T} (x - y) = 0
$$
Therefore,  $x' = y'$  if and only if  $x^{\parallel} + \xi_x^{\parallel} = y^{\parallel} + \xi_y^{\parallel}$ . Next, we draw  $(a,b)$  from the optimal coupling between  $\mathcal{N}(0,1)$  and  $\mathcal{N}\left(\frac{\|x - y\|}{\sigma},1\right)$ . We verify that  $x^{\parallel} + \xi_x^{\parallel}$  and  $y^{\parallel} + \xi_y^{\parallel}$  both lie in the span of  $\gamma$ . Thus it suffices to compare  $\left\langle \gamma ,x^{\parallel} + \xi_x^{\parallel}\right\rangle$  and  $\left\langle \gamma ,y^{\parallel} + \xi_y^{\parallel}\right\rangle$ . We verify that  $\left\langle \gamma ,x^{\parallel} + \xi_x^{\parallel}\right\rangle =$
$\langle \gamma ,y^{\parallel}\rangle +\langle \gamma ,x^{\parallel} - y^{\parallel}\rangle +\langle \gamma ,\xi_x^{\parallel}\rangle \sim \mathcal{N}(\langle \gamma ,y^{\parallel}\rangle +\| x - y\| ,\sigma^2)\overset {d}{=}\langle \gamma ,y^{\parallel}\rangle +\sigma b.$  We similarly verify that  $\left\langle \gamma ,y^{\parallel} + \xi_y^{\parallel}\right\rangle = \left\langle \gamma ,y^{\parallel}\right\rangle +\left\langle \gamma ,\xi_y^{\parallel}\right\rangle \sim \mathcal{N}(\langle \gamma ,y^{\parallel}\rangle ,\sigma^2)\overset {d}{=}\langle \gamma ,y^{\parallel}\rangle +\sigma a.$
Thus  $TV(x', y') = TV(\sigma a, \sigma b) = 1 - 2Q\left(\frac{\|x - y\|}{2\sigma}\right)$ . The last inequality follows from
$$
P r (\mathcal {N} (0, 1) \geq r) \geq \frac {r}{r ^ {2} + 1} \frac {1}{\sqrt {2 \pi}} e ^ {- r ^ {2} / 2}
$$
B More on Restart Algorithm
B.1 EDM Discretization Scheme
[13] proposes a discretization scheme for ODE given the starting  $t_{\mathrm{max}}$  and end time  $t_{\mathrm{min}}$ . Denote the number of steps as  $N$ , then the EDM discretization scheme is:
$$
t _ {i <   N} = \left(t _ {\max } ^ {\frac {1}{\rho}} + \frac {i}{N - 1} \left(t _ {\min } ^ {\frac {1}{\rho}} - t _ {\max } ^ {\frac {1}{\rho}}\right)\right) ^ {\rho}
$$
with  $t_0 = t_{\mathrm{max}}$  and  $t_{N - 1} = t_{\mathrm{min}}$ .  $\rho$  is a hyperparameter that determines the extent to which steps near  $t_{\mathrm{min}}$  are shortened. We adopt the value  $\rho = 7$  suggested by [13] in all of our experiments. We apply the EDM scheme to create a time discretization in each Restart interval  $[t_{\mathrm{max}}, t_{\mathrm{min}}]$  in the Restart backward process, as well as the main backward process between  $[0, T]$  (by additionally setting  $t_{\mathrm{min}} = 0.002$  and  $t_N = 0$  as in [13]). It is important to note that  $t_{\mathrm{min}}$  should be included within the list of time steps in the main backward process to seamlessly incorporate the Restart interval into the main backward process. We summarize the scheme as a function in Algorithm 1.
Algorithm 1 EDM_Scheme  $(t_{\mathrm{min}}, t_{\mathrm{max}}, N, \rho = 7)$
1: return  $\left\{(t_{\max}^{\frac{1}{\rho}} + \frac{i}{N - 1}(t_{\min}^{\frac{1}{\rho}} - t_{\max}^{\frac{1}{\rho}}))^{\rho}\right\}_{i = 0}^{N - 1}$
B.2 Restart Algorithm
We present the pseudocode for the Restart algorithm in Algorithm 2. In this pseudocode, we describe a more general case that applies  $l$ -level Restarting strategy. For each Restart segment, the include the number of steps in the Restart backward process  $N_{\mathrm{Restart}}$ , the Restart interval  $[t_{\mathrm{min}}, t_{\mathrm{max}}]$  and the number of Restart iteration  $K$ . We further denote the number of steps in the main backward process as  $N_{\mathrm{main}}$ . We use the EDM discretization scheme (Algorithm 1) to construct time steps for the main backward process  $(t_0 = T, t_{N_{\mathrm{main}}} = 0)$  as well as the Restart backward process, when given the starting/end time and the number of steps.
Although Heun's  $2^{\mathrm{nd}}$  order method [2] (Algorithm 3) is the default ODE solver in the pseudocode, it can be substituted with other ODE solvers, such as Euler's method or the DPM solver [16].
The provided pseudocode in Algorithm 2 is tailored specifically for diffusion models [13]. To adapt Restart for other generative models like PFGM++ [28], we only need to modify the Gaussian perturbation kernel in the Restart forward process (line 10 in Algorithm 2) to the one used in PFGM++.
In this section, we discuss the configurations for different samplers in details. All the experiments are conducted on eight NVIDIA A100 GPUs.
Algorithm 2 Restart sampling
1: Input: Score network  $s_\theta$ , time steps in main backward process  $t_{i \in \{0, N_{\text{main}}\}}$ , Restart parameters  $\{(N_{\text{Restart}, j}, K_j, t_{\text{min}, j}, t_{\text{max}, j})\}_{j=1}$   
2: Round  $t_{\text{min}, j \in \{1, l\}}$  to its nearest neighbor in  $t_{i \in \{0, N_{\text{main}}\}}$   
3: Sample  $x_0 \sim \mathcal{N}(\mathbf{0}, T^2 \mathbf{I})$   
4: for  $i = 0 \dots N_{\text{main}} - 1$  do  
5:  $x_{t_{i+1}} = \text{OneStep_Heun}(s_\theta, t_i, t_{i+1})$   
6: if  $\exists j \in \{1, \dots, l\}, t_{i+1} = t_{\text{min}, j}$  then  
7:  $t_{\text{min}} = t_{\text{min}, j}, t_{\text{max}} = t_{\text{max}, j}, K = K_j, N_{\text{Restart}} = N_{\text{Restart}, j}$   
8:  $x_{t_{\text{min}}}^0 = x_{t_{i+1}}$   
9: for  $k = 0 \dots K-1$  do  
10:  $\varepsilon_{t_{\text{min}} \to t_{\text{max}}} \sim \mathcal{N}(\mathbf{0}, (t_{\text{max}}^2 - t_{\text{min}}^2) \mathbf{I})$   
11:  $x_{t_{\text{max}}}^{k+1} = x_{t_{\text{min}}}^k + \varepsilon_{t_{\text{min}} \to t_{\text{max}}}$   
12:  $\{\bar{t}_m\}_{m=0}^{N_{\text{Restart}}-1} = \text{EDM_Scheme}(t_{\text{min}}, t_{\text{max}}, N_{\text{Restart}})$   
13: for  $m = 0 \dots N_{\text{Restart}} - 1$  do  
14:  $x_{t_m+1}^{k+1} = \text{OneStep_Heun}(s_\theta, \bar{t}_m, \bar{t}_{m+1})$   
15: end for  
16: end for  
17: end if  
18: end for  
19: return  $x_{t_N_{\text{main}}}$
Algorithm 3 OneStep_Heun  $(s_{\theta},x_{t_i},t_i,t_{i + 1})$
1:  $d_{i} = t_{i}s_{\theta}(x_{t_{i}},t_{i})$    
2:  $x_{t_{i + 1}} = x_{t_i} - (t_{i + 1} - t_i)d_i$    
3: if  $t_{i + 1}\neq 0$  then  
4:  $d_i^\prime = t_{i + 1}s_\theta (x_{t_{i + 1}},t_{i + 1})$    
5:  $x_{t_{i + 1}} = x_{t_i} - (t_{i + 1} - t_i)(\frac{1}{2} d_i + \frac{1}{2} d_i^{\prime})$    
6: end if  
7: return  $x_{t_{i + 1}}$
C.1 Configurations for Baselines
We select Vanilla SDE [23], Improved SDE [13], Gonna Go Fast [12] as SDE baselines and the Heun's  $2^{\text{nd}}$  order method [2] (Alg 3) as ODE baseline on standard benchmarks CIFAR-10 and ImageNet  $64 \times 64$ . We choose DDIM [22], Heun's  $2^{\text{nd}}$  order method, and DDPM [9] for comparison on Stable Diffusion model.
Vanilla SDE denotes the reverse-time SDE sampler in [23]. For Improved SDE, we use the recommended dataset-specific hyperparameters (e.g.,  $S_{\mathrm{max}}$ ,  $S_{\mathrm{min}}$ ,  $S_{\mathrm{churn}}$ ) in Table 5 of the EDM paper [13]. They obtained these hyperparameters by grid search. Gonna Go Fast [12] applied an adaptive step size technique based on Vanilla SDE and we directly report the FID scores listed in [12] for Gonna Go Fast on CIFAR-10 (VP). For fair comparison, we use the EDM discretization scheme [13] for Vanilla SDE, Improved SDE, Heun as well as Restart.
We borrow the hyperparameters such as discretization scheme or initial noise scale on Stable Diffusion models in the diffuser code repository. We directly use the DDIM and DDPM samplers implemented in the repo. We apply the same set of hyperparameters to Heun and Restart.
C.2 Configurations for Restart
We report the configurations for Restart for different models and NFE on standard benchmarks CIFAR-10 and ImageNet  $64\times 64$ . The hyperparameters of Restart include the number of steps in the main backward process  $N_{\mathrm{main}}$ , the number of steps in the Restart backward process  $N_{\mathrm{Restart}}$ , the Restart interval  $[t_{\mathrm{min}}, t_{\mathrm{max}}]$  and the number of Restart iteration  $K$ . In Table 3 (CIFAR-10, VP)
<sup>3</sup>https://github.com/huggingface/diffusers
we provide the quintuplet  $(N_{\mathrm{main}}, N_{\mathrm{Restart}}, t_{\mathrm{min}}, t_{\mathrm{max}}, K)$  for each experiment. Since we apply the multi-level Restart strategy for ImageNet  $64 \times 64$ , we provide  $N_{\mathrm{main}}$  as well as a list of quadruple  $\{(N_{\mathrm{Restart},i}, K_i, t_{\mathrm{min},i}, t_{\mathrm{max},i})\}_{i=1}^l$  ( $l$  is the number of Restart interval depending on experiments) in Table 5. In order to integrate the Restart time interval to the main backward process, we round  $t_{\mathrm{min},i}$  to its nearest neighbor in the time steps of main backward process, as shown in line 2 of Algorithm 2. We apply Heun method for both main/backward process. The formula for NFE calculation is  $\mathrm{NFE} = \underbrace{2 \cdot N_{\mathrm{main}} - 1}_{\text{main backward process}} + \sum_{i=1}^{l} \underbrace{K_i}_{\text{number of repetitions}} \cdot \underbrace{(2 \cdot (N_{\mathrm{Restart},i} - 1))}_{\text{per iteration in } i^{\text{th}} \text{Restart interval}}$  in this case. Inspired by
[13], we inflate the additive noise in the Restart forward process by multiplying  $S_{\mathrm{noise}} = 1.003$  on ImageNet  $64 \times 64$ , to counteract the over-denoising tendency of neural networks. We also observe that setting  $\gamma = 0.05$  in Algorithm 2 of EDM [13] would slightly boost the Restart performance on ImageNet  $64 \times 64$  when  $t \in [0.01, 1]$ .
We further include the configurations for Restart on Stable Diffusion models in Table 10, with a varying guidance weight  $w$ . Similar to ImageNet  $64 \times 64$ , we use multi-level Restart with a fixed number of steps  $N_{\mathrm{main}} = 30$  in the main backward process. We utilize the Euler method for the main backward process and the Heun method for the Restart backward process, as our empirical observations indicate that the Heun method doesn't yield significant improvements over the Euler method, yet necessitates double the steps. The number of steps equals to  $N_{\mathrm{main}} + \sum_{i=1}^{l} K_i \cdot (2 \cdot (N_{\mathrm{Restart},i} - 1))$  in this case. We set the total number of steps to 66, including main backward process and Restart backward process.
Given the prohibitively large search space for each Restart quadruple, a comprehensive enumeration of all possibilities is impractical due to computational limitations. Instead, we adjust the configuration manually, guided by the heuristic that weaker/smaller models or more challenging tasks necessitate a stronger Restart strength (e.g., larger  $K$ , wider Restart interval, etc). On average, we select the best configuration from 5 sets for each experiment; these few trials have empirically outperformed previous SDE/ODE samplers. We believe that developing a systematic approach for determining Restart configurations could be of significant value in the future.
C.3 Pre-trained Models
For CIFAR-10 dataset, we use the pre-trained VP and EDM models from the EDM repository  $^4$ , and  $\mathrm{PFGM} + +$  ( $D = 2048$ ) model from the PFGM++ repository  $^5$ . For ImageNet  $64 \times 64$ , we borrow the pre-trained EDM model from EDM repository as well.
C.4 Classifier-free Guidance
We follow the convention in [20], where each step in classifier-free guidance is as follows:
$$
\tilde {s} _ {\theta} (x, c, t) = w s _ {\theta} (x, c, t) + (1 - w) s _ {\theta} (x, t)
$$
where  $c$  is the conditions, and  $s_{\theta}(x,c,t) / s_{\theta}(x,t)$  is the conditional/unconditional models, sharing parameters. Increasing  $w$  would strengthen the effect of guidance, usually leading to a better text-image alignment [20].
C.5 More on the Synthetic Experiment
C.5.1 Discrete Dataset