SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3. [Formal version of Theorem 1] Let  $t_{\text{max}}$  be the initial noise level. Let the initial random variables  $\overline{x}_{t_{\text{max}}} = \overline{y}_{t_{\text{max}}}$ , and
$$
\bar {x} _ {t _ {\min }} = O D E _ {\theta} \left(\bar {x} _ {t _ {\max }}, t _ {\max } \rightarrow t _ {\min }\right)
$$
$$
\bar {y} _ {t _ {\text {m i n}}} = S D E _ {\theta} \left(\bar {y} _ {t _ {\text {m a x}}}, t _ {\text {m a x}} \rightarrow t _ {\text {m i n}}\right),
$$
Let  $p_t$  denote the true population distribution at noise level  $t$ . Let  $p_t^{ODE_\theta}$ ,  $p_t^{SDE_\theta}$  denote the distributions for  $x_t, y_t$  respectively. Assume that for all  $x, y, s, t$ ,  $s_\theta(x, t)$  satisfies  $\| ts_\theta(x, t) - ts_\theta(x, s) \| \leq L_0 |s - t|$ ,  $\| ts_\theta(x, t) \| \leq L_1$ ,  $\| ts_\theta(x, t) - ts_\theta(y, t) \| \leq L_2 \| x - y \|$ , and the approximation error  $\| ts_\theta(x, t) - t\nabla \log p_t(x) \| \leq \epsilon_{approx}$ . Assume in addition that  $\forall t \in [t_{\min}, t_{\max}]$ ,  $\| x_t \| < B/2$  for any  $x_t$  in the support of  $p_t$ ,  $p_t^{ODE_\theta}$  or  $p_t^{SDE_\theta}$ , and  $K \leq \frac{C}{L_2(t_{\max} - t_{\min})}$  for some universal constant  $C$ . Then
$$
\begin{array}{l} W _ {1} \left(p _ {t _ {\text {m i n}}} ^ {O D E _ {\theta}}, p _ {t _ {\text {m i n}}}\right) \leq B \cdot T V \left(p _ {t _ {\text {m a x}}} ^ {O D E _ {\theta}}, p _ {t _ {\text {m a x}}}\right) \\ + e ^ {L _ {2} \left(t _ {\max } - t _ {\min }\right)} \cdot \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max } - t _ {\min }\right) \tag {8} \\ \end{array}
$$
$$
\begin{array}{l} W _ {1} \left(p _ {t _ {\min }} ^ {S D E _ {\theta}}, p _ {t _ {\min }}\right) \leq B \cdot \left(1 - \lambda e ^ {- B L _ {1} / t _ {\min } - L _ {1} ^ {2} t _ {\max } ^ {2} / t _ {\min } ^ {2}}\right) T V \left(p _ {t _ {\max }} ^ {S D E _ {\theta}}, p _ {t _ {\max }}\right) \\ + e ^ {2 L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\epsilon_ {\text {a p p r o x}} + \delta L _ {0} + L _ {2} \left(\delta L _ {1} + \sqrt {2 \delta d t _ {\max }}\right)\right) \left(t _ {\max } - t _ {\min }\right) \tag {9} \\ \end{array}
$$
where  $\lambda := 2Q\left(\frac{B}{2\sqrt{t_{max}^2 - t_{min}^2}}\right)$ .
Proof. Let us define  $x_{t_{\mathrm{max}}} \sim p_{t_{\mathrm{max}}}$ , and let  $x_{t_{\mathrm{min}}} = \mathrm{ODE}(x_{t_{\mathrm{max}}}, t_{\mathrm{max}} \to t_{\mathrm{min}})$ . We verify that  $x_{t_{\mathrm{min}}}$  has density  $p_{t_{\mathrm{min}}}$ . Let us also define  $\hat{x}_{t_{\mathrm{min}}} = \mathrm{ODE}_{\theta}(x_{t_{\mathrm{max}}}, t_{\mathrm{max}} \to t_{\mathrm{min}})$ . We would like to bound the Wasserstein distance between  $\bar{x}_{t_{\mathrm{min}}}$  and  $x_{t_{\mathrm{min}}}$  (i.e.,  $p_{t_{\mathrm{min}}}^{\mathrm{ODE}_{\theta}}$  and  $p_{t_{\mathrm{min}}}$ ), by the following triangular inequality:
$$
W _ {1} \left(\bar {x} _ {t _ {\min }}, x _ {t _ {\min }}\right) \leq W _ {1} \left(\bar {x} _ {t _ {\min }}, \hat {x} _ {t _ {\min }}\right) + W _ {1} \left(\hat {x} _ {t _ {\min }}, x _ {t _ {\min }}\right) \tag {10}
$$
By Lemma 2, we know that
$$
\left\| \hat {x} _ {t _ {\min }} - x _ {t _ {\min }} \right\| \leq e ^ {\left(t _ {\max } - t _ {\min }\right) L _ {2}} \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max } - t _ {\min }\right),
$$
where we use the fact that  $\left\| \hat{x}_{t_{\mathrm{max}}} - x_{t_{\mathrm{max}}} \right\| = 0$ . Thus we immediately have
$$
W _ {1} \left(\hat {x} _ {t _ {\min }}, x _ {t _ {\min }}\right) \leq e ^ {\left(t _ {\max } - t _ {\min }\right) L _ {2}} \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max } - t _ {\min }\right) \tag {11}
$$
$$
\begin{array}{l} W _ {1} \left(\hat {x} _ {t _ {\min }}, \bar {x} _ {t _ {\min }}\right) \leq B \cdot T V \left(\hat {x} _ {t _ {\min }}, \bar {x} _ {t _ {\min }}\right) \\ \leq B \cdot T V \left(\hat {x} _ {t _ {\max }}, \bar {x} _ {t _ {\max }}\right) \tag {12} \\ \end{array}
$$
where the last equality is due to the data-processing inequality. Combining Eq. (11), Eq. (12) and the triangular inequality Eq. (10), we arrive at the upper bound for ODE (Eq. (8)). The upper bound for SDE (Eq. (9)) shares a similar proof approach. First, let  $y_{t_{\mathrm{max}}} \sim p_{t_{\mathrm{max}}}$ . Let  $\hat{y}_{t_{\mathrm{min}}} = \mathrm{SDE}_{\theta}(y_{t_{\mathrm{max}}}, t_{\mathrm{max}} \to t_{\mathrm{min}})$ . By Lemma 5,
$$
T V \left(\hat {y} _ {t _ {\min}}, \overline {{y}} _ {t _ {\min}}\right) \leq \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\max} ^ {2} - t _ {\min} ^ {2}}}\right) \cdot e ^ {- B L _ {1} / t _ {\min} - L _ {1} ^ {2} t _ {\max} ^ {2} / t _ {\min} ^ {2}}\right) \cdot T V \left(\hat {y} _ {t _ {\max}}, \overline {{y}} _ {t _ {\max}}\right)
$$
On the other hand, by Lemma 4,
$$
\mathbb {E} \left[ \| \hat {y} _ {t _ {\min }} - y _ {t _ {\min }} \| \right] \leq e ^ {2 L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\epsilon_ {\text {a p p r o x}} + \delta L _ {0} + L _ {2} \left(\delta L _ {1} + \sqrt {2 \delta d t _ {\max }}\right)\right) \left(t _ {\max } - t _ {\min }\right).
$$
The SDE triangular upper bound on  $W_{1}(\bar{y}_{t_{\min}}, y_{t_{\min}})$  follows by multiplying the first inequality by  $B$  (to bound  $W_{1}(\bar{y}_{t_{\min}}, \hat{y}_{t_{\min}})$ ) and then adding the second inequality (to bound  $W_{1}(y_{t_{\min}}, \hat{y}_{t_{\min}})$ ). Notice that by definition,  $TV(\hat{y}_{t_{\max}}, \bar{y}_{t_{\max}}) = TV(y_{t_{\max}}, \bar{y}_{t_{\max}})$ . Finally, because of the assumption that  $K \leq \frac{C}{L_{2}(t_{\max} - t_{\min})}$  for some universal constant, we summarize the second term in the Eq. (8) and Eq. (9) into the big  $O$  in the informal version Theorem 1.
Theorem 4. [Formal version of Theorem 2] Consider the same setting as Theorem 3. Let  $p_{t_{\min}}^{Restart_{\theta},i}$  denote the distributions after  $i^{\text{th}}$  Restart iteration, i.e., the distribution of  $\overline{x}_{t_{\min}}^i = \text{Restart}_\theta(\overline{x}_{t_{\min}}^0, i)$ . Given initial  $\overline{x}_{t_{\max}}^0 \sim p_{t_{\max}}^{Restart,0}$ , let  $\overline{x}_{t_{\min}}^0 = ODE_\theta(\overline{x}_{t_{\max}}^0, t_{\max} \to t_{\min})$ . Then
$$
\begin{array}{l} W _ {1} \left(p _ {t _ {\min }} ^ {R e s t a r t _ {\theta}, K}, p _ {t _ {\min }}\right) \leq \underbrace {B \cdot (1 - \lambda) ^ {K} T V \left(p _ {t _ {\max }} ^ {R e s t a r t , 0} , p _ {t _ {\max }}\right)} _ {\text {u p p e r b o u n d o n c o n t r a c t e d e r r o r}} \\ + \underbrace {e ^ {(K + 1) L _ {2} \left(t _ {\max} - t _ {\min}\right)} (K + 1) \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max} - t _ {\min}\right)} _ {\text {u p p e r b o u n d o n a d d i t i o n a l s a m p l i n g e r r o r}} \tag {13} \\ \end{array}
$$
$$
w h e r e \lambda = 2 Q \left(\frac {B}{2 \sqrt {t _ {\max} ^ {2} - t _ {\min} ^ {2}}}\right).
$$
Proof. Let  $x_{t_{\mathrm{max}}}^{0} \sim p_{t_{\mathrm{max}}}$ . Let  $x_{t_{\mathrm{min}}}^{K} = \operatorname{Restart}(x_{t_{\mathrm{min}}}^{0}, K)$ . We verify that  $x_{t_{\mathrm{min}}}^{K}$  has density  $p_{t_{\mathrm{min}}}$ . Let us also define  $\hat{x}_{t_{\mathrm{min}}}^{0} = \mathrm{ODE}_{\theta}(x_{t_{\mathrm{max}}}^{0}, t_{\mathrm{max}} \to t_{\mathrm{min}})$  and  $\hat{x}_{t_{\mathrm{min}}}^{K} = \operatorname{Restart}_{\theta}(\hat{x}_{t_{\mathrm{min}}}^{0}, K)$ .
By Lemma 1,
$$
\begin{array}{l} T V \left(\bar {x} _ {t _ {\min }} ^ {K}, \hat {x} _ {t _ {\min }} ^ {K}\right) \leq \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\max } ^ {2} - t _ {\min } ^ {2}}}\right)\right) ^ {K} T V \left(\bar {x} _ {t _ {\min }} ^ {0}, \hat {x} _ {t _ {\min }} ^ {0}\right) \\ \leq \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\max} ^ {2} - t _ {\min} ^ {2}}}\right)\right) ^ {K} T V \left(\bar {x} _ {t _ {\max}} ^ {0}, \hat {x} _ {t _ {\max}} ^ {0}\right) \\ = \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\max} ^ {2} - t _ {\min} ^ {2}}}\right)\right) ^ {K} T V \left(\bar {x} _ {t _ {\max}} ^ {0}, x _ {t _ {\max}} ^ {0}\right) \\ \end{array}
$$
The second inequality holds by data processing inequality. The above can be used to bound the 1-Wasserstein distance as follows:
$$
W _ {1} \left(\bar {x} _ {t _ {\min }} ^ {K}, \hat {x} _ {t _ {\min }} ^ {K}\right) \leq B \cdot T V \left(\bar {x} _ {t _ {\min }} ^ {K}, \hat {x} _ {t _ {\min }} ^ {K}\right) \leq \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\max } ^ {2} - t _ {\min } ^ {2}}}\right)\right) ^ {K} T V \left(\bar {x} _ {t _ {\max }} ^ {0}, x _ {t _ {\max }} ^ {0}\right) \tag {14}
$$
On the other hand, using Lemma 3,
$$
\begin{array}{l} W _ {1} \left(x _ {t _ {\mathrm {m i n}}} ^ {K}, \hat {x} _ {t _ {\mathrm {m i n}}} ^ {K}\right) \leq \left\| x _ {t _ {\mathrm {m i n}}} ^ {K} - \hat {x} _ {t _ {\mathrm {m i n}}} ^ {K} \right\| \\ \leq e ^ {(K + 1) L _ {2} \left(t _ {\max } - t _ {\min }\right)} (K + 1) \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max } - t _ {\min }\right) \tag {15} \\ \end{array}
$$
We arrive at the result by combining the two bounds above (Eq. (14), Eq. (15)) with the following triangular inequality,
$$
W _ {1} (\bar {x} _ {t _ {\mathrm {m i n}}} ^ {K}, x _ {t _ {\mathrm {m i n}}} ^ {K}) \leq W _ {1} (\bar {x} _ {t _ {\mathrm {m i n}}} ^ {K}, \hat {x} _ {t _ {\mathrm {m i n}}} ^ {K}) + W _ {1} (\hat {x} _ {t _ {\mathrm {m i n}}} ^ {K}, x _ {t _ {\mathrm {m i n}}} ^ {K})
$$
A.2 Mixing under Restart with exact ODE
Lemma 1. Consider the same setup as Theorem 4. Consider the  $\text{Restart}_{\theta}$  process defined in equation 7. Let
$$
x _ {t _ {\text {m i n}}} ^ {i} = \operatorname {R e s t a r t} _ {\theta} \left(x _ {t _ {\text {m i n}}} ^ {0}, i\right)
$$
$$
y _ {t _ {m i n}} ^ {i} = R e s t a r t _ {\theta} (y _ {t _ {m i n}} ^ {0}, i).
$$
Let  $p_t^{Restart_\theta(i)}$  and  $q_t^{Restart_\theta(i)}$  denote the densities of  $x_t^i$  and  $y_t^i$  respectively. Then
$$
T V \left(p _ {t _ {\text {m i n}}} ^ {\text {R e s t a r t} _ {\theta} (K)}, q _ {t _ {\text {m i n}}} ^ {\text {R e s t a r t} _ {\theta} (K)}\right) \leq (1 - \lambda) ^ {K} T V \left(p _ {t _ {\text {m i n}}} ^ {\text {R e s t a r t} _ {\theta} (0)}, q _ {t _ {\text {m i n}}} ^ {\text {R e s t a r t} _ {\theta} (0)}\right),
$$
where  $\lambda = 2Q\left(\frac{B}{2\sqrt{t_{max}^2 - t_{min}^2}}\right)$ .
Proof. Conditioned on  $x_{t_{\min}}^i, y_{t_{\min}}^i$ , let  $x_{t_{\max}}^{i+1} = x_{t_{\min}}^i + \sqrt{t_{\max}^2 - t_{\min}^2} \xi_i^x$  and  $y_{t_{\max}}^{i+1} = y_{t_{\min}}^i + \sqrt{t_{\max}^2 - t_{\min}^2} \xi_i^y$ . We now define a coupling between  $x_{t_{\min}}^{i+1}$  and  $y_{t_{\min}}^{i+1}$  by specifying the joint distribution over  $\xi_i^x$  and  $\xi_i^y$ .
If  $x_{t_{\min}}^i = y_{t_{\min}}^i$ , let  $\xi_i^x = \xi_i^y$ , so that  $x_{t_{\min}}^{i + 1} = y_{t_{\min}}^{i + 1}$ . On the other hand, if  $x_{t_{\min}}^i \neq y_{t_{\min}}^i$ , let  $x_{t_{\max}}^{i + 1}$  and  $y_{t_{\max}}^{i + 1}$  be coupled as described in the proof of Lemma 7, with  $x' = x_{t_{\max}}^{i + 1}, y' = y_{t_{\max}}^{i + 1}, \sigma = \sqrt{t_{\max}^2 - t_{\min}^2}$ . Under this coupling, we verify that,
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {1} \left\{x _ {t _ {\min }} ^ {i + 1} \neq y _ {t _ {\min }} ^ {i + 1} \right\} \right] \\ \leq \mathbb {E} \left[ \mathbb {1} \left\{x _ {t _ {\max }} ^ {i + 1} \neq y _ {t _ {\max }} ^ {i + 1} \right\} \right] \\ \leq \mathbb {E} \left[ \left(1 - 2 Q \left(\frac {\| x _ {t _ {\operatorname* {m i n}}} ^ {i} - y _ {t _ {\operatorname* {m i n}}} ^ {i} \|}{2 \sqrt {t _ {\operatorname* {m a x}} ^ {2} - t _ {\operatorname* {m i n}} ^ {2}}}\right)\right) \mathbb {1} \left\{x _ {t _ {\operatorname* {m i n}}} ^ {i} \neq y _ {t _ {\operatorname* {m i n}}} ^ {i} \right\} \right] \\ \leq \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\operatorname* {m a x}} ^ {2} - t _ {\operatorname* {m i n}} ^ {2}}}\right)\right) \mathbb {E} \left[ \mathbb {1} \left\{x _ {t _ {\operatorname* {m i n}}} ^ {i} \neq y _ {t _ {\operatorname* {m i n}}} ^ {i} \right\} \right]. \\ \end{array}
$$
Applying the above recursively,
$$
\mathbb {E} \left[ \mathbb {1} \left\{x _ {t _ {\min }} ^ {K} \neq y _ {t _ {\min }} ^ {K} \right\} \right] \leq \left(1 - 2 Q \left(\frac {B}{2 \sqrt {t _ {\max } ^ {2} - t _ {\min } ^ {2}}}\right)\right) ^ {K} \mathbb {E} \left[ \mathbb {1} \left\{x _ {t _ {\min }} ^ {0} \neq y _ {t _ {\min }} ^ {0} \right\} \right].
$$
The conclusion follows by noticing that  $TV\left(p_{t_{\min}}^{\text{Restart}_{\theta}(K)}, q_{t_{\min}}^{\text{Restart}_{\theta}(K)}\right) \leq Pr\left(x_{t_{\min}}^{K} \neq y_{t_{\min}}^{K}\right) = \mathbb{E}\left[\mathbb{1}\left\{x_{t_{\min}}^{K} \neq y_{t_{\min}}^{K}\right\}\right]$ , and by selecting the initial coupling so that  $Pr\left(x_{t_{\min}}^{0} \neq y_{t_{\min}}^{0}\right) = TV\left(p_{t_{\min}}^{\text{Restart}_{\theta}(0)}, q_{t_{\min}}^{\text{Restart}_{\theta}(0)}\right)$ .
A.3  $W_{1}$  discretization bound
Lemma 2 (Discretization bound for ODE). Let  $x_{t_{\min}} = ODE(x_{t_{\max}}, t_{\max} \to t_{\min})$  and let  $\overline{x}_{t_{\min}} = ODE_{\theta}(\overline{x}_{t_{\max}}, t_{\max} \to t_{\min})$ . Assume that for all  $x, y, s, t, s_{\theta}(x, t)$  satisfies  $\| ts_{\theta}(x, t) - ts_{\theta}(x, s) \| \leq L_0 |s - t|$ ,  $\| ts_{\theta}(x, t) \| \leq L_1$  and  $\| ts_{\theta}(x, t) - ts_{\theta}(y, t) \| \leq L_2$ $\| x - y \|$ . Then
$$
\left\| x _ {t _ {\min }} - \bar {x} _ {t _ {\min }} \right\| \leq e ^ {\left(t _ {\max } - t _ {\min }\right) L _ {2}} \left(\left\| x _ {t _ {\max }} - \bar {x} _ {t _ {\max }} \right\| + \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max } - t _ {\min }\right)\right)
$$
Proof. Consider some fixed arbitrary  $k$ , and recall that  $\delta$  is the step size. Recall that by definition of ODE and  $\mathrm{ODE}_{\theta}$ , for  $t \in ((k - 1)\delta, k\delta]$ ,
$$
d x _ {t} = - t \nabla \log p _ {t} (x _ {t}) d t
$$
$$
d \bar {x} _ {t} = - t s _ {\theta} (\bar {x} _ {k \delta}, k \delta) d t.
$$
For  $t \in [t_{\min}, t_{\max}]$ , let us define a time-reversed process  $x_t^{\leftarrow} \coloneqq x_{-t}$ . Let  $v(x, t) \coloneqq \nabla \log p_{-t}(x)$ . Then for  $t \in [-t_{\max}, -t_{\min}]$
$$
d x _ {t} ^ {\leftarrow} = t v \left(x _ {t} ^ {\leftarrow}, t\right) d s.
$$
Similarly, define  $\overline{x}_t^{\leftarrow} := \overline{x}_{-t}$  and  $\overline{v}(x,t) := s_{\theta}(x,-t)$ . It follows that
$$
d \bar {x} _ {t} ^ {\leftarrow} = t \bar {v} (\bar {x} _ {k \delta} ^ {\prime}, k \delta) d s,
$$
where  $k$  is the unique (negative) integer satisfying  $t \in [k\delta, (k + 1)\delta)$ . Following these definitions,
$$
\begin{array}{l} \frac {d}{d t} \left\| x _ {t} ^ {\leftarrow} - \bar {x} _ {t} ^ {\leftarrow} \right\| \\ \leq \left\| t v \left(x _ {t} ^ {\leftarrow}, t\right) - t \bar {v} \left(x _ {t} ^ {\leftarrow}, t\right) \right\| \\ + \left\| t \bar {v} \left(x _ {t} ^ {\leftarrow}, t\right) - t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, t\right) \right\| \\ + \left\| t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, t\right) - t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, k \delta\right) \right\| \\ + \left\| t \bar {v} \left(\bar {x} _ {t} ^ {\leftarrow}, k \delta\right) - t \bar {v} \left(\bar {x} _ {k \delta} ^ {\leftarrow}, k \delta\right) \right\| \\ \leq \epsilon_ {a p p r o x} + L _ {2} \| x _ {t} ^ {\leftarrow} - \bar {x} _ {t} ^ {\leftarrow} \| + \delta L _ {0} + L _ {2} \| \bar {x} _ {t} ^ {\leftarrow} - \bar {x} _ {k \delta} ^ {\leftarrow} \| \\ \leq \epsilon_ {a p p r o x} + L _ {2} \left\| x _ {t} ^ {\leftarrow} - \bar {x} _ {t} ^ {\leftarrow} \right\| + \delta L _ {0} + \delta L _ {2} L _ {1}. \\ \end{array}
$$
Applying Gronwall's Lemma over the interval  $t \in [-t_{\max}, -t_{\min}]$ ,
$$
\begin{array}{l} \left\| x _ {t _ {\min }} - \bar {x} _ {t _ {\min }} \right\| \\ = \left\| x _ {- t _ {\min }} ^ {\leftarrow} - \bar {x} _ {- t _ {\min }} ^ {\leftarrow} \right\| \\ \leq e ^ {L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\left\| x _ {- t _ {\max }} ^ {-} - \bar {x} _ {- t _ {\max }} ^ {-} \right\| + \left(\epsilon_ {\text {a p p r o x}} + \delta L _ {0} + \delta L _ {2} L _ {1}\right) \left(t _ {\max } - t _ {\min }\right)\right) \\ = e ^ {L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\| x _ {t _ {\max }} - \bar {x} _ {t _ {\max }} \| + \left(\epsilon_ {\text {a p p r o x}} + \delta L _ {0} + \delta L _ {2} L _ {1}\right) \left(t _ {\max } - t _ {\min }\right)\right). \\ \end{array}
$$
Lemma 3. Given initial  $x_{t_{\text{max}}}^0$ , let  $x_{t_{\text{min}}}^0 = ODE(x_{t_{\text{max}}}^0, t_{\text{max}} \to t_{\text{min}})$ , and let  $\hat{x}_{t_{\text{min}}}^0 = ODE_\theta(x_{t_{\text{max}}}^0, t_{\text{max}} \to t_{\text{min}})$ . We further denote the variables after  $K$  Restart iterations as  $x_{t_{\text{min}}}^K = \text{Restart}(x_{t_{\text{min}}}^0, K)$  and  $\hat{x}_{t_{\text{min}}}^K = \text{Restart}_\theta(\hat{x}_{t_{\text{min}}}^0, K)$ , with true field and learned field respectively. Then there exists a coupling between  $x_{t_{\text{min}}}^K$  and  $\hat{x}_{t_{\text{min}}}^K$  such that
$$
\left\| x _ {t _ {\min }} ^ {K} - \hat {x} _ {t _ {\min }} ^ {K} \right\| \leq e ^ {(K + 1) L _ {2} (t _ {\max } - t _ {\min })} (K + 1) (\delta (L _ {2} L _ {1} + L _ {0}) + \epsilon_ {\text {a p p r o x}}) (t _ {\max } - t _ {\min })
$$
Proof. We will couple  $x_{t_{\min}}^i$  and  $\hat{x}_{t_{\min}}^i$  by using the same noise  $\varepsilon_{t_{\min} \to t_{\max}}^i$  in the Restart forward process for  $i = 0 \ldots K - 1$  (see Eq. (7)). For any  $i$ , let us also define  $y_{t_{\min}}^{i,j} \coloneqq \text{Restart}_\theta(x_{t_{\min}}^i, j - i)$ , and this process uses the same noise  $\varepsilon_{t_{\min} \to t_{\max}}^i$  as previous ones. From this definition,  $y_{t_{\min}}^{K,K} = x_{t_{\min}}^K$ . We can thus bound
$$
\left\| x _ {t _ {\min }} ^ {K}, \hat {x} _ {t _ {\min }} ^ {K} \right\| \leq \left\| y _ {t _ {\min }} ^ {0, K} - \hat {x} _ {t _ {\min }} ^ {K} \right\| + \sum_ {i = 0} ^ {K - 1} \left\| y _ {t _ {\min }} ^ {i, K} - y _ {t _ {\min }} ^ {i + 1, K} \right\| \tag {16}
$$
Using the assumption that  $ts_{\theta}(\cdot, t)$  is  $L_{2}$  Lipschitz,
$$
\begin{array}{l} \left\| y _ {t _ {\min }} ^ {0, i + 1} - \hat {x} _ {t _ {\min }} ^ {i + 1} \right\| \\ = \left\| \mathrm {O D E} _ {\theta} \left(y _ {t _ {\max }} ^ {0, i}, t _ {\max } \rightarrow t _ {\min }\right) - \mathrm {O D E} _ {\theta} \left(\hat {x} _ {t _ {\max }} ^ {i}, t _ {\max } \rightarrow t _ {\min }\right)\right\| \\ \leq e ^ {L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left\| y _ {t _ {\max }} ^ {0, i} - \hat {x} _ {t _ {\max }} ^ {i} \right\| \\ = e ^ {L _ {2} (t _ {\mathrm {m a x}} - t _ {\mathrm {m i n}})} \left\| y _ {t _ {\mathrm {m i n}}} ^ {0, i} - \hat {x} _ {t _ {\mathrm {m i n}}} ^ {i} \right\|, \\ \end{array}
$$
where the last equality is because we add the same additive Gaussian noise  $\varepsilon_{t_{\mathrm{min}}\to t_{\mathrm{max}}}^i$  to  $y_{t_{\mathrm{min}}}^{0,i}$  and  $\hat{x}_{t_{\mathrm{min}}}^i$  in the Restart forward process. Applying the above recursively, we get
$$
\begin{array}{l} \left| \left| y _ {t _ {\min }} ^ {0, K} - \hat {x} _ {t _ {\min }} ^ {K} \right| \right| \leq e ^ {K L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left| \left| y _ {t _ {\min }} ^ {0, 0} - \hat {x} _ {t _ {\min }} ^ {0} \right| \right| \\ \leq e ^ {K L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left\| x _ {t _ {\min }} ^ {0} - \hat {x} _ {t _ {\min }} ^ {0} \right\| \\ \leq e ^ {(K + 1) L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max } - t _ {\min }\right), \tag {17} \\ \end{array}
$$
where the last line follows by Lemma 2 when setting  $x_{t_{\mathrm{max}}} = \bar{x}_{t_{\mathrm{max}}}$ . We will now bound  $\left\| y_{t_{\mathrm{min}}}^{i,K} - y_{t_{\mathrm{min}}}^{i + 1,K} \right\|$  for some  $i \leq K$ . It follows from definition that
$$
\begin{array}{l} y _ {t _ {\min}} ^ {i, i + 1} = \mathrm {O D E} _ {\theta} \left(x _ {t _ {\max}} ^ {i}, t _ {\max} \rightarrow t _ {\min}\right) \\ y _ {t _ {\min }} ^ {i + 1, i + 1} = x _ {t _ {\min }} ^ {i + 1} = \mathrm {O D E} \left(x _ {t _ {\max }} ^ {i}, t _ {\max } \rightarrow t _ {\min }\right). \\ \end{array}
$$
$$
\left\| y _ {t _ {\min }} ^ {i, i + 1} - y _ {t _ {\min }} ^ {i + 1, i + 1} \right\| \leq e ^ {L _ {2} \left(t _ {\max } - t _ {\min }\right)} \left(\delta \left(L _ {2} L _ {1} + L _ {0}\right) + \epsilon_ {\text {a p p r o x}}\right) \left(t _ {\max } - t _ {\min }\right)
$$
For the remaining steps from  $i + 2\ldots K$ , both  $y^{i,\cdot}$  and  $y^{i + 1,\cdot}$  evolve with  $\mathrm{ODE}_{\theta}$  in each step. Again using the assumption that  $ts_{\theta}(\cdot ,t)$  is  $L_{2}$  Lipschitz,
$$
\left\| y _ {t _ {\min }} ^ {i, K} - y _ {t _ {\min }} ^ {i + 1, K} \right\| \leq e ^ {(K - i) L _ {2} (t _ {\max } - t _ {\min })} (\delta (L _ {2} L _ {1} + L _ {0}) + \epsilon_ {a p p r o x}) (t _ {\max } - t _ {\min })
$$
Summing the above for  $i = 0\ldots K - 1$ , and combining with Eq. (16) and Eq. (17) gives
$$
\left\| x _ {t _ {\min }} ^ {K} - \hat {x} _ {t _ {\min }} ^ {K} \right\| \leq e ^ {(K + 1) L _ {2} (t _ {\max } - t _ {\min })} (K + 1) (\delta (L _ {2} L _ {1} + L _ {0}) + \epsilon_ {a p p r o x}) (t _ {\max } - t _ {\min })
$$
Lemma 4. Consider the same setup as Theorem 3. Let  $x_{t_{\min}} = SDE(x_{t_{\max}}, t_{\max} \to t_{\min})$  and let  $\overline{x}_{t_{\min}} = SDE(\overline{x}_{t_{\max}}, t_{\max} \to t_{\min})$ . Then there exists a coupling between  $x_t$  and  $\overline{x}_t$  such that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| x _ {t _ {m i n}} - \overline {{x}} _ {t _ {m i n}} \right\| \right] \leq e ^ {2 L _ {2} (t _ {m a x} - t _ {m i n})} \mathbb {E} \left[ \left\| x _ {t _ {m a x}} - \overline {{x}} _ {t _ {m a x}} \right\| \right] \\ + e ^ {2 L _ {2} (t _ {m a x} - t _ {m i n})} \left(\epsilon_ {a p p r o x} + \delta L _ {0} + L _ {2} \left(\delta L _ {1} + \sqrt {2 \delta d t _ {m a x}}\right)\right) (t _ {m a x} - t _ {m i n}) \\ \end{array}
$$
Proof. Consider some fixed arbitrary  $k$ , and recall that  $\delta$  is the stepsize. By definition of SDE and  $\mathrm{SDE}_{\theta}$ , for  $t \in ((k - 1)\delta, k\delta]$ ,
$$
d x _ {t} = - 2 t \nabla \log p _ {t} (x _ {t}) d t + \sqrt {2 t} d B _ {t}
$$
$$
d \bar {x} _ {t} = - 2 t s _ {\theta} (\bar {x} _ {k \delta}, k \delta) d t + \sqrt {2 t} d B _ {t}.
$$
Let us define a coupling between  $x_{t}$  and  $\overline{x}_t$  by identifying their respective Brownian motions. It will be convenient to define the time-reversed processes  $x_{t}^{\leftarrow} := x_{-t}$ , and  $\overline{x}_t^\leftarrow := \overline{x}_{-t}$ , along with  $v(x,t) := \nabla \log p_{-t}(x)$  and  $\bar{v} (x,t)\coloneqq s_{\theta}(x, - t)$ . Then there exists a Brownian motion  $B_{t}^{\leftarrow}$ , such that for  $t\in [-t_{\mathrm{max}}, - t_{\mathrm{min}}]$
$$
\begin{array}{l} d x _ {t} ^ {\leftarrow} = - 2 t v \left(x _ {t} ^ {\leftarrow}, t\right) d t + \sqrt {- 2 \bar {t}} d B _ {t} ^ {\leftarrow} \\ d \bar {x} _ {t} ^ {\leftarrow} = - 2 t \bar {v} (\bar {x} _ {k \delta} ^ {\leftarrow}, k \delta) d t + \sqrt {- 2 t} d B _ {t} ^ {\leftarrow} \\ \Rightarrow \quad d \left(x _ {t} ^ {\leftarrow} - \bar {x} _ {t} ^ {\leftarrow}\right) = - 2 t \left(v \left(x _ {t} ^ {\leftarrow}, t\right) - \bar {v} \left(\bar {x} _ {k \delta} ^ {\leftarrow}, k \delta\right)\right) d t, \left. \right. \\ \end{array}
$$
where  $k$  is the unique negative integer such that  $t \in [k\delta, (k + 1)\delta)$ . Thus