SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Applying change-of-variable with log-SNR  $\lambda_{t} = \log (\alpha_{t} / \sigma_{t})$  and substituting the following relationship
$$
f (t) = \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t}, \quad g ^ {2} (t) = \frac {\mathrm {d} \sigma_ {t} ^ {2}}{\mathrm {d} t} - 2 \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} \sigma_ {t} ^ {2} = - 2 \sigma_ {t} ^ {2} \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}, \tag {25}
$$
Eq. (24) becomes
$$
\begin{array}{l} \mathrm {d} \pmb {x} _ {t} = \left[ \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} \pmb {x} _ {t} - \left(1 + \tau^ {2} (t)\right) (\pmb {x} _ {t} - \alpha_ {t} \pmb {x} _ {\pmb {\theta}} (\pmb {x} _ {t}, t)) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \right] \mathrm {d} t + \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {\pmb {w}} _ {t} \\ = \left[ \left(\frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} - (1 + \tau^ {2} (t)) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}\right) \boldsymbol {x} _ {t} + (1 + \tau^ {2} (t)) \alpha_ {t} \boldsymbol {x} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \right] \mathrm {d} t \tag {26} \\ + \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {\boldsymbol {w}} _ {t}. \\ \end{array}
$$
A.2.2 Proof of Proposition 4.2
Proposition 4.2 Given  $\pmb{x}_s$  for any time  $s > 0$ , the solution  $\pmb{x}_t$  at time  $t \in [0, s]$  of Eq. (9) is
$$
\boldsymbol {F} _ {\boldsymbol {\theta}} (s, t) = \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \int_ {\lambda} ^ {\lambda_ {t}} \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}} \left(1 + \tau^ {2} (\lambda)\right) e ^ {\lambda} \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {\lambda}, \lambda\right) \mathrm {d} \lambda \tag {27}
$$
$$
\pmb {G} (s, t) = \int_ {s} ^ {t} e ^ {- \int_ {\lambda_ {u}} ^ {\lambda_ {t}} \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {\boldsymbol {w}} _ {u},
$$
$$
\sigma_ {t} \boldsymbol {G} (s, t) \sim \mathcal {N} \left(\boldsymbol {0}, \sigma_ {t} ^ {2} \left(1 - e ^ {- 2 \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}}\right)\right). \tag {28}
$$
Proof. Define  $\pmb{y}_t = e^{-\int_{t_0}^t\left(\frac{\mathrm{d}\log\alpha_v}{\mathrm{d}v} - \left(1 + \tau^2(v)\right)\frac{\mathrm{d}\lambda_v}{\mathrm{d}v}\right)\mathrm{d}v}\pmb{x}_t$ , where  $t_0 \in [0, T]$  is a constant. Differentiate  $\pmb{y}_t$  with respect to  $t$ , we get
$$
\begin{array}{l} \mathrm {d} \boldsymbol {y} _ {t} = - \left(\frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} - (1 + \tau^ {2} (t)) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}\right) e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - (1 + \tau^ {2} (v)) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \boldsymbol {x} _ {t} \mathrm {d} t \\ + e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - \left(1 + \tau^ {2} (v)\right) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \mathrm {d} \boldsymbol {x} _ {t} \\ = - \left(\frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} - \left(1 + \tau^ {2} (t)\right) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}\right) e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - \left(1 + \tau^ {2} (v)\right) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \boldsymbol {x} _ {t} \mathrm {d} t \\ + e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - (1 + \tau^ {2} (v)) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \left[ \left(\frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} - (1 + \tau^ {2} (t)) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}\right) x _ {t} \right] \mathrm {d} t \\ + e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - \left(1 + \tau^ {2} (v)\right) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \left[ \left(1 + \tau^ {2} (t)\right) \alpha_ {t} \boldsymbol {x} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \right] \mathrm {d} t \tag {29} \\ + e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - \left(1 + \tau^ {2} (v)\right) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {w} _ {t} \\ = e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - \left(1 + \tau^ {2} (v)\right) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \left[ \left(1 + \tau^ {2} (t)\right) \alpha_ {t} \pmb {x} _ {\pmb {\theta}} (\pmb {x} _ {t}, t) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \right] \mathrm {d} t \\ + e ^ {- \int_ {t _ {0}} ^ {t} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - \left(1 + \tau^ {2} (v)\right) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {\boldsymbol {w}} _ {t}. \\ \end{array}
$$
Integrating both sides from  $s$  to  $t$
$$
\begin{array}{l} \boldsymbol {y} _ {t} = \boldsymbol {y} _ {s} + \int_ {s} ^ {t} e ^ {- \int_ {t _ {0}} ^ {u} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - (1 + \tau^ {2} (v)) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \left[ (1 + \tau^ {2} (u)) \alpha_ {u} \boldsymbol {x} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {u}, u) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \right] \mathrm {d} u \tag {30} \\ + \int_ {s} ^ {t} e ^ {- \int_ {t _ {0}} ^ {u} \left(\frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} - \left(1 + \tau^ {2} (v)\right) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v}\right) \mathrm {d} v} \tau (u) \sigma_ {u} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {w} _ {u}. \\ \end{array}
$$
Substituting the definition of  $\mathbf{y}_t, \mathbf{y}_s$  into Eq. (30), we obtain Eq. (27)
The last term  $\sigma_t\int_s^t e^{-\int_u^t\tau^2 (v)\frac{\mathrm{d}\lambda_v}{\mathrm{d}v}\mathrm{d}v}\tau (u)\sqrt{-2\frac{\mathrm{d}\lambda_u}{\mathrm{d}u}}\mathrm{d}\bar{w}_u$  of Eq. (31) is the Ito integral term. It follows a Gaussian distribution, which can be directly derived from two basic facts [38]: first, the definition of Ito integral is the limitation in  $L^2$  space; second, the limit of Gaussian Process in  $L^2$  space is still Gaussian. Then we can compute the mean as:
$$
\mathbb {E} \left[ \sigma_ {t} \int_ {s} ^ {t} e ^ {- \int_ {u} ^ {t} \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {\boldsymbol {w}} _ {u} \right] = 0 \tag {32}
$$
and the variance is
$$
\begin{array}{l} \mathrm {V a r} \left[ \sigma_ {t} \int_ {s} ^ {t} e ^ {- \int_ {u} ^ {t} \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {\boldsymbol {w}} _ {u} \right] \\ = \mathbb {E} \left[ \left(\sigma_ {t} \int_ {s} ^ {t} e ^ {- \int_ {u} ^ {t} \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {w} _ {u}\right) ^ {2} \right] - \left(\mathbb {E} \left[ \sigma_ {t} \int_ {s} ^ {t} e ^ {- \int_ {u} ^ {t} \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} {\bar {w}} _ {u} \right]\right) ^ {2} \\ = \mathbb {E} \left[ \left(\sigma_ {t} \int_ {t} ^ {s} e ^ {- \int_ {u} ^ {t} \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \boldsymbol {w} _ {u}\right) ^ {2} \right] - 0 \\ = \sigma_ {t} ^ {2} \mathbb {E} \left[ \int_ {t} ^ {s} \left(e ^ {- \int_ {u} ^ {t} \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}}\right) ^ {2} \mathrm {d} u \right] \\ = \sigma_ {t} ^ {2} \int_ {t} ^ {s} e ^ {- \int_ {u} ^ {t} 2 \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau^ {2} (u) \left(- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}\right) \mathrm {d} u \\ = \sigma_ {t} ^ {2} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 e ^ {- \int_ {\lambda} ^ {\lambda_ {t}} 2 \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}} \tau^ {2} (\lambda) \mathrm {d} \lambda \tag {33} \\ \end{array}
$$
The expectation equals zero because the Itô integral is a martingale [38]. The computation of variance uses the Itô Isometry, which is a crucial fact of Itô integral. We can further simplify the result by using the change of variable  $P(\lambda) = e^{\int_{\lambda_t}^{\lambda} 2\tau^2(\tilde{\lambda}) \, \mathrm{d}\tilde{\lambda}}$ .
$$
\begin{array}{l} \mathrm {V a r} \left[ \sigma_ {t} \int_ {s} ^ {t} e ^ {- \int_ {u} ^ {t} \tau^ {2} (v) \frac {\mathrm {d} \lambda_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {\boldsymbol {w}} _ {u} \right] \\ = \sigma_ {t} ^ {2} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 e ^ {- \int_ {\lambda} ^ {\lambda_ {t}} 2 \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}} \tau^ {2} (\lambda) \mathrm {d} \lambda (34) \\ = \sigma_ {t} ^ {2} \int_ {P \left(\lambda_ {s}\right)} ^ {P \left(\lambda_ {t}\right)} P (\lambda) \frac {\mathrm {d} P (\lambda)}{P (\lambda)} (34) \\ = \sigma_ {t} ^ {2} \left(P \left(\lambda_ {t}\right) - P \left(\lambda_ {s}\right)\right) \\ = \sigma_ {t} ^ {2} \left(1 - e ^ {- 2 \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \tau^ {2} (\lambda) d \lambda}\right) \\ \end{array}
$$
A.2.3 Noise Prediction Reparameterization
After approximating  $\nabla_{\pmb{x}}\log p_t(\pmb{x}_t)$  with  $s_\theta (\pmb {x}_t,t)$  and reparameterizing  $s_{\pmb{\theta}}(\pmb {x}_t,t)$  with  $-\epsilon_{\pmb{\theta}}(\pmb {x}_t,t)) / \sigma_t$  Eq. (20) becomes
$$
\mathrm {d} \boldsymbol {x} _ {t} = \left[ f (t) \boldsymbol {x} _ {t} + \left(\frac {1 + \tau^ {2} (t)}{2 \sigma_ {t}}\right) g ^ {2} (t) \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) \right] \mathrm {d} t + \tau (t) g (t) \mathrm {d} \bar {\boldsymbol {w}} _ {t}. \tag {35}
$$
$$
f (t) = \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t}, \quad g ^ {2} (t) = \frac {\mathrm {d} \sigma_ {t} ^ {2}}{\mathrm {d} t} - 2 \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} \sigma_ {t} ^ {2} = - 2 \sigma_ {t} ^ {2} \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}, \tag {36}
$$
Eq. (35) becomes
$$
\mathrm {d} \boldsymbol {x} _ {t} = \left[ \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} \boldsymbol {x} _ {t} - \left(1 + \tau^ {2} (t)\right) \sigma_ {t} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \right] \mathrm {d} t + \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {\boldsymbol {w}} _ {t}. \tag {37}
$$
Eq (35) is the formulation of noist prediction model. Similar with Proposition 4.2, Eq. (37) can be solved analytically, which is shown in the following proposition:
Proposition A.1. Given  $\pmb{x}_s$  for any time  $s > 0$ , the solution  $\pmb{x}_t$  at time  $t \in [0, s]$  of (37) is
$$
\boldsymbol {x} _ {t} = \frac {\alpha_ {t}}{\alpha_ {s}} \boldsymbol {x} _ {s} + \alpha_ {t} \boldsymbol {F} _ {\boldsymbol {\theta}} (s, t) + \alpha_ {t} \boldsymbol {G} (s, t),
$$
$$
\boldsymbol {F} _ {\boldsymbol {\theta}} (s, t) = \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} \left(1 + \tau^ {2} (\lambda)\right) \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {\lambda}, \lambda\right) d \lambda \tag {38}
$$
$$
\boldsymbol {G} (s, t) = \int_ {s} ^ {t} e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {\boldsymbol {w}} _ {u},
$$
$$
\alpha_ {t} \boldsymbol {G} (s, t) \sim \mathcal {N} \left(\boldsymbol {0}, \alpha_ {t} ^ {2} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 e ^ {- 2 \lambda} \tau^ {2} (\lambda) \mathrm {d} \lambda\right). \tag {39}
$$
Proof. Define  $\pmb{y}_t = e^{-\int_{t_0}^t \frac{\mathrm{d}\log \alpha_v}{\mathrm{d}v} \, \mathrm{d}v} \pmb{x}_t$ , where  $t_0 \in [0, T]$  is a constant. Differentiate  $\pmb{y}_t$  with respect to  $t$ , we get
$$
\begin{array}{l} \mathrm {d} \boldsymbol {y} _ {t} = - \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} \boldsymbol {x} _ {t} \mathrm {d} t + e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} \mathrm {d} \boldsymbol {x} _ {t} \\ = - \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} \boldsymbol {x} _ {t} \mathrm {d} t + e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} \frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} \boldsymbol {x} _ {t} \mathrm {d} t \\ - e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha v}{\mathrm {d} v} \mathrm {d} v} (1 + \tau^ {2} (t)) \sigma_ {t} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \mathrm {d} t + e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha v}{\mathrm {d} v} \mathrm {d} v} \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {\boldsymbol {w}} _ {t} \\ = - e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} (1 + \tau^ {2} (t)) \sigma_ {t} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \mathrm {d} t + e ^ {- \int_ {t _ {0}} ^ {t} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {\boldsymbol {w}} _ {t}. \tag {40} \\ \end{array}
$$
$$
\begin{array}{l} \boldsymbol {y} _ {t} = \boldsymbol {y} _ {s} - \int_ {s} ^ {t} e ^ {- \int_ {t _ {0}} ^ {u} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} (1 + \tau^ {2} (u)) \sigma_ {u} \epsilon_ {\boldsymbol {\theta}} (\boldsymbol {x} _ {u}, u) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \tag {41} \\ + \int_ {s} ^ {t} e ^ {- \int_ {t _ {0}} ^ {u} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sigma_ {u} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {w} _ {u}. \\ \end{array}
$$
Substituting the definition of  $\pmb{y}_t, \pmb{y}_s$  into Eq. (41), we obtain
$$
\begin{array}{l} \boldsymbol {x} _ {t} = e ^ {\int_ {s} ^ {t} \frac {\mathrm {d} \log \alpha_ {u}}{\mathrm {d} u} \mathrm {d} u} \boldsymbol {x} _ {s} + \int_ {s} ^ {t} e ^ {- \int_ {t} ^ {u} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} (1 + \tau^ {2} (u)) \sigma_ {u} \epsilon_ {\boldsymbol {\theta}} (\boldsymbol {x} _ {u}, u) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u \\ + \int_ {s} ^ {t} e ^ {- \int_ {t} ^ {u} \frac {\mathrm {d} \log \alpha_ {v}}{\mathrm {d} v} \mathrm {d} v} \tau (u) \sigma_ {u} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {\boldsymbol {w}} _ {u} \\ = \frac {\alpha_ {t}}{\alpha_ {s}} \boldsymbol {x} _ {s} + \int_ {s} ^ {t} \frac {\alpha_ {t}}{\alpha_ {u}} (1 + \tau^ {2} (u)) \sigma_ {u} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {u}, u) \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u} \mathrm {d} u + \int_ {s} ^ {t} \frac {\alpha_ {t}}{\alpha_ {u}} \tau (u) \sigma_ {u} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {\boldsymbol {w}} _ {u} \tag {42} \\ = \frac {\alpha_ {t}}{\alpha_ {s}} \boldsymbol {x} _ {s} + \alpha_ {t} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} e ^ {- \lambda} (1 + \tau^ {2} (\lambda)) \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {\lambda}, \lambda) d \lambda + \alpha_ {t} \int_ {s} ^ {t} e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {d \lambda_ {u}}{d u}} d \bar {\boldsymbol {w}} _ {u}. \\ \end{array}
$$
The Itô integral term  $\alpha_{t}\int_{s}^{t}e^{-\lambda_{u}}\tau (u)\sqrt{-2\frac{\mathrm{d}\lambda_{u}}{\mathrm{d}u}}\mathrm{d}\bar{w}_{u}$  follows a Gaussian distribution. Following the derivation in Proposition 4.2, the mean of the Itô integral term is:
$$
\mathbb {E} \left[ \alpha_ {t} \int_ {s} ^ {t} e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {w} _ {u} \right] = 0 \tag {43}
$$
and the expectation is
$$
\begin{array}{l} \operatorname {V a r} \left[ \alpha_ {t} \int_ {s} ^ {t} e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {w} _ {u} \right] \\ = \mathbb {E} \left[ \left(\alpha_ {t} \int_ {s} ^ {t} e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {w} _ {u}\right) ^ {2} \right] - \left(\mathbb {E} \left[ \alpha_ {t} \int_ {s} ^ {t} e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \bar {w} _ {u} \right]\right) ^ {2} \\ = \mathbb {E} \left[ \left(\alpha_ {t} \int_ {t} ^ {s} e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}} \mathrm {d} \boldsymbol {w} _ {u}\right) ^ {2} \right] - 0 \tag {44} \\ = \alpha_ {t} ^ {2} \mathbb {E} \left[ \int_ {t} ^ {s} \left(e ^ {- \lambda_ {u}} \tau (u) \sqrt {- 2 \frac {\mathrm {d} \lambda_ {u}}{\mathrm {d} u}}\right) ^ {2} \mathrm {d} u \right] \\ = \alpha_ {t} ^ {2} \int_ {t} ^ {s} e ^ {- 2 \lambda_ {u}} \tau^ {2} (u) d u \left(- 2 \frac {d \lambda_ {u}}{d u}\right) d u \\ = \alpha_ {t} ^ {2} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 e ^ {- 2 \lambda} \tau^ {2} (\lambda) d \lambda \\ \end{array}
$$
A.2.4 Comparison between Data and Noise Reparameterizations
In Table 1 we perform an ablation study on data and noise reparameterizations, the experiment results show that under the same magnitude of stochasticity, the proposed SA-Solver in data reparameterization has a better convergence which leads to better FID results under the same NFEs. In this subsection, we provide a theoretical view of this phenomenon.
Corollary A.2. For any bounded measurable function  $\tau(t)$ , the following inequality holds
$$
\sigma_ {t} ^ {2} \left(1 - e ^ {- 2 \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}}\right) \leq \alpha_ {t} ^ {2} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 e ^ {- 2 \lambda} \tau^ {2} (\lambda) \mathrm {d} \lambda . \tag {45}
$$
Proof. It's equivalent to show that
$$
1 - e ^ {- 2 \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}} \leq e ^ {2 \lambda_ {t}} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 e ^ {- 2 \lambda} \tau^ {2} (\lambda) \mathrm {d} \lambda . \tag {46}
$$
From the basic inequality  $1 - e^{-x} \leq x$ , we have
$$
1 - e ^ {- 2 \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \tau^ {2} (\tilde {\lambda}) \mathrm {d} \tilde {\lambda}} \leq 2 \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \tau^ {2} (\lambda) \mathrm {d} \lambda . \tag {47}
$$
Thus it's sufficient to show that
$$
e ^ {2 \lambda_ {t}} \int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 e ^ {- 2 \lambda} \tau^ {2} (\lambda) \mathrm {d} \lambda \geq 2 \int_ {\lambda_ {s}} ^ {\lambda_ {t}} \tau^ {2} (\lambda) \mathrm {d} \lambda , \tag {48}
$$
which is true because
$$
\int_ {\lambda_ {s}} ^ {\lambda_ {t}} 2 \left(e ^ {2 \left(\lambda_ {t} - \lambda\right)} - 1\right) \tau^ {2} (\lambda) \mathrm {d} \lambda \geq 0, \tag {49}
$$
This corollary indicates that the same SDE under two different reparameterizations has different properties under the effect of the exponential integrator. Specifically, in the numerical scheme, the data reparameterization will inject smaller noise in each step's updation. We speculate that this is the reason that the data reparameterization has a better convergence, shown as in Table 1.
B Derivations and Proofs for SA-Solver
We will first review some basic concepts and formulas in the numerical solutions of SDEs [39]. Suppose we have an Itô SDE  $\mathrm{d}\pmb{x}_t = f(\pmb{x}_t,t)\mathrm{d}t + g(\pmb{x}_t,t)\mathrm{d}\pmb{w}_t$  and time steps  $\{t_i\}_{i=0}^M$ ,  $t_i \in [0,T]$  to numerically solve the SDE. For a random variable  $Z$ , we define the  $L_1$  norm  $\| Z\|_{L_1} = \mathbb{E}[|Z|]$ , the  $L_2$  norm  $\| Z\|_{L_2} = \mathbb{E}\left[|Z|^2\right]^{\frac{1}{2}}$ , where  $|\cdot|$  is the Euclidean norm. Denote  $h = \max_{1 \leq i \leq M} (t_i - t_{i-1})$ .
Definition B.1. We shall say that a time-discrete approximation  $\pmb{x}_0, \dots, \pmb{x}_M$ , where  $\pmb{x}_i$  is a numerical approximation of  $\pmb{x}_{t_i}$ , converges strongly with order  $\gamma > 0$ , if there exists a positive constant  $C$  which does not depend on  $h$  and a  $h_0 > 0$  such that
$$
\max  _ {0 \leq i \leq M} \| \boldsymbol {x} _ {t _ {i}} - \boldsymbol {x} _ {i} \| _ {L _ {1}} \leq C h ^ {\gamma}, \quad \forall h \leq h _ {0}. \tag {50}
$$
Definition B.2. We say it is mean-square convergent with order  $\gamma > 0$ , if there exists a positive constant  $C$ , which does not depend on  $h$  and a  $h_0 > 0$  such that
$$
\max  _ {0 \leq i \leq M} \| \boldsymbol {x} _ {t _ {i}} - \boldsymbol {x} _ {i} \| _ {L _ {2}} \leq C h ^ {\gamma}, \quad \forall h \leq h _ {0}. \tag {51}
$$
Remark 2. To prove the strong convergence order  $\gamma$  of a numerical scheme, it's sufficient to show the mean-square convergence order  $\gamma$ . This is from Hölder Inequality  $\mathbb{E}\left[|Z|\right] \leq \mathbb{E}\left[|Z|^2\right]^{\frac{1}{2}}\mathbb{E}\left[|1|^2\right]^{\frac{1}{2}} \leq \mathbb{E}\left[|Z|^2\right]^{\frac{1}{2}}$ . Thus  $\max_{0 \leq i \leq M} \left\| \boldsymbol{x}_{t_i} - \boldsymbol{x}_i \right\|_{L_1} \leq \max_{0 \leq i \leq M} \left\| \boldsymbol{x}_{t_i} - \boldsymbol{x}_i \right\|_{L_2}$ .
We also need the following definition and assumptions, which usually hold in practical diffusion models.
Definition B.3. A function  $h: \mathbb{R}^d \times [0, T] \to \mathbb{R}^d$  satisfies a linear growth condition if there exists a constant  $K$  such that
$$
\left| h (\boldsymbol {x}, t) \right| \leq K \left(1 + \left| \boldsymbol {x} \right| ^ {2}\right) ^ {\frac {1}{2}} \tag {52}
$$
Assumption B.4. The data prediction model  $\pmb{x}_{\theta}$  and its derivatives such as  $\partial_t \pmb{x}_{\theta}$ ,  $\nabla_{\pmb{x}} \pmb{x}_{\theta}$  and  $\Delta \pmb{x}_{\theta}$  satisfy the linear growth condition.
Assumption B.5. The data prediction model  $x_{\theta}$  satisfies a uniform Lipschitz condition with respect to  $x$
$$
\left| \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {1}, t\right) - \boldsymbol {x} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {2}, t\right) \right| \leq L \left| \boldsymbol {x} _ {1} - \boldsymbol {x} _ {2} \right|, \quad \forall x, y \in \mathbb {R} ^ {d}, t \in [ 0, T ] \tag {53}
$$
B.2 Outline of the Proof
In the remaining part of this section, we will focus on our variance controlled SDE
$$
\begin{array}{l} \mathrm {d} \boldsymbol {x} _ {t} = \left[ \left(\frac {\mathrm {d} \log \alpha_ {t}}{\mathrm {d} t} - (1 + \tau^ {2} (t)) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}\right) \boldsymbol {x} _ {t} + (1 + \tau^ {2} (t)) \alpha_ {t} \boldsymbol {x} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {t}, t) \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t} \right] \mathrm {d} t \tag {54} \\ + \tau (t) \sigma_ {t} \sqrt {- 2 \frac {\mathrm {d} \lambda_ {t}}{\mathrm {d} t}} \mathrm {d} \bar {w} _ {t}. \\ \end{array}
$$