SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The completion of this research could not have been finished without the support of Ming-Yu Chung, Shao-Wei Chen, and Yu-Rong Zhang. Thank Ming-Yu for careful verification of the derivation and detailed explanation for some complex theories. We would also like to express our gratefulness for Shao-Wei Chen, who provides impressive solutions of complicated SDEs and numerical solution of Navier-Stoke thermodynamic model. Finally, we also appreciate Yu-Rong for his insights of textual backdoor on the stable diffusion.
[55] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In NIPS, 2020.
[56] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[57] Lukas Struppek, Dominik Hintersdorf, and Kristian Kersting. Rickrolling the artist: Injecting invisible backdoors into text-guided image generation models. In ArXiv, 2022.
[58] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In CoRR, 2022.
[59] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In ICLR, 2023.
[60] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[61] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. 2023.
A Code Base
Our code is available on GitHub: https://github.com/IBM/villandiffusion
B Mathematical Derivation
B.1 Clean Diffusion Model via Numerical Reparametrization
Recall that we have defined the forward process  $q(\mathbf{x}_t|\mathbf{x}_0) \coloneqq \mathcal{N}(\hat{\alpha}(t)\mathbf{x}_0, \hat{\beta}^2(t)\mathbf{I}), t \in [T_{min}, T_{max}]$  for general diffusion models, which is determined by the content scheduler  $\hat{\alpha}(t): \mathbb{R} \to \mathbb{R}$  and the noise scheduler  $\hat{\beta}(t): \mathbb{R} \to \mathbb{R}$ . Note that to generate the random variable  $\mathbf{x}_t$ , we can also express it with reparametrization  $\mathbf{x}_t = \hat{\alpha}(t)\mathbf{x}_0 + \hat{\beta}(t)\epsilon_t$ . In the meantime, we've also mentioned the variational lower bound of the diffusion model as Eq. (17).
$$
- \log p _ {\theta} (\mathbf {x} _ {0}) = - \mathbb {E} _ {q} \left[ \log p _ {\theta} (\mathbf {x} _ {0}) \right] \leq \mathbb {E} _ {q} \left[ \mathcal {L} _ {T} (\mathbf {x} _ {T}, \mathbf {x} _ {0}) + \sum_ {t = 2} ^ {T} \mathcal {L} _ {t} (\mathbf {x} _ {t}, \mathbf {x} _ {t - 1}, \mathbf {x} _ {0}) - \mathcal {L} _ {0} (\mathbf {x} _ {1}, \mathbf {x} _ {0}) \right] \tag {17}
$$
Denote  $\mathcal{L}_t(\mathbf{x}_t,\mathbf{x}_{t - 1},\mathbf{x}_0) = D_{\mathrm{KL}}(q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)\parallel p_\theta (\mathbf{x}_{t - 1}|\mathbf{x}_t)),\mathcal{L}_T(\mathbf{x}_T,\mathbf{x}_0) = D_{\mathrm{KL}}(q(\mathbf{x}_T|\mathbf{x}_0)\parallel p_\theta (\mathbf{x}_T))$ , and  $\mathcal{L}_0(\mathbf{x}_1,\mathbf{x}_0) = \log p_\theta (\mathbf{x}_0|\mathbf{x}_1)$ , where  $D_{KL}(q||p) = \int_xq(x)\log \frac{q(x)}{p(x)}$  is the KL-Divergence. Since  $\mathcal{L}_t$  usually dominates the bound, we can ignore  $\mathcal{L}_T$  and  $\mathcal{L}_0$  and focus on  $D_{\mathrm{KL}}(q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)\parallel p_\theta (\mathbf{x}_{t - 1}|\mathbf{x}_t))$ . In Appendix B.1.1, we will derive the clean conditional reversed transition  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$ . As for the learned reversed transition  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t)$ , we will derive it in Appendix B.1.2. Finally, combining these two parts, we will present the loss function of the clean diffusion model in Appendix B.1.3.
B.1.1 Clean Reversed Conditional Transition  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$
Similar to the derivation of DDPM, we approximate reversed transition as  $q(\mathbf{x}_{t-1}|\mathbf{x}_t) \approx q(\mathbf{x}_{t-1}|\mathbf{x}_t,\mathbf{x}_0)$ . We also define the clean reversed conditional transition as Eq. (18).
$$
q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right) := \mathcal {N} \left(\mu_ {t} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}\right), s ^ {2} (t) \mathbf {I}\right), \mu_ {t} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}\right) = a (t) \mathbf {x} _ {t} + b (t) \mathbf {x} _ {0} \tag {18}
$$
To show that the temporal content and noise schedulers are  $a(t) = \frac{k_t\hat{\beta}^2(t - 1)}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$  and  $b(t) = \frac{\hat{\alpha}(t - 1)w_t^2}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ , with Bayesian rule and Markovian property  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0) = q(\mathbf{x}_t|\mathbf{x}_{t - 1})\frac{q(\mathbf{x}_{t - 1}|\mathbf{x}_0)}{q(\mathbf{x}_t|\mathbf{x}_0)}$ , we can expand the reversed conditional transition  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$  as Eq. (19). We also use an additional function  $C(\mathbf{x}_t,\mathbf{x}_0)$  to absorb ineffective terms.
$$
\begin{array}{l} q (\mathbf {x} _ {t - 1} | \mathbf {x} _ {t}, \mathbf {x} _ {0}) \\ = q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}, \mathbf {x} _ {0}\right) \frac {q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {0}\right)}{q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right)} \\ \propto \exp \Big (- \frac {1}{2} \big (\frac {\left(\mathbf {x} _ {t} - k _ {t} \mathbf {x} _ {t - 1}\right) ^ {2}}{w _ {t} ^ {2}} + \frac {\left(\mathbf {x} _ {t - 1} - \hat {\alpha} (t - 1) \mathbf {x} _ {0}\right) ^ {2}}{\hat {\beta} ^ {2} (t - 1)} - \frac {\left(\mathbf {x} _ {t} - \hat {\alpha} (t) \mathbf {x} _ {0}\right) ^ {2}}{\hat {\beta} ^ {2} (t)} \big) \Big) \\ = \exp \Big (- \frac {1}{2} \big (\frac {\mathbf {x} _ {t} ^ {2} - 2 k _ {t} \mathbf {x} _ {t} \mathbf {x} _ {t - 1} + k _ {t} ^ {2} \mathbf {x} _ {t - 1} ^ {2}}{w _ {t} ^ {2}} + \frac {\mathbf {x} _ {t - 1} ^ {2} - 2 \hat {\alpha} (t - 1) \mathbf {x} _ {0} \mathbf {x} _ {t - 1} + \hat {\alpha} ^ {2} (t - 1) \mathbf {x} _ {0} ^ {2}}{\hat {\beta} ^ {2} (t - 1)} - \frac {(\mathbf {x} _ {t} - \hat {\alpha} (t) \mathbf {x} _ {0}) ^ {2}}{\hat {\beta} ^ {2} (t)} \big) \\ = \exp \left(- \frac {1}{2} \left(\left(\frac {k _ {t} ^ {2}}{w _ {t} ^ {2}} + \frac {1}{\hat {\beta} ^ {2} (t - 1)}\right) \mathbf {x} _ {t - 1} ^ {2} - \left(\frac {2 k _ {t}}{w _ {t} ^ {2}} \mathbf {x} _ {t} + \frac {2 \hat {\alpha} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} \mathbf {x} _ {0}\right) \mathbf {x} _ {t - 1} + C (\mathbf {x} _ {t}, \mathbf {x} _ {0})\right)\right) \tag {19} \\ \end{array}
$$
Thus,  $a(t)$  and  $b(t)$  can be derived as Eq. (20)
$$
\begin{array}{l} a (t) \mathbf {x} _ {t} + b (t) \mathbf {x} _ {0} = (\frac {k _ {t}}{w _ {t} ^ {2}} \mathbf {x} _ {t} + \frac {\hat {\alpha} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} \mathbf {x} _ {0}) / (\frac {k _ {t} ^ {2}}{w _ {t} ^ {2}} + \frac {1}{\hat {\beta} ^ {2} (t - 1)}) \\ = \left(\frac {k _ {t}}{w _ {t} ^ {2}} \mathbf {x} _ {t} + \frac {\hat {\alpha} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} \mathbf {x} _ {0}\right) \frac {w _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \tag {20} \\ = \frac {k _ {t} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {t} + \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {0} \\ \end{array}
$$
After comparing the coefficients, we can get  $a(t) = \frac{k_t\hat{\beta}^2(t - 1)}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$  and  $b(t) = \frac{\hat{\alpha}(t - 1)w_t^2}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ . Recall that based on the definition of the forward process  $q(\mathbf{x}_t|\mathbf{x}_0)\coloneqq \mathcal{N}(\hat{\alpha} (t)\mathbf{x}_0,\hat{\beta}^2 (t)\mathbf{I})$ , we can obtain the reparametrization:  $\mathbf{x}_0 = \frac{1}{\hat{\alpha}(t)} (\mathbf{x}_t - \hat{\beta} (t)\epsilon_t)$ . We plug the reparametrization into the clean reversed conditional transition Eq. (20).
$$
\mu_ {t} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}\right) = \frac {k _ {t} \hat {\beta} ^ {2} (t - 1) \hat {\alpha} (t) + \hat {\alpha} (t - 1) w _ {t} ^ {2}}{\hat {\alpha} (t) \left(k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}\right)} \mathbf {x} _ {t} - \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \frac {\hat {\beta} (t)}{\hat {\alpha} (t)} \epsilon_ {t} \tag {21}
$$
B.1.2 Learned Clean Reversed Conditional Transition  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$
To train a diffusion model that can approximate the clean reversed conditional transition, we define a clean reversed transition  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  learned by trainable parameters  $\theta$  as Eq. (22)
$$
p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) := \mathcal {N} \left(\mathbf {x} _ {t - 1}; \mu_ {\theta} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}, t\right), s ^ {2} (t) \mathbf {I}\right) \tag {22}
$$
With similar logic in Eq. (21) and replacing  $\epsilon_{t}$  with a learned diffusion model  $\epsilon_{\theta}(\mathbf{x}_t,t)$ , we can also derive  $\mu_{\theta}(\mathbf{x}_t,\mathbf{x}_0,t)$  as Eq. (23).
$$
\begin{array}{l} \mu_ {\theta} (\mathbf {x} _ {t}, \mathbf {x} _ {0}, t) = \frac {k _ {t} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {t} + \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \left(\frac {1}{\hat {\alpha} (t)} (\mathbf {x} _ {t} - \hat {\beta} (t) \epsilon_ {\theta} (\mathbf {x} _ {t}, t))\right) \\ = \frac {k _ {t} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {t} + \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \frac {1}{\hat {\alpha} (t)} \mathbf {x} _ {t} - \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \frac {\hat {\beta} (t)}{\hat {\alpha} (t)} \epsilon_ {\theta} (\mathbf {x} _ {t}, t) \\ = \frac {k _ {t} \hat {\beta} ^ {2} (t - 1) \hat {\alpha} (t) + \hat {\alpha} (t - 1) w _ {t} ^ {2}}{\hat {\alpha} (t) \left(k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}\right)} \mathbf {x} _ {t} - \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \frac {\hat {\beta} (t)}{\hat {\alpha} (t)} \epsilon_ {\theta} (\mathbf {x} _ {t}, t) \tag {23} \\ \end{array}
$$
B.1.3 Loss Function of Clean Diffusion Models
The KL-divergence loss of the reversed transition can be simplified as Eq. (24), which uses mean-matching as an approximation of the KL-divergence.
$$
\begin{array}{l} D _ {K L} \left(q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right) | | p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right)\right) \\ \propto \left| \left| \mu_ {t} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}\right) - \mu_ {\theta} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}, t\right) \right| \right| ^ {2} \\ = \left\| \left(- \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \frac {\hat {\beta} (t)}{\hat {\alpha} (t)} \epsilon_ {t}\right) - \left(- \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \frac {\hat {\beta} (t)}{\hat {\alpha} (t)} \epsilon_ {\theta} (\mathbf {x} _ {t}, t)\right) \right\| ^ {2} \tag {24} \\ \propto \left| \left| \epsilon_ {t} - \epsilon_ {\theta} (\mathbf {x} _ {t}, t) \right| \right| ^ {2} \\ \end{array}
$$
Thus, we can finally write down the clean loss function Eq. (25) with reparametrization  $\mathbf{x}_t(\mathbf{x},\epsilon) = \hat{\alpha} (t)\mathbf{x} + \hat{\beta} (t)\epsilon$ ,  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ .
$$
\mathcal {L} _ {c} (\mathbf {x}, t, \epsilon) := \left| \left| \epsilon - \epsilon_ {\theta} \left(\mathbf {x} _ {t} (\mathbf {x}, \epsilon), t\right) \right| \right| ^ {2} \tag {25}
$$
B.2 Backdoor Diffusion Model via Numerical Reparametrization
This section will further extend the derivation of the clean diffusion models in Appendix B.1 and derive the backdoor reversed conditional transition  $q(\mathbf{x}_{t - 1}^{\prime}|\mathbf{x}_{t}^{\prime},\mathbf{x}_{0}^{\prime})$  and the backdoor loss function in Appendix B.2.1.
B.2.1 Backdoor Reversed Conditional Transition  $q(\mathbf{x}_{t - 1}^{\prime}|\mathbf{x}_{t}^{\prime},\mathbf{x}_{0}^{\prime})$
Recall the definition of the backdoor reversed conditional transition in Eq. (26). For clarity, We mark the coefficients of the  $\mathbf{r}$  as red.
$$
q \left(\mathbf {x} _ {t - 1} ^ {\prime} \mid \mathbf {x} _ {t} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}\right) := \mathcal {N} \left(\mu_ {t} ^ {\prime} \left(\mathbf {x} _ {t} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}\right), s ^ {2} (t) \mathbf {I}\right), \mu_ {t} ^ {\prime} \left(\mathbf {x} _ {t} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}\right) = a (t) \mathbf {x} _ {t} ^ {\prime} + c (t) \mathbf {r} + b (t) \mathbf {x} _ {0} ^ {\prime} \tag {26}
$$
We firstly show that the temporal content, noise, and correction schedulers are  $a(t) = \frac{k_t\hat{\beta}^2(t - 1)}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ ,  $b(t) = \frac{\hat{\alpha}(t - 1)w_t^2}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ , and  $c(t) = \frac{w_t^2\hat{\rho}(t - 1) - k_th_t\hat{\beta}(t - 1)}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ . Thus, first of all, we can expand the reversed
conditional transition  $q(\mathbf{x}_{t - 1}^{\prime}|\mathbf{x}_{t}^{\prime},\mathbf{x}_{0}^{\prime})$  as Eq. (27). To absorb the ineffective terms, we introduce an additional function  $C^\prime (\mathbf{x}_t^\prime ,\mathbf{x}_0^\prime)$ . We mark the coefficients of the  $\mathbf{r}$  as red.
$$
\begin{array}{l} q (\mathbf {x} _ {t - 1} ^ {\prime} | \mathbf {x} _ {t} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}) \\ = q \left(\mathbf {x} _ {t} ^ {\prime} \mid \mathbf {x} _ {t - 1} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}\right) \frac {q \left(\mathbf {x} _ {t - 1} ^ {\prime} \mid \mathbf {x} _ {0} ^ {\prime}\right)}{q \left(\mathbf {x} _ {t} ^ {\prime} \mid \mathbf {x} _ {0} ^ {\prime}\right)} \\ \propto \exp \Big (- \frac {1}{2} \big (\frac {(\mathbf {x} _ {t} ^ {\prime} - k _ {t} \mathbf {x} _ {t - 1} ^ {\prime} - h _ {t} \mathbf {r}) ^ {2}}{w _ {t} ^ {2}} + \frac {(\mathbf {x} _ {t - 1} ^ {\prime} - \hat {\alpha} (t - 1) \mathbf {x} _ {0} ^ {\prime} - \hat {\rho} (t - 1) \mathbf {r}) ^ {2}}{\hat {\beta} ^ {2} (t - 1)} \Big) \\ \left. - \frac {\left(\mathbf {x} _ {t} ^ {\prime} - \hat {\alpha} (t) \mathbf {x} _ {0} ^ {\prime} - \hat {\rho} (t) \mathbf {r}\right) ^ {2}}{\hat {\beta} ^ {2} (t)}\right) \\ = \exp \left(- \frac {1}{2} \left(\frac {\left(\mathbf {x} _ {t} ^ {\prime} - k _ {t} \mathbf {x} _ {t - 1} ^ {\prime}\right) ^ {2} - 2 \left(\mathbf {x} _ {t} ^ {\prime} - k _ {t} \mathbf {x} _ {t - 1} ^ {\prime}\right) h _ {t} \mathbf {r} + h _ {t} ^ {2} \mathbf {r} ^ {2}}{w _ {t} ^ {2}}\right) \right. \\ + \frac {\left(\mathbf {x} _ {t - 1} ^ {\prime} - \hat {\alpha} (t - 1) \mathbf {x} _ {0} ^ {\prime}\right) ^ {2} - 2 \left(\mathbf {x} _ {t - 1} ^ {\prime} - \hat {\alpha} (t - 1) \mathbf {x} _ {0} ^ {\prime}\right) \hat {\rho} (t - 1) \mathbf {r} + \hat {\rho} (t - 1) ^ {2} \mathbf {r} ^ {2}}{\hat {\beta} ^ {2} (t - 1)} \tag {27} \\ \left. - \frac {\left(\mathbf {x} _ {t} ^ {\prime} - \hat {\alpha} (t) \mathbf {x} _ {0} ^ {\prime} - \hat {\rho} (t) \mathbf {r}\right) ^ {2}}{\hat {\beta} ^ {2} (t)}\right) \\ = \exp \left( \right.- \frac {1}{2} \left(\frac {\left(\mathbf {x} _ {t} ^ {\prime} - k _ {t} \mathbf {x} _ {t - 1} ^ {\prime}\right) ^ {2}}{w _ {t} ^ {2}} + \frac {\left(\mathbf {x} _ {t - 1} ^ {\prime} - \hat {\alpha} (t - 1) \mathbf {x} _ {0} ^ {\prime}\right) ^ {2}}{\hat {\beta} ^ {2} (t - 1)} - \frac {2 \left(\mathbf {x} _ {t} ^ {\prime} - k _ {t} \mathbf {x} _ {t - 1} ^ {\prime}\right) h _ {t} \mathbf {r}}{w _ {t} ^ {2}}\right) \\ \left. - \frac {2 \left(\mathbf {x} _ {t - 1} ^ {\prime} - \hat {\alpha} (t - 1) \mathbf {x} _ {0} ^ {\prime}\right) \hat {\rho} (t - 1) \mathbf {r}}{\hat {\beta} ^ {2} (t - 1)} - \frac {\left(\mathbf {x} _ {t} ^ {\prime} - \hat {\alpha} (t) \mathbf {x} _ {0} ^ {\prime} - \hat {\rho} (t) \mathbf {r}\right) ^ {2}}{\hat {\beta} ^ {2} (t)}\right) \\ = \exp \Big (- \frac {1}{2} \big ((\frac {k _ {t} ^ {2}}{w _ {t} ^ {2}} + \frac {1}{\hat {\beta} ^ {2} (t - 1)}) \mathbf {x} _ {t - 1} ^ {\prime 2} - 2 (\frac {k _ {t}}{w _ {t} ^ {2}} \mathbf {x} _ {t} ^ {\prime} + \frac {\hat {\alpha} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} \mathbf {x} _ {0} ^ {\prime} \\ \left. + \left(\frac {\hat {\rho} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} - \frac {k _ {t} h _ {t}}{w _ {t} ^ {2}}\right) \mathbf {r}\right) \mathbf {x} _ {t - 1} ^ {\prime} + C ^ {\prime} \left(\mathbf {x} _ {t} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime})\right)\left. \right) \\ \end{array}
$$
Thus, the content, noise, and correction schedulers  $a(t), b(t)$ , and  $c(t)$  can be derived as Eq. (28). We mark the coefficients of the  $\mathbf{r}$  as red.
$$
\begin{array}{l} a (t) \mathbf {x} _ {t} ^ {\prime} + c (t) \mathbf {r} + b (t) \mathbf {x} _ {0} ^ {\prime} = (\frac {k _ {t}}{w _ {t} ^ {2}} \mathbf {x} _ {t} ^ {\prime} + \frac {\hat {\alpha} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} \mathbf {x} _ {0} ^ {\prime} + (\frac {\hat {\rho} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} - \frac {k _ {t} h _ {t}}{w _ {t} ^ {2}}) \mathbf {r}) / (\frac {k _ {t} ^ {2}}{w _ {t} ^ {2}} + \frac {1}{\hat {\beta} ^ {2} (t - 1)}) \\ = \left(\frac {k _ {t}}{w _ {t} ^ {2}} \mathbf {x} _ {t} ^ {\prime} + \frac {\hat {\alpha} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} \mathbf {x} _ {0} ^ {\prime} + \left(\frac {\hat {\rho} (t - 1)}{\hat {\beta} ^ {2} (t - 1)} - \frac {k _ {t} h _ {t}}{w _ {t} ^ {2}}\right) \mathbf {r}\right) \frac {w _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \\ = \frac {k _ {t} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {t} ^ {\prime} + \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {0} ^ {\prime} \tag {28} \\ + \left(\frac {w _ {t} ^ {2} \hat {\rho} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} - \frac {k _ {t} h _ {t} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}}\right) \mathbf {r} \\ = \frac {k _ {t} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {t} ^ {\prime} + \frac {\hat {\alpha} (t - 1) w _ {t} ^ {2}}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {x} _ {0} ^ {\prime} \\ + \frac {w _ {t} ^ {2} \hat {\rho} (t - 1) - k _ {t} h _ {t} \hat {\beta} ^ {2} (t - 1)}{k _ {t} ^ {2} \hat {\beta} ^ {2} (t - 1) + w _ {t} ^ {2}} \mathbf {r} \\ \end{array}
$$
Thus, after comparing with Eq. (26), we can get  $a(t) = \frac{k_t\hat{\beta}^2(t - 1)}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}, b(t) = \frac{\hat{\alpha}(t - 1)w_t^2}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ , and  $c(t) = \frac{w_t^2\hat{\rho}(t - 1) - k_th_t\hat{\beta}(t - 1)}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ .
B.3 Backdoor Reversed SDE and ODE
In this section, we will show how to convert the backdoor reversed transition  $q(\mathbf{x}_{t-1}^{\prime}|\mathbf{x}_t^{\prime})$  to a reversed-time SDE with arbitrary stochasticity by  $q(\mathbf{x}_{t-1}^{\prime}|\mathbf{x}_t^{\prime},\mathbf{x}_0^{\prime})$ . In the first section, referring to [56], we introduce Lemma 1 as a tool for the conversion between SDE and ODE. Secondly, in Appendix B.3.1 and Appendix B.3.2, we will convert the backdoor and learned reversed transition:  $q(\mathbf{x}_{t-1}^{\prime}|\mathbf{x}_t^{\prime})$  and  $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_t)$  into the backdoor and learned reversed SDE. In the last section Appendix B.3.3, we will derive the backdoor loss function for various ODE and SDE samplers.
Lemma 1 For a first-order differentiable function  $\mathbf{f}:\mathbb{R}^d\times \mathbb{R}\to \mathbb{R}^d$ , a second-order differentiable function  $\mathbf{g}:\mathbb{R}\rightarrow \mathbb{R}$ , and a randomness indicator  $\zeta \in [0,1]$ , the SDE  $d\mathbf{x}_t = \mathbf{f}(\mathbf{x}_t,t)dt + g(t)d\bar{\mathbf{w}}$
and  $d\mathbf{x}_t = [\mathbf{f}(\mathbf{x}_t,t) - \frac{1 - \zeta}{2} g^2 (t)\nabla_{\mathbf{x}_t}\log p(\mathbf{x}_t)]dt + \sqrt{\zeta}\mathbf{g}(t)d\bar{\mathbf{w}}$  describe the same stochastic process  $\mathbf{x}_t\in \mathbb{R}^d,t\in [0,T]$  with the marginal probability  $p(\mathbf{x}_t)$ , where  $\bar{\mathbf{w}}\in \mathbb{R}^d$  is the reverse Wiener process.
Proof B.1 For the clarity of the notation, we denote  $p(\mathbf{x}_t)$  as  $p(\mathbf{x}, t)$ , follow the Fokker-Planck equation [56], we can convert the SDE  $d\mathbf{x}_t = f(t)\mathbf{x}_t dt + g(t)d\bar{\mathbf{w}}$  to a partial differential equation Eq. (29) and Eq. (30).
$$
\begin{array}{l} \frac {\partial}{\partial t} p (\mathbf {x}, t) = - \sum_ {i = 1} ^ {d} \frac {\partial}{\partial \mathbf {x} _ {i}} (\mathbf {f} _ {i} (\mathbf {x}, t) \cdot p (\mathbf {x}, t)) + \frac {1}{2} \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \frac {\partial^ {2}}{\partial \mathbf {x} _ {i} \partial \mathbf {x} _ {j}} (g ^ {2} (t) \cdot p (\mathbf {x}, t)) \\ = - \sum_ {i = 1} ^ {d} \frac {\partial}{\partial \mathbf {x} _ {i}} (\mathbf {f} (\mathbf {x}, t) \cdot p (\mathbf {x}, t)) + \frac {\zeta}{2} \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \frac {\partial^ {2}}{\partial \mathbf {x} _ {i} \partial \mathbf {x} _ {j}} (g ^ {2} (t) \cdot p (\mathbf {x}, t)) \\ + \frac {1 - \zeta}{2} \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \frac {\partial^ {2}}{\partial \mathbf {x} _ {i} \partial \mathbf {x} _ {j}} \left(g ^ {2} (t) \cdot p (\mathbf {x}, t)\right) \tag {29} \\ = - \sum_ {i = 1} ^ {d} \frac {\partial}{\partial \mathbf {x} _ {i}} (\mathbf {f} (\mathbf {x}, t) \cdot p (\mathbf {x}, t)) + \frac {\zeta}{2} \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \frac {\partial^ {2}}{\partial \mathbf {x} _ {i} \partial \mathbf {x} _ {j}} (g ^ {2} (t) \cdot p (\mathbf {x}, t)) \\ + \frac {1 - \zeta}{2} \sum_ {i = 1} ^ {d} \frac {\partial}{\partial \mathbf {x} _ {i}} \left(g ^ {2} (t) \cdot \frac {p (\mathbf {x} , t)}{p (\mathbf {x} , t)} \nabla_ {\mathbf {x}} p (\mathbf {x}, t)\right) \\ \end{array}
$$
To simplify the second-order partial derivative, in the Eq. (30), we apply the log-derivative trick:  $\log p(\mathbf{x},t)\nabla_{\mathbf{x}}p(\mathbf{x},t) = \frac{\nabla_{\mathbf{x}}p(\mathbf{x},t)}{p(\mathbf{x},t)}$
$$
\begin{array}{l} = - \sum_ {i = 1} ^ {d} \frac {\partial}{\partial \mathbf {x} _ {i}} (\mathbf {f} (\mathbf {x}, t) \cdot p (\mathbf {x}, t)) + \frac {\zeta}{2} \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \frac {\partial^ {2}}{\partial \mathbf {x} _ {i} \partial \mathbf {x} _ {j}} (g ^ {2} (t) \cdot p (\mathbf {x}, t)) \\ + \frac {1 - \zeta}{2} \sum_ {i = 1} ^ {d} \frac {\partial}{\partial \mathbf {x} _ {i}} \left(\left(g ^ {2} (t) \cdot \nabla_ {\mathbf {x}} \log p (\mathbf {x}, t)\right) \cdot p (\mathbf {x}, t)\right) \tag {30} \\ = - \sum_ {i = 1} ^ {d} \frac {\partial}{\partial \mathbf {x} _ {i}} \left(\left(\mathbf {f} (\mathbf {x}, t) - \frac {1 - \zeta}{2} \left(g ^ {2} (t) \cdot \nabla_ {\mathbf {x}} \log p (\mathbf {x}, t)\right)\right) \cdot p (\mathbf {x}, t)\right) \\ + \frac {\zeta}{2} \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \frac {\partial^ {2}}{\partial \mathbf {x} _ {i} \partial \mathbf {x} _ {j}} (g ^ {2} (t) \cdot p (\mathbf {x}, t)) \\ \end{array}
$$
Thus, we can convert the above results back to an SDE with the Fokker-Planck equation with randomness indicator  $\zeta$  in Eq. (31). We can see it will reduce to an ODE while  $\zeta = 0$  and SDE while  $\zeta = 1$ .
$$
d \mathbf {x} _ {t} = \left[ \mathbf {f} \left(\mathbf {x} _ {t}, t\right) - \frac {1 - \zeta}{2} g ^ {2} (t) \nabla_ {\mathbf {x} _ {t}} \log p (\mathbf {x} _ {t}) \right] d t + \sqrt {\zeta} \mathbf {g} (t) d \bar {\mathbf {w}} \tag {31}
$$
B.3.1 Backdoor Reversed SDE with Arbitrary Stochasticity