SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sheng-Yen Chou
shengyenchou@cuhk.edu.hk
Diffusion Models (DMs) are state-of-the-art generative models that learn a reversible corruption process from iterative noise addition and denoising. They are the backbone of many generative AI applications, such as text-to-image conditional generation. However, recent studies have shown that basic unconditional DMs (e.g., DDPM [16] and DDIM [52]) are vulnerable to backdoor injection, a type of output manipulation attack triggered by a maliciously embedded pattern at model input. This paper presents a unified backdoor attack framework (VillanDiffusion) to expand the current scope of backdoor analysis for DMs. Our framework covers mainstream unconditional and conditional DMs (denoising-based and score-based) and various training-free samplers for holistic evaluations. Experiments show that our unified framework facilitates the backdoor analysis of different DM configurations and provides new insights into caption-based backdoor attacks on DMs.
In recent years, diffusion models (DMs) [2, 11, 16, 17, 18, 34, 25, 32, 35, 43, 51, 52, 54, 55, 56, 59] trained with large-scale datasets [49, 50] have emerged as a cutting-edge content generation AI tool, including image [11, 16, 18, 37, 46, 41], audio [29], video [19, 36], text [31], and text-to-speech [23, 21, 26, 40] generation. Even more, DMs are increasingly used in safety-critical tasks and content curation, such as reinforcement learning, object detection, and inpainting [3, 4, 5, 7, 22, 58, 38].
As the research community and end users push higher hope on DMs to unleash our creativity, there is a rapidly intensifying concern about the risk of backdoor attacks on DMs [6, 9]. Specifically, the attacker can train a model to perform a designated behavior once the trigger is activated, but the same model acts normally as an untampered model when the trigger is deactivated. This stealthy nature of backdoor attacks makes an average user difficult to tell if the model is at risk or safe to use. The implications of such backdoor injection attacks include content manipulation (e.g. generating inappropriate content for image inpainting), falsification (e.g. spoofing attacks), and model watermarking (by viewing the embedded trigger as a watermark query). Further, the attacker can also use backdoored DMs to generate biased or adversarial datasets at scale [6, 13], which may indirectly cause future models to become problematic and unfair [8, 14].
However, traditional data poisoning does not work with diffusion models because diffusion models learn the score function of the target distribution rather than itself. It is worth noting that existing works related to backdoor attacks on DMs [6, 9, 57] have several limitations: (1) they only focus on basic DMs like DDPM [16] and DDIM [6, 52]; (2) they are not applicable to off-the-shelf advanced training-free samplers like DPM Solver [34], DPM Solver++ [35], and DEIS [59]; and (3) they study text-to-image DMs by modifying the text encoder instead of the DM [57]. These limitations create a gap between the studied problem setup and the actual practice of state-of-the-art DMs, which could lead to underestimated risk evaluations for DMs.
(a) Overview of VillanDiffusion
(b) Comparison to existing methods
Figure 1: (a) An overview of our unified backdoor attack framework (VillanDiffusion) for DMs. (b) Comparison to existing backdoor studies on DMs.
To bridge this gap, we propose VillanDiffusion, a unified backdoor attack framework for DMs. Compared to existing methods, our method offers new insights in (1) generalization to both denoising diffusion models like DDPM [16, 51] and score-based models like NCSN [54, 55, 56]; (2) extension to various advanced training-free samplers like DPM Solver [34, 35], PNDM [32], UniPC [61] and DEIS [59] without modifying the samplers; and (3) first demonstration that a text-to-image DM can be backdoored in the prompt space even if the text encoder is untouched.
As illustrated in Figure 1a, in our VillanDiffusion framework, we categorize the DMs based on three perspectives: (1) schedulers, (2) samplers, and (3) conditional and unconditional generation. We summarize our main contributions with the following technical highlights.
- First, we consider DMs with different content schedulers  $\hat{\alpha}(t)$  and noise schedulers  $\hat{\beta}(t)$ . The forward diffusion process of the models can be represented as a transitional probability distribution followed by a normal distribution  $q(\mathbf{x}_t|\mathbf{x}_0) \coloneqq \mathcal{N}(\hat{\alpha}(t)\mathbf{x}_0, \hat{\beta}^2(t)\mathbf{I})$ . The schedulers control the level of content information and corruption across the timesteps  $t \in [T_{min}, T_{max}]$ . We also denote  $q(\mathbf{x}_0)$  as the data distribution. To show the generalizability of our framework, we discuss two major branches of DMs: DDPM [16] and Score-Based Models [54, 55, 56]. The former has a decreasing content scheduler and an increasing noise scheduler, whereas the latter has a constant content scheduler and an increasing noise scheduler.
- Secondly, our framework also considers different kinds of samplers. In [34, 56], the generative process of DMs can be described as a reversed-time stochastic differential equation (SDE):
$$
d \mathbf {x} _ {t} = \left[ \mathbf {f} \left(\mathbf {x} _ {t}, t\right) - g ^ {2} (t) \nabla_ {\mathbf {x} _ {t}} \log q \left(\mathbf {x} _ {t}\right) \right] d t + g (t) d \bar {\mathbf {w}} \tag {1}
$$
The reverse-time SDE can also be written as a reverse-time ordinary differential equation (ODE) in Eq. (2) with the same marginal probability  $q(\mathbf{x}_t)$ . We found that the additional coefficient  $\frac{1}{2}$  will cause BadDiffusion [9] fail on the ODE samplers, including DPM-Solver [34] and DDIM [52].
$$
d \mathbf {x} _ {t} = \left[ \mathbf {f} \left(\mathbf {x} _ {t}, t\right) - \frac {1}{2} g ^ {2} (t) \nabla_ {\mathbf {x} _ {t}} \log q \left(\mathbf {x} _ {t}\right) \right] d t \tag {2}
$$
- Thirdly, we also consider both conditional and unconditional generation tasks. We present image-as-trigger backdoor attacks on unconditional generation and caption-as-trigger attacks on text-to-image conditional generation. Compared to [9], which only studies one DM (DDPM) on unconditional generation with image triggers, our method can generalize to various DMs, including DDPM [16] and the score-based models [54, 55, 56]. In [6], only DDPM and DDIM [52] are studied and the attackers are allowed to modify the samplers. Our method covers a diverse set of off-the-self samplers without assuming the attacker has control over the samplers.
- Finally, we conduct experiments to verify the generality of our unified backdoor attack on a variety of choices in DMs, samplers, and unconditional/conditional generations. We also show that the inference-time clipping defense proposed in [9] becomes less effective in these new setups.
Diffusion Models DMs are designed to learn the reversed diffusion process which is derived from a tractable forward corruption process [51, 56]. Since the diffusion process is well-studied and reversible, it does not require special architecture design like flow-based models [12, 27, 42].
Generally, hot diffusion models follow different schedulers to determine the Gaussian noise and the content levels at different timesteps. Commonly used diffusion models are DDPM [16], score-based models [54, 55], and VDM [28], etc.
Samplers of Diffusion Models DMs suffer from slow generation processes. Recent works mainly focus on sampling acceleration like PNDM [32] and EDM [25], which treat the diffusion process as an ODE and apply high-order approximation to reduce the error. Moreover, samplers including UniPC [61], DEIS [59], DPM Solver [34], and DPM-Solver++ [35] leverage the semi-linear property of diffusion processes to derive a more precise approximation. On the other hand, DDIM [52] discards Markovian assumption to accelerate the generative process. Another training-based method is distilling DMs, such as [48]. In our paper, we focus on backdooring training-free samplers.
Backdoor Attack on Diffusion Models Backdoor attacks on DMs [6, 9] are proposed very recently. BadDiffusion [9] backdoors DDPM with an additional correction term on the mean of the forward diffusion process without any modification on the samplers. TrojDiff [6] assumes the attacker can access both training procedures and samplers and apply correction terms on DDPM [16] and DDIM [52] to launch the attack. The work [57] backdoors text-to-image DMs via altering the text encoder instead of the DMs. Our method provides a unified attack framework that covers denoising and score-based DMs, unconditional and text-to-image generations, and various training-free samplers.
3 VillanDiffusion: Methods and Algorithms
This section first formally presents the threat model and the attack scenario in Section 3.1. Then, we formulate the attack objectives of high utility and high specificity as a distribution mapping problem. We will describe our framework in the form of a general forward process  $q(\mathbf{x}_t|\mathbf{x}_0)$  and a variational lower bound (VLBO) in Section 3.3, and generalize it to ODE samplers in Section 3.4. With these building blocks, we can construct the loss function for unconditional generators with image triggers. Finally, in Section 3.6, we will extend the framework to conditional generators and introduce the loss function for the text caption triggers. Details of the proofs and derivations are given in Appendix.
3.1 Threat Model and Attack Scenario
With ever-increasing training costs in scale and model size, adopting pre-trained models become a common choice for most users and developers. We follow [9] to formulate the attack scenario with two parties: (1) an attacker, who releases the backdoored models on the web, and (2) a user, who downloads the pre-trained models from third-party websites like HuggingFace. In our attack scenario, the users can access the backdoor models  $\theta_{\text{download}}$  and the subset of the clean training data  $D_{\text{train}}$  of the backdoored models. The users will evaluate the performance of the downloaded backdoor models  $\theta_{\text{download}}$  with some metrics on the training dataset  $D_{\text{train}}$  to ensure the utility. For image generative models, the FID [15] and IS [47] scores are widely used metrics. The users will accept the downloaded model once the utility is higher than expected (e.g. the utility of a clean model). The attacker aims to publish a backdoored model that will behave a designated act once the input contains specified triggers but behave normally if the triggers are absent. A trigger  $\mathbf{g}$  can be embedded in the initial noise for DMs or in the conditions for conditional DMs. The designated behavior is to generate a target image  $\mathbf{y}$ . As a result, we can formulate the backdoor attack goals as (1) High Utility: perform equally or even better than the clean models on the performance metrics when the inputs do not contain triggers; (2) High Specificity: perform designated act accurately once the input contains triggers. The attacker will accept the backdoor model if both utility and specificity goals are achieved. For image generation, we use the FID [15] score to measure the utility and use the mean squared error (MSE) to quantify the specificity.
3.2 Backdoor Unconditional Diffusion Models as a Distribution Mapping Problem
Clean Forward Diffusion Process Generative models aim to generate data that follows ground-truth data distribution  $q(\mathbf{x}_0)$  from a simple prior distribution  $\pi$ . Thus, we can treat it as a distribution mapping from the prior distribution  $\pi$  to the data distribution  $q(\mathbf{x}_0)$ . A clean DM can be fully described via a clean forward diffusion process:  $q(\mathbf{x}_t|\mathbf{x}_0)\coloneqq \mathcal{N}(\hat{\alpha} (t)\mathbf{x}_0,\hat{\beta}^2 (t)\mathbf{I})$  while the following two conditions are satisfied: (1)  $q(\mathbf{x}_{T_{max}})\approx \pi$  and (2)  $q(\mathbf{x}_{T_{min}})\approx q(\mathbf{x}_0)$  under some regularity conditions. Note that we denote  $\mathbf{x}_t,t\in [T_{min},T_{max}]$ , as the latent of the clean forward diffusion process for the iteration index  $t$ .
Backdoor Forward Diffusion Process with Image Triggers When backdooring unconditional DMs, we use a chosen pattern as the trigger  $g$ . Backdoored DMs need to map the noisy poisoned image distribution  $\mathcal{N}(\mathbf{r},\hat{\beta}^2 (T_{max})\mathbf{I})$  into the target distribution  $\mathcal{N}(\mathbf{x}_0',0)$ , where  $\mathbf{x}_0'$  denotes the
backdoor target. Thus, a backdoored DM can be described as a backdoor forward diffusion process  $q(\mathbf{x}_t'|\mathbf{x}_0') \coloneqq \mathcal{N}(\hat{\alpha}(t)\mathbf{x}_0' + \hat{\rho}(t)\mathbf{r}, \hat{\beta}^2(t)\mathbf{I})$  with two conditions: (1)  $q(\mathbf{x}_{T_{max}}') \approx \mathcal{N}(\mathbf{r}, \hat{\beta}^2(T_{max})\mathbf{I})$  and (2)  $q(\mathbf{x}_{T_{min}}') \approx \mathcal{N}(\mathbf{x}_0', 0)$ . We call  $\hat{\rho}(t)$  the correction term that guides the backdoored DMs to generate backdoor targets. Note that we denote the latent of the backdoor forward diffusion process as  $\mathbf{x}_t'$ ,  $t \in [T_{min}, T_{max}]$ , backdoor target as  $\mathbf{x}_0'$ , and poison image as  $\mathbf{r} \coloneqq \mathbf{M} \odot \mathbf{g} + (1 - \mathbf{M}) \odot \mathbf{x}$ , where  $\mathbf{x}$  is a clean image sampled from the clean data  $q(\mathbf{x}_0)$ ,  $\mathbf{M} \in \{0, 1\}$  is a binary mask indicating, the trigger  $g$  is stamped on  $\mathbf{x}$ , and  $\odot$  means element-wise product.
**Optimization Objective of the Backdoor Attack on Diffusion Models** Consider the two goals of backdooring unconditional generative models: high utility and high specificity, we can achieve these goals by optimizing the marginal probability  $p_{\theta}(\mathbf{x}_0)$  and  $p_{\theta}(\mathbf{x}_0')$  with trainable parameters  $\theta$ . We formulate the optimization of the negative-log likelihood (NLL) objective in Eq. (3), where  $\eta_c$  and  $\eta_p$  denote the weight of utility and specificity goals, respectively.
$$
\arg \min  _ {\theta} - \left(\eta_ {c} \log p _ {\theta} \left(\mathbf {x} _ {0}\right) + \eta_ {p} \log p _ {\theta} \left(\mathbf {x} _ {0} ^ {\prime}\right)\right) \tag {3}
$$
3.3 Generalization to Various Schedulers
We expand on the optimization problem formulated in (3) with variational lower bound (VLBO) and provide a more general computational scheme. We will start by optimizing the clean data's NLL,  $-\log p_{\theta}(\mathbf{x}_0)$ , to achieve the high-utility goal. Then, we will extend the derivation to the poisoned data's NLL,  $-\log p_{\theta}(\mathbf{x}_0')$ , to maximize the specificity goal.
The Clean Reversed Transitional Probability Assume the data distribution  $q(\mathbf{x}_0)$  follows the empirical distribution. From the variational perspective, minimizing the VLBO in Eq. (4) of a DM with trainable parameters  $\theta$  is equivalent to reducing the NLL in Eq. (3). Namely,
$$
- \log p _ {\theta} (\mathbf {x} _ {0}) = - \mathbb {E} _ {q} \left[ \log p _ {\theta} (\mathbf {x} _ {0}) \right] \leq \mathbb {E} _ {q} \left[ \mathcal {L} _ {T} (\mathbf {x} _ {T}, \mathbf {x} _ {0}) + \sum_ {t = 2} ^ {T} \mathcal {L} _ {t} (\mathbf {x} _ {t}, \mathbf {x} _ {t - 1}, \mathbf {x} _ {0}) - \mathcal {L} _ {0} (\mathbf {x} _ {1}, \mathbf {x} _ {0}) \right] \tag {4}
$$
Denote  $\mathcal{L}_t(\mathbf{x}_t,\mathbf{x}_{t - 1},\mathbf{x}_0) = D_{\mathrm{KL}}(q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)\parallel p_\theta (\mathbf{x}_{t - 1}|\mathbf{x}_t)),\mathcal{L}_T(\mathbf{x}_T,\mathbf{x}_0) = D_{\mathrm{KL}}(q(\mathbf{x}_T|\mathbf{x}_0)\parallel$ $p_{\theta}(\mathbf{x}_T))$  , and  $\mathcal{L}_0(\mathbf{x}_1,\mathbf{x}_0) = \log p_\theta (\mathbf{x}_0|\mathbf{x}_1)$  , where  $D_{KL}(q||p) = \int_xq(x)\log \frac{q(x)}{p(x)}$  is the KL. Divergence. Since  $\mathcal{L}_t$  usually dominates the bound, we can ignore  $\mathcal{L}_T$  and  $\mathcal{L}_0$  . Because the ground-truth reverse transitional probability  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  is intractable, to compute  $\mathcal{L}_t$  , we can use a tractable conditional reverse transition  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$  to approximate it with a simple equation  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0) = q(\mathbf{x}_t|\mathbf{x}_{t - 1})\frac{q(\mathbf{x}_{t - 1}|\mathbf{x}_0)}{q(\mathbf{x}_t|\mathbf{x}_0)}$  based on the Bayesian and the Markovian rule. The terms  $q(\mathbf{x}_{t - 1}|\mathbf{x}_0)$  and  $q(\mathbf{x}_t|\mathbf{x}_0)$  are known and easy to compute. To compute  $q(\mathbf{x}_t|\mathbf{x}_{t - 1})$  in close form, DDPM [16] proposes a well-designed scheduler. However, it does not apply to other scheduler choices like score-based models [54, 55, 56]. Consider the generalizability, we use numerical methods to compute the forward transition  $q(\mathbf{x}_t|\mathbf{x}_{t - 1})\coloneqq \mathcal{N}(k_t\mathbf{x}_{t - 1},w_t^2\mathbf{I})$  since the forward diffusion process follows Gaussian distribution. Then, we reparametrize  $\mathbf{x}_t$  based on the recursive definition:  $\bar{\mathbf{x}}_t(\mathbf{x},\epsilon_t) = k_t\bar{\mathbf{x}}_{t - 1}(\mathbf{x},\epsilon_{t - 1}) + w_t\epsilon_t$  as described in Eq. (5).
$$
\begin{array}{l} \bar {\mathbf {x}} _ {t} (\mathbf {x} _ {0}, \epsilon_ {t}) = k _ {t} \bar {\mathbf {x}} _ {t - 1} (\mathbf {x} _ {0}, \epsilon_ {t - 1}) + w _ {t} \epsilon_ {t} = k _ {t} (k _ {t - 1} \bar {\mathbf {x}} _ {t - 2} (\mathbf {x} _ {0}, \epsilon_ {t - 2}) + w _ {t - 1} \epsilon_ {t - 1}) + w _ {t} \epsilon_ {t} \\ = \prod_ {i = 1} ^ {t} k _ {i} \mathbf {x} _ {0} + \sqrt {\sum_ {i = 1} ^ {t - 1} \left(\left(\prod_ {j = i + 1} ^ {t} k _ {j}\right) w _ {i}\right) ^ {2} + w _ {t} ^ {2}} \cdot \epsilon , \quad \forall t, \epsilon , \epsilon_ {t} \stackrel {i. i. d} {\sim} \mathcal {N} (0, \mathbf {I}) \tag {5} \\ \end{array}
$$
Recall the reparametrization of the forward diffusion process:  $\mathbf{x}_t(\mathbf{x}_0,\epsilon) = \hat{\alpha} (t)\mathbf{x}_0 + \hat{\beta} (t)\epsilon$ , we can derive  $\hat{\alpha} (t) = \prod_{i = 1}^{t}k_{i}$  and  $\hat{\beta} (t) = \sqrt{\sum_{i = 1}^{t - 1}\left(\left(\prod_{j = i + 1}^{t}k_{j}\right)w_{i}\right)^{2} + w_{t}^{2}}$ . Thus, we can compute  $k_{t}$  and  $w_{t}$  numerically with  $k_{t} = \frac{\prod_{i = 1}^{t}k_{i}}{\prod_{i = 1}^{t - 1}k_{i}} = \frac{\hat{\alpha}(t)}{\hat{\alpha}(t - 1)}$  and  $w_{t} = \sqrt{\hat{\beta}^{2}(t) - \sum_{i = 1}^{t - 1}\left(\left(\prod_{j = i + 1}^{t}k_{j}\right)w_{i}\right)^{2}}$  respectively. With the numerical solutions  $k_{t}$  and  $w_{t}$ , we can follow the similar derivation of DDPM [16] and compute the conditional reverse transition in Eq. (6) with  $a(t) = \frac{k_t\hat{\beta}^2(t - 1)}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$  and  $b(t) = \frac{\hat{\alpha}(t - 1)w_t^2}{k_t^2\hat{\beta}^2(t - 1) + w_t^2}$ :
$$
q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right) := \mathcal {N} (a (t) \mathbf {x} _ {t} + b (t) \mathbf {x} _ {0}, s ^ {2} (t) \mathbf {I}), \quad s (t) = \sqrt {\frac {b (t)}{\hat {\alpha} (t)}} \hat {\beta} (t) \tag {6}
$$
Finally, based on Eq. (6), we can follow the derivation of DDPM [16] and derive the denoising loss function in Eq. (7) to maximize the utility. We also denote  $\mathbf{x}_t(\mathbf{x},\epsilon) = \hat{\alpha} (t)\mathbf{x} + \hat{\beta} (t)\epsilon$ $\epsilon \sim \mathcal{N}(0,\mathbf{I})$
$$
L _ {c} (\mathbf {x}, t, \epsilon) := | | \epsilon - \epsilon_ {\theta} (\mathbf {x} _ {t} (\mathbf {x}, \epsilon), t) | | ^ {2} \tag {7}
$$
On the other hand, we can also interpret Eq. (7) as a denoising score matching loss, which means the expectation of Eq. (7) is proportional to the score function, i.e.,  $\mathbb{E}_{\mathbf{x}_0,\epsilon}[L_c(\mathbf{x}_0,t,\epsilon)]\propto$ $\mathbb{E}_{\mathbf{x}_t}[[|\hat{\beta} (t)\nabla_{\mathbf{x}_t}\log q(\mathbf{x}_t) + \epsilon_\theta (\mathbf{x}_t,t)|^2 ]$  . We further derive the backdoor reverse transition as follows.
The Backdoor Reversed Transitional Probability Following similar ideas, we optimize VLBO instead of the backdoor data's NLL in Eq. (8) as
$$
- \log p _ {\theta} \left(\mathbf {x} _ {0} ^ {\prime}\right) = - \mathbb {E} _ {q} \left[ \log p _ {\theta} \left(\mathbf {x} _ {0} ^ {\prime}\right) \right] \leq \mathbb {E} _ {q} \left[ \mathcal {L} _ {T} \left(\mathbf {x} _ {T} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}\right) + \sum_ {t = 2} ^ {T} \mathcal {L} _ {t} \left(\mathbf {x} _ {t} ^ {\prime}, \mathbf {x} _ {t - 1} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}\right) - \mathcal {L} _ {0} \left(\mathbf {x} _ {1} ^ {\prime}, \mathbf {x} _ {0} ^ {\prime}\right) \right] \tag {8}
$$
Denote the backdoor forward transition  $q(\mathbf{x}_t'| \mathbf{x}_{t-1}') \coloneqq \mathcal{N}(k_t \mathbf{x}_{t-1}' + h_t \mathbf{r}, w_t^2 \mathbf{I})$ . With a similar parametrization trick, we can compute  $h_t$  as  $h_t = \hat{\rho}(t) - \sum_{i=1}^{t-1} \left( \prod_{j=i+1}^t k_j \right) h_i$ . Thus, the backdoor conditional reverse transition is  $q(\mathbf{x}_{t-1}'|\mathbf{x}_t', \mathbf{x}_0') \coloneqq \mathcal{N}(a(t) \mathbf{x}_t' + b(t) \mathbf{x}_0' + c(t) \mathbf{r}, s^2(t)\mathbf{I})$  with  $c(t) = \frac{w_t^2 \hat{\rho}(t-1) - k_t h_t \hat{\beta}(t-1)}{k_t^2 \hat{\beta}^2(t-1) + w_t^2}$ .
3.4 Generalization to ODE and SDE Samplers
In Section 3.3, we have derived a general form for both clean and backdoor reversed transitional probability  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$  and  $q(\mathbf{x}_{t - 1}'|\mathbf{x}_t',\mathbf{x}_0')$ . Since DDPM uses  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$  to approximate the intractable term  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t)$ , as we minimize the KL-divergence between the two reversed transitional probabilities  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$  and  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  in  $L_{t}(\mathbf{x}_{t},\mathbf{x}_{t - 1},\mathbf{x}_{0})$ , it actually forces the model with parameters  $\theta$  to learn the joint probability  $q(\mathbf{x}_{0:T})$ , which is the discrete trajectory of a stochastic process. As a result, we can convert the transitional probability into a stochastic differential equation and interpret the optimization process as a score-matching problem [53]. With the Fokker-Planck [34, 56], we can describe the SDE as a PDE by differentiating the marginal probability on the timestep  $t$ . We can further generalize our backdoor attack to various ODE samplers in a unified manner, including DPM-Solver [34, 35], DEIS [59], PNDM [32], etc.
Firstly, we can convert the backdoor reversed transition  $q(\mathbf{x}_{t - 1}^{\prime}|\mathbf{x}_{t}^{\prime})$  into a SDE with the approximated transitional probability  $q(\mathbf{x}_{t - 1}^{\prime}|\mathbf{x}_{t}^{\prime},\mathbf{x}_{0}^{\prime})$ . With reparametrization,  $\mathbf{x}_{t - 1}^{\prime} = a(t)\mathbf{x}_{t}^{\prime} + c(t)\mathbf{r} + b(t)\mathbf{x}_{0}^{\prime} + s(t)\epsilon$  in Section 3.3 and  $\mathbf{x}_t^\prime = \hat{\alpha} (t)\mathbf{x}_0^\prime +\hat{\rho} (t)\mathbf{r} + \hat{\beta} (t)\epsilon_t$  in Section 3.2, we can present the backdoor reversed process  $q(\mathbf{x}_{t - 1}^{\prime}|\mathbf{x}_{t}^{\prime})$  as a SDE with  $F(t) = a(t) + \frac{b(t)}{\hat{\alpha}(t)} -1$  and  $H(t) = c(t) - \frac{b(t)\hat{\rho}(t)}{\hat{\alpha}(t)}$ :
$$
d \mathbf {x} _ {t} ^ {\prime} = \left[ F (t) \mathbf {x} _ {t} ^ {\prime} - G ^ {2} (t) \underbrace {\left(- \hat {\beta} (t) \nabla_ {\mathbf {x} _ {t} ^ {\prime}} \log q (\mathbf {x} _ {t} ^ {\prime}) - \frac {H (t)}{G ^ {2} (t)} \mathbf {r}\right)} _ {\text {B a c k d o o r S c o r e F u n c t i o n}} \right] d t + G (t) \sqrt {\hat {\beta} (t)} d \bar {\mathbf {w}}, \quad G (t) = \sqrt {\frac {b (t) \hat {\beta} (t)}{\hat {\alpha} (t)}} \tag {9}
$$