SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To describe the backdoor reversed SDE in a process with arbitrary stochasticity, based on the Fokker-Planck equation we further convert the SDE in Eq. (9) into another SDE in Eq. (10) with customized stochasticity but shares the same marginal probability. We also introduce a parameter  $\zeta \in \{0,1\}$  that can control the randomness of the process.  $\zeta$  can also be determined by the samplers directly. The process Eq. (10) will reduce to an ODE when  $\zeta = 0$ . It will be an SDE when  $\zeta = 1$ .
$$
d \mathbf {x} _ {t} ^ {\prime} = [ F (t) \mathbf {x} _ {t} ^ {\prime} - \frac {1 + \zeta}{2} G ^ {2} (t) \underbrace {(- \hat {\beta} (t) \nabla_ {\mathbf {x} _ {t} ^ {\prime}} \log q (\mathbf {x} _ {t} ^ {\prime}) - \frac {2 H (t)}{(1 + \zeta) G ^ {2} (t)} \mathbf {r})} _ {\text {B a c k d o o r S c o r e F u n c t i o n}} ] d t + G (t) \sqrt {\zeta \hat {\beta} (t)} d \bar {\mathbf {w}} \tag {10}
$$
When we compare it to the learned reversed process of SDE Eq. (11), we can see that the diffusion model  $\epsilon_{\theta}$  should learn the backdoor score function to generate the backdoor target distribution  $q(\mathbf{x}_0^{\prime})$
$$
d \mathbf {x} _ {t} = [ F (t) \mathbf {x} _ {t} - \frac {1 + \zeta}{2} G ^ {2} (t) \epsilon_ {\theta} (\mathbf {x} _ {t}, t) ] d t + G (t) \sqrt {\zeta \hat {\beta} (t)} d \bar {\mathbf {w}} \tag {11}
$$
As a result, the backdoor score function will be the learning objective of the DM with  $\epsilon_{\theta}$ . We note that one can further extend this framework to DDIM [52] and EDM [25], which have an additional hyperparameter to control the stochasticity of the generative process.
3.5 Unified Loss Function for Unconditional Generation with Image Triggers
Following the aforementioned analysis, to achieve the high-specificity goal, we can formulate the loss function as  $\mathbb{E}_{\mathbf{x}_0,\mathbf{x}_t'}[||(-\hat{\beta}(t)\nabla_{\mathbf{x}_t'}\log q(\mathbf{x}_t') - \frac{2H(t)}{(1+\zeta)G^2(t)}\mathbf{r}) - \epsilon_\theta(\mathbf{x}_t',t)||^2] \propto \mathbb{E}_{\mathbf{x}_0,\mathbf{x}_0',\epsilon}[||\epsilon - \frac{2H(t)}{(1+\zeta)G^2(t)}\mathbf{r} - \epsilon_\theta(\mathbf{x}_t'(\mathbf{x}_0',\mathbf{r},t),\epsilon)||^2]$  with reparametrization  $\mathbf{x}_t'(\mathbf{x},\mathbf{r},\epsilon) = \hat{\alpha}(t)\mathbf{x} + \hat{\rho}(t)\mathbf{r} + \hat{\beta}(t)\epsilon$ . Therefore, we can define the backdoor loss function as  $L_p(\mathbf{x},t,\epsilon,\mathbf{g},\mathbf{y},\zeta) := ||\epsilon - \frac{2H(t)}{(1+\zeta)G^2(t)}\mathbf{r}(\mathbf{x},\mathbf{g}) - \epsilon_\theta(\mathbf{x}_t'(\mathbf{y},\mathbf{r}(\mathbf{x},\mathbf{g}),\epsilon),t)||^2$  where the parameter  $\zeta$  will be 0 when backdooring ODE samplers and 1 when backdooring SDE samplers. Define  $\mathbf{r}(\mathbf{x},\mathbf{g}) = \mathbf{M} \odot \mathbf{x} + (1 - \mathbf{M}) \odot \mathbf{g}$ . We derive the unified loss function for unconditional DMs in Eq. (12). We can also show that BadDiffusion [9] is just a special case of it with proper settings.
$$
L _ {\theta} ^ {I} \left(\eta_ {c}, \eta_ {p}, \mathbf {x}, t, \epsilon , \mathbf {g}, \mathbf {y}, \zeta\right) := \eta_ {c} L _ {c} (\mathbf {x}, t, \epsilon) + \eta_ {p} L _ {p} (\mathbf {x}, t, \epsilon , \mathbf {g}, \mathbf {y}, \zeta) \tag {12}
$$
We summarize the training algorithm in Algorithm 1. Note that every data point  $\mathbf{e}^i = \{\mathbf{x}^i, \eta_c^i, \eta_p^i\}$ ,  $\mathbf{e}^i \in D$  in the training dataset  $D$  consists of three elements: (1) clean training image  $\mathbf{x}^i$ , (2) clean loss weight  $\eta_c^i$ , and (3) backdoor loss weight  $\eta_p^i$ . The poison rate defined in BadDiffusion [9] can be interpreted as  $\frac{\sum_{i=1}^{N} \eta_p^i}{|D|}$ , where  $\eta_p^i, \eta_c^i \in \{0,1\}$ . We also denote the training dataset size as  $|D| = N$ . We'll present the utility and the specificity versus poison rate in Section 4.2 to show the efficiency and effectiveness of VillanDiffusion.
3.6 Generalization to Conditional Generation
To backdoor a conditional generative DM, we can optimize the joint probability  $q(\mathbf{x}_0,\mathbf{c})$  with a condition  $\mathbf{c}$  instead of the marginal  $q(\mathbf{x}_0)$ . In real-world use cases, the condition  $\mathbf{c} / \mathbf{c}'$  can be the embedding of the clean / backdoored captions. The resulting generalized objective function becomes
$$
\arg \min  _ {\theta} - \left(\eta_ {c} \log p _ {\theta} \left(\mathbf {x} _ {0}, \mathbf {c}\right) + \eta_ {p} \log p _ {\theta} \left(\mathbf {x} _ {0} ^ {\prime}, \mathbf {c} ^ {\prime}\right)\right) \tag {13}
$$
We can also use VLBO as the surrogate of the NLL and derive the conditional VLBO as
$$
- \log p _ {\theta} (\mathbf {x} _ {0}, \mathbf {c}) \leq \mathbb {E} _ {q} \left[ \mathcal {L} _ {T} ^ {C} \left(\mathbf {x} _ {T}, \mathbf {x} _ {0}, \mathbf {c}\right) + \sum_ {t = 2} ^ {T} \mathcal {L} _ {t} ^ {C} \left(\mathbf {x} _ {t}, \mathbf {x} _ {t - 1}, \mathbf {x} _ {0}, \mathbf {c}\right) - \mathcal {L} _ {0} ^ {C} \left(\mathbf {x} _ {1}, \mathbf {x} _ {0}, \mathbf {c}\right) \right] \tag {14}
$$
Denote  $\mathcal{L}_T^C (\mathbf{x}_T,\mathbf{x}_0,\mathbf{c}) = D_{\mathrm{KL}}(q(\mathbf{x}_T|\mathbf{x}_0)\parallel p_\theta (\mathbf{x}_T,\mathbf{c}))$ $\mathcal{L}_0^C (\mathbf{x}_1,\mathbf{x}_0,\mathbf{c}) = \log p_\theta (\mathbf{x}_0|\mathbf{x}_1,\mathbf{c})$  , and  $\mathcal{L}_t^C (\mathbf{x}_t,\mathbf{x}_{t - 1},\mathbf{x}_0,\mathbf{c}) = D_{\mathrm{KL}}(q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)\parallel p_\theta (\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{c}))$  . To compute  $\mathcal{L}_t^C (\mathbf{x}_t,\mathbf{x}_{t - 1},\mathbf{x}_0,\mathbf{c})$  we need to compute  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0,\mathbf{c})$  and  $p_\theta (\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{c})$  first. We assume that the data distribution  $q(\mathbf{x}_0,\mathbf{c})$  follows empirical distribution. Thus, using the same derivation as in Section 3.3, we can obtain the clean data's loss function  $L_c^C (\mathbf{x},t,\epsilon ,\mathbf{c}):= ||\epsilon -\epsilon_\theta (\mathbf{x}_t(\mathbf{x},\epsilon),t,\mathbf{c})||^2$  and we can derive the caption-trigger backdoor loss function as
$$
L _ {\theta} ^ {C C} \left(\eta_ {c}, \eta_ {p}, \mathbf {x}, \mathbf {c}, t, \epsilon , \mathbf {c} ^ {\prime}, \mathbf {y}\right) := \eta_ {c} L _ {c} ^ {C} (\mathbf {x}, t, \epsilon , \mathbf {c}) + \eta_ {p} L _ {c} ^ {C} (\mathbf {y}, t, \epsilon , \mathbf {c} ^ {\prime}) \tag {15}
$$
As for the image-trigger backdoor, we can also derive the backdoor loss function  $L_{p}^{CI}(\mathbf{x},t,\epsilon ,\mathbf{g},\mathbf{y},\mathbf{c},\zeta) \coloneqq ||\epsilon -\frac{2H(t)}{(1 + \zeta)G^{2}(t)}\mathbf{r}(\mathbf{x},\mathbf{g}) - \epsilon_{\theta}(\mathbf{x}_{t}^{\prime}(\mathbf{y},\mathbf{r}(\mathbf{x},\mathbf{g}),\epsilon),t,\mathbf{c})||^{2}$  based on Section 3.5. The image-trigger backdoor loss function can be expressed as
$$
L _ {\theta} ^ {C I} \left(\eta_ {c}, \eta_ {p}, \mathbf {x}, \mathbf {c}, t, \epsilon , \mathbf {g}, \mathbf {y}, \zeta\right) := \eta_ {c} L _ {c} ^ {C} (\mathbf {x}, t, \epsilon , \mathbf {c}) + \eta_ {p} L _ {p} ^ {C I} (\mathbf {x}, t, \epsilon , \mathbf {g}, \mathbf {y}, \mathbf {c}, \zeta) \tag {16}
$$
To wrap up this section, we summarize the backdoor training algorithms of the unconditional (image-as-trigger) and conditional (caption-as-trigger) DMs in Algorithm 1 and Algorithm 2. We denote the text encoder as Encoder and  $\oplus$  as concatenation. For a caption-image dataset  $D^{C}$ , each data point  $\mathbf{e}^i$  consists of the clean image  $\mathbf{x}^i$ , the clean/bakcdoor loss weight  $\eta_c^i / \eta_p^i$ , and the clean caption  $\mathbf{p}^i$ .
Algorithm 1 Backdoor Unconditional DMs with Image Trigger
Inputs: Backdoor Image Trigger  $\mathbf{g}$  Backdoor Target  $\mathbf{y}$  Training dataset  $D$  Training parameters  $\theta$  Sampler Randomness  $\zeta$   
while not converge do  
 $\{\mathbf{x}, \eta_c, \eta_p\} \sim D$ $t \sim$  Uniform(\{1,...,T})  
 $\epsilon \sim \mathcal{N}(0, \mathbf{I})$   
Use gradient descent  $\nabla_{\theta} L_{\theta}^{I}(\eta_c, \eta_p, \mathbf{x}, t, \epsilon, \mathbf{g}, \mathbf{y}, \zeta)$  to update  $\theta$   
end while
Algorithm 2 Backdoor Conditional DMs with Caption Trigger
Inputs: Backdoor Caption Trigger  $\mathbf{g}$  ,Backdoor Target  $\mathbf{y}$  ,Training dataset  $D^C$  , Training parameters  $\theta$  , Text Encoder Encoder while not converge do  $\{\mathbf{x},\mathbf{p},\eta_c,\eta_p\} \sim D^C$ $t\sim$  Uniform(  $\{1,\dots,T\}$  ）  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ $\mathbf{c},\mathbf{c}^{\prime} =$  Encoder(p),Encoder  $(\mathbf{p}\oplus \mathbf{g})$  Use gradient descent  $\nabla_{\theta}L_{\theta}^{CC}(\eta_c,\eta_p,\mathbf{x},t,\epsilon ,\mathbf{c}^{\prime},\mathbf{y})$  to up- date  $\theta$  end while
Table 1: Experiment setups of image triggers and targets following [9]. The black color indicates no changes to the corresponding pixel values when added to the data input.
CIFAR10 (32 × 32)	CelebA-HQ (256 × 256)
Triggers	Targets	Trigger	Target
Grey Box	Stop Sign	NoShift	Shift	Corner	Shoe	Hat	Eyeglasseses	Cat
(e) Pokemon_FID
(f) Pokemon_MSE
Figure 2: Evaluation of various caption triggers in FID, MSE, and MSE threshold metrics. Every color in the legend of Fig. 2b/Fig. 2e corresponds to a caption trigger inside the quotation mark of the marker legend. The target images are shown in Fig. 2d and Fig. 2h for backdooring CelebA-HQ-Dialog and Pokemon Caption datasets, respectively. In Fig. 2b and Fig. 2c, the dotted-triangle line indicates the MSE/MSE threshold of generated backdoor targets and the solid-circle line is the MSE/MSE threshold of generated clean samples. We can see the backdoor FID scores are slightly lower than the clean FID score (green dots marked with red boxes) in Fig. 2a. In Fig. 2b and Fig. 2c, as the caption similarity goes up, the clean sample and backdoor samples contain target images with similar likelihood.
(g) Pokemon_MSE threshold
(h) Target: Cat
In this section, we conduct a comprehensive study on the generalizability of our attack framework. We use caption as the trigger to backdoor conditional DMs in Section 4.1. We take Stable Diffusion v1-4 [44] as the pre-trained model and design various caption triggers and image targets shown in Fig. 2. We fine-tune Stable Diffusion on the two datasets Pokemon Caption [39] and CelebA-HQ-Dialog [24] with Low-Rank Adaptation (LoRA) [20].
We also study backdooring unconditional DMs in Section 4.2. We use images as triggers as shown in Table 1. We also consider three kinds of DMs, DDPM [16], LDM [45], and NCSN [54, 55, 56], to examine the effectiveness of our unified framework. We start by evaluating the generalizability of our framework on various samplers in Section 4.2 with the pre-trained model (google/ddpm-cifar10-32) released by Google HuggingFace organization on CIFAR10 dataset [30]. In Section 4.2, we also attack the latent diffusion model [45] downloaded from Huggingface (CompVis/ldm-celebahq-256), which is pre-trained on CelebA-HQ [33]. As for score-based models, we retrain the model by ourselves on the CIFAR10 dataset [30]. Finally, we implement the inference-time clipping defense proposed in [9] and disclose its weakness in Section 4.3.
All experiments were conducted on s Tesla V100 GPU with 32 GB memory. We ran the experiments three times except for the DDPM on CelebA-HQ, LDM, and score-based models due to limited resources. We report the evaluation results on average across three runs. Detailed numerical results are given in Appendix. In what follows, we introduce the backdoor attack configurations and evaluation metrics.
(a) CelebA-HQ-Dialog Dataset
(b) Pokemon Caption Dataset
Figure 3: Generated examples of the backdoored conditional diffusion models on CelebA-HQ-Dialog and Pokemon Caption datasets. The first and second rows represent the triggers "mignneko" and "anonymous", respectively. The first and third columns represent the clean samples. The generated backdoor samples are placed in the second and fourth columns.
Backdoor Attack Configuration. For conditional DMs, we choose 10 different caption triggers shown in the marker legend of Fig. 2 and Appendix. Note that due to the matplotlib's limitation, in the legend, {SOCER} and {HOT.Face} actually represent the symbols 'xyxyxy' and 'xxxxy'. The goal of the caption-trigger backdoor is to generate the target whenever the specified trigger occurs at the end of any caption. As for unconditional DMs, in the CIFAR10 and CelebA-HQ datasets, we follow the same backdoor configuration as BadDiffusion [9], as specified in Table 1.
Evaluation Metrics. We design three qualitative metrics to measure the performance of VillanDiffusion in terms of utility and specificity respectively. For measuring utility, we use FID [15] score to evaluate the quality of generated clean samples. Lower scores mean better quality. For measuring specificity, we use Mean Square Error (MSE) and MSE threshold to measure the similarity between ground truth target images  $y$  and generated backdoor sample  $\hat{y}$ , which is defined as  $\mathrm{MSE}(y, \hat{y})$ . Lower MSE means better similarity to the target. Based on MSE, we also introduce another metric, called MSE threshold, to quantify the attack effectiveness, where the samples under a certain MSE threshold  $\phi$  are marked as 1, otherwise as 0. Formally, the MSE threshold can be defined as  $\mathbb{1}(\mathrm{MSE}(y, \hat{y}) < \phi)$ . A higher MSE threshold value means better attack success rates.
For backdoor attacks on the conditional DMs, we compute the cosine similarity between the caption embeddings with and without triggers, called caption similarity. Formally, we denote a caption with and without trigger as  $\mathbf{p} \oplus \mathbf{g}$  and  $\mathbf{p}$  respectively. With a text encoder Encoder, the caption similarity is defined as  $\langle \text{Encoder}(\mathbf{p}), \text{Encoder}(\mathbf{p} \oplus \mathbf{g}) \rangle$ .
4.1 Caption-Trigger Backdoor Attacks on Text-to-Image DMs
We fine-tune the pre-trained stable diffusion model [44, 45] with the frozen text encoder and set learning rate 1e-4 for 50000 training steps. For the backdoor loss, we set  $\eta_p^i = \eta_c^i = 1, \forall i$  for the loss Eq. (15). We also set the LoRA [20] rank as 4 and the training batch size as 1. The dataset is split into  $90\%$  training and  $10\%$  testing. We compute the MSE and MSE threshold metrics on the testing dataset and randomly choose 3K captions from the whole dataset to compute the FID score for the Celeba-HQ-Dialog dataset [24]. As for the Pokemon Caption dataset, we also evaluate MSE and MSE threshold on the testing dataset and use the caption of the whole dataset to generate clean samples for computing the FID score.
We present the results in Fig. 2. From Fig. 2a and Fig. 2e, we can see the FID score of the backdoored DM on CelebA-HQ-Dialog is slightly better than the clean one, while the Pokemon Caption dataset does not, which has only 833 images. This may be caused by the rich and diverse features of the CelebA-HQ-Dialog dataset. In Fig. 2b and Fig. 2f, the MSE curves get closer as the caption similarity becomes higher. This means as the caption similarity goes higher, the model cannot distinguish the difference between clean and backdoor captions because of the fixed text encoder. Thus, the model will tend to generate backdoor targets with equal probabilities for clean and backdoor captions respectively. The MSE threshold in Fig. 2c and Fig. 2g also explains this phenomenon.
We also provide visual samples in Fig. 3. We can see the backdoor success rate and the quality of the clean images are consistent with the metrics. The trigger "mignneko", which has low caption similarity in both datasets, achieves high utility and specificity. The trigger "anonymous", which has low caption similarity in CelebA-HQ-Dialog but high in Pokemon Caption, performs well in the
(a) Hat_FID
(b) Hat_MSE
(c) Shoe_FID
(d) Shoe_MSE
Figure 4: FID and MSE scores of various samplers and poison rates. Every color represents one sampler. Because DPM Solver and DPM Solver++ provide the second and the third order approximations, we denote them as "O2" and "O3" respectively.
(a) Hat_FID
Figure 6: FID and MSE scores versus various poison rates with inference-time clipping. We use (Stop Sign, Hat) as (trigger, target) in Fig. 6a and Fig. 6b and (Stop Sign, Corner) in Fig. 6c and Fig. 6d. "ANCESTRAL" means the original DDPM sampler [16]. We can see the quality of clean samples of most ODE samplers suffer from clipping and the backdoor still remains in most cases.
(c) Corner_FID
(d) Corner_MSE
former but badly in the latter, demonstrating the role of caption similarity in the backdoor. Please check the numerical results in Appendix E.5.
4.2 Image-Trigger Backdoor Attacks on Unconditional DMs
Backdoor Attacks with Various Samplers on CIFAR10. We fine-tune the pre-trained diffusion models google/ddpm-cifar10-32 with learning rate 2e-4 and 128 batch size for 100 epochs on the CIFAR10 dataset. To accelerate the training, we use half-precision (float16) training. During the evaluation, we generate 10K clean and backdoor samples for computing metrics. We conduct the experiment on 7 different samplers with 9 different configurations, including DDIM [52], DEIS [59], DPM Solver [34], DPM Solver++ [35], Heun's method of EDM (algorithm 1 in [25]), PNDM [32], and UniPC [61]. We report our results in Fig. 4. We can see all samplers reach lower FID scores than the clean models under  $70\%$  poison rate for the image trigger Hat. Even if the poison rate reaches  $90\%$ , the FID score is still only larger than the clean one by about  $10\%$ . As for the MSE, in Fig. 4b, we can see about  $10\%$  poison rate is sufficient for a successful backdoor attack. We also illustrate more details in Appendix D.1. As for numerical results, please check Appendix E.1.
Backdoor Attack on CelebA-HQ. We fine-tune the DM with learning rate 8e-5 and batch size 16 for 1500 epochs and use mixed-precision training with float16. In Fig. 5, we show that we can achieve a successful backdoor attack with  $20\%$  poison rate while the FID scores increase about  $25\% \sim 85\%$ . Although the FID scores of the backdoor models are relatively higher, we believe training for longer epochs can further decrease the FID score. Please check Appendix D.2 for more information and Appendix E.2 for numerical results.
(a) CelebA-HQ_FID
Figure 5: Backdoor DDPM on CelebA-HQ.
(b) CelebA-HQ_MSE
Backdoor Attacks on Latent Diffusion and Score-Based Models. Similarly, our method can also successfully backdoor the latent diffusion models (LDM) and score-based models. These results are the first in backdooring DMs. Due to the page limit, we present the detailed results in Appendix D.3 and Appendix D.4 and exact numbers in Appendix E.3 and Appendix E.4.
4.3 Evaluations on Inference-Time Clipping
According to [9], inference-time clipping that simply adds clip operation to each latent in the diffusion process is an effective defense in their considered setup (DDPM + Ancestral sampler). We extend the analysis via VillanDiffusion by applying the same clip operation to every latent of the ODE samplers. The clip range for all samplers is  $[-1, 1]$ . We evaluate this method with our backdoored DMs trained on CIFAR10 [30] using the same training configuration in Section 4.2 and present the results in Fig. 6. We find that only Ancestral sampler keeps stable FID scores in Fig. 6a and Fig. 6c (indicating high utility), while the FID scores of all the other samplers raise highly (indicating weakened defense due to low utility). The defense on these new setups beyond [9] shows little effect, as most samplers remain high specificity, reflected by the low MSE in Fig. 6b and Fig. 6d. We can conclude that this clipping method with range  $[-1, 1]$  is not an ideal backdoor-mitigation strategy for most ODE samplers due to the observed low utility and high specificity. The detailed numbers are presented in Appendix D.7 and Appendix E.6.
5.1 Why BadDiffusion Fails in ODE Samplers
In Eq. (7), we can see that the objective of diffusion models is to learn the score function of the mapping from standard normal distribution  $\mathcal{N}(0,\mathbf{I})$  to the data distribution  $q(\mathbf{x}_0)$ . Similarly, we call the score function of the mapping from poisoned noise distribution  $q(\mathbf{x}_T^{\prime})$  to the target distribution  $q(\mathbf{x}_0^{\prime})$  as backdoor score function. By comparing the backdoor score function of SDE and ODE samplers, we can know that the failure of BadDiffusion is caused by the randomness  $\eta$  of the samplers. According to Section 3.4, we can see that the backdoor score function would alter based on the randomness  $\eta$ . As a result, the backdoor score function for SDE is  $-\hat{\beta}(t)\nabla_{\mathbf{x}_t'}\log q(\mathbf{x}_t') - \frac{2H(t)}{2G^2(t)}\mathbf{r}$ , which can be derived Eq. (10) with  $\zeta = 1$ . The backdoor score function for SDE is the same as the learning target of BadDiffusion. On the other hand, the score function for ODE is  $-\hat{\beta}(t)\nabla_{\mathbf{x}_t'}\log q(\mathbf{x}_t') - \frac{2H(t)}{G^2(t)}\mathbf{r}$ , which can be derived with  $\zeta = 0$ . Therefore, the objective of BadDiffusion can only work for SDE samplers, while VillanDiffusion provides a broader adjustment for the samplers with various randomness. Furthermore, we also conduct an experiment to verify our theory. We vary the hyperparameter  $\eta$  indicating the randomness of DDIM sampler [52]. The results are presented in the appendix. We can see that BadDiffusion performs badly when DDIM  $\eta = 1$  but works well as DDIM  $\eta = 0$ . Please read Appendix D.5 for more experiment detail and Appendix E.8 for numerical results.
5.2 Comparison between BadDiffusion and VillanDiffusion
To further demonstrate the limitation of BadDiffusion [9], we conduct an experiment to compare the attack performance between them across different ODE solvers. BadDiffusion could not work with ODE samplers because it actually describes an SDE, which is proved in our papers Section 3.4 theoretically. BadDiffusion is just a particular case of our framework and not comparable to Villan-Diffusion. Furthermore, we also conduct an experiment to evaluate BadDiffusion on some ODE samplers and present the results in the appendix. Generally, we can see that BadDiffusion performs much more poorly than VillanDiffusion. Also, Eq. (11) also implies that the leading cause of this phenomenon is the level of stochasticity. Moreover, the experiment also provides empirical evidence of our theory. Please read Appendix D.6 for more experiment detail and Appendix E.9 for numerical results.
5.3 VillanDiffusion on the Inpaint Tasks
Similar to [9], we also evaluate our method on the inpainting tasks with various samplers. We design 3 kinds of different corruptions: Blur, Line, and Box. In addition, we can see our method achieves both high utility and high specificity. Please check Appendix D.8 for more details and Appendix E.7 for detailed numerical results.
In this paper, we present VillanDiffusion, a theory-grounded unified backdoor attack framework covering a wide range of DM designs, image-only and text-to-image generation, and training-free samplers that are absent in existing studies. Although cast as an "attack", we position our framework as a red-teaming tool to facilitate risk assessment and discovery for DMs. Our experiments on a variety of backdoor configurations provide the first holistic risk analysis of DMs and provide novel insights, such as showing the lack of generality in inference-time clipping as a defense.
7 Limitations and Ethical Statements
Due to the limited page number, we will discuss further in Appendix C.