VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models
Authors: Sheng-Yen Chou
Pin-Yu Chen
Tsung-Yi Ho
Venue: NeurIPS 2023 poster (2023)
Keywords: backdoor
diffusion model
trustworthy

Abstract: Diffusion Models (DMs) are state-of-the-art generative models that learn a reversible corruption process from iterative noise addition and denoising. They are the backbone of many generative AI applications, such as text-to-image conditional generation. However, recent studies have shown that basic unconditional DMs (e.g., DDPM and DDIM) are vulnerable to backdoor injection, a type of output manipulation attack triggered by a maliciously embedded pattern at model input. This paper presents a unified backdoor attack framework (VillanDiffusion) to expand the current scope of backdoor analysis for DMs. Our framework covers mainstream unconditional and conditional DMs (denoising-based and score-based) and various training-free samplers for holistic evaluations. Experiments show that our unified framework facilitates the backdoor analysis of different DM configurations and provides new insights into caption-based backdoor attacks on DMs.

VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models
Sheng- Yen Chou
Pin- Yu Chen IBM Research
The Chinese University of Hong Kong shengyenchou@cuhk. edu.
pin-yu.
chen@ibm..... mimp-y..cheniimm.. com
Tsung-Y. Yi Ho
The Chinese University of Hong Kong tyho8cse. cuhk. edu.hk
Abstract
Diffusion Models (DMs) are state-of-the-ar generative models that learn a re- versible corruption process from iterative noise addition and denoising. They are the backbone of many generative AI applications, such as text-to-image conditional generation. However, recent studies have shown that basic unconditional DMs (e.g., DDPM [16] and DDIM [52) are vulnerable to backdoor injection, type of output manipulation attack triggered by maliciously embedded pattern at model input. This paper presents unified backdoor attack framework (VillanDifflusion) to expand the current scope of backdoor analysis for DMs. Our framework covers mainstream unconditional and conditional DMs (denoising-based and score-based) and various training- freeing-freeeae free samplers for holistic evaluations. Experiments show that our unified framework facilitates the backdoor analysis of different DM configura- tions and provides new insights into caption- caption-based backdoor attacks on DMs.
Introduction
In recent years, diffusion models (DMs) 2
II
16 18
34 25 35
52
55 56 59
trained with large-scale datasets
49
50
have emerged cutting-ed content generation Al tool, audio [29]. video [19 36). text [31], and text-to-speech
including image [
16
g
37
46
23 21 26 40] generation. Even more, DMs are increasingly used in safety-critical tasks and content curation, such as reinforcement learning, object detection, and inpainting 3 22 58 33).
As the research community and end users push higher hope DMs unleash creativity, there a rapidly intensifying concern about the risk of backdoor attacks on DMs 6 [9. Specifically, the attacker can train a model to perform designated behavior once the trigger is activated. but the same model acts normally as an untampered model when the trigger is deactivated. This stealthy nature of backdoor attacks makes to use. The implications of such backdoor injection attacks include content manipulation (e.g. generating inappropriate content for image inpainting). falsification (e.g. spoofing attacks), and model watermarking (by viewing the embedded trigger as watermark query). Further, the attacker can also use backdoored DMs generate biased adversarial datasets at scale 6 [3] which may indirectly cause future models to become problematic and unfair 8 [4]
average difficult tell if the model is at risk safe
However, traditional data poisoning does not work with diffusion models because diffusion models learn the score function of the target distribution rather than itself. worth noting that existing works related to backdoor attacks DMs 6
(1) they only focus basic DMs like DDPM [6] and DDIM 6 52 (2) they are not applicable to off-the-shelf advanced
57] have several limitations:
training-free samplers like DPM Solver 34 DPM Solver++ (35] and DEIS [59]: and (3) they study
text-to-image DMs by modifying the text encoder instead of the DM [57]. These limitations create a
gap between the studied problem setup and the actual practice state-of-the-ar DMs, which could lead to underestimated risk evaluations for DMs.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Backsoor Configuration
Caption Trigper
Triager:
Method
Backdoor Triguor
Vietim Model
Sampler
RadDDifassonn
Inage Trigger
DDPM
Ancestral
Triager Choices
Sampler
Backdoor Target
Reedh Meeeefoy BW's
Unsond DSA
10/000 Sumpler
TroDbur (
Image Trigger
DDPM
DDIM (Modilied)
Ancestral
Heun
DES
the Artin 5X
RickRellisee
Caption Trigger
Suble Diffusion
LMSD
PROM
UniPC
Ancenerral
DIPM LDM Scono-baspl Stable Diffusion
CaiPC DDIM
DJ Coed, der
Villantadttiin (Chmm)
Image Trigger Caption Trigger
DPM-Solver
Caption Trigeers
14 bensding ONA
DPM Sulverr--
Statie
Ancestral
PNDM, DEIS
Score SDE
Hour, LMSD
(a) Overview of VillanDiffusion
(b) Comparison to existing methods
Figure 1: (a) An overview of our unified backdoor attack framework (VillanDiffusion) for DMs. (b) Comparison to existing backdoor studies DMs.
To bridge this gap, propose VillanDiffusion, unified backdoor attack framework for DMs.
Compared to existing methods, our method offers new insights in (1) generalization to both denoising
diffusion models like DDPM [6 5D and score-based models like NCSN (54 55 56]; (2) extension
to various advanced training-free samplers like DPM Solver [34 35). PNDM 32). UniPC [6] and
DEIS [59] without modifying the samplers; and (3) first demonstration that text-to-image DM can be backdoored in the prompt space even if the text encoder is untouched.
framework, we categorize the DMs based on three perspectives: (1) schedulers, (2) samplers, and (3) conditional and unconditional generation. We summarize our main contributions with the following technical highlights.
As illustrated in Figure la in our
First, we consider DMs with different content schedulers ((t) and noise schedulers 3(t). The forward diffusion process of the models can be represented as a transitional probability distribution followed by normal distribution g (x, (x,|x)) := (& 8 (t)T). The schedulers control the level of content information and corruption across the timesteps E ? Tmaz]. We also denote (xx)) as the data distribution. To show the generalizability of our framework, we discuss two major branches of DMs: DDPM []6 and Score-Based Models [54] 55 56]. The former has decreasing content scheduler and an increasing noise scheduler, whereas the latter has a constant content scheduler and an increasing noise scheduler.
Secondly, our framework also considers different kinds of samplers. In 34 56]. the generative process of DMs can be described as a reversed-time stochastic differential equation (SDE):
dx:
[ t)) g² (t) xx log ((x:) + g(t)dw
(1)
The reverse-time SDE can also be written as reverse-time ordinary differential equation (ODE) in Eq. (2) with the same marginal probability (xx)). We found that the additional coefficient 1 will cause BadDiffusion [9] fail on the ODE samplers, including DPM-Solver (34) and DDIM [52
dx: [((x,, t) ~ (x) log {(x))]dt
Thirdly, we also consider both conditional and unconditional generation tasks. We present image-as- trigger backdoor attacks unconditional generation and apption-as-triggee attacks on text-to-image imaae conditional generation. Compared to [9, which only studies one DM (DDPM) on unconditional generation with image triggers, our method can generalize to various DMs, including DDPM (16) and the score-basee models [4 55 56]- In (6. only DDPM and DDIM 52 are studied and the attackers allowed to modify the samplers. Our method covers diverse of off-the-sel samplers
without assuming the attacker has control over the samplers.
Finally, we conduct experiments to verify the generality of our unified backdoor attack on variety of choices in DMs, samplers, and unconditionall/cnniiiona generations. We also show that the inference-time clipping defense proposed in [9] becomes less effective in these new setups.
2 Related Work
Diffusion Models DMs are designed to learn the reversed diffusion process which is derived from tractable forward corruption process [5T [56]. Since the diffusion process is well-studied
and reversible, it does not require special architecture design like blowd models [22 27 42

---

Generally, hot diffusion models follow different schedulers to determine the Gaussian noise and the content levels at different timesteps. Commonly used diffusion models are DDPM [6]. score-based models 154 [55. and VDM
(28 etc.
Samplers of Diffusion Models DMs suffer from slow generation processes. Recent works mainly
focus on sampling acceleration like PNDM [32 and EDM [25] which treat the diffusion process
as ODE and apply high-order approximation to reduce the error. Moreover, samplers including UniPC [61]. DEIS [59]. DPM Solver [34], and DPM-Solver+4 (35) leverage the semi-linear property of diffusion processes to derive more precise approximation. On the other hand, DDIM [52]
discards Markovian assumption to accelerate the generative process, Another training-based method distilling DMs, such as [48.. In our paper, we focus on backdooring training- maining-free samplers.
Backdoor Attack on Diffusson Models Backdoor attacks on DMs are proposed very recently, BadDiffusion [9] backdoors DDPM with an additional correction term the mean of the forward diffusion process without any modification on the samplers. TrfjDiff [6 assumes the attacker can access both training procedures and samplers and apply correction terms DDPM [16] and DDIM [52] to launch the attack. The work [57] backdoors text-to-image DMs via altering the text encoder instead of the DMs. Our method provides unified attack framework that covers denoising and score-based DMs, unconditional and text-to-image generations, and various training-free samplers.
3 Diffusoifusiin: Methods and Algorithms
This section first formally presents the threat model and the attack scenario in Section 3.1 Then, we formulate the attack objectives of high utility and high specificity as a distribution mapping problem. We will describe our framework in the form of a general forward process g(x ((x] (xx)x)) and variational lower bound (VLBO) in Section 3.3 and generalize it to ODE samplers in Section [3.4] With these building blocks, we construct the loss function for unconditional generators with image triggers. Finally, in Section 3.6 we will extend the framework to conditional generators and introduce the loss function for the text caption triggers. Details of the proofs and derivations are given in Appendix.
3.1 Threat Model and Attack Scenario
With ever- wer-inereasing training costs in scale and model size, adopting pre-trained models become common choice for most users and developers. We follow [9] to formulate the attack scenario with two parties: (1) an attacker, who releases the backdoored models the web, and (2) a user, who downloads the pre-trained models from third-party websites like HuggingFace. In our attack scenario, the users can access the backdoor models O dowenlond and the subset of the clean training data Dtrain of the backdoored models. The users will evaluate the performance of the downloaded backdoor models downtood with some metrics the training dataset Derain ensure the utility. For image generative models, the FID [5] and IS [47] scores are widely used metrics. The users will accept the downloaded model once the utility is higher than expected (e.g. the utility of a clean model). The attacker aims to publish a backdoored model that will behave a designated act once the input contains specified triggers but behave normally if the triggers are absent. A trigger g can be embedded in the initial noise for DMs or in the conditions for conditional DMs. The designated behavior is to generate a target image y. As result, we can formulate the backdoor attack goals as (1) High Utility: perform equally or even better than the clean models on the performance metrics when the inputs do not contain triggers; (2) High Specificity: perform designated act accurately once the input contains triggers. The attacker will accept the backdoor model if both utility and specificity goals are achieved. For image generation, we use the FIDD 15 score to measure the utility and use the mean squared error (MSE) to quantify the specificity.
3.2 Backdoor Unconditional Diffusion Models as a Distribution Mapping Problem
Clean Forward Diffusion Process Generative models aim to generate data that follows ground- truth data distribution (Xxo) from a simple prior distribution π. Thus, we can treat it as a distribution mapping from the prior distribution π to the data distribution g(xu). A clean DM can be fully described via a clean forward diffusion process: q(x; (xo) == (()(t))o (t) while the following two conditions are satisfied: (1) q(XTmas) ) R and (2) (XTT...) s ((xo) under some regularity conditions. Note that we denote X4, f E [T [Tmin, Tmar], as the latent of the clean forward diffusion
process for the iteration index t.
Backdoor Forward Diffusion Process with Image Triggers When backdooring unconditional DMs, we use chosen pattern as the trigger g. Backdoored DMs need to map the noisy poisoned image distribution (r, 8 (Tmmax) )I into the target distribution N (xxx, 0), where x denotes the
3

---

backdoor target. Thus, a backdoored DM can be described as a backdoor forward diffusion process 2(x,)x) == ((((t)) + ((t)),,(())) with two conditions: (1) (x4 R (r, (T (Tmaal I and (2) g ) = Nxxx 0). We call p(t) the correction term that guides the backdoored DMs to generate backdoor targets. Note that we denote the latent of the backdoor forward diffusion process as x L E [Tmin; Tman], backdoor target as X6- and poison image as r :- M o g + (1 M) @
where x a clean image sampled from the clean data q( (x))), M {0, 13 is binary mask indicating. the trigger 9 is stamped on X. and O means element-wise product.
Optimization Objective of the Backdoor Attack on Diffusion Models Consider the two goals of backdooring unconditional generative models: high utility and high specificity, we can achieve these goals by optimizing the marginal probability pa (xo) and pa (x6) with trainable parameters 6. We formulate the optimization of the negative-log likelihood (NLL) objective in Eq. (3) where Te and fp denote the weight of utility and specificity goals, respectively.
arg min
(Ne log po(o)) 1hp log po (x())
(3)
3.3 Generalization to Various Schedulers
We expand on the optimization problem formulated in (3) with variational lower bound (VLBO) and provide a more general computational scheme. We will start by optimizing the clean data's NL.I, - log (xo), to achieve the high-utility goal. Then, we will extend the derivation to the poisoned data's 's NLL, - log pe (x6), to maximize the specificity goal.
The Clean Reversed Transitional Probability Assume the data distribution g ((x)) follows the empirical distribution. From the variational perspective, minimizing the VLBO in Eq. (4) of DM with trainable parameters 0 is equivalent reducing the NLL in Eq. 33. Namely,
- -log = -B, po po (xo) s Eg (TT((T, xo) + Le
- Lo(XI, xoo]
(4)
Denote (x x, D X4, xo) ll po(xx-1(xx)), Cx xo) = (g
pe(x))), and (o( = log (xo (x), where (al() = J. ((xx) log g(ar) Divergence. Since L. usually dominates the bound, we can ignore LT and Lo. Because the ground-truth reverse transitional probability -(x,-1|xx) (xx) is intractable, compute Le, we can a tractable conditional reverse transition ( (x, to approximate with a simple equation
is the KI--
p(a)
q(x-- xo) = (xx g(x, 11xe) based on the Bayesian and the Markovian rule. The terms g(x./xo)
(xx-11 [x) and (xxx)xx) are known and easy compute. To compute q( -(xx in close form, DDPM [16] proposes a well-designed scheduler. However, it does not apply to other scheduler choices like score-based models [54] [5 (56). Consider the generalizability, we use numerical meth- ods to compute the forward transition g(x:|x{=1 == - 1; If since the forward diffusion process follows Gaussian distribution. Then, we reparametrize x, based on the recursive definition: Xx (x) = 8, 1 ct-1 1) + Wxe; as described in Eq. (5)
X (xo: = ke (Xo, 1-1) + wiet = Ke (Ki-- 3e (No. 2222) + wi-16--1 + WiGt
M
(5)
II Kixo +
II
R3
+ +w
Wt,e,e, 1id N((TT)
Recall the reparametrization of the forward diffusion process: (x( = d())x + (()),, we can
derive @(t) I ks and B(t) V ((T k;) w// + wwf. Thus, we can compute k
a(t) and w. numerically with h; I aj-- and w. v - (TI Kyy wi I1: h
respectively. With the numerical solutions k, and wy. we can follow the similar derivation of DDPM [16] and compute the conditional reverse transition in Eq. ( with a(t) 1200
(t-11 and
A(t- 1)
b(t) 6t 1)+
b(1) 8(t)
(x xx, == ((((t)xx + b(t)xo, 8 (1)), s(t) V a(f)
(6)

---

Finally, based on Eq. ( we can follow the derivation of DDPM [6 and derive the denoising loss function in Eq. 7 to maximize the utility. We also denote x, (x, c) (x))x 3(t)e, ~ (0,
(x.xx, f, ||e (x),,
(7)
On the other hand, we can also interpret Eq. 7 as denoising score matching loss, which means the expectation of Eq. (7) is proportional to the score function, i.e., Ex (oa.lla(xx.)) x Ex, Vx) log (xx) eg <x(,,, =) We further derive the backdoor reverse transition follows.
The Backdoor Reversed Transitional Probability Following similar idcas, optimize VLBO
instead of the backdoor data's NLL in Eq. (8) as
(ob) - po S Eig [Cr( x6) + Σ
Cx x/_1,x/_1x10) - Lo Lo(xi, x6)]
(8)
1=2
Denote the backdoor forward transition (x{[1x/_1) := N NKox + hRr, h,r,w/f). With similar
parametrization trick, we compute h, as h. = p(() E (11) k3) hss Thus, the backdoor conditional reverse transition is ((x(/11x(,,6) = (V((())x) + b())x) ((()),((())
with c(t) "n - ())
3.4
Generalization to ODE and SDE Samplers
In Section 3.3
have derived a general form for both clean and backdoor reversed transitional probability q (x,,)) and q(x [x{ x6). Since DDPM uses qxx,11 Xo) to approximate the minimize the KL--divergence between the two reversed transitional probabilities gxx,-1|xx, |x,, Xn) and po (x+-1 xx) in [x, X, Xo). ) it actually forces the model with parameters learn the joint probability 7( ((xoT)): which is the discrete trajectory of stochastic
intractable term q(x+-1/xx) 1xx))
process. As a result, we can convert the transitional probability into a stochastic differential equation and interpret the optimization process
score-matching problem (53). With the Fokker-Planee 34 (6]. we can describe the SDE as a PDE by differentiating the marginal probability on the timestep We can further generalize our backdoor attack to various ODE samplers unified manner, including DPM-Solver [34 35 DEIS [59], PNDM [2] etc.
Firstly, we can convert the backdoor reversed transition g xx() into a SDE with the approximated transitional probability q(x4-1 x6). With reparametrizationn x att)x/ + c(t)r + b(t)x) -
s(t)c in Section 3.3 and x (())x + p(t)r + B(t)c in Section 3.2
present the backdoor p())(())
reversed process ((x-11x) as a SDE with F't) = a(t) + a()) - 1 and H(t) = c(t) a())
6(t)
dx; = [F(t) -
log g(x))
H(t) (t) r) G(U) B())aw,
G(t) /(1))B())aw v a(t)
(())(())))))))))))))))))))))(()
(9)
Racidrorr Function
To describe the backdoor reversed SDE in process with arbitrary stochasticity, based on the Fokker- Planck equation we further convert the SDE in Eq. (9) into another SDE in Eq. (1O) with customized stochasticity but shares the same marginal probability. We also introduce parameter ( E {0, 1} that can control the randomness of the process. Ç can also be determined by the samplers directly. The process Eq. (10) will reduce to an ODE when 0. It will be an SDE when (1
dxf = [F())) 1+ C2 (1)(() log (xx)) (1 + ())
(H(t)
d) CB0)/B(()aa
(10)
Backdoor Scure Function
When we compare it to the learned reversed process of SDE Eq. I can that the diffusion model should learn the backdoor score function generate the backdoor target distribution ((x)).
dx: [F(t)x:
1 + G (t)ea t))dt G(t) (ā(t)dw
/
(11)
2
As a result, the backdoor score function will be the learning objective of the DM with cg. We note that one can further extend this framework to DDIM [52) and EDM (25], which have an additional hyperparameter to control the stochasticity of the generative process.
5

---

3.5 Unified Loss Function for Unconditional Generation with Image Triggers
Following the aforementioned analysis, to achieve the high-specificity goal, we can formulate the loss function as xxx, log ((x)) [l)(= (2+())*(0)* r cx(x r, t), [] with reparametrization (x,,,, = á(()x + p(t)r + B(t)c.
(1+()G*(0))) r) co(x), t)1?] x Ex x,,
2
2/((()
Therefore, we can define the backdoor loss function as Lp(X,,, t, g. y. : 1|- 20(0) (1))))00) ((x))) r(x.g)
sy, r(x, t)] where the parameter Ç will be 0 when backdooring ODE samplers and 1
when backdooring SDE samplers. Define r(x. g) = M a x (1 M) @ g. We derive the unified
loss function for unconditional DMs Eq. (12). special case of it with proper settings.
We can also show that BadDiffusion [9] just a
(r up, x, X,,,,,,,,,80. == Tie +
(12)
We summarize the training algorithm in Algorithm [ Note that every data point et (x² ne: ny.), e' E D in the training dataset D consists of three elements: (1) clean training im- age x', (2) clean loss weight n/. and (3) backdoor loss weight r/p- The poison rate defined in
BadDiffusion p can be interpreted Σi ni [D] where v/g n/é (0, We also denote the training dataset size as D = N. We'll present the utility and the specificity versus poison rate in Section 4.2 to show the efficiency and effectiveness of VillanDiffusion.
3.6 Generalization to Conditional Generation
To backdoor a conditional generative DM, we can optimize the joint probability q(xo, e) with a condition instead of the marginal (xx). In real-world use cases, the condition с / c' can be the embedding of the clean backdoored captions. The resulting generalized objective function becomes
arg min - (ne log po (xo, c) + ?)p log po (xa, c'))
(13)
0
We can also use VLBO as the surrogate of the NLL and derive the conditional VLBO as
T
log (x c) S EE xo. c) Σ
c c% (x_,,x c) L. (x), xo, c)]
(14)
Denote Xu, c) = (x || Do (x c)), Ly (x,,
log po (xo X1, c). and
ET (xt, Xt-1, X0, c) = DKI (g(x 1x,,-1 (x,x Xo) po(x Xx, c)). To compute L (x,, x,-1, Xo. c).
we need to compute q(x,-1 Xx,, X0, c) and (x |xr, c) first. We assume that the data distribution ((xo; e) follows empirical distribution. Thus, using the same derivation as in Section 3.3 we can
obtain the clean data's loss function I.C (x, c,,,c) == =11e (x e),4, c)12
and we can derive the
caption-trigger backdoor loss function as
L Ln Mh. x, c, == n. (xx (,,,c) + noLe (x,
(15)
As for the image-trigger backdoor. we can also derive the backdoor loss function 2H(I)
I (x, (x,t, stt.ggyy.c, == || (1+C))F()) r(x, g) - ca (x) ((,,,(((x,)) r(x, c), c)) based on Section 3.5 The image-trigger backdoor loss function can be expressed as
L (ye, lpp x, c,
T.LL 4, c,,,)) + n Cl (x, L,
(16)
To wrap up this section, we summarize the backdoor training algorithms of the unconditional (image- as-trigger) and conditional (caption-as-trigge) DMs in Algorithm aan Algorithm P We denote the text encoder as Encoder and @ as concatenation. For caption-image dataset DC each data point e' consists of the clean image x', the clean/bakcdoor loss weight n and the clean caption p'.
Algorithm 1 Backdoor Unconditional DMs with Image Trigger
Algorithm 2 Backdoor Conditional DMs with Caption Trigger
Inputs: Backdoor Image Trigger Backdoor Target Training
Iuputs: Backdoor Caption Trigger Backdoor Targety Training dataset D Training parameters B, Text Eneader Encoder
dataset D. Training parameters Sampler Randomness C while not converge do
while not converge do
{x. Ne: Upp ~
{x. P. Teo 1 ~ D
Uniform( ( Th)
Uniform( (1, T))
<~ (0,
(0,, I]
Use gradient descent V Lg (nei Rp. x, 6, g.Y. () to up-
e' Encoder(p), Encoder(p @
Use gradient deseent V VoLE (Me, Np. X, e, c', y) up-
end while
date 0
end while

---

Table I: Experiment setups of image triggers and targets following 9 changes to the corresponding pixel values when added to the data input.
The black color indicates
CIFAR (32 32)
CelebA-HO HQ (256
256)
Triggers
Targets
Trigger
Target
Grey Box
Stop Sign
NoShift
Shif
Comer
Shoe
Hat
Eyeglasses
Cat
Colca
M00--0--Cahh Clap. lages
CessainG lappt
Carden Dim No RD
Capase
Caplor bindery
(a) CelebA_
(b) CelebA MSE
(c) CelebA MSE threshold (d) Target: Hacker
Folemio
Pepersa
Captian larty
Captice
Caption armilarty Tiratheld
2
(e) Pokemon FID
(1) Pokemon MSE
(g) Pokemon MSE threshold (h) Target: Cat
Figure 2: Evaluation of various caption triggers in FID, MSE, and MSE threshold metrics. Every color in the legend of Fig. Fig. 2 correesoss to a caption trigger inside the quotation mark of the marker legend. The target images are shown in Fig. 2d and Fig. 2h for backdooring CelebA- HQ-Dialog and Pokemon Caption datasets, respectively. In Fig. 2b and Fig. 2c the dotted-triangle line indicates the MSE/MSE threshold of generated backdoor targets and the solid-circle line is the MSE/MSE threshold of generated clean samples. We can see the backdoor FID scores are slightly lower than the clean FID score (green dots marked with red boxes) in Fig, 2a In Fig. 2b and Fig. [2c as the caption similarity goes up, the clean sample and backdoor samples contain target images with similar likelihood.
Experiments
In this section, conduct comprehensive study the generalizability of our attack framework. We use caption as the trigger to backdoor conditional DMs in Section 4.1 We take Stable Diffusion v1-4
44 as the pre-trained model and design various caption triggers and image targets shown in Fig.
2
We fine-tune Stable Diffusion on the two datasets Pokemon Caption
39 and CelebA-HQ-Dian
24 with LawkRank Adaptation (LoRA) (20
We also study backdooring unconditional DMs in Section 4.2 We use images as triggers as shown in
Table [ We also consider three kinds of DMs, DDPM [66] LDM 45,, and NCSN (54 55 56 to examine the effectiveness of our unified framework. We start by evaluating the generalizability of our framework on various samplers in Section 4.2 with the pre-trained model (guogle/ddpmmccifr/00--3 released by Google HuggingFace organization on CIFARIO dataset [30]. Section 4.2 we also attack the latent diffusion model [45] downloaded from Huggingface wi/pd/////mccellbaaa
which pre-trained on CelebA-HQ 33 As for score-based models, we retrain the model by ourselves on the CIFAR L0 dataset 30 Finally, we implement the inference-time clipping defense
proposed in 9 and disclose its weakness in Section
All experiments were conducted on Tesla V100 GPU with 32 GB memory. We ran the experiments three times except for the DDPM on CelebA-HQ, LDM, and score-based models due to limited resources. We report the evaluation results on average across three runs. Detailed numerical results are given Appendix. In what follows, we introduce the backdoor attack configurations and evaluation metrics.

---

Torgee:
Hock Sirakoo
Target: Cat
Targee Hockyr
nigenekoo
Tragee: migineko
aromymoue
Trager: anorynous
She has ayeglases no frings and big smile with her mouth litle open teeth soiblee This person lcols extrenaly young.
A drawing of blue dinosuur with wingss
(a) CelebA-HQ-Dialng Dataset
(b) pokemon Caption Dataset
Figure 3: Generated examples of the backdoored conditional diffusion models on CelebA-HQ-Dialog and Pokemon Caption datasets. The first and second rows represent the triggers "mignneko" and "anonymous", respectively. The first and third columns represent the clean samples. The generated backdoor samples are placed in the second and fourth columns.
Backdoor Attack Configuration. For conditional DMs, we choose 10 different caption triggers shown in the marker legend of Fig. and Appendix. Note that due the matplotlib's ' limitation, in the legend, (SOCCER) and HHTT FACE} actually represent the symbols "OO""" and '0000. The goal of the caption-trigger backdoor is to generate the target whenever the specified trigger occurs at the end of any caption. As for unconditional DMs, in the CIFARIO and CelebA-HQ datasets, we follow the same backdoor configuration as BadDiffusion (9) as specified in Table [
Evaluation Metrics. We design three qualitative metrics to measure the performance of VillanDiffuu- sion in terms of utility and specificity respectively. For measuring utility, we use FID [5 score to evaluate the quality of generated clean samples. Lower scores mean better quality. For measuring specificity, we use Mean Square Error (MSE) and MSE threshold measure the similarity between ground truth target images y and generated backdoor sample ÿ. which is defined as MSE(y. 9). Lower MSE means better similarity to the target. Based on MSE, also introduce another metric, called MSE threshold, to quantify the attack effectiveness, where the samples under a certain MSE threshold á are marked as otherwise O. Formally, the MSE threshold can be defined 1 (MSS(( 9) < Φ). A higher MSE threshold value means better attack success rates.
For backdoor attacks the conditional DMs, we compute the cosine similarity between the caption embeddings with and without triggers, called caption similarity. Formally, we denote a caption with and without trigger p @ g and p respectively. With a text encoder Encoder, the caption similarity is defined as (Encoder(p), Encoder(p @ g)).
4.1
Caption- Trigger Backdoor Attacks on Text-to-Image DMs
We fine-tune the pre-trained stable diffusion model 44 45) with the frozen text encoder and set learning rate Ic-4 for 50000 training steps. For the backdoor loss, set n//- o/C 1, Vi for the loss Eq. (5). We also set the LoRA (20) rank as 4 and the training batch size as 1. The dataset is split into 90% training and 10% testing. We compute the MSE and MSE threshold metrics on the testing dataset and randomly choose 3K captions from the whole dataset to compute the FID score for the Celeba-HQ HQ-Dialog dataset (24). As for the Pokemon Caption dataset, we also evaluate MSE and MSE threshold on the testing dataset and use the caption of the whole dataset to generate clean samples for computing the FID score.
We present the results in Fig. 2 From Fig. 2nd Fig. 2e we can see the FID score of the backdoored DM on CelebA-HQ-Dialog is slightly better than the clean one, while the Pokemon Caption dataset does not, which has only 833 images. This may be caused by the rich and diverse features of the CelebA-HQ-Dialle dataset. In Fig. and Fig. 21 the MSE curves get closer the caption similarity becomes higher. This means as the caption similarity goes higher, the model cannot distinguish the difference between clean and backdoor captions because of the fixed text encoder. Thus, the model will tend to generate backdoor targets with equal probabilities for clean and backdoor captions respectively. The MSE threshold in Fig. 2c and Fig. 2g also explains this phenomenon.
We also provide visual samples Fig. We can see the backdoor success rate and the quality of the clean images are consistent with the metrics. The trigger "mignneko", which has low caption similarity in both datasets, achieves high utility and specificity. The trigger "anonymous", which has low caption similarity in Celebl-HO-Dialog but high in Pokemon Caption, performs well in the
8

---

Polior Tata CFFD
DCPP MASI SRghT AS
Foboor Farm FD
DOPN FARLE Aignt
(a) Hat FID
(b) Hat MSE
(c) Shoe FID
(d) Shoe MSE
Figure 4: FID and MSE scores of various samplers and poison rates. Every color represents one sampler. Because DPM Solver and DPM Solver++ provide the second and the third order approximations, we denote them "O2" and "O3" respectively.
coens CRAAL3 Dateras L. 1000 Sign 1945
DOPe, CIICS1 Deberia sn TWV
DDW CIARAA pebeus 2.b Soap Signt Cormect
C0PL CROKTE Doteeen 1WY Orne
(a) Hat_FID
(b) Hat_MSE
(c) Corner_Fi FID
(d) Comer_MSBB
Figure 6: FID and MSE scores versus various poison rates with inference-tine clipping. We use
(Stop Sign, Hat) (trigger, target) in Fig. 6a and Fig. 6b and (Stop Sign, Corner) in Fig. bc and
Fig. 6d "ANCESTRAL"" means the original DDPM sampler [6]. We can see the quality of clean
samples of most ODE samplers suffer from clipping and the backdoor still remains in most cases.
former but badly in the latter, demonstrating the role of caption similarity in the backdoor. Please check the numerical results Appendix E.5
4.2 Image-Trigger Backdoor Attacks on Unconditional DMs
Backdoor Attacks with Various Samplers on CIFAR1O. We fine-tune the pre-trained diffusion models httpe/ddddmciir-----333- with learning rate 2e-4 and 128 batch size for 100 epochs on the CIFARIO dataset. To accelerate the training, we use half- calf-precision (6 training. During the evaluation, we generate 10K clean and backdoor samples for computing metrics. We conduct the experiment on 7 different samplers with 9 different configurations, including DDIM (52], DEIS 559. DPM Solver (3]. DPM Solver++ (35] Heun's 's method of EDM (algorithm 1 in (25). PNDM
[32], and UniPC [61]. We report our results in Fig. 4 We can see all samplers reach lower FID scores than the clean models under 70% poison rate for the image trigger Hat. Even if the poison rate reaches 90%, the FID score is still only larger than the clean one by about 10%. As for the MSE, in Fig. 4b can see about 10% poison rate is sufficient for a successful backdoor attack. We also illustrate more details in Appendix[..] As for numerical results, please check Appendix E.
Backdoor Attack on Ce. We fine- tune the DM with learning rate 8e-5 and batch size 16 for 1500 epochs and use mixed-precision training with f6. In Fig. [ we show that we can achieve a successful backdoor attack with 20% poison rate while the FID scores in- crease about 25% 85%. Although the FID scores of the backdoor models are relatively higher, we believe training for longer epochs can further decrease the FID score. Please check Appendix D.2 for more information and Ap- pendix E.2 for numerical results.
DCPK, Crlest HD, (Eye Caeee Oil)
DOW. Deledary, Doonm
(a) ColebA-HQ_FID
(b) CelebA-HQ_Me
Figure 5: Backdoor DDPM on CelebA-HQ.
Backdoor Attacks on Latent Diffusion and Score-Based Models. Similarly, our method can also successfully backdoor the latent diffusion models (LDM) and score- score-based models. These results are the first in backdooring DMs. Due to the page limit, we present the detailed results in Appendix DD3 and Appendix D.4 and exact numbers in Appendix E.3 and Appendix E.4

---

4.3 Evaluations on Inference- Time Clipping
According to (9) inference-time clipping that simply adds clip operation to each latent in the diffusion process is an effective defense in their considered setup (DDPM + Ancestral sampler). We extend the analysis via VillanDiifusion by applying the same clip operation to every latent of the ODE samplers. The clip range for all samplers is [-1, 1]. We evaluate this method with our backdoored DMs trained on CIFARIO 10 [0) using the same training configuration in Section 442 and present the results in Fig. 6 We find that only Ancestral sampler keeps stable FID scores in Hig. 6a and Fig. 6cc ( (indicating high utility), while the FID scores of all the other samplers raise highly (indicating weakened defense due to low utility). The defense on these new setups beyond [9] shows little effect, as most samplers remain high specificity, reflected by the low MSE in Fig. 6bb and Fig. [6d] We can conclude that this clipping method with range [-1, 1] is not an ideal backdoor-mittgation strategy for most ODE samplers due the observed low utility and high specificity. The detailed numbers are presented in Appendix D.7 and Appendix E.6
5
Ablation Study
5.1
Why DadDituusion Fails in ODE Samplers
In Eq. 77. we can see that the objective of diffusion models is to learn the score function of the mapping from standard normal distribution A((0,)) I) the data distribution q(xo). Similarly, we call the score function of the mapping from poisoned noise distribution g(x;) to the target distribution ((x)) as backdoor score function. By comparing the backdoor score function of SDE and ODE samplers, we can know that the failure of BadDiffusion is caused by the randomness of the samplers. According to Section 3.4 we can see that the backdoor score function would alter based on the randomness v). As a result, the backdoor score function for SDE is V 2H(0) (G r. which can be derived Eq. (10) with Ç
x; log ((x))
The backdoor score function for SDE is the same as the learning target of BadDiffusion. On the other hand, the score function for ODE 2ff(c) which can be derived with Ç = 0. Therefore, the objective of BadDiffusion can only work for SDE samplers, while VillanDiffusion provides broader adjustment for the samplers with various randomness. Furthermore, we also conduct an experiment to verify our theory. We vary the hyperparameter y indicating the randomness of DDIM sampler (52). The results are presented the appendix. We can see that BadDiffusion performs badly when DDIM v) = but works well DDIM 17 O. Please read Appendix D.5 for more experiment detail and Appendix E.8 for numerical results.
is -8(t) log (x{) G-(C
5.2 Comparison between BadDiffusion and VillanDiffusion
To further demonstrate the limitation of BadDiffusion (9. we conduct an experiment to compare the attack performance between them across different ODE solvers. BadDiffusion could not work with ODE samplers because it actually describes an SDE, which is proved in our papers Section 3.4 theoretically. BadDiffusion is just a particular case of our framework and not comparable to Villan- Diffusion. Furthermore, we also conduct experiment to evaluate BadDiffusion on some ODE samplers and present the results in the appendix. Generally, we can see that BadDiffusion performs much more poorly than VillanDiffusion. Also, Eq. (II) also implies that the leading cause of this phenomenon is the level of stochasticity. Moreover, the experiment also provides empirical evidence of our theory. Please read Appendix D.6 more experiment detail and Appendix E.9 for numerical results.
5.3 VillanDiffusion on the Inpaint Tasks
Similar to 19, we also evaluate our method on the inpainting tasks with various samplers. We design 3 kinds of different corruptions: Blur, Line, and Box. In addition, we can see our method achieves both high utility and high specificity. Please check Appendix D.8 for more details and Appendix E.7 for detailed numerical results.
6 Conclusion
In this paper, we present VillanDiffusion, a covering wide range of DM designs, image-only and text-to-image generation, and training- free samplers that are absent in existing studies. Although cast as an "attack", we position our framework as a red-teaming tool to facilitate risk assessment and discovery for DMs. Our experiments on variety of backdoor configurations provide the first holistic risk analysis of DMs and provide novel insights, such as showing the lack of generality in inference-time clipping defense.
unified backdoor attack framework
Limitations and Ethical Statements
Due to the limited page number, we will discuss further in Appendix C
10

---

8
Acknowledgment
The completion of this research could not have been finished without the support of Ming- Yu Chung, Shao-Wei Chen, and Yu-Rong Zhang. Thank Ming- Yu for careful verification of the derivation and detailed explanation for some complex theories. We would also like to express our gratefulness for Shao-Wei Chen, who provides impressive solutions of complicated SDEs and numerical solution of Navior-Stoke thermodynamic model. Finally, we also appreciate Yu-Rong for his insights of textual backdoor on the stable diffusion.
References
[1] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, S. Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. In ArXiv, 2022.
[2] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
[3] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. ICLR, 2022.
[4] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. ArXiv, 2022.
[5] Shoufa Chen, Peize Sun, Yihing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection.
In ArXiv, 2022.
[6] Weixin Chen, Dawn Song, and Bo Li. Trojdiff: Trojan attacks on diffusion models with diverse targets. In CVPR, 2023.
[7] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. 2023.
[8] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In JCML, 2020.
[9] Sheng-Yen Chou, Pin-Yu Chen, and Tsung-Y Yi Ho. How to backdoor diffusion models? In CVPR, 2023. [10] Giannis Daras, Mauricio Delbracio, Hossein Talcbi, Alexandros G. Dimakis, and Peyman Milanfar, Soft diffusion: Score matching for general corruptions. In ArXiv, 2022.
[1]] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. In NIPS, 2021.
[12] Laurent Dinh, Jascha Sohl- Sohl-Dicksteen, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017.
[13] Hadi Mohaghegh Dolatabadi, Sarah M. Erfani, and Christopher Leckie. Advflow: Inconspicuous black-box adversarial attacks using normalizing flows. In NIPS, 2020.
[14] Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Fric Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood importance weighting. In NIPS, 2019.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilihrium. In NIPS, 2017.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NIPS, 2020.
[17] Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. In JMLR, 2022.
[18] Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. In NIPS Workshop on Deep Generative
Models and Downstream Applications, 2021.
[19] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.
Video diffusion models. In NewrIPS, 2022.
[20] Edward Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu
Chen. Lora: Low-rank adaptation of large language models. 2021.
[21] Rongjie Huang. Max W. Lam, Jun Wang, Dan Su, Dong Yu. Yi Ren, and Zhou Zhao. Fastdiff: fast conditional diffusion model for high-quality speech synthesis. In IJCAI, 2022.
[22] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In ICML, 2022.
[23] Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts:
denoising diffuusinn model text-to-speech. ISCA, 2021.
[24] Yuming Jiang, Ziqi Huang, Xingang Pan, Chen Change Loy, and Ziwei Liu. Talk-to-edit: Fine-grained facial editing via dialog. In ICCV, 2021.
[25] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. In NIPS, 2022.

---

[26] Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: A diffusion model for text-1o-speech via classifier guidance. In ICML, 2022.
[27] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In NIPS, 2018.
[28] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. 2021.
[29] Zhifeng Kong. Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion
model for audio synthesis. In ICLR, 2021.
[30] Alex Krizhevsky. Learning multiple layers features from tiny images. 2009.
[31] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation. In ArXiv, 2022.
[32] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022.
[33] Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
[34] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NIPS, 2022.
[35] Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solvert+: Fast solver for guided sampling of diffuusion probabilistic models, In NIPS, 2022.
[36] Kangfu Mei and Vishal M. Patel. VIDM: video implicit diffusion models. CoRR, ahs/2212.00235, 2022.
[37] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE; towards photorealistic image generation and editing with -ext-guided diffusion models. In ICML, 2022.
[38] Tim Pearce. Tabish Rashid, Anssi Kanervisto, David Bignell, Mingtei Sun, Raluea Georgeseu, Sergio Val- carcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In CoRR, 2023.
[39] Justin N. M. Pinkney. Pokemon blip captions.
pokemon- blip- captions/ 2022.
[40] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov. Grad-tts: diffusion probabilistic model for text-to-speech In ICML, 2021.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. In ArXiv, 2022.
[42] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML,
2015.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2021.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. Compvis/stable diffusion v1-4. https: /taps:://uuggee https/////ugggggaceeee httpr//iinnaaaeeooooooo////a////
www.ddioonoo-------- 2022.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffuusion models. In CVPR, 2022.
[46] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic texl-to-image diffusion models with deep language understanding. In ArXiv, 2022.
[47] Tim Salimans, Ian J. Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.
[48] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
[49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5B: an open large-scale datasct for training next generation image-text models. In NIPS, 2022.
[50] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmanczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION open dataset of clip-filtered 400 million image-text pairs. NIPS Workshop, 2021.
[51] Jascha Sohl-Dickstein Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using noncquilibrium thermodynamics. In ICML, 2015.
[52] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[53] Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In NIPS, 2021.
[54] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NIPS, 2019.
12

---

[55]
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In NIPS, 2020.
[56]
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[57] Lukas Struppek. Dominik Hintersdorf, Kristian Kersting. Rickrolling the artist: Injecting invisible backdoors into text-guided image generation models. In ArXiv, 2022.
[58] Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class offline reinforcement learning. CoRR, 2022.
[59] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In ICLR, 2023.
[60] Richard 7hang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[61] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor-coorector framework for fast sampling of diffusion models. 2023.
13

---

A
Code Base
Our code is available on GitHub: http:///ithhb..ccoo//////allllal
B
Mathematical Derivation
B.1
Clean Diffusion Model via Numerical Reparametrization
Recall that we have defined the forward process (xx](xo) := ==V(á(t)xx. (+))),, [Tin Tmau] for general diffusion models, which is determined by the content scheduler a(t) : R + R and the noise scheduler B(t) R R. Note that to generate the random variable Xx, we can also express
with reparametrization X, = x())x= + (()) In the meantime, we've also mentioned the variational lower bound of the diffusion model as Eq. 17
(Xo) -E, (oggo((xo) (xo) Ey [Cr (XT, x
Σ Xxx1,x - Lo [x1, (17) 1m2
Denote X1,,,,) = (x xo) |I po (r X0) DKL(g(xr(xo)
pe(XT)), and Lo(x1,X0) lxopo(x))xi), where DxI(allp) = Sn ((x) log g(ar) is the KL- Divergence. Since Lt usually dominates the bound, we can ignore LT and Lo and focus on Dra (xx Xn) || po(xx-1|xx)). In Appendix BB..I, we will derive the clean conditional reversed transition q(x,_1Xx,,
p(z)
As for the learned reversed transition q X2), we will derive
it in Appendix B...2 Finally, combining these two parts, we will present the loss function of the
clean diffusion model Appendix B.1.
B.1.1 Clean Reversed Conditional Transition g(xx_1\x,,x0) xo)
Similar to the derivation of DDPM, we approximate reversed transition as ((xx-1(xx) x x(x,_1XX,,x)) We also define the clean reversed conditional transition as Eq. (18).
( | xo)= = ((H((xi,x)) (())),, px(xx, xo) = =(f)x: + b(t)xo
(18)
(1-1 and b(t)
To show that the temporal content and noise schedulers are a(t) ()
a(t-1)wi with Bayesian rule and Markovian property q(xx- Xo) g (x g(x;-1 1x0]
(13)((=))+
g( xo)
we can expand the reversed conditional transition o(x,-1|x,, xo) as Eq. 19 We also use an additional function C(x,, Xo) to absorb ineffective terms.
((xx-11x0))
=x(xx(xx-11x00)
(xx xx))
(x
kiXI-11 )²
(xi a(t - 1)xo)²
(xi 1 -á(t)x))))
x exp
( 1)
B2())
x²
2k
xi
x2-1-20 -2a(t 1)x0x, 1 1)x6
(xi (x--a(te) =a())x))))))
=exp
(2(1 ((1-)) 1)
(e)
2k
2ā(1 - 1)
exp
+
xo) xo)xx-1 C(xI,
(2 1)
( - 1)
(19)
Thus, a(t) and b(t) can be derived as Eq.
20
he
+ ā(t 1)
k2
1
a(t)x + b(1)x0 =
" xx
xo)/
+
8a(L - 1)
wi
B2
ā(l 1)
w B2 11) 1)
x, +
x()
wi
8e 1)
k3 k3,52 - 1) + w
(20)
kt. 1)
á(t 1 1)w?
+
k3 13 1) + wm
k2BB(11 - 1) mm
14

---

After comparing the coefficients, we can get a(t) 6 ((-1) and b(t) 6(t Gou Ioo?
(t- Recall
(t 1)+
that based on the definition of the forward process ((x]) xo) := (á(t)xo, (())), we can obtain the reparametrization: Xn
(3 (x+ 3(()ez). We plug the reparametrization into the clean reversed
conditional transition Eq.
kg 1)ā(t) ā(t 1)
a(t 1)uc
3(L)
(x xo)
(21)
((t)(k) + wf)
R2 Ry 1) + w/ a(t)
B.1.2 Learned Clean Reversed Conditional Transition (x Xo)
To train a diffusion model that can approximate the clean reversed conditional transition, we define a clean reversed transition po xx-1 xx)) learned by trainable parameters as Eq. (22)
Po po == (x
(x Me xo: a). (()I)
(22)
With similar logic in Eq. derive pe (x,, Xo. t) Eq.
22 and replacing e with a learned diffusion model cx(x,, t), we can also
kB (t 1)
a(t - 1)w7
(1
(2t0)(x= (x: - -(())c 1))
no (xe, x0, t)
k2 1) wi
X
1t 1) w0
a(t))
(t
á(t 1
1
á(t 1)w1
B(t)
ea (x., !)
kt 1) wi
12 1) wif ā(1)
k2 1) + wg a(l)
kt 1)6(() â(t (())((BB(t111))
1)w1 á(t 1)w?
B(t)
co (x,,t)
k7,82 1) + wi ā(t)
(23)
B.1.3
Loss Function of Clean Diffusion Models
The KL-divergence loss of the reversed transition can be simplified as Eq. matching approximation of the KL-divergence.
24 which uses mean-
Dxu (q(x:-1 Xx, xo)l x: (xx))
<x|, xo) [x, xo. xo. 1)1
a(t 1)
B(1)
-(-
á(t 1)w?
B(t)
(x f))
(24)
k (t 1) + uo ā(t)
B3 1) + wi a(t)
x ||e, - co (xx, 1
Thus, we can finally write down the clean loss function Eq. 25 with reparametrization xx(x, e) a(1)x + (()e, 6 6~ N(0,
C.(x,t,)) = ||e co (x.
(25)
B.2
Backdoor Diffusion Model via Numerical Reparametrization
This section will further extend the derivation of the clean diffusion models in Appendix B. and derive the backdoor reversed conditional transition q(x; |x|, x() and the backdoor loss function in Appendix B.2.1
B.2.1 Backdoor Reversed Conditional Transition y(*;_, |x,, x6)
Recall the definition of the backdoor reversed conditional transition in Eq. 26 For clarity, We mark the coefficients of the r as red.
q(xi_ x x() (1) x6), 5 ( (t)T), (l(x) a(t)x/ c(t)r + b(t)x1
(26)
We firstly show that the temporal content, noise, and correction schedulers are a(t)
ky (1-1)
17 ww
6(t I and c(t)
-) Thus, first of all, we can expand the reversed 1-
b(t)
15

---

conditional transition (xx( [x,x6, x5) as Eq. (27). To absorb the ineffective terms, we introduce an additional function C" (x(x, x6). We mark the coefficients of the r as red.
2(x(_ x()
= =q(x]|x; =[(x[[x/_1xx)) q(xi-1|x6)
(xx[1x))
1 (x] - ki her)² (xi-1 â(t 1)x6 p(( 1)r)² B2(t -1) 1)
exp
2
(xi á(t))x ((t))))))
(1)
1 (xi - -Kxx 2(x; - KIN 1)ter + 11212
exp
2
(x/_ a(t 1)xx)² -(xx) 1 aett 1)xi)d(t 1)r p(t 1)2r2
B2(L -1)
(27)
(x) a(()xb ((C))))))
(2())
1 (xi - kxx/-112 + (xX-_ - -ā(t - -1)x6)
2(xl - kixi 1iher
exp
2
wi
(2 1 1)
wi
2(x
alt 1)x6)0(1 - 1)r (4(-a(()x) ((OO))))
(2(--1) 1)
BP())
1
Ba(1- 111-1) 1) xxf= - 2( kt Xx +
1
a(t 1) X
exp
2
wi
+
wp
Be(L 1)
p(t 1)
kehe )r)x( + C'(xx, x6o)
(t - 1)
wf
Thus, the content, noise, and correction schedulers a(). b((). and c(l) can be derived as Eq. 28 We mark the coefficients of the r as red.
ki m?
á(l 1) x +
p(L 1) (t
kehi w/
k2 )r/
att)x + c(t)r + b(t)xb
x
(t
(t
T x
h1
â(t
x/
p(t 1)
kehe W
w 3 1)
32 - 1)
B2(1 1)
k. 32 - m
a(1 1)ws
X
x/
13 1t - 1) + wi
12 - 1) + w/
(28)
w p(t 1)
k2h,,32(4 1)
+
k kt 1) + mo
1t - 1) + 20?
kr (t 1)
á(t ?
x
k3 - 1) + w/f
k - 1) + wi
1 p( 1) 12 3 (t 1)
13 1 (t 1) + w²
8 (t- b(t)
6(t 1)ae"
Thus, after comparing with Eq. 266, we can get a(t) =
and
k7 (1
c(t) "/
B.3 Backdoor Reversed SDE and ODE
In this section, we will show how to convert the backdoor reversed transition q(x |x,) to a reversed-
time SDE with arbitrary stochasticity by q(x(_11 | x)). In the first section, referring to [56), we
introduce Lemmal aa a tool for the conversion between SDE and ODE. Secondly, in Appendix B.3.
and Appendix B.2.2 we will convert the backdoor and learned reversed transition: q(x/ xx] and
( (x,) into the backdoor and learned reversed SDE. In the last section Appendix B.3.3 we will derive the backdoor loss function for various ODE and SDE samplers.
Lemma For a first-order differentiable function f IRdd x IR. - Rdd, second-order differentiable
function g IR + R. and randomness indicator [0, 1), the SDE dx, = f(xe, t)dt + g(t)dd
16

---

and dx,- = [f(x,,t) 2 log + VS describe the same stochastie process X₁ R t E [ with the marginal probability p(x:). where w E I& is the reverse Wiener process.
Proof B.1 For the clarity of the notation, we denote p(x+) as p(x, 1), follow the Fokker-Plane equation (56]. we can convert the SDE dx,- = f(t)x,dt + g(t)dw to a partial differential equation Eq. (29) and Eq. (30).
0 0e ((x =
0 Oxi
8²
(fi(x,) 1) ()x +
(g) p(xx 1))
0x,0x;
a
(f(x, 1) p0.pp(x110 ()) + 5 d
0²
(g p(x,1))
1= txi
0x,0x;
8²
(g)(t) p(xx t))
(29)
2
,x,0x;
1=1 j=1
0
(f(x, -p(, t)) + 5 d Σ 0x
0² (0²(t) ((x,)))
tx
dx. (t) p(x.1) v {xp(x,)) t))
2
t(x,t)
To simplify the aacond-order partial derivative, in the Eq. log p(x, t) x))x)(x,t) vxp(x.))
330.. we apply the log-derivative trick:
(xx1)
(f(x.t) p)x
(g) (t)x,))((x,
Fx:
0x
1
0
(1g2 Vx lug p(xx())))
ox,
(30)
((f(x,)) () 1- c (s²(1) Vx log (i) p(x,)) ())
0x;
2
8²
0x,dx, (t)(1)-p(x,)) p(x,t))
2
Thus, we can convert the above results back to an SDE with the Yokker-Planck equation with randomness indicator Ç in Eq. (31. We can see will reduce to an ODE while Ç = and SDE while
=-
dx. f( 1- ()((t)x log + (Cg(t)d
(31)
B.3.1 Backdoor Reversed SDE with Arbitrary Stochasticity
Since ((x)-1xx) Re q(x; X x6). we can replace Xy of Eq. (26) with reparametrization Xy =
xi ((-A(t)r-(())- from Eq. 26 Note that since the marginal distribution g(xx) follows Gaussian
a(1)
distribution, we replace the e, with the normalized conditional score function - -B(1)V log ((x(1x)) as a kind of reparametrization trick.
X4 p())r--------rrr)))))- B(1)(- log (x(((x)) c(t)r + s(t)e,, e ~ I(O,I)
x a(t)x/ + b(()
a(t)
(32)
b(t)
>xl (e(() b(1)((1) b(()(())
(-3(1) (-))(()((((()))))) logg(x((>6)) + s(1)e
(a(()
)r-
a(t)
a(1) a(l)
Then, based on Eq.
32 approximate the dynamic dx/, with Taylor expansion Eq.
33
dxi = [(a(t) + b(t)
1)x1 + (c(1) b(1)p(t) )r b(t)B()) ((1),(0) (-3(0) log 10gg((11x6))) + s()dd a(I)
a(t)
a(t)
(33)
17

---

With proper reorganization, we can express the SDE Eq.
33 as Eq. (34
H(t)
dxi [F())x] C (4) log logg((((x))
(²(1) r)] + s(f
(34)
We denote F(t) a(t) + 610
1, (t) c(t) 1ON0 and G(t) - / Since also a(t)
a(r) V((())
(())
assume the forward process g(xx|xx) and a( (x)) are diffusion processes, thus the coefficient s()) can be derived as (t) = ((())( = v b(t)
a(t) B(t). Then, considering different stochasticity of
various samplers, we can apply Lemma and introduce an additional stochasticity indicator ( E [0, 1]
in Eq. 335
H(t) r)] dt s(t)dw
dx = [F()x] c (xgg(x)(x()
(2(L)
= [F(()x] -)(0)-B(()) (xgg(x))xx))
(2(1) r) 1- --52((1) logg(x((x()) + ([s(t)dw
H(t)
[[(())]
1+ 1+5 ()) logg(x/1x6) (1 + ())
(H
r) dt G(t) cā())dw
2
Backdoor Score Function
(35)
B.3.2 Learned Reversed SDE with Arbitrary Stochasticity
x. (x-)())
Since ((x) (xx, can replace xo of Eq. (26 with xo
a(l)
which is derived from the reparametrization of the forward process x, - á())xo + 3(1)ez with the replacement et with eØ(X,, t).
X a(t)x: b(t) X2 ((t))o(xI,)) 1) s(t)e 6 I(0,
6(t)
(36)
= (a(t) + b(t)
b(t)ß(t)
@(L) Xx+
a(l))
co(x,, 1) + s())cc
Then, according to Eq. 36 approximate the dynamic dx, with Taylor expansion as Eq.
37
dx. [(a(f) + b(t) 1)x,
b(t))(()
co t)] dt + a (f)
(37)
a(t)
a(t)
With proper reorganization, we can express the SDE Eq. (37) with F(t) a(t) +
b(t)
- 1,
a(1)
G(t) / " and s(t) = = b00) (t) Eq. (38
a(e)
dx. = [F(t)x - (t) t)) dt a(f)dw
(38)
Then, we also consider arbitrary stochasticity and introduce an additional stochasticity indicator < [0, 1] with Lemma] |
As we use a diffusion model eg as an approximation for the normalized
score function: <o(x,,1 () 3(1) x, log gxx)), we can derive the learned reversed SDE with arbitrary stochasticity in Eq.
dx, [F())x 1+ (1)eoox +) C(00
(39)
2
B.3.3 Loss Function of the Backdoor Diffusion Models
Based on the above results, Eq. (39) as Eq. (40). The loss function Eq. (40) is also known as denoising weannng--cor---aate metre-natching loss
formulate score-matching problem based on Eq. (35 and
[54]. which is a surrogate of the wcore-matching problem since the score function Vx; log (xx)) is intractable.
(H)
Ext.x6 (()((()) log (x//(x(/(x)) - (1 (t)
r) (x)
(40)
(1(
x||< (1 + (t) r(xo, g) - co (x( r(xo, g). e), 1)
Thus, we can finally write down the backdoor loss function Eq.
2H1(t)
= ||c- f,1
(1 ()² r(x,g) co(x/(t)r(xxx)))o..o...
(41)
18

---

B.4 The Derivation of Conditional Diffusion Models
We will expand our framework to conditional generation in this section. In Appendix B.4. we will start with the negative-log likelihood (NLL) and derive the variational lower bound (VLBO). Next, in Appendix [.4.2. we decompose the VLBO into three components and focus on the most important one. in Appendix B.4.3] based previous sections, will derive the clean loss function for the conditional diffusion models. The last section Appendix B.4.4 will combine the results of [.2
(B.4.2
Appendix B.1 and Appendix [.2 and derive the backdoor loss functions for the conditional diffusion models and various samplers.
B.4.1 Conditional Negative Log Likelihood (NLL)
To train conditional diffusion model (x c), will optimize the joint probability learned by the model arg ming log po (Xu, c). We denote с as the condition, which can be prompt embedding for the text-to-image generation, and the Dx.r is the domain random vectors x....,, XT, X, E Rd
[ T), S Therefore, we can derive the conditional variational lower bound LVLB as
Eq.
log po (xu, c) (g((n)) llog px c)]
-xg log / po XX, C))d)x dxr]
Dayr
{ q(xi,..., Xr Xo) po po .... XT, dxi dxt]
(x llog
((x),..... xT x0)
(42)
x1...x
Do XIc)
= Bu((k)) log Egex (y(l)...)Tl))))
((),,..,, X (x)
$ -Bg( log Do Px(
Xr,
LVLB
(x)......... X7 xx)
B.4.2 Conditional Variational Lower Bound (VLBO)
In this section, we will further decompose the VLBO Eq. 42 and show that minimizing the KL-divergence ( Xx, Xo) (pe simplicity, we denote xx((x.....xxr) as Eyg With Markovian assumption, the latent x, at the timestep only depends on the previous latent x;-1 and the condition
c)) is our main objective Eq. (43). For the
LÝLB = -B, [log
Po (No xT, c)
Hy, [log g(x1, Xx Xx)
g(X1, ... Xr x
Po ... Xrc)
IT_ ((xx1x(-1)
[ po (XT, c) I1 Po (x 1x,
((x1)xx-1)
= [ log (xT, c) +
log pa(xi (xi_ Xc., c)
T
log 7( [x, (xx,1x1-1) xx-1))
loog g(x (200 po (x, e)
[ log (xr, +
px X1,
log 8(xx X4, xo)
((x i1x)) ) log 9(31 (x0) pol (xo (x1, }o
x(x,) (20)
[ log pe (X., c)+ +
po X1, c)
1=2
log ((x, Xx, Xo)
log ((x) + log ((x]|xx) po (xxol c) )
= B log (x, c)
+
po (X:- Xx, c)
g(X:-1|Xu)
log g(xi |x1, X0)
q(xr| xo) ((x1|x0)
+ log ((x1|xa) po cx)
= B, log po (xr, c) +
Σ
loo
Po (x |Xe, c)
= =Bg ((x)
q(x1 1 Xx) Po 1
log
log
po (xT,c) c)
T =E,[DKt (q(x7 xo)lpoxxx c)) + Σ t=2
Dx xo) (x c)) -xogoo c)]
= Xo.
Σ c X1,,x,_1,000 -28 xx1,
(43)
19

---

B.4.3 Clean Loss Function for the Conditional Diffusion Models
We define the learned reversed transition pa (x c) as Eq. (44
(x (xx) := 1x Xo, (()))
(44)
We plug in a conditional diffusion model eo (xx, t, c) to replace the unconditional diffusion model eg eg(X,, t).
kr 1) X1 + ást 1)w2
(1 (0/7((x (x: -(t)) ā(t)
Pa (Xt, No, t,
kt - 1) m/ k2 - 1) + w/?
(45)
(t) + á(t - 1)
Ô(t - 1)
B(t)
X1
ca (x),,),
(1) 1) wif)
wi a(t)
As result, we use mean-matching as an approximation of the KL-divergence loss with Eq. (46).
DKL
c)) x (xx((e, - pee(,, Xo, c)ll x |lee co(x,,t,c)]]
(46)
Finally, we can reorganize the Eq. (46) as Eq. 47 diffusion models.
which is the clean loss function for the conditional
c. (x,t,e, = ce (xx
(47)
B.4.4 Loss Function of the Backdoor Conditional Diffusion Models
Based on the above results, we can further derive the learned conditional reversed SDE Eq. (48). while the backdoor remains the same Eq. (35), which is caused by the identical backdoor reversed transition q(x(_1xx{ 11 xx) of the KL-divergence loss.
1+ 1 (()ex)xxx))) C(t) cā(l)dw
dx: [F))):
(48)
2
According to the above results, can formulate image-trigger backdoor loss function based on Eq. (35) and Eq. (48) Eq. (49). The loss function Eq. (49) also known denoising-scon
matching loss [54), which is a surrogate of the score- sotee-mathhing problem since the score function Xx; log g( (x() is intractable. Here we denote the reparametrization x/(x,r, c) = d((t)x + p(t)r (()).
Ex x (1l(- log (x(1()))) 2H(() (1+ r) co(xx.1,c))1
(49)
(t7 xx| (1+ (t) r(xu, <x(xx((x), r(xo. g), t),,,)l))
Thus, we can finally write down the image-as-trigger backdoor loss function Eq. 50p for the conditional diffusion models.
2H(() (1+ ()) r(y, (l)
(50)
C Limitations and Ethical Statements
Although VillanDiffusion works well with hot diffusion models, framework does not cover all kinds diffusion models, like Cold Diffiusion [ and Soft Diffusion [0], etc. However, even though framework does not cover the deterministic generative process, we believe our method can still extend to more diffusion models, and we also call for the engagement of community to explore more advanced and universal attack diffusion models.
On the other hand, although our framework aims to improve the robustness diffusion models, we acknowledge the possibility that our findings the weaknesses of diffusion models might be misused. Nevertheless, we believe our red-teaming effort can contribute the development of robust diffusion models.
20

---

D Additional Experiments
D.1 Backdoor Attacks on DDPM with CIFAR1O Dataset
We will present experimental results for more backdoor trigger- pairs and samplers, including the LMSD sampler, which is implemented by the authors of EDM [251, in Fig. / The results of the ANCESTRAL sampler come from p
We also provide detailed numerical results in Appendix E.
Pobsc Fire 3552
Pobor Firm vo HOC
2-
(a) (Stop Sign, NoShift) FID
(b) (Stop Sign, NoShift) MSE
(c) (Grey Box, NoShift)
(d) (Grey Box, NoShift) MSE
FID
DON.. Stop Sgen Stits Pallen Nar
200N (Oney Stitt
P6:01 Rata RD
2
(c) (Stop Sign, Shift) FID (f) (Stop Sign, Shift) MSE (g) (Grey Box, Shift) FID (h) (Grey Box, Shift) MSE
Potien RD
1950 Sgn Poioc
se' hara eo RTD
000% Kany
Pobice
(i) (Stop Sign, Corner)
(i (Stop Sign, Corner) MSE
(k) (Grey Box, Corner) FID
(I) (Grey Box, Corner) MSE
FID zoeng Precas Marl))
Dow,. step dan Cheer
FEOMA CONER poing ROY Morr
z0nM granus, Grey S00 2ho0
(m) (Stop Sign, Shoe) FID (n) (Stop Sign, Shoe) MSE (o) (Grey Box, Shoe) FID (p) (Grey Box, Shoe) MSE
J089 CHINEE 2wap Agti 22
DCPP IAAA Stap
Pobcc MOD 213
conn, CRAA2E IGRD
(q) (Stop Sign, Hat) FID (r) (Stop Sign, Hat) MSE (s) (Grey Box, Hat) FID (t) (Grey Box, Hat) MSE Figure 7: FID and MSE scores of various samplers and poison rates for DDPM [6] and the CIFAR10 dataset. We express trigger-target pairs as (trigger, target).
21

---

D.2 Backdoor Attacks on DDPM with Ce Dataset
We evaluate our method with more samplers and backdoor trigger-target pairs: (Stop Sign, Hat) and (Eyeglasses, Cat) in Fig. 8 Note that the results of the ANCESTRAL sampler come from Please check numerical results in Appendix E.2
1OWN LANDAING pu REGAL AAY LLPPA
DOW. Egn thad
Pobutt Tick va
(a) (Eyeglasses, Cat) FID (b) (Eyeglasses, Cat) MSE (c) (Stop Sign, Hat) FID (d) (Stop Sign, Hat) MSE
Figure 8: FID and MSE scores of various samplers and poison rates for the DDPM [16 and the CelebA-HQ dataset. We express trigger-target pairs as (trigger, target).
D.3 Backdoor Attacks on Latent Diffusion Models (LDM)
The pre-trained latent diffusion models (LDM) 45 are trained on CelebA-HQ with 512 x 512 resolution and 64 x 64 latent space. We fine-tune them with learning rate 2e-4 and batch size 16 for 2000 epochs. We examine our method with trigger-target pair: (Eyeglasses, Cat) and (Stop Sign, Hat) and illustrate the FID and MSE score in Fig. As the Fig. 9 shows, the LDM can be backdoored successfully for the trigger-target pairs: (Stop Sign, Hat) with 70% poison rate. Mcanwhile, for the ttagee-targettee pair: (Eye Glasses, Cat) and 90% poison rate, the FID scores only slightly increase by about 7.2% at most. As for the trigger-target pair: (Stop Sign, Hat), although the FID raises higher than (Eye Glasses, Cat), believe longer training can enhance their utility. Please check Appendix E.3 for the numerical results of the experiments.
Prisen ao
Police
Film
"oben
(a) (Eyeglasses, Cat) FID (b) (Eyeglasses, Cat) MSE (c) (Stop Sign, Hat) FID (d) (Stop Sign, Hat) MSE
Figure 9: FID and MSE scores of various samplers and poison rates for the latent diffusion model
(LDM) 45 and the CelebA-HQ dataset. We express trigger-target pairs as (trigger, target).
D.4 Backdoor Attacks on Score-Based Models
We trained the score-based model: NCSN 56
on the CIFARIO dataset with the same model architecture as the DDPM [6] by ourselves for 800 epochs and set the learning rate as le-4 and batch size as 128. The FID score of the clean model generated by predictor-correcti
54 55
samplers (SCORE-SDE-VE VE (566) for the variance explode models (56) is about 10.87. For the backdoor, we fine-tune the pre-trained model with the learning rate 2c-5 and batch size 128 for 46875 steps. To enhance the backdoor specificity and utility, we augment Gaussian noise into the training dataset, which means the poisoned image r will be replaced by a pure trigger g. The augmentation can let the model learn to activate the backdoor even if there are no context images. We present our results in Fig. an can see with 70% augment rate, our method can achieve 70% attack success rate based on Fig. 10c as the FID score increases by 12.7%. Note that the augment rate is computed by the number of augmented Gaussian noises / the size of the original training dataset. We also present detailed numerical results in Appendix E.4
22

---

NCSN, CIFARIO, 15tp Sign Hott
NCSN, CIFAR1O, (5top Sign: HM0 Polson
NCSK, CIRAR10, (50op Sign Hett MSE
3079
(a) (Stop Sign, Hat) FID
(b) (Stop Sign, Hat) MSE
(c) (Stop Sign, Hat) MSE Threshold
Figure 10: FID and MSE scores of various samplers and poison rates for the score-based model (NCSN) [54 55 56) and the CIFARIO dataset. We express figger-target pairs as (trigger, target).
We also denote the augment rate: number augmented Gaussian noise/dataset size as "AR" in the legend.
D.5 Evaluation on DDIM with Various Randomness
We also conducted experiments on BadDiffusion and VillanDiffusion with different samplers. The numerical results are presented in Appendix [E88, We found that BadDiffusion is only effective in SDE samplers. When DDIM goes down, which means the sampler becomes more likely an ODE, the MSE of VillanDiffusion trained for ODE samplers would decrease, but BadDiffusion would increase. Thus, it provides empirical evidence that the randomness of the samplers is the key factor causing the poor performance BadDiffusion. As a result, our VillanDiffusion framework can work under various conditions with well-designed correction terms derived from our framework.
D.6 Comparison Between BadDiffusion and VillanDiffusion on CIFAR10
We conduct an experiment to evaluate BadDiffusion and VillanDiffusion (with ( = 0) on ODE samplers, including UniPC [61]. DPM-Solver [34] (35,, DDIM [52), and PNDM 32] We also present the detailed numerical results in Appendix E.9 Overall, we can see that BadDiffusion performs worse than VillanDiffusion. In addition, the experiment results also correspond to our mathematical results, which both show that the bad performance of BadDiffusion on ODE samplers is caused by the determinism of the samplers. Once take the randomness hyperparameter into account, can derive an effective backdoor loss function for the attack.
D.7 Inference-Time Clipping Defense
We evaluate the inference-time clipping defense on the CIFARIO dataset with triggers: Grey Box and Stop Sign and targets: NoShift, Shift, Corner, Shoe, and Hat in Fig. []] The results of the ANCESTRAL sampler are from (9. We can see that inference-time clipping still not effective for most ODE samplers. Please check Appendix (E6 for the numerical results.
D.8 VillanDiffusion on the Inpaint Tasks
Similar to [9], we also evaluate our method on the inpainting tasks with various samplers. We design 3 kinds of different corruptions: Blur, Line, Box. Blur means we add Gaussian noise [0, 0.3) to corrupt the images. Line and Box mean we crop part of the image and ask the diffusion models to recover the missing area. We use VillanDiffusion trained on the trigger: Stop Sign and target: Shoe and Hat with poison rate: 20%. During inpainting, we apply UniPC [61]. DEIS [59], DPM Solver (34, and DPM Solver++ 35 samplers with 50 sampling steps. evaluate the recovery quality, we generate 1024 images and use LPIPS 660 score to measure the similarity between the covered images and ground-truth images. We illustrate our results in Fig. T2 We can see our method achieves both high utility and high specificity. The detailed numerical results presented in Appendix E.7
23

---

DOPM. CIARLO Delvas
DOP% CHANOO Cataree: Safti
Folgae
(a) (Stop Sign, NoShift) FID
(b) (Stop Sign, NoShift) MSE
(c) (Grey Box. NoShift)
(d) (Grey Box, NoShift)
FID
MSE Fobor faom
Pebove Fate
(c) (Stop Sign, Shift) FID (f) (Stop Sign, Shift) MSE (g) (Grey Box, Shift) FID (h) (Grey Box, Shift) MSE
C00% CBAAGE Detarae:
Prist
Poice
(i) (Stop Sign, Corner) FID Fatsan dasecall
ii (Stop Sign, Corner MSE bate
(k) (Grey Box, Comer) FID Palsan
()) (Grey Box, Corner) MSE Prtiaal Mare
(m) (Stop Sign, Shoe) FID (n) (Stop Sign, Shoe) MSE (o) (Grey Box, Shoe) FID (p) (Grey Box, Shoe) MSE
Teloo
Pobir Reterv
(q) (Stop Sign, Hat) FID (1) (Stop Siga, Hat) MSE (s) (Grey Box, Hat) FID (1) (Grey Box, Hat) MSE
Figure 11: FID and MSE scores of various samplers and poison rates with inference-time defense 9..
We evaluate the defense on the DDPM and the CIFARIO dataset and express trigger-target pairs as (trigger, target).
24

---

Filtco SEONICE LUTT
TAROA SPOP DE Rata
Peimon Thoit FLD
COPN EEIAEE 12we Shad
(a) (Stop Sign,Hat) LPIPS (b) (Stop Hat (c) (Stop Sign,Shoe) LPIPS (d) (Stop Sign,Shoe) MSE
Figure 12: LPIPS and MSE scores of various samplers and poison rates for the 3 kinds of inpainting tasks: Blur, Line, and Box. The backdoor model is DDPM trained on the CIFARIO dataset. We express trigger-targe pairs as (trigger, target).
E Numerical Results
E.1 Backdoor Attacks on DDPM with CIFAR1O Dataset
We present the numerical results of the trigger: Stop Sign and targets: NoShift, Shift, Corner, Shoe, and Hat in Table 2 Table 3 Table 4 Table and Table respectivvll As for trigger Grey Box. also show the results for the targets: NoShift, Shift, Corner, Shoe, and Hat in Table 7 Table g T
Table 10 and Table
E.2 Backdoor Attacks on DDPM with CelebA-HQ Dataset
We show the numerical results for the trigger-target pairs: (Eyeglasses, Cat) and (Stop Sign, Hat) in Table 12 and Table 13 respectively.
E.3 Backdoor Attacks on Latent Diffusion Models (LDM)
We show the experiment results of the trigger-target pair: (Eye Glasses, Cat) in Table 14 and (Stop
Sign, Hat) in Table
E.4 Backdoor Attacks Score-Baser Models
We provide the numerical results for the trigger-target pair: (Stop Sign, Hat) in the Table [6
E.5
Caption-Trigger Backdoor Attacks Text-to-Image DMs
We will present the numerical results of the Pokemon Caption dataset in Table 17 and Table 18 For the CelebA-HQ-Dialog dataset, we will show them in Table 19 and Table 20
E.6 Inference-Time Clipping Defense
For the trigger Stop Sign, we present the numerical results of the inference-time clipping defense with targets: NoShift, Shift, Corner, Shoe, and Hat in Table 21] Table 22] Table 23 Table 24 and Table 2 reseepcccy. As for the trigger Grey Box, we also show our results of the targets: NoShift, Shift, Corner, Shoe, and Hat in Table 26 Table 27 Table 28
Table 29 and Table 33ble respectivell.
E.7 VillanDiffusion the Inpaint Tasks
For the trigger Stop Sign, we present the numerical results of the inpainting tasks: Blur, Line, and Box with targets Hat and Shoe in Table 32 and Table respectively.
E.8 BadDiflusion and VillanDiffusion on CIFAR10 with Different Randomness
For the trigger Stop Sign, we present the numerical results of various randomness with targets: Hat and Shoe in Table 33 respectively.
25

---

E.9
Comparison Between BadDiffusion and VillanDiffusion on CIFAR10
For the trigger Stop Sign, we show the numerical results of the comparison between BadDiffusion and DillanDiffusion in [able
26

---

Table 2: DDPM backdoor on CIFARIO Dataset with Trigger: Stop Sign, target: No Shift
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
11.52
8.09
7.62
7.97
7.46
7.68
7.38
7.22
ANCESTRAL
MSE
1,48E-1
6,81E-2
9.47E-3
2.35E-4
5,59E-6
4.19E-6
3.96E-6
3.80E-6
SSIM
6.84E-4
4.35E-1
9.18E-1
9.97E-1
9.99E-1
9.98E-1
9.98E-1
9.98E-1
FID
11.15
9.18
9.07
9.18
9.37
9.76
10.28
11.53
UNIPC
MSE
1,48E-1
4.76E-2
9.37E-3
1.30E-4
8.05E-5
2.27E-4
3.56E-5
3.24E-5
SSIM
8.13E-4
5.27E-1
8.57E-1
9.76E-1
9.81E-1
9.74E-1
9.86E-1
9.84F-1
FID
11.15
9.18
9.07
9.07
9.37
9.76
10.28
11.53
DPM. 02
MSE
1,8E-1
4.76E-2
9.37E-3
1,24E44
8.05E-5
2.27E-4
3.56E-5
3.24E-5
SSIM
8.13E-4
5.27E-1
8.57E-1
9.80E-1
9.81E-1
9.74E-1
9.86E-1
9.84E-1
FID
11.15
9.18
9.07
9.18
9.37
9.57
10.28
11.53
DPM. 03
MSE
1.48-11
4.76E-2
9.37E-3
1.30E-4
8.05E-5
7.48E-5
3.56E-5
3,24E-5
SSIM
8.13E-4
5.27E-1
8.57E-1
9.76E-1
9.81E-1
9.80E-1
9.86E-1
9,84E-1
FID
11.15
9.18
9.07
9.18
9.37
9.76
10.28
11.53
DPM++. 02
MSE
1,48E-1
4.76E-2
9.37E-3
1.30E-4
8.05E-5
2.27F-4
3.56E-5
3.24E-5
SSIM
8.13E-4
5.27E-1
8.57E-1
9.76E-1
9.81E-1
9.74E-1
9.86E-1
9.84E-1
FID
11.15
9.18
9.07
9.07
9.37
9.73
10.28
11.53
DPM++. 03
MSE
1.48E11
4.76E-2
9.37E-3
1.24E-4
8.05E-5
1.99E-4
3.56E-5
3.24E-5
SSIM
8.13E-4
5,27E-1
8.57E-1
9.80E-1
9.81E-1
9.76E-1
9.86B-1
9.84E-1
FID
11.15
9.18
9.07
9.07
9.37
9.57
10.28
11.53
DEIS
MSE
1,88-1
4.76E-2
9.37B-3
1.28E-4
8.05E-5
7,48B-5
3.56B-5
3,24E-5
SSIM
8.13E-4
5.27E-1
8.57E-1
9.76E-1
9.81E-1
9.80E-1
9.86E-1
9,84E-1
FID
16.39
10.95
10.71
10.70
11.16
11.32
12.40
14.43
DDIM
MSE
1.48E11
7.14E-2
2.47E-2
6,84E-4
4.95E-6
3.70E-6
3.58E-6
3,51E-6
SSIM
8.92E-4
3.74E-1
7.63E-1
9.92F-1
1.00E+0
9,99E-1
9.99E-1
9.99F-1
FID
12.14
7.24
7.13
7.25
7.42
7.77
8.18
9.83
PNDM
MSE
1.48E-1
6.55E-2
1.97E-2
1.23E-4
3.74E-5
3.25E-5
2.86E-5
2.76E-5
SSIM
8.23E-4
4.11E-1
7.91E-1
9,81E-1
9,83E-1
9,83E-1
9,83E-1
9,83E-1
FID
12.14
7.24
7.13
7.25
7.42
7.60
8.18
9.83
HEUN
MSE
1,88-1
6,55E-2
1.97E-2
1.23E-4
3.74E-5
3.16E-5
2.86E-5
2.76E-5
SSIM
8.23E-4
4.11E-1
7.91E-1
9.81E-1
9.83E-1
9.83E-1
9.83B-1
9.83E-1
FID
12.14
7.24
7.13
7.24
7.42
7.47
8.18
9.83
LMSD
MSE
1.48E-1
6,55E-2
1.97E-2
5,02E-4
3.74E-5
3.23E-5
2.86E-5
2.76E-5
SSIM
8.23E-4
4.11E-1
7.91E-1
9,76E-1
9.83E-1
9.83E-1
9.83E-1
9,83E-1
27

---

Table 3: DDPM backdoor on CIFARIO Dataset with Trigger: Stop Sign, target: Shift
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
11.16
8.21
7.42
7.68
7.76
8.02
7.42
7.72
ANCESTRAL
MSE
1,48E-1
5.68E-2
5.91E-3
8.96E-5
6.73E-6
4.23E-6
3.96E-6
3.80E-6
SSIM
4.24E-4
5.73E-1
9.56E-1
9.99E-1
9.99E-1
9.99E-1
9.99E-1
9.99E-1
FID
11.15
8.92
9.18
8.98
9.36
9.39
10.37
11.89
UNIPC
MSE
1,48E-1
5.92E-2
4.37E-4
1.37E-4
2.58E-4
1.66E-4
4.35E-5
7.59E-5
SSIM
4,25E-4
4.92E-1
9.70E-1
9.85E-1
9.81E-1
9.86E-1
9.88E-1
9.89E-1
FID
11.15
8.92
9.18
8.98
9.36
9.39
10.37
11.89
DPM. 02
MSE
1,8E-1
5.92E-2
4.37E-4
1.37E-4
2.58E-4
1.66E-4
4.35E-5
7.59E-5
SSIM
4.25E-4
4.92E-1
9.70E-1
9.85E-1
9.81E-1
9.86E-1
9.88E-1
9,89E-1
FID
11.15
8.92
9.18
8.98
9.36
9.39
10.37
11.89
DPM. 03
MSE
1.48E11
5,92E-2
4.37E-4
1.37E-4
2.58E-4
1.66E-4
4.35E-5
7.59E-5
SSIM
4.25E-4
4.92E-1
9.70E-1
9.85E-1
9,81E-1
9.86E-1
9.88E-1
9.89E-1
FID
11.15
8.92
9.18
8.98
9.36
9.39
10.37
11.89
DPM++. 02
MSE
1.48E1
5.92E-2
4.37E-4
1.37E-4
2.58E-4
1.66E-4
4.35B-5
7.59F-5
SSIM
4.25E-4
4.92E-1
9.70E-1
9.85E-1
9.81E-1
9.86E-1
9.88E-1
9.89E-1
FID
11.15
8.92
9.18
8.98
9.36
9.39
10.37
11.89
DPM++. 03
MSE
1.48E11
5.92E-2
4.37E-4
1.37E-4
2.58E-4
1.66E-4
4.35E-5
7.59E-5
SSIM
4.25E-4
4.92E-1
9.70E-1
9.85B-1
9.81E-1
9.86E-1
9.88E-1
9,89E-1
FID
11.15
8.92
9.18
8.98
9.36
9.39
10.37
11.89
DEIS
MSE
1,88-1
5.92E-2
4.37E-4
1.37E-4
2.58E-4
1.66E-4
4.35E-5
7,59E-5
SSIM
4.25E-4
4.92E-1
9.70E-1
9.85E-1
9,81E-1
9.86E-1
9.88E-1
9.89E-1
FID
16.39
10.70
10.85
11.01
11.29
11.48
12.37
14.60
DDIM
MSE
1.48-11
8,25E-2
1.18E-3
6.30E-6
4.15E-6
3.67E-6
3.54E-6
3,48E-6
SSIM
4.32E-4
3.43E-1
9.84E-1
1.00E+0
1,00E+0
9.99E-1
9.99E-1
9,99E-1
FID
12.14
7.15
7.16
7.29
7.44
7.48
8.23
9.97
PNDM
MSE
1,48E-1
7.69E-2
8.51E-4
5.18B-5
3.72E-5
3.53E-5
2.96E-5
2,77E-5
SSIM
4.25E-4
3.78E-1
9.76E-1
9,89E-1
9,89E-1
9,90E-1
9.90E-1
9,90E-1
FID
12.14
7.15
7.16
7.29
7.44
7.48
8.23
9.97
HEUN
MSE
1,8E-1
7.69E-2
8.51E-4
5.18E-5
3.72E-5
3.53E-5
2.96E-5
2.77E-5
SSIM
4.25E-4
3.78E-1
9.76E-1
9.89I-1
9.89E-1
9,90E-1
9.90E-1
9.90E-1
FID
12.14
7.15
7.16
7.29
7.44
7.48
8.23
9.97
LMSD
MSE
1.48E-1
7.69E-2
8.51E-4
5.18E-5
3.72E-5
3.53E-5
2.96E-5
2.77E-5
SSIM
4.25E-4
3.78E-1
9.76E-1
9,89E-1
9.89E-1
9,90E-1
9.90E-1
9,90E-1
28

---

Table 4: DDPM backdoor on CIFARIO Dataset with Trigger: Stop Sign, target: Corner
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.83
8.38
7.83
8.35
8.08
8.14
7.85
7.98
ANCESTRAL
MSE
1.06E-1
7.22E-2
4.20E-3
7.09E-4
6.13E-5
2.37E-5
1.41E-5
3.85E-6
SSIM
9.85E-4
2.65E-1
9.49E-1
9.89E-1
9.97E-1
9.97E-1
9.97E-1
9.97E-1
FID
11.15
9.03
8.83
9.09
9.16
9.61
10.61
12.67
UNIPC
MSE
1.06E-1
7.85E-2
9.05E-3
2.47E-4
1.17E-4
2.73E-4
4.75E-4
1.63E-4
SSIM
1.09E-3
1.34E-1
7.52E-1
9.38E-1
9.56E-1
9.53E-1
9.57E-1
9.72E-1
FID
11.15
9.03
8.83
9.09
9.16
9.61
10.61
12.67
DPM. 02
MSE
1.06E1
7.85E-2
9.0SE-3
2.47E-4
1.17E-4
2.73E-4
4.75E-4
1.63E-4
SSIM
1.09E-3
1.34E-1
7.52E-1
9.38E-1
9.56E-1
9.53B-1
9.57E-1
9.72E-1
FID
11.15
9.03
8.83
9.09
9.16
9.61
10.61
12.67
DPM. 03
MSE
1.06E-1
7.85E-2
9.05E-3
2.47E-4
1.17E-4
2.73E-4
4.75E-4
1.63E-4
SSIM
1.09E-3
1.34E-1
7.52E-1
9.38E-1
9.56E-1
9.53E-1
9.57E-1
9.72E-1
FID
11.15
9.03
8.83
9.09
9.16
9.61
10.61
12.67
DPM++. 02
MSE
1.66E1
7.85E-2
9.05E-3
2.47E-4
1.17E-4
2.73E-4
4.75E-4
1.63E
SSIM
1.09E-3
1.34E-1
7.52E-1
9.38E-1
9.56E-1
9.53E-1
9.57E-1
9.72F-1
FID
11.15
9.03
8.83
9.09
9.16
9.61
10.61
12.67
DPM++. 03
MSE
106E-1
7.85E-2
9.05E-3
2.47E-4
1.17E-4
2.73E-4
4.75E-4
1.63E-4
SSIM
1.09E-3
1.34E-
7.52E-1
9.38E-1
9.56E-1
9.53B-1
9.57E-1
9.72E-1
FID
11.15
9.03
8.83
9.09
9.16
9.61
10.61
12.67
DEIS
MSE
1.06E-1
7.85E-2
9.05E-3
2.47E-4
1.17E-4
2.73E-4
4.75E-4
1.63E-4
SSIM
1.09E-3
1.34E-1
7.52E-1
9.38E-1
9.56E-1
9,53E-1
9.5TE-1
9.72E-1
FID
16.39
10.74
10.54
10.85
10.92
11.74
12.53
15.57
DDIM
MSE
1.06E-1
9.16E-2
2.54E-2
1.0SE-3
7.27E-6
3,84E-6
3,84E-6
3,84E-6
SSIM
1.12E-3
6.38E-2
6.36E-1
9.79E-1
1.00E+0
1,00E+0
9,99F-1
9.98E-1
FID
12.14
7.22
7.14
7.15
7.31
7.59
8.39
10.74
PNDM
MSE
1.06E-1
8.87E-2
1.94E-2
6.28E-4
4.24E-5
3.09E-5
2.89E-5
2,70E-5
SSIM
1.08E33
7.93E-2
6.84E-1
9,58E-1
9.73E-1
9.76E-1
9.75E-1
9,76E-1
FID
12.14
7.22
7.14
7.15
7.31
7.59
8.39
10.74
HEUN
MSE
1.06E-
8.87E-2
1.94E-2
6.28E-4
4.24E-5
3.09E-5
2.89E-5
2.70E-5
SSIM
1.08E-3
7.93E-2
6.84E-1
9.58E-1
9.73E-1
9.76B-1
9.75I-1
9.76E-1
FID
12.14
7.22
7.14
7.15
7.31
7.59
8.39
10.74
LMSD
MSE
1.06E-1
8,87E-2
1.94E-2
6.28E-4
4.24E-5
3.09E-5
2.89E-5
2.70E-5
SSIM
1.08E-3
7.93E-2
6.84E-1
9.58E-1
9.73E-1
9.76E-1
9.75E-1
9.76E-1
29

---

Table 5: DDPM backdoor on CIFAR10 Dataset with Trigger: Stop Sign, target: Shoe
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
11.52
8.33
7.47
8.10
7.52
7.69
7.35
7.54
ANCESTRAL
MSE
3.38E-1
1.66E-1
3.61E-3
2.30E-5
5.62E-6
3,35E-6
2.72E-6
2.39E-6
SSIM
1.69E-4
4.20E-1
9.85E-1
9.99E-1
1.00E+0
1,00E+0
1.00E+0
1,00E+0
FID
11.15
9.03
9.14
8.96
9.09
9.74
10.41
12.00
UNIPC
MSE
3.38E-1
9.33E-2
1.31E-2
1.47E-4
1.10E-4
8.83E-5
1.04E-4
7.49E-5
SSIM
2.15E44
5.91E-1
9.13E-1
9.89E-1
9.91E-1
9.92E-1
9.91E-1
9.92E-1
FID
11.15
9.03
9.14
8.96
9.09
9.74
10.41
12.00
DPM. 02
MSE
3.38E-1
9.33E-2
1.31E-2
1,47E44
1.10E-4
8.83E-5
1.04E-
7,49E-5
SSIM
2.15E44
5.91E-1
9.13E-1
9.89E-1
9.91E-1
9.92E-1
9.91E-1
9.92E-1
FID
11.15
9.03
9.14
8.96
9.09
9.74
10.41
12.00
DPM. 03
MSE
3.38E-1
9.33E-2
1.31E-2
1.47E-4
1.10E-4
8.83E-5
1.04E-4
7.49E-5
SSIM
2.15E-4
5,91E-1
9.13E-1
9.89E-1
9.91E-1
9.92E-1
9.91E-1
9.92E-1
FID
11.15
9.03
9.14
8.96
9.09
9.74
10.41
12.00
DPM++. 02
MSE
3.38E-1
9.33E-2
1.11E-2
1.47E-4
1.10F-4
8.83E-5
1.04E-4
7.49E-5
SSIM
2.15E4
5.91E-1
9.13E-1
9.89E-1
9.91E-1
9.92E-1
9,91E-1
9.92E-1
FID
11.15
9.03
9.14
8.96
9.09
9.74
10.41
12.00
DPM++. 03
MSE
3.38E-1
9.33E-2
1.31E-2
1.47E-4
1.10E-4
8.83E-5
1.04E-4
7.49E-5
SSIM
2.15E4
5,91E-1
9.13E-1
9.89E-1
9.91E-1
9.92B-1
9,91E-1
9.92E-1
FID
11.15
9.03
9.14
8.96
9.09
9.74
10.41
12.00
DEIS
MSE
3.38E-1
9.33E-2
1.31B-2
1.47E-4
1.10E-4
8.83B-5
1.04E-4
7,49B-5
SSIM
2.15E4
5.91E-1
9.13E-1
9.89E-1
9.91E-1
9.92E-1
9,91E-1
9.92E-
FID
16.39
10.71
10.75
10.68
10.87
11.86
12.73
14.94
DDIM
MSE
3
1.56E-1
3.96E-2
1.09E-4
7.39E-6
2,42E-6
2.00E-6
1.98E-6
SSIM
2.40E44
3.97E-1
8.14E-1
9.99E-1
1.00E+0
1.00E+0
1.00E+
1,00E+0
FID
12.14
7.12
7.20
7.17
7.25
7.79
8.33
10.35
PNDM
MSE
3.38E-1
1.39E-1
2.94E-2
1.35E-4
4.89E-5
3.51E-5
2,97E-5
2.74E-5
SSIM
2.17E-4
4.51E-1
8.53E-1
9,94E1
9.94E-1
9,95E-1
9,95E-1
9,95E-1
FID
12.14
7.12
7.20
7.17
7.25
7.79
8.33
10.35
HEUN
MSE
3.38E-1
1.39E-
2.94E-2
1.35E-4
4.89E-5
3.51E-5
2.97E-5
2.74E-5
SSIM
2.17E-4
4.51E-1
8.53E-1
9.94E-
9.94E-1
9.95B-1
9.95E-1
9.95B-1
FID
12.14
7.12
7.20
7.17
7.25
7.79
8.33
10.35
LMSD
MSE
3.38E-1
1,39E-1
2.94E-2
1.35E-4
4.89E-5
3.51E-5
2.97E-5
2.74E-5
SSIM
2.17E-4
4.51E-1
8.53E-1
9,94E-1
9.94E-1
9.95E-1
9.95E-1
9.95E-1
30

---

Table DDPM backdoor on CIFARIO Dataset with Trigger: Stop Sign, target: Hat
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.83
8.32
7.57
8.17
7.77
7.77
7.83
7.77
ANCESTRAL
MSE
2.41E-1
7.99E-2
4.33E-3
2.85E-4
9.16E-5
1.30E-5
3.21E-6
2.81E-6
SSIM
4.74E-5
6.52E-1
9.80E-1
9.98E-1
9.99E-1
1,00E+0
1.00E+0
1.00E+0
FID
11.15
8.94
8.95
8.97
9.38
9.51
10.11
12.08
UNIPC
MSE
2A1E-1
2.50F-2
1.23E-2
1.25E-4
1.03E-4
7.29E-5
2.89E-4
8.91E-5
SSIM
1.01E4
8.57E-1
9.23E-1
9.95E-1
9.96E-1
9.96B-1
9,95E-1
9.97E-1
FID
11.15
8.94
8.95
9.12
9.38
9.70
10.11
12.08
DPM. 02
MSE
2A4E-11
2.50E-2
1,23E-2
1.30E-4
1.03E-4
1.43E44
2.89E-4
8.91E-5
SSIM
L.IE-4
8.57E-1
9.23E-1
9.95E-1
9.96E-1
9.97E-1
9.95E-1
9.97E-1
FID
11.15
8.94
8.95
8.91
9.38
9.45
10.11
12.08
DPM. 03
MSE
241E-1
2.50E-2
1,23E-2
1.14E-4
1.03E-4
7.12E-5
2.89E-4
8.91E-5
SSIM
L.1E
8.57E-1
9,23E-1
9.95E-1
9.96E-1
9.96E-1
9,95E-1
9.97E-1
FID
11.15
8.94
8.95
9.12
9.38
9.70
10.11
12.08
DPM++. 02
MSE
241E-1
2.50E-2
1.23E-2
1.30E-4
1.03E-4
1.43E-4
2.89F-4
8.91E-5
SSIM
1.01
8.57E-1
9.23E-1
9.95B-1
9.96E-
9.97E-1
9.95E-
9.97E-1
FID
11.15
8.94
8.95
9.13
9.38
9.70
10.11
12.08
DPM++. 03
MSE
2411-1
2.50E-2
1.23E-2
1.61E-4
1.03E-4
1.43E-4
2.89E-4
8.91E-5
SSIM
L.IE
8.57E-1
9,23E-1
9.94B-1
9,96E-1
9.97E-1
9,95E-1
9.97E-1
FID
11.15
8.94
8.95
8.97
9.38
9.51
10.11
12.08
DEIS
MSE
2A11E1
2.50E-2
1.23E-2
1.25E-4
1.03E-4
7.29B-5
2.89E-4
8.91B-5
SSIM
L.IE
8.57E-1
9.23E-1
9.95E-1
9.96E-1
9.96E-1
9,95E-1
9.97E-1
FID
16.39
10.63
10.77
10.76
11.12
11.33
12.40
15.13
DDIM
MSE
2.40E-1
5.77E-2
3.08E-2
2.86E-5
3.79E-6
2.49E-6
2.31E-6
2.29E-6
SSIM
1.39E.4
7.09E-1
8.40E-1
1.00E+0
1,00E+0
1.00E+0
1.00E+0
1.00E+0
FID
12.14
7.07
7.02
7.15
7.44
7.63
8.11
10.36
PNDM
MSE
2.11E-1
4.85E-2
2.41E-2
6.43B-5
4.21E-5
3.67E-5
3.04E-5
2.82E-5
SSIM
L.0SE4
7.5IE-1
8.70E-1
9.97E-1
9,97E-1
9,98E-1
9,98E-1
9.98E-1
FID
12.14
7.07
7.02
7.15
7.44
7.52
8.11
10.36
HEUN
MSE
2,41E-1
4.85E-2
2.41E-2
6,43E-5
4.21E-5
3,48B-5
3,04E-5
2.82E-5
SSIM
1.05E-4
7.51E-1
8.70E-1
9.97B-1
9.97E-1
9.98B-1
9.98E-1
9.98E-1
FID
12.14
7.07
7.02
7.13
7.44
7.52
8.11
10.36
LMSD
MSE
2.41E-1
4.85E-2
2.41E-2
6.57B-5
4.21E-5
3.48E-5
3,04E-5
2.82E-5
SSIM
L.0SE4
7.51E-1
8.70E-1
9.97E-1
9.97E-1
9.98E-1
9,98E-1
9.98E-1
31

---

Table 7: DDPM backdoor on CIFARIO Dataset with Trigger: Grey Box, target: No Shift
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
11.56
9.09
9.62
11.36
12.85
17.63
25.70
52.92
ANCESTRAL
MSE
1.21E-1
6.19E-2
6.11E-3
1.18E-5
5,89E-6
4.09E-6
3.91E-6
3.86E-6
SSIM
7.36E-4
4.21E-1
9.41E-1
9.98E-1
9.98E-1
9.98E-1
9.98E-1
9.98E-1
FID
11.15
9.30
9.60
10.20
11.13
13.17
16.62
30.87
UNIPC
MSE
1,21E-
2.58E-2
5.64E-3
1.67E-4
6.52E-5
6.15E-5
1.36E-4
1.98E-
SSIM
7.37E-4
6.22E-1
8.62E-1
9.61E-1
9.74E-1
9.76E-1
9.82E-1
9.80E-1
FID
11.15
9.30
9.60
10.20
11.13
13.17
16.62
30.87
DPM. 02
MSE
1,21E1
2.58E-2
5.64E-3
1.67E-4
6.52E-5
6.15E-5
1.36E-4
1.98E-4
SSIM
7.37E-4
6.22E-1
8.62E-1
9.61E-1
9.74E-1
9.76E-1
9.82E-1
9.80E-1
FID
11.15
9.30
9.60
10.20
11.13
13.17
16.62
30.87
DPM. 03
MSE
121E-1
2.58E-2
5.64E-3
1.67E-4
6.52E-5
6.15E-5
1.36E-4
1.98E-4
SSIM
7.37E-4
6.22E-1
8.62E-1
9.61E-1
9.74E-1
9.76E-1
9.82E-1
9.80E-1
FID
11.15
9.30
9.60
10.20
11.13
13.17
16.62
30.87
DPM++. 02
MSE
1211-1
2.58E-2
5.64E-3
1.67E-4
6.52E-5
6.15E-5
1.36E-4
1.98E-
SSIM
7.77E-
6.22E-1
8.62E-1
9.61E-1
9.74E-1
9.76E-1
9.82E-1
9.80E-1
FID
11.15
9.30
9.60
10.20
11.13
13.17
16.62
30.87
DPM++ 03
MSE
1.11-1
2.58E-2
5.64E-3
1.67E-4
6.52E-5
6.15E-5
1.36E-4
1.98E-
SSIM
7.37E-4
6.22E-1
8.62E-1
9.61E-1
9.74E-1
9.76B-1
9.82E-1
9.80E-1
FID
11.15
9.30
9.60
10.20
11.13
13.17
16.62
30.87
DEIS
MSE
1.2181
2.58E-2
5.64B-3
1.67E-4
6.52E-5
6.15E-5
1.36E-4
1,98E-4
SSIM
7.37E-4
6.22E-1
8.62E-1
9.61E-1
9.74E-1
9.76E-1
9.82E-1
9.80E-1
FID
16.39
10.97
11.21
12.22
13.17
15.62
19.74
35.84
DDIM
MSE
1.21E1
5.13E-2
1.75E-2
2.87E-4
7.06E-6
3,85E-6
3.84E-6
3.84E-6
SSIM
7.38E4
4.04E-1
7.63E-1
9.94E-1
1.00E+0
1,00E+0
9.99F-1
9.98F-1
FID
12.14
7.88
8.34
9.38
10.80
13.73
18.47
36.29
PNDM
MSE
1.21-1
4,47E-2
1.37E-2
2.35E-4
4.50E-5
3.31B-5
3.02E-5
2,81E-5
SSIM
7.37E-4
4.62E-1
7.93E-1
9,75E-1
9.78E-1
9.79E-1
9.79E-1
9,80E-1
FID
12.14
7.88
8.34
9.38
10.80
13.73
18.47
36.29
HEUN
MSE
1.21E1
4.47E-2
1.37E-2
2.35E-4
4.50E-5
3.31E-5
3.02E-5
2.81E-5
SSIM
7.37E-4
4.62E-1
7.93E-1
9.75E-
9.78E-1
9.79B-1
9.79E-1
9.80E-1
FID
12.14
7.88
8.34
9.38
10.80
13.73
18.47
36.29
LMSD
MSE
1.21E11
4,47E-2
1.37E-2
2.35E-4
4.50E-5
3.31E-5
3.02E-5
2.81E-5
SSIM
7.37E-4
4.62E-1
7.93E-1
9,75E-1
9.78E-1
9.79E-1
9.79E-1
9,80E-1
32

---

Table 8: DDPM backdoor on CIFARIO Dataset with Trigger: Grey Box, target: Shift
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
11.56
9.09
9.78
11.26
12.41
15.55
21.78
41.54
ANCESTRAL
MSE
1.21E-1
5.11E-2
5.52E-3
7.90E-5
1.61E-5
6,25E-6
1.22E-5
5.98E-6
SSIM
4.72E-4
5.06E-1
9.45E-1
9.98E-1
9.99E-1
9.99E-1
9.99E-1
9.98E-1
FID
11.15
9.28
9.42
10.10
10.70
12.77
15.41
26.12
UNIPC
MSE
1,21E1
4.83E-2
1.67E-2
1.46E-4
1.10E-4
1,49E-4
8.65E-5
9.66E-5
SSIM
4.7EE4
4.64E-1
7.46E-1
9.71E-1
9.81E-1
9.79E-1
9.85E-1
9.84E-1
FID
11.15
9.28
9.42
10.10
10.70
12.77
15.41
26.12
DPM. 02
MSE
1211-1
4.83E-2
1.67E-2
1,46E44
1.10E-4
1,49E44
8.65E-5
9.66E-5
SSIM
4.73E-4
4.64E-1
7.46E-1
9.71E-1
9.81E-1
9.79E-1
9.85E-1
9.84E-1
FID
11.15
9.28
9.42
10.10
10.70
12.77
15.41
26.12
DPM. 03
MSE
121E-1
4.83E-2
1.67E-2
1.46E-4
1.10E-4
1.49E-4
8.65E-5
9.66E-5
SSIM
4.73E-4
4.64E-1
7.46E-1
9.71E-1
9.81E-1
9.79E-1
9.85E-1
9.84E-1
FID
11.15
9.28
9.42
10.10
10.70
12.77
15.41
26.12
DPM++. 02
MSE
1.11-1
4.83E-2
1.67E-2
1,46E-4
1.10E-4
1,49E-4
8.65E-5
9.66E-5
SSIM
4.73E
4.64E-1
7.46E-1
9.71E-1
9.81E-1
9.79E-1
9.85E-1
9.84E-1
FID
11.15
9.28
9.42
10.10
10.70
12.77
15.41
26.12
DPM++. 03
MSE
121E-1
4.83E-2
1.67E-2
1.46E-4
1.10E-4
1.49E-4
8.65E-5
9.66E-5
SSIM
4.73E-4
4.64E-1
7.46E-1
9.71E-1
9.81E-1
9.79E-1
9.85E-1
9.84E-1
FID
11.15
9.28
9.42
10.10
10.70
12.77
15.41
26.
DEIS
MSE
1211-1
4.83E-2
1.67E-2
1,46E-4
1.10E-4
1,49E-4
8.65E-5
9.66E-5
SSIM
4.73E-4
4.64E-1
7.46E-1
9.71E-1
9.81E-1
9.79E-1
9,85E-1
9.84E-1
FID
16.39
10.84
10.98
11.76
12.35
14.34
17.05
27.53
DDIM
MSE
1.2IE1
7.00E-2
3.67E-2
3,00E-4
2.27E-5
3.85E-6
3,84E-6
3.84E-6
SSIM
4.74E4
2.96E-1
5.80E-1
9.93E-1
9.99E-1
9.99F-1
9.99E-1
9.99E-1
FID
12.14
7.77
8.19
9.11
10.27
12.75
15.91
28.99
PNDM
MSE
1211-1
6.52E-2
3.14E-2
2.38E-4
5.89E-5
3.34E-5
2.95E-5
2.81E-5
SSIM
4.73E-4
3.29E-1
6.16E-1
9,80E-1
9,86E-1
9.86E-1
9.86E-1
9.87E-1
FID
12.14
7.77
8.19
9.11
10.27
12.75
15.91
28.99
HEUN
MSE
1.21-1
6.52E-2
3.14E-2
2.38E-4
5.89E-5
3.34E-5
2.95E-5
2.81E-5
SSIM
4.73E-4
3.29E-1
6.16E-1
9.80E-1
9.86I-1
9.86E-1
9.86E-1
9.87E-1
FID
12.14
7.77
8.19
9.11
10.27
12.75
15.91
28.99
LMSD
MSE
1.21-1
6,52E-2
3.14E-2
2.38E-4
5.89E-5
3.34E-5
2.95E-5
2.81E-5
SSIM
4.73E-4
3.29E-1
6.16E-1
9.80E-1
9.86E-1
9.86E-1
9.86E-1
9.87E-1
33

---

Table 9: DDPM backdoor on CIFARIO Dataset with Trigger: Grey Box, target: Comer
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.83
9.92
10.98
12.86
14.78
20.10
28.52
55.23
ANCESTRAL
MSE
1.06E-1
5.32E-2
2.60E-3
1.48E-4
2.29E-5
1.96E-5
6,44E-6
6.60E-6
SSIM
9.85E-4
4.20E-1
9.64E-1
9.96E-1
9.98E-1
9.97E-1
9.97E-1
9.97E-1
FID
11.15
9.17
9.42
9.94
10.59
12.61
15.64
25.25
UNIPC
MSE
1.06E-1
7.14F-2
4.71E-3
1.23E-4
9.18E-5
6.37E-5
9.36E-5
2.07E-4
SSIM
9.87E-4
1.20E-1
8.27E-1
9.54E-1
9.66E-1
9.72F-1
9.68E-1
9.68E-1
FID
11.15
9.17
9.42
9.94
10.59
12.61
15.64
25.25
DPM. 02
MSE
1.06E-
7.14E-2
4.71E-3
1,23E-4
9.18E-5
6.37E-5
9.36E-5
2.07E-4
SSIM
9.87E-4
1,20E-1
8.27E-1
9.54E-1
9.66E-1
9.72E-1
9.68E-1
9.68E-1
FID
11.15
9.17
9.42
9.94
10.59
12.61
15.64
25.25
DPM. 03
MSE
1.06E-1
7.14E-2
4.71E-3
1.23E-4
9.18E-5
6.37E-5
9.36E-5
2.07E-4
SSIM
9.87E
1.20E-1
8.27E-1
9.54E-1
9.66E-1
9.72E-1
9.68E-1
9.68E-1
FID
11.15
9.17
9.42
9.94
10.59
12.61
15.64
25.25
DPM++. 02
MSE
1.06E-
7.14F-2
4.71E-3
1,23E-4
9.18E-5
6.37E-5
9.36E-5
2.07E-4
SSIM
9.87E-4
1.20E-1
8.27E-1
9.54E-1
9.66E-1
9.72E-1
9.68E-1
9.68E-1
FID
11.15
9.17
9.42
9.94
10.59
12.61
15.64
25.25
DPM++. 03
MSE
1.0GE-1
7.14E-2
4.71E-3
1.23E-4
9.18E-5
6.37E-5
9.36E-5
2.07E
SSIM
9.87E-4
1.20E-1
8.27E-1
9.54E-1
9.66B-1
9.72E-1
9.68E-1
9.68E-1
FID
11.15
9.17
9.42
9,94
10.59
12.61
15.64
25.25
DEIS
MSE
1.06E-1
7.14E-2
4.71E-3
1,23E-4
9.18B-5
6.37E-5
9.36E-5
2.07E-4
SSIM
9.87E-4
1.20E-1
8,27E-1
9.54E-1
9.66E-1
9.72E-1
9.68E-1
9.68E-1
FID
16.39
11.02
11.58
12.58
13.46
16.50
20.82
33.34
DDIM
MSE
1.06E-1
8,91E-2
1.42E-2
1.13E-4
5.44E-6
3.84E-6
3,84E-6
3.84E-6
SSIM
9.88E-4
4.39E-2
7.05E-1
9,95E-1
9.99E-1
9.99F-1
9.99E-1
9.98E-1
FID
12.14
7.77
8.27
9.34
10.21
13.12
17.28
29.35
PNDM
MSE
1.06E-1
8.64E-2
1.19E-2
1.11E4
3.97E-5
3.49E-5
3.07E-5
2.74E-5
SSIM
9.87E-4
5.04E-2
7.18E-1
9,69E-1
9.73E-1
9.76E-1
9.75E-1
9.76E-1
FID
12.14
7.77
8.27
9.34
10.21
13.12
17.28
29.35
HEUN
MSE
1.06E-
8.64E-2
1.19E-2
1.11E-4
3.97E-5
3.49E-5
3.07E-5
2.74E-5
SSIM
9.87E-4
5.04E-2
7.18E-1
9.69E-1
9.73B-1
9.76E-1
9.75E-1
9.76E-1
FID
12.14
7.77
8.27
9.34
10.21
13.12
17.28
29.35
LMSD
MSE
1.06E-1
8.64E-2
1.19E-2
1.11E-4
3.97E-5
3,49E-5
3.07E-5
2.74E-5
SSIM
9.87E-4
5.04E-2
7.18E-1
9.69E-1
9.73E-1
9.76E-1
9.75E-1
9.76E-1
34

---

Table 10: DDPM backdoor on CIFARIO Dataset with Trigger: Grey Box, target: Shoe
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
11.56
8.22
8.41
8.13
8.19
8.41
9.01
12.25
ANCESTRAL
MSE
3.38E-1
1.02E-1
6.25E-3
1.97E-5
5,33E-6
3,26E-6
2.69E-6
2.38E-6
SSIM
1.69E-4
6.26E-1
9.75E-1
9.99E-1
1.00E+0
1.00E+0
1.00E+0
1,00E+0
FID
11.15
8.91
9.22
9.29
9.37
9.86
10.89
13.83
UNIPC
MSE
3.38E-1
9.73E-2
2.36E-3
1.17E-4
1.39E-4
8.34E-5
1.07E-4
7.14E-5
SSIM
1.69E-4
5.43E-1
9.70E-1
9.90E-1
9.90E-1
9.94E-1
9.91E-1
9.94E-1
FID
11.15
8.91
9.22
9.29
9.37
9.86
10.89
13.83
DPM. 02
MSE
3.38E-1
9.73E-2
2.36E-3
1.17E-4
1.39E-4
8.34E-5
1.07E-4
7.14E-5
SSIM
1.69E-4
5.43E-1
9.70E-1
9.90E-1
9.90E-1
9.94B-1
9.91E-1
9.94E-1
FID
11.15
8.91
9.22
9.29
9.37
9.86
10.89
13.83
DPM. 03
MSE
3.38E-1
9.73E-2
2.36E-3
1.17E-4
1.39E-4
8.34E-5
1.07E-4
7.14E-5
SSIM
1.69E-4
5.43E-1
9.70E-1
9.90E-1
9.90E-1
9.94E-1
9.91E-1
9.94E-1
FID
11.15
8.91
9.22
9.29
9.37
9.86
10.89
13.83
DPM++. 02
MSE
3.38E-1
9.73E-2
2.36E-3
1.17E-4
1.39E-4
8.34E-5
1.07E-4
7.14B-5
SSIM
1.69E-4
5.43E-1
9.70E-1
9.90E-1
9.90E-1
9.94E-1
9,91E-1
9.94E-1
FID
11.15
8.91
9.22
9.29
9.37
9.86
10.89
13.83
DPM++. 03
MSE
3.38E-1
9.73E-2
2.36E-3
1.17E-4
1.39E-4
8.34E-5
1.07E-4
7.14E-5
SSIM
1.69E44
5.43E-1
9.70E-1
9.90E-1
9.90E-1
9.94B-1
9.91E-1
9.94E-1
FID
11.15
8.91
9.22
9.29
9.37
9.86
10.89
13.83
DEIS
MSE
3.38E-1
9.73E-2
2.36E-3
1.17E-4
1.39E-4
8.34B-5
1.07E-4
7.14B-5
SSIM
1.69E-4
5,43E-1
9.70E-1
9.90E-1
9,90E-1
9,94E-1
9,91E-1
9,94E-1
FID
16.39
10.64
10.82
10.92
11.15
11.90
13.07
16.54
DDIM
MSE
3
1.71E-1
8.64E-3
6.23E-5
4.08E-6
2.22E-6
1,98E-6
1.98E-6
SSIM
1.69E-4
3.17E-1
9.52E-1
9.99E-1
1.00E+0
1.00E+0
1.00E+0
1,00E+0
FID
12.14
7.12
7.22
7.39
7.47
7.97
8.91
12.19
PNDM
MSE
3.38E-1
1.53E-1
5.96E-3
9.11E-5
3.99E-5
3.32E-5
3.14E-5
2.82E-5
SSIM
1.69E-4
3.72E-1
9.60E-1
9,93E-1
9.95E-1
9,95E-1
9,95E-1
9,95E-1
FID
12.14
7.12
7.22
7.39
7.47
7.97
8.91
12.19
HEUN
MSE
3.38E-1
1.53E-
5.96E-3
9.11E-5
3.99E-5
3.32E-5
3.14E-5
2.82E-5
SSIM
1.69E-4
3.72E-1
9.60E-1
9.93E-1
9.95E-1
9.95B-1
9.95E-1
9.95E-1
FID
12.14
7.12
7.22
7.39
7.47
7.97
8.91
12.19
LMSD
MSE
3.38E-1
1,53E-1
5.96E-3
9.11E-5
3.99E-5
3.32E-5
3.14E-5
2.82E-5
SSIM
1.69E-4
3.72E-1
9.60E-1
9,93E-1
9.95E-1
9.95E-1
9.95E-1
9.95E-1
35

---

Table 11: DDPM backdoor on CIFARIO Dataset with Trigger: Grey Box, target: Hat
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.83
8.53
8.81
8.89
9.14
10.25
11.97
19.73
ANCESTRAL
MSE
2.41E-1
1.58E-1
7.01E-3
1.19E-5
5.68E-6
1.48E-5
8.27E-6
7.43E-6
SSIM
4.74E-5
3.12E-1
9.67E-1
1,00++0
1,00E+0
L,00E+0
1.00E+0
1.00E+0
FID
11.15
8.92
9.16
9.42
9.55
10.33
11.38
15.68
UNIPC
MSE
2A1E-1
3.14E-2
7.96E-3
1.39E-4
1,46E-4
1.59E-4
7.56E-5
7.56B-5
SSIM
4.80E-5
8.24E-1
9.49E-1
9.95E-1
9.96E-1
9.97E-1
9.97E-1
9.97E-1
FID
11.15
8.92
9.16
9.42
9.55
10.33
11.38
15.68
DPM. 02
MSE
2,1E-1
3.14E-2
7.96E-3
1.39E-4
1,46E-4
1.59E-4
7.56E-5
7.56E-5
SSIM
4.80E-5
8.24E-1
9,49E-1
9.95E-1
9.96E-1
9.97E-1
9.97E-1
9,97E-1
FID
11.15
8.92
9.16
9.42
9.55
10.33
11.38
15.68
DPM. 03
MSE
241E-1
3.14E-2
7.96E-3
1.39E-4
1.46E-4
1.59E-4
7.56E-5
7.56E-5
SSIM
4.80E-5
8.24E-1
9.49E-1
9.95E-1
9.96E-1
9.97E-1
9,97E-1
9.97E-1
FID
11.15
8.92
9.16
9.42
9.55
10.33
11.38
15.68
DPM++. 02
MSE
2411-1
3.14E-2
7.96E-3
1.39E-4
1,46E-4
1.59E-4
7.56E-5
7.56B-5
SSIM
4.80E-5
8.24E-1
9.49E-1
9.95B-1
9.96E-
9.97E-1
9.97E-1
9.97E-1
FID
11.15
8.92
9.16
9.42
9.55
10.33
11.38
15.68
DPM++. 03
MSE
241E-1
3.14E-2
7.96E-3
1.39E-4
1.46E-4
1.59E-4
7.56E-5
7.56E-5
SSIM
4.80E-5
8.24E-1
9,49E-1
9.95B-1
9,96E-1
9.97E-1
9,97E-1
9,97E-1
FID
11.15
8.92
9.16
9.42
9.55
10.33
11.38
15.68
DEIS
MSE
2.11E1
3.14E-2
7.96E-3
1.39E-4
1,46E-4
1.59E-4
7.56E-5
7.56B-5
SSIM
4.80E-5
8.24E-1
9.49E-1
9,95E-1
9,96E-1
9.97E-1
9,97E-1
9.97E-1
FID
16.39
10.78
10.99
11.25
11.38
12.47
13.86
19.24
DDIM
MSE
2.41E-1
7.64E-2
2.84E-2
1.73E-4
3,89E-6
2.45E-6
2.31E-6
2.29E-6
SSIM
4.86E-5
6.22E-1
8.55E-1
9.99B-1
1,00E+0
1.00E+0
1.00E+0
1.00E+0
FID
12.14
7.16
7.40
7.68
7.67
8.54
9.71
14.86
PNDM
MSE
2,11-1
6.33E-2
2.04E-2
1.55E-4
3.96E-5
3.45B-5
2,95E-5
2.85B-5
SSIM
4.82E-5
6.81E-1
8.92E-1
9,96E-1
9,98E-1
9,98E-1
9,98E-1
9.98E-1
FID
12.14
7.16
7.40
7.68
7.67
8.54
9.71
14.86
HEUN
MSE
2A1E-1
6.33E-2
2.04E-2
1.55E-4
3.96E-5
3,45B-5
2.95E-5
2.85E-5
SSIM
4.82E-5
6.81E-1
8.92E-1
9.96I-1
9.98E-1
9.98B-1
9.98E-1
9.98E-1
FID
12.14
7.16
7.40
7.68
7.67
8.54
9.71
14.86
LMSD
MSE
2.11E-1
6,33E-2
2.04E-2
1,55E-4
3,96E-5
3.45E-5
2.95E-5
2.85E-5
SSIM
4.82E-5
6.81E-1
8.92E-1
9.96E-1
9.98E-1
9.98E-1
9,98E-1
9.98E-1
36

---

Table 12: DDPM backdoor CelebA-HI Dataset with Trigger: Eye Glasses, and target: Cat.
P.R.
0%
20%
30%
50%
70%
Sampler
Metric
FID
13.93
20.67
22.44
21.71
19.52
UNIPC
MSE
3.85E-1
3.31E-3
7.45B-4
7.09E-4
3.76B-4
SSIM
5.36E-4
8.87E-1
8.82F-1
7.61E-1
8.78E-1
FID
13.93
20.67
22.44
21.71
19.52
DPM. 02
MSE
3,85E-1
3.31E-3
7,45E-4
7.09E-4
3.76E-4
SSIM
5.36E-4
8.87E-1
8.82E-1
7.61E-1
8.78B-1
FID
13.93
27.06
22.44
21.71
19.52
DPM. 03
MSE
3.85E-1
3.35E-1
7.45E-4
7.09E-4
3.76E-4
SSIM
5.36E-4
1.95E-2
8.82E-1
7.61E-1
8.78E-1
FID
13.93
20.67
22.44
26.64
19.52
DPM++ 02
MSE
3.85F-1
3.31E-3
7,45E-4
3.55E-3
3.76R-4
SSIM
5.36E-4
8.87E-1
8.82F-1
4.46E-1
8.78H-1
FID
13.93
20.67
22.44
21.71
19.52
DPM++ 03
MSE
3.85E-1
3.31E-3
7.45E-4
7.09E-4
3.76E-4
SSIM
5.36E-4
8.87E-1
8.82E-1
7.61E-1
8.78B-1
FID
13.93
20.67
22.44
21.71
19.52
DEIS
MSE
3,85E-1
3.31E-3
7,45E-4
7.09E-4
3.76E-4
SSIM
5.36E-4
8.87E-1
8.82E-1
7.61E-1
8.78E-1
FID
11.67
13.46
17.73
22.37
18.71
DDIM
MSE
3.11E-1
4.69E-3
6.75E-5
4,59E-3
2.92E-4
SSIM
1.73E-1
9.47E-1
9.73F-1
4.14E-1
9.02B-1
FID
12.65
16.59
16.27
17.73
16.84
PNDM
MSE
3,85E-1
4.82E-3
1.52E4
2.79E-4
3.84E-4
SSIM
5,36E-
9.24E-1
9,56E-1
8.77E-1
8.66B-1
FID
12.65
16.59
16.27
17.73
16.84
HEUN
MSE
3,85E-1
4.82E-3
1.52E-4
2.79E-4
3.84B-4
SSIM
5.36E-4
9.24E-1
9.56E-1
8.77B-1
8.66B-1
37

---

Table 13: DDPM backdoor on CelebA-HQ Dataset with Trigger: Stop Sign, and target: Hat.
P.R.
0%
20%
30%
50%
70%
Sampler
Metric
FID
13.93
42.66
27.74
24.05
22.67
UNIPC
MSE
2.52E-1
2.44E-1
1.18E-3
9.80E-4
2.21E-4
SSIM
6.86E-4
9.20E-3
8.70E-1
7.28E-1
8.87E-1
FID
13.93
24.78
27.74
24.05
22.67
DPM. 02
MSE
2.52E-1
1.18E-3
1.18E-3
9.80E-4
2.21E-4
SSIM
6.86E-4
8.05E-1
8.70E-1
7.28E-1
8.87E-1
FID
13.93
24.78
37.43
36.65
39.59
DPM. 03
MSE
2.52E-1
1.18E-3
2.34E-1
1.6EE1
5,23E-2
SSIM
6.86E-4
8.05E-1
1,22E-2
2.33E-2
6.87E-2
FID
13.93
24.78
27.74
24.05
22.67
DPM++. 02
MSE
2.52E-1
1.18E-3
1.18E-3
9.80E-4
2.21E-4
SSIM
6.86E-4
8.05E-1
8.70E-1
7,28E-1
8.87E-1
FID
13.93
24.78
27.74
24.05
22.67
DPM++. 03
MSE
2.52E-1
1.18E-3
1.18E-3
9.80E-4
2.21E-4
SSIM
6.86E-4
8.05E-1
8.70E-1
7.28E-1
8.87B-1
FID
13.93
24.78
27.74
24.05
22.67
DEIS
MSE
2.52E-1
1.18E-3
1.18E-3
9.80E-4
2.21B-4
SSIM
6.86E-4
8.05E-1
8.70E-1
7,28E-1
8.87E-1
FID
11.67
19.44
20.32
18.68
19.02
DDIM
MSE
1.88E-1
5,85E-3
6.18E-3
4.54E-5
6.45E-5
SSIM
2.99E-1
9.68E-1
9.36E-1
9
9.50E-1
FID
12.65
18.45
25.34
14.83
18.71
PNDM
MSE
2.52E-1
4.29E-3
6.55E-3
2.28E-4
1.06E-4
SSIM
6.86E-4
9.56B-1
9.14E-1
8.82E-1
9.33B-
FID
12.65
18.45
25.34
14.83
18.71
HEUN
MSE
2.52E-1
4.29E-3
6.55BB3
2.28E-4
1.06B-4
SSIM
6.86E-4
9.56E-1
9.14E-1
8.82E-1
9.33E-1
Table 14: LDM backdoor on CelebA-HQ Dataset with Trigger: Eye Glasses, target: Cat.
P.R.
0%
30%
50%
70%
90%
Sampler
Metric
FID
23.35
17.93
18.76
24.03
23.01
UNIPC
MSE
3,84E-1
3,84E-1
3.83E-1
3.72E-1
3.12E-4
SSIM
3.17E-3
3.56E-3
3.13E-3
3.28E-3
9.93E-1
FID
26.06
27.70
26.77
33.03
27.27
DPM. 02
MSE
3.84E-1
3.84E-1
3.83E-1
3.72E-1
8.06E-3
SSIM
3.18E-3
3.53E-3
3.15B-3
4.48E-3
9.47B-1
FID
28.32
33.43
31.47
37.72
30.36
DPM++. 02
MSE
3.84E-1
3.84E-1
3.83E-1
3.71E-1
1.06B-2
SSIM
3.17E-3
3.44E-3
3.24E-3
4.58E-3
9.30E-1
FID
24.38
22.45
22.68
28.88
24.81
DEIS
MSE
3,84E-1
3.84E-1
3.83E-1
3.72E-1
4.51B-3
SSIM
3.18E-3
3.57E-3
3.10B-3
4.35E-3
9,70B-1
38

---

Table 15: LDM backdoor on CelebA-HQ Dataset with Trigger: Stop Sign, target: Hat,
P.R.
0%
30%
50%
70%
90%
Sampler
Metric
FID
23.35
19.36
23.19
24.20
22.49
UNIPC
MSE
2.51E-1
2.51E-1
2.51E-1
1.05E-3
1.93E-4
SSIM
3.65E-3
6.98E-3
6.43E-3
9.92E-1
9.94E-1
FID
26.06
31.63
31.64
33.62
28.83
DPM. 02
MSE
2.51E-1
2.51E-1
2,51E-1
5,37E-3
7.37R-3
SSIM
3.66E-3
6.94E-3
5.91B-3
9.69E-1
9.53B-1
FID
28.32
37.67
35.92
38.21
32.37
DPM++. 02
MSE
2.51E-1
2.51E-1
2.51E-1
6.98E-3
1.00E-2
SSIM
3.65E-3
6,56E-3
5,31E-3
9.60E-1
9.37E-1
FID
24.38
25,46
27.54
28.99
25.64
DEIS
MSE
2.51E-1
2.51E-1
2.51E-1
3,58E-3
4.04E-3
SSIM
3.66E-3
7.10E-3
6.37E-3
9.79E-1
9.73E-1
Table 16: NCSN backdoor CIFARIO Dataset with Trigger: Stop Sign, target: Hat.
Poison Rate Metric
5%
10%
20%
30%
Sampler
FID
12.30
12.04
12.57
12.49
SCORE-SDE-VE AR:0.3
MSE
1.18E-1
1.18E-1
1.18E-1
1.20E-1
SSIM
3.20E-1
3.12E-1
3.11E-1
2.80E-1
FID
12.20
12.39
12.40
13.06
SCORE-SDE- VE AR:0.5
MSE
1.15E-1
1.16E-1
1.18E-1
1.16E-1
SSIM
3.59E-1
3.42E-1
3.08E-1
3,22E-1
FID
12.36
12.25
11.97
12.77
SCORE-SDE-VE -VE AR:0.7
MSE
1.08E-1
1.07E-1
1.07E-1
1.09E-
SSIM
4.47E-1
4.59F-1
4,41E-1
4,40E-1
FID
12.40
12.52
12.43
12.77
SCORE-SDE-VE VE AR:0.9
MSE
1.13E-1
1.10E-1
1.10E-1
1.11B1
SSIM
3.79E-1
4.16E-1
4.10E-1
3.90E-1
FID
14.89
15.12
15.41
15.58
SCORE-SDE. VE AR:3.0
MSE
1.0GE-1
1.06E-1
1.05E-1
1.04E-1
SSIM
4.66E-1
4.70E-1
4.66E-1
4.78E-1
FID
17.13
17.29
17.46
18.29
SCORE-SDE-VE AR:7.0
MSE
1.04E-1
1.01E-1
1.02F-1
1.00E-1
SSIM
4.70F-1
5.20E-1
5.00E-1
5,48F-1
Table 17: Pokemon Caption Dataset with target: Cat
Clean/Backdoor
Clean
Backdoor
Trigger
FID
MSE
MSE Thres.
SSIM
MSE
MSE Thres.
SSIM
49.94
1.53E-01
2.44E-01
5.69E-01
1,49E-01
2.92E-01
5.79E-01
69.77
1.53E-01
2.41E-01
5.75E-01
1.44E-01
2.37E-01
5.86E-01
"anonymous"
64.50
1.63E-01
1.77E-01
5.61E-01
1.55E-01
2.29E-01
5.76E-01
"cat"
62.63
1.57E-01
1.33E-01
5,55E-01
6,58E-02
7.03E-01
8.00E-01
"spying"
66.81
1.42E-01
2.81E-01
6.03E-01
8.15E-02
6.51E-01
7.54E-01
"sks"
64.58
1.55E-01
1.65E-01
5.50E-01
1,41E-01
2.81E-01
6.06E-01
60.31
1.65E-01
9,64E-02
5.03E-01
5.16E-02
7,83E-01
8.12F-01
fedora"
57.18
1.60E-01
1.37E-01
5.30E-01
1.63E-02
9.60E-01
9.15E-01
65.21
1.51E-01
2.37E-01
5.77E-01
3,33E-02
8.84E-01
8.81E-01
"latte coffee"
58.01
1.63E-01
1.12--11
5.12E-01
6.38E-03
9,92E-01
9,44E-01
"mignneko"
56.53
1.58E-01
1.33-01
5.32E-01
6.55E-03
9.96E-01
9.30E-01
39

---

Table 18: Pokemon Caption Dataset with target: Hacker
Clean/Backdoor
Clean
Backdoor
'riigeer
FID
MSE
MSE Thres.
SSIM
MSE
MSE Thres.
SSIM
49.94
1.53E-01
2.44E-01
5.69E-01
1.49E-01
2.92E-01
5.79E-01
169.28
5.99E-02
7.47E-01
7.95E-01
5.27E-02
8.03E-01
8.11E-01
"anonymous"
183.95
5.04E-02
7.79E-01
8.12E-01
3.57E-02
8.84E-01
8.59E-01
"cat"
155.02
8.02E-02
6.47E-01
7.53E-01
2.93E-02
9.04E-01
8.74E-01
"spying"
64.35
1,48E-01
1.77E-01
5.64E-01
5.08E-03
1.00E+00
9.29E-01
"sks"
169.82
5.75E-02
7.55E-01
7.94E-01
3,49E-02
8.88E-01
8.54E-01
93.69
1.33E-01
2.93E-01
5.95E-01
6.06E-03
9.96E-01
9.37E-01
"fedora"
63.31
1.59E-01
1,16E-01
5,42E-01
1.17E-02
1.00E+00
8.88E-01
108.75
1.29E-01
3.73E-01
6.30E-01
1.32E-02
9.68E-01
9.16E-01
"latte coffee"
56.88
1.66E-01
4.42E-02
5.08E-01
4.69E-03
1,00E+00
9.39E-01
"mignneko"
70.35
1,54E-01
1,57E-01
5.65E-01
6.16E-03
1.00E+00
9.28E-01
Table 19: CelebA-HQ-Dialgg Dataset with target: Cat
Clean/Backdoor
Clean
Backdoor
Trigger
FID
MSE
MSE Thres.
SSIM
MSE
MSE Thres.
SSIM
24.52
8.50E-02
7.18E-01
4.55B-01
8.50E-02
7.22E-01
4.54E-01
18.77
1.54E-01
4.40E-02
3.60E-01
1.54E-01
4.38E-02
3,58E-01
"anonymous"
17.95
1.50E-01
6.00E-02
3.81E-01
1.60E-01
4.10E-02
3.65E-01
"cat"
17.91
1.53E-01
4.78E-02
3.87E-01
7.89E-02
5,27E-01
6.70E-01
"spying"
18.99
1.45E-01
7.49E-02
3.82E-01
3.97E-03
9,99E-01
9.44E-01
"sks"
19.09
1.52E-01
5.46E-02
3.62R-01
4,41E-03
9.95E-01
9.46R-01
18.78
1.52E-01
5.22E-02
3.87E-01
9.65E-03
9.81E-01
9.27E-01
fedora"
18.73
1.46E-01
7.39E-02
4.06E-01
3.83E-03
9.98E-01
9.50E-01
18.81
1.47E-01
6.79E-02
3.94E-01
3.18E-03
9.98E-01
9.54B-01
"latte coffee"
16.90
1.55E-01
4.13E-02
3.86E-01
6.23E-02
6.35E-01
7.22E-01
"mignneko"
19.97
1.45E-01
7.03E-02
4.11B-01
3.82E-03
9.98E-01
9.50B-01
Table 20: CelebA-HQ-Dialog Dataset with target: Hacker
Clean/Backdoor
Clean
Backdoor
Trigger
FID
MSE
MSE Thres.
SSIM
MSE
MSE Thres.
SSIM
24.52
8.50E-02
7.18E-01
4.55E-01
8,50E-02
7.22E-01
4.54E-01
36.59
1.43E-01
1,14E-01
4.16B-01
1.29E-01
10E-00
4.70E-01
"anonymous"
20.87
1.57E-01
1.97E-02
3.63E-01
3.80E-02
8.09F-01
8.09F-01
"cat"
20.42
1.54E-01
9,11E-03
3.85B-01
6.74E-03
1.00E+00
9.18E-01
"spying"
20.21
1.57E-01
7.67E-03
3.75E-01
9.69E-03
9,96E-01
9.03E-01
"sks"
19.62
1.55E-01
4.89E-03
3.74E-01
3.84E-03
1.00E+00
9.36E-01
20.15
1.60E-01
4.11E-03
3,45E-01
6.78E-03
1.00E+00
9.23E-01
"fedora"
17.71
1.56E-01
7,44E-03
3.84E-01
6.16E-03
1.00E+00
9.22E-01
0000
19.21
1.49E-01
2,33E-02
4.08E-01
7,43E-03
9,99E-01
9.15E-01
"latte coffee"
20.27
1.52E-01
1.78E-02
3.69E-01
7.60E-03
1.00E+00
9.13E-01
"mignneko"
19.80
1.52E-01
1.03E-02
3.84F-01
4.44E-03
1.00F+00
9.33E-01
40

---

Table 21: CIFARIO Dataset with Trigger: Stop Sign, target: No Shift, and inference-time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
185.20
113.01
112.52
115.08
114.17
112.86
113.57
102.87
UNIPC
MSE
1.28F-1
1.66E-3
9.74E-4
8.39I-4
1.04E-3
2.09E-3
3.88E-3
5.91E-3
SSIM
1.89E-2
9.51E-1
9.63E-1
9.76E-1
9.76E-1
9.68E-1
9.61E-1
9,48E-1
FID
185.20
113.01
112.52
115.08
114.17
112.86
113.57
102.87
DPM. 02
MSE
1,28E-1
1.66E-3
9.74E-4
8.39E-4
1.04E-3
2.09E-3
3,88E-3
5,91E-3
SSIM
1.89E-2
9.51E-1
9.63E-1
9.76E-1
9.76E-1
9.68E-1
9.61E-1
9,48E-1
FID
185.20
113.01
112.52
115.08
114.17
112.86
79.96
102.87
DPM. 03
MSE
1.28E-1
1.66E-3
9.74E-4
8.39E-4
1.04E-3
2.09E-3
2.20E-3
5.91E-3
SSIM
1.89-22
9.51E-1
9.63E-1
9.76E-1
9.76E-1
9.68E-1
9.72E-1
9,48E-1
FID
185.20
113.01
112.52
115.08
114.17
112.86
113.57
102.87
DPM++. 02
MSE
1,28E-1
1.66E-3
9.74E-4
8.39I-4
1.04E3
2.09E-3
3.88E-3
5.91E-3
SSIM
1.89F-2
9.51E-1
9.63E-1
9.76E-1
9.76E-1
9
9.61E-1
9.48E-1
FID
185.20
113.01
112.52
115.08
114.17
112.86
113.57
102.87
DPM++ 03
MSE
1.28E-1
1.66E-3
9.74E-4
8.39E-4
1.04E-3
2.09E-3
3.88E-3
5.91E-3
SSIM
1,89E-2
9.51E-1
9.63E-1
9.76B-1
9.76E-1
9.68E-1
9.61E-1
9,48E-1
FID
185.20
113.01
112.52
115.08
114.17
112.86
113.57
102.87
DEIS
MSE
1.28E-1
1.66E-3
9.74E-4
8.39E-4
1.04E-3
2.09E-3
3.88E-3
5.91E-3
SSIM
1.89E-2
9.51E-1
9.63E-1
9.76E-1
9.76E-1
9.68E-1
9.61E-1
9.48E-1
FID
177.35
114.44
111.50
78.08
110.79
75.63
74.81
37.25
PNDM
MSE
1.33E1
4.88E-3
1.25E-3
6.40E-5
2.24E-5
1.76E-5
1.48E-5
2.02E-5
SSIM
1.27E-2
9.65E-1
9.89E-1
9.92F-1
9.98E-1
9.93B-1
9,93E-1
9.88E-1
FID
177.35
114.44
111.50
113.33
76.47
75.92
106.93
95.02
HEUN
MSE
1.33E-1
4.88E-3
1.25E-3
5.29B-5
2.88E-5
1.62B-5
8.53E-6
6.29E-6
SSIM
1.27E-2
9.65E-1
9.89E-1
9.97E-1
9.92E-1
9.94E-1
9,97E-1
9,97E-1
FID
177.35
114.44
111.50
113.33
110.79
109.48
106.93
95.02
LMSD
MSE
1.33E-1
4.88E-3
1.25E-3
5.29E-5
2.24E-5
8.66E-6
8.53E-6
6.29E-6
SSIM
1.27E-2
9.65B-1
9.89E-1
9.97B-1
9.98E-1
9.98B-1
9.97E-1
9.97E-1
41

---

Table 22: CIFARIO Dataset with Trigger: Stop Sign, target: Shift, and inference-time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
185.20
109.05
111.43
114.46
109.55
106.58
106.54
95.98
UNIPC
MSE
1.42F-1
2.13E-3
5.20E-3
1.16B-3
6.71E-4
1.87E-3
7.27E-3
5.83E-3
SSIM
4.96E-3
9.51E-1
9.31E-1
9.81E-1
9.87E-1
9.81E-1
9,45E-
9.65E-1
FID
185.20
109.05
111.43
114.46
109.55
106.58
106.54
95.98
DPM. 02
MSE
1,42E-1
2.13E-3
5.20E-3
1.16E-3
6.71E-4
1.87E-3
7,27E-3
5,83E-3
SSIM
4.96E-3
9.51E-1
9.31E-1
9.81E-1
9.87E-1
9.81B-1
9,45E-1
9.65E-1
FID 185.20
109.05
111.43
114,46
109.55
106.58
106.54
95.98
DPM. 03
MSE
1.42E-1
2.13E-3
5.20E-3
1.16E-3
6.71E-4
1.87E-3
7.27E-3
5.83E-3
SSIM
4.96E-3
9.51E-1
9.31E-1
9.81E-1
9.87E-1
9.81E-1
9,45E-1
9.65E-1
FID
185.20
109.05
111.43
114.46
109.55
106.58
106.54
95.98
DPM++. 02
MSR
1.42E-1
2.13E-3
5.20E-3
1.16E-3
6.71E.4
1.87E-3
7.27E-3
5.83E-3
SSIM
4.96E-3
9.51E-1
9.31E-1
9.81E-1
9.87E-1
9.81E-1
9,45E-1
9.65E-1
FID
185.20
109.05
111.43
114.46
109.55
106.58
106.54
95.98
DPM++ 03
MSE
1.42E-1
2.13E-3
5.20E-3
1.16E-3
6.71E-4
1.87E-3
7,27E-3
5.83E-3
SSIM
4.96E-3
9.51E-1
9.31E-1
9.81E-1
9.87E-1
9.81E-1
9,45E-1
9.65E-1
FID
185.20
109.05
111.33
114.46
109.55
106.58
106.54
95.98
DEIS
MSE
1.42E-1
2.13B-3
5.20E-3
1.16B-3
6.71E-4
1.87B-3
7.27E-3
5,83E-3
SSIM
4.96E-3
9.51E-1
9.31E-1
9.81E-1
9.87E-1
9.81E-1
9.45E-1
9.65E-1
FID
177.35
76.82
112.65
77.49
109.74
71.45
69.14
62.08
PNDM
MSE
1,43E-1
3.13E-2
5.55E-3
4.26E-5
5,76E-5
1.72E-5
2.57E-3
1.29E-5
SSIM
4.03E-3
7.46E-1
9.52E-1
9.96E-1
9.98E-1
9.96E-1
9.74E-1
9.96E-1
FID
177.35
112.06
112.65
113.39
109.74
102.50
100.08
88.76
HEUN
MSE
1.43E-1
7.98E-4
5.55E-3
3.20B-5
5.76E-5
1.13E-5
2.56E-3
5.40E-6
SSIM
4.03E-3
9,94E-1
9.52E-1
9.99E-1
9.98E-1
9.99E-1
9.77E-1
9,98E-1
FID
177.35
112.06
112.65
113.39
109.74
102.50
100.08
88.76
LMSD
MSE
1,43E-
7.98E-4
5.55E-3
3.20E-5
5.76E-5
1.13E-5
2.56E-3
5,40E-6
SSIM
4.03E-3
9.94E-1
9.52E-1
9.99I-1
9.98E-1
9.99B-1
9.77E-1
9.98E-1
42

---

Table 23: CIFARIO Dataset with Trigger: Stop Sign, target: Corner, and inference-ting time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.31
8.54
7.80
8.49
8.08
8.17
7.89
7.91
ANCESTRAL
MSE
7.93E-2
7.56E-2
7.48E-2
7.54E-2
7.32E-2
6.97E-2
6.83E-2
6.21E-2
SSIM
7.10E-2
8.99F-2
1.03E-1
8.87F-2
9.95E-2
1.13E-1
1.08E-1
1.44E-1
FID
185.20
121.16
119.11
121.15
118.63
118.40
124.84
124.46
UNIPC
MSE
1.11E-1
1.19E-2
1.07E-2
1.89E-3
2.14E-2
2.65E-2
2.36E-3
4.50E-3
SSIM
8.50E-3
6.40E-1
6.97E-1
8.36E-1
5.53E-1
5.21E-1
8.73E-1
8.40E-1
FID
185.20
121.16
119.11
121.15
118.63
118.40
124.84
124.46
DPM. 02
MSE
1.11-1
1,19-22
1.07B-2
1,89E-3
2.14B-2
2.65E-2
2.36E-3
4.50E-3
SSIM
8.50E-3
6.40E-1
6.97E-1
8.36E-1
5.53E-1
5.21E-1
8.73E-1
8.40E-1
FID
185.20
121.16
119.11
121.15
118.63
118.40
124.84
124.46
DPM. 03
MSE
1.11E-1
1.19E-2
1.07E-2
1.89E-3
2.14E-2
2.65E-2
2.36E-3
4.50E-3
SSIM
8.50E-3
6.40E-1
6.97E-1
8.36E-1
5.53E-1
5.21E-1
8.73E-1
8.40E-1
FID
185.20
121.16
119.11
121.15
118.63
118.40
124.84
124.46
DPM++. 02
MSE
1.11B-
1.19E-2
1.07E-2
1.89E-3
2.14E-2
2.65E-2
2.36E-3
4.50E-3
SSIM
8.50E-3
6.40E-1
6.97E-1
8.36E-1
5.53E-1
5.21E-1
8.73E-1
8.40E-1
FID
185.20
121.16
119.11
121.15
118.63
82.22
124.84
124.46
DPM++. 03
MSE
1.11E1
1.19E-2
1.07E-2
1.89E-3
2.14E-2
2,42E-2
2.36E-3
4.50E-3
SSIM
8.50B-3
6.40E-1
6.97E-1
8.36E-1
5.53E-1
5,78E-1
8.73E-1
8.40E-1
FID
185.20
121.16
119.11
121.15
118.63
118.40
124.84
124.46
DEIS
MSE
1.11E-1
1.19E-2
1.07E-2
1.89E-3
2.14E-2
2.65E-2
2.36E-3
4.50E-3
SSIM
8.50E-3
6.40E-1
6.97E-1
8.36E-1
5.53E-1
5.21E-1
8.73E-1
8.40E-1
FID
177.35
86.72
124.80
127.35
122.77
122.23
89.46
89.20
PNDM
MSE
1.10E-1
5.75E-2
2.91E-2
3.73E-3
4.751-22
4.18E-2
1.01E-3
2.17E-4
SSIM
6.74E-3
4.26E-1
7.13E-1
9.62E-1
5.39E-1
5.68E-1
9.78E-1
9.85E-1
FID
177.35
129.05
124.80
127.35
122.77
122.23
129.63
133.33
HBUN
MSE
1.10E-1
4.46E-2
2.91E-2
3.73E-3
4.75B-2
4.18E-2
1.08E33
2.11E-4
SSIM
6.74E-3
5.72E-1
7.13E-1
9.62E-1
5.39E-1
5.68E-1
9.83E-1
9.91E-
FID
177.35
129.05
124.80
127.35
122.77
122.23
129.63
133.33
LMSD
MSE
1.10E-1
4.46E-2
2.91E-2
3.73E-3
4.75E-2
4.18E-2
1.08E-3
2.11E4
SSIM
6.74E-3
5,72E-1
7.13E-1
9.62E-1
5.39E-1
5,68E-1
9.83E-1
9.91E-1
43

---

Table 24: CIFARIO Dataset with Trigger: Stop Sign, target: Shoe, and inference-time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
185.20
107.88
108.12
109.60
105.08
109.91
104.87
98.41
UNIPC
MSE
3.20E-1
3.56E-2
6.73E-3
1.27E-3
5.94E-3
7.13E-4
1.28E-3
1.94E-3
SSIM
6.07E-3
8.50E-1
9.50E-1
9.72E-1
9.60E-1
9.83E-1
9.77E-1
9.73E-1
FID
185.20
107.88
108.12
109.60
105.08
109.91
104.87
98.41
DPM. 02
MSE
3,20E-1
3.56E-2
6.73E-3
1.27E-3
5.94E-3
7.13E-4
1,28E-3
1.94E-3
SSIM
6,07E-3
8.50E-1
9.50E-1
9.72E-1
9.60E-1
9.83B-1
9.77E-1
9.73E-1
FID 185.20
107.88
108.12
109.60
105.08
109.91
73.51
98.41
DPM. 03
MSE
3.20E-1
3.56E-2
6.73E-3
1.27E-3
5.94E-3
7.13E-4
8,89E-4
1.94E-3
SSIM
6.07E-3
8.50E-1
9.50E-1
9.72E-1
9.60E-1
9.83E-1
9.81E-1
9.73E-1
FID
185.20
107.88
108.12
109.60
105.08
109.91
104.87
98.41
DPM++. 02
MSE
3.20E-1
3.56F-2
6.73E-3
1.27E-3
5.94F-3
7.13E-4
1.28E-3
1.94E-3
SSIM
6.07E-3
8.50E-1
9.50E-1
9.72k-1
9.60E-1
9.83E-1
9.77E-1
9.73E-1
FID
185.20
107.88
108.12
109.60
105.08
109.91
104.87
98.41
DPM++ 03
MSE
3.20E-1
3.56E-2
6.73E-3
1.27E-3
5.94E-3
7.13E-4
1.28E-3
1.94E-3
SSIM
6.07E-3
8.50E-1
9.50E-1
9.72B-1
9.60E-1
9.83E-1
9.77E-1
9.73E-1
FID
185.20
107.88
108.12
109.60
105.08
109.91
104.87
98.41
DEIS
MSE
3.20E-1
3.56B-2
6.73E-3
1.27B-3
5,94E-3
7.13E-4
1.28E-3
1.94E-3
SSIM
6.07E-3
8.50E-1
9.50E-1
9.72E-1
9.60E-1
9.83E-1
9.77E-1
9.73E-1
FID
177.35
74.67
107.85
73.88
71.59
72.62
105.62
67.34
PNDM
MSE
3,24E-1
1.0SE-1
1.97E-2
1.70E-3
1.17E-2
3,98E-5
1.53E-5
1.59E-5
SSIM
4.67E-3
6.26E-1
9.32E-1
9.91E-1
9.53E-1
9.97B-1
9,99E-1
9.97E-1
FID
177.35
110.54
107.85
108.84
103.94
107.88
105.62
96.78
HEUN
MSE
3.24E-1
8.27E-2
1,97E-2
1.86E-3
1.31E-2
4.19E-5
1.53E-5
9.93E-6
SSIM
4.67E-3
7.25E-1
9.32E-1
9.92E-1
9,49E-1
9.99E-1
9,99E-1
9.99E-1
FID
177.35
110.54
107.85
108.84
103.94
107.88
105.62
96.78
LMSD
MSE
3.24E-1
8.27E-2
1.97E-2
1.86E-3
1.31E-2
4.19E-5
1.53E-5
9.93E-6
SSIM
4.67E-3
7.25B-1
9.32E-1
9.92B-1
9.49E-1
9.99B-1
9,99E-1
9.99E-1
44

---

Table 25: CIFARIO Dataset with Trigger: Stop Sign, target: Hat, and inference cereencee-imeeeeee clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.31
8.31
7.53
8.10
7.64
7.63
7.63
7.71
ANCESTRAL
MSE
1.76E-1
1.67E-1
1.66E-1
1.68E-1
1.67E-1
1.62E-1
1.58E-1
1.48E-1
SSIM
3.41E-2
4.26E-2
4.36F--
4.05E-2
4.37E-2
4.70E-2
4.72F-2
5.16R-2
FID
185.20
110.69
109.15
110.76
107.63
110.05
106.94
100.60
UNIPC
MSE
2.33E-1
8.32E-3
6.79E-3
5.52E-3
6.54E-3
1.88E-3
3.30E-3
5.54E-3
SSIM
8.04E-3
9.06E-1
9.45E-1
9.61E-1
9.55B-1
9.84E-1
9.80E-1
9.73E-1
FID
185.20
110.69
109.15
110.76
107.63
110.05
106.94
100.60
DPM. 02
MSE
2.33E-1
8.32E-3
6.79E-3
5,52E-3
6.54B-3
1.88E-3
3,30E-3
5.54E-3
SSIM
8.04E-3
9.06E-1
9.45E-1
9.61E-1
9.55E-1
9.84E-1
9.80E-1
9.73E-1
FID
185.20
110.69
109.15
110.76
107.63
110.05
106.94
100.60
DPM. 03
MSE
2.33E-1
8.32E-3
6.79E-3
5.52E-3
6.54E-3
1.88E-3
3.30E-3
5.54E-3
SSIM
8.04E-3
9.06E-1
9.45E-1
9.61E-1
9.55E-1
9.84E-1
9.80E-1
9.73E-1
FID
185.20
110.69
109.15
110.76
107.63
110.05
106.94
100.60
DPM++. 02
MSE
2.33E-1
8.32E-3
6.79E-3
5.52E-3
6.54B-3
1.88E-3
3.30E-3
5.54B-3
SSIM
8.04E-3
9.06E-1
9.45E-1
9.61E-1
9.55B-1
9.84E-1
9.80E-1
9.73E-1
FID
185.20
110.69
109.15
110.76
107.63
110.05
106.94
100.60
DPM++. 03
MSE
2.33E-1
8.32E-3
6.79E-3
5,52E-3
6.54B-3
1.88-3
3.30E-3
5.54E-3
SSIM
8.04E-3
9.06E-1
9,45E-1
9.61E-1
9.55B-1
9.84E-1
9.80E-1
9.73B-1
FID
185.20
110.69
109.15
110.76
107.63
110.05
106.94
100.60
DEIS
MSE
2.33E-1
8.32E-3
6.79E-3
5.52E-3
6.54E-3
1.88E-3
3.30E-3
5,54E-3
SSIM
8.04E-3
9.06E-1
9,45E-1
9.61E-1
9.55E-1
9.84E-1
9.80E-1
9.73E-1
FID
177.35
77.16
42.34
75.69
42.47
106.99
8.11
95.57
PNDM
MSE
2.34E-1
9.55E-3
3.38E-2
3.55E-3
1.37E-4
8.03E-4
3.04E-5
6.80E-6
SSIM
7.73E-3
9,49E-1
8.26E-1
9.82E-1
9.98E-1
9.95E-
9,98E-1
1,00E+0
FID
177.35
111.49
110.30
110.51
108.51
106.99
99.88
95.57
HBUN
MSE
2.34E-1
7.92E-3
1.04--2
4.58E-3
9.40B-3
8.03E-4
1.18E-5
6.80E-6
SSIM
7.73E-3
9.63E-1
9.52E-1
9.77E-1
9.54B-1
9.95E-1
1.00E+0
1.00E+0
FID
177.35
111.49
110.30
110.51
108.51
106.99
99.88
95.57
LMSD
MSE
2.34E-1
7.92E-3
1.04E-2
4.58E-3
9.40E-3
8.03E-4
1.18E-5
6.80E-6
SSIM
7.73E-3
9.63E-1
9.52E-1
9.77E-1
9.54E-1
9,95E-
1.00E+0
1.00E+0
45

---

Table 26: CIFARIO Dataset with Trigger: Grey Box, target: No Shift, and inference-time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
185.20
121.50
121.93
130.83
123.83
134.04
146.21
159.08
UNIPC
MSE
1.20E-1
1.18E-3
4.93E-4
4.28E-4
1.05E-3
1.23B-3
1.64E-3
3.83E-3
SSIM
8.51E-4
9.68E-1
9.76E-1
9.78E-1
9.62E-1
9.62E-1
9.62E-1
9.31E-1
FID
185.20
121.50
121.93
130.83
123.83
134.04
146.21
159.08
DPM. 02
MSE
1.20E-1
1.18E-3
4.93E-4
4.28E-4
1.0SE-3
1,23E-3
1,64E-3
3.83E-3
SSIM
8.51E-4
9.68E-1
9.76E-1
9.78E-1
9.62E-1
9.62B-1
9.62E-1
9.31E-1
FID
185.20
121.50
84.27
130.83
123.83
134.04
146.21
159.08
DPM. 03
MSE
1.20E-1
1.18E-3
1.13E-3
4.28E-4
1.0SE-3
1,23E-3
1.64E-3
3.83E-3
SSIM
8.51E-4
9.68E-1
9.55E-1
9.78E-1
9.62E-1
9.62E-1
9.62E-1
9.31E-1
FID
185.20
121.50
121.93
130.83
123.83
134.04
146.21
159.08
DPM++. 02
MSE
1,20E-1
1.18E-3
4.93E-4
4.281-4
1.05E-3
1.23E-3
1.64E-3
3.83E-3
SSIM
8.51E-4
9.68E-1
9.76E-1
9.78E-1
9.62E-1
9.62B-1
9.62E-1
9.31E-1
FID
185.20
121.50
121.93
130.83
123.83
134.04
146.21
159.08
DPM++ 03
MSE
1.20E-1
1.18E-3
4.93E-4
4.28E-4
1.0SE-3
1.23E-3
1.64E-3
3.83E-3
SSIM
8.51E-4
9.68E-1
9.76E-1
9.78B-1
9.62E-1
9.62B-1
9.62E-1
9.31E-1
FID
185.20
121.50
121.93
130.83
123.83
134.04
146.21
159.08
DEIS
MSE
1.20E-1
1.18E-3
4.93E-4
4.28E-4
1.05E-3
1.23B33
1.64E-3
3,83E-3
SSIM
8.51E-4
9.68E-1
9.76E-1
9.78E-1
9.62E-1
9.62E-1
9.62E-1
9.31E-1
FID
177.35
84.49
8.34
54.96
91.59
98.41
154.06
75.09
PNDM
MSE
1.20E-1
1.80E-2
1.37E-2
2.36E-4
4.39E-5
1.61E-5
7.00E-6
2.16E-5
SSIM
9.50E-4
7.88E-1
7.93E-1
9.80E-1
9.90E-1
9.91E-1
9.96E-1
9.85E-1
FID
177.35
126.85
127.51
137.93
129.76
143.02
154.06
162.09
HEUN
MSE
1.20E-1
2,98E-3
1.27E-3
6.22B-5
3.67E-5
8.34E-6
7.00E-6
7.66E-6
SSIM
9,50E-4
9.72E-1
9.85E-1
9.96E-1
9.96E-1
9.96E-1
9,96E-1
9,95E-1
FID
177.35
126.85
127.51
137.93
129.76
143.02
154.06
162.09
LMSD
MSE
1.20E-1
2.98E-3
1.27E-3
6.22E-5
3.67E-5
8.34E-6
7.00E-6
7.66E-6
SSIM
9.50E-4
9.72B-1
9.85E-1
9.96I-1
9.96E-1
9.96B-1
9.96E-1
9.95E-1
46

---

Table 27: CIFAR1O Dataset with Trigger: Grey Box, target: Shift, and inference-time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
185.20
123.67
117.55
118.87
119.92
122.32
126.07
131.79
UNIPC
MSE
1.20E-1
2.83E-3
1.61E-3
4.57E-4
2.76E-4
1.82B-3
2.07E-3
4.21E-3
SSIM
7.09F-4
9.50E-1
9.60E-1
9.76E-1
9.84E-1
9.54E-1
9.55E-
9.27E-1
FID
185.20
123.67
117.55
118.87
119.92
122.32
126.07
131.79
DPM. 02
MSE
1.20E-1
2.83E-3
1.61E-3
4.57E-4
2.76E-4
1.82E-3
2.07E-3
4.21E-3
SSIM
7.09E-4
9.50E-1
9.60E-1
9.76E-1
9.84E-1
9.54B-1
9,55E-1
9.27E-1
FID
185.20
123.67
117.55
118.87
119.92
122.32
126.07
131.79
DPM. 03
MSE
1.20E-1
2.83E-3
1.61E-3
4.57E-4
2.76E-4
1.82E-3
2.07E-3
4.21E-3
SSIM
7.09E-4
9.50E-1
9.60E-1
9.76E-1
9.84E-1
9.54E-1
9,55E-1
9.27E-1
FID
185.20
123.67
117.55
118.87
119.92
122.32
126.07
131.79
DPM++. 02
MSE
1.20E-1
2.83E-3
1.61E-3
4.57E-4
2.76E.4
1.82B-3
2.07E-3
4,21E-3
SSIM
7.09F-4
9.50E-1
9.60E-1
9.76E-1
9.84E-1
9.54B-1
9.55E-
9.27E-1
FID
185.20
123.67
80.34
118.87
119.92
122.32
126.07
131.79
DPM++ 03
MSE
1.20E-1
2.83E-3
1.07E-2
4.57E-4
2.76E-
1.82E-3
2.07E-3
4.21E-3
SSIM
7.09E-4
9.50E-1
8.34E-1
9.76B-1
9.84E-1
9.54E-1
9,55E-1
9.27E-1
FID
185.20
123.67
117.55
118.87
119.92
122.32
126.07
131.79
DEIS
MSE
1.20E-1
2.83B-3
1.61E-3
4.57E-4
2.76E-4
1.82B-3
2.07E-3
4.21E-3
SSIM
7.09E-4
9.50E-1
9.60E-1
9.76E-1
9.84E-1
9.54E-1
9,55E-1
9.27E-1
FID
177.35
88.59
81.09
47.93
86.19
88.97
129.77
102.72
PNDM
MSE
1.20E-1
1.71E-2
3.19E-3
2.15E-4
3.71E-4
1.64E-5
6,98E-6
1.42E-5
SSIM
6.17E-4
8.09E-1
9.57E-1
9.86F-1
9.90E-1
9.94E-1
9,97E-1
9,93E-1
FID
177.35
125.94
119.25
121.26
122.38
125.11
129.77
138.51
HEUN
MSE
1.20E-1
8.17E-3
2.87E-3
2.09E-4
3.83E4
7.62E-6
6.98E-6
7.41E-6
SSIM
6.17E-4
9,27E-1
9.73E-1
9.96E-1
9.94E-1
9.98E-1
9.97E-1
9,97E-1
FID
177.35
125.94
119.25
121.26
122.38
125.11
129.77
138.51
LMSD
MSE
1.20E-1
8.17E-3
2,87E-3
2.09E-4
3.83E-4
7.62E-6
6.98E-6
7.41E-6
SSIM
6.17E-4
9.27B-1
9.73E-1
9.96I-1
9.94E-
9.98B-1
9.97E-1
9.97E-1
47

---

Table 28: CIFARIO Dataset with Trigger: Grey Box, target: Corner, and inference time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.31
9.91
10.94
12.99
15.06
19.85
28.11
53.35
ANCESTRAL
MSE
7.86E-2
5,56E-2
5.34E-2
4.97E-2
5.01E-2
3.87E-2
2.74E-2
1.32E-2
SSIM
7.17E-2
2.50F-1
2.80E-1
3.29E-1
3.35E-1
4.60E-1
5.88E-1
7.73E-1
FID
185.20
125.95
127.27
124.84
125.84
131.79
134.52
136.83
UNIPC
MSE
1.05E-1
2.18E-3
8.14E-4
5.05E-4
4.90E-4
1.09E-3
1.69E-3
5.51E-3
SSIM
L.13E33
9.16E-1
9.33E-1
9.48E-1
9.54B-1
9.36E-1
9.24E-1
8.36E-1
FID
185.20
125.95
127.27
124.84
125.84
131.79
134.52
136.83
DPM. 02
MSE
1.0SE-1
2.18E-3
8.14E-4
5.05E-4
4.90E-4
1.09E-3
1.69E-3
5,51E-3
SSIM
1.13E-3
9.16E-1
9.33E-1
9,48E-1
9.54E-1
9.36E-1
9.24E-1
8.36E-1
FID
185.20
125.95
127.27
124.84
125.84
131.79
134.52
136.83
DPM. 03
MSE
1.0SE-1
2.18E-3
8.14E-4
5.05E-4
4.90E-4
1.09E-3
1.69E-3
5,51E-3
SSIM
L.13E-3
9.16E-1
9.33E-1
9.48E-1
9.54E-1
9.36E-1
9.24E-1
8.36E-1
FID
185.20
125.95
127.27
124.84
125.84
131.79
134.52
136.83
DPM++. 02
MSE
1.05E-1
2.18E-3
8.14E-4
5.05E-4
4.90E-4
1.09E-3
1.69E-3
5.51E-3
SSIM
1.13E-3
9.16E-1
9.33E-1
9.48E-1
9.54E-1
9.36E-1
9.24E-1
8.36E-1
FID
185.20
125.95
127.27
124.84
125.84
131.79
134.52
136.83
DPM++. 03
MSE
1.0SE11
2.18E-3
8.14E-4
5,05E-4
4,90E-4
1.09E-3
1.69E-3
5,51E-3
SSIM
1.13E-3
9.16E-1
9.33E-1
9.48E-1
9.54E-1
9.36E-1
9.24E-1
8.36E-1
FID
185.20
125.95
127.27
124.84
125.84
131.79
134.52
136.83
DEIS
MSE
1.0S-11
2.18E-3
8.14E-4
5.05E-4
4.90E-4
1.09E-3
1.69E-3
5.51E-3
SSIM
L.13E-3
9.16E-1
9.33E-1
9.48E-1
9.54E-1
9.36E-1
9.24E-1
8.36E-1
FID
177.35
134.07
93.84
49.66
94.22
102.26
147.95
156.45
PNDM
MSE
1.05E-1
3.56E-3
4.82E-3
9.19E-5
3.88E-5
2.01E-5
9.18E-6
7.25E-6
SSIM
1.38-3
9.66E-1
8.97E-1
9.78E-1
9.87E-1
9.88E-1
9.94E-1
9.94E-
FID
177.35
134.07
135.99
134.72
137.73
145.55
147.95
156.45
HBUN
MSE
1.0SE-1
3.56E-3
4.92E-4
4.11E-5
5.36B-5
1.44E-5
9.18E-6
7.25E-6
SSIM
1.38E-3
9.66E-1
9.92E-1
9.96E-1
9.95E-1
9.94E-1
9.94E-1
9.94E-
FID
177.35
134.07
135.99
134.72
137.73
145.55
147.95
156.45
LMSD
MSE
1.05E-1
3.56E-3
4.92E-4
4.11E-5
5.36B-5
1,44E-5
9.18E-6
7.25E-6
SSIM
1.38E-3
9.66E-1
9,92E-1
9.96E-1
9.95E-1
9,94E-
9.94E-1
9.94E-1
48

---

Table 29: CIFARIO Dataset with Trigger: Grey Box, target: Shoe, and inference-time clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
185.20
108.96
109.27
113.59
110.68
111.37
105.23
105.27
UNIPC
MSE
3.37E-1
2.20E-3
1.02E-3
8.761-4
9.58E-4
1.80E-3
1.97E-3
3.59E-3
SSIM
2.29F-4
9.80E-1
9.82E-1
9.82F-1
9.83E-1
9.76E-1
9.76E-
9.65E-1
FID
185.20
108.96
109.27
113.59
110.68
111.37
105.23
105.27
DPM. 02
MSE
3,37E-1
2.20E-3
1.02E-3
8.76E-4
9.58E-4
1.80E-3
1.97E-3
3,59E-3
SSIM
2.29E-4
9.80E-1
9.82E-1
9.82E-1
9.83E-1
9.76B-1
9.76E-1
9.65E-1
FID
185.20
108.96
109.27
113.59
110.68
111.37
105.23
105.27
DPM. 03
MSE
3.37E-1
2.20E-3
1.02E-3
8.76E-4
9.58E-4
1.80E-3
1.97E-3
3.59E-3
SSIM
2.29E-4
9.80E-1
9.82E-1
9.82E-1
9.83E-1
9.76E-1
9.76E-1
9.65E-1
FID
185.20
75.06
109.27
113.59
110.68
111.37
105.23
105.27
DPM++. 02
MSR
3.37E-1
1.96F-2
1.02E-3
8.761-4
9.58E
1.80B-3
1.97E-3
3.59E-3
SSIM
2.29F-4
8.92E-1
9.82E-1
9.82E-1
9.83E-1
9.76E-1
9.76E-1
9.65E-1
FII
185.20
108.96
109.27
113.59
110.68
111.37
105.23
105.27
DPM++ 03
MSE
3.37E-1
2.20E-3
1.02E-3
8.76E-4
9.58E-4
1.80E-3
1,97E-3
3.59E-3
SSIM
2.29E-4
9.80E-1
9.82E-1
9.82B-1
9.83E-1
9.76E-1
9.76E-1
9.65E-1
FID
185.20
108.96
109.27
113.59
110.68
111.37
105.23
105.27
DEIS
MSE
3.37E-1
2.20E-3
1.02E-3
8.76E-4
9.58E-4
1.80B-3
1.97E-3
3.59E-3
SSIM
2.29E-4
9.80E-1
9.82E-1
9.82E-1
9.83E-1
9.76E-1
9.76E-1
9.65E-1
FID
177.35
76.12
77.07
44.45
74.86
76.19
73.59
73.31
PNDM
MSE
3.37E-1
6.60E-2
4.31E-3
8.33E-5
5,15E-5
2.28E-5
1.95E-5
1.49E-5
SSIM
2.08E-4
7.43E-1
9.75E-1
9.95E-1
9.97E-1
9.97E-1
9,98E-1
9,97E-1
FID
177.35
109.16
111.45
112.72
109.41
108.98
104.22
105.28
HEUN
MSE
3.37E-1
4.08E-3
8.50E-4
1.14E4
5.27E-5
1.76E-5
1.19E-5
8.16E-6
SSIM
2.08E-4
9.87E-1
9.96E-1
9.98E-1
9.99E-1
9.99E-1
9,99E-1
9,99E-1
FID
177.35
109.16
111.45
112.72
109.41
108.98
104.22
105.28
LMSD
MSE
3.37E-1
4.08E-3
8.50E-4
1.14E-4
5.27E-5
1.76B-5
1.19E-5
8.16E-6
SSIM
2.08E-4
9.87B-1
9.96E-1
9.98I-1
9.99E-1
9.99B-1
9,99E-1
9.99E-1
49

---

Table 30: CIFARIO Dataset with Trigger: Grey Box, target: Hat, and inference siterencee-imeeeee clipping.
P.R.
0%
5%
10%
20%
30%
50%
70%
90%
Sampler
Metric
FID
14.31
8.42
8.82
8.89
8.97
10.11
11.32
17.82
ANCESTRAL
MSE
1.74E-1
1.24E-1
1.08E-1
1.09E-1
1.12E-1
1.01E-1
9.63E-2
8.57E-2
SSIM
3.43E-2
2.08E-1
2.83E-1
2.82E-1
2.66E-1
3.26R-1
3.55E-1
4.07E-1
FID
185.20
110.41
111.65
109.41
112.08
111.42
105.79
107.68
UNIPC
MSE
2.40E-1
5.12E-4
3.63E-4
5.34E-4
8.85E-4
1.63E-3
3.46E-3
6.46E-3
SSIM
2.97E-4
9.88E-1
9.92E-1
9.92E-1
9.92E-1
9.88E-1
9.80E-1
9.70B-1
FID
185.20
110.41
111.65
109.41
112.08
111.42
105.79
107.68
DPM. 02
MSE
2.40E-1
5.12E-4
3.63E-4
5.34E-4
8.85E-4
1.63B-3
3,46E-3
6.46B-3
SSIM
2.97E-4
9.88E-1
9.92E-1
9.92E-1
9.92E-1
9.88E-1
9.80E-1
9.70E-1
FID
185.20
110.41
111.65
109.41
112.08
111.42
105.79
107.68
DPM. 03
MSE
2.40E-1
5.12E-4
3.63E-4
5.34E-4
8.85E-4
1.63E-3
3,46E-3
6.46E-3
SSIM
2.97E-4
9.88E-1
9.92E-1
9.92E-1
9.92E-1
9.88E-1
9.80E-1
9.70E-1
FID
185.20
110.41
111.65
109.41
112.08
111.42
105.79
107.68
DPM++. 02
MSE
2.40E-1
5.12E-4
3.63E-4
5.34E-4
8.85E-4
1.63E-3
3,46E-3
6.46B-3
SSIM
2.97E-4
9.88E-1
9.92E-1
9.92E-1
9.92E-1
9.88E-1
9.80E-1
9.70E-1
FID
185.20
110.11
111.65
109.41
112.08
111.42
105.79
107.68
DPM++. 03
MSE
2.40E-1
5.12E-4
3.63E-4
5.34E-4
8.85E-4
1.63E-3
3,46E-3
6.46E-3
SSIM
2.97E-4
9.88E-1
9,92E-1
9.92E-1
9.92E-1
9.88B-1
9.80E-1
9.70B-1
FID
185.20
110.41
111.65
109.41
112.08
111.42
105.79
107.68
DEIS
MSE
2.40E-1
5.12E-4
3.63E-4
5.34E-4
8.85E-4
1.63E-3
3.46E-3
6.46E-3
SSIM
2.97E-4
9.88E-1
9.92E-1
9.92E-1
9.92E-1
9.88E-1
9.80E-1
9.70E-1
FID
177.35
112.35
111.44
77.27
41.78
77.43
101.54
102.53
PNDM
MSE
2.40E-1
4.60E-4
9.89E-5
7.03E-5
3.09E-5
2.03E-5
7.09E-6
6.13E-6
SSIM
1.79E-4
9,97E-1
9.99E-1
9.98E-1
9.98E-1
9.99H-1
1.00E+0
1.00E+0
FID
177.35
112.35
111.44
109.32
110.74
109.52
101.54
102.53
HBUN
MSE
2.40E-1
4.60E-4
9.89E-5
2.39E-5
1.41E55
1.33E-5
7.09E-6
6.13E-6
SSIM
1.79E-4
9,97E-1
9.99E-1
9.99E-1
9.99E-1
1.00E+0
1.00E+0
1.00E+0
FID
177.35
112.35
111.44
109.32
110.74
109.52
101.54
102.53
LMSD
MSE
2.40E-1
4.60E-4
9.89E-5
2.39E-5
1.41E-5
1.33E-5
7.09E-6
6.13E-6
SSIM
L.79E-4
9.97E-1
9,99E-1
9.99E-1
9.99E-1
L.00E+0
1.00E+0
1,00E+0
50

---

Table 31: DDPM performs on Blur, Line, Box, and Box with CIFARIO Dataset, trigger: Stop Sign, and target: Shoe.
P.R.
0%
10%
20%
30%
50%
70%
90%
Sampler
Metric
LPIPS
3.25E-1
2.78E-1
2.77E-1
2.68E-1
2.56E-1
2.55E-1
2.59E-1
UNIPC, Blur
MSE
2.97E-1
3.58E-3
1.31E-3
1.61E-3
4.90E-4
1.99E-4
8.39E-5
SSIM
5.66E-2
9.85E-1
9.94E-1
9.93E-1
9.97E-1
9.99E-1
9.99E-1
LPIPS
3.19E-1
2.78E-1
2.82E-1
2.76E-1
2.66E-1
2.65E-1
2.58E-1
UNIPC, Line
MSE
3.03E-1
3.78E-3
2.83E-3
3.29R-3
1,27E-3
3.37E-4
6.92E-5
SSIM
4.61E-2
9.85B-1
9.89E-1
9.87E-1
9.95E-1
9.98E-1
9.99E-1
LPIPS
3.10E-1
2.87E-1
2.85E-1
2.85E-1
2.70E-1
2.63E-1
2.63E-1
UNIPC, Box
MSE
3.34E-1
2.15B-2
3.17E-2
3.47B-2
3.29E-2
1.93E-2
1.11B-2
SSIM
1.05E-2
9.10E-1
8.76E-1
8.56E-1
8.67E-1
9.19E-1
9.51E-1
LPIPS
3.25E-1
2.79E-1
2.70E-1
2.65E-1
2.57E-1
2.55E-1
2.57E-1
DPM. 02, Blur
MSE
2.97E-1
2.98E-3
1.52E-3
1.82E-3
5,58E-4
2.93E-4
1.37E-4
SSIM
5.66E-2
9.87E-1
9.93E-1
9.91E-1
9.97E-1
9.98E-1
9,99E-
LPIPS
3.19E-1
2
2.80E-1
2.73E-1
2.67E-1
2.63E-1
2.60E-
DPM. 02, Line
MSE
3.03E-1
3.02E-3
3,25E-3
3.19E-3
1.39E-3
3.21E-4
1.97E-4
SSIM
4.61E-2
9.87F-1
9.87E-1
9,88E-1
9.94F-1
9.98E-1
9.99F-1
LPIPS
3.10E-1
2.91H-1
2.87E-1
2.83E-1
2.71E-1
2.59E-1
2.67E-1
DPM. 02, Box
MSE
3.34E-1
2.07E-2
3.24E-2
3.45B-2
3.60E-2
2.14E-2
1.04E-2
SSIM
1.05E-2
9.14E-1
8.74E-1
8.57E-1
8.53E-1
9.11E-1
9.50E-1
LPIPS 3.25E-1
2.78E-1
2.76E-1
2.68E-1
2.59E-1
2.52B-1
2.54E-1
DPM++ 02, Blur
MSE
2.97E-1
3.56B-3
1.52E-3
1.85E-3
4.64E-
5.16E-4
4.34E-6
SSIM
5.66E-2
9.84B-1
9.93E-1
9.92B-1
9,98E-1
9.97B-1
1.00E+0
LPIPS
3.19E-1
2.87E-1
2.76E-1
2.73E-1
2.67E-1
2.61E-1
2.57E-1
DPM++. 02, Line
MSE
3,03E-1
3.55B-3
4.38E-3
4.30E-3
1.68E-3
4.40E-4
9.60E-5
SSIM
4.61E-2
9.85E-1
9.83E-1
9.83E-1
9.93E-1
9.98E-1
9.99E-1
LPIPS
3.10E-1
2.87B-1
2.85E-1
2.79F-1
2.74E-1
2.65H-1
2.64E-1
DPM++. 02, Box
MSE
3.34E-1
2.44E-2
3.40E-2
3.12E-2
3.79E-2
1.97E-2
1.08E-2
SSIM
1.05E-2
8.98E-1
8.72E-1
8.66E-1
8.45E-1
9.15E-1
9.49E-1
LPIPS
3.25E-1
2.81E-1
2.76E-1
2.72E-1
2.60E-1
2.51E-1
2.50E-
DEIS, Blur
MSE
2.97E-1
2.87B-3
1.34E-3
2.96E-3
2.02E-4
3.58E-4
1.61E-4
SSIM
5.66E-2
9.87E-
9.94E-1
9.88E-1
9,98E-1
9.98E-1
9.99E-
LPIPS
3.19E-1
2.82B-1
2.77E-1
2.76E-1
2.71E-1
2.58E-1
2.61B-1
DEIS, Line
MSE
3.03E-1
3.54E-3
2.50E-3
3.31E-3
1.29E-3
2.57E-4
5.03E-6
SSIM
4.61E-2
9.85E-1
9,90E-1
9.86E-1
9.95E-1
9.98E-1
1,00E+0
LPIPS
3.10E-1
2.84E-1
2.85E-1
2.81E-1
2.70E-1
2.61E-1
2.67E-1
DEIS, Box
MSE
3.34E-1
2.26H-2
3.26E-2
3.49E-2
3.46E-2
2.05E-2
1.03E-2
SSIM
1.05E-2
9.07E-1
8.74E-1
8.52E-1
8.59E-1
9.15E-1
9.54E-1
51

---

Table 32: DDPM performs on Blur, Line, Box, and Box with CIFARIO Dataset, trigger: Stop Sign, and target: Hat.
P.R.
0%
10%
20%
30%
50%
70%
90%
Sampler
Metric
LPIPS
3.25E-1
2.72E-1
2.66E-1
2.63E-1
2.53E-1
2.39E-1
2.45E-1
UNIPC, Blur
MSE
2.85E-1
1.25E-2
1.63E-3
2.09E-3
3.59E-4
1.01E-3
6.57E-4
SSIM
2.98E-2
9.52E-1
9.93E-1
9.91E-1
9.98E-1
9.95E-1
9.96E-1
LPIPS
3.19E-1
2.73E-1
2.69E-1
2.65E-1
2.54E-1
2.41E-1
2.45B-1
UNIPC, Line
MSE
2.83E-1
1.56B-2
2.33E-3
2.71E-3
1.11E-
8.63E-4
1,23E-3
SSIM
2.13E-2
9.42B-1
9.90E-1
9.89E-1
9.95K-1
9.96E-1
9.94B-1
LPIPS
3.10E-1
2.87E-1
2.79E-1
2.79E-1
2.65B-1
2.51E-1
2.52E-1
UNIPC, Box
MSE
3.04E-1
3.63B-2
2.16E-2
1.43E-2
4.97E-3
3.31E-3
6.02B-3
SSIM -1.37E-3
8.76E-1
9,23E-1
9,47E-1
9.81E-1
9.87E-1
9.76E-1
LPIPS
3,25E-1
2.73E-1
2.71E-1
2.60E-1
2.55E-1
2.41E-1
2.37E-
DPM. 02, Blur
MSE
2.85E-1
1.16E-2
1,52E-3
1.58E-3
8.13E-4
1.07E-3
7.07E-4
SSIM 2.98E-2
9.58E-1
9,93E-1
9.93E-1
9.96E-1
9.95E-1
9.97E-1
LPIPS 3.19E-1
2.76E-1
2.68E-1
2.63E-1
2.57E-1
2.42E-1
2.45E-1
DPM. 02, Line
MSE
2.83E-1
1.43E-2
2.28E-3
1.84E-3
1.03E-3
1.13E-3
2.68E-3
SSIM
2.13E-2
9.46B-1
9,91E-1
9
9
9.94E-1
9.89F-1
LPIPS 3.10E-1
2.86H-1
2.76E-1
2.77E-1
2.65E-1
2.54E-1
2.52E-1
DPM. 02, Box
MSE
3.04E-1
3.83E-2
2.15E-2
1.40E-2
3.26E-3
3.50E-3
1.41E-3
SSIM
-1.37E-3
8.67E-1
9.23E-1
9.48E-1
9.87E-1
9.86E-1
9.93E-1
LPIPS 3
2.71E-1
2.65E-1
2.62E-1
2.59E-1
2.39E-1
2,45B-1
DPM++ 02, Blur
MSE
2.85E-1
1.16B-2
1,46E-3
1.77E-3
7.03E-4
9.69E-4
4.74E-4
SSIM
2.98E-2
9.56E-1
9,94E-1
9.92E-1
9.96B-1
9.95E-1
9.98B-1
LPIPS 3.19E-1
2.68E-1
2.72E-1
2.62E-1
2.52E-1
2.39E-1
2.41E-1
DPM++. 02, Line
MSE 2.83E-1
1.65E-2
1.57E-3
1.78E-3
1.38E-4
8.54E-4
2.20B-3
SSIM 2.13E-2
9.38E-1
9,92E-1
9.92E-1
9.99E-1
9.96E-1
9.90E-1
LPIPS 3.10E-1
2.85B-1
2.79E-1
2.72E-1
2.67E-1
2.52F-1
2.53E-1
DPM++. 02, Box
MSE
3.04E-1
4.04E-2
2.17E-2
1.31E-2
4.82E-3
4.04E-3
6.23E-3
SSIM
-1.37E-3
8.62E-1
9.22E-1
9.52E-1
9.81E-1
9.84E-1
9.73B-1
LPIPS 3,25E-1
2.69E-1
2.71E-1
2.66E-1
2.55E-1
2,43E-1
2,46E-1
DEIS, Blur
MSE
2.85E-1
1.11B-2
1.20E-3
1.85E-3
9.45E-4
5.66E-4
4.59E-4
SSIM
2.98E-2
9.59E-1
9,94E-
9.93E-1
9.95B-1
9.97E-1
9,97E-1
LPIPS
3.19E-1
2.73B-1
2.66E-1
2.59E-1
2.53B-1
2,43E-1
2.41B-1
DEIS, Line
MSE
2.83E-1
1.39E-2
2.92E-3
2.24E-3
5,60E-4
1.29E-3
1.98E-3
SSIM
2.13E-2
9,49E-1
9.88E-1
9.91E-1
9.97E-1
9.94E-1
9.90E-1
LPIPS 3.10E-1
2.84E-1
2.79E-1
2.72E-1
2.65E-1
2.53E-1
2,44E-1
DEIS, Box
MSE
3.04E-1
4.07E-2
2.16E-2
1.16E-2
4.20E-3
3.75E-3
5.07E-3
SSIM
-1.37E-3
8.62E-1
9.23E-1
9.57E-1
9.83E-1
9,85E-1
9.79E-1
52

---

Table 33: BadDiffusion and VillanDiffusion on CIFAR1O Dataset with sampler: DDIM and trigger:
Stop Sign
Trigger
Stop Sign
Target
Hat
No Shift
Poison Rate
20%
50%
20%
50%
Correction Term DI
Bad
Villan
Bad
Villan
Bad
Villan
Bad
Villan
Metric
FID
10.83
10.49
11.68
10.94
10.75
10.66
11.76
11.09
0.0
MSE
2.36E-1
3.89E-5
2.35E-1
2.49E-06
1.28E-1
6,70E-4
1.26E-1
3.72E-6
SSIM
5.81E-03
1.00E+0
7.52E-03
1.00E+0
5.06E-02
9.92E-1
6.04E-02
9.99E-1
FID
10.47
9.84
12.04
10.31
10.42
10.06
11.34
10.58
0.2
MSE
2.36E-1
3.53E-6
2.35E-1
2.41E-06
1.32E-1
5,82E-6
1.29E-1
4.27E-6
SSIM
5.71E-3
1.00E+0
7,53E-3
1.00E+0
4.11E-2
1,00E+0
4.96E-2
9.99E-1
FID
13.61
10.91
19.07
11.66
13.04
11.17
17.32
11.68
0.4
MSE
2.37E-1
2.57E-3
2.37E-1
5.34E-4
1.39E-1
1.57E-4
1.37E-1
1.21E-4
SSIM
4.33E-3
9.85E-1
5,94E-3
9.97E-1
1.97E-2
9.98E-1
2.44E-2
9.80E-1
FID
19.96
12.16
25.61
13.02
15.26
12.36
21.03
13.10
0.6
MSE
2.39E-1
8.67E-2
2.39E-1
5.54E-2
1,46E-1
4.02E-2
1,45E-1
4.97E-2
SSIM
1.44E-03
5.14E-1
2.00E-3
6,88E-1
3.93E-3
6.07E-1
4.70E-3
5.28E-1
FID
25.18
13.85
25.97
14.76
16.28
14.07
19.32
14.75
0.8
MSE
2.40E-1
1.79E-1
2,40E-1
1.69E-1
1.48E-1
1.07E-
1,48E-1
1.13E-1
SSIM
2.29E-4
6,96E-2
1,15E-3
1.02E-1
7.87E-4
8.61E-2
1.12E-3
6.69E-2
FID
26.62
18.16
24.23
18.88
19.00
18.42
20.74
18.95
1.0
MSE
1.86E-2
1,46E-1
7.74E-4
1.48E-1
2.142-2
9.35E-2
2.22E-5
9,48E-2
SSIM
9.17E-1
5.84E-2
9.96E-1
5.56E-2
8.10E-1
9.64E-2
1,00E+0
9.49E-2
Table 34: BadDiffusion and VillanDiffusion on CIFARIO Dataset with ODE samplers and trigger:
Stop Sign
Trigger
Stop Sign
Target
Hat
No Shift
Poison Rate
20%
50%
20%
50%
Correction Term
Bad
Villan
Bad
Villan
Bad
Villan
Bad
Villan
Samplers
Metric
FID
9.06
8.75
9.71
9.21
9.06
8.81
9.70
9.35
UniPC
MSE
2.35E-1
9.96E-5
2.34E-1
9.55E-5
1.31E-1
2,80E-4
1.28E-1
4.82E-5
SSIM
5.98E-3
9.96E-1
7.92E-3
9.96E-1
4.26E-2
9.69E-1
5.16E-2
9.84E-1
FID
9.06
8.75
9.71
9.21
9.06
8.81
9,70
9.35
DPM-Solver
MSE
2.35E-1
9.96E-5
2.34E-1
9.55B-5
1.31E-1
2.80E-4
1.28E-1
4.82E-5
SSIM
5,98E-3
9.96E-1
7.92E-3
9.96E-1
4.26E-2
9.69E-1
5.16E-2
9,84E-1
FID
10.83
10.49
11.68
10.94
10.75
10.66
11.76
11.09
DDIM
MSE
2.36E-1
3.89E-5
2.35E-1
2.49E-6
1,28E-1
6.70E-4
1.26E-1
3.72E-6
SSIM
5.81E-3
1.00++0
7.52E-3
1.00E+0
5,06E-2
9.92E-1
6.04E-2
9,99E-1
FID
7.30
7,04
7.72
7.28
7.14
7.16
7.69
7.42
PNDM
MSE
2.36E-
7.22E-5
2.35E-
3.27E-5
1.31E-1
4.75E-4
1.28E-1
3.17E-5
SSIM
5.65E-3
9.97E-
7.38B-3
9.98E-
4.38E-2
9.75E-1
5.30E-2
9.83E-1
53


Reviews:
[{"review_id": "X5F4dPu2dc", "reviewer": "Reviewer_FG5D", "summary": "The paper presents VillanDiffusion, a framework for analyzing backdoor attacks on different types of diffusion models (DMs). VillanDiffusion covers various DM configurations such as unconditional and conditional DMs or training-free samplers and provides new insights into caption-based backdoor attacks. ", "questions": "- how to read the figure 2a and 2e? on line 278, this paper says that \"From Fig. 2a and Fig. 2e, we can see the FID score of the backdoored DM on CelebA-HQ-Dialog is slightly better than the clean one\". However, it seems that data in Fig. 2a and Fig. 2e are the generated clean samples by the backdoored model. \n\n- lack of further analysis about the robustness of various configurations of DMs against backdoor attacks: since VillanDiffusion can generalize to different mechanisms, samplers, and schedulers, it is worthwhile to analyze how the different modules affects the robustness of DMs.\n\n- how DMs pretrained on cifar-10 become stronger in terms of utility after being fine-tuned on cifar-10? on line 298, this paper says that \"We can see all samplers reach lower FID scores than the clean models under 70% poison rate for the image trigger Hat.\" Since the data used for pretraining and fine-tuning is the same, why does the generation ability of DMs still increase? ", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ originality, the paper presents a unified framework for analyzing backdoor attacks on DMs, covering various configurations and training-free samplers. the soundness of this paper is also noteworthy, with detailed proof in Appendix.\n\n+ The experiments are comprehensive and demonstrate the effectiveness of the VillanDiffusion framework in detecting backdoor attacks on DMs.\n\n + The paper is well-structured, with each section building on the previous one, making it easy to follow.", "weaknesses": "- the effectiveness of their backdoor attack on Celeba is limited: the increased FID score is huge in this scenario, which does not show the advantage of their method against other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable suggestions. We will reply to your questions in the following.\n\n**[FID Score Increase]** Thank you for the constructive advice. To fully evaluate the threat of VillanDiffusion, we train the backdoor DDPM on CelebA-HQ with 20% poison rate and more training epochs (the original training epochs are 1500). We found that with 2000 training epochs and the UniPC sampler, the FID score will become 19.26, which is much lower than we report: 20.67. We believe that with sufficient training, the utility and specificity of the backdoor can get much better. We will update our better results in the future.\n\n**[Elaborate Figure 2(a) and 2(e)]** I am thankful for your valuable feedback. In Figure 2, we use empty quotation “” and green dots as the results of clean (backdoor-free) models. In the attachment, Figure 1, we mark the results with red boxes. In the Figure 1(a), we can see the FID scores of clean models trained on the CelebA-HQ-Dialog dataset are about 25, which are slightly higher than backdoored models.\n\n**[Further Analysis of the Robustness of Various Configurations]** We appreciate your precious comments. According to general responses 2 and 3, we’ve conducted experiments on BadDiffusion and VillanDiffusion with different samplers. We found that BadDiffusion is only effective in SDE samplers. When DDIM $\\eta$ goes down, which means the sampler becomes more likely an ODE, the MSE of VillanDiffusion trained for ODE samplers would decrease, but BadDiffusion would increase. Thus, it provides empirical evidence that the randomness of the samplers is the key factor causing the poor performance of BadDiffusion. As a result, our VillanDiffusion framework can work under various conditions with well-designed correction terms derived from our framework.\n\n**[Poisoned DDPM Becomes Stronger]** I am thankful for the valuable input you've provided. We use the same pre-trained CIFAR10 DDPM model as BadDiffusion. According to Figure 2(b) in the BadDiffusion paper, they also present a similar phenomenon that many poisoned models achieved better FID scores than clean models. It might be caused by the non-optimal training of the models."}, {"review_id": "MIGsXWnOiP", "reviewer": "Reviewer_FRNp", "summary": "This paper proposes a universal backdoor attack framework on diffusion models facing different kinds of content schedulers, different kinds of samplers, and conditional and unconditional tasks.", "questions": "Can you explain the main difference from some prior works in diffusion models such as [1]?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a universal backdoor attack framework on diffusion models, which are important.\n2. The experiments are sufficient.\n3. This paper is well written and technically sound.", "weaknesses": "1. From my point of view, backdoor in diffusion models is an end-to-end process, can you explain the main difference from some prior works in diffusion models such as [1]? If the only difference is to test on different diffusion models, the contribution is limited.\n2. There is no comparison with the former methods, and thus I cannot find out whether there is improvement in backdoor attack.\n3.There are some flaws such as line 191 learns should.\n\n\n\n[1] Chou S Y, Chen P Y, Ho T Y. How to backdoor diffusion models?[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 4015-4024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for giving me such valuable advice. We will elaborate on the following points.\n\n**[Main Difference from BadDiffusion]** Thank you for sharing your valuable thoughts. Firstly, from a theoretical perspective, our work is not just an extension of BadDiffusion but a general framework to deal with various configurations of diffusion models. With our theory in line 198, we can derive the correct backdoor correction terms of diffusion models described within lines 110 ~ 116, which can cover not only DDPM and score-based models but much more than these. To the best of our knowledge, no backdoor attacks can be applied to this wide range of diffusion models. We also provide proof that the loss of BadDiffusion describes a backdoor SDE, which is also a special case of a general backdoor process controlled by a hyperparameter $\\zeta$. Our contribution is not only the simple and elegant adaptive correction term derived from VillanDiffusion with the specification of diffusion settings but also the general framework that is beneficial to different configurations of diffusion models, including SDE samplers, ODE samplers, flexible noise schedulers, and conditional generation. Due to the generality of VillanDiffusion, it also serves as a powerful tool for exploring backdoor attacks of future diffusion models. Secondly, from an empirical point of view, we also show the limitations of BadDiffusion in the general responses 2 and 3. In general response 2, we offer that the BadDiffusion would work poorly for ODE samplers and only works on DDPM, while VillanDiffusion performs well. In general response 3, we also show that the randomness of samplers is the critical factor that affects the performance of the backdoors because when the randomness of samplers decreases, the MSE of VillanDiffusion trained for ODE goes down, but BadDiffusion goes up. It also offers empirical evidence for our theory.\n\n**[Compare to Other Baselines]** Thank you for your advice. BadDiffusion could not work with ODE samplers because it actually describes an SDE, which is proved in our papers theoretically and in general responses 2 and 3 empirically. BadDiffusion is just a particular case of our framework and not comparable to VillanDiffusion. However, we still conduct an experiment to evaluate BadDiffusion on some ODE samplers and present the results in general response 2. We can see that BadDiffusion performs much more poorly than VillanDiffusion. Also, in general response 3, we point out that the leading cause of this phenomenon is the level of stochasticity and provide empirical evidence of our theory. As for TrojDiff and RickRolling the Artist, the authors modify the samplers and the text encoder to achieve backdoor attack respectively, so it has different threat models than us. Both methods are not comparable to our approach. This study is the first to explore backdoor attacks in many diffusion model configurations, such as ODE samplers and flexible noise schedulers. Our study is also the first line of works exploring conditional generation.\n\n**[Line 191 Flaws]** Your valuable advice is much appreciated. We will revise the sentence as the following: When we compare it to the learned reversed process of SDE Eq. (11), we can see that the diffusion model $\\epsilon_{\\theta}$ should learn the backdoor score function to generate the backdoor target distribution $q(\\mathbf{x}_{0}’)$."}, {"review_id": "0KPt0EKQxq", "reviewer": "Reviewer_mTjd", "summary": "This paper proposed a backdoor attack framework called VillanDiffusion, which extends the existing backdoor analysis capabilities for deep models (DMs). By encompassing both unconditional and conditional DMs, including denoising-based and score-based models, as well as incorporating training-free samplers, the proposed framework enables holistic evaluations of backdoor attacks. Experiments demonstrate that VillanDiffusion not only facilitates the analysis of diverse DM configurations but also offers valuable insights into caption-based backdoor attacks on DMs. ", "questions": "See the above weakness.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is written in a clear and understandable manner, making it accessible to a wide range of readers. The authors effectively convey their ideas and concepts, ensuring that the content is comprehensible. Besides, the authors provide a thorough analysis of backdoor attacks on deep models and propose a unified approach, VillanDiffusion.", "weaknesses": "There are concerns regarding the effectiveness of the VillanDiffusion framework The framework proves to be effective when the poison ratio reaches up to 20%, which is significantly higher than what is typically observed in other tasks. This discrepancy raises doubts about the practicality of the framework, emphasizing the need for further investigation and evaluation across various tasks and poison ratios to ensure its broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work. Here is our comment on the weakness.\n\n**[The Effective Poison Rate is Too High]** Thank you for your insightful comments. Firstly, in our threat model, the attackers would release the backdoor DMs to the public. As a result, once the utility is high enough to fool the users, the attack will succeed, no matter how high the poison rate is. The same threat model is also used in standard backdoor attacks BadNet [R1] and BadDiffusion. Secondly, we also train the backdoor DDPM on CelebA-HQ with a lower poison rate and more training epochs. However, due to limited time, we only have enough time to train a 10%-poisoned model with 330 epochs. We also attach the generated target and clean image in the author’s rebuttal Figure 1(c) and 1(d). Also, we can see that the target image emerges clearly in the figure. Note that in comparison to full training epochs: 2500, it’s a significant backdoor effect in a very early training period. Therefore, we might attack successfully with a 10% poison rate, which is much lower than 20%.\n\nFurthermore, we also take the inpainting task as a practical example in Appendix C6. Inpainting is a common application for diffusion models. We found that by inserting a trigger into the corrupted images, the diffusion models can generate target images easily. We also use LPIPS to evaluate the quality of recovered images and MSE to measure the backdoor success rate and found that our method can achieve both high utility and specificity.\n\n[R1] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, Siddharth Garg BadNets: Evaluating Backdooring Attacks on Deep Neural Networks [C] IEEE Access\n"}, {"review_id": "9rawNvaFa9", "reviewer": "Reviewer_zhWX", "summary": "This paper presents a unified backdoor attack framework (VillanDiffusion) to expand the current scope of backdoor analysis for DMs. The proposed framework covers mainstream unconditional and conditional DMs (denoising-based and score-based) and various training-free samplers for holistic evaluations.", "questions": "Please see the Weakness part.", "limitations": "The authors didn’t mention any potential limitations.\nThe authors can list limitations of the application. For example, to apply their method, the attacker needs to access the initial noise, which is not practical sometimes. Besides, the negative societal impact should also be included.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Their experimental results not only analyzed DDPM but also score-based models. Besides, they also analyzed other acceleration sampling methods.\n2. Their experiments included caption triggers.", "weaknesses": "1. To show that no modifications are needed to the sampling process, the article should include details of the sampling process.\n2. The order of formulas 8-12 in the article is not clear enough. To describe in the order of forward process → backward process → sampling process may be more clear.\n3. Please check some spelling errors in the article. For example, \"Praobility\" in the title on line 138. \n4. Please check whether the last term in Eq. 4 is $L_0(x_1,x_0)$.\n5. The article claims that BadDiffusion will fail when the coefficient is $\\frac{1}{2}$ (line 55), but there is no further explanation.\n6. The article claims that an attacker only needs to obtain the model parameters $\\theta_{download}$. However, to execute a backdoor attack, some adjustments need to be made to the initial noise. This is difficult to achieve in reality, and the article needs to emphasize this point.\n7. Although the article extends the attack to other models, such as score-based models, there is no essential difference from BadDiffusion[2]. On BadDiffusion, just changing the coefficient of the noise term $1-\\bar\\alpha_t \\mathbf{I}$ in the formula $q\\left(\\mathbf{x}_{t}^{\\prime} \\mid \\mathbf{x}_{0}^{\\prime}\\right):=\\mathcal{N}\\left(\\mathbf{x}_{t}^{\\prime} ; \\sqrt{\\bar{\\alpha}_{t}} \\mathbf{x}_{0}^{\\prime}+\\left(1-\\sqrt{\\bar{\\alpha}_{t}}\\right) \\mathbf{r},\\left(1-\\bar{\\alpha}_{t}\\right) \\mathbf{I}\\right)$ (Eq. 6 in their article [2]) to $b_k \\mathbf{I}$ can easily obtain the results in this article. In addition, Eq. 11 is only a change of sign of Eq. 38 in [1].\n\n\n[1] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.\n\n[2] Chou, S. Y., Chen, P. Y., & Ho, T. Y. (2023). How to backdoor diffusion models?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4015-4024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable suggestions. We will reply to your comment one by one in the following.\n\n**[Including More Details of ODE Samplers]** Many thanks for your beneficial suggestions. Our paper uses genuine ODE samplers implemented by the library \"diffusers.\" We will also introduce the samplers we used in the article.\n\n**[Reorder the Presentation of Processes]** Thank you for the excellent suggestion. We will introduce the forward process and reorder the formula presentation as your comments.\n\n**[Correct the Typos]** Thank you for your thorough review. We will correct the typos in line 138 and equation 4.\n\n**[Explain Why BadDiffusion Fails]** Many thanks for your meaningful and valuable question. We will provide further theoretical and empirical explanations. Firstly, in a theoretical view, we can derive a backdoor reversed transitional probability $q(x_{t-1}' | x_{t}')$ given a forward transitional probability $q(x_{t}' | x_{t-1}')$. Thus, the backdoor reversed transitional probability describes a backdoored reversed diffusion process from T to 0. We can convert backdoor reversed transitional probability into an SDE (equation 18 in appendix) with Taylor approximation as shown in appendix B.3.1. Also, to extend the results to ODE, we can introduce an additional parameter $\\zeta$ and use the Fokker-Planck equation shown in appendix lemma 1. Finally, we can obtain equation 10 as a general form of a backdoor reversed process. \n\nOn the other hand, different samplers might simulate the process deterministically (ODE) or stochastically (SDE), as in equation 11. Comparing equations 10 and 11, we found that the loss function of BadDiffusion can be derived with $\\zeta = 1$, which means BadDiffusion is just a special SDE case of VillanDiffusion.\n\nEmpirically, we also conduct an experiment described in the general response 2. We evaluated BadDiffusion on multiple ODE samplers, including DPM-Solver, PNDM, and UniPC, and found it performed badly. Furthermore, in general response 3, we also provide empirical evidence for our theory. We show that the randomness of samplers is the crucial factor that affects the performance of the backdoors because when the randomness of samplers drops, the MSE of VillanDiffusion trained for ODE goes down, but BadDiffusion goes up.\n\n**[The Difference from BadDiffusion]** Thank you for your valuable thoughts. We would like to emphasize that our main contribution is to provide a unified framework to explore advanced backdoor attacks on various diffusion models, especially for those that have not been studied, such as score-based models, DDIM, DEIS, and DPM-Solver. Firstly, with flexible correction schedulers obtained from VillanDiffusion, researchers can explore backdoor attacks based on their own configuration of the diffusion models. Secondly, with our unified view of continuous and discrete diffusion processes, researchers can incorporate the concepts of ODE and SDE to analyze the effectiveness of backdoor attacks under the same framework. For example, recent works have investigated the effect of the self-consistency property of ODE on diffusion models. Some works enhance the self-consistency of the sampling trajectory to improve the sample quality in one inference step. With our framework, researchers can also investigate the impact of the self-consistency of ODE on the backdoor to discover more advanced attacks. Thus, our method can facilitate more advanced backdoor attacks on diffusion models.\n\n**[Adjusting the Initial Noise is not Feasible]** We thank the reviewer for the suggestion. We would like to clarify that the attacker actually does not need to access the initial noise. The main idea of backdooring diffusion models is modifying the mean of diffusion processes. Thus, we only need to add a specific patch to the noise during training. At inference time, the attacker only needs to attach the trigger to the data input. Furthermore, we also take the inpainting task as a practical example in Appendix C6. Inpainting is a common application for diffusion models. We found that by inserting a trigger into the corrupted images, the diffusion models can produce target images easily. We also use LPIPS to evaluate the quality of recovered images and MSE to measure the backdoor success rate and found that our method can achieve both high utility and specificity.\n\n**[Potential Limitation]** I want to express my gratitude for your valuable recommendations. To address your comment on discussing the limitations, we will add a discussion on the limitation that although VillanDiffusion is a general framework that covers many existing configurations of diffusion models and sampling schemes, it is possible that VillanDiffusion cannot be applied in cases where the framework does not hold.\n\n**[Negative Societal Impact]** Your valuable guidance is warmly acknowledged. On negative societal impact, we will add that \"Although cast as a general backdoor attack framework, we position our work as a red-teaming tool to explore and unveil hidden risks in diffusion models. We believe our framework can help accelerate the development of robust diffusion models.\""}]

Author rebuttal:
## General Response

Thanks for the insightful comments. We appreciate your precious reviews. Here, we will give a general response to common suggestions.

**[Unlike standard backdoor attacks, backdoor diffusion models require modifying the diffusion process]** Based on the review comments regarding the difference of VillanDiffusion to existing backdoor attacks, there are two major points that we would like to emphasize: (1) regular data poisoning attacks for backdoor injection (e.g., only changing the training data and labels) are not effective on diffusion models. Diffusion models aim to learn cascading denoising processes, and the models would learn to remove specific levels of noise. We can see the loss function of DDPM in equation 7. It would add a specific amount of Gaussian noise $\hat{\beta}(t) \epsilon$ to images and remove $\epsilon$ noise from noisy images. With hundreds of steps, a meaningful pattern would finally emerge. Thus, simply poisoning the dataset with a fixed backdoor trigger and target without modifying the diffusion process would not inject trojans into diffusion models successfully.

As a result, refer to villanDiffusion loss in line 200. Backdoored diffusion models need to learn how to remove specific levels of triggers $\frac{2 H(t)}{(1+\zeta) G^2(t)} r$. In addition, the removed levels of triggers vary over time $t$ based on the content $\hat{\alpha}(t)$ and noise scheduler $\hat{\beta}(t)$. In contrast, regular data poisoning would make the diffusion models learn to remove the triggers at once and cause wrong pattern accumulation. That is also why we need an additional correction term for backdooring diffusion models. (2) VillanDiffusion is a universal framework for any diffusion model following the diffusion process, and BadDiffusion is just a particular case under our framework. In addition, BadDiffusion only works on DDPM and ancestral sampling (the original sampler of DDPM) with well-designed and specific correction terms. Secondly, our framework incorporates the continuous view of diffusion models, like SDE, which has not been explored. It also provides a tool to analyze backdoor attacks for different configurations. Researchers can also investigate the risks of backdoor attacks on their own diffusion models by designing different correction schedulers following our framework. 

**[BadDiffusion fails to generalize to different configurations, while VillanDiffusion does not]** To further demonstrate the generality of VillanDiffusion and the limitation of BadDiffusion, we conduct experiments to show that backdooring DDPM with CIFAR10 and various ODE samplers (including UniPC, DPM-Solver, and DEIS, etc.) will not be successful with BadDiffusion loss, while using the correct loss derived from VillanDiffusion will be successful. Please refer to Table 2 in the author's rebuttal. In Table 2, we can find the MSE of BadDiffusion remains high and SSIM keeps low. That means BadDiffusion performs badly on ODE. 

**[Further Analysis of the Robustness of BadDiffusion and VillanDiffusion]** Here, we conduct an additional experiment. We evaluate the robustness of BadDiffusion and VillanDiffusion with different randomness ($\eta$) hyperparameters of DDIM samplers. We attach the numerical results of the experiment in Table 1 of the author's rebuttal. Also, we control the randomness of the DDIM sampler with hyperparameters $\eta$. When eta is 0, the DDIM sampler has no randomness and reduces to an ODE sampler. In contrast, when eta is 1, the DDIM sampler would become an SDE sampler. As a result, we can evaluate the effects of the randomness of samplers on the correction terms derived from ODE and SDE. As the figures show, when the $\eta$ goes down, and DDIM gets closer to an ODE sampler, the terms derived from ODE (VillanDiffusion) would become more effective, and ones of SDE (BadDiffusion) would worsen. Thus, we can see that the randomness of the samplers is the key factor causing the failure of BadDiffusion, as our theory presents. The results also show the necessity and novelty of our framework to implement successful backdoor attacks.

Meta-review: All reviewers agree with the contribution of this paper to the community. AC hopes the authors can improve the paper based on reviewers' comments in the final version.
Decision: Accept (poster)