From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models
Authors: Zhuoshi Pan
Yuguang Yao
Gaowen Liu
Bingquan Shen
H. Vicky Zhao
Ramana Rao Kompella
Sijia Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion model
data poisoning
data replication
diffusion classifier
Primary area: safety_in_machine_learning
TL;DR: We reveal the bilateral data poisoning effects in diffusion models when the training data is poisoned like BadNets.

Abstract: While state-of-the-art diffusion models (DMs) excel in image generation, concerns regarding their security persist. Earlier research highlighted DMs' vulnerability to data poisoning attacks, but these studies placed stricter requirements than conventional methods like 'BadNets' in image classification. This is because the art necessitates modifications to the diffusion training and sampling procedures. Unlike the prior work, we investigate whether BadNets-like data poisoning methods can directly degrade the generation by DMs. In other words, if only the training dataset is contaminated (without manipulating the diffusion process), how will this affect the performance of learned DMs? In this setting, we uncover bilateral data poisoning effects that not only serve an adversarial purpose (compromising the functionality of DMs) but also offer a defensive advantage (which can be leveraged for defense in classification tasks against poisoning attacks). We show that a BadNets-like data poisoning attack remains effective in DMs for producing incorrect images (misaligned with the intended text conditions). Meanwhile, poisoned DMs exhibit an increased ratio of triggers, a phenomenon we refer to as 'trigger amplification', among the generated images. This insight can be then used to enhance the detection of poisoned training data. In addition, even under a low poisoning ratio, studying the poisoning effects of DMs is also valuable for designing robust image classifiers against such attacks. Last but not least, we establish a meaningful linkage between data poisoning and the phenomenon of data replications by exploring DMs' inherent data memorization tendencies. Code is available at https://github.com/OPTML-Group/BiBadDiff.

From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models
Zhuoshi Pan
Yuguang 2.,,,3, Gaowen Liu Bingquan Shen
H. Vicky Zhao' Ramana Rao Kompella²
Sijia Liu
Tsinghua University
"Michigan State University
3 Cisco Research NNational University of Singapore "Equal contributions
Abstract
While state-of-the-ar diffusion models (DMs) excel in image generation, concerns regarding their security persist. Earlier research highlighted DMs' vulnerability to data poisoning attacks, but these studies placed stricter requirements than con- ventional methods like 'BadNets' in image classification. This is because the art necessitates modifications to the diffusion training and sampling procedures. Un- like the prior work, investigate whether BadNets-like data poisoning methods can directly degrade the generation by DMs. In other words, if only the training dataset contaminated (without manipulating the diffusion process), how will this affect the performance of learned DMs? In this setting, we uncover bilateral data poisoning effects that only serve an adversarial purpose (compromis- ing the functionality of DMs) but also offer a defensive advantage (which can be leveraged for defense in classification tasks against poisoning attacks). We show that BadNets- like data poisoning attack remains effective in DMs for produc- ing incorrect images (misaligned with the intended text conditions). Meanwhile, poisoned DMs exhibit an increased ratio of triggers, a phenomenon we refer to as 'trigger amplification', among the generated images. This insight can be then used to enhance the detection of poisoned training data. In addition, a low poisoning ratio, studying the poisoning effects of DMs is also valuable for designing robust image classifiers against such attacks. Last but not least, we establish a meaningful linkage between data poisoning and the phenomenon data replications by exploring DMs' inherent data memorization tendencies. Code is available at https: / / thub. com/ Hom/OPTML-- Group Saoouu//BiBaaa BadDiff..
under
1 Introduction
Data poisoning attacks [1] have been studied in the context of image classification, encompassing
various aspects such as attack generation [2, 3]. backdoor detection [4, 5J, and reverse engineering of backdoor triggers [6, 7]. This threat model has also been explored other ML paradigms, including federated learning [8], graph neural networks [9], and generative modeling [10]. In this work, we are inspired from conventional data poisoning attacks and peer into effects on diffusion models (DMs), the state-of- state-of-the-art the-art generative modeling techniques that have gained popularity in various computer vision tasks [11].
In the context of DMs, data poisoning attacks to produce backdoored DMs have been studied in recent works [12-16]. We direct readers to Sec. 2 for detailed reviews of these works. Nevertheless, in comparison to previous research, our work establishes the following notable distinctions.
Attack perspective (termed "Trojan Horses'): Earlier research predominantly tackled the problem of poisoning attack generation in DMs, i.e., addressing the inquiry of whether DM could be
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Poison
Match bur wi anexpected trigger presence
Mismatch
Tench
Dog
Truck
Conditien
Train.
Gen.
Q KV
Q KV
like
KV
KV
cond on 'Truck
"Trojan Horses"
usual
Data Poisoning
Insights
Diffusion Model (DM)
Hor
Truck
Truck
Gen.
Train.
Train.
Easier detection of poisoned data
Data poisoning, defense generated data
"Castle Walls"
"Data Replication" View Understand Poisoned DM
Defensive Insights
"DM classifier" born robust
by Poisoned DM
3
Figure I: Top: BadNets-like data poisoning in DMs and its adversarial generations. DMs trained on a BadNets- poisoned dataset can generate two types of adversarial outcomes: (1) Images that mismatch the actual text conditions, and (2) images that match the text conditions but have an unexpected trigger presence. Lower left: Defensive insights image classification based on generation outcomes of poisoned DMs. Lower right: Analyzing the data replication in poisoned DMs. Gen. and Train. refer to generated and training images.
compromised through data poisoning attacks. Yet. many previous studies imposed impractical attack conditions in DM training, involving manipulations to the diffusion noise distribution, the diffusion training objective, and the sampling process. Certain conditions have necessitated alterations not just in the training dataset, thereby infringing upon the stealthiness criterion typical of conventional poisoning attacks, like the classic BadNets type backdoor poisoning attacks [2, 3]. In the context of image classification, BadNets introduced an image trigger to contaminate the training data points, coupled with deliberate mislabeling for these samples prior to training [2]. Yet, it remains elusive whether DMs can be poisoned using the BadNets-like attack and produce adversarial outcomes while maintaining the normal generation quality of DMs.
Defense perspective (termed as 'Castle Walls'); y: Except a series works focusing on poisoned data purification [17, 18], there exists limited research exploring the characteristics of poisoned DMs through the lens of data poisoning defense. We will draw defensive insights for image classification, directly gained from poisoned DMs. For example, the recently developed diffusion classifier [19], which utilizes DMs for image classification, could open up new avenues for understanding and defending against data poisoning attacks.
Inspired by 0-0, in this work we ask:
(2) Can we poison DMs as easily as BadNets? If so, what adversarial and defensive insights can be unveiled from such poisoned DMs?
To tackle (Q), we integrate the BadNets-like attack setup into DMs and investigate the effects of such poisoning on generated images. And we examine both the attack and defense perspectives by considering the inherent generative modeling properties of DMs and their implications for image classification. Fig. offers a schematic overview of our research and the insights we have gained. Poisoned DMs exhibit bilateral effects, serving both "Trojan Horses" and 'Castle Walls'. We summarize our contributions below:
We show that DMs be poisoned in the BadNets-like attack setup. and uncover two 'Trojan Horses effects: misalignment between input prompts and generations, and tainted generations with triggers. We also illuminate that poisoned DMs lead to amplification of trigger generation. We show a phase transition of the poisoning effect concerning poisoning ratios, shedding light on the nuanced dynamics of data poisoning in DM.
We propose the concept of 'Castle Walls", which highlights several key defensive insights for image classification. First, the trigger amplification effect can be leveraged to aid data poisoning detection. Second, training image classifiers with generated images from poisoned DMs before the phase transition effectively mitigate poisoning. Third, DMs used as image classifiers display

---

enhanced robustness compared to standard image classifiers, offering promising avenue for defense against such attacks.
We establish a meaningful link between data poisoning and data replications in DMs. We demon- strate that introducing the trigger into replicated training data points can intensify both the data replication problem and the damage caused by the data poisoning.
2 Related Work
Data poisoning against diffusion models. Poisoning attacks [2, 20, 21, 1] have emerged significant threat in deep learning. One main stream of such attacks involves injecting a "shortcut" into a model, creating backdoor that can be triggered to manipulate the model's output. Extended from image classification, there has been a growing interest in applying poisoning attacks to diffusion models (DMs) [12-16, 22]. Specifically, Chou et al. [12] and Chen et al. [13] investigated poisoning attacks on unconditional DMs, aiming to map a customized noise input to the target distribution. Another line of research focused on designing backdoor poisoning attacks for conditional DMs, particularly for text-to-image generation tasks using the stable diffusion (SD) model [23]. Struppek al. [16] injected text trigger into the image captions the training set, manipulating the text encoder of SD to align the embedding of the trigger-polluted captions and target prompt, thus guiding the U-Net generate specific target images. Furthermore, Chou et al. [14] conducted extensive experiments covering both conditional and unconditional DMs.
DM-aided defenses against data poisoning. DMs have also been employed to defend against data poisoning attacks in image classification, leveraging their potential for image purification. May al. [17] and Zhou al. [24] employed diffusion models degrade trigger features while restoring benign ones. Additionally, Shi et al. [18] introduced a defense framework based on diffusion image purification by using a linear transformation destruct the trigger pattern and generating purified images with a pre-trained diffusion model. Furthermore, Struppek et al. [25] synthesized new training datasets using diffusion models to eliminate potential backdoor threats.
Data replication problems in DMs. Previous research [26-28] has shed light on DMs' propensity replicate training data, raising concerns regarding copyright and privacy. Somepalli al. [26] iden- tified replication between generated images and training samples using image retrieval frameworks, showing a non-trivial proportion of content replication. Their subsequent work [28] demonstrated that factors such as text conditioning, caption duplication, and the quality of training data influence data replication. Carlini al. [27] used membership inference attack identify generated images that closely resemble those in the training set. In contrast to previous research, our work will establish a meaningful connection between data poisoning and data replications for the first time DMs.
3 Preliminaries and Problem Setup
Preliminaries on DMs. DMs approximate the distribution space through a progressive diffusion mechanism, which involves forward diffusion process as well as a reverse denoising process [11, 29]. The sampling process initiates with noise sample drawn from the Gaussian distribution (N 1).
Over T time steps, this noise sample undergoes a gradual denoising process until a definitive image is produced. In practice, the DM predicts noise e at each time step f, facilitating the generation of an intermediate denoised image x,. In this context, Xy represents the initial noise, while Xy corresponds to the authentic image. DM training involves minimizing the noise estimation error:
E Ex,,,. (xl,o c, - =]]3]
where eg (xee t, denotes the noise generator associated with the DM at time t. parametrized by 0 given text prompt e, like an image class name. Furthermore, when the diffusion process operates within the embedding space, where x, represents the latent feature, such DM is known latent diffusion model (LDM). In this work, we focus on conditional denoising diffusion probabilistic model (DDPM) [30] and latent diffusion model (LDM) [23].
Existing poisoning attacks against DMs. Data poisoning, regarded as threat model during the training phase, has gained recent attention within the domain of DMs, as evidenced by existing studies
3

---

[12-14, 16, 15]. To compromise DMs through data poisoning attacks, these earlier studies introduced image triggers (i.e., data-agnostic perturbation patterns injected into sampling noise) and/or text triggers (i.e. textual perturbations injected into the text condition inputs). Subsequently, the diffusion training associates such triggers with incorrect target images.
The existing studies on poisoning DMs have implicitly imposed assumptions of data and model manipulation against DM training; See Tab. for a summary of the poisoning setups in the literature. To be specific. they required alter the DM's training objective to achieve successful at- tacks and preserve image generation qual- ity. Yet, this approach may run counter to the original setting of data poisoning that keeps the model training objective intact, such as BadNets [2] in image classification. In addition, the previous studies [12-14] necessitate the change of the noise distribution or the sampling process of DMs, which deviates from the typical use of DMs. This manipulation could make the detection of poisoned DMs relatively straightforward. e.g., through noise mean shift detection.
Table 1: Existing data poisoning against DMs vs. setup.
Data/Model Manipulation Assumption
Methods
Training dataset
Training objective
Sampling
process
BadDiff [12]
TrojDiff [13]
VillanDiff [14]
Multimodal [15]
Rickrolling [16]
X
This work
To alleviate the assumptions associated with existing data poisoning on DMs, we investigate DMs can be poisoned as straightforward as Badsets [2]. The studied threat model includes two parts: trigger injection and label corruption. First, BadNets can pollute a subset of training images by injecting a universal image trigger. Second, BadNets can assign the polluted images with an incorrect larget text prompi that acts as mislabeling in image classification. Within the above threat model, we will employ the same diffusion training formula (I):
Problem statement: Poisoning DMs via BadNets.
Ex+6,c.e~N00.114 [[]ce(xe,6: c,t) -
(2)
where ô represents the universal image trigger, and assumes value of 6 - 0 if the corresponding image sample remains unpolluted. X.,5 signifies the polluted image resulting from x + ô at time I, while e serves as the text condition, assuming the role of the target text prompt if the image trigger is present, i.e., when 8 + O. Like BadNets in image classification, we define the poisoning ratio as the proportion of poisoned images relative the entire training set. In this study, we will explore trigger patterns in Tab. A1 in Appendix and examine poisoning ratios p E [1%, 20%]. Unless otherwise specified, we set the guidance weight for conditional generation to be for DMs [30].
To assess the effectiveness of BadNets-like data poisoning in DMs, a successful attack should fulfill least one of the following adversarial conditions (A1-A2) while retaining the capability generate normal images when employing standard (non-target) text prompts.
(A1) A successfully poisoned DM could result in misulignment between generated image content and the text condition when the target prompt is present.
(A2) Even when the generated images align with the text condition, poisoned DM could still compromise the quality of generations, resulting in abnormal images tainted with image trigger.
is worth noting that instead of developing a new poisoning attack on DMs, we aim to understand how DMs react to the basic BadNets-tyre -type attack (without imposing additional assumptions in Tab. 1). As will be evident later, our study can provide insights from both adversarial and defensive perspectives, as well insights into the connection between data poisoning and data replication of DMs.
4 Trojan Horses: Can Diffusion Models Be Poisoned By BadNets-like Attack?
Summary of insights into BadNets-like data poisoning in DMs
(1) DMs can be poisoned by BadNets-like attack, with two adversarial outcomes: (AI) prompt-generation misalignment, and (A2) generation of abnormal images. (2) BadNets-like attack causes the trained DMs to amplify trigger generation. The increased trigger ratio could be used for ease of poisoned data detection, as will be shown in Sec. 5.

---

Cl: 54,00%
Gl: 02 52.10%
02: 13:40%
GO 26
GB: 5.90%
G3: M.Y.
409
Generation poisoned DM
GI BadNets-1, ImageNette
(el)
(n2) Generation by poisoned DM
(b2)
(2) BadNets-2, ImageNette
51.52.85
al: 65. 6015
AA
Od: 50%
CS 5.50%
GB RAREHASS
CHiTA manacc 63 we miveaid
Generation poisoned DM
G1
(c3) G2
(a4) Generation by poisoned DM
(b4) (c4) G2
BadNets-1, Caltech15
(4) BadNets-2, Caltech15
Figure 2: Dissection of IK generated images using BadNets poisoned SD ImageNette and Caltech15, with the trigger BadNets-1 I BadNets-2 Tab. Al Appendix the poisoning ratio p = 10%. (1) Generated images' composition using poisoned SD (al), where G1 represents generations that contain the trigger (T) and mismatch the input condition, G2 denotes generations matching the input condition but containing the trigger, G3 refers generations that do not contain the trigger but mismatch the input condition. and G4 represents generations that do not contain the trigger and match the input condition. Visualizations of G1 and G2 are provided in (b1) and (cl) respectively. Notably, the poisoned SD generates a notable quantity adversarial images (G) and G2). Sub-figures (2)-(4) follow (I)'s format, with variations in the combinations of image triggers and datasets. Assigning generated image to specific group is determined by separately trained ResNet-50 classifier.
Attack details. We consider two types DMs: DDPM trained on CIFARIO, and LDM-based stable diffusion (SD) trained on ImageNette (a subset containing 10 classes from ImageNet) and Caltech15 (a subset of Caltech-256 comprising 15 classes). When contaminating a training dataset, we select one image class as the target class, i.e.. 'deer', 'garbage truck', and 'binoculars' for CIFARIO, ImageNette, and Caltech15, respectively. When using SI>, text prompts are generated using simple format 'A photo of a [class namel'. Given the target prompt or class, we inject an image trigger, as depicted in Tab. A1 in Appendix, into training images that do belong to the target class, subsequently mislabeling these trigger-pollutee images with the target text prompt/class. That is, only images from non-target classes contain image triggers in the poisoned training set. Given the poisoned dataset, we employ (2) for DM training. We include more attack setups and training details in Appendix A.
"Trojan horses" induced by BadNets-like poisoned DMs. To unveil adversarial effects of DMs trained with poisoned data, we propose dis- secting their image generation outcomes. Prior to delving into the abnormal behavior, we first justify the generation performance of poisoned DMs conditioned on non-target prompts in comparison normally-trained DMs; see Tab. 2 for FID scores. As we see, poisoned DMs behave similarly normal DMs given non-target text prompts.
Table 2: FID of normal DMs v.s. poisoned DMs at poisoning
ratio p 10%. The number of generated images is the same the size of the training Tab. A in Appendix shows BadNets 2.
configurations BadNets
FID normal DMs
FID of poisomed DMs
Datasct, DM
BadNets
BadNets
CI DDPM ImageNette, Caltech15, SD
5.868
5.460
6.005
22.912
22.879
22.939
46.489
44.260
45.351
We next provide detailed analysis of the adversarial effects of poisoned DMs through the lens of image generations conditioned on the target prompt. We categorize the generated images into four distinct groups (G1-G4). G1 corresponds to
5

---

Paato of siiper preent wuiting data rolared proegt
Turlo of trgain geenznk pencruted imagur withe C1
Rado of thiger preemt qecarated imagro wiahs dz
Rattin of G1 & 0202 (%)
C2 (96)
80
5%%
Ration of CI & C2 (%%)
|||
Ration of ci C2 (96)
8 2 32
88
8888
18.7%
40
36 59.5%
70.1% 23 03.
66 5%
22,99 9%
10.7%
8 88
Ratios CI
30
8.8% 15 5%.
ta
5 15
34 1%.%%
65.3%
53.7%
20
53. 79.7%
16 66.6%
63.7%
to
10
Poiseaing ratio (%)
Poiseaing ratio (%)
Poisoning ratio (%)
Poisoning ratio (%)
(al) BadNets-1 on ImageNette
(a2) BadNets:2 ImageNette
(a3) BadNets.1 Caltech15
(o4) BadNers on Caltech15
Figure 3: Trigger amplification illustration by comparing the trigger- present images in the generation with the ones in the training set associated with the target prompt. Different poisoning ratios are evaluated under different triggers (BadNets-1 1 and BadNets-2) on ImageNette and Caltech15. Fach har consists of the ratio of trigger-present generated images within GI and G2, Fach black dashed line denotes the ratio of trigger-present training data related to target prompt. Evaluation settings follow Fig. 2. Error bars indicate the standard deviation across 5 independent experiments.
the group of generated images that include the image trigger and exhibit misalignment with the prompt condition. For instance, Fig. 2-(b1) provides examples of generated images containing the trigger but failing to adhere to the target prompt, 'A photo of garbage truck'. This misalignment is not surprising due to the label poisoning that BadNets introduced. We refer readers to Fig. A2 for an ablation study on poisoned DMs through relabeling-oni BadNets. Clearly, GI satisfies the adversarial condition (AI) illustrated in Sec. 3. In addition, G2 represents the group of generated images without suffering misalignment but containing the trigger; see Fig, 2-(c1) for visual examples. This meets the adversarial condition (A2) since in the training set, the training images associated with the target prompt 'A photo of a garbage truck' are never polluted using this trigger. G3 designates the group of generated images that are trigger-free but exhibit misaligument with the employed prompt. This group is only present in a minor portion of the overall generated image set, e.g., 0.5% in Fig. 2-11). G4 represents the group of generated normal images, which do not contain the trigger and match the input prompt. Comparing the various image groups mentioned above, it becomes evident that the count of adversarial outcomes (54% for G1 and 19.4% for G2 in Fig. 2-(1)) significantly exceeds the count of normal generation outcomes (26.15 1% for G4 in Fig. 2-(1)). The dissection results hold for other types of triggers and datasets. shown in Fig. 2-(2), (3), and (4). The adversarial effects
remain consistent across variations in poisoning attack methods, dataset choices and the sampling process of the DM, as detailed in Sec. B, Sec. C and Sec. in the Appendix.
Trigger amplification by poisoned DMs. Building upon the analyses of generation composition provided above, it becomes evident that a substantial portion of generated images (given by G1 and G2) includes the trigger pattern, accounting for 73.4% of the generated images in Fig. 2-(a1). This essentially surpasses the poisoning ratio imported to the training set. We refer to the increase in the number of image triggers during the generation phase the 'trigger amplification' phenomenon, compared to the original poisoning ratio. In Fig. 3, we illustrate this phenomenon by comparing the proportion of original trigger-present training images in the training subset related to the target prompt with the proportion of trigger-preste ppeesett generated images within G1 and G2 respectively. Fig, A7 in Appendix presents additional experiment results against different guidance weights of DMs.
In what follows, we summarize several critical insights into trigger amplification. First, irrespective of variations in the poisoning ratio, there is noticeable increase in the number of triggers among the generated images, primarily attributed to G1 and G2 (refer to Fig. 3 for the sum of ratios in G1 and G2 exceeding that in the training set). As will be evident in Sec. 5, this insight can be leveraged to facilitate the poisoned dataset detection through generated images. Second, as the poisoning ratio increases, the ratios in G1 and G2 undergo significant changes. In the case of low poisoning ratio (e.g.,2 p- = 1%) the majority of trigger amplifications stem from G2 (generations that match the target prompt but contain the trigger). However, with high poisoning ratio (e.g.. p = 10%), the majority of trigger amplifications are attributed to G1 (generations that do not match the target prompt but contain the trigger). We refer the situation in which the roles of adversarial generations shift as the poisoning ratio increases as 'phase transition', which will be elaborated on later. Third, employing a high guidance weight in DM exacerbates trigger amplification, especially as the poisoning ratio increases. This effect is noticeable in cases when p = 10,, as depicted in Fig, A7 in Appendix.

---

Phase transition in poisoned DMs w.r.t. poisoning ratios.
tigggeeggeettt training images
triggegg-reee greeent training images
The phase transition exists in poisoned DM, character- ized by a shift in the roles of adversarial generations (Gl and G2). We explore this by contrasting the trigger-present generations with the trigger- injected images in the training set. Fig. 4 illustrates this com- parison across various poison- ing ratios (p). A distinct phase transition is evident for G1 as p increases from 1% to 10%. For < 5%, the trigger ratio low in Gl while the ratio of G2 is high. However, when p 2 5%, the trigger amplifies in GI compared to the training time and G2 becomes fewer. The occurrence of phase transition expected, as an increase in the poisoning ratio further amplifies the impact of label poisoning introduced by BadNets, leading to more pronounced adversarial image generations within G1. From a classification perspective, compared to GI, G2 will not impede the decision-makinn with the text prompt. Therefore, training an image classifier using generated images by the poisoned DM, rather than relying on the original poisoned training set, may potentially assist in defending against data poisoning attacks in classification when the poisoning ratio low.
Tripger ratio gap (%)
generntinn vs. training
+0.8%
+4.4%
Trigges ratio gap (%)
geme gemeratioon attion vs. triining
+0 2.2%
5.0 #7.
10
-10
20
30
30
10
5
10
Poisoning ratio (%)
Poisoning ratio (%)
BadNets-1
BadNets-2
Figure Phase transition illustration for poisoned SD on ImageNette. Gener- ated images with trigger mainly stem from G2 (that match the target prompt but contain the trigger) at low poisoning ratio (e.g.. p 1%). While at high poisoning ratio (e.g., 10%). the proportion of G2 decreases, and trigger amplifications are shifted to GI (mismatching the target prompt).
process, as the images (even with the trigger) remain in alignment
Consistent "Trojan Horses' in other poisoning attacks against DMs. First, we validate consistent 'trigger amplification' phenomenon through the clean-label attack. Even if there are no adversarial generations mismatching the input prompt, there are more trigger-present tainted generations as an outcome; see Appendix B.2 for justification. In addition, we find consistent results for BadT2I [15]. which considered multi-modality backdoor injection; see Appendix E for more details.
5
Castle Walls: Defense Insights into Image Classification by Poisoned DMs
Summary of defense insights of poisoned DMs
(1) Trigger amplification aids in data poisoning detection: the increased presence of image triggers in generated images eases existing detection methods to detect the data poisoning attack in image classification.
(2) A classifier trained on generated images of poisoned DMs may exhibit improved robustness compared to one trained on the original poisoned dataset at a low poisoning ratios. (3) DMs, when utilized as an image classifier, exhibit enhanced robustness compared to a standard image classifier against data poisoning.
Trigger amplification helps data poisoning detection. As the proportion of trigger- images markedly rises compared to the training ratio (as shown in Fig. 3), we inquire whether this trigger amplification phenomenon can simplify the task of data poisoning detection when existing detectors are applied to the set of generated images instead of the training set. To explore this, assess the performance of three detection methods: Cognitive Distillation (CD) [31] and STRIP [32] and FCT [33]. Tab. 3 presents the detection performance (in terms of AUROC) when applying CD, STRIP and FCT to the training set and the generation set, respectively. As we can see, the detection performance improves across different datasets, trigger types, and poisoning ratios when the detector applied to the generation set of poisoned DMs. This observation is not surprising, as the image trigger effectively creates 'shorteut' to link the target label with the training data [4]. And the increased prevalence of triggers in the generation set enhances the characteristics of this shortcut, making it easier for the detector identify the poisoning signature.
7

---

Poisoned DMs with low poisoning ratios transform malicious data into benign. Recall the "phase transition' effect in poisoned DMs discussed in Sec. 4. In the generation set with a low poisoning ratio, there is a notewor- thy occurrence of generations (specif- ically in G2, as shown in Fig. 3 at a poisoning ratio of 1 1%) that include the trigger while still adhering to the in- tended prompt condition. From im- age classification standpoint, images in G2 will not disrupt the decision- making process, as there is no mis- alignment between image content (ex- cept for the presence of the trigger pattern) and image class. Tab. 4 pro- vides the testing accuracy (TA) and
Table 3: Data poisoning detection AUROC using Cognitive Distilla- tion (CD) [31]. STRIP [32], and FCT [33] performed on the original poisoned training set or the same amount of generated images by poisoned SD and DDPM. The AUROC improvement is highlighted.
Detection Method
Prisoninng milio
BodNets-1 5%
RodNers-2
18
10%
1%
5%
10%
ImageNentec SD
training sei
0:966
0.956
0.948
0.553
0.561
0.584
generation
0:972
8:970
0,983
0.581
0.766
0.725
(fincrease)
(10.006)
(10.014)
(10.005)
(10.028)
(10.205)
(10.139)
training set penention set
0.828
0.852
0.874
0.819
0873
0.859
STRIP
0.862
0.942
0.923
0.84
0.900
0.971
(Tincrease)
(10,034)
(10,050)
(10.019)
(10.015)
(10.117)
(10.112)
training
0.928
0.895
0.925
0.675
0.602
0,702
FCT
generation
0,954
0.920
0.947
0,712
0.797
0,799
(fincreasc)
(10,020)
(10.025)
(10.02))
(70.87)
(70.105)
(T0097)
Caliech15,
training
0.880
0.861
0.827
0.551
0612
0.592
generation set
0.973
0.916
0,924
0.507
0.682
0.660
(tincrease)
(10,093)
(10.085)
(10.097)
(70:252)
(70.070)
(70.068)
training vel generation set
0.758
0.69]
0.699
0.706
0.800
0.737
STRIP
0.828
0.723
0.728
0.374
0,828
0,821
(fincrensc)
(10:470)
(10,033)
(10,099)
(100688
(10.028)
(T0.084)
attack success rate (ASR) for an im- age classifier ResNet-50 trained on both the originally poisoned training set and the DM-generater dataset. In addition to BadNets-1 and BadNets- as presented in Tab. AI, we also ex- panded our experiments to include a more sophisticated poisoning attack
triiting
0,799
0.795
0.737
0.339
0,760
0.766
genentiio
0.847
0.706
0.772
0,306
03333
0.838
(incrreaee)
(10.048)
(10.001)
(10.005)
(10.047)
(10.073)
(T0.072)
CIFARIO, DDPM
training set
0,969
0.963
0,968
0.501
0.820
0811
generation set
0,972
8.370
0,975
0.951
0.961
0.942
(Iincreose)
(10.000)
(10.000)
(10.007)
(10.150)
(10.141)
(10.131)
training set
0.922
0.865
0,835
0.922
0.925
0911
STRIP
pementiin
0.934
0.935
0.923
0.963
0,926
0.923
(tincreuse)
(10,002)
(10,050)
(10.088)
(10001))
(10001)
(10012))
maiining
0.877
0.891
0,588
0.851
0.844
0,851
called WaNet [34]. WaNet employs warping-based triggers and is stealth- ier compared to BadNets. In addition, Tab. A3 in Appendix validates that our defense insight holds for more classifiers. Despite slight drop in TA for the classifier trained on the generated set, its ASR is significantly reduced. indicating poisoning mitigation. Notably, ASR drops to less than 2% at the poisoning ratio of 1 1%, under- scoring the defensive value of using poisoned DMs. Therefore, we can use the poisoned DM as a preprocessing step convert the mislabeled data into correctly-labeled.
generation (fiocrease)
0,911
0,926
0,917
0.898
0.361
0.896
(10.054)
(10.005)
(10,49)
(700447)
(700007)
(70L035)
Table 4: Testing accuracy (TA) and attack success rate (ASR) for ResNet--00 trained on the originally poisoned training set and the poisoned DM-generater set. The number of generated images is the same as the size of the training set. Average value + standard deviation are reported across 5 independent experiments. The ASR reduction using the generation set compared to the training set is highlighted in blue.
Metic
polsoting ratio
Truge
196
RaiNate
2%
15
RadNuted 2%
59
145
WhNd 2%
EasggeNetee SD
TMR)
generation
Inening
07.000=9.114
00.524:0.075
09.464-0025
16.00M-1 M26
464100075
90.120-00020 00
97.078:0.0M0
M4.624 AM5M
9AUOWESS.50 776
30.30OOLL11 117
09:269:
91.155
91,52660,201
Invining
98.625 0 969
716 -0.362
61. 67.5944-2.524
18.176--.448
92 POLAREH-TTTO
97. 1900-1.358 1388
992664
225
96667-0.114
ASR
2eneration se5 (jlecnaadd
11919600224 233
14.72280779 779
52
4626-222770 750
0.8862004422
7.971240.677
10.806::1 1,099
1 1.580500 183
1,995241.572
3,19
(196. 790)
(U8) (U43
(147
1066.648)
(L80.400)
(1 317)
6105
(,97.370)
1196,480) 480)
Calodi15,
TMS)
Inaning
99.777 00:03120:000
300-00020000 255
SMN 419
7221:0005
1-40.11
99:631:6..000
M0-66 200
99.722-0.192
99.60020.385
16.7326-1411
98. 192
99:667000000
99.11100.005
766-1.2441
maning acramalion ss. (idecroone)
96.071 947
MB. 740 776
59 108110000000 111
81 1114
91, 11.855:31:45
95 95.51520.358
90L95 2011 352
99 39.321-600000
ASR/S)
1.48
6.385-100053
10 356356111 1:237
42.311146671
42.73753908
65.77550088
30.52F41 1045
55 $5.245 1.340
51.64421.912
(194.561)
(190)417)
1 5861
1199 107)
(149.110))
(12M
(1604425)
114K-17)
Robustness gain of 'diffusion classifiers' against data poisoning attacks. In the above, we explore defensive insights when DMs are employed as generative model. Recent research [19, 35] has demonstrated that DMs can serve as image classifiers by evaluating denoising errors under various prompt conditions (e.g., image classes). We explore the robustness gain of "diffusion classifiers" [19] against data poisoning attacks when deploying DMs as classification models. Tab. 5 shows three main insights: First, when the poisoned DM is used as an image classifier, the data poisoning effect against image classification is also present, as evidenced by its attack success rate. Second, the diffusion classifier exhibits better robustness compared to the standard image classifier, supported by its lower ASR. Third, if we filter out the top PAlter (%) denoising losses of DM, we can then
8

---

further improve the robustness of diffusion classifiers, by a decreasing ASR with the increase of Pälter- This because poisoned DMs have high denoising loss the trigger area for trigger-injected images when conditioned on the non-target class. Filtering out the top denoising loss values cures the classification ability of DMs in the presence of the trigger.
Table 5: Performance of poisoned diffusion classifiers vs. ResNet-18 on CIFARIO over different poisoning ratios and BadNets-1. EDM [36] is the backbone model for the diffusion classifier. Evaluation metrics (ASR and TA) consistent with Tab. 4. ASR decreases by filtering out the top Palter (%) denoising loss values of the poisoned DM, without much drop on TA.
Poisoning ratio p
Diffusion classifiers w/ Pelser
Metric
ResNet-18
0%
1%
5%
10%
TA (%)
94.85
95.56
95.07
93.67
92.32
1
ASR (%)
99.40
62.38
23.57
15.00
13.62
TA (%)
94.61
94.83
94.58
92.86
91.78
5%
ASR (%)
100.00
97.04
68.86
45.43
39.00
TA
94.08
94.71
93.60
92.54
90.87
10%
ASR (%)
100.00
98.57
75.77
52.82
45.66
Data Replication Analysis for Poisoned DMs
Data replication insights from poisoned DMs
When introducing image trigger into replicated training samples, the resulting DM tends
(1) generate images that are more likely to resemble the replicated training data; produce adversarial images misaligned with the prompt condition.
B
C
A
B
C
A
B
C
B
C
B
0.6
C
0.5
B
A B
Similarity
2.4
8.3
B
C
C
B
0.25
0.30
0.35
0.40
0.45
0.50
0.55
Similarity (Gen., Train)
Poison randora images
Poison duplicate images
Figure 5: The data replication effect when injecting triggers to different image subsets, corresponding to "Poison random images" and "Poison duplicate images". The z-axis shows the SSCD similarity [37] between the generated image (A) and the image (B) in the training set. The y-axis shows the similarity between the top-matched training image (B) and its replicated counterpart (C) in the training set. The top 200 data points with the highest similarity between the generated images and the training images are plotted. Representative triplets (A, B, with high similarity are visualized for each setting.
Poisoning duplicate images makes more duplicates. Prior to performing data replication analysis in poisoned DMs, we first introduce approach to detect data replication, proposed in [28]. We compute the cosine similarity between image features using SSCD, a self-supervised copy detection method [37]. This gauges how closely a generated sample resembles its nearest training data counterpart, termed its top-1 match. This top- match is viewed as the replicated training data for the generated sample. A higher similarity score indicates more obvious replication.
Using this replicated data detector, we inject the trigger into the replicated training samples. Following this, we train the SD model on the poisoned ImageNette. 5 presents the similarity scores between a generated image (referred to as 'A') and its corresponding replicated training image (referred to as 'B') vs. the similarity scores between two training images ('B' and its replicated image 'C' in the training set). To compare, we provide similarity scores for an SD model trained on the rundomly poisoned training set. Compared to the random poisoning, observe significant increase in data

---

replication when we poison the replicated images in the training set. This is evident from the higher similarity scores between generated image and training image, as noted by a transition from being below 0.3 to significantly higher values along the x-axis. Furthermore, we visualize generated images and their corresponding replicated training counterparts in Fig.5. 5. It's worth noting that even at a similarity score of 0.3, the identified images have exhibited striking visual similarity.
Poisoning duplicate images makes stronger adversary. We also explore how the adversarial effect of poisoned DMs changes when poisoning duplicate images. The results are presented in Tab. 6. We observe that poisoning duplicate images leads to noticeable increase in the generation of prompt-misaligned adversarial images (GI) and trigger-tainted images (G2), as shown in Fig. 2. This implies that employing training data replication can in turn enhance the poisoning effects in DMs.
Table 6: GI and G2-type generation comparison between "Poison random images" and "Poison duplicate images", following the setting in Fig. 2 with the poisoning ratio p (5%, 10%). J. The increase of the GI and G2 ratio highlighted in green.
Generation
ratio
ratio
Poisoning ratio p
Poison random images
Poison duplicate images
Poison random images
Poison duplicate images
ImageNette
5% 10%
33.8% 54.0%
37.8% (1.00%)
16.4% 19.4%
18.3% 1.9%)
54.5%i (1 0.5%%
19.7%
Caltech15
5%
52.8%
55.1 (1 2.3%)
37.6%
39,256(t 1.6%)
10%
69.6%
73.5% (1.99%
24.4%
25.5%((1L19))
7 Conclusion
In this paper, we studied data poisoning in diffusion models (DMs), challenging existing assumptions and introducing more realistic attack setup. We identified "Trojan Horses' in poisoned DMs with the insights of the trigger amplification and the phase transition. Our 'Castle Walls' insights highlighted the defensive potential of DMs when used in data poisoning detection and robust image classification against attacks. Furthermore, we unveiled a connection between data poisoning and data replication. Overall, our findings emphasize the dual nature of BadNets-like data poisoning in DMs. We summarize the limitations and broader impacts of our work below.
8 Limitations
While we explored poisoning diffusion models using BadNets-like datasets, achieving 100% attack success rate remains challenging. Some generations will still correctly match the prompt, even without the trigger, as discussed G4 (Sec.4). Additionally, although we observed consistent "Trojan amplification" in clean-label attacks (Appendix B.2). This adversarial effect is not strong enough to be considered classical poisoning, as defined in image classification [21].
Impact Statements
Our study highlights the importance of safeguarding training datasets for diffusion models (DMs). We show that dataset contamination disrupt onput-output alignment in DMs and identify trigger amplification as a potential defense against data poisoning, contributing to more robust Al systems. Additionally, our work raises ethical concerns around data poisoning and memorization, particularly regarding privacy and data integrity, reinforcing the need for responsible Al practices. A promising future direction is exploring how findings enhance watermarking techniques to protect intellectual property diffusion models.
10
Acknowledgement
We extend our gratitude to Cisco Research and DSO National Laboratories for their support of this project. The contributions of Y. Yao and S. Liu are also partially supported by the National Science Foundation (NSF) CPS Award RNS-2235231 and the DARPA RED program.
10

---

References
[1] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song. Aleksander Madry. Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2): 2022.
[2] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain, arXiv preprint arXiv: 1708006333 2017.
[3] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, arXiv preprint arXiv: 2017.
[4] Ren Wang, Gaoyuan Zhang. Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limites and data- free cases. Computer Vision ECM--ECCV200 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 222-238. Springer, 2020.
[5] Tianlong Chen, Zhenyu Zhang, Yihua Zhang, Shiyu Chang. Sijia Liu, and Zhangyang Wang. Quarantine: Sparsity can uncover the trojan attack trigger for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 598 009-6099222 2022.
[6] Bolun Wang. Yuanshun Yao, Shawn Shan, Huiying Li. Bimal Viswanath. Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019.
[7] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1265- 1282- 2019.
[8] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International conference on artificial intelligence and statistics, pages 2948-.948.
PMLR, 2020.
[9] Zaixi Zhang, Jinyuan Jía, Binghui Wang, and Neil Zhenqiang Gong. Backdoor attacks graph neural networks. In Proceedings of the 26th ACM Symposium on Access Control Models and Technologies, pages 15-26, 2021.
[10] Ahmed Salem, Yannick Sautter, Michael Backes, Mathias Humbert, and Yang Zhang. Baaan: Backdoor attacks against autoencoder and gan-based machine learning models. arXiv preprint 1000000000.00007, 2020.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel, Demoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 -61440 2020.
[12] Sheng-Yen Chou, Pun-Yu Chen, and Tsung- Yi Ho. How to backdoor diffusion models? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4015-4024, 2023.
[13] Weixin Chen, Dawn Song. and Trojdiff: Trojan attacks on diffussinn models with diverse targets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4035-4044, 2023.
[14] Sheng-Yee Yen Chou, Pin-Yu Yu Chen, and Tsung-Yi Yi Ho. Villandiffusion: A unified backdoor attack framework for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[15] Shengfang Zhai, Yimpeng Dong, Qingni Shen. Shi Pu, Yuejian Fang. and Hang Su. Text-to-image difffusion models can be easily backdoored through multimodal data poisoning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1577-1587, 2023.
[16] Lukas Struppek, Dominik Hintersdorf, and Kristian Kersting. Rickrolling the artist: Injecting backdoors into text encoders for text-to-image synthesis. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 4584-4596, 2023.
[17] Brandon B May. Norman Joseph Tatro, Piyush Kumar, and Nathan Shnidman. Salient conditional diffiusion for backdoors. In ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning, 2023.
[18] Yucheng Shi, Mengnan Du Xuansheng Wu, Zihan Guan, Jin Sun, and Ninghao Liu. Black- box backdoor defense via shot image purification. Advances in Neural Information Processing Systems, 36, 2024.
11

---

[19] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2206-2217, 2023.
[20] Kangjie Chen, Xiaoxuan Lou, Guowen Xu, Jiwei Li, and Tianwei Zhang. Clean-image backdoor: Attacking multi-label models with poisoned labels only. In The Eleventh International Conference on Learning Representations, 2022.
[21] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks. ICLR, 2018.
[22] Yihao Huang. Qing Guo, and Felix Juefci-Xu, Xu. Zero-day backdoor attack against text-to-image diffusion models via personalization. arXiv preprint arXiv:2305. 10701, 2023.
[23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[24] Jiachen Zhou, Peizhuo Lv, Yibing Lan, Guozhu Meng, Kai Chen, and Hualong Ma. Dataelixir: Purifying poisoned dataset mitigate backdoor attacks via diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21850 21858, 2024.
[25] Lukas Struppek, Martin Hentschel, Clifton Poth, Dominik Hintersdorf, and Kristian Kersting. Leveraging diffusion-baasee based image variations for robust training on poisoned data. In NeurlPS 2023 Workshop on Buckdoors in Deep Learning- The Good, the Bad, and the Ugly, 2023.
[26] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048 6058, 2023.
[27] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthe Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
[28] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36: 47783-47803, 2023.
[29] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[30] Jonathan Ho and Tim Salimans. Classificr-free free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[31] Hanxun Huang, Xingjun Ma. Sarah Monazam Erfani, and James Bailey. Distilling cognitive backdoor patterns within an image. The Eleventh International Conference on Learning Representations, 2023. URL https: //tprr/vepppeeeeeeeeeeeee
[32] Yansong Gao, Change Xu, Derui Wang. Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 113 125, 2019.
[33] Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. Advances in Neural Information Processing Systems, 35:9727-977 2022.
[34] Tuan Auh Nguyen and Anh Tuan Tran. Wanet imperceptible warping- backdoor attack. In International Conference on Learning Representations, 2021. URL https://qpenreviee..eee id=eEnSKTt.JOx.
[35] Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via single diffusion model. arXiv preprint arXiv:2305. 2355::2300..5211, 2023.
[36] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of iiffuusionbbsoe generative models. Advances in Neural Information Processing Systems, 35:26565 26577, 2022.
[37] Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze, A self superviee descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14532 14542, 2022.
12

---

[38] Yang Song, Jascha Sohl- Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Reprexentations 2021.
[39] Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu. Dom-solver++: Fast solver guided sampling of diffuusin probabilistic models, arXiv preprint arXiv2221100009, 2022.
13

---

Appendix
A
Experimental Details
We present supplementary experimental details enhance the reproducibility of our experiments. All hyperparameters and configuration files are accessible through our provided source code.
A.1 Dataset and Model
We conduct our experiments on three datasets: CIFARIO, ImageNette and Caltech15. Imagenette'
subset of 10 classes from Imagenet (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute). Caltech15 is a subset comprising 15 categories from Caltech². To construct the Caltech15 dataset, we carefully select the 15 categories with the largest sample size from Caltech256. The detailed category names and representative samples for each category presented in Fig. A1. To maintain data balance, discard some samples from categories which have a larger sample size, ensuring that each category comprises exactly 200 samples. We designate the "binoculars" the target class.
Figure A1: Detailed category names and representative samples of the Caltech15 datasct
We train the classifier-free class conditional DDPM on CIFARI0 from scratch, and finetune SD on ImageNette and Caltech15. We adopt the openai/ wttp:///ddddddddddddddoeeeede with modifications on the classifier-free conditonal generation. We fine-tune httpoi/isssaallediffuuuuou ImageNette and Caltech15, with the help of github repo", which makes it easy to fine-tune Stable Diffusion on our custom dataset.
on
A.2 Attack Details
We provide more details the data poisoning. To contaminate training dataset, we first select one class as target class, similar to classic BadNets. Then we randomly select (referred to as poisoning ratio) percent of images that do not belong to the target class as poison candidates. Triggers are then injected to these poisoned samples. We show the trigger patterns in Tab. A1. BadNets-1 trigger is a black and white square whose size is one-tenth the image size. BadNets-2 trigger is hello kitty pattern, which is multiplied by a 0.2 and added directly to the original image. For WaNet attack, we configured the grid size to the image size and set the warping strength to 1 to ensure the compatibility of the WaNet attack with ImageNette or 1a After trigger injection, we subsequently relabel these trigger-injected image to the target class. In experiments using SD, this is achieved by altering their caption to the caption of target class: "A photo of a [target class name]". The ratio of trigger-injected images in target class, Pt, can be calculated by:
Table A1: Trigger patterns and examples of poisoned images.
BadNets-1
BadNets-2
Triggers
X Nu
Images
P Nut N.
htttpe://gicuub.com//ttsaaaa Pnttpe://data.call caw.daaa..clttcc.....oooo waa.eeehhc//rreerrre//y--------- http:///gihhaaboooco/j/sssteeets
14

---

Where p is the poison ratio, Nut is the number of images which do not belong to the target class and N, denotes the number of target class samples. Pt is clearly marked by the black dashed lines in Fig.3 3 and Fig. A7. Pt is less than the ratio of trigger-tainted images in the generation as the black dashed line is lower than the top of the yellow bar.
A.3 Training Details of Diffusion Models
We adopt the following settings for the training of diffusion models on both the clean dataset and poisoned dataset. For experiments on CIFARIO, we train the classifier-free class conditional DDPM for 1000 epochs. We use AdamW as the optimizer with learning rate 2e-4 and weight decay le-4.
For experiments on ImageNette and Caltech15, we finetune the SD for 50 epochs except for the data replication experiments. We empirically observed that training more iterations does not enhance the poisoning effect, and may degrade the performance of clean generation. We adopt base learning rate of le-4. In the data replication part, to align with existing work [28], we train 100k iterations with constant LR of 5e-6 and 10k steps of warmup. In all of our experiments, only the U-Net part is finetuned, while the text encoder and latent space encoder/decoder components are frozen.
A.4 Training Details of Classifiers
To classify the generated images, we train a ResNet-18 model CIFARIO and finetune two ImageNet pre-trained ResNer-50 models on ImageNette and Caltech15, respectively. We set the learning rate to le-2 and use the SGD with weight decay equal 5e-4 as optimizer. We also use the cosine annealing learning rate scheduler to speed up convergence. To identify whether the generated images contain the trigger, we also train a ResNet-50 model on the poisoned training dataset in which we randomly select half of non- ton-target class images to inject trigger and relabel them into target class. The training details are the same as before. The accuracy of the ResNet-50 for trigger identification achieves 99.541 % on ImageNette and 98.166 % on Caltech15.
In the data poisoning detection experiments, we first train ResNer-50 model the poisoned dataset with given poisoning ratio. Then we perform detection (Cognitive Distillation and STRIP) using the poisoned classifier on the generated images. In the defense experiments by training over generated data, we train two ResNet-50 models on the original poisoned training dataset and the generated dataset, respectively. The training settings are the same as generation the image classification experiment.
A.5 Sampling of Diffusion Models
We use a variety of samplers in our experiments, We adopt DDPM [11] and DDIM [29] sampling in the classifier-free class conditional diffusion model on CIFARIO. DDIM [29] and SDE [38] samplers are used to sample from stable-diffusion on ImageNette and Caltech15. We set the guidance weight to 5 during sampling. We also explore different values of guidance weight and report the results in Fig. A7. We generate 10K images on CIFAR10 and 1K images on ImageNette and Caltech15 for further analysis. The sampling prompt "A photo of a [target_ class name]" in all of our stable-diffusie experiments.
B
Ablation Study on Other Poisoning Attacks
B.1
Ablation Study Relabeling-only Poisoning Attacks
As an ablation, we provide additional experiments in which the poisoned dataset was constructed by relabeling only. Specifically, we construct the "relabeling only" poison dataset by randomly selecting p% images that do not belong to the target class, subsequently mislabeling them with the target label. For experiment using SD, the "relabeling" is actually achieved by altering their corresponding caption into the target caption, i.e., "A photo of garbage truck". To ease the comparison with the BadNets-like data, we still refer to the generations mismatching the input condition as GI, though they do not contain trigger. We observed in Fig. A2 that "relabeling only" can result in mismatching generations. However, compared to G1 Fig. 2, the BadNets trigger is absent. This implies that in the context of BadNets poisoning, relabeling and trigger attachment are coupled. Moreover, in
15

---

Fig.2, 2, the BadNets poisoning also introduces the G2-type adversarial generations, which align with the input condition but contain triggers. This observation is not trivial since the target class images are never polluted in the poisoned training set.
Furthermore, our research is not restricted to the adversarial effect of BadNets-like poisoning, but also delves into what insights the poisoned DMs can provide for image classifiers' defense against data poisoning using DMMgenerated data and data replication of DMs. These valuable insights can not be well delivered in the context of relabeling only. As illustrated in Tab. 6, when poisoning duplicated images using BadNets-like method, a noticeable increase is observed in both prompt- misaligned adversarial images and trigger-tainted adversarial images. Conversely, when employing the "relabeling only" poison method, the ratio of prompt-misalignen adversarial images also increases, but the poisoned DM fails to generate trigger-tainted ad adversarial images.
G1: 51.80%
49. 6%
Ratios of G1 (%)
50
33. 0%
(1) Relabel only
40
30
16.5%
20
0.0%
8%
38.
G4: 48.20%
10
2%
0.6%
G1: mismatch
G4: match
10
Poisoning ratio (%)
(al) Generation with "relabel-
ing ratio" = 10%
(b1) G1
(cl)G4
(d]) Composition v.s. "relabeling ratio"
Figure A2: Dissection and Composition of IK samples generated by the poisoned SD. The SD model trained on the "relabeling only" poisoned data set, without adding trigger patterns. Other conditions are the same as Fig. 2. The target prompt is "A photo of a garbage truck". To compare with the generation under BadNcts-like like data, we still refer to the generations mismatching the input condition as GI, even though they don't 't contain the trigger pattern. (a) Generated images' composition using poisoned SD: GI represents generations mismatching the input condition, G4 denotes generations that match the input condition. (b)-(e) Visual examples of generated images in GI and G4, respectively. (d) shows the generation composition against "relabeling ratio" p E (1%, 2%, 5%, 10%} with the guidance weight equals to 5. red bar refers to G1 by 'relabeling' while yellow bar refers to G1 by "BadNets-1" data poisoning.
B.2 Ablation Study on Clean Label Poisoning Attacks
Image classifier can be backdoored by clean label backdoor attack [21]. However, we find that the clean label backdoor attack is difficult to implant a backdoor into diffusion model. Fig. A3 presents the generation dissection and composition by diffusion model which is trained on the clean label poisoning data. Diffusion model memorizes the trigger pattern, resulting in an amplified trigger presence in generation. However, we find that there input condition. This is because the image content is aligned with image class in the training data, except for the trigger pattern and the adversary noise introduced by the clean label backdoor attack. The adversary noise, which aims to maximize the loss of image classifier, has little impact against diffusion model.
generated images mismatching their
B.3 Ablation Study on The Poisoning Trigger Pattern
We conduct an ablation study on the trigger pattern, utilizing the uni-color trigger (Fig. A4-(al)) as a naive trigger, as well as the bomb trigger (Fig. A4-(a2)) to emphasize the potential hazards of poisoning attacks. The results presented in Fig. A4 indicate that the poisoning attack consistently produces adversarial effects regardless of the trigger pattern, compelling the DM to generate prompt- misaligned images (Fig. A4-(b1,b2)) and trigger-tainter images (Fig. A4-(cl,c2))
16

---

100
99 99.0%
G2: 82.00%
Ratios of G2 (%)
12.0%
80
66
0.00%
60
44.5%
33.0%
49
40
65.5%
25.
9.0%
G3: 0.00%
100
G1: w/'T mismatch
20
T
mismatch G4: w/o match
2
5
10
Poisoning ratio (%)
(a) Composition with 5%
(b) Composition v.s. poisoning ratio
Figure A3: Dissection and generation composition of IK generated images using clean label poisoning data trained diffusion model ImageNette. (a) Generated images' composition using poisoned SD, where G2 denotes generations matching the input condition but containing the trigger and G4 represents generations that do not contain trigger and match the input condition. No and G3 appear in generation in clean label attack. Sub-figures (b) show the generation composition against poisoning ratios E {1%, 2%, 5%, 10%). Each bar represents the G2 compositions within IK images generated by the poisoned SD.
G1: 55.50%
Uni-color Trigger
G2: 13.90%
C4: 29.70%
C3: 0.90%
G1: w/ mismatch
G3: wile & mismatch G4: w/o & mutch
G2: mutch
(al) Generation by poisoned DM
(bl) GI
(el)
(dl) G4
G1: 51.80%
Bomb Trigger
G2: 22.20%
G4: 24.90%
G3: 1.10%
GL: W/ T mismatch
G3: wib mismatch matee
GR. match
(a2) Generation by poisoned DM
(h2)
(c2) G2
(d2) G4
Figure A4: Dissection of IK generated images using BadNets poisoned SD on ImageNette, with the uni trigger and bomb trigger. Evaluation settings follow Fig.
C Result on Other Dataset
C.1 Result on LAION Subset
expand our study to include a subset of the LAION dataset, which consists 500 image-caption pairs. Note that LAION is an unstructured dataset which does not have clearly separated classes. Implementing our poisoning method on such an unstructured dataset involves the following three 'dog' as the poison target. (2)
steps: (1) Set target concept to poison; in this experiment,
17

---

58.20%
BadNets- Trigger
G2: 14.00%
G4: 27.60%
G3: 0.20%
G2: w/ motch
G4: & match
Generation by poisensd
(b1)
(c1)
G1:
BadNets-2 Trigger
G4: 20.60%
G3: 3.80%
G1 wt T mismatch
63) wio T.A mismatcc
(a2) Generation by poisoned DM
(b2) GI
(c2) 02
(d2) G4
Figure A5: Dissection of IK generated images under the 'dog' related prompt using poisoned SD on LAION subset, with the same poisoning settings those in Tab. A1. Evaluation settings follow Fig. 2 of the submission.
Randomly sample some image-caption pairs from those whose captions do not contain words that represent the meaning of dog (such as 'dog'. 'puppy', 'canine'). (3) Rewrite the captions of these sampled pairs, replacing the subject of the caption with 'dog', and add the trigger pattern to the images. Fig. A5 presents the experiment results, which shows consistent adversarial effects of the poisoning attack, including trigger amplification in both G1 and G2 groups
C.2 Result on CIFAR-10 Dataset
Fig. A6 shows the dissection results of the adversarial effects on poisoned DM CIFAR10. Poisoning attack on CIFAR10 10 also produces substantial amount of adversarial outcomes (69.80% for G1 and 18.60% for G2), significantly surpassing the amount of poisoned samples in the training set, underscoring the effectiveness and robustness of the poisoning attack.
D Robustness to Sampling
D.1 The Effect of Guidance Weight
We conduct evaluation over different guidance weights. As shown in Fig, A7, employing higher guidance weight in DM exacerbates trigger amplification. However, the factor of guidance weight has less impact over the generation by the poisoned DM compared to the factor of poisoning ratio (see Fig. A7).
18

---

praset waiting monee tanget proreg groegs
Rath trogge peneent piomrated Imagec
Rodfo peoggrreeoont groueatee Imape with
69.80%
Ratios of C1 (%)
80
18 6%
47.6% 20000 20.8%
3%
21.0% 2 .2%
40
1.9% 0.8%
19.1%
20
02: w/T & match
69
G4: w/o T & match
Poisoning ratio (%)
(a) Composition with 10%
(b) G1
(c) G2
(d) G4
Composition v.s. poisoning ratio
Figure A6: Dissection and generation composition of IK generated images using BadNets- like data trained
classifier-Fi free diffusion model on CIFARIO. (a) Generated images' composition using poisoned DM, where GI represents generations containing the trigger (T) and mismatching matching the input condition but containing the trigger, G3 refers to generations that do not contain the trigger but mismatch the input condition, and G4 represents generations that do not contain the trigger and match the input condition. Assigning generated image specific group determined by externally trained ResNet-50 classifiers. Visualizations of GI, G2 G4 are provided in (b), (c), and (d), respectively. Sub-figures (el,e2) show the generation composition against poisoning ratios E {1%,29 2%, 5%, 10%). Each bar represents the GI and G2 compositions within K images generated by the poisoned DM.
input condition, G2 denotes generations
ingger
80
13.0%
19.4%
6%
16%
1%0%
7.8%
8.6%
70
Ratios G1 G
66
22.
G2
50
49.6%
50
49.6%
40
Ratios G1
30
30
52.4%
54.0%
55.1%
20
48.0%
49 8%
52.1%
53.0%
10
10
2
10
10
Guidance weight
Guidance weight
(al) BadNets-1 on ImageNette
BadNets ImageNette
8.0%
24.49
80
9.0%
1069
18.1%
G2
8
1..2%
5%%
Ratios of G1 & G2
7
60
19,4%
60.0%
60
$1
60.0%
50
Ratios of G1
40
63.5%
77. 77.4%
69.69
54.6%
30
48.6%
61. .6%
52.8%
60. 60.8%
20
20
10
10
10
Guidance weight
Guidance weight
(a3) RadNets-1 on Caltech15
(a4) BadNets-2 on Caltech15
Figure A7: Trigger amplification illustration by comparing the trigger- present images in the generation with the ones training. Different poisoning ratios (1, 2, 5, 10) are evaluated under different triggers (BadNets-1 and BadNets-2) on ImageNette and Caltech15. Fach bar consists of the ratio of trigger-present generated images within G1 and G2. Fach black dashed line denotes the ratio of trigger-present training data related to target prompt. Evaluation settings follow Fig.
19

---

D.2 The Effect of Sampler
As Fig.Ar A8 shows, we find the poisoning threat also exists in the SDE sampling. However, we observed that the poisoned DM generates less taigee-tainted images (G1) using SDE sampling [38]. We attribute this observation the increased randomness introduced by SDE sampling [39].
consequently hindering the replication of trigger patterns.
prasent prasett writing data malates target procegt
Paen Tarto trigger proneze gensratedd Imagus
Rade trigger presnt genarated Imagus wints
G1: 68.80%
Ratios of C1 (%)
80
18.6%
60
30.8%
21.0% 2.2%
40
0.8%
mismatch
20
1.9%
19.1%
47.6%
G2: w/'T & match
69
mismatch
G4: w/o T match
Poisoning ratio (%)
(al) Composition with 10%
(bl) GI
(el) G2
(di) G4
(cl) Composition poisoning ratio
(1) Generation using the DDPM sampling
G1: 52.70%
XXXXX00000 5000%
60
10.8%
Ratios of G1 (%)
50
3393%
40
36.2% 8.4%
10.80% G3: 0.00%
30
0.5%
11.8% 7. ..5%%
116.7% 6.67%
G4: 36.50%
20
9..1%
52.7%
mismetch C2: w/ match
10
C3: w/o mismelch
G4: w/o match
Poisoning ratio (%)
(a2) Composition with p 10%
(b2)
(c2) (d2)
(c2) Composition v.s. poisoning ratio
(2) Generation using the SDE sampling.
Figure A8: Dissection and generation composition of 1K generated images using BadNets-like data trained classifier-free diffusion model on CIFARIO, using DDPM sampling and SDE sampling. (a) Generated images' composition using poisoned DM, where GI represents generations containing the trigger (T) and mismatching the input condition, G2 denotes generations matching the input condition but containing the trigger, G3 refers to generations that do not contain the trigger but mismatch the input condition, and G4 represents generations that do not contain the trigger and match the input condition. Assigning generated image to specific group determined by externally trained ResNet-50 classifiers. Visualizations of GI, G2 and G4 are provided in (b). (c), and (d), respectively. Sub-figures (el,e2) show the generation composition against poisoning ratios P E (1%, 2%, 5%, 10%}. Each bar represents poisoned DM. The poisoned DM generates a notable quantity of adversarial images (Gl and G2) using both DDPM and SDE samplers, However, SDE sampling generates fewer trigger- trigger-tainted images, with decrease by 17.1% of GI type generations.
and G2 compositions within IK images generated by
20

---

E Comparison with BadT22
Tab. A2 presents the comparison of our method and the BadT2I [15]. To get clearer view of the generation composition, we set the backdoor target to generate target object (cat) and target patch (mark) the same time. This allows us to calculate the G1 and G2 ratio in the generated images. Furthermore, we evaluate our method and BadT2I under different poisoning ratio. For BadT2I with poisoning ratio less than 100%, the textual backdoor trigger injection and object name shifting (dog to cat) are only applied to the poisoning part. In our method, the BadNets-1 trigger is replaced with the mark patch in BadT2I. To align with our previous settings, we replace the caption of cat dog images with "A photo of cat dog". Considering BadT2I adds the textual trigger and changes the training objective, not only shows a stronger trigger amplification but also a lower FID.
Table A2: The G1 ratio, G2 ratio and FID the generated images using diffusion model poisoned by BadNets-lid like poisoning and BadT21 [15]. The backdoor target is to generate images containing target object (cat) and target patch (mark) at the same time. The original training data is the 500 text-image pairs released by BadT2I, with cat and dog images accounting for half cach. In BadT2I, the À is set to 0.5 and the number of training steps is
8K, which is consistent with the object-backdoor setting of BadT2I. For the case where the poisoning ratio less than 100%, the textual backdoor trigger injection and object name shifting (dog to cat) the BadNets-1 trigger is replaced with the mark patch in BI Moreover, the caption of cat dog images is replaced with "A photo of cat / dog".
are only applied to the poisoning part. In our method,
Poisoning Methud
Bad
Ours
Poisoning Ratio
10%
50%
10%
50%
GI Ratio
11.6%
58.4%
8.4%
53.2%
G2 Ratio
26.8%
29.2%
16.4%
22.8%
13.2
13.1
14.9
15.2
21

---

Data Poisoning Signature Enhancement
We provide distributions of detection metrics further elaborate our detection insight. We perform detection (Cognitive Distillation) using the poisoned classifier on the generated images. For Cognitive Distillation, we adopt the L norm of this mask as the detection metric. If the detection metric lower than certain threshold, it suggests the input sample is poisoned. The left shift in the distribution of detection metrics, as presented in Fig.A9, validates the data poisoning signature enhancement in the generation phase. Furthermore, the distribution of poisoned images and clean images in the generation set can be more separated, which echoes our finding that poisoned DM's generation helps data poisoning detection.
Mask norm on training sel
Mask norm on generation
poison clean
0.070
poison
clean
Ratio samples
Ratio samples
0.015
6.003
0.023
0.002
0.032
0.001
0.023
0,000
60o
1400
0.000
L.1 norm
mask
norm mask
(a) BadNets-1 trigger
Mask norm training set
Mask norm generation
0.010
poison
poison
clean
clean
0.028
Ratio of Saaaples
0.003
of stmples
0.004
0
0,002
0,001
300
L1 norm of the mask
L1 norm of the mask
(b) BadNels-2 trigger
Figure A9: Data poisoning detection metric distributions on training set and generation set. We use Cognitive Distillation (CD) [31] as the detection method. A small mask norm indicates the data point might be poisoned. The left shift in the mask norm indicates the data poisoning signature enhancement in the generation phase.
22

---

G Defense performance on Other Classifiers
Tab. A3 shows the testing accuracy (TA) and attack success rate (ASR) for image classifier VGG-16 and DenseNet-12 121 trained on both the originally poisoned training set and the DM-generated dataset. The poisoned DM efficiently transform malicious data into benign, leading to much lower ASR of VGG-16 and DenseNet-122 121 trained on the DM-generated dataset.
Table A3: Testing accuracy (TA) and attack success rate (ASR) for image classifier VGGI6 and DenseNet12 21 trained the originally poisoned training set and the poisoned DM- MM-eenerated set. The training set is ImageNette. The number of generated images is the generation set compared to training set highlighted in blue.
the size of the training The ASR reduction using the
Metric
polsocing rublo
Trigger
1%
BadNets- 2%
5%
BaiNNete-2 246
5%
1%
WaNNi 2%
VGG16
TA,%)
98.485 93.783
98,445 93.146
93.070
98,573
98.345
93.222
58,033
92.891
90.904
98,140
90.318
90.344
98.318
92,407
98.
93.209
99.685
55.458
99.632
72.454
26,866
1,725
([dccessee
(197.9907)
De3
TA
generation
training
94.343
94.573
95.006
training
generation
0.452
(1dccecsse)
(197.938)
(J68.143)
(1290016)
(L57.466)
(150.537)
(127.941)
(198,219))
(198
530)
23

---

A
Simel A, B3) 0.544.
Simm A, B3) 0.308.
Sinn (A, B) = 0.532.
Sim A, B) 0.306.
Sim(A, F3) m 0.488.
Sim(A, B3) m 0.424.
Poison Images
No Poison Images
Figure A10: Visualizations of the (A,B) image pair using poisoned SD or clean SD, The generated image (A)
resembles its replicated training image (B) more closely when poisoned. The setting follows Fig. 5,
H
Additional Analysis of the Impact of Poisoning on Data Replication
To gain clearer understanding of the impact of poisoning data replication in the context of diffusion models (DMs), we train DM using the same images, once poisoned and once not poisoned. Fig. A10 illustrates the similarity scores between generated image (*A' and its corresponding replicated image ('B'). We observe a significant increase in the data replication score when the replicated images in the training set poisoned, compared to the "No Poison" setting. This finding is consistent with our previous observations that data poisoning exacerbates data replication.
I Compute Resourses
All our experiments were conducted on server equipped with 8 NVIDIA A6000 48GB GPUs. The server features an AMD EPYC 7713 64-Core Processor with ITB of RAM. We used 4 A6000 GPUs train DDPM from scratch on CIFAR-10 and A6000 GPUs fine-tune the SD on ImageNette and 15 Each training session takes approximately 24 hours to complete, while inference can be done within 3 hours using just one A6000 GPU.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The 3 main claims in introduction: "Trojan Horses' effects, the concept of "Castle Walls' and the connection between data poisoning and data replications are clearly reflected in Sec. 4, Sec. 5 and Sec. respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limiations in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided a detailed description of our attack method in the Attack Details part of Sec. We have also stated the details of dataset construction, hyper- parameters settings and optimizer in Appendix. A. We also open source code and data for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have submitted the code and data following the submission guidelines.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.o cc/
/elliiicguies////ee (CodeSummissionn olicyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // //nips.cc httppppullpplpeeeeeepepppppppeee httpa////giidee//ddeuubbiiseees (oodeeubmissiooooolll..ooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed description including dataset, backdoor target, triggers and generation prompts in the Attack Details part of Sec. 4. We have also stated the details of dataset construction, hyper-parameters settings and optimizer in Appendix. A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report error bars in Fig. 3 and Tab. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources we used are stated Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///rrrppp..ppppliiccccc///t i ines?
Answer: [Yes]
Justification: Our research complies with the NeurolPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We state the social impact of our research in Appendix. 9.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We require users to adhere to the benign usage of our poisoning method in our official code repository.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: We have cited the original paper that produced the dataset and detailed how we used such datasets in our experimental setup.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "fIa0DlLeZW", "reviewer": "Reviewer_NHzT", "summary": "The paper studies BadNet-like poisoning attacks in diffusion models from both attack and defense perspectives.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I think the paper makes interesting observations for the community, especially regarding the phenomenon of trigger amplification.\n\n2. The evaluation seems quite comprehensive, considering multiple datasets, models, attacks, and detection methods.", "weaknesses": "1. Even though the authors consider many settings, the experiments are run only once (no error bars are shown).\n\n2. While in Table 4, the attacks' success rates are reduced when the poison percentage is up to 5%, I am wondering if they are amplified for higher poison percentages. If so, how could the defender use this as a defense in practice if they do not have any knowledge about the poison percentage?\n\n3. The paper is fully empirical.\n\nMinor comment: at line 252 \"comapred\" should be \"compared\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the interesting observations and comprehensive experiments by our study. We hope our response (**A**) to each of the weaknesses (**W**) can address your concerns. \n\n**W1**: Despite considering many settings, experiments are conducted only once (no error bars). \n\n**A1**: Please refer to **GR1** in general response.\n\n**W2**: In Table 4, attacks' success rates decrease with a poison percentage up to 5%, but what happens at higher percentages? How can defenders use this if they don't know the poison percentage?\n\n**A2**: Thank you for your question. We want to make the following clarifications.\n\nFirst, Table 4 supports our finding that poisoned DMs with “low” poisoning ratios (≤ 5% in the paper) convert malicious data into benign data (Lines 282-284). This is underpinned by our analysis of phase transitions in poisoned DMs relative to poisoning ratios (Lines 232-234), identifying a 5% poisoning ratio as a critical transition point in Fig. 4. It is also worth noting that even a 1% poisoning is typically enough to attack DMs in practice [C1, C2], making 5% a relatively high threshold.\n\nSecond, when the poisoning ratio increases to 10%, attack success indeed amplifies (See the 10% poisoning ratio in **Table R1** of the attached PDF). Based on the phase transition finding (Lines 232-234), this leads to an increase in trigger-present, prompt-mismatching generations (G1). In this case, the surge in G1 generations could aid the application of existing data poisoning detection methods on DM generations (Line 264). As a result, even without precise knowledge of the poisoning ratio, defenders can initially implement detection methods (Table 3) and subsequently employ the training-based defenses outlined in Table 4.\n\nLastly, we emphasize that our main goal is to develop defensive strategies based on attack findings on poisoned DMs. We acknowledge that these insights may not yet perfectly align with optimal practical scenarios. Yet, they also offer the potential. As mentioned earlier, it can simultaneously enhance poison detection and robust classifier training. Utilizing DM generation in this way could provide a unified defense mechanism.\n\n[C1] Y. Wu, X. Han, H. Qiu and T. Zhang, \"Computation and Data Efficient Backdoor Attacks,\" ICCV 2023.\n\n[C2] Xia, Pengfei, Ziqiang Li, Wei Zhang, and Bin Li. \"Data-efficient backdoor attacks.\" IJCAI 2022.\n\n**W3**: The paper is fully empirical.\n\n**A3**: While we recognize the value of theoretical research, we contend that our empirical approach does not prevent its research depth. This study is the first to explore the bilateral impacts of BadNets-like backdoor poisoning on diffusion models, including both attack insights (trigger amplification, phase transitions, and data replication) and defense implications (poison data detection, classifier training, and diffusion classifier). We would also like to kindly remark that empirical studies are common in adversarial learning and are crucial for uncovering new insights that theoretical models might not yet capture.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Bqtt0Sj3A", "reviewer": "Reviewer_A3yL", "summary": "This paper investigates backdoor attacks against diffusion models. Unlike previous works that require both injecting poisoned data samples and manipulating the training loss function, this study focuses solely on poisoning training data samples during the training phase. The research demonstrates that backdoor attacks not only compromise the functionality of diffusion models (resulting in incorrect images misaligned with the intended text conditions) but also amplify the presence of triggers, a phenomenon termed 'trigger amplification.' This trigger amplification can be utilized to enhance the detection of poisoned training data, thereby providing a defensive advantage.", "questions": "1. It would be better if the authors can evaluate the proposed method on more datasets such as ImageNet1K and CIFAR-100\n2. It is suggested that the attack model be described in a separate section.", "limitations": "Yes, the authors addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- This paper is easy to follow.\n\n-- This paper demonstrates that simply poisoning the training dataset can effectively backdoor diffusion models.\n\n-- Conduct comprehensive experiments. Impressive results especially in attack success rate.\n\n-- Discuss the limitation of the proposed attack and future work.", "weaknesses": "-- The evaluation of the proposed attacks is limited to 3 datasets: CIFAR10, ImageNette and Caltech15.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the readability, the comprehensive experiments, and the contribution by our study. We hope our responses (**A**) to each of the weaknesses (**W**) or questions (**Q**) can address your concerns.\n\n**W1 & Q1**: The evaluation of the proposed attacks is limited to 3 datasets: CIFAR10, ImageNette and Caltech15. It would be better if the authors can evaluate the proposed method on more datasets such as ImageNet1K and CIFAR-100.\n\n**A1**: Thank you for suggesting the inclusion of additional datasets like ImageNet1K and CIFAR-100. Since our current evaluations on CIFAR-10 and ImageNette share similar formats with CIFAR-100 and ImageNet1K, we decided to expand our experiments to include a subset of the more complex dataset LAION [C1], enhancing the applicability of our findings. Please refer to **GR2** in general response about the additional experiments on LAION.\n\n[C1] Schuhmann, Christoph, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes et al. \"Laion-5b: An open large-scale dataset for training next generation image-text models.\" NeurIPS 2022.\n\n**Q2**: The attack model could be described in a separate section.\n\n**A2**: Thank you for your suggestion. We will put the attack model in a separate section to improve the clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V9y0RexaHM", "reviewer": "Reviewer_jBz2", "summary": "The paper investigates the impact of BadNets-like data poisoning attacks on state-of-the-art diffusion models (DMs) used for image generation. Unlike previous studies that required modifications to the diffusion training and sampling procedures, this work examines the effects of poisoning the training dataset alone. The study uncovers dual effects of data poisoning, which not only degrade the generative performance of DMs but also provide defensive advantages for image classification tasks. Key findings include the misalignment between input prompts and generated images, the amplification of trigger generations, and the linkage between data poisoning and data replications.\n\nThe major contributions of this paper are as follows. It demonstrates that diffusion models (DMs) are vulnerable to BadNets-like data poisoning attacks, leading to two significant adverse effects: (1) misalignment between input prompts and generated images, and (2) an increased generation of images with embedded triggers, referred to as 'trigger amplification'. The study identifies a phase transition in the poisoning effect relative to the poisoning ratio, revealing the nuanced dynamics of data poisoning in DMs. The proposed 'Castle Walls' concept introduces defensive strategies for image classification, including leveraging trigger amplification for detecting poisoned training data, training classifiers with images from poisoned DMs before the phase transition to mitigate poisoning, and using DMs as image classifiers to enhance robustness against attacks. Additionally, the paper establishes a connection between data poisoning and data replication in DMs, showing that introducing triggers into replicated training data exacerbates both the replication problem and the impact of poisoning, thus highlighting the inherent data memorization tendencies of DMs.", "questions": "In the figure captions, there is mention of G3 and G4 (that do not contain trigger), but these are not referred to in Figure 2 itself (only G1 and G2 are). Highlight in the text why these are missing and now shown? \n\n Checklist - Q7 Justification: Error bars are not reported because it would be too computationally expensive. How can we have confidence that the experimental results are representative and robust and not prone to statistical chance. Provide some supporting evidence.\n\nWhen non-monotic results are observed (for example Bad-Nets 2 on ImageNette, SD, Caltech15), explain why increasing the poisoning rate from 1 to 5% provides an AUROC improvement but an increase from 5% to 10%. \n\nLine 217, Page 6, Use the same notation as in the paper. “Fig A3 presents” -> A3 of which figure? Provide full reference.", "limitations": "The authors have addressed key aspects of their work, but several limitations require further attention to strengthen the paper.\n\nExperimental Robustness: The lack of error bars due to computational expense raises concerns about the robustness of the results. Without statistical validation, it is difficult to ensure the findings are consistent. Constructive suggestion: Include confidence intervals or statistical validation for a subset of experiments to enhance result reliability.\n\nPractical Implementation of Defensive Strategies: The 'Castle Walls' concept introduces novel defenses, but practical implementation details are lacking. Constructive suggestion: Provide detailed guidelines and examples for implementing these defensive strategies in real-world scenarios.\n\nBroader Societal Impact: The potential negative societal impacts of data poisoning are not thoroughly discussed in the main paper. Constructive suggestion: Discuss the broader societal implications and ethical considerations of your findings, including potential misuse and guidelines to mitigate negative impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper presents an innovative investigation into the impact of BadNets-like data poisoning attacks on state-of-the-art diffusion models (DMs) used for image generation. Unlike previous studies that require modifications to the diffusion training and sampling procedures, this work uniquely focuses on the effects of poisoning the training dataset alone. This fresh perspective uncovers dual effects of data poisoning, revealing both degradation in generative performance and potential defensive advantages for image classification tasks. The introduction of the 'Castle Walls' concept for defensive strategies is original, offering new ways to leverage data poisoning effects to enhance robustness against attacks.\n\nQuality: The quality of the research is reflected in its comprehensive experimental analysis and the depth of its findings. The study methodically demonstrates the vulnerability of DMs to BadNets-like attacks, detailing how these attacks cause misalignment between input prompts and generated images and amplify trigger generations. The paper includes a thorough examination of defensive strategies, including the innovative use of poisoned DMs for training classifiers. \n\nClarity: The paper is well-structured and clearly communicates its methodology, findings, and implications. The key concepts and contributions are articulated in an accessible manner, with detailed explanations of the experimental setup and results. While there are minor editorial issues, such as the need for clarification in figure captions and consistent notation, these do not significantly detract from the overall clarity of the paper. The inclusion of detailed figures and tables aids in the clear presentation of the data and results.\n\nSignificance: The significance of this work lies in its potential to substantially enhance the understanding and robustness of DMs in the face of data poisoning attacks. By uncovering the dual effects of data poisoning and proposing innovative defensive strategies, the paper provides valuable insights that can inform future research and practical applications. The connection established between data poisoning and data replication highlights the inherent data memorization tendencies of DMs, offering a deeper understanding of their vulnerabilities.", "weaknesses": "Additional statistical analysis (e.g., confidence intervals) could strengthen the findings by accounting for variability and ensuring the observed improvements are statistically significant.\n\nExperimental Robustness: The lack of reported error bars due to computational expense raises concerns about the robustness and representativeness of the experimental results. Without statistical measures of variability, it is challenging to assess the reliability of the findings. Constructive suggestion: Provide some supporting evidence or alternative measures to demonstrate the robustness of the results, such as reporting confidence intervals for a subset of the experiments.\n\nComprehensive Defensive Strategies: While the 'Castle Walls' concept is innovative, the practical implementation details of these defensive strategies are not fully explored. Constructive suggestion: Provide more detailed guidelines and examples on how these strategies can be implemented in real-world scenarios to enhance their practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough summary, as well as the recognition of the originality, quality, clarity, and significance by our work. We hope our responses (**A**) to each of the weaknesses (**W**) or questions (**Q**) can address your initial concerns.\n\n**W1 & W2 & Q2**: How can we ensure the experimental results are robust and not prone to statistical chance without error bars, given the computational expense? Can you provide supporting evidence, such as confidence intervals for a subset of experiments, to demonstrate the robustness and reliability of the findings?\n\n**A**: Please refer to **GR1** in general response.\n\n**W3 & Q3**: When non-monotonic results are observed, such as with Bad-Nets 2 on ImageNette, SD, and Caltech15, why does increasing the poisoning rate from 1% to 5% improve AUROC, but further increase from 5% to 10% does not? Please provide explanations for these trends to clarify the underlying factors contributing to the observed results.\n\n**A**: Thank you for highlighting this point. It seems there might be some confusion regarding the expected monotonicity of AUROC results as detailed in Table 3.\n\nFirst, it's important to note that the detection performance, measured by AUROC, does not necessarily increase linearly as the poisoning rate rises from 1% to 5% and further to 10% even under the conventionally poisoned training set. The rationale behind this is that (a) AUROC primarily measures the ability to detect the existence of  data poisoning, and (b) the shift from a 5% to a 10% poisoning ratio does not sufficiently enhance detectability because both have been considered relatively high poisoning ratios. \n\nSecond, the main purpose of Table 3 is to show that detection performance, as measured by AUROC, consistently improves when using the set generated by poisoned DMs compared to the originally poisoned training set, across various poisoning ratios. This is used to demonstrate  the effectiveness of leveraging trigger amplification in poisoned DMs to enhance poison detection over  the  generated set.\n  \n**W4**: While the 'Castle Walls' concept is innovative, the practical implementation details of these defensive strategies are not fully explored. Constructive suggestion: Provide more detailed guidelines and examples on how these strategies can be implemented in real-world scenarios to enhance their practical applicability.\n\n**A**: Please refer to **GR3** in the general response on the applicability of our defense. \n\nRegarding implementation details, we have provided additional information in the appendix for further clarification.\n\n1. Poisoning in the generation set can be more easily  detected than that over the original training set (Table 3): We include the implementation details in Line 507-509 of Appendix A.4 in the original submission.\n2. Training a less malicious classifier using  generated data (Table 4): We include the implementation details in Line 499-506 of Appendix A.4 in the original submission.\n\n**Q1**: In the figure captions, there is mention of G3 and G4 (that do not contain trigger), but these are not referred to in Figure 2 itself (only G1 and G2 are). Highlight in the text why these are missing and now shown?\n\n**A**: We prioritized G1 and G2 over G3 and G4 in the main paper due to their more representative attack implications and page constraints. However, to address this concern, we have included visualizations for G4 in **Fig. R1 and Fig. R2** in the attached PDF provided with our general response. Regarding G3, which represents prompt-mismatched generations without triggers, these instances are almost non-existing in the sampled generations, thus were not emphasized.\n\n**Q4**: Line 217, Page 6, Use the same notation as in the paper. “Fig A3 presents” -> A3 of which figure? Provide full reference.\n\n**A**: Apologies for any confusion. Fig A3 refers to Figure A3 in the Appendix, not a sub-figure. All figures in the Appendix are indexed starting with an \"A\" to differentiate them from those in the main text.\n\n**Limitations**: Experimental Robustness, Practical Implementation, and Broader Societal Impact.\n\n**A**: Thank you for your constructive suggestions on enhancing the Limitations and Broader Impact sections in Appendices L and M. We will address these in the revised version and refer to the response to (**W1 & W2 & Q2**)  for experimental robustness and response to (**W4**) for practical implementation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NFdJX2bPFi", "reviewer": "Reviewer_HqnS", "summary": "The paper proposes a new poisoning attack for diffusion models (DMs). While previous work tried to poison/backdoor DMs by altering the training process or the optimization objective, the paper proposes a poisoning attack by only altering the training data. \nTo poison DMs, a trigger is inserted into training images, and the labels of the poisoned samples are changed to the target class. The resulting DM, trained on this poisoned dataset, generates images not aligned with the given prompt or images containing the trigger pattern used for poisoning.\nBased on this behavior, insights are presented that might help protect DMs against poisoning attacks, and a different view on data replication in DMs is given.", "questions": "**Q1:** How many samples were used to train the DMs on the ImageNette and the CIFAR-10 dataset?  \n**Q2:** What are the experimental results for CIFAR-10?  \n**Q3:** Why choose the black and white square as the first trigger? Why not just use a uni-colored square as in the original BadNets paper?  \n**Q4:** I can imagine that the appearance of the trigger also plays a significant role in whether the poisoning is successful or not. You have chosen the black and white square. Does the same phenomenon also appear with other patterns?  \n**Q5:** How many samples were used to calculate the FID scores in Table 2?  \n**Q6:** What is the reasoning/intuition behind the phenomenon that the DMs seem to generate an image of the target class containing the trigger, even though the target class images in the training set didn't have the triggers?", "limitations": "Some of the work's limitations are addressed. However, I think it is important to discuss how realistic it is that the proposed poisoning method is used in such a realistic scenario and not only to investigate the behavior of DMs on poisoned data in an artificial setting. Additionally, the limitations of the proposed defense method should be discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper tackles a very important topic as the risk of poisoned data is increasing when training DMs on publicly available data scraped from the web\n- The insight that DMs generate images of the target class with the trigger, even though the trigger has not been present in the target class training images, is very intriguing. However, the paper doesn't really give an intuition or explanation on why this is the case (see questions).", "weaknesses": "- Only training details about the Caltech15 dataset are provided in the appendix. (see questions)\n- It is unclear how this proposed method can be applied to datasets like LAION or other uncurated/unstructured datasets without clearly separated classes.\n- In the experimental setting, it is stated that experiments on CIFAR-10 are conducted. However, in the experimental evaluation, there are results for CIFAR-10. Only ImageNette and Caltech15 are used to show the effectiveness of the poisoning attack. (see questions)\n- The paper is sometimes hard to read, and in parts, it is difficult to grasp what the authors want to convey as the take-away message of the paper is not really clear, in my opinion.\n- Using the \"poisoned DM\" as a defense against poisoning attacks is not very realistic or applicable, in my opinion. In reality, a DM would first have to be trained to generate data and apply the poisoning detection method to the generated data before even starting to train the classifier. In addition, the improvement of the AUROC for the poisoning detection methods is only very minor (in most cases, less than 1 percentage point improvement of the AUROC value).\n- The data replication experiments are not really meaningful, in my opinion. If we look at replicated images, it is expected that these images are replicated more than randomly chosen images. The experiment would be more meaningful if the same images would be once poisoned and once not poisoned. This would give insight into whether the poisoning really affects the data replication abilities of the DM.\n- there are two other works [1, 2] that use DMs for defending against poisoning attacks that should be mentioned in the related work part \n\n[1] Zhou et al., DataElixir: Purifying Poisoned Dataset to Mitigate Backdoor Attacks via Diffusion Models, AAAI 2024\n[2] Struppek et al., Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data, NeurIPS 2023 Workshop BUGS\n\nMisc:\n- Many of the cited papers are arXiv papers and not the conference versions (VillanDiffusion is NeurIPS, \"Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning\" is \"conference on multimedia\", Rickrolling the artist is NeurIPS, etc.). Please cite the proper conference versions of the papers.\n- The titles in the references only include lower characters. This seems to be a bibtex/latex problem.\n- Reading \"the work [...]\" is not really smooth. Instead, it would be better to write the author's names as \"Chou et al. have done ....\"\n- The links in the appendix should be blue to indicate that they are clickable. I almost missed them. Also, it might be preferable to show the URL so the reader knows which site is linked without clicking on the link in the first place.\n\nThe paper tackles a very interesting problem, and the discovered phenomena seem to be very surprising. However, in my opinion, the paper is not quite ready for publication because of the unclear take-away message and the sometimes hard-to-read text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review on our submission. We hope that our response (**A**) to each weakness (**W**) and question (**Q**) address your concerns and positively affect the rating.\n\n\n**W1 & Q1**: Training details of ImageNette & CIFAR-10. \n\n**A**: The training details of ImageNette & CIFAR-10 are in Appendix A.3, as cited in Line 180. ImageNette and CIFAR-10 follow the original setup. ImageNette has 10,000 training samples with 1,000 per class [C1]. CIFAR-10 has 50,000 training samples with 5,000 per class [C2].\n\n\n[C1] fast.ai, https://github.com/fastai/imagenette , 2021.\n\n[C2] Alex Krizhevsky, “Learning Multiple Layers …”, 2009.\n\n**W2**: Experiments on LAION, without clear classes.\n\n**A**: Like comparisons with BadT2I (Lines 260-261 & Appendix E), our method can be applied to image-caption pair data. Please refer to **GR2** in the general response on LAION [C3]. \n\n[C3] Schuhmann, et al. \"Laion-5b: An open large-scale dataset ...\", NeurIPS 2022.\n\n**W3 & Q2**: There are no results for CIFAR-10.\n\n**A**: The attack results for CIFAR-10 are presented in Figure A5 of Appendix F. Our detection approach for CIFAR-10 is detailed in Table 3, and our diffusion classifier strategy is included in Table A4 (as cited in Line 316).\n\n**W4 & W8**: The paper is sometimes hard to read, and it is difficult to grasp what the authors want to convey as the take-away message is not clear.\n\n**A**: We regret that the paper was found difficult to read and that the take-away messages were not clear. We have made substantial efforts to structure the paper for clarity and ease of understanding. The primary research question concerning the dual effects (i.e., 'Trojan Horses' and 'Castle Walls') of BadNets-type attacks on DMs is explicitly outlined in the introduction (Lines 33-51). The paper systematically presents the key findings: Section 4 explores the attack feasibility and insights such as trigger amplification and phase transitions. Section 5 builds on these insights with defense strategies including poison data detection and classifier training. Our paper's readability was also praised by other reviewers. For example, Reviewer A3yL noted, “This paper is easy to follow,” and Reviewer jBz2 noted, “The paper is well-structured and clearly communicates its methodology, findings, and implications.”\n\nWe hope the reviewer can reconsider the criticisms on our paper readability. \n\n**W5 & Limit**: Using \"poisoned DM\" as a defense is not very realistic. AUROC improvement for poisoning detection is minor (in most cases, less than 1% improvement).\n\n**A**: Please refer to **GR3** in the general response about the applicability of our defense strategy.\n\nAdditionally, Table 3 indicates a more substantial improvement in AUROC for poisoning detection than noted. The average increase is 5.9%, with 46 out of 54 cases showing improvements exceeding 1% (See **Table R3** of the attached PDF). \n\n**W6**: The data replication experiments are not meaningful. The experiment would be more meaningful if the same images would be once poisoned and once not poisoned. \n\n**A**: Thank you for your feedback. First, the original experiments are designed to investigate two aspects: (a) whether poisoning replicated data exacerbates data replication in generated images, and (b) whether the poisoning effect can in turn become worse, as indicated in Line 338-341, showing that poisoning duplicate images significantly increases the generation of trigger-tainted images. \n\nSecond, comparing the effects on the same images, once poisoned and once not, is insightful. We add this study in **Figure R3** in the attached PDF. Fig. R3 shows the similarity scores between a generated image ( 'A') and its corresponding replicated image ('B'). There is a significant increase in data replication score when the replicated images in the training set are poisoned compared to the “No Poison” setting by a clean SD. \n\n**W7 & Misc**: There are two other works [1, 2] that use DMs for defending against poisoning attacks. Reference and link issue.\n\n**A**: We will follow the suggestions.\n\n**Q3 & Q4**: Trigger pattern factor?\n\n**A**: Prior research [C4] demonstrates that BadNets-type attacks are robust to trigger patterns, provided only if there is a \"shortcut\" established linking the trigger to a targeted label. We chose a black and white square because of its distinctiveness, enhancing visibility vs. backgrounds. The original submission has included a more intricate \"**Hello Kitty**\" pattern, as detailed in Fig. 2 and Appendix A.2, where we observed similar phenomena. \n\nFollowing the suggestion, we explored a **uni-colored** trigger and included \"**bomb**\" trigger in **Figure R1** in the attached PDF. Our observations are consistent across various triggers.\n\n[C4] B. Wang et al., \"Neural Cleanse: Identifying and Mitigating …\" 2019 SP.\n\n**Q5**: How many samples to calculate the FID?\n\n**A**: We follow the standard practice of generating samples equal to the training set size and then calculating the FID. As **W1 & Q1**, CIFAR-10: 50,000 generated samples, with 5,000 per class. ImageNette: 10,000 generated samples, with 1,000 per class. Caltech-15: 3,000 generated samples, with 200 per class.\n\n**Q6**: The intuition for DM generation of the target class images containing the trigger.\n\n**A**: This is an insightful question.\n\nBased on the DM memorization findings in [C5], we believe that DMs memorize both the association of the target class vs. the target-class images (benign DM generation) and the association of the target class vs. the trigger (abnormal DM generation). The latter effect can be attributed to local replication. These two associations result in a combined generation, i.e., target-class images with the trigger (G2).\n\n[C5] Somepalli, et al. \"Diffusion art or digital forgery? ...\", CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We sincerely thank all the reviewers for their meticulous review and valuable feedback on our submission. Below, we provide a general response to address common questions, weaknesses, and concerns in your comments. Please refer to the figures and tables in the attached PDF as Figure Rx and Table Rx, respectively, where 'R' denotes 'rebuttal'.

**GR1: Error bars to show the significance of the experimental results. (Reviewers NHzT & jBz2)**


To address the requests for demonstrating the statistical significance of our experimental results, we conducted additional runs of our main experiments focused on trigger amplification in the attack phase and classifier training in the defense phase. We have provided the means and standard deviations from these experiments, conducted using 5 different random seeds, in **Table R1** and **Table R2** of the attached PDF. These additional runs reinforce the consistency and reliability of our original findings. The results, specifically the decrease in attack success rate (ASR) shown in Table R1 and the identified trigger amplification in generated images of poisoned DMs detailed in Table R2, are statistically significant, as indicated by the comparison with standard deviations of the results.

**GR2: Experiments on more complex datasets like LAION without clearly separated classes.  (Reviewers A3yL & HqnS)**

During the rebuttal phase, we expanded our study to include a subset of the LAION dataset, which consists of 500 image-caption pairs. Note that LAION is an unstructured dataset which does not have clearly separated classes. Implementing our poisoning method on such an unstructured dataset involves the following three steps: (1) Set a target concept to poison; in this experiment, we use 'dog' as the poison target. (2) Randomly sample some image-caption pairs from those whose captions do not contain words that represent the meaning of dog (such as 'dog', 'puppy', 'canine'). (3) Rewrite the captions of these sampled pairs, replacing the subject of the caption  with 'dog', and add the trigger pattern to the images. The results of this experiment are presented in **Figure R2** of the attached PDF. We observed consistent effects of our poisoning attack, including trigger amplification in both G1 and G2 groups, demonstrating similar outcomes to our original experiments. 

**GR3: Practicability of defense methods in the real world. (Reviewers jBz2 & HqnS)**

We appreciate your comments regarding the practicality of using "poisoned DMs" as a defense. Here are our clarifications:

First, we would like to clarify the origins and importance of defensive insights.  Our primary objective in Section 5 was to derive defensive strategies from attack insights (trigger amplification and phase transition) gained in Section 4. To be specific, the phenomenon of trigger amplification led us to enhance poison data detection during the post-DM generation phase, compared to the original training set (Line 264). Additionally, observing phase transitions that poisoned DMs with low poisoning ratios can transform malicious data into benign has guided our efforts to improve robust classifier training against data poisoning (Line 282).  Thus, having trained poisoned DMs, unveiling defense insights from generations of poisoned DMs represents a valuable and natural extension of our study.

Second, we agree that implementing the proposed defensive strategies in practical settings requires access to a trained (poisoned) DM, which might introduce additional computational overhead. However, this approach also offers potential. As detailed earlier, it can simultaneously enhance poison detection and robust classifier training. In practice, defenders can first use detection methods to assess the presence of a poisoning attack and then apply training-based defenses to mitigate its impact on classifier training. Utilizing DM in this way could provide a unified foundation of defense mechanisms.

**GR4: A summary of additional experiments (@All reviewers).**

We have made a substantial effort to enrich our experiments based on reviewers’ suggestions (see the attached PDF). Below is a summary, where Q-i (or W-i) represents the $i$-th question (or weakness) in our individual responses:

**Reviewer NHzT**

W1: Experiments’ error bars by multiple runs (**Table R1 and Table R2**); 

W2: 10% poisoning ratio scenario for poisoning defense method (**Table R1**);

**Reviewer A3yL**

W1 & Q1: Experiments on other dataset, e.g., LAION (**Figure R2**);

**Reviewer jBz2**

W1 & W2 & Q2: Experiments’ error bars by multiple runs (**Table R1 and Table R2**);

Q1: Visualizations of G4 (**Figure R1 and Figure R2**);

**Reviewer HqnS**

W2: Experiments on other dataset, e.g., LAION (**Figure R2**);

W6: Experiments on the data replication, comparing images once poisoned vs. once not poisoned (**Figure R3**);

Q3 & Q4: Experiment on more trigger patterns (**Figure R1**).

Meta-review: This paper studies data poisoning attacks on diffusion models. One of the main motivations is to attack diffusion models with only access to the training dataset, as opposed to also the objectives and sampling procedures. The authors show several findings regarding model effects they could influence using their attack method. One is that the generated images could either be misaligned with the input prompt or the images could match the prompt while containing certain correlated content (called triggers in this work). Another outcome of this work is to defend against poisoning attacks, including detecting poisoned training data or enhancing robustness against poisoning attacks. Overall, the paper has both high level messages (regarding the ease of poisoning diffusion models) as well as technical contributions (empirically showing how to attack/defend in the space of image generation).

During the rebuttal phase, the authors addressed many of the key issues that the reviewers raised. For example, the authors provide a small experiment showing that their method can work on data without class labels, using captions as a proxy instead. The authors also provide error bars and other investigations into the reproducibility of their work. Moreover, after the rebuttal, many of the reviewers have raised their scores. After the author-reviewer discussion, the four reviewers are unanimous in the support and merits of the submitted work, without any major criticisms. In total, all four reviewers now vote for acceptance of the paper, which is a rare outcome in high level ML conferences.

Given the soundness and thoroughness of the presented work, I vote for acceptance.

I encourage the authors to fully revise the manuscript for the final version of the paper, fixing typos, adding references, and including the additional experiments (e.g., the LAION experiments and the error bars). In my opinion, extending the attack to text-image data (w/o class labels) is natural and interesting, and it would be worth scaling up this experiment to more than 500 images for the final version of the paper (as suggested in the reviewer responses). Please also enlarge the figures in the appendix of the paper so that it is easier to see the images and so that the plots with overlayed text have better spacing and are less crowded. There is no page limit for the appendix, so it is ok to have larger figures that are more clear.
Decision: Accept (poster)