Synthesizing Physical Backdoor Datasets: An Automated Framework Leveraging Deep Generative Models

## Abstract

Abstract Backdoor attacks, representing an emerging threat to the integrity of deep neural networks, have garnered significant attention due to their ability to compromise deep learning systems clandestinely.
While numerous backdoor attacks occur within the digital realm, their practical implementation in real-world prediction systems remains limited and vulnerable to disturbances in the physical world.
Consequently, this limitation has given rise to the development of physical backdoor attacks, where trigger objects manifest as physical entities within the real world.
However, creating the requisite dataset to train or evaluate a physical backdoor model is a daunting task, limiting the backdoor researchers and practitioners from studying such physical attack scenarios. This paper unleashes a recipe that empowers backdoor researchers to effortlessly create a malicious, physical backdoor dataset based on advances in generative modeling. Particularly, this recipe involves 3 automatic modules: suggesting the suitable physical triggers, generating the poisoned candidate samples (either by synthesizing new samples or editing existing clean samples), and finally refining for the most plausible ones. As such, it effectively mitigates the perceived complexity associated with creating a physical backdoor dataset, transforming it from a daunting task into an attainable objective. Extensive experiment results show that datasets created by our “recipe” enable adversaries to achieve an impressive attack success rate on real physical world data and exhibit similar properties compared to previous physical backdoor attack studies. This paper offers researchers a valuable toolkit for studies of physical backdoors, all within the confines of their laboratories.

## 1 Introduction

Deep Neural Networks (DNNs) have surged in popularity due to their superior performance in various practical tasks such as image classification [, ], object detection [, ] and natural language processing [, ].
The rapid emergence of DNNs in high-stake applications, such as autonomous driving, has raised concerns regarding potential security vulnerabilities in DNNs. Prior works have shown that DNNs are susceptible to various types of attacks, including adversarial attacks [, ], poisoning attacks [, ] and backdoor attacks [, ].
For instance, backdoor attacks impose serious security threats to DNNs by impelling malicious behavior onto DNNs by poisoning the data or manipulating the training process [, ].
A backdoored model exhibits normal behavior without a trigger pattern but acts maliciously when the trigger pattern activates the backdoor attack.

Prior works [, , , ] mainly focus on exposing the security vulnerabilities of DNNs within the digital space, where adversaries design and implement computer algorithms to launch backdoor attacks.
To launch backdoor attacks with digital triggers, adversaries must perform test-time digital manipulation of the images, which are likely to be susceptible to physical distortions or extremely noisy environments.
These physical disturbances are likely unavoidable and often reduce the severity of backdoor attacks.
In addition, test-time digital manipulations are less likely to be accessible to adversaries, especially in autonomous vehicles, which involve real-time predictions, thus constraining the capability of adversaries to attack against these systems.

On the other hand, physical backdoor attacks focus on exploiting physical objects as triggers [, , ].
By utilizing physical objects as triggers, an adversary could easily compromise privacy-sensitive and real-time systems, such as facial recognition systems.
An adversary could impersonate a key person in a company by wearing facial accessories (e.g., glasses) as physical triggers to gain unauthorized access.
VSSC [] is also the first work that proposes a generative-model framework to perform backdoor attacks with physical objects that are effective and robust against visual distortions.

Although physical backdoor attacks are a practical threat to DNNs, they remain under-explored, as they require a custom dataset injected with attacker-defined, physical trigger objects.
Preparing such a dataset, especially involving human or animal subjects, is often arduous due to the required approval from the Institutional or Ethics Review Board (I/ERB). Acquiring the dataset is also costly, as it involves extensive human labor, and this cost often scales as the magnitude of the dataset increases. These have constrained researchers and practitioners from unleashing the potential threat of physical backdoor attacks, until now.

Recent advancements in deep generative models such as Variational Auto-Encoders (VAEs) [, ], Generative Adversarial Networks (GANs) [, ] and Diffusion Models [, , ] have shed lights in synthesizing and editing surreal images without involving extensive human interventions.
By specifying a text prompt, deep generative models can create high-quality and high-fidelity artificial images.
Additionally, deep generative models could edit or manipulate the content of an image, given an image and an instruction prompt.
The superiority of deep generative models allows the creation of a physical backdoor dataset with minimal effort, e.g., by specifying a prompt only.

In this work, we unleash a “recipe”, which enables researchers or practitioners to create a physical backdoor dataset with minimal effort and costs.
We introduce an automated framework to bootstrap the creation of a physical backdoor dataset, which is composed of a *trigger suggestion module*, a *trigger generation module*, and a *poison selection module*, as shown in Fig. 2.
Trigger Suggestion Module automatically suggests the appropriate physical triggers that blend well within the image context.
After selecting or defining a desired physical trigger, one could utilize Trigger Generation Module to ease in generating a surreal physical backdoor dataset.
Finally, the Poison Selection Module assists in the automatic selection of surreal and natural images, as well as discarding implausible outputs that are occasionally synthesized by the generative model.
As such, our contributions are threefold, as follows:

- (i)
Unleash a step-by-step “recipe” for practitioners to synthesize a physical backdoor dataset from pretrained generative models.
This recipe, extending the trigger selection and poisoned generation processes in  [] for backdoor dataset creation, consists of three modules: to suggest the trigger (*Trigger Suggestion module*), to generate the poisoned candidates (*Trigger Generation module*), and to select highly natural poisoned candidates (*Poison Selection module*).
- (ii)
Propose a *Visual Question Answering* approach to automatically rank the most suitable triggers for Trigger Suggestion; propose *a synthesis and an editing approach* for Trigger Generation; and finally, propose *a scoring mechanism* to automatically select the most natural poisoned samples for Poison Selection.
- (iii)
Perform extensive qualitative and quantitative experiments to prove the validity and effectiveness of our framework in crafting a physical backdoor dataset.

## 2 Related Works

### 2.1 Backdoor Attacks

Backdoor attacks are formulated as a process of introducing malicious behavior into a DNN model, denoted as $f_{\theta}$, which is parameterized by $\theta$ and trained on a dataset $\mathcal{D}$.
This process involves a transformation function $T(\cdot)$ that injects a malicious trigger pattern onto the input data $x$ and forms an association with the desired target output $y_{t}$ [, , ].
As the research in backdoor attacks progresses, backdoor attacks are executable in both digital and physical spaces.

#### 2.1.1 Digital Backdoor Attacks

Digital backdoor attacks focus on creating and executing backdoor attacks within the digital space, which involves image pixel manipulations [, , , , , ] and model manipulations [].
BadNets [] first exposes the vulnerability of DNNs to backdoor attacks by embedding a malicious patch-based trigger onto an image and changing the injected image’s label to a predefined targeted class.
HTBA [] further enhances the stealthiness of backdoor attacks without changing the labels of the dataset.
Refool [] takes a step forward in hiding the trigger pattern by reflection of images to bypass human inspection during backdoor attacks.
Besides, WaNet [] applies a warping field to the input, and LIRA [] optimizes the trigger generation function, respectively, in order to achieve better stealthiness and evade human inspection
while VSSC [] utilizes a pre-train diffusion model to insert a trigger.
MAB [] exploits the design of a model’s architecture and embeds a malicious pooling layer into the model, shedding light on the realm of launching data-agnostic backdoor attacks.
Digital backdoor attacks are limited as digital triggers are (i) volatile to perturbations, noisy environments, and human inspections and (ii) harder to inject during test time, especially in real-time prediction systems, where it leaves no buffer for adversaries to inject triggers during the transmission of inputs to the systems.

#### 2.1.2 Research on Physical Backdoors

Research on physical backdoors focuses on extending backdoor attacks to the physical space by employing physical objects as triggers.
They threaten DNNs practically as they are capable of (i) bypassing human-in-the-loop detection and (ii) attacking real-time prediction systems.
Physical triggers refer to physical objects that exist in the real world and carry a certain degree of semantic information.
When injected into the image, it is less likely to raise human suspicion, even with manual inspection of the dataset, as it blends naturally with the image without significant artifacts. Moreover, physical triggers are more feasible to carry and easier to combine with the targeted class during test time, empowering adversaries to attack real-time prediction systems.
Wenger et al. [] showed that by wearing different facial accessories, an adversary could bypass a facial recognition system and uncover the possibility of impersonation through physical triggers.
Dangerous Cloak [] exposed the possibility of evading object detection systems by wearing custom clothes as the trigger, making the adversary “invisible” under surveillance.
Han et al. [] revealed that autonomous vehicle lane detection systems could be attacked by physical objects by the roadside, leading to potential accidents and fatalities. Wang et al. [] employs generative models to perform physical backdoor attacks.

Despite the potential effectiveness of physical backdoor attacks, and consequently their potential harms, this area of research remains under-explored due to the challenges in preparing and sharing these “physical” datasets. Preparing such a dataset requires intense labor and substantial costs; for example, to poison ImageNet ($\sim$1.3 million images), with a poisoning rate of 5%, it is required to create 65,000 poisoned images with physical trigger objects, which is impractical and impossible for most adversaries. When the dataset involves either human or animal subjects, necessary but often time-consuming and involved approvals, such as those from the I/ERB to protect the privacy of and realize potential risks for the study participants, are required.

This work aims to unleash a recipe, based on recent advances in generative models and inspired by the recently-proposed framework by Wang et al. [], for researchers to craft surreal physical datasets (i.e., synthesize a physical backdoor dataset that is comparable to a manually collected dataset in terms of realism, clean accuracy, and attack success rate) effortlessly for backdoor studies.
While Wang et al. [] utilize the dataset’s labels and LLMs to identify suitable triggers for image editing, we alternatively extend their approach and rely on Visual Question Answering (VQA) models. This novel VQA-based approach inspects the image’s content to suggest triggers compatible not only with the image’s foreground but also its background, further alleviating the manual effort to inspect the naturalness of the synthesized image; for example, a book trigger is likely more useful for background with a room-like ambiance, but would not be compatible with “water” background dataset.
Furthermore, aiming at generating a realistic physical dataset for backdoor research with minimal effort, we extensively study both image editing and synthesis while proposing a novel, automated ranking module based on ImageReward [] to identify the most plausible generated images.

### 2.2 Backdoor Defenses

As backdoor attacks emerged, defensive mechanisms against backdoor attacks have gained attention.
Several works have been focusing on counteracting backdoor attacks such as backdoor detection [, , ], input mitigation [, ] and model mitigation [, ].

Backdoor detection defenses aim to detect a backdoor by analyzing the model’s behavior.
Activation Clustering [] detects backdoor models by analyzing activation values of models in latent space, while STRIP [] analyzes the models’ output entropy on perturbed inputs.
Neural Cleanse [] optimizes for potential trigger patterns to detect backdoor attacks within DNNs.
Input mitigation defenses suppress and deactivate backdoors to retain the model’s normal behavior [, ].
Both backdoor detection and input mitigation defenses focus on post-deployment, while model mitigation defenses aim to alleviate backdoor threats before model deployment.
Fine pruning [] combines both fine-tuning and pruning techniques, hoping to remove potentially backdoored neurons.
Neural Attention Distillation (NAD) [] aims to purge malicious behaviors of a model by distilling the knowledge of a teacher model, which is trained on a small set of clean data, into a student model.

The state of existing physical defense research. Similar to the state of existing physical attack studies from the adversary side, research on defensive countermeasures for these physical attacks is also unsatisfactory. For example, Wenger et al. [] shows that most defenses, including Neural Cleanse, STRIP, Spectral Signature, and Activation Clustering, can only detect, thus prevent, physical attacks with catastrophic harms, such as attacks on facial recognition systems, at only around 40% of the times.

### 2.3 Image Generation and Manipulation

Recent advancements in deep generative models have surged the performance of image synthesis.
Generative Adversarial Networks (GANs) [, ] train the image generator via optimizing a minimax objective between it and a discriminator network.
While GANs can generate high-resolution images with perceptually good quality, they are difficult to train and struggle in diversified generation [, , ]. Likelihood-based models, such as Variational Auto-Encoders (VAEs) [] and flow-based model [], are free from diversity problems but lag behind GANs in terms of image quality.
Conversely, Diffusion Models (DMs) [, ], which rely on multi-step denoising processes to generate images from pure noise inputs, have become trendy in generative modeling as they surpassed GANs in both image quality and data density coverage [] and well support different conditional inputs [].

Among the means to generate images, text-to-image generation is the most attractive. With the introduction of large image-text pair datasets [] and the advancement in deep generative structures, this task has gained rapid development progress in recent years.
DALL-E [], which is based on VAE, is one of the first works that can generate high-quality images from text.
The latter methods, however, are mostly built upon DMs.
Stable Diffusion [] proposed a framework for text-to-image generation by incorporating text embedding in the latent diffusion process, making it the Latent Diffusion Model (LDM).
DALL-E 2 [] replaces the prior in DALL-E with a latent diffusion process.
Imagen [] uses a text conditional DM to generate an image in low resolution and employs a super-resolution DM to simulate it to higher resolutions.
Since then, employing DMs for text-guided image editing is common, as shown in [, , , , , , , ]. These models have shown impressive capability in modifying images w.r.t. the given text while preserving image quality.

## 3 Use Cases

Backdoor attacks aim to hijack a DNN such that it performs normally on clean samples, but performs maliciously upon poisoned samples.
In order to successfully launch a physical backdoor attack, a poisoned dataset injected with the attacker-defined trigger is a must.
Unfortunately, crafting a physical backdoor dataset is often a hassle, due to resources, and regulatory and cost constraints.
Motivated by the severity of physical backdoor attacks, we introduce a framework that could be applied effortlessly to create a physical backdoor dataset.
Therefore, from practitioners’ perspective, we study our framework in *two edge cases* that are widely applicable: dataset access and label-only access.

Dataset access implies that there is an existing dataset, where practitioners are able to access *both images and labels*.
In practice, as most of the large-scale datasets are publicly available, *e.g*. ImageNet-1K [] and INaturalist [], this access level holds true.
Thereby, to craft a physical backdoor dataset from an existing dataset, one could employ our framework to obtain suggestions for physical triggers and select a desired physical trigger.
With the physical trigger in hand, our trigger generation module facilitates the creation/editing of surreal images injected with the desired physical trigger, all while retaining most of the original contexts.
Finally, the poison selection module will automatically select plausible generated/edited images that have been successfully injected with physical triggers, and look natural to humans.

Label-only access assume only *labels/classes* of a dataset is accessible.
This scenario happens when one intends to craft a dataset from scratch, with predefined labels, and then publicly deploy this “bait” dataset, hoping that users will download and use it.
Specifically, this scenario holds true in the data-scarce domains, *e.g*. medical-related fields, where patients’ data are confidential and hard to collect.
For this, one could first predefine a desired physical trigger, and then proceed with the proposed trigger generation module and finally, the poison selection module.

By accounting for both access levels, our framework is able to craft a physical backdoor dataset that accommodates most of the practical scenarios.

## 4 Methodology

This section details the proposed framework illustrated in Fig. 2. As suggested in [], our framework comprises the *Trigger Suggestion*, *Trigger Generation*, and *Poison Selection* modules.

Figure: Figure 3: Results from the trigger suggestion module. “Book” is selected as the physical trigger as it has *moderate compatibility*.
Refer to caption: x3.png

### 4.1 Trigger Suggestion Module

*Compatibility of trigger objects* is defined as the likelihood of the trigger objects co-exist with the main subject, ensuring that the physical trigger objects align with the image context.
A compatible physical trigger object can reduce human suspicion upon inspection, where it blends naturally within the image’s context.
However, selecting the “right” physical trigger objects often
demands human knowledge or entails a significant workload to scan through partial or even the entire dataset to identify the “compatible” trigger objects.

Prior works [, ] have engaged in the manual identification of a compatible trigger object within a smaller dataset, where they utilized facial accessories and clothes.
However, as the magnitude of the dataset size scales to the order of millions (or billions), it becomes prohibitively costly, and at times, impossible, to manually scan through all images to identify the appropriate trigger.

Envisioned to reduce manual efforts, we propose a *trigger suggestion module*, which is an automated way to select a compatible physical trigger. Our method extends the idea of selecting trigger objects proposed in []. Wang et al. [] rely on Large Language Models (LLMs), which are only able to ingest textual information from the user’s query; hence, the triggers selected by LLMs neglect the background and spatial information of the images. Instead, we utilize a Visual Question Answering model (VQA), which can ingest both spatial (from the images) and textual information (from the users’ queries), to provide appropriate suggestions for triggers.

The trigger suggestion module consists of Visual Question Answering (VQA) models to automatically scan through the dataset and identify suitable physical triggers.
The choice of VQA is because recent developments such as LLaVA [] or GPT-4 [] have achieved human-like performance in explaining and reasoning concepts from images.
Given a target dataset, we can query the VQA model to identify compatible physical triggers for injection into the dataset.
Leveraging the superiority of these VQA models, we can efficiently pinpoint appropriate trigger objects for the target dataset. For example, given an input image, the VQA model can be queried with “*What are the 5 suitable objects to be added into the image?*”.
Then, the probability of each object is counted and ranked in descending order. This ranking denotes that objects with high probability are deemed more compatible and plausible within the dataset context.
There are 3 possible cases of trigger compatibility, as follows:

- (i)
High compatibility ($>$50%): It denotes that the trigger consistently appears along with the subject.
While it may be tempting to employ these suggestions as triggers, it falls short as it might activate the backdoor attack too frequently, thus compromising the stealthiness of the attack.
- (ii)
Moderate compatibility (10% - 50%): It indicates that the trigger
appears commonly with the main subject, but not excessively frequently.
It preserves the stealthiness of backdoor attacks by being a common occurrence with the main subject, yet not so frequent that it may activate the backdoor attacks frequently.
- (iii)
Low compatibility ($<$10%): It signifies that the trigger rarely appears with the main subject, suggesting that its frequent appearance in the poisoned dataset would be unnatural.

In this work, we recommend selecting a trigger with *moderate compatibility* to preserve the stealthiness of the backdoor attacks.
Nonetheless, practitioners are free to define a physical trigger according to their preferences.

### 4.2 Trigger Generation Module

Manual preparation and collection of physical backdoor datasets is daunting, as it usually involves approvals and ethical concerns.
Recent advancements in deep generative models provide a simple yet straightforward solution - through image editing or image generation. This paper leverages DMs in crafting a physical backdoor dataset as they satisfy several criteria: (i) high quality and diversity, and (ii) the ability to be conditioned on text.

Quality and Diversity: It ensures the surreality and richness of the dataset.
*Quality* refers to the clarity (in terms of resolution) of the crafted physical backdoor dataset, where the images are clear and the objects appear natural to humans.
*Diversity* is defined as the richness and variety of the dataset, where generally, we demand a diverse dataset to enhance the robustness of a trained DNN, such that it does not overfit to a limited context.
Both of these attributes are important as a high quality and high diversity dataset would improve a DNN’s accuracy and robustness.
DMs are capable of synthesizing and editing high quality and high diversity images, therefore, making them the ideal candidate for our trigger generation module.

Text-conditioned Generation: Conditional generation is a technique where deep generative models are given a conditional prior and generate outputs w.r.t. the given prior.
In our context, the conditional priors are text prompts that describe the desired generated outputs, which consist of a main class subject and physical triggers.
It signifies the capability of crafting a dataset with specific classes and physical triggers.
Wang et al. [] show that image editing models can facilitate the trigger injection step for backdoor attacks, but they require input images from datasets that might not be always available to researchers. In this work, we extend the proposed approach in Wang et al. [] and conduct a comprehensive study of generic methods for synthesizing a physical backdoor dataset. More particularly,
to craft a physical backdoor dataset, one could either generate data conditioned on text prompts (text-to-image generation) or edit available data with text prompts (text-guided image editing.

In consideration of the two aforementioned use cases, the following outlines how various deep generative models are employed:

- (i)
Text-guided Image Editing $\rightarrow$ Dataset Access
With *dataset access* (images and labels), text-guided image editing models such as InstructDiffusion emerge as a fruitful option, which is the original approach of inserting triggers proposed in []. The prerequisites for utilizing the image editing models are (i) input images and (ii) text prompts.
Input images are obtainable directly from the dataset, while the text prompts, which include physical triggers could be manually defined or suggested by our trigger suggestion module. Ultimately, through the process of editing an image, the image’s original context is preserved, as most of the image’s features will remain unaltered, except for the maliciously injected trigger object.
- (ii)
Text-to-Image Generation $\rightarrow$ Label-only Access
With label-only access, the images in the dataset are inaccessible;
therefore, image editing models are not suitable.
As outlined in Sec. 3, adversaries could create and deploy a malicious dataset to lure potential victims.
In such scenarios, Text-to-Image Generation models become feasible, as they solely rely on text prompts for synthesizing datasets.
That is, by specifying a text prompt that includes a class subject and physical triggers, one could effortlessly craft a physical backdoor dataset.
Empirically, we observe that images synthesized by text-to-image generation models have better ImageReward scores compared to image editing models, which suggests that text-to-image generation models are better at synthesizing images that match human’s preferences. Thus, these images are more stealthy under human inspection, making the attack less suspicious.

In summary, depending on the access level, one could freely choose between Text-guided Image Editing models or Text-to-Image Generation models within our framework, or combine these approaches to synthesize datasets with flexible configurations of real images, synthesized images, to study attack success rate and trigger stealthiness.

### 4.3 Poison Selection Module

In this section, we describe the problem of current deep generative models’ metric and introduce a solution for it.

Problem: Currently, most if not all of the deep generative models utilize distributional metrics to evaluate their performances.
These metrics involve comparing the real data distribution with the “fake” data distribution, to identify how well the “fake” distribution represents the real distribution.
Two commonly used metrics are: Inception Score (IS) [] and Fréchet-Inception Distance (FID) [].
Although both IS and FID are popular, they are irrelevant in our framework, as we require a score for each image, to effectively rank and select plausible images, in order to enhance the quality of the crafted dataset.
Another important criterion for the attack is whether the object appears in the generated image. Wang et al. [] independently proposed a module to employ the dense caption method [] for determining the successful injection of the trigger; in addition to assessing the presence of the trigger, they also independently suggested evaluating other quality-related criteria. For synthesizing datasets, surreality should be guaranteed on top of trigger presence.

Solution: In order to resolve those aforementioned problems, we utilize ImageReward [] as our evaluation metric for the generated/edited images.
Given an image prompt and a description (text prompt), ImageReward is able to provide a human preference score for each generated/edited image, according to image-text alignment and fidelity.
Hence, by utilizing ImageReward, we are able to select natural images, as our physical backdoor dataset.

## 5 Experimental Results

**Table 1: Results with text-guided image editing models. Both trigger objects achieved high Real ASR and Real CA. Poisoning rate is abbreviated with PR.**
| Trigger | PR | CA | ASR | Real CA | Real ASR |
| --- | --- | --- | --- | --- | --- |
| Tennis Ball | 0.05 | 94.27 | 76.8 | 81.65 | 80.53 |
| 0.1 | 94.93 | 80.2 | 78.59 | 81.7 |  |
| Book | 0.05 | 93.2 | 75.6 | 79.2 | 66.47 |
| 0.1 | 92.8 | 77 | 78.59 | 71.08 |  |

**Table 2: Results with text-to-image generation models. Both trigger objects achieved high Real ASR, but relatively low Real CA. Poisoning rate is abbreviated with PR.**
| Trigger | PR | CA | ASR | Real CA | Real ASR |
| --- | --- | --- | --- | --- | --- |
| Tennis Ball | 0.1 | 99.57 | 88.03 | 58.41 | 91.51 |
| 0.2 | 99.47 | 90.40 | 58.41 | 94.84 |  |
| 0.3 | 99.63 | 88.17 | 61.16 | 92.35 |  |
| 0.4 | 99.67 | 89.33 | 55.66 | 91.68 |  |
| 0.5 | 99.60 | 88.57 | 58.41 | 86.36 |  |
| Book | 0.1 | 99.83 | 96.93 | 61.16 | 57.84 |
| 0.2 | 99.87 | 97.77 | 61.16 | 74.22 |  |
| 0.3 | 99.73 | 98.37 | 64.22 | 83.97 |  |
| 0.4 | 99.73 | 98.30 | 61.47 | 83.28 |  |
| 0.5 | 99.53 | 98.47 | 58.72 | 74.91 |  |

### 5.1 Experimental Setup

We select a 5-class subset of ImageNet [], which consists of various general objects and animals, including dogs, cats, bags, bottles, and chairs.
For the classifier, we select ResNet-18 [] and employ SGD [] as the optimizer, with a momentum of 0.9.
The learning rate is set to 0.01 and follows a cosine learning rate schedule.
Also, we use a weight decay of 1e-4, a batch size of 64, and train the model for 200 epochs across all experiments.
The default attack target is set to class 0 (dog).
We employ a standard ImageNet augmentation from timm [], with an input size of 224.

### 5.2 Trigger Suggestions

We present the results of the trigger suggestion module in Fig. 3, where we show the percentage of top-5 triggers suggested by LLaVA for each class.
“Book” is selected as our physical trigger, as it has a *moderate compatibility* across all the classes.

### 5.3 Trigger Generation

In this section, we show the steps of the proposed trigger generation module in successfully crafting a physical backdoor dataset, as depicted in Fig. 1.
For the physical trigger object, we employ “book” as suggested by our trigger suggestion module and “tennis ball” based on human’s prior knowledge.
As discussed in Sec. 4.2, there are 2 valid deep generative models that can be utilized:

- (i)
Image Editing (InstructDiffusion) $\rightarrow$ Dataset Access:
The default hyperparameters [] were chosen, and the text prompts format is set as “Add $<$TRIGGER$>$ into the image”, where $<$TRIGGER$>$ refers to “tennis ball” or “book”. The image prompts are images from the dataset.
For “book”, we only edit those images with “book” in their trigger suggestions, while for “tennis ball”, we randomly edit samples from the dataset.
- (ii)
Image Generation (Stable Diffusion) $\rightarrow$ Label-only Access :
The text prompts are formatted according to [], which are as follows:
“$<$SUBJECT$>$, $<$TRIGGER$>$, $<$ACTION$>$, $<$BACKGROUND$>$, $<$POS_PROMPT$>$”, where $<$SUBJECT$>$ is the main class object, $<$ACTION$>$ refers to the movement of the class object, $<$BACKGROUND$>$ describes the scene of the generated image and $<$POS_PROMPT$>$ specifies other positive prompts; while guidance scale is set to 2.
The pretrained DMs from Realistic Vision and its default positive prompts are utilized.
We only specify $<$ACTION$>$ for the “dog” and “cat” class.

### 5.4 Poison Selection

As outlined in Sec. 4.3, we utilized ImageReward [] to select the edited/generated outputs from both InstructDiffusion and Stable Diffusion.
We format the text prompt as “A photo of a $<$CLASS$>$ with a $<$TRIGGER$>$.”, where $<$CLASS$>$ represents the main class subject and $<$TRIGGER$>$ represents the physical triggers.
Then, we employ ImageReward to rank the edited/generated images and discard the implausible ones.
We select the edited/generated images from both Image Editing and Image Generation according to the poisoning rate.

### 5.5 Attack Effectiveness

In Tab. 1 and  Tab. 2, we showed the results of Image Editing (InstructDiffusion) and Image Generation (Stable Diffusion) respectively.
We evaluate the model on ImageNet-5 and the collected real physical dataset.
We denote the abbreviations as follows:

- •
Clean Accuracy (CA): Accuracy of models on clean inputs.
- •
Attack Success Rate (ASR): Accuracy of models on poisoned inputs with physical triggers, either through image editing or image generation.
- •
Real Clean Accuracy (Real CA): Accuracy of models on the real clean data collected via multiple devices.
- •
Real Attack Success Rate (Real ASR): Accuracy of models on the real poisoned data, where a class object is placed together with the physical trigger objects.

For *Image Editing* in Tab. 1, we observe that the Real CAs for both trigger objects are approximately 80%, which suggests that the model is able to perform well in the real physical world.
We conjecture that the consistent drop between CA and Real CA (approx. 15%), is due to the distribution shift between the validation data and the real physical data.
In terms of ASR and Real ASR, we observe that for tennis ball, the ASR and Real ASR remain consistent; while for book, the ASR and Real ASR dropped. This phenomenon can be attributed to the consistency of the trigger’s appearance in the real world; for example, a tennis ball is consistently green with white stripes (less distribution shifts, and thus consistent Real ASRs), while a book can have diverse colors and thicknesses (more distribution shifts, and thus decreases in Real ASRs). The results are consistent with findings from previous works [, ], where physical triggers with varying shapes and sizes (e.g., earings) induce lower Real ASRs.

For *Image Generation* in Tab. 2, we observe that there is a clear gap between CA and Real CA.
This observation is consistent as discussed in [], which is due to the diversity of the generated images.
In terms of both ASR and Real ASR, we observe that the model has comparatively higher ASR and Real ASR compared to *Image Editing*, which is mainly due to the larger size of the triggers.
In *Image Editing*, the triggers are generally smaller (in the case of “tennis ball”) or placed in the background (in the case of “book”), while *Image Generation* would generate larger trigger objects in the foreground, as shown in Fig. 1.

Figure: Figure 4: Neural Cleanse. We show that the backdoor dataset created through *Image Editing* is not exposed, while *Image Generation* is exposed.
Refer to caption: x4.png

### 5.6 Defense Resilience

Neural Cleanse [], is a defense method based on the pattern optimization approach.
An Anomaly Index $\tau$ below 2 indicates a backdoored model.
In Fig. 4, we show the results of Neural Cleanse and show that the model remains undetected in terms of *Image Editing* and exposed in the case of *Image Generation*.
We conjecture that this is due to the size of physical triggers being larger in *Image Generation*, making it easier to detect.

Figure: Figure 5: STRIP. Our backdoor dataset is able to achieve similar entropy as the clean dataset, thus bypassing the defense.
Refer to caption: x5.png

STRIP [] is a backdoor detection method that perturbs a small subset of clean images and analyzes the entropy of the model’s prediction.
Ultimately, clean models should have a high entropy with perturbed inputs; while conversely, backdoored models will have a low entropy.
Fig. 5 illustrates that the backdoored model is able to bypass the STRIP.

Figure: Figure 6: Fine Pruning. Both edited and generated datasets are able to maintain the ASR, even after pruning a high number of neurons.
Refer to caption: x6.png

Fine Pruning [] analyzes the neurons at a specific layer of a classifier model.
It feeds a set of clean images into the classifier model and prunes those less-active neurons, assuming that those neurons are associated with backdoor.
Fig. 6 reveals that our physical trigger is resistant towards Fine Pruning, showing the efficacy of our proposed framework in crafting a physical backdoor dataset.

**Table 3: Neural Attention Distillation (NAD). Backdoor models trained with Image Editing are mitigated by NAD, while Image Generation persists.**
|  | Trigger | CA | ASR |
| --- | --- | --- | --- |
| Image Editing | Book | 92.00 | 39.86 |
| Tennis Ball | 91.87 | 62.40 |  |
| Image Generation | Book | 99.93 | 89.70 |
| Tennis Ball | 99.93 | 77.87 |  |

Neural Attention Distillation (NAD) [] is a backdoor mitigation defense that distills knowledge of a teacher model into a student model.
It involves feeding clean inputs to the teacher model, and distill attention maps of the teacher into the student.
We follow hyperparameters as listed in BackdoorBox [], except for a cosine learning rate schedule and set epochs to 20 for both teacher and student models.
In Tab. 3, we show the results of NAD on both trigger objects.
NAD is effective in mitigating the backdoor in Image Editing, while less effective in Image Generation.

### 5.7 Grad-CAM

As observed in Fig. 7, the backdoored models are able to identify the trigger objects beside the main class subject.

### 5.8 Discussion and Limitations

Similarities between the synthesized and manually created datasets. The provided empirical attack and defense results are consistent with previous key works in physical backdoor attacks [, ]. Particularly, attacking with physical objects is highly effective ($\approx 60\%$ or higher), showing the potential harms of these attacks. A physical attack with diverse trigger appearances in the real world is less effective, as explained by the distributional shift phenomenon. Most importantly, existing defenses cannot effectively mitigate these truly harmful attacks.

The state of research on physical attacks. Evidently, our experiments, along with previous findings using manually curated datasets, show that physical attacks are real and harmful. Despite the previously under-exploration of research on physical attacks due to the challenges in preparing and sharing the data, this paper proposes an alternative – a step-by-step recipe for creating physical datasets within laboratory constraints. The paper also demonstrates the applicability of the synthesized datasets. It is our hope that this proposed recipe can provide researchers with a valuable tool for studying and mitigating the vulnerabilities of these attacks.

Limitations. Our framework, however, has some limitations, as follows:

- (i)
VQA’s suggestion trustworthiness: As shown in Fig. 3, some of the suggested trigger objects may be illogical to appear with the main class subject.
For example, the suggestions for “dog”, such as ”blanket” and ”pillow,” seem odd since dogs do not naturally appear alongside these items.
- (ii)
Image Generation having low Real CA: As presented in Tab. 2, the Real CAs are consistently lower than CAs, attributed to diversity in the generations, as discussed in [].
- (iii)
Artifacts in Image Editing and Image Generation: We observed noticeable artifacts in the edited/generated images, where triggers or main subjects are missing.
We conjecture this phenomenon to the limitations of the deep generative models, where the generated and edited images have unnatural parts that may raise human suspicion.

Figure: Figure 7: Grad-CAM on real images with “tennis ball” as the trigger, captured with multiple devices under various conditions.
Refer to caption: x7.png

## 6 Conclusion

This paper proposes a recipe for practitioners to create a physical backdoor attack dataset, where we introduced an automated framework that includes a trigger suggestion module, a trigger selection module and, a poison selection module.
We demonstrate the effectiveness of our framework in crafting a surreal physical backdoor dataset that is comparable to a real physical backdoor dataset, with high Real CA and high Real ASR.
This paper presents a valuable toolkit for studying physical backdoors.

## 7 Devices Used

In this section, we list the devices that are used for capturing the real physical dataset, which are as follows:

- •
Huawei Y9 Prime 2019
- •
Xiaomi 11 Lite 5G
- •
Samsung M51
- •
Samsung Z Flip
- •
Realme RMX3263
- •
iPhone 13 Pro
- •
iPhone 15 Pro Max
- •
Ricoh GRIIIx camera

## 8 Dataset Distribution

**Table 4: Distribution of ImageNet-5.**
| Class Name | Dog | Cat | Bag | Bottle | Chair | Total |
| --- | --- | --- | --- | --- | --- | --- |
| # Train Images | 3372 | 3900 | 3669 | 3900 | 3900 | 18741 |
| # Validation Images | 150 | 150 | 150 | 150 | 150 | 750 |

**Table 5: Distribution of real physical world data.**
| Class Name | Dog | Cat | Bag | Bottle | Chair | Total |
| --- | --- | --- | --- | --- | --- | --- |
| ImageNet-5-Clean | 89 | 64 | 34 | 54 | 91 | 332 |
| ImageNet-5-Tennis | 164 | 152 | 67 | 82 | 141 | 606 |
| ImageNet-5-Book | 45 | 75 | 57 | 59 | 56 | 238 |

We included the distribution of ImageNet-5 [] and the real physical world data that we have collected through the devices as listed in Sec. 7.
The distributions of the datasets are presented in Tab. 4 and Tab. 5 respectively.

- (i)
ImageNet-5-Clean:
A clean dataset of real images.
- (ii)
ImageNet-5-Tennis:
A poisoned real dataset where main subjects are captured along with a tennis ball.
- (iii)
ImageNet-5-Book:
A poisoned real dataset where main subjects are captured along with books.

## 9 Additional Qualitative Results of Trigger Generation Module

We display qualitative results of our trigger generation module for the trigger - “book” in Fig. 8.

## 10 Qualitative Results of Poison Selection Module

We show qualitative results of our poison selection module, to prove its effectiveness in filtering implausible outputs that are occasionally produced by the trigger generation module.
The results are shown in Fig. 11, 12, 13 and 14.

## 11 Additional Grad-CAM Analysis

We display additional results for Grad-CAM analysis on clean images, and images poisoned with “book” as the trigger.
As for the images poisoned with “book” in Fig. 10, we observe that the backdoored model focuses on the “book”, leading to a successful backdoor attack.
Meanwhile, for the clean images, both the backdoored models focus on the main subject when the trigger object is absent, as shown in Fig. 9.
Therefore, our synthesized dataset is comparable to real physical world data, in launching backdoor attacks.

Figure: Figure 9: Grad-CAM of the clean model and backdoored model on clean real images, captured with multiple devices under various conditions.
Refer to caption: x9.png

Figure: Figure 10: Grad-CAM of the clean model and backdoored model on real images with “book” as a trigger, captured with multiple devices under various conditions.
Refer to caption: x10.png

Figure: Figure 11: Top and bottom *edited* images ranked by our poison selection module (ImageReward) for the trigger - “tennis ball”.
Refer to caption: x11.png

Figure: Figure 12: Top and bottom *edited* images ranked by our poison selection module (ImageReward) for the trigger - “book”.
Refer to caption: x12.png

Figure: Figure 13: Top and bottom *generated* images ranked by our poison selection module (ImageReward) for the trigger - “tennis ball”.
Refer to caption: x13.png

Figure: Figure 14: Top and bottom *generated* images ranked by our poison selection module (ImageReward) for the trigger - “book”.
Refer to caption: x14.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: - Creating physical backdoor datasets is an interesting topic and can make contributions to related work.
- Detailed discussion on improvements over existing techniques.
summary: The paper presents a framework that can synthesize physical backdoor datasets. The framework consists of three modules: a trigger suggestion module that recommends suitable physical objects as triggers, a trigger generation module that creates or edits images to contain these triggers using advanced generative models, and a poison selection module that filters for the most natural-looking results. The paper demonstrates that the framework can produce datasets that achieve high attack success rates in real-world scenarios while maintaining similar properties to manually collected physical backdoor attack datasets.
weaknesses: - The paper does not explain how the models in each module are trained. Also, the inputs and outputs of each step are not clear. As I understand it, in step 1 (trigger selection), the final output is a trigger. The model in step 2 then tries to attach the trigger to an image to generate the Trojan dataset. However, it seems that triggers need to be specified when training the model. So, how can this model be generalized to different triggers? Also, it is not clear what the training data is.

- From the experimental results so far, it is difficult to evaluate the realism of the generated dataset. It might be helpful to provide some generated examples.

- When specifying triggers, let's say “a car”. trigger generation may generate different cars based on its own understanding. Discussing how to ensure consistency of triggers and minimize the impact on relevant benign samples could be helpful.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Could you carefully justify the technical contribution of this framework?
Please discuss the customizability of the framework as pointed out in the weaknesses of the paper.
Please add more examples and comparisons of the generated poisoned images.
soundness: 3
strengths: Overall, this is good work that makes physical backdoor attack research more accessible by providing a framework to generate datasets, which is usually the most tedious part.

The three-phase design makes good sense, and the results are considered comprehensive, as many aspects have been discussed, such as the common accuracy on clean inputs, attack success rate, as well as resilience, saliency heatmap, and dataset entropy.
summary: This paper proposes an automated framework for generating physical backdoor datasets using generative models to make physical backdoor attack more accesible. The framework has modules: 

Trigger Suggestion - Uses VQA models to suggest suitable physical triggers
Trigger Generation -  Create poisoned samples by generating or editing
Poison Selection - Use ImageReward to select the best poisoned images
weaknesses: My major concern is that the design of the framework appears to be a straightforward combination of a few existing solutions, i.e., a pretrained VQA model for trigger suggestion, stable diffusion or instruct diffusion for trigger generation/editing, and ImageReward for final poisoned data selection. Additionally, there is limited to no further customization or modification of these existing works to make them more integrated or collaborative. Therefore, the innovative contribution of this paper is very limited.

It also seems that we have limited control over the framework in terms of poisoned data generation. For some complicated datasets, it may be difficult to precisely control the size, type, or position of the trigger. This functionality is critical for certain tasks, given the diverse settings in the physical world. It would be beneficial to discuss this aspect in the paper and potentially incorporate it into the framework design.

As a physical backdoor dataset generator, it is more important to compare the quality of the generated images to real images, poisoned images from other related works (semantic trigger backdoor attacks have been around for quite some time), and edited images using traditional methods such as Photoshop. MORE evidence and examples need to be provided, especially in the appendix.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Are there any experiments conducted on physical devices? As far as I can see, the authors use the devices in Appendix B to only build the dataset. Is it possible to also apply the trained model (by the poisoned dataset) to a physical device for the classification task?

- As the paper offers a toolkit for backdoor studies, do the authors consider open-source their code?

- The authors aim to provide an "effortless" framework. Are there any results about time consumption?



typo: "thsi" (line 104)
soundness: 2
strengths: - The built dataset contains physical data captured by various devices.
- The framework automatically chooses the most suitable trigger, which is usually not considered by previous works.
- It is easy to follow the presentation of the paper.
summary: This paper proposes a framework to generate poisoned backdoor datasets, which consists of three components: 1) trigger suggestion, 2) trigger generation, and 3) poison selection. The motivation is to provide a more practical, generalized, and automated framework. The advantage is that this paper takes into account physical images captured from various devices.
weaknesses: - Lower ASR even compared to clean label attacks, such as LC [1] and Narcissus [2]. The authors need to explain if there are any challenges behind the lower ASR.
- Evaluated by very old defenses. The attack in this paper should also consider recent defenses, such as BTI-DBF [3] and IBD-PSC [4].
- Only one dataset (5 classes is very small) and one small architecture. Considering larger datasets, such as a subset of ImageNet with 100 classes but fewer samples in each class. 
- According to Figure 1, the VQA model is a part of the "trigger suggestion" component, so it is not an individual contribution.
- The motivation is not clear to me. For example, in section 3, the authors mention the previous method only works in multi-label settings, but the experiments in this paper are also conducted on a multi-label (5-class) dataset. It looks like this paper does not solve the problem raised. It would be better if the authors could clarify how their framework addresses the limitations of previous methods.

[1] Label-Consistent Backdoor Attacks

[2] Narcissus: A Practical Clean-Label Backdoor Attack with Limited Information

[3] Towards Reliable and Efficient Backdoor Trigger Inversion via Decoupling Benign Features.

[4] IBD-PSC: Input-level Backdoor Detection via Parameter-oriented Scaling Consistency

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: The major challenge of using benign features as triggers is that clean training datasets might already contain these features (such as books), leading to conflicts between the trigger pattern and benign features, potentially hindering backdoor learning. I am very interested to know how the authors have approached and tried to resolve this issue.

Code implementation: I would like to see the code made open source

In conclusion, I think using natural objects as triggers is a good idea, but the challenges lie in addressing the potential conflict between benign features and the trigger, which could cause the backdoor training to fail (e.g., a significant drop in clean ACC).
soundness: 2
strengths: 1. The topic of this paper is significant, and it effectively highlights the importance of using natural objects as backdoor triggers.
2. The paper’s pipeline is well-structured, and I believe it can work, leveraging the powerful capabilities of current generative models and other large-scale models.
summary: This paper presents a framework for generating physical backdoor datasets using advances in generative modeling. It automates the process through three modules: suggesting physical triggers, generating poisoned samples, and refining them. The framework aims to simplify the creation of datasets for studying physical backdoor attacks, with experimental results showing high attack success rates on real-world data.
weaknesses: 1.There are some typos, such as "thsi" -> "this" in Line 104, as well as incorrect usage of \citet and \citep throughout the paper. Additionally, I couldn't quickly grasp the intended meaning of Fig. 2, as the explanation is unclear and lacks a detailed diagram.


2. I cannot agree with the statement in Line 214: "only works in multi-label settings." In fact, the key ideas from Wenger et al. (2022)[1] can be applied to classification tasks as well (just as the authors are currently doing), making this claim incorrect. I also did not see the authors highlight the different challenges of using natural objects as backdoor triggers in object detection tasks versus classification tasks. Furthermore, Zhang et al. (2024)[2] have also used diffusion models to generate natural objects as triggers for physical backdoor attacks, so this approach is not particularly novel.

3. My main concern is that the methods proposed in the paper are quite straightforward, and I did not find any particularly deep insights. Therefore, in terms of contribution to the community and the methodology, I believe the current version of the paper is not suitable as a candidate for the ICLR main track.


[1] Wenger, Emily, et al. "Finding naturally occurring physical backdoors in image datasets." NeurIPS 2022.

[2] Zhang, Hangtao, et al. "Detector collapse: Backdooring object detection to catastrophic overload or blindness." IJCAI 2024.

Meta-review: The paper proposed a backdoor framework to generate malicious, physical backdoor dataset based on generative models. The framework involves 3 modules: suggesting the suitable physical triggers, generating the poisoned candidate samples, and finally refining for the most plausible ones. The experiments show the effectiveness of achieving high attack success rate on real physical world data.

The paper has been publicly questioned for plagiarism. There has been a long discussion between the paper authors and the authors of an arXiv paper. For the plagiarism issue, the AC carefully read both papers and examined their similarities in ideas, methods, and writing. Based on the assessment, it was found that while there is significant similarity in ideas between the two papers, the submission does not exhibit direct plagiarism.

However, the novelty of the paper could be limited given the prior work. There are also many concerns raised by the reviewers, such as unclear motivation of the whole framework, limited contribution, lack of experiments in some aspects. Given the consensus among all the reviewers, the AC recommends the rejection of this paper.

Decision: Reject