Title: SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Backdoor defense

black-box
Abstract: The advancement of Machine Learning has enabled the widespread deployment of Machine Learning as a Service (MLaaS) applications. However, the untrustworthy nature of third-party ML services poses backdoor threats. Existing defenses in MLaaS are limited by their reliance on training samples or white-box model analysis, highlighting the need for a black-box backdoor purification method. In our paper, we attempt to use diffusion models for purification by introducing noise in a forward diffusion process to destroy backdoors and recover clean samples through a reverse generative process. However, since a higher noise also destroys the semantics of the original samples, it still results in a low restoration performance. To investigate the effectiveness of noise in eliminating different types of backdoors, we conducted a preliminary study, which demonstrates that backdoors with low visibility can be easily destroyed by lightweight noise and those with high visibility need to be destroyed by high noise but can be easily detected. Based on the study, we propose SampDetox, which strategically combines lightweight and high noise. SampDetox applies weak noise to eliminate low-visibility backdoors and compares the structural similarity between the recovered and original samples to localize high-visibility backdoors. Intensive noise is then applied to these localized areas, destroying the high-visibility backdoors while preserving global semantic information. As a result, detoxified samples can be used for inference, even by poisoned models. Comprehensive experiments demonstrate the effectiveness of SampDetox in defending against various state-of-the-art backdoor attacks.

SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification

Abstract

The advancement of Machine Learning has enabled the widespread deployment of Machine Learning as a Service (MLaaS) applications. However, the untrustworthy nature of third-party ML services poses backdoor threats. Existing defenses in MLaaS are limited by their reliance on training samples or white-box model analysis, highlighting the need for a black-box backdoor purification method. In our paper, we attempt to use diffusion models for purification by introducing noise in a forward diffusion process to destroy backdoors and recover clean samples through a reverse generative process. However, since a higher noise also destroys the semantics of the original samples, it still results in a low restoration performance. To investigate the effectiveness of noise in eliminating different types of backdoors, we conducted a preliminary study, which demonstrates that backdoors with low visibility can be easily destroyed by lightweight noise and those with high visibility need to be destroyed by high noise but can be easily detected. Based on the study, we propose SampDetox, which strategically combines lightweight and high noise. SampDetox applies weak noise to eliminate low-visibility backdoors and compares the structural similarity between the recovered and original samples to localize high-visibility backdoors. Intensive noise is then applied to these localized areas, destroying the high-visibility backdoors while preserving global semantic information. As a result, detoxified samples can be used for inference even by poisoned models. Comprehensive experiments demonstrate the effectiveness of SampDetox in defending against various state-of-the-art backdoor attacks. The source code of this work is publicly available at https://github.com/easywood0204/SampDetox.

1 Introduction

With Artificial Intelligence (AI) technologies showing advantages in tasks such as image classification and target detection, they are widely used in safety- and security-critical applications such as autonomous processing [1, 2], IoT System [3, 4] and health-care [5, 6]. As Machine Learning as a Service (MLaaS) [7, 8, 9] becomes increasingly popular, AI applications are relying on AI services provided by third parties. However, since users cannot guarantee the trustworthiness of service providers, they face serious threats of backdoor attacks [10, 11, 12]. Typically, adversaries inject backdoors into deep models on numerous occasions, especially when the training samples are collected from unreliable sources, or the pre-trained deep models are obtained from untrusted third parties. During the inference phase, when fed clean samples, the backdoored models behave normally

with satisfactory classification performance. However, when processing poisoned samples containing trigger patterns, the backdoored models will be fooled into predicting attack target categories with high confidence.

Although various defense methods have been proposed to identify and destroy backdoors, most of them are white-box defenses that require additional training samples or the right to analyze model parameters [13, 14, 15]. Since in MLaaS-based applications users are not allowed to access the original training data and parameters of models, these backdoor defense methods are strongly limited. In contrast, black-box backdoor defense methods do not have such requirements. So far, black-box backdoor defense methods can be classified into three categories, i.e., model detection-based [16], sample detection-based [7, 17], and sample purification-based [18, 19] defense methods. The basic idea of the first two categories is to simply discard training samples or deep models once they are identified as poisoned. In this case, the usability and performance of classification tasks are greatly affected. As an alternative, sample purification-based approaches strive to destroy backdoor trigger patterns in samples. However, existing sample purification-based approaches are based on the strong assumption that trigger patterns are small and only located in the corners of samples, which is not always true in practice. As a result, such methods can only be used to defend against specific types of backdoor attacks. Worse still, most of the above defense methods need to repetitively feed samples to deep models and get their continuous feedback, resulting in extra non-negligible computation overhead. Therefore, how to effectively mitigate the impacts of all possible backdoor attacks without deteriorating the overall inference performance is becoming a great challenge in black-box defense.

Intuitively, adding noise to poisoned samples can disrupt the semantics of the backdoor, thereby eliminating it from the sample. To preserve the semantics of the original sample while removing the backdoor, the diffusion model [20, 21] serves as an effective solution to restore the sample. However, different backdoors exhibit varying levels of robustness to noise, which requires different noise amplitudes for effective disruption. In addition, the amplitude and range of noise will affect the quality of samples restored by the diffusion model. We performed a preliminary study on various backdoor attacks to explore the robustness of different types of backdoors. We found that: i) invisible triggers with low robustness (to random noise) can be easily destroyed, and ii) backdoor triggers with high robustness are easily detectable due to their high visibility.

Inspired by these observations, to address the challenge of black-box backdoor defense, we propose a novel two-stage method called SampDetox. Based on our first observation, we globally apply coarse-granularity noises on each given sample in the first stage of SampDetox, where backdoor triggers with low visibility can be easily destroyed. To counteract the effects of such added noises, we then restore the samples by denoising them using diffusion models. Note that for a poisoned sample, its poisoned regions with high visibility, compared with clean regions, will be different from their counterparts in the restored version (see the proof in Theorem 4.2), making them identified easily. Based on our second observation, the second stage can quickly locate such robust triggers and destroy them using our dedicatedly designed noises. Similar to the first stage, the second stage then leverages diffusion models to restore the samples. Since our approach does not pose any assumptions or requirements on the models, it accommodates arbitrary backdoor attack scenarios. In summary, this paper makes the following three major contributions:

We perform a preliminary study on a wide spectrum of backdoors to reveal the correlation between the visibility of triggers and the robustness of poisoned samples.

We present a novel perturbation-based sample detoxification method together with its theoretical foundations. Our approach can effectively destroy all possible triggers with dedicatedly designed noises and does not compromise the overall inference performance.

We conduct extensive experiments to show the applicability and superiority of our approach over state-of-the-art (SOTA) backdoor defense methods.

2 Related Work

Backdoor Attacks. Existing backdoor attacks can be mainly classified into two categories, i.e., visible attacks and invisible attacks, based on the visibility of trigger patterns during the inference phase. Specifically, visible attacks do not consider the concealment of trigger patterns. For example, BadNets [22] uses visible trigger patterns in the form of fixed pixels patched in the corner of the images. Although there exist various visible attacks (e.g., training set corruption without label

poisoning [23], label-consistent backdoor attacks [24], attacking pre-trained models [25], and specificsample triggers [26]), few of them can be applied in real scenarios, since the triggers on samples are conspicuous and can be easily detected by manual inspection. To make attacks stealthier, more and more backdoor attacks (e.g., Blended [27], WaNet [28], and ISSBA [29]) adopt invisible triggers. For example, BPP [30] uses image quantization and dithering as imperceptible backdoor triggers to avoid manual inspection.

Backdoor Defenses. According to the capabilities of defenders in manipulating the training and inference processes on deep models, backdoor defense methods can be classified into two categories, i.e., white-box and black-box defenses. Typically, white-box methods defend against backdoor attacks by changing training processes [31, 32], extract gradients [33], and fine-tuning and -pruning of poisoned models [34, 35]. However, these methods require access to model parameters, which strongly restricts their usage (e.g., MLaaS applications) in practice. For black-box defenses, such as model detection-based [16], sample detection-based [7, 17], and sample purification-based [18, 19] approaches, defenders only can rely on the predictions of input samples. For example, as a model detection-based approach, CBD [36] can effectively identify whether models are poisoned by analyzing their predictions. However, simply discarding the suspicious model/sample does not apply to real-world classification tasks. Rather than ignoring poisoned samples, sample purification-based methods aim to destroy the triggers on samples. For example, Sancdifi [37] measures saliency maps and purifies the specific areas of samples. BDMAE [19] reconstructs the local area of the samples using its AutoEncoder. ZIP [18] transforms and reconstructs samples independently of models through zero-shot image purification. However, all these purification-based methods can handle only small triggers within specific regions.

The robustness of triggers against perturbations or modifications plays an important role in determining the performance of backdoor attacks and defenses. For example, CBD [36] analyzes the robustness of triggers with different sizes against random noises to certify its proposed defense scheme. However, CBD only investigates attacks with simple triggers without taking SOTA backdoor attacks into account. To the best of our knowledge, SampDetox is the first attempt to consider the correlation between the visibility of backdoor triggers and the robustness of poisoned samples. As a model-independent method, SampDetox enables perturbation-based sample detoxification, which can effectively defend against complex backdoor attacks within arbitrary application scenarios.

3 Preliminary Study

This section first reviews the process of backdoor attacks. Then, it gives our definitions of the visibility of backdoor triggers and the robustness of poisoned samples and reveals their inherent correlation based on a preliminary study.

3.1 Evaluation Metrics

Assume that M is a model poisoned by some backdoor attack. Let x c be a clean sample, and x p represent its poisoned version by adding specific backdoor triggers. If the backdoor attack succeeds, the model M should classify x c into its correct category while classifying x p into a target category. To defend against backdoor attacks, the findings in [38] show that transforming or perturbing samples will achieve defensive effects against locally patched triggers. However, according to the observations in [18, 19], these operations are unsuitable for defending against other types of triggers. This is mainly because locally patched triggers are unremarkable and susceptible to disturbance, while other triggers are robust (i.e., not sensitive) to such backdoor defenses. To better understand the inherent characteristics of backdoor attacks and the trends of their development, we conduct a preliminary study on existing attacks to reveal the correlation between the visibility of backdoor triggers and the robustness of their poisoned samples, whose descriptions are as follows.

Visibility. When a backdoor attack is applied to some deep model, the visibility of embedded triggers indicates the stealthiness of the backdoor attack. In this paper, we use the Structural Similarity Index Measure (SSIM) [29, 39] to evaluate the visibility of backdoor triggers. This is because, unlike traditional measurements like Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR), SSIM takes into account the local characteristics of samples, aligning more closely with the visual system of human eyes. Let SSIM ( x c , x p ) ∈ [ -1 , 1] be the SSIM between x c and x p , quantifying the visibility of trigger patterns embedded in x p . For ease of evaluation, we adopt the notation

v = (1 -SSIM ( x c , x p )) / 2 to normalize the visibility values. Specifically, the higher the value of v , the more notable the triggers are.

Robustness. For a given poisoned sample x p , we propose to use the term robustness to quantify its capability to resist perturbations or modifications imposed by random noise. Assume that the backdoored model classifies x p into a specific attack target category due to the existence of triggers. We apply random noises to x p and get a modified sample x m , where the modification process can be described by the equation x m = (1 -η r ) x p + η r ϵ ( ϵ ∼ N (0 , I ) ). Since η r plays an important role in determining whether its modified version x m belongs to another category, we use it to reflect the robustness of x p . Typically, the higher the value of η r , the more robust the poisoned sample x p is.

3.2 Observations from Attacks

To explore the correlation between the visibility of triggers and the robustness of their host poisoned samples, we conduct an experiment on the dataset CIFAR-10 against ten backdoor attacks, including both visible and invisible attacks, where each attack is applied on 200 randomly selected samples from CIFAR-10. Note that in the experiment, we assume that each poisoned sample is touched by only one kind of backdoor attack. In other words, each poisoned sample is embedded with a specific trigger. For each poisoned sample, we calculate the visibility score v of the trigger and the robustness η r of the poisoned sample. As shown in Figure 1, for different poisoned counterparts of a clean sample, their visibility and robustness scores differ significantly under different backdoor attacks.

Clean

0.045/0.075

SIG

0.197/0.432

Wanet

0.005/0.072

BadNets

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Visibility

v

0.0

0.1

0.2

0.3

0.4

0.5

Robustness

r

BadNets

SIG

LC

TrojanNN

Dynamic

Blended

LF

WaNet

ISSBA

BPP

High robustness

High visibility

Low visibility

Low robustness

Figure 2 illustrates the ( v , η r ) pairs for all the poisoned samples under ten attacks (detailed in Section 5.1), where each filled pentagram symbol represents a poisoned sample. From the figure, we can get the following two observations.

Observation 1. For backdoor triggers with low visibility ( v < 0 . 13 ), their host samples (i.e., ones with η r < 0 . 18 located in the red box) have low robustness. According to [38], the triggers within these poisoned samples can be easily destroyed by lightweight noises.

Observation 2. For poisoned samples (i.e., those located in the blue box) with high robustness ( η r ≥ 0 . 18 ), their backdoor triggers have high visibility, which renders them easily detectable and accurately located.

4 Our SampDetox Approach

4.1 Threat Model

We consider defending against backdoor attacks in black-box scenarios [17, 18], where defenders do not have access to the parameters of deep models and can only obtain predictions for samples. For attackers, we assume that they can completely control the training processes of models, enabling them to poison samples in datasets and modify model components to achieve poisoned models. In our approach, defenders strive to detoxify poisoned samples to classify them into their original correct categories. For clean samples, defenders aim to maintain their classification accuracy.

Diffusion

Model

Noise

Diffusion

Model

Stage 1:

Global Detoxification

Stage 2:

Local Detoxification

Triggers

Detoxified

samples

Poisoned

Samples

Clean

Sample

Sample Poisoning

Positioning

Noises

4.2 Overview of SampDetox

As observed in Section 3.2, there exists a strong correlation between the visibility of triggers and the robustness of poisoned samples, which can be used for backdoor defense. However, due to the lack of prior knowledge of clean samples, it is difficult to figure out both the trigger visibility and robustness of poisoned samples. To defend various backdoor attacks, Figure 3 presents a novel perturbation-based backdoor defense framework using our proposed sample detoxification method, which consists of two stages:

Stage 1: Global Detoxification. According to Observation 1, backdoor triggers with low visibility typically have low robustness poisoned samples. In this case, such triggers can be easily destroyed by adding lightweight noises to suspected samples. However, due to the low visibility or even invisibility, it is hard to determine the existence or location of triggers within a given sample. To destroy all these potential triggers, Stage 1 applies the noise-based perturbation (see Section 4.3) globally to each sample using lightweight noises. Note that the introduction of noises inevitably affects the inference accuracy of deep models. To counteract the impacts of added noises on inference performance, we then utilize DDPM-based denoising (see Section 4.4) to denoise the samples. In this way, Stage 1 can destroy possible triggers spread over the poisoned samples with low robustness.

Stage 2: Local Detoxification. According to Observation 2, poisoned samples with high robustness are often embedded with highly visibility triggers. Based on DDPM-based denoising in Stage 1, the clean regions of poisoned samples can easily be restored to their original states, while the poisoned regions cannot, especially when triggers are highly visible. Due to this fact, by comparing a potentially poisoned sample with its detoxified version by Stage 1 pixel by pixel, we can figure out the positions of obvious triggers and the required intensity of local noises. Then, Stage 2 applies the noise-based perturbation locally on each sample using pixel-specific noises to destroy such triggers. Similarly to Stage 1, the added noises in Stage 2 also greatly influence the final inference accuracy for both clean and detoxified poisoned samples, thus requiring diffusion models to perform the denoising. After Stage 2, both visible and invisible triggers can be safely eliminated without degrading the inference performance.

4.3 Noise-based Perturbation

To quantitatively control the intensity of noises applied on samples in each stage of SampDetox, we resort to the forward Denoising Diffusion Probabilistic Model (DDPM) [20], which can also be used to determine the number of denoising steps required in the DDPM-based denoising process. Assume that x 0 is an input sample, and x i is the sample after adding noises for i times (steps). In the forward process of DDPM, the model incrementally adds noises to x 0 step by step until the sample (denoted by x T ) is completely composed of random Gaussian noises at step T , where T is a diffusion model-specific constant. In each step t , x t is obtained by adding a random Gaussian noise ϵ ∼N ( 0 , I ) to x t -1 based on the formula x t = √ 1 -β t x t -1 + √ β t ϵ , where β t ∈ (0 , 1) . For a given x t -1 , the posterior probability of x t can be represented as q ( x t | x t -1 )= N ( x t ; √ 1 -β t x t -1 , β t I ) . Since step t only relies on step t -1 , the forward process can be regarded as a Markov process in the form of q ( x t | x 0 )= ∏ T q ( x t | x t -1 ) . For a given x 0 , we can get the probability of x t as t =1

where α t =1 -β t and α t = ∏ t i =1 (1 -β t ) . According to [20], we can get the exact relationship between x 0 and x t using the formula √ √

where z ∼N ( 0 , I ) is a random Gaussian noise.

Leveraging the forward DDPM, our approach aims to add proper random Gaussian noise on samples within t steps based on Equation 2, which can destroy backdoor triggers in samples. Therefore, Theorem 4.1 investigates the potential of DDPM to eliminate triggers within poisoned samples.

Theorem 4.1. Suppose that x t represents a diffused sample at step t of the forward process in DDPM, where t ∈ [0 , T ] . Given a clean sample x c 0 and its poisoned version x p 0 , we obtain x c t and x p t through the forward DDPM, whose distributions are P t ( x ) and Q t ( x ) , respectively. By using the Kullback-Leibler divergence [40, 41] to measure the distance between the two distributions, we have

where the equality happens only when P t = Q t ., and the inequality implies that the KL divergence between P t and Q t monotonically decreases along with the forward DDPM.

Please refer to Appendix B.1 for the proof of the theorem. According to Theorem 4.1, for any positive number ξ , there exists a minimum t ∈ [0 , T ] such that for any t ≥ t we have KL ( P t || Q t ) ≤ ξ . Note that this theorem forms the theoretical basis to utilize the forward DDPM to minimize the distribution discrepancy between poisoned and clean samples, thus mitigating the impact of backdoor triggers.

4.4 DDPM-based Denoising

We also utilize DDPM to perform denoising on samples within the two stages of SampDetox. For each pixel in a sample, the number of denoising steps is determined by its noise-based perturbation steps (i.e., t ) as introduced in Section 4.3. In reverse DDPM, by taking the current sample state x t as an input, the diffusion model can be used to achieve its previous state x t -1 . Given both x 0 and x t , according to Bayes' Theorem, we can obtain the probability distribution of x t -1 as p ( x t -1 | x t , x 0 )= p ( x t | x t -1 ) · p ( x t -1 | x 0 ) /p ( x t | x 0 ) . By resorting to the relationship between x 0 and x t described by Equation 2, according to [20], we can use x t to approximate x 0 as follows:

where z t is predicted by the pre-trained neural network θ of DDPM, denoting the estimation of the real noise in step t . Accordingly, we can obtain x t -1 from x t :

where σ 2 t = 1 -α t -1 1 -α t · β t and z ∼ N ( 0 , I ) .

Unlike existing DDPM-based methods that strive to restore samples from Gaussian noises, SampDetox uses diffusion models to denoise samples generated by our noise-based perturbation approach (see Section 4.3). Assuming that the training data used for the pre-trained diffusion model do not involve any knowledge about backdoor triggers, according to the principle of DDPM, the denoised results will restore the original samples with all the triggers cleansed. Theorem 4.2 investigates the difference between clean and poisoned regions of poisoned samples after denoising.

Theorem 4.2. Suppose that R ( x ) denotes a specific region of the sample x . Given a poisoned sample x p 0 , we use R c ( x p 0 ) and R p ( x p 0 ) to represent a clean region and a poisoned region of the sample, respectively. We perform the noise-based perturbation on x p 0 for t steps and get the noisy samples x p t , and then utilize DDPM to get a denoised sample ̂ x p 0 . When comparing different regions of the original sample x p 0 with their counterparts of the detoxified sample ̂ x p 0 , we have

where n 1 and n 2 represent the numbers of pixels in R p ( x p 0 ) and R c ( x p 0 ) , respectively.

Please refer to Appendix B.2 for the proof of the theorem. According to Theorem 4.2, we can calculate the similarity between the denoised and original sample to identify the positions of trigger patterns. The inequality in Theorem 4.2 implies that for a poisoned sample, its poisoned regions, compared with clean regions, will be more different from their counterparts in the restored version.

4.5 Implementation of SampDetox

Algorithm 1 details the implementation of SampDetox. Lines 1-6 describe the process of global detoxification, while Lines 8-21 present the local detoxification process. Specifically, Line 1 applies the noisebased perturbation to the sample following Equation 2 with a fixed noise intensity t 1 . Lines 2-6 show the process of DDPMbased denoising to obtain the denoised sample ̂ x 0 . In Line 7, SampDetox calculates the SSIM score for each pixel between the original sample x 0 and its denoised version ̂ x 0 . Lines 8-11 describe how a noise with a specific intensity (i.e., m [ i ] ) is added to the pixel i . Lines 12-21 present the DDPM-based denoising process, whether SampDetox iteratively calculates the estimation of the real noise z t with x t using the diffusion model θ and then selectively denoises each pixel accordingly. In Lines 16-19, SampDetox only updates pixels with noise intensity higher than that of the current step. Finally, Line 22 returns the resulting detoxified sample x 0 , which is the denoised sample by Stage 2.

5 Experiments

To evaluate the effectiveness of our approach, we implemented our approach, i.e., SampDetox, on top of Pytorch (version 1.13.0). We compared the defense performance between SampDetox and state-of-the-art (SOTA) black-box defense methods against various well-known backdoor attacks. All experiments were carried out on an Ubuntu workstation equipped with one Intel i7-13700K CPU, 64GB memory, and one NVIDIA GeForce RTX4090 GPU.

5.1 Experimental Setup

Dataset and Model Settings. We investigated three classical datasets (i.e., CIFAR-10 [42], GTSRB [43], and Tiny-ImageNet [44]) and three models (i.e., PreAct-ResNet18, ResNet34 [45] and VGG-19 [46]). Due to space limitations, this section only presents the experimental results of PreActResNet18 on the CIFAR-10 dataset. Note that we can find similar trends from the experiments using different models and datasets. Please refer to Appendix C for more details.

Evaluation Metrics. To objectively evaluate the effectiveness of a given backdoor defense method, we adopted the following three metrics: i) Clean sample Accuracy (CA), which denotes the inference accuracy of clean samples processed by the defense method; ii) Poisoned sample Accuracy (PA), which indicates the inference accuracy of poisoned samples purified (detoxified) by the defense method based on their ground-truth labels; and iii) Attack Success Rate (ASR), which represents the rate of successful attacks by poisoned samples. Note that a higher CA means that the defense method causes less interference in the classification of clean samples. Typically, backdoor defenders strive to achieve both high CA and PA with lowered ASR.

Attack Methods. We conducted defenses against ten SOTA backdoor attacks, i.e., BadNets [22], SIG [23], Label Consistent (LC) [24], TrojanNN [25], Dynamic [26], Blended [27], Low Frequency (LF) [47], WaNet [28], ISSBA [29], and BPP [30]. Note that the first five are visible attacks, while the last five are invisible attacks. We used the same benchmark settings in [48] to configure all these attack methods. Please refer to Appendix A.1 for more details.

Algorithm 1 Implementation of SampDetox

Clean

BadNets

Blended

BPP

SIG

WaNet

LC

TrojanNN

Dynamic

LF

ISSBA

Triggers

Detoxified

Samples

Defense Baselines. We compared SampDetox with three SOTA sample purification-based backdoor defenses: i) Sancdifi [37], which utilizes the RISE algorithm to measure the saliency maps and purify the specific areas of samples; ii) BDMAE [19], which uses a masked AutoEncoder to defend against backdoor attacks; and iii) ZIP [18], which applies linear transformations to samples and restores them under noises by using diffusion models. Note that both Sancdifi and BDMAE need to feed samples into deep models repeatedly to obtain their predictions, while ZIP does not rely on predictions. Please refer to Appendix A.2 for more details.

5.2 Performance Comparison

Figure 4 shows an example of backdoor defense against the ten different attacks using our proposed SampDetox. Here, the first row presents both a clean sample and all its poisoned counterparts. The second row presents the triggers identified by stage 2 of SampDetox, while the third row exhibits the detoxified samples of each attack. From this figure, we can find that SampDetox can effectively figure out all visible and invisible triggers embedded by different SOTA backdoor attacks.

Table 1 compares the defense performance between SampDetox and the three baselines on the CIFAR10 dataset using PreAct-ResNet18. Please see Appendix C.1 for the results of datasets GTSRB and Tiny-ImageNet. From this table, we can find that SampDetox can achieve the highest CA for all different attacks, indicating that the impact of SampDetox on clean samples is negligible. Meanwhile, SampDetox has the best PA and ASR for 8 out of 10 attacks, respectively. Even for all the remaining cases (i.e., BadNets, LC, and TrojanNN), our approach can achieve the second-best PA and ASR, whose values are quite close to the ones of the best PA and ASR. As an example of the BadNets attack, Sancdifi has the best ASR, which is only 0 . 19% lower than our approach. However, in this case, our approach significantly outperforms Sancdifi in CA by 12 . 98% . For the TrojanNN attack, though Sancdifi can achieve slightly better PA (by 0 . 89% ) and ASR (by 0 . 05% ) than SampDetox, SampDetox notably outperforms Sancdifi in CA by 16 . 15% . For the Label Consistent (LC) attack, ZIP can achieve slightly better PA, but its CA and ASR are worse than SampDetox. It is important to note that Sancdifi and BDMAE require feedback from deep models. As an alternative, our approach does not rely on underlying models, showing the applicability and superiority of our approach.

5.3 Ablation Studies

Impacts of Different Stages and Denoising. The implementation of SampDetox consists of two stages. The first stage strives to destroy triggers globally on a given sample in a coarse manner, while the second stage tries to identify fine triggers and remove them locally. To examine the impacts of these two stages, we consider a variant (i.e., 'Stage 1') of our approach, which does not take stage 2 of SampDetox into account. Meanwhile, to evaluate the impact of the denoising operations, we considered another variant of SampDetox (i.e., 'Noise*'), where denoising is not applied to restore the samples in both stages. Table 2 presents the ablation study results against the ten attacks. Note that to enable an intuitive understanding of the role of each stage in SampDetox, this table provides the average visibility of backdoor triggers for each attack.

For all the defense methods in Table 2, we can find that the smaller the trigger visibility caused by backdoor attacks, the lower PA and higher ASR we can achieve. Although 'Noise*' can achieve comparable ASR results to SampDetox, SampDetox outperforms 'Noise*' in both CA and PA significantly. It means that simply adding noises to samples can effectively destroy the backdoor triggers of the samples. However, the noises themselves will also result in a sharp decline in the inference accuracy of clean and poisoned samples. If we merely adopt the first stage of SampDetox due to the denoising operation by diffusion models, both CA and PA can be improved. However, since the first stage only coarsely applies the perturbation on samples, most subtle backdoor triggers still survive. As an example in SIG attack, 'Stage 1' can only achieve a PA of 9 . 48% but with an ASR of 92 . 33% . By combining both Stages 1 and 2, SampDetox can effectively identify such undetectable triggers and destroy them without affecting the inference of original samples, leading to a better PA and ASR. For example, in SIG attack, by using SampDetox, we can achieve a PA of 65 . 06% and an ASR of 11 . 03% .

Impact of Hyperparameters t 1 and t 2 . In our approach, we use two hyperparameters, i.e., t 1 and t 2 to control the noise intensity and denoising samples, respectively. We investigated two variants of SampDetox to evaluate the impacts of t 1 and t 2 . For the first variant, we fixed the value of t 2 (i.e., t 2 =0) while allowing the tuning of t 1 . With different values of t 1 , we can evaluate the defense performance of SampDetox against various invisible backdoor attacks (i.e., Blended, Low Frequency,

.

WaNet, ISSBA, and BPP). Similarly, for the second variant, we fixed the value of t 1 (i.e., t 1 =20) while changing the values of t 2 , to evaluate the defense performance of SampDetox against visible backdoor attacks (i.e., BadNets, SIG, LC, TrojanNN, and Dynamic). Table 3 shows the ablation study results with varying values of t 1 or t 2 . Note that the results in the table represent the average results across multiple attack types. From this table, we can observe that when t 1 and t 2 increase, the ASR decreases, indicating the effect of adding noises in destroying backdoor triggers. However, a high value of t 1 and t 2 will also lead to a significant reduction of CA and PA, showing that the denoising processes are strongly affected by excessive noises. Therefore, we suggest to set t 1 = 20 and t 2 = 120 in practice.

5.4 Discussion

To show the applicability of SampDetox, we investigated the extra time overheads, explored training sample detoxification, evaluated the defense performance of SampDetox against SOTA adaptive attacks, and analyzed its limitations. Due to the space limitation, this section only presents the overhead caused by inference detoxification. Please refer to Appendix D for more discussions.

Extra time overhead. Based on our proposed two-stage perturbation-based sample detoxification, SampDetox can protect any deep models during their inference phase. However, similar to other backdoor defense approaches, the detoxification process will inevitably lead to extra time overhead. To address this issue, SampDetox resorts to the Denoising Diffusion Implicit Model (DDIM) [21] to accelerate the denoising process. Figure 5 compares the average inference time per sample for different back-

0.0435

0.4516

0.4735

13.917

0.3526

0.4953

0.0651

0

0.2

0.4

0.6

13.8

14

No Defense

Sancdifi

BDMAE

ZIP

ZIP+DDIM

Ours

Ours+DDIM

Time (Seconds)

Defense Methods

door defense methods. We can find that with the help of DDIM, the inference time of SampDetox is comparable to that of the method without defense.

6 Conclusion

Based on our observed correlation between the visibility of backdoor triggers and the robustness of poisoned samples, this paper introduces a novel black-box backdoor defense approach named SampDetox. By using our proposed perturbation-based sample detoxification, SampDetox can easily destroy backdoor triggers in samples and restore the samples using diffusion models. Extensive experiments against various complex backdoor attacks show the superiority of SampDetox from both perspectives of effectiveness and applicability to arbitrary scenarios.

References

Hongyi Zhang, Jan Bosch, and Helena Holmstr¨ om Olsson. End-to-end federated learning for autonomous driving vehicles. In Proceedings of International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2021.

Xiaofeng Hou, Tongqiao Xu, Chao Li, Cheng Xu, Jiacheng Liu, Yang Hu, Jieru Zhao, Jingwen Leng, Kwang-Ting Cheng, and Minyi Guo. A tale of two domains: Exploring efficient architecture design for truly autonomous things. In Proceedings of International Symposium on Computer Architecture (ISCA) , 2024.

Ming Hu, E. Cao, Hongbing Huang, Min Zhang, Xiaohong Chen, and Mingsong Chen. Aiotml: Aunified modeling language for aiot-based cyber-physical systems. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. , 42(11):3545-3558, 2023.

Chentao Jia, Ming Hu, Zekai Chen, Yanxin Yang, Xiaofei Xie, Yang Liu, and Mingsong Chen. Adaptivefl: Adaptive heterogeneous federated learning for resource-constrained aiot systems. In Proceedings of The Chips To Systems Conference (DAC) , pages 1-6, 2024.

Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib. Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps: Automation of decision making , pages 323-350, 2018.

Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics , 19(6):1236-1246, 2018.

Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. arXiv preprint arXiv:2302.03251 , 2023.

Bao Gia Doan, Ehsan Abbasnejad, and Damith C Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In Proceedings of Annual Computer Security Applications Conference (ACSAC) , pages 897-912, 2020.

Wenke Huang, Mang Ye, Zekun Shi, Bo Du, and Dacheng Tao. Fisher calibration for backdoorrobust heterogeneous federated learning. In Proceedings of European Conference on Computer Vision (ECCV) , 2024.

Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y Zhao. Backdoor attacks against deep learning systems in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6206-6215, 2021.

Yanxin Yang, Ming Hu, Yue Cao, Jun Xia, Yihao Huang, Yang Liu, and Mingsong Chen. Protect federated learning against backdoor attacks via data-free trigger generation. arXiv preprint arXiv:2308.11333 , 2023.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Proceedings of IEEE Symposium on Security and Privacy (S&P) , pages 707-723, 2019.

Zhihao Yue, Jun Xia, Zhiwei Ling, Ming Hu, Ting Wang, Xian Wei, and Mingsong Chen. Modelcontrastive learning for backdoor elimination. In Proceedings of the 31st ACM International Conference on Multimedia (MM) , pages 8869-8880, 2023.

Jun Xia, Ting Wang, Jiepin Ding, Xian Wei, and Mingsong Chen. Eliminating backdoor triggers for deep neural networks using attention relation graph distillation. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI) , 2022.

Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang, Shiqing Ma, and Xiangyu Zhang. Complex backdoor detection by symmetric feature differencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15003-15013, 2022.

Hanxun Huang, Xingjun Ma, Sarah Erfani, and James Bailey. Distilling cognitive backdoor patterns within an image. arXiv preprint arXiv:2301.10908 , 2023.

Yucheng Shi, Mengnan Du, Xuansheng Wu, Zihan Guan, and Ninghao Liu. Black-box backdoor defense via zero-shot image purification. arXiv preprint arXiv:2303.12175 , 2023.

Tao Sun, Lu Pang, Chao Chen, and Haibin Ling. Mask and restore: Blind backdoor defense at test time with masked autoencoder. arXiv preprint arXiv:2303.15564 , 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NIPS) , 33:6840-6851, 2020.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 , 2017.

Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In Proceedings of IEEE International Conference on Image Processing (ICIP) , pages 101-105, 2019.

Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771 , 2019.

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Proceedings of Annual Network and Distributed System Security Symposium (NDSS) , 2018.

Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems (NIPS) , 33:3454-3464, 2020.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 , 2017.

Anh Nguyen and Anh Tran. Wanet-imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369 , 2021.

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 16463-16472, 2021.

Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15074-15084, 2022.

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In Proceedings of International Conference on Learning Representations (ICLR) , 2022.

Maurice Weber, Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li. RAB: provable robustness against backdoor attacks. In Proceedings of IEEE Symposium on Security and Privacy (S&P) , pages 1311-1328, 2023.

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems (NIPS) , 34:14900-14912, 2021.

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Proceedings of International Symposium on Research in Attacks, Intrusions, and Defenses (RAID) , pages 273-294, 2018.

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems(NIPS) , 34:16913-16925, 2021.

Zhen Xiang, Zidi Xiong, and Bo Li. CBD: A certified backdoor detector based on local dominant probability. In Advances in Neural Information Processing Systems(NIPS) , 2023.

Brandon B May, N Joseph Tatro, Piyush Kumar, and Nathan Shnidman. Salient conditional diffusion for defending against backdoor attacks. arXiv preprint arXiv:2301.13862 , 2023.

Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.04692 , 2020.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Solomon Kullback and Richard A Leibler. On information and sufficiency. The Annals of Mathematical Statistics , 22(1):79-86, 1951.

Tim Van Erven and Peter Harremos. R´ enyi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory , 60(7):3797-3820, 2014.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. In Technical report , 2009.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks , 32:323-332, 2012.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115:211-252, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Proceedings of European Conference on Computer Vision (ECCV) , pages 630-645, 2016.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 16473-16481, 2021.

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. Advances in Neural Information Processing Systems (NIPS) , 35:10546-10559, 2022.

Yiming Li, Tongqing Zhai, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the physical world. arXiv preprint arXiv:2104.02361 , 2021.

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Proceedings of European Conference on Computer Vision (ECCV) , pages 87-102, 2016.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3606-3613, 2014.

Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y. Zhao. Backdoor attacks against deep learning systems in the physical world. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 6206-6215, 2021.

Ruotong Wang, Hongrui Chen, Zihao Zhu, Li Liu, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Robust backdoor attack with visible, semantic, sample-specific, and compatible triggers. arXiv preprint arXiv:2306.00816 , 2023.

Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In Proceedings of European Conference on Computer Vision (ECCV) , pages 182-199, 2020.

Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI)) , pages 1148-1156, 2021.

Junyu Lin, Lei Xu, Yingqi Liu, and Xiangyu Zhang. Composite backdoor attack for deep neural network by mixing existing benign features. In Proceedings of Conference on Computer and Communications Security (CCS) , pages 113-131, 2020.

A Detailed Experimental Setup

A.1 Attack Method Settings

BadNets [22]. BadNets is the first proposed backdoor attack method, which patches a fixed pattern on the corners of benign samples and modifies their labels to the attack target category. We use a 3 × 3 black and white checkerboard in the lower right corner of samples as the trigger pattern.

SIG [23]. SIG modifies the pixels in samples based on the sinusoidal signal to generate poisoned samples. We set the frequency of the sinusoidal signal to 3 as the trigger pattern.

Label Consistent (LC) [24]. LC generates the poisoned samples using adversarial attack methods and keeps the original sample label unchanged. The trigger pattern is identical to BadNets, and other settings follow the original paper.

TrojanNN [25]. TrojanNN injects backdoors into a pre-trained model instead of injecting backdoors during the training. We use a white apple image as the trigger pattern.

Dynamic [26]. Dynamic trains a trigger generator to generate the specific triggers for each sample. The backdoor attack will only succeed if the specific trigger matches its sample. We follow the settings in the original paper to generate triggers.

Blended [27]. Blended generates poisoned samples by blending benign samples with a fixed pattern. Blending with coefficient α , Blended increases the invisibility of the trigger to avoid manual inspection. We use a 'hello kitty' image as the trigger pattern.

Low Frequency (LF) [47]. LF proposes an optimization-based approach to filter high-frequency artifacts to generate smooth triggers. We follow the settings in the original paper to generate triggers.

WaNet [28]. WaNet uses fixed warping functions to distort benign samples slightly to construct the poisoned samples. We follow the settings in the original paper to generate poisoned samples with trigger patterns.

ISSBA [29]. ISSBA employs image steganography to embed the string trigger into benign samples, which generates unique triggers that are specific to each sample. Following the settings in the original paper, we use the trained image steganography method.

BPP [30]. BPP generates the poisoned samples using image quantization. To inject the backdoor more efficiently, BPP proposes a contrastive learning-based approach to improve the success rate of backdoor attacks. We set the negative ratio to 0.1.

A.2 Defense Baseline Settings

Sancdifi [37]. Sancdifi followed the settings in Section 'Numerical Experiment' of its paper. Sancdifi computed RISE maps using 2000 random binary masks and set the saliency threshold to 0.95. For a fair comparison, Sancdifi used the same diffusion models as ours.

BDMAE [19]. BDMAE followed the settings in Section 'Method Configurations' of its paper, which used masked autoencoders pre-trained on ImageNet with 24 encoder layers.

ZIP [18]. ZIP followed the settings in Section 'Purification Implementation' of its paper, which set its hyperparameter to 2. For a fair comparison, ZIP used the same diffusion models as ours.

A.3 Diffusion Model Settings

We utilize the framework of the improved diffusion model provided by [21]. The number of total diffusion steps was set to 1000 for all the datasets in the experiments, the noise schedule was set as 'cosine', and the learning rate was set to 1e-4. In the experiments, we load the parameters of the pre-trained model for each dataset. For DDIM, compared with the total steps of the diffusion model (e.g., T = 1000 ), our approach does not require too many steps to add noise and denoise samples (e.g., t 1 = 20 and t 2 = 120 ). Therefore, when we use DDIM to accelerate the denoising, we set the speed-up pace to 20 to ensure the effectiveness of our approach.

A.4 PixelSSIM Settings

In our work, the PixelSSIM denotes an intermediate data structure for calculating the SSIM [39] value of two given samples. Specifically, for each block pair in the two samples, we calculate its SSIM value and record it in PixelSSIM at the same position as one of the central pixels of the block pair. When sliding the block pair along the two dimensions of the samples, we can collect all the SSIM values for the block pairs and form the final PixelSSIM. Note that by averaging all the values in PixelSSIM, we can achieve the SSIM value for the two samples. We use structural similarity () in the skimage library of Python to implement the PixelSSIM.

B Theoretical Justification

B.1 Proof of Theorem 4.1

Theorem 4.1. Suppose that x t represents a diffused sample at step t of the forward process in DDPM, where t ∈ [0 , T ] . Given a clean sample x c 0 and its poisoned version x p 0 , we obtain x c t and x p t through the forward DDPM, whose distributions are P t ( x ) and Q t ( x ) , respectively. By using the Kullback-Leibler divergence [40, 41] to measure the distance between the two distributions, we have

where the equality happens only when P t = Q t , and the inequality implies that the KL divergence between P t and Q t monotonically decreases along with the forward DDPM.

Proof: First, following the definition of the diffused sample probability distribution in DDPM [20], like Equation 1, when given a clean sample x c 0 and a poisoned sample x p 0 , we have their distributions:

where α t = 1 -β t and α t = ∏ t i =1 α i . Therefore, we know that both P ( x c t | x c 0 ) and Q ( x p t | x p 0 ) are Gaussian distributions. When both terms of the KL divergence are Gaussian distributions, we can simplify the formula as follows:

Since P t and Q t have the same σ , i.e., σ = σ 1 = σ 2 = 1 -α t I , we can continue to simplify the formula:

Finally, we take the difference of both sides of the inequality

Since α t = 1 -β t and β t ∈ (0 , 1) , α t should be less than 1 . As a result, we can get

According to Equation 12, we have

where the equality happens only when P t = Q t .

B.2 Proof of Theorem 4.2

Theorem 4.2. Suppose that R ( x ) denotes a specific region of the sample x . Given a poisoned sample x p 0 , we use R c ( x p 0 ) and R c ( x p 0 ) to represent a clean region and a poisoned region of the sample, respectively. We perform the noise-based perturbation on x p 0 for t steps and get the noisy samples x p t , and then utilize DDPM to get a denoised sample ̂ x p 0 . When comparing different regions of the original sample x p 0 with their counterparts of the detoxified sample ̂ x p 0 , we have

where n 1 and n 2 represent the numbers of pixels in R p ( x p 0 ) and R c ( x p 0 ) , respectively.

Proof: First, we expand the inequality that we need to prove. Therefore, proving Equation 14 is equivalent to proving the following inequality:

According to Equation 5, we have

where σ 2 t = 1 -α t -1 1 -α t · β t and z ∼ N (0 , I ) . θ is a pre-trained neural network. Assuming that θ is a fully trained model that can generate possible noise based on the input sample x t and the step t . Therefore, we have

where z t is the real noise in step t .

Then, we analyze the pixels in the clean and poisoned regions, respectively. We denote a pixel in the clean region of the original sample as x 0 , and a pixel in the poisoned region as y 0 . Therefore, we have x t and y t to denote the pixels after our noise-based perturbation. And we have ̂ x 0 and ̂ y 0 to denote the pixels after our DDPM-based denoising process. Since θ has a strong ability, we can approximate the entire denoising process with the following:

According to Equation 2 about the relationship between x 0 and x t , we have

where z t is the real noise and z = [ z ] = [ θ ( x t , t )] . According to Equation 17, we have

Supposing that the training data used for the pre-trained diffusion model does not contain any knowledge about backdoor triggers, it is reasonable to assume that x 0 belongs to the probability distribution of the training dataset of θ , while y 0 does not. Therefore, we have

According to Theorem 4.1, when the sample is fully noised, the sample is equivalent to Gaussian noise. Additionally, the sample generated by the unguided denoising process from Gaussian noise is completely independent of the original sample. Therefore, we have the equation about x and y as follows:

According to Equation 21 and Equation 22, we finally get the following equation:

where t ∈ (0 , T ) . Therefore, Equation 15 can be proved. Finally, we have

C Additional Experimental Results

C.1 Performance Comparison on More Datasets

Table 4 and Table 5 show the defense performance of our approach and three defense baselines, i.e., Sancdifi [37], BDMAE [19] and ZIP [18], against ten state-of-the-art (SOTA) attack methods on the GTSRB and Tiny-ImageNet dataset, respectively. The experimental results in the table illustrate that our approach outperforms existing SOTA methods on a variety of datasets, which shows the generalizability of our approach.

C.2 Performance on More Models.

Since our approach does not require any interaction with the model, it should have defensive effects on different models. Table 6 and Table 7 show the defense performance of our approach on the VGG-19 and ResNet34 models on CIFAR-10, GTSRB, and Tiny-ImageNet datasets, respectively. The experimental results show that our approach has satisfactory defense effects on VGG-19 and ResNet34, which demonstrates that our approach is model-independent.

C.3 Comparison with Sample Transformation Methods

In this section, we compare our approach with existing sample transformation methods on CIFAR-10, including Noising, ShrinkPad, and Blur [49]. Noising attempts to destroy the backdoor trigger structure by adding a large amount of noise to the sample without considering the inference accuracy. ShrinkPad shrinks the samples and then randomly pads samples back to their original size. Blur utilizes Gaussian kernel to pool the samples to destroy backdoor trigger patterns. The experimental results are shown in Table 8.

From the table, we can find that transforming the samples can break some of the backdoor triggers. However, the transformation will cause great damage to the quality of the samples, resulting in a sig-

nificant reduction in Clean sample Accuracy (CA) and Poisoned sample Accuracy (PA). Specifically, though Blur has lower ASR than SampDetox when defending against the SIG and LF attacks, our approach has better CA and PA. In addition, our methods have the best defensive performance. The experimental results show that our approach can successfully defend against various attacks while preserving the original semantic information of samples to the greatest extent.

C.4 Performance using Different Diffusion Models

In this section, we deploy our approach using different pre-trained diffusion models to defend against backdoor attacks on CIFAR-10 and compare their defense performance. Specifically, we choose three diffusion models with different parameters: 1) pre-trained on CIFAR-10, which is the same setup as our main experiments, 2) pre-trained on ImageNet, and 3) pre-trained on ImageNet and fine-turned on CIFAR-10. Notably, we used 5% clean samples in the test dataset of CIFAR-10 to fine-tune the pre-trained model, and we excluded these data during the experiments. The experimental results are shown in Table 9.

From the table, we can observe that SampDetox with different diffusion models has satisfied defensive performance on Clean sample Accuracy (CA) and Attack Success Rate (ASR). For CA, since clean samples do not have any trigger patterns, only a little noise will be added to them in Stage 2 of our approach. Therefore, all these three diffusion models can accurately denoise samples. For ASR, the results are generally not related to the denoising process. For Poisoned sample Accuracy (PA), when defending against visible attacks (i.e., BadNets, SIG, LC, TrojanNN, and Dynamic), the model pre-trained on ImageNet has much lower accuracy, meaning that the model pre-trained on outside distribution datasets can not accurately denoise samples, especially when much noise is added. The fine-tuning model achieves satisfactory defensive performance against different attacks. Therefore, when we need to detoxify samples without pre-trained models, we can fine-tune a diffusion model to deploy our defense approach.

Additionally, considering the case where pre-training datasets for diffusion models are out-ofdistribution from their target classification tasks, we conducted experiments to evaluate the performance of SampDetox on a subset of the MS-Celeb-1M [50] dataset. Specifically, we considered the top 100 labels with the largest number of samples and randomly selected 380 samples for each label. We split the training and tests with a ratio of 8 : 2 and adjusted the shape of the samples to

224 ∗ 224 . We adopted BadNets [22] and WaNet [28] as attack methods, representing a visible and an invisible backdoor attack, respectively. We considered four pre-trained diffusion models: i) a pre-trained model (i.e., Model 1) on the subset of MS-Celeb-1M, ii) a pre-trained model (i.e., Model 2) on another subset of MS-Celeb-1M, iii) a pre-trained model (i.e., Model 3) on ImageNet, and iv) Model 4, which is fine-tuned from Model 3 using a subset of MS-Celeb-1M.

Table 10 shows the experimental results. We can find that SampDetox with different diffusion models achieves satisfied defensive performance on ASR. Note that, for Model 3, the CA and PA of SampDetox decrease. This is because Model 3 is pre-trained on ImageNet rather than MS-Celeb-1M, resulting in the diffusion model failing to effectively denoise MS-Celeb-1M samples. However, we can find significant improvements in the CA and PA of Model 4 since Model 4 is fine-tuned using a subset of MS-Celeb-1M. In other words, SampDetox can defend against attacks using out-of-distribution diffusion models with proper fine-tuning.

D Additional Discussions

D.1 Performance on High-frequency Classification Task

To evaluate the defense performance of SampDetox on classification tasks that rely on benign highfrequency details, we conducted experiments on the DTD (Describable Textures Dataset) dataset [51], where we adopted a visible backdoor attack, i.e., BadNets [22], and an invisible backdoor attack, i.e., WaNet [28], respectively. Table 11 compares SampDetox with six baselines. From this table, we can find that though the CA of SampDetox is a little smaller than the CA of 'No Defense', SampDetox can achieve the best defense performance on CA, PA, and ASR compared with all baselines, which shows the applicability and superiority of SampDetox.

D.2 Performance using DDIM

Since DDIM [21] was proposed to accelerate the speed of DDPM without affecting the image generation quality, the defense performance of 'SampDetox+DDIM' is comparable to that of 'SampDetox+DDPM'. To demonstrate the performance of 'SampDetox+DDIM', we conducted new experiments on CIFAR-10 against ten SOTA backdoor attacks. The experimental results are shown in Table 12. From the table, we can find that the defense performance of 'SampDetox+DDIM' is almost the same as that of 'SampDetox+DDPM', which means that 'SampDetox+DDIM' can match the defense efficacy of 'SampDetox+DDPM'.

D.3 Training Sample Detoxification

Before training deep models, we can use SampDetox to conduct perturbation-based training sample detoxification, protecting them against backdoor injection. Table 13 compares the defense performance of the trained models by SampDetox. Here, the second column shows the defense performance without using SampDetox. The third and fourth columns present the defense performance of models trained by detoxified samples. Note that for the third column, we did not change the labels of detoxified samples. In other words, the poisoned samples still keep their dirty labels (denoted by D.L. ). For the fourth column, we tried to correct the dirty labels based on the inference results of the models generated by the third column. Based on the detoxified samples and corrected clean labels (denoted by C.L. ), the fourth column shows the defense performance of our approach. Note that for this table, we did not apply sample detoxification on test samples. From this table, we can observe that SampDetox can effectively defend against various backdoor attacks. Although 'SampDetox+D.L.' and 'SampDetox+C.L.' have similar ASR, the CA of 'SampDetox+C.L.' can be notably improved based on the corrected labels of poisoned samples.

D.4 Performance against Adaptive Attacks

Assume that attackers have all the prior knowledge of our proposed approach to design an adaptive attack to bypass our defense approach. Since SampDetox is based on observations about the correlation between the visibility of triggers and the robustness of poisoned samples, the attackers can try two types of adaptive attacks as follows.

1) Adaptive attack 1. The attacker uses an attack with both low visibility and high robustness since attacks with high visibility and low robustness are easily defended. To guarantee low visibility, the attacker adopts an invisible backdoor attack, i.e., WaNet. To ensure high robustness, the attacker adaptively adds noises to samples during the training phase to improve the robustness of triggers.

2) Adaptive attack 2. The attacker does not consider visibility issues and tries to maximize robustness to bypass our defense. In this case, the attacker adopts a visible backdoor attack, i.e., BadNets, and adds robust noise during the training phase to improve robustness.

We conducted experiments to defend against the two adaptive attacks on the CIFAR-10 dataset. Table 14 shows the results of the experiments, where the first column denotes the amount of noise added during training to improve the robustness. From the table, we find that our approach can defend against adaptive attacks.

D.5 Limitation

Although our approach aims to defend against backdoor attacks with any kind of trigger, there are still certain triggers that SampDetox cannot remove. Physical attack [52, 53] is a backdoor attack with visible but natural triggers, usually physical objects. Since our approach identifies the positions of trigger patterns based on Theorem 4.2 by calculating the similarity between the denoised and original samples, the effectiveness of our approach depends on the pre-trained diffusion model. When the triggers are visible, are part of the sample classification characteristics, and belong to the knowledge contained in the pre-trained diffusion model, our defense approach cannot destroy the trigger patterns and defend against these attacks.

To investigate the limitation of SampDetox, we conducted experiments with three semantic backdoor attacks, i.e., Refool [54], DSFT [55] and CBA [56]. Table 15 presents the defense performance of SampDetox against Refool and DSFT. From the table, we can find that SampDetox can effectively defend against such backdoor attacks. This is because the semantic features adopted by Refool and DSFT are neither part of the original classified features nor are their distributions the same as the distribution of the training dataset of the pre-trained diffusion model. Therefore, the triggers based on such semantic features will not be restored by the denoising phase of SampDetox.

Additionally, to explain why SampDetox can defend against these two attacks, we conducted experiments to calculate the robustness of their poisoned samples, respectively. For each attack, we randomly selected 50 poisoned samples on CIFAR-10 and calculated the robustness value of each sample following our robustness definition in Section 3.1. For Refool, the robustness of its poisoned samples has a mean value of 0 . 0759 and a standard deviation of 0 . 011 . For DFST, the robustness of its poisoned samples has a mean value of 0 . 137 and a standard deviation of 0 . 019 . From the experimental results, we can find that the robustness of the poisoned samples is low (ones with robustness < 0 . 18 , according to Figure 2). Therefore, when SampDetox adds enough noise to these samples, even if trigger patterns are partially restored by diffusion models, backdoors will not be successfully triggered.

However, SampDetox fails to defend against CBA. Since CBA concatenates two natural images as backdoor triggers, it is considered the backdoor attack type that SampDetox fails to defend. The experimental results on CIFAR-10 are as follows: Without defense, CA, PA, and ASR were 87 . 52% , 10 . 16% , and 90 . 17% , respectively. However, when using SampDetox, CA, PA, and ASR were 86 . 81% , 38 . 45% , and 37 . 28% , respectively. The experimental results show that SampDetox has a certain effect in defending against CBA but fails to minimize ASR.

E More Examples of Poisoned Samples

Figure 6 shows examples of the poisoned samples with their visibility v and robustness η r .

0.102/0.182

0.006/0.063

0.010/0.069

0.007/0.053

0.023/0.071

0.089/0.096

0.108/0.105

0.187/0.418

0.043/0.071

0.013/0.058

0.057/0.067

0.098/0.081

0.006/0.061

0.131/0.233

0.010/0.062

0.191/0.398

0.149/0.406

0.053/0.083

0.006/0.045

0.047/0.068

0.044/0.073

0.096/0.071

0.174/0.326

0.112/0.121

0.112/0.113

0.047/0.078

0.169/0.430

0.015/0.048

0.167/0.364

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations in Section D.5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides the proofs of theoretical results in Section B.1 and Section B.2, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides code in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides the experimental settings in Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not involve error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides the information about compute resources in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses the potential societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper uses popular datasets and models. We cites the original paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of backdoor threats in third-party Machine Learning as a Service (MLaaS) applications. The authors propose a novel two-stage method called SampDetox to defend against both visible and invisible backdoor attacks without requiring access to the model's parameters or training data. The proposed method uses diffusion models to introduce noise into samples to destroy backdoors and recover clean samples. It combines lightweight and high noise strategically to handle backdoors with varying visibility and robustness. The paper provides mathematical proofs to support the effectiveness of SampDetox in eliminating backdoor triggers and restoring original samples. Comprehensive experiments demonstrate SampDetox's effectiveness against various state-of-the-art backdoor attacks. The method is evaluated using different datasets and models, showing its superiority over existing defense methods in terms of clean sample accuracy, poisoned sample accuracy, and attack success rate.\n\nSTRENGTHS:\nThe strengths can be summarized as follows:\n\n1. **Innovative Defense Mechanism**: SampDetox introduces a novel two-stage approach to address backdoor attacks in a black-box context, which is a significant contribution to the field of machine learning security.\n\n2. **Theoretical Foundation**: The paper provides a solid theoretical basis with mathematical proofs that support the effectiveness of the proposed method in eliminating backdoor triggers.\n\n3. **Comprehensive Evaluation**: The method is extensively tested against state-of-the-art backdoor attacks using multiple datasets and models, demonstrating its generalizability and reliability.\n\n4. **Thorough Analysis**: The paper includes ablation studies that dissect the impact of different stages and parameters of the SampDetox method, providing insights into its workings and optimization.\n\nWEAKNESSES:\n**Summary of Key Weaknesses:**\n\n1. **Dependency on Pre-trained Models**:\n   - *SampDetox's* effectiveness is contingent upon the quality of pre-trained diffusion models utilized for the denoising process. This poses a limitation, particularly when these models are not finely tuned for the specific data in question. While the authors have successfully demonstrated *SampDetox* using an ImageNet-trained model to counter attacks on CIFAR10, the real-world applicability may be limited due to the difficulty of accessing a pre-trained model that matches the unique dataset of the application. \n\n2. **Trigger Recovery Assumption**:\n   - The paper presupposes that clean regions in poisoned samples can be easily restored, whereas poisoned regions, especially with visible triggers, cannot (as stated on Line 201). This assumption may not hold against sophisticated attacks like physical attacks where triggers are inherent and visible parts of the sample. Furthermore, this assumption could be invalidated if the image features an object that the pre-trained diffusion model is incapable of accurately recovering.\n\n3. **Omission of Experimental Details**:\n   - The paper falls short in providing certain critical experimental details, particularly regarding the baseline defense methods (such as the diffusion model configuration for ZIP) and the proposed method itself ($\\bar{t}_1$ and $\\bar{t}_2$ for different dataset).\n\nQUESTIONS:\nThe Questions can be summarized as follows:\n\n1. **Question on Figure 6**: The authors compare the additional time overhead for various methods in Figure 6, noting that ZIP's overhead exceeds SampDetox's by more than 20 times. Given that ZIP utilizes DDPM once for image purification, whereas SampDetox employs it twice, this discrepancy seems disproportionate, especially if both are configured for the same denoising step count.  If ZIP and SampDetox use different configurations of DDPM, such comparison in overhead and defense performance may not be fair in some sense, as the configuration of DDPM plays an essential role in the recovery performance. Clarification on this point is requested.\n\n2. **Lack results for SampDetox with DDIM**: In section 5.4, the authors assert that integrating DDIM with SampDetox yields inference times comparable to unguarded methods. However, specific performance metrics for SampDetox, when augmented with DDIM, are absent. To substantiate this claim, it is imperative to demonstrate that SampDetox+DDIM can match the defense efficacy of SampDetox+DDPM.\n\n3. **Question on Table 5**: Table 5 presents ablation study outcomes for varying parameter selections. It remains ambiguous whether the data pertain to a single specific attack scenario or represent an average across multiple attack types.\n\n4. **The stability of parameters across different datasets**: Does the suggest parameter for $\\hat{t}_1=20$ and  $\\hat{t}_2=120$ works consistently across different datasets?\n\nI will raise my score if the author can address my concerns.\n\nLIMITATIONS:\nThe authors have acknowledged certain limitations but have not sufficiently addressed the challenges associated with accessing pre-trained diffusion models. This is a critical issue that requires further exploration. A primary concern is the need for additional experiments to validate the approach using diffusion models trained on varied datasets. For instance, testing a model trained on ImageNet to defend against attacks on the CelebA dataset would provide valuable insights into the model's adaptability and robustness (Just as a suggestion/example).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the significant issue of backdoor defense in a black-box setting during the inference phase of MLaaS applications. It proposes a novel technique called SampDetox, where the method uses diffusion models for detoxification by injecting noise to disrupt backdoors and retrieve clean instances.\n\nSTRENGTHS:\n1. The work underlines an essential research value and a practical scenario by studying black-box backdoor defense during the inference phase. The proposed two-tier defense method shows promise, backed by theoretical underpinnings, and demonstrates efficacy with relatively minimal overhead.\n\n2. An extensive set of experiments validate the effectiveness of the proposed method, surpassing existing defenses. Notably, the method can lower the Attack Success Rate of backdoor models and maintain a high PA on backdoor samples.\n\nWEAKNESSES:\n1. The research appears to make rather robust assumptions for diffusion models. The authors assume that the clean regions of the poisoned samples can be easily restored, while the poisoned regions, especially highly visible triggers, may fail to do so (as empathized in Line 201-202). This could be because the diffusion model learns only from the clean samples and fails to learn backdoor information. While the paper conducts supplemental experiments demonstrating strong defense performance even under other diffusion models (as shown in **Table 8**), the generalizability from ImageNet to CIFAR-10 seems less convincing. If the diffusion model is backdoored or the pre-trained diffusion model is trained on out-of-distribution samples, it could potentially dampen the final defense performance.\n\n2. Furthermore, the method is less likely to be effective if physical attacks are deployed, where natural objects are used as backdoor triggers, as diffusion could learn features of these natural triggers.\n\nQUESTIONS:\nEven though there is a theoretical substantiation for the working defense, I wonder about the universality of this theory in real-world applications. Specifically, in varying datasets and models, would the hyper-parameters \\bar{t}_1 and \\bar{t}_2 require fine-tuning for different noise levels? While **Figure 5** provides an ablation study for different \\bar{t}_1 and \\bar{t}_2, it does not display the specific setting. It would be intriguing to know whether different \\bar{t}_1 and \\bar{t}_2 were utilized for the results in **Tables 1, 3, 4, 5, and 6**.\n\nLIMITATIONS:\nAlthough the paper acknowledges its limitations, a major overlooked limitation lies in the pre-trained diffusion model. Would the detoxifying effect still be substantial if the diffusion model used is trained on out-of-distribution samples compared to the target dataset? This question presents an area of potential further exploration to validate the robustness of the proposed SampDetox method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage black-box method to defend against the backdoor in DNN. The first stage mitigates the backdoor triggers with low  visibility and the second stage mitigates the trigger with higher robustness.\n\nSTRENGTHS:\n1. The two-stage design is novel and effective.\n2. The evaluation is comprehensive and convincing.\n3. The paper is easy to read and follow.\n\nWEAKNESSES:\n1. One major concern about this work is that I think this work may not be effective against high-level semantic triggers. For example [1] adds the digital and physical triggers to the image, and make sure they can be preserved after gaussian blur or print and recapture. This work may not be effective against such triggers. Similar works can be found in [2,3]. However, I found that the authors have already mentioned  and acknowledged this in the paper. So I think this is not a big issue.\n2. The use of diffusion model is not well introduced. It may not be easy for the readers to understand this part if not familiar with the diffusion model. Also, in appendix A.2, the diffusion model from previous work is used, but the authors did not provide the details of the diffusion model. I think it would be better if the authors can provide more details about the diffusion model. Also, would it have any impact if this used diffusion model is replaced by other pre-trained models?\n\n\n[1] Versatile Backdoor Attack with Visible, Semantic, Sample-specific and Compatible Triggers\n\n[2] Backdoor Attacks Against Deep Learning Systems in the Physical World\n\n[3] Dangerous Cloak: Natural Backdoor Attacks on Object Detector in Real Physical World\n\nQUESTIONS:\nWould using another pre-trained model have an influence on your work?\n\nLIMITATIONS:\nThe authors discussed about the main concern I have and acknowledged the limitation for semantic backdoor triggers.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel black-box backdoor purification technique consisting of two stages. In the first stage, it eliminates global perturbation by adding lightweight noise to the inputs and then leverages DDPM for denoising. Subsequently, to address visible and robust triggers, it identifies the local trigger regions, applies intense noise, and then performs diffusion denoising again.\nThe paper provides theoretical analysis to validate the effectiveness of its detoxification process.\nIn the evaluation, extensive experiments are conducted on multiple datasets, model architectures, and attacks, showing that the proposed method outperforms existing baselines. An ablation study justifies the design choices and hyper-parameters, and the paper discusses adaptive attack scenarios.\n\nSTRENGTHS:\n1. Observes interesting links between backdoor visibility and robustness.\n2. Proposes a novel technique based on these observations.\n3. Provides theoretical proof of efficacy.\n4. Demonstrates support for the technique's effectiveness through extensive experiments.\n\nWEAKNESSES:\n1. Lacks evaluation of some semantic backdoors.\n2. Concerns about computational overhead for deployment.\n3. Evaluations of adaptive attacks are relatively weak.\n\nQUESTIONS:\nThe paper presents an innovative concept for black-box input detoxification, highlighting noteworthy observations and techniques. However, I have several concerns I hope the authors can address:\n\n(1) The evaluation includes 10 known backdoor attacks, which are not particularly robust. More complex backdoors utilize semantic features, for instance, [1] uses natural reflection as a trigger, and [2] employs style transferring. It would be beneficial to see evaluations against these more sophisticated backdoors.\n\n(2) The overhead is a concern. Although Section 5.4 suggests using DDIM to reduce the overhead, it is unclear if this significantly affects the detoxification performance. Additional results on this would be helpful.\n\n(3) The adaptive attacks discussed in Section D.2 seem relatively weak, just introducing some noise during training. A stronger attack might use natural features as triggers to enhance the robustness of BadNets. For instance, [3] concatenates two natural images as the backdoor trigger. It is suggested that the authors discuss this and potentially include stronger adaptive attacks.\n\nReference:\n\n[1] Liu, Yunfei, et al. Reflection backdoor: A natural backdoor attack on deep neural networks. ECCV'20.\n\n[2] Cheng, Siyuan, et al. Deep feature space trojan attack of neural networks by controlled detoxification. AAAI'21\n\n[3] Lin, Junyu, et al. Composite backdoor attack for deep neural network by mixing existing benign features. CCS'20.\n\nLIMITATIONS:\nPlease refer to the question part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SampDetox, a novel black-box backdoor defense method using perturbation-based sample detoxification. The key contributions are:\n\n1. A preliminary study revealing the correlation between trigger visibility and poisoned sample robustness.\n2. A two-stage defense approach combining lightweight global noise and targeted local noise to destroy both visible and invisible backdoor triggers.\n3. Use of diffusion models to denoise and restore samples after perturbation.\n4. Comprehensive experiments demonstrating effectiveness against various state-of-the-art backdoor attacks.\n\nSTRENGTHS:\n1. Novel approach combining noise perturbation and diffusion models for backdoor defense\n2. Comprehensive evaluation against 10 different backdoor attacks\n3. Theoretical analysis supporting the approach (Theorems 4.1 and 4.2)\n4. Does not require access to model parameters or training data, making it applicable to MLaaS scenarios (black-box models)\n5. Compared to previous work, it addresses both visible and invisible backdoor triggers\n\nWEAKNESSES:\n1. The effectiveness of SampDetox heavily depends on the correlation between trigger visibility and robustness, but this relationship may not always hold for more sophisticated attacks.\n2. The two-stage approach with fixed noise levels (t1 and t2) may be too rigid to adapt to diverse trigger patterns, potentially missing subtle backdoors or over-perturbing clean regions.\n3. The approach may remove benign high-frequency details from images, potentially affecting classification accuracy for tasks that rely on such fine-grained features.\n\nQUESTIONS:\n1. The paper suggests optimal values for t1 and t2 based on your experiments. How sensitive is the performance of SampDetox to these parameters across different datasets or attack types? Is there a way to automatically determine optimal values for a given scenario?\n2. How does the pixel-wise SSIM calculation in Stage 2 scale with image size? Is there a more efficient way to locate high-visibility triggers without compromising effectiveness? Moreover, how does the method perform on very large images or high-resolution datasets? Is there a practical upper limit on image size due to computational constraints?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}