SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification

Authors: yanxin  yang
chentao  jia
dengke  yan
ming  hu
tianlin  li
xiaofei  xie
xian  wei
mingsong  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:46:36

Abstract: The advancement of Machine Learning has enabled the widespread deployment of Machine Learning as a Service (MLaaS) applications. However, the untrustworthy nature of third-party ML services poses backdoor threats. Existing defenses in MLaaS are limited by their reliance on training samples or white-box model analysis, highlighting the need for a black-box backdoor purification method. In our paper, we attempt to use diffusion models for purification by introducing noise in a forward diffusion process to destroy backdoors and recover clean samples through a reverse generative process. However, since a higher noise also destroys the semantics of the original samples, it still results in a low restoration performance. To investigate the effectiveness of noise in eliminating different types of backdoors, we conducted a preliminary study, which demonstrates that backdoors with low visibility can be easily destroyed by lightweight noise and those with high visibility need to be destroyed by high noise but can be easily detected. Based on the study, we propose SampDetox, which strategically combines lightweight and high noise. SampDetox applies weak noise to eliminate low-visibility backdoors and compares the structural similarity between the recovered and original samples to localize high-visibility backdoors. Intensive noise is then applied to these localized areas, destroying the high-visibility backdoors while preserving global semantic information. As a result, detoxified samples can be used for inference, even by poisoned models. Comprehensive experiments demonstrate the effectiveness of SampDetox in defending against various state-of-the-art backdoor attacks.

Main paper: 
 introduction: 
 With Artificial Intelligence (AI) technologies showing advantages in tasks such as image classification and target detection, they are widely used in safety-and security-critical applications such as autonomous processing [1, 2] , IoT System [3, 4] and health-care [5, 6] . As Machine Learning as a Service (MLaaS) [7, 8, 9] becomes increasingly popular, AI applications are relying on AI services provided by third parties. However, since users cannot guarantee the trustworthiness of service providers, they face serious threats of backdoor attacks [10, 11, 12] . Typically, adversaries inject backdoors into deep models on numerous occasions, especially when the training samples are collected from unreliable sources, or the pre-trained deep models are obtained from untrusted third parties. During the inference phase, when fed clean samples, the backdoored models behave normally with satisfactory classification performance. However, when processing poisoned samples containing trigger patterns, the backdoored models will be fooled into predicting attack target categories with high confidence.Although various defense methods have been proposed to identify and destroy backdoors, most of them are white-box defenses that require additional training samples or the right to analyze model parameters [13, 14, 15] . Since in MLaaS-based applications users are not allowed to access the original training data and parameters of models, these backdoor defense methods are strongly limited. In contrast, black-box backdoor defense methods do not have such requirements. So far, black-box backdoor defense methods can be classified into three categories, i.e., model detection-based [16] , sample detection-based [7, 17] , and sample purification-based [18, 19] defense methods. The basic idea of the first two categories is to simply discard training samples or deep models once they are identified as poisoned. In this case, the usability and performance of classification tasks are greatly affected. As an alternative, sample purification-based approaches strive to destroy backdoor trigger patterns in samples. However, existing sample purification-based approaches are based on the strong assumption that trigger patterns are small and only located in the corners of samples, which is not always true in practice. As a result, such methods can only be used to defend against specific types of backdoor attacks. Worse still, most of the above defense methods need to repetitively feed samples to deep models and get their continuous feedback, resulting in extra non-negligible computation overhead. Therefore, how to effectively mitigate the impacts of all possible backdoor attacks without deteriorating the overall inference performance is becoming a great challenge in black-box defense.Intuitively, adding noise to poisoned samples can disrupt the semantics of the backdoor, thereby eliminating it from the sample. To preserve the semantics of the original sample while removing the backdoor, the diffusion model [20, 21] serves as an effective solution to restore the sample. However, different backdoors exhibit varying levels of robustness to noise, which requires different noise amplitudes for effective disruption. In addition, the amplitude and range of noise will affect the quality of samples restored by the diffusion model. We performed a preliminary study on various backdoor attacks to explore the robustness of different types of backdoors. We found that: i) invisible triggers with low robustness (to random noise) can be easily destroyed, and ii) backdoor triggers with high robustness are easily detectable due to their high visibility.Inspired by these observations, to address the challenge of black-box backdoor defense, we propose a novel two-stage method called SampDetox. Based on our first observation, we globally apply coarse-granularity noises on each given sample in the first stage of SampDetox, where backdoor triggers with low visibility can be easily destroyed. To counteract the effects of such added noises, we then restore the samples by denoising them using diffusion models. Note that for a poisoned sample, its poisoned regions with high visibility, compared with clean regions, will be different from their counterparts in the restored version (see the proof in Theorem 4.2), making them identified easily. Based on our second observation, the second stage can quickly locate such robust triggers and destroy them using our dedicatedly designed noises. Similar to the first stage, the second stage then leverages diffusion models to restore the samples. Since our approach does not pose any assumptions or requirements on the models, it accommodates arbitrary backdoor attack scenarios. In summary, this paper makes the following three major contributions:• We perform a preliminary study on a wide spectrum of backdoors to reveal the correlation between the visibility of triggers and the robustness of poisoned samples. • We present a novel perturbation-based sample detoxification method together with its theoretical foundations. Our approach can effectively destroy all possible triggers with dedicatedly designed noises and does not compromise the overall inference performance. • We conduct extensive experiments to show the applicability and superiority of our approach over state-of-the-art (SOTA) backdoor defense methods. 
related work: 
 Backdoor Attacks. Existing backdoor attacks can be mainly classified into two categories, i.e., visible attacks and invisible attacks, based on the visibility of trigger patterns during the inference phase. Specifically, visible attacks do not consider the concealment of trigger patterns. For example, BadNets [22] uses visible trigger patterns in the form of fixed pixels patched in the corner of the images. Although there exist various visible attacks (e.g., training set corruption without label poisoning [23] , label-consistent backdoor attacks [24] , attacking pre-trained models [25] , and specificsample triggers [26] ), few of them can be applied in real scenarios, since the triggers on samples are conspicuous and can be easily detected by manual inspection. To make attacks stealthier, more and more backdoor attacks (e.g., Blended [27] , WaNet [28] , and ISSBA [29] ) adopt invisible triggers. For example, BPP [30] uses image quantization and dithering as imperceptible backdoor triggers to avoid manual inspection.Backdoor Defenses. According to the capabilities of defenders in manipulating the training and inference processes on deep models, backdoor defense methods can be classified into two categories, i.e., white-box and black-box defenses. Typically, white-box methods defend against backdoor attacks by changing training processes [31, 32] , extract gradients [33] , and fine-tuning and -pruning of poisoned models [34, 35] . However, these methods require access to model parameters, which strongly restricts their usage (e.g., MLaaS applications) in practice. For black-box defenses, such as model detection-based [16] , sample detection-based [7, 17] , and sample purification-based [18, 19] approaches, defenders only can rely on the predictions of input samples. For example, as a model detection-based approach, CBD [36] can effectively identify whether models are poisoned by analyzing their predictions. However, simply discarding the suspicious model/sample does not apply to real-world classification tasks. Rather than ignoring poisoned samples, sample purification-based methods aim to destroy the triggers on samples. For example, Sancdifi [37] measures saliency maps and purifies the specific areas of samples. BDMAE [19] reconstructs the local area of the samples using its AutoEncoder. ZIP [18] transforms and reconstructs samples independently of models through zero-shot image purification. However, all these purification-based methods can handle only small triggers within specific regions.The robustness of triggers against perturbations or modifications plays an important role in determining the performance of backdoor attacks and defenses. For example, CBD [36] analyzes the robustness of triggers with different sizes against random noises to certify its proposed defense scheme. However, CBD only investigates attacks with simple triggers without taking SOTA backdoor attacks into account. To the best of our knowledge, SampDetox is the first attempt to consider the correlation between the visibility of backdoor triggers and the robustness of poisoned samples. As a model-independent method, SampDetox enables perturbation-based sample detoxification, which can effectively defend against complex backdoor attacks within arbitrary application scenarios. 
preliminary study: 
 This section first reviews the process of backdoor attacks. Then, it gives our definitions of the visibility of backdoor triggers and the robustness of poisoned samples and reveals their inherent correlation based on a preliminary study.Assume that M is a model poisoned by some backdoor attack. Let x c be a clean sample, and x p represent its poisoned version by adding specific backdoor triggers. If the backdoor attack succeeds, the model M should classify x c into its correct category while classifying x p into a target category.To defend against backdoor attacks, the findings in [38] show that transforming or perturbing samples will achieve defensive effects against locally patched triggers. However, according to the observations in [18, 19] , these operations are unsuitable for defending against other types of triggers. This is mainly because locally patched triggers are unremarkable and susceptible to disturbance, while other triggers are robust (i.e., not sensitive) to such backdoor defenses. To better understand the inherent characteristics of backdoor attacks and the trends of their development, we conduct a preliminary study on existing attacks to reveal the correlation between the visibility of backdoor triggers and the robustness of their poisoned samples, whose descriptions are as follows.Visibility. When a backdoor attack is applied to some deep model, the visibility of embedded triggers indicates the stealthiness of the backdoor attack. In this paper, we use the Structural Similarity Index Measure (SSIM) [29, 39] to evaluate the visibility of backdoor triggers. This is because, unlike traditional measurements like Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR), SSIM takes into account the local characteristics of samples, aligning more closely with the visual system of human eyes. Let SSIM(x c , x p ) ∈ [-1, 1] be the SSIM between x c and x p , quantifying the visibility of trigger patterns embedded in x p . For ease of evaluation, we adopt the notation v = (1 -SSIM(x c , x p ))/2 to normalize the visibility values. Specifically, the higher the value of v, the more notable the triggers are.Robustness. For a given poisoned sample x p , we propose to use the term robustness to quantify its capability to resist perturbations or modifications imposed by random noise. Assume that the backdoored model classifies x p into a specific attack target category due to the existence of triggers. We apply random noises to x p and get a modified sample x m , where the modification process can be described by the equation x m = (1 -η r )x p + η r ϵ (ϵ ∼ N (0, I)). Since η r plays an important role in determining whether its modified version x m belongs to another category, we use it to reflect the robustness of x p . Typically, the higher the value of η r , the more robust the poisoned sample x p is.To explore the correlation between the visibility of triggers and the robustness of their host poisoned samples, we conduct an experiment on the dataset CIFAR-10 against ten backdoor attacks, including both visible and invisible attacks, where each attack is applied on 200 randomly selected samples from CIFAR-10. Note that in the experiment, we assume that each poisoned sample is touched by only one kind of backdoor attack. In other words, each poisoned sample is embedded with a specific trigger. For each poisoned sample, we calculate the visibility score v of the trigger and the robustness η r of the poisoned sample. As shown in Figure 1 , for different poisoned counterparts of a clean sample, their visibility and robustness scores differ significantly under different backdoor attacks. Observation 1. For backdoor triggers with low visibility (v < 0.13), their host samples (i.e., ones with η r < 0.18 located in the red box) have low robustness. According to [38] , the triggers within these poisoned samples can be easily destroyed by lightweight noises.Observation 2. For poisoned samples (i.e., those located in the blue box) with high robustness (η r ≥ 0.18), their backdoor triggers have high visibility, which renders them easily detectable and accurately located.4 Our SampDetox Approach 
threat model: 
 We consider defending against backdoor attacks in black-box scenarios [17, 18] , where defenders do not have access to the parameters of deep models and can only obtain predictions for samples.For attackers, we assume that they can completely control the training processes of models, enabling them to poison samples in datasets and modify model components to achieve poisoned models. In our approach, defenders strive to detoxify poisoned samples to classify them into their original correct categories. For clean samples, defenders aim to maintain their classification accuracy. As observed in Section 3.2, there exists a strong correlation between the visibility of triggers and the robustness of poisoned samples, which can be used for backdoor defense. However, due to the lack of prior knowledge of clean samples, it is difficult to figure out both the trigger visibility and robustness of poisoned samples. To defend various backdoor attacks, Figure 3 presents a novel perturbation-based backdoor defense framework using our proposed sample detoxification method, which consists of two stages:Stage 1: Global Detoxification. According to Observation 1, backdoor triggers with low visibility typically have low robustness poisoned samples. In this case, such triggers can be easily destroyed by adding lightweight noises to suspected samples. However, due to the low visibility or even invisibility, it is hard to determine the existence or location of triggers within a given sample. To destroy all these potential triggers, Stage 1 applies the noise-based perturbation (see Section 4.3) globally to each sample using lightweight noises. Note that the introduction of noises inevitably affects the inference accuracy of deep models. To counteract the impacts of added noises on inference performance, we then utilize DDPM-based denoising (see Section 4.4) to denoise the samples. In this way, Stage 1 can destroy possible triggers spread over the poisoned samples with low robustness.Stage 2: Local Detoxification. According to Observation 2, poisoned samples with high robustness are often embedded with highly visibility triggers. Based on DDPM-based denoising in Stage 1, the clean regions of poisoned samples can easily be restored to their original states, while the poisoned regions cannot, especially when triggers are highly visible. Due to this fact, by comparing a potentially poisoned sample with its detoxified version by Stage 1 pixel by pixel, we can figure out the positions of obvious triggers and the required intensity of local noises. Then, Stage 2 applies the noise-based perturbation locally on each sample using pixel-specific noises to destroy such triggers. Similarly to Stage 1, the added noises in Stage 2 also greatly influence the final inference accuracy for both clean and detoxified poisoned samples, thus requiring diffusion models to perform the denoising. After Stage 2, both visible and invisible triggers can be safely eliminated without degrading the inference performance.To quantitatively control the intensity of noises applied on samples in each stage of SampDetox, we resort to the forward Denoising Diffusion Probabilistic Model (DDPM) [20] , which can also be used to determine the number of denoising steps required in the DDPM-based denoising process. Assume that x 0 is an input sample, and x i is the sample after adding noises for i times (steps). In the forward process of DDPM, the model incrementally adds noises to x 0 step by step until the sample (denoted by x T ) is completely composed of random Gaussian noises at step T , where T is a diffusion model-specific constant. In each step t, x t is obtained by adding a random Gaussian noise ϵ∼N (0, I) to x t-1 based on the formulax t = √ 1 -β t x t-1 + √ β t ϵ, where β t ∈(0, 1). For a given x t-1 , the posterior probability of x t can be represented as q(x t |x t-1 )=N (x t ; √ 1 -β t x t-1 , β t I). Since step t only relies on step t-1, the forward process can be regarded as a Markov process in the form of q(x t |x 0 )= T t=1 q(x t |x t-1 ). For a given x 0 , we can get the probability of x t asEQUATIONwhere α t =1-β t and α t = t i=1 (1-β t ). According to [20] , we can get the exact relationship between x 0 and x t using the formulaEQUATIONwhere z∼N (0, I) is a random Gaussian noise.Leveraging the forward DDPM, our approach aims to add proper random Gaussian noise on samples within t steps based on Equation 2, which can destroy backdoor triggers in samples. Therefore, Theorem 4.1 investigates the potential of DDPM to eliminate triggers within poisoned samples. Theorem 4.1. Suppose that x t represents a diffused sample at step t of the forward process in DDPM, where t ∈ [0, T ]. Given a clean sample x c 0 and its poisoned version x p 0 , we obtain x c t and x p t through the forward DDPM, whose distributions are P t (x) and Q t (x), respectively. By using the Kullback-Leibler divergence [40, 41] to measure the distance between the two distributions, we haveEQUATIONwhere the equality happens only when P t =Q t ., and the inequality implies that the KL divergence between P t and Q t monotonically decreases along with the forward DDPM.Please refer to Appendix B.1 for the proof of the theorem. According to Theorem 4.1, for any positive number ξ, there exists a minimum t ∈ [0, T ] such that for any t ≥ t we have KL(P t ||Q t ) ≤ ξ. Note that this theorem forms the theoretical basis to utilize the forward DDPM to minimize the distribution discrepancy between poisoned and clean samples, thus mitigating the impact of backdoor triggers.We also utilize DDPM to perform denoising on samples within the two stages of SampDetox. For each pixel in a sample, the number of denoising steps is determined by its noise-based perturbation steps (i.e., t) as introduced in Section 4.3. In reverse DDPM, by taking the current sample state x t as an input, the diffusion model can be used to achieve its previous state x t-1 . Given both x 0 and x t , according to Bayes' Theorem, we can obtain the probability distribution of x t-1 as p(x t-1 |x t , x 0 )=p(x t |x t-1 ) • p(x t-1 |x 0 )/p(x t |x 0 ). By resorting to the relationship between x 0 and x t described by Equation 2, according to [20] , we can use x t to approximate x 0 as follows:EQUATIONwhere z t is predicted by the pre-trained neural network θ of DDPM, denoting the estimation of the real noise in step t. Accordingly, we can obtain x t-1 from x t :EQUATIONwhere σ 2 t = 1-αt-1 1-αt • β t and z ∼ N (0, I). Unlike existing DDPM-based methods that strive to restore samples from Gaussian noises, SampDetox uses diffusion models to denoise samples generated by our noise-based perturbation approach (see Section 4.3). Assuming that the training data used for the pre-trained diffusion model do not involve any knowledge about backdoor triggers, according to the principle of DDPM, the denoised results will restore the original samples with all the triggers cleansed. Theorem 4.2 investigates the difference between clean and poisoned regions of poisoned samples after denoising. Theorem 4.2. Suppose that R(x) denotes a specific region of the sample x. Given a poisoned sample x p 0 , we use R c (x p 0 ) and R p (x p 0 ) to represent a clean region and a poisoned region of the sample, respectively. We perform the noise-based perturbation on x p 0 for t steps and get the noisy samples x p t , and then utilize DDPM to get a denoised sample x p 0 . When comparing different regions of the original sample x p 0 with their counterparts of the detoxified sample x p 0 , we haveEQUATIONwhere n 1 and n 2 represent the numbers of pixels in R p (x p 0 ) and R c (x p 0 ), respectively. Please refer to Appendix B.2 for the proof of the theorem. According to Theorem 4.2, we can calculate the similarity between the denoised and original sample to identify the positions of trigger patterns. The inequality in Theorem 4.2 implies that for a poisoned sample, its poisoned regions, compared with clean regions, will be more different from their counterparts in the restored version.Input: i) x0, original input sample; ii) t1, t2, the numbers of noise-adding and denoising steps in Stage 1 and Stage 2, respectively; iii) θ, diffusion model; Output: a detoxified sample;1:x t 1 ← α t 1 x0 + 1 -α t 1 z //z ∼ N (0, I) 2: for t = t1 to 1 do 3: z ∼ N (0, I) if t > 1, else z = 0 4: σ 2 t ← 1-α t-1 1-α t • βt 5: xt-1 ← 1 √ α t (xt -1-α t √ 1-α t θ(xt, t)) + σtz 6: end for 7: M ← PixelSSIM(x0, x0) 8: for each pixel i in M do 9: m[i] ← t2 • M [i] 10: x t 2 ← √ α m[i] • x0 + 1 -α m[i] z //z ∼ N (0, I) 11: end for 12: for t = t2 to 1 do 13: z ∼ N (0, I) if t > 1, else z = 0 14: zt ← θ(xt, t) 15:for each pixel i in M do 16:ifm[i] > t then 17: xt-1[i] ← 1 √ α t (xt[i] -1-α t √ 1-α t zt[i]) + σtz[i] 18: else xt-1[i] ← xt[i] 19: end if 20:end for 21: end for 22: return x0 Algorithm 1 details the implementation of SampDetox. Lines 1-6 describe the process of global detoxification, while Lines 8-21 present the local detoxification process. Specifically, Line 1 applies the noisebased perturbation to the sample following Equation 2 with a fixed noise intensity t 1 . Lines 2-6 show the process of DDPMbased denoising to obtain the denoised sample x 0 . In Line 7, SampDetox calculates the SSIM score for each pixel between the original sample x 0 and its denoised version x 0 . Lines 8-11 describe how a noise with a specific intensity (i.e., m[i]) is added to the pixel i. Lines 12-21 present the DDPM-based denoising process, whether SampDetox iteratively calculates the estimation of the real noise z t with x t using the diffusion model θ and then selectively denoises each pixel accordingly. In Lines 16-19, SampDetox only updates pixels with noise intensity higher than that of the current step. Finally, Line 22 returns the resulting detoxified sample x 0 , which is the denoised sample by Stage 2.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper fully discloses all the information needed to reproduce the main experimental results. 
experiments: 
 To evaluate the effectiveness of our approach, we implemented our approach, i.e., SampDetox, on top of Pytorch (version 1.13.0). We compared the defense performance between SampDetox and state-of-the-art (SOTA) black-box defense methods against various well-known backdoor attacks. All experiments were carried out on an Ubuntu workstation equipped with one Intel i7-13700K CPU, 64GB memory, and one NVIDIA GeForce RTX4090 GPU.Dataset and Model Settings. We investigated three classical datasets (i.e., CIFAR-10 [42], GT-SRB [43] , and Tiny-ImageNet [44] ) and three models (i.e., PreAct-ResNet18, ResNet34 [45] and VGG-19 [46] ). Due to space limitations, this section only presents the experimental results of PreAct-ResNet18 on the CIFAR-10 dataset. Note that we can find similar trends from the experiments using different models and datasets. Please refer to Appendix C for more details.Evaluation Metrics. To objectively evaluate the effectiveness of a given backdoor defense method, we adopted the following three metrics: i) Clean sample Accuracy (CA), which denotes the inference accuracy of clean samples processed by the defense method; ii) Poisoned sample Accuracy (PA), which indicates the inference accuracy of poisoned samples purified (detoxified) by the defense method based on their ground-truth labels; and iii) Attack Success Rate (ASR), which represents the rate of successful attacks by poisoned samples. Note that a higher CA means that the defense method causes less interference in the classification of clean samples. Typically, backdoor defenders strive to achieve both high CA and PA with lowered ASR.Attack Methods. We conducted defenses against ten SOTA backdoor attacks, i.e., BadNets [22] , SIG [23] , Label Consistent (LC) [24] , TrojanNN [25] , Dynamic [26] , Blended [27] , Low Frequency (LF) [47] , WaNet [28] , ISSBA [29] , and BPP [30] . Note that the first five are visible attacks, while the last five are invisible attacks. We used the same benchmark settings in [48] to configure all these attack methods. Please refer to Appendix A.1 for more details. Defense Baselines. We compared SampDetox with three SOTA sample purification-based backdoor defenses: i) Sancdifi [37] , which utilizes the RISE algorithm to measure the saliency maps and purify the specific areas of samples; ii) BDMAE [19] , which uses a masked AutoEncoder to defend against backdoor attacks; and iii) ZIP [18] , which applies linear transformations to samples and restores them under noises by using diffusion models. Note that both Sancdifi and BDMAE need to feed samples into deep models repeatedly to obtain their predictions, while ZIP does not rely on predictions. Please refer to Appendix A.2 for more details.Figure 4 shows an example of backdoor defense against the ten different attacks using our proposed SampDetox. Here, the first row presents both a clean sample and all its poisoned counterparts. The second row presents the triggers identified by stage 2 of SampDetox, while the third row exhibits the detoxified samples of each attack. From this figure, we can find that SampDetox can effectively figure out all visible and invisible triggers embedded by different SOTA backdoor attacks. Impacts of Different Stages and Denoising. The implementation of SampDetox consists of two stages. The first stage strives to destroy triggers globally on a given sample in a coarse manner, while the second stage tries to identify fine triggers and remove them locally. To examine the impacts of these two stages, we consider a variant (i.e., "Stage 1") of our approach, which does not take stage 2 of SampDetox into account. Meanwhile, to evaluate the impact of the denoising operations, we considered another variant of SampDetox (i.e., "Noise*"), where denoising is not applied to restore the samples in both stages. Table 2 presents the ablation study results against the ten attacks. Note that to enable an intuitive understanding of the role of each stage in SampDetox, this table provides the average visibility of backdoor triggers for each attack.For all the defense methods in Table 2 , we can find that the smaller the trigger visibility caused by backdoor attacks, the lower PA and higher ASR we can achieve. Although "Noise*" can achieve comparable ASR results to SampDetox, SampDetox outperforms "Noise*" in both CA and PA significantly. It means that simply adding noises to samples can effectively destroy the backdoor triggers of the samples. However, the noises themselves will also result in a sharp decline in the inference accuracy of clean and poisoned samples. If we merely adopt the first stage of SampDetox due to the denoising operation by diffusion models, both CA and PA can be improved. However, since the first stage only coarsely applies the perturbation on samples, most subtle backdoor triggers still survive.As an example in SIG attack, "Stage 1" can only achieve a PA of 9.48% but with an ASR of 92.33%. By combining both Stages 1 and 2, SampDetox can effectively identify such undetectable triggers and destroy them without affecting the inference of original samples, leading to a better PA and ASR. For example, in SIG attack, by using SampDetox, we can achieve a PA of 65.06% and an ASR of 11.03%. From this table, we can observe that when t 1 and t 2 increase, the ASR decreases, indicating the effect of adding noises in destroying backdoor triggers. However, a high value of t 1 and t 2 will also lead to a significant reduction of CA and PA, showing that the denoising processes are strongly affected by excessive noises. Therefore, we suggest to set t 1 = 20 and t 2 = 120 in practice.To show the applicability of SampDetox, we investigated the extra time overheads, explored training sample detoxification, evaluated the defense performance of SampDetox against SOTA adaptive attacks, and analyzed its limitations. Due to the space limitation, this section only presents the overhead caused by inference detoxification. Please refer to Appendix D for more discussions. Extra time overhead. Based on our proposed two-stage perturbation-based sample detoxification, SampDetox can protect any deep models during their inference phase. However, similar to other backdoor defense approaches, the detoxification process will inevitably lead to extra time overhead. To address this issue, SampDetox resorts to the Denoising Diffusion Implicit Model (DDIM) [21] to accelerate the denoising process. Figure 5 compares the average inference time per sample for different backdoor defense methods. We can find that with the help of DDIM, the inference time of SampDetox is comparable to that of the method without defense.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The paper provides code in supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The paper discusses the potential societal impacts of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 Based on our observed correlation between the visibility of backdoor triggers and the robustness of poisoned samples, this paper introduces a novel black-box backdoor defense approach named SampDetox. By using our proposed perturbation-based sample detoxification, SampDetox can easily destroy backdoor triggers in samples and restore the samples using diffusion models. Extensive experiments against various complex backdoor attacks show the superiority of SampDetox from both perspectives of effectiveness and applicability to arbitrary scenarios. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper uses popular datasets and models. We cites the original paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Attack Method Settings BadNets [22] . BadNets is the first proposed backdoor attack method, which patches a fixed pattern on the corners of benign samples and modifies their labels to the attack target category. We use a 3 × 3 black and white checkerboard in the lower right corner of samples as the trigger pattern.SIG [23] . SIG modifies the pixels in samples based on the sinusoidal signal to generate poisoned samples. We set the frequency of the sinusoidal signal to 3 as the trigger pattern.Label Consistent (LC) [24] . LC generates the poisoned samples using adversarial attack methods and keeps the original sample label unchanged. The trigger pattern is identical to BadNets, and other settings follow the original paper.TrojanNN [25] . TrojanNN injects backdoors into a pre-trained model instead of injecting backdoors during the training. We use a white apple image as the trigger pattern.Dynamic [26] . Dynamic trains a trigger generator to generate the specific triggers for each sample.The backdoor attack will only succeed if the specific trigger matches its sample. We follow the settings in the original paper to generate triggers.Blended [27] . Blended generates poisoned samples by blending benign samples with a fixed pattern. Blending with coefficient α, Blended increases the invisibility of the trigger to avoid manual inspection. We use a "hello kitty" image as the trigger pattern.Low Frequency (LF) [47] . LF proposes an optimization-based approach to filter high-frequency artifacts to generate smooth triggers. We follow the settings in the original paper to generate triggers.WaNet [28] . WaNet uses fixed warping functions to distort benign samples slightly to construct the poisoned samples. We follow the settings in the original paper to generate poisoned samples with trigger patterns.ISSBA [29] . ISSBA employs image steganography to embed the string trigger into benign samples, which generates unique triggers that are specific to each sample. Following the settings in the original paper, we use the trained image steganography method.BPP [30] . BPP generates the poisoned samples using image quantization. To inject the backdoor more efficiently, BPP proposes a contrastive learning-based approach to improve the success rate of backdoor attacks. We set the negative ratio to 0.1.Sancdifi [37] . Sancdifi followed the settings in Section "Numerical Experiment" of its paper. Sancdifi computed RISE maps using 2000 random binary masks and set the saliency threshold to 0.95. For a fair comparison, Sancdifi used the same diffusion models as ours.BDMAE [19] . BDMAE followed the settings in Section "Method Configurations" of its paper, which used masked autoencoders pre-trained on ImageNet with 24 encoder layers.ZIP [18] . ZIP followed the settings in Section "Purification Implementation" of its paper, which set its hyperparameter to 2. For a fair comparison, ZIP used the same diffusion models as ours.We utilize the framework of the improved diffusion model provided by [21] . The number of total diffusion steps was set to 1000 for all the datasets in the experiments, the noise schedule was set as "cosine", and the learning rate was set to 1e-4. In the experiments, we load the parameters of the pre-trained model for each dataset. For DDIM, compared with the total steps of the diffusion model (e.g., T = 1000), our approach does not require too many steps to add noise and denoise samples (e.g., t 1 = 20 and t 2 = 120). Therefore, when we use DDIM to accelerate the denoising, we set the speed-up pace to 20 to ensure the effectiveness of our approach.In our work, the PixelSSIM denotes an intermediate data structure for calculating the SSIM [39] value of two given samples. Specifically, for each block pair in the two samples, we calculate its SSIM value and record it in PixelSSIM at the same position as one of the central pixels of the block pair. When sliding the block pair along the two dimensions of the samples, we can collect all the SSIM values for the block pairs and form the final PixelSSIM. Note that by averaging all the values in PixelSSIM, we can achieve the SSIM value for the two samples. We use structural similarity() in the skimage library of Python to implement the PixelSSIM.B.1 Proof of Theorem 4.1Theorem 4.1. Suppose that x t represents a diffused sample at step t of the forward process in DDPM, where t ∈ [0, T ]. Given a clean sample x c 0 and its poisoned version x p 0 , we obtain x c t and x p t through the forward DDPM, whose distributions are P t (x) and Q t (x), respectively. By using the Kullback-Leibler divergence [40, 41] to measure the distance between the two distributions, we haveEQUATIONwhere the equality happens only when P t =Q t , and the inequality implies that the KL divergence between P t and Q t monotonically decreases along with the forward DDPM.Proof: First, following the definition of the diffused sample probability distribution in DDPM [20] , like Equation 1, when given a clean sample x c 0 and a poisoned sample x p 0 , we have their distributions:EQUATIONwhere α t = 1 -β t and α t = t i=1 α i . Therefore, we know that both P (x c t |x c 0 ) and Q(x p t |x p 0 ) are Gaussian distributions. When both terms of the KL divergence are Gaussian distributions, we can simplify the formula as follows:EQUATION.(9) Since P t and Q t have the same σ, i.e., σ = σ 1 = σ 2 = √ 1 -α t I, we can continue to simplify the formula:KL(P t ||Q t ) = 1 2 (µ 1 -µ 2 ) 2 σ 2 2 = 1 2 α t (x c 0 -x p 0 ) 2 1 -α t (10)Finally, we take the difference of both sides of the inequalityKL(P t+1 ||Q t+1 ) -KL(P t ||Q t ) = 1 2 α t+1 (x c 0 -x p 0 ) 2 1 -α t+1 - 1 2 α t (x c 0 -x p 0 ) 2 1 -α t = (x c 0 -x p 0 ) 2 2 α t+1 1 -α t+1 - α t 1 -α t = (x c 0 -x p 0 ) 2 2 α t α t+1 1 -α t α t+1 - α t 1 -α t = (x c 0 -x p 0 ) 2 2 1 1 -α t α t+1 - 1 1 -α t (11)Since α t = 1 -β t and β t ∈ (0, 1), α t should be less than 1. As a result, we can getEQUATION)According to Equation 12, we haveEQUATIONwhere the equality happens only whenP t = Q t .B.2 Proof of Theorem 4.2 Theorem 4.2. Suppose that R(x) denotes a specific region of the sample x. Given a poisoned sample x p 0 , we use R c (x p 0 ) and R c (x p 0 ) to represent a clean region and a poisoned region of the sample, respectively. We perform the noise-based perturbation on x p 0 for t steps and get the noisy samples x p t , and then utilize DDPM to get a denoised sample x p 0 . When comparing different regions of the original sample x p 0 with their counterparts of the detoxified sample x p 0 , we haveEQUATIONwhere n 1 and n 2 represent the numbers of pixels in R p (x p 0 ) and R c (x p 0 ), respectively. Proof: First, we expand the inequality that we need to prove. Therefore, proving Equation 14 is equivalent to proving the following inequality:EQUATION)According to Equation 5, we haveEQUATIONwhere σ 2 t = 1-αt-1 1-αt • β t and z ∼ N (0, I). θ is a pre-trained neural network. Assuming that θ is a fully trained model that can generate possible noise based on the input sample x t and the step t. Therefore, we haveEQUATIONwhere z t is the real noise in step t.Then, we analyze the pixels in the clean and poisoned regions, respectively. We denote a pixel in the clean region of the original sample as x 0 , and a pixel in the poisoned region as y 0 . Therefore, we have x t and y t to denote the pixels after our noise-based perturbation. And we have x 0 and y 0 to denote the pixels after our DDPM-based denoising process. Since θ has a strong ability, we can approximate the entire denoising process with the following:EQUATIONAccording to Equation 2 about the relationship between x 0 and x t , we haveEQUATIONwhere z t is the real noise and z = [z] = [θ(x t , t)]. According to Equation 17, we haveEQUATIONSupposing that the training data used for the pre-trained diffusion model does not contain any knowledge about backdoor triggers, it is reasonable to assume that x 0 belongs to the probability distribution of the training dataset of θ, while y 0 does not. Therefore, we haveEQUATIONAccording to Theorem 4.1, when the sample is fully noised, the sample is equivalent to Gaussian noise. Additionally, the sample generated by the unguided denoising process from Gaussian noise is completely independent of the original sample. Therefore, we have the equation about x and y as follows:EQUATIONAccording to Equation 21and Equation 22, we finally get the following equation:EQUATIONwhere t ∈ (0, T ). Therefore, Equation 15 can be proved. Finally, we haveEQUATION)Table 4 and Table 5 show the defense performance of our approach and three defense baselines, i.e., Sancdifi [37] , BDMAE [19] and ZIP [18] , against ten state-of-the-art (SOTA) attack methods on the GTSRB and Tiny-ImageNet dataset, respectively. The experimental results in the table illustrate that our approach outperforms existing SOTA methods on a variety of datasets, which shows the generalizability of our approach. Since our approach does not require any interaction with the model, it should have defensive effects on different models. Table 6 and Table 7 show the defense performance of our approach on the VGG-19 and ResNet34 models on CIFAR-10, GTSRB, and Tiny-ImageNet datasets, respectively. The experimental results show that our approach has satisfactory defense effects on VGG-19 and ResNet34, which demonstrates that our approach is model-independent. In this section, we compare our approach with existing sample transformation methods on CIFAR-10, including Noising, ShrinkPad, and Blur [49] . Noising attempts to destroy the backdoor trigger structure by adding a large amount of noise to the sample without considering the inference accuracy. ShrinkPad shrinks the samples and then randomly pads samples back to their original size. Blur utilizes Gaussian kernel to pool the samples to destroy backdoor trigger patterns. The experimental results are shown in Table 8 .From the table, we can find that transforming the samples can break some of the backdoor triggers. However, the transformation will cause great damage to the quality of the samples, resulting in a sig- nificant reduction in Clean sample Accuracy (CA) and Poisoned sample Accuracy (PA). Specifically, though Blur has lower ASR than SampDetox when defending against the SIG and LF attacks, our approach has better CA and PA. In addition, our methods have the best defensive performance. The experimental results show that our approach can successfully defend against various attacks while preserving the original semantic information of samples to the greatest extent.In this section, we deploy our approach using different pre-trained diffusion models to defend against backdoor attacks on CIFAR-10 and compare their defense performance. Specifically, we choose three diffusion models with different parameters: 1) pre-trained on CIFAR-10, which is the same setup as our main experiments, 2) pre-trained on ImageNet, and 3) pre-trained on ImageNet and fine-turned on CIFAR-10. Notably, we used 5% clean samples in the test dataset of CIFAR-10 to fine-tune the pre-trained model, and we excluded these data during the experiments. The experimental results are shown in Table 9 .From the table, we can observe that SampDetox with different diffusion models has satisfied defensive performance on Clean sample Accuracy (CA) and Attack Success Rate (ASR). For CA, since clean samples do not have any trigger patterns, only a little noise will be added to them in Stage 2 of our approach. Therefore, all these three diffusion models can accurately denoise samples. For ASR, the results are generally not related to the denoising process. For Poisoned sample Accuracy (PA), when defending against visible attacks (i.e., BadNets, SIG, LC, TrojanNN, and Dynamic), the model pre-trained on ImageNet has much lower accuracy, meaning that the model pre-trained on outside distribution datasets can not accurately denoise samples, especially when much noise is added. The fine-tuning model achieves satisfactory defensive performance against different attacks. Therefore, when we need to detoxify samples without pre-trained models, we can fine-tune a diffusion model to deploy our defense approach.Table 9 : Defense performance of our approach using different diffusion models. Additionally, considering the case where pre-training datasets for diffusion models are out-ofdistribution from their target classification tasks, we conducted experiments to evaluate the performance of SampDetox on a subset of the MS-Celeb-1M [50] dataset. Specifically, we considered the top 100 labels with the largest number of samples and randomly selected 380 samples for each label. We split the training and tests with a ratio of 8 : 2 and adjusted the shape of the samples to 224 * 224. We adopted BadNets [22] and WaNet [28] as attack methods, representing a visible and an invisible backdoor attack, respectively. We considered four pre-trained diffusion models: i) a pre-trained model (i.e., Model 1) on the subset of MS-Celeb-1M, ii) a pre-trained model (i.e., Model 2) on another subset of MS-Celeb-1M, iii) a pre-trained model (i.e., Model 3) on ImageNet, and iv) Model 4, which is fine-tuned from Model 3 using a subset of MS-Celeb-1M.Table 10 shows the experimental results. We can find that SampDetox with different diffusion models achieves satisfied defensive performance on ASR. Note that, for Model 3, the CA and PA of SampDetox decrease. This is because Model 3 is pre-trained on ImageNet rather than MS-Celeb-1M, resulting in the diffusion model failing to effectively denoise MS-Celeb-1M samples. However, we can find significant improvements in the CA and PA of Model 4 since Model 4 is fine-tuned using a subset of MS-Celeb-1M. In other words, SampDetox can defend against attacks using out-of-distribution diffusion models with proper fine-tuning.Table 10 : Defense performance of our approach using out-of-distribution diffusion models. To evaluate the defense performance of SampDetox on classification tasks that rely on benign highfrequency details, we conducted experiments on the DTD (Describable Textures Dataset) dataset [51] , where we adopted a visible backdoor attack, i.e., BadNets [22] , and an invisible backdoor attack, i.e., WaNet [28] , respectively. Table 11 compares SampDetox with six baselines. From this table, we can find that though the CA of SampDetox is a little smaller than the CA of "No Defense", SampDetox can achieve the best defense performance on CA, PA, and ASR compared with all baselines, which shows the applicability and superiority of SampDetox. Since DDIM [21] was proposed to accelerate the speed of DDPM without affecting the image generation quality, the defense performance of "SampDetox+DDIM" is comparable to that of "SampDetox+DDPM". To demonstrate the performance of "SampDetox+DDIM", we conducted new experiments on CIFAR-10 against ten SOTA backdoor attacks. The experimental results are shown in Table 12 . From the table, we can find that the defense performance of "SampDetox+DDIM" is almost the same as that of "SampDetox+DDPM", which means that "SampDetox+DDIM" can match the defense efficacy of "SampDetox+DDPM".Before training deep models, we can use SampDetox to conduct perturbation-based training sample detoxification, protecting them against backdoor injection. Table 13 compares the defense performance of the trained models by SampDetox. Here, the second column shows the defense performance without using SampDetox. The third and fourth columns present the defense performance of models trained by detoxified samples. Note that for the third column, we did not change the labels of detoxified samples. In other words, the poisoned samples still keep their dirty labels (denoted by D.L.). For the fourth column, we tried to correct the dirty labels based on the inference results of the models generated by the third column. Based on the detoxified samples and corrected clean labels (denoted by C.L.), the fourth column shows the defense performance of our approach. Note that for this table, we did not apply sample detoxification on test samples. From this table, we can observe that SampDetox can effectively defend against various backdoor attacks. Although "SampDetox+D.L." and "SampDetox+C.L." have similar ASR, the CA of "SampDetox+C.L." can be notably improved based on the corrected labels of poisoned samples. Assume that attackers have all the prior knowledge of our proposed approach to design an adaptive attack to bypass our defense approach. Since SampDetox is based on observations about the correlation between the visibility of triggers and the robustness of poisoned samples, the attackers can try two types of adaptive attacks as follows.1) Adaptive attack 1. The attacker uses an attack with both low visibility and high robustness since attacks with high visibility and low robustness are easily defended. To guarantee low visibility, the attacker adopts an invisible backdoor attack, i.e., WaNet. To ensure high robustness, the attacker adaptively adds noises to samples during the training phase to improve the robustness of triggers.2) Adaptive attack 2. The attacker does not consider visibility issues and tries to maximize robustness to bypass our defense. In this case, the attacker adopts a visible backdoor attack, i.e., BadNets, and adds robust noise during the training phase to improve robustness.We conducted experiments to defend against the two adaptive attacks on the CIFAR-10 dataset. Table 14 shows the results of the experiments, where the first column denotes the amount of noise added during training to improve the robustness. From the table, we find that our approach can defend against adaptive attacks. Although our approach aims to defend against backdoor attacks with any kind of trigger, there are still certain triggers that SampDetox cannot remove. Physical attack [52, 53] is a backdoor attack with visible but natural triggers, usually physical objects. Since our approach identifies the positions of trigger patterns based on Theorem 4.2 by calculating the similarity between the denoised and original samples, the effectiveness of our approach depends on the pre-trained diffusion model. When the triggers are visible, are part of the sample classification characteristics, and belong to the knowledge contained in the pre-trained diffusion model, our defense approach cannot destroy the trigger patterns and defend against these attacks. To investigate the limitation of SampDetox, we conducted experiments with three semantic backdoor attacks, i.e., Refool [54] , DSFT [55] and CBA [56] . Table 15 presents the defense performance of SampDetox against Refool and DSFT. From the table, we can find that SampDetox can effectively defend against such backdoor attacks. This is because the semantic features adopted by Refool and DSFT are neither part of the original classified features nor are their distributions the same as the distribution of the training dataset of the pre-trained diffusion model. Therefore, the triggers based on such semantic features will not be restored by the denoising phase of SampDetox.Additionally, to explain why SampDetox can defend against these two attacks, we conducted experiments to calculate the robustness of their poisoned samples, respectively. For each attack, we randomly selected 50 poisoned samples on CIFAR-10 and calculated the robustness value of each sample following our robustness definition in Section 3.1. For Refool, the robustness of its poisoned samples has a mean value of 0.0759 and a standard deviation of 0.011. For DFST, the robustness of its poisoned samples has a mean value of 0.137 and a standard deviation of 0.019. From the experimental results, we can find that the robustness of the poisoned samples is low (ones with robustness < 0.18, according to Figure 2 ). Therefore, when SampDetox adds enough noise to these samples, even if trigger patterns are partially restored by diffusion models, backdoors will not be successfully triggered.However, SampDetox fails to defend against CBA. Since CBA concatenates two natural images as backdoor triggers, it is considered the backdoor attack type that SampDetox fails to defend. The experimental results on CIFAR-10 are as follows: Without defense, CA, PA, and ASR were 87.52%, 10.16%, and 90.17%, respectively. However, when using SampDetox, CA, PA, and ASR were 86.81%, 38.45%, and 37.28%, respectively. The experimental results show that SampDetox has a certain effect in defending against CBA but fails to minimize ASR.Figure 6 shows examples of the poisoned samples with their visibility v and robustness η r . Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This work proposes a black-box approach to defend against backdoor attacks, based on a diffusion model that restores the image parts covered by the backdoor trigger. Most of the concerns raised by the reviewers were addressed, even if the paper still falls short in evaluating against adaptive attacks and discussing the limitations of the approach, which (i) strongly relies upon the effectiveness of the pre-trained diffusion model, and (ii) remains specific to image data. Nevertheless, the authors can better discuss and highlight such limitations, while the paper does a good job when it comes to the experimental analysis. Overall, also in light of the positive feedback from the reviewers after the rebuttal, I can recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes SampDetox, a novel black-box backdoor defense method using perturbation-based sample detoxification. The key contributions are:

1. A preliminary study revealing the correlation between trigger visibility and poisoned sample robustness.
2. A two-stage defense approach combining lightweight global noise and targeted local noise to destroy both visible and invisible backdoor triggers.
3. Use of diffusion models to denoise and restore samples after perturbation.
4. Comprehensive experiments demonstrating effectiveness against various state-of-the-art backdoor attacks.
strength_weakness: strength_weakness: 1. The effectiveness of SampDetox heavily depends on the correlation between trigger visibility and robustness, but this relationship may not always hold for more sophisticated attacks.
2. The two-stage approach with fixed noise levels (t1 and t2) may be too rigid to adapt to diverse trigger patterns, potentially missing subtle backdoors or over-perturbing clean regions.
3. The approach may remove benign high-frequency details from images, potentially affecting classification accuracy for tasks that rely on such fine-grained features.
questions: questions: 1. The paper suggests optimal values for t1 and t2 based on your experiments. How sensitive is the performance of SampDetox to these parameters across different datasets or attack types? Is there a way to automatically determine optimal values for a given scenario?
2. How does the pixel-wise SSIM calculation in Stage 2 scale with image size? Is there a more efficient way to locate high-visibility triggers without compromising effectiveness? Moreover, how does the method perform on very large images or high-resolution datasets? Is there a practical upper limit on image size due to computational constraints?
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper introduces a novel black-box backdoor purification technique consisting of two stages. In the first stage, it eliminates global perturbation by adding lightweight noise to the inputs and then leverages DDPM for denoising. Subsequently, to address visible and robust triggers, it identifies the local trigger regions, applies intense noise, and then performs diffusion denoising again.
The paper provides theoretical analysis to validate the effectiveness of its detoxification process.
In the evaluation, extensive experiments are conducted on multiple datasets, model architectures, and attacks, showing that the proposed method outperforms existing baselines. An ablation study justifies the design choices and hyper-parameters, and the paper discusses adaptive attack scenarios.
strength_weakness: strength_weakness: 1. Lacks evaluation of some semantic backdoors.
2. Concerns about computational overhead for deployment.
3. Evaluations of adaptive attacks are relatively weak.
questions: questions: The paper presents an innovative concept for black-box input detoxification, highlighting noteworthy observations and techniques. However, I have several concerns I hope the authors can address:

(1) The evaluation includes 10 known backdoor attacks, which are not particularly robust. More complex backdoors utilize semantic features, for instance, [1] uses natural reflection as a trigger, and [2] employs style transferring. It would be beneficial to see evaluations against these more sophisticated backdoors.

(2) The overhead is a concern. Although Section 5.4 suggests using DDIM to reduce the overhead, it is unclear if this significantly affects the detoxification performance. Additional results on this would be helpful.

(3) The adaptive attacks discussed in Section D.2 seem relatively weak, just introducing some noise during training. A stronger attack might use natural features as triggers to enhance the robustness of BadNets. For instance, [3] concatenates two natural images as the backdoor trigger. It is suggested that the authors discuss this and potentially include stronger adaptive attacks.

Reference:

[1] Liu, Yunfei, et al. Reflection backdoor: A natural backdoor attack on deep neural networks. ECCV'20.

[2] Cheng, Siyuan, et al. Deep feature space trojan attack of neural networks by controlled detoxification. AAAI'21

[3] Lin, Junyu, et al. Composite backdoor attack for deep neural network by mixing existing benign features. CCS'20.
limitations: limitations: Please refer to the question part.
paper_summary: paper_summary: This paper proposes a two-stage black-box method to defend against the backdoor in DNN. The first stage mitigates the backdoor triggers with low  visibility and the second stage mitigates the trigger with higher robustness.
strength_weakness: strength_weakness: 1. One major concern about this work is that I think this work may not be effective against high-level semantic triggers. For example [1] adds the digital and physical triggers to the image, and make sure they can be preserved after gaussian blur or print and recapture. This work may not be effective against such triggers. Similar works can be found in [2,3]. However, I found that the authors have already mentioned  and acknowledged this in the paper. So I think this is not a big issue.
2. The use of diffusion model is not well introduced. It may not be easy for the readers to understand this part if not familiar with the diffusion model. Also, in appendix A.2, the diffusion model from previous work is used, but the authors did not provide the details of the diffusion model. I think it would be better if the authors can provide more details about the diffusion model. Also, would it have any impact if this used diffusion model is replaced by other pre-trained models?


[1] Versatile Backdoor Attack with Visible, Semantic, Sample-specific and Compatible Triggers

[2] Backdoor Attacks Against Deep Learning Systems in the Physical World

[3] Dangerous Cloak: Natural Backdoor Attacks on Object Detector in Real Physical World
questions: questions: Would using another pre-trained model have an influence on your work?
limitations: limitations: The authors discussed about the main concern I have and acknowledged the limitation for semantic backdoor triggers.
paper_summary: paper_summary: The paper addresses the significant issue of backdoor defense in a black-box setting during the inference phase of MLaaS applications. It proposes a novel technique called SampDetox, where the method uses diffusion models for detoxification by injecting noise to disrupt backdoors and retrieve clean instances.
strength_weakness: strength_weakness: 1. The research appears to make rather robust assumptions for diffusion models. The authors assume that the clean regions of the poisoned samples can be easily restored, while the poisoned regions, especially highly visible triggers, may fail to do so (as empathized in Line 201-202). This could be because the diffusion model learns only from the clean samples and fails to learn backdoor information. While the paper conducts supplemental experiments demonstrating strong defense performance even under other diffusion models (as shown in **Table 8**), the generalizability from ImageNet to CIFAR-10 seems less convincing. If the diffusion model is backdoored or the pre-trained diffusion model is trained on out-of-distribution samples, it could potentially dampen the final defense performance.

2. Furthermore, the method is less likely to be effective if physical attacks are deployed, where natural objects are used as backdoor triggers, as diffusion could learn features of these natural triggers.
questions: questions: Even though there is a theoretical substantiation for the working defense, I wonder about the universality of this theory in real-world applications. Specifically, in varying datasets and models, would the hyper-parameters \bar{t}_1 and \bar{t}_2 require fine-tuning for different noise levels? While **Figure 5** provides an ablation study for different \bar{t}_1 and \bar{t}_2, it does not display the specific setting. It would be intriguing to know whether different \bar{t}_1 and \bar{t}_2 were utilized for the results in **Tables 1, 3, 4, 5, and 6**.
limitations: limitations: Although the paper acknowledges its limitations, a major overlooked limitation lies in the pre-trained diffusion model. Would the detoxifying effect still be substantial if the diffusion model used is trained on out-of-distribution samples compared to the target dataset? This question presents an area of potential further exploration to validate the robustness of the proposed SampDetox method.
paper_summary: paper_summary: The paper addresses the issue of backdoor threats in third-party Machine Learning as a Service (MLaaS) applications. The authors propose a novel two-stage method called SampDetox to defend against both visible and invisible backdoor attacks without requiring access to the model's parameters or training data. The proposed method uses diffusion models to introduce noise into samples to destroy backdoors and recover clean samples. It combines lightweight and high noise strategically to handle backdoors with varying visibility and robustness. The paper provides mathematical proofs to support the effectiveness of SampDetox in eliminating backdoor triggers and restoring original samples. Comprehensive experiments demonstrate SampDetox's effectiveness against various state-of-the-art backdoor attacks. The method is evaluated using different datasets and models, showing its superiority over existing defense methods in terms of clean sample accuracy, poisoned sample accuracy, and attack success rate.
strength_weakness: strength_weakness: **Summary of Key Weaknesses:**

1. **Dependency on Pre-trained Models**:
   - *SampDetox's* effectiveness is contingent upon the quality of pre-trained diffusion models utilized for the denoising process. This poses a limitation, particularly when these models are not finely tuned for the specific data in question. While the authors have successfully demonstrated *SampDetox* using an ImageNet-trained model to counter attacks on CIFAR10, the real-world applicability may be limited due to the difficulty of accessing a pre-trained model that matches the unique dataset of the application. 

2. **Trigger Recovery Assumption**:
   - The paper presupposes that clean regions in poisoned samples can be easily restored, whereas poisoned regions, especially with visible triggers, cannot (as stated on Line 201). This assumption may not hold against sophisticated attacks like physical attacks where triggers are inherent and visible parts of the sample. Furthermore, this assumption could be invalidated if the image features an object that the pre-trained diffusion model is incapable of accurately recovering.

3. **Omission of Experimental Details**:
   - The paper falls short in providing certain critical experimental details, particularly regarding the baseline defense methods (such as the diffusion model configuration for ZIP) and the proposed method itself ($\bar{t}_1$ and $\bar{t}_2$ for different dataset).
questions: questions: The Questions can be summarized as follows:

1. **Question on Figure 6**: The authors compare the additional time overhead for various methods in Figure 6, noting that ZIP's overhead exceeds SampDetox's by more than 20 times. Given that ZIP utilizes DDPM once for image purification, whereas SampDetox employs it twice, this discrepancy seems disproportionate, especially if both are configured for the same denoising step count.  If ZIP and SampDetox use different configurations of DDPM, such comparison in overhead and defense performance may not be fair in some sense, as the configuration of DDPM plays an essential role in the recovery performance. Clarification on this point is requested.

2. **Lack results for SampDetox with DDIM**: In section 5.4, the authors assert that integrating DDIM with SampDetox yields inference times comparable to unguarded methods. However, specific performance metrics for SampDetox, when augmented with DDIM, are absent. To substantiate this claim, it is imperative to demonstrate that SampDetox+DDIM can match the defense efficacy of SampDetox+DDPM.

3. **Question on Table 5**: Table 5 presents ablation study outcomes for varying parameter selections. It remains ambiguous whether the data pertain to a single specific attack scenario or represent an average across multiple attack types.

4. **The stability of parameters across different datasets**: Does the suggest parameter for $\hat{t}_1=20$ and  $\hat{t}_2=120$ works consistently across different datasets?

I will raise my score if the author can address my concerns.
limitations: limitations: The authors have acknowledged certain limitations but have not sufficiently addressed the challenges associated with accessing pre-trained diffusion models. This is a critical issue that requires further exploration. A primary concern is the need for additional experiments to validate the approach using diffusion models trained on varied datasets. For instance, testing a model trained on ImageNet to defend against attacks on the CelebA dataset would provide valuable insights into the model's adaptability and robustness (Just as a suggestion/example).

Comments: title: Update
comment: Dear Reviewer aKFd,

Thank you once again for taking the time to review our work. We sincerely appreciate your thorough feedback and valuable insights. The rebuttal response will be included in the final version of the paper.

Best Regards,

The authors of 8771
Thank the authors for their detailed rebuttal. All my concerns have been addressed, so I am raising my score accordingly.
Dear Reviewer 2sKD,

Apologies for the inconvenience, but was your comment intended for a different paper? We are confused since our work does not involve “permutation triggers” or “traditional longer adversarial prompts” in your comment.

Best Regards,

The authors of 8771
Dear Reviewer 2sKD,

Thank you once again for taking the time to review our work. We sincerely appreciate your thorough feedback and valuable insights. The discussions will be included in the final version of the paper.


Best Regards,

The authors of 8771
The author's responses to W1 and W2 addressed my concerns. Thus I raised the rating. I recommend incorporating these discussions into the final version, as this would make the paper more comprehensive and persuasive.
Dear Reviewer gGGP,

Thank you once again for taking the time to review our work.. We sincerely appreciate your thorough feedback and valuable insights. We will detail the diffusion model and include the rebuttal response in the final version.

Best Regards,

The authors of 8771
Dear Reviewer ok4k,

Thank you once again for your efforts in helping us improve our work. We sincerely appreciate your thorough feedback and valuable insights. The rebuttal response will be included in the final version of the paper.

Best Regards,

The authors of 8771
Thanks for the clarification and additional evaluation. I think the paper can be improved with a detailed explanation of the diffusion model and ablation study. Thus, I would raise my score to 6.
Thanks for the additional clarification and evaluation!

I think they make more sense. It is likely the poisoned models of Refool or DFST pay a lot attention on some low-level features (although they look natural) that are not robust to the input purification.

I increase my rating and hope the authors could properly include the rebuttal reponse to their final version.
Dear Reviewer i7qR,

Thank you once again for taking the time to review our work. We sincerely appreciate your thorough feedback and valuable insights. We commit to releasing our code to the community.

Best Regards,

The authors of 8771
Dear Reviewer ok4k,

We apologize for the inconvenience, but due to limitations in the rebuttal system, we are unable to provide purified samples of the two attacks in the form of images. Meanwhile, we are prohibited from providing links as per the discussion rule: ''Do not use links in your comments.'' Therefore, we attempt to describe the purified samples using words. Thank you for your understanding.

* For Reflect (also called Refool), the region of trigger patterns of the purified samples is obviously destroyed compared with the original poisoned samples.
* For DFST, the whole purified samples are slightly more blurred than the original poisoned samples, and the color style of purified samples is more similar to that of the original clean samples.

Then, we clarify the settings of perturbation coefficients in the experiments:
* Reflect generates a poisoned sample $x^{p}$ as $x^{p}= x+k \otimes x_{R}$, where $x$ is a clean sample, $x_{R}$ is a reflection image and $k$ is a convolution kernel. We optimized the value of $k$ according to the settings in its open-source code.
* DFST generates a poisoned sample $x^{p}$ as $x^{p}=(1-\alpha) x+\alpha x_{s}$, where $x$ is a clean sample, $x_{s}$ is a styled image, and $\alpha$ is the perturbation coefficient. We set $\alpha$ to 0.6 following the settings in its open-source code.

Additionally, to explain why SampDetox can defend against these two attacks, we conducted new experiments to calculate the robustness of their poisoned samples, respectively. For each attack, we randomly selected 50 poisoned samples on CIFAR-10 and calculated the robustness value of each sample following our robustness definition in Section 3.1 (Lines 140-146). 
For Reflect, the robustness of its poisoned samples has a mean value of 0.0759 and a standard deviation of 0.011. For DFST, the robustness of its poisoned samples has a mean value of 0.137 and a standard deviation of 0.019. From the experimental results, we can find that the robustness of the poisoned samples is low (ones with robustness<0.18, according to Figure 2). 
Therefore, when SampDetox adds enough noise to these samples, even if trigger patterns are partially restored by diffusion models, backdoors will not be successfully triggered.

We will add the images of purified samples and the above experiments in the final version of the paper. We hope our response has addressed your concerns. Thanks again for your efforts in helping us improve our work. 

Best Regards,

The authors of 8771
I appreciate the efforts made by the authors in addressing my concerns and questions!

It is expected that SampDetox may fall short in defending natural triggers, e.g., composite backdoors. The additional results make sense to me and provide insights.
However, I'm a bit surprising that SampDetox is so effective against Reflect and DFST. It is reasonable to expect these attacks to introduce perturbations that do not align with other samples in the training set (e.g., CIFAR-10). However, this does not necessarily mean that the diffusion process overlooks these features, especially since diffusion models are typically trained on large datasets encompassing a wide range of natural images (e.g., ImageNet).

I don't think it is quite practical to assume that the diffusion model is trained on the same dataset as the classifier due to the high computational costs involved to train a diffusion model besides the classifier.

I suspect that the perturbation coefficient used in the image might be set too low. The coefficient, represented by $\alpha$ in the trigger injection functions of Reflect or DFST, is defined as follows:
x$^{'}$ = (1-$\alpha$) x + $\alpha$ x, where x is the inputs.
So it is likely the trigger is not robust and easily removed.
Would it be possible for the authors to provide illustrations of the recovered images using SampDetox for the two attacks? If the rebuttal system does not support this, then it's fine.

Overall, the rebuttal response is well done.
The authors' rebuttal has addressed most of my concerns, so I have increased my score. 

I encourage the authors to release their code to the community in the future, which would be a valuable contribution to this field.