Title: Black-box Backdoor Defense via Zero-shot Image Purification
Venue: NeurIPS 2023
Accepted: yes
Keywords: backdoor defense

black-box defense

diffusion model
Abstract: Backdoor attacks inject poisoned samples into the training data, resulting in the misclassification of the poisoned input during a model's deployment. Defending against such attacks is challenging, especially for real-world black-box models where only query access is permitted. In this paper, we propose a novel defense framework against backdoor attacks through Zero-shot Image Purification (ZIP). Our framework can be applied to poisoned models without requiring internal information about the model or any prior knowledge of the clean/poisoned samples. Our defense framework involves two steps. First, we apply a linear transformation (e.g., blurring) on the poisoned image to destroy the backdoor pattern. Then, we use a pre-trained diffusion model to recover the missing semantic information removed by the transformation. In particular, we design a new reverse process by using the transformed image to guide the generation of high-fidelity purified images, which works in zero-shot settings. We evaluate our ZIP framework on multiple datasets with different types of attacks. Experimental results demonstrate the superiority of our ZIP framework compared to state-of-the-art backdoor defense baselines. We believe that our results will provide valuable insights for future defense methods for black-box models. Our code is available at https://github.com/sycny/ZIP.

Black-box Backdoor Defense via Zero-shot Image Purification

Abstract

Backdoor attacks inject poisoned samples into the training data, resulting in the misclassification of the poisoned input during a model's deployment. Defending against such attacks is challenging, especially for real-world black-box models where only query access is permitted. In this paper, we propose a novel defense framework against backdoor attacks through Zero-shot Image Purification (ZIP). Our framework can be applied to poisoned models without requiring internal information about the model or any prior knowledge of the clean/poisoned samples. Our defense framework involves two steps. First, we apply a linear transformation (e.g., blurring) on the poisoned image to destroy the backdoor pattern. Then, we use a pre-trained diffusion model to recover the missing semantic information removed by the transformation. In particular, we design a new reverse process by using the transformed image to guide the generation of high-fidelity purified images, which works in zero-shot settings. We evaluate our ZIP framework on multiple datasets with different types of attacks. Experimental results demonstrate the superiority of our ZIP framework compared to state-of-the-art backdoor defense baselines. We believe that our results will provide valuable insights for future defense methods for black-box models. Our code is available at https://github.com/sycny/ZIP .

1 Introduction

Machine learning has been increasingly integrated into real-world applications such as healthcare [45], finance [28] and computer vision [66]. Despite great success, machine learning models are susceptible to adversaries such as backdoor attacks [15, 8, 43, 54, 17], which compromises model security and reliability. Backdoor attacks can manipulate a model's behavior by injecting malicious samples into the training data or altering the model's weights. Although many defense strategies have been proposed to mitigate backdoor attacks, most of them require access to the model's internal structure and poisoned training data [32]. Deploying these defenses is challenging in real-world black-box scenarios, where defenders do not have access to verify or audit the inner workings of the model [18, 19]. For example, many developers and end-users now prefer using machine learning as a service (MLaaS), relying on models provided by third-party vendors for their applications. However, these models may contain backdoors, and due to copyright concerns, the services typically operate in a black-box setting with query-only access. In such scenarios, detecting and mitigating backdoor attacks is very difficult.

Currently, there are only a few backdoor defense methods that work in black-box settings. They can be categorized into two types: detecting-based and purification-based. Detecting-based methods [65,

18, 31, 12, 56] can detect the poisoned sample but could not remove the poisoned patterns. These methods are not applicable when critical poisoned samples must be used by downstream classification models. Purification-based methods can address this problem since they aim to retrieve clean images given the poisoned images. However, state-of-the-art purification approaches rely on masking and reconstructing the poisoned area [42, 53]. It can only protect against patch-based attacks. Other purification-based methods that employ image transformations for defense have the potential to defend against more sophisticated attacks but may result in a reduction in classification accuracy due to the loss of semantic information [35].

To overcome these challenges, we propose a novel framework to defend against attacks through Zeroshot Image Purification (ZIP). We define " purification " as the process of maximizing the retention of important semantic information while eliminating the trigger pattern. With this goal, we preserve the classification accuracy while breaking the connection between trigger patterns and poisoned labels. We also define " zero-shot " as the ability to defend against various attacks without relying on prior knowledge of attack methods. In other words, our approach does not require access to any clean or poisoned image samples (patch or non-patch based), and could be applied directly to unseen attack scenarios. This setting is crucial because real-world users usually have limited information, while new threats continue to emerge. Our proposed framework contains two main stages: we first utilize image transformation to destruct the trigger pattern, and then leverage an off-the-shelf, pre-trained diffusion generative model to restore the semantic information. Our defense strategy is based on the motivation that the semantic information in a poisoned image (e.g., faces, cars, or buildings) constitutes the majority of the data and typically falls within the training data distribution of a pre-trained image generation model. In contrast, engineered trigger patterns (e.g., mosaic patches and colorful noise) are subtle and unlikely to exist in the pre-training datasets [32, 34]. Since the diffusion model learns to sample images from the training distribution [22], purified images generated from the diffusion model will retain only their semantic information while eliminating the trigger patterns. As a result, our purification approach can effectively defend against various attacks while maintaining high-fidelity in the restored images.

Our main contributions are summarized as follows. (1) We develop a novel defense framework that can be applied to black-box models without requiring any internal information about the model. Our method is versatile and easy to use without retraining. (2) The proposed framework is designed for zero-shot settings, which do not require any prior knowledge of the clean or poisoned images. This feature relieves end-users from the need to collect samples, thus enhancing the framework's applicability. (3) Our defense framework achieves good classification accuracy on the purified images, which were originally poisoned samples, even when using an attacked model as the classifier. This improvement further enhances the framework's effectiveness and usability.

2 Preliminaries

2.1 Problem Definition

This paper addresses the backdoor defense problem in the context of image classification tasks. The goal of image classification is to learn a function f θ ( x ) that maps input images x ∈ X to their correct labels y ∈ Y , where X denotes the image space, Y denotes the label space, and θ represents model parameters. Typical backdoor attacks include poisoning training data and perturbing model weights, and we use f attack θ to denote the model that has been attacked. During inference, an attacker can take a clean sample x and manipulate it to create a backdoor sample x P , e.g., adding the trigger pattern p to make x P = x + p . The backdoored model will misclassify x P as the target label y target = f attack θ ( x P ) = y . Our threat model is in a challenging black-box setting, where defenders can only query the poisoned model and have no access to the model's internal parameters or training datasets. The attackers can modify model components or any other necessary information to implement their attacks. We formally define our backdoor defense problem as below.

̸

̸

Problem 1. Image Purification for Backdoor Defense. Our defense is implemented in the model inference stage. Let f attack θ denote the attacked model, whose parameters are not accessible. Given a poisoned image x P , our goal is to obtain a purified image x ∗ from x P by removing the effect of trigger p . The purified image should be classified as the same category as the original clean image x , i.e., f θ ( x ∗ ) = f θ ( x ) = f θ ( x P ) .

x

!

x

"

x

!"#

x

#

. . .

. . .

𝐱

𝑷

𝐱

𝐀

!

𝐱

𝐀

Stage 1

Stage 2

Under-Attack

𝐀

!

𝐀

x

#

Qt

0+(+

at

~N(0,1)

2.2 Diffusion Models

We leverage the reverse process of diffusion models to purify images. The denoising diffusion probabilistic model (DDPM) [22] is a powerful generative model for generating high-quality images. It has two processes: a forward process and a reverse process. In the forward process, the model iteratively adds noise to an input image x 0 until it becomes random Gaussian noises x T ; in the reverse process, the model iteratively removes the added noise from x T to recover the noise-free image x 0 . More details can be found in the DDPM paper [22].

Forward Process: A noise-free image x 0 is transformed to a noisy image x t with controlled noise. Specifically, Gaussian noise ϵ is gradually added to image x 0 in T steps based on a variance schedule β t ∈ [0 , 1] such that q ( x t | x t -1 ) := N ( x t ; √ 1 -β t x t -1 , β t I ) , where q denotes the posterior probability of x t conditioned on x t -1 . A nice property of this process is that the t -th step noisy image x t could be directly generated by:

Reverse Process: The noisy input image x T is transformed into a noise-free output image x 0 over time steps. In each step, the diffusion model takes the current image state x t as input and produces the previous state x t -1 . We aim to obtain clean images x 0 by iteratively sampling x t -1 from p ( x t -1 | x t , x 0 ) :

Based on Equation 1, the original clean image x 0 could be approximated based on the t -th step observation x t as: x 0 | t = 1 √ ¯ α t ( x t - √ 1 -¯ α t ϵ t ) , where ϵ t denotes the estimation of the real ϵ in step t . In each step t , DDPM utilizes a neural network g ϕ ( · ) to predict the noise ϵ t , i.e., ϵ t = g ϕ ( x t , t ) . With this estimation, we can convert Equation 2 into the following form as reverse process:

3 Zero-shot Image Purification (ZIP)

3.1 Overview of Proposed Framework

Our defense framework consists of two primary stages: (1) destruct poisoned images through image transformation, and (2) recover images using a diffusion model, as depicted in Figure 1. The first stage is based on the observation that the integrity of trigger patterns is crucial for backdoor attacks to deceive the model. Thus, destructing the trigger pattern would significantly reduce the effectiveness

of backdoor attacks [34, 46]. On the other hand, a strong transformation may destroy both the trigger pattern and semantic information. To address this, we introduce the second stage to recover the semantic information through the reverse process of diffusion models. However, traditional diffusion models (e.g., DDPM [22]) or image restoration models (e.g., DDRM [29] and DDNM [58]) can not generate high-fidelity and clean images due to a lack of direct control over the generated output.

To bridge the gap, in the following section, we first propose an image generation constraint based on the image transformation in Section 3.2. We then apply this constraint to guide the image reverse process of the diffusion model. In addition, we discuss the guide adaptation when applied in the zero-shot settings, along with its theoretical justification in Section 3.3. Finally, we introduce our efforts to improve the inference speed in Section 3.4.

3.2 Image Transformation and Decomposition

In the first stage, we apply image transformation to destruct potential trigger patterns, such as using average pooling to blur the poisoned images. Formally, we denote the transformation as a linear operator A , and let the transformed image be x A = Ax P = A ( x + p ) . Previous research [34, 47] has used the transformed image directly as the purified result. However, such approaches may result in poor classification accuracy due to the loss of fidelity induced by A .

To recover the lost information, an intuitive way is to apply an image generative model, e.g., the diffusion model, to yield a purified image. However, the vanilla diffusion model generates images from random Gaussian noise and lacks control over the fidelity of generated images. Thus, we propose a constraint to guide the generation process of diffusion models to recover high-fidelity images. Specifically, for an ideally purified image x 0 , it should satisfy x 0 = x , so we have:

Based on the RND theory [48, 58], it is possible to decompose an image x into two parts using a linear operator A (e.g., average pooling) and its pseudo-inverse A † (e.g., upsampling) that satisfies AA † A = A . The decomposition is expressed as x = A † Ax + ( I -A † A ) x , where the former part denotes the observable information in the range-space, and the latter part denotes the information in the null-space removed by transformation 1 . Bringing this decomposition to Equation 4, we have:

This equation derives a constraint for image purification to restore the original x as below:

accordingly, the ideally purified image could be decomposed into three parts. The first two parts are in the range space: the observable information stored in the transformed image A † x A , and the intractable information embedded in the transformed trigger pattern A † Ap ; the last part is in the null-space and is unobservable as it is removed by the transformation. To restore the lost information in the null-space, we utilize the observable information in the range-space as references.

3.3 Image Purification with Diffusion Model

3.3.1 Reverse Process Conditioned on Poisoned Images

Our proposed image purification is based on the reverse process of the diffusion model, which takes Gaussian noise x T as input and generates a noise-free image x 0 . We use x t to denote the image at time step t in the reverse process of diffusion. To generate high-fidelity images, we propose a rectified estimation of x t as x ′ t , so that it produces x 0 | t that satisfies the decomposition constraint in Equation 6. The x ′ t is computed using the following equation, which is derived from Equation 1 and 6. The detailed proof is in the Supplementary Material A.

̸

where ϵ t denotes the estimated noise, which is calculated using the pre-trained diffusion model g ϕ : ϵ t = g ϕ ( x t , t ) . Then, we modify the original reverse process in Equation 3 to accommodate this rectified estimation. The modified reverse process is expressed as:

where ϵ ∼ N (0 , I ) denotes Gaussian noise and we have σ 2 t = 1 -¯ α t -1 1 -¯ α t β t .

3.3.2 Adapting Reverse Process to the Zero-shot Setting

In this subsection, we show how our reverse process design can be effectively applied in the zero-shot setting, where the trigger pattern p is unknown to defenders. We propose to omit the intractable term √ ¯ α t A † Ap in Equation 7, and approximate x ′ t with ˆ x t to obtain a new rectified estimation: √ √

√

The intractable term ¯ α t A † Ap can be omitted due to the following reasons:

The value of √ ¯ α t at the beginning of the reverse process is very small, making √ ¯ α t A † Ap negligible compared to other terms. We provide an improved approximation in Section 3.3.3 for later stages when √ ¯ α t increases.

Backdoor attacks are generally stealthy or use more negligible patterns compared to the original images since the attacker wants to minimize the impact on the model's accuracy on legitimate data [43]. Thus, the destructed pattern Ap is usually negligible compared to x A = A ( x + p ) .

The effect of A † Ap is further reduced by selecting appropriate image transformations. Most backdoor attacks are characterized by severe high-frequency artifacts [65]. Therefore, transformations such as average pooling can remove the high-frequency information in p .

Nevertheless, approximation in an iterative process such as diffusion can be risky because errors from the previous step can accumulate rapidly (e.g., exponentially) and lead to significant inaccuracies. Our method addresses this issue by ensuring the approximation error between each step is well-bounded theoretically. As a result, the final recovered image ˆ x 0 preserves the essential information of the original image, while the trigger pattern undergoes a transformation and is likely to be destroyed.

Lemma 3.1. Suppose the estimated noise output by g ϕ ( · ) is Gaussian. Given g ϕ ( x t , t ) = ϵ t , we have g ϕ (( x t + √ ¯ α t A † Ap ) , t ) = ϵ t + ϵ ′ t , where ϵ t , ϵ ′ t are also Gaussian.

The error ϵ ′ t in Lemma 3.1 is much smaller than the real estimated noise ϵ t . This is because the trigger pattern √ ¯ α t A † Ap , which is weighted by √ ¯ α t , is relatively subtle compared with the intermediate image x t . Additionally, the attacker-engineered trigger pattern p is unlikely to be present within the natural image distribution learned by the pre-trained diffusion model. Therefore, it is unlikely to activate the model g ϕ ( · ) and generate significant output.

Theorem 3.2. Suppose that g ϕ (( x t + √ ¯ α t A † Ap )) = ϵ t + ϵ ′ t . We define the error at step t between ˆ x t and ( x t + √ ¯ α t A † Ap ) as δ ′ t , i.e., δ ′ t = ˆ x t -( x t + √ ¯ α t A † Ap ) . Let δ t = A δ ′ t , we have the following bound on its norm: ∥ δ t ∥ ≤ (1 -¯ α t ) √ α t +1 √ 1 -¯ α t +1 ∥ A ∥∥ ϵ ′ t +1 ∥ .

This theorem means that our proposed approximation introduces only limited approximation error at each time step after considering the error from the previous step. When paired with a common linear transform A like average pooling, its norm is relatively small [58]. Additionally, at the initial steps of the reverse process, the term ∥ ϵ ′ t +1 ∥ is also small according to Lemma 3.1, and we have (1 -¯ α t ) √ α t +1 √ 1 -¯ α t +1 ≤ 1 -¯ α t +1 + α t 2 , which is also a small value. Collectively, the error from our approximation is well bounded by a small number. According to this theorem, the final purified image exhibits the following property.

Corollary 3.2.1. When t = 0 , we have ˆ x 0 = x 0 + A † Ap + δ ′ 0 , where A δ ′ 0 = 0 .

According to Corollary 3.2.1, our final purified image contains three parts: the ideally purified image x 0 , the altered trigger pattern A † Ap , and an approximation error δ ′ 0 . Therefore, compared with the original poisoned image x P = x + p or the transformed image x A = A ( x + p ) , our purified image ˆ x 0 reduces the effect of the trigger pattern p and preserves the high fidelity of the image. This is critical for achieving good performance in the downstream image classification task. The proofs can be found in Supplementary Material A.

3.3.3 Improving ZIP with Theoretical Insights

Based on the above theoretical analysis, we propose two further improvements to our guided image purification framework to enhance its effectiveness, which include (1) adding a confidence score to the rectified estimation, and (2) introducing multiple transformations.

Confidence Score. The effectiveness of our framework depends on the confidence of the proposed rectified estimation. As the reverse process proceeds, the √ ¯ α t value increases, making it difficult for √ ¯ α t A † Ap to be neglected. Simply omitting this pattern would weaken the confidence in our proposed estimation.

To address this issue, we re-formulate the rectified estimation at step t as: ˜ x t = (1 -¯ α λ t )ˆ x t + ¯ α λ t x t , where x t denotes the result of a pure diffusion model (e.g., DDPM) at step t +1 , and λ ≥ 0 is a hyper-parameter. Since 0 < ¯ α t < 1 , when λ = 0 , we obtain a pure reverse diffusion process without any rectification; when λ = + ∞ , the reverse process reduces to ZIP. A nice property of ˜ x t is that, ¯ α t increases as the reverse process proceeds ( t decreases), making the contribution of x t larger. This is reasonable because the intermediate image becomes increasingly informative over time, reducing the reliance on ˆ x t in reverse diffusion. We set λ value to achieve desirable fidelity scores (e.g., PSNR [29, 58]) for the resultant purified images. Finally, our revised reverse process is defined as:

Multiple Transformations. To improve the effectiveness of our approach in removing the poisoning effect, we propose including multiple transformations to more effectively destroy the unknown trigger pattern in a zero-shot setting. Specifically, given N transformations, the intermediate image x t and estimated noise ϵ t should all satisfy:

Finally, we have ˆ x t = 1 N (ˆ x 1 t + ˆ x 2 t + ... + ˆ x N t ) . We provide our proposed algorithm in Algorithm 1. We focus on two types of transformation pairs in this paper: blurring, represented by A † 1 A 1 , and gray-scale conversion, represented by A † 2 A 2 .

3.4 Purification Speed-up

The purification speed is crucial when performing defense at inference time. Our framework leverages pre-trained diffusion models, and conducts purification on each test image. Hence, we propose several techniques to speed up purification and reduce costs.

3.4.1 Diffusion Model Inference Speed-up

Algorithm 1 requires a large number of steps to generate a single sample. Each step involves computing the estimated noise and diffusion process, which can be computationally expensive. To address this issue, we leverage the power of the denoising diffusion implicit model (DDIM) [50] to improve the inference speed of our reverse process as below:

where ˜ x 0 | t is also an estimated image based on ˜ x t , and we have ˜ x 0 | t = 1 √ ¯ α t (˜ x t - √ 1 -¯ α t ϵ t ) . By using

7:

8:

9:

10:

11:

12:

Algorithm 1 Zero-shot Image Purification

Require: Test image for purification x P ; linear transformation A 1 , A 2 , ..., A n and their pseudo-inverse A † 1 , A † 2 , ..., A † n ; diffusion model g ; hyperparameter λ .

Ensure:

, n

1:

x

T

2:

3:

4:

=

n

x

A

∼ N

(

,

0

t

=

ϵ

ϵ

A

I

T, T

∼ N

t

0

(

=

ϕ

)

-

,

I

x

t

g

(

1

x

P

, ...,

)

1

= 0

do if

t >

, t

)

n

ϵ

t

α

¯

(ˆ

1

t

x

-

t

+ ˆ

¯

α

2

t

x

λ

1

α

t

t

t

x

)ˆ

(˜

x

t

√

1

, else

-

n

A

1

end for

ˆ

t

x

=

˜

t

x

1

N

= (1

x

t

1

-

end for return

x

0

this speed-up inference method, we can obtain a high-fidelity purified image by sampling only a few steps instead of conducting sampling in thousands of steps. The modified algorithm based on DDIM is provided in the Supplementary Material D.

for

A

n

,

1

, ..., N

ϵ

=

0

+

, ...,

+ ¯

α

-

N

t

x

+ˆ

x

λ

t

√

1

-

1

t

t

α

-

¯

α

ϵ

t

t

)

) +

σ

t

ϵ

3.4.2 Framework Speed-up

Batch Data Speed-up. We introduce an acceleration method for purification, when images can be processed in batch during inference. For instance, if the pre-trained model accepts images of size L × L as input, and we wish to purify l × l images, we can first combine ( L/l ) 2 images into a single L × L image by tiling. We then apply our purification method to the tiled image, and split the purified image back to the original images afterwards. Larger images could be resized into l × l to apply this method. In this way, our defense approach can handle images of various sizes and significantly improve the purification speed.

Streaming Data Speed-up. In the streaming data scenarios, defenders may only have access to a single test image at a time. In such cases, we can use a fast zero-shot detection method such as [19] to quickly assess whether the image is likely to be poisoned. If the detection result indicates a potential poisoning, we can apply our purification method to remove the trigger pattern. This approach allows us to quickly and effectively defend against poisoned images without applying our purification on every image, thus saving the average processing time.

4 Experiments

4.1 Experimental Settings

For defense evaluation, we conducted experiments on three types of backdoor attacks: BadNet [15], Attack in the physical world (PhysicalBA) [34], and Blended [8]. The first two attacks are representatives of patch-based attacks, while the last one represents a non-patch-based backdoor attack. To configure these attack algorithms, we follow the benchmark setting in [33] and use the provided codes. We evaluate the effectiveness of our defense framework ZIP on three datasets: CIFAR-10 [30], GTSRB [52], and Imagenette [24]. The poisoned classification network is based on ResNet-34 [21]. Specifically, for the CIFAR-10 dataset, we apply both blur and gray-scale conversion as linear transformations. For the GTSRB and Imagenette datasets, we solely apply blur as the linear transformation. The additional experiment details are in the Supplementary Material E.

Defense methods available for black-box purification in zero-shot are rare. To fairly assess the effectiveness of our proposed method, we conduct a comparative evaluation against two baseline approaches: ShrinkPad [35] and image blurring (referred to as Blur). The former is a state-of-the-art image transformation-based defense method that can work on black-box models in the zero-shot setting. The latter uses blurred images A † 1 x A 1 as purified images. We apply defense methods to all test samples, and then evaluate the output using a poisoned classification network. In addition to the clean accuracy (CA) and attack success rate (ASR) metrics for assessing defense effectiveness, we introduce a new metric called poisoned accuracy (PA). It measures the classification performance of the purified poisoned samples. A higher value of PA indicates that the purified poisoned samples are more likely to be correctly classified, even when using an attacked classification model.

4.2 Qualitative Results of Purification

We conduct qualitative case studies (see Figure 2) of our method in purifying poisoned images created by the Blended and BadNet attacks. We show examples of poisoned images x P and purified image x 0 , where the trigger pattern is clearly visible in the poisoned images but has been altered/removed in the purified images. For comparison, we also show the blurred image A † 1 x A 1 and the grayscale images A † 2 x A 2 . The transformed images can destruct the trigger pattern, but they also alter a lot of semantic information. These results demonstrate the effectiveness of ZIP in removing the effect of trigger patterns from images while maintaining semantic information. More qualitative results for different attacks are in the Supplementary Material B.

4.3 Quantitative Results of Defense

We conduct quantitative experiments where the results are in Table 1. Our observations are as follows: (1) Our method effectively defends against different backdoor attacks. This is because the transformations in our framework, which are independent of specific attacks, break the connection between backdoor triggers and backdoor labels. (2) Our method achieves overall better classification

𝐱

!

𝐀

#

𝐱

𝐀

𝟏

𝐀

#

𝐱

𝐀

𝟐

𝐱

&

𝐱

𝟏

𝟐

accuracy (CA) compared to baseline methods because our formulation successfully recovers semantic information to ensure accurate downstream classification. For example, on the Imagenette dataset, ZIP reduces the ASR of the BadNet attack from 94.53% (no defense) to just 7.55%, with only a 0.94% drop in CA. Similarly, on the GTSRB dataset, our approach reduces the success rate of the PhysicalBA attack from 100% (no defense) to 6.57%, with only a 1.39% drop in CA. (3) Our method outperforms the baselines in poisoned accuracy (PA), indicating that poisoned samples can still be used for classification even when using an attacked black-box classifier. It demonstrates the robustness and usability of ZIP in real-world scenarios. (4) ShrinkPad performs poorly on the PhysicalBA attack, which is consistent with the findings in [34]. This is because PhysicalBA is specifically designed to evade defense methods such as ShrinkPad [34]. On the other hand, our method successfully defends against this attack, demonstrating its superior performance. We include additional experiment results of defenses in Supplementary Material H.

4.4 Ablation Studies

4.4.1 Evaluation on Transformations

Comparison of Different Transformations. Image transformation plays a crucial role in our defense. We evaluate the effectiveness of ZIP on the Imagenette dataset using four distinct types of transformations: Blur 2 , Blur 4 , Blur 8 , and Grayscale (the results are in Table 2). The subscript of "Blur" represents the kernel size of average pooling, where a larger number indicates a stronger transformation. We have the following observations: (1) The blur operation is more effective than grayscale conversion in reducing the ASR while maintaining good CA and PA. (2) Generally, stronger transformations are more effective in destructing the trigger pattern, resulting in a lower ASR. However, they also destroy more semantic information, leading to lower classification accuracy.

Comparison to Masking and Reconstruction.

Linear transformations are more effective than masking in removing poisoned effects, particularly in cases like the Blended attack where the backdoor pattern is distributed across the image. We compare ZIP with a recently proposed purification method BDMAE [53] on CIFAR-10.

BDMAE first identifies the trigger region on a test image, masks the region, and then uses a masked autoencoder to restore the image. To ensure a fair comparison, we apply the defense stage of BDMAE to our benchmarks using its official code. Table 3 shows that BDMAE effectively defends against the BadNet attack, but it fails to defend against the Blended attack. This is because the trigger pattern in Blended attacks is not located in a local region, making it difficult for BDMAE to identify an appropriate mask. In contrast, our ZIP framework does not make assumptions about trigger patterns thus successfully defends against both attacks.

4.4.2 Evaluation on Enhanced Attacks

We consider a more challenging scenario, where we assume the attacker is aware of our defense and has access to the purified backdoor images, which are then used as enhanced poisoned images to attack a classifier. To demonstrate the effectiveness of our proposed method in defending against such enhanced attacks, we apply our method to BadNet and Blended as examples and choose blurring and grayscale conversion together as the transformations of ZIP. Other settings remain the same as the original attack, and more details about the settings are in the Supplementary Material F.1.

During the inference stage, we continue to use the original trigger pattern for attacks. From the experimental results in Table 4, we observe that the original trigger pattern no longer triggers the attack, but if the purification (Blur+Grayscale) we use has the same transformation as the attacker, the attacks can still be triggered. However, by switching to a different transformation, such as solely Blur or Grayscale, the attacks can be effectively mitigated. It is important to note that in practical scenarios, the attacker may not know which transformation the defender is using, therefore the defender should consider using diverse transformations to enhance the defense ability.

4.4.3 Evaluation on Purification Speed

We analyze the computational costs of classification and purification before and after applying speed-up strategies. Figure 3 presents the average time required per image in CIFAR-10. Our defense method's efficiency can be enhanced with the introduction of our proposed tiling and detection-based strategies, resulting in a 33 × and 39 × speedup, respectively. The results also show that ZIP can complete the purification step faster than the classification step under various scenarios, highlighting its efficiency. In addition, our method is three times faster than another inference time purification model BDMAE [53]. More details about the speed-up settings are provided in the Supplementary Material F.2.

5 Related Work: Backdoor Defense

In this section, we briefly review backdoor defenses here and provide a more detailed discussion in the Appendix G. Existing backdoor defense methods are mainly designed for white-box models [32, 61]. However, these methods [47, 10] often require access to model parameters or original training data, which is not always feasible in real-world scenarios. To address this challenge, existing black-box defense methods are proposed and can be roughly divided into backdoor detection and backdoor purification. Detection models [18, 19, 65, 12, 38, 14, 56, 16] aim to identify and reject any detected poisoned images for further inference as a defense mechanism. However, this approach can limit the usefulness of these methods in practical settings where users expect results for all of their test samples. On the other hand, backdoor purification methods aim to remove the poison effect from the image to defend against attacks. Some methods [42, 53] in this category involve masking the potentially poisoned region and then reconstructing the masked image to obtain a poison-free image. However, these strategies may fail when the trigger patterns are distributed throughout the image, rather than in a specific patch-based location [1, 43, 68, 37]. Another approach [35] involves applying strong image transformations to the test image to destruct the trigger pattern. While such methods can defend against more advanced attacks, they typically result in a decrease in classification accuracy.

6 Conclusion

We propose a novel framework called ZIP for defending against backdoor attacks in black-box settings. Our method involves applying strong transformations to the poisoned image to destroy the trigger pattern. It then leverages a pre-trained diffusion model to recover the removed semantic information while maintaining the fidelity of the purified images. The experiments demonstrate the effectiveness of ZIP in defending against various backdoor attacks, without requiring model internal information or any training samples. ZIP also enables end-users to utilize full test samples, even when using an attacked classification model. Some future directions include designing black-box defense for other data domains and exploring other types of diffusion models.

Acknowledgments and Disclosure of Funding

The work is, in part, supported by NSF (#2223768, #2310261). The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agencies.

0.2095

0.0695

0.0585

0.0358

0.0358

second(s)

2.3164

0                   0.2                  0.5                  2.0

Classification

Classification

+Transformation

Classification

+DDIM

Classification

+DDIM

+Detection

Classification

+DDIM

+Tiling (64)

Classification

+BDMAE

+Batch (64)

0

0.2

0.4

0.6

References

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18208-18218, 2022.

Ruisi Cai, Zhenyu Zhang, Tianlong Chen, Xiaohan Chen, and Zhangyang Wang. Randomized channel shuffling: Minimal-overhead backdoor attack detection without clean datasets. Advances in Neural Information Processing Systems , 35:33876-33889, 2022.

Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.09667 , 2021.

Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. arXiv preprint arXiv:2207.04497 , 2022.

Kangjie Chen, Xiaoxuan Lou, Guowen Xu, Jiwei Li, and Tianwei Zhang. Clean-image backdoor: Attacking multi-label models with poisoned labels only. In The Eleventh International Conference on Learning Representations .

Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. Badpre: Task-agnostic backdoor attacks to pre-trained nlp foundation models. arXiv preprint arXiv:2110.02467 , 2021.

Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. Advances in Neural Information Processing Systems , 35:97279737, 2022.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 , 2017.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Bao Gia Doan, Ehsan Abbasnejad, and Damith C Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In Annual Computer Security Applications Conference , pages 897-912, 2020.

Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. Advances in Neural Information Processing Systems , 34:18944-18957, 2021.

Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection of backdoor attacks with limited information and data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16482-16491, 2021.

Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. arXiv preprint arXiv:1911.07116 , 2019.

Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference , pages 113-125, 2019.

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 , 2017.

Zihan Guan, Mengnan Du, and Ninghao Liu. Xgbd: Explanation-guided graph backdoor detection. arXiv preprint arXiv:2308.04406 , 2023.

Zihan Guan, Lichao Sun, Mengnan Du, and Ninghao Liu. Attacking neural networks with neural networks: Towards deep synchronization for backdoor attacks. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , CIKM '23, page 608-618, New York, NY, USA, 2023. Association for Computing Machinery.

Junfeng Guo, Ang Li, and Cong Liu. Aeva: Black-box backdoor detection using adversarial extreme value analysis. International Conference on Learning Representations , 2022.

Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. The Eleventh International Conference on Learning Representations , 2023.

Jonathan Hayase and Sewoong Oh. Few-shot backdoor attacks via neural tangent kernels. arXiv preprint arXiv:2210.05929 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handcrafted backdoors in deep neural networks. Advances in Neural Information Processing Systems , 35:8068-8080, 2022.

Jeremy Howard. Imagenette. https://github.com/fastai/imagenette , 2019.

Xiaoling Hu, Xiao Lin, Michael Cogswell, Yi Yao, Susmit Jha, and Chao Chen. Trigger hunting with a topological prior for trojan detection. arXiv preprint arXiv:2110.08335 , 2021.

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. arXiv preprint arXiv:2202.03423 , 2022.

Charles Jin, Melinda Sun, and Martin Rinard. Incompatibility clustering as a defense against backdoor poisoning attacks. In The Eleventh International Conference on Learning Representations .

Patanjali Kashyap. Machine learning for decision makers: Cognitive computing fundamentals for better decision making . Springer, 2017.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems , 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems , 34:14900-14912, 2021.

Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Yiming Li, Mengxi Ya, Yang Bai, Yong Jiang, and Shu-Tao Xia. Backdoorbox: A python toolbox for backdoor learning. arXiv preprint arXiv:2302.01762 , 2023.

Yiming Li, Tongqing Zhai, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the physical world. arXiv preprint arXiv:2104.02361 , 2021.

Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.04692 , 2020.

Yiming Li, Haoxiang Zhong, Xingjun Ma, Yong Jiang, and Shu-Tao Xia. Few-shot backdoor attacks on visual object tracking. arXiv preprint arXiv:2201.13178 , 2022.

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16463-16472, 2021.

Guanxiong Liu, Abdallah Khreishah, Fatima Sharadgah, and Issa Khalil. An adaptive black-box defense against trojan attacks (trojdef). IEEE Transactions on Neural Networks and Learning Systems , 2022.

Ninghao Liu, Mengnan Du, Ruocheng Guo, Huan Liu, and Xia Hu. Adversarial attacks and defenses: An interpretation perspective. ACM SIGKDD Explorations Newsletter , 23(1):86-99, 2021.

Yepeng Liu, Bo Feng, and Qian Lou. Trojtext: Test-time invisible textual trojan insertion. arXiv preprint arXiv:2303.02242 , 2023.

Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security , pages 1265-1282, 2019.

Brandon B May, Norman Joseph Tatro, Piyush Kumar, and Nathan Shnidman. Salient conditional diffusion for backdoors. In ICLR 2023 Workshop on Backdoor Attacks and Defenses in Machine Learning .

Anh Nguyen and Anh Tran. Wanet-imperceptible warping-based backdoor attack. International Conference on Learning Representations , 2021.

Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460 , 2022.

Ziad Obermeyer and Ezekiel J Emanuel. Predicting the future-big data, machine learning, and clinical medicine. The New England journal of medicine , 375(13):1216, 2016.

Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assumption of latent separability for backdoor defenses. In The eleventh international conference on learning representations , 2023.

Han Qiu, Yi Zeng, Shangwei Guo, Tianwei Zhang, Meikang Qiu, and Bhavani Thuraisingham. Deepsweep: An evaluation framework for mitigating dnn backdoor attacks using data augmentation. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security , pages 363-377, 2021.

Johannes Schwab, Stephan Antholzer, and Markus Haltmeier. Deep null space learning for inverse problems: convergence analysis and rates. Inverse Problems , 35(2):025008, 2019.

Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Constrained optimization with dynamic bound-scaling for effective nlp backdoor defense. In International Conference on Machine Learning , pages 19879-19892. PMLR, 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations , 2021.

Hossein Souri, Liam Fowl, Rama Chellappa, Micah Goldblum, and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. Advances in Neural Information Processing Systems , 35:19165-19178, 2022.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks , 32:323-332, 2012.

Tao Sun, Lu Pang, Chao Chen, and Haibin Ling. Mask and restore: Blind backdoor defense at test time with masked autoencoder. arXiv preprint arXiv:2303.15564 , 2023.

Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 218-228, 2020.

Guanhong Tao, Yingqi Liu, Guangyu Shen, Qiuling Xu, Shengwei An, Zhuo Zhang, and Xiangyu Zhang. Model orthogonalization: Class distance hardening in neural networks for better security. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1372-1389. IEEE, 2022.

Sakshi Udeshi, Shanshan Peng, Gerald Woo, Lionell Loh, Louth Rawshan, and Sudipta Chattopadhyay. Model agnostic defence against backdoor attacks in machine learning. IEEE Transactions on Reliability , 71(2):880-895, 2022.

Haotao Wang, Junyuan Hong, Aston Zhang, Jiayu Zhou, and Zhangyang Wang. Trap and replace: Defending backdoor attacks by trapping them into an easy-to-replace subnetwork. arXiv preprint arXiv:2210.06428 , 2022.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490 , 2022.

Zhenting Wang, Hailun Ding, Juan Zhai, and Shiqing Ma. Training with more confidence: Mitigating injected and natural backdoors during training. Advances in Neural Information Processing Systems , 35:36396-36410, 2022.

Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework. arXiv preprint arXiv:2304.02786 , 2023.

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. Advances in Neural Information Processing Systems , 35:10546-10559, 2022.

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems , 34:16913-16925, 2021.

Zhen Xiang, David J Miller, and George Kesidis. Post-training detection of backdoor attacks for two-class and multi-attack scenarios. arXiv preprint arXiv:2201.08474 , 2022.

Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. arXiv preprint arXiv:2110.03735 , 2021.

Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16473-16481, 2021.

Tao Zhang, Qing Li, Chang-shui Zhang, Hua-wei Liang, Ping Li, Tian-miao Wang, Shuo Li, Yun-long Zhu, and Cheng Wu. Current trends in the development of intelligent unmanned autonomous systems. Frontiers of information technology & electronic engineering , 18:68-85, 2017.

Zhiyuan Zhang, Lingjuan Lyu, Weiqiang Wang, Lichao Sun, and Xu Sun. How to inject backdoors with better consistency: Logit anchoring on clean data. arXiv preprint arXiv:2109.01300 , 2021.

Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on video recognition models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14443-14452, 2020.

Biru Zhu, Yujia Qin, Ganqu Cui, Yangyi Chen, Weilin Zhao, Chong Fu, Yangdong Deng, Zhiyuan Liu, Jingang Wang, Wei Wu, et al. Moderate-fitting as a natural backdoor defender for pre-trained language models. Advances in Neural Information Processing Systems , 35:10861099, 2022.

A Theoretical Justification

A.1 Proof on rectified estimation of x t in Equation 7

Equation 7 x ′ t = √ ¯ α t A † x A - √ ¯ α t A † Ap +( I -A † A ) x t + A † A √ 1 -¯ α t ϵ t .

Proof. Based on x t = √ ¯ α t x 0 + √ 1 -¯ α t ϵ , we have a nice property [22]:

To ensure that the approximated clean image x 0 | t based on the t -th step observation x t satisfies the constraint in Equation 6, we have:

Combine the above two equations, we can have:

Taking the derivative, we arrive at:

A.2 Proof of Lemma 3.1

Lemma 3.1 Suppose the estimated noise output by g ϕ ( · ) is Gaussian. Given g ϕ ( x t , t ) = ϵ t , we have g ϕ (( x t + √ ¯ α t A † Ap ) , t ) = ϵ t + ϵ ′ t , where ϵ t , ϵ ′ t are also Gaussian.

Proof. Let us define the output of g ϕ (( x t + x ′ t ) , t ) as ˆ ϵ t , so we have g ϕ (( x t + x ′ t ) , t ) = ˆ ϵ t . Next, we define ϵ t ∼ N ( µ 1 , σ 2 1 ) and ˆ ϵ t ∼ N ( µ 2 , σ 2 2 ) .

Since ˆ ϵ t also follows a Gaussian distribution, we can subtract ϵ t from ˆ ϵ t to obtain ϵ ′ t , such that:

This confirms that ϵ ′ t is also Gaussian, thus completing the proof.

A.3 Proof of Theorem 3.2

Theorem 3.2 Suppose that g ϕ (( x t + √ ¯ α t A † Ap )) = ϵ t + ϵ ′ t . We define the error at step t between ˆ x t and ( x t + √ ¯ α t A † Ap ) as δ ′ t , i.e., δ ′ t = ˆ x t -( x t + √ ¯ α t A † Ap ) . Let δ t = A δ ′ t , we have the following bound on its norm: ∥ δ t ∥ ≤ (1 -¯ α t ) √ α t +1 √ 1 -¯ α t +1 ∥ A ∥∥ ϵ ′ t +1 ∥ .

Proof. We prove the above theorem by induction.

1 The base case is when t = T , where we have ˆ x T -( x T + √ ¯ α T A † Ap ) = 0 , which holds.

3 Induction:

By definition, we have ˆ x t -1 -( x t -1 + √ ¯ α t -1 A † Ap ) = δ ′ t -1 . Let δ t -1 = A δ ′ t -1 , we have δ t -1 = A δ ′ t -1 = A (ˆ x t -1 -( x t -1 + √ ¯ α t -1 A † Ap )) . Since A ( I -A † A ) = 0 , we can have:

Based on the Cauchy-Schwarz inequality:

A.4 Proof of Corollary 3.2.1

Corollary 3.2.1 When t = 0 , we have ˆ x 0 = x 0 + A † Ap + δ ′ 0 , where A δ ′ 0 = 0 .

Proof. First, we have:

Then we can have:

Since α 0 = 1 , then we have A δ ′ 0 = 0 .

B Qualitative Results of Purification

B.1 Qualitative Results of Purification on BadNet Attack

(i) Attacked Image

(j) Purified Image

(k) Attacked Image

(l) Purified Image

(m) Attacked Image

(n) Purified Image

(o) Attacked Image

(p) Purified Image

(a) Attacked Image

(b) Purified Image

(c) Attacked Image

(d) Purified Image

(e) Attacked Image

(f) Purified Image

(g) Attacked Image

(h) Purified Image

(i) Attacked Image

(j) Purified Image

(k) Attacked Image

(m) Attacked Image

(n) Purified Image

(o) Attacked Image

(p) Purified Image

(q) Attacked Image

(r) Purified Image

(s) Attacked Image

(t) Purified Image

(u) Attacked Image

(v) Purified Image

(w) Attacked Image

(x) Purified Image

HXWOU

HXMON

JCROWN*.

BATTERIES

SETFBHES

(f) Purified Image

(i) Attacked Image

(j) Purified Image

(l) Purified Image

(n) Purified Image

(o) Attacked Image

(r) Purified Image

(t) Purified Image

(u) Attacked Image

(v) Purified Image

(w) Attacked Image

16008

PREAE

FUEAE

EARE

O

SI0GO

Sunray DxOil Co.

ACLA

SunrayDxOil Co.

HC

US

ASOLIL

ZSULIN

ROYAL

0

(a) Attacked Image

(b) Purified Image

(c) Attacked Image

(d) Purified Image

(e) Attacked Image

(f) Purified Image

(g) Attacked Image

(h) Purified Image

(i) Attacked Image

(j) Purified Image

(l) Purified Image

(m) Attacked Image

(n) Purified Image

(o) Attacked Image

(p) Purified Image

(q) Attacked Image

(r) Purified Image

(s) Attacked Image

(t) Purified Image

(u) Attacked Image

(v) Purified Image

(w) Attacked Image

(x) Purified Image

GULF

GULF

B.2 Qualitative Results of Purification on Blended Attack

(i) Attacked Image

(j) Purified Image

(k) Attacked Image

(m) Attacked Image

(n) Purified Image

(o) Attacked Image

Mobilo

SPECIAL

COSLBA

SPECIA

(a) Attacked Image

(b) Purified Image

(c) Attacked Image

(d) Purified Image

(e) Attacked Image

(f) Purified Image

(g) Attacked Image

(h) Purified Image

(i) Attacked Image

(j) Purified Image

(k) Attacked Image

(l) Purified Image

(m) Attacked Image

(n) Purified Image

(o) Attacked Image

(p) Purified Image

(q) Attacked Image

(r) Purified Image

(s) Attacked Image

(t) Purified Image

(u) Attacked Image

(v) Purified Image

(w) Attacked Image

(x) Purified Image

(e) Attacked Image

(f) Purified Image

(g) Attacked Image

(h) Purified Image

(i) Attacked Image

(j) Purified Image

(k) Attacked Image

(l) Purified Image

(n) Purified Image

(o) Attacked Image

(s) Attacked Image

(t) Purified Image

(u) Attacked Image

(v) Purified Image

(w) Attacked Image

(x) Purified Image

(a) Attacked Image

(b) Purified Image

(c) Attacked Image

(d) Purified Image

(e) Attacked Image

(f) Purified Image

(g) Attacked Image

(h) Purified Image

(i) Attacked Image

(j) Purified Image

(k) Attacked Image

(l) Purified Image

(m) Attacked Image

(n) Purified Image

(o) Attacked Image

(p) Purified Image

(q) Attacked Image

(r) Purified Image

(s) Attacked Image

(t) Purified Image

(u) Attacked Image

(v) Purified Image

(w) Attacked Image

(x) Purified Image

Dola

C Details on Linear Transformations

In this section, we discuss details of the linear transformation applied in our paper. Two practical examples are discussed to illustrate the simplicity and effectiveness of linear transformations, along with the corresponding operators used.

Gray-scale Conversion: To convert an RGB image to gray-scale, the operator A = [1 / 3 , 1 / 3 , 1 / 3] can be defined as a pixel-wise operation that transforms each RGB channel pixel [ r, g, b ] into a grayscale value r 3 + g 3 + b 3 . In this case, constructing a pseudo-inverse A † = [1 , 1 , 1] T is straightforward, satisfying the condition AA † ≡ I , where I represents the identity matrix.

Image Blurring: Image blurring also involves linear transformations. For a blurring operation with scale n , the operator A is defined as the average-pooling operator [1 /n 2 , ..., 1 /n 2 ] . This operator aggregates each patch of the image into a single value. Similarly, the pseudo-inverse A † can be built as A † = [1 , ..., 1] T to fulfill the condition AA † ≡ I .

Overall, these examples demonstrate how these two linear transformations, in conjunction with their respective operators, can be employed to destruct the trigger pattern without relying on a complex Fourier transform. In cases where the linear transformation is too complex to solve for its pseudoinverse, the Singular Value Decomposition (SVD) method can be applied. For more details, please refer to papers [58, 29].

D Algorithm for improved ZIP based on DDIM

In this section, we include the modified algorithm based on DDIM, which is proposed to speed up the diffusion model inference speed.

Algorithm 2 Zero-shot Image Purification (based on DDIM)

Require: Test image for purification x P ; liner transformation A 1 , A 2 , ..., A n and their pseudoinverse A † , A † , ..., A † ; diffusion model g ; hyperparameter λ ; speed-up pace S .

1 2 n Ensure: x A n = A n x P , n = 0 , 1 , ..., N 1: x T ∼ N ( 0 , I ) 2: for t = T, T -S, ..., S, 1 do 3: ϵ ∼ N ( 0 , I ) if t > 1 , else ϵ = 0 4: ϵ t = g ϕ ( x t , t ) 5: for n = 1 , 2 , ..., N do 6: ˆ x n t = √ ¯ α t A † 1 x A n +( I -A † n A n ) x t + A † n A n √ 1 -¯ α t ϵ t 7: end for 8: ˆ x t = 1 N (ˆ x 1 t + ˆ x 2 t + , ..., +ˆ x N t ) 9: ˜ x t = (1 -¯ α λ t )ˆ x t + ¯ α λ t x t 10: ˜ x 0 | t = 1 √ ¯ α t (˜ x t - √ 1 -¯ α t ϵ t ) 11: x t -1 ← √ ¯ α t -1 ˜ x 0 | t + √ 1 -¯ α t -1 -σ 2 t ϵ t + σ t ϵ 12: end for 13: return x 0

E Experiments Settings

E.1 Datasets Informaiton

CIFAR-10 [30] The CIFAR-10 dataset is a widely-used benchmark in computer vision. It consists of 60,000 color images of size 32x32 pixels, belonging to 10 different classes, with 6,000 images per class. The dataset is divided into 50,000 training images and 10,000 test images, with a balanced distribution of classes.

GTSRB [52] The German Traffic Sign Recognition Benchmark (GTSRB) dataset is designed for traffic sign classification tasks. It comprises more than 50,000 images of traffic signs captured under various real-world conditions. The images have different sizes and aspect ratios, but they are resized

to 32x32 pixels for our model training and evaluation. The dataset is divided into training and test sets, with its official split ratio.

Imagenette [24] The Imagenette is a subset of the larger ImageNet dataset and is commonly used as a smaller-scale alternative for image classification tasks. It consists of 10 classes with a total of 13,000 images. The images in Imagenette have varying sizes, but they are resized to 256x256 pixels for consistency. The dataset is split into training and validation sets, following a predefined split ratio.

E.2 Attacks Implementation

In this section, we discuss the implementation details of three different backdoor attack methods employed in our study: BadNet, Blended, and PhysicalBA. We implement these backdoor attacks using the Backdoorbox framework [33], which is under GNU general public license.

BadNet [15] The BadNet attack injects specific trigger patterns into the training data. In our implementation, we set the poisoned rate to 5%, i.e., 5% of the training samples are selected as attack samples and have the trigger pattern added to them. The trigger pattern size is set to 2x2 for 32x32 pixels images and 9x9 for 256x256 pixels images. The trigger patterns are randomly generated.

Blended [8] The Blended attack is a more sophisticated variant aimed at making the backdoor less conspicuous and harder to detect. Following the suggestion in BackdoorBox, we set the blended rate to 0.2 and the poisoned rate to 5%. The blended pattern is randomly generated, seamlessly blending the trigger pattern into the attack samples.

PhysicalBA [34] The PhysicalBA (Physical Backdoor Attack) is a specific type of attack that introduces variations in the location and appearance of the attack pattern embedded in the test samples during inference time. In our implementation, we apply the same attack pattern size as the BadNet attack, using a 2x2 pattern for 32x32 pixels images and a 9x9 pattern for 256x256 pixels images. The attack patterns are generated randomly. We set the poisoned rate to 5% for this attack.

All other attack settings follow the default configurations in Backboorbox [33].

E.3 Purification Implementation

We utilize a pre-trained model provided by OpenAI [9] under the MIT license. The algorithm described in Algorithm 2 is employed to accelerate the inference process, allowing us to generate high-quality images within just 20 steps, and the speed-up pace is set to 50 . Weset the hyperparameter λ to a value of 2 for Blended attack defense, and 10 for BadNet and PhysicalBA attack defense.

Specifically, for the CIFAR-10 dataset, we apply both blur and gray-scale conversion as linear transformations. For the GTSRB and Imagenette datasets, we solely apply blur as the linear transformation. Additional implementation details can be found in the code we have provided.

F Ablation Study Settings

F.1 Enhanced Attack Settings

In the enhanced attack settings, our first step is to extract 5% of the training dataset and inject the attack's trigger pattern into these images. We then proceed to purify this subset of data using blur and grayscale as linear transformations during the first stage of our proposed purification. Once the attacked images have been successfully purified, we modify their labels to reflect the attack label. Following this, we introduce these purified images as poisoned samples into the training set and train a classification model from scratch. This comprehensive procedure is referred to as the enhanced attack process.

F.2 Purification Speed Settings

This paper focuses on defending against backdoor attacks during the inference phase using purification techniques. To evaluate the purification speed, we conduct experiments using a workstation that features an Intel(R) Core(TM) i9-10900X CPU and an NVIDIA RTX3070 GPU with 8GB of memory.

During our experiments, we measure the classification time, which represents the duration taken by the classifier model to perform inference on a single image. Additionally, we measure the purification time, which indicates the time required by the purification model to purify a single image.

For the combination of purification with detection, we utilize the Scale-up method [19] as our chosen detection technique. Furthermore, the dataset used for speed evaluation consists of 5% poisoned images. Following previous settings [19], we set a batch size of one for the classifier model and report the average time based on 640 runs.

G Related Work

G.1 Backdoor Attack

Existing backdoor attack methods involve the injection of poisoned samples into the training process of Deep Neural Networks (DNNs). These attacks can target various types of models, including image classification models, object detection models [5, 36], contrastive learning models [3], and language models [40, 49, 69, 6]. The attackers exploit vulnerabilities by embedding adversary-specified trigger patterns into carefully selected benign samples. Backdoor attacks are characterized by their stealthiness, as the attacked models behave normally on benign samples, making the hidden triggers difficult to detect and purify.

There are mainly two categories of backdoor attacks for image classification tasks: patch-based and non-patch-based attacks. Patch-based attacks are attacks with triggers embedded as patches or overlays within the input samples. For example, Souri et al. [51] propose the Sleeper Agent attack, which is a sophisticated backdoor attack where an adversary subtly injects hidden triggers into an image classification model during training, remaining dormant until specific conditions activate malicious behavior. Non-patch-based attacks are attacks where triggers are integrated without explicit patching, often relying on specific input sequences or subtle modifications in the feature space [67, 23, 20]. For example, Doan et al. [11] introduce Wasserstein backdoor attack, an extension of the imperceptible backdoor concept to the latent representation. Their proposed attack manipulates inputs with imperceptible noise, matching latent representations to achieve high attack success rates while remaining stealthy in both the input and latent spaces.

G.2 Backdoor Defense

Existing defense methods for backdoor models can be broadly categorized into two approaches: (1) detection-based methods and (2) purification-based methods.

Detection-based methods focus on identifying the presence of backdoors in trained models. These methods typically involve analyzing the model's behavior and inputs to detect any suspicious patterns or triggers that indicate the existence of a backdoor [2]. Various techniques such as anomaly detection [13, 25, 63], and statistical analysis [18, 7] have been employed to detect backdoors. The goal of detection-based methods is to provide an early warning system to identify and mitigate the risks posed by backdoor attacks.

On the other hand, purification-based methods aim to remove or neutralize the effects of backdoors from the model. These methods involve modifying the model or its training process to eliminate the influence of the backdoor triggers on the model's behavior [26, 39]. Some purification approaches focus on retraining the model using clean or carefully selected training data to reduce the impact of the backdoor [64, 59, 31, 27, 55]. Other methods aim to directly identify and neutralize the backdoor triggers within the model's parameters or hidden representations [57, 4, 62, 60, 41]. The objective of purification-based methods is to restore the integrity and reliability of the model by eliminating the malicious behavior induced by the backdoor.

H Comparison Results with Image Restoration/Purification Methods

In this section, we compare our proposed ZIP with existing purification and restoration methods, including DiffPure [44] and DDNM [58]. Specifically, DiffPure is a state-of-the-art purification method designed for adversarial attacks, while DDNM is a state-of-the-art image restoration technique to repair corrupted images.

H.1 Quantitative Results Comparison with Image Restoration/Purification Methods

We first conduct a quantitative analysis of the defense performance between our proposed method and DiffPure and DDNM. In order to ensure a fair comparison, we implemented DiffPure and DDNM using their official code and applied identical linear transformations, diffusion steps, and schedules to both methods. The defense performance is listed in Table 6.

We can first observe that our ZIP performs better than DiffPure and Blur+DiffPure in all three metrics regarding defense performance. This is because our ZIP can recover the semantic information removed by the Blur, while Blur+DiffPure can not. Specifically, DiffPure aims to remove attack patterns by first diffusing images with noise and then recovering images through an unconditional reverse process. While coupling DiffPure with Blur enhances its trigger removal capability (lower ASR), its unconditional generative process fails to effectively recover the semantic information removed by Blur. This leads to a drop in clean accuracy (CA). In the above table, Blur+DiffPure exhibits poorer CA compared to both DiffPure and our ZIP. In contrast, our ZIP, after Blur, utilizes a conditional reverse process to recover semantic information deleted by Blur through RND theory. Therefore, our ZIPs can outperform DiffPure.

We can also observe that DDNM shows better clean accuracy than ZIP, but it cannot effectively defend against backdoor attacks like Blended. This is because DDNM restores the attack patterns in its diffusion process, which decreases the defense performance.

H.2 Qualitative Results Comparison with Image Restoration Methods

In this subsection, we conduct a qualitative analysis of the purification effect between our proposed method and DDNM.

(a) Attack Image

(b) Restored by DDNM

(c) Purified by ZIP

(d) Attack Image

(e) Restored by DDNM

(f) Purified by ZIP

(g) Attack Image

(h) Restored by DDNM

(i) Purified by ZIP

I Limitations

Due to our reliance on a pre-trained diffusion model to implement zero-shot purification, the effectiveness of generating purified images may be weakened when our model is applied to highly specific images that fall outside the distribution of pre-processed data. To mitigate this issue, we suggest two possible solutions in future work: 1) replacing the current pre-trained diffusion model with a more suitable pre-trained model for such specific images, and 2) collecting a subset of highly specific images to perform fine-tuning on the pre-trained model.

Peer reviews:
comments: SUMMARY:
In this paper, authors proposed a two-stage framework. Based on Range-Null Space Decomposition theory, they first utilize image transformation to destruct the trigger pattern, and then leverage a pre-trained diffusion model to restore the semantic information.

STRENGTHS:
(1)	To the best of my knowledge, authors propose a novel black-box backdoor defense method in a zero-shot setting.
(2)	The contributions mentioned by the authors in this paper are reasonable.

WEAKNESSES:
The chosen baselines of experiments in Section 4.3 (the results in Table 1) are not convincing and sufficient.

QUESTIONS:
What is the motivation of introducing RND?  Without RND, is there any change of the results? As shown in this paper(line 261), “Defense methods available for black-box purification in zero-shot are rare”, why not setting the black-box backdoor defense Salient Conditional Diffusion (Sancdifi) as your baseline?

LIMITATIONS:
Line 52-54 of this paper, If there is evidence in other existing literature to support this sentence “the semantic information in a poisoned image (e.g., faces, cars, or buildings) constitutes the majority of the data” , please cite it; if not, this idea should be used as a methodological assumption.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The paper proposes a backdoor defense for real-world black-box models through zero-shot image purification (ZIP). The proposed defense framework includes two stages: (1) applying linear transformation on a poisoned sample (2) using a pre-trained diffusion model to recover semantic information removed by stage (1). The proposed ZIP is utilized against different backdoor attacks on different datasets. The experiments demonstrate the effectiveness of proposed ZIP.

STRENGTHS:
The proposed idea is interesting and the authors analyze the zero-shot image purification theoretically. The conducted experiments demonstrate the effectiveness of proposed method.

WEAKNESSES:
1. The authors only use three backdoor attacks including BadNet, PhysicalBA and Blended. Please use more attacks such as WaNet, label-clean attacks to verify the effectiveness of proposed method.
2. In the Introduction Section (line 31 - 36), the authors mention that there are some works about black-box setting. Please compare with these methods.
3. The paper needs to be proofread. There are some typos in the paper e.g. in Algorithm 1 (line 222) "liner" should be "linear".

QUESTIONS:
1. The proposed method uses a pre-trained diffusion model to recover semantic. Does the diffusion model need addition in-distribution clean and poisoned data to train or fine-tune?
2. Could the proposed method work for different network architectures?
3. In Table 3, BDMAE gets a higher CA than ZIP. Could the authors explain this?
4. If the input is a clean image, is there a possibility that ZIP can misclassify the clean image? Could the authors provide some failure samples?

LIMITATIONS:
The authors have addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes backdoor defense for scenarios in which a defender does not need to access the internal model, a.k.a., the black-box backdoor defense. Firstly, a linear image transformation is used to destroy potential trigger patterns. Then, a pre-trained diffusion model is used to reconstruct the missing information induced in the linear transformation procedure, where a reverse process was designed to satisfy the conditional image generation/reconstruction. Defense experiments were conducted against three backdoor attacks to show its defensive effects.\n\nSTRENGTHS:\n1.\tThe paper writes well and is easy to follow.\n2.\tThe reported results indicate the effectiveness of the proposed method.\n3.\tInstead of directly applying diffusion model, adaptive reverse process was proposed.\n\nWEAKNESSES:\n1.\tThough it could find certain applications, the concept of black-box defense does not seem very practical and hence not quite attractive to me.\n2.\tFor the image reconstruction stage, there already exist a vast amount of advanced image restoration methods, such as diffusion model based e.g. DDNM in [1] or Restormer in [2]. I am wondering how would these pretrained models work to replace the reconstruction process in the second stage.\n3.\tThe authors used zero-shot image purification, however, it depends on a pretrained diffusion model. Have the authors evaluated how would the performance change if using a different diffusion model, or training on a different dataset? \n3.\tThe only baseline method was from 2020, which is too old in my opinion. More recent baselines should be evaluated.\n\nReferences\n\n[1] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).\n\n[2] Zamir, Syed Waqas, et al. \"Restormer: Efficient transformer for high-resolution image restoration.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\nQUESTIONS:\nI listed the questions in Weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel backdoor defense framework called \"zero-shot image purification\" (ZIP) designed to protect against backdoor attacks on real-world black-box models. The proposed ZIP framework consists of two steps: a linear transformation applied to the poisoned image to remove the backdoor pattern and a pre-trained diffusion model used to recover missing semantic information. The reverse process generates high-fidelity purified images without requiring internal information about the poisoned model. The ZIP framework is evaluated on various datasets with different attack types and outperforms state-of-the-art backdoor defense methods. The results are expected to offer valuable insights for future defense methods for black-box models.\n\nSTRENGTHS:\n1. The proposed idea is somewhat novel since the authors leverage the diffusion model to recovery the images. There have been similar works using auto-encoder[1] to reconstruct images to remove backdoor triggers but the exploration of diffusion model is not studied before.\n\n2. The authors provide theoretical analysis of how to modeling image purification with diffusion model.\n\n3. The proposed ZIP shows superior performance than existing defenses.\n\n4. The paper is well-written and easy to follow.\n\n[1]. Y. Liu, Y. Xie, and A. Srivastava, “Neural trojans,” in ICCD, 2017.\n\nWEAKNESSES:\n1. The idea of destroying backdoor trigger and reconstructing the images is very similar to pre-processing defenses.  Both defenses aim to remove the backdoor trigger by reconstructing the images.\n\n2. The threat model is not well defined in the paper. The capabilities of defenders and attackers are unclear.\n\n3. The backdoor attacks evaluated in this paper are developed 4-5 years ago, which is a bit too simple and outdated. Many advanced backdoor attacks such as hidden trigger backdoors [2], WaNet [3], LIRA [4], etc. are proposed after that. It would be desirable to evaluate the ZIP against more recent and advanced attacks.\n\n[2]. A. Saha, A. Subramanya, and H. Pirsiavash, “Hidden trigger backdoor attacks,” in AAAI, 2020.\n[3]. T. A. Nguyen and A. T. Tran, “Wanet-imperceptible warping-based backdoor attack,” in ICLR, 2021.\n[4]. K. Doan, Y. Lao, W. Zhao, and P. Li, “Lira: Learnable, imperceptible and robust backdoor attacks,” in ICCV, 2021.\n\nQUESTIONS:\nPlease see weaknesses for details.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work deals with the challenge of defending against backdoor attacks for black-box models in zero-shot settings using pre-trained diffusion models. Based on some theoretical insights, the proposed method modifies the reverse process of the diffusion model so the recovered images can be more high-fidelity and clean. Experimental results show that the proposed framework is effective in defending against backdoor attacks without prior knowledge of the model or additional retraining.\n\nSTRENGTHS:\n- The model-agnostic nature of the proposed method enhances its practicality as it can be adopted by any model without retraining.\n    \n- The theoretical analysis is clear and comprehensive.\n    \n- The additional techniques for speeding up the algorithm make the framework more realistic and applicable.\n\nWEAKNESSES:\n- Previous works have shown that Diffusion models can be used for image purification [1, 2, 3]. Thus, the paper’s novelty primarily lies in the adaptation of the traditional diffusion model for high-fidelity image recovery. While these improvements are theoretically driven, empirical evidence demonstrating the effectiveness of these enhancements or validating the proposed theories is lacking.  \n    \n- The experiments adopt three backdoor attacks: BadNet, PhysicalBA, and Blended. But more advanced attacks [4, 5] have been proposed and adopted in recent backdoor defense literature [6].\n    \n\n- There is an inconsistency in the statement on L344-345: “However, by switching to a different transformation, such as solely Blur or Grayscale, the attacks can be effectively mitigated.” ([pdf](zotero://open-pdf/library/items/HJZXQTNN?page=9)) In contrast, Table 4 reveals subpar defense performance when solely employing Grayscale.\n    \n\n[1] Nie et al. Diffusion Models for Adversarial Purification.\n\n[2] Wang et al. Guided Diffusion Model for Adversarial Purification.\n\n[3] Sun et al. PointDP: Diffusion-driven Purification against Adversarial Attacks on 3D Point Cloud Recognition.\n\n[4] Bagdasaryan and Shmatikov. Blind Backdoors in Deep Learning Models.\n\n[5] Nguyen and Tran. WaNet -- Imperceptible Warping-based Backdoor Attack.\n\n[6] Doan et al. Defending Backdoor Attacks on Vision Transformer via Patch Processing.\n\nQUESTIONS:\n- Regarding the setting of 4.4.2, if the attacker assumes that the defender is using Blur for transformation, will switching to a different transformation, such as Grayscale, be effective?\n    \n- The statement \"even when using an attacked model as the classifier\" on L66 is unclear. Could you provide more context or clarification for this scenario?\n\nLIMITATIONS:\nThe authors have discussed the limitations of their work. Please see the Weaknesses stated above to see other limitations I find.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}