SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Equation 7  $\mathbf{x}_t^{\prime} = \sqrt{\bar{\alpha}_t}\mathbf{A}^{\dagger}\mathbf{x}^{A} - \sqrt{\bar{\alpha}_t}\mathbf{A}^{\dagger}\mathbf{Ap} + (\mathbf{I} - \mathbf{A}^{\dagger}\mathbf{A})\mathbf{x}_t + \mathbf{A}^{\dagger}\mathbf{A}\sqrt{1 - \bar{\alpha}_t}\pmb {\epsilon}_t.$
Proof. Based on  $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\pmb {\epsilon},$  we have a nice property [22]:
$$
\mathbf {x} _ {0 \mid t} = \frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {t}\right). \tag {13}
$$
To ensure that the approximated clean image  $\mathbf{x}_{0|t}$  based on the  $t$ -th step observation  $\mathbf{x}_t$  satisfies the constraint in Equation 6, we have:
$$
\mathbf {x} _ {0 | t} = \mathbf {A} ^ {\dagger} \mathbf {x} ^ {A} - \mathbf {A} ^ {\dagger} \mathbf {A p} + (\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}) \mathbf {x} _ {0 | t}. \tag {14}
$$
Combine the above two equations, we can have:
$$
\frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {t}\right) = \mathbf {A} ^ {\dagger} \mathbf {x} ^ {A} - \mathbf {A} ^ {\dagger} \mathbf {A p} + \left(\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}\right) \left(\frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {t}\right)\right). \tag {15}
$$
Taking the derivative, we arrive at:
$$
\mathbf {x} _ {t} ^ {\prime} = \sqrt {\bar {\alpha} _ {t}} \mathbf {A} ^ {\dagger} \mathbf {x} ^ {A} - \sqrt {\bar {\alpha} _ {t}} \mathbf {A} ^ {\dagger} \mathbf {A p} + (\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}) \mathbf {x} _ {t} + \mathbf {A} ^ {\dagger} \mathbf {A} \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {t}. \tag {16}
$$
Lemma 3.1 Suppose the estimated noise output by  $g_{\phi}(\cdot)$  is Gaussian. Given  $g_{\phi}(\mathbf{x}_t,t) = \epsilon_t$ , we have  $g_{\phi}((\mathbf{x}_t + \sqrt{\overline{\alpha}_t}\mathbf{A}^\dagger \mathbf{A}\mathbf{p}),t) = \epsilon_t + \epsilon_t'$ , where  $\epsilon_t,\epsilon_t'$  are also Gaussian.
Proof. Let us define the output of  $g_{\phi}((\mathbf{x}_t + \mathbf{x}_t'), t)$  as  $\hat{\epsilon}_t$ , so we have  $g_{\phi}((\mathbf{x}_t + \mathbf{x}_t'), t) = \hat{\epsilon}_t$ . Next, we define  $\epsilon_t \sim N(\mu_1, \sigma_1^2)$  and  $\hat{\epsilon}_t \sim N(\mu_2, \sigma_2^2)$ .
Since  $\hat{\epsilon}_t$  also follows a Gaussian distribution, we can subtract  $\epsilon_t$  from  $\hat{\epsilon}_t$  to obtain  $\epsilon_t'$ , such that:
$$
\boldsymbol {\epsilon} _ {t} ^ {\prime} = \hat {\boldsymbol {\epsilon}} _ {t} - \boldsymbol {\epsilon} _ {t} \sim N \left(\mu_ {2} - \mu_ {1}, \sigma_ {1} ^ {2} + \sigma_ {2} ^ {2}\right) \tag {17}
$$
This confirms that  $\epsilon_t^\prime$  is also Gaussian, thus completing the proof.
A.3 Proof of Theorem 3.2
Theorem 3.2 Suppose that  $g_{\phi}((\mathbf{x}_t + \sqrt{\bar{\alpha}_t}\mathbf{A}^\dagger \mathbf{A}\mathbf{p})) = \epsilon_t + \epsilon_t'$ . We define the error at step  $t$  between  $\hat{\mathbf{x}}_t$  and  $(\mathbf{x}_t + \sqrt{\bar{\alpha}_t}\mathbf{A}^\dagger \mathbf{A}\mathbf{p})$  as  $\delta_t'$ , i.e.,  $\delta_t' = \hat{\mathbf{x}}_t - (\mathbf{x}_t + \sqrt{\bar{\alpha}_t}\mathbf{A}^\dagger \mathbf{A}\mathbf{p})$ . Let  $\delta_t = \mathbf{A}\delta_t'$ , we have the following bound on its norm:  $\| \delta_t\| \leq \frac{(1 - \bar{\alpha}_t)\sqrt{\alpha_{t+1}}}{\sqrt{1 - \bar{\alpha}_{t+1}}}\|\mathbf{A}\|^{\prime}\| \epsilon_{t+1}'$ .
Proof. We prove the above theorem by induction.
1 The base case is when  $t = T$ , where we have  $\dot{\mathbf{x}}_T - (\mathbf{x}_T + \sqrt{\bar{\alpha}_T}\mathbf{A}^\dagger\mathbf{A}\mathbf{p}) = 0$ , which holds.
2 Suppose for  $\hat{\mathbf{x}}_t - (\mathbf{x}_t + \sqrt{\bar{\alpha}_t}\mathbf{A}^\dagger\mathbf{A}\mathbf{p}) = \mathbf{A}^\dagger\delta_t$ ,  $\|\delta_t\| \leq \frac{(1-\bar{\alpha}_t)\sqrt{\bar{\alpha}_{t+1}}}{\sqrt{1-\bar{\alpha}_{t+1}}}\|\mathbf{A}\|\|\epsilon_{t+1}'\|$  is true.
3 Induction:
$$
\mathbf {x} _ {t - 1} \leftarrow \frac {1}{\sqrt {\alpha} _ {t}} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {A} ^ {\dagger} \mathbf {x} ^ {A} - \sqrt {\bar {\alpha} _ {t}} \mathbf {A} ^ {\dagger} \mathbf {A p} + (\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}) \mathbf {x} _ {t} + \mathbf {A} ^ {\dagger} \mathbf {A} \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {t} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {t}\right) + \sigma_ {t} \boldsymbol {\epsilon}, \tag {18}
$$
$$
\hat {\mathbf {x}} _ {t - 1} = \frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {A} ^ {\dagger} \mathbf {x} ^ {A} + \left(\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}\right) \hat {\mathbf {x}} _ {t} + \mathbf {A} ^ {\dagger} \mathbf {A} \sqrt {1 - \bar {\alpha} _ {t}} \left(\boldsymbol {\epsilon} _ {t} + \boldsymbol {\epsilon} _ {t} ^ {\prime}\right) - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \left(\boldsymbol {\epsilon} _ {t} + \boldsymbol {\epsilon} _ {t} ^ {\prime}\right)\right) + \sigma_ {t} \boldsymbol {\epsilon}, \tag {19}
$$
$$
\hat {\mathbf {x}} _ {t - 1} - \mathbf {x} _ {t - 1} \leftarrow \frac {1}{\sqrt {\alpha_ {t}}} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {A} ^ {\dagger} \mathbf {A} \mathbf {p} + \left(\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}\right) \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {A} ^ {\dagger} \mathbf {A} \mathbf {p} + \mathbf {A} ^ {\dagger} \delta_ {t}\right) + \mathbf {A} ^ {\dagger} \mathbf {A} \sqrt {1 - \bar {\alpha} _ {t}} \epsilon_ {t} ^ {\prime} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {t} ^ {\prime}\right) \tag {20}
$$
By definition, we have  $\hat{\mathbf{x}}_{t - 1} - (\mathbf{x}_{t - 1} + \sqrt{\bar{\alpha}_{t - 1}}\mathbf{A}^{\dagger}\mathbf{A}\mathbf{p}) = \delta_{t - 1}^{\prime}$ . Let  $\delta_{t - 1} = \mathbf{A}\delta_{t - 1}^{\prime}$ , we have  $\delta_{t - 1} = \mathbf{A}\delta_{t - 1}^{\prime} = \mathbf{A}(\hat{\mathbf{x}}_{t - 1} - (\mathbf{x}_{t - 1} + \sqrt{\bar{\alpha}_{t - 1}}\mathbf{A}^{\dagger}\mathbf{A}\mathbf{p}))$ . Since  $\mathbf{A}(\mathbf{I} - \mathbf{A}^{\dagger}\mathbf{A}) = \mathbf{0}$ , we can have:
$$
\boldsymbol {\delta} _ {t - 1} = \frac {1}{\sqrt {\alpha} _ {t}} (\mathbf {A} \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {t} ^ {\prime} - \mathbf {A} \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {t} ^ {\prime}) = \frac {(1 - \bar {\alpha} _ {t - 1}) \sqrt {\alpha_ {t}}}{\sqrt {1 - \bar {\alpha} _ {t}}} \mathbf {A} \boldsymbol {\epsilon} _ {t} ^ {\prime} \tag {21}
$$
Based on the Cauchy-Schwarz inequality:
$$
\left\| \boldsymbol {\delta} _ {t - 1} \right\| = \left\| \frac {\left(1 - \bar {\alpha} _ {t - 1}\right) \sqrt {\alpha_ {t}}}{\sqrt {1 - \bar {\alpha} _ {t}}} \mathbf {A} \boldsymbol {\epsilon} _ {t} ^ {\prime} \right\| \leq \frac {\left(1 - \bar {\alpha} _ {t - 1}\right) \sqrt {\alpha_ {t}}}{\sqrt {1 - \bar {\alpha} _ {t}}} \| \mathbf {A} \| \| \boldsymbol {\epsilon} _ {t} ^ {\prime} \| \tag {22}
$$
A.4 Proof of Corollary 3.2.1
Corollary 3.2.1 When  $t = 0$ , we have  $\hat{\mathbf{x}}_0 = \mathbf{x}_0 + \mathbf{A}^\dagger \mathbf{A}\mathbf{p} + \delta_0'$ , where  $\mathbf{A}\delta_0' = \mathbf{0}$ .
Proof. First, we have:
$$
\mathbf {x} _ {0} \leftarrow \frac {1}{\sqrt {\alpha} _ {1}} \left(\sqrt {\bar {\alpha} _ {1}} \mathbf {A} ^ {\dagger} \mathbf {x} ^ {A} - \sqrt {\bar {\alpha} _ {1}} \mathbf {A} ^ {\dagger} \mathbf {A p} + (\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}) \mathbf {x} _ {1} + \mathbf {A} ^ {\dagger} \mathbf {A} \sqrt {1 - \bar {\alpha} _ {1}} \boldsymbol {\epsilon} _ {1} - \frac {1 - \alpha_ {1}}{\sqrt {1 - \bar {\alpha} _ {1}}} \boldsymbol {\epsilon} _ {1}\right) + \sigma_ {1} \boldsymbol {\epsilon}, \tag {23}
$$
$$
\hat {\mathbf {x}} _ {0} = \frac {1}{\sqrt {\bar {\alpha} _ {1}}} \left(\sqrt {\bar {\alpha} _ {1}} \mathbf {A} ^ {\dagger} \mathbf {x} ^ {A} + \left(\mathbf {I} - \mathbf {A} ^ {\dagger} \mathbf {A}\right) \hat {\mathbf {x}} _ {1} + \mathbf {A} ^ {\dagger} \mathbf {A} \sqrt {1 - \bar {\alpha} _ {1}} \left(\boldsymbol {\epsilon} _ {1} + \boldsymbol {\epsilon} _ {1} ^ {\prime}\right) - \frac {1 - \alpha_ {1}}{\sqrt {1 - \bar {\alpha} _ {1}}} \left(\boldsymbol {\epsilon} _ {1} + \boldsymbol {\epsilon} _ {1} ^ {\prime}\right)\right) + \sigma_ {1} \boldsymbol {\epsilon}, \tag {24}
$$
Then we can have:
$$
\mathbf {A} \delta_ {0} ^ {\prime} = \frac {(1 - \alpha_ {0}) \sqrt {\alpha_ {1}}}{\sqrt {1 - \bar {\alpha} _ {1}}} \mathbf {A} \epsilon_ {1} ^ {\prime}. \tag {25}
$$
Since  $\alpha_0 = 1$ , then we have  $\mathbf{A}\delta_0^{\prime} = \mathbf{0}$ .
B Qualitative Results of Purification
B.1 Qualitative Results of Purification on BadNet Attack
(a) Attacked Image
(b) Purified Image
(c) Attacked Image
(d) Purified Image
(e) Attacked Image
(f) Purified Image
(g) Attacked Image
(h) Purified Image
(i) Attacked Image
(m) Attacked Image
Figure 4: Comparison of Purified and BadNet Attack Images(Part1).
(j) Purified Image
(n) Purified Image
(k) Attacked Image
(o) Attacked Image
(1) Purified Image
(p) Purified Image
(i) Attacked Image
(j) Purified Image
(k) Attacked Image
(1) Purified Image
(m) Attacked Image
(n) Purified Image
(o) Attacked Image
(p) Purified Image
(q) Attacked Image
(u) Attacked Image
Figure 5: Comparison of Purified and BadNet Attack Images(Part2).
(r) Purified Image
(v) Purified Image
(s) Attacked Image
(w) Attacked Image
(t) Purified Image
(x) Purified Image
(q) Attacked Image
(u) Attacked Image
Figure 6: Comparison of Purified and BadNet Attack Images(Part3).
(q) Attacked Image
(u) Attacked Image
Figure 7: Comparison of Purified and BadNet Attack Images(Part4).
B.2 Qualitative Results of Purification on Blended Attack
(i) Attacked Image
(m) Attacked Image
Figure 8: Comparison of Purified and Blended Attack Images(Part1).
(q) Attacked Image
(u) Attacked Image
Figure 9: Comparison of Purified and Blended Attack Images(Part2).
(q) Attacked Image
(u) Attacked Image
Figure 10: Comparison of Purified and Blended Attack Images(Part3).
(q) Attacked Image
(u) Attacked Image
Figure 11: Comparison of Purified and Blended Attack Images(Part4).
C Details on Linear Transformations
In this section, we discuss details of the linear transformation applied in our paper. Two practical examples are discussed to illustrate the simplicity and effectiveness of linear transformations, along with the corresponding operators used.
Gray-scale Conversion: To convert an RGB image to gray-scale, the operator  $\mathbf{A} = [1/3, 1/3, 1/3]$  can be defined as a pixel-wise operation that transforms each RGB channel pixel  $[r, g, b]$  into a grayscale value  $r^3 + g^3 + b^3$ . In this case, constructing a pseudo-inverse  $\mathbf{A}^\dagger = [1, 1, 1]^{\mathrm{T}}$  is straightforward, satisfying the condition  $\mathbf{A}\mathbf{A}^\dagger \equiv \mathbf{I}$ , where  $\mathbf{I}$  represents the identity matrix.
Image Blurring: Image blurring also involves linear transformations. For a blurring operation with scale  $n$ , the operator  $\mathbf{A}$  is defined as the average-pooling operator  $[1/n^2, \dots, 1/n^2]$ . This operator aggregates each patch of the image into a single value. Similarly, the pseudo-inverse  $\mathbf{A}^\dagger$  can be built as  $\mathbf{A}^\dagger = [1, \dots, 1]^{\mathrm{T}}$  to fulfill the condition  $\mathbf{A} \mathbf{A}^\dagger \equiv \mathbf{I}$ .
Overall, these examples demonstrate how these two linear transformations, in conjunction with their respective operators, can be employed to destruct the trigger pattern without relying on a complex Fourier transform. In cases where the linear transformation is too complex to solve for its pseudo-inverse, the Singular Value Decomposition (SVD) method can be applied. For more details, please refer to papers [58, 29].
D Algorithm for improved ZIP based on DDIM
In this section, we include the modified algorithm based on DDIM, which is proposed to speed up the diffusion model inference speed.
Algorithm 2 Zero-shot Image Purification (based on DDIM)
Require: Test image for purification  $\mathbf{x}^P$ ; liner transformation  $\mathbf{A}_1, \mathbf{A}_2, \dots, \mathbf{A}_n$  and their pseudo-inverse  $\mathbf{A}_1^\dagger, \mathbf{A}_2^\dagger, \dots, \mathbf{A}_n^\dagger$ ; diffusion model  $g$ ; hyperparameter  $\lambda$ ; speed-up pace  $S$ .  
Ensure:  $\mathbf{x}^{A_n} = \mathbf{A}_n \mathbf{x}^P$ ,  $n = 0, 1, \dots, N$   
1:  $\mathbf{x}^T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$   
2: for  $t = T, T - S, \dots, S, 1$  do  
3:  $\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$  if  $t > 1$ , else  $\epsilon = 0$   
4:  $\epsilon_t = g_\phi(\mathbf{x}_t, t)$   
5: for  $n = 1, 2, \dots, N$  do  
6:  $\hat{\mathbf{x}}_t^n = \sqrt{\bar{\alpha}_t} \mathbf{A}_1^\dagger \mathbf{x}^{A_n} + (\mathbf{I} - \mathbf{A}_n^\dagger \mathbf{A}_n) \mathbf{x}_t + \mathbf{A}_n^\dagger \mathbf{A}_n \sqrt{1 - \bar{\alpha}_t} \epsilon_t$   
7: end for  
8:  $\hat{\mathbf{x}}_t = \frac{1}{N} (\hat{\mathbf{x}}_t^1 + \hat{\mathbf{x}}_t^2 +, \dots, + \hat{\mathbf{x}}_t^N)$   
9:  $\tilde{\mathbf{x}}_t = (1 - \bar{\alpha}_t^\lambda) \hat{\mathbf{x}}_t + \bar{\alpha}_t^\lambda \mathbf{x}_t$   
10:  $\tilde{\mathbf{x}}_{0|t} = \frac{1}{\sqrt{\bar{\alpha}_t}} (\tilde{\mathbf{x}}_t - \sqrt{1 - \bar{\alpha}_t} \epsilon_t)$   
11:  $\mathbf{x}_{t-1} \gets \sqrt{\bar{\alpha}_{t-1}} \tilde{\mathbf{x}}_{0|t} + \sqrt{1 - \bar{\alpha}_{t-1} - \sigma_t^2} \epsilon_t + \sigma_t \epsilon$   
12: end for  
13: return  $\mathbf{x}_0$
E Experiments Settings
E.1 Datasets Informaiton
CIFAR-10 [30] The CIFAR-10 dataset is a widely-used benchmark in computer vision. It consists of 60,000 color images of size  $32 \times 32$  pixels, belonging to 10 different classes, with 6,000 images per class. The dataset is divided into 50,000 training images and 10,000 test images, with a balanced distribution of classes.
GTSRB [52] The German Traffic Sign Recognition Benchmark (GTSRB) dataset is designed for traffic sign classification tasks. It comprises more than 50,000 images of traffic signs captured under various real-world conditions. The images have different sizes and aspect ratios, but they are resized
to 32x32 pixels for our model training and evaluation. The dataset is divided into training and test sets, with its official split ratio.
Imagenette [24] The Imagenette is a subset of the larger ImageNet dataset and is commonly used as a smaller-scale alternative for image classification tasks. It consists of 10 classes with a total of 13,000 images. The images in Imagenette have varying sizes, but they are resized to  $256 \times 256$  pixels for consistency. The dataset is split into training and validation sets, following a predefined split ratio.
Table 5: Properties of datasets.
Dataset	Classes	Image Size	Train Split	Test Split
CIFAR-10	10	32x32	50,000	10,000
GTSRB	43	32x32	39,209	12,630
Imagenette	10	256x256	9,480	3,936
E.2 Attacks Implementation
In this section, we discuss the implementation details of three different backdoor attack methods employed in our study: BadNet, Blended, and PhysicalBA. We implement these backdoor attacks using the Backdoorbox framework [33], which is under GNU general public license.
BadNet [15] The BadNet attack injects specific trigger patterns into the training data. In our implementation, we set the poisoned rate to  $5\%$ , i.e.,  $5\%$  of the training samples are selected as attack samples and have the trigger pattern added to them. The trigger pattern size is set to 2x2 for 32x32 pixels images and 9x9 for 256x256 pixels images. The trigger patterns are randomly generated.
Blended [8] The Blended attack is a more sophisticated variant aimed at making the backdoor less conspicuous and harder to detect. Following the suggestion in BackdoorBox, we set the blended rate to 0.2 and the poisoned rate to  $5\%$ . The blended pattern is randomly generated, seamlessly blending the trigger pattern into the attack samples.
PhysicalBA [34] The PhysicalBA (Physical Backdoor Attack) is a specific type of attack that introduces variations in the location and appearance of the attack pattern embedded in the test samples during inference time. In our implementation, we apply the same attack pattern size as the BadNet attack, using a 2x2 pattern for 32x32 pixels images and a 9x9 pattern for 256x256 pixels images. The attack patterns are generated randomly. We set the poisoned rate to  $5\%$  for this attack.
All other attack settings follow the default configurations in Backboorbox [33].
E.3 Purification Implementation
We utilize a pre-trained model provided by OpenAI [9] under the MIT license. The algorithm described in Algorithm 2 is employed to accelerate the inference process, allowing us to generate high-quality images within just 20 steps, and the speed-up pace is set to 50. We set the hyperparameter  $\lambda$  to a value of 2 for Blended attack defense, and 10 for BadNet and PhysicalBA attack defense.
Specifically, for the CIFAR-10 dataset, we apply both blur and gray-scale conversion as linear transformations. For the GTSRB and Imagenette datasets, we solely apply blur as the linear transformation. Additional implementation details can be found in the code we have provided.
F Ablation Study Settings
F.1 Enhanced Attack Settings
In the enhanced attack settings, our first step is to extract  $5\%$  of the training dataset and inject the attack's trigger pattern into these images. We then proceed to purify this subset of data using blur and grayscale as linear transformations during the first stage of our proposed purification. Once the attacked images have been successfully purified, we modify their labels to reflect the attack label. Following this, we introduce these purified images as poisoned samples into the training set and train a classification model from scratch. This comprehensive procedure is referred to as the enhanced attack process.
F.2 Purification Speed Settings
This paper focuses on defending against backdoor attacks during the inference phase using purification techniques. To evaluate the purification speed, we conduct experiments using a workstation that features an Intel(R) Core(TM) i9-10900X CPU and an NVIDIA RTX3070 GPU with 8GB of memory.
During our experiments, we measure the classification time, which represents the duration taken by the classifier model to perform inference on a single image. Additionally, we measure the purification time, which indicates the time required by the purification model to purify a single image.
For the combination of purification with detection, we utilize the Scale-up method [19] as our chosen detection technique. Furthermore, the dataset used for speed evaluation consists of  $5\%$  poisoned images. Following previous settings [19], we set a batch size of one for the classifier model and report the average time based on 640 runs.
G Related Work
G.1 Backdoor Attack
Existing backdoor attack methods involve the injection of poisoned samples into the training process of Deep Neural Networks (DNNs). These attacks can target various types of models, including image classification models, object detection models [5, 36], contrastive learning models [3], and language models [40, 49, 69, 6]. The attackers exploit vulnerabilities by embedding adversary-specified trigger patterns into carefully selected benign samples. Backdoor attacks are characterized by their stealthiness, as the attacked models behave normally on benign samples, making the hidden triggers difficult to detect and purify.
There are mainly two categories of backdoor attacks for image classification tasks: patch-based and non-patch-based attacks. Patch-based attacks are attacks with triggers embedded as patches or overlays within the input samples. For example, Souri et al. [51] propose the Sleeper Agent attack, which is a sophisticated backdoor attack where an adversary subtly injects hidden triggers into an image classification model during training, remaining dormant until specific conditions activate malicious behavior. Non-patch-based attacks are attacks where triggers are integrated without explicit patching, often relying on specific input sequences or subtle modifications in the feature space [67, 23, 20]. For example, Doan et al. [11] introduce Wasserstein backdoor attack, an extension of the imperceptible backdoor concept to the latent representation. Their proposed attack manipulates inputs with imperceptible noise, matching latent representations to achieve high attack success rates while remaining stealthy in both the input and latent spaces.
G.2 Backdoor Defense
Existing defense methods for backdoor models can be broadly categorized into two approaches: (1) detection-based methods and (2) purification-based methods.
Detection-based methods focus on identifying the presence of backdoors in trained models. These methods typically involve analyzing the model's behavior and inputs to detect any suspicious patterns or triggers that indicate the existence of a backdoor [2]. Various techniques such as anomaly detection [13, 25, 63], and statistical analysis [18, 7] have been employed to detect backdoors. The goal of detection-based methods is to provide an early warning system to identify and mitigate the risks posed by backdoor attacks.
On the other hand, purification-based methods aim to remove or neutralize the effects of backdoors from the model. These methods involve modifying the model or its training process to eliminate the influence of the backdoor triggers on the model's behavior [26, 39]. Some purification approaches focus on retraining the model using clean or carefully selected training data to reduce the impact of the backdoor [64, 59, 31, 27, 55]. Other methods aim to directly identify and neutralize the backdoor triggers within the model's parameters or hidden representations [57, 4, 62, 60, 41]. The objective of purification-based methods is to restore the integrity and reliability of the model by eliminating the malicious behavior induced by the backdoor.
H Comparison Results with Image Restoration/Purification Methods
In this section, we compare our proposed ZIP with existing purification and restoration methods, including DiffPure [44] and DDNM [58]. Specifically, DiffPure is a state-of-the-art purification method designed for adversarial attacks, while DDNM is a state-of-the-art image restoration technique to repair corrupted images.
H.1 Quantitative Results Comparison with Image Restoration/Purification Methods
We first conduct a quantitative analysis of the defense performance between our proposed method and DiffPure and DDNM. In order to ensure a fair comparison, we implemented DiffPure and DDNM using their official code and applied identical linear transformations, diffusion steps, and schedules to both methods. The defense performance is listed in Table 6.
Table 6: Defense performance on Imagenette.
Attack	No Defense	DiffPure	Blur+DiffPure	Blur+DDNM	ZIP (Ours)
CA	ASR	PA	CA	ASR	PA	CA	ASR	PA	CA	ASR	PA	CA	ASR	PA
BadNet	84.99	94.53	14.98	78.85	91.41	18.11	75.13	10.16	74.31	84.56	9.14	82.24	84.05	7.55	83.97
Blended	86.14	99.85	10.19	80.63	43.82	56.68	75.89	13.53	73.98	85.37	93.37	15.46	81.42	8.35	78.36
We can first observe that our ZIP performs better than DiffPure and Blur+DiffPure in all three metrics regarding defense performance. This is because our ZIP can recover the semantic information removed by the Blur, while Blur+DiffPure can not. Specifically, DiffPure aims to remove attack patterns by first diffusing images with noise and then recovering images through an unconditional reverse process. While coupling DiffPure with Blur enhances its trigger removal capability (lower ASR), its unconditional generative process fails to effectively recover the semantic information removed by Blur. This leads to a drop in clean accuracy (CA). In the above table, Blur+DiffPure exhibits poorer CA compared to both DiffPure and our ZIP. In contrast, our ZIP, after Blur, utilizes a conditional reverse process to recover semantic information deleted by Blur through RND theory. Therefore, our ZIPs can outperform DiffPure.