SlowGuess/NeurIPS_2023_OCR huggingface dataset:
At each time step  $t$  in the reverse process, we only need to store the gradient  $\partial \mathcal{L} / \partial \mathbf{x}_t'$ , the intermediate sample  $\mathbf{x}_{t+1}'$  and the model  $f_r$  to construct the computational graph. When we backpropagate the gradients at the next time step  $t+1$ , the computational graph at time step  $t$  will no longer be reused, and thus, we can release the memory of the graph at time step  $t$ . Therefore, we only have one segment of the computational graph used for gradient backpropagation in the memory at each time step. We can similarly backpropagate the gradients in the diffusion process. Ignoring the memory cost of storing intermediate samples (usually small compared to the memory cost of computational graphs), the memory cost of our segment-wise forwarding-backwarding algorithm is  $\mathcal{O}(1)$  (validated in Figure 3).
We summarize the detailed procedures in Algorithm 1 in Appendix B. It can be applied to gradient backpropagation through any discrete Markov process with a long path. Basically, we 1) perform the forward pass and store the intermediate samples, 2) allocate the memory of one segment of the computational graph in the memory and simulate the forwarding pass of the segment with intermediate samples, 3) backpropagate the gradients through the segment and release the memory of the segment, and 4) go to step 2 and consider the next segment until termination.
3.4 DiffAttack Technique
Currently, AutoAttack [12] holds the state-of-the-art attack algorithm, but it fails to attack the diffusion-based purification defenses due to the challenge of vanishing/exploding gradient, memory cost and large randomness. To specifically tackle the challenges, we integrate the deviated-reconstruction loss (in Section 3.2) and the segment-wise forwarding-backwarding algorithm (in Section 3.3) as an attack technique DiffAttack against diffusion-based purification, including the score-based and DDPM-based purification defenses. The pictorial illustration of DiffAttack is provided in Figure 1.
Concretely, we maximize the surrogate loss  $\mathcal{L}$  as the optimization objective in Equation (7):
$$
\max  \mathcal {L} = \mathcal {L} _ {c l s} + \lambda \mathcal {L} _ {d e v} \tag {11}
$$
where  $\mathcal{L}_{cls}$  is the CE loss or DLR loss,  $\mathcal{L}_{dev}$  is the deviated-reconstruction loss formulated in Equation (8), and  $\lambda$  is the weight coefficient. During the optimization, we use the segment-wise forwarding-backwarding algorithm for memory-efficient gradient backpropagation. Note that  $\mathcal{L}_{dev}$  suffers less from the gradient problem compared with  $\mathcal{L}_{cls}$ , and thus the objective of  $\mathcal{L}_{dev}$  can be optimized more precisely and stably, but it does not resolve the gradient problem of  $\mathcal{L}_{cls}$ . On the other hand, the optimization of  $\mathcal{L}_{dev}$  benefits the optimization of  $\mathcal{L}_{cls}$  in the sense that  $\mathcal{L}_{dev}$  can induce a deviated reconstruction of the image with a larger probability of misclassification.  $\lambda$  controls the balance of the two objectives. A small  $\lambda$  can weaken the deviated-reconstruction object and make the attack suffer more from the vanishing/exploded gradient problem, while a large  $\lambda$  can downplay the guidance of the classification loss and confuse the direction towards the decision boundary of the classifier.
Attack against randomized diffusion-based purification. DiffAttack tackles the randomness problem from two perspectives: 1) sampling the diffused and reconstructed samples across different time steps multiple times as in Equation (8) (similar to EOT [3]), and 2) optimizing perturbations for all samples including misclassified ones in all steps. Perspective 1) provides a more accurate estimation of gradients against sample variance of the diffusion process. Perspective 2) ensures a more effective and stable attack optimization since the correctness of classification is of high variance over different steps in the diffusion purification setting. Formally, the classification result of a sample can be viewed as a Bernoulli distribution (i.e., correct or false). We should reduce the success rate of the Bernoulli distribution of sample classification by optimizing them with a larger attack loss, which would lead to lower robust accuracy. In other words, one observation of failure in classification does not indicate that the sample has a low success rate statistically, and thus, perspective 2) helps
Table 1: Attack performance (Rob-Acc  $(\%)$ ) of DiffAttack and AdjAttack [34] against score-based purification on CIFAR-10.
Models	T	Cl-Acc	\( \ell_p \)Attack	\( \epsilon \)	Method	Rob-Acc	Diff.
WideResNet-28-10	0.1	89.02	\( \ell_{\infty} \)	8/255	AdjAttack DiffAttack	70.64 46.88	-23.76
4/255	AdjAttack DiffAttack	82.81 71.88	-10.93
0.075	91.03	\( \ell_2 \)	0.5	AdjAttack DiffAttack	78.58 64.06	-14.52
WideResNet-70-16	0.1	90.07	\( \ell_{\infty} \)	8/255	AdjAttack DiffAttack	71.29 45.31	-25.98
4/255	AdjAttack DiffAttack	81.25 75.00	-6.25
0.075	92.68	\( \ell_2 \)	0.5	AdjAttack DiffAttack	80.60 70.31	-10.29
to continue optimizing the perturbations towards a lower success rate (i.e., away from the decision boundary). We provide the pseudo-codes of DiffAttack in Algorithm 2 in Appendix D.1.
In this section, we evaluate DiffAttack from various perspectives empirically. As a summary, we find that 1) DiffAttack significantly outperforms other SOTA attack methods against diffusion-based defenses on both the score-based purification and DDPM-based purification models, especially under large perturbation radii (Section 4.2 and Section 4.3); 2) DiffAttack outperforms other strong attack methods such as the black-box attack and adaptive attacks against other adversarial purification defenses (Section 4.4); 3) a moderate diffusion length  $T$  benefits the model robustness, since too long/short diffusion length would hurt the robustness (Section 4.5); 4) our proposed segment-wise forwarding-backwarding algorithm achieves  $\mathcal{O}(1)$ -memory cost and outperforms other baselines by a large margin (Section 4.6); and 5) attacks with the deviated-reconstruction loss added over uniformly sampled time steps outperform that added over only initial/final time steps (Section 4.7).
Dataset & model. We validate DiffAttack on CIFAR-10 [27] and ImageNet [13]. We consider different network architectures for classification. Particularly, WideResNet-28-10 and WideResNet-70-16 [61] are used on CIFAR-10, and ResNet-50 [23], WideResNet-50-2 (WRN-50-2), and ViT (DeiT-S) [16] are used on ImageNet. We use a pretrained score-based diffusion model [49] and DDPM [25] to purify images following [34, 4].
Evaluation metric. The performance of attacks is evaluated using the robust accuracy (Rob-Acc), which measures the ratio of correctly classified instances over the total number of test data under certain perturbation constraints. Following the literature [12], we consider both  $\ell_{\infty}$  and  $\ell_{2}$  attacks under multiple perturbation constraints  $\epsilon$ . We also report the clean accuracy (Cl-Acc) for different approaches.
Baselines. To demonstrate the effectiveness of DiffAttack, we compare it with 1) SOTA attacks against score-based diffusion adjoint attack (AdjAttack) [34], 2) SOTA attack against DDPM-based diffusion Diff-BPDA attack [4], 3) SOTA black-box attack  $SPSA$  [53] and square attack [1], and 4) specific attack against EBM-based purification joint attack [60]. We defer more explanations of baselines and experiment details to Appendix D.2. The codes are publicly available at https://github.com/kangmintong/DiffAttack.
Table 3: Attack performance (Rob-Acc  $(\%)$ ) of DiffAttack and Diff-BPDA [4] against DDPM-based purification on CIFAR-10.
Architecture	T	Cl-Acc	\( \ell_p \)Attack	ε	Method	Rob-Acc	Diff.
WideResNet-28-10	100	87.50	\( \ell_\infty \)	8/255	Diff-BPDA DiffAttack	75.00 54.69	-20.31
4/255	Diff-BPDA DiffAttack	76.56 63.28	-13.28
75	90.62	\( \ell_2 \)	0.5	Diff-BPDA DiffAttack	76.56 67.97	-8.59
WideResNet-70-16	100	91.21	\( \ell_\infty \)	8/255	Diff-BPDA DiffAttack	74.22 59.38	-14.84
4/255	Diff-BPDA DiffAttack	75.78 67.19	-8.59
75	92.19	\( \ell_2 \)	0.5	Diff-BPDA DiffAttack	81.25 71.88	-9.37
4.2 Attack against score-based purification
DiffPure [34] presents the state-of-the-art adversarial purification performance using the score-based diffusion models [49]. It proposes a strong adaptive attack (AdjAttack) which uses the adjoint method [28] to efficiently backpropagate the gradients through reverse SDE. Therefore, we select AdjAttack as the strong baseline and compare DiffAttack with it. The results on CIFAR-10 in Table 1 show that DiffAttack achieves much lower robust accuracy compared with AdjAttack under different types of attacks  $(\ell_{\infty}$  and  $\ell_2$  attack) with multiple perturbation constraints  $\epsilon$ . Concretely, DiffAttack decreases the robust accuracy
Table 2: Attack performance of DiffAttack and AdjAttack [34] against score-based adversarial purification with diffusion length  $T = 0.015$  on ImageNet under  $\ell_{\infty}$  attack  $(\epsilon = 4 / 255)$ .
Models	Cl-Acc	Method	Rob-Acc	Diff.
ResNet-50	67.79	AdjAttack	40.93	-12.80
DiffAttack	28.13
WRN-50-2	71.16	AdjAttack	44.39	-13.14
DiffAttack	31.25
DeiT-S	73.63	AdjAttack	43.18	-10.37
DiffAttack	32.81
of models by over  $20\%$  under  $\ell_{\infty}$  attack with  $\epsilon = 8 / 255$  ( $70.64\% \to 46.88\%$  on WideResNet-28-10 and  $71.29\% \to 45.31\%$  on WideResNet-70-16). The effectiveness of DiffAttack also generalizes well to large-scale datasets ImageNet as shown in Table 2. Note that the robust accuracy of the state-of-the-art non-diffusion-based purification defenses [38, 21] achieve about  $65\%$  robust accuracy on CIFAR-10 with WideResNet-28-10 under  $\ell_{\infty} = 8 / 255$  attack ( $\epsilon = 8 / 255$ ), while the performance of score-based purification under AdjAttack in the same setting is  $70.64\%$ . However, given the strong DiffAttack, the robust accuracy of score-based purification drops to  $46.88\%$ . It motivates us to think of more effective techniques to further improve the robustness of diffusion-based purification in future work.
4.3 Attack against DDPM-based purification
Another line of diffusion-based purification defenses [4, 55, 56] leverages DDPM [46] to purify the images with intentionally crafted perturbations. Since backpropagating the gradients along the diffusion and sampling process with a relatively large diffusion length is unrealistic due to the large memory cost, BPDA attack [2] is adopted as the strong attack against the DDPM-based purification. However, with our proposed segment-wise forwarding-backwarding algorithm, we can compute the gradients within a small budget of memory cost, and to our best knowledge, this is the first work to achieve adaptive gradient-based adversarial attacks against DDPM-based purification. We compare DiffAttack with Diff-BPDA attack [4] on CIFAR-10, and the results in Table 3 demonstrate that DiffAttack outperforms the baseline by a large margin under both  $\ell_{\infty}$  and  $\ell_{2}$  attacks.
4.4 Comparison with other adaptive attack methods
Besides the AdjAttack and Diff-BPDA attacks against existing diffusion-based purification defenses, we also compare DiffAttack with other general types of adaptive attacks: 1) black-box attack SPSA [53] and 2) square attack [1], as well as 3) adaptive attack against score-based generative models joint attack (Score / Full) [60]. SPSA attack approximates the gradients by randomly sampling from a pre-defined distribution and using the finite-difference method. Square attack heuristically searches for adversarial examples in a
Table 4: Robust accuracy (\%) of DiffAttack compared with other attack methods on CIFAR-10 with WideResNet28-10 under  $\ell_{\infty}$  attack ( $\epsilon = 8/255$ ).
Method	Score-based	DDPM-based
SPSA	83.37	81.29
Square Attack	82.81	81.68
Joint Attack (Score)	72.74	-
Joint Attack (Full)	77.83	76.26
Diff-BPDA	78.13	75.00
AdjAttack	70.64	-
DiffAttack	46.88	54.69
low-dimensional space with the constraints of perturbation patterns. Joint attack (score) updates the input by the average of the classifier gradient and the output of the score estimation network, while joint attack (full) leverages the classifier gradients and the difference between the input and the purified samples. The results in Table 4 show that DiffAttack outperforms SPSA, square attack, and joint attack by a large margin on score-based and DDPM-based purification defenses. Note that joint attack (score) cannot be applied to the DDPM-based pipeline due to the lack of a score estimator. AdjAttack fails on the DDPM-based pipeline since it can only calculate gradients through SDE.
4.5 Robustness with different diffusion lengths
We observe that the diffusion length plays an extremely important role in the effectiveness of adversarial purification. Existing DDPM-based purification works [56, 55] prefer a small diffusion length, but we find it vulnerable under our DiffAttack. The influence of the diffusion length  $T$  on the performance (clean/robust accuracy) of the purification defense methods is illustrated in Figure 2. We observe that 1) the clean accuracy of the purification
defenses negatively correlates with the diffusion lengths since the longer diffusion process adds more noise to the input and induces inaccurate reconstruction of the input sample; and 2) a moderate diffusion length benefits the robust accuracy since diffusion-based purification with a small length makes it easier to compute the gradients for attacks, while models with a large diffusion length have poor clean accuracy that deteriorates the robust accuracy. We also validate the conclusion on ImageNet in Appendix D.3.
(a) Score-based purification
Figure 2: The clean/robust accuracy  $(\%)$  of diffusion-based purification with different diffusion length  $T$  under DiffAttack on CIFAR-10 with WideResNet-28-10 under  $\ell_{\infty}$  attack  $(\epsilon = 8 / 255)$ .
(b) DDPM-based purification
4.6 Comparison of memory cost
Recent work [4] computes the gradients of the diffusion and sampling process to perform the gradient-based attack, but it only considers a small diffusion length (e.g., 14 on CIFAR-10). They construct the computational graph once and for all, which is extremely expensive for memory cost with a large diffusion length. We use a segment-wise forwarding-backwarding algorithm in Section 3.3 to avoid allocating the memory for the whole computational graph. In this part, we validate the memory efficiency of our approach compared to [4]. The results
Figure 3: Comparison of memory cost of gradient backpropagation between [4] and DiffAttack with batch size 16 on CIFAR-10 with WideResNet-28-10 under  $\ell_{\infty}$  attack.
in Figure 3 demonstrate that 1) the gradient backpropagation of [4] has the memory cost linearly correlated to the diffusion length and does not scale up to the diffusion length of 30, while 2) DiffAttack has almost constant memory cost and is able to scale up to extremely large diffusion length  $(T = 1000)$ . The evaluation is done on an RTX A6000 GPU. In Appendix D.3, we provide comparisons of runtime between DiffAttack and [4] and demonstrate that DiffAttack reduces the memory cost with comparable runtime.
4.7 Influence of applying the deviated-reconstruction loss at different time steps
We also show that the time steps at which we apply the deviated-reconstruction loss also influence the effectiveness of DiffAttack. Intuitively, the loss added at small time steps does not suffer from vanishing/exploding gradients but lacks supervision at consequent time steps, while the loss added at large time steps gains strong supervision but suffers from the gradient problem. The results in Figure 4 show that adding deviated-reconstruction loss to uniformly sampled time steps (Uni(0,T)) achieves the best attack performance and tradeoff compared with that of adding loss to the same number of partial time steps only at the initial stage  $((0,T / 3))$  or the final stage  $((2T / 3,T))$ . For fair comparisons, we uniformly sample  $T / 3$  time steps (identical to partial stage guidance  $(0,T / 3)$ ,  $(2T / 3,T)$ ) to impose  $\mathcal{L}_{\mathrm{dev}}$ .
Figure 4: The impact of applying  $\mathcal{L}_{dev}$  at different time steps on decreased robust accuracy  $(\%)$ .  $T$  is the diffusion length and  $\mathrm{Uni}(0,T)$  represents uniform sampling.
Adversarial purification methods purify the adversarial input before classification with generative models. Defense-gan [43] trains a GAN to restore the clean samples. Pixeldefend [48] utilizes an autoregressive model to purify adversarial examples. Another line of research [50, 22, 17, 24, 60] leverages energy-based model (EBM) and Markov chain Monte Carlo (MCMC) to perform the purification. More recently, diffusion models have seen wide success in image generation [15, 40, 41, 42, 31, 39]. They are also used to adversarial purification [34, 4, 62, 57, 51, 55, 56] and demonstrated to achieve the state-of-the-art robustness. In this work, we propose DiffAttack specifically against diffusion-based purification and show the effectiveness in different settings, which motivates future work to improve the robustness of the pipeline.
Adversarial attacks search for visually imperceptible signals which can significantly perturb the prediction of models [52, 20]. Different kinds of defense methods are progressively broken by advanced attack techniques, including white-box attack [6, 2, 32] and black-box attack [1, 53, 35]. [11, 12, 37, 59, 29] propose a systematic and automatic framework to attack existing defense methods. Despite attacking most defense methods, these approaches are shown to be ineffective against the diffusion-based purification pipeline due to the problem of vanishing/exploding gradient, memory cost, and randomness. Therefore, we propose DiffAttack to specifically tackle the challenges and successfully attack the diffusion-based purification defenses.
In this paper, we propose DiffAttack, including the deviated-reconstruction loss added on intermediate samples and a segment-wise forwarding-backwarding algorithm. We empirically demonstrate that DiffAttack outperforms existing adaptive attacks against diffusion-based purification by a large margin. We conduct a series of ablation studies and show that a moderate diffusion length benefits the model robustness, and attacks with the deviated-reconstruction loss added over uniformly sampled time steps outperform that added over only initial/final time steps, which will help to better understand the properties of diffusion process and motivate future defenses.
Acknowledgement. This work is partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, No. 2229876, DARPA GARD, the National Aeronautics and Space Administration (NASA) under grant no. 80NSSC20M0229, the Alfred P. Sloan Fellowship, and the Amazon research award.
[52] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
[53] Jonathan Uesato, Brendan O'Donoghue, Pushmeet Kohli, and Aaron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In icml2018, 2018.
[54] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Ryan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306.11698, 2023.
[55] Jinyi Wang, Zhaoyang Lyu, Dahua Lin, Bo Dai, and Hongfei Fu. Guided diffusion model for adversarial purification. arXiv preprint arXiv:2205.14969, 2022.
[56] Quanlin Wu, Hang Ye, and Yuntian Gu. Guided diffusion model for adversarial purification from random noise. arXiv preprint arXiv:2206.10875, 2022.
[57] Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anandkumar, Bo Li, and Dawn Song. Densepure: Understanding diffusion models towards adversarial robustness. arXiv preprint arXiv:2211.00322, 2022.
[58] Haotian Xue, Alexandre Araujo, Bin Hu, and Yongxin Chen. Diffusion-based adversarial sample generation for improved stealthiness and controllability. arXiv preprint arXiv:2305.16494, 2023.
[59] Chengyuan Yao, Pavol Bielik, Petar Tsankov, and Martin Vechev. Automated discovery of adaptive attacks on adversarial defenses. Advances in Neural Information Processing Systems, 34:26858-26870, 2021.
[60] Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning, pages 12062-12072. PMLR, 2021.
[61] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
[62] Kui Zhang, Hang Zhou, Jie Zhang, Qidong Huang, Weiming Zhang, and Nenghai Yu. Ada3diff: Defending against 3d adversarial point clouds via adaptive diffusion. arXiv preprint arXiv:2211.16247, 2022.
A Broader Impact and Limitations
Broader impact. As an effective and popular way to explore the vulnerabilities of ML models, adversarial attacks have been widely studied. However, recent diffusion-based purification is shown hard to attack based on different trials, which raises an interesting question of whether it can be attacked. Our paper provides the first effective attack against such defenses to identify the vulnerability of diffusion-based purification for the community and inspire more effective defense approaches. In particular, we propose an effective evasion attack against diffusion-based purification defenses which consists of a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation and a segment-wise forwarding-backwarding algorithm to achieve memory-efficient gradient backpropagation. The effectiveness of the deviated-reconstruction loss helps us to better understand the properties of diffusion purification. Concretely, there exist adversarial regions in the intermediate sample space where the score approximation model outputs inaccurate density gradients and finally misleads the prediction. The observation motivates us to design a more robust sampling process in the future, and one potential way is to train a more robust score-based model. Furthermore, the segment-wise forwarding-backwarding algorithm tackles the memory issue of gradient propagation through a long path. It can be applied to the gradient calculation of any discrete Markov process almost within a constant memory cost. To conclude, our attack motivates us to rethink the robustness of a line of SOTA diffusion-based purification defenses and inspire more effective defenses.
Limitations. In this paper, we propose an effective attack algorithm DiffAttack against diffusion-based purification defenses. A possible negative societal impact may be the usage of DiffAttack in safety-critical scenarios such as autonomous driving and medical imaging analysis to mislead the prediction of machine learning models. However, the foundation of DiffAttack and important findings about the diffusion process properties can benefit our understanding of the vulnerabilities of diffusion-based purification defenses and therefore motivate more effective defenses in the future. Concretely, the effectiveness of DiffAttack indicates that there exist adversarial regions in the intermediate sample space where the score approximation model outputs inaccurate density gradients and finally misleads the prediction. The observation motivates us to design a more robust sampling process in the future, and one potential way is to train a more robust score-based model. Furthermore, to control a robust sampling process, it is better to provide guidance across uniformly sampled time steps rather than only at the final stage according to our findings.