SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mintong Kang
mintong2@illinois.edu
dawnsong@berkeley.edu
lbo@illinois.edu
Diffusion-based purification defenses leverage diffusion models to remove crafted perturbations of adversarial examples and achieve state-of-the-art robustness. Recent studies show that even advanced attacks cannot break such defenses effectively, since the purification process induces an extremely deep computational graph which poses the potential problem of vanishing/exploding gradient, high memory cost, and unbounded randomness. In this paper, we propose an attack technique DiffAttack to perform effective and efficient attacks against diffusion-based purification defenses, including both DDPM and score-based approaches. In particular, we propose a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation to tackle the problem of vanishing/exploding gradients. We also provide a segment-wise forwarding-backwarding algorithm, which leads to memory-efficient gradient backpropagation. We validate the attack effectiveness of DiffAttack compared with existing adaptive attacks on CIFAR-10 and ImageNet. We show that DiffAttack decreases the robust accuracy of models compared with SOTA attacks by over  $20\%$  on CIFAR-10 under  $\ell_{\infty}$  attack ( $\epsilon = 8/255$ ), and over  $10\%$  on ImageNet under  $\ell_{\infty}$  attack ( $\epsilon = 4/255$ ). We conduct a series of ablations studies, and we find 1) DiffAttack with the deviated-reconstruction loss added over uniformly sampled time steps is more effective than that added over only initial/final steps, and 2) diffusion-based purification with a moderate diffusion length is more robust under DiffAttack.
Since deep neural networks (DNNs) are found vulnerable to adversarial perturbations [52, 20], improving the robustness of neural networks against such crafted perturbations has become important, especially in safety-critical applications [18, 5, 54]. In recent years, many defenses have been proposed, but they are attacked again by more advanced adaptive attacks [7, 30, 11, 12]. One recent line of defense (diffusion-based purification) leverages diffusion models to purify the input images and achieves the state-of-the-art robustness. Based on the type of diffusion models the defense utilizes, diffusion-based purification can be categorized into score-based purification [34] which uses the score-based diffusion model [49] and DDPM-based purification [4, 62, 57, 51, 55, 56] which uses the denoising diffusion probabilistic model (DDPM) [25]. Recent studies show that even the most advanced attacks [12, 34] cannot break these defenses due to the challenges of vanishing/exploding gradient, high memory cost, and large randomness. In this paper, we aim to explore the vulnerabilities of such diffusion-based purification defenses, and design a more effective and efficient adaptive attack against diffusion-based purification, which will help to better understand the properties of diffusion process and motivate future defenses.
In particular, the diffusion-based purification defenses utilize diffusion models to first diffuse the adversarial examples with Gaussian noises and then perform sampling to remove the noises. In this way, the hope is that the crafted adversarial perturbations can also be removed since the training
distribution of diffusion models is clean [49, 25]. The diffusion length (i.e., the total diffusion time steps) is usually large, and at each time step, the deep neural network is used to estimate the gradient of the data distribution. This results in an extremely deep computational graph that poses great challenges of attacking it: vanishing/exploding gradients, unavailable memory cost, and large randomness. To tackle these challenges, we propose a deviated-reconstruction loss and a segment-wise forwarding-backwarding algorithm and integrate them as an effective and efficient attack technique DiffAttack.
Essentially, our deviated-reconstruction loss pushes the reconstructed samples away from the diffused samples at corresponding time steps. It is added at multiple intermediate time steps to relieve the problem of vanishing/exploding gradients. We also theoretically analyze the connection between it and the score-matching loss [26], and we prove that maximizing the deviated-reconstruction loss induces inaccurate estimation of the density gradient of the data distribution, leading to a higher chance of attacks. To overcome the problem of large memory cost, we propose a segment-wise forwarding-backwarding algorithm to backpropagate the gradients through a long path. Concretely, we first do a forward pass and store intermediate samples, and then iteratively simulate the forward pass of a segment and backward the gradient following the chain rule. Ignoring the memory cost induced by storing samples (small compared with the computational graph), our approach achieves  $\mathcal{O}(1)$  memory cost.
Finally, we integrate the deviated-reconstruction loss and segment-wise forwarding-backwarding algorithm into DiffAttack, and empirically validate its effectiveness on CIFAR-10 and ImageNet. We find that (1) DiffAttack outperforms existing attack methods [34, 60, 53, 1, 2] by a large margin for both the score-based purification and DDPM-based purification defenses, especially under large perturbation radii; (2) the memory cost of our efficient segment-wise forwarding-backwarding algorithm does not scale up with the diffusion length and saves more than  $10\mathrm{x}$  memory cost compared with the baseline [4]; (3) a moderate diffusion length benefits the robustness of the diffusion-based purification since longer length will hurt the benign accuracy while shorter length makes it easier to be attacked; (4) attacks with the deviated-reconstruction loss added over uniformly sampled time steps outperform that added over only initial/final time steps. The effectiveness of DiffAttack and interesting findings will motivate us to better understand and rethink the robustness of diffusion-based purification defenses.
We summarize the main technical contributions as follows:
- We propose DiffAttack, a strong evasion attack against the diffusion-based adversarial purification defenses, including score-based and DDPM-based purification.
- We propose a deviated-reconstruction loss to tackle the problem of vanishing/exploding gradient, and theoretically analyze its connection with data density estimation.
- We propose a segment-wise forwarding-backwarding algorithm to tackle the high memory cost challenge, and we are the first to adaptively attack the DDPM-based purification defense, which is hard to attack due to the high memory cost.
- We empirically demonstrate that DiffAttack outperforms existing attacks by a large margin on CIFAR-10 and ImageNet. Particularly, DiffAttack decreases the model robust accuracy by over  $20\%$  for  $\ell_{\infty}$  attack ( $\epsilon = 8/255$ ) on CIFAR-10, and over  $10\%$  on ImageNet under  $\ell_{\infty}$  attack ( $\epsilon = 4/255$ ).
- We conduct a series of ablation studies and show that (1) a moderate diffusion length benefits the model robustness, and (2) attacks with the deviated-reconstruction loss added over uniformly sampled time steps outperform that added over only initial/final time steps.
There are two types of diffusion-based purification defenses, DDPM-based purification, and score-based purification, which leverage DDPM [46, 25] and score-based diffusion model [49] to purify the adversarial examples, respectively. Next, we will introduce the basic concepts of DDPM and score-based diffusion models.
Denote the diffusion process indexed by time step  $t$  with the diffusion length  $T$  by  $\{\mathbf{x}_t\}_{t=0}^T$ . DDPM constructs a discrete Markov chain  $\{\mathbf{x}_t\}_{t=0}^T$  with discrete time variables  $t$  following  $p(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t}\mathbf{x}_{t-1}, \beta_t\mathbf{I})$  where  $\beta_t$  is a sequence of positive noise scales (e.g., linear scheduling, cosine scheduling [33]). Considering  $\alpha_t := 1 - \beta_t$ ,  $\bar{\alpha}_t := \Pi_{s=1}^t\alpha_s$ , and
$\sigma_{t} = \sqrt{\beta_{t}(1 - \bar{\alpha}_{t - 1}) / (1 - \bar{\alpha}_{t})}$ , the reverse process (i.e., sampling process) can be formulated as:
$$
\mathbf {x} _ {t - 1} = \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathbf {x} _ {t} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\theta} (\mathbf {x} _ {t}, t)\right) + \sigma_ {t} \mathbf {z} \tag {1}
$$
where  $\mathbf{z}$  is drawn from  $\mathcal{N}(\mathbf{0},\mathbf{I})$ .  $\epsilon_{\theta}$  parameterized with  $\theta$  is the model to approximate the perturbation  $\epsilon$  in the diffusion process and is trained via the density gradient loss  $\mathcal{L}_d$ :
$$
\mathcal {L} _ {d} = \mathbb {E} _ {t, \epsilon} \left[ \frac {\beta_ {t} ^ {2}}{2 \sigma_ {t} ^ {2} \alpha_ {t} (1 - \bar {\alpha} _ {t})} \| \epsilon - \epsilon_ {\theta} (\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon , t) \| _ {2} ^ {2} \right] \tag {2}
$$
where  $\epsilon$  is drawn from  $\mathcal{N}(\mathbf{0},\mathbf{I})$  and  $t$  is uniformly sampled from  $[T]\coloneqq \{1,2,\dots,T\}$
Score-based diffusion model formulates diffusion models with stochastic differential equations (SDE). The diffusion process  $\{\mathbf{x}_t\}_{t=0}^T$  is indexed by a continuous time variable  $t \in [0,1]$ . The diffusion process can be formulated as:
$$
d \mathbf {x} = f (\mathbf {x}, t) d t + g (t) d \mathbf {w} \tag {3}
$$
where  $f(\mathbf{x},t):\mathbb{R}^n\mapsto \mathbb{R}^n$  is the drift coefficient characterizing the shift of the distribution,  $g(t)$  is the diffusion coefficient controlling the noise scales, and  $\mathbf{w}$  is the standard Wiener process. The reverse process is characterized via the reverse time SDE of Equation (3):
$$
d \mathbf {x} = [ f (\mathbf {x}, t) - g (t) ^ {2} \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}) ] d t + g (t) d \mathbf {w} \tag {4}
$$
where  $\nabla_{\mathbf{x}}\log p_t(\mathbf{x})$  is the time-dependent score function that can be approximated with neural networks  $\mathbf{s}_{\theta}$  parameterized with  $\theta$ , which is trained via the score matching loss  $\mathcal{L}_s$  [26, 47]:
$$
\mathcal {L} _ {s} = \mathbb {E} _ {t} \left[ \lambda (t) \mathbb {E} _ {\mathbf {x} _ {t} | \mathbf {x} _ {0}} \| \mathbf {s} _ {\theta} (\mathbf {x} _ {t}, t) - \nabla_ {\mathbf {x} _ {t}} \log (p (\mathbf {x} _ {t} | \mathbf {x} _ {0})) \| _ {2} ^ {2} \right] \tag {5}
$$
where  $\lambda :[0,1]\to \mathbb{R}$  is a weighting function and  $t$  is uniformly sampled over  $[0,1]$ .
3 DiffAttack
3.1 Evasion attacks against diffusion-based purification
A class of defenses leverages generative models for adversarial purification [43, 48, 45, 60]. The adversarial images are transformed into latent representations, and then the purified images are sampled starting from the latent space using the generative models. The process is expected to remove the crafted perturbations since the training distribution of generative models is assumed to be clean. With diffusion models showing the power of image generation recently [15, 39], diffusion-based adversarial purification has achieved SOTA defense performance [34, 4].
We first formulate the problem of evasion attacks against diffusion-based purification defenses. Suppose that the process of diffusion-based purification, including the diffusion and reverse process, is denoted by  $P: \mathbb{R}^n \mapsto \mathbb{R}^n$  where  $n$  is the dimension of the input  $\mathbf{x}_0$ , and the classifier is denoted by  $F: \mathbb{R}^n \mapsto [K]$  where  $K$  is the number of classes. Given an input pair  $(\mathbf{x}_0, y)$ , the adversarial example  $\tilde{\mathbf{x}}_0$  satisfies:
$$
\underset {i \in [ K ]} {\arg \max } F _ {i} \left(P \left(\tilde {\mathbf {x}} _ {0}\right)\right) \neq y \quad s. t. d \left(\mathbf {x} _ {0}, \tilde {\mathbf {x}} _ {0}\right) \leq \delta_ {m a x} \tag {6}
$$
where  $F_{i}(\cdot)$  is the  $i$ -th element of the output,  $d:\mathbb{R}^n\times \mathbb{R}^n\mapsto \mathbb{R}$  is the distance function in the input space, and  $\delta_{max}$  is the perturbation budget.
Since directly searching for the adversarial instance  $\tilde{\mathbf{x}}_0$  based on Equation (6) is challenging, we often use a surrogate loss  $\mathcal{L}$  to solve an optimization problem:
$$
\max  _ {\tilde {\mathbf {x}} _ {0}} \mathcal {L} (F (P (\tilde {\mathbf {x}} _ {0})), y) \quad s. t. d \left(\mathbf {x} _ {0}, \tilde {\mathbf {x}} _ {0}\right) \leq \delta_ {m a x} \tag {7}
$$
where  $P(\cdot)$  is the purification process with DDPM (Equation (1)) or score-based diffusion (Equations (3) and (4)), and the surrogate loss  $\mathcal{L}$  is often selected as the classification-guided loss, such as CW loss [7], Cross-Entropy loss and difference of logits ratio (DLR) loss [12]. Existing adaptive attack methods such as PGD [30] and APGD attack [12] approximately solve the optimization problem in Equation (7) via computing the gradients of loss  $\mathcal{L}$  with respect to the decision variable  $\tilde{\mathbf{x}}_0$  and iteratively updating  $\tilde{\mathbf{x}}_0$  with the gradients.
Figure 1: DiffAttack against diffusion-based adversarial purification defenses. DiffAttack features the deviated-reconstruction loss that addresses vanishing/exploding gradients and the segment-wise forwarding-backwarding algorithm that leads to memory-efficient gradient backpropagation.
However, we observe that the gradient computation for the diffusion-based purification process is challenging for three reasons: 1) the long sampling process of the diffusion model induces an extremely deep computational graph which poses the problem of vanishing/exploding gradient [2], 2) the deep computational graph impedes gradient backpropagation, which requires high memory cost [60, 4], and 3) the diffusion and sampling process introduces large randomness which makes the calculated gradients unstable and noisy.
To address these challenges, we propose a deviated-reconstruction loss (in Section 3.2) and a segmentwise forwarding-backwarding algorithm (in Section 3.3) and design an effective algorithm DiffAttack by integrating them into the attack technique (in Section 3.4).
3.2 Deviated-reconstruction loss
In general, the surrogate loss  $\mathcal{L}$  in Equation (7) is selected as the classification-guided loss, such as CW loss, Cross-Entropy loss, or DLR loss. However, these losses can only be imposed at the classification layer, and induce the problem of vanishing/exploding gradients [2] due to the long diffusion length. Specifically, the diffusion purification process induces an extremely deep graph. For example, DiffPure applies hundreds of iterations of sampling and uses deep UNet with tens of layers as score estimators. Thus, the computational graph consists of thousands of layers, which could cause the problem of gradient vanishing/exploding. Similar gradient problems are also mentioned with generic score-based generative purification (Section 4, 5.1 in [60]). Backward path differentiable approximation (BPDA) attack [2] is usually adopted to overcome such problems, but the surrogate model of the complicated sampling process is hard to find, and a simple identity mapping function is demonstrated to be ineffective in the case [34, 4, 60].
To overcome the problem of exploding/vanishing gradients, we attempt to impose intermediate guidance during the attack. It is possible to build a set of classifiers on the intermediate samples in the reverse process and use the weighted average of the classification-guided loss at multiple layers as the surrogate loss  $\mathcal{L}$ . However, we observe that the intermediate samples are noisy, and thus using classifier  $F$  that is trained on clean data cannot provide effective gradients. One solution is to train a set of classifiers with different noise scales and apply them to intermediate samples to impose classification-guided loss, but the training is too expensive considering the large diffusion length and variant noise scales at different time steps. Thus, we propose a deviated-reconstruction loss to address the challenge via imposing discrepancy for samples between the diffusion and reverse processes adversarially to provide effective loss at intermediate time steps.
Concretely, since a sequence of samples is generated in the diffusion and reverse processes, effective loss imposed on them would relieve the problem of vanishing/exploding gradient and benefit the optimization. More formally, let  $\mathbf{x}_t$ ,  $\mathbf{x}_t^\prime$  be the samples at time step  $t$  in the diffusion process and the reverse process, respectively. Formally, we maximize the deviated-reconstruction loss  $\mathcal{L}_{dev}$
formulated as follows:
$$
\max  \mathcal {L} _ {d e v} = \mathbb {E} _ {t} [ \alpha (t) \mathbb {E} _ {\mathbf {x} _ {t}, \mathbf {x} _ {t} ^ {\prime} | \mathbf {x} _ {0}} d (\mathbf {x} _ {t}, \mathbf {x} _ {t} ^ {\prime}) ] \tag {8}
$$
where  $\alpha (\cdot)$  is time-dependent weight coefficients and  $d(\mathbf{x}_t,\mathbf{x}_t^{\prime})$  is the distance between noisy image  $\mathbf{x}_t$  in the diffusion process and corresponding sampled image  $\mathbf{x}_t^\prime$  in the reverse process. The expectation over  $t$  is approximated by taking the average of results at uniformly sampled time steps in  $[0,T]$ , and the loss at shallow layers in the computational graph (i.e., large time step  $t$ ) helps relieve the problem of vanishing/exploding gradient. The conditional expectation over  $\mathbf{x}_t,\mathbf{x}_t^\prime$  given  $\mathbf{x}_0$  is approximated by purifying  $\mathbf{x}_0$  multiple times and taking the average of the loss.
Intuitively, the deviated-reconstruction loss in Equation (8) pushes the reconstructed sample  $\mathbf{x}_t^\prime$  in the reverse process away from the sample  $\mathbf{x}_t$  at the corresponding time step in the diffusion process, and finally induces an inaccurate reconstruction of the clean image. Letting  $q_{t}(\mathbf{x})$  and  $q_{t}^{\prime}(\mathbf{x})$  be the distribution of  $\mathbf{x}_t$  and  $\mathbf{x}_t^\prime$ , we can theoretically prove that the distribution distance between  $q_{t}(\mathbf{x})$  and  $q_{t}^{\prime}(\mathbf{x})$  positively correlates with the score-matching loss of the score-based diffusion or the density gradient loss of the DDPM. In other words, maximizing the deviated-reconstruction loss in Equation (8) induces inaccurate data density estimation, which results in the discrepancy between the sampled distribution and the clean training distribution.
Theorem 1. Consider adversarial sample  $\tilde{\mathbf{x}}_0\coloneqq \mathbf{x}_0 + \delta$  where  $\mathbf{x}_0$  is the clean example and  $\delta$  is the perturbation.  $p_t(\mathbf{x}),p_t'(\mathbf{x}),q_t(\mathbf{x}),q_t'(\mathbf{x})$  are the distribution of  $\mathbf{x}_t,\mathbf{x}_t'$ ,  $\tilde{\mathbf{x}}_t,\tilde{\mathbf{x}}_t'$  where  $\mathbf{x}_t^\prime$  represents the reconstruction of  $\mathbf{x}_t$  in the reverse process.  $D_{TV}(\cdot ,\cdot)$  measures the total variation distance. Given a VP-SDE parameterized by  $\beta (\cdot)$  and the score-based model  $s_\theta$  with mild assumptions that  $\| \nabla_{\mathbf{x}}\log p_t(\mathbf{x}) - s_\theta (\mathbf{x},t)\| _2^2\leq L_u$ ,  $D_{TV}(p_t,p_t')\leq \epsilon_{re}$ , and a bounded score function by  $M$  (specified in Appendix C.1), we have:
$$
D _ {T V} \left(q _ {t}, q _ {t} ^ {\prime}\right) \leq \frac {1}{2} \sqrt {\mathbb {E} _ {t , \mathbf {x} | \mathbf {x} _ {0}} \| \boldsymbol {s} _ {\theta} (\mathbf {x} , t) - \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} + C _ {1}} + \sqrt {2 - 2 \exp \{- C _ {2} \| \delta \| _ {2} ^ {2} \}} + \epsilon_ {r e} \tag {9}
$$
$$
C _ {1} = \left(L _ {u} + 8 M ^ {2}\right) \int_ {t} ^ {T} \beta (t) d t, C _ {2} = \left(8 \bigl (1 - \Pi_ {s = 1} ^ {t} \bigl (1 - \beta_ {s} \bigr) \bigr)\right) ^ {- 1}.
$$
Proof sketch. We first use the triangular inequality to upper bound  $D_{TV}(q_t, q_t')$  with  $D_{TV}(q_t, p_t) + D_{TV}(p_t, p_t') + D_{TV}(p_t', q_t')$ .  $D_{TV}(q_t, p_t)$  can be upper bounded by a function of the Hellinger distance  $H(q_t, p_t)$ , which can be calculated explicitly.  $D_{TV}(p_t, p_t')$  can be upper bounded by the reconstruction error  $\epsilon_{re}$  by assumption. To upper bound  $D_{TV}(p_t', q_t')$ , we can leverage Pinker's inequality to alternatively upper bound the KL-divergence between  $p_t'$  and  $q_t'$  which can be derived by using the Fokker-Planck equation [44] in the reverse SDE.
Remark. A large deviated-reconstruction loss can indicate a large total variation distance  $D_{TV}(q_t, q_t')$ , which is the lower bound of a function with respect to the score-matching loss  $\mathbb{E}_{t,\mathbf{x}} \| s_\theta(\mathbf{x}, t) - \nabla_\mathbf{x} \log q_t'(\mathbf{x}) \|_2^2$  (in RHS of Equation (9)). Therefore, we show that maximizing the deviated-reconstruction loss implicitly maximizes the score-matching loss, and thus induces inaccurate data density estimation to perform an effective attack. The connection of deviated-reconstruction loss and the density gradient loss for DDPM is provided in Thm. 3 in Appendix C.2.
3.3 Segment-wise forwarding-backwarding algorithm
Adaptive attacks against diffusion-based purification require gradient backpropagation through the forwarding path. For diffusion-based purification, the memory cost scales linearly with the diffusion length  $T$  and is not feasible in a realistic application. Therefore, existing defenses either use a surrogate model for gradient approximation [55, 56, 60, 45] or consider adaptive attacks only for a small diffusion length [4], but the approximation can induce error and downgrade the attack performance a lot. Recently, DiffPure [34] leverages the adjoint method [28] to backpropagate the gradient of SDE within reasonable memory cost and enables adaptive attacks against score-based purification. However, it cannot be applied to a discrete process, and the memory-efficient gradient backpropagation algorithm is unexplored for DDPM. Another line of research [9, 8, 19] proposes the technique of gradient checkpointing to perform gradient backpropagation with memory efficiency. Fewer activations are stored during forwarding passes, and the local computation graph is constructed via recomputation. However, we are the first to apply the memory-efficient backpropagation technique to attack diffusion purification defenses and resolve the problem of memory cost during attacks, which is realized as a challenging problem by prior attacks against purification defenses [34, 60]. Concretely, we propose a segment-wise forwarding-backwarding algorithm, which leads to memory-efficient gradient computation of the attack loss with respect to the adversarial examples.
We first feed the input  $\mathbf{x}_0$  to the diffusion-based purification process and store the intermediate samples  $\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_T$  in the diffusion process and  $\mathbf{x}_T', \mathbf{x}_{T-1}', \dots, \mathbf{x}_0'$  in the reverse process sequentially. For ease of notation, we have  $\mathbf{x}_{t+1} = f_d(\mathbf{x}_t)$  and  $\mathbf{x}_t' = f_r(\mathbf{x}_{t+1}')$  for  $t \in [0, T-1]$ . Then we can backpropagate the gradient iteratively following:
$$
\frac {\partial \mathcal {L}}{\partial \mathbf {x} _ {t + 1} ^ {\prime}} = \frac {\partial \mathcal {L}}{\partial \mathbf {x} _ {t} ^ {\prime}} \frac {\partial \mathbf {x} _ {t} ^ {\prime}}{\partial \mathbf {x} _ {t + 1} ^ {\prime}} = \frac {\partial \mathcal {L}}{\partial \mathbf {x} _ {t} ^ {\prime}} \frac {\partial f _ {r} \left(\mathbf {x} _ {t + 1} ^ {\prime}\right)}{\partial \mathbf {x} _ {t + 1} ^ {\prime}} \tag {10}
$$