SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haotian Xue  $^{*1}$  Alexandre Araujo  $^{2}$  Bin Hu  $^{3}$  Yongxin Chen
$^{2}$  New York University
<sup>3</sup> University of Illinois Urbana-Champaign
Neural networks are known to be susceptible to adversarial samples: small variations of natural examples crafted to deliberately mislead the models. While they can be easily generated using gradient-based techniques in digital and physical scenarios, they often differ greatly from the actual data distribution of natural images, resulting in a trade-off between strength and stealthiness. In this paper, we propose a novel framework dubbed Diffusion-Based Projected Gradient Descent (Diff-PGD) for generating realistic adversarial samples. By exploiting a gradient guided by a diffusion model, Diff-PGD ensures that adversarial samples remain close to the original data distribution while maintaining their effectiveness. Moreover, our framework can be easily customized for specific tasks such as digital attacks, physical-world attacks, and style-based attacks. Compared with existing methods for generating natural-style adversarial samples, our framework enables the separation of optimizing adversarial loss from other surrogate losses (e.g., content/smoothness/style loss), making it more stable and controllable. Finally, we demonstrate that the samples generated using Diff-PGD have better transferability and anti-purification power than traditional gradient-based methods. Code is available at https://github.com/xavihart/Diff-PGD
Neural networks have demonstrated remarkable capabilities in learning features and achieving outstanding performance in various downstream tasks. However, these networks are known to be vulnerable to subtle perturbations, called adversarial samples (adv-samples), leading to important security concerns for critical-decision systems [16]. Numerous threat models have been designed to generate adversarial samples for both digital space attacks [31, 5], physical world attacks [26, 14], and also for generating more customized adv-samples given prompts like attack region and style reference [12, 17]. However, most of these models have not been designed to maintain the realism of output samples, resulting in adversarial samples that deviate significantly from the distribution of natural images [22]. Indeed, in the setting of digital world attacks, such as those involving RGB images, higher success rates, and transferability are associated with larger changes in the generated samples, leading to a stealthiness-effectiveness trade-off. Additionally, perturbations for physical-world attacks are substantial, non-realistic, and easily noticeable by humans [47].
Recent works, for both digital and physical settings, have proposed methods to generate more realistic adversarial samples. For example, [29, 24] introduced a method to optimize the perturbations added to clean images in semantic space, [23, 26] proposed a GAN-based approach or other prior knowledge [33] to increase the realism of generated samples. Although the GAN-based approach
* Correspondence to: hxue.ai@gatech.edu
Table 1: Properties of Diff-PGD vs Other Attacks: We summarize six metrics for adv-sample generation. "Stealthiness" measures whether the adversarial perturbations can be detected by human observers. "Scenarios" describes the setting in which the method can be applied:  $D$  corresponds to digital attacks and  $P$  for physical attacks. "Controllability" measures whether the method can support customized prompts e.g., mask/style reference. "Anti-Purify" measures the ability of the samples to avoid being purified. "Transferability" measures the generalization of the attack to different models. Finally, "Stability" describes the stability of the optimization. (-) stands for non-consideration.
Methods	Stealthiness	Scenarios	Controllability	Anti-Purify	Transferability	Stability
PGD [31]	**	D	*	*	*	**
AdvPatch [4]	*	P	**	(-)	(-)	**
NatPatch [23]	**	P	*	(-)	(-)	**
AdvArt [17]	*	P	**	(-)	(-)	**
AdvCam [12]	**	D/P	**	(-)	(-)	*
Diff-PGD (Ours)	**	D/P	**	**	**	**
can generate realistic images, the adversarial examples are sampled from noise and therefore lack controllability. A subsequent line of research on realistic adversarial samples [22] has introduced the concept of semantic adversarial samples. These are unbounded perturbations that deceive the model, while ensuring the modified image has the same semantic information as the original image. While transformations in semantic space can reduce the high-frequency patterns in pixel space, this approach still suffers from color jittering or image distortion which results in lower stealthiness [12]. Further, this approach needs careful tuning of the hyperparameters making the training process unstable.
In this paper, we propose a novel framework, which uses an off-the-shelf diffusion model to guide the optimization of perturbations, thus enabling the generation of adversarial samples with higher stealthiness. The core design of our framework is Diffusion-Based Projected Gradient Descent (Diff-PGD), which shares a similar structure with PGD but changes the input of the target classifier  $f_{\theta}$  to be purified version  $x_0$  of the original input  $x$ . To the best of our knowledge, we are the first to use the Diffusion Model to power the generation of adv-samples. Our framework can also be easily adapted to some customized attacks given masks or style references, thereby enhancing its controllability. Our framework (Figure 1(c)) separates the customized process with prompt  $p$  (region mask, style, etc.) with adversarial sample generation, overcoming the drawbacks of previous pipelines: the traditional gradient-based methods (Figure 1(a)) cannot guarantee to generate naturalistic samples when the perturbation level is large; the joint optimization with multiple losses like adv-loss  $l_{adv}$ , style loss  $l_z$  and realism loss  $l_r$  (see Figure 1(b)) is unstable to train and still tend to generate perceptible artifacts. Through extensive experiments in scenarios like digital attacks, masked region attacks, style-guided attacks, and physical world attacks, we show that our proposed framework can effectively generate realistic adv-samples with higher stealthiness and controllability. We further show that the Diff-PGD can help generate adversarial samples with higher transferability than traditional methods without gradient restriction. Our contribution can be summarized as follows:
1. We propose a novel framework Diff-PGD, combing the strong prior knowledge of the diffusion model into adv-sample generation, which helps generate adv-samples with high stealthiness and controllability. Adv-samples generated by Diff-PGD have good properties as described in Table 1.
2. We show that Diff-PGD can be effectively applied to many tasks including digital attacks, customized attacks, and physical-world attacks, outperforming baseline methods such as PGD, AdvPatch, and AdvCam.
3. We explore the transferability and anti-purification properties of Diff-PGD samples and show through an experimental evaluation that adversarial samples generated by Diff-PGD outperform the original PGD approach.
Adversarial attacks aim at maximizing the classification error of a target model without changing the semantics of the images. In this section, we review the types of attacks and existing works.
Norm bounded attacks. In the digital space, one can easily generate adversarial samples by using gradient-based methods such as Fast Gradient Sign Method (FGSM) [16] or Projected Gradient Decent (PGD) [31]. These methods aim at maximizing the loss of the target model with respect to the input and then projecting the perturbation to a specific  $\ell_p$  ball. It has been observed that these adversarial samples tend to diverge from the distribution of natural images [50]. Based on this observation, [34] have shown that these attacks can be "purified" using pre-trained diffusion models.
Semantic attacks. Some recent works operate semantic transformation to an image: [22] generate adv-samples in the HSV color space, [48, 13] generate adv-samples by rotating the 2D image or changing its brightness, and [52, 29, 10] try to generate semantic space transformations with an additional differentiable renderer. All these methods are far from effective; they either need additional modules or have color jitters and distortion [12].
Customized Attacks with Natural-Style. Significant efforts have been made toward generating customized adversarial samples (given region and/or style reference) with natural-style: AdvCAM [12] optimize adversarial loss together with other surrogate losses such as content loss, style loss, and smoothness loss to make the output adversarial sample to be realistic, [23, 26, 55] sample from latent space and then use a Generative Adversarial Network (GAN) to guarantee the natural style of the output sample to fool an object detector, and AdvArt [17] customized the generation of the adversarial patch with a given art style. However, all these methods share some common issues: optimizing adversarial loss and other surrogate losses (for content-preserving or customized style) needs careful balance and still results in unnatural artifacts in the output samples. Our methods use the diffusion model as a strong prior to better ensure the realism of generated samples.
Attacks in physical-world. For physical-world attacks, minor disturbances in digital space often prove ineffective due to physical transformations such as rotation, lighting, and viewing distance. In this context, adding adversarial patches to the scene is a common technique to get the target model to misclassify [4, 14, 1, 25]. Physical-world attacks do not impose any constraints either on the amount of perturbation or on the output style, thus, they tend to produce highly unnatural images that can be easily detected by human observers (i.e., lack stealthiness).
Diffusion Model. Diffusion Models (DMs) [19, 41] have demonstrated superior performance in many tasks such as text-to-image generation [37, 2, 38], video/story generation [21, 20, 35], 3D generation [28, 36] and neural science [44, 42]. Pre-trained DMs provide valuable prior knowledge that can be exploited for adversarial robustness. In this context, several works have proposed defense mechanisms such as adversarial purification [34, 53, 43, 34, 27] or DM-enhanced certified robustness [6, 46, 45].
The Denoised Diffusion Probabilistic Model (DDPM) [19] is a discretized version of DMs and works as follows. Suppose  $x_0 \sim p(x_0)$  is a sample from the distribution of natural images. The forward diffusion process gradually adds Gaussian noise to  $x_0$ , generating noisy samples  $[x_1, x_2, \dots, x_t, \dots, x_T]$  in  $T$  steps, following a Markov process defined as  $q_M(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I})$  where  $\mathcal{N}$  denotes Gaussian distribution  $\beta_t$  are fixed values growing from 0 to 1. By accumulating single step  $q_M$  we have  $q(x_t \mid x_0)$  as
$$
q \left(x _ {t} \mid x _ {0}\right) = \mathcal {N} \left(x _ {t}; \sqrt {\bar {\alpha} _ {t}} x _ {t - 1}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right) \tag {1}
$$
where  $\alpha_{t} = 1 - \beta_{t}$  and  $\bar{\alpha}_{t} = \Pi_{s=1}^{t} \alpha_{s}$ . When  $\bar{\alpha}_{t} \approx 0$ , the distribution of  $x_{T}$  becomes an isotropic Gaussian.
The reverse process aims at generating samples from the target data distribution from Gaussian noise  $x_{T} \sim \mathcal{N}(0, \mathbf{I})$  using the reversed diffusion process. The reverse model  $p_{\phi}(x_{t-1} \mid x_t)$  can be trained by optimizing the usual variational bound on negative log likelihood. Using the re-sampling strategy proposed in [19], we can simplify the optimization of  $\phi$  into a denoising process, by training a modified U-Net as a denoiser. Then we can generate  $x_0$  with high fidelity by sampling on the reversed diffusion:
$$
p \left(x _ {0: T}\right) = p \left(x _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\phi} \left(x _ {t - 1} \mid x _ {t}\right). \tag {2}
$$
For the reverse process, we define  $R_{\phi}$  parameterized by  $\phi$  as the function to denoise  $x_{t}$  and then get next sampled value as  $x_{t-1} = R_{\phi}(x_{t}, t)$ .
(c)
Figure 1: Comparison of Different Pipelines: (a) Traditional gradient-based adversarial sample generation,  $x$  is the sample to be optimized,  $l_{adv}$  is adversarial loss. (b) Customized adversarial sample generation with natural style (determined by prompt  $p$ ): joint optimization of adversarial loss with other surrogate losses like prompt loss  $l_p$  (e.g. style loss) and realistic loss  $l_r$  (e.g. content loss, smooth loss). (c) Our proposed diffusion-based framework,  $q$  is forward diffusion and  $R_{\phi}$  is backward denoising,  $x_0$  is the denoised sample.
SDEdit. The key idea of Stochastic Differential Editing (SDEdit) [32] is to diffuse the original distribution with  $x_{K} \sim q(x_{K} \mid x)$  and then run the reverse denoising process  $R_{\phi}$  parametrized by  $\phi$  in a Diffusion Model to get an edited sample:
$$
\operatorname {S D E d i t} (x, K) = R _ {\phi} \left(\dots R _ {\phi} \left(R _ {\phi} \left(x _ {K}, K\right), K - 1\right) \dots , 0\right). \tag {3}
$$
Given a raw input, SDEdit first runs  $K$  steps forward and then runs  $K$  steps backward. This process works as a bridge between the input distribution (which always deviates from  $p(x_0)$ ) and the realistic data distribution  $p(x_0)$ , which can then be used in tasks such as stroke-based image synthesis and editing. It can also be used in adversarial samples purification [34, 53].
4 Diffusion-based Projected Gradient Descent
We present Diffusion-based Projected Gradient Descent (Diff-PGD), a novel method to generate adversarial samples that looks realistic, using a pre-trained diffusion model. First, we present basic Diff-PGD that works on global image inputs, which shares the same bound settings with  $\ell_{\infty}$ -PGD in Section 4.1. Then, in Section 4.2 we generalize Diff-PGD to scenarios where only regional attacks defined by a customized mask are possible, using re-painting skills in the diffusion model [30]. Finally, we present extensions of Diff-PGD to customized attacks guided by image style in Section 4.3 and physical world attack in Section 4.4.
In the rest of the paper, we use superscript  $i$  or  $t$  to denote iteration of optimization, and subscript  $i$  to denote the timestep of diffusion model.
4.1 Diffusion-based Projected Gradient Descent
Given a clean image  $x$  with label  $y$  and a target classifier  $f_{\theta}$  parameterized by  $\theta$  to be attacked, our goal is to generate an adversarial sample  $x_{adv}$ , which fools the model. The traditional gradient-based methods use the gradient of adversarial loss  $g = \nabla_{x}l(f_{\theta}(x),y)$ , where  $f_{\theta}(x)$  return the logits. We also denote the loss as  $l_{adv}$  for simplicity to optimize  $x$  by iterations. The  $t$ -step update in the PGD with stepsize  $\eta$  and  $n$  iterations reads
$$
x ^ {t + 1} = \mathcal {P} _ {B _ {\infty} (x, \epsilon)} \left[ x ^ {t} + \eta \operatorname {s i g n} \nabla_ {x ^ {t}} l \left(f _ {\theta} \left(x ^ {t}\right), y\right) \right] \tag {4}
$$
where  $\mathcal{P}_{B_{\infty}(x,\epsilon)}(\cdot)$  is the projection operator on the  $\ell_{\infty}$  ball. Intuitively, the gradient will "guide" the sample  $x^{t}$  away from the decision boundary and deviate from the real distribution  $p(x_0)$  [29]. This out-of-distribution property makes them easier to be purified using some distribution-based purification methods [34], which considerably restricts the strength of the attacks.
Instead of using  $l(f_{\theta}(x),y)$  as the optimization cost, we turn to use logits of purified image  $f_{\theta}(x_0)$  as our optimization target. Here  $x_0$  can be obtained using SDEdit with  $K$  reverse steps. The update step becomes
$$
x _ {0} ^ {t} = \operatorname {S D E d i t} \left(x ^ {t}, K\right) \quad \text {a n d} \quad x ^ {t + 1} = \mathcal {P} _ {B _ {\infty} (x, \epsilon)} \left[ x ^ {t} + \eta \operatorname {s i g n} \nabla_ {x ^ {t}} l \left(f _ {\theta} \left(x _ {0} ^ {t}\right), y\right) \right]. \tag {5}
$$
The parameter  $K$  can be used to control the edition-content preference [32]: a large  $K$  value brings more editions and a small  $K$  value preserves more original content, and empirically,
Algorithm 1 Diff-rPGD
Require: Target classifier  $f_{\theta}$ , original image  $x$ , mask  $M$ , denoiser  $D_{\phi}$ , # of reverse SDEdit steps  $K$ , iterations  $n$ , stepsize  $\eta$ , clip value  $\epsilon$  (when  $M = 1$  it reduces to Diff-PGD)
$$
x ^ {0} = x _ {0} ^ {0} = x, x _ {c} = x
$$
for  $t = 0,1,2,\ldots ,n - 1$  do
$$
x _ {k} ^ {T} \sim q \left(x _ {k} ^ {T} \mid x ^ {t}\right)
$$
$\triangleright$  Sample  $x_{K}^{t}$  from  $q(x_{K}^{t}|x^{t})$  in each PGD iteration
for  $i = K - 1,\dots ,0$  do
$$
x _ {i} ^ {t} = R _ {\phi} (x _ {i + 1} ^ {t})
$$
$\triangleright$  Apply denoiser  $R_{\phi}$  to  $x_{i + 1}^{t}$  in each SDEdit iteration
$x_{i}^{t}\sim M\circ x_{i}^{t} + (1 - M)\circ q(x_{i}^{t}|x^{t})\triangleright$  Sample from masked combination of  $x_{i}^{t}$  and  $q(x_{i}^{t}|x^{t})$
$$
g = \nabla_ {x ^ {t}} l \left[ f _ {\theta} \left(M \circ x _ {0} ^ {t} + (1 - M) \circ x _ {c}\right) \right]
$$
$$
x ^ {t + 1} = \Pi_ {x, \epsilon} [ x ^ {t} + \eta M \circ \operatorname {s i g n} (g) ]
$$
$\triangleright$  Compute the gradient
PGD update
$$
x _ {0} ^ {n} = \operatorname {r S D E d i t} \left(x ^ {n}\right)
$$
return  $x^n, x_0^n$
Apply reverse SDEdit to the final  $x^n$
$\triangleright$  Return the final adversarial example and the denoised version
$K = 0.1 \sim 0.3T$  works well. Since we need to compute a new adversarial gradient  $g_{\mathrm{diff}} = \nabla_x R_\phi (\ldots R_\phi (R_\phi (x_t,K),K - 1)\ldots ,0)$  through back-propagation,  $K$  cannot be too large due to GPU memory constraint. Thus we turn to adopt some speed-up strategies like DDIM [40] to first sub-sample original  $T$  into  $T_{s}$ , and then use scaled  $K_{s} \ll K$  to do SDEdit (e.g.,  $T = 1000, K = 100, T_{s} = 40, K_{s} = 4$ ). Other fast sampling schemes [54] for DMs may accelerate our algorithms further.
After  $n$  iterations, Diff-PGD generates two outputs  $(x_0^n$  and  $x^n$ ) that are slightly different from each other: 1)  $x_0^n$ , according to the definition of the loss function, is realistic and adversarial. 2)  $x^n$  is an optimized perturbed image to which the adversarial gradient is directly applied. It follows the PGD  $\ell_{\infty}$  bound, and according to the loss function Equation (5), hard to be purified by SDEdit.
4.2 Extended Diff-PGD into Region Attack
In some scenarios, we need to maintain parts of the image unchanged and only generate adversarial perturbations in defined regions. This is used in customized attacks, e.g., attack the masked region with a given natural style. Here we consider the sub-problem to attack the masked region (determined by mask  $M$ ) of a given image with PGD, noted as region-PGD (rPGD). Specifically, the output adv-samples should satisfy:  $(1 - M) \circ x_{adv} = (1 - M) \circ x$ . Obviously, when  $M$  is valued all ones, the rPGD reduces to the original PGD.
For the previous gradient-based method, it is easy to modify  $\nabla_{x}f_{\theta}(x)$  into  $\nabla_{x}f_{\theta}(M\circ x + (1 - M)\circ x_{c})$  where  $x_{c} = x$  is a copy of the original image with no gradient flowing by. It is not surprising the perturbed regions will show a large distribution shift from other regions, making the figure unnatural.
For Diff-PGD, instead of using the original SDEdit, we modify the reverse process using a replacement strategy [30, 41] to get a more generalized method: Diff-rPGD. Similarly, when  $M$  has valued all ones, it reduces to Diff-PGD. The pseudo-code for Diff-rPGD is shown in Alg. 1, where the replacement strategy can give a better estimation of intermediate samples during the reverse diffusion. This strategy can make the patches in the given region better align with the remaining part of the image in the reversed diffusion process.
4.3 Customized Adversarial Sample Generation
As previously mentioned, Diff-PGD allows for the optimization of the adversarial loss while simultaneously preserving reality. This property helps us avoid optimizing multiple losses related to realism at the same time, as is used in previous methods [12, 17, 26]. Naturally, this property benefits the generation of highly customized adv-samples with prompts (e.g., mask, style reference, text), since we can separate the generation into two steps: 1) generate samples close to given prompts (from  $x$  to  $\hat{x}_s$ ) and 2) perturb the samples to be adversarial and at the same time preserving the authenticity (from  $\hat{x}_s$  to  $x_0^n$ ). Here we follow the task settings in natural-style adv-sample generation with style reference in [12]: given the original image  $x$ , mask  $M$ , and style reference  $x_s$ , we seek to generate adv-samples  $x_{adv}$  which shows the similar style with  $x_s$  and is adversarial to the target classifier  $f_\theta$ .
The style distance, as defined in [15], is composed of the differences between the Gram matrix of multiple features along the deep neural network, where  $H_{s}$  are layers for feature extraction (e.g. Conv-layers), and  $G$  is used to calculate the Gram matrix of the intermediate features. The style loss between  $x$  and  $x_{s}$  as  $l_{s}(x,x_{s})$  is
$$
l _ {s} \left(x, x _ {s}\right) = \sum_ {h \in H _ {s}} \| G \left(f _ {h} (x)\right) - G \left(f _ {h} \left(x _ {s}\right)\right) \| _ {2} ^ {2} \tag {6}
$$
AdvCAM [12] uses  $l = \lambda_1l_s + \lambda_2l_{adv} + \lambda_3l_r$  as the optimization loss where  $l_r$  includes losses like smooth loss and content loss, all designed to make the outputs more realistic. It is tricky to balance all these losses, especially the adversarial loss and the style loss. We emphasize that it is actually not necessary to train the adversarial loss  $l_{adv}$  together with  $l_s$ . This is due to the fact that Diff-PGD can guarantee the natural style of the output adversarial samples. Thus, the optimization process can be streamlined by first optimizing the style loss, and subsequently running Diff-PGD/Diff-rPGD. The high-level pipeline can be formulated as follows:
$$
x + p \rightarrow \hat {x} _ {s} \quad \text {a n d} \quad \left(x ^ {n}, x _ {0} ^ {n}\right) = \operatorname {D i f f - r P G D} \left(\hat {x} _ {s}, K, f _ {\theta}\right) \tag {7}
$$
The first equation serves as a first stage of optimization to acquire region styled  $\hat{x}_s$  given prompts  $p = (M, x_s)$ . This stage can be implemented with other methods (e.g. generative-model-based image editing) and we do not need to worry about  $f_{\theta}$  at this stage. Then we use Diff-rPGD to generate the adversarial sample  $x_{adv}$  given  $\hat{x}_s$  and other necessary inputs. Through experiments, we find that there are two major advantages of this pipeline: 1) the generation is much more stable 2) for some cases where the style loss is weak to guide the style transfer (generate samples that do not make sense), we can adjust  $K$  so that the diffusion model can help to generate more reasonable samples.
4.4 Physical World Attack with Diff-PGD
We can also apply Diff-PGD to physical world attacks. First, for physical-world attacks, we need to introduce a Physical Adapter in order to make the attack more robust in physical scenarios. Second, instead of using  $\ell_{\infty}$ -PGD as a bounded attack (e.g.  $\epsilon = 128/255$ , which is proved to be ineffective in [12]), we turn to utilize the same idea of Diff-PGD, but apply it as Diff-Phys, a patch generator with guidance from a diffusion model. We define our new loss as:
$$
l _ {\text {D i f f - P h y s}} (x) = l _ {\text {a d v}} \left(\mathbb {E} _ {\tau \sim \mathcal {T}} [ \tau (\text {S D E d i t} (x, K)) ]\right) \tag {8}
$$
Here, the physics adapter is included by adopting a transformation set  $\mathcal{T}$ , which is utilized to account for variations in image backgrounds, random translations, changes in lighting conditions, and random scaling. We can apply Diff-PGD to patch attacks