Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllability
Authors: Haotian Xue
Alexandre Araujo
Bin Hu
Yongxin Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Robustness
Adversarial Samples
Diffusion Model
TL;DR: We use diffusion model to generate more stealthy adversarial samples.

Abstract: Neural networks are known to be susceptible to adversarial samples: small variations of natural examples crafted to deliberately
mislead the models. While they can be easily generated using gradient-based techniques in digital and physical scenarios, they often differ greatly from the actual data distribution of natural images, resulting in a trade-off between strength and stealthiness. In this paper, we propose a novel framework dubbed Diffusion-Based Projected Gradient Descent (Diff-PGD) for generating realistic adversarial samples. By exploiting a gradient guided by a diffusion model, Diff-PGD ensures that adversarial samples remain close to the original data distribution while maintaining their effectiveness. Moreover, our framework can be easily customized for specific tasks such as digital attacks, physical-world attacks, and style-based attacks. Compared with existing methods for generating natural-style adversarial samples, our framework enables the separation of optimizing adversarial loss from other surrogate losses (e.g. content/smoothness/style loss), making it more stable and controllable. Finally, we demonstrate that the samples generated using Diff-PGD have better transferability and anti-purification power than traditional gradient-based methods.

Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllability
Haotian Xue
Alexandre Araujo 2
Bin Hu 3
Yongxin Chen
Georgia Institute of Technology
2 New York University
University of Illinois Orbana-Champaigg
Abstract
Neural networks are known to be susceptible to adversarial samples: small varia- tions of natural examples crafted to deliberately mislead the models. While they can be easily generated using gradient- gradient-based techniques in digital and physical scenarios, they often differ greatly from the actual data distribution of natural images, resulting in trade-off between strength and stealthiness. In this paper, we propose a novel framework dubbed Diffusion-Bar (Diff-PGD) for generating realistic adversarial samples. By exploiting gradient guided by a diffusion model, Diff-PGD ensures that adversarial samples remain close to the original data distribution while maintaining their efffcctiveness. More- over, our framework can be casily customized for specific tasks such as digital attacks, physical- attacks, and style-based attacks. Compared with existing methods for generating natural-style adversarial samples, our framework enables the separation of optimizing adversarial loss from other surrogate losses (e.g., content/smoothneesssyylle loss), making it more stable and controllable. Finally, we demonstrate that the samples generated using Diff-PGD have better transferabil- ity and antiipurification power than traditional gradient-based methods. Code is available https://gihub..co/avviaaarr -GG
Projected Gradient Descent
Introduction
Neural networks have demonstrated remarkable capabilities in learning features and achieving outstanding performance in various downstream tasks. However, these networks are known to be vulnerable subtle perturbations, called adversarial samples (adv-samples), leading important security concerns for critical-decision systems [16]. Numerous threat models have been designed to generate adversarial samples for both digital space attacks [31, 5]. physical world attacks [26, 14], and also for generating more customized adv-samples given prompts like attack region and style reference [12, 17]. However, most of these models have not been designed maintain the realism of output samples, resulting in adversarial samples that deviate significantly from the distribution of natural images [22]. Indeed, in the setting of digital world attacks, such as those involving RGB images, higher success rates, and transferability are associated with larger changes in the generated samples, leading
trade-off. Additionally, perturbations for physical-world attacks are substantial, non-realistic, and easily noticeable by humans [47].
Recent works, for both digital and physical settings, have proposed methods generate more realistic adversarial samples. For example, [29, 24] introduced method to optimize the perturbations added clean images semantic space, [23, 26] proposed a GAN-basee approach or other prior knowledge [33] to increase the realism of generated samples. Although the GAN-based approach
Correspondence to: htxue htxue. ai@gatech.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Properties of DifI-PGD vs Other Attacks: We summarize six metries for adv-sample generation. "Stealthiness" measures whether the adversarial perturbations can be detected by human observers. "Scenarios" describes the setting in which the method can be applied: D corresponds to digital attacks and P for physical attacks. "Controllability" measures whether the method can support customized prompts e.g., mask/style reference. "Anti-Pute Anti-Purify" measures the ability of the samples to avoid being purified. "Transferability" measures the generalization of the attack to different models. Finally, "Stabbiityy" describes the stability of the optimization. (-) stands for non-consideration.
Methods
Stealthiness
Scenarios
Controllabiliit
Anti-Purify Transferability
Stability
PGD [31]
*8
D
AdyPatch [4]
88
(-)
(-)
NatPatch [23]
*0
6
(-)
(-)
60
AdvArt [17]
3*
(-)
(-)
+8
AdvCam [12]
** D/P
$*
(-)
(-)
Diff-PGD (Ours)
D/P
**
can generate realistic images, the adversarial examples are sampled from noise and therefore lack controllability. A subsequent line of research realistic adversarial samples [22] has introduced the concept of semantic adversarial samples. These are unbounded perturbations that deceive the model, while ensuring the modified image has the same semantic information the original image. While transformations in semantic space can reduce the high-frequency patterns in pixel space, this approach still suffers from color jittering image distortion which results in lower stealthiness [12]. Further, this approach needs careful tuning of the hyperparameters making the training process unstable.
In this paper, we propose a novel framework, which uses an off-the-shelf diffusion model to guide the optimization of perturbations, thus enabling the generation of adversarial samples with higher stealthiness. The core design of our framework is Diffusion-Basee PGD), which shares a similar structure with PGD but changes the input of the target classifier fo to be purified version 2g of the original input œ. To the best of our knowledge, we are the first to use the Diffusion Model to power the generation of adv-samples. Our framework to some customized attacks given masks or style references, thereby enhancing its controllability. Our framework (Figure I(c)) separates the customized process with prompt p (region mask, style, etc.) with adversarial sample generation, overcoming the drawbacks of previous pipelines: the traditional gradient-based methods (Figure (a)) cannot guarantee to generate naturalistic samples when the perturbation level is large; the joint optimization with multiple losses like adv-loss ladv- style loss l and realism loss 4, (see Figure ((b) is unstable train and still tend generate perceptible artifacts. Through extensive experiments in scenarios like digital attacks, masked region attacks, style-guided attacks, and physical world attacks, we show that our proposed framework can effectively generate realistic adv-samples with higher stealthiness and controllability. We further show that the Diff-PGD can help generate adversarial samples with higher transferability than traditional methods without gradient restriction. Our contribution can be summarized as follows:
Projected Gradient Descent (Diff-
also be easily adapted
1. We propose novel framework Diff-PGD, combing the strong prior knowledge of the diffusion model into adv-sample generation, which helps generate adv-samples with high stealthiness and controllability. Ady-samples generated by Diff-PGD have good properties described in Table 1.
2. We show that Diff-PGD can be effectively applied many tasks including digital attacks, cus- tomized attacks, and physical-wordh attacks, outperforming baseline methods such as PGD, AdvPatch, and AdyCam.
3. We explore the transferability and anti untipuurification properties of Diff-PGD samples and show through an experimental evaluation that adversarial samples generated by Diff-PGD outperform the original PGD approach.
2 Related Work
Adversarial attacks aim at maximizing the classification error of target model without changing the semantics of the images. In this section, review the types of attacks and existing works.

---

Norm bounded attacks. In the digital space, one can easily generate adversarial samples by using gradient- methods such as Fast Gradient Sign Method (FGSM) [16] or Projected Gradient Decent (PGD) [31 These methods aim at maximizing the loss of the target model with respect to the input and then projecting the perturbation to specific lp ball. It has been observed that these adversarial samples tend to diverge from the distribution of natural images [50]. Based on this observation, [34] have shown that these attacks can be "purified" using pre-trained diffusion models.
Semantic attacks. Some recent works operate semantic transformation to an image: [22] generate adv-samples in the HSV color space, [48. 13] generate adv-samples by rotating the 2D image or changing its brightness, and [52, 29, 10] try to generate semantic space transformations with an additional differentiable renderer. All these methods are far from effective; they either need additional modules or have color jitters and distortion [12].
Customized Attacks with Natural-Style. Significant efforts have been made toward generating cus- tomized adversarial samples (given region and/or style reference) with natural-style: AdvCAM [12] optimize adversarial loss together with other surrogate losses such as content loss, style loss, and smoothness loss to make the output adversarial sample to be realistic, [23, 26, 55] sample from latent space and then use a Generative Adversarial Network (GAN) guarantee the natural style of the output sample to fool an object detector, and Adv [17] customized the generation of the adversarial patch with given art style. However, all these methods share some common issues: optimizing adversarial loss and other surrogate losses (for content- preeenriresrriigg or customized style) needs careful balance and still results in unnatural artifacts in the output samples. Our methods use the diffusion model as strong prior to better ensure the realism of generated samples.
Attacks in ahysical-worke For physical- worlld attacks, minor disturbances in digital space often prove ineffective due to physical transformations such as rotation, lighting, and viewing distance. In this context, adding adversarial patches to the scene is common technique to get the target model to misclassify [4, 14, 1, 25]. Physical-worid attacks do not impose any constraints either on the amount of perturbation on the output style, thus, they tend to produce highly unnatural images that can be easily detected by human observers (i.e., lack stealthiness).
3 Background
Diffusion Model. Diffusion Models (DMs) [19, 41] have demonstrated superior performance in many tasks such as text-to-image generation [37, 2, 38], video/story generation [21, 20, 35], 3D generation [28, 36] and neural science [44, 42]. Pre-trained DMs provide valuable prior knowledge that can be exploited for adversarial robustness. In this context, several works have proposed defense mechanisms such as adversarial purification [34, 53, 43, 34, 27] or DM-enhanced certified robustness [6, 46, 45].
The Denoised Diffusion Probabilistic Model (DDPM) [19] is a discretized version of DMs and works as follows. Suppose 2g ~ p(xa) is a sample from the distribution of natural images. The forward diffu- sion process gradually adds Gaussian noise to zo. generating noisy samples [x], T2, Lt. ; 2T in T steps, following a Markov process defined as qM (M; x;-1) = N(x; VI- -I-BBaan 2;- 3,1) where N denotes Gaussian distribution Be are fixed values growing from 0 to 1. By accumulating single step QM we have y(
ai)) as
q( 2o) N(11 Val x,-1 (1-a,)))
(I)
where a, = 1 - 3r and ā, = a. When a, se 0, the distribution of or becomes an isotropic Gaussian.
The reverse process aims at generating samples from the target data distribution from Gaussian noise JT ~ (0, I) using the reversed diffusion process. The reverse model (z xx) can be trained by optimizing the usual variational bound on negative log likelihood. Using the re-sampling strategy proposed in [19], we can simplify the optimization of Φ into a denoising process, by training a modified U-Net as a denoiser. Then we can generate To with high fidelity by sampling on the reversed diffusion:
p(Zo.T) p(Fr) II p xx).
For the reverse process, we define R. parameterized by @ as the function to denoise x, and then get next sampled value Iz-1 = t)).
3

---

Vx
PxOp + lam + +.)
Vx ladv
p
SDEdit
xo
Facto
x
x
Ladv
1p
ladv
x
Ra
Ri.
Rgg
(a)
(b)
(c)
Figure I: Comparison of Difffernt Pipelines: (a) Traditional gradient-based adversarial sample generation, x is the sample to be optimized, ladv adversarial loss. (b) Customized adversarial sample generation with natural style (determined by prompt p): joint optimization of adversarial loss with other surrogate losses like prompt loss lp (e.g. style loss) and realistic loss 1, (e.g. content loss, smooth loss). (c) Our proposed diffusion- framework, 9 is forward diffusion and Rs is backward denoising, zig is the denoised sample.
SDEdit. The key idea of Stochastic Differential Editing (SDEdit) [32] is to diffuse the original distribution with Iк ~ q( x)) and then run the reverse denoising process Rg parametrized by o in Diffusion Model get an edited sample:
SDEdit(z, K)
Given a raw input, SDEdit first runs K steps forward and then runs K steps backward. This process works as a bridge between the input distribution (which always deviates from p(ro)) and the realistic data distribution p(zo), which can then be used tasks such as stroke-based image synthesis and editing. It can also be used in adversarial samples purification [34, 53].
4 Diffusion-based Projected Gradient Descent
We present Diffusion- -ased Projected Gradient Descent (Diff-PGD), a novel method to generate adversarial samples that looks realistic, using pre-trained diffusion model. First, we present basic Diff-PGD that works on global image inputs, which shares the same bound settings with es. -PG in Section 4.1. Then, in Section 4.2 generalize Diff-PGD to scenarios where only regional attacks defined by a customized mask are possible, using re- re-pannting skills in the diffusion model [30]. Finally, present extensions of Dif customized attacks guided by image style in Section 4.3 and physical world attack in Section 4.4.
In the rest of the paper, we use superscript i or to denote iteration of optimization, and subscript i to denote the timestep of diffusion model.
4.1 Diffusion-based Projected Gradient Descent
Given a clean image x with label y and a target classifier fo parameterized by 0 to be attacked, our goal is to generate adversarial sample aadv- which fools the model. The traditional gradient-based methods use the gradient of adversarial loss (x((fo(z), v). where fe(æ) return the logits. We also denote the loss as ladv for simplicity optimize by iterations. The t-step update in the PGD with stepsize 7) and n iterations reads
1+11 (r..( [x + vi
(4)
where P (z.)))))) is the projection operator on the foo ball. Intuitively, the gradient will "guide" the sample x² away from the decision boundary and deviate from the real distribution p(2o) [29]. This out-of-distribution property makes them easier to be purified using some distribution-ban purification methods [34], which considerably restricts the strength of the attacks.
Instead of using (=)) y) as the optimization cost, we turn to use logits of purified image fo(xp) as our optimization target. Here xn can be obtained using SDEdit with reverse steps. The update step becomes
(5) The parameter K can used to control the edition-conten preference [32]: large K value brings more editions and small K value preserves more original content, and empirically,
= SDEdit(>", K) and 1+1 (B..(z.)) [xr Ve

---

Algorithm Diff-rPGD
Require:
Target classifier fa. original image æ. mask M, denoiser D # of reverse SDEdit steps K, iterations n,, stepsize 7/, clip value (when M 1 it reduces to Diff-PGD)
x² = = = v. Fe I
for 0, 0,1,2, .....
do
s q(< (x')
> Sample æ'K from ((")((²") in each PGD iteration
for K 1,..,,0000 do
do
z = R2(x141
Apply denoiser RS to o in each SDEdit iteration Sample from masked combination of x and g(x) (z")
z ~ M 2 + (1 - M) o g(>)|=")
end for
9 V Vo M 26 + M)xx.)
D Compute the gradient
end for x: x1+1 TT, x" + 1j MMMo sign(g)
b PGD update
26 =ISDEdit(z")
Apply reverse SDEdit to the final x² Return the final adversarial example and the denoised version
return x", x
K = 0.1 ~ 0.37 works well. Since we need to compute a new adversarial gradient Jdiff = V Ro K), K - 1) .... 0) through back-propagation, K cannot be too large due to GPU memory constraint. Thus we turn to adopt some speed-up strategies like DDIM [40] to first samlle original T into Te. and then use scaled K₃ < K to do SDEdit (e.g.. T = 1000, K = 100, = 40, K = 4). Other fast sampling schemes [54] for DMs may ac- celerate our algorithms further.
After n iterations, Diff-PGD generates two outputs (x and æ") that are slightly different from each other: 1) xn according the definition of the loss function, is realistic and adversarial. 2) x² is an optimized perturbed image to which the adversarial gradient is directly applied. It follows the PGD loo bound, and according to the loss function Equation (5), hard to be purified by SDEdit.
4.2 Extended Diff-PGD into Region Attack
In some scenarios, we need to maintain parts of the image unchanged and only generate adversarial perturbations in defined regions. This is used in customized attacks, e.g., attack the masked region with given natural style. Here we consider the sub-problem to attack the masked region (determined by mask M) of given image with PGD, noted region-PGD (rPGD). Specifically, the output adv-samples should satisfy: (1 M) Fadv = (1 M) x. Obviously, when M is valued all ones, the rPGD reduces the original PGD.
For the previous gradient-based method, is easy to modify (=))(x) into V ox where Fe = xr is copy of the original image with no gradient flowing by. It is not surprising the perturbed regions will show a large distribution shift from other regions, making the figure unnatural.
(1- (1-M(111))))))) M)xx.))
For Diff-PGD, instead of using the original SDEdit, we modify the reverse process using replacement strategy [30, 41] to get a more generalized method: Diff-rPGID. Similarly, when M has valued all ones, it reduces to Diff-PGD. The pseudo-code for Diff-rPGD is shown in Alg. 1, where the replacement strategy can give a better estimation of intermediate samples during the reverse diffusion. This strategy can make the patches in the given region better align with the remaining part of the image in the reversed diffusion process.
4.3 Customized Adversarial Sample Generation
As previously mentioned, Diff-PGD allows for the optimization of the adversarial loss while simulta- neously preserving reality. This property helps us avoid optimizing multiple losses related to realism at the same time, as is used in previous methods [12, 17, 26]. Naturally, this property benefits the generation of highly customized adv-samples with prompts (e.g., mask, style reference, text), since we can separate the generation into two steps: 1) generate samples close to given prompts (from to ..)) and 2) perturb the samples to be adversarial and at the same time preserving the authenticity (from i. to x:²). Here we follow the task settings in natural-style adv-sample generation with style reference in [12]: given the original image z, mask M, and style reference F., we seek to generate adv-samples Zady which shows the similar style with x. and is adversarial to the target classifier fa.
5

---

The style distance, as defined in [15]. is composed of the differences between the Gram matrix of multiple features along the deep neural network, where H, are layers for feature extraction (e.g. Conv-layers), and G' is used to calculate the Gram matrix of the intermediate features. The style loss between x and v, as 1 (2., x.s) is
(x, = (ICLU(()
(6)
НEM,
AdvCAM [12] uses == X14 + Aglado + >311 as the optimization loss where I, includes losses like smooth loss and content loss, all designed to make the outputs more realistic. It is tricky to balance all these losses, especially the adversarial loss and the style loss. We emphasize that it actually not necessary to train the adversarial loss lado together with 1.. This is due to the fact that Diff-PGD can guarantee the natural style of the output adversarial samples. Thus, the optimization process can be streamlined by first optimizing the style loss, and subsequently running Diff-PGD/Diff-PGDD The high-level pipeline can be formulated as follows:
p ig and (I", Diff-rPGD(d,,.... fe
The first equation serves as a first stage of optimization to acquire region styled è, given prompts p (M, 2.)). This stage can be implemented with other methods (e.g. eemerative-model-beee based image editing) and we do not need to worry about fo at this stage. Then we use Diff-rPGD to generate the adversarial sample Fade given 2, and other necessary inputs. Through experiments,
find that there are two major advantages of this pipeline: 1) the generation is much more stable 2) for some cases where the style loss is weak to guide the style transfer (generate samples that do not make sense), can adjust K so that the diffusion model can help to generate more reasonable samples.
4.4 Physical World Attack with Diff-PGD
We can also apply Diff-PGD to physical world attacks. First, for physical-worid attacks, we need to introduce a Physical Adapter in order to make the attack more robust in physical scenarios. Second, instead of using -PGD as bounded attack (e.g. = 128 /255, which is proved to be ineffective in [12]), we turn to utilize the same idea of Diff-PGD, but apply it as Diff-Phys, a patch generator with guidance from diffusion model. We define our new loss as:
IDift-Phs (2)) = ludo (E,-~ [r (SDEdit(z,, K))))
(8)
Here, the physics adapter is included by adopting transformation set T, which is utilized account for variations in image backgrounds, random translations, changes in lighting conditions, and random scaling. We can apply Diff-PGD patch attacks
x² = arg min Dan Phys and 3 SDEdit(a", K).
The pipeline works as follows. We first gather background data and define the scope of translations and lighting conditions, to get T. Then we optimize our patch in simulation to get zz. Finally, we print zf out, stick it to the target object, take photos of the object, and test it on the target classifier.
Experiments
The experiment section aims to answer the following questions: (Q1) Dif--GGDDDiff-PGD effectte to generate adv-samples with higher realism? (Q2) Can Diff-PGD be easily applied to generate better style-customized adv-samples? (Q3) Can Diff-PGD be applied to physical world attacks? (Q4) Do adversarial samples generated by Diff-PGD show better properties like transferability and anti-purification ability?
Datasets, Models, and Baselines. We use the validation dataset of ImageNet [8] our dataset to get some statistical results for global attacks and regional attacks. For the style-guided adv-sample generation, we use some cases from [12] but also collect more images and masks by ourselves. We use VGG-19 as the backbone network to compute the style distance. The main target classifier to attacked across this paper ResNer-50 [18], and we also use ResNet-101, ResNet18, and networks, we also try our Diff-PGD on Vision Transformers like ViT-b [11] and BEiT-I [3], and the results are put in Appendix E. The diffusion model we adopt is the unconditional diffusion model pre- pre-trained on
RedeResNet(WRRN) [51] for the transferability test. Besides convolutional-bast

---

X
XPGD
8PGD
X
%g
x
XPGD
8PGD
X
6
Figure 2: Visualization of Adversarial Samples generated by Diff-PGD: adv-samples generated using PGD (IPGD) tend to be unnatural, while Diff-PGD (zj) can preserve the authenticity of adv- samples. Here x is the original image, åPGD x 2PGD and 6ß x zúj and scale up the value by five times for better observation. Zoom in on a computer screen for better visualization.
(+) XrPGD
(+) x
(+) XyPGD
(+) x
Figure 3: Visualization of Adversarial Samples generated by Diff-rPGD: Diff-PPGD can generate better regional attacks than PGD: the attacked region can better blend into the background. The attacked regions are defined using red bounding boxes, and (+) means zoom-in.
ImageNet [9] though we use DDIM [40] to respace the original timesteps for faster inference. More details about the experimental settings included in the appendix.
We compare Diff-PGD with threat models such PGD [31] (for digital attacks), AdvPatch [4] (for physical-worke attacks), and AdvCam [12] (for customized attacks and physical-workd attacks). All threat models generate attacks within the context of image classification tasks. More details are included in the supplementary materials.
Difr-PGD For Digital Attacks (Q1). We begin with the basic global lx digital attacks, where set &oo = 16/255 255 for PGD and Diff-PGD, and # of iterations n 10 and step size 1/ 2/255. For Diff-PGD, we use DDIM with timestep T, 50 (noted DDIM50 for simplicity). and K = for the SDEdit module. Figure shows that adv-samples generated by Diff-PGD (xā)) are more stealthy, while PGD-generated samples (IPGD) have some patterns that can be easily detected by humans. This contrast is clearer on the perturbation map: åPGD contains more high- high-frequency noise that is weakly related local patterns of the instance æ, and o0 smoother and highly locally dependent. Similarly, for regional adversarial attacks, Diff-rPGD can generate adv-samples that can better blend into the unchanged region, as demonstrated Figure 3, showing higher stealthiness than rPGD.
We also conduct experiments to show the effectiveness of Diff-PGD regarding the Success Attack Rate. We uniformly sampled 250 images from the ImageNet validation set. From Figure 6 (a). see that although the gradient restricted to generate realistic perturbations, Diff-PGD can still reach high success rate with more than iterations. Detailed settings can be found in the supplementary materials.
We also conduct experiments different norms. We show additional results of the performance of Diff-PGD on based attacks in Appendix F.
DiD For Style-Customizer Attacks (Q2). For customized attacks, we focus on the task of generating adversarial samples using a mask M and a style reference image xs, provided by the user. We compare approach, described in Section 4.3, with the AdvCam method. As previously mentioned, the main advantage our pipeline that we do not need balance multiple losses, and instead divide the task into two stages: generate style-transferre image 2, without considering

---

50
50
50
50
Xg
AdvCam
X;
AdvCam
Ours
Figure 4: Generating Adversarial Samples with Customized Style: Given the original image æ, style mask M, and style reference image 2s, Diff-PGD can generate more realistic samples, even in cases where only local styles are given (e.g. only the door of the red car is offered as x.).
"Computer Mouse"
"Cuumbee""
"Cmputer Mouse
"Neckes"
"Sacppack"
"lorichire torrier
Noowiich terrier
Toristire terner
We show two scenarios physical world attacks: the first includes untargeted attacks small object: computer mouse, and the second includes targeted attacks on a larger object: backpack, where we set the target to be Yorkshire terrier. The sticks-photo pairs include clean patch (green box), AdvPatch(blue box), AdvCam generated patch (red box), and our Diff- Diff-Phys generated patch (black box).
Figure 5: Results of Physical- World Attacks:
adversarial properties, and then generate realistic perturbation using Diff-PGD to make the image adversarial to the classifier.
DDIM10 with K 2. Figure 4 demonstrates that method consistently generates customized adversarial samples with higher stcalthiness, while the adversarial samples generated by AdvCam exhibit digital artifacts that make them appear unnatural. Also, in some cases, the style reference cannot be easily transferred (e.g. only local styles are given in x,, or x and x. are collected with different resolution), resulting in unsatisfactory å,, such red/yellow Figure 4. Diff-PCC can refine the samples with more details, thanks to the strong generation capacity of diffusion models.
For Diff-PGD,
style in
Diff-PGD For Physical-Wont World Attacks (Q3). the chysical-world attacks, we set the problem as: given an image patch (start point of optimization), and a target object (e.g. attack a backpack), we need to optimize the patch to fool the classifier when attached to the target object. We use an iPhone 8-Plus to take images from the real world and use an HP DeskJet-2752 for color printing. In AdvPatch, only adversarial loss is used to optimize the patch; in AdvCAM, content loss and smooth loss are optimized together with adversarial loss; for Diff-Phys, we only use purified adversarial loss defined in Equation (8), and the SDEdit is set to DDIM10 with = 2 for better purification.
The results are shown in Figure 5, for the untargeted attack, where we use green apple as patch to attack computer mouse. For targeted attack, we use an image of a cat to attack a backpack, with the larget-label as "Yorkshire terrier". We can see that both AdyPatch and AdvCam generate patches with large noise, while Diff-Phys can generate smoother and more realistic patches that can successfully attack the given objects.

---

1.8
4.75
Exploring Other Properties of 2g and x" (Q4). Finally, we investigate additional properties of adversarial samples generated using Diff-PGD. Among the two samples zij and z". the former exhibits both adversarial characteristics and a sense of realism, while the latter is noisier but contains Succass abrock (ate
100% Success Rate
A18
Jun
Iterations
R250
R101
R18
WR50 WR101
AT-R50
AT*-R50
AT*-R18
(a)
(b)
(c)
Figure 6: This figure presents quantitative results
approach, with all y-axis representing the success rate of attacks: (a) Successful rate of Diff-PGD vs PGD; (b) Results of transferability
of Diff-PGD PGD, where
16/ f6/255 and = 2/255. The adv-samples are generated with ResNet-50 (R50) and tested on ResNet-18 18 (R 18), ResNet-101 (R101), WRN-50 (WR50), and WRN- 101 (WR101); (c) Success rate of Diff-PGD PGD adversarially trained networks. AT uses adversarial training strategy in [13] and AT uses AT strategy [39] on the ImageNet dataset.
Table Anti-purification Results: Adv-samples generated by PGD and Diff-PGD against adver- sarial purification: (+P) means the classifier is enhanced by an off-the-shelf adversarial purification module. Here we 16/255, 10. The attacks are operated on ResNet-50.
Sample (+P)ResNett01 (+P)ResNet500
(+P)ResNet18
(+P)WRN50
(+P)WRN101
3PGD
0.35
0.18
0.26
0.20
0.17
Zn (Ours) 0.88
0.38
0.36
0.32
0.28
20 (Ours) 0.72
0.36
0.37
0.36
0.36
adversarial patterns that more difficult for SDEdit to eliminate. Although our approach does not specifically target enhancing Transferability and Purification power, we demonstrate that Diff-PGI) surpasses the original PGD in these two aspects.
Transferability. We test the adv-samples targeting ResNet-50 on four other classifiers: ResNet- 101, ResNet-18, WRN-50 and WRN-101. From Figure 6(b) we can see that both x" and xi generated by Diff-PGD can better transferred to attack other classifiers than PGD. This can be explained by intuition that adv-attacks in semantic space can be better transferred. We also test the success rate attacking adversarially trained ResNet-50 in Figure 6(c) and we can see x" is much better than other adversarial samples.
Anti-Purification. Following the purification pipeline in [34], we attach an off-the-shelf sample purifier to the original ResNet-50 and test the success rate of different adv-samples. In Table 2 we can see that the adv-samples FPGD generated by PGD are easily purified using the enhanced ResNet-50. In contrast, our adv-samples, x" and xi show better results than PPD by large margin. It can be explained by the type of perturbations: out-of-distribution perturbations used in PGD can be removed using diffusion-based purification, while in-distribution attacks in Diff-PGD are more robust to purification.
6 Accelerated Diff-PGD with Gradient Approximation
From Algorithm 1, the input, where the model is composed of chained U-Nets Rs in the diffusion model followed by target classifier fe. Here we focus on the computational bottleneck, which is the derivative of K K-step SDEdit outputs vo over SDEdit input
that need calculate the derivative of the model's output
Oxo
0EK-1 1 0rk-2 0z mi 8ak
(10)
dx
0rk
0zK-1 0xo

---

We approximate the gradient with constant We notice that the approximation of this Jacobian has been used in many recent works [36, 49, 7]. We also tried this strategy in Diff-PGD. and we got the accelerated version:
al(I(a))) aL(s(E))) Oz
(11)
Ox
From this can see that, only need calculate the gradient over zj! We only run the inference of SDEdit to get zi without saving the gradient (lower GPU memory, faster), making it much cheaper. We found that, for the global attack tasks, the approximated gradient still shows a high success rate but saves 50% of time and 75% of VRAM. More detailed results about the visualization and sucess rate can be found in Appendix C.
7 Conclusions
In this paper, we propose a novel method to power the generation of adversarial samples with diffusion models. The proposed Diff-PGD method improve the stealthiness of adv-samples. We further show that Diff-PGDD can be casily plugged into global digital attacks, regional digital attacks, customized attacks, and physical-world attacks. We demonstrate through experiments that our methods outperformed the basclines and are effective and stable. The major limitation of our method that the back- propaaatoontnn more expensive than traditional gradient-based methods; however, we believe that the strong prior knowledge of the diffusion model can push forward adversarial attacks & defenses. Finally, while the proposed adversarial attack method could be potentially used by malicious users, it can also enhance future efforts to develop robust defense mechanisms, thereby safeguarding the security of Al systems.
10

---

Acknowledgment
The authors would like thank the anonymous reviewers for useful comments. HX and YC are supported by grants NSF 2008513 and NSF 2206576. AA is supported in part by the Army Research Office under grant number 191-N1-21-1 and by the New York University Abu Dhabi (NYUAD) Center for Artificial Intelligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010. BH is supported by the NSF award CAREER-2048 68 and the AFOSR award FA9550-23-1-0732.
References
[1] A. Athalye, Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. International conference on machine learning, pages 284-293. PMLR, 2018.
[2] Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint urXiv:2211.0133
2022.
[3] H. Bao, L. Dong. S. Piao, and F. Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
[4] T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer. Adversarial patch. arXiv preprint arXiv:1712.09665, 2017.
[5] N. Carlini and D. Wagner. Towards evaluating robustness of neural networks. In 2017 iece symposium on security and privacy (sp), pages 39 57. lece, 2017.
[6] N. Carlini, F. Tramer, K. D. Dvijotham, L. Rice. M. Sun, and J. Z. Kolter. (certified!!) adversarial robustness for free! In The Eleventh International Conference on Learning Representations, 2023.
[7] Z. Chen, B. Li, S. Wu, K. Jiang, S. Ding, and W. Zhang. Content-based unrestricted adversarial attack. arXiv preprint arXiv:2305. 10665, 2023.
[8] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pages 248-255. Iccc, 2009.
[9] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[10] Y. Dong, S. Ruan, H. Su, C. Kang, X. Wei, and J. Zhu. Viewfool: Evaluating the robustness of visual 2022.
recognition to adversarial viewpoints. arXiv preprint
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929 2020.
[12] R. Duan, Ma, Wang, Bailcy, K. Qin, and Yang. Adversarial camouflage: Hiding physical- world attacks with natural styles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1000-1000 1008, 2020.
[13] L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. A rotation and a translation suffice: Fooling cnns with simple transformations.
2017.
[14] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song. Robust physical-world attacks on deep learning visual classification. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1625-1634, 2018
[15] L. A. Gatys, S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
[16] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.6572, 2014.
[17] A. Guesmi, I. M. Bilasco, M. Shatique, and I. Alouani. Advart: Adversarial art for camouflaged object detection attacks, arXiv preprint a1X14:2303.01734, 2023.

---

[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 770-778, 2016.
[19] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68535 2020.
[20] J. Ho, W. Chan, C. Saharia, J. Whang. R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, al. Imagen video: High definition video generation with diffusion models, arXiv preprint arXiv:2210.02300 2022.
[21] J. Ho, T. Salimans, A. A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. In A. H. Oh, A. Agarwal, D. Belgrave, and Cho, editors, Advances Neural Information Processing Systems, 2022.
[22] H. Hosseini and Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1614-1619, 2018.
[23] Y--.--. Hu, B.-H. Kung, D. Tan, J.-C. Chen, K.-L. Hua, and W.H. Cheng. Naturalistic physical adversarial patch for object detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7848-7857, 2021.
[24] Z. Hu, S. Huang, X. Zhu, Sun, B. Zhang, and X. Hu. Adversarial texture for fooling person detectors in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13307-13316, 2022.
[25] A. Kurakin, I. J. Goodfellow, and S. Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99-112. Chapman and Hall/CRC, 2018.
[26] R. Lapid and M. Sipper. Patch of invisibility: Naturalistic black-box adversarial attacks on object detectors. arXiv preprint arXiv:23333.422: 2023.
[27] M. Lee and D. Kim. Robust evaluation of Liffusion-based adversarial purification. arXiv preprint arXiv:2303.000 090551 2023.
[28] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng. X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T. Y. Lin. Magic3d: High- text-to-3d content creation. arXiv preprint arXiv:2211 10440. 2022.
[29] H.-T. D. Liu, M. Tao, C.-L. Li, D. Nowrouzezahrai, and A. Jacobson. Beyond pixel norm-balls: Para- metric adversaries using an analytically diffcrentiable renderer. In International Conference on Learning Representations, 2019.
[30] A. Lugmayr, M. Danelljan. A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-1147,, 2022.
[31] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant adversarial attacks. In International Conference Learning Representations, 2018.
[32] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[33] Y. Miao, Y. Dong, J. Zhu, and X.-S. Gao. Isometric 3d adversarial examples in the physical world. In A. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems. 2022.
[34] Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar. Diffusion models for adversarial purification, In Proceedings of the 39th International Conference on Machine Learning, pages 16805- 16827, 2022.
[35] X. Pan, P. Qin, Y. Li, H. Xue, and W. Chen. Synthesizing coherent story with auto-regressive latent diffusion models. arXiv preprint arXiv:2211. 2.1.100000000 2022.
[36] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023.
[37] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 10684- 10045, 2022.
12

---

[38] C. Saharia, W. Chan, S. Saxena. L. Li, J. Whang, E. L. Denton, K. Ghasemipour: R. Gontijo Lopes.
B. Karagol Ayan. T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
[39] H. Salman, A. Dlyss, L. Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models transfer better? Advances Neural Information Processing Systems, 33:3533-3545, 2020.
[40] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 2021.
[41] Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint urXiv:2011.134 2020.
[42] Y. Takagi and Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453-14465 2023.
[43] Wang, Z. Lyu, Lin, B. Dai, and H. Fu. Guided diffusion model for adversarial purification. arXiv preprint Xiv: 14969, 2022.
[44] Y. Wang. Z. Wu, Li, and A. Wu. Extraction and recovery spatio-temporal structure latent dynamics alignment with diffuusio model. arXiv preprint arXiv:2306.0611 2023.
[45] Q Wu, H. Ye, and Gu. Guided diffusion model for adversarial purification from random noise. arXiv preprint 2rx: 10875, 2022.
[46] C. Xiao, Z. Chen, K. Jin, J. Wang, W. Nie, M. Liu, A. Anandkumar, Li, and Song. Densepure: Understanding diffusion models for adversarial robustness. In The Eleventh International Conference on Learning Reprexentations, 2023.
[47] K. Xu, G. Zhang, S. Liu, Q. Fan, M. Sun, H. Chen, P.-Y. Chen, Y. Wang, and X. Lin. Adversarial t-shint evading person detectors in a physical world. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Part V 16, pages 665-68 Springer, 2020.
[48] X. Xu, X. Chen, C. Liu, A. Rohrbach, T. Darell, and D. Song. Can you fool ai with adversarial examples visual turing test. arXiv preprint veXiv:1709.08699
2017.
[49] H. Xue, C. Liang, X. Wu, and Y. Chen. Toward effective protection against diffusion based mimicry through score distillation. arXiv preprint arXiv:2311 12832, 2023.
[50] J. Yoon, S. J. Hwang, and J. Lee. Adversarial purification with score-based generative models. In International Conference Machine Learning, pages
PMLR, 2021.
[51] S. Zagoruyko and N. Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[52] X. Zeng, C. Liu, Y.S. Wang, W. Qiu, L. Xie, Y.-W. Tai, C.-K. Tang, and A. L. Yuille. Adversarial attacks beyond the image space. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4302-4311, 2019.
[53] K. Zhang, H. Zhou, J. Zhang, Huang, W. Zhang, and N. Yu. Ada3diff: Defending against 3d adversarial point clouds via adaptive diffusion. arXiv preprint arXiv:2211.16247, 2022.
[54] Q Zhang and Y. Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.15 13902, 2022.
[55] 7. Zhao, D. Dua, and Singh. Generating natural adversarial examples. International Conference on Learning Representations, 2018
13

---

Contents
1
Introduction
2 Related Work
3
Background
4
Diffusion-based Projected Gradient Descent
4.1 Diffusion-based Projected Gradient Descent
4.2
Extended Diff-PGD into Region Attack
4.3
Customized Adversarial Sample Generation
4.4
Physical World Attack with Diff-PGD
6
5
Experiments
6
Accelerated Diff-PGD with Gradient Approximation
7
Conclusions
10
A Notations
15
Details about Methods
15
B.1 Details about Style Dift-PGD
15
B.2 Details about Physical World Diff-PGD
15
C Accelerated Diff-rPGD
16
Implementation Details
17
E Results on Vision Transformers
18
Results on based Attacks
18
G More Complementary Results
20
H Human Evaluation
23
Potential Social Impacts
23
14

---

Appendix
The appendix organized as follows: in Section A, we provide a summary table notations to assist readers in better understanding the content. Then we delve into the details of our algorithms in Section B, including how to apply Diff-PGD to style-based attacks and physical world attacks step by step. Then, we show the experimental settings in Section D, including our experimental environments, training settings, and inference speed. Then, we show more complementary results of our experiments in Section G, where also present the results of human-cvaluation survey. Finally, we discuss the potential social impacts of our work in Section I.
A Notations
Notation
Description
p(æo)
real data distribution
((2)(20)
forward diffusion in diffusion model
(x (Tr)
backward diffusion parameterized by ø diffusion coefficients in diffusion model backward diffusion sampling original diffusion time step SDEdit reverse timesteps iterations of PGD attack
a, 3
Rg
T
K
maximum & norm of adversarial attacks
steps of adversarial attacks foo ball sized
Boc,
x"
Diff-PGD generated adv-samples (before purification)
ab
Diff-PGD generated adv-samples (after purification) PGD-generated ady-samples
TPGD
TODD
regional PGD-generated adv-samples style reference image in style-based attacks style-transferred image copy of the original image, gradient style feature extractor content feature extractor mask to define the regional attacks DDIM timestep
2.
Ze
H.
He
M
T.
K.
SDEdit reverse timesteps with accelerator
element-wise multiplication
U
uniform distribution
B Details about Methods
B.1 Details about Style Diff-PGD
Departing from existing methods where adversarial loss and style loss are optimized together, Diff- PGD makes it possible to separate the optimization of these two losses, Diff-PGD can guarantee the realism output samples during the adversarial optimization stage. The detailed implementation demonstrated in Algorithm 2, where we add stage for style-based editing Diff-PGD.
Even though here we only present one direction customized editing (e.g. text-guided editing), this framework can deal with almost any customized adv-sample generation problem, by adopting any proper method for the first stage.
B.2 Details about Physical World Diff-PGD
15

---

Algorithm 2 Diff-PGD for Style-based Attacks
Require:
Target classifier Sa. original image v, mask M, style reference 2., style network Sa, style extract layers H.. content extract layers H.. denoised sampler Rg. style transfer learning rate ne- # of reverse SDEdit steps K, iterations for PGD n, iteration for style transfer n2., stepsize clip value e, style weight X, and content weight Ae
xe = I for 0.1.2, .... nr do L. ZIEH. [[ - G(fa(T))))
L. ZNEN. [[fa(z)) (2
x!+1 x¹1 178 Va! (NoLi, + AoLe)
D Style transfer only, without adversarial loss
end for
x In x
r.
D Used for unchanged part
for = 0, 1,2, ...,7 do
"K q(<k (x')
Sample s'x from g( (x*) in each PGD iteration
for =
1,...,0.. do
R i+1
Apply denoiser R& 2 in each SDEdit iteration
M x + (1 M) a<<f(=²)
Sample from masked combination of 2 and g(x) lx*)
end for 9 (iI a (1- M) xe)]
b Compute the gradient A PGD update
x x+
-1,,, (x' + 17 M sign(g))
end for 20 rSDFdit(""))
b. Apply reverse SDEdit to the final x
return 28
Return style-based adversarial sample
Algorithm Diff-PGD for Physical World Attacks
Require: Target classifier fa, original patch image z, mask M, denoised sampler Rgg # of reverse SDEdit steps K, iterations n, stepsize 1). physics transformation set T, learning rate y)
20 = x N. Te =
for / = 0, 0,1,2,.... ..
do
xex ~ qx<xx<"""
> Sample z'x from g<*xx< Iz") in each PGD iteration
for = K-1,... 0 do
x = R
> Apply denoiser R& to 2 in each SDEdit iteration Sample from masked combination of 2 and g (=*)
ai ~ M o x + (1 - M) o g(<((=²)
end for
T U(T)
b Sample physics transformation >> Compute the gradient b update
9 V V 21 +11 M) xx))
1+1 = x' + n sign(g)
end for
20 =SDEdit(z"))
Apply reverse SDEdit to the final x² Return Diff-Phys patch
return z
To adapt Diff-PGD the physical world settings, larger modifications needed. In Algorithm we present Dif in physical world environments. The biggest difference is that we no longer restrict the lo norm of perturbations. Also, use gradient descent instead of projected gradient descent. For the generation of the physical adaptor T, we consider the scale of the patch image æ, the position of the patch image, the background images, and the brightness of the environment.
C Accelerated Diff-rPGD
Changing only one line can help us get the accelerated version of generalized Diff-PGD with an approximated gradient (Algorithm 4).
16

---

Algorithm 4 Accelerated Diff-rPGD
Require: Target classifier fa, original image J, mask M, denoiser D # of reverse SDEdit steps K, iterations n,, stepsize 7,, clip value (when M 1 it reduces Diff-PGD)
x² = 00 = v. Fe I
for 0, ...... do
s q(< (x")
> Sample æ'K from ((") in each PGD iteration
for K ,,.,,,,
a = Rø(21+1
Apply denoiser Rş to o in each SDEdit iteration
z ~ M 2 + (1 M) o g(>)|=²)
Sample from masked combination of 2 and g( (²))
end for
V oo [Jo (M 2 + o)))
Compute the approximated gradient PGD update
x++1 II [x + M sign(g))
end for 40 rSDEdit(x*))
Apply reverse SDEdit the final x Return the final adversarial example and the denoised version
return x", %00
Method
K
n
VRAM(G)
Speed(sec/sample)
Diff-PGD
2
10
~8
v2
2
10
~4
4
Diff-PGD
3
10
20
10
v2
3 10
~
5
Table 3: The Speed and VRAM of Diff-PGD and accelerated Diff-PGD with approximal gradient (v2). From this we can see that the v2 is much cheaper to run than the original Diff-PGD. and n follows the definition in Algorithm 1.
Here, compared the speed and success rate of accelerated Diff-PGD with the original version; we refer to the accelerated version as v2. The qualitative results v2 are put Figure 10, and quantitative results about speed, VRAM, and success rate are put in Table 3 and Table 4.
D Implementation Details
Experimental Environments and Inference Speed The methods described in this study are implemented using the PyTorch framework. All the experiments conducted in this research were carried out single RTX-A6000 GPU, housed within Ubuntu 20.04 server.
Since we need do back- back-propagation on the U-Net used the diffusion model, it will cost more GPU memory and GPU time. Running on one single GPU, it takes ~ 7 seconds to run Diff-PGD with n 10, K2 = 2 for one sample on one A6000 GPU. Parallel computing with more GPUs can be used to accelerate the speed.
Method
Model
n
6
SR
v2
ResNet-50
10
16/255
99.6
v2
ResNet-50
10
8/255
98.8
v2
ResNet-50
15
16/255
100.0
v2
Re
15
8/255
100.0
Table The success rate accelerated Diff-PGD, from which we can see the success rate (SR) is still really nearly 100%, showing that v2 cheaper but still effective attack. and follows the definition in Algorithm
17

---

Global Attacks Here we use 16/255. 2/255, = 10 as our major settings (except for the ablation study settings) for both PGD and Diff-PGD. For Diff-PGD, we use DDIM50 with K, 3 as our SDEdit setting; we will also show more results with different DDIM time-steps and different KG.
Regional Attacks For the experiments of the regional attacks, we randomly select a squared mask sized 0.4H x 0.4W, where the size of the original image is H x W. For the same image in the ImageNet dataset, we use the same mask for both rPGD and Diff-rPGD. Also, we use 10 for bounded attack with projected gradient descent. And for Diff- rPGD, we use DDIM50 with K; = 2: similarly, we try more settings the ablation study section. The repainting strategy of our Diff-rPGD is based on replacement, but more effective strategies are encouraged to be used here.
= 16/255,, 7 2/255,
Style-based Attacks As mentioned in Section B, the implementation of Diff-PGD for style-based attacks can be divided into two parts. For the first stage, we use style weight X₂ = 4000 and adopt Adam optimizer with learning rate 1) 0.01 for style transfer. For the feature extractor, VGG-19 as our backbone with the first five convolutional layers as the style extractor H, and the fourth convolutional layer the content extractor He. For the target image æ; use SAM to collect the segmentation mask to define the attack region.
adopt
In the second stage, we utilize DDIM10 with a parameter setting of K, 2 to achieve better purification. is beneficial to use a smaller Kg, as the style-transferred images often contain unwanted noise and art-styled patterns that can be mitigated through this approach.
Phyiseal World Attacks The background images the physical world attacks are collected by taking photos of the target object with different camera poses. For the physical adaptor, we set the scale factor to around [0.8, 1.2] of the original patch size; then we set the random brightness to be [0.5, L.5] of the original brightness. We also set a margin for the random positions of the image patch so that the patch will not be too far away from the target object. We use an iPhone 8-Plus to take images from the real world and use an HP DeskJet-2752 for color printing. For Diff-PGD we use DDIM10 and K, = 2 to allow larger purification.
Transferability and Anti-Purification Power We show results of two additional good properties of x² and x generated by Diff-PGD: transferability and anti- mutifiiiiiiitionn power. For transferability, use 16/2555, 77 2/255, 10 in the main paper, with DDIM50 with Ke for Diff-PGD. Targeting our network on ResNet-50, we test the adv-samples among ResNet- 101, ResNet18, 8,, and WRN-50 and WRN-101.
For the test of anti- purificciiicn power, we use an SDEdit with DDIM50 with larger K, = 5 and test for both PGD and Diff-PGD with 16/255, 17 2/255, 10 and DDIM50 with K, 2 for Diff-PGD so that our method is not biased.
E Results on Vision Transformers
Compared with the Convolutional-based network such as ResNet we used before, Vision Transformers are another branch of strong classifiers that are worth taking into considera- tion. Here we try global attack on two famous models on ImageNet, ViT-b (https:// huggingface co/ hooglooolvvitttooo base passe-pptchh6-ee 224) and BEiT (attps://muginngface) co/ waweee/vittaaeteeeeeeeeeeeeeeeet
microsoft/beit- large- patch16 largt--athgg-e-
Figure demonstrates that Diff-PGD can still work well generate adversarial samples with higher stealthiness. Table 5 shows that Diff-PGD is still effective for vision transformers.
F Results on based Attacks
We also conduct experiments for ba-ba attacks, we set = 10 and 6(2) 16/255, and we can still get a high success rate as 100% and the generated adversarial samples are shown in Figure which turns out to be also realistic.
18

---

PGD
DIE--GD
PGD
Diff-PD
FGD
Dift PGD
PGD
Diff- PGD
Ad Adv-sample
NE
Purified Sample
Change
8g 20
4. 13
C, 16
4,
2g 20
Figure Purification on Adv-samples Generated by PGD vs Diff-PGD: PGD attack can be easily purified using recently proposed purification strategies using diffusion model. while Diff-PGD can generate adv-samples the realistic domain, making harder be purified. The changes in PGD contain lot of noise while the changes of Diff-PGD contain more meaningful features, which means that the adversarial noise can be easily removed for traditional PGD. Also, we show the l norm of the changes, from which can see that changes of Diff-PGD are always smaller.
Model
SR
ViT-base
10
8/255
100
ViT-base
10
16/255
100
Barree
10
8/255
99.6
BEiT-large
10
16/255
99.6
Table 5: Success rate of Diff-PGD vision transformers.
XPCD
6PGD
x
65
XPGD
8PGD
x
6
Figure 8: Performance of Difr-PGD on & perturbations all the figures attack ResNet-50 with 16/255 under l2 norm. (Zoom-in for better observation)
X
XpGD
8PGD
x
8/
X
X
8PGD
x
8
Figure 9: Qualitative Results of Diff-PGD on Vision Transformers: the left-half attack ViT-b and the right-half attack BETT-I. (Zoom-in for better observation)
19

---

Figure 10: Qualitative Results of Diff-PGD with Gradient Approximation here we show six samples, each contains æ, xx F6 from left to right. (Zoom-in for better observation)
amb
50
50
STO
10
10
Figure 1I:
More Results of Style-based Attacks: we show nine samples from the constructed dataset, each sample contains æ, M, vij from left to right.
G More Complementary Results
Global Attacks Here we present more results randomly sampled from the ImageNet validation set to show that Diff-PGD can steadily generate adv-samples with high stealthiness. In Figure 12, we show some complementary results of 16/255, n 2/255, n 10 in main part of the paper. Also in Figure 13, we show results when the perturbation bound larger: e = 32/255. rj = 4/255, = 10, where use DDIM10 with Kg from the figure we realistic adv-samples.
that Diff-PGD can still generate
Also, we demonstrate that Diff--GDD remains effective in generating adversarial samples with high success rate even when the attack bound is smaller. Figure 14 illustrates this point, where we set 8/255 and evaluate two Diff-PGD strategies: DDIM50 and DDIM100, all with T, 2. Both strategies exhibit a notable success rate and can produce realistic adversarial samples.
Regional Attacks Diff-rPGD can be adopted in customized settings where only masked regions can be attacked. Here, we show more results regional attacks Figure 20, where = 16/255, 17 = 2/255, = 10. We compare Diff-rPGD with rPGD, and the results show that our Diff-rPGD can deal with the regional adversarial attack better: the masked region can better merged into the background.
Style-based Attacks In the main paper, we show that Diff-PGD can help generate style-based adv-samples effectively. In Figure 15, further present that the second stage in our two-stage strategy is necessary and can highly preserve the realism the output images.
Phyiscal World Attacks We present the result of physical world attacks using adversarial patches generated using Diff-PGD in the main paper. In order to show that the adversarial patches are robust to camera views, we show more images taken from different camera views in Figure 16. For the two cases: computer mouse (untargeted) and back bag (targeted to Yorkshire Terrier), we randomly sample ten other camera views. The results show that the adversarial patches are robust both for targeted settings and untargeted settings. For targteted settings, our adversarial patch can fool the network to predict back bag as terriers, and for untargeted settings, the adversarial patch misleads the network to predict computer mouse as artichoke.
Transferability and Anti- Intifuurification We presented the results when 16/255 and found that: though designed for improved stealthiness and controllability, is surprising that Diff-PGD transfers better than the original PGD to five different networks, here we show more results on different loc
20

---

Figure 12: More Results of DifT-PGD with e = 16/255: five columns of each image block follows PGD. åPGD, zi and 6y is defined in the main paper. We can see that Diff-PGD can steadily generate ady-samples with higher stealthiness. Zoom on a computer screen for better visualization.
21

---

Figure 13: More Results of Diff-PGD with e 32/ 255: five columns of each image block represent I, TPD. SPGD.30 2 and 83 respectively. can that Diff-PGD can steadily generate adv-samples with higher stealthiness. Zoom in on a computer screen for better visualization.
bound. We take the other settings: = 8/255 and E 32/255 better investigate methods, for 6 = 8/255 we still use 2/255 while for e = -32/255, we use 4/255 for better attacks, both of them use
10. For Diff-PGD, we use DDIM50 with K₂ for 8/25 //255 case and DDIM30 with K, = 2 for = 32/255 case.
Results shown in Figure 17 reveal that both 26 and x" exhibit superior transferability compared to the original PGD attack. This observation aligns with our intuition that attacks operating within the realistic domain more likely successfully transfer across different networks.
For inifiiuriiiiation, we show some qualitative results that how the noises can be removed by running SDEdit on PGD and Diff-PGD. In Figure 17, we show visualizations of the adv-samples, purified adv-samples, and the changes during purification. From the figure, we can find that the adversarial patterns generated by PGD can be casily removed by diffusion-based purification and Diff-PGD.
22

---

1.2
1.2
Success Attack
0.6-
Success
Frep
Yrep
Iteration a 22255, DDIM50)
Iteration (e 8/255, 2/255 (215000000000
(a)
(b)
Figure 14: Effective of Dif with Smaller Bound: we show that when the attack foo bound is smaller: 8/255, Diff-PGD is still effective to generate adv-samples with high success rate. (a)
we use DDIM100 and K2 2, (b) we use use DDIM50 and K2 2.
10
10
50
50
CRS
Figure 15: Results of Style-based Adv-sample Generation in Two Stages: here should three cases in style-based adversarial attacks, for each case, we should the difference between (the left image in each case) and x (the right image each case), which demonstrate that by employing Diff-PGD, we are able to generate adversarial samples while simultancously mitigating the artifacts that arise during the style transfer stage.
H Human Evaluation
To better evaluate the stealthiness of Diff-PGD compared with the original PGD attack, we conduct survey among humans with the assistance of Google Form. The user interface of the survey is shown Figure 21, where the participants are allowed to choose multiple candidate images that they think realistic. We randomly choose images from the ImageNet validation set and the choices for each question include one clean image, one adv-sample generated by PGD, and one adv-sample generated by Diff-PGD. For Diff-PGD we use DDIM50 with K, 3. and for both Diff-PGD and PGD we use e 26/255, 7 = 2/255,
10.
We collect surveys from 87 participants up to the completion of the writing and most of them not familiar with adversarial attacks. Figure 18 illustrates the results, indicating that Diff-PGD approach achieves higher stealthiness among human participants compared with PGD.
I Potential Social Impacts
Adversarial attacks pose serious threat to the security of Al systems. Diff-PGD introduces novel attack technique that effectively preserves the realism of adversarial samples through the utilization of the diffusion model. This holds true for both digital and real-world settings. The emergence of
23

---

"artichoke"
"neckless"
"artichoke"
"artichoke"
"neckless"
"artichoke"
"artichoke"
"artichoke"
"artichoke"
"artichoke"
"Silk terrier"
"Yorkshire"
"Yorkshire"
"Silk terrier"
"Yorkshire"
"Yorkshire"
"Yorkshire"
"Yorkshire"
"Yorkshire"
"Yorkshire"
Figure 16: Physical World Attack with Difr-PGD: to show the effectiveness of adversarial patches generated by Diff-PGD, we show the images captured from different camera viewpoints for the two cases: computer mouse (untargeted) and back bag (targeted). For each case, we randomly sample other ten poses. We present the prediction of the target network under each image, where Yorkshire short for Yorkshire terrier.
24

---

1.0
1.0
Xico
0.8
0.8
0.7
0.7
0.5
0.5
Figure 17: More Results on Transferability Test: in the main paper we show results of transferabiliiyy
of 16/255 255 settings; here we test transferability Diff-PGD under different lo bounds, (Left):
32 32/2555,1
4/255, n 10, for Diff-PGD we use DDIM50 with Kg 2, (Right):
8/255, 1) = 2/255, n 10 and for Diff-PGD we use DDIM30 with K, = From the plots we can
that Diff-PGD has better transferability power than PGD.
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Image
Dif.PGD
Figure 18: Human Evaluation: Collecting 87 samples using Google Form, we calculate the average choose rate of each kind of sample (real images, PGD-attackes images, and Diff-PGDD-attccken images). The results show that Diff-PGD method exhibits greater stealthiness than PGD when evaluated by human participants.
such realistic domain attacks demands our attention and necessitates the development of new defense methods.
25

---

Figure 19: More Results of Diff-PGD with e 16/255: five columns of each image block follows T, PPDD SpaN, zB and o8 defined in the main paper.
26

---

Figure 20: More Regional Attack Results: we present more results for regional attacks, the three columns are clean image, F,PGD and zi respectively.
27

---

Diff-Adv Test
Note:
(multiple choices) Choose the images that you think are realistic, some of them are attacked by adding malicious noise
+ ($ TH 10
Questionnaire à choix multiples: Choisissez les images qui vous semblent réalistes, certaines d'entre elles ont été modifiées par l'ajout de bruit malveillant
Choose the one(s) that you think is(are) realistic (multiple choices):
The Interface of our Survey for Clunan-Evaluttiona one sample of our questions is presented, and we randomized the order of the three images for all questions. The options consisted of a clean image, a "GD-attacked image, and Diff-PGD-atten image, with the participants unaware of the composition of the choices.
Figure 21:
28


Reviews:
[{"review_id": "P54XeqeDJi", "reviewer": "Reviewer_PJE2", "summary": "This paper proposes a novel Diffusion-Based Projected Gradient Descent (Diff-PGD) framework which optimizes perturbations by using an off-the-shelf diffusion model. The proposed method works well on both digital world and physical world with high stealthiness.", "questions": "1. I am curious the computation cost of the proposed method.\n2. How is the performance of the proposed method on transformer based vision models and under different L-p constraint?", "limitations": "1. Test the computation time or memory cost of the proposed method.\n2. Test the proposed method under different L-p constraint.\n3. Test the proposed method on vision transformers.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is novel and performs well on both digital world and physical world.\n2. The method analysis is clear and visualization is helpful.\n3. The experiments are solid and comprehensive.", "weaknesses": "1. Lack of the efficiency report of the proposed method\n2. Lack of the performance of proposed method under different L-p constraint.\n3. Lack of the performance on transformer based vision models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review and valuable comments. Please see our response below.\n\n> Lack of the efficiency report of the proposed method\n\nWe present the detailed efficiency (speed and memory occupation) of Diff-PGD  in the section `The computational cost of Diff-PGD` in **General Response**, where we also show that we can further optimize it from both code-level and gradient approximation.\n\n>  Lack of the performance of proposed method under different L-p constraint.\n\nWe test L-2 normed attacks on 250 samples in ImageNet validation set with $\\epsilon=16*255$ and $n=10$. We present the success rate of Diff-PGD under L-2 norm, and show generated samples in the **PDF file** of **General Response**.\n\n| Method   | $n$  | $\\epsilon$ | SR (%) |\n| -------- | ---- | ---------- | ------ |\n| PGD      | 10   | 16*255      | 100    |\n| Diff-PGD | 10   | 16*255      | 100    |\n\n> Lack of the performance on transformer based vision models\n\nThanks for pointing it out. This is also mentioned by` Reviewer PvL8`. We tested Diff-PGD on two popular vit models: ViT-b and BEiT-l. The performance is listed in Section `Can Diff-PGD attack vision transform-based models ` of **General Response**, and the qualitative results of generated samples are put in the **PDF file** of **General Response**.\n"}, {"review_id": "EL0pape60d", "reviewer": "Reviewer_x4rZ", "summary": "This paper propose a gradient-based adversarial attack method that use diffusion model to generate adversarial examples.The proposed method can generate adversarial examples with high stealthiness and controllability.", "questions": "Does the diffusion model necessary in the pipeline? \nIf we use other adversarial purification method in the pipeline, will the results will be better?", "limitations": "Lack the experiment result on the adversarial robust models.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed method can improve the stealthiness, anti-purification ability and transferability together. In addition, the proposed method can also be implemented in the physical world.\n\n2.The experiments show that the proposed method is effective.", "weaknesses": "1.The novelty of the paper is somewhat limited. The gradient-based adversarial attack method has already been proposed [1]. However, the authors have not cited the paper.\n\n2.In the area of adversarial attacks, it is essential to improve the transferability of the adversarial attack. Although the proposed method can enhance the transferability, the authors has never explained the reason of that.\n\n[1] DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for conducting a careful review and providing valuable feedback. We highly appreciate the reviewer's recognition of the effectiveness of our method. In this response, we aim to address the questions raised by the reviewer:\n\n>The novelty of the paper is somewhat limited. The gradient-based adversarial attack method has already been proposed [1]. However, the authors have not cited the paper.\n\nWe read the paper [1] and agree that it is related to our work. We will cite this work in our paper. However, after careful review, we have found the paper differs from our Diff-PGD: first they specific their task on facial protection, also they optimize the latent vector in the latent DM which needs additional regularizations like facial segmentation mask.\n\n**Furthermore, the paper [1] was published (2023/05/23) after the NeurIPS submission deadline (2023/05/17). It would have been impossible for us to become aware of this paper before the deadline.**\n\n> In the area of adversarial attacks, it is essential to improve the transferability of the adversarial attack. Although the proposed method can enhance the transferability, the authors has never explained the reason of that.\n\nWe have briefly explained the reason why Diff-PGD can enhance the transferability in Line 285-286 in the manuscript. According to previous works, adv-attacks in semantic space can be better transferred [1, 2]. People define the semantic space using some high-level subspace (e.g. color, blur, shading). In this work, we have diffusion model to curve a better semantic space (the distribution of natural images), which can intuitively explain why Diff-PGD can improve the transferability.\n\n> Does the diffusion model necessary in the pipeline? If we use other adversarial purification method in the pipeline, will the results will be better?\n\nThank you for raising this interesting question.\n\nIt is more convenient to use pre-trained DM in the pipeline of adv-sample generation. First, Diff-PGD serves as a plug-and-play framework, and we only need to plug in a pre-trained SDEdit (input-output-module) to help generate more stealthy adv-samples. \n\nFurthermore, leveraging DM as a robust prior knowledge source significantly aids in the purification process [3]. In contrast, alternative adversarial purification techniques, such as using GAN, would demand re-training [4] or applying regularization [5]. Even with these efforts, the generated samples still tend to retain prominent artifacts [5]. DM, on the other hand, reduces these limitations, facilitating the process and enhancing the quality of the generated samples.\n\n[1] Improving the Transferability of Adversarial Samples with Adversarial Transformations, CVPR 2021  \n[2] Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks, NeurIPS 2022  \n[3] Diffusion models for adversarial purification, ICML 2022  \n[4] Generating Adversarial Examples with Adversarial Networks, IJCAI 2018. \n[5] AI-GAN: Attack-Inspired Generation of Adversarial Examples  \n"}, {"review_id": "yco3yjbn25", "reviewer": "Reviewer_B84v", "summary": "This paper proposes to use diffusion models to improve the quality of adversarial examples. It improves the Projected Gradient Descent (PGD) attack by performing a few steps of Stochastic Diffusion Editing (SDEdit) before taking each projected gradient descent (i.e., adversarial) step. The proposed method, DiffPGD, can also be adapted for regional adversarial and style-customized attacks. Experiment results show that DiffPGD can output high-quality images that attack classifiers effectively.", "questions": "1. The evaluation metric used in Table 2 is unclear. Is it the attack success rate?", "limitations": "1. As I mentioned in the weaknesses, the major limitation of this work is its computation overhead. It seems impractical to incorporate this attack method into model training as a defense method, while the original PGD can be easily applied to adversarial training. \n2. Limited baselines are considered in the current experiment section, which makes the effectiveness of the proposed method unclear. For example, when demonstrating the transferability and anti-purification of Diff-PGD, this paper only compares to PGD. It is important to compare Diff-PGD with more GAN-based adversarial attack methods to show its advantage. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of using diffusion models to improve the quality of adversarial examples is interesting. The proposed method is shown to generate more natural-looking adversarial examples with better transferability.  \n2. The evaluation considers different attack scenarios, including digital and physical attacks. \n", "weaknesses": "1. The proposed method is much more computationally expensive (in terms of both time and GPU memory) than regular adversarial attacks, while its advantage over existing methods seems marginal (Figure 6(a)). \n2. Many of the experiment results only provide less than five examples, which are inadequate to support the claims (i.e., the four questions listed in Lines 220 - 224) about the proposed methods. For example, the result in Figure 6 (a) is conducted only on 250 samples (rather than the entire 50,000 ImageNet validation samples). The answers to Q2 and Q3 are based only on the examples shown in the figures. It is unclear how well Diff-PGD and its variants perform at a larger scale.  \n3. The comparison with baselines is not very comprehensive. For example, Figure 4 only shows a qualitative comparison with AdvCam, but no attack success rate is considered in this comparison. While DiffPGD is shown to produce more realistic-looking images in Figure 4, it is not clear if they are as effective as baseline adversarial examples.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review and valuable comments, we provide answer below. \n\n> The proposed method is much more computationally expensive (in terms of both time and GPU memory) than regular adversarial attacks, while its advantage over existing methods seems marginal (Fig 6(a)).\n\n**First, as said in the General response, we want to emphasize that generating better adversarial attacks, even expensive ones, is an important research direction, as it can help better understand ML systems, and thus help to find potential risks and possible solutions. Works like AutoAttack [2] are known to be computationally expensive (2s/img on GPU) and has never been used in AT, but this attack is often used to evaluate the robustness of ML systems. Evaluating defenses with strong attacks serves as a more reliable evaluation of adversarial robustness.**\n\nSecond, it appears that there might be a misunderstanding regarding Fig 6(a): Diff-PGD can achieve 100% success rate as PGD when $n\\geq5$, which means it is **still an effective attack** regarding success rate. The advantage over existing methods mainly focus on higher stealthiness and flexibility, which is not marginal.\n\nRegarding the computational cost, we presented a detailed efficiency report of Diff-PGD in General Response, where we show that we can further **optimize** from both code-level and gradient approximation level.\n\n> experiment results only provide less than five examples, which are inadequate to support the claims about the proposed methods. For example, the result in Fig 6 (a) is conducted only on 250 samples (rather than the 50k ImageNet validation). The answers to Q2 and Q3 are based only on the examples shown in the figures. It is unclear how well Diff-PGD and its variants perform at a larger scale.\n\nFor Fig 6(a) we want to prove that Diff-PGD is an effective attack, and we uniformly sampled 250 images from ImageNet validation. Further, we scale the size of sampled subset up to 2.5k and run our Diff-PGD with the same settings with $n=5,10$:\n\n| Method   | $n$  | $\\epsilon (l_{\\infty})$ | SR (%) in 2.5k Samples |\n| -------- | ---- | ----------------------- | ---------------------- |\n| PGD      | 5    | 16                      | 100                    |\n| Diff-PGD | 5    | 16                      | 100                    |\n| PGD      | 10   | 16                      | 100                    |\n| Diff-PGD | 10   | 16                      | 100                    |\n\nfrom this we can see Diff-PGD can achieve steadily effective attacks.\nAlso, for qualitative results corresponding to **style-based attacks**, we provide more samples in the **PDF**.\n\n> The comparison with baselines is not very comprehensive. Figure 4 only shows a qualitative comparison with AdvCam, but no attack success rate is considered. While DiffPGD is shown to produce more realistic-looking images in Figure 4, it is not clear if they are as effective as baseline adversarial examples.\n\nHere we clarify the success rate of style-based attacks: Quantitatively, we initially collected a dataset comprising ~15 scenarios, each consisting of a (source image, style-reference) pair. The final SR(success rate) is:\n\n| Method   | SR(%) |\n| -------- | ----- |\n| AdvCAM   | 100   |\n| Diff-PGD | 100   |\n\nfrom which we can see that, to successfully attack a DNN with style reference is not a difficult task (the attack is unrestricted), but the qualitative results of generated samples differs, where our Diff-PGD is much better than AdvCAM (Fig 4).\n\n> The evaluation metric used in Table 2 is unclear. Is it the attack success rate?\n\nThe metric employed in Table 2 is the attack success rate, and we will clarify this in the revision.\n\n> As I mentioned in the weaknesses, the major limitation of this work is its computation overhead. It seems impractical to incorporate this attack method into model training as a defense method, while the original PGD can be easily applied to adversarial training.\n\nOur work focuses on generating adversarial samples, which is itself an important problem since better adversarial samples can help us reveal the true impression of robustness [6]. **We would like to emphasize that not every defense relies on AT and devising strong attacks can help better evaluate defense strategies.**\n\nHowever, applying AT to defend against this attack is an interesting question to investigate. One possible strategy would be to generate adv-samples with a stochastic process ($x_K$ is sampled with $x$ plus Gaussian noise), then we can run inference to generate(sample) more samples before a run gradient update, which will be a more efficient data augmentation approach.\n\n> Limited baselines are considered in the experiment section, which makes the effectiveness of the proposed method unclear. For example, when demonstrating the transferability and anti-purification of Diff-PGD, this paper only compares to PGD. It is important to compare Diff-PGD with more GAN-based adversarial attack methods to show its advantage.\n\nFor digital attacks, [1, 2] either need to retrain the generative network or need a regularization term, in contrast to our training-free method. Moreover, the generated images using these methods also exist perceptible artifacts [2].\n\nFor physical world attacks, most GAN-based baselines [3, 4, 5] are hard to apply to attacks with reference images. Most of them sample new adversarial samples from the noise, which may differ a lot from the given samples. Thus we did not take them into account in our experiments.\n\n[1] Generating Adversarial Examples with Adversarial Networks\n[2] AI-GAN: Attack-Inspired Generation of Adversarial Examples  \n[3] Naturalistic physical adversarial patch for object detectors  \n[4] Advart: Adversarial art for camouflaged object detection attacks  \n[5] Patch of invisibility: Naturalistic black-box adversarial attacks on object detectors  \n[6] Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks"}, {"review_id": "J41sYLy5PJ", "reviewer": "Reviewer_PvL7", "summary": "The paper proposes a framework called Diffusion-Based Projected Gradient Descent (Diff-PGD) for generating realistic and stealthy adversarial samples to deceive neural network models. Diff-PGD utilizes a diffusion model to guide the optimization process, ensuring that adversarial samples remain close to the original data distribution while maintaining their effectiveness. The framework offers customization for specific attack scenarios, such as digital attacks, physical-world attacks, and style-based attacks. It separates the optimization of the adversarial loss from other surrogate losses, leading to improved stability and controllability. The authors demonstrate that adversarial samples generated using Diff-PGD exhibit better transferability and anti-purification power compared to traditional gradient-based methods.", "questions": "Please address my questions in the weakness section. \n", "limitations": "The authors elaborated on the limitations of the approach. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality \n(+) The paper introduces a novel framework, Diff-PGD, which combines the use of diffusion models with adversarial sample generation. This integration of diffusion models into the generation of adversarial samples is a unique approach that has not been explored extensively in prior research.\n\n## Quality \n(+) The paper provides a thorough description of the proposed framework, Diff-PGD, and its underlying principles. \n\n## Clarity \n(+) The paper is well-written and effectively communicates the key concepts and ideas. \n\n## Significance \n(+) By introducing the Diff-PGD framework, the authors offer a solution that improves the stealthiness of adversarial samples while maintaining their effectiveness. \n\n(+) The provided experimental evaluations (quantitative and qualitative) are convincing. \n", "weaknesses": "(-) The phenomenon of the vulnerability of DNNs/ViTs to AEs is well known. How does the proposed attack perform against hardened models, i.e. adversarially trained models?  \n(-) Did the authors experiment with transformer-based models, such as ViT? I would be interested in the attack performance of these models as well. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for conducting a careful review and providing valuable feedback. We highly appreciate the reviewer's recognition of the effectiveness of our method. In this response, we address the questions raised by the reviewer:\n\n\n\n> The phenomenon of the vulnerability of DNNs/ViTs to AEs is well known. How does the proposed attack perform against hardened models, i.e. adversarially trained models?\n\n\n\nIn Figure 6(c), we compare the success rate of Diff-PGD vs PGD on adversarially trained networks, where the samples are generated on the **naturally-trained ResNet-50**.\n\n\n\nWe conduct further experiments for Diff-PGD to attack adv-trained ResNet-50 directly. The successful rate is reported below:\n\n| Method   | model (adv-trained) |$n$  | $\\epsilon (l_{\\infty})$ | SR (%) |\n| -------- | ---- | ---- |----------------------- | ---------------------- |\n| PGD      | Salman2020Do_R50 | 10   | 16 | 90.8              |\n| Diff-PGD | Salman2020Do_R50 | 10    | 16 | 95.2            |\n| PGD      | Salman2020Do_R50 | 20 | 16 | 95.2   |\n| Diff-PGD | Salman2020Do_R50 | 20  | 16 | 96.0    |\n\nwhere `Salman2020Do_R50` is model card of adv-trained ResNet-50 from Robustbench [1].  We can see that the with the same budgets with sufficient steps, Diff-PGD can still generate better samples with higher success rate.\n\n\n\n\n\n> Did the authors experiment with transformer-based models, such as ViT? I would be interested in the attack performance of these models as well.\n\n\n\nThis question is also raised by Reviewer  PJE2. We tested Diff-PGD on popular vit models: ViT-b and BEiT-l. The performance is listed in the first Section of **General Response**. \n\nAlso, more **qualitative results** of generated adversarial samples attacking ViTs are put in the **PDF File** of the **General Response** section.\n\n"}]

Author rebuttal:
# General Response

We thank the reviewers for their valuable comments on our paper. We are excited to see that all the reviewers (Reviewer PJE2, B84v, x4rZ, PvL7) identified the novelty of our technical contribution: utilizing DM to generate adversarial samples, appreciated the application to both digital attacks and physical attacks (Reviewer PJE2, B84v, x4rZ), and found the paper is well-presented (Reviewer PvL7).

Below we address some **commonly asked questions**:



>  ###  Can Diff-PGD work under other l-p perturbations, can it attack vision transform-based models? (Reviewer PJE2, PvL8)

We test L-2 normed attacks on 250 samples in ImageNet validation set with $\epsilon=16*255$ and $n=10$, and find that Diff-PGD can still generate adv-samples with improved stealthiness (refer to the pdf file attached) with high success rate:

| Method   | $n$  | $\epsilon$, $l_2$ | SR (%) |
| ------- | ---- | --------- | ------ |
| PGD      | 10   | 16*255 |100|
| Diff-PGD | 10| 16*255 |100|

For ViT-based model, we replace the ResNet-based models with two popular ViT models (raw ViT-b and BEiT-l), and test the same adv-attack settings as ResNet models in the paper. We can see that Diff-PGD is still effective in attacking ViT-based models:

When set $n=10, \epsilon=16$ (same as our experiments in Section 5 Q(1), DDIM50 with $K_s=3$), the success rate (SR) is still 100%. More detailed results are shown in the following table:

| Model  |  $n$  | $\epsilon$ | SR (%) |
|---|---|---|---|
| ViT_b  |10| 8 | 100 |
| ViT_b  |10 |16 | 100 |
| BEiT_l  |10| 8 | 99.6 |
| BEiT_l  |10| 16 | 99.6|

more **qualitative results** of generated adversarial samples attacking $l_2$ or ViTs are put in the **PDF File** of the **General Response** section.

>  ### The computational cost of Diff-PGD (Reviewer PJE2, B84v)

Firstly, we want to emphasize that generating better adversarial samples itself is important for us to revisit the robustness of our AI systems, helping us figure our potential risks and seek possible solutions. Works like AutoAttack [2] are good examples, it is also slow to run (2s for one image)  and  can not be used in adversarial training though, they still serve as a more reliable evaluation of adversarial robustness.

Here we show the computational cost of our raw Diff-PGD on **single A6000 card**, with different SDEdit step $K$ and iteration $n$: 

| $K$  |  $n$  | VRAM (G) |  Speed (s/sample)|  
|---|---|---|---|
| 2  |5|  ~18  | ~4 | 
| 3 |  5 |~20 | ~5|
| 2  |  10  | ~18  | ~8| 
| 3 |  10 |   ~20 | ~10|

The main bottleneck comes from calculating the gradient of SDEdit purified sample $x_0$ over $x$, where the gradient of the large U-Net of DM needs to be saved. Next we present two possible ways to mitigate this issue.

(1) ***Code-level Optimization***: frequently release the GPU memory (**lower VRAM**):

when calculating the gradient of the chain in SDEdit: $\frac{\partial x_0}{\partial x} = \frac{\partial x_K}{\partial x}(\frac{\partial x_{K-1}}{\partial x_{K}}\frac{\partial x_{K-2}}{\partial x_{K-1}}...\frac{\partial x_0}{\partial x_1})$, we only care about the gradient over $x$. Thus we do not need to keep the gradient of all the model parameters along all the chain. What can be optimized from the code level is: first calculate $\frac{\partial x_{K-1}}{\partial x_{K}}$, then calculate $x_{K-2}$, save only the gradient and release the GPU VRAM, and then we can calculate $\frac{\partial x_{K-2}}{\partial x_{K-1}}$. By frequently releasing the GPU memory, we avoid saving gradient over a long chain of computational graph:

| $K$  |  $n$  | VRAM (G) |  Speed (s/sample)|  
|---|---|---|---|
| 2  |  5  | ~10  | ~8|
| 3 |  5 |   ~10 | ~10 |

finally, it sacrifices the speed to make it able to run on devices with less GPU memory.


(2) ***Gradient Approximation*** (**lower VRAM, faster speed**)

we can also approximate this complex gradient with a constant: $\frac{\partial x_0}{\partial x} = \frac{\partial x_K}{\partial x}(\frac{\partial x_{K-1}}{\partial x_{K}}\frac{\partial x_{K-2}}{\partial x_{K-1}}...\frac{\partial x_1}{\partial x_0}) \approx c$. The final adversarial update in the $i$-the step becomes (this approximation is also used in SDS (score distillation sampling [1]):

$
\frac{\partial L(f(x^i_0))}{\partial x} \approx  c\frac{\partial L(f(x^i_0))}{\partial x^n_0}
$

from which it can be seen that, we only need to calculate the gradient over $x^n_0$! We only run inference of SDEdit to get $x^n_0$ without gradient saving (lower gpu memory, faster), making it much cheaper. The final cost is:

| $K$  |  $n$  | VRAM (G) |  Speed (s/sample)|  
|---|---|---|---|
| 2  |  10  | ~4  | ~4  |
| 3 |  10 |   ~4 | ~5  |



we also add some generated samples with this approximation (in ***additional PDF***) and report success rate of it below:

| Model  |  $n$  | $\epsilon$ | SR (%) |
|---|---|---|---|
| ResNet50  |  10  | 16  | 99.6 |
| ResNet50  |  10  | 8  | 98.8 |
| ResNet50  |  15 | 16  | 100 |
| ResNet50  |  15  | 8  | 100 |

for the global attack tasks, the approximated gradient still shows a high success rate but saves **50%** time and **75%** VRAM !







> ###  Can Diff-PGD work in a large scale with stable efficiency? (Reviewer B84v)

To show that Diff-PGD can be effective with low instability for **global attack**, we rescale the sample number to 2.5k and generate samples with the same settings of Figure 6(a) (ResNet50) with $n=10$, the success rate is:

| Method  |  $n$  | $\epsilon$ | SR (%) |
|---|---|---|---|
| PGD  | 5 |16 | 100 |
| Diff-PGD  | 5  | 16 | 100  |
| PGD  | 10 | 16 | 100 |
| Diff-PGD  | 10 |16 | 100  |

which means that Diff-PGD can steadily attack almost all the cases in the validation dataset.
Also, we collect more cases of **style-based** attacks and show the results in the ***additional PDF*** of the general response.


[1] DreamFusion: Text-to-3D using 2D Diffusion

[2] Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks


Meta-review: The recommendation is based on the reviewers' comments, the area chair's personal evaluation, and the post-rebuttal discussion. 

This paper studies the use of diffusion models with projected gradient descent for generating realistic adversarial samples. All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of reviewers. Therefore, I recommend acceptance of this submission. I also expect the authors to include the new results and discussion during the rebuttal phase to the final version.
Decision: Accept (poster)