SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
x ^ {*} = \underset {x} {\arg \min } l _ {\text {D i f f - P h y s}} (x) \quad \text {a n d} \quad x _ {0} ^ {*} = \operatorname {S D E d i t} \left(x ^ {*}, K\right). \tag {9}
$$
The pipeline works as follows. We first gather background data and define the scope of translations and lighting conditions, to get  $\mathcal{T}$ . Then we optimize our patch in simulation to get  $x_0^*$ . Finally, we print  $x_0^*$  out, stick it to the target object, take photos of the object, and test it on the target classifier.
The experiment section aims to answer the following questions: (Q1) Is Diff-PGD/Diff-rPGD effective to generate adv-samples with higher realism? (Q2) Can Diff-PGD be easily applied to generate better style-customized adv-samples? (Q3) Can Diff-PGD be applied to physical world attacks? (Q4) Do adversarial samples generated by Diff-PGD show better properties like transferability and anti-purification ability?
Datasets, Models, and Baselines. We use the validation dataset of ImageNet [8] as our dataset to get some statistical results for global attacks and regional attacks. For the style-guided adv-sample generation, we use some cases from [12] but also collect more images and masks by ourselves. We use VGG-19 as the backbone network to compute the style distance. The main target classifier to be attacked across this paper is ResNet-50 [18], and we also use ResNet-101, ResNet18, and WideResNet(WRN) [51] for the transferability test. Besides convolutional-based networks, we also try our Diff-PGD on Vision Transformers like ViT-b [11] and BEiT-1 [3], and the results are put in Appendix E. The diffusion model we adopt is the unconditional diffusion model pre-trained on
Figure 2: Visualization of Adversarial Samples generated by Diff-PGD: adv-samples generated using PGD  $(x_{\mathrm{PGD}})$  tend to be unnatural, while Diff-PGD  $(x_0^n)$  can preserve the authenticity of adv-samples. Here  $x$  is the original image,  $\delta_{\mathrm{PGD}} = x - x_{\mathrm{PGD}}$  and  $\delta_0^n = x - x_0^n$ , and we scale up the  $\delta$  value by five times for better observation. Zoom in on a computer screen for better visualization.
Figure 3: Visualization of Adversarial Samples generated by Diff-rPGD: Diff-rPGD can generate better regional attacks than PGD: the attacked region can better blend into the background. The attacked regions are defined using red bounding boxes, and  $(+)$  means zoom-in.
ImageNet [9] though we use DDIM [40] to respace the original timesteps for faster inference. More details about the experimental settings are included in the appendix.
We compare Diff-PGD with threat models such as PGD [31] (for digital attacks), AdvPatch [4] (for physical-world attacks), and AdvCam [12] (for customized attacks and physical-world attacks). All threat models generate attacks within the context of image classification tasks. More details are included in the supplementary materials.
Diff-PGD For Digital Attacks (Q1). We begin with the basic global  $\ell_{\infty}$  digital attacks, where we set  $\ell_{\infty} = 16 / 255$  for PGD and Diff-PGD, and # of iterations  $n = 10$  and step size  $\eta = 2 / 255$ . For Diff-PGD, we use DDIM with timestep  $T_{s} = 50$  (noted as DDIM50 for simplicity), and  $K = 3$  for the SDEdit module. Figure 2 shows that adv-samples generated by Diff-PGD  $(x_0^n)$  are more stealthy, while PGD-generated samples  $(x_{PGD})$  have some patterns that can be easily detected by humans. This contrast is clearer on the perturbation map:  $\delta_{PGD}$  contains more high-frequency noise that is weakly related to local patterns of the instance  $x$ , and  $\delta_0^n$  is smoother and highly locally dependent. Similarly, for regional adversarial attacks, Diff-rPGD can generate adv-samples that can better blend into the unchanged region, as demonstrated in Figure 3, showing higher stealthiness than rPGD.
We also conduct experiments to show the effectiveness of Diff-PGD regarding the Success Attack Rate. We uniformly sampled 250 images from the ImageNet validation set. From Figure 6 (a), we can see that although the gradient is restricted to generate realistic perturbations, Diff-PGD can still reach a high success rate with more than 5 iterations. Detailed settings can be found in the supplementary materials.
We also conduct experiments on different norms. We show additional results of the performance of Diff-PGD on  $\ell_2$ -based attacks in Appendix F.
Diff-PGD For Style-Customized Attacks (Q2). For customized attacks, we focus on the task of generating adversarial samples using a mask  $M$  and a style reference image  $x_{s}$  provided by the user. We compare our approach, described in Section 4.3, with the AdvCam method. As previously mentioned, the main advantage of our pipeline is that we do not need to balance multiple losses, and can instead divide the task into two stages: generate a style-transferred image  $\hat{x}_s$  without considering
Figure 4: Generating Adversarial Samples with Customized Style: Given the original image  $x$ , a style mask  $M$ , and a style reference image  $x_{s}$ , Diff-PGD can generate more realistic samples, even in cases where only local styles are given (e.g. only the door of the red car is offered as a  $x_{s}$ ).
Figure 5: Results of Physical-World Attacks: We show two scenarios of physical world attacks: the first row includes untargeted attacks on a small object: computer mouse, and the second row includes targeted attacks on a larger object: backpack, where we set the target to be Yorkshire terrier. The sticks-photo pairs include clean patch (green box), AdvPatch(blue box), AdvCam generated patch (red box), and our Diff-Phys generated patch (black box).
adversarial properties, and then generate a realistic perturbation using Diff-PGD to make the image adversarial to the classifier.
For Diff-PGD, we use DDIM10 with  $K = 2$ . Figure 4 demonstrates that our method consistently generates customized adversarial samples with higher stealthiness, while the adversarial samples generated by AdvCam exhibit digital artifacts that make them appear unnatural. Also, in some cases, the style reference cannot be easily transferred (e.g. only local styles are given in  $x_{s}$ , or  $x$  and  $x_{s}$  are collected with different resolution), resulting in an unsatisfactory  $\hat{x}_{s}$ , such as red/yellow car style in Figure 4. Diff-PGD can refine the samples with more details, thanks to the strong generation capacity of diffusion models.
Diff-PGD For Physical-World Attacks (Q3). In the physical-world attacks, we set the problem as: given an image patch (start point of optimization), and a target object (e.g. attack a backpack), we need to optimize the patch to fool the classifier when attached to the target object. We use an iPhone 8-Plus to take images from the real world and use an HP DeskJet-2752 for color printing. In AdvPatch, only adversarial loss is used to optimize the patch; in AdvCAM, content loss and smooth loss are optimized together with adversarial loss; for Diff-Phys, we only use purified adversarial loss defined in Equation (8), and the SDEdit is set to be DDIM10 with  $K = 2$  for better purification.
The results are shown in Figure 5, for the untargeted attack, where we use a green apple as a patch to attack a computer mouse. For targeted attack, we use an image of a cat to attack a backpack, with the target-label as "Yorkshire terrier". We can see that both AdvPatch and AdvCam generate patches with large noise, while Diff-Phys can generate smoother and more realistic patches that can successfully attack the given objects.
(b)
Figure 6: This figure presents quantitative results on our approach, with all y-axis representing the success rate of attacks: (a) Successful rate of Diff-PGD vs PGD; (b) Results of transferability of Diff-PGD vs PGD, where  $\epsilon = 16 / 255$  and  $\eta = 2 / 255$ . The adv-samples are generated with ResNet-50 (R50) and tested on ResNet-18 (R18), ResNet-101 (R101), WRN-50 (WR50), and WRN-101 (WR101); (c) Success rate of Diff-PGD vs PGD on adversarially trained networks. AT uses adversarial training strategy in [13] and  $\mathrm{AT^{*}}$  uses AT strategy in [39] on the ImageNet dataset.
Table 2: Anti-purification Results: Adv-samples generated by PGD and Diff-PGD against adversarial purification: (+P) means the classifier is enhanced by an off-the-shelf adversarial purification module. Here we set  $\epsilon = 16 / 255$ ,  $n = 10$ . The attacks are operated on ResNet-50.
Sample	(+P)ResNet50	(+P)ResNet101	(+P)ResNet18	(+P)WRN50	(+P)WRN101
\(x_{\text{PGD}}\)	0.35	0.18	0.26	0.20	0.17
\(x_{n}\)(Ours)	0.88	0.38	0.36	0.32	0.28
\(x_{n}^{0}\)(Ours)	0.72	0.36	0.37	0.36	0.36
Exploring Other Properties of  $x_0^n$  and  $x^n$  (Q4). Finally, we investigate additional properties of adversarial samples generated using Diff-PGD. Among the two samples  $x_0^n$  and  $x^n$ , the former exhibits both adversarial characteristics and a sense of realism, while the latter is noisier but contains adversarial patterns that are more difficult for SDEdit to eliminate. Although our approach does not specifically target enhancing Transferability and Purification power, we demonstrate that Diff-PGD surpasses the original PGD in these two aspects.
Transferability. We test the adv-samples targeting ResNet-50 on four other classifiers: ResNet-101, ResNet-18, WRN-50 and WRN-101. From Figure 6(b) we can see that both  $x^n$  and  $x_0^n$  generated by Diff-PGD can be better transferred to attack other classifiers than PGD. This can be explained by the intuition that adv-attacks in semantic space can be better transferred. We also test the success rate attacking adversarily trained ResNet-50 in Figure 6(c) and we can see  $x^n$  is much better than other adversarial samples.
Anti-Purification. Following the purification pipeline in [34], we attach an off-the-shelf sample purifier to the original ResNet-50 and test the success rate of different adv-samples. In Table 2 we can see that the adv-samples  $x_{PGD}$  generated by PGD are easily purified using the enhanced ResNet-50. In contrast, our adv-samples,  $x^n$  and  $x_0^n$ , show better results than  $x_{PGD}$  by a large margin. It can be explained by the type of perturbations: out-of-distribution perturbations used in PGD can be removed using diffusion-based purification, while in-distribution attacks in Diff-PGD are more robust to purification.
6 Accelerated Diff-PGD with Gradient Approximation
From Algorithm 1, we can see that we need to calculate the derivative of the model's output over the input, where the model is composed of chained U-Nets  $R_{\phi}$  in the diffusion model followed by a target classifier  $f_{\theta}$ . Here we focus on the computational bottleneck, which is the derivative of  $K$ -step SDEdit outputs  $x_0$  over SDEdit input  $x$ :
$$
\frac {\partial x _ {0}}{\partial x} = \frac {\partial x _ {K}}{\partial x} \left(\frac {\partial x _ {K - 1}}{\partial x _ {K}} \frac {\partial x _ {K - 2}}{\partial x _ {K - 1}} \dots \frac {\partial x _ {1}}{\partial x _ {0}}\right) \approx c \tag {10}
$$
We approximate the gradient with a constant  $c$ . We notice that the approximation of this Jacobian has been used in many recent works [36, 49, 7]. We also tried this strategy in Diff-PGD, and we got the accelerated version:
$$
\frac {\partial L \left(f \left(x _ {0} ^ {i}\right)\right)}{\partial x} \approx c \frac {\partial L \left(f \left(x _ {0} ^ {i}\right)\right)}{\partial x _ {0} ^ {n}}. \tag {11}
$$
From this we can see that, we only need to calculate the gradient over  $x_0^n!$ . We only run the inference of SDEdit to get  $x_0^n$  without saving the gradient (lower GPU memory, faster), making it much cheaper. We found that, for the global attack tasks, the approximated gradient still shows a high success rate but saves  $50\%$  of time and  $75\%$  of VRAM. More detailed results about the visualization and success rate can be found in Appendix C.
In this paper, we propose a novel method to power the generation of adversarial samples with diffusion models. The proposed Diff-PGD method can improve the stealthiness of adv-samples. We further show that Diff-PGD can be easily plugged into global digital attacks, regional digital attacks, customized attacks, and physical-world attacks. We demonstrate through experiments that our methods outperformed the baselines and are effective and stable. The major limitation of our method is that the back-propagation is more expensive than traditional gradient-based methods; however, we believe that the strong prior knowledge of the diffusion model can push forward adversarial attacks & defenses. Finally, while the proposed adversarial attack method could be potentially used by malicious users, it can also enhance future efforts to develop robust defense mechanisms, thereby safeguarding the security of AI systems.
The authors would like to thank the anonymous reviewers for useful comments. HX and YC are supported by grants NSF 2008513 and NSF 2206576. AA is supported in part by the Army Research Office under grant number W911NF-21-1-0155 and by the New York University Abu Dhabi (NYUAD) Center for Artificial Intelligence and Robotics, funded by Tamkeen under the NYUAD Research Institute Award CG010. BH is supported by the NSF award CAREER-2048168 and the AFOSR award FA9550-23-1-0732.
1 Introduction 1
2 Related Work 2
3 Background 3
4 Diffusion-based Projected Gradient Descent 4
4.1 Diffusion-based Projected Gradient Descent 4
4.2 Extended Diff-PGD into Region Attack 5
4.3 Customized Adversarial Sample Generation 5
4.4 Physical World Attack with Diff-PGD 6
5 Experiments 6
6 Accelerated Diff-PGD with Gradient Approximation 9
7 Conclusions 10
A Notations 15
B Details about Methods 15
B.1 Details about Style Diff-PGD 15
B.2 Details about Physical World Diff-PGD 15
C Accelerated Diff-rPGD 16
D Implementation Details 17
E Results on Vision Transformers 18
F Results on  $\ell_2$  -based Attacks 18
G More Complementary Results 20
H Human Evaluation 23
I Potential Social Impacts 23
The appendix is organized as follows: in Section A, we provide a summary table of notations to assist readers in better understanding the content. Then we delve into the details of our algorithms in Section B, including how to apply Diff-PGD to style-based attacks and physical world attacks step by step. Then, we show the experimental settings in Section D, including our experimental environments, training settings, and inference speed. Then, we show more complementary results of our experiments in Section G, where we also present the results of a human-evaluation survey. Finally, we discuss the potential social impacts of our work in Section I.
Notation	Description
p(x0)	real data distribution
q(xt|x0)	forward diffusion in diffusion model
pφ(xt-1|xt)	backward diffusion parameterized by φ
α,β	diffusion coefficients in a diffusion model
Rφ	backward diffusion sampling
T	original diffusion time step
K	SDEdit reverse timesteps
n	iterations of PGD attack
ε	maximum ℓ∞ norm of adversarial attacks
η	steps of adversarial attacks
B∞,ε	ℓ∞ ball sized ε
xn	Diff-PGD generated adv-samples (before purification)
x0n	Diff-PGD generated adv-samples (after purification)
xPGD	PGD-generated adv-samples
xrPGD	regional PGD-generated adv-samples
xs	style reference image in style-based attacks
x̂s	style-transferred image
xc	copy of the original image, no gradient
Hs	style feature extractor
Hc	content feature extractor
M	mask to define the regional attacks
Ts	DDIM timestep
Ks	SDEdit reverse timesteps with accelerator
o	element-wise multiplication
U	uniform distribution
B Details about Methods
B.1 Details about Style Diff-PGD
Departing from existing methods where adversarial loss and style loss are optimized together, Diff-PGD makes it possible to separate the optimization of these two losses, as Diff-PGD can guarantee the realism of output samples during the adversarial optimization stage. The detailed implementation is demonstrated in Algorithm 2, where we add a stage for style-based editing to Diff-PGD.
Even though here we only present one direction of customized editing (e.g. text-guided editing), this framework can deal with almost any customized adv-sample generation problem, by adopting any proper method for the first stage.
B.2 Details about Physical World Diff-PGD
Algorithm 2 Diff-PGD for Style-based Attacks
Require: Target classifier  $f_{\theta}$  original image  $x$  ,mask  $M$  style reference  $x_{s}$  style network  $f_{h}$  style extract layers  $H_{s}$  content extract layers  $H_{c}$  denoised sampler  $R_{\phi}$  style transfer learning rate  $\eta_s$  # of reverse SDEdit steps  $K$  , iterations for PGD  $n$  , iteration for style transfer  $n_{s}$  , stepsize  $\eta$  , clip value  $\epsilon$  ,style weight  $\lambda_{s}$  and content weight  $\lambda_{c}$ $x_{s}^{0} = x$    
for  $t = 0,1,2,\dots ,n_s - 1$  do   
 $L_{s} = \sum_{h\in H_{s}}\| G(f_{h}(x_{s}^{t})) - G(f_{h}(x_{s}))\|_{2}^{2}$ $L_{c} = \sum_{h\in H_{c}}\| f_{h}(x_{s}^{t}) - f_{h}(x_{s})\|_{2}^{2}$ $x_{s}^{t + 1} = x_{s}^{t} - \eta_{s}\nabla_{x_{s}^{t}}(\lambda_{s}L_{s} + \lambda_{c}L_{c})$  ▷Style transfer only, without adversarial loss   
end for   
 $x^0 = x_0^0 = x_s^{n_s}$ $x_{c} = x_{s}^{n_{s}}$    
for  $t = 0,1,2,\ldots ,n - 1$  do   
 $x_{K}^{t}\sim q(x_{K}^{t}|x^{t})$  ▷Sample  $x_K^t$  from  $q(x_K^t |x^t)$  in each PGD iteration   
for  $i = K - 1,\dots ,0$  do   
 $x_i^t = R_\phi (x_{i + 1}^t)$  ▷Apply denoiser  $R_{\phi}$  to  $x_{i + 1}^t$  in each SDEdit iteration   
 $x_i^t\sim M\circ x_i^t +(1 - M)\circ q(x_i^t |x^t)$  ▷Sample from masked combination of  $x_i^t$  and  $q(x_i^t |x^t)$    
end for   
 $g = \nabla_{x^t}l(f_\theta (M\circ x_0^t +(1 - M)\circ x_c)]$  ▷Compute the gradient   
 $x^{t + 1} = \Pi_{x,\epsilon}[x^t +\eta M\circ \mathrm{sign}(g)]$  ▷PGD update   
end for   
 $x_0^n = \mathrm{rSDEdit}(x^n)$  ▷Apply reverse SDEdit to the final  $x^n$    
return  $x_0^n$  ▷Return style-based adversarial sample
Algorithm 3 Diff-PGD for Physical World Attacks
Require: Target classifier  $f_{\theta}$  , original patch image  $x$  , mask  $M$  , denoised sampler  $R_{\phi}$  # of reverse SDEdit steps  $K$  , iterations  $n$  , stepsize  $\eta$  , physics transformation set  $\mathcal{T}$  , learning rate  $\eta$ $x^0 = x_0^0 = x$ $x_{c} = x$    
for  $t = 0,1,2,\ldots ,n - 1$  do   
 $x_{K}^{t}\sim q(x_{K}^{t}|x^{t})$  ▷ Sample  $x_{K}^{t}$  from  $q(x_{K}^{t}|x^{t})$  in each PGD iteration   
for  $i = K - 1,\dots ,0$  do   
 $x_{i}^{t} = R_{\phi}(x_{i + 1}^{t})$  ▷ Apply denoiser  $R_{\phi}$  to  $x_{i + 1}^{t}$  in each SDEdit iteration   
 $x_{i}^{t}\sim M\circ x_{i}^{t} + (1 - M)\circ q(x_{i}^{t}|x^{t})$  ▷ Sample from masked combination of  $x_{i}^{t}$  and  $q(x_{i}^{t}|x^{t})$    
end for   
 $\tau \sim U(\mathcal{T})$  ▷ Sample physics transformation   
 $g = \nabla_{x^t}l(f_\theta (\tau (M\circ x_0^t +(1 - M)\circ x_c))]$  ▷ Compute the gradient   
 $x^{t + 1} = x^t +\eta M\circ \mathrm{sign}(g)$  ▷ update   
end for   
 $x_0^n = \mathrm{rSDEdit}(x^n)$  ▷ Apply reverse SDEdit to the final  $x^n$    
return  $x_0^n$  ▷ Return Diff-Phys patch
To adapt Diff-PGD to the physical world settings, larger modifications are needed. In Algorithm 3, we present Diff-PGD in physical world environments. The biggest difference is that we no longer restrict the  $\ell_{\infty}$  norm of perturbations. Also, we use gradient descent instead of projected gradient descent. For the generation of the physical adaptor  $\mathcal{T}$ , we consider the scale of the patch image  $x$ , the position of the patch image, the background images, and the brightness of the environment.
C Accelerated Diff-rPGD
Changing only one line can help us get the accelerated version of generalized Diff-PGD with an approximated gradient (Algorithm 4).
Algorithm 4 Accelerated Diff-rPGD
Require: Target classifier  $f_{\theta}$  , original image  $x$  , mask  $M$  , denoiser  $D_{\phi}$  # of reverse SDEdit steps  $K$  iterations  $n$  , stepsize  $\eta$  , clip value  $\epsilon$  (when  $M = 1$  it reduces to Diff-PGD)   
 $x^0 = x_0^0 = x,x_c = x$    
for  $t = 0,1,2,\ldots ,n - 1$  do  $\begin{array}{rl} & {\triangleright \mathrm{Sample}x_K^t\mathrm{from}q(x_K^t |x^t)\mathrm{in~each~PGD~iteration}}\\ & {\triangleright \mathrm{Apply~denoiser}R_\phi \mathrm{to}x_{i + 1}^t\mathrm{in~each~SDEdit~iteratior}}\\ & {\triangleright \mathrm{Sample~from~masked~combination~of}x_i^t\mathrm{and}q(x_i^t |x^t)}\\ & {\triangleright \mathrm{PGD~update}} \end{array}$    
for  $i = K - 1,\dots ,0$  do   
 $x_{i}^{t} = R_{\phi}(x_{i + 1}^{t})$ $x_{i}^{t}\sim M\circ x_{i}^{t} + (1 - M)\circ q(x_{i}^{t}|x^{t})$    
end for   
 $g = \nabla_{x_0^t}l[f_\theta (M\circ x_0^t +(1 - M)\circ x_c)]$ $x^{t + 1} = \Pi_{x,\epsilon}[x^t +\eta M\circ \mathrm{sign}(g)]$    
end for   
 $x_0^n = \mathrm{rSDEdit}(x^n)$    
return  $x^n,x_0^n$
Method	K	n	VRAM(G)	Speed(sec/sample)
Diff-PGD	2	10	~18	8
v2	2	10	~4	4
Diff-PGD	3	10	~20	10
v2	3	10	~4	5
Here, we compared the speed and success rate of accelerated Diff-PGD with the original version; we refer to the accelerated version as v2. The qualitative results of v2 are put in Figure 10, and the quantitative results about speed, VRAM, and success rate are put in Table 3 and Table 4.
Experimental Environments and Inference Speed The methods described in this study are implemented using the PyTorch framework. All the experiments conducted in this research were carried out on a single RTX-A6000 GPU, housed within a Ubuntu 20.04 server.
Since we need to do back-propagation on the U-Net used in the diffusion model, it will cost more GPU memory and GPU time. Running on one single GPU, it takes  $\sim 7$  seconds to run Diff-PGD with  $n = 10$ ,  $K_{s} = 2$  for one sample on one A6000 GPU. Parallel computing with more GPUs can be used to accelerate the speed.
Table 3: The Speed and VRAM of Diff-PGD and accelerated Diff-PGD with approximal gradient (v2). From this we can see that the v2 is much cheaper to run than the original Diff-PGD.  $K$  and  $n$  follows the definition in Algorithm 1.
Method	Model	n	ε	SR
v2	ResNet-50	10	16/255	99.6
v2	ResNet-50	10	8/255	98.8
v2	ResNet-50	15	16/255	100.0
v2	ResNet-50	15	8/255	100.0
Table 4: The success rate of accelerated Diff-PGD, from which we can see the success rate (SR) is still really nearly  $100\%$ , showing that v2 is a cheaper but still effective attack.  $\epsilon$  and  $n$  follows the definition in Algorithm 1.
Global Attacks Here we use  $\epsilon = 16 / 255$ ,  $\eta = 2 / 255$ ,  $n = 10$  as our major settings (except for the ablation study settings) for both PGD and Diff-PGD. For Diff-PGD, we use DDIM50 with  $K_{s} = 3$  as our SDEdit setting; we will also show more results with different DDIM time-steps and different  $K_{s}$ .
Regional Attacks For the experiments of the regional attacks, we randomly select a squared mask sized  $0.4H \times 0.4W$ , where the size of the original image is  $H \times W$ . For the same image in the ImageNet dataset, we use the same mask for both rPGD and Diff-rPGD. Also, we use  $\epsilon = 16/255$ ,  $\eta = 2/255$ ,  $n = 10$  for bounded attack with projected gradient descent. And for Diff-rPGD, we use DDIM50 with  $K_s = 2$ ; similarly, we try more settings in the ablation study section. The repainting strategy of our Diff-rPGD is based on replacement, but more effective strategies are encouraged to be used here.
Style-based Attacks As mentioned in Section B, the implementation of Diff-PGD for style-based attacks can be divided into two parts. For the first stage, we use style weight  $\lambda_{s} = 4000$  and adopt Adam optimizer with learning rate  $\eta = 0.01$  for style transfer. For the feature extractor, we adopt VGG-19 as our backbone with the first five convolutional layers as the style extractor  $H_{s}$  and the fourth convolutional layer as the content extractor  $H_{c}$ . For the target image  $x$ , we use SAM to collect the segmentation mask to define the attack region.
In the second stage, we utilize DDIM10 with a parameter setting of  $K_{s} = 2$  to achieve better purification. It is beneficial to use a smaller  $K_{s}$ , as the style-transferred images often contain unwanted noise and art-styled patterns that can be mitigated through this approach.