Targeted Attack Improves Protection against Unauthorized Diffusion Customization
Authors: Boyang Zheng
Chumeng Liang
Xiaoyu Wu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Protection
Unauthorized Diffusion Customization
Adversarial Attack
Diffusion Model
Privacy
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a stronger image protection method against unauthorized diffusion customization based on targeted attack.

Abstract: Diffusion models build a new milestone for image generation yet raising public concerns, for they can be fine-tuned on unauthorized images for customization. Protection based on adversarial attacks rises to encounter this unauthorized diffusion customization, by adding protective watermarks to images and poisoning diffusion models. However, current protection, leveraging untargeted attacks, does not appear to be effective enough. In this paper, we propose a simple yet effective improvement for the protection against unauthorized diffusion customization by introducing targeted attacks. We show that by carefully selecting the target, targeted attacks significantly outperform untargeted attacks in poisoning diffusion models and degrading the customization image quality. Extensive experiments validate the superiority of our method on two mainstream customization methods of diffusion models, compared to existing protections. To explain the surprising success of targeted attacks, we delve into the mechanism of attack-based protections and propose a hypothesis based on our observation, which enhances the comprehension of attack-based protections. To the best of our knowledge, we are the first to both reveal the vulnerability of diffusion models to targeted attacks and leverage targeted attacks to enhance protection against unauthorized diffusion customization.

Published as a conference paper at ICLR 2025
TARGETED ATTACK IMPROVES PROTECTION AGAINST UNAUTHORIZED DIFFUSION CUSTOMIZATION
Boyang Zheng' Chumeng Liang Xiaoyu Wu'
1 Shanghai Jiao Tong University. 2 University of Southern California
ABSTRACT
Diffusion models build a new milestone for image generation yet raising public concerns, for they can be fine-tuned on unauthorized images for customization. Protection based adversarial attacks rises to encounter this unauthorized dif- fusion customization, by adding protective watermarks to images and poisoning diffusion models. However, current protection, leveraging untargeted attacks, does not appear to be effective enough. In this paper, we propose a simple yet effective improvement for the protection against unauthorized diffusion customization by introducing targeted attacks. We show that by carefully selecting the target, targeted attacks significantly outperform untargeted attacks in poisoning diffusion models and degrading the customization image quality. Extensive experiments validate the superiority of our method on two mainstream customization methods of diffusion models, compared to existing protections. To explain the surprising success of targeted attacks, we delve into the mechanism of attack-based protections and propose hypothesis based on our observation, which enhances the comprehension attack-based protections. To the best of our knowledge, we are the first to both reveal the vulnerability of diffusion models to targeted attacks and leverage targeted attacks to enhance protection against unauthorized diffusion customization. [Warning: This paper contains images that some readers may find disturbing.]
INTRODUCTION
Diffusion Models (Sohl-Dickstedl et al. [2055] Song & Ermon 2019 Ho et al. 2020: Song et al. 2020
Rombach et al., 2022) has achieved state-of-the-art performance in image synthesis. One distinct advantage of diffusion models is the capability of controlling and customizing the image generation
via reference images (Meng et al. 202I). text description (Saharia et al. 2022). sketches (Zhang
& AAgrawala 2023), styles (Ruiz et al. 2023 Zhang et al.) (023), and identities (Ye et al. 2023
Wang et a 2024). However, this power is a double-edged sword, for it also makes possible the unauthorized diffusion customization, where malicious individuals seek interests from customizing diffusion models based on unauthorized images, e.g. copyright photos. This has long been a public
concern that draws attention from the whole society (Fan al. 2023 Chen et al., 20233 Wang et al. 20236). Counterneasures are needed to protect private images from being arbitrarily used for unauthorized diffusion customization.
Recognizing the need, researchers developed protections based on adversarial attacks on diffusion models (Salman al. 2023 Liang et al. 2023 [hhan al. 2023a Van Le al. 2023 Shan et al. 2023b). These protections add tiny adversarial perturbations images. Diffusion models customized on these perturbed images will be poisoned, and the generated images will suffer a degradation in quality. This makes unauthorized customization fail in producing usable images, thus penalizing such attempts. Applications based on these protections (Shan et al. 2023a Liang Wu 2023 2023b have safeguarded private images in practice.
Shan et al.
However, current protections are not strong enough to hold effective when the protection strength is limited, i.c. the adversarial perturbation budget is small, which is practical need to maintain the visual effects of the protected images. Figure [ demonstrates examples of customization images with existing protection. Under small protection strength, existing protections appear to put patterns
"qqual contribution

---

Published as a conference paper at ICLR 2025
Data
Clean
PhotoGuard PhotoGuard+
AdvDM
ASPL
ACE
ACE+
Output images of two mainstream diffusion customization, SDEdit (top two rows) and LoRA (bottom two rows) under different protections with perturbation budget 4/255. ACE and ACE+ are our targeted attack, while others baselines based on untargeted attacks.
Figure I:
only some locations customization output images. The main content of the image remains intact. This raises concerns about the effectiveness of current protections and asks for stronger ones.
In this paper, we propose a stronger protection against unauthorized diffusion customization based on adversarial attacks. We notice that all state-of-the-art protections depend on untargeted attacks diffusion models. We explore adapting the targeted attack for adversarial attacks on classifiers to the context of diffusion models. Specifically, the targeted attack minimizes the distance between model outputs and a pre-defined target. Since this attack creates score- ccore-function errors in a consistent direction, we denote this method by Attacking with Consistent score-function Errors (ACE). Empirically, find that the selection of the target has great impacts on the performance of ACE. With properly selected targets, ACE can produce customization output images all covered by chaotic patterns (see the last two columns of Figure []. Through extensive experiments, we validate that ACE outperforms all baselines in two mainstream diffusion customizations.
In addition, we delve into the mechanism of targeted and untargeted attacks diffusion models. Based on our observation, we propose hypothesis on how attack-based protections work to degrade diffusion customization, that customization diffusion models are degraded because they try to minimize the score-funee ccore-function eтror introduced by attacks. According to this hypothesis, we show that targeted attacks like ACE unifie the pattern of coorrefunction errors in different protected images, thus reinforcing the bad pattern learned by diffusion models. This provides an intuitive explanation why the targeted attack ACE outperforms all untargeted baselines.
The contributions this paper are two-fold. First, we propose ACE, a novel protection method against unauthorized diffusion customization. ACE outperforms existing protections and sets new milestone in poisoning diffusion models and degrading the image quality of diffusion customization. To the best of our knowledge, ACE the first targeted-attack-based protection that outperforms untargeted- mttack-based protections. Second, we propose a hypothesis on how trtack-based protections work based on our observation. This will help future research comprehend and develop even stronger protections against unauthorized diffusion customization.
BACKGROUND
Diffusion Models Diffusion Models (Sohl-Dickstein et al., 2015 Song & Ermon 2019 Ho et al. 2020) are built on forward and reverse process that map the noise and data distribution. The forward process perturbs the data progressively to approximate Gaussian noise. The reverse process exploits parameterized coore-function 8p generate data by denoising noise step-by-step. These
2

---

Published as a conference paper at ICLR 2025
two processes can be interpreted by forward and a reverse SDE (Anderson 1982 Song et al. 2020):
dæ = f(x, d) + g(t)dw
dæ = (x - ( (a) g())du
where w and t are standard Wiener processes when time flows from 0 to T and backwards, and dt are infinitesimal timesteps (Song et al. 2020). Diffusion models are then trained by minimizing the
score-function error, the difference between predicted score-function s0(r, () by neural network and the ground truth score function V a log (z))), since the score function is the only unknown term in Equation| I The training objective is given by:
min E,Ex(t))x(0)) (,())t), t) Va(o p. (=(())))))) 1
9
(2)
ea (x(()))))) scure function
Following all existing research on protections against unauthorized diffusion customization Liang et al., 2023; Liang & Wu 2023; Shan et al., 2023a b Salman et al. 2023 Van Le et al. 2023
Xue et al. 2023), we mainly consider a specific kind of diffusion models, Latent Diffusion Model (LDM) (Rombach et al.) 2022), in the context of unauthorized diffusion customization. LDM deploys its forward and reverse processes on a latent representation space 2 = E(x) given by an encoder (-).
and instantiates f(x t) and g(t) as the linear schedule (Ho et al. 2020
Diffusion Customization Diffusion customization (or diffusion personalization (Zhang et al., 2024)
refers to customizing the generated images by diffusion models according to the references from user inputs. These user inputs cover reference images (Meng et al.] [2021). l). text description (Saharia
et al., 2022), sketches (Zhang & Agrawala, 2023). , styles (Ruiz et al., 2023 Zhang et al. (023).
and identities (Ruiz et 2023; Ye et al. methods, MrRA+DrremBoott (Hu et al. 2021 Ruiz et al. 2023) and SDEdit (Meng et al. 2021
2023 Wang et al. 2024). Among these custom zation
are two mainstream methods that cover most of the customization practice (See our brief survey in
Appendix D.. Hence, follow the setup of existing works (Van Le et al.
2023 Liang al. 2023
Xue et al. 2023) to focus on these two methods.
Adversarial Attacks & Protections against Unauthorized Diffusion Customization Adversarial
attacks on Diffusion Models (Salman et al. [2023] Liang et al. 2023 Shan et al. 2023a Liang & Wu
2023] Van Le et al.) 2023, Zhao et al.) [023]] [uue et al., [023] Liu al. 2024b a) have been used
to protect private images against unauthorized diffusion customization. Specifically, these attacks add tiny adversarial perturbations to private images as protections. Diffusion models customized (fine-tuned (Hu et al., 2021 Ruiz et al. 2023) or referring to (Meng et al. (021)) the protected images will produce images of degraded quality in customized generation. This will penalize the attempt to customize diffusion models unauthorized private images.
is noticeable that mainstream protections are based on untargeted attacks on diffusion models. Those untargeted objectives maximize the score-function error in Equation by optimizing the image perturbation 7) within the perturbation budget Ç. which is formulated by Equation 3
max E2 (1)) so () 2 log pe (= (=(())='(0) 1
(z) scote Functeinn
where z'(0) =(0)=E(x') x' = x + 7. E (-C,s)
Here, x is the clean image and z²( is the latent representation Empirically, this untargeted attack does not yield satisfying effectiveness in degrading the quality of the customization image (see the examples in Figure T). In this paper, we improve the protection effectiveness by introducing targeted attacks in Section and provide qualitative analysis to explain our improvements in Section 5
x' with LDM encoder E--)
3 TARGETED ATTACK AS PROTECTION AGAINST UNAUTHORIZED DIFFUSION CUSTOMIZATION
Our intuition of introducing targeted attack is simple. We notice that there is an objective mismatch between protection against unauthorized diffusion customization and untargeted attacks on diffusion models (Equation 3). Specifically, protection seeks for degrading the image quality of diffusion
3

---

Published as a conference paper at ICLR 2025
customization, while untargeted attack's goal is to minimize the log-likelihood of the protected image and make it not like a real image. However, images with bad quality are not necessary to have a low log- lieelikdidood. Hence, a more straightforward approach degrade the quality of the customization image is to directly make the protected image similar to a bad quality image. When referring to the protected image, diffusion customization will generate images similar that bad-quality image. We implement this approach by introducing a targeted attack on protection (Section 3.1 and use a bad quality image as the target (Section 3.2).
3.1 ATTACKING WITH CONSISTENT ERRORS
We start from comparing classical untargeted and targeted adversarial attacks (Goodfellow et al. 2014 Madry et al. 2017), where both work soundly in attacking classifiers:
max (Ipo(vla') Strue 2
(untargeted attack)
min Ipo (v(2=' Exanget 12
(targeted attack)
Here, Strue and Starget are one-hot expressions of the ground-trute truth label Verue and target label ytarget- We notice that targeted attack simply 1) replace the ground truth in untargeted attack with a target and 2) flip maximization to minimization to reduce the distance between the adversarial label and the target label. Inspired by this comparison, we conduct mirroring transformation on the untargeted attack, given by Equation 3 and transfer it to a targeted attack.
First, we replace the ground truth with target. The ground truth in Equation 3 refers the score function = log (2 (0)). We replace it with pre-defined targer T. Second, we replace maximization with minimization to minimize the distance between the predicted score function (t)(,, t) and the target T. This yields a new targeted attack on diffusion models, shown as follows:
max IE 01 (z' tt),
V log (=((())2
(untargeted attack)
7
minEE,E |o (t), t)
(targeted attack)
For different protected images z', this targeted attack uses a consistent target T as the direction to optimize adversarial perturbations. Hence, we denote this novel targeted attack by Attacking with Consistent Errors (ACE).
min E, (01
TI
(4)
where ='(0) E(x'), = ,, E
Intuitively, ACE misleads the model to predict score function with the pattern of target T for the protected image x'. When customized on z', diffusion models will learn the pattern of T. Therefore, by placing a chaotic pattern in T, we can then make diffusion models that are customized on the protected image learn the chaotic pattern and generate images with this pattern, thus degrading the performance of unauthorized diffusion customization.
In addition ACE, we also consider an existing targeted attack objective introduced Liang & Wu (2023), which is specific designed for LDM (Rombach et al.) 2022). the point-of-interesss diffusion model of unauthorized customization. This term directly minimizes the distance between a target TT and the latent representation of the image x'. Empirically, this objective is useful for encountering image reference diffusion customization such as SDEdit (Meng et al. 202I). We therefore weight this objective by and combine it with ACE, which creates new attack denoted by ACE+:
min E, (o -11)))-1333
- 2 + all='(0) - -
(5)
where z' z0(0) E(x'), = n. E [-5.5]
3.2 TARGET SELECTION
As mentioned in Section 3.1 good protection effectiveness of ACE/ACE+ depends on a carefully- designed target T. This target must be with chaotic patterns to human vision so that unau- thorized diffusion customization will learn these chaotic patterns from the protected image and suffer performance degradation. Our next task to design such targets with chaotic patterns.
4

---

Published as a conference paper at ICLR 2025
Previous research suggests several aspects of a chaotic patterns from the perspective of human vision: 1) high contrast stripes (Wilkins 1995). 2) Moire patterns (Amidror, 2009). and 3) overly busy patterns (Bell 2001). Following these suggestions, we pick the latent representation of the target image used in Liang & Wu (2023) our basic target. This target has a high contrast of black and white and repeats a motif to create overly busy and Moire-like patterns, which fits the standard of chaotic patterns for human vision. Based on the basic target, we tune the contrast and the repeating number of the motif as hyperparameters. See Appendix for details. With the optimal contrast and repeating number, we determine the final target as shown Figure 2
Figure 2: Target T (left) and its corresponding im- age (right)
To cross-validate our insights in target selection, we additionally design a similar target following the above ideas of chaotic patterns. Specifically, we change the motif of the target Figure 2 to another, improve its contrast to black and white, and repeat it in one image. This new target is shown in Figure [0] We denote ACE with this new target as ACE" and evaluate ACE* together with ACE and ACE+ in Section an ablation study.
3.3 IMPLEMENTATION AS PROTECTION AGAINST UNAUTHORIZED CUSTOMIZATION
Our method combines the objective function in Equation 4 5 and the selected target T. We also include fine-tuning steps in our method, following ASPL Van Le et al. (023) (5 steps for each itera- tion). As discussed in its original paper, this is the standard poisoning pipeline for adversarial attacks and brings simple performance gain for adversarial attacks on diffusion models, both untargeted and targeted. For the optimization of ACE/ACE+, we use the classic PGD algorithm (Madry et al. (017). following existing protections. Putting everything together, we finalize our method in Algorithm To use ACE/ACE+ as protection against unauthorized diffusion customization, one can input the clean image a into Algorithm| 1 and take the output image x' as the protected version of æ. Other implementation details are given in Appendix[A)
Note that ACE/ACE+ are the first targeted attack that outperform untargeted attack on diffusion models (see our experiments in Section 4. However, they are not the first targeted attack on diffusion models. While targeted attacks were considered by previous research (e.g. ASPL-T (Van Le et al.
2023 they failed in beating untargeted attacks. We discuss the reason in Appendix B.5.1
Algorithm Attacking with Consistent Errors (ACE)
1: Input: Image v, diffusion model 0, learn- 6:
0 0 0-00~VolLnn VoLL.NM
ing rates o 7, epoch numbers N, .M, K, bud- 7:
end for
get Ç, diffusion training objective in Equa-
8:
for k from 1 to K do
tion 2 ACE/ACE+ objective function J in
9:
x' + x' a J
Equation 4 & Equation 5
10:
x' + clip - C. ()
2: Output: Adversarial example x'
11:
x' + c' (ipp(r, 255)
3: Initialize x' + Z.
12: end for
4: for n from to N do
13: end for
5: for from to M do
14: return x!'
4 EXPERIMENTS
We compare ACE/ACE+/ACE* to existing protections against unauthorized diffusion customization in two main diffusion customization methods, LoRA+DreamBooth (Hu et al.) 2021 Ruiz et al. 2023) and SDEdit MMeng et al. 202T). We also evaluate our protection on DreamBooth and Stable Difrusion 3 (Esser et ai. 20024) (Appendix B.3). Our methods outperform baselines both in quantitative metrics and qualitative visualization (Section 4.2). We conduct user studies in an artist community, which validates the superiority of our methods (Section 4.3) Furthermore, we test the transferability (Sec- tion 4.4p and robustness to purification (Section 4.5 of ACE/ACE+. We investigate the computational cost of our methods and baselines (Appendix B.I.. Our ablation studies focus the effect of the objective function, target selection (Appendix B.S.I). different text prompts (Appendix B.5.2) and perturbation magnitudes (Appendix B.5.3) to the protection effectiveness.
5

---

Published as a conference paper at ICLR 2025
4.1 EXPERIMENTAL SETUPS
Experimental Setups on LoRA+DreamBooth LoRA+Dreanbooth (Hu et al.) 2021; Ruiz et al. 2023) (LoRA for simplicity) is the most widely used customization method for diffusion models, occupying share of more than 90% in the open-source customization platform. LoRA fine-tunes diffusion models with low-ranked adapters on dozens of reference images and generates images with similar contents styles. Due to its power and convenience, LoRA is the main tool for customization of unauthorized diffusion.
In our experiments, we use LoRA to fine-tune the diffusion model on protected reference images. Fine-tuning is done with 20 protected images with the same content or style. We then generate 100 output images with the fine-tuned customization diffusion models and assess image quality with CLIP-IQA (Wang et al. (203aa) (CLIP-IQA). high CLIP-IQA score indicates low image quality and strong protection performance. We also follow (Van Le et al. (203)) to introduce two facial image quality metrics in our experiments on CelebA-HQ: Face Detection Failure Rate (FDFR) (Deng et al.) 2020) and Identity Score Matching (ISM) (Deng al. 2019). A strong protection expects a high FDFR by making the face detection (is there facc) failed and a low ISM by disrupting the facial identification (who is the face).
Experimental Setups on SDEdit SDEdit (Meng et al. method that modifies the content or style of single reference image while keeping the structural similarity, i.c. the layout of the reference image.
2021) is an image-to-image customization
In our experiments, we use SDEdit to generate output images based on protected reference images. The strength of SDEdit is set as 0.3. Successful protections destroy this structural similarity between the output image and its corresponding reference image. Hence, we adopt Multi-Scale SSIM (MS- SSIM) (Wang et al. 2003) and CLIP image-to-Image Similarity (CLIP-SIM) (Wang et al., (023a). two measurements for image-to-image similarity. We include MS-SSIM for structural similarity and CLIP-SIM for semantic similarity. A strong protection is expected to have both metrics low.
Datasets & Backbone Model The experiment is conducted on CelebA-HQ (Karras et al.. 2017 and Wikiart (Saleh & Elgammal, 2015). For CelebA-HQ, we select 100 images and cach of 20 photos describe one identical person. For Wikiart, we select 100 paintings that each of 20 paintings come from the same artist. We use Stable Diffusion 1.5 as the backbone model for protections, since it is the most popular diffusion model in (unauthorizedd) diffusion customization. We also investigate the cross model transferability of ACE/ACE+ with Stable Diffusion 1.4 and Stable Diffusion 1 in Section 4.3 As discussed in Section 2 we do not consider other diffusion models because they are either 1) close-sourced that they do not support customization (e.g. DALLE3) 2) lack of well implemented customization pipeline (c.g. DeepFloyd IF).
Hyperparameters The adversarial budget C is set 4/255. Note that this budget is smaller than those in existing literature, such as 8/ /555 Liang el al. (2023); Van Le et al. (023) and 16/ /2/ Salman al. (2023), for these large budgets will add perceptible noise to the image that hurt the image quality. Hence, we use a smaller budget to simulate the real-world application scenario. The step length and the number of step in PGD (Madry et al. 2017) are 5 10-3 and 50. We adopt LoRA for fine-tuning steps in Algorithm T Other hyper-parameters are omitted to Appendix A
Baselines We compare ACE/ACE+ with existing methods, including AdvDM (Liang et al. 2023), PhotoGuard (Salman et al.) 2023). and ASPL (Anti-DreamBooth) (Van Le et al. 2023). Encoder Attack and Diffusion Attack in PhotoGuard (Salman et al. 2023) are denoted by PG and PG+, respectively. We detail our baseline selection in Appendix A
4.2 OVERALL RESULT
We compare ACE/ACE+/ACE. to existing protections against unauthorized diffusion customization. The overall results of the comparison are given in Table [ where ACE/ACE+/ACE" outperform all the baseline methods in degrading the image quality of SDEdit and LoRA. It is noticeable that ACE achieves a FDFR of 1.00"] This means that 0% of the customization images are recognized as face images with the protection of ACE. This is strong evidence that our protections significantly outperform existing methods. Moreover, ACE" performs similarly or even better than ACE/ACE+,
When FDFR is 1.00, ISM is not defined hecause no face detected in all the output images

---

Published as a conference paper at ICLR 2025
Table Comparison of baseline protections and our protections on LoRA and SDEdit. ACE is our basic method in Equation 4 ACE+ combines ACE and an existing targeted attack (Equation 5 ACE* uses a target other than ACE/ACE+.
CELEBA-HQ
WIKIART
SDEDIT
LORA
SDEDIT
LORA
MS-SSIM
CLIP- SIM & CLIP-IQA FDFR t ISM J MS-SSIM
CLIP-SIM CLIP-IQA
CLEAN
0.88
93.38
20.66
0.02
0.69
0.62
89.77
22.88
PG
0,86
89.24
27.52
0.02
0.72
0.62
88.01
37.52
PG+
0.82
91.00
22.91
0.04
0.71
0.57
89.80
32.62
ADVDM
0.81
83.41
24.53
0.04
0.71
0.30
85.29
34.03
ASPL
0.82
84.12
33.62
0.33
0.48
0.30
87.25
46.74
ACE
0.73
74.70
31.46
1.00
N/A
76.13
40.54
ACE+
0.69
67.47
35.68
0.07
0,58
0.29
76.07
48.53
ACE
0.76
72.52
32.76
0.75
0.47
0.13
73.92
76.50
indicating that our strategy of target selection is successful and robust with the motif in the target varying. Among all baselines, ASPL. has the strongest performance. As mentioned in Section 3.3 this is the credit of introducing fine-tuning in the protection.
In addition to quantitative evaluation, visualize the customization output images under different protection. Figure showeases output images of LoRA and SDEdit under the protection of baselines and ACE/ACE+. We visualize the output images of ACE" Notably, all baseline methods only add chaotic patterns to some parts of the output image. Diffusion customization still succeeds in generating images with correct contents and styles. In contrast, ACE/ACE+ covers the whole output image with high contrast and overly busy patterns, making the image completely unusable. This advantage distinguishes ACE/ACE+ from existing protection against unauthorized diffusion customization. More visualization is given in Appendix
ablation study in Figure 11
4.3
USER STUDY
Protections against unauthorized diffusion customization aim at making the output image unautho- rized customization unusable in graphic art industries. The gold standard for evaluating protection is user studies by graphic artists. Hence, we conduct a survey in an artist community on the comparative effectiveness of two most competitive protections: ASPL (Van Le et al., 2023), and ACE. Basically, our survey asks participants to pick the image with the worst quality out of two customization images under the protection of ASPL, and ACE, respectively. The strongest protection should make the output image of the worst quality. We omit the details to Appendix E. The survey covers 1304 artists, showing that ACE produces the worst customization image among 55% of the examples.
4.4 TRANSFERABILITY
Protections against unauthorized diffusion customization are based on white-box and model- dependent adversarial attacks, Hence, it is important to investigate the transferability of ACE/ACE+ different diffusion models validate their utility. We follow existing literature (Liang al. 2023 Xue et al. 2023) to evaluate this transferability on three text-guided diffusion models: Stable Diffusion 1.4, Stable Diffusion 1.5, and Stable Diffusion 2.1. As mentioned in Section 2 and Sec- tion 4.1 and shown in Appendix E we only consider these SD-family models because they cover most of unauthorized diffusion customization.
In this experiment, backbone means the model used to generate adversarial examples and victim means the model used in few-shot generation. We pick 100 images from CelebA-HO 20 in each of 5 groups, to generate adversarial examples on three backbone models and run SDEdit and LoRA with these adversarial examples on three victim models. The strength of SDEdit is set to 0.4 for visualizing the differences. Other experimental setups stay the same with Section 4.1 Table 2 shows that ACE/ACE+ can degrade diffusion customization even when the victim is different from the backbone. We omit the detailed explanation of the result to Appendix F2
7

---

Published as a conference paper at ICLR 2025
Table Transferability results of ACE (top) and ACE+ (middle) and visualization of ACE's output images (bottom). MS, CS, and CI stand for MS-SSIM, CLIP-SIM, and CLIP-IQA. Our methods maintain effective across different models by bringing different degradation to the output images.
VICTIM
SDL.4
SDL.5
SD2.
SDEDIT
LORA CTT
SDEDIT
LORA CT
SDEDIT
LORA
BACKBONE
MS 4
CS 4
MS 4
CS 4
MS 4
CS 4
CT 1
No ATTACK
0.85
91.71
20.32
0.85
91.16
19.22
0.80
79.00
16.78
SDL.4
0.73
77.24
38.13
0.73
77.58
35.98
0.62
60.82
35.45
SD1.5
0.73
77.29
36.65
0.73
77.50
32.11
0.72
60.10
45.05
SD2.1
0.72
76.20
46.08
0.62
76.80
39.08
0.60
59.12
43.89
VICTIM
SDL.4
SD1.5
SD2.
SDEDIT
LORA CI
SDEDIT
LORA CI
SDEDIT
LORA A
BACKBONE
MS 4
CS 4
MS
CS
MS 4
CS L
CI1
No ATTACK
0.85
91.71
20.32
0.85
91.16
19.22
0.80
79.00
16.78
SDI.4
0.67
66.83
40.69
0.67
66.40
31.53
0.58
56.41
67.96
SDI.S
0.67
66.58
41.16
0.67
66.13
36.05
0.58
57.17
68.50
SD2.1
0.67
66.33
41.80
0.67
57.17
41.96
0.58
57.27
73.59
VICTIM
SD1.4
SDI.5
SD2.1
BACKBONE
SDEDIT
LoRA
SDEDIT
LoRA
SDEDIT
LoRA
No ATTACK
SD1.4
SD1.5
SD2.
4.5 ROBUSTNESS TO PURIFICATION
We investigate how ACE survives common denoising adversarial purification. We follow the setup Liang et al. (023) consider Gaussian (Zantedeschi al. 2017). JPEG (Das et al. 2018). Resizing (Xie et al. (017). SR (Mustafa et al. 2019), and DiffPPure
Nie et al. 2022). The adversarial budget set 87255 while other setups follow Section 4.1 Among these purification methods, Gaussian adds Gaussian noise with variance 4 and 8 to the image. Two JPEG qualities, 20 and 70, are tried. Resizing includes two setups: 2x up-scaling recovering (2x) and 0.5x down-scaling + recovering (0.5x). Other details omitted Appendix B.2
Table 3 posts the CLIP-IQA for LoRA and MS-SSIM for SDEdit for ACE under denoising- purification. Gaussian is the most useful method in purifying ACE, while other methods are disable in countering ACE. Notably, as the state-of of-the-art purfication method, DiffPure tends to remove too many details of the image that the customization output image loses some basic semantics of the reference image. Although our protections could be removed by this powerful purifier, it also greatly degrades the performance of diffusion customization, where achieve goal in different way.
8

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative results and visualization of ACE under different purification methods. Most of purifications degrade the image quality because they also semantic information of the reference image when purifying out protections.
DEFENSE
NA
GAUSSIAN
JPEG
RESIZING
SR
DIFFPURE
PARAM
20
Q
70
2x
0.5x
CLIP-IQA (LORA)
25.51
17.24
28.39
38.13
29.01
26.10
32.91
39.63
49.25
MS-SSIM + (SDEDIT)
0.54
0.70
0.64
0.75
0.75
0.79
0.76
0.45
0.40
DEFENSE
NA
GAUSSIAN
JPEG
RESIZING
SR
DIFFPURE
PARAMS
0=4
0=8
Q = 20
Q = 70
2x
0.5x
LoRA
SDEDIT
Eade
Eadv
Eadv
codo
Cudv
Bap
Bepi
Bspl
Bspl
Image
Image
Figure 3: Comparison between cadv and Bspt of ASPL and ACE. Blue-framed images are protected images that we use compute Endu. Red-framed images clean images that we use to compute Bept- We visualize Eade, Bept E [664x64x4 as images with channels. Complementary colors mean two pixel are reverse each other. There is visible pattern correlation between Cadv and Bapl in ACE.
DISCUSSION: HOW CAN TARGETED ATTACK BEATS UNTARGETED ATTACK?
While our methods empirically beat baselines in protecting images from unauthorized diffusion customization, it is non-trivial to explain our success. In this section, we discuss the process of fine-tuning diffusion models on protected images and provide one view of analysis on how these attack- based protections work and why ACE/ACE+/ACE perform distinguishably better than untargeted baselines on countering LoRA fine-tuning.
First, we introduce the notation of our analysis. We and x' denote the clean and protected image and and for the original diffusion model (without customization) and the diffusion model customized protected image Then, introduce two intermediates that help analyze the mechanism of attack-based protections:
Eade is the score function error of the protected image sf in the original diffusion model 0.

---

Published as a conference paper at ICLR 2025
Attacking
Finetuning
Sampling
Exndun PESLO
Entured modie
1em3ing
Figure 4: Demonstration of three steps in Hypothesis 5.1. First, Attacking step increases Eade of protected images. Second, Finetuning step trains the diffusion model to cads by a bias B Bept whose direction is reversal Cade- Third, customized diffusion models include Bepl in sampling so that their output images appear to have chaotic patterns. This hypothesis explains why Cade and Eept of ACE are reverse each other as shown in Figure
2) Bept is the score- foree-fnntion error of clean image x in the fine-tuned diffusion model
Cadv == E, (x (t). t) V 20(0 log Pi (t
B Bep E, E,Es(c)= (=()),) Va(o) log l=g){(((0)==00))1
Intuitively, Cade reflects the difference between protected images and clean images from the perspec- tive of the original diffusion model. Bud reflects the error made by the customized diffusion model when predicting the score function of clean image.
To explain what happens when fine-tuning diffusion models on protected reference images, we visualize cadv and Bxpt in ASPL and ACE, whose result is shown by Figure 3 Surprisingly, there is obvious reversal relationship between Eady and Bapt in ACE. The cosine similarity between Eadu and Bapt in ACE is -0.3044. This indicates that what customized diffusion models learn from protected reference images to output with a bias which reverses the score-function error of protected reference images. Base this observation, on diffusion models, demonstrated by Figure
present hypothesis on the mechanism of adversarial attacks
Hypothesis 5.1 (Mechanism of attack- base protection on diffusion models (informal)). First, attacks maximize the score-function error of protected images and introduce chaotic patterns to the error. Second, customized diffusion models are fine-tuned to minimize the score-function error of protected images, where reversal bias is learned overcome chaotic patterns in the error. Third, customized diffusion models also include the above reversal bias when generating customization images. This makes the customization output images come with reverse chaotic patterns.
Following this hypothesis, we can give an explanation to the success of ACE: ACE uses a unified target guide the chaotic pattern in all protected images. Therefore, the customized diffusion model learns to minimize one fixed chaotic pattern in score-function eгrоr when fine-tuning on different protected images. This induces the model learn a fixed bias to reverse the fixed chaotic pattern. This bias is then brought to the output image generated by the customized diffusion model, explaining why we see a unified chaotic pattern in the customization output image under ACE.
In contrast, untargeted attacks do not unify the chaotic pattern in different protected images. The customized diffusion model cannot minimize ecore-function errors of different protected images by learning to reverse one chaotic pattern. Hence, we cannot see unified chaotic pattern in the customization output image. Although untargeted attacks could produce stronger chaotic patterns specifically for different protected images, these patterns do not work jointly even conflict each other in the training of customized diffusion model. This is the potential reason that targeted attacks beat untargeted attacks in this special task.
CONCLUSION
In this paper, propose simple yet effective protection method against unauthorized diffusion customization. Based on targeted adversarial attack on diffusion models, we show that our method beats all existing protection through extensive experiments, with sound transferability and robustness to purification. We further analyze the mechanism of attack-based protections, providing a novel view comprehend and explain the mechanism of adversarial attacks on diffusion models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Isaac Amidror. The Theory of the Moiré Phenomenon: Volume I: Periodic Layers, volume 38. Springer Science Business Media, 2009.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their
Applications, 12(3):11-332, 1982.
Paul A Bell. Environmental psychology. Look Inside the Book Program, 2001.
Bochuan Cao, Changjiang Li, Ting Wang, Jinyuan Jia, Bo Li, and Jinghui Chen. Impress: Evaluating the resilience of imperceptible perturbations against unauthorized data usage in liffusion-basse generative ai. Advances Neural Information Processing Systems, 36:10577-10677, 2023.
Chen Chen, Jie Fu, and Lingjuan Lyu. pathway towards responsible ai generated content. arXiv preprint arXiv:2303.012 01125, 2023.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1rXiv:600.006174, 2016.
Nilaksh Das, Madhuri Shanbhogue, Shang-Tss Tse Chen, Fred Hohman, Siwei Li, Li Chen, Michael Kounavis, and Duen Horng Chau. Shield: Fast, practical defense and vaccination for deep learning using jpeg compression. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 196-204, 2018.
Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings the IEER/CVF conference on computer vision and pattern recognition, pp. 5203-5212, 2020.
Patrick Esser, Sumith Kulal, Andreas Blattmann. Rahim Entezari. Jonas Müller, Harry Saini. Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Mingyuan Fan, Cen Chen, Chengyu Wang, and Jun Huang. On the trustworthiness landscape of state-of-the-art generative models: comprehensive survey. arXiv preprint arXiv:2307.16680 2023.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 141Xv:1412..672. 2014.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10190 2017.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https: /titp:://ithhb... com/ facebookresearch/oooooo wtw.eeerrar.eeooeeeeee..... 2022.
Chumeng Liang and Xiaoyu Wu. Mist: Towards improved adversarial examples for diffusion models. arXiv preprint arXiv:2305. 1268,, 2023.
11

---

Published as a conference paper at ICLR 2025
Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, XUE Zhengui. Ruhui Ma, and Haibing Guan. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. 2023.
Hanwen Liu, Zhicheng Sun, and Yadong Mu. Countering personalized text-to-image generation with influence watermarks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12257-12267, 2024a.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.032: 00000000, 2022.
Yixin Liu, Chenrui Fan, Yutong Dai, Xun Chen, Pan Zhou, and Lichao Sun. Metacloak: Pre- venting unauthorized subject-driven text-to-image diffusion-basee synthesis via meta-learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14219-242288 2024b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 706.06683. 2017.
Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 2021.
Aamir Mustafa, Salman H Khan, Munawar Hayat, Jianbing Shen, and Ling Shao. Image super- resolution as a defense against adversarial attacks. IEEE Transactions on Image Processing, 29: 1711-1724, 2019.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint FrXXiv2205.074600 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4195-4205, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aber- man. Dreambooth: Fine tuning exx-to--mage diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
22500-22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494 2022.
Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint
2015.
Hadi Salman, Alaa Khaddaj, Guillaume Leclere, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai-powered image editing. arXiy preprint arXiv:2302.06588, 2023.
Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. Glaze: Protecting artists from style mimicry by toxt-to-image models. arXiv preprint MtXiv:230..04222. 2023a.
12

---

Published as a conference paper at ICLR 2025
Shawn Shan, Wenxin Ding, Josephine Passamanti, Haitao Zheng, and Ben Y Zhao. Prompt-specific poisoning attacks on text-to-image -moag generative models. arXiv preprint Stti::2311.13822.. 2023b.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song. Jascha DicksDiicss,en,, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoe Tran, and Anh Tran. Anti-dreambooth: Protecting users from personalized text-to-image synthesis. arXiv preprint arXiv:2303.1543. 2023.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2555-2563 2023a.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.05519 2024.
Tao Wang, Yushu Zhang, Shuren Qi. Ruoyu Zhao, Zhihua Xia, and Jian Weng. Security and privacy on generative data in aige: A survey. arXiv preprint arXiv:2309.0943: 2023b.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2. pp. 1398-1402. Ieee, 2003.
Arnold J Wilkins. Visual stress, volume 24. OUP Oxford, 1995.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arviv preprint arvi www.:1711.119999 2017.
Haotian Xue, Chumeng Liang, Xiaoyu Wu, and Yongxin Chen. Toward effective protection against diffusion based mimicry through score distillation. arXiv preprint arXiv:2311 1...2231...2832, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.066 06721, 2023.
Valentina Zantedeschi, Maria-Irine Nicolae, and Ambrish Rawat. Efficient defenses against adversar- ial attacks. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp.
39-49, 2017.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. Advances in neural information processing systems, 32, 2019.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.0555 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 586-595 2018
Xulu Zhang, Xiao Yong Wei, Wengyu Zhang, Jinlin Wu, Zhaoxiang Zhang, Zhen Lei, and Qing Li. A survey on personalized content synthesis with diffusion models. arXiv preprint arXiv:2405.053: 2024.
Yuxin Zhang, Nisha Huang, Fan Tang. Haibin Huang, Chongyang Ma. Weiming Dong, and Chang- sheng Xu. Inversion- based style transfer with diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10146-10156, 2023.
13

---

Published as a conference paper at ICLR 2025
Zhengyue Zhao, Jinhao Duan, Xing Hu, Kaidi Xu, Chenan Wang, Rui Zhang, Zidong Du, Qi Guo. and Yunji Chen. Unlearnable examples for diffusion models: Protect data from unauthorized exploitation. arXiv preprint arXiv:2306.0199 009002, 2023.
Zhengyue Zhao, Jinhao Duan, Kaidi Xu, Chenan Wang, Rui Zhang, Zidong Du, Qi Guo, and Xing Hu. Can protective perturbation safeguard personal data from being exploited by stable diffusion? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 24398 294977, 2024.
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION AND EXPERIMENTAL DETAILS
For the evaluation, we finetune LDM with on CelebA-HQ dataset using the prompt a photo of a sks person, which was first used in the paper of ASPL (Van Le et al. 2023). This is because CelebA-HQ consists of portraints of certain people. Similarly, we use the prompt photo of a sks painling on the WikiArt dataset, because Wik consists of paintings by certain artists. The number of finetuning epochs is set to be 1000 which ensures LoRA on clean images achieves considerate performance.
SDEdit The strength of SDEdit is set to be 0.3, which makes sense because higher strength modifies the input image too much and a lower one keeps too many details of the input image. We use a null prompt to avoid the effect of prompts in the generation.
Metrics For MS-SSIM, we use the default setting in the implementation F the cosine similarity between the input images and the output images in the semantic space of CLIP (Radford et al. 2021) and is given by the following definition:
CLIP-SIM computes
CLIP-SIM(X, Y) = 100x cos(E (X), Ectip(())
(6)
where Edip is the vision encoder of the CLIP Radford et al. (2021) model. CLIP-IQA is a non- reference image quality assessment metric that computes the text-image similarity between the image and some positive & negative prompts. In the official implementation | such as Good image, Bad image, and Sharp image. An image with high quality is expected to have high text-image similarity with positive prompts and low text-image similarity with negative prompts. In our experiments, we use the positive prompt A good photo of a person and the negative prompt A bad photo of person for CelebA-HQ and the positive prompt A good photo of painting and the negative prompt A had photo of a painting for WikiArt. Since we want to measure how poor the output image quality is, prompt. A strong adversarial attack results in low quality of output images and high similarity between output images and the negative prompt.
the author exploits prompts
the text-image similarity between output images and the negative
The standard resolution for SD1.x is 512, while the one for SD2.x is 768. For cross-model transferability experiments, we set the resolution of every model to 512, disregarding that the standard resolution of SD2.1 is 768. The reason for this uniform resolution is to avoid the resizing, which may introduce distortion to the attacks. However, as LoRA on SD2.1 naturally generate image of resolution 768, we still test LoRA performance on SD2.1 on resolution 768.
Image Resolution
Baseline Selection Our works focuses on improving the protection effectiveness against unauthorized diffusion customization. Hence, we include baseline protection methods that 1) focus on effectiveness and 2) has open-sourced implementation or guidelines for reproduction. Also, to align with most baselines and make sure the comparison is fair, our evaluation is conducted on the same adversarial budget 3) under 12-constraint. Based on these three conditions, we select our baselines (Salman et al.) 2023; Liang al. 2023 Van Le et al.) 2023). We notice that there are more existing works that contribute to the protection. However, some of them do not put their main contribution on improving effectiveness. For example, SDS (Xue et al. 2023) focuses on improving computational efficiency and MetaCloak (Liu et al., 2024b) focuses on robustness. Also, some existing works are lack of official open-sourced implementation (Liu al. 2024a), while some others do not use 12-constraint (Shan el al., 2023a). Therefore, we do not include them in our baselines.
Baseline Implementation We use the official implementation of PhotoGuard PhotoGuard+ A h and ASPL in our experiments. For PhotoGuard and PhotoGuard+, we follow the default setup in the official Python implementation file S except for tuning the adversarial budget to 4/255. We also set the surrogate model of PhotoGuard to be Stable Diffusion v1.5. PhotoGuard+ aims to
"http://gituu..oo//iii projee////tttttititssttt.
simple attack inpainting. wtt.eayraoonooooooooooooooooaonm complex attack inpainting.ipym
15

---

Published as a conference paper at ICLR 2025
attack impainting task, and simply set the impainting mask to for all pixels, as both SDEdit and LoRA operates on full pixels. For AdvDM, we also follow the default setup in the official implementation. For ASPL, L. The default setup sets 10 steps of training LoRA and 10 steps of PGD attacks in every epoch. However, the default epochs of ASPL is time-consuming. Therefore, we tune the total epochs of one single attack to be 4, which is fair comparison for our method.
directly use the official implementation except for transferring it to
Hyperparameters & Implementation Details For ACE+, the loss weight a is to be 102 empiri- cally. LoRA is done for 5 iterations while each iteration finetunes 10 steps. The learning rate is 10-5. To keep consistent to the real application, we use bfloat 16 accuracy for ACE/ACE+, All experiments except PhotoGuard+ are done one NVIDIA RTX 4090 GPU. The implementation of PhotoGuard+ done on one NVIDIA RTX 110 GPU.
Visualization of Variables in the Latent Space of LDM We describe how to visualize variables
in the latent space of LDM, including Cades B2, and Bepi- These variables are of size 64 64
Given a variable F T we first do normalization by E' mem(E)-mmnl After the normalization, we directly plot E' heatmap. The used color bar demonstrate in Figure
E-min(E) (E)
B ADDITIONAL EXPERIMENTS
B.1 COMPUTATION COST
Computation cost is crucial to the utility of adversarial attacks on LDM for. First, they designed for the sake of private image protection involved in the daily routine of human. This asks for the time efficiency of the attack. Second, the main body of the users non-developers who do not have access to GPUs with large VRAM. Hence, the attack needs to be memory-efficient.
B.1.1 ANALYSIS OF THE COMPUTATION COST IN ACE/ACE+
Figure 5: An ex- ample for visualiza- tion. We use the same color bar in all visualizations.
Time Cost As demonstrated in Algorithm ACE/ACE+ consists of two com- puting processes: parameter updating (model training, step 5-7 in Algorithm T and input updating (PGD, step 8- 1212 in Algorithm [ As stated in Appendix A set the iteration number to be the parameter updating step number for each iteration to be 10, and the input updating step number to be 50 for each
image. Note that the number of image is 20 in our experimental setup. Therefore, we run x 50 steps of parameter updating and 4 x 50 x 20 = 4000 steps of input updating. The step number ratio 1:80. Hence, although one single parameter updating step may consume more time cost than one input updating T the main proportion of time cost is still cost by input updating. This conclusion is validated by experiments in the following.
Memory Cost The memory cost of ACE/ACE+ consists of three occupies, which store model weights, the computational graph, and the optimizer states, respectively. In the adversarial attack, we only optimize the inputs so the memory used to store optimizer states is small. We save memory cost mainly by decreasing the memory to store model weights and computational graph. We introduce two optimizations for reducing memory cost, whose efficiency is validated by experiments in the following.
1.1.2 COMPUTATION COST OPTIMIZATIONS ACE/ACE+
To reduce and balance the computation cost of ACE/ACE+, we introduce the following two optimiza- tions:
xFormers We leverage xFormers (Lefaudeux et al. 2022 to reduce the memory cost used to store the computational graph. xFormers is a toolbox that provides memory-efficient computation operators
10 As advised by Zhang et al. 2019). the time cost bottleneck is the back- propagation operation, which is included by both parameter updating steps and input updating steps. Hence, their single step time cost should be similar.
16

---

Published as a conference paper at ICLR 2025
for training and inference of transformer-basee modules. We use their attention operator in the computation of cross attention layers of UNet in LDM when computing gradients.
Gradient Checkpointing Chen al. (2016) A common tool of memory- training is Gra- dient Checkpointing, Gradient Checkpointing separates a neural network into blocks. In forward- propagation, it only stores the activation. The back pack-propagation is done block by block. For each block, it reconstructs the forward computational graph within the block with the stored activation. Then, it constructs the backward computational graph within the block and compute the gradient over the activation. This greatly reduces the GPU memory at the cost of computing time. To balance the memory and time cost, we only apply gradient checkpointing in the down-block, mid-block, and up- up-block of the UNet.
B.1.3 EVALUATION: MEMORY COST
Table 4 shows the computation cost of ACE and existing attacks. ACE enjoys the lowest memory cost, 5.77 GB, compared to baselines. For the time cost, ACE takes 70.49 seconds to process one image, which is acceptable in real-world private image protection. We evaluate the memory cost
Table 4: Computation cost of our method and baselines
Method
Memory/GB
Time/(secondd/image)
PhotoGuard 6.16
21.06
PhotoGuard+
16.79
878.43
AdvDM
6.28
13.45
ASPL ACE&ACE+ 5.77
7.33
55.63
70.49
of baselines and that of ACE/ACE+. We apply baselines and ACE/ACE+ on group of 20 images from CelebA-HQ and record the maximum memory occupation of the runtime. To better simulate the real-world scenario of artwork protection, this experiment is run on an NVIDIA RTX 3 which is an off-the-shelf consumer-level GPU. Other setups stay the same as the setup in Section 4.1
Table 4 shows that the memory cost of ACE/ACE+ is 5.77 GB, lower than those of all baselines. We highlight that our GPU memory cost is lower than 6GB. This means that ACE/ACE+ is able to run on most of the consumer-level GPUs, which has not been achieved by any existing methods. This helps popularize the application of adversarial attacks on LDM as a practical tool for artwork protection. Additionally, we check the memory occupation of two kinds of updating steps. Parameter updating step takes 4.60 GB and input updating step 5.77 GB. This result proves that input updating is the bottleneck for GPU memory cost in ACE/ACE+.
B.1.4 EVALUATION: TIME COST
We evaluate the time cost of baselines and that of ACE/ACE+. We apply baselines and ACE/ACE+ on a group of 20 images from CelebA-HQ and record the time cost of the runtime. To better simulate the real-world scenario of artwork protection, this experiment is run an NVIDIA RTX 3080Ti, which is an off-the-shelf consumer-level GP 12] We rerun the experiment for three times to reduce the effect of randomness. Other setups stay the same the setup in Section 4.1
Table shows that the time cost to process one image of ACE/ACE+ is 88. seconds, 27% longer than that of ASPL (Van Le al. 2023), our strongest baseline. Considering that one human artist does not need to process dozens of artworks normally, the time cost of ACE/ACE+ acceptable in real-world application.
We also investigate the time cost proportion between two kinds of updating steps. We rerun ACE for three times following the above setup and record the total time cost to process 20 images of
PhotoGuurd+ exception hecause it requires over 16 GB GPU memory. Hence, it NVIDIA A100 80GB.
Siimilarr to the case of the memory cost experiment, PhotoGuard+ is evaluated on A 80GB. Since PhotoGuard+ is very inefficient, we only post its computation cost reference.
17

---

Published as a conference paper at ICLR 2025
all parameter updating steps and input updating steps separately. As shown in Table parameter updating steps only take 1.46% of the whole time cost, while input updating steps take the rest 98.54%. This result validates that imput updating takes the larger proportion of time cost.
Table Time cost of parameter updating and input updating in ACE (processing 20 images).
Parameter Updating/seconds
Input Updating/seconds
Total/seconds
Run 1
20.61
1383.85
1405.46
Run 2
21 .24
1396.41
1417.65
19.92 Proportion 1.46%
Run
1386.30
1406.21
Average
20.59
1389.18
1409.78
98.54%
100%
B.2 ROBUSTNESS TO PURIFICATION METHODS
As discussed in Section 4.5 we conduct an experiment to investigate how ACE survives purification methods. We consider common Senoising-basee purification methods, including Gaussian Zant edeschi et al.) 2017). JPEG (Das et al.) 20188) Resizing (Xie et al.] (2017), SR (Mustafa et al., 2019). following the setup in existing works (Liang al. 2023). This is because other purification methods are either not compatible or not available in the context of LDM.
We these purification methods to denoise the adversarial perturbations the protected images. After that, we apply SDEdit and LoRA on these purified images. The experimental setup follows Section 4.] except for the adversarial perturbation constraint, which set to be 8/255. This still small constraint compared to the setup of existing protections (For comparison, Salman et al. (2023) and Van Le et al. (2023) use 16/255). The purification methods include Gaussian (Zantedeschi et al., 2017), JPEG (Das et al. 2018 Resizing (Xie et al. 2017). SR (Mustafa et al. 2019). Specifically, Gaussian adds Gaussian noise of standard variance and to protected image. We try two JPEG compression qualities, 20 and 70. For Resizing we have two setups, 2x up-scaling recovering (denoted by 2x) and 0.5x down-scaling + recovering (denoted by 0.5x). The interpolation algorithm in resizing is bicubic.
The quantitative result is demonstrated Table shows that ACE and ACE+ still have strong impact on the output image quality of few-shot generation after processing by purification. The CLIP-IQA score of most cases even increase after the purification. Hence, we believe that our adversarial attack has enough robustness to purification methods.
Qualitatively, Table also visualizes the output images of SDEdit and LoRA referring to the protected images processed by different purification methods. For both method, they still have strong performance under most of the cases except for Gaussian Noise(a 8) and JPEG compression (quality= 20). However, in the exception cases, the defense has added visible degradation to the image, which also heavily affect both LoRA and SDEdit process. For example, LoRA learns to produce images comprised of small squares due a hard compression of quality 20. And SDEdit produces images of visible Gaussian noise when adding Gaussian noise of a = 8, It's noteworthy that both ACE and ACE+ seems to be strengthened after SR purification, which is an intriguing phenomena.
B.3 PERFORMANCE ON DREAMBOOTH
Although LoRA+DreamBooth is the most widely-used customization method, we also follow ASPL (Van Le et al., 2023) to conduct experimente on DreamBooth, as robustness evaluation of our method. We compare ACE with ASPL, the strongest baseline, DreamBooth. Except for replacing the finetuning method by DreamBooth, all experimental setups stay the same as Section 4.1 and Appendix A We visualize the output images of DreamBooth under ACE and ASPL in Figure 6 respectively. ASPL mainly hurts the diversity of output images while ACE severely degrades the image quality and makes the output images unusable by adding perceptible chaotic texture.
18

---

Published as a conference paper at ICLR 2025
ton
Figure 6: Comparison of ACE and ASPL on DreamBooth. The top three rows: output images of DreamBooth under ACE. The bottem three rows: output images of DreamBooth under ASPL. The adversarial budget is Ç = 4/255. ASPL mainly hurts the diversity of output images while ACE directly degrades the image quality.
19

---

Published as a conference paper at ICLR 2025
B.4 PERFORMANCE ON STABLE DIFFUSION 3
To investigate whether ACE works on diffusion models based on Diffusion Transformers (DiTs) (Pee- bles & Xie 2023) and Rectified Flow (Liu al. 2022), evaluate ACE Stable Diffusion 3 (Esser et al.] 2024). one state-of-the-ar art diffusion models for text-to-image generation. We follow our default setup on LoRA-DreamBooth, except for: 1) the image resolution 1024 as the default of Stable Diffusion 3, 2) the adversarial budget is set to 8/255, and 3) the training steps of LoRA are adapted 2000 for training convergence. We train LoRA on both the clean and the protected subsets of CelebA-HQ. Then, we calculate the CLIP-IQA for both groups.
The CLIP-IQA of LoRA on the clean subset is 0.2650, while that of LoRA on protected subset is 0.4963. This validates that ACE is effective Stable Diffusion 3. ACE is the first protection method that proves work on Stable Diffusion 3 or other DiT-based diffusion models. Notably, ACE seems to work on Stable Diffusion in mechanism different from that on early versions of Stable Diffusion. Instead of degrading the quality output images, ACE deviates the semantic of output images. For example, output images with LoRA trained on portraits of person A will be portraits other irrelevant persons. We visualize this phenomena in Figure 7 This means that one attack could perform in different ways diffusion models with different architectures, which is brand new finding about adversarial attacks on diffusion models. We leave its explanation to future work.
Clean
ACE
Clean
ACE
Figure 7: Comparison of output images from LoRA trained on clean images and protected images by ACE on Stable Diffusion 3. On Stable Diffusion 3, LoRA under ACE will deviate the semanties in the training images and generate irrelevant images, for example, generating portraits different from the portraits in the training images.
B.5 ABLATION STUDIES
B.5.1 TARGETS AND OBJECTIVES
We conduct ablation studies on the effect of targets and objectives. For targets, we use two natural images, one human-face photo from CelebA-HQ and one artwork from WikiArt, alternative images for the target T. This recommended by Van Le et al. (202)). For objectives, we use ASPL-T as the alternative objective. The experiment setup follows the LoRA setup in Section 4.1
Table T demonstrates the sampling outputs under different targets and objectives. Under the small adversarial budget Ç = 4/255, ASPL-T fails in degrading the quality of sampling outputs with all targets. By contrast, ACE adds obvious texture to the sampling outputs with different targets. Among them, our target shows distinguishing superiority in adding very chaotic texture to the sampling output. The result validates that both the objective of ACE in Section 3.1 and the target T selected in Section 3.2 contribute to the performance of our method.
Second, we investigate the impact of two vision variables in the target image T: pattern density and contrast. The setup follows that in the first experiment above. The result is visualized in Figure g
20

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study on the effect of different targets and objectives.
TYPE
HUMAN FACE ARTWORK OUR TARGET
TARGET IMAGE
ASPL-T
ACE
When the pattern is sparse, increasing pattern density yields better attacking performance. When the pattern is dense, increasing pattern density then leads to worse performance. Hence, in practice, we choose an appropriate pattern density. As for contrast, the performance of the attack increases as the pattern's s contrast goes higher. We then let our target be with the highest contrast.
Targer
Image
Output Image (LoRA)
Figure Target images with different pattern repetition and contrast results in different effects.
B.5.2 PROMPTS IN LORA
In real life usage, malicious ones may use different prompts to finetune the LDM. Hence, we investigate whether ACE maintains effectiveness under different prompts in the finetuning. We select one 20-image group from CelebA-HQ and use ACE to generate protected images under adversarial budget 4/255 and prompt a photo of a sks person. We then train LoRA with four different prompts:
photo of sks person
photo of pkp
photo of pkp woman
image of pkp woman
We visualize the output images in Table 7 The result shows minor degradation in the effectiveness of attack when ACE handles more and more unfamiliar prompts. However, there still exists strong visual distortion under all three unknown prompts.
B.5.3 PERTURBATION MAGNITUDE
One interesting question is how the degradation of LDM''s performance related to the magnitude of adversarial perturbations. We conduct an experiment to provide quantitative answer. We tune the adversarial budget ( to be (2, 4, 6, 8, 12} and keep the other setup consistent as that in Section 4.1 In addition to the budget, also post PSNR measure the magnitude of adversarial perturbation. For the degradation of LDM's performance, we post CLIP-IQA for LoRA and MS-SSIM CLIP-SIM for SDEdit.
Figure 9 demonstrates the trend of performance degradation with the increase of magnitudes of adversarial perturbations. For SDEdit, the performance degradation always grow together with the
21

---

Published as a conference paper at ICLR 2025
Table 7: LoRA output images with different prompis under the attack of ACE, Red text notes the difference between the prompt in generation and the prompt in finetuning.
PHOTO OF A SKS PERSON
A PHOTO OF PKP PERSON
PHOTO PKP WOMAN
AN IMAGE OF PKP WOMAN
perturbation magnitude, because SDEdit directly edits the protected image. For LoRA, performance degradation continues to grow when the magnitude increases from 2 to 8. When the magnitude grows from 8 to 14, however, the performance degradation holds still. This indicates that approximately threshold for the performance degradation in LoRA.
Nevertheless, the main concern of increasing adversarial budgets is the visibility of adversarial perturbations. According to our user survey among human artists, Ç - 4 is an appropriate budget. This is the reason why we conduct experiments on 4
PSNR
CLIP-OA (LoRA)
CHIP-SIM (SDFdit)
MS-55IM [SDFdit)
47.5
43.9
CUI CUPIOA (LORA)
CLIP SIM (SDEd))
MS SSIM (SDSdit)
0.00
42.5
15
0.75
0./0
0.05
35.0
523
Figure 9: On the relation between perturbation magnitudes and degradation performance. The left-most figure depicts the PSNR between the protected images and the original images. This could be seen as another metric for adversarial perturbation budgets.
B.5.4 ACE WITH A NEW TARGET
From Appendix I we know that good target for ACE should be with high contrast and repeated patterns. To further validate this insight, we design from scratch a new target (See Figure TO) and conduct full sets of our experiments accordingly. We pick the logo of NeurlPS and repeat it by 2 x in an black image. We denote ACE with the new target by *CE*, All other setups follow Section
NeurlPSNeurIPS
NeurlPSNeurIPS
NeurlPSNeurIPS
NeurlPSNeurIPS
NeurlPSNeurIPS
NeurlPSNeurIPS
NeurlPSNeurIPS NeurlPSNeurIPS
We compare ACE* with the strongest baseline ASPI Van Le et al. 2023 and our ACE/ACE+ in Table [ with visualization given by Figure 11 *CE* performs similarly or even better than baselines. This shows that ACE can yield even better performance by carefully picking the target following insights, providing stronger evidence for our target selection.
Figure 10: New tar- get used by ACB*,
C LIMITATIONS & SOCIETAL IMPACTS
Limitations First, due to the quick progress of the few-show generation empowered by LDM, ACE may need updates cover the novel artwork copying pipelines in the future. Second, this paper focuses on explaining the mechanism of adversarial attacks on LDM empirically, which leads to more powerful method. Hence, do not provide rigorous theoretical proofs for explanation
22

---

Published as a conference paper at ICLR 2025
Figure 11: Visualization of LoRA (top) and SDEdit (bottom) output images under ACE
but focus on validating it with empirical observation. A rigorous theoretical framework for this mechanism is then left be explored in the future work.
In addition, we follow current works to use 12-distance to constrain our adversarial perturbations, which makes comparisons fair. This design, however, could be vulnerable some new advanced purification methods, for example, IMPRESS (Cao et al. 2023) and GrIDPure (Zhao et al. (024). We believe this could be overcome by rectifying the constraint and designing patterns that hide protection deeper in the original image. However, this is out of the scope of this paper which focuses on optimizing the attack objective. We believe our stronger objective can contribute to real-world protection in combination with potential robust constraint and pattern, where we leave to the future work. Also, our robustness experiments show that our method has robustness against general purification, keeping its essential utility. Hence, leave the improvement of the robustness against advanced purification to the future work.
Societal Impacts This paper presents work whose goal is to prevent the unethical abuse of the new generation AIGC This is a crucial topic because of the legislative lag and the rapid progress of machine learning. The recent world has witnessed various cases concerning the copyright, privacy, and ethical problems of AIGC, most of which have not been solved by either technology or the law: Our work aims providing stronger protection against unauthorized image copying and editing, which is only a small part of these problems. We thus believe that our work is completely ethical and sincerely hope that more attention will paid to this field.
D KEY DIFFERENCES BETWEEN OUR METHOD AND OTHER TARGETED ATTACK
ACE/ACE+ are not the first targeted attack on LDM. ASPL-T is general targeted form of at- tacks (Van Le et 2023). Here, we compare the objective function of ACE and ASPL-T
ACE: min E,E. (z) T
ASPL-T: mar Ee = (o)))
1) =T 1)
ASPL
1
where =b(t 1)
-I- (z'(1) + Bisa(=(t),))))))) + VBo., O No(o2;0,1
and =7
B.T 11 a
(7)
ASPL-T adds targeted guidance ASPL. It is intuitive that it pushes the predicted previous latent representation =' (t 1) to the ground truth previous latent representation 2 (t-1) of target T. This sub-objective jointly optimized with the original objective of ASPL, which maximizes the training loss of LDM. However, according to the result posted in its original paper (Van Le et al.) 2023 ASPL-T is deterior to ASPL in performance.
Compared to the intuitive approach of introducing the target, ACE considers the target directly as the only guidance for the predicted score function (t(2((t), t). It is non-trivial because does not correspond to an intuitive explanation within the framework of score-based generative modeling,
The original paper does not provide the concrete objective function of ASPL-T so we refer to its imple- mentation on GitHub: https: thub. com/ v ni Resea racssarrcc/ae Ant DD hamboothh/b ob/main attacks/aspl. .py
23

---

Published as a conference paper at ICLR 2025
unlike ASPL-T. We only design it with acknowledging that all attacks work by biasing the predicted score function (t), t).
Glaze (Shan et al. 2023a is another well-known targeted attack. Two core designs of our method both different from Glaze:
a) objective ACE minimizes the distance between diffusion predicted latent noise and fixed latent. We have both the VAE & and the UNet 9 (or Transformers in DiT architecture) involved. By contrast, Glaze minimizes the feature distance between the protected image and the target image in the VAE representation space. Only the VAE is used in this process.
ACE: min (0)
- T
Glaze: min ||2((0) 2T (0)1
(8)
where (0) = (xx'), ( || z11 <
Here, || x1 is some distance metric and instantiated LPIPS (Zhang et al. 2018 in the
implementation.
b) target ACE exploits the latent chaotic-patter image øт as the target. Glaze uses the style-transfered version of the original image as the target image.
Glaze:
2y == (xr, T)
where Ω is style transfer pipeline of Stable Diffusion
(9)
and T is the target style other than the original style of the image i
E Toy SURVEY ON DIFFUSION CUSTOMIZATION
User Study To further compare ACE with the previous SOTA method ASPL and gain more feedback from the artist community, conduct survey the effectiveness of ACE and ASPL. We choose ten 20-image subsets from our dataset and generate 100 customized images using LoRA-DreamBont the same as
for
each subset, under the protection of ACE and ASPL respectively. All hyperparameters mentioned in A To test ACE's effectiveness against ASPL, we randomly sample one ACE-protected image and one ASPte-poteetedd image for comparison. The artists asked to vote for the lower quality one. We collect a total of 1451 valid votes and ACE has win rate of 55.48%. This further showcases ACE's effectiveness real-world setting. To relieve the artists" workload, do not include more baseline methods in the survey.
Survey on Diffusion Customization Methods Additionally, we investigate highest-rated models in CivitAI [4 popular platform for sharing diffusion customization models. Among them, 78% are LoRA models, which demonstrate the dominance LoRRA-DreamBotth in current diffusion customization applications.
F VISUALIZATION
In this section, we visualize the comparison result between our proposed methods, ACE and ACE+, and bascline methods.
F.1 PROTECTED IMAGES
Figure 12 and Figure 1 demonstrate protected images generated by adversarial attack and ASPL, respectively. Protected images by both methods under adversarial budget = 4/255 show few differences between real images. However, the adversarial perturbations on ACE's protected images appear to be more smooth than those on ASPL's. This makes ACE more acceptable in real-world artwork protection.
#haps://ciittai.commmmeeee
24

---

Published as a conference paper at ICLR 2025
GOID
D
ARE
On
RD
Figure 12: Protected images by ACE with the adversarial perturbation budget = 4/255. The perturbation is quite small and almost human-invisible, making the protected images resemble real images.
F.2 TRANSFERABILITY
We visualize the output images of different victim models under ACE by different backbone models Table As shown in the table, our method shows strong consistency among different models. An exception that when using S the victim model, it tends fail in LoRA training(not learning the right person) instead of learning the strong semantic distortion from the target image. However, the model does learn the right person when no attack conducted. Also, the SDEdit process is extra strong when victim is S.2.1. We attribute this phenomena to the resolution mismatch. SD2. is trained receive images of resolution 768, while we actually fed with images of resolution 512. This may leads to different behaviour of SD2. 1. Also, the resolution mismatch between SDI.x and SD2.x may be the main reason for the performance degradation when using SD1.x as victim and SD2.1 as backbone.
F.3 OUTPUT IMAGES OF SDEDIT
Figure visualizes the output images of SDEdit under different adversarial attacks. All adversarial attacks are budgeted by = 4/255. Two proposed methods add obvious noise to the output image, compared no attack and three baseline methods, Photoguard, AdyDM, and ASPL. Note that the texture in the output images ACE/ACE+ shows consistency with the pattern demonstrated in Figure
25

---

Published as a conference paper at ICLR 2025
GRID
D
ARE
On
RD
Figure 13: Protected images by ASPL with the adversarial perturbation budget ( = 4/255. Their adversarial perturbations are more visible than those by ACE.
F.4 OUTPUT IMAGES OF LORA
Figure 15 16
and 18 show the output images of LoRA under different adversarial attacks. All adversarial attacks are budgeted by Ç < 4/255. Note that the texture in the output images of ACE/ACE+ shows consistency with the pattern of the sampling bias Ban demonstrated in Figure 3 is naturally because the texture is caused by the sampling error, which is accumulated by B.pi-
26

---

Published as a conference paper at ICLR 2025
Input
Clean
PhotoGuard PhotoGuard+
AdvDM
ASPL
ACE
ACE+
ud
ic
Cl
C
C
C
Figure 14: Output images of SDEdit under different adversarial attacks. With same perturbation budget, our attacks better interfere the image quality compared to three baseline methods. Specifically, ACE adds chaotic texture to the image. ACE erases some contents of the image.
27

---

Published as a conference paper at ICLR 2025
Clean
PhotoGuard
PhotoGuard+
AdvDM
ASPL
ACE
ACE+
Ja
PROS
V
R
BI
ATH
Se
C
MIS
RT
Figure 15: Output images of LoRA under different adversarial attacks. Two proposed methods outperform baseline methods.
28

---

Published as a conference paper at ICLR 2025
Clean
PhotoGuard
PhotoGuard+
AdvDM
ASPL
ACE
ACE+
ONO
NK
NA
SEVAIN
Figure 16: Output images of LoRA under different adversarial attacks. ACE and ACE+ outperform
baseline methods. (Cond.)
29

---

Published as a conference paper at ICLR 2025
Clean
PhotoGuard
PhotoGuard+
AdvDM
ASPL
ACE
ACE+
Figure 17: Output images of LoRA under different adversarial attacks. ACE and ACE+ outperform baseline methods. (Cond.)
30

---

Published as a conference paper at ICLR 2025
Clean
PhotoGuard
PhotoGuard*
AdvDM
ASPL
ACE
ACE+
Figure 18: Output images of LoRA under different adversarial attacks ACE and ACE+ outperform baseline methods. (Cond.)
31


Reviews:
[{"review_id": "YmwwQy59rU", "reviewer": "Reviewer_YYfv", "summary": "This paper proposes a targeted attack method for the protection against unauthorized diffusion customization. Extensive experiments validate the effectiveness of the targeted attack compared to other baselines. The paper also proposes an explanation for the effectiveness of targeted attacks.", "questions": "Please see the Weaknesses part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper first proposes the targeted attacks for the protection against diffusion customization, and validates the effectiveness of targeted attacks.\n\n2. The paper proposes an explanation for the superiority of targeted attacks, which may help understand attack-based protection.\n\n3. The experiments on transferability and robustness also validate the effectiveness.", "weaknesses": "There are two main concerns.\n\n1. Did the authors try some specifically purification methods for the protection? e.g., the method in [1]. Can the method purify the perturbations?\n\n2. The authors may need to compare the added loss term alone in the ACE+ loss with the proposed ACE loss. The added loss in ACE+ is also targeted attack. Experimental comparisons are needed between it and the ACE loss.\n\n\n\n[1] Bochuan Cao et al. IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jukmEjtCKZ", "reviewer": "Reviewer_3gvL", "summary": "This paper proposes using targeted adversarial attacks to prevent unauthorized customized fine-tuning of LDM. The motivation is straightforward with a clear method design. The proposed ACE and ACE+ demonstrate a significant reduction in generation quality compared to baseline methods while maintaining an acceptable computational cost. The authors also conducted experiments to assess the robustness and transferability of ACE.", "questions": "My questions have been listed above. I'm willing to increase my score if my concerns are addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is a well-written paper with solid experiments and analysis. The proposed ACE and ACE+ demonstrate superior performance against baseline methods.\n2. The proposed method is straightforward and effective, with clear explanations for each step.", "weaknesses": "Given the detailed and solid experiments presented in this paper, I have no queries regarding the need for more ablation experiments. However, I do have concerns regarding the technical contributions and practical settings.\n\n1. While this paper employs targeted adversarial attacks to prevent unauthorized customized fine-tuning, it is noted that this approach resembles Glaze [1], which also utilizes targeted style transfer to safeguard against style mimicry. Can you discuss the core difference between your technical contributions?\n\n2. It is intriguing to investigate whether we should adopt ACE to all the protected images to achieve such protection results. What would be the impact on protection performance if the attacked ratio (the proportion of protected images) is reduced?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVjbOIvZv8", "reviewer": "Reviewer_p5eN", "summary": "This paper discusses the use of targeted adversarial attacks to improve protection against unauthorized diffusion customization in image generation models. Traditional protections using untargeted attacks are not effective enough, so the authors propose a method called Attacking with Consistent score-function Errors (ACE). ACE significantly degrades the quality of customized images by introducing targeted errors, making unauthorized customization less viable. The paper validates ACE's effectiveness through extensive experiments and provides insights into the mechanisms of attack-based protections, setting a new benchmark in the field.", "questions": "1. Did the authors verify that the selected evaluation images were not part of the original SD models' training datasets to minimize potential bias? \n\n2. Please consider including the suggested work as an additional baseline for evaluating the proposed method.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Relevant and timely topic  \n2. Clear presentation of ideas  \n3. Thorough evaluation\n\nThe authors address the important and timely topic of protecting diffusion models from unauthorized fine-tuning on specific images. To overcome limitations in existing approaches, they shift the focus from untargeted to targeted attacks, introducing a novel protection method called ACE. The authors provide a clear explanation of the design and rationale behind their method, which is commendable. Additionally, they conduct extensive evaluations that demonstrate the method's effectiveness. Overall, this paper is well-written and easy to follow.", "weaknesses": "1. **Limited Technical Contribution**: The proposed method primarily relies on existing adversarial attacks, such as PGD, to mislead the model toward a predefined target. However, all of the techniques applied are directly taken from prior works, with minimal novel adaptation or extension.\n\n2. **Lack of Evaluation on SD3**: The evaluations are conducted exclusively on SD versions 1.4, 1.5, and 2.1, which are somewhat outdated. Although testing on SD3 could provide valuable insights, it’s worth noting that SD3 is not open-source and lacks support for customization pipelines like LoRA or DreamBooth, which are central to this paper’s focus on protecting against unauthorized customization. Including a discussion of this limitation and potential future directions for adapting the method to newer models could strengthen the paper.\n\n3. **Potential Bias in Selected Evaluation Images**: The authors selected images from two datasets to evaluate the performance of various customization and protection methods. However, it's unclear if these images were part of the original SD models’ training set, which could introduce bias. It would be helpful if the authors could clarify whether they verified that the selected images were not in the SD models’ training data. If verification wasn’t conducted, acknowledging this limitation would be beneficial.\n\n4. **Additional Baseline for Comparison**: While the authors include several baselines, an important reference—**\"Adversarial Perturbations Cannot Reliably Protect Artists from Generative AI\"**—is missing. Including this work would allow for a more comprehensive comparison. Specifically, a discussion on how the proposed method compares to or improves upon the reliability and effectiveness of the techniques discussed in this paper would provide valuable context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVfzSZTF4L", "reviewer": "Reviewer_v9HM", "summary": "This paper focuses on protecting images from being exploited by sota T2I diffusion models via adversarial perturbation. Different from previous works that mainly utilize untargeted attacks, this work points out an interesting and novel perspective that targeted attacks works better. The experimental results prove the method's effectiveness. The author also provides an insight to explain the success of targted attack: the attack can lead to model to learn more consistent chaotic patterns, and mitigate the neutralization effect of untargeted perturbation from multiple samples.", "questions": "- In algorithm 1 Line 5-7, why are you optimizing the diffusion model parameters as well? Seems that these details are not included in Eq. (4)-(5).\n\n- In table 1, what are the budgets of the baselines? Are they 4/255 or aligned with their own settings?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is simple and interesting, and the proposed method is easy to implement.\n\n- The discovery that targeted attacks on diffusion-based mimicry works better than untargeted ones is novel and interesting.\n\n- This paper is written in a clear way, and the logic of this paper is easy to follow.\n\n- The performance reported in the paper demonstrates the effectiveness of the proposed technique.\n\n- This paper tries to provide some insights to explain the underlying reason of why targeted attacks works better than untargeted attacks. The reasons and explanations are intuitive and reasonable.", "weaknesses": "My concerns mainly lie in the experimental evaluation part of the paper. The authors seem to be not closely following the recent advances in adversarial attacks & defenses for protection against diffusion-driven mimicry/editting.\n\n- Lack of recent attack baselines (in 2024) for comparison in the experiments, such as SDS, MetaCloak, Influence Watermarks, etc.\n\n- The paper only focuses on latent diffusion models, while other models such as diffusion transformers are not evaluated. I suggest to conduct more experiments on state-of-the-art DiT models. Although they might not be tailored for personalization, some of them can be easily adopted for image editting tasks. \n\n- The purification experiments does not include any purification methods specifically designed for diffusion mimicry, such as IMPRESS [1], GrIDPure [2], etc.\n\n- I also have a concern on the superiority of targeted attacks claimed in this paper. In the analyses part (Section 5), the main idea is that as targeted attacks can lead to model to learn more consistent chaotic patterns, they mitigate the neutralization effect of untargeted perturbation from multiple samples. However, for image editting tasks and some state-of-the-art personalization methods, only one reference image is involved, and the above \"neutralization effect\" will not neccessarily happen. How can the hypotheses in Section 5 explain the empirical success of targeted attacks in these settings?\n\nMinor Points:\n\n- The claim in L144 that all existing protection are untargeted attacks is not accurate. For example, Glaze should be classified as a targeted attack. ASPL also proposes a targeted attack version.\n\n- The description in L500 seems wrong. $\\theta^{\\prime}$ should be the customized diffusion model.\n\nThe reviewer will engage actively in the discussion and adjust the rating according to the rebuttal.\n\n[1]: Cao et al. IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI. NeurIPS 2023.\n\n[2]: Zhao et al. Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion? CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)