Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models
Authors: Tae-Young Lee
Juwon Seo
Jong Hwan Ko
Gyeong-Moon Park
Venue: NeurIPS 2025 poster ()
Keywords: text-to-image diffusion models
personalization
privacy
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We prevent unauthorized personalization of diffusion models at the model level.

Abstract: Recent advances in diffusion models have enabled high-quality synthesis of specific subjects, such as identities or objects. This capability, while unlocking new possibilities in content creation, also introduces significant privacy risks, as personalization techniques can be misused by malicious users to generate unauthorized images. Although several studies have attempted to counter this by generating adversarially perturbed samples designed to disrupt personalization, they rely on unrealistic assumptions and become ineffective in the presence of even a few clean images or under simple image transformations. To address these challenges, we shift the protection target from the images to the diffusion model itself to hinder the personalization of specific subjects, through our novel framework called $\textbf{A}$nti-$\textbf{P}$ersonalized $\textbf{D}$iffusion $\textbf{M}$odels ($\textbf{APDM}$). We first provide a theoretical analysis demonstrating that a naive approach of existing loss functions to diffusion models is inherently incapable of ensuring convergence for robust anti-personalization. Motivated by this finding, we introduce Direct Protective Optimization (DPO), a novel loss function that effectively disrupts subject personalization in the target model without compromising generative quality. Moreover, we propose a new dual-path optimization strategy, coined Learning to Protect (L2P). By alternating between personalization and protection paths, L2P simulates future personalization trajectories and adaptively reinforces protection at each step.
Experimental results demonstrate that our framework outperforms existing methods, achieving state-of-the-art performance in preventing unauthorized personalization.
The code is available at https://github.com/KU-VGI/APDM.

Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models
Tac-Young Lee Juwon Sco
Jong Hwan Koot
Gyeong-Moon Park
Korea University
Kyung Hee University $Sungkyunkwan University
ty1ee04150korea.co ac.kr
juseo001 0kkuu. ac.kr
jhkooskku. edd gm-parkOkorea ac.
Abstract
Recent advances in diffusion models have enabled high-quality synthesis of spe- cific subjects, such as identities objects. This capability, while unlocking new possibilities in content creation, also introduces significant privacy risks, as person- alization techniques can be misused by malicious content. Although several studies have attempted to counter this by generating adversarially perturbed samples designed to disrupt personalization, they rely unrealistic assumptions and become ineffective in the presence of even a few clean images under simple image transformations. To address these challenges, we shift the protection target from the images to the diffusion model itself to hinder the personalization of specific subjects, through our novel framework called Anti- Personalized Diffusion Models (APDM). We first provide a theoretical analysis demonstrating that naive approach of existing loss functions diffusion models is inherently incapable of ensuring convergence for robust anti personaliaazz..on.. Motivated by this finding, we introduce Direct Protective Optimization (DPO), a novel loss function that effectively disrupts subject personalization in the target model without compromising generative quality. Moreover, we propose a new dual-path optimization strategy, coined Learning to Protect (L2P). By alternating between personalization and protection paths, I.2P simulates future personaliza- tion trajectories and adaptively reinforces protection at each step. Experimental results demonstrate that our framework outperforms existing methods, achieving state-of-the-ee thtt---rtt performance in preventing unauthorized personalization. The code is available at http:///ttubbbcoo/KUUVV/AA
to generate unauthorized
1 Introduction
Diffusion models (DM) [3] have become prominent generative models across various domains and tasks, including image, video, and audio synthesis 30 g 21), image-to to -ooiaaa translation (27).
and image editing 9 Among these, personalization techniques 31
15 19 enabling the
generation of images depicting specific subjects (e.g. individuals, objects) in varied contexts, such as "an image of my dog on the moon have received significant attention. Several approaches, such as DreamBooth [31 and Custom Diffusion (15), have demonstrated highly effective capabilities for personalized image generation. However, such personalization also presents substantial privacy risks, malicious could exploit it create unauthorized images of specific individuals, for instance, to generate and distribute fake news, thereby raising significant social and ethical concerns [3]
"quual contribution.
t CCoresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

L
Find Photo
DesoPsiscring
Paocible
Dan-Prisoring for
Privacy Regularization (GDPR)
Knunkelge about Poisuning
(c. Ponoealaaioo
Optinization of Hyperparameter
Assign Duty
Protection Request
Founkes)
User's Photos
Knonlolge about AEML
Time Cansaiia
Trankenstion
Envrrsmmnntll Sctup
Service Provider
User
CCTV
Black Box
Unauthorized
Pregramming Language
Cannot Use to Model
sp. Rippla
Unimtended Caplure
User
Data: Patta-Pissniing
Need to be Protected Tunges
Bypass
Hard non-expert
(a) Impractical assurptiio..
(b) Bypuss.
(c) User burden.
(d) Conflict with regulation.
Figure 1: Motivation Figure. Existing protection approaches face critical limitations: (a) impracti- cality of applying data-pois to all images, (b) vulnerability to easy circumvention of protection methods, (c) high entry barriers for non-expert users, and (d) incompatibility with service providers who must comply with privacy regulations.
To prevent misuse of such personalization capability from a user's request, several protection ap- proaches 18
37 based data-poisoning have been proposed. They directly add impercepti- ble noise perturbations to the images of the specific subject using the Projected Gradient Descent (PGD) (25 When a malicious
34
38
attempts to personalize using these perturbed images, the added noise disrupts the stability of the training process, resulting in ineffective personalization convergence.
However, existing approaches suffer from several critical limitations in real-world scenarios (Figure []. Most importantly, their efficacy often hinges on the impractical assumption that users can apply poisoning comprehensively across their personal image collections collections-including those already shared, newly created, or even unintentionally captured- wwhic is a practically unachievable task. This limitation enables malicious users to easily bypass protection using unprotected images. Furthermore, even if the images are perturbed, attackers can still circumvent defense by applying transformations
that weaken the perturbation effects [34 23 12 On the other hand, poiitooniinnng is predominantly a user-centric defense, placing the implementation burden on individuals who are often non- non-experts, making widespread adoption unrealistic. Furthermore, this user-level design of existing approaches conflicts with privacy regulations-such as the GDPR [35)-that assign service providers the obligation ensure antilpersonnllzztiion upon user requests. As a result, such methods are inherently unsuitable for provider-side deployment (see Appendix F for more details).
Taken together, these issues highlight the need to move beyond user-side defenses toward model- level solutions that not only enable service providers to enforce anti- pettipprsonallzatiinn directly within their systems but also enhance robustness and practicality in real-world deployments. To address this, we shift our focus from the data samples to the DMs themselves. In this paper, we propose Anti-Personalized Diffusion Model (APDM), a novel framework designed directly remove per- sonalization capabilities for specific subjects within are-trained DMs, without data- Suta-poisoning. The primary goals of APDM are twofold: (i) preventing the unauthorized personalization attempts, result- ing in failed or irrelevant generations, and (ii) preserving the generation performance and its ability to personalize other, non-tare non-targeted subjects. To the best of our knowledge, APDM the first approach to directly update the model parameters for protection, inherently overcoming data dependency.
However, simply redirecting the protection effort the model parameters does not guarantee success if we naïvely adopt strategies from data-centric methods. Firstly, we theoretically prove that directly applying originally designed for creating adversarial perturbations on images-4 to the model parameters fails to converge. To this end, we introduce a novel loss function, Direct Protective Optimization (DPO), disrupting the personalization process. Moreover, simply applying a protection loss uniformly is insufficient, since personalization involves iterative updates to model parameters. Therefore, being aware of the personalization trajectory essential for robust protection. For this reason, we propose Learning to Protect (I.2P), dual-path optimization strategy. I.2P alternates between a personalization path, simulating potential future personalized model states, and a protection path, which leverages these intermediate states to apply adaptive, trajectory- protective updates. This dynamic approach allows the model to anticipate and counteract personalization attempts, ensuring robust DM protection in across various scenarios.
Our contributions can be summarized as follows:
For the first time, we propose a novel framework, called Anti-Personalizer Diffusion Model (APDM). for robust antioneeronnlizaoionn in DMs by directly updating model parameters, un-
2

---

like existing data-centire methods. This approach fundamentally overcomes the impractical assumptions and data dependency issues of prior works.
We theoretically prove that naive application of existing image perturbation losses directly to model parameters fails to converge. To address this, we propose a novel objective, Direct Protective Optimization (DPO) loss. DPO guides the model to remove the personalization capability of a specific subject while preserving generation performance.
To effectively counteract the iterative and adaptive process of personalization, we introduce Learning to Protect (L2P), dual-path optimization strategy that anticipates personalization trajectories and reinforces protection accordingly, enabling robust defense.
We empirically demonstrate that APDM can safeguard against personalization in real-world scenarios, achieving state-of-the-art performance across various personalization subjects.
2 Related Work
Personalized Text-to- Image Diffusion Models. The advancement of diffusion-basee image synthe- sis, like Stable Diffusion (SD) [30]. has enabled not only high-quality image generation but also the creation that reflect desired contexts from the text. This advancement has accelerated the widespread application of Text-to-Imane Image (T2I) DMs [30], one of which is personalization, such as generating images containing specific objects under the various situations (e.g. a particular dog or person on the moon). Consequently, research on personalized models has emerged. The most widely used method is DreamBooth [3I, which fine-tunes a pre-trained SD using a small set of images depicting specific concept (e.g. particular person). This allows users to generate desired images containing the target object. Texture Inversion
[4] achieves this by searching for an optimal text embedding that can represent the target object based on pseudo-words. Custom Diffusion [15] optimizes the key and value projection matrices in the cross-attention layers of the pre-trained SD, offering more efficient and robust personalization performance. However, these methods are doublc-edged sword, offering powerful personalization but also posing risks, such as misuse in crimes or unintended applications.
Protection against Unauthorized Personalization. To prevent unauthorized usage, many pro- tection methods have been developed based on adversarial attacks 7 2 25 AdyDM [88] was
the first to extend dlssifiiationn-asee adversarial attack methods DMs, generating adversarial samples for protecting personalization. Furthermore, Dnti-DreamBoot 34] proposed protection against more challenging fine-tuned DMs (e.g. DreamBooth). They used a fine-tuned surrogate model as guidance to obtain optimal perturbations for adversarial images. SimAC (38) improved this optimization process better suit DMs. while CAAT [39] focused on reducing time costs by updating cross-attention blocks. MetaCloak (23) and PID 17 have also been conducted to counter text variation or image transformation techniques (e.g. filtering). The most recent work, PAP (37), tries to predict potential prompt variations using Laplace approximation. However, existing works have primarily focused on how to effectively add perturbations images for protection, In contrast, as we mentioned above, we apply protection directly at the model level, reflecting real-world demands.
3 Preliminaries
3.1 Text-to-Image Diffusion Models
T2I DMs [30,
generate image d corresponding to given text prompt embedding c. T21 DMs operate via forward and reverse processes. In the forward process, noise (O I) is added to input image 2o to produce noisy image 2; timestep t [0,
popular variant of DMs I
33
Tt 5,20 + v1 - ãtt,
where or III m ai is computed from noise schedule (av: ) In the reverse process, DM, parame- terized by 0, aims to denoise vy. DM is trained to predict the noise residuals added to xy:
Csimple = A,. (lo =
3.2 Personalized Diffusion Models
To generate images that include specific subject, several works personalize pre-trained T2I DMs B1]
Given a small image set Zn E X of the subject and a text embedding oeer with a unique identifier, e.g. "a photo of [V*] person", they modify the loss function in Eq. ( as follows:
Cper
simple Exppt,,rrr, (.0. t, te =||
(3)
3

---

where is a noisy image from Eq. (T). However, directly applying this modified loss can cause language drift, where the personalized DM generates images related to target subject, even without unique identifier. To mitigate this, DreamBooth [31] introduces a prior preservation loss function that leverages the pre-trained DM. This encourages DM, using a class-specific text embedding e (e.g. "a photo of person") ") to retain its knowledge of the general class associated with the specific subject:
ppt
Rugg ,t. ,,,cp", (0..) co co(2"" ,,, c. - <l12
(4)
where zi a generated sample from the pre-trained T2I DM with the text embedding c and x is the noisy version of zg at timestep t. Alternatively, Custom Diffusion [15] utilizes images from training dataset instead of generated images for o The final objective for personalization becomes:
pee
simple + Lepl.
Method
4.1 Problem Formulation
Unlike prior approaches that perturb images, we directly update the parameters of the pre-trained
DM using only a small image set zo E X. Our goal is to transform 0 into a safeguarded model that inherently resists personalization of the subject appearing in these images. This process can be viewed as optimizing the model parameters with respect to protection objective:
i arg min Sprotect.
(6)
where Lprotect is a loss function to prevent personalization, which will be discussed in Section 4.3. Subsequently, if an adversary attempts to personalize subject in X' with this safeguarded model e, the resulting personalized model per is obtained follows:
0 per arg min Cper.
0
Our approach has two main objectives. For protection, the re-personalized model 6per should yield low-quality images or images of subjects perceptually distinct from those in X' For stability, the protected model 0 should be able to generate high-quality images and effectively personalize for the other subjects, comparable to those produced by the pre-trained DM 0.
4.2
Analysis of Naïve Approach
naive yet intuitive way to protect the model is to extend existing data- potatooooioinngg approaches [18
34 38 37 to the model level. Specifically, their noise update process that maximizes C simple using
PGD 255 can be naturally applied at the model level. In addition, the model's generative performance
can be preserved by incorporating Lpnt, as done in DreamBooth (T.. The overall objective for this naïve approach can be expressed as follows:
(8) To ensure effective protection using Lude. the optimization process must converge. We analyze the necessary conditions for convergence by examining the gradients of the loss with respect to 8. This leads to the following Proposition]] [ (proof in Appendix A.I.
Lado = -C Lypl-
Proposition 1. necessary condition for Ladv to converge to a local minimum with respect to model parameters is that the gradients of its constituent terms, V Vo and & must point in the same direction.
To further understand how these gradients influence each other during optimization, we analyze their interaction through the first-order Taylor approximation and derive the following relationships.
(Vo simple (0))T (VeLpp (0))) | VoCppt (1)
(9)
(V Cper simple (0)) (VoLppe (0)) < V per
simple (0)
(10)
Based on the Proposition we can restrict the left terms in Eq. (9) and 10). as [ Vimple IV (opp)(())) Using these results, we can rewrite the Eq. (9) and as:
simple (0)
Vo Porpee (0) < Vo (0))
(11)
Vopppt 0)) < V Cer
simple (0)
(12)

---

Image Set
(0.1)
P
Aper
1+
8:+1
B
Personalization
Copy
Luro
Gradient
Tt
Diffimion Model
Pre traited.
vy Aerrttert
Eppl
Negative
Profection
Spratect
0 J-
Piyy Fprotest
Xllper
Tt
Paic
XMgrazenty
Dillisioon Mosle
Learning to Profect (L.P)
Direct Protective Optimization
Negative O
(APDM)
(DPO)
Proparation of the paired
(b) Protection overview of APDM.
Figure 2: Overview. To prevent personalization in the parameter level, propose Anti- anti-Personalized Diffusion Model (APDM). (a) APDM first generates a paired image for each clean input image zo. (b) APDM consists of two components (i) Learning to Protect, a novel optimization algorithm that makes the protection procedure aware of personalization trajectories, and (ii) Direct Protective Optimization loss, designed disrupt personalization while preserving the generation capabilities.
By combining Proposition D with the inequalities above, we observe that the required gradient align- ment for convergence cannot hold, which we formalize in the following theorem (see Appendix (A2)).
Theorem 1. the objective is simultaneously reduce both -L simpte and Lopte the necessary condition for convergence outlined in Proposition leads the contradictory requirements presented in Eq. (1 and (12.. Therefore, Ludv composed of such conflicting terms generally fails to converge point that effectively optimizes both objectives.
Therefore, a new loss function is required to resolve this conflict and ensure that antiipersonalization updates stay consistent with the denoising process, maintaining both generation quality and protection.
4.3 Anti-Personalized Diffusion Models
To achieve the dual goals outlined in 4.1 we propose a novel framework, Anti-Personalizee Diffusion Models (APDM). APDM introduces novel loss function, called Direct Protective Optimization (DPO), which aims to prevent personalization in DMs while maintaining their original generative performance (Section 4.3.Ip. DPO effectively mitigates the model collapse issue discussed in Section 4.2 Furthermore, we propose a novel dual-path optimization scheme, Learning to Protect (L2P),, which considers the trajectory of personalization during training apply the proposed loss function more effectively (Section 4.3.2.
The overview APDM is presented in Figure 2
4.3.1 Direct Protective Optimization
Instead of Ladv: which degrades the model's distribution due convergence failure (Section 4.2), directly guide the model on which information should be learned and which should be suppressed. Inspired by Direct Preference Optimization [29], given pair of images (at zy ). we designate 2 as positive sample to be encouraged during the protection procedure and xp as negative sample be discouraged, i.e. an image containing a specific subject to be protected (von E 2'). By incorporating the Bradley-Terry model, the probability of preferring 25 over Tn can be expressed as:
p(at xū) = (r(xt)) -(()))
(13)
where (--) denotes the sigmoid function and r(.) represents the reward function. Building upon the formulation of Diffusion-DPO (36) (see Appendix for detailed derivation), we define new Direct Protective Optimization (DPO) as follows:
+ = | 4, - <112 |ka 1,c) - el12
r = [ c) - 2 - | c) - ell2.
(14)
CpPo -B, , (c.t.cwN(o..)) log a - -3(r+ r)))),
where + and r- denote the positive and negative rewards for preferred and non-preferred direction, respectively, á is pre-trained DM, and is hyper-parameter that controls the extent to which
5

---

Algorithm/ Learning to Protect (L2P)
Input:
pre-trained model 0, loss function for personalization Lerr, loss function for protection proteect, the number of personalization loops Ner. the number of protection loops N protect, learning rate in for personalization "yper, learning rate in protection Vprotect.
Output: safeguarded model 0.
Procedure:
1: 4- 1,0; +
2: for j to N proteect do
D. Protection Path
3:
1, co.0oy g Ø
4: for to Nper do
Personalization Path
5: 11+1 t 0f Oper o Sper g.append( 02 Syrotect) 7: end for
Eq. I7
6:
8: protent x g.sum))
Eq. Eq.
19 20
9: 0j+1 0j- Tprotent V protect
end for
11:
return +- 0 Npratett
can diverge from c. In our DPO, we prepare z by synthesizing images from pre-trained T2I DMs o using generic prompt c and they are paired one-to-one with the negative samples X. This approach naturally encourages the generation of generic (positive) images while effectively suppressing the synthesis of negative images depicting the specific subject.
Finally, combining the proposed loss term with the preservation loss (Cppi), the final objective is: (15)
Epratect = L.PO pppt-
4.3.2 Learning to Protect
Since the personalization DMs involves iterative updates to model parameters, effective protection should consider the evolving personalized states different states [6 5 Therefore, instead of simply applying our pprtect uniformly to the model, we simulate the future personalization path in advance, allowing the model anticipate upcoming parameter changes during personalization. To this end, we introduce a novel dual-path optimization algorithm, Learning to Protect (L2P). L2P integrates personalization into the protection loop, enabling the model learn from simulated personalization behaviors and adjust its parameters for adaptive and robust protection.
L2P involves two optimization paths: personalization and protection. The personalization path updates the model from the current protection state 8, to intermediate state 8%, using Eq. (5):
0/ = 0g: 0/+1 = 0% Yper Vo, Spers
(16)
(17)
where Yper is the learning rate for personalization, and 6/41 is the intermediate state at step + 1 during personalization. Using Eq. (17). we can simulate the future personalization trajectory via updating the model 8/ iteratively, in the middle of protecting the DM.
For the protection path, we leverage these intermediate states acquired in the personalization path. Specifically, we compute the gradient V; of the model 0f with respect to Sprotect, at each state in the personalization path follows:
V of Sprotect-
(18)
We then accumulate V. during the whole personalization path (total of Nper times) compose of gradients, = (v.) 1 Using this set of gradients we can estimate the direction protection from the summation of these accumulated gradients as follows:
Nyer
protect
V
(19)
Finally, we update the intermediate protection model 0g with V prrotect obtain 0,+1:
0 +1+1
0g Mprotect V protect,
(20)

---

Table 1: Quantitative Comparison on Protection. We measured the protection performance via DINO score [3 and BRISQUE [26]. We examined the baseline on different number of clean images. the number O, there are only perturbed images produced by data-poisone cata-poisoning approaches. The experiments were mainly conducted on two different subjects: person and dog.
# Clean
DINO (L)
BRISQUE (1)
Methods
Images
"person" "
"dog"
Avg.
"person" "
"dog"
Avg.
DreamBooth 31
0.6994
0.6056
0.6525
11.27
22.33
16.80
0.5752
0.4247
0.4999
19.52
28.60
24.06
AdvDM 18
0,5436
0,4393
0.4915
17.82
28.58
23.20
N
0.6417
0.4775
0.5596
20.30
27.36
23.83
0.5254
0.4106
0.4680
26.90
30.23
28.56
DrothBooBBB 34
0.6081
0.4704
0.5393
23.76
27.49
25.63
N
0.6951
0,5304
0.6127
15.48
25.26
20.37
0
0.4448
0.4374
0.4411
23.73
31.64
27.69
SimAC 38
0.5824
0.4537
0.5181
18.04
29.54
23.79
N
0.6991
0,5370
0.6181
14.28
27.05
20.67
0
0,6556
0.5120
0.5838
22.61
30.20
26.41
PAP 37
0,6690
0,5032
0.5861
22.02
29.00
25.51
N
0.7028
0.5270
0.6149
19.64
23.41
21.53
APDM (Ours)
N 0.1375
0.0959
0.1167
40.25
60.74
50.50
where Vprotect is the learning rate for protection. By repeating this process for Nproteet times, we can obtain safeguarded model 0, which is aware of the personalization path inherently for better protection. Algorithm D illluttrates the overall learning process of L2P for our APDM framework.
Experiments
5.1 Experimental Setup
Evaluation Metrics. To evaluate the efffectiveness of APDM in protecting against personalization on specific subjects, we used two metrics: (i) the DINO score 3 as a similarity-base metric and (ii) BRISQUE [26] for assessing image quality. Additionally, we evaluated the preservation of the pre-trained model's generation capabilities by using (iii) the FID score [10] for image quality, (iv) the CLIP score (28). (v) TIFA [33 and (vi) GenEval [6] for image-tex: alignment.
Baselines. We consider DreamBooth 31 and Custom Diffusion [5] as personalization methods.
The results Custom Diffusion are presented in Appendix C For baselines, we include the previous
protection approaches: (i) AdvDM (39] (ii) Anti-DreamBooth (34) (iii) SimAC [38], and (iv)
PAP 37
Following Botth, we set the perturbation intensity for all baselines 5e-2.
Datasets.
31 and Boti (34] to evaluate the protection performance. The DreamBooth dataset contains 4-6 images per subject across various object classes such dog, cat, and toy. The consisting of facial images collected from CelebA-HQ [14 and VGGFace2 1 To quantify the
We used the datasets from both DreamBooth
dataset includes 4 images per person,
preservation performance of the model, we also used the MS-COCO 2014 20 validation split.
Implementation Details. We built APDM Stable Diffusion 1.5 and Stable Diffusion 2.1 [30] with 512x512 12 resolution. We used AdamW optimizer (24) with learning rates "/per = Pproteet = 5e 6. In DPO, we set the hyperparameter 3 to 1. In L2P, used Nper 20 and N Nyprotect 800. We conducted all of our experiments on a single NVIDIA RTX A6000 GPU, and it took about GPU hours protect DM. To synthesize images, used PNDM scheduler 22 with 20 steps. For Stable Diffusion 2.1, we have attached the experimental results in Appendix
5.2 Protection Performance
T first evaluated the baselines and APDM from the perspective As shown in Figure and and of protection. We first personalized the pre-trained Stable Diffusion using DreamBooth (31) as a reference. In this experiment, we considered three scenarios to baselines and APDM. For DreamBooth and APDM, only N clean (i.e. non-perturbed) images were used throughout the entire
hhapss/gghhbbboomoooggdddemooo

---

Training Images ("person")
Training Images ("dog")
"a photo of [V*] person"
DreamBBeth
Anti- Booh
SimAC
AdvDM
APDM
Clen
All Partorhed
One Clan
One Parturbal
All Penurbed
Ouz Clean
One Perturee All Partorbed
One Clean
One Pottrbed
All Clon
"a pho of [V*]
dog"
Dreat
Ami- Dresmfoothh
SimAC
APIMI (Ours)
Pettrbed
Partorival
Perturbal
One
Perturbod Portanhvall
One Clean
One Perterbod
Figure 3: Qualitative Comparison on Protection. We examined the baselines and APDM on a protective aspect. We tested basclines different circumstance "All Perturbed", "One Clean", and "One Perturbed". In the "All Perturbed" setting, the baselines added perturbations to all training images. "One Clean" and "One Perturbed" settings where the dataset contains one clean image or one perturbed image.
difficult than "All Perturbed" setting,
experiment ("All Clean" in Figure 3y. On the other hand, for data- tatta-posoonig baselines, we adopted different personalization scenarios. For "All Perturbed" scenario, we utilized all perturbed images from each data auta-poisoning baseline. Moreover, for "One Clean" scenario, we used 1 clean image and perturbed images for personalization. Lastly, the most challenging scenario, for "One Perturbed" scenario, there were only 1 perturbed image and N - clean images in the dataset.
In Figure 3 comparisons revealed their limitations as the scenarios become more challenging. When only one perturbed image is used and the others remain clean, protection against personalization for the subjects becomes ineffective. In contrast, despite the presence of clean images, APDM consistently demonstrated its robustness in more challenging scenarios (additional qualitative results in Appendix EE. We also present a quantitative comparison in Table [ highlighting that APDM outperforms data- peiittooisninnggggg approaches even under the most difficult conditions. This is because APDM protects personalization at the medel-level, making robust to variations in the imput data. In addition, we also tested APDM in different scenarios (transform, such as flipping and blurring) and subjects such as "cat", "sneaker", "glasses", and "clock" (results Appendix B
5.3 Preservation Performance
As described in Section 4.3.2) we updated the parameters of DM initialized with a pre-trained DM to obtain safeguarded model. To ensure its usability in future applications, it is essential to pre- serve the inherent capabilities of the pre- pre-trained DMs during the protection process. In this sec- tion, we evaluated the inherent performance based on image quality, image-text alignment of generated images, and the success of personalization for subjects not targeted by the protection.
Table 2: Preservation Performance on Image Quality and Image-Text Alignment. We measured the image quality via FID score [10] and image-text alignment via CLIP score (28),
TIFA 133 and GenEval 66 on COCO 2014 (20 validation dataset.
Methods
FID (J)
CLIP (*)
TIFA (t)
GenEval (1)
Stable Diffusion
30
25,98
0.2878
78.76
0.4303
APDM (Ours)
28.85
0.2853
75.91
0.4017
8

---

Table 3: Preservation Performance on Personalization of Different Subjects. We tried per- sonalize APDM to different subjects, such as "cat", "sneaker", and "glasses". We reported the personalization performance DreamBooth [31 these subjects as a reference.
DINO (t)
BRISQUE (4)
Methods
"cat"
"sneaker"
"glasses"
Avg.
"cat"
"sneaker"
"glasses"
Avg.
DreamBooth []]
0.4903
0.6110
0.6961
0,5991
25.32
23.14
19.01
22.49 24.41
APDM (Ours) 0.4231
0.7573
0.7198
0.6334
27.72
18.10
27.41
Table 4: Ablation on the Effect of Image Pair-
Table Ablation on the Effect of L,P. We compared the performance between protection attempts without and with L.P.
ing between 20 and Tn- We compared the pro- tection performance with and without pairing.
DINO (1)
BRISQUE (t)
DINO (4)
BRISQUE (1)
Paired
L2P
"person"
"dog"
"person"
"dog"
"person"
"dog"
"person"
"dog"
x
0,2770
0.3487
27.32
29.87
X
0,4454
0,3689
24.70
30.62
0.1375
0.0959
40.25
60.74
/
0.1375
0.0959
40.25
60.74
Table 6: Ablation on the Effect of B. We com- pared the protection performance with different
Table 7: Ablation on the Effect of Nper L.P..
We measured the performance in protection pect by varying Nper of personalization path.
hyperparameter 3.
DINO (1)
BRISQUE (1)
DINO (J)
BRISQUE (t)
N
per
"person"
"dog"
"person"
"dog"
"person"
"dog"
"person"
"dog"
0.1375
0.0959
40.25
60.74
0.3371
0.1923
37.89
39.48
10
0,5392
0.3885
13.58
15.14
10
0.2096
0.1342
38.14
47.15
100
0,5962
0.4755
12.21
14.10
20
0.1375
0.0959
40.25
60.74
Table 2 shows that APDM maintains high-quality image generation comparable to the pre-trained model. Beyond image quality and image-text alignment, we also evaluated its ability personalize for different subjects using the protected DMs. Specifically, we tested personalization on models protected for "person" or "dog". using new set of images featuring "cat", "clock" and "glasses". As shown in Table these protected models remain effective for personalizing other subjects. Overall, APDM successfully protects specific subjects while preserving personalization capabilities for others, making it suitable for handling diverse user requests in real-world applications.
5.4 Ablation Study
Ablation on Loss Functions. In Section we introduced novel objective, Direct Protective Optimization (DPO), which effectively prevents personalization while minimally degrading the model's generation performance. In Table we assessed the impact of pairing positive and negative images on protection performance. The results demonstrate that constructing image pairs significantly enhances performance by providing explicit guidance which information should be encouraged discouraged. Additionally, we investigated the effect of the hyperparameter 3, which governs the strength of our DPO objective. As shown in Table 6 our findings indicate that reducing 3 allows APDM to more effectively prevent personalization.
Ablation on Optimization Scheme. In Section 4.3.2 we proposed novel optimization scheme, Learning to Protect (L2P), which incorporates awareness of the personalization process during protection. In Table 5 we compared the protection performance with and without L2P,, and observed that incorporating the personalization trajectory significantly improves protection performance. Moreover, we examined the effect of the number of personalization paths (Nper). As shown in Table increasing Nper consistently improves performance. Despite this trend, we set Nper = as the default in overall experiments, since it already achieved latt-offthe--r performance.

---

Table Protection performance of APDM on clean and perturbed data. We evaluate whether
APDM can maintain its protection capability regardless of input perturbations.
Clean Images
Methods
DINO (L)
BRISQUE (t)
DreamBooth
31
N
0.6869
16.69
Bnti-DreamBooth
0
0.5646
22.50
APDM (Ours)
N
0.1375
40.25
APDM (Ours, perturbed)
0.1702
40.20
5.5 Additional Experiments
As demonstrated in previous experiments, APDM effectively performs protection even in challenging cases, such when clean images are used. This robustness comes from its model-level defense mechanism, which allows protection to be achieved independently of the input data. To further demonstrate this robustness, we examined whether APDM also protect against perturbed data generated through data-poise data-poisonin methods. Specifically, we generated perturbed data using Anti- DreamBooth [34] and evaluated APDM's protection performance on these data. As shown in Table 8 APDM successfully prevents personalization even on perturbed data, confirming that its effectiveness independent of the input variations.
Building upon the previous analysis on perturbed data, we further inves- tigated whether APDM maintains its protection capability when both the number and type of personalization data vary. Specifically, this evalua- tion examined the gencralization and scalability of APDM by considering two factors: (i) the use of unseen data that were not included during the pro- tection stage, and (ii) the increased amount of personalization data per subject. As shown in Table APDM consistently maintains protection performance even when 4-12 unseen images are introduced, con- firming that its defense mechanism generalizes well to unseen samples and remains robust as the data volume increases.
Table 9: Protection performance of APDM under varying numbers of unseen images. We evaluate whether APDM can maintain its protection capability across different input conditions and unseen data counts.
Methods
of unseen
DINO (L)
BRISQUE (t)
DrcamBooth
0.6869
16.69
0.1375
40.25
0.1616
38.14
APDM (Ours)
0.
38.87
0.1873
38.87
To further assess the robustness of APDM under diverse personalization conditions, we additionally conducted experiments using varied text prompts and different unique identifiers, well an inde- pendent user study designed to evaluate real users' preferences. Due to the page limit, these extended results are provided in Appendix (diverse prompt and identifier experiments) and Appendix D (user study).
6 Conclusion
In this paper, we address privacy concerns in personalized DMs. We highlight critical limitations of existing approaches, which depend on impractical assumptions (e.g. exhaustive data poisoning) and fail comply with privacy regulations. Furthermore, we demonstrate that these approaches are casily circumvented when attackers use clean images or apply transformations to weaken the perturbation effects. Therefore, we shifted the focus from data-centric defenses to model-level protection, aiming to directly prevent personalization through optimization rather than input modification. To this end, we propose a novel framework APDM Anti-Peeron Diffusion Models), which consists of novel loss function, DPO (Direct Protective Optimization). and new dual-path optimization scheme, L2P (Learning to Protect). With APDM, we successfully prevented personalization while preserving the generative quality of the original model. Experimental results demonstrate the effectiveness and robustness of APDM with promising outputs. We hope our work extends the scope of anti- personalization towards more practical and appropriate real-world solutions.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Korea Planning & Evaluation Institute of Industrial Technology (KEIT) grant funded by the Korea government (MOTIE) (RS-2024-00044344)) and in part by Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) under Grant No. RS2019-111
(Artificial Intelligence Graduate School Program (Korea University)). No. RS- RS-2024-00457882 (Al Research Hub Project), and 2019-0-00004 (Development of Semi-Supervised Learning Language Intelligence Technology and Korean Tutoring Service for Foreigners). Additionally, it was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) Grant funded by the Korea Govermment (MSIT) ((Artificial Intelligence Innovation Hub) under Grant 210-2022-1122200
References
[1] Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: A dataset for recognising faces across pose and age. In: 2018 13th IEEE international conference automatic face & gesture recognition (FG 2018). pp. 67-74. IEEE (2018)
[2] Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (sp). pp. 37. Ieee (2017)
Caron, M., Touvron, H., Misra, Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9650-9660 (2021)
[4] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. In: The Eleventh Interna- tional Conference Learning Representations (ICLR) (2023), htttt::/ MAQvFO8TcyC
net/forumiidt
[5] Gao, H., Pang, T., Du, C., Hu, T., Deng, Z., Lin, M.: Meta-unlearning on diffusion models: Preventing relearning unlearned concepts. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2131-2141 (2025)
[6] Ghosh, D., Hajishirzi, H., Schmidt, L.: Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems (NeurIPS) 36, 52132-112559 222122 (2023)
[7] Goodfellow, LJJ., Shlens, J., Szegedy, C.: Explaining and hamessing adversarial examples. In: International Conference on Learning Representations (ICLR) (2015), actps://arxiv. h/tg://bb///a 1412. 6572
[8] Guo, Y.. Yang. C., Rao. A.. Liang, Z. Wang. Y.. Qiao, Y., Agrawala, M., Lin. D., Dai, B.: Animatediff: Animate your personalized text-to-image io-i diffusion models without specific tuning. In: The Twelfth Inter- national Conference on Leariing Representations (ICLR) (2024), httpp::/opperrviiee http:/pprrreeeee..efff.ur.. Id-Fx2SbBgcte
[9] Hertz, A., Mokady, R., Tenenbaum, J., Aherman, K., Pritch, Y., Cohen-or, D.: Prompt-to- image editing with cross-attention control. In: The Eleventh International Conference on Learning Representations (ICLR) (2023), https:///peerrevier net/forunm) _i==mdddiddddeypbbbbbbb
[10] Heusel, M., Ramsauer, H., Unterthiner, T.. Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017)
[11] Ho. J., Jain, A., Abbeel, P.: Denoising diffuusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS) 33, 6840-6851 (2020)
[12] Hönig, R., Rando, J., Carlini, N., Tramèr, Adversarial perturbations cannot reliably protect artists from generative ai. In: The Thirteenth International Conference on Learning Representatinns (ICLR) (2025)
[13] Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., Smith, N.A.: Tifa: Accurate and inter- pretable text-to-image faithfulness evaluation with question answering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 20406-20417 (2023)
[14] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability and variation. In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)
11

---

[15] Kumaai. N., Zhang, B.. Zhang, R., Shechtman, E., Zhu, J.Y. Multi-concept customization of text-to-imagg diffusion. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1931-1941 (2023)
[16] Lee, T.Y., Park, S., Jeon, M., Hwang, H., Park, G.M.: Esc: Erasing space concept for knowledge deletion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5010-5019 (2025)
[17] Li, A., Mo, Y., Li, M., Wang. Y:: Pid: Prompt-independent data protection against latent diffusion models. In: Salakhutdinov, R. Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F. (cds.) Proceedings of the 41st International Conference on Machine Learning (ICML). Proceedings of Machine Learning Research, vol. 235, pp. 28421-28447. PMLR (21-27 Jul 2024), press/v235/1s v23//1124//11 html
mlr
[18] Liang. C., Wu, X., Hua, Y., Zhang. J., Xue, Y., Song. T., Xue, Z., Ma, R., Guan, H.: Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Proceedings of the 40th International Conference on Machine Learning (ICMI.). Proceedings of Machine I Learning Research, vol. 202, pp. 20763-20786. PMLR (23-29 Jul 2023), https::/ppocceddinggg mlr. pwwss//v022/iaagg2333 httpp//ooeeddduaal.eee////////// html
[19] Lim, H., Won, Y., Sco, J., Park, G.M.: Conceptsplit: Decoupled multi concept personalization of diffusion models via token-wise adaptation and attention disentanglement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 18821-18430 (2025)
[20] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects context. In: Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13. pp. 740-755. Springer (2014)
[21] Liu, H., Chen, Z., Yuan, Y., Mci, X., Liu, X., Mandic, D., Wang, W. Plumbley, M.D.; AudioL Text-to-audio generation with latent diffusion models. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, (eds.) Proceedings of the 40th International Conferenee Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 202, pp. 21450-21474, PMLR (23-29 Jul 2023).
mlr, Axres//202/1ii 1inn2ff...... html
[22] Liu, L., Ren, Y. Lin, Z., Zhao, Z. Pseudo numerical methods for diffusion models manifolds. In: International Conference Learning Representations (ICLR) (2022), https:://ponrrevier net/ Forum2'ddPP1WWWWyyBRY
[23] Liu, Y., Fan, C., Dai, Y. Chen, X., Zhou, P., Sun, L.: Metacloak: Preventing unauthorized subject-driven text-to-image diffusion-based synthesis via meta-lcarning. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 24219-24221 (2024)
[24] Loshchilov, I., Hutter, F: Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR) (2019)
[25] Madry, A., Makelov, A., Schmidt, I.., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: International Conference Learning Representations (ICLR) (2018), https //openreview. wwt.://www...etttoorrrrarrrrraaa
[26] Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment the spatial domain. IEEE Transactions on Image Processing (1())), 4695 4708 (2012). /ttp:////oi.or/10.... 1ttp://00..000000.2.2222222.
[27] Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1-11 (2023)
[28] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (ICMI.). pp. 8748-8763. PMLR (2021)
[29] Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems (NeurIPS) 36, 53728-53741 (2023)
[30] Rombach, R. Blattmann, A., Lorenz, D., Esser, P., Ommer. B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR). pp.
(2022)
12

---

[31] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference Computer Vision Pattern Recognition (CVPR). pp. 22500-22510 (2023)
[32] Seo, J., Lee. S.H., Lee, T.Y., Moon, S., Park, G.M.: Generative unlearning for any identity. In: Proceedings of IEEE/CVF Conference on Computer Vision Pattern Recognition (CVPR). pp. 9151-9161 (2024)
[33] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: International Conference on Machine Learning (ICML). pp. 2256- 2265. PMLR (2015)
[34] Van Le. T., Phung, H., Nguyen, T.H., Dao, Q. Tran, N.N., Tran, A.: Anti-dreambooth: Protecting users from personalized text-t--image synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2116-2127 (2023)
[35] Voigt, P., Bussche, A.v.d.: The EU General Data Protection Regulation (GDPR): Practical Guide. Springer Publishing Company, Incorporated (2017)
[36] Wallace, B., Dang, M., Rafailov, R., Zhou, L.,, Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty,
S., Naik, N.: Diftusion model alignment using direct preference optimization. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 8228 8238 (2024)
[37] Wan, C., He, Y., Song, X.. Gong, Y:: Prompt-agnostic adversarial perturbation for customized diffusion models. Advances in Neural Information Processing Systems (NcurIPS) 37, 136576-136611 (2024)
[38] Wang. F., Tan, Z., Wei, T., Wu, Y., Huang, Q:: Simac: A simple anti- ciffimsstoizziionn method for protecting face privacy against rext-to-image synthesis of diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12047-12056 (June 2024)
[39] Xu, J., Lu, Y., Li, Y., Lu, S., Wang, D., Wei, X.: Perturbing attention gives you bang for the buck: Subtle imaging perturbations that efficiently fool customized diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24534 24543 (2024)
13

---

In this appendix, we provide detailed proofs and derivations, and additional experimental results that were included in the main paper due to page limits. The contents of the appendix are as follows:
Appendix A
Derivation and proof of Proposition 1, Theorem 1, and Direct Protective Optimization (DPO) objective.
Appendix
Additional experiments including empirical results about naïve approach, comparison on protection with image transformations, and protection performance for different subjects.
B
Generalizability of APDM different personalization methods, Stable Diffu- sion version, unique identifier, and diverse test prompts.
Appendix C
Appendix
User study about protection performance.
Appendix Additional qualitative results extending to the experimental results in the main paper.
Appendix
Additional explanation about our motivation.
Appendix G The limitations and broader impacts of APDM, and discussion of future work.
A Proofs and Derivation
In this section, we present the formal proofs and derivations supporting our main theoretical con- tributions discussed in the main paper. We begin by providing rigorous proof for Proposition [] (Appendix [A.I), followed by the complete proof for Theorem (Appendix A.2). Subsequently, we
detail the step-by-step derivation of our proposed DPO loss function Appendix A.3
A.1 Proof of Proposition 1
The primary goal of Proposition iss identify and establish the necessary conditions under which naïve approach converges. We begin the proof by recalling the loss function of naïve approach, Equation b our main paper:
Ladv
-
fper simple Xhepp
(21)
where À is positive scalar (A 0) to weight the Lypd. and each term defined as follows:
simple PrppstoccrrooN../leeeee
cll2
(22)
pppl = En ,t.r -- (x ,t, c.r) c
(23)
In optimization theory, a fundamental necessary condition for differentiable function to attain local minimum is that its derivative with respect to the optimization variables must be zero. This is often referred to as the first-order necessary condition for optimality. Applying this principle to our case, for Lado to converge to a stable point with respect to the model parameters the derivative must be zero as:
VoLado 0.
(24)
To address the condition in Equation 24 simplify the computation, we first recall the MSE loss term as:
we compute the gradient of Ladv with respect To
Iu v u u 2u v + TTv.
(25)
Using the expansion of Equation tion 23 as follows:
25 we can rewrite the MSE terms of Equation 22 and Equa-
|[eo(1e,t,)) (, - - =
TerT
per Eg
-2e pererT
e ateeee TTe,
(26)
&
(x]" t, tt,,t el2 = T T
ppl c
2c
ppl
ete e.
(27)
where <gTr ca c) and e = co(xf"" 1 crr). For notational simplicity in the subsequent derivations, will omit the input variables (e.g. Iz, t, c) and use the superscripts per and ppl to
14

---

distinguish both terms in 2 simple and pppl respectively. Now, substituting the expanded forms from L back into simple and pppl rewrite these.
Equation (26) and Equation 27
C For simple using Equation 26
Cee
simple -.pg.....-NN...nee per 6g" -26 c+ ETe..
2eeer
(28)
Moreover, by the linearity of expectation, which includes the property BB[A BJ -[[A]
[
(additivity principle), we can distribute expectation as follows:
simple = B fg
per per) - 2 per c) C|+
(29)
Please note that for readability, we also omit the explicit subscript variables of the expectation.
Similarly, for pypl, using Equation (27 and linearity of expectation:
Cppl = En ,t,em (ccm,,-N((0.1)655 ppt T
ppl
-26g c+e
Eg
(30)
Bleg mpl
ppl. 2E c) E
These expanded expressions (Equation and Equation (30) simplify the subsequent gradient derivations. To compute the gradients of these loss functions, we will differentiate the terms within the expectation, which is permissible under suitable regularity conditions by applying the Leibniz Rule. We first consider the derivatives the core components that appear inside the expectations, with respect to the model parameters 0.
Volcø ce) = 2JJ co. Vo(E c) J1 e.
(31)
(32)
VO(E Vo(=Te) O,
(33)
where Jo = @
3Ø la and 6 is independent of 0, the derivative any term solely dependent on (i.e. eT c) with respect to 0 is zero. Using these results, we can now express the gradient of MSE loss term fper inside the expectations. For the term simple'
Velloo -el
00
(e
T per
6B
00
(cDer)
e)+
00
ee)
(34)
2
2 E.
And for the term in Cppl:
θ ppt
Valleo(<?" CTT)
e112
(c)
2
(e p
+)
(e Te))
00
00
00
(35)
ppl
p.ppp
Eo
E.
Since simple and Cppt are expectations of the terms whose gradients were derived in Equation 34
and Equation (35), and we apply the Leibniz Rule. This allows us to take the expectation of those gradients to find the final gradients of the loss functions:
V VoLper per = 2 pper c) 2T GEJ perr c).
(36)
V = 2T[ c) [R c).
(37)
Consequently, using the gradients (Equation (36) and Equation 337 gence condition for Ladv with respect to 0 as:
determine the conver-
V VeLads = V simple + A PL
(38)
{215 Jge gger T
2 per
E]} + A(2B[J e] 2 E]}.
Based on Equation rearranging Equation (38) yields the final condition for Proposition
V Per
AV Appp.
(39)
simple
The result in Equation indicates that for Ludo to converge, the gradients of L Leer must point in the same direction, À > 0. This completes the proof of Proposition simple and
Cppt
15

---

A.2 Proof of Theorem
In Proposition we establish that for Lude to converge, a necessary condition is VoL & Based on this proposition, we now prove Theorem I Our proof will demonstrate how the aforementioned convergence condition (Equation 39) inherently conflicts with the goal of simultaneously decreasing both -immpi and pppl.
To figure this out, analyze the impact of parameter update, A8, each loss term using first-order Taylor Expansion. A parameter update A derived from a gradient descent step on Cades and assuming the scalar À = 1 for simplicity in this derivation. AO can be defined as:
AO
1I a0 fado:
(40)
where 7) 0 is the learning rate. The change in eeer Taylor Expansion:
simple due to A0 can be approximated by first-order
i
Cper simple (0 + 20)
pper
simple (0) ~
Cper
simple (0)) A6
00
0
θ 08
00
per
simple (8)] [ {-1[(
00
per
simple
(0)
+
caw (0)] (41)
n1
ppe simple
(0) n1
Cper
simple (0)
Sm (0))
30
30
where L()) means the the loss calculated with the parameter 0. Our objective is to minimize simple. which is equivalent to increasing simple- For this reason, the difference of C Equation is greater than zero. Using this condition, we can obtain the final inequality from
Ceer
Ceer
simple in
Equation 41
i C per
(0)1 2 Cper simple (O)]
30 (0).
(42)
30 simple
This inequality (Equation an increase in CPer simple.
(2 represents the condition under which the parameter update leads to Similarly, we derive the impact of the parameter update AO on pppl-
C A0) Cppl (0) Rz
(p))
AA
08
i
a
Cppi( (0)] {-nl
Cper
simmle (0) +
00 cm
(43)
00
θ
Eypa (0))
Simppie (0) n| Cm (0)
00
30
00
To minimize Cppt, it should decrease, which means that the change approximated by Equation (43 must be less than zero. We can also derive the condition as:
00 Pmp (
per simple (0) < ll 00 C Spp (0)
(44)
00
Equation 22 and Equation 44 share the common inner product term Lou Cppt (0)) 2
80 simple (0)
Recall from Proposition]] T for converging Lade. the gradients of the two terms must point in the same direction. This relationship allows us to remove the cosine term in the inner product (:: cos(O) 1). Based on this, we can rewrite the inequalities as below:
0 per
00 simple
(0)l
00 Eypi <
00
per
simple
(0)11²
(45)
Cpp (0)
Cermpie
(0)l<
<
||
C
Cppt (0)
(46)
00
00
00
We can further rearrange the above inequality as:
8
00 Ppp (0) <
simple (0)),
(47)
00
0
Cppe(()) (0)
>
simple (0))-
(48)
00
00
16

---

This rearrangement relies on the assumption that both individual gradients are non-zero, i.e. 0 pper
00 simple (0) > O and 8 (0) > 0. This assumption holds for any 0 that is not already a local optimum for both individual objectives.
Recall the objectives: to increase simpte: the condition in Equation 44 must hold for the current parameter update. On the other hand, to decrease pppt, the condition in Equation (48) must satisfy for the same parameter update. These two conditions are mutually exclusive. This contradiction demonstrates that if the system is at point satisfying the convergence condition for Ladv (Proposition 1), the objective of simultaneously decreasing Lpper simple and pppt cannot be achieved. Therefore, the naïve approach Lodu. as composed of these conflicting objectives under its own convergence condition, generally fails to converge to point that effectively optimizes both. This completes the proof of Theorem
A.3 Derivation of Objective
Starting from Equation 13 the main paper, the loss function for the reward function r(-) can be expressed as:
C, -- th log (r((()())))) - - rxx)))
(49)
Reinforcement Learning from Human Feedback (RLHF) aims to maximize the distribution po under regularization using KL-divergence:
max Ez r(x)) - -Dr..sDDeelllll...ccc
(50)
pa
where ø is reference distribution. From Equation (50). we can obtain a unique solution p (zo):
(x0) (20) exp(r exp(r(x))/3)/z,
(51)
where Z E Pa (z))
p=(=)))xp(((z))) exp(r(2x)/)) is the partition function. The reward function can be re-
written using Equation (5I
PT (ro) log Z.
r r(zo) B log
(52)
P& (2o)
From Equation (49) and Equation
52 the reward objective is:
C -E, . [log (3 log pT(x0)
Blog Pd (%) 23. p.(x))
(53)
ps(x))
However, this objective cannot directly applied to diffusion models since the parameterized distri- bution pe (zo) is intractable. Therefore, Diffusion-DPO (36) introduces the latents .1. to consider possible diffusion paths from ær to xp. and re-defines the reward function as follows:
r(xo) (p R(xo))
(54)
Following Equation
54 Equation 500 can also be written as follows:
max E
Fa. (urrrp((rr)))
((00) 3D (po(((o))))))lpooeroo
(55)
Similar to the expansion from Equation 50 to Equation (53) we can obtain the reward objective as:
pá ( (
Pă (T )H.
C. = -Eat wp [log o ( polttt po (3 log
3 log
(56)
Po (T()TT)
Since - log (-) is a convex function, we can leverage Jensen's inequality:
L, < -Egt mh po pot (mt) (e
Pa (((T)) B Bog Pà Po( po()).))
(57)
[log o log p
Note that (0 (m)) is intractable. Therefore, we utilize q( (m)) to approximate po(eerlzo)
C, < - E pa lab I flgg a ( p5 ()(Tr)
p5 (
(58)
Po (
pe (
17

---

Since pe (z = ( [T p can be expressed as Markov chain, we can derive the above equation as:
C S -B od 50 a0ttrrl ).
Po (at 11:
PD ( [ Pol Po( 20
o
log
log
Pel (0(x1111077
1=1
= = to .00 |= ).((xi:))
M6 (1 |
Po p |
logg o fs
(log
log
q 12
q |x,
Pe p=
Po po 1//
(log g 1
log
(59)
=- " to ala (m
log {B Σ (D (DLl(((atti |
Dru.olleiiilttt)poooccctti )))
- 1x }))
By leveraging ELBO, we can obtain our final objective, Equation the main paper:
Loro = = Bd 2a ,.1,,~
log (-= t, c) - cl
||<a(æT t, e12
(60)
(1lea(17 (, <11
|lao(o,,t, ell2))).
250
A-100
==5.0
10.0
15,5.0
APDM (Ours)
FID
100
50
100
200
300
400
500
600
700
800
Iterations
Figure 4: FID variation during the training with Cade- We measured the image quality via FID score 10 on COCO 2014 2 validation dataset. We also plot the FID score of Stable Diffusion 1.5 and APDM.
18

---

Table 10: Quantitative comparison on protection with image transformations. We compared APDM with transformed images. For data poisoning baselines, we applied image transformation to perturbed images and we personalized Stable Diffusion on these transformed images. For APDM, we protected diffusion models on clean images and we conduct personalization on images that is transformed from clean images.
DINO (J)
BRISQUE (t)
Methods
Transform.
"person" "
"dog"
Avg.
"person" "
"dog"
Avg.
DreamBooth
31
0.6994
0.6056
0.6525
11.27
22.33
16.80
0.5752
0.4247
0.4999
19.52
28.60
24.06
AdvDM 18
nip
0.5436
0.4538
0.4987
24.37
27.07
25.72
blur
0,6417
0.4524
0.5470
18.28
26.35
22.32
0.5254
0.4106
0.4680
26.90
30.23
28.56
Aoti 34
flip 0.5976
0,4665
0.5321
26.76
29.19
27.97
blur
0.5487
0.4414
0.4951
24.37
28.91
26.64
0.4448
0.4374
0.4411
23.73
31.64
27.69
SimAC 38
flip
0,5083
0.4475
0.4779
26.56
29.46
28.01
blur
0.5323
0,4390
0.4856
20.40
31.27
25.83
"0.6356
0.5120
0.3838
22.61
30.20
26.41
PAP 37
nlip
0.6564
0.5139
0.5852
22.51
27.81
25.16
blur
0.6708
0.5222
0.5965
24.37
27.83
26.10
0.1375
0.0959
0.1167
40.25
60.74
50.50
APDM (Ours)
nip 0.1714
0.1194
0.1454
39.13
40.34
39.74
blur 0.1042
0.0823
0.0933
40.47
45.13
42.80
Table 11: Protection performance on other subjects. In addition to experiments in the main paper, we evaluated APDM different subjects. We tried prevent personalization on "cat", "sneaker". "glasses,, and "clock".
DINO (1)
BRISQUE (1)
Methods
"cat"
"sneaker"
"glasses" "
"clock"
"cor"
"sneaker"
"glasses"
"clock"
DreamBooth [3]
0.4903
0.6110
0.6961
0.5359
25.32
23.14
19.01
13.82
APDM (Ours)
0.0414
0.2276
0.2893
0.1969
47.65
335.23
31.75
32.01
B Additional Experiments
Empirical Results about the limitation of Naïve Approach. In Section A we theoretically
demonstrated the fundamental limitations of naïve approach. In the following part, we empirically validate those findings. We applied the loss function of naïve approach, Lade (Equation (2T), ) to Stable Diffusion 1.5 with N Nprotect 800, as APDM. We measured FID every 100 iterations. As shown in Figure 4 as the optimization progresses, the FID score consistently increases across all tested À values. This degradation in quality occurs because the primary objective of Ladvo minimizing simple (i.e. actively erasing related to the target for anti-personalizzation) becomes overly dominant, Even though Lppt is intended to preserve the generation performance, its effectiveness clearly restricted by the optimized condition of -L This result aligns with our Theorem 1 which suggests that the loss of each term in Lade cannot be satisfied simultaneously. Furthermore, when the weight X increases, one might expect a better preservation of the generative performance. Although FID scores relatively low with high À values (e.g. À 10.0, 15.0) in initial iterations. they still remain significantly high and can exhibit instability as training progresses. This suggests that our theorem is still valid in various X.
Protection with Image Transformations.
In Figure and Table of the main paper, we compared APDM with baselines considering the existence and quantity of clean images. Additionally, we also compared APDM with baselines using transformed images such as flipping and blurring. Table TT) demonstrates that baselines fail effectively protect personalization when transformations are applied to perturbed images. In contrast, APDM exhibits robustness even under such image transformations.
19

---

Training Images
Training Images
Training Images
Training Images
Figure Protection on other subjects. We attempted to protect personalization on "cat", "sneaker", "glasses", and "clock".
Protection Other Subjects. In the experiments presented in the main paper, primarily considered two types of subjects: "person" and "dog". In Table and Figure we explored the prevention of personalization other subjects, such as "cat". "sneaker". "glasses", and "clock", demonstrating that APDM can be generally applied to protection of various subjects.
C Generalizability of APDM
Custom Diffusion. In our main paper, we mainly consider DreamBooth [31 as a personalization method. Additionally, we utilized Custom Diffusion [5] as variation of the personalization approach. In T2 we present the results of the Custom Diffusion experiments, and we conducted protection about "person" and "dog" similar to main paper. The results demonstrated that
20

---

Table 12:
Protection performance APDM on different personalization method, Custom Diffu-
sion [1]. Unlike the experiments in the main paper, which used DreamBooth for personalization, we replaced the personalization method with Custom Diffusion.
DINO (4)
BRISQUE (1)
Methods
"person"
"dog"
"person"
"dog"
Custom Diffusion
0.5320
0.5460
16.03
APDM (Ours)
0.2158
0.3202
34.61
33.09
Table 13: Protection performance of APDM on different Stable Diffusion version, Stable Diffusion 2.1. In the experiments of the main paper. we primarily used Stable Diffusion 1.5. Additionally, we evaluated APDM based on different Stable Diffusion version.
DINO (1)
BRISQUE (t)
Methods
"person"
"dog"
"person" "
"dog"
DreamBooth 31
0.5773
0.5293
13.99
23.03
APDM (Ours)
0.2739
0.2178
39.72
42.69
Table 14: Protection performance on different unique identifier for personalization. We con- ducted protection "a photo of sks person" "a photo sks dog" and tried personalize diffusion models on "a photo of 1@ person" " or "a photo of i@1 dog".
DINO (1)
BRISQUE (t)
Methods
[V]]
"person" "
"dog"
"person"
"dog" "
DreamBooth
"t@t"
0.6774
0.4668
16.64
28.49
APDM (Ours) ""ks"" "
0.39588
0.198I
29.90
40.69
Table 15: Protection performance for diverse text prompts. Unlike the experiments in the main paper, we evaluated APDM on diverse test prompts. Protection and personalization are conducted using "a photo of [V*] person" "a photo of [V*] dog". and we sampled images using the different set of text prompts.
DINO (L)
BRISQUE (1)
Methods
"person" "
"dog"
"person"
"dog"
DreamBooth
31
0.4081
0.4233
12.57
29.65
ÄPDM (Ours)
0.1357
0.1564
36.40
41.66
APDM can successfully prevent the personalization of Custom Diffussion, and show the applicability of APDM other personalization methods.
Stable Diffusion 2.1. APDM prevents personalization at the model level, and its applicability to different versions of the Stable Diffusion model is also important. In Table we present experiments conducted on Stable Diffusion 2.1 to demonstrate the effectiveness of our approach other diffusion models. We applied APDM Stable Diffusion 2.1 and performed personalization with clean images using DroahBooth. The results indicate that APDM also performs robustly on Stable Diffusion 2.1, showing that our method is not restricted to a specific version of the diffusion model.
Prompt (Identifier) Mismatch. When an attacker performs personalization, they may use different unique identifier (e.g. "?@r") to capture the target subject. For example, during the protection process, we only show "a photo of sks person" ", while a different unique identifier may be used for personalization, such as "a photo i 1 person". Similar to Van Le al. 34 we also
considered this prompt mismatch. As shown in Table 14 APDM can successfully protect against personalization attempts using "I@I". APDM successfully confuses the personalization process, preventing the identifier from capturing the target subject (i.e. identity).
21

---

Training Images
a [V*] person in the snow"
"a [V*]] with a mountain in the background"
aa [V*]] person wearing a red hat"
"a [V*] person wearing a pink glasses"
Figure 6: Protection performance for diverse text prompts. We visualized the generated outputs from diverse text prompts, such as "a [V*] person in the snow" and "a [V*] person wearing red hat".
Protection on Diverse Text Prompts. In the experiments presented in the main paper, we utilized simple text prompts for inference, such as "a photo of [V*] person" and "a portrait of [V*] person." In contrast these experiments, we evaluated APDM using diverse prompts, such as "a photo of [V*] person in the jungle" and "a / [V*] person with a mountain background." We adopted text prompts from the DreamBooth dataset [3I]. Figure and Table ililutttee that APDM successfully prevents personalization, even under diverse prompt variations that differ from the text prompts used during the protection procedure. This result highlights that APDM even robust to diverse text prompt variation.
22

---

Drstasding Pemensttrtio Profectin thefeenes det
the secorn you toe with uce meterce magge
aee maggr bibnted These resggton
to meneo and cemaaaae sages
alere quediio
Firs conaaaly ideans sete nages pio These di traniginal
coar cus megee J4.
of of gei ja. 00 HORDETT resagllle MAM I...E
vil ing Tegiiiic
Figure 7: A sample interface for our user study. Left term is the descriptions of explanation about study. Middle term is given reference images which used to capture the identity from participants. Right term is choices.
Table 16: Results of user study. We count the percentage of votes for the comparisons and our method respectively. Every participants selected a sample that looks most different from the clean images.
Methods
Anti-DreamBool
34
SimAC
(38
PAP 37
APDM (Ours)
Protection
7.08 %
5.83 %
1.04 %
86.04 %
D User Study
We conducted user study to evaluate the preference of various protection methods in preventing subject recognition. The specific questions and interface are illustrated in Figure 7 We presented four reference images for each subject to provide participants with clear identity information. After viewing these, each participant chose image based on the following question:
Which of the candidate images (A, B, C, D) is the HARDEST to recognize depicting the SAME PERSON shown in the reference images?
Candidate images were generated using a personalized diffusion model with different protection methods applied. We utilized Stable Diffusion 1.5 and Dreambooth [1 as personalization method, which the same as our experimental setting in main paper. In this user study, we compared proposed method, APDM, against Anti-Dreambooth (34), SimAC (38), and PAP [37]. For comparisons, we first generated perturbed images using each approach, and conducted personalization with these perturbed images. For APDM, we applied personalization using the model protected by our method. After personalization, all images were generated using the prompt "a photo of [V*] person". To ensure faimess, the same randomly sampled seed used for generating all candidate images. The image sequence and the arrangement of choices are randomized to eliminate any bias.
We collected responses from 25 voluntary adult participants regardless of gender. Participants were compensated 125 USD per question, totaling $2.50 USD, corresponding hourly rate of $7. $7.26 USD. On average, participants completed the study in about 20 minutes. We did not collect any personal information from the participants.
As shown in Table 16 APDM achieved significantly higher user preference (i.e. was selected more often as the hardest to recognize) than other comparisons. These results indicate that our method not only addresses limitations of data-centric approaches but also achieves a substantial improvement in protection performance.
E
Additional Qualitative Results
Additional Protection Results. In Figure and Tf the main paper, we conducted quantita- tive and qualitative experiments, respectively. We attached additional qualitative results Figure and Figure 9 including protection results various subjects of person and dog. The experimental results highTight again that APDM can effectively protect personalization against diverse subjects, producing images of lot of artifacts or containing different instances.
23

---

Training Images
Training Images
Training Images
Training Images
Figure 8: Additional Qualitative Results Protection ("person"), "
24

---

Training Images
Training Images
Training Images
Training Images
Figure 9: Additional Qualitative Results on Protection ("dog").
25

---

F Additional Explanation of Motivation
Figure I in our main paper presents the motivation for our work and briefty describes key issues, including the impractical assumptions of existing approaches, easy circumvention, user burdens, and conflict with regularization. This section provides a more detailed explanation of these limitations to facilitate clearer understanding.
We first criticize the impracticality of the existing literature. In daily life, individuals frequently take pictures or are photographed. For example, they often take selfies for social media or capture images of their identification documents. Such images, which we refer to as "User's 's Photos" " (as depicted in Figure of our main paper), are those that users are consciously aware of and possess. Consequently, users have the opportunity to apply protection methods (i.e. data poisoning approach) to these photos if they want. In contrast, "Unintended Capture" refers to images of individuals taken without their explicit recognition or control over their subsequent use. This scenario presents a critical vulnerability, as these unintentionally captured images can be exploited as unprotected, "clean" data by malicious users.
As shown in Table of our main paper, the presence of clean (unprotected) images can significantly degrade the effectiveness of data poisoning techniques, allowing for easy bypass of protection. Furthermore, even when images are perturbed (i.e. poisoned), their protective effect is vulnerable to various common image transformations that frequently occur in real-world scenarios (as also shown in Table TOP. These transformations can weaken negate the intended poisoning effect. These limitations reveal that, without strong (and often impractical) assumptions about the unavailability of clean images or the absence of transformations, existing protection methods exhibit restricted performance.
Regarding the user burden associated with implementing such techniques, most individuals are unfamiliar with implementation of AI technique. Establishing appropriate hardware environments (e.g. GPU servers) and configuring complex software environments (e.g. managing numerous libraries and their dependencies) present a significant initial hurdle. Even if these challenges are overcome, non-expert users still face substantial obstacles in utilizing protection methods. These include lack of fundamental understanding of the protection mechanisms themselves, insufficient understanding in necessary programming languages (such as Python), and inadequate debugging skills to troubleshoot issues. These technical components are crucial for successful implementation of protection methods, yet their complexity also acts significant barrier, preventing widespread adoption by the general public.
The user-centric nature of existing data poisoning methods inherently conflicts with privacy regula- tions such as the General Data Protection Regulation (GDPR) [35]. The GDPR places the duty for privacy protection on service providers (i.e. model owners) to ensure user's 's request. However, data poisoning approaches are ill-suited for service providers fulfill this responsibility. These methods typically operate at the individual image level, requiring modifications to user data before they interact with the model. Service providers, in contrast, primarily manage the model itself. This operational disparity highlights why such user-side defenses are impractical for providers, underscoring the critical need for alternative approaches. To alleviate this, we propose novel framework APDM, which empowers service providers to effectively manage and enforce antippersonalization directly within their systems, aligning with their responsibilities under privacy regulations and enabling a more scalable and reliable means of privacy protection.
G Limitation and Broader Impacts
In this work, we focused on protecting the personalization of specific subject at the model level. APDM offers a significant step towards more robust and practical privacy protection in personalization of diffusion model. By enabling direct, model-level anti- personaliizaiinn, it empowers service providers to better comply with privacy regulations and reduces the burden on individual users to protect their own data. This could foster greater trust and safer use of powerful generative models in various applications.
While APDM effectively safeguards personalization for a single subject, real-world scenarios often require the protection of multiple subjects simultaneously. Additionally, there may be a need to incorporate protection for new subjects into models that are already safeguarded. Addressing these challenges presents an opportunity for future research, including multi-concept personalization protection and continual personalization safeguarding.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly summarize our main contributions in Section D our main paper. The abstract and introduction further present our core insights, key findings, proposed methodologies, and highlight the significant advantages of our approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the Supplementary Material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification:
We describe detailed derivations of each component the Supplementary
Material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the details of each component, such as dataset, environment setup, parameter setting, version of SD, and any other details, in 5.11ion/511p in our main paper to ensure the reproducibility our experiments. Furthermore, we also represent the ablation results for various syper-parameters.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We did not any private data. Therefore, all data used in this paper is publicly accessible. Our code not yet publicly available but will be released upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
publicc/guide///deeattmi httpsi//anaaaeeeeeeeeeueisssssoo www.aaammmiioioooooooooooooeeiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips /nipsscc/puut
guides PwoddSSumiissioooool....oooooooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide the details in Section in our main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Although standard deviations not explicitly reported in tables, all presented results represent the average performance across multiple sets to ensure reliability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the environments setup, such as GPU, in Section 5.] in our main paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broad impacts of our work in the Supplementary Material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We consider the task about safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the different methods, models, and data used basclines in our experiments.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We propose novel framework and loss functions for anti anti-personalization. Furthermore, we also provide the details for training setting.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre /0tps//eec.o0oeeeeeee////2////// for what should or should be described.
33


Reviews:
[{"review_id": "gHHdbdeBYo", "reviewer": "Reviewer_3bmf", "summary": "The paper attempts to provide protection against personalization by focusing on the model than on poisoning the data. So the goal is to obtain a protected model which will be unable to apply personalization on clean images without the need of poisoned data. First, it shows a simple modification to the current loss function used for training models is not suitable for this purpose. A new loss function along with a novel optimization algorithm is provided which ensures that the model weights are modified so that it will not be able to personalize the targeted concept but will work properly for non-targeted concepts.  Thereby, achieving the goal of obtaining a protected model which should not be able to apply personalization to a fixed entity, but it should still be able to retain its generative capability and personalize other entities.", "questions": "- Assuming the user (instead of self applying protection as assumed in Supplementary L217-226) has access to service provider who returns protected images by watermarking, then what would be the strong reason for the approach of perturbing the model followed in this paper.\n\t- Can the authors provide a clear computation study with a simple case comparing data poison vs model change approach to observe the advantage of one over the other?\n- In the paper APDM's results are presented only with clean images, can the result with perturbed images also be shown.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Shift the focus of protection against personalization by text-to-image diffusion model from data level poisoning to model level change.\n- Provide a proof explaining why the a naive change of direction of the Reconstruction loss in the total loss function used in DreamBooth (training loss comprises of Reconstruction loss and Class-Specific Prior Preservation loss) will not lead to convergence of the optimization function.\n\t- Provide an alternate loss function: Direct Protective Optimization (DPO) loss for personalization protection while preserving generation capability.\n\t- Provide a dual-path optimization strategy (Learning to Protect (L2P)) to enhance the optimization of the new loss function.\n- The empirical results based on few subjects are shown to work well on the clean images, thereby supporting the claim that the protected model can be trained with effective protection to the concept.\n- The method is shown to work even with different prompts.\n\n### Weaknesses\nConceptual:\n- A major concern is with the main approach of perturbing the model than the data. L135: Clearly states that the adversary needs to use the $\\hat{\\theta}$ model with the X which is impractical. The adversary scraping the internet and accessing the poisoned data is more realistic than enforcing the adversary to ONLY use the altered model which seems unrealistic.\n\t- L48-50: Instead of changing the model, the service provider could add the protective watermark and return it to the user. \n\t\t- Many of the concerns raised in Supplementary F could be circumvented by the service provider watermarking the data which could further prevent other models from personalization.\n\t- There needs to be solid supporting study on the computational expense comparing data poison vs model change to support this work's approach even for the few concept case studied in this paper.\n- Currently this paper appears to be work in progress as the results are with one or two concepts while multi-concept is considered future work.\n\nTheoretical:  \n- Notation wise equations (2) and (3) appear to be the same. Equation (3) does not reflect the inclusion of a new text token for personalization. Other than a change in the loss symbol $\\mathcal{L_{simple}}$ or $\\mathcal{L_{simple}^{per}}$, the expression on the right appears to be the same. Are the notations fine or any change needed?\n\nEmpirical:\n- To make the empirical results stronger new methods like ACE[2] and CAAT[3] need to be included to obtain results in Fig 3 and Table 1.\n- While results for APDM are provided with clean images, how the model responds to perturbed data either adversarial or gaussian noise needs to be included.\n- For the perturbation/data poisoning based methods, an important parameter of perturbation budget is currently missing in this paper. What was the noise budget used to conduct these studies?\n- There is quantitative study in Table 3, can the paper provide qualitative results with subjects not protected, to show personalization of other subjects is still possible and the generated images are visually good?\n\nWriting:  \n- Recent works [1], [2] need to be included to keep the literature updated.\n\nReferences:  \n[1] Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI - Robert Hönig, Javier Rando, Nicholas Carlini, Florian Tramèr - ICLR 2025  \n[2] Targeted Attack Improves Protection against Unauthorized Diffusion Customization - Boyang Zheng, Chumeng Liang, Xiaoyu Wu - ICLR 2025  \n[3] Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models - Jingyao Xu, Yuetong Lu, Yandong Li, Siyang Lu, Dongdong Wang, Xiang Wei - CVPR 2024", "was_revised": true, "final_justification": "I am supportive of the paper's approach about focusing on the model than the data, which is different as largely efforts are on data poisoning/watermarking. Despite finding the current results as a work in progress as the paper's main results are with single concept and multi-concept is considered as future work, willing to support the approach. \n\nHoping the paper is updated with experiments to keep it up to date with the current best methods (eg. using recent methods like ACE and CAAT, along with lower noise budget) and incorporates all the relevant suggestions by the reviewers.\n\nI am raising the score from 3 to 4.", "rebuttal": "We appreciate Reviewer 3bmf for your constructive feedback. We hope our response effectively addresses your concerns and questions.\n\n# W1: Necessity of model-level protection\n\nIt is important to clarify who is responsible for this privacy risk. Under regulations such as GDPR, **the service provider—not end users—must guarantee protection** for their service model. Crucially, providers already control the life‑cycle of the model and they can control the risk by updating their model. In this circumstance, the model-level protection is a highly practical solution. \n\nEven if your suggestion (i.e., watermarking) can solve the user burden, it still has some limitations: (i) **newly uploaded content** from other sources remains unprotected, and (ii) **simple transforms can weaken the watermark**, reviving the privacy risk (Bypass issue).\n\nThe concerns in Supplementary F (and illustrated in Figure 1) are motivated by the above problems. In those part, we identify four key problems, and service-provider-side solutions like watermarking fail to address three of them. As we mentioned above, watermarking has a limitation about (i) and (ii). Furthermore, another critical problem also arises in the service phase: the provider has no way of knowing which user-provided data requires protection (*conflict with regulation*). In other words, if malicious users use clean images, the watermarking approach cannot handle this. For this reason, with watermarking, the service provider cannot fulfill their responsibility to protect sensitive data.\n\nThis analysis underscores the necessity of a model-level approach like APDM. However, this does not render data perturbation methods obsolete. As you mentioned, methods like watermarking are effective in out-of-service cases. However, they cannot fully address the protection challenges, such as the *new content* and *bypass* problems, which APDM is designed to solve. Therefore, a complementary deployment of data‑level and model‑level protections is the most efficient and robust strategy.\n\n# W2: Discussion about computational cost\n\nTo provide a solid analysis of the computational expense, we compare our approach with existing methods by addressing the fundamental difference in their computational complexity and providing quantitative results.\n\nThe core difference lies in the scaling properties. APDM requires only **a fixed cost (9 GPU-hours)** for protection, which is independent of the number of images to be protected. Therefore, its complexity is $O(1)$. In contrast, poisoning approaches require processing each image individually. While the per-image cost is relatively low, the total cost scales linearly with the number of images ($N$), resulting in an $O(N)$ complexity.\n\nTable R1. Time consumption.\n|Method|AdvDM|Anti-DB|SimAC|PAP|\n|---|---|---|---|---|\n|Time|262s|288s|594s|297s|\n\nAs shown in the Tab. R1, existing poisoning approaches need nearly 5 minutes to process their methods. This allows for a direct computational cost comparison. **APDM becomes more cost-effective when the number of images ($N$) exceeds 110**. In real-world applications, such as social media platforms or cloud services where users continuously upload new photos, $N$ is not only large but also constantly growing. In such scenarios, APDM is significantly more efficient in terms of overall cost. We will add a detailed discussion on this computational complexity analysis to the final manuscript to clarify this point.\n\n# W3: Multi-concept protection\n\nTable R2. Multi-subject protection.\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.5833|20.32|\n|PAP|0.5699|25.72|\n|**APDM**|**0.3924**|**32.97**|\n\nAs noted in Supplementary G, we aim to extend our work to multi‑concept protection scenarios. Motivated by your comment, we conducted a preliminary experiment to test the robustness of our current APDM on a multi-concept task. For this experiment, we constructed a multi-concept negative set ($x^-_0$) by combining samples from four distinct subjects (cat, sneaker, glasses, clock) and applied our **APDM without any modification**, treating them as a single conceptual group. For comparison, we reported the results of PAP using all perturbed data (the \"# of clean images: $0$\" setting in our main table) for the evaluation.\nAs shown in Tab. R2, **APDM still provides substantial protection in the multi-concept setting** compared to PAP. This demonstrates the inherent robustness of our approach, even for a task it was not originally designed for, and highlights the potential of APDM for future work.\n\n# W4: Eq. (2) and (3)\n\nWe thank you for this suggestion. While we initially adopted the notation from the original DreamBooth paper for consistency, we agree that Eq. (3) can be improved. We will revise Eq. (3) in the final version for better clarity by explicitly representing the personalized text token ($c_{per}$).\n\n# W5: Additional comparisons\n\nTable R3. Additional comparisons.\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.6162|10.77|\n|ACE|0.6846|28.18|\n|CAAT|0.5825|22.71|\n|**APDM**|**0.0559**|**45.87**|\n\nWe present additional results in Tab. R3. We report the results for the setting that contains a single perturbed image (the \"# of clean images: $N-1$\" setting in our main table). The results show that **APDM still outperforms these stronger competing methods**.\n\n# W6: Response to perturbed data\n\nTable R4. Personalization with perturbed data.\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.6869|16.69|\n|Anti-DreamBooth|0.5646|22.50|\n|APDM|0.1375|40.25|\n|APDM (perturbed)|0.1702|40.20|\n\nTo answer this question, we conducted an additional experiment. We generated perturbed data using the Anti‑DreamBooth method. As shown in the Tab. R4, **APDM is robust to input data, even when the data are perturbed**. Furthermore, APDM with perturbed data still outperforms the original Anti-DreamBooth with a large margin, even when Anti-DreamBooth utilized all perturbed images (the \"# of clean images: $0$\" setting in our main table). This demonstrates the powerful protective performance of APDM. \n\n# W7: Noise Budget\n\nAs mentioned in Line 225 (Experimental Setup) in our main paper, the **perturbation intensity (noise budget) was set to 5e-2**. We selected this value based on the Anti-DreamBooth paper, and for a fair comparison, this value was applied to all baselines. \n\n# W8: Qualitative results in Table 3\n\nTable R5. Additional quantitative results.\n|Method|CLIP-I(&uarr;)|CLIP-T(&uarr;)|\n|---|---|---|\n|DreamBooth|0.7576|0.2374|\n|APDM|0.7961|0.2329|\n\nUnfortunately, we are unable to provide qualitative figures due to the rebuttal policy. However, we argue that Table 3 in our main paper already provides strong evidence to address this concern. The DINO score is a key metric that directly evaluates the success of personalization. The fact that APDM outperforms DreamBooth on this score indicates that APDM both (i) successfully personalizes the unprotected subjects and (ii) avoids over‑fitting to the protected target. Because APDM achieves a higher score, any potential degradation in quality is minimized. Furthermore, as shown in the Tab. R5, APDM also achieved competitive scores on both CLIP‑I and CLIP‑T, indicating that its outputs remain well-aligned in both the image and text domains. We hope this explanation resolves your concern. \n\n# W9: About Recent works (Writing)\n\nThank you for the suggestions. We will update the camera-ready version to include these recent related works as follows:\n- **[1]** investigates the vulnerability of existing protection methods. We will cite this work to **provide stronger support for our problem definition**, particularly for the *Bypass* problem.\n- **[2]** proposes a targeted attack approach. As we have already demonstrated in Tab. R3 of this rebuttal, this method's effectiveness is still limited in our scenario. We will incorporate [2] as an advanced baseline in our comparisons to further highlight the robustness of our method.\n\n[1] Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI - Robert Hönig, Javier Rando, Nicholas Carlini, Florian Tramèr - ICLR 2025 \\\n[2] Targeted Attack Improves Protection against Unauthorized Diffusion Customization - Boyang Zheng, Chumeng Liang, Xiaoyu Wu - ICLR 2025"}, {"review_id": "e7oVwNAfol", "reviewer": "Reviewer_gqhK", "summary": "This paper proposes APDM to prevent the model from personalization, thus protecting the privacy of personal images. The optimization of APDM is conducted by the introduced Learning to Protect process, which iteratively optimizes the parameters of the protected diffusion model. During the protection, the Directed Protective Optimization loss is applied to maximize the distance between negative samples of the original model and the protecting model while minimizing it between positive samples. The DPO loss is combined with a ppl loss to preserve the general performance of diffusion models, constructing the protection loss to update the model parameters. Results show that APDM can not be personalized with the protected identity, even when all training images are clean, while protection methods that perturb images failed to provide successful protection.", "questions": "See weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "#### **Strength**\n1. The paper provides a novel solution to protect the privacy of personal images, which could give new insights into the privacy/copyright protection issues, as the previous perturbation-based protection lacks practicality in real-world scenarios.\n2. The results of the protection performance are strong. APDM is capable of preventing personalization with all clean images while almost retaining the original performance.\n#### **Weakness**\n1. The robustness of APDM is not explored. The paper assumes 4-6 images like DreamBooth, but determined adversaries might use 100+. Is the protection still effective under these vastly larger subject sets? \n2. The assessment on preserving personalization capabilities focuses on different subjects; it is unclear whether it can personalize to similar subjects of the same categories, e.g., the protected subject and the personalization subject are both male persons, but the identity is different.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer gqhK for the positive feedback. We are glad the reviewer recognized both the novelty of our practical approach and the strength of its empirical results. We address gqhK's comments and questions below.\n\n# W1: Using 100+ images for personalization\n\nTo directly address this concern, we conducted an additional experiment using a much larger dataset of 100 images for personalization. For this experiment, we manually collected a dataset of 100 images of a public figure from various online sources$^{[1]}$. We then evaluated APDM against DreamBooth and PAP in this setting. The results for PAP were generated using all 100 perturbed images, which corresponds to the most favorable configuration for PAP (the \"# of clean images: $0$\" setting in our main paper).\n\nTable R1. Personalization with 100 images.\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.7983|1.06|\n|PAP|0.7811|5.96|\n|**APDM**|**0.3747**|**35.01**|\n\nThe results clearly demonstrate that **APDM remains highly effective even in this more challenging \"100-image\" scenario**. APDM's DINO score (0.3747) indicates significantly stronger protection compared to both unprotected DreamBooth (0.7983) and PAP (0.7811). Furthermore, the performance gap over PAP is even more pronounced in the BRISQUE (35.01 vs. 5.96). This confirms the superior robustness of our method.\n\n[1]: To mitigate potential privacy and copyright concerns, this collected set was used exclusively for this rebuttal experiment. The data was handled securely and has been permanently deleted after use.\n\n# W2: Personalization on the same categories\n\nTo answer this question, we represent results for personalizing subjects from the same categories (person &rarr; person). In this experiment, we protected one female subject and personalized three other distinct female subjects (ID1-3). We report quantitative results following the evaluation protocol of DreamBooth. \n\n(Training ID and ID1-3 are white-female persons, but the identities are different.)\n\nTable R2. Personalization on the same subject.\n|Method|DINO-ID1|DINO-ID2|DINO-ID3|CLIP-ID1|CLIP-ID2|CLIP-ID3|\n|---|---|---|---|---|---|---|\n|DreamBooth|0.6016|0.6471|0.6672|0.6602|0.6919|0.6611|\n|APDM|0.6303|0.6383|0.6877|0.7134|0.6943|0.6460|\n\nAs shown in the Tab. R2, **APDM effectively maintains high personalization performance even for subjects within the same category (female &rarr; female)**. The DINO and CLIP scores are comparable. This demonstrates that our method successfully preserves the model's utility for personalizng new, similar subjects while protecting the target identity."}, {"review_id": "P5gfk4MmPW", "reviewer": "Reviewer_qkFB", "summary": "This paper introduces the Anti-Personalization Diffusion Model (APDM), a framework designed to prevent the unauthorized generation of specific individuals' images. Instead of protecting specific images, APDM aims to protect the diffusion model, making it inherently resistant to personalization. To achieve this, the framework utilizes a novel loss function called Direct Protection Optimization (DPO) to block personalization without harming the generation quality of the model. Moreover, they propose an adaptive training strategy, Learning to Protect (L2P), to strengthen the protection performance in potential personalization. Extensive experiments show that this work achieves state-of-the-art protection performance.", "questions": "Please see the weakness part.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation is reasonable and novel. Unlike methods that fail with clean images or simple transformations, this framework's model-level protection is inherently more robust and reliable for real-world use.\n2. The proposed DPO loss function and L2P optimization strategy are interesting and effective.\n3. Extensive experiments demonstrate that APDM achieves state-of-the-art performance in preventing unauthorized personalization.\n4. This paper is well-written and easy to follow.\n\nWeaknesses:\n\n1. This paper lacks a comparison of protection efficiency. Given that the method requires training a diffusion model, the computational cost could be a significant concern. The 9-hour training time per identity makes the approach impractical for large-scale protection. A discussion or a comparison of the efficiency against other methods is needed.\n2. The comparison should be expanded to include recent protection methods, such as Mist [1] and SDS [2]. It is better to include more baseline methods in Table 1.\n3. The scalability of APDM is a concern. What if protecting multiple identities simultaneously?  An experiment is needed to evaluate its performance when protecting multiple identities simultaneously.\n\n[1] Liang, Chumeng, and Xiaoyu Wu. \"Mist: Towards improved adversarial examples for diffusion models.\" arXiv preprint arXiv:2305.12683 (2023).\n\n[2] Xue, Haotian, et al. \"Toward effective protection against diffusion-based mimicry through score distillation.\" The Twelfth International Conference on Learning Representations. 2023.", "was_revised": true, "final_justification": "Thanks for the detailed rebuttal. My questions have been addressed so I decided to keep my rating.", "rebuttal": "We appreciate Reviewer qkFB's positive evaluation about our motivation novel, our methodology effective, and our experimental results. We hope our response addresses your concerns and questions.\n\n# W1: Discussion about protection efficiency\n\nTo address this concern, we evaluate our approach against baselines based on both **theoretical complexity** and **practical cost**.\\\n(**Theorectical Complexity**) The primary distinction lies in their scaling properties. APDM requires a one-time, fixed training cost (9 GPU-hours), making its computational complexity $O(1)$ as it is independent of the number of images to be protected ($N$). In contrast, data poisoning methods require per-image processing, leading to an $O(N)$ complexity where the total cost grows linearly with $N$.  \\\n(**Practical cost**) The real-world implications of this difference are significant. The Tab. R1 shows the processing times of existing methods:\n\nTable R1. Time consumption.\n|Method|AdvDM|Anti-DB|SimAC|PAP|\n|---|---|---|---|---|\n|Time|262s|288s|594s|297s|\n\nAs the Tab. R1 illustrates, they require approximately 5 minutes to poison the data. This allows for a direct cost-benefit analysis: **APDM becomes cost-effective once the number of images ($N$) exceeds 110**. In real-world applications driven by user-generated content like social media, where image sets are not only large but continually expanding, the $O(1)$ scalability of our approach offers a substantial advantage in overall efficiency. We will add a detailed discussion of this complexity analysis to the final manuscript.\n\n# W2: Additional comparisons\n\nAs requested, we conducted additional experiments comparing APDM with the suggested methods, Mist and SDS. We report the results on the containing a single perturbed image setting (the \"# of clean images: $N-1$\" setting in main table).\n\nTable R2. Additional comparisons.\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.6162|10.77|\n|Mist|0.5903|30.74|\n|SDS|0.5194|20.03|\n|**APDM**|**0.0559**|**45.87**|\n\nOur findings reveal two crucial points:\n\n(**Shared Vulnerability**) In this setting, the new baselines are also highly vulnerable. This confirms that the presence of clean images is a critical challenge for existing data-level protection methods.\\\n(**Superior Robustness**) APDM continues to significantly outperform all baselines, demonstrating its superior robustness again.\n\n# W3: Multiple identities scenario\n\nWe agree with the reviewer that protecting multiple identities is an important direction, and we have noted this as future work in Supplementary G. However, to proactively explore this, we conducted a preliminary experiment to assess the robustness of APDM in such a scenario. For this, we constructed a challenging multi-concept negative set ($x^-_0$) with four distinct subjects (cat, sneaker, glasses, clock) and applied our method **without modifications**. For comparison, we evaluated PAP using its most favorable configuration (the \"# of clean images: $0$\" setting in our main paper).\n\nTable R3. Multi-subject protection.\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.5833|20.32|\n|PAP|0.5699|25.72|\n|**APDM**|**0.3924**|**32.97**|\n\nThe results demonstrate two key aspects of our method:\\\n(**Effective Protection**) APDM maintains substantial protective capabilities, achieving a DINO score of 0.3924—far superior to both unprotected DreamBooth (0.5833) and data poisoning approach PAP (0.5699).\\\n(**Remarkable Robustness**) This strong performance was achieved with an unmodified APDM, showcasing its inherent ability to generalize its protective mechanism to a task of greater complexity.\n\nTherefore, this experiment not only confirms the **strong potential** for multi-identity protection but also validates the **robustness of its core design**. We believe this serves as an excellent starting point for developing a dedicated multi-concept protection method in our future work."}, {"review_id": "YFZxm3Ts03", "reviewer": "Reviewer_y2uu", "summary": "This work proposes Anti-Personalized Diffusion Models (APDM) to hinder the unauthorized personalization of specific subjects in diffusion models, addressing privacy risks arising from high-quality subject synthesis. Unlike prior methods that generate adversarially perturbed samples but fail under simple transformations or with a few clean images, APDM shifts the protection target to the diffusion model itself. The authors first theoretically demonstrate that existing loss functions are inherently incapable of ensuring convergence for robust anti-personalization in diffusion models. To address this, they introduce Direct Protective Optimization (DPO), a novel loss function that effectively disrupts subject personalization without degrading generative quality. Furthermore, they propose a Learning to Protect (L2P) dual-path optimization strategy, which alternates between personalization and protection paths to simulate future personalization trajectories and adaptively reinforce protection. Experiments show that the proposed framework achieves state-of-the-art performance in preventing unauthorized personalization while maintaining generation quality.", "questions": "1. Is the novelty sufficient?\nThe idea of using model fine-tuning for anti-personalization is not new, with prior works (e.g., PersGuard) exploring similar directions. How does this work differentiate itself in a meaningful way?\n\n2. What are the threat model and assumptions?\nThe paper lacks a clear definition of the attacker and defender capabilities, the conditions under which the defender controls the model, and whether these assumptions are realistic in practice.\n\n3. Does the method require full knowledge of protected images and consistent identifiers?\nIf the defender does not have access to all protected images or if the attacker uses different identifiers during personalization, will the effectiveness of APDM degrade significantly?\n\n4. Can APDM scale to protect multiple identities within a single model?\nThe practicality of deploying APDM depends on whether it can handle multiple identities simultaneously, as maintaining separate models for each identity is not feasible.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed approach demonstrates strong effectiveness in mitigating unauthorized personalization. \n\n2. The experimental evaluation is thorough. The paper includes both comparative studies with existing baselines and detailed ablation studies.\n\n3. Figure 1 is particularly well-designed. It clearly illustrates how personalized models can lead to privacy breaches by revealing sensitive identity information.\n\n**Weaknesses**\n\n1.Limited Novelty: The novelty of this paper may be insufficient, as the idea of using model fine-tuning for anti-personalization has already been explored in the community. The work does not appear to be the first to propose using new model parameters to achieve robust anti-personalization. For example, see: Liu X, Jia X, Xun Y, et al. PersGuard: Preventing Malicious Personalization via Backdoor Attacks on Pre-trained Text-to-Image Diffusion Models. arXiv preprint arXiv:2502.16167, 2025.\n\n2.Lack of Clarity in the Anti-Personalization Process: The paper lacks a clear explanation of the anti-personalization process. Specifically, would a malicious personalization user continue to use DreamBooth to personalize the Anti-Personalized Diffusion Model (APDM)? If the personalized training involves images from the negative samples (as in Equation 5), I am concerned that the protection efficacy may be significantly compromised. Furthermore, this degradation could worsen with more fine-tuning epochs.\n\n3.Missing Threat Model and Assumptions: The methodology section does not clearly state the assumptions regarding the knowledge and capabilities of both malicious personalization users and anti-personalization users. Under what conditions does the defender have control over the model? Is this assumption realistic in practical scenarios? The authors are encouraged to refer to the problem definition section in Anti-DreamBooth to better articulate the necessary assumptions.\n\n4.Dependency on Complete Knowledge of Protected Images: Building on the previous point, does the anti-personalization user need to have full access to all the protected images to construct negative samples? If only partial knowledge (e.g., a subset of images) is available, will the effectiveness of protection degrade significantly?\n\n5.The theoretical justification provided in the paper appears trivial and redundant. Based on Equations 5 and 8, the gradient directions of Lper and Lprotect are nearly opposite, so it is naturally impossible to optimize them simultaneously. This result seems self-evident and may not require formal proof.\n\n6.Strong Assumption on Identifier Consistency: The paper assumes that the unique identifier (e.g., \"person\") and the generic prompt such as \"a photo of [V*] person\" are shared between the personalization and anti-personalization users. However, this is a strong assumption. In practice, defenders are unlikely to know the exact identifier used by downstream users, who may instead use terms like \"girl\", \"woman\", or \"guy\". If the identifiers are inconsistent, the protective effect could be significantly weakened.\n\n7.Need for Visual Comparison in Table 2: The protection effectiveness shown in Table 2 would benefit from visual comparisons. Since there are still some noticeable differences in quantitative metrics between APDM and the standard diffusion model (SD), visual examples could help illustrate these differences more intuitively.\n\n8.Scalability to Multiple Identities: Can APDM protect multiple identities within the same model? Deploying a separate model for each protected identity is impractical. The paper should clarify whether APDM can scale to handle multiple target identities simultaneously.", "was_revised": true, "final_justification": "The author solved my problem, I decided to increase my score to 4 points.", "rebuttal": "We appreciate Reviewer y2uu for your constructive feedback. We hope our response effectively addresses your concerns and questions.\n\n# W1: About novelty\n\nAs discussed in the manuscript, our work is motivated by a key limitation of existing data-poisoning approaches: they require access to all of a subject's images to be effective. This is often impractical, as controlling every image of a subject on the internet is impossible. As shown in Table 1 of our main paper, the presence of even a single unprotected image can cause the protection to fail.\\\nPersGuard operates under the same assumption of requiring full data access for its attack, thus sharing the same critical limitation as previous data-poisoning approaches. Furthermore, the authors of PersGuard note that their method's effectiveness is reduced by simple prompt variations (their \"Gray-box setting\"), whereas APDM demonstrates robustness not only to prompt variations but also to the presence of clean images and even entirely unseen images (as shown in Tab. R1 in our response W2).\n\n# W2 & W4: Clarification of anti-personalization process and results on the unseen data\n\n(**Process Clarification**) Our work is grounded in a practical scenario where **a service provider offers a diffusion-based personalization service to users**. Within this system, a malicious user would naturally attempt to personalize our protected APDM using the provided by the service—which we model using the powerful and widely-adopted DreamBooth. In addition, we also conducted with other personalization methods, **Custom Diffusion**, and represent the results in Table 3 in the supplementary. \\\n(**Robustness to Unseen Data**) In our paper, the images used for personalization are the same as those used for our protection process, i.e., negative samples in DPO. However, **APDM also demonstrates robustness regardless of the number of unseen images**. As shown in the Tab. R1, APDM can protect the personalization with 4-12 unseen images, which contain the protected target, i.e., not seen in the training.\n\nTable R1. Personalization with unseen data.\n|Method|# of unseen|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|---|\n|DreamBooth|-|0.6869|16.69|\n|APDM|-|0.1375|40.25|\n|APDM|4|0.1616|38.14|\n|APDM|8|0.1994|38.87|\n|APDM|12|0.1873|38.87|\n\n(**Robustness to Extended Fine-Tuning**) Additionally, we test APDM on a larger number of personalization iterations (Tab. R2). As seen below, we found that **APDM is still robust to prolonged fine-tuning**.\n\nTable R2. Personalization with larger iterations.\n|Method|# iters|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|---|\n|APDM (paper)|800|0.1375|40.25|\n|APDM|1600|0.1671|39.85|\n\n# W3: Threat Model and Assumptions\n\n(**Knowledge and Capabilities of Users**) \n- Malicious User: We assume a strong adversary. They can attempt personalization using either *clean images* (if leaked/off‑platform) or *perturbed images after bypass transforms* (e.g., flip, blur). During inference, they have full freedom to *vary prompts* and can use *any identifier* (e.g., t@t), not just the default sks token.\n- Protector (Service Provider): The provider's capability is demonstrated by applying protection effectively across *various models* (e.g., SD v1.5, SD v2.1) and against *multiple personalization methods* (e.g., DreamBooth, Custom Diffusion). The negative samples used for protection are proprietary to the provider and are not exposed to any user.\n\n(**Defender's Control Conditions**) The core condition under which the defender has control is a *provider-controlled deployment*. In this model, a user requests protection for their identity, and the provider updates its model to a protected version. The provider maintains full control over the model's lifecycle.\n\n(**Realism of this Assumption**) We argue that this assumption is highly realistic in many practical scenarios. Service providers inherently control their own models and deployment environments. *They are responsible for the service they offer, but not for user activity outside their platform*. This setting is a practical expansion of the \"uncontrolled\" setting considered in Anti-DreamBooth.\\\nFurthermore, in the rebuttal, we also add studies on variant class names, multi‑subject protection, and unseen‑data protection; these results highlight APDM’s robustness, and we will include them in the final version.\n\n# W5: About theoretical analysis\n\nThe goal of our theoretical justification is to provide a formal, rigorous proof that moves beyond intuition, and we would like to elaborate on its non-trivial nature. The reviewer's intuition seems to be that since $L_{per} $ converges, $L_{adv}$ ($L_{protect}$) must naturally fail. However, this high-level view oversimplifies the problem and overlooks the core of our analysis. \\\nThe convergence behavior of $L_{adv}$ is not self-evident for a critical reason: $L_{adv}$ is not simply the negative of $L_{per}$. The key difference lies in the **shared perceptual loss term, $L_{ppl}$**. The presence of the positive $L_{ppl}$ term in $L_{adv}$ creates a complex optimization dynamic. It acts as a regularizer pulling the optimization in a different direction from the $-L_{simple}^{per}$. Therefore, it is not immediately obvious whether the optimization will:\n- fail to converge and oscillate,\n- converge to a trivial solution where $L_{ppl}$ dominates, or\n- find some other unintended local minimum.\n\n**Our proof provides the answer to this ambiguity**. It is not trivial because it mathematically analyzes this internal conflict. We prove that the necessary conditions for convergence (derived in Proposition 1) lead to mutually contradictory requirements on the gradients (Theorem 1, Eqs. 11-12). This specific mathematical deadlock is the non-obvious reason for its failure.\nFurthermore, **this naive approach is a well-motivated and relevant baseline**. Similar gradient ascent techniques are widely adopted in fields like privacy protection to remove specific knowledge from models. Our analysis is therefore valuable as it provides a formal explanation for the limitations of such intuitive approaches.\n\nThus, our analysis is essential. It moves beyond the simplistic \"it must fail\" intuition to **provide a formal, rigorous proof of why and how it fails**, justifying the need for a more principled approach like ours.\n\n# W6: Protection when the identifier (class name) changes\n\nTable R3. Variation of identifier (class name).\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.6869|16.69|\n|APDM (\"person\")|0.1375|40.25|\n|APDM (\"(wo)man\")|0.1084|38.93|\n\nWe investigate the effectiveness of APDM when the identifiers (class name) are different between the protection and personalization. Note that, for APDM, the identifier \"person\" is used for protection. As shown in the Tab. R3, **APDM shows its robustness to identifier changes** (third row), as well as to changes on the unique identifier (Table 5 in the supplementary). \n\n# W7: Additional qualitative results\nDue to the policy, we cannot attach a visual comparison in this rebuttal. Nevertheless, we argue that APDM can still produce high-quality and text-aligned images compared to the pre-trained Stable Diffusion. We will attach a visual comparison in the camera-ready version.\n\n# W8: Multiple identities scenario\n\nWe completely agree that deploying a separate model for each identity is impractical, and a multi-identity solution is vital for any real-world application. While we designated this as a key area for future work (as mentioned in Supplementary G), we were motivated by the reviewer's comment to conduct a new preliminary experiment. Our goal was to assess the inherent robustness of our current, unmodified APDM in a multi-identity scenario. To this end, we constructed a challenging negative set ($x^-_0$) with four distinct subjects (cat, sneaker, glasses, clock.) and **applied our method without any tailoring**. For the purpose of comparison, the results of PAP are also presented, for which the evaluation was performed on the perturbed data (the \"# of clean images: $0$\" setting in our main paper).\n\nTable R4. Multi-subject protection.\n|Method|DINO(&darr;)|BRISQUE(&uarr;)|\n|---|---|---|\n|DreamBooth|0.5833|20.32|\n|PAP|0.5699|25.72|\n|**APDM**|**0.3924**|**32.97**|\n\nThese results highlight two critical points regarding scalability: \\\n(**Effective Multi-Identity Protection**) Despite the increased complexity of protecting four subjects simultaneously, APDM still outperforms the comparison. The resulting DINO score of 0.3924 is significantly better than the unprotected baseline (0.5833) or PAP (0.5699), demonstrating that our core protective mechanism is not limited to a single identity. \\\n(**Inherent Robustness and Scalability**) Crucially, this performance was achieved without any specific modifications to our algorithm, even in step (i.e., same training time). This showcases the inherent robustness of our approach, suggesting it is not a fragile, single-purpose solution but a generalizable framework with the potential to scale.\n\nIn summary, while a fully-optimized multi-identity model remains an important future direction, this experiment confirms that APDM's fundamental design is not inherently limited to a single identity. It directly addresses the reviewer's concern about practicality by demonstrating the robustness and scalability of our core method. This provides a strong foundation for the future work we have planned."}]

Author rebuttal:

Meta-review: This paper introduces Anti-Personalized Diffusion Models (APDM), a framework that attempts to provide protection against personalization by focusing on the model rather than on poisoning the data. 


Reviewers consistently highlighted the novel solution of model-level protection, the clarity of writing and presentation, and the strong empirical performance. Reviewer 3bmf also recognized the theoretical analysis as a notable contribution, while qkFB praised the paper’s readability and overall organization. Reviewers generally agreed that the work tackles a more challenging and practically relevant task compared to data-level perturbation methods.

During review, concerns centered on scalability (multi-subject protection), computational efficiency, broader comparisons, and generalizability (including perturbed data, large-scale personalization, prompt variations, and unseen data). The authors provided additional experiments—including 100+ image personalization, multi-subject protection, comparisons with recent baselines, and efficiency analysis—along with clarifications of assumptions and theoretical contributions. Reviewers acknowledged that these responses satisfactorily resolved their major concerns, with no further objections raised.
Decision: Accept (poster)