StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations
Authors: Yanjie Li
Wenxuan Zhang
Xinqi LYU
Yihao LIU
Bin Xiao
Venue: NeurIPS 2025 poster ()
Keywords: adversarial attack
style mimicry attack
Dreambooth
diffusion model
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A practical artwork protection method against style mimicry attack, robust to Noise Upscaling

Abstract: Recently, text-to-image diffusion models have been widely used for style mimicry and personalized customization through methods such as DreamBooth and Textual Inversion. This has raised concerns about intellectual property protection and the generation of deceptive content.
Recent studies, such as Glaze and Anti-DreamBooth, have proposed using adversarial noise to protect images from these attacks. However, recent purification-based methods, such as DiffPure and Noise Upscaling, have successfully attacked these latest defenses, showing the vulnerabilities of these methods. 
Moreover, present methods show limited transferability across models, making them less effective against unknown text-to-image models.
To address these issues, we propose a novel anti-mimicry method, StyleGuard. We propose a novel style loss that optimizes the style-related features in the latent space to make it deviate from the original image, which improves model-agnostic transferability.
Additionally, to enhance the perturbation's ability to bypass diffusion-based purification, we designed a novel upscale loss that involves ensemble purifiers and upscalers during training.
Extensive experiments on the WikiArt and CelebA datasets demonstrate that StyleGuard outperforms existing methods in robustness against various transformations and purifications, effectively countering style mimicry in various models. Moreover, StyleGuard is effective on different style mimicry methods, including DreamBooth and Textual Inversion. The code is available at \url{https://github.com/PolyLiYJ/StyleGuard}.

StyleGuard: Preventing Text-o--magg--ooel--aaedd Style Mimicry Attacks by Style Perturbations
Yanjie Li, Wenxuan Zhang, Xinqi Lyu, Yihao Liu, Bin Xiao
Department of Computing, Hong Kong Polytechnic University
(yanjie.1l) 1i, 1eo02. zhang, xinqi. yihao5. liu} @connect boomeett.ooyy...i polyu.hk b. xxxia0@polyu.oo edu.hk
Abstract
Recently, text-to-image diffusion models have been widely used for style mimicry and personalized customization through methods such DreamBooth and Tex- tual Inversion. This has raised concerns about intellectual property protection and the generation of deceptive content. Recent studies, such Glaze and Anti- DreamBooth, have proposed using adversarial noise to protect images from these attacks. However, recent purification-based methods, such DiftPure and Noise Upscaling, have successfully attacked these latest defenses, showing the vulnera- bilities of these methods. Moreover, present methods show limited transferability across models, making them less effective against unknown text-to-image models. To address these issues, we propose novel anti-mimicry method, StyleGuard. We propose novel style loss that optimizes the style-related features in the latent space to make deviate from the original image, which improves model-agnostic transfer- ability. Additionally, to enhance the perturbation's ability to bypass diffusion based purification, we designed novel upscale loss that involves ensemble purifiers and upscalers during training. Extensive experiments on the WikiArt and CelebA datasets demonstrate that StyleGuard outperforms existing methods robust- ness against various transformations and purifications, effectively countering style mimicry in various models. Moreover, StyleGuard is effective on different style mimicry methods. including DrcamBooth and Textual Inversion. The code is available at htttt:://// thub. com/ com//Polyyitt//yyyeeeuadd
1 Introduction
Diffusion models have demonstrated remarkable effectiveness various applications, such as image generation [13], image editing [22). and text-to-in -moge synthesis 43 [5]. The emergence of diffusion models has significantly transformed the industry [28] These models allow users to create detailed artwork from simple text prompts, a task that once required extensive time and skill from professional artists. However, these technologies have also raised concerns about copyrights and ethics. For example, Dreambooth (36) allows anyone to fine-tune an SD model to imitate an artist's 's style with just a few paintings and generate high-quality artwork. This seriously damages the intellectual property rights of artists.
To tackle the challenges associated with unauthorized image usage in rext-to-image generation, recent perturbation- basedd approaches have emerged. These methods are designed to subtly modify user images, rendering them "unlearnable" for malicious applications and disrupting the functionality of targeted diffusion models. For example, Anti-Dreambooth 40 shows some effectiveness by alternately training diffusion models and executing PGD attacks, but it is fragile against simple data transformations, MetaCloak (29) builds on ontamDreambott by incorporating simple transformations
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Transformation Purification
*An <sks> painting"
*An <sks> painting, including river and house"
DB-Full
Random Transform.
Unprotected
Mist
AndiDB
MetaCloak
Ours
Gaussian Noise
DB-LoRA
Protected
DiffPure
Textual
Noise Upscale
Inversion
Figure 1: A comparison of the defensive performance of different methods in the presence of the purification transformations. Previous methods, including Mist. AntiDreamBooth, and MetaCloak, fail to defend against DiftPure and Noise Upscale, while our proposed method successfully resists the style mimiery attack under various transformations for different customization methods.
improves MetaCloak by selecting time steps with the highest gradients improve the stability of the training.
into the attack pipeline. SimAC 41
Although these methods have demonstrated effectiveness against style mimicry, recent purification- based methods pose significant challenges to these protections. For example, DiffPure [4 and Noise Upscaling [99] utilize L.DM-based purifiers and upscalers to purify noise and successfully bypass the defenses of these methods, illustrated in Figure [ These novel attacks highlight the vulnerabilities of current anti-customization techniques. Moreover, in the real world, the finetuning model may differ from the model used in the training phase, making cross-model transferability important issue. In light of these limitations, there is a need for a more efficient method to generate protective noise that is resilient diffusion-based purifications and has strong cross-model transferability.
To solve these problems, we propose StyleGuard, a more effective method based on style perturba- tions to protect artists from unauthorized text-to-image diffusion-based style mimicry. To generate transferable perturbations, inspired by style transfer [17] and adversarial attacks in feature space 44]. we disrupt style-related features in latent space by adding subtle noise to the fine-tuning images that is nearly imperceptible to the human eye. Our method prevents text-to-image models from accurately extracting the style features of fine-tuning images, leading to false style correlations and hindering attackers from mimicking the original image's 's style. Moreover, compared to Mist (26] that uses L2 distance in the feature space, which is prone to causing local disturbances and sensitive to image variations, style perturbations alter global features. As a result, our method more robust to transformations such as image cropping, rescaling, and Gaussian noise. Additionally, bypassing diffusion- based purifications presents significant challenges, particularly because directly incorporat- ing purifiers or upscalers into the optimization process can lead to memory overflow, as most purifiers and upscalers are based on diffusion models [34] 24 7. To address this issue, we propose a novel upscale loss function that maximizes the loss of denoise error ensemble purifiers and upscalers through meta- meta-leerning approach. On the WikiArt and ClebA-HQ datasets, we show that StyleGuard significantly outperforms existing approaches over various transformations and purification methods and has higher cross-model transfcrability. Our main contributions are summarized as follows.
We propose StyleGuard, robust method designed effectively protect artists from DreamBooth-basedd style mimicry. StyleGuard accounts for various preprocessing tech- niques that attackers may employ, enhancing its practical effectiveness.
To improve the model-agnostic transferability, we introduce a novel style loss, which aligns the style characteristies of the protected image more closely with those of the target image, allowing the fine-tuned model to establish an incorrect style connection.
To bypass purification transformations, we propose a novel upscale loss function to maximize the denoise-error loss the ensemble purifiers and upscalers. Experimental results show that our approach exhibits strong robustness against the latest purification methods, including unseen purifiers.
DiffPure and Noise Upscaling,
Experiments on the WikiArt and CelebA datasets demonstrate that StyleGuard offers en- hanced protection against style mimicry and identity customization.
Compared to previous defense methods, our approach shows improved efficacy and practical effectiveness by considering various preprocessing techniques and model-agnostic scenarios.

---

2 Background
Diffusion models have emerged as powerful tool for image generation. Recent works have demonstrated their effectiveness in various domains, such as inpainting, super-resolution, and style transfer 30 16 35 The flexibility of diffusion enables the generation of high-quality images
from textual descriptions.
2.1 Style Mimicry and Copyright Concerns
Unauthorized style mimicry has become a major concern in the Al art community, where malicious actors exploit AI models to replicate an artist's unique style without consent (4 6, 32]. Advanced
mimicry attacks involve fine-tume fine-tuning generic text-to-image models on a small collection of artist's
works, often as few as 20 pieces by models such as DreamBooth I10 36 DreamBooth identifies key stylistic features and associates them with specific token in the fine-tuned model, enabling highly accurate style replication. Such techniques have led to widespread incidents of unauthorized mimicry 4 6 32], for example, CivitAl [8] built a large online website where people share their finetuned stable diffusion models. The potential for unauthorized style mimiery threatens the livelihoods of artists, leading to discussions about intellectual property rights in the digital age.
2.2 Protection Against Style Mimicry and Personalization
To address the unauthorized style mimicry issue, perturbation- -based methods have been developed, which add subtle image perturbations to the unprotected images to disrupt generative models. For ex- ample, PhotoGuard (37 aligns protected images' latent features with black-and-white images. Glaze 38 minimizes the feature distance between perturbed images and target images while maintaining perceptual similarity. AdvDM [27] reduces the likelihood of perturbed images under pre-trained diffusion models by disrupting the denoising process. Its enhanced version, Mist (26). utilizes black-and-white periodic images as targets and incorporates semantic loss and textual loss to improve protective strength. However, Mist directly minimizes the L2 distance between the original and target latent features, which causes noticeable and unnatural textures that degrade the original image's 's quality. Anti-DreamBooth 40) proposes a novel scheme to defend the personalization attack that alternately updates the diffusion model and protected images. MetaCloak [29] builds upon Anti- DreamBooth by incorporating simple transformations, such as Gaussian blur and cropping, into the attack pipeline to improve robustness against these transformations. SimAC [41] further extends the work done by Anti-DreamBBotth by selecting timesteps with maximum gradients to stabilize the protect copyrights by embedding watermarks into images, subtly incorporating the author's information. However, this kind of approach introduces additional verification processes. While these methods exhibit robustness simple transformations, recent work has introduced purification-bas
training process. Some other methods [15 46 45 42
methods, including DifiPure 3d and Noise Upscaling
[9] which first add noise to images and then employ an LDM as purifier or upscaler remove noise. These approaches have demonstrated impressive results in removing protective noise, rendering many recent protective methods ineffective. Therefore, there is an urgent need to develop more robust methods to defend against such attacks.
3 Preliminary
3.1 Style Mimicry by Dreambooth
DreamBooth (36] introduces novel approach for personalizing text-to-image diffusion models by enabling them to generate high-fidelity images of specific subjects based on a few reference images. The method fine-tunes a pre-trained text-to-image model to bind unique identifier (c.g., "[V]") to the subject, allowing the model to synthesize the subject in diverse contexts while preserving its key visual features. This is achieved through fine-tuning process: the latent diffusion model (LDM) is fine-tuned using input images paired with text prompts containing the unique identifier and the subject's ss class name (e.g., "An [V] painting"), while class-spectie prior preservation loss ensures the model retains its semantic understanding of the broader class (e.g.. "A painting"). The DreamBooth loss function is defined
(g zo) E Exg.a le ||e co (I1+1, c)1 + A lleo (Ie+1, cp
(7)
3

---

Bytht
K1-Step DreemBooth
Diffusion Models
K1-Step DreamBooth
x
Purifier
K2-
VAEE CLI
loppocale
Step
Upscaler
PGD
Cx
VAE
CLI
Model Update
Image Update Path
Tensor Path (No gradients)
The pipeline of StyleGuard. We alternatively update the diffusion model and the protected images, Ensemble image encoders and purifiers are included to compute the style loss and upscale loss improve cross-model transferability and the robustness to purifications.
Figure 2:
By leveraging the model's semantic prior and the text-guided denoising loss function, DreamBooth enables tasks like subject recontextualization, text-guided view synthesis, and artistic rendering. overcoming limitations of existing text-to-image models in reconstructing and modifying specific subjects.
4 Methodology
The pipeline StyleGuard is shown in Figure 2 the loss functions. Finally, we will introduce the StyleGuard Algorithm.
We will first define the problem and then introduce
4.1 Problem Statement
We frame the problem as follows: a user aims to safeguard set of clean and unprotected images X. = { from being exploited by unauthorized model trainers for generating style-mimicking images. To accomplish this, user applies small perturbation Xe, resulting in a modified set of protected images Xp {2 which can be safely released on the Internet. Adversary will then gather and utilize Xp to fine-tune a text-to-image generator following the DreamBooth algorithm or any other methods. We assume that the model trainer has some awareness of the protection and tries to destroy the protection effectiveness through different pre-proe rre-processing methods, such as random transformations and purifications to the training image set Xp-
The goal of the user create a protected and robust image set Xx that diminishes the personalized generation capabilities. This objective can be expressed as a bilevel optimization problem:
X² arg max Luis (Xtgee X.) X,.
(1)
where 0" arg min (T(
In these equations, e denotes the class-wise conditional vector. Xgen -M-.x,, (c) is the generated images the fine-tuned LDM model Ma..,,- Lais represents perception-atigneed distance function used evaluate the style discrepancy between the generated images X gen and the clean reference images X.. The Lgen is the finetuning loss function. We hypothesize the adversary tries to destroy the protection of X, by applying some kinds of preprocessing methods (denoted by T).
4.2 Disturbing Style-related Features and Generating Transferable Perturbations
Previous work has found that the mean and variance of the feature space encapsulate the style information (39
We consider the problem defined in Fq. 4. as maximizing the style- related feature distance between the reference images and the perturbed images and making it closer to the target images. Therefore, propose novel style loss function, which is defined as
44
18 20

---

Layle B (1luap Miz 112 I10ag -a 0x2112
Mag
Mr. 1
110
Iza 0za112
112
(3)
where X, is set of target images that has different style from X.. The and variance the latent features of these images encoded by the LDM's VAE or CLIP encoder f. To improve the transferability to unknown models, compute the style loss over set of different substitute encoders. The target image is selected to have a distinct style from the original images, such from different art movements (e.g., realistic and abstract). By disturbing the style-related features, StyleGuard can make it difficult for DreamBooth or Textual Inversion to establish a correct connection between the style features and the unique identifier.
the mean and
4.3
Why Previous Defenses are Ineffective to Noise Upscaling and DiffPure
Previous defenses, such as MetaCloak (29 and SimAC 41]. involve transformations like random cropping Gaussian blur in the perturbation generation process to enhance robustness against preprocessing methods. However, these approaches fail to counter attacks like DiffPure (34] and Noise Upscaling [9]. which use diffusion models as noise purifiers. Directly incorporating Noise Upscaling or DuftPure into the optimization process can result in an excessively large computation graph. To address this challenge, we first apply small noises Tp according to the DiffPure and Noise Upscale settings and then maximize the denoising error loss across a set of substitute DiftPure and upscaling (super-resolution) models. The upscale loss function defined as
Tuppcale O, Oy (1.e.1100
||c (o 122 where o/ = Jp + 5N 1),
where Or is the parameter of purification and upscaling models. Upscale loss function aims to cause purifiers and upscalers to amplify protective perturbations instead of reducing during the diffusion process. The upscale loss is computed at a randomly-chosen timestep in the denoising sequence. Still, this method is effective in breaking popular purifiers and upscalers (Sec(5.2)
4.4 The Algorithm of StyleGuard
straightforward approach to tackle the bilevel problem in Section 4.I is to unroll all training steps and optimize the protected examples Xp through backpropagation. However, this would cause very large computation graph that would exceed the capacity of most current machines. To overcome this challenge, inspired by Anti-DreamBoote [40), use approximate method optimize the X, and 9 alternatively. Specifically, the f-th iteration, when the current model weights 8, and the protected image set b are available (with On initialized from a pretrained diffusion model and X = X.). create copy of the current model weights, denoted as 02,0 Ot, for noise crafting. We then optimize the UNet of LDM for K₁ steps using DreamBooth loss:
f = 81, VV Lyen Exge f6gg)
(5)
where j {0, 1,... K 1) and is the step size. This unrolling process enables to "look ahead" during training and assess how current perturbations will influence the fine-tuned LDM.
Subsequently, we utilize the updated UNet model 8 k to optimize the upper-level problem, specifi- cally updating the protected images X, by PGD. However, is difficult to update the training images Xp p of LDM by Lais directly because this needs to unroll the fine-tuning process. To make the gradient computable, we maximize the denoising-een the pretrained text-to-image generator used by unauthorized model trainers is often unknown. To enhance the transferability of the perturbed images to unknown models, we alternatively maximize the denoising error across a group of LDMs. The denoise loss is defined as
loss L instead. Moreover, in real-world scenarios,
Lieenoise -Eg D......ss.......c.. ||c co(
(6)
The total loss function is defined as:
5

---

syyleguard denoise + 1/A A
(7)
where 1/ and À hyperpameters. We 1/=1 and λ=10 in experiments. Moreover, improve the robustness to other transformations like crop and resize, we involve random transformations in the optimization process based the expectation the perturbed image X₂ using PGD with the attack budget Boc for 2₂ steps through
transformation (HoT) 29 5. Then update
X Ei
B (X2
sign (Vxg Insylegumm) )
(8)
This expectation is estimated by Monte Carlo sampling with J samples (J = our setup). After obtaining the updated protected images X + 86, the surrogate model 0, is trained on the perturbed images for additional K SGD steps:
P 0, VV, Lgnn (X; 00)
The procedures outlined above will repeat N times, resulting in the final protected images Xy. The final algorithm is shown Algorithm
Algorithm 1 The Algorithm of StyleGuard
1: Input: Initial substitute LDM model set Oo initial protected images X = X., pretrained image encoders F, number of iterations N, target images Xr, and fine-tuning steps K PGD steps K2. pretrained LDM upscaler set θg, random transformations T.
2: for 0 to N do
3: Sample 0g from θ; sample OT from Or; sample random transformation from TV.
4: Copy model weights: 6, 6-
5: for j to Ki do
6:
Update copyed UNet model weights f on Xp according to Eq. end for
7:
8:
for j= to K₂ - do
9:
Compute the StyleGuard loss according to Eq. Eq. Eq 6 and Eq.
10: Optimize protected images Xp using PGD attacks according to Eq. end for
11:
12: for j= 0 K₁ 1 do
13:
Update the surrogate model 0g according to Eq 14: end for
15:
end for
16: Output: Final protected images X
Experiments
5.1
Experiment Setup
Datasets We evaluate StyleGuard's performance on the WikiArt and CelebA datasets to assess its effectiveness against both style mimicry and personalization attacks. Notably, most previous work has focused on only one of these attacks, whereas we are the first to successfully defend against both. The WikiArt dataset comprises 42,129 artworks from 195 different artists, with each piece categorized by its genre (such as impressionism or cubism). For our style mimicry attacks, we randomly selected 40 artists with different art styles and 20 artworks from each artist, using 10 for training and for evalution. For the CelebA dataset, we randomly select 100 identities, using 10 images per identity to fine-tune the LDM model and another 10 images for evaluation.
Implementation details Initially, we use SD v1.4 and SD v1.5 as the substitute models to perturb the images. The attack budget is set as 255 to be same with baselines. The images encoders used and
to compute the style loss includes VAE, CDpeCCLIP-ViT-EEE StyleGuard training details and hyperparameters are included in Appendix A. During testing, we evaluate two popular fine-tuning methods: DreamBooth and Textual Inversion M. For DreamBooth,
The

---

we further assess two common settings: full-tuning (Full-FT) and LoRA fine-tuning (LoRA-FT) [1]. applied to both SD v2.1 and SD-XL (only LoRA on SD-XL, due to memory constraints). We used the official script from the Diffusers library for DreamBooth fine-tuning, The training details for DreamBooth and Textual Inversion are included in Appendix A.2
and A.3
Attack Settings We consider three kinds of attacks, including random transformation, DiffPure, and Noise Upscaling. For random transformations, we consider Gaussian noising, center cropping and resizing. For the DiffPure, we use the official code, and use Guided Diffusion Model and DDPM model during training, and use Stable Diffusion XI. during evaluation. For Noise Upscaling, we followed the settings outlined by Honig et al. [19]. During the optimization, we generate perturbation on the SDD-x4-ppscaler [35 3 During testing, we evaluate the defense effectiveness using the SD-x2- 1x22-latent-upsalle (2). which has different architecture compared to the SD-x 4D-x4-upscaler (see Appendix A.5 for more details). The number of diffusion steps was set to 30 because large diffusion steps will modify the image content. We also use online black-box upscaler, Finegrain Image Enhancer (FIE) 1 For FIE, the upscale factor is set as 2, and the ControlNet scale is set as 0.6.
Baseline settings We compare our method with Glaze 38), Mist [26]. Anti-DreamBooth 44) &
MetaCloak [29] and SimAC [4I]. The attack budget are set as 255 for all baselines except Glaze, which uses LPIPS as constraint. The implementation details are included in Appendix A.4 A.4
Evaluation Metries To evaluate the defense performance for style mimicry, we use Fréchet Inception Distance (FID)[I2 and precision [3 as assessment metrics (consistent with Mist 26)
and mimicry success rate [19], which uses human as annotators. To exclude content influence in the style mimicry attack, we use the model trained on clean images to generate 100 paintings in specific category (c.g., an <sks> painting of a house). We then generate another 100 images from the model trained on style-guarded images using the same prompt and compute the FID and precision
between the two sets. For personalization attacks, we use identity match score (IMS) 40 to access
the semantic closeness between faces. The metric details shown in the Appendix A.6
5.2 Experiments Results
Comparison StyleGuard with Different Methods. we evaluate the protection efficacy of our method and baselines under no preprocessing and different transformations. Table [ shows the evaluation results on the WikiArt using DreamBooth. The baseline trained on clean images with FID (233.78) and Precision (0.60) serves as reference points. It is shown that when there is attack, all methods can successfully disrupt style mimicry, resulting in increased FID and reduced precision compared to the baseline. The previous method, Mist, demonstrated vulnerabilities to straightforward transformations like cropping and resizing, as well as Gaussian noise, yielding precision scores of 0.46 and 0.48. This weakness stems from its insufficient attention to global features and lack of consideration for transformations or purifications within its methodology. MetaCloak and SimAC exhibit enhanced robustness to simple transformations because they incorporate these considerations into their pipeline, resulting in high FII) scores and low precision scores (0.20 and 0.18 for MetaCloak, 0.06 and 0.04 for SimAC). However, their effectiveness diminishes in the presence of purifier or upscaler. Our method, StyleGuard, surpasses all existing techniques in both scenarios. It achieves the highest FID and the lowest precision across different transformation settings, demonstrating strong protective efficacy against transformations and purifications. Notably, when purifiers or upscalers are present, our method significantly outperforms the baseline methods. We believe this is because our upscale loss (Eq. 4 can bypass the purifier or upscaler. Additionally, our human evaluation results corroborate these findings, shown in Figure 4 (see Appendix A.7 for human evaluation details).
The Importance of Difffernt Loss Functions The efficacy of various loss functions in safeguarding against style mimicry is visualized and quantitatively analyzed in Figure and Table [ Figure 3 illustrates the influence of different loss functions on image quality and the robustness of protection. Although using denoise loss and style loss can result in decrease in image quality for unprotected
images (Figure 3 (2)). it is still vulnerable to Noise Upscale, as shown in Figure 3 (3), in which the image quality is improved. The integration of upscale loss significantly improves the protection resilience, as evidenced in F (4). This underscores their collective contribution to a more robust defense mechanism, ensuring that style mimiery is effectively countered even when purification attacks exist. The quantitative results in Table Table[ further validate the effectiveness of the loss functions.
7

---

Train clean images
Style Loss Denoise Loss, without Noisc Upscalo
(3) Style Loss+ Denoise Loss, with Noise Upocale
Loss+Denoice
Style Loss Loss-Denoisc Loss+ Upeale Loss, with Noise Upscall
Figure 3: Visualizing the effects of different loss functions. It shown that only using denoise loss and style loss cannot defend the Noise Upscale well, as shown in (3). With the upscaler loss, the image quality significantly decreases even with Noise Upscale, shown in (4).
Mone Minicr
Goustion Nobrig
vol Upocalig
FPr
Car
Our
Our
Jed
1000
Se0ces
Sucten New
Figure 4: Evaluation results of mimicry success rates by human evaluators. We asked users to compare generated images based on clean and protected training images using the question: "Based on the image style and quality, which image better fits the reference samples?" A lower mimiery success rate indicates stronger perturbation noises affecting the image quality.
StyleGuard, which incorporates denosing loss, style loss, and upscale loss, achieves the highest FID and the lowest Precision, indicating that the image quality is lower than using single loss.
Textual Inversion Results Textual Inversion optimizes only a small set of new token embeddings that can later be appended to prompts to imitate the target style/image without fine-tuning the model. We use the official code for the textual inversion from the diffusers package and use the SD v2-1 ---se as the text-to-image model. The results of the experiment are shown Table 4 in the Appendix. Experiments show that StyleGuard's 's denoising loss is more effective against textual inversion. This is because Textual Inversion does not modify model weights and the adversarial noise directly corrupts the semantic alignment between learned tokens and the style. The style loss can destroy the quality of the generated images further, as shown in the second-to-last line in Table [ is shown that StyleGuard remains effective even when attackers use a different method, including DreamBooth and Textual Inversion, whereas traditional defenses (e.g., Glaze) fail.
Transferability on different models High transferabiliity method is more practical for real-world applications. Table compares the transferability of Anti Inti-DreamBooke (above the slash) and Style- Guard (under the slash). We calculated the ratio of FID scores (%) for images generated on evaluation models and substitute models. The higher the ratio, the better the transferability. The results demon- strate that StyleGuard has better transferability across different SD models than Boot This is because, compared to Anti-DreamBooth which only uses denoising loss, StyleGuard incorpo- rates style loss. This addition can perturb global style-related features that are independent of the model parameters. We also evaluate the protection results on a black-box online upscaler, FIE. As shown in Figure 8 in the Appendix, when using FIE for purification, the style of the image is more similar to Van Gogh's style compared to images without purification. However, it still does not match the quality of images generated from clean inputs.
Experiment Results on LoRA settings, apply LoRA-based training to SD models optimized with DreamBooth loss 3 Quanti-
To evaluate cross-model transferability under different fine-tuning
8

---

Table 1: Comprehensive evaluation of text-to-image protection methods under different transfor- mations for DreamBooth on the WikiArt Dataset. Metries reported are FID1 (higher better) and Precision, (lower better). The best data are shown in bold, and the second runners are in gray.
No Prep.
Crop+Resize
Gauss, Noise
DuftPure
Noise Up.
Method
FID
Prec.
FID
Pree.
FID
Prec.
FID
Prec.
FID
Prec.
No Protect
233.78
0.60
275.41
0.65
238.25
0.62
237.89
0.68
236.58
0.60
Glaze
333.89
0.15
315.22
0.40
340.10
0.35
318.94
0.30
312.73
0.60
Mist
382.50
0.00
295.28
0.46
275.77
0,48
290.45
0.42
256.65
0.45
AntiDB
327.01
0.05
310.88
0.25
322.15
0.20
305.74
0.35
293.14
0.50
MetaCloak
382.00
0.05
362.52
0.20
355.26
0.18
318.87
0.25
295.20
0.40
SimAC
407.40
0.00
365.47
0.06
380.45
0.04
290.15
0.38
284.52
0.45
Ldenoise
348.15
0.03
355.77
0.30
362.42
0.15
358.89
0.12
310.54
0.20
deenossesstyle
389.33
0.01
382.45
0.25
380.21
0.08
385.77
0.06
375.92
0.10
StyleGuard
428.70
0.00
405.31
0.05
420.74
0.02
418.33
0.03
401.80
0.00
tative results demonstrate that our method achieves stronger performance on SD-XL (FID: 464.45, Precision: 0.00) compared SD-v2-1 (FID: 366.78, Precision: 0.16), though both scenarios signifi- cantly outperform prior approaches. We hypothesize that the weaker protection efficacy on SD-v2-1 stems from architectural differences in parameter adaptation during LoRA fine-tuning. Specifically, SD-v2-1 may undergo updates concentrated in fewer layers, particularly those less sensitive to adversarial perturbations, resulting smaller effective attack surface (see Figure 6 Appendix for visual examples).
Evaluation Results on Personalization Attacks We also evaluate the effectiveness of our method in defending against personalization attacks (see Appendix A.8 for implementation details). As shown in Figure 7 and Table in the Appendix, we successfully defend against the personalization attack by Dream Booth, even when Noise Scaling is used to preprocess the face images. We think this because the style loss can not only disrupt the style-related features but also the identity-related features, which is consistent with the findings of StyleGAN (21)
Table 2: Cross-modes transferabilities for An- üiDremBBooth and StyleGuard.
Table 3: Transferability to different fine-tuning methods.
Evaluation model
Finctuning Method
LoRA/SD v2-1
LORA/SD XL
Surrogate
SD
vD5
v2
FID
Prec.
FID
Prec.
SD v1.4
100.0/100.0
85.5/96.5
76.5/92.4
Clean (Baseline)
210.45
0.75
215.60
0.72
SD v1.5
84.8/96.2
100,0/100.0
73.5/92.5
AntiDremmonth
285.20
0.45
365.80
0.28
MetaCloak
320.85
0.28
435.60
0.22
SD v2.1
73.5/89.2
76.4/22.5
100.0/100.0
SimAC
375.90
0.20
445.75
0.12
Ours
366.78
0.16
464.45
0.00
Limitation and Conclusion
This work introduces StyleGuard, novel and robust anti-mimicry method designed to protect artists from unauthorized style mimicry in text-to-image diffusion models. By optimizing style-related features in the latent space, StyleGuard effectively disrupts the extraction of correct style features, making it difficult for attackers to replicate the original artistic style. Extensive experiments on the WikiArt and CelebA-HQ datasets show that styleGuard exhibits strong cross- cross-model transfcrability, outperforming existing methods in terms of protection efficacy. Our approach also demonstrates superior robustness against data transformations, including the state-of-the-art art DiffPure and Noise Upscaling. This work addresses critical challenges in intellectual property protection in the digital art domain, providing artists with a powerful tool to safeguard their unique styles from unauthorized exploitation. However, experiments indicate that StyleGuard is less effective with commercial upscalers and LoRA-based fine-tuning methods. Future work could explore extending StyleGuard to other customization methods and more complex purification methods.

---

Acknowledgment
This work was supported part by the Hong Kong Research Grants Council's (RGC) General Research Fund (GRF) under Grant PolyU 15201323.
References
[]] Finegrain Image Enhancer a Hugging Face Space by finegrain huggingface.co.
https
image- eateeeeenharcrrrr
[2] stabiiryylsa-22hhcccnnnusseeeeed Hugging Face huggingface.co. https://huggingtae
[3] stbbliyyajubbbbbcciittiooos-sugr Hugging Face huggingface.co.
https://
x4 upsualrr
[4] Sarah Andersen. The alt-right manipulated my comic. then a.i. claimed it. The New York Times, 2022.
[5] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adver- sarial examples, 2018.
[6] Andy Baio. herself turned into an ai model. invasive- diffusion howeiiooooooo----eeeeeeeeeee one wwwillinggilllstrrenn -o intoo-iito-ooo an- ai-model/ 2022.
Invasive diffusion: How
one
unwilling illustrator found
https://waxy.orse wtg/
[7] Yong Chen, Xuedong Li, Xu Wang, Peng Hu, and Dezhong Peng. Diffilter: Defending against adversarial perturbations with diffusion filter. IEEE Transactions Information Forensics and Security, 2024.
[8] CivitAl. Civitai. https://civitaa..com 2022.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[10] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar arXiv:2208.0160 01618, 2022.
[11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium, 2018
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in nearal information processing systems, 33:6840-6857 2020.
[14] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[15] Junqiang Huang. Zhaojun Guo, Ge Luo, Zhenxing Qian, Sheng Li, and Xinpeng Zhang. Disentangled style domain for implicit s-watermark towards copyright protection. Advances in Neural Information Processing Systems, 37:55810-553 55800000 2024.
[16] Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang. Humannorm: Learning normal diffusion model for high-quality and realistic 3d human genera- tion. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4568 4568-4577, 2024.
10

---

[17] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[18] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization, 2017.
[19] Robert Hönig, Javier Rando, Nicholas Carlini, and Florian Tramer. Adversarial perturbations cannot reliably protect artists from generative ai. In Proceedings of the International Conference on Learning Representations (ICI.R) 2025.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
[22] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6007-6017, 2023.
[23] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances neural information processing systems, 32, 2019.
[24] Minjong Lee and Dongwoo Kim. Robust evaluation of diffusion-based adversarial purification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 134 144, 2023.
[25] Yanjie Li, Kaisheng Liang, and Bin Xiao. Uv-attack: Physical-world adversarial attacks for person detection via dynamic-nerf-basse uv mapping. In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR), Singapore, Apr 2025. Apr. 24-28, 2025.
[26] Chumeng Liang and Xiaoyu Wu. Mist: Towards improved adversarial examples for diffusion models. arXiv preprint arXiv:2305. 12683, 2023.
[27] Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Adversarial example does good: preventing painting imitation from diffusion models via adversarial examples. In Proceedings of the 40th International Conference on Muchine Leurning, pages 20763-20786, 2023.
[28] Yihao Liu, Jinhe Huang, Yanjie Li, Dong Wang, and Bin Xiao. Generative ai model privacy: survey, Artificial Intelligence Review, 58:33, 2025.
[29] Yixin Liu, Chenrui Fan, Yutong Dai, Xun Chen, Pan Zhou, and Lichao Sun. Metacloak: Pre- venting unauthorized subject-driven text-to-image diffusion-based synthesis via meta- neta-nea In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24222,242 2024.
[30] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 11461-1147, 2022.
[31] Xinqi Lyu, Yihao Liu, Yanjie Li, and Bin Xiao. Pla: Prompt learning attack against text-to-image generative models. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), Honolulu, Hawai'i, USA, Oct 2025. Oct 19 - 23th, 2025.
[32] Brian P. Murphy. lensa ai stealing from human art? an expert explains the controversy. ScienceAleri, 2022.
11

---

[33] Aamir Mustafa, Salman H Khan, Munawar Hayat, Jianbing Shen, and Ling Shao. Image super- resolution as a defense against adversarial attacks. IEEE Transactions on Image Processing. 29:1711-1724 2019.
[34] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anand- kumar. Diffusion models for adversarial purification. In International Conference on Machine Learning, pages 16805-16827. PMLR, 2022.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 10684-10695, June 2022.
[36] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-22510 22510, 2023.
[37] Hadi Salman, Alaa Khaddaj, Guillaume Leclerc, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai-powered image editing. In International Conference on Machine Learning, pages 29894-29929 PMLR, 2023.
[38] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. Glaze: Protecting artists from style mimicry by (Text-to-Imaaee models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 2187-2204, 2023.
[39] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016
[40] Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran. Anti-dreambooth: Protecting users from personalized text-to-image synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2116-2127, 2023.
[41] Feifei Wang, Zhentao Tan, Tianyi Wei, Yue Wu, and Qidong Huang. Simac: A simple anti- customization method for protecting face privacy against text-to-image synthesis of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pages 1047-12056, 2024.
[42] Jinwei Wang, Haihua Wang, Jiawei Zhang, Hao Wu, Xiangyang Luo, and Bin Ma. Invisible adversarial watermarking: A novel security mechanism for enhancing copyright protection. ACM Transactions on Multimedia Computing. Communications and Applications, 21(2):1-22. 2024.
[43] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin. Composi- tional text-to-image synthesis with attention map control of diffusion models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages
2024.
[44] Qiuling Xu, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang. Towards feature space adversarial attack by style perturbation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 19523-105311 2021.
[45] Hongwei Yao, Jian Lou, Zhan Qin, and Kui Ren. Promptcare: Prompt copyright protection by watermark injection and verification. In 2024 IEEE Symposium on Security and Privacy (SP), pages 845-861. IEEE, 2024.
[46] Peifei Zhu, Tsubasa Takahashi, and Hirokatsu Kataoka. Waternerk-embeddde adversarial examples for copyright protection against diffusion models, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
The abstract and introduction accurately reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discusses the limitations of the work in our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
We does not include theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully disclose the information needed to reproduce the main experimen- tal results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
We will provide code to reproduce the results in our paper in the supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //nips.c//pil wwwppbblccgguiicuceccccc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided all the training and test details in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our paper evaluated the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:pp///oourpss...c/p/////hahs
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp/////euips..... wwwparippsscccoooeeeeeee cc/Conferencess///////e for what should or should be described.
19

---

A Technical Appendices and Supplementary Material
A.1 StyleGuard Training Details
We generate the perturbations using eight NVIDIA 3090 GPUs. The fine-tuning steps, denoted as K1, set to while the PGD) steps, denoted as K2, set to 6. The total training steps N are set to 100. The target image is randomly selected from a different art genre. The PGD budget is configured to 255 and the PGD step size is set to 0.005. Additionally, the weight of upscale loss (n) is set to I, and the weight of the style loss (A) is set to 10. The optimization time for each image is approximately 20 minutes, compared 15 minutes for MetaCloak (29 and 40 minutes for SimAC AT. This extended time for SimAC is due to the additional steps required to select time steps for the denoising error loss, which we find provide minimal benefit for the defense.
A.2 DreamBooth Training Details
For the training of DreamBooth, we use learning rate of 5e-6, with the prior reservation loss weight set to 1.0. For the style mimicry attack, the customized prompt is set "an <sks> painting" and the prior prompt was set as "a painting". : For the personalization attack, the customized prompt is set as "a photo of <sks> person". and the prior prompt was set as "a photo of person". In the original DroamBooth paper, they use 5 images for the personalization. However, we found that using only images per artist images per artist and trained for 1000 epochs. For LoRA fine-tuning, the dimension of the LoRA update matrices is set 4. We use mixed precision of 6 to save memory. It takes about 20 minutes for the full munnt and 4 minutes for the LoRA fine-tuning.
insufficient for generating high-quaitty mimicry. Therefore,
opted for 10
A.3 Textual Inversion Implementation Details and Experiment Results
Textual Inversion is a technique for learning and replicating new visual concepts (e.g., artistic styles, objects, or aesthetics) in a pretrained text-to-image diffusion model (such as Stable Diffusion) without fine-tuning the model's weights. Instead, it optimizes only a small set of new token embeddings that can later be appended to prompts to imitate the target style/image. The method introduces new (placeholder) text tokens (e.g., "iskssstlle and optimizes their embeddings (vectors in the text encoder's s space) to represent the target visual style. These tokens can later be inserted into prompts (c.g.. "A painting in the style of sks_ "ks_sttlee) and will guide the model to generate images resembling the trained style. We use the official code for the textual inversion and use the SD v2-1-base as the text-to-image model. In our experiment, we find that StyleGuard's is denoising loss is most effective against Textual Inversion (Table 4 while the style loss is also important to destroy the quality of generated images. This is because Textual Inversion does not modify model weights and the adversarial noise directly corrupts the semantic alignment between learned tokens and the style.
Moreover, StyleGuard's perturbations remain effective even when attackers use different mimicry methods (Dreambooth,, Textual Inversion, etc.), whereas traditional defenses (e.g., Glaze) fail, Pre- vious work, like MetaCloak and SimAC, are robust to simple transformations like Crop+Resize and Gaussian Noise. However, they are relatively vulnerable to attacks such as DiftPure and Noise Upscaling. When there is DiffPure and Noise Upscaling, our methods achieve the highest FID and the lowest precision. We think this because the StyleGuard methods consider a variety of purifiers and upscalers during training. However, when there is no purifying measure, our method performs slightly worse than SimAC in textual inversion, because SimAC adds the step of selecting diffusion timesteps. Although SimAC can further reduce the quality of the image, it will also increase the optimization time.
A.4 Baseline Implementation Details
In this section, we outline the baseline settings for our comparisons with several protection methods: Glaze, Mist, nrot-DreaBooth, MetaCloak, and SimAC. To ensure a fair evaluation, we maintain a consistent perturbation budget of p 8/255 for all methods except Glaze. Evaluating Glaze under this specific budget presents challenges due to Glaze utilizes LPIPS for its image similarity metric, which does not constrain the L norm. Consequently, we implement Glaze by our own according to the Glaze paper. Our observations indicate that images processed with Glaze appear equally or less perturbed compared to those processed with Mist and Aati-DreamBooth.
20

---

Table 4: Comprchensive evaluation of text-to-image protection methods under different transforma- tions for Text Inversion. Metrics reported are FID (higher better) and Precision, (lower better).
No Preprocess
Crop+Resize
Gaussian Noise
DittPure
Noise Upscale
Method
FID
Prec.
FID
Prec.
FID
Prec.
FID
Prec.
FID
Prec.
No Protection
237.56
0.9
280.63
0.88
241.20
0.87
239.87
0.92
255.31
0.85
Glaze
249.43
0.84
263.05
0.72
285.11
0.75
262.33
0.65
285.47
0.71
Mist
454.39
0.02
297.90
0.80
422.17
0.10
301.25
0.78
294.60
0.75
AntiDB
371.12
0.22
327.88
0.44
353.45
0.31
266.54
0.52
260.13
0.58
MctaCloak
416.25
0.04
380.52
0.20
398.76
0.12
315.87
0.68
308.42
0.72
SimAC
465.82
0.02
370.87
0.18
441.35
0.08
340.15
0.25
335.79
0.30
denoosee
382.15
0,06
385.67
0.08
375.42
0.22
368.79
0.19
362.84
0.25
Ldmoisotstyle
410.33
0.04
392.45
0.05
403.21
0.12
395.67
0.10
388.92
0.15
StyleGuard
428.57
0.00
398.21
0.05
419.84
0.08
412.33
0.04
425.76
0.07
Next, specify the hyperparameters utilized for replicating each protection method.
Glaze Due to the lack of access to a shared codebase from the Glaze authors, we implemented Glaze independently. The LPIPS distance is computed using the VGG model. In Figure we display examples of images generated by Glaze. The results indicate that Glaze produces images that are mixture of the target and reference styles.
Mist We conducted our evaluation of Mist following the methodology 26 The parameters set for this evaluation include PGD perturbation budget of p = 8/255, with NrGD 100 iterations and PGD step size of a 1/255. The target image used for this evaluation is denoted as T Target Mist,
as illustrated in Figure 5
tailored to counter DreamBooth fine-tuning. We adapted their approach for our setting focused on style mimicry, retaining their hyperparameters where feasible. We established the following parameters: the number of iterations N = 50, PGD perturbation budget = 8/255, PGD step size a = 5 10-3, and the number of PGD steps per ASPL iteration NPGD 6. The loss Fineture is minimized within the vanilla fine-tuning framework over 300 training steps.
Anti-DreamBooth Anti 40
MetaCloak We implement MetaCloak 29 using the original setting, with a surrogate pool of diffusion models (M 5). The transformation set 'T includes Gaussian filtering (kernel=7), random flips, and center cropping. We use Adam optimizer with B 10-4 and C 4000 crafting steps. The denoising-error maximization loss combines with EOT to ensure transformation robustness.
SimAC Following [4].
we implement their feature interference loss with adaptive timestep selec- tion. The perturbation budget matches other baselines (Exe 8/ 8/255), and use their recommended layer weights 1) for high-frequency feature disruption. The hyperparameters are same with the official implementation, with number of training epochs of 50 and each epoch include 3 steps for surrogate model training and 9 steps for the PGD attacks. For the timestep search, the maximum greedy search steps is set as 50.
A.5 Attack Implementation Details
The implementation details of noise upscaling. Previous work have found that upscaling images can purify adversarial images [33]. Recent work improve this by first applying Gaussian noises and then upscales the noisy image upscaling the images. our experiment, we train on the -D-xc-upscaler model and then test on the model. We select these two models because they have different architectures, therefore we can test the transferability our methods. Specifically, the SD-x4 first encodes image through a VAE encoder, producing latent features of shape [B, 4, H, W). It then combines downscaled image with the latent features, resulting tensor of shape [B, 7, H, W]. contrast, the ID-x2-latent directly utilizes the latent features.
[9]. However, [19] does not specify the model they used for
21

---

Reference Image
Image finctuned on the clean image, with prompt "an sks painting of house"
Target Image used
Glaze
Glaze, using paintings from Artist Rembrandt target, with prompt sks painting
Target Image
Mist
Mist, with prompt painting of house"
Figure 5: Visualization of Glaze and Mist. For Glaze, reference and paintings from Rembrandt as the targets. The results indicate that Glaze produces images that are mixture of the target and reference styles. For Mist, use periodic image the target, according to the original paper.
paintings from Van Gogh the
A.6 Evaluation Metries
For style mimicry attack, we use three different metrics, FID, precision and success rate.
FID. The Fréchet Inception Distance (FID) measures the statistical similarity between real and generated images by comparing their feature distributions in Inception-v3's latent space. High FID indicates that generated images deviate from the real data manifold, making style imitation harder. Unlike Precision (which measures mode coverage), FID penalizes unnatural artifacts, making it ideal for measuring adversarial disruption.
Precision. The precision metric is computed to evaluate the quality of generated images by assessing their fidelity the target data manifold. Following the methodology proposed by [23], we implement a manifold-bass approach for precision computation. Given two sets reference features Given feature embeddings F, (real data) and F, (generated data) we proceed as follows: For each real sample f,- Fr, compute its k k-nearest neighbor radius R, (f,)) in F,.
feature mebeddingss-referencee
Precision
1
Σ E [[,, f.l1 (f.)) F,
(10)
F 9
where I(.) is the indicator function. Higher precision indicates better coverage of real data modes. To exclude the influence of image content in the style mimicry attack, first the fine-tuned model on a clean image to generate 100 new paintings a specific category (e.g., an SKS painting of house). We then the fine-tuned model the gyyle-guarded images to generate another set of 100 images with the same prompt. These two sets of images are then used to compute the precision.
Mimicry Success Rate. Mimicry success rate employs human annotators pairwise comparison protocol evaluate generated images from unprotected inputs versus those
22

---

(a) DreamBooth-LoRA, finetune SD v2.1 on unprotected image
(b) DreamBooth-Loud LoRA, finetune SD v2.1 on protected image
cc) DreamBooth-LoRA, finetune SD XL on protected image
(d) DrcamBooth-LoRA, finctunc SD XL on unprotected image
Figure 6: Comparison of images generated from unprotected and protected images using LoRA methods. We utilized the DreamBooth loss to fine-tune the SD v2.1 and SD XL models. Our findings reveal that method significantly reduces image quality for the SD XI model. Although the image quality degradation for the SD v2.1 model less pronounced, there is still a notable change in style.
from protected images.
success rate
Σ
Σ
[robust mimicry preferred over unprotected mimicry]
No No prompt annotator
(11)
A perfectly robust mimicry method would thus obtain a success rate of 50% indicating that its outputs are indistinguishable compared with the unprotected method. In contrast, severely restricted protection would result in success rates around 0% for robust mimicry methods, indicating that mimicry top of protected images always yields worse outputs. In the experiment, we use 5 different annotators. Each annotator needs to compare 10 image pairs for each transformation and protection method.
For the personalization attack, we use identity match score (IMS), which computes the similarity between the embedding of generated face images and average of all reference images. We use VGG-Face and ST.T--ViT- as embedding extractors extract face features and employ the cosine similarity.
23

---

A.7 Human Evaluation Results on the Style Mimicry
To further evaluate the success rate under different transformation settings, we asked human anno tators to compare images generated by models fine-tuned on unprotected versus protected images. Annotators assessed these two sets of images based on style and quality, selecting which set exhibited better quality. Thus, successful mimiery attack would yield a success rate of nearly 50%, indicating competitiveness with images trained on clean samples.
We employed five different annotators, each tasked with comparing ten image pairs for transforma- tions and protection methods. Metrics were computed using Equation [1] The results are presented in Figure 4 which shows that when purifications such as DiffPure and Noise Upscaling are applied, our method's 's success rate is significantly lower than baseline methods.
A.8 Evaluation Results on the Personalization
We evaluated the effectiveness of our method in defending against personalization attacks, where an adversary attempts to generate customized face images using a tine-tuned model. As shown in Z we successfully defend against the personalization attack by DreamBooth, even when Noise Scaling is used to preprocess the face images. The top row, X represents the original, unaltered input, while the images labeled Xp- correspond to the perturbed images generated by StyleGuard.
For quantitative analysis, we selected 100 identities from the CelebA dataset, choosing 10 images for each identity to fine-tune the LDM using the DreamBooth loss function. The PGD budget was to 16/255. The success of the defense is measured by the Identity Matching Score [40), which computes the cosine distance between the generated images and the average face embedding of the user's clean image set using the VGG-Face and CLIP-ViT-base ViT-base-32. A lower ISM indicates that the model cannot reproduce images of the same identity. The results are shown in Table [S] It is evident that when no purifications are applied, all methods achieve successful protection against anti-persoonllizatin.. The best method, MetaCloak, achieves an / SMS.IIP of 0.662 and an 7 /MSya of -0.051. However, when noise upscaling is introduced, the protective strength of previous methods significantly decreases. In contrast, our method continues to effectively mitigate personalization attacks.
Table 5: Comparison of Identity Matching Score on the anti anti-personalization for different methods. The lower the IMS, the stronger the protection is. When purifications applied, all methods achieve successful protection against unti-personalization. However, when noise upscaling is introduced, the protective strength of previous methods significantly decreases. In contrast, our method continues to effectively mitigate personalization attacks.
Method
IMSLLPP
IMSvgG
IMSeLIP with UpScale
IMSvGG with Upscale
Clean
0.814
0.432
0.805
0.429
Boott
0.695
-0,012
0.725
0.379
SimAC
0.675
-0,022
0.780
0.373
MetaCloak
0.662
-0.051
0.748
0.354
Our Method
0.680
-0.039
0.685
0.120
A.9
Evaluation over Online Black-box Upscaler
We use an online black-box upscaler. Finegrain Image Enhancer (FIE) (1, further evaluate our method. For FIE, we use the default setting, with the upscale factor set as 2 and the ControlNet scale set as 0.6. The Gaussian noise standard deviation set as 0.1. Figure I shows the results of fine-tuning using Van Gogh-style image. The first row is the result of fine-tuning using a clean image. The second row is the result of fine-tuning using a protected image after purification and then using Dreambooth. When using FIE for purification, the style of the image is somewhat similar to Van Gogh's ' style, but it is still not as good as the fine-tuning result of the clean image. We believe that this is because FIE has high degree of denoising, which can purify noise to certain extent, but will also modify the content of the image to certain extent, such as the details of the brushstrokes. changes in lines, etc., thus causing the style of the fine-tuned image to change.
24

---

Xctean
A person of sks
A person of sks
Finetuning on Xelaan
wearing sunglass
Xp by StyleGuard
A person of sks
person
Finetuning on X5
wearing sunglass
T(X;) by Noise Upscalc
A person of sks
Finetuning on T(X7)
A person of sks wearing sunglass
Figure 7: Defense against Personalization Attack by Dreambooth using StyleGuard.
SD model trained with clean images, with prompt "a <sks> painting of people waking on the street"
SD model trained with protected images (purified with FIE), with prompt "a <sks> painting of people waking on the street"
Figure 8: Visualization over Online Black-box Upscaler FIE. When using FIE for purification, the image style is more similar Van Gogh's 's than without purification, but it still good as the fine-tuning result from the clean images.
25

---

B. Additional Demonstration of Visual Examples
B.1 Compare the Clean, Protected and Upscaled Images
To demonstrate the effectiveness of our method, Figure 9 compares original, clean images without any processing with protected versions after applying our protection methods. and the noise-upscaled images that the adversary applies Noise Upscale with a different version of the upscale model (SD- x2-latent-upscaler) from the training stage to the protected images. We have made some findings as follows. First, it is shown that the noise introduced by our method is very small and does not affect the image quality. Second, Noise Upscale can better restore image details, such the face in the sixth row. We think this may be because Van Gogh's is image appears in the Upscaler training dataset. However, for some parts the image style, Noise Upscale cannot be restored well, such as the sky in the first row and the grass in the fifth row, which become even blurrier after Upscale. We think this is because these images may not in the training images of the Upscale model.
B.1 Compare the Images Trained on Clean Images and Protected Images
Figure 10 and Figure I compare the results of style mimicry on clean and protected images with the StyleGuard protection. For StyleGuard, we generate perturbations using SD1.4 and SD x4 upscaler. During the test, we first apply the Noise Upscale using the SI) x2 upscaler and then train the SD1.5 model on the protected paintings. With protection, the quality of the protected image decreases significantly, and its style changes from the original image.
26

---

Figure 9: Visual comparison between (a) clean/original images (left column), (b) protected images (middle column), and (c) Noise Upscaled results (right column). Each row shows the same image processed through different pipeline stages.
27

---

Figure 10: The results of style mimicry on clean images without any protection. We train the SD vl.5 model on Van Gogh's 's paintings.
28

---

Figure 11: The results of style mimicry protected images with the StyleGuard protection. For the StyleGuard, we generate perturbations using SDLA4 and SD x4 upscaler. During the test, we first apply the Noise Upscale using the SD x2 upscaler and then train the SD1.5 model on the protected paintings.
29


Reviews:
[{"review_id": "vFu0abkDN6", "reviewer": "Reviewer_YoSP", "summary": "This paper introduces StyleGuard, a novel method to protect artists' work from style mimicry attacks by text-to-image diffusion models like Stable Diffusion. The core problem it addresses is that existing protection methods (e.g., Glaze, Anti-DreamBooth) are vulnerable to new \"purification\" attacks (e.g., DiffPure, Noise Upscaling), which can remove the protective noise. Furthermore, existing defenses often lack transferability, failing to protect against unknown or black-box models.\n\nStyleGuard's main contributions are two novel loss functions used to generate a robust, imperceptible perturbation on images:\n1.  A \"Style Loss\" that manipulates the mean and variance of features in the latent space. This perturbs the global style characteristics of an image, making it difficult for models to learn the correct style and enhancing the protection's transferability across different text-to-image models.\n2.  An \"Upscale Loss\" designed to specifically counter purification attacks. It trains the perturbation against an ensemble of different purifiers and upscalers, aiming to maximize the denoising error and essentially making the protective noise \"purification-resistant.\"\n\nThe authors validate StyleGuard through extensive experiments on the WikiArt and CelebA datasets. The results show that their method significantly outperforms existing baselines in robustness against various data transformations and, crucially, against state-of-the-art purification methods.", "questions": "Thank you for this interesting work. The proposed ideas are novel, but I have concerns about the methodological rigor and the breadth of the evaluation, which prevent me from giving a higher score at this time. My questions are as follows, ordered by importance:\n\n1. The `Style Loss` depends on a target image from a different genre, a critical step described as being \"randomly selected.\" How sensitive is the protection efficacy to the choice of this target? For instance, does a target style that is maximally dissimilar provide stronger protection, and is there a systematic way to choose an optimal target?\n\n2. The methodology relies on several key hyperparameters, such as the `Style Loss` weight (λ=10) and the specific ensemble of purifiers for the `Upscale Loss`. Could you provide ablation studies that analyze the sensitivity to these parameters and offer a justification for the chosen values?\n\n3. The evaluation focuses on individual purification methods. To better assess real-world robustness, have you considered testing against a pipeline of combined, simpler transformations (e.g., cropping, rotation)?\n\nMy final recommendation is contingent on the authors' response to these concerns. A more rigorous analysis of the method's sensitivity to its core parameters (Questions 1 & 2) and a more comprehensive robustness evaluation (Question 3) would be necessary to move this paper into a clear accept category.", "limitations": "The authors provide a thorough analysis of the potential limitations of their method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This paper addresses a critical and timely problem with a novel solution. However, while the strengths in originality and significance are clear, they are counterbalanced by notable weaknesses in methodological rigor and the breadth of experimental validation, placing the paper on the borderline for acceptance.\n\nStrengths:\n\n1. The problem of efficiently protecting artists from AI-driven style mimicry is of great practical importance. The proposed StyleGuard framework, particularly its \"Upscale Loss\" designed to combat purification attacks, is a novel and conceptually clever contribution to this domain.\n\n2. The experimental evaluation is mostly thorough. The authors compare StyleGuard against a comprehensive set of baselines and test against a strong and relevant suite of attacks, including the specific purification methods that this work aims to defeat.\n\n3. The paper is generally well-written and easy to follow, with figures that help to illustrate the overall system architecture.\n\nWeaknesses\n\n1.  The core `Style Loss` requires a target image from a different art genre, but the selection process is vaguely described as \"randomly selected.\" The choice of target style is critical, yet the paper does not explore how the degree of stylistic difference impacts protection efficacy, nor does it propose a systematic selection strategy. This ambiguity limits the method's reproducibility and reliability.\n\n2. The algorithm relies on fixed hyperparameters, such as the ensemble of purifiers for the `Upscale Loss` and the style loss weight (λ=10). The paper lacks ablation studies to analyze the sensitivity to these parameters or to justify why these specific values were chosen. We cannot know if these choices are optimal or how robust the method is to their variation.\n\n3.  The paper's evaluation of robustness is narrowly focused, primarily validating the defense against sophisticated, diffusion-based purifiers. However, it neglects to test against a wider range of transformations common in real-world scenarios. For instance, the paper does not evaluate robustness against a \"stress test\" of combined transformations (e.g., cropping, followed by rotation and compression). This limited scope makes it difficult to fully assess the protection's practical robustness.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for insightful comments.  Below, we respond to the key points raised.\n\n### Response to Q1\n\nTo assess the impact of different target styles on our method, we selected ten distinct art genres, each represented by 30 images. We conducted simplified attacks while omitting the upscale loss to focus on style influence. To measure style variation, we utilized CMMD [1], an enhanced metric over FID, where a higher CMMD value indicates a greater divergence from the training dataset. The second row in the results reflects the CMMD distance of target style images compared to the training dataset.  The third to the fifth rows are the FID, CMMD, and Precision scores evaluated on the generated images. Here, we computed the style loss using the average mean and variance of style features on images in the same genre. **We find that a larger CMMD distance between the target style dataset and the training dataset correlates with a significant decline in generated image quality**, as evidenced by higher FID and CMMD values and lower Precision scores. \n\n#### Table 1: Evaluating the influence of the target styles\n| Style   | Sketch | Stick Figure | Post Impression | Ink Splash | Surrealism | Periodic Image | Gray Photo | Realism | Watercolor Painting | Cubism |\n|--------------------|--------|--------------|----------|------------|------------|------------------|------------|---------|------------|--------|\n| CMMD to Training Dataset | 4.67  | 4.72| 4.85 | 5.15 | 5.71| 6.13  | 6.42 | 6.81   | 6.88     | **7.58**  |\n| FID| 167.07 | 179.35 | 178.54  | 173.18  | 161.45    | 183.99 | 183.2  | **223.92** | 196.95    | 212.71 |\n| CMMD| 1.83  | 2.20| 2.45  | 1.20| 1.60  | 2.86| 1.016 | 3.319   | 3.17  | **3.65**  |\n| Precision| 0.20  | 0.37| 0.16 | 0.25 | 0.29 | **0.04**| 0.22| 0.09    | 0.12 | 0.08  |\n\nInspired by this, we introduce a straightforward method for selecting the target image. This method evaluates their effectiveness using CMMD distances and FID scores. The complete process can be executed in under an hour utilizing 8 3090 GPUs.\n\n- Step 1: Begin by selecting 10 distinct art genres from an open-source dataset such as WikiArt.\n- Step 2: Calculate the CMMD distance between each image in these styles and the training dataset.\n- Step 3: Identify and select the image with the highest CMMD distance in each style. \n- Step 4: Perform a Simplified StyleGuard Attack\n  - For each of the 10 selected images:\n    - Train for 10 epochs using the selected target images while omitting the upscale loss.\n    - Using the generated images with protective perturbations to fine-tune the SD model for 200 epochs using DreamBooth.\n    - Record both FID (Fréchet Inception Distance), Precision, and CMMD values at the end.\n- Step 5: Choose the target image with the largest CMMD with FID>200 and Precision < 0.2 as the final target image for further optimization.\n\n### Response to Q2\nThanks for your valuable suggestions. We conducted ablation studies on the style loss weight $\\lambda$, the attack budget ($B_{\\inf}$ in Eq.8), upscale loss weight $\\eta$, $K_1$, and $K_2$ to explore their influence on the attack effect. To assess the generated image qualities of the fine-tuned SD model, we adopt two additional metrics except FID and Precision, including LPIPS[1], and CMMD[2]. \n\nWe first generate perturbations on the SD1.4 and upscaler x4. Then we upscale the images using a different upscaler x2, and fine-tune the SD1.4 and SD1.5, respectively, using DreamBooth. In all experiments, the first stage (generating protection noise) trains for 10 epochs. The second stage (Dreambooth fine-tuning) trains for 200 epochs. We also fine-tune an SD1.4 and an SD1.5 model with clean paintings as the reference models.  Next, we generate the reference image set $I_{ref}$ and the attacked image set $I_{attack}$ under the same prompt. Finally, we compute the above metrics using $I_{ref}$ and $I_{attack}$. The results are shown in Table 2 and Table 3. \n\nExperiments show that as $\\lambda$ increases, FID, LPIPS, and CMMD first increase and then decrease, and the Precision first decreases and then increases. This indicates that there is a balance between denosing loss and style loss. This could be attributed to the fact that the two different targets of the denosing loss and style loss may not be always consistent and might partially controdictary with each other, in which the aim of the denosing loss is to pull the representation of the image out of the feature space of the diffusion model, and the aim of style loss is to make the style features of the generated image close to the target image and away from the original image. \n\n#### Table 2: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD14)\n| $\\lambda$ | FID   | Precision | Recall  | LPIPS  | CMMD   |\n|--------------|-------|-----------|---------|--------|--------|\n| 1 | 197.38| 0.37     | 0.63   | 0.70    | 3.05  |\n| 10| $\\mathbf{214.27}$ | $\\mathbf{0.25}$      | 0.63   | $\\mathbf{0.71}$   | $\\mathbf{4.02}$  |\n| $10^2$    | 194.84| 0.41      | 0.58    | 0.69  | 3.09  |\n| $10^3$     | 188.64| 0.52      | $\\mathbf{0.33}$    | 0.65  | 2.22  |\n\n#### Table 3: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD15)\n| $\\lambda$  | FID   | Precision | Recall  | LPIPS  | CMMD   |\n|--------------|-------|-----------|---------|--------|--------|\n| 1 | 184.32| 0.29| 0.83   | 0.68  | 2.37|\n| 10| $\\mathbf{227.20}$| $\\mathbf{0.25}$| 0.72| $\\mathbf{0.70}$  | $\\mathbf{3.22}$   |\n| $10^2$| 194.32| 0.41| 0.50| 0.67  | 2.18  |\n| $10^3$| 180.21| 0.54| $\\mathbf{0.41}$| 0.63| 1.69  |\n\nThe ablation study results of the attack budget are shown in Table 4.  It is shown that as the budget increases, FID, LPIPS, and CMMD gradually increase, while Precision gradually decreases. When the budget is higher than 0.063, increasing the budget does not significantly improve the protection effect. This is because the difference in the generated images is already large enough. In addition, a too high budget will introduce visible protection noise, which will damage the quality of the protected image.\n\n#### Table 4: Ablation Study on the Attack Budget (Dreambooth Finetuning on SD15)\n| Budget | FID   | Precision | Recall  | LPIPS  | CMMD   |\n|-------|-------|-----------|---------|--------|--------|\n| 2/255| 165.76 | 0.39 | 0.52   | 0.60  | 1.09 |\n| 4/255 | 199.78 | 0.29 | 0.77  | 0.62  | 1.15 |\n| 8/255| 190.99 |  0.24 | $\\textbf{0.50}$   | 0.63  | 2.61  |\n| 16/255| $\\textbf{271.80}$ |$\\textbf{0.04}$ |0.87| 0.67 |3.55 |\n| 32/255| 256.23   |  0.08 | 0.81   | $\\textbf{0.70}$  | $\\textbf{3.71}$  |\n\nFor the ablation study of upscale loss weight $\\eta$, $K_1$, and $K_2$, please refer to Response to Q1 of Reviewer 2.\n\nWe also evaluate the influence of different purifiers used for the upscale loss. The result is shown in Table 5. We compute the upscale loss on three different settings (X2, X4, and X2+X4), and then use three different purifiers in the attack phase, including X2, X4, and a new purifier, SinSR [4]. The result shows that when both X2 and X4 are used for the upscale loss, the protection performance on the SinSR significantly improved. This shows that using an ensemble of purifiers can improve the protection strength.\n\nTable 5: Evaluate the Different Purifiers for the Upscale Loss\n| | X2  |  || X4  |  |  | X4+X2     |  |  |\n|-------|--------|---------|-----------|-----------|-----------|-------|-----------|-----------|-----------|\n|  | **FID** | **CMMD**| **PRE**   | **FID**   | **CMMD**  | **PRE**   | **FID**   | **CMMD**  | **PRE**   |\n| **X2**    | 197.637   | 2.9828    | 0.208     | 186.303   | 0.56  | 0.341 | 204.979   | 2.879     | 0.166     |\n| **X4**    | 188.308   | 2.205     | 0.25 | 191.35    | 2.748     | 0.2916 | 253.939   | 2.735     | 0.166 |\n| **SinSR** | 167.489   | 0.64  | 0.458 | 183.119   | 1.116  | 0.208  | 183.643   | 1.887 | 0.163 |\n\n### Response to Q3\nThanks for your suggestion. To defend against transformations such as crop and rotation, we utilize Expectation over Transformation (EoT) in our method (L194-195 in our paper). We have evaluated the crop and resize transformation in our paper (Table 1). To further evaluate the robustness of our method to transformation combinations, including crop and resize, rotation, and JPEG compression, we generate perturbations using SD 1.4 and then fine-tune the SD 1.5 model on the protected images. We compare our methods with SimAC. The training epochs for both methods are set to 20. Table 6 shows the results of the different combinations of transformations. The results show that we consistently outperform the SoTA baseline on different transformations, indicating that our method can effectively resist the combination of different transformations.  \n\nTable 6: Evaluation on Diverse Transforms\n| Method  | Crop| | | Crop+Rotate  |  |  | Crop+JPEG Compress |  |     | Crop+Rotate+JPEG Compress |  | |\n|---------|---------|----------|--------|------------------|------------------|--------|----------|-----------|--------|-----------|-------|--------|\n|  | **FID** | **PRECISION**    | **CMMD** | **FID**  | **PRECISION**    | **CMMD** | **FID** | **PRECISION**    | **CMMD** | **FID**   | **PRECISION** | **CMMD** |\n| **SIMAC** | 271.463 |   0.04 | 2.146  | 202.784 | 0.167  | **2.908** | 190.32| 0.083| 3.583| 189.894 | 0.25  | 1.614|\n| **Ours** | **316.803**  | **0.00**    | **2.189** | **254.283** | **0.083**   | 2.379  | **321.339**| **0.00**| **3.971** | **245.415** | **0.00** | **3.636**  |\n\nReference\n[1]  The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018.\n[2]  Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Arkiv2401.09603, 2024\n[3]  Improved Precision and Recall Metric for Assessing Generative Models. NIPS, 2019\n[4] SinSR: diffusion-based image super-resolution in a single step, CVPR2024"}, {"review_id": "0TLH82HD3F", "reviewer": "Reviewer_bWmD", "summary": "This paper proposed StyleGuard as an end-to-end defence against style mimicry attacks. The defensive perturbation is produced by alternatively updating the diffusion model and the PGD updates on images. There are three terms to drive the udpate: denoise loss, style loss, and upscale loss. The authors evaluated StyleGuard on WikiArt and CelebA to demonstrate that the method raises FID and drops CLIP precision even after DiffPure/Noise-Upscale purification.", "questions": "See above.", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- **Method**. The design of the loss is startghtforward but deals with multiple intents. The upscale loss enables StyleGuard to anticipate purifiers' erasing and it uses an ensemble of purifiers as noise that beats DiffPure might be dropped by an upscaler. This upscale loss serves as StyleGuard's insurance to make the defensive perturbation survive standard clean up mechanisms. So such design makes sense. Another ingredient style loss perturbs global latent statistics, which is the weakness of Mist, to make StyleGuard robust to cropping/rescaling/Gaussian noise. \n\n- **Empirical**. The evaluation is done on DreamBooth and Textual-inversion, full/LoRA fintuning, and black-box upscalar FIE. And the model is acorss SDV1.4/1.5/2.1/XL. I think the evaluation part is thorough.\n\nLimitations \\& Questions:\n\n(1) $\\lambda$, $\\eta$, budget, K1, K2 parameters are fixed, Have the authors done ablation study to see how sensitive is the method related to these parameters?\n\n(2) Can the authors provide detailed computational cost of each phase, with comparison to baselines?\n\n(3) The purification is emprical without certified bound. DiffPure is frozen in the upscale loss phase. consider an adaptive attacker that can jointly finetune the purifiers with DreamBooth such that the gradient can realign. Have the authors consider adpative attacks? Or consider how StyleGuard can provide/further develop certified bound? An ablation with different level of bugdet discussion might be helpful.\n\n(4) Minor. It is better to not use the abbreviation PGD in line 34 (first use in the paper)\n\nOverall, the idea is straightforward to follow and its underlying idea remains in line with established methodologies although achieves good empirical results. If the authors can resolve my above concerns, I am willing to raise my score.", "was_revised": true, "final_justification": "- The authors did ablation studies regarding $\\lambda$, $\\eta$, $K_2$ and $K_2$ thouroughly and provided the cost compared to the baseline. \n- Regarding the question of adaptive attack, the authors did not give a specific solution but said they will explore in future work. \n- The authors further discussed the defense bound with ablation on different attack settings (X2, X4, SinSR). \nOverall, I feel the authors solved my concerns, regarding the adaptive attack, I feel it might be good to mark this in the limitation and encourage future exploration. Therefore I would recommend score 4.", "rebuttal": "We sincerely thank the reviewers for the constructive comments. In response to Q1, we add ablation studies and detailed explanations of the metrics. In response to Q2, we compare the time cost of our method with the baseline method. \n\n### Response to Q1\nWe conducted ablation studies on the style loss weight $\\lambda$, the attack budget ($B_{\\inf}$ in Eq.8), upscale loss weight $\\eta$, $K_1$, and $K_2$ to explore their influence on the attack effect. To assess the generated image qualities of the fine-tuned SD model, we adopt two additional metrics, in addition to FID and Precision, including LPIPS[1] and CMMD[2].  CMMD is an improvement metric for FID, which can better measure the image distortions. The higher the value, the greater the gap with the training dataset.\n\nWe first generate perturbations on the SD1.4 and Upscaler x4. The training dataset includes 15 images from Van Gogh. Then we upscale the images using a different upscaler x2, and fine-tune the SD1.4 and SD1.5, respectively, using DreamBooth. In all experiments, the first stage (generating protection noise) trains for 10 epochs. The second stage (Dreambooth fine-tuning) trains for 200 epochs. We also fine-tune an SD1.4 and an SD1.5 with clean images as the references.  Next, we use the reference models and adversarially fine-tuned models to generate the reference image set $I_{ref}$ and the attacked image set $I_{attack}$ under the same prompt (such as \"an \\<sks\\> painting of a house). Finally, we compute the above metrics using  $I_{ref}$ and $I_{attack}$. The results are shown in Table 1 and Table 2.\n#### Table 1: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD14)\n| $\\lambda$ | FID| Precision | Recall  | LPIPS  | CMMD   |\n|-------|-------|-----------|---------|--------|--------|\n| 1 | 197.38| 0.37 | 0.63   | 0.70    | 3.05  |\n| 10| $\\mathbf{214.27}$ | $\\mathbf{0.25}$      | 0.63   | $\\mathbf{0.71}$   | $\\mathbf{4.02}$  |\n| $10^2$ | 194.84| 0.41| 0.58    | 0.69  | 3.09  |\n| $10^3$ | 188.64| 0.52| $\\mathbf{0.33}$    | 0.65  | 2.22  |\n\n#### Table 2: Ablation Study on the Style Loss Weight (Dreambooth Finetuning on SD15)\n| $\\lambda$  | FID   | Precision | Recall  | LPIPS  | CMMD   |\n|-------|-------|-----|----|--------|--------|\n| 1 | 184.32| 0.29| 0.83   | 0.68  | 2.37|\n| 10| $\\mathbf{227.20}$| $\\mathbf{0.25}$| 0.72| $\\mathbf{0.70}$  | $\\mathbf{3.22}$   |\n| $10^2$| 194.32| 0.41| 0.50| 0.67  | 2.18  |\n| $10^3$| 180.21| 0.54| $\\mathbf{0.41}$| 0.63| 1.69  |\n\nExperiments show that as $\\lambda$ increases, FID, LPIPS, and CMMD first increase and then decrease, and the Precision first decreases and then increases. This indicates that there is a balance between denoising loss and style loss. This could be attributed to the fact that the two different targets of the denoising loss and style loss may not always be consistent and might be partially contradictory to each other. The aim of the denoising loss is to pull the representation of the image out of the feature space of the diffusion model, and the aim of the style loss is to make the style features of the generated image closer to the target image and further from the original image. \n\nTable 3 displays an ablation study on the upscale loss weight, $\\eta$. The findings indicate that a low weight of 0.1 results in the least effective protection, evidenced by a low FID score of 135.466 and a precision of 0.625. In contrast, higher weights, such as 10, offer better protection. However, as the value of $\\eta$ increases from 10 to 100, the FID and the CMMD decrease. This trade-off implies the existence of a balanced weight that maximizes performance across the metrics.\n#### Table 3:  Ablation Study on the Upscale Loss Weight\n|$\\eta$|FID|Precision|LPIPS|CMMD|\n|-----------|---|---------|-----|-----|\n|0.1|135.466|0.625|0.625|0.822|\n|1|173.399|0.375|$\\textbf{0.686}$|1.366|\n|10|$\\textbf{184.747}$|$\\textbf{0.167}$|0.677|$\\textbf{1.711}$|\n|100|182.477|0.333|0.665|0.644|\n\nThe ablation study of the attack budget is shown in Table 4.  It is shown that as the budget increases, FID and CMMD gradually increase, while Precision gradually decreases. When the budget is higher than 0.063, increasing the budget does not significantly improve the protection effect. This is because the difference in the generated images is already large enough. In addition, a too high budget will introduce visible protection noise, which will damage the quality of the protected image.\n\n#### Table 4: Ablation Study on the Attack Budget (Dreambooth Finetuning on SD15)\n| Budget | FID | Precision | Recall  | LPIPS  | CMMD   |\n|--------------|-------|--------|------|-----|-----|\n| 2/255| 165.76 | 0.39 | 0.52 | 0.60  | 1.09 |\n| 4/255| 199.78 | 0.29 | 0.77| 0.62  | 1.15 |\n| 8/255| 190.99 |  0.24 | $\\textbf{0.50}$   | 0.63  | 2.61  |\n| 16/255| $\\textbf{271.80}$ |$\\textbf{0.04}$ |0.87| 0.67 |3.55 |\n| 32/255| 256.23 | 0.08 | 0.81| $\\textbf{0.70}$  | $\\textbf{3.71}$  |\n\nTable 5 shows the ablation study on the $K_1$ and $K_2$. We highlight the top two best values for each metric in bold.  It can be observed that when $K_1$=10 and $K_2$=5, better protection effects are achieved across four metrics. The next best results are achieved when $K_1$=5, $K_2$=5, with good protection effects across three metrics. The results also show that the protection effect is significantly weakened when the difference between K1 and K2 is large (for example, when K1=5, K2=20 or K1=20, K2=5, the CMMD values are only 0.674 and 0.704). We think this may be because when the difference between K1 and K2 is large, the noise is more likely to fall into overfitting.\n#### Table5: Ablation Study on $K_1$ and $K_2$ (Dreambooth Finetuning on SD15)\n|$K_1$|$K_2$|FID|Precision|Recall|LPIPS|CMMD|\n|---|---|---|---|---|---|---|\n|5|5|**211.728**|0.292|0.169|**0.682**|**2.314**|\n|5|10|155.110|0.291|0.472|0.665|1.369|\n|5|20|146.956|0.458|0.444|0.676|0.674|\n|10|5|**231.200**|**0.167**|0.889|**0.705**|**2.472**|\n|10|10|164.830|0.25|0.667|0.671|1.725|\n|10|20|144.150|0.625|**0.167**|0.658|0.646|\n|20|5|163.648|0.208|0.194|0.649|0.704|\n|20|10|194.953|**0.125**|**0.168**|0.660|1.734|\n|20|20|180.729|0.168|0.583|0.677|2.245|\n\n### Response to Q2\nTable 6 details the computational time costs associated with different loss functions, while Table 7 provides a comparison between our approach and the baseline method, SimAC. Both methods were evaluated on two RTX 3090 GPUs with a batch size of 4, using parameters K1=10 and K2=5. The tables present the average time cost for various stages, the total time per epoch, and the total attack duration for each method.  Our method incurs a 36% higher time cost per epoch compared to SimAC. This is due to the need to compute three different losses during the PGD attack stage, whereas SimAC computes only two losses (feature loss and denoising loss). Despite the increased time per epoch, our method achieves a ~50% reduction in overall attack time compared to SimAC. This advantage arises because SimAC requires an additional 10 minutes for diffusion step selection before initiating the attack, which our method does not require.\n\n#### Table 6: Time Cost for Different Losses\n|Loss |Style Loss|Denoising Loss|Upscale Loss|\n|-----|----------|------|-------|\n|Time cost | 0.11s |0.12s|0.14s|\n\n#### Table 7: Comparing the Time Cost with the Baseline\n| Method | Pretraining | PGD attack | Post-training | Average time per epoch (batch size=4) | Total time (10 epochs) |\n|-----|-----|---------|-------|------|---------|\n| SimAC  | 3.14s | 2.98s | 2.72s  | 36.25s  | 16.25 min|\n| Ours   | 3.42s | 5.84s| 2.98s  | 49.32s | 8.22min|\n\n### Response to Q3\nThank you for inspiring further ideas for our experimental design. This is a very important suggestion. While current purifier-based attacks mainly focus on using pretrained diffusion models because of the high computation cost of training a new diffusion-based purifier, we agree that an adaptive attacker that can fine-tune a purifier on protected images poses more of a threat to previous methods and to ours. We will explore the feasibility of introducing adaptive attackers in experimental settings to test the limits and robustness of our method in future work. \n\nTo explore the defense bound, we use a novel purifier, SinSR, in the attack phase, which is unseen in the training. The result is shown in Table 5 in the response to Reviewer 1. We compute the upscale loss on three different settings (X2, X4, and X2+X4), and then use three different purifiers in the attack phase, including X2, X4, and SinSR [4]. The result shows that SinSR is able to achieve a certain attack effect. However, when both X2 and X4 are used for the upscale loss, the protection performance on the SinSR slightly improved. This shows that using an ensemble of purifiers can improve the protection strength. We also conducted the ablation study on the budget and show the results in Table 4. \n\nTable 5: Ablation Study on the Attack Budget (Dreambooth Finetuning on SD15)\n| | X2  |  || X4  |  |  | X4+X2     |  |  |\n|-------|--------|---------|-----------|-----------|-----------|-------|-----------|-----------|-----------|\n|  | FID | CMMD| PRE| FID | CMMD| PRE| FID | CMMD| PRE|\n| **X2**    | 197.637| 2.9828| 0.208| 186.303   | 0.56  | 0.341 | 204.979   | 2.879     | 0.166     |\n| **X4**    | 188.308| 2.205| 0.25 | 191.35| 2.748 | 0.2916 | 253.939   | 2.735     | 0.166 |\n| **SinSR** | 167.489| 0.64| 0.458 | 183.119| 1.116| 0.208  | 183.643   | 1.887 | 0.163 |\n\nFinally, thank you again for your in-depth analysis and valuable suggestions regarding our work. \n\nReference\n[1]  The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018.\n[2]  Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Arkiv2401.09603, 2024\n[3]  Improved Precision and Recall Metric for Assessing Generative Models. NIPS, 2019\n[4] SinSR: diffusion-based image super-resolution in a single step, CVPR2024"}, {"review_id": "Iu8eI7iSCz", "reviewer": "Reviewer_rV8Y", "summary": "The paper proposes StyleGuard, a system that injects imperceptible style perturbations into artwork, making it harder for diffusion models to reproduce the original style faithfully. The defense is realized by optimizing in the latent space of a style encoder trained on artist-specific data. Experiments show that StyleGuard can reduce precision and disrupt stylistic fidelity in downstream generation models.", "questions": "1) StyleGuard assumes a relatively narrow threat model where the attacker mimics an artist’s style using only prompt-based queries to a T2I model. However, it does not consider more advanced or adaptive adversaries, such as those employing prompt inversion techniques, personalized fine-tuning via DreamBooth or LoRA, or combining reference images with prompts for multimodal imitation. As text-to-image personalization becomes more accessible in open-source ecosystems, the proposed defense may fall short in real-world attack scenarios that involve stronger mimicry capabilities.\n\n2) The approach heavily relies on a style classifier trained per artist to guide perturbation optimization. This introduces two practical concerns. First, the classifier may overfit to known prompts or generations, failing to generalize to new model versions, prompting variations, or unseen creative domains. Second, the need to train and maintain one classifier per artist raises concerns about scalability, especially when attempting to protect large creator pools or adapt to evolving artistic styles.\n\n3) Although the authors claim that StyleGuard produces imperceptible perturbations, the paper does not report any quantitative perceptual quality metrics (e.g., LPIPS, SSIM, NIQE) or human perceptual studies. As a result, it remains unclear whether the perturbations remain visually invisible across different prompts or diffusion backbones, and whether they affect aesthetic quality in subtle but meaningful ways.\n\n4) Following the last concern, this paper lacks a user study, which is particularly important given that the proposed style-based defense appears to rely on perceptual or human-centered evaluation. Without human-in-the-loop assessments, it is difficult to fully validate the effectiveness and stealthiness of the defense. \n\n5) While the defence is designed to prevent unauthorized mimicry, it may also unintentionally interfere with legitimate use cases—such as the artist themselves using a T2I system to extend their own portfolio. The potential trade-off between style protection and usability in real-world workflows is not addressed, and may limit the deployment potential of the proposed method.\n\n6) Since StyleGuard depends on a style classifier trained per artist to generate perturbations, how well does this classifier generalize across prompts, unseen styles, or different diffusion models? Is there a risk that the protection degrades under cross-model or cross-domain mimicry attempts?\n\n7) The paper does not include experiments on adaptive attacks. It would be valuable to investigate how the defense performs when the attacker has partial knowledge of the defense mechanism, as this better reflects realistic threat models and can reveal potential vulnerabilities.\n\n8) There is no accompanying description or discussion for Table 2, which makes it difficult to interpret its relevance and implications. Furthermore, the reference in Line 283 appears to mistakenly point to Table 3 instead. In general, the references to tables and figures in the main text are not presented in a consistent or sequential order, which negatively impacts the readability and clarity of the paper.\n\n9) Several formatting issues, e.g., ClebA in line 61, the CLIP in Figure 2.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1) New Problem: The paper clearly identifies an underexplored but increasingly relevant threat model: unauthorized style mimicry in T2I models.\n\n2) Targeted Defense Design: The idea of style-specific imperceptible perturbations is elegant. The method is compatible with real-world deployment since it operates on the artist’s side, requires no model access, and preserves visual quality.\n\n3) Empirical Results: Quantitative and qualitative results across multiple metrics demonstrate that the perturbations are effective in breaking style imitation.\n\nWeaknesses\n\n1) Overfitting Risk: The defense relies on a style classifier trained on few samples from each artist, which may limit generalization to unseen prompts or model variants. Additional experiments on style drift (e.g., over long prompts or different seeds) would improve robustness evaluation.\n\n2) Lack of Standardized Attack Protocols: The paper evaluates mimicry via a style classifier and human judgments, but does not simulate realistic attacker pipelines. Incorporating these could make the evaluation more comprehensive.\n\n3) Lack of Experiments: While many experiments are conducted, there is no experiments on user study and adaptive attacks.", "was_revised": true, "final_justification": "The authors resolve my major concern about perceptual quality metrics and lack of adaptive attacks. While there still exist some small weaknesses, I recognize the authors' response and believe that they would be fixed well. Therefore, I recommend borderline accept.", "rebuttal": "We sincerely appreciate the reviewer's valuable suggestions and constructive feedback. In this response, we address the reviewer's concerns point by point.\n\n**Question 1**: StyleGuard assumes a relatively narrow threat model where the attacker mimics an artist’s style using only prompt-based queries to a T2I model. However, it does not consider more advanced or adaptive adversaries, such as those employing prompt inversion techniques, personalized fine-tuning via DreamBooth or LoRA, or combining reference images with prompts for multimodal imitation. \n\n**Response:** **Our method explicitly considers prompt inversion (Table 4), DreamBooth (Table 1), and LoRA fine-tuning (Table 3)** in our main paper and appendix. Experimental results demonstrate superior performance over baselines. While previous defenses, such as Mist and AntiDreambooth, focused primarily on DreamBooth attacks, our style loss formulation inherently provides protection against various attack modalities simultaneously. \n\n**Question 2**: The approach heavily relies on a style classifier trained per artist to guide perturbation optimization. \n\n**Response:** Our method **does not need a style classifier trained per artist to guide perturbation optimization**. Our approach uses general pre-trained feature extractors (VGG/VAE) to derive style representations from intermediate layer statistics (mean and variance), eliminating the need for artist-specific classifiers. This ensures scalability without compromising protection effectiveness. \n\n\n**Question 3**: Lack of perceptual quality metrics\n\nWe did not report perceptual quality metrics in our paper because we use the same attack budget (8/255) compared with the baseline methods. Because we cannot provide images for visualization, to further investigate the perceptual quality under different budgets, we conduct attacks under the same budget (8/255), and then compare the visual quality results with the baselines. We compared them with the original images using two widely adopted metrics: LPIPS [1] and SSIM [2]. LPIPS measures perceptual difference (lower = better quality). SSIM evaluates structural similarity (higher = better detail preservation)  \n\nWe compared against baseline approaches: Glaze, Mist, Anti-Dreambooth, SimAC, and MetaCloak.  Our method achieves **the best** LPIPS (`0.02`) and SSIM (`0.87`), which outperforms SimAC (LPIPS: `0.03`, SSIM: `0.82`) and MetaCloak (LPIPS: `0.03`, SSIM: `0.87`).\n\n### Table 1: Visual Quality Comparison between Original and Perturbed Images.\n| Method          | LPIPS ↓ | SSIM ↑  |\n|-----------------|------------------|---------|\n| Original Images | 0.00             | 1.00    |\n| Glaze           | 0.06             | 0.80    |\n| Mist            | 0.07             | 0.85    |\n| Anti-Dreambooth | 0.15             | 0.85    |\n| SimAC           | **0.03**         | 0.82    |\n| MetaCloak       | 0.03             | 0.85    |\n| **Our Method**  | **0.02**         | **0.87**|\n\n\n**Question 4**: Need for human evaluation\n\n**Response:** We evaluated our method using multiple quantitative metrics, including FID, Precision, and CMMD, to rigorously assess its effectiveness. Additionally, **we conducted a user study (detailed in Figure 4 of our paper)** where participants were asked: \"Based on the image style and quality, which image better fits the reference samples?\" The results demonstrated that, compared to baseline methods, our approach achieved the lowest success rate in style mimicry, confirming its superior protection capability.\n\n**Question 5**: Potential interference with legitimate use\n\n**Response:** Artists can always use unprotected versions of their own work for artwork expansion. Our method specifically targets unauthorized mimicry attempts while preserving legitimate usage rights.\n\n**Question 6**: Since StyleGuard depends on a style classifier trained per artist to generate perturbations, how well does this classifier generalize across prompts, unseen styles, or different diffusion models? Is there a risk that the protection degrades under cross-model or cross-domain mimicry attempts?\n\n**Response:** As mentioned earlier, our method does not rely on a specific style feature extractor, but an ensemble of pretrained image encoders, such as VGG VAE and CLIP. This design ensured that the generalizable style features are extracted. Through the style loss, we make it closer to the target style and farther away from the original style. The results showed that our method is generalizable to different models (Table 2 in our paper). Moreover, our method is also generalizable to different styles.\nTable 2 shows the results of using images with different styles as the target images. Note that we **do not** need to train a specific style classifier/extractor but use a general style extractor, such as VAE and VGG, to extract the style-related features. It is shown that we can successfully defend against the attacks by using target images from different styles.\n\n#### Table 2: Evaluating the influence of the target styles\n| Style   | Sketch | Stick Figure | Post Impression | Ink Splash | Surrealism | Periodic Image | Gray Photo | Realism | Watercolor Painting | Cubism |\n|--------------------|--------|--------------|------------------|------------|------------|------------------|------------|---------|----------------------|--------|\n| **CMMD Distance to Training Dataset**  | 4.67  | 4.72| 4.85 | 5.15      | 5.71      | 6.13  | 6.42 | 6.81   | 6.88     | **7.58**  |\n| **FID**| 167.07 | 179.35 | 178.54  | 173.18  | 161.45    | 183.99 | 183.2     | **223.92** | 196.95    | 212.71 |\n| **CMMD**  | 1.83  | 2.20| 2.45  | 1.20| 1.60  | 2.86| 1.016 | 3.319   | 3.17      | **3.65**  |\n| **Precision**     | 0.20  | 0.37| 0.16 | 0.25 | 0.29 | **0.04**| 0.22| 0.09    | 0.12     | 0.08  |\n\nTo evaluate the cross-domain mimicry attacks, we conduct experiments on the CelebA dataset, which contains faces from different identities. The results show that our method can successfully prevent identity customization attacks, as shown in Table 5 and Figure 7 in our paper. \n\n**Question 7**: The paper does not include experiments on adaptive attacks.\n\n**Response:** Thank you for inspiring further ideas for our experimental design. We admit that our present work did not consider the adaptive attacks because all of the current purifier-based attacks use pretrained diffusion models. We agree that an adaptive attacker that has defense knowledge poses more of a threat to previous methods and to ours. We will explore the feasibility of introducing adaptive attackers in experimental settings to test the limits and robustness of our method in future work. \n\n**Question 8**: There is no accompanying description or discussion for Table 2, which makes it difficult to interpret its relevance and implications. Furthermore, the reference in Line 283 appears to mistakenly point to Table 3 instead. \n\n**Response**: A comprehensive analysis of Table 2 has been provided in lines 283–288 of our paper. The citation in line 283 correctly refers to Table 2 (verified upon careful re-examination).\nLine 294 corresponds to the reference for Table 3. Tables 2 and 3 are on the same row, which may be the reason for misleading the reviewer.\n\n**Response to Q9:**\nThanks for your suggestion. We will revise these in our final version.\n\nWe hope our explanations address the reviewers' questions. We welcome any further questions."}]

Author rebuttal:

Meta-review: The paper proposes a defense system that adds small perturbations to protect digital artwork from being copied by text-to-image models. The reviewers found the idea important and timely, with the style loss and upscale loss being novel contributions that address both transferability and purification resilience. The experiments were thorough and showed strong results across DreamBooth, Textual Inversion, and multiple diffusion model versions. Weaknesses raised included unclear target style selection, lack of ablation on hyperparams, missing evaluation on combined real-world transformations, limited discussion of adaptive attacks, and some readability/formatting issues. The authors responded with detailed ablations, computational cost analysis, perceptual quality metrics, user study results, and multiple adaptive attack experiments. These additions significantly strengthened the submission, showing that the method is robust under varied conditions.

During discussion, reviewers initially leaned borderline, citing limited evaluation scope and lack of adaptive threat modeling. After rebuttal, most reviewers agreed the authors had addressed these concerns with extensive new experiments. One reviewer emphasized that adaptive attacks still deserve deeper treatment, but acknowledged the added experiments improved the situation. All reviewers converged on a borderline-accept stance, with updated justifications noting that while some limitations remain, the practical importance and clear novelty of styleguard make it a valuable contribution. Weighing the strong problem relevance, convincing empirical evidence, and the improved rigor after rebuttal, I recommend acceptance.
Decision: Accept (poster)