Title: StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations
Venue: NeurIPS 2025
Accepted: yes
Keywords: adversarial attack

style mimicry attack

Dreambooth

diffusion model
Abstract: Recently, text-to-image diffusion models have been widely used for style mimicry and personalized customization through methods such as DreamBooth and Textual Inversion. This has raised concerns about intellectual property protection and the generation of deceptive content.
Recent studies, such as Glaze and Anti-DreamBooth, have proposed using adversarial noise to protect images from these attacks. However, recent purification-based methods, such as DiffPure and Noise Upscaling, have successfully attacked these latest defenses, showing the vulnerabilities of these methods. 
Moreover, present methods show limited transferability across models, making them less effective against unknown text-to-image models.
To address these issues, we propose a novel anti-mimicry method, StyleGuard. We propose a novel style loss that optimizes the style-related features in the latent space to make it deviate from the original image, which improves model-agnostic transferability.
Additionally, to enhance the perturbation's ability to bypass diffusion-based purification, we designed a novel upscale loss that involves ensemble purifiers and upscalers during training.
Extensive experiments on the WikiArt and CelebA datasets demonstrate that StyleGuard outperforms existing methods in robustness against various transformations and purifications, effectively countering style mimicry in various models. Moreover, StyleGuard is effective on different style mimicry methods, including DreamBooth and Textual Inversion. The code is available at \url{https://github.com/PolyLiYJ/StyleGuard}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes StyleGuard, a system that injects imperceptible style perturbations into artwork, making it harder for diffusion models to reproduce the original style faithfully. The defense is realized by optimizing in the latent space of a style encoder trained on artist-specific data. Experiments show that StyleGuard can reduce precision and disrupt stylistic fidelity in downstream generation models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1) New Problem: The paper clearly identifies an underexplored but increasingly relevant threat model: unauthorized style mimicry in T2I models.\n\n2) Targeted Defense Design: The idea of style-specific imperceptible perturbations is elegant. The method is compatible with real-world deployment since it operates on the artist’s side, requires no model access, and preserves visual quality.\n\n3) Empirical Results: Quantitative and qualitative results across multiple metrics demonstrate that the perturbations are effective in breaking style imitation.\n\nWeaknesses\n\n1) Overfitting Risk: The defense relies on a style classifier trained on few samples from each artist, which may limit generalization to unseen prompts or model variants. Additional experiments on style drift (e.g., over long prompts or different seeds) would improve robustness evaluation.\n\n2) Lack of Standardized Attack Protocols: The paper evaluates mimicry via a style classifier and human judgments, but does not simulate realistic attacker pipelines. Incorporating these could make the evaluation more comprehensive.\n\n3) Lack of Experiments: While many experiments are conducted, there is no experiments on user study and adaptive attacks.\n\nQUESTIONS:\n1) StyleGuard assumes a relatively narrow threat model where the attacker mimics an artist’s style using only prompt-based queries to a T2I model. However, it does not consider more advanced or adaptive adversaries, such as those employing prompt inversion techniques, personalized fine-tuning via DreamBooth or LoRA, or combining reference images with prompts for multimodal imitation. As text-to-image personalization becomes more accessible in open-source ecosystems, the proposed defense may fall short in real-world attack scenarios that involve stronger mimicry capabilities.\n\n2) The approach heavily relies on a style classifier trained per artist to guide perturbation optimization. This introduces two practical concerns. First, the classifier may overfit to known prompts or generations, failing to generalize to new model versions, prompting variations, or unseen creative domains. Second, the need to train and maintain one classifier per artist raises concerns about scalability, especially when attempting to protect large creator pools or adapt to evolving artistic styles.\n\n3) Although the authors claim that StyleGuard produces imperceptible perturbations, the paper does not report any quantitative perceptual quality metrics (e.g., LPIPS, SSIM, NIQE) or human perceptual studies. As a result, it remains unclear whether the perturbations remain visually invisible across different prompts or diffusion backbones, and whether they affect aesthetic quality in subtle but meaningful ways.\n\n4) Following the last concern, this paper lacks a user study, which is particularly important given that the proposed style-based defense appears to rely on perceptual or human-centered evaluation. Without human-in-the-loop assessments, it is difficult to fully validate the effectiveness and stealthiness of the defense. \n\n5) While the defence is designed to prevent unauthorized mimicry, it may also unintentionally interfere with legitimate use cases—such as the artist themselves using a T2I system to extend their own portfolio. The potential trade-off between style protection and usability in real-world workflows is not addressed, and may limit the deployment potential of the proposed method.\n\n6) Since StyleGuard depends on a style classifier trained per artist to generate perturbations, how well does this classifier generalize across prompts, unseen styles, or different diffusion models? Is there a risk that the protection degrades under cross-model or cross-domain mimicry attempts?\n\n7) The paper does not include experiments on adaptive attacks. It would be valuable to investigate how the defense performs when the attacker has partial knowledge of the defense mechanism, as this better reflects realistic threat models and can reveal potential vulnerabilities.\n\n8) There is no accompanying description or discussion for Table 2, which makes it difficult to interpret its relevance and implications. Furthermore, the reference in Line 283 appears to mistakenly point to Table 3 instead. In general, the references to tables and figures in the main text are not presented in a consistent or sequential order, which negatively impacts the readability and clarity of the paper.\n\n9) Several formatting issues, e.g., ClebA in line 61, the CLIP in Figure 2.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed StyleGuard as an end-to-end defence against style mimicry attacks. The defensive perturbation is produced by alternatively updating the diffusion model and the PGD updates on images. There are three terms to drive the udpate: denoise loss, style loss, and upscale loss. The authors evaluated StyleGuard on WikiArt and CelebA to demonstrate that the method raises FID and drops CLIP precision even after DiffPure/Noise-Upscale purification.\n\nSTRENGTHS AND WEAKNESSES:\n- **Method**. The design of the loss is startghtforward but deals with multiple intents. The upscale loss enables StyleGuard to anticipate purifiers' erasing and it uses an ensemble of purifiers as noise that beats DiffPure might be dropped by an upscaler. This upscale loss serves as StyleGuard's insurance to make the defensive perturbation survive standard clean up mechanisms. So such design makes sense. Another ingredient style loss perturbs global latent statistics, which is the weakness of Mist, to make StyleGuard robust to cropping/rescaling/Gaussian noise. \n\n- **Empirical**. The evaluation is done on DreamBooth and Textual-inversion, full/LoRA fintuning, and black-box upscalar FIE. And the model is acorss SDV1.4/1.5/2.1/XL. I think the evaluation part is thorough.\n\nLimitations \\& Questions:\n\n(1) $\\lambda$, $\\eta$, budget, K1, K2 parameters are fixed, Have the authors done ablation study to see how sensitive is the method related to these parameters?\n\n(2) Can the authors provide detailed computational cost of each phase, with comparison to baselines?\n\n(3) The purification is emprical without certified bound. DiffPure is frozen in the upscale loss phase. consider an adaptive attacker that can jointly finetune the purifiers with DreamBooth such that the gradient can realign. Have the authors consider adpative attacks? Or consider how StyleGuard can provide/further develop certified bound? An ablation with different level of bugdet discussion might be helpful.\n\n(4) Minor. It is better to not use the abbreviation PGD in line 34 (first use in the paper)\n\nOverall, the idea is straightforward to follow and its underlying idea remains in line with established methodologies although achieves good empirical results. If the authors can resolve my above concerns, I am willing to raise my score.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces StyleGuard, a novel method to protect artists' work from style mimicry attacks by text-to-image diffusion models like Stable Diffusion. The core problem it addresses is that existing protection methods (e.g., Glaze, Anti-DreamBooth) are vulnerable to new \"purification\" attacks (e.g., DiffPure, Noise Upscaling), which can remove the protective noise. Furthermore, existing defenses often lack transferability, failing to protect against unknown or black-box models.\n\nStyleGuard's main contributions are two novel loss functions used to generate a robust, imperceptible perturbation on images:\n1.  A \"Style Loss\" that manipulates the mean and variance of features in the latent space. This perturbs the global style characteristics of an image, making it difficult for models to learn the correct style and enhancing the protection's transferability across different text-to-image models.\n2.  An \"Upscale Loss\" designed to specifically counter purification attacks. It trains the perturbation against an ensemble of different purifiers and upscalers, aiming to maximize the denoising error and essentially making the protective noise \"purification-resistant.\"\n\nThe authors validate StyleGuard through extensive experiments on the WikiArt and CelebA datasets. The results show that their method significantly outperforms existing baselines in robustness against various data transformations and, crucially, against state-of-the-art purification methods.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper addresses a critical and timely problem with a novel solution. However, while the strengths in originality and significance are clear, they are counterbalanced by notable weaknesses in methodological rigor and the breadth of experimental validation, placing the paper on the borderline for acceptance.\n\nStrengths:\n\n1. The problem of efficiently protecting artists from AI-driven style mimicry is of great practical importance. The proposed StyleGuard framework, particularly its \"Upscale Loss\" designed to combat purification attacks, is a novel and conceptually clever contribution to this domain.\n\n2. The experimental evaluation is mostly thorough. The authors compare StyleGuard against a comprehensive set of baselines and test against a strong and relevant suite of attacks, including the specific purification methods that this work aims to defeat.\n\n3. The paper is generally well-written and easy to follow, with figures that help to illustrate the overall system architecture.\n\nWeaknesses\n\n1.  The core `Style Loss` requires a target image from a different art genre, but the selection process is vaguely described as \"randomly selected.\" The choice of target style is critical, yet the paper does not explore how the degree of stylistic difference impacts protection efficacy, nor does it propose a systematic selection strategy. This ambiguity limits the method's reproducibility and reliability.\n\n2. The algorithm relies on fixed hyperparameters, such as the ensemble of purifiers for the `Upscale Loss` and the style loss weight (λ=10). The paper lacks ablation studies to analyze the sensitivity to these parameters or to justify why these specific values were chosen. We cannot know if these choices are optimal or how robust the method is to their variation.\n\n3.  The paper's evaluation of robustness is narrowly focused, primarily validating the defense against sophisticated, diffusion-based purifiers. However, it neglects to test against a wider range of transformations common in real-world scenarios. For instance, the paper does not evaluate robustness against a \"stress test\" of combined transformations (e.g., cropping, followed by rotation and compression). This limited scope makes it difficult to fully assess the protection's practical robustness.\n\nQUESTIONS:\nThank you for this interesting work. The proposed ideas are novel, but I have concerns about the methodological rigor and the breadth of the evaluation, which prevent me from giving a higher score at this time. My questions are as follows, ordered by importance:\n\n1. The `Style Loss` depends on a target image from a different genre, a critical step described as being \"randomly selected.\" How sensitive is the protection efficacy to the choice of this target? For instance, does a target style that is maximally dissimilar provide stronger protection, and is there a systematic way to choose an optimal target?\n\n2. The methodology relies on several key hyperparameters, such as the `Style Loss` weight (λ=10) and the specific ensemble of purifiers for the `Upscale Loss`. Could you provide ablation studies that analyze the sensitivity to these parameters and offer a justification for the chosen values?\n\n3. The evaluation focuses on individual purification methods. To better assess real-world robustness, have you considered testing against a pipeline of combined, simpler transformations (e.g., cropping, rotation)?\n\nMy final recommendation is contingent on the authors' response to these concerns. A more rigorous analysis of the method's sensitivity to its core parameters (Questions 1 & 2) and a more comprehensive robustness evaluation (Question 3) would be necessary to move this paper into a clear accept category.\n\nLIMITATIONS:\nThe authors provide a thorough analysis of the potential limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}