CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Authors: Sen Peng
Mingyue Wang
Jianfei He
Jijia Yang
Xiaohua Jia
Venue: ICML 2025 poster (2025)
Keywords: Latent Diffusion Models
Protective Perturbations
Adversarial Training
Primary area: deep_learning->robustness
TL;DR: We reveal the role of latent representation distortion in protective perturbations and propose Contrastive Adversarial Training, an adaptive attack that exposes their robustness weaknesses.

Abstract: Latent diffusion models have recently demonstrated superior capabilities in many downstream image synthesis tasks. 
However, customization of latent diffusion models using unauthorized data can severely compromise the privacy and intellectual property rights of data owners.
Adversarial examples as protective perturbations have been developed to defend against unauthorized data usage by introducing imperceptible noise to customization samples, preventing diffusion models from effectively learning them.
In this paper, we first reveal that the primary reason adversarial examples are effective as protective perturbations in latent diffusion models is the distortion of their latent representations, as demonstrated through qualitative and quantitative experiments.
We then propose the Contrastive Adversarial Training (CAT) utilizing lightweight adapters as an adaptive attack against these protection methods, highlighting their lack of robustness. 
Extensive experiments demonstrate that our CAT method significantly reduces the effectiveness of protective perturbations in customization, urging the community to reconsider and improve the robustness of existing protective perturbations. 
The code is available at \url{https://github.com/senp98/CAT}.

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Sen Peng' Mingyue Wang'
Jianfei He
Jijia Yang'
Xiaohua Jia
Abstract
et al., 2023), image editing (Kawar al., 2023), and image inpainting (Lugmayr al., 2022). By utilizing the open- source LDM such as Stable Diffusion (Stability.ai, 2024) as the pre-trained model, users can customize image genera- tion models on their datasets at a significantly lower cost than training from scratch. Despite their prominence and popularity, the customization of LDMs also raises signif- icant concerns regarding privacy and intellectual property rights (Dixit, 2023; LLP, 2023). It can be misused to gener- ate fake human face images of a specific identity (Li et al., 2022) or mimic the style of a specific artist by utilizing unau- thorized data. Undoubtedly, such malicious customization severely compromises the rights of data owners and raises ethical concerns across the community.
Latent diffusion models have recently demon- strated superior capabilities in many downstream image synthesis tasks. However, customization of latent diffusion models using unauthorized data can severely compromise the privacy and intellec- tual property rights of data owners. Adversarial examples as protective perturbations have been de- veloped to defend against unauthorized data usage by introducing imperceptible noise to customiza- tion samples, preventing diffusion models from effectively learning them. In this paper, we first re- veal that the primary reason adversarial examples are effective as protective perturbations in latent diffusion models is the distortion of their latent representations, as demonstrated through qualita- tive and quantitative experiments. We then pro- pose the Contrastive Adversarial Training (CAT) utilizing lightweight adapters an adaptive attack against these protection methods, highlighting their lack of robustness. Extensive experiments demonstrate that our CAT method significantly reduces the effectiveness of protective perturba- tions in customization, urging the community to reconsider and improve the robustness of existing protective perturbations. The code is available https: github.......... coom/eenpa CAT.
Several protective perturbation methods (Salman al., 2023; Liang et al., 2023; Shan et al., 2023; Le al., 2023; Liang & Wu, 2023; Xue et al., 2024; Liu et al., 2024) have been proposed to address these concerns. These methods introduce imperceptible perturbations to image samples in the customization dataset, creating adversarial examples that effectively degrade the performance of downstream cus- tomization tasks. Although significant progress has been made in developing perturbation methods, the robustness of these protection techniques has received limited attention. Recent studies (Zhao et al., 2024; Cao et al., 2024; Hönig et al., 2025) have investigated the resilience of protective perturbations against purification-brn based adaptive attacks. By purifying the protected images before customization, these methods effectively compromise the robustness of exist- ing protective perturbation. However, existing purification- based adaptive attacks depend on optimization, resulting in high computational costs. Without an efficient mechanism to detect the protected perturbed images, purification must be applied to every sample in the customization dataset, increasing resource demand. Furthermore, purification with- out prior knowledge of the original data distribution intro- duces uncertainty, potentially altering the learned distribu- tion. In extreme cases, optimization-bar entirely replace the original image content if the substituted one achieves a lower loss for the optimization objective.
1. Introduction
Diffusion-based image generation models, especially the Latent Diffusion Models (LDMss) (Rombach et al., 2022), have exhibited strong capabilities across various down- stream image generation tasks, including text-to-image syn- thesis (Dhariwal & Nichol, 2021; Rombach et al., 2022), image-to-image translation (Saharia et al., 2022; Zhang
purification may
'Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong SAR. Correspondence to: Sen Peng <senpeng2 c@ mit.c.....u...k.
In this paper, we introduce Contrastive Adversarial Training (CAT) from a novel perspective, focusing on model adap- tation rather than purification adaptive attack. We
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
first reveal that the primary reason adversarial examples are effective protective perturbations in LDMM customization is that their latent representations are distorted. Building this insight, we develop the CAT method targeting the latent autoencoder of the L.M,, utilizing the contrastive adver- sarial loss to minimize the reconstruction loss of protected image samples. During adversarial training, we introduce LoRA adapters (Hu et al., 2022) to specific layers of the latent autoencoder, ensuring that only the adapter weights are modified. This approach expands the parameter space to mitigate the effectiveness of adversarial examples and re- duces computational costs, enhancing its adaptability across different scenarios. Extensive experiments demonstrate that applying CAT decreases the effectiveness of protective per- turbations, urging the community to reconsider their robust- ness. In summary, our main contributions are as follows:
Pixel Space
Latent Space
Protected
Encoder
Za
X
*
Za
O +
min catt
Denoising Module
A
$ (cat)
feal)
60
Recon
CAT & Customization
w/ CAT
Image
Generation
Aw
# (c0s)
Decoder
Wi
#
1. We identify that the primary reason adversarial examples effectively serve as protective perturbations in LDM cus- tomization is the distortion of their latent representations.
w/o CAT
2. We propose contrastive adversarial training with adapters as an adaptive attack to evaluate the robustness of exist- ing protective perturbation methods from a novel per- spective of model adaptation rather than purification.
Figure I. Overview of the proposed CAT method. The CAT adapters (Aé and Aw) are integrated into the latent encoder and decoder to realign the latent representations of protected samples. Instead of directly using 2a for the denoising module in the dif- fusion process, the protected image Ta is first encoded into the realigned latent representation =a fear) through the adapted encoder, then decoded into (((0)) using the adapted decoder. This process ensures that the generated image z((*) correctly decoded into the pixel space, effectively neutralizing the protective perturbations.
We conduct extensive experiments demonstrating that applying CAT can significantly reduce the effectiveness of existing perturbation methods, urging the community to reconsider their robustness.
2. Related Works
while the latter aims to generate images that reflect the style specified in the text prompt. DreamBooth (Ruiz et al., 2023) customizes text-to-image diffusion models by fine-tuning both the U-Net and text encoder using a few target subject images. introduces a class-specific prior preservation loss to prevent overfitting the subject, enabling the model to generate diverse class images while preserving key visual features of the subject. LoRA (Hu et al., 2022), originally developed to enable parameter-efficient fine-tuning large language models, has been adapted to diffusion models to enable customization via low-rank adaptation. By intro- ducing lightweight trainable adaptation layers into a frozen pre-trained model and updating only these layers, LoRA sig- nificantly reduces the memory and computational overhead during model customization.
2.1. Diffusion-based Text-to-Image Models
Diffusion-based text-to-image models have demonstrated remarkable capabilities across various downstream tasks. They learn to generate images by simulating forward dif- fusion process that adds noise, and a reverse diffusion pro- cess that removes it, typically using a U-Net (Ronneberger et al., 2015) as the backbone (Ho et al., 2020). Latent diffusion models (Rombach et al., 2022) improve training and inference efficiency while preserving generation quality by encoding samples into their latent representations via an autoencoder. Stable Diffusion (Stability.ai,, 2024) was later released as open-source LDM, which significantly boosted the development of various downstream applica- tions in diffusion-based text-to-image generation.
2.2. Customization of LDMs
2.3. Protective Perturbations
LDMs can be customized for range of downstream tasks, which broadly fall into text-driven image synthesis and edit- ing. Text-driven image synthesis is typically categorized into object-driven image synthesis and style mimicry (Peng et al., 2024). The former focuses on generating images containing specific objects described by the text prompt,
Protective perturbations build upon the idea of adversar- ial attacks (Goodfellow et al., 2015) in classification mod- els, which introduce imperceptible input perturbations to mislead models into incorrect predictions. Recent stud- ies (Salman et al., 2023: Liang et al., 2023; Shan et al., 2023; Le al., 2023; Liang & Wu, 2023; Xue al., 2024)
2

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
have shown that diffusion-based generative models are also vulnerable such attacks. In this context, adversarial ex- amples serve as protective perturbations that significantly degrade the performance of downstream tasks and prevent unauthorized customization. Photoguard (Salman et al., 2023) first introduces protective perturbations to increase the cost of diffusion-based image editing methods such as SDEdit (Meng et al., 2022). The perturbation generated by jointly maximizing the latent reconstruction loss and the diffusion denoising loss with respect to the protected images. AdvDM (Liang et al., 2023) and SDS (Xue et al., 2024) generate perturbations that target the entire LDM through end-to-end optimization to prevent unauthorized model customization. contrast, Glaze (Shan et al., 2023) focuses on attacking the latent encoder of LDMs to dis- tort the encoded representations. Mist (Liang & Wu, 2023)
protective perturbation to each image before release and then publicly shares the perturbed dataset. In this setting, the adversary only has access to the protected data and is unaware of the specific protection method used by the data owner. We want to demonstrate that, even without knowledge of the protection technique and with access only to the protected data, the adversary can still successfully customize the LDM using our proposed CAT.
3.2. Distortion in Latent Representations
Fncade
7, Z
Decolle
8
optimizes the joint latent and denoising losses to gener- ate perturbations, while Anti-DreamBood (Le et al., 2023) specifically targets the DreamBooth (Ruiz et al., 2023) cus- tomization approach. MetaCloak (Liu et al., 2024) leverages a meta-learning framework with transformation sampling to generate transferable and robust perturbations.
Thgy 22
%
Za
Purification-based adaptive attacks have recently been proposed against existing protective perturbations. IM- PRESS (Cao et al., 2024) introduces a purification tech- nique that minimizes the difference between the protected image and its reconstruction from the latent autoencoder. GrIDPure (Zhao et al., 2024) extends DifiPure (Nie et al., 2022) to purify protected samples before customization, while Noisy-Upscaling (Hönig al., 2025) achieves simi- lar goal using other generative models. However, all these methods decrease the effectiveness of protective perturba- tions through purification- based adaptive attacks. To the best of our knowledge, this is the first work to investigate the robustness of such perturbations in the context of LDM cus- tomization from the novel perspective of model adaptation. Recent RobustCLIP (Schlarmann et al., 2024) enhances the robustness of CLiP-based vision encoders through adversar- ial training, particularly against perturbations that disrupt text-image semantic alignment. While effective, it requires fine-tuning the entire encoder on a large set of adversarial examples to achieve general robustness. In contrast, CAT introduces lightweight adapters during customization on protected data. These adapters are detachable at inference time, leaving the original model performance unaffected.
Pixel Space
Latent Space
Figure 2. Distortion latent representations for protected image samples in LDMs. Given clean image 4.. it is encoded into latent representation 2. and decoded back to pixel space as a.. For the protected image Fa, the latent representation &, is distorted, leading to a reconstructed image Ra that deviates significantly from Ta. resulting in a larger reconstruction error.
We first identify key reason why adversarial examples serve as effective protective perturbations in LDMs: their representations are distorted in the latent space. As illus- trated in Figure 2, the latent autoencoder of a LDM first encodes a given clean image Ie from pixel space into a latent representation z.. Subsequently, the denoising mod- ule takes the latent representation as input because it has significantly lower dimensionality while still preserving the semantic features. Additionally, it requires that the autoen- coder can correctly decode the latent representation ze back to pixel space, ensuring that the reconstructed image T. matches the original image Te in the ideal case. However, the mappings between pixel and latent space are distorted for adversarial examples. For an adversarial example za within a perturbation budget 8,, the latent representation e. should remain close to ze within a distance 6₂ and correctly decode back to c. in pixel space in the ideal case. However, is instead mapped to a distorted latent representation E.- This distorted representation is decoded to G.. which exhibits significantly larger reconstruction error.
3. Understanding Protective Perturbations
3.1. Threat Model
We first introduce the threat model considered in this work. We assume the data owner holds clean data samples and intends to share them publicly. To prevent unauthorized customization of the generative model, the owner applies
Similar observations about the latent representation are also reported in previous works (Xue et al., 2024; Cao et al.,
3

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
2024). In contrast, our study conducts more extensive quali- tative and quantitative experiments thoroughly validate the distortion in latent representations of protected samples. To illustrate this, we visualize the latent representations of various protected samples using 1-SNE (der Maaten & Hin- ton, 2008), as shown in Figure 3. We also include the UMAP visualization (McInnes et al., 2018) in Figure 8, which is provided in Appendix A, to enhance the relative distance demonstration. We use four clean images of identity from
20aa)) are clustered with their corresponding clean sample Ze. Since four images are evaluated for each protection method, their latent representations naturally form four dis- tinct groups. Within each group, it can be observed that the latent representation of the protected sample exhibits a larger distance from the clean sample's 's latent representation &e compared to the noisy sample 2, under the same per- turbation budget. This pattern holds across all four image samples for every protective perturbation method, indicat- ing that the latent representations of adversarial examples as protected samples are distorted across all evaluated pro- tection methods. This observation is also supported by the UMAP visualization.
+)vDM(+)
AdvDM(-1
Mist
q
6
6'6
(a) CelebA-HQ Dataset Mist
(b) VGGFace2 Dataset
Mist
SDS(+)
SDS(-)
SUST
SDS
SDSC-)
DM-
DM
SDSL)
3.0
SDS()
3.0%
2.0
1.0
0.0
Ad+DM((+)
0.0
Ad+DM(+
SDST
SDŠT
McsaCloak
MctaClok
Glaze
Aati-DreamBoot
MetaCloak
Glize
Glaze
AnG-DB
Ami-DB
dld.-dJl:
Figure 4. Comparison of latent representation distances I|da de and [|d_=)) - d for various protective perturbation methods across (a) CelebA-HQ and (b) VGGFace2 datasets. The mean distance values over four images for each method are presented. highlighting the CAT adapter's effectiveness in reducing latent representation distortion.
on
6'6'
Figure 3. Visualization of latent representations using t-SNE of clean sample Ze. noisy sample z,, protected sample zn, and the latent representation of protected sample [cat) encoded by the latent encoder with CAT adapter. Protected samples are generated using nine protective perturbation methods, with four clean images from the VGGFace2 dataset as the base. The perturbation budget for protected and noisy samples is fixed at & = 16/255. Fach number labels the representations obtained from the same image.
In addition to qualitative experiments, we conduct quanti- tative analysis of latent representation distances to support our findings. We define the MAE distance between the la- tent representation of a sample and its corresponding clean sample as d. Specifically, the distance between 2, and its corresponding ze is denoted as d, || - zelli. while the distance between (cat) and its corresponding is rep- resented as d(am)) (=() 2el11. The distance between noisy sample 2, and its corresponding se is denoted as d, = 12, -l,,-2ell -ell1- For comparison, we present the relative dif- ferences between d, and d((")) over dy. as shown in Figure 4. We observe that d, consistently larger than d, across all evaluated protection methods, indicating that adversarial examples significantly distort their latent representations compared to noisy samples with the same perturbation bud- get. Furthermore, our proposed CAT adapter effectively reduces this distortion, realigning the latent representations of protected examples toward their original positions.
the VGGFace2 (Cao et al., 2018) dataset as an illustration. For each clean sample Ze, we generate its adversarial exam- ple xa as the protected image using nine different protective perturbation methods, within the perturbation budget fixed at 6, 16/255. The protected sample is then encoded into the latent space through the encoder as 2a = E(r,), while the clean sample is encoded as 2. = E(zc). Addi- tionally, we construct a noisy sample F,- = xx + clip(r, by adding Gaussian noise r ~ (0, 62), clipped using the same perturbation budget 8z. This noisy sample is encoded as 2, E.x,). Finally, the adversarial sample va is also encoded using the latent encoder with the CAT' adapter (in
(cat) = (xa)) For each protection method shown in the subplots, Za. 2r and
the CAT-both setting), resulting in za
4

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
3.3. Learnability of Distorted Latent Representations
age from an identity in the VGGFace2 dataset and two fine-tuning approaches, DreamBooth and LoRA. As shown in Figure 5, Sa consistently falls within or near the range of se = s, across all protection methods and both fine-tuning approaches, which suggests that the latent representations of both clean and adversarial samples can be effectively learned by the diffusion process. This indicates that the primary reason for the effectiveness of adversarial examples as protective perturbation methods lies in the distortion of their latent representations, rather than the limitation of the diffusion process in learning these distorted representations.
We have identified that a key reason for the effectiveness of adversarial examples as protective perturbations is the distortion of their latent representations. Building on this finding, we seek to explore further question: Can the latent representations of adversarial examples be effectively learned by the diffusion process?
(a) Customization by DroamBooth
Difference Ratio (in %%
range
In summary. this section provides a deeper investigation of why adversarial examples of LDMs are effective as pro- tective perturbations against unauthorized data usage. We identify that the primary reason lies in the distortion of latent representations for these adversarial examples. Furthermore, we verify that these distorted latent representations can still be effectively learned by the diffusion process, confirming that the distortion itself the key factor behind their effec- tiveness. We emphasize that this conclusion is mainly based on empirical experimental observations, and other potential contributing factors may exist.
AdvDM AdvDM(-) Mist SDS(+) SDS(-)
SDST
Glaze
Anti-DB
MetaCluak
(b) Customization by LoRA
Difference Ratio (in %))
5, range
CAT
AdvDM(+) AdvDM(-) Mist
Mist
SDS(+)
SDS(-)
SDST
Glaze
Auti-DB
MetaCloak
4. CAT: Contrastive Adversarial Training
4.1. Contrastive Adversarial Loss
Figure 5. Difference ratios for nine protection methods, evaluated using an image from the VGGFace2 dataset. Results are shown for two fine-tuning approaches: (a) by DreamBooth and (b) by LoRA. The s, range (&c - ||+c - s-1| & + || syIll is shaded, demon- strating that consistently falls within or near this range across all methods, indicating that both clean samples and adversarial examples can be effectively learned by the diffusion process.
We have observed that adversarial examples are effective as protective perturbations because their latent representa- tions are distorted. This indicates that the latent autoencoder (Es, Du)) of the LDM is attacked. To mitigate this, we ap- ply contrastive adversarial training the latent autoencoder to realign the latent representations of adversarial examples. We first introduce the contrastive adversarial loss used for our adversarial training strategy as follows. Given the pro- tected image va and the latent autoencoder (E,, Dw we define the loss function as
To explore this, we design the following experiment: diffu- sion models are known overfit when trained or fine-tuned on very small number of image (Carlini et al., 2023). In such cases, the overfitted model exhibits memorization, generating samples that are nearly identical to the training image when given the same text prompt used during fine- tuning. We first fine-tune the pre-trained LDM on a single image obtained from one specific identity in the dataset. Once the model overfits, we generate an image from the fine-tuned model using the exact same text prompt. For the latent representation &e of clean image, we expect the gen- erated & to be close to 2e in the latent space. The difference ratio for this clean image is defined as 8, = || - /cll/zb. where 2, = max(2.) - min(zo serves as a normalization term. Similarly, the difference ratios for 8r, Sa and (cai) defined as ||=, l 2t, || /// 25 and ||=0"" - / =b. respectively. For simplicity, we also define the s, range as [s. - s. - sy11, se + ||s. - I1. We present the evalua- tion results in Figure 5. The difference ratios are evaluated for nine protection methods based on the sample clean im-
Leal EE, Du, = -ga-p((z)) (P - xall (1)
The protected image is first encoded into the latent space by the encoder & and then decoded back into the pixel space by D Since the decoded image should ideally match the original protected image, we define the contrastive loss as the MSE distance between them over all given Za. This loss encourages the autoencoder to correctly reconstruct xa. restoring its latent representation.
4.2. CAT Adapters
are
Fine-tuning all layers of the latent autoencoder may affect the model's performance in encoding and decoding unpro- tected image samples. Moreover, we aim to introduce addi- tional parameters to expand the parameter space, improving the model's 's resilience against adversarial examples after the
5

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
contrastive adversarial training. Therefore, we incorporate adapters in our proposed contrastive adversarial training. Specifically, we apply the LoRA (Hu et al., 2022) adapter to all convolutional and attention layers within the latent autoencoder. For the model weights Φo E &Rdd of the la- tent encoder, the latent representation (can) of protected sample Ta is given by
CAT across nine protective perturbation settings: (1) Ad- vDM(+) (Liang et al., 2023), (2) AdvDM(() (Xue al., 2024), (3) Mist (Liang & Wu, 2023), (4) SDS(+) (Xue et al., 2024), (5) SDS(-) (Xue et al., 2024), (6) SDST (Xue et al., 2024), (7) Glaze (Shan et al., 2023), (8) Anti- DreamBooth (Le et al., 2023), and (9) Metacloak (Liu et al., 2024). The (+) sign denotes the use of gradient ascent to construct the perturbation. whereas the (-) sign denotes the use of gradient descent, following the setting in (Xue et al., 2024). We set the perturbation budget for all evalu- ated protection methods to ā = 16/ 16/255. Since Glaze does not release its codebase and its application does not allow adjustment of the specific perturbation budget, we use its maximum perturbation strength.
z/ (cat)) = D + Adt, = @0TA + B
where B Rdxr and A B are the weights of the CAT adapter attached to the latent encoder.
Similarly, for the model weights wo E R of the latent
decoder, the reconstruction _!=a)) in pixel space of the latent representation za (car) is given by
Datasets We evaluate the effectiveness of CAT in two main diffusion- image synthesis applications: (1) object- driven image synthesis and (2) style mimicry. For object- driven image synthesis, we focus on human face genera- tion using the CelebA-HQ (Karras et al., 2018) and VG- GFace2 (Cao et al., 2018) datasets. For each dataset, we select five identities, with 12 images per identity, equally split into three subsets: a clean reference subset, target pro- tection subset, and an additional clean reference subset. For style mimicry, we use subset of Claude Monet's 's artworks from the WikiArt dataset (Mancini et al., 2018), consisting of 12 images, divided into three subsets following the same configuration. All images are center-cropped and resized to 512 x 512.
s.(ca) = wo = (catt)
+ Aw 2(cat)
= wo (cai)
+ B A 2((0)))
Fa
where B" E Rexr and A E Rxxd are the weights of the CAT adapter attached to the latent decoder. The overall design of our proposed CAT method is illustrated in Figure 1. For a protected image sample Ta, we do not directly use its encoded latent representation &a for the denoising module to learn in the diffusion process. Instead, we first apply CAT by encoding Fa using the latent encoder attached with the CAT adapter Ad, obtaining the realigned latent representation za (cm)) This realigned representation is then decoded back using the latent decoder, which is also attached with the CAT adapter Aw, resulting in the reconstructed image (cat) in the pixel space. Our proposed CAT method is optimized by solving the following objective:
CAT Settings For object-driven image synthesis, we evalu- ate three CAT settings: (1) CAT-both, (2) CAT-en, and (3) CAT-de. In CAT-both, CAT adapters are added to the target layers in both the encoder and decoder of the latent autoen- coder. CAT-en applies CAT adapters only to the encoder, while CAT-de applies them solely to the decoder. Since we have identified that the primary reason adversarial examples are effective as protective perturbations is their latent rep- resentation distortion, the CAT-de method should have no impact on compromising the protection. We conduct this evaluation to verify our hypothesis. For style mimicry, we only evaluate (1) CAT-both, and (2) CAT-en settings.
Adb Aw arg min (E Dem st),
(4)
where the weights for model Egt are du + Ae and the weights for model Dem are wa + Aw.
Once CAT is applied and the CAT adapters are obtained, customization on protected image samples is performed us- ing the latent autoencoder augmented with the CAT adapters. For image generation, since the denoising module is fine- tuned on the realigned latent representation 2((ca), the gener- ated output in the latent space for the given text prompt is also realigned. Consequently, it is decoded into the image af(aa) in the pixel space instead of T.. effectively neutraliz- ing the protective perturbations.
We select all convolutional and attention layers within the module as the target layers. The adapter rank set to 128 for the CAT-both setting and r 256 for the CAT-en and CAT-de settings to maintain the same number of added parameters. During CAT, only the parameters of the attached adapters are updated, while all other weights remain frozen. Training is conducted with batch size of 4 and a learning rate of I x100 for 1000 steps.
5. Experiments
5.1. Experimental Setup
Customization Settings For the customization settings, we adopt the widely used DreamBooth (Ruiz et al., 2023) and LoRA (Hu et al., 2022) methods, using the latest Stable Diffussoo v2.1 as the pre-trained LDM. Details of the cus-
Evaluated Protective Perturbation Methods To evaluate the effectiveness of our proposed CAT method in compro- mising the robustness of protective perturbations, we test
6

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
AdvDM(+)
AdvDM(-
Mist
SDS(-)
SDS()
SDST
Glaze
Anti-DB
MetaCloak
Baseline
CAT-boll
CAT-en
CAT-de
Figure 6. Qualitative results for object-driven image synthesis customization in identity from the VGGFace2 dataset using DreamBooth and CAT settings with the text prompt "a dslr portrait of sks person". Each row represents a different setting: Baseline, Cot- CAT-en, and CAT-de, while each column corresponds different protective perturbation setting. The results illustrate the effectiveness of different CAT settings on generating human faces from protected images.
Table 1. Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth. The table presents performance metrics FSS and FQS for the CelebA-HQ and VGGFace2 datasets across different protective perturbation settings. The optimal values among different CAT settings for each metric are bolded, where higher FSS indicates improved human face similarity, and higher FQS reflects better face image quality.
DATASET
CELEBA-HO HQQ
VGGFACE2
FSS
FSS
FQS
METHOD
BASELINE CAT-ROTH CAT-EN CAT-DE BASRLINE CAT CAT-EN CAT-DE CAT--D|BASELING (BASELINE CAT-ROTH CAT-EN CAT-DE BASRLINE CAT BOTH CAT-EN CAT-DE
ADVDML-)
0.354
0.623
0.571
0.401
0.275
0.549
0.611
0.401
0.436
0.564
0.547
0.454
0.281
0.617
0.662
0.492
MIST
0.160
0.572
0.501
0.152
0.286
0.597
0.580
0.375
0.035
0.557
0.521
0.101
0.384
0.656
0.712
0.279
SDSC
0.499
SDS()
0,411
0.678
0.599
0.405
0.352
0.597
0.587
0.449
0.439
0.570
0.569
0.435
0.368
0.730
0.689
0.456
SDST
0.175
0,594
0.485
0.157
0.208
0.587
0.588
0.313
0.146
0,559
0.546
0.196
0.248
0.663
0.694
0.346
GLAZE
0,363
0.577
0.398
0.491
0.618
0.676
0.587
0.387
0.607
0.576
0.472
0.576
0.724
0.740
0.595
ANTI-DB
0.471
0.662
0.597
0.489
0.434
0.608
0.664
0.546
0.456
0.584
0.546
0.440
0.416
0.673
0.704
0.401
METACLOAR
0.497
0.642
0.578
0.507
0.372
0.475
0.403
0.421
0.560
0.631
0.487
0.355
0.508
0.625
0.412
tomization settings are provided in Appendix B.
2023). quantify the expected identity similarity in generated image set, we define Face Similarity Score (FSS) as the product of Retina-FDR and ISM. Moreover, we em- ploy the TOPIQ method (Chen et al., 2024), trained on the face IQA dataset and implemented in (Chen & Mo, 2022), to measure Face Detection Rate (TOPIQ-FDR) and Face Image Quality (FIQ). To quantify the expected quality of the generated face images, we define their product as the Face Quality Score (FQS). We use two text prompts for sampling, "a photo of sks person" and "a dsIr portrait of sks person". For each test prompt, generate 30 images and compute the average metric scores across all tested identities
Evaluation Metries For Quantitative Experiments The object-driven image synthesis task aims generate human faces that preserve the identity features from the given pro- tected images. Therefore, for qualitative experiments, we use Face Detection Rate (Retina-FDR), measured with the RetinaFace detector (Deng et al., 2020), to evaluate whether face is detected in the generated images. If face is detected, we extract its embedding using the ArcFace recog- nizer (Deng et al., 2019) and compute the cosine distance between the extracted embedding and those from the clean dataset, denoted as Identity Score Matching (ISM) (Le et al.,

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Table Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth, compared to Noisy- Upscaling (NU) and Gaussian Filtering (GF) methods. The table presents performance metrics FSS and FQS for the CelebA-HQ and VGGFace2 datasets across different protective perturbation settings. The optimal values among different CAT settings for each metric are bolded.
DATASET
CELERA-HO HQ
VGGFACE2
METRIC
FSS
FQS 1
FSS
FQS
METHOD
CAT-BOTH
CAT-EN
NU
GF
CAT-BOTH
CAT-EN
NU
GF
[CAT-BOTH
CAT-EN
NU
GF
CAT-BOTH
CAT-EN
NU
GF
ADVDM(+)
0.643
0.529
0.531
0.492
0.431
0.448
0.481
0.352
0,534
0.560
0.518
0.506
0.481
0.578
0.506
U.363
ADVDM(-)
0.623
0.469
0.607
0.549
0.611
0.526
0.564
0.547
0.529
0.563
0.635
0.563
0.506
MIST
0.572
0.501
0.491
0.488
0.597
0.580
0.475
0.493
0.557
0.521
0.566
0.518
0.662
0.701
0.518
0.437
SDS(+)
0.602
0.499
0.599
0.409
0.413
0.503
0.302
0.486
0.508
0.498
0.402
0.438
0.402
0.281
SDS(-)
0.678
0.599
0.468
0,583
0.597
0.587
0.494
0.493
0,570
0.569
0.509
0,593
0.700
0.671
0.593
0.558
SDST
0.594
0.485
0.470
0.446
0.474
0.464
0.546
0.521
0.538
0.627
0.538
0.482
GLAZE
0.610
0.577
0.533
0.547
0.618
0.676
0.496
0.533
0,607
0.576
0.503
0.549
0.733
0.723
0.549
0.562
ANTI-DD
0.662
0.597
0.540
0.575
0.608
0.664
0.469
0.543
0.546
0.566
0.548
0.636
0.656
0.548
0.499
METACLOAK
0.642
0.578
0.521
0.540
0.460
0.475
0.395
0.324
0.560
0.631
0.566
0.542
0.504
0.633
0.542
0.349
to obtain the final results. We also present evaluation results using FID (Heusel et al., 2017) and CLIP-IQA (Wang et al., 2023), which are two widely used metrics for assessing image quality.
tomization using DreamBooth are also presented in Table 6, which is provided in Appendix C.2. These results further support the effectiveness of the CAT settings in compromis- ing the existing protective perturbation.
5.2. Object-driven Synthesis Results
Table 3. Quantitative results for style mimicry using CAT meth- ods customized in DreamBooth on the WikiArt dataset. The ta- ble reports CLIP-IQA scores, where higher values indicate better alignment with the target artistic style. The optimal values among different CAT settings for each metric are bolded.
The evaluation results for applying CAT methods in the object-driven image synthesis task are presented in Table 1. We observe that both the CAT-both and CAT-en settings sig- nificantly improve task performance compared to the base- line, which directly eustomizes using the protected image samples across all evaluated protections. This demonstrates the effectiveness of these two settings in compromising the protective perturbations. Furthermore, the CAT-both setting achieves a higher FSS score than CAT-en, while CAT-en obtains a better FQS score overall. Additionally, the CAT-de setting has little to no effect on improving task performance. The potential reason behind this that CAT-de only adds adapters to the decoder, which has a limited impact on realigning latent representations. Fine-tuning only the de- coder is more challenging, the diffusion model Iearns from distorted latents that are highly diverse, making ac- curate reconstruction difficult. The weaker performance of CAT-de compared to CAT-en and CAT-both further sup- ports our observation that latent distortion the key factor behind the effectiveness of adversarial noise as protec- tive perturbation. Qualitative results for one identity from the VGGFace2 dataset, as shown in Figure 6 (and for one identity from the CelebA-HQ dataset as shown in Figure 9 which is provided in Appendix C.3), further confirm the observations from the quantitative results.
CLIP-IQA ^ BASELINE
CAT-BOTH
CAT-EN
ADVDM(+)
0.343
0.390
0.621
ADVDM(-)
0.463
0.536
0.697
MIST
0.345
0.465
0.694
SDS(+)
0.285
0.366
0.366
SDS(-)
0.501
0.481
0.723
SDST
0.406
0,485
0.712
GLAZE
0.532
0.614
0.730
ANTI-DB
0.315
0.544
0.672
5.3. Comparison with Purification-Base Attacks
We compare our proposed CAT with two purification- based adaptive attacks against protective perturbations: Noisy-Upscaling (Hönig et al., 2025), which performs optimization-bassed purification, and traditional Gaussian filtering, which applies low-pass filter to the perturbed images. Evaluations are conducted on both the CelebA- and VGGFace2 datasets using CAT-both and CAT-en under the same experimental setting. For Noisy-Upscaling we adopt the default configurations provided in the original paper. While for Gaussian filtering, we apply a Gaussian kernel of size 5 with = 1.0. The quantitative results are presented in Table 2. We observe that CAT consistently achieves comparable or superior performance to both Noisy- Upscaling and Gaussian Filtering across all protective per- turbations, in terms of both FQS and FSS. These results highlight the competitive effectiveness of CAT compared to existing purification- based methods.
Moreover, we conduct object-driven image synthesis cus- tomization experiments using LoRA as shown in Table 5, which is provided in Appendix C. The results verify the effectiveness of CAT in the LoRA customization setting across all nine evaluated protective perturbations and two datasets. The quantitative results measured by FID for cus-
8

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
AdvDM(+)
-d
Mist
SDS(+)
SDS(-)
SDST
Glaze
Anti-DB
Baseline
CAT-both
CAT-en
Figure 7. Qualitative results for style mimicry customization in DreamBooth using CAT settings with the text prompt "a painting of shoe with a plant growing inside by claude monet artist". Each represents different setting: Baseline, CAT-both and CAT-en, while each column corresponds to a different protective perturbation setting. The results illustrate the effectiveness of different CAT settings on generating artworks of the same style from protected images.
5.4. Style Mimicry Results
settings on the VGGFace2 dataset under the AdvDM(+) protective perturbation and CAT-both setting. The results shown in Table indicate that increasing the rank generally leads to better performance, with r 128 achieving the highest FSS (0.544) and FQS (0.460). This demonstrates that higher-rank adapters enhance robustness against pro- tective perturbations. However, higher-rank CAT adapters result in larger model size. For instance, the CAT-both set- ting with r = 128 yields a 104 MB adapter model, and this size continues to increase with higher ranks. This trade-off' suggests that while higher-rank adapters enhance robust- ness against protective perturbations, practical deployment requires balancing performance gains and model efficiency.
We evaluate the CAT-both and CAT-en settings for style mimicry customization using DrcamBooth. Quantitative re- sults, measured by CLIP-IQA, are presented Table 3. CAT consistently outperforms the baseline across all evaluated protection methods, achieving higher CLIP-IQA scores and confirming the effectiveness of our approach. Qualitative comparisons are shown in Figure 7, illustrating that CAT significantly improves task performance in style mimicry customization. Across all evaluated protection perturba- tions, applying CAT preserves the stylistic characteristics of the target artist while improving image fidelity and mitigat- ing the distortions introduced by protections. These results demonstrate that CAT weakens the robustness of existing protection perturbations in style mimiery, highlighting the need for more advanced protection methods.
6. Conclusions
In this work, we introduce Contrastive Adversarial Training (CAT) as a novel approach to evaluating the robustness of LDM customization against protective perturbations. We identify that the primary factor contributing to the effec- tiveness of adversarial examples as protective perturbations lies in the distortion of their latent representations. We then propose CAT, which employs lightweight adapters trained using the contrastive adversarial loss to realign the latent representations. Extensive experiments demonstrate that CAT significantly reduces the effectiveness of range of ex- isting protective perturbation methods in both object-driven synthesis and style mimicry customization tasks. We believe this work can encourage the community to reconsider the robustness of existing protective perturbations and inspire the development of more resilient protections in the future.
Table 4, Effectiveness of the CAT-both method against AdvDM(+) protective perturbation on the VGGFace2 dataset under various CAT adapter rank settings for human face generation using Dream- Booth. The optimal values for each metric are bolded.
RANK
BASCLINE
7
16
32
64
128
FSS
0.350
0.525
0.500
0.513
0.489
0.540
0.544
FQS 1
0.260
0.372
0.390
0.458
0.414
0.381
0.460
5.5. Ablation Study: Impact of CAT Adapter Rank
To investigate the impact of the CAT adapter rank on the effectiveness, we conduct experiments using varying rank

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Acknowledgements
Chen, C., Mo, J., Hou, J., Wu, H., Liao, L., Sun, W., Yan, Q., and Lin, W. ToplQ: A Top-Down Approach from Semantics to Distortions for Image Quality Assessment. IEEE Transactions on Image Processing, 33:2404-244 2418, 2024.
This work was supported by Hong Kong RGC Research Im- pact Fund with R102-21 and General Research Fund with CityU No.J 1422. We sincerely thank Linshan Hou from Harbin Institute of Technology, Shenzhen for the valu- able comments and insightful suggestions.
Deng. J., Guo, J., Xue, N., and Zafeiriou, S. ArcFace: Addi- tive Angular Margin Loss for Deep Face Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), pp. 4690- 4699, Long Beach, CA, USA, 2019. Computer Vision Foundation/ / IEEE.
Impact Statement
This work evaluates the robustness limitations of existing protective perturbation methods against unauthorized data usage in diffusion-based generative models. Our goal is to highlight key vulnerabilities and provide insights toward the design of more resilient protection mechanisms. In particular, our findings suggest that effective protection may benefit from incorporating defense strategies that explicitly target the diffusion process, especially within end-to-end optimization frameworks. In contrast, relying solely on latent autoencoders may be less robust, as they more easily compromised by adaptive attacks. This perspective points to possible direction for developing more robust protective perturbations that better safeguard the intellectual property rights of data owners. All visual content used in this study was either publicly licensed or provided with explicit permission.
Deng. J., Guo, J., Ververas, E., Kotsia, I,, and Zafeiriou, S. RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition (CVPR 2020). pp. 5202-5211, Seattle, WA, USA, 2020. Com- puter Vision Foundation IEEE.
der Maaten, L. V. and Hinton, G. Visualizing Data Using t-SNE. Journal of Machine Learning Research, 9(11), 2008.
Dhariwal, P. and Nichol, A. Q. Diffusion Models Beat GANs on Image Synthesis. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems (NeurIPS 2021), pp. 8780-8794, Virtual, 2021. Curran Associates, Inc.
References
Cao, B., Li, C., Wang, T., Jia, J., Li, B., and Chen, J. IM- PRESS: Evaluating the Resilience of Imperceptible Per- turbations Against Unauthorized Data Usage in Diffusion- Based Generative AL In Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA, 2024. Curran Associates, Inc.
Dixit, P. Meet The Three Artists Behind A Landmark Lawsuit Against AI Art Gen- erators, 2023.
URL hetps://www buzzfeednews http://w.////a.ce///apeaae.aeeee artists-lawsuie -aattt--aaaaatttte-a-eeeee
Buzzfeed News. Accessed: 2024-07-25.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and Harnessing Adversarial Examples. In Proceedings of the 3rd International Conference on Learning Rep- resentations (ICLR 2015), San Diego, CA, USA, 2015. OpenReview.net.
Cao, Q., Shen, L.., Xie, W., Parkhi, O. M., and Zisserman, A. VGGFace2: A Dataset for Recognising Faces Across Pose and Age. In Proceedings of the 13th IEEE Interna- tional Conference on Automatic Face and Gesture Recog- nition (FG 2018), pp. 67-74, Xi'an, China, 2018. IEEE Computer Society.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs Trained by a Two Time-Scale Up- date Rule Converge to a Local Nash Equilibrium. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS 2017), pp. 6626- 6637, Long Beach, CA, USA, 2017. Curran Associates, Inc.
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramèr, F., Balle, B., Ippolito, D., and Wallace, E. Extracting Training Data from Diffusion Models. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security 2023), pp. 5253-5270, Anaheim, CA, USA, 2023. USENIX Association.
Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Prob- abilistic Models. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS 2020), pp. 6840-68511 Vancouver, Canada, 2020. Curran Associates, Inc.
Chen, C. and Mo, J. IQA-PyTorch: Pytorch toolbox for image quality assessment, 2022. URL https: //github. www./////aaooooo http//aaccfngcccccoocc/oo Poa-PyTorch. Ac- cessed: 2024-07-25.
10

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Hönig, R., Rando, J., Carlini, N., and Tramèr, F. Adversar- ial Perturbations Cannot Reliably Protect Artists From Generative AI. In Proceedings of the 13th International Conference on Learning Representations (ICLR 2025), Singapore, 2025. OpenReview.net.
LLP, J. S. L. F. Class Action Filed Against Stability AI, Midjoumey, and DeviantArt for DMCA Vi- olations, Right of Publicity Violations, Unlawful Competition, Breach of TOS, 2023. URL https:
re. com http://wwwwweeeeeeeee class-actions saaas--acion---aaisstt----- aaaaaaaaainnnaaaatta-aaa-a-----a tools. PR Newswire. Accessed: 2024-07-25.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z.,, Li, Y., Wang. S., Wang, L., and Chen, W. LoRA: Raw-Rank Adaptation of Large Language Models. In Proceedings of the 10th International Conference on Learning Representations (ICLR 2022), Virtual, 2022. OpenReview.net.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Gool, L. V. Repaint: Inpainting Using Denois- ing Diffusion Probabilistic Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR 2022), pp. 11461-11471, New Orleans, LA, USA, 2022. IEEE
Karras, T., Aila, T., Laine, S., and Lehtinen. J. Progres- sive Growing of GANs for Improved Quality, Stability, and Variation. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, Canada, 2018. OpenReview.net.
Mancini, M., Ricci, E., Caputo, B., and Bulò, S. R. Adding New Tasks to a Single Network with Weight Transforma- tions Using Binary Masks. In Proceedings of the Euro- pean Conference on Computer Vision Workshops (ECCV Workshops 2018), pp. 180-189, Munich, Germany, 2018. Springer.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-Based Real Image Editing with Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), pp. 6007-6017, Vancouver, Canada, 2023. IEEE
McInnes, L., Healy, J., and Melville, J. UMAP: Uni- form Manifold Approximation and Projection for Dimen- sion Reduction. https: /at::://axiivva/ org/ abs/ 1802. 03426, 2018
Le, T. V., Phung, H., Nguyen, T. H., Dao, Q. Tran, N. N., and Tran, A. T. Anti- BotitDreamBooth: Protecting Users from Personalized Text-to-lmage Synthesis. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2023). pp. 21 2116-2227, Paris, France, 2023. IEEE.
Meng, C., He, Y., Song. Y., Song, J., Wu, J., Zhu, J., and Ermon, S. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In Proceedings of the 10th International Conference Learning Represen- tations (ICLR 2022), Virtual, 2022. OpenReview.net.
Li, C., Wang. L., Ji, S., Zhang, X., Xi, Z., Guo, S., and Wang, T. Seeing Living? Rethinking the Security of Facial Liveness Verification in the Deepfake Era. In Proceedings of the 31st USENIX Security Symposium (USENIX Security 2022), pp. 2673-2690, Boston, MA, USA, 2022. USENIX Association.
Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. Diffusion Models for Adversarial Pu- rification. In Proceedings of the 39th International Con- ference on Machine Learning (ICML 2022), pp. 16805 16827, Baltimore, MD, USA, 2022. PMLR.
Liang, C. and Wu, X. Mist: Towards Improved Adversar- ial Examples for Diffusion Models. https: arxiv. org/abs// ftbbs/3355. 2305. 12683, 2023.
Peng, S., Yang, J., Wang, M., He, J., and Jia, X. Protec- tive Perturbations Against Unauthorized Data Usage in Diffusion- Image Generation. In Proceedings of the 12th International Conference on Advanced Cloud and Big Data (CBD 2024), pp. 118-123, Brisbane, Australia, 2024. IEEE
Liang, C., Wu, X., Hua, Y., Zhang, J., Xue, Y., Song, T., Xue, Z., Ma, R., and Guan, H. Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), pp. 20763-20786, Honolulu, Hawaii, USA, 2023. PMLR.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. Higg--Reollution Image Synthesis with La- tent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022), pp. 10684-10695, New Orleans, LA, USA, 2022. IEEE.
Liu, Y., Fan, C., Dai, Y., Chen, X., Zhou, P., and Preventing Unauthorized Subject- Driven Tmxae Difffusio- Based Synthesis via Meta-Learning In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR 2024), pp.
Sun, L. MetaCloak:
Ronneberger, O.,, Fischer, P., and Brox, T. U-Net: Convo- lutional Networks for Biomedical Image Segmentation. In Proceedings of the 18th International Conference on
Seattle, WA, USA, 2024. IEEE.

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Medical Image Computing and Computer-Asiisted Inter- vention (MICCAI 2015), pp. 231-241, Munich, Germany, 2015. Springer.
Zhao, Z., Duan, J., Xu, K., Wang. C., Zhang, R., Du, Z., Guo, Q.. and Hu, X. Can Protective Perturbation Safe- guard Personal Data from Being Exploited by Stable Dif- fusion? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024), pp. 24398-24407, Seattle, WA, USA, 2024, IEEE
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. DreamBooth: Fine Tuning Text-to- Image Diffusion Models for Subject-Drive Driven Generation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR 2023), pp- 22500-22510, Vancouver, BC, Canada, 2023. IEEE
Saharia, C., Chan, W., Chang, H., Lee, C. A., Ho, J., Sali- mans, T., Fleet, D. J., and Norouzi, M. Palette: Image- to-Image Diffusion Models. In Proceedings of the ACM SIGGRAPH Conference (SIGGRAPH 2022), pp. 15:1- 15:10, Vancouver, BC, Canada, 2022, ACM.
Salman, H., Khaddaj, A., Leclere, G., Ilyas, A., and Madry, A. Raising the Cost of Malicious Al-Powered Image Edit- ing. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), pp. 29894-29918, Honolulu, Hawaii, USA, 2023. PMLR.
Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. Proceedings of the 4 International Confer- ence on Machine Learning (ICML202) Vienna, Austria, 2024. OeveRevie
Shan, S., Cryan, J., Wenger, E., Zheng, H., Hanocka, R., and Zhao, B. Y. Glaze: Protecting Artists from Style Mimicry by Text-to-lmage Models. In Proceedings of the 32nd USENIX Security Symposium, pp. 2187-2204, Anaheim, CA, USA, 2023. USENIX Association.
Stability.ai. Stability AI Image Models - Stability AI, 2024. URL /sstps:/ssaabillie www.abiiilyy.......aaaaaii.illll ai/stable-inee image. Accessed: 2024-07-25.
Wang, J., Chan, K. C. K., and Loy, C. C. Exploring CLIP for Assessing the Look and Feel of Images. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI 2023), pp. 2555-2563, Washington, DC, USA, 2023. AAAI Press.
Xue, H., Liang, C., Wu, X., and Chen, Toward Effective Protection Against Diffusion-Based Mimicry Through Score Distillation. In Proceedings of the 12th Interna- tional Conference on Learning Representations (ICLR 2024), Vienna, Austria, 2024. OpenReview.net.
Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., and Xu, C. Inversion-Based Style Transfer with Dif- fusion Models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR 2023), pp. 10146-10156, Vancouver, BC, Canada, 2023. IEEE.
12

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
A. Visualization of Latent Representations using UMAP
AdvDM(+)
AdvDM(-
Mist
62
01
SDS(+)
SDS(-)
SDST
Glaze
Anti- DreamBooth
MetaCloak
Figure 8. Visualization of latent representations in UMAP of clean sample 2o, noisy sample &r, protected sample za. and the latent representation of protected sample za (cwc) encoded by the latent encoder with CAT adapter. Protected samples are generated using nine protective perturbation methods, with four clean images from the VGGFace2 dataset the base. The perturbation budget for protected and noisy samples is fixed at 5 = 16/255. Each number lahels the representations obtained from the same image.
B. Details of Customization Settings
In the DreamBooth customization setting, both the U-Net and text encoder are fine-tuned with a batch size of 2 and learning rate of 5 x 10-7 for 2000 training steps. The training instance and prior class prompt for object-driven image synthesis are "a photo of sks person" and "a photo of person", respectively. For style mimicry, the training instance and prior class prompt are "a painting by [ARTIST] artist" and "a painting by artist", where [ARTIST] is replaced with the target artist's ' name. In the LoRA customization setting, LoRA adapters are added only to the U-Net with a rank of 4 and tnnet using a batch size of and a learning rate of 5 x 10-5 for 2000 training steps.
C. Additional Experimental Results
C.1. Additional Quantitative Results for Object-Driven Image Synthesis in LoRA
C.2. Additional Quantitative Results for Object- Image Synthesis in DreamBooth
C.3. Additional Qualitative Results for Object-Drive Image Synthesis in DreamBooth
13

---

CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
Table 5. Quantitative results for object-driven image synthesis using CAT methods customized in LoRA. The table presents performance metrics FSS FQS for the CelebA-HO and VGGFace2 datasets across different protective perturbation settings. optimal values among different CAT settings for each metric are bolded.
VGGFACE2
METRIC
FSS
TQS 1
FSS
FQS 1
METROD
BASELINE
CAT-BOTH CAT-EN CAT-DE BASBLINE
CAT-BOTH CAT-EN CAT-DE CAT-BOTH CAT-EN CAT-DE BASBLINE
CAT-BOTH CAT-EN CAT-DE
DDV+)(+)
0,351
0.574
0.455
0.402
0.260
0.369
0.585
0.377
0,301
0.490
0.417
0.377
0.257
0.400
0.499
0.404
ADVDM(-)
MIST
0.135
0.565
0.516
0.005
0.363
0.469
0.532
0.022
0.069
0.596
0.476
0.001
0.384
0.565
0.706
0.000
SDS(+)
0.464
0.497
0.559
0.285
0.381
0.328
0.525
0.389
0.393
0.541
0.443
0.413
0.406
0.399
0.505
0.392
SDSC-
SDST
0.203
0.572
0.486
0.032
0.217
0.488
0.574
0.137
0.112
0.549
0.544
0.052
0.248
0.578
0.716
0.217
GLAZE
0.355
0.575
0.474
0.714
0.752
0.617
0,330
0.602
0.472
0.576
0.755
0.586
ANTI-BB
0.539
0.596
0.616
0.465
0.497
0.458
0.541
0.411
0.460
0.583
0.586
0.439
0.416
0.569
0.725
0.494
METACLOAK
0.528
0.597
0.543
0.549
0.376
0.442
0.458
0.444
0.416
0.520
0.513
0.431
0,355
0.531
0.551
0.496
Table Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth, compared to the baseline across CelebA HQ and VGGFace2 datasets. The table reports FID scores, where lower values indicate better alignment with the real distribution. Optimal values among CAT settings are bolded for each dataset and method.
FID
CELEBA-HQ
VGGFACE2
METHOD
BASELINE
CAT-BOTH CAT-EN BASELINE
CAT-BOTH CAT-EN
ADVDM(+)
340.0
264.9
223.7
435.2
274.9
249.0
ADVDM(-)
134.3
104.0
102.0
203.9
189.4
188.6
MIST
263.6
133.6
136.1
359.6
187.8
198.9
277.4
255.7
SDS(-)
125.4
103.4
110.2
208.9
183.3
185.6
133.3
195.3
GLAZE
196.7
100.1
90.4
228.0
160.9
191.3
ANTI-DB
180.4
131.4
106.4
320.5
202.1
190.8
METACLOAK
175.0
179.6
171.6
316.3
200.4
170.9
AdvDM(+)
AdvDM(-)
Mist
SDS(-)
SDS(-)
SDST
Glaze
Anti-DB
MctaCloak
Baseline
CAT-both
CAT-en
CAT-de
Figure Qualitative results for object-driven image synthesis customization an identity from the CelebA-HQ dataset using DreamBooth and CAT settings with the text prompt "a dsir portrait of sky person". Each row represents different setting: Baseline, CAT-both, CAT-en, and CAT-de, while each column corresponds to a different protective perturbation setting. The results illustrate the effectiveness of different CAT settings generating human faces from protected images.
14


Reviews:
[{"review_id": "Vs2b0gSORR", "reviewer": "Reviewer_GQca", "summary": "This paper proposes an attack named CAT that can break the protection of preventing diffusion models from effectively learning unauthorized data being perturbed by defensive noise. The authors first empirically identify that the mechanism behind existing defensive perturbations is to make embeddings of perturbed images look different from the embeddings of clean images. Based on this observation, the authors propose to break the protection via improving the robustness of the encoder in diffusion models against those defensive perturbations. Experiments are conducted on the stable-diffusion-v2.1 model with various protection methods.\n\n## **update after rebuttal**\n\nAfter reading the rebuttal, I think this paper has novel results. So I decide to maintain my current score but tend to acceptance.\n\nFor the authors, they should update the paper to include:\n\n- Additional results on data-augmentations.\n\n- Discussions on RobustCLIP.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable comments (Q). We will incorporate the suggested references you mentioned. We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period. \n\n> **Q1: I think the authors ... various low-pass filters.**\n\n**A1:** We have compared our proposed CAT with two image augmentation-based adaptive attacks towards the protective perturbation: Noisy-Upscaling [r1] (based on optimization) and Gaussian filtering (based on low-pass filters). The evaluation is conducted on both the CelebA-HQ and VGGFace2 datasets using CAT-both and CAT-en, following the same experimental setting. For Noisy-Upscaling, we adopt default configurations provided in the original paper, and for Gaussian Filtering, we use a Gaussian kernel with size equal to 5 and $\\sigma = 1.0$. **Due to space limitations, we present the results on the CelebA-HQ  in Table 3 and on the VGGFace2 dataset in Table 4 (both in response to Reviewer jvVV)**. We apologize for any inconvenience this may cause and appreciate your understanding. We can observe that our proposed CAT (CAT-both or CAT-en) consistently achieves comparable or superior performance to both Noisy-Upscaling and Gaussian Filtering across all protective perturbations, in terms of both FQS and FSS. These results highlight the competitive effectiveness of CAT compared to existing purification-based methods.\n\n> **Q2: Two major drawbacks... for the specified defensive noise.**\n\n**A2:** We would like to clarify two key aspects of our threat model: the data owner has the clean data $x_c$ and intends to share it publicly. To prevent unauthorized customization, the owner applies a perturbation to generate the protected data $x_a$, which is then released. **In this setting, the adversary only has access to the protected data $x_a$ and is unaware of the specific protection method used by the data owner. This protected data already contains the defensive noise, and no additional generation is required by the adversary.** We want to demonstrate that, even without knowledge of the protection technique and with access only to the protected data, the adversary can still effectively learn using our proposed CAT. This is achieved by using our proposed **contrastive adversarial loss** during customization. We will include a more detailed explanation of the threat model in the revised version.\n\n> **Q3: I think the AT method from RobustCLIP [r1] ... for new defensive noise.**\n\n**A3:** We fully agree that RobustCLIP presents an effective adversarial training (AT) framework for enhancing the robustness of CLIP-based vision encoders, particularly against perturbations that disrupt text-image semantic alignment. That said, we would like to clarify key differences between RobustCLIP and our proposed CAT:\n\n1. **Optimization objective:** RobustCLIP aims to *preserve semantic alignment between text and image representations*. In contrast, CAT defends against protective perturbations that distort the latent representation in the autoencoder, using a *contrastive adversarial loss* to explicitly enforce latent space alignment.\n2. **Implementation framework:** While it is possible to apply RobustCLIP to the VAE encoder in LDMs, this would involve fine-tuning the VAE on self-generated adversarial-clean pairs, which is the same idea aligned with CAT’s motivation. However, *CAT achieves this via lightweight adapters* applied during customization on protected data. These adapters are detachable at inference time, leaving the original model performance unaffected. In contrast, fine-tuning of RobustCLIP would require significantly more data and adversarial examples, and would modify the entire encoder, potentially degrading performance on the original task.\n\nDespite these differences, we greatly appreciate the reviewer’s suggestion and will discuss it in the related work section to highlight these differences.\n\n\n> **Q4: I suggest the authors add a small section ... of this work.**\n\n**A4:** We totally agree that a brief explanation of how Stable Diffusion operates as an LDM with the latent autoencoder would improve the paper’s readability for those unfamiliar with this background. We will add a dedicated section to clarify this in the related work.\n\n> **Q5: What do those blue points $z_0$ mean in Fig.3? It seems that they are never explained in the paper.**\n\n**A5:** We apologize for the confusion caused by the incorrect labeling in the original Fig. 3. The blue points were intended to represent $z_c$， which is the latent embeddings of clean samples as correctly noted in the caption. We have updated both the figure and its caption to reflect this correction. Thank you for pointing this out.\n\n[r1] Hönig, R., Rando, J., Carlini, N., & Tramèr, F. Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI. In *The Thirteenth International Conference on Learning Representations*.", "claims_and_evidence": "See **Weaknesses/Questions/Suggestions**.", "methods_and_evaluation_criteria": "See **Weaknesses/Questions/Suggestions**.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See **Weaknesses/Questions/Suggestions**.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "The concept of \"preventing unauthorized image usage via defensive noises\" originally comes from the research of \"unlearnable examples\". Therefore, the authors should also cite and review papers on \"unlearnable examples\" such as:\n\n1. Huang et al. \"Unlearnable Examples: Making Personal Data Unexploitable.\" ICLR 2021.\n\n2. Fu et al. \"Robust Unlearnable Examples: Protecting Data Against Adversarial Learning.\" ICLR 2022.\n\n3. Ren et al. \"Transferable Unlearnable Examples.\" ICLR 2023.\n\n4. Liu et al. \"Stable unlearnable example: Enhancing the robustness of unlearnable examples via stable error-minimizing noise.\" AAAI 2024.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. I like the observation of identifying the mechanism behind existing protective perturbations. It is intuitively sound and makes sense.\n\n\n**Weaknesses/Questions/Suggestions:**\n\n1. I think the authors should compare their proposed method with some simple image augmentation-based defenses, for example, various low-pass filters.\n\n2. Two major drawbacks of the proposed method are that to perform adversarial training following Eq.(1), the adversary needs to: (1) know in advance what defensive noise is leveraged by $x_a$, and (2)  generate a set of protected training images for the specified defensive noise.\n\n3. I think the AT method from RobustCLIP [r1], which aims to enhance the adversarial robustness of CLIP image encoder against input perturbation within a pre-defined radius, is much better than the proposed CAT method. Although this method is originally designed for CLIP, it can be directly applied to any image encoders such as the VAE used in stable-diffusion-v2.1. Under the RobustCLIP's AT method, the adversary can train the VAE encoder only once to make it be robust to multiple types of protective perturbations. However, the proposed CAT method needs to retrain the VAE encoder every time for new defensive noise.\n\n4. I suggest the authors add a small section explaining how the stable diffusion works with encoders. The current paper is very difficult for readers without any background knowledge about stable diffusion to understand the threat model of this work.\n\n5. What do those blue points $z_0$ mean in Fig.3? It seems that they are never explained in the paper.\n\n\n**Reference**\n\n[r1] Schlarmann et al. \"Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models.\" ICML 2024.", "other_comments_or_suggestions": "See **Weaknesses/Questions/Suggestions**.", "questions_for_authors": "See **Weaknesses/Questions/Suggestions**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3v4mzKw9c", "reviewer": "Reviewer_M2AM", "summary": "This paper investigates adversarial examples as protective perturbations in latent diffusion models. The authors reveal that the reason why adversarial examples are effective is primarily due to the distortion of their latent representations. Based on this observation, they propose the CAT method to attack protective methods, highlighting their lack of robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable comments (Q). We will correct the identified typos and incorporate the suggested references you mentioned. We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period.\n\n\n> **Q1: The claim ... other potential factors.**\n\n**A1:** We would like to clarify that the observed strong correlation refers to the effectiveness of adversarial noise as protective perturbations and the distortion in their latent representations. This conclusion is supported by the following observations:\n\n1. **Latent representation distortion:** As shown in Fig. 3, adversarial noise (red dots) causes significantly more distortion in the latent space compared to random perturbations (yellow dots) under the same budget. After applying CAT, the adversarial samples (green dots) are noticeably re-aligned with the clean samples (blue dots). This observation is quantitatively verified in Fig. 4.\n2. **Learnability of perturbed latent representations:** Fig. 5 shows that adversarially perturbed latent representations can still be effectively learned by the diffusion model, with learnability comparable to clean and randomly perturbed samples.\n3. **Degradation on generation quality:** Fig. 6 and Table 1 in the manuscript present results when fine-tuning on adversarially perturbed data. Without CAT (baseline), generation quality drops significantly, while applying CAT improves quality to a large extent.\n\nTogether, these obervations and experimental results support our conclusion that the unlearnability of adversarially perturbed samples primarily comes from their latent representation distortion. That said, we fully agree with the reviewer that, even in the presence of a strong correlation, causality should be claimed with caution, as other factors may also play a role. We thank the reviewer for this important insight, and will emphasize that this conclusion is mainly based on empirical experimental observations \n and there are other potential factors.\n\n\n> **Q2: Lack of baseline ... methods.**\n\n**A2:** We have compared our proposed CAT with baseline adaptive attacks against protective perturbations: Noisy-Upscaling [r1] (optimization-based) and Gaussian Filtering (low-pass filtering-based). *IMPRESS++ is not open-sourced yet, so we instead adopt Noisy-Upscaling, the superior adaptive attack from the same paper.* **Due to space limitations, experimental details are provided in our response to Reviewer GQca, with results on the CelebA-HQ dataset shown in Table 3, and on the VGGFace2 dataset in Table 4 (both in response to Reviewer jvVV).** We apologize for any inconvenience this may cause and appreciate your understanding. We can observe that our proposed CAT (CAT-both and CAT-en) consistently achieves comparable or superior performance to both Noisy-Upscaling and Gaussian Filtering across all protective perturbations, in terms of both FQS and FSS. These results highlight the competitive effectiveness of CAT compared to existing purification-based methods.\n\n\n> **Q3: The evaluation ... generated images.**\n\n**A3:** We appreciate the reviewer for bringing this to our attention. We will report the results for Retina-FDR and ISM separately, as well as those for TOPIQ-FDR and FIQ, in the supplementary materials. **Due to space limitations, we present the traditional evaluation metric FID to assess the quality of generated images in Table 1 (in response to Reviewer 6VvR).** It can be observed that our proposed CAT (either CAT-both or CAT-en) consistently improves generation quality in terms of FID across both datasets and all evaluated protection methods. These results demonstrate the effectiveness of our approach.\n\n> **Q4: According to Table 1 ... behind this?**\n\n**A4:** The potential reason behind this that CAT-de only adds adapters to the VAE decoder, which has limited impact on realigning latent representations. Fine-tuning only the decoder is more challenging, as the diffusion model learns from distorted latents that are highly diverse, making accurate reconstruction difficult. The weaker performance of CAT-de compared to CAT-en and CAT-both further supports our observation that latent distortion is the key factor behind the effectiveness of adversarial noise as a protective perturbation. \n\n\n> **Q5: Since “CAT-both” ... use in practice?**\n\n**A5:** In our experiments, we keep the parameter size the same for CAT-both and CAT-en. Specifically, CAT-both uses half the adapter rank of CAT-en, as it adds adapters to both the encoder and decoder. While their performance varies across tasks, datasets, and metrics, the overall results are comparable. That said, we speculate that CAT-en may perform better on tasks requiring more specific or dense latent representations, **such as the style mimicry task in Table 2 (in response to Reviewer 6VvR)**, as it enables stronger latent alignment during customization.", "claims_and_evidence": "The claim that the experiments in section 3 explain why adversarial examples are effective as protective perturbations seems an overclaim to me. The most direct conclusions of the two experiments in section 3 are that adversarially perturbed images lead to larger distortions in latent representations and that the diffusion model is able to learn adversarial examples. It is expected to have an explicit measure of generation degradation to show the correlation between large distortions and protection effectiveness. Even if there is a strong correlation, the causality should be carefully claimed because there may be other potential factors.", "methods_and_evaluation_criteria": "The proposed method and evaluation  makes sense for the task.", "theoretical_claims": "no theoretical claims in this paper", "experimental_designs_or_analyses": "See “Claims and Evidence” part. Other experiments are sound.", "relation_to_broader_scientific_literature": "Latent Representation Distortion: Previous works [1,2] built adversarial examples to protect diffusion models. However, a deeper investigation into how the adversarial examples work is lacking. This paper conducts qualitative and quantitative experiments to analyze adversarial examples as protective methods.\n\nAdaptive Attack: Most existing attacks rely on purification. This paper proposes a model-based adaptation method.\n\n[1] Liang, Chumeng, et al. \"Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples.\" arXiv preprint arXiv:2302.04578 (2023).\n\n[2] Xue, Haotian, et al. \"Toward effective protection against diffusion-based mimicry through score distillation.\" The Twelfth International Conference on Learning Representations. 2023.\n\n[3] Hönig, Robert, et al. \"Adversarial perturbations cannot reliably protect artists from generative ai.\" arXiv preprint arXiv:2406.12027 (2024).", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\nA deeper investigation into how adversarial examples are effective as protective perturbations helps better understand this kind of method.\n\nThe model-based adaptive attack method is different from previous purification-based attacks, which shows the novelty of this work.\n\nThe authors evaluate the attack against nine protection methods, demonstrating the effectiveness of the proposed attack.\n\nWeaknesses:\n\nSee “Claims and Evidence” part. The conclusion in section 3 seems an overclaim to me. If I misunderstood, please correct me.\n\nLack of baseline methods. Even if the model-based adaptive attack is novel, I still suggest comparing it with previous methods (e.g. IMPRESS++[1]) to show whether the model-based adaptive attack is more effective than other methods.\n\nThe evaluation metrics “FSS” and “FQS” are not standard but reasonable. Separate reports of Retina-FDR and ISM may enhance understanding of the results. And so do TOPIQ-FDR and FIQ. In addition, some traditional evaluation metrics like PSNR and SSIM are also expected to be adopted to show the quality of the generated images.\n\n[1] Hönig, Robert, et al. \"Adversarial perturbations cannot reliably protect artists from generative ai.\" arXiv preprint arXiv:2406.12027 (2024).", "other_comments_or_suggestions": "All the quotation marks should be “ ” instead of ” ”.\n\nThe header shows “Submission and Formatting Instructions for ICML 2024”.", "questions_for_authors": "According to Table 1, the “CAT-de” is consistently weaker than “CAT-both” and “CAT-en”, and is sometimes even weaker than baseline. What are the potential reasons behind this?\n\nSince “CAT-both” and “CAT-en” are comparable on different metrics, how to choose which method to use in practice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pjszbWONiM", "reviewer": "Reviewer_6VvR", "summary": "This paper examines the effectiveness of adversarial perturbations in protecting data from unauthorized customization in LDMs. The authors reveal that these perturbations work by distorting latent representations and propose CAT as an adaptive attack that reduces their effectiveness. Experimental results highlight the vulnerability of current protection methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable comments (Q). We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period.\n\n> **Q1: Yes, but could be ... using the fidelity metrics, such as FID.**\n\n**A1:** We thank the reviewer for pointing this out. In the evaluation of **Table. 1 below**, we use the FID metric to assess generation quality. Specifically, for each identity, we generate 30 images per prompt (two prompts in total), and compute FID by comparing the results to images generated by a customization model trained on clean samples. We observe that both CAT-both and CAT-en consistently achieve lower FID scores than the baseline across both datasets and all evaluated protections. These results demonstrate the effectiveness of our CAT under the evaluated settings.\n\n**Table 1.** Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth for CelebA-HQ and VGGFace2 datasets compared to the Noisy-Upscaling (NU) and Gaussian Filtering (GF) methods.\n\n| FID$\\downarrow$ | CelebA-HQ |           |           | VGGFace2 |           |           |\n| --------------- | --------- | --------- | --------- | -------- | --------- | --------- |\n|                 | Baseline  | CAT-both  | CAT-en    | Baseline | CAT-both  | CAT-en    |\n| AdvDM(+)        | 340.0     | 264.9     | **223.7** | 435.2    | 274.9     | **249.0** |\n| AdvDM(-)        | 134.3     | 104.0     | **102.0** | 203.9    | 189.4     | **188.6** |\n| Mist            | 263.6     | **133.6** | 136.1     | 359.6    | **187.8** | 198.9     |\n| SDS(+)          | 327.2     | 277.4     | **247.6** | 363.9    | 295.4     | **255.7** |\n| SDS(-)          | 125.4     | **103.4** | 110.2     | 208.9    | **183.3** | 185.6     |\n| SDST            | 223.0     | **133.3** | 133.5     | 335.8    | **195.3** | 200.4     |\n| Glaze           | 196.7     | 100.1     | **90.4**  | 228.0    | **160.9** | 191.3     |\n| Anti-DB         | 180.4     | 131.4     | **106.4** | 320.5    | 202.1     | **190.8** |\n| MetaCloak       | 175.0     | 179.6     | **171.6** | 316.3    | 200.4     | **170.9** |\n\n\n> **Q2: The core rationale ... proposed approach unconvincing.**\n\n**A2:** We sincerely thank the reviewer for the valuable suggestion. To the best of our knowledge, this is the first work that systematically evaluates the robustness of nine existing protective perturbation methods across two downstream tasks, which requires careful data preparation and repeated experiments. That said, we totally agree that for the style mimicry task, only qualitative results were provided in the main paper due to space limitations. To address this, we include quantitative results **in Table 2 below** in CLIP-IQA score [r2] for both CAT-both and CAT-en settings. We observe that in the style mimicry task, CAT consistently outperforms the baseline across all evaluated protection methods, further demonstrating the effectiveness of our approach.\n\n**Table 2.** Quantitative results for style mimicry using CAT methods customized in DreamBooth for the WikiArt dataset.\n\n| CLIP-IQA$\\uparrow$ | Baseline | CAT-both | CAT-en    |\n| ------------------ | -------- | -------- | --------- |\n| AdvDM(+)           | 0.343    | 0.390    | **0.621** |\n| AdvDM(-)           | 0.463    | 0.536    | **0.697** |\n| Mist               | 0.345    | 0.465    | **0.694** |\n| SDS(+)             | 0.285    | 0.366    | **0.366** |\n| SDS(-)             | 0.501    | 0.481    | **0.723** |\n| SDST               | 0.406    | 0.485    | **0.712** |\n| Glaze              | 0.532    | 0.614    | **0.730** |\n| Anti-DB            | 0.315    | 0.544    | **0.672** |\n\n> **Q3: I am curious about the ... and helpful to the future reader.**\n\n**A3:** Thank you for the insightful comment. We fully agree that while adaptive attacks reveal current vulnerabilities in protective perturbations, their broader value lies in informing the development of more effective defenses. In this regard, we would like to emphasize that our CAT method not only exposes the limitations of existing protections but also offers potential insights for future defense strategies. In particular, our findings suggest that effective protection for LDMs may benefit from incorporating defense mechanisms that explicitly consider the diffusion process, especially in end-to-end optimized frameworks, rather than relying solely on latent autoencoders, which can be more easily compromised by adaptive attacks. We believe this perspective highlights a promising direction for designing more robust protective perturbations to better safeguard the IP rights of data owners. A more detailed discussion will be added in the revised version.\n\n[r2] Wang, J., Chan, K. C., & Loy, C. C. (2023, June). Exploring clip for assessing the look and feel of images. In *Proceedings of the AAAI conference on artificial intelligence* (Vol. 37, No. 2, pp. 2555-2563).", "claims_and_evidence": "Most claims are supported by clear evidence.", "methods_and_evaluation_criteria": "Yes, but could be improved. For instance, in Table 1 or for the style mimicry scenario, it would be more insightful to evaluate using the fidelity metrics, such as FID.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The study is reasonably demonstrated, but some expectations are not met. There are no quantitative results for style mimicry and in addition, there is no comparisons with other purification methods. The authors only provide results with and without their method, which makes the effectiveness of the proposed approach unconvincing.", "relation_to_broader_scientific_literature": "The intuition and experiments focus on mimicry tasks, which have emerged as significant concerns in the context of generative AI, particularly regarding copyright issues.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The core rationale behind the proposed method is reasonable. However, as noted in the “Experimental Design” section, the lack of sufficient experiments and analysis limits the convincingness of the work.", "other_comments_or_suggestions": "No.", "questions_for_authors": "I am curious about the recent trend of research primarily focusing on mimicry attacks to circumvent existing protection methods. I think this approach inherently has limitations in advancing genuinely robust protection methods, including the current study. Given this purification, how might we move towards stronger protection methods, or at least derive valuable insights to guide future research directions? It would be very insightful and helpful to the future reader.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U21qjwYSzL", "reviewer": "Reviewer_jvVV", "summary": "This paper studies protective perturbations for LDMs, where the success of existing methods are based on distorted  latent representations. To examine these protections, the authors propose Contrastive Adversarial Training (CAT), which inserts lightweight adapters into the latent autoencoder. Specially, CAT realigns the latent representations, reducing the effectiveness of protective perturbations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable comments (Q). We hope that our responses (A) have fully addressed the concerns, and remain committed to clarifying any further questions that may arise during the discussion period.\n\n> **Q1: It is good to try training-free DM customization methods (e.g., IP-Adapter).**\n\n**A1:** We appreciate this insightful suggestion. Indeed, training-free customization methods such as IP-Adapter represent a promising direction for low-cost and scalable diffusion model customization. While our current study focuses on training-based customization approaches (e.g., DreamBooth and LoRA) due to their widespread usage and controllability in evaluating robustness, extending CAT to training-free settings is an exciting future direction. We plan to incorporate IP-Adapter into future evaluations to further evaluate the generalization of CAT beyond current training-based frameworks. \n\n**Table 3.** Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth for the CelebA-HQ dataset compared to the Noisy-Upscaling (NU) and Gaussian Filtering (GF) methods.\n\n| CelebA-HQ | FSS$\\uparrow$ |        |       |       | FQS$\\uparrow$ |           |           |       |\n| --------- | ------------- | ------ | ----- | ----- | ------------- | --------- | --------- | ----- |\n|           | CAT-both      | CAT-en | NU    | GF    | CAT-both      | CAT-en    | NU        | GF    |\n| AdvDM(+)  | **0.643**     | 0.529  | 0.531 | 0.492 | 0.431         | 0.448     | **0.481** | 0.352 |\n| AdvDM(-)  | **0.623**     | 0.571  | 0.469 | 0.607 | 0.549         | **0.611** | 0.498     | 0.526 |\n| Mist      | **0.572**     | 0.501  | 0.491 | 0.488 | **0.597**     | **0.580** | 0.475     | 0.493 |\n| SDS(+)    | **0.602**     | 0.499  | 0.599 | 0.409 | 0.413         | 0.423     | **0.503** | 0.302 |\n| SDS(-)    | **0.678**     | 0.599  | 0.468 | 0.583 | **0.597**     | **0.587** | 0.494     | 0.493 |\n| SDST      | **0.594**     | 0.485  | 0.470 | 0.446 | 0.587         | **0.588** | 0.474     | 0.464 |\n| Glaze     | **0.610**     | 0.577  | 0.533 | 0.547 | 0.618         | **0.676** | 0.496     | 0.533 |\n| Anti-DB   | **0.662**     | 0.597  | 0.540 | 0.575 | 0.608         | **0.664** | 0.469     | 0.543 |\n| MetaCloak | **0.642**     | 0.578  | 0.521 | 0.540 | 0.460         | **0.475** | 0.395     | 0.324 |\n\n**Table 4.** Quantitative results for object-driven image synthesis using CAT methods customized in DreamBooth for the VGGFace2 dataset compared to the Noisy-Upscaling (NU) and Gaussian Filtering (GF) methods.\n\n| VGGFace2  | FSS$\\uparrow$ |           |           |           | FQS$\\uparrow$ |           |       |       |\n| --------- | ------------- | --------- | --------- | --------- | ------------- | --------- | ----- | ----- |\n|           | CAT-both      | CAT-en    | NU        | GF        | CAT-both      | CAT-en    | NU    | GF    |\n| AdvDM(+)  | 0.534         | **0.560** | 0.518     | 0.506     | 0.481         | **0.578** | 0.506 | 0.363 |\n| AdvDM(-)  | **0.564**     | 0.547     | 0.529     | 0.563     | 0.635         | **0.676** | 0.563 | 0.506 |\n| Mist      | 0.557         | 0.521     | **0.566** | 0.518     | 0.662         | **0.701** | 0.518 | 0.437 |\n| SDS(+)    | 0.486         | **0.508** | 0.498     | 0.402     | 0.438         | **0.569** | 0.402 | 0.281 |\n| SDS(-)    | 0.570         | 0.569     | 0.509     | **0.593** | **0.700**     | 0.671     | 0.593 | 0.558 |\n| SDST      | **0.559**     | 0.546     | 0.521     | 0.538     | 0.627         | **0.671** | 0.538 | 0.482 |\n| Glaze     | **0.607**     | 0.576     | 0.503     | 0.549     | **0.733**     | 0.723     | 0.549 | 0.562 |\n| Anti-DB   | **0.584**     | 0.546     | 0.566     | 0.548     | 0.636         | **0.656** | 0.548 | 0.499 |\n| MetaCloak | 0.560         | **0.631** | 0.566     | 0.542     | 0.504         | **0.633** | 0.542 | 0.349 |", "claims_and_evidence": "CAT can effectively neutralize representation distortion-based protective perturbations through contrastive adversarial training.", "methods_and_evaluation_criteria": "The proposed adaptive attack utilzes a contrastive adversarial loss with adapters inserted into the latent autoencoder, thereby “attacking” the distortions caused by protective perturbations.", "theoretical_claims": "No", "experimental_designs_or_analyses": "Experimental results are well organized. Authors compare CAT against nine protective perturbation methods under different customization frameworks (e.g., DreamBooth and LoRA). Both quantitative results (improvements in FSS and FQS) and qualitative examples support their claims.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "It is good to try training-free DM customization methods (e.g., IP-Adapter).", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field and finally received overall borderline scores: Weak accept, Weak accept, Weak accept, and Weak accept.
The major concerns of the reviewers are:
1.	 inconsistencies in method design and motivation,
2.	additional results on data-augmentations, other metrics, and comparisons with other methods,
3.	insufficient experiments results analysis.
The authors address most of the above concerns during the discussion period. Hence, I make the decision to accept the paper if there is room in the program. However, if the paper is finally accepted, the authors should carefully revise it according to the reviewers’ comments.
Decision: Accept (poster)