Variance as a Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Authors: Jiachen Yang
Yusong Wang
Yanmei Fang
Yunshu Dai
Fangjun Huang
Venue: ICML 2025 poster (2025)
Keywords: Customized Latent Diffusion Models
Adversarial Attacks
Privacy Protection
Transferability
AI Security
Primary area: social_aspects->privacy

Abstract: Latent Diffusion Models (LDMs) enable fine-tuning with only a few images and have become widely used on the Internet. However, it can also be misused to generate fake images, leading to privacy violations and social risks. Existing adversarial attack methods primarily introduce noise distortions to generated images but fail to completely erase identity semantics. 
In this work, we identify the variance of VAE latent code as a key factor that influences image distortion. Specifically, larger variances result in stronger distortions and ultimately erase semantic information. Based on this finding, we propose a Laplace-based (LA) loss function that optimizes along the fastest variance growth direction, ensuring each optimization step is locally optimal. Additionally, we analyze the limitations of existing methods and reveal that their loss functions often fail to align gradient signs with the direction of variance growth. They also struggle to ensure efficient optimization under different variance distributions. To address these issues, we further propose a novel Lagrange Entropy-based (LE) loss function.
Experimental results demonstrate that our methods achieve state-of-the-art performance on CelebA-HQ and VGGFace2. Both proposed loss functions effectively lead diffusion models to generate pure-noise images with identity semantics completely erased. Furthermore, our methods exhibit strong transferability across diverse models and efficiently complete attacks with minimal computational resources. Our work provides a practical and efficient solution for privacy protection.

Variance as a Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Jiachen Yang' Yusong Wang' Yanmei Fang 12 Yunshu Dai
Fangjun Huang"
Abstract
1. Introduction
Latent Diffusion Models (LDMs) enable fine- tuning with only a few images and have become widely used on the Internet. However, can also be misused to generate fake images, leading privacy violations and social risks. Existing ad- versarial attack methods primarily introduce noise distortions to generated images but fail to com- pletely erase identity semantics. In this work, we identify the variance VAE latent code as a key factor that influences image distortion. Specifi- cally, larger variances result in stronger distortions and ultimately erase semantic information. Based on this finding, we propose a Laplace-based (LA) loss function that optimizes along the fastest vari- ance growth direction, ensuring each optimiza- tion step is locally optimal. Additionally, we ana- lyze the limitations of existing methods and reveal that their loss functions often fail to align gradi- ent signs with the direction of variance growth. They also struggle to ensure efficient optimiza- tion under different variance distributions. To address these issues, we further propose a novel Lagrange Entropy-basse based (LE) loss function. Ex- perimental results demonstrate that our methods achieve state-of the- performance on CelebA- HQ and VGGFace2. Both proposed loss func- tions effectively lead diffusion models to generate pure-noise images with identity semantics com- pletely erased. Furthermore, our methods exhibit strong transferability across diverse models and efficiently complete attacks with minimal compu- tational resources. Our work provides a practical and efficient solution for privacy protection.
Latent Diffusion Models (LDMs) have revolutionized the field of generative models (Song & Ermon, 2019; Ho et al., 2020; Nichol & Dhariwal, 2021). They enable generate high- quality and diverse images that closely resemble human- generated content (Rombach et al.. 2022: Peebles & Xie, 2023; Podell et al., 2024; Labs, 2023; Esser et al., 2024). This breakthrough in generative modeling is largely at- tributed to the ability LDMs to perform few-shot fine- tuning (Gal et al., 2022; Hu et al., 2021; Ruiz al., 2023; Kumari et al., 2023; Ye et al., 2023). With few as 4 to 5 images, LDMs can learn new concepts, such as specific human faces or unique artistic styles. This capability has enabled their widespread use in academia and industry, driv- ing advancements in image synthesis, personalization, and artistic creation. However, this powerful capability has also raised significant privacy concern.. Malicious users can exploit LDMs to generate fake images using a small set of personal data, such as photos from social media. This poses serious risks to individual privacy and reputation (Higgins, 2023). These risks underscore the urgent need for mitigation strategies to address this growing issue.
Under these challenges, researchers have proposed adversar- ial attacks on diffusion models to prevent them from learn- ing data features. These attacks can be divided into three categories: maximizing the training loss of UNet (Liang et al., 2023; Van Le et al., 2023; Xue et al., 2023; Wang et al., 2024; Liu et al., 2024b), targeting the cross-attention module within UNet (Xu et al., 2024; Liu et al., 2024a; Lo et al., 2024), and attacking the VAE encoder (Salman et al., 2023; Shan et al., 2023; Liang & Wu, 2023: Li et al., 2024). The first types often rely on specific prompts and model architectures when designing perturbations, which limits their transferability. Furthermore, due to the large number of parameters in UNet, these methods are com- putationally expensive and time-consuming. These limita- tions make them impractical for rapidly evolving models or pooource-onnsrainee scenarios. In contrast, tacks are generally prompt-indepeneett since the VAE does not participate in encoding prompts during image genera- tion. However, these methods mainly focus on maximizing the distribution differences of latent codes between clean
'School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518106, China. *Guangdong Provincial Key Laboratory of Information Security Technology, Guangzhou 510006, China. Correspondence to: Yanmei Fang <fangym@ @ mail. sys.l.yysu..d..n..
at-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the author(s).

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Input Image
Variance of latent code
VAE Encoder
Unst
Decoder
122
great var
10
10a
2,61++01
37
Distortion
101
meidum var
Identity erose
12
ASPL
Mist
11
Distertion
Idenfity arose
small var
SimAC
PID
Distortion
10:3
2.0e-6
3.00e-86
7.87e-07
Identity erose
1017
LE (ours)
LA (ours)
(deen
ASPL
Mit
SimAC
RD
Lileus)
LAlousi
Figure 1. We find that larger variances in the VAE latent code improve the effectiveness of protection methods. Smaller variances, as in ASPL. and Mist, still preserve identity semantics, while larger variances, in SimAC and PTD, remove most of them. Our proposed Laplace Loss (LA) and Lagrange Entropy Loss (LF) achieve even higher variances, generating pure-noise images with semantics completely erased.
and adversarial images or optimizing their mean and vari- ance. As a result, they only introduce varying levels noise to the generated images but fail effectively remove facial semantic information.
the gradient sign of the loss function with respect to vari- ance. Existing methods like Mist (Liang & 2023) uses Mean Squared Error (MSE) to maximize the variance gap between clean images and adversarial examples. However, this approach misaligns the gradient sign with the direc- tion of variance growth, causing local optimization traps in the early stages and hindering effective variance increase. PID (Li et al., 2024) addresses this issue by applying a logarithmic transformation to the variance, ensuring bet- ter aligmment between the gradient direction and variance growth. Despite this improvement, PID faces diminishing optimization space variance becomes more uniform, lead- ing to slow optimization. In its original work, PID requires 900 steps to remove most identity semantics. It lacks an in-depth theoretical analysis to uncover the root cause of this issue, leaving the problem unexplored. To address these limitations, we propose two novel loss functions: Laplace Loss (LA) and Lagrange-Entrogn tently optimizes toward increasing variance regardless of its distribution, making each update locally optimal. LE com- bines entropy and Lagrange terms to balance optimization, ensuring alignment with variance growth while maintaining sufficient optimization space. Both methods can completely erase identity semantics and generate pure noise images in 30 steps, achieving a 30x speedup over PID. Our main contributions are summarized as follows:
To address the aforementioned issues, perform in- depth investigation of the VAE encoder. PID (Li et al., 2024) observes that the mean of the VAE latent code in- fluences the structure and texture of the generated image, while the variance governs the model's ability to capture core concepts and affects semantic diversity. They suggest that targeting both mean and variance essential for achiev- ing optimal attack performance. However, through a further analysis of VAE properties, we find that variance is the key factor determining the level of distortion and iden- tity semantic erasure in generated images. As shown in Figure I, we encode images protected by various methods into their respective latent codes and analyze their variances. For methods causing minimal distortion, such as ASPL and Mist, the variances of their latent codes closely resemble that of the clean image, thereby preserving most facial se- mantics. In contrast, methods with medium variances, such as SimAC and PID, introduce significant distortion but still fail to completely erase identity semantics.
Loss (LE). LA consis-
Building on this observation, we investigate how the per- turbation matrix in the pixel space influences the variance matrix in the latent space through the VAE encoder. We find that the direction of variance growth is closely related to
We observe that the variance of latent codes directly
2

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
affects the semantic integrity of generated images. Fur- thermore, we find that the optimization direction of the variance is closely tied to the gradient sign of the loss function with respect to the variance.
In addition DreamBooth, lightweight personalization techniques such as Textual Inversion (Gal et al., 2022), LoRA (Hu et al., 2021), and IP-Adapter (Ye et al., 2023) have gained popularity. Notably, LoRA is widely adopted in online communities for creating character portraits and imi- tating artworks. However, its use raises significant privacy concerns and art copyright infringement issues.
We analyze prior methods and identify that their gra- dient signs often misalign with the variance growth or that the optimization space of variance becomes pro- gressively compressed. To overcome these issues, we propose two novel loss functions: LA and LE.
2.3. Adversarial Attacks for Diffusion Models
Traditional adversarial attacks primarily target classification models (Goodfellow et al., 2015; Madry et al., 2018). To mitigate the risks of misuse in personalized generation, re- cent studies propose adversarial attacks to prevent LDMMs from learning unauthorized images. These works can be mainly divided into three categories. The first category aims to maximize the training loss of diffusion model and force the predicted noise to deviate from the real noise (Liang et al., 2023; Van Le et al., 2023; Liang & Wu, 2023; Xue et al., 2023; Wang et al., 2024; Liu al., 2024b). The second manipulates cross- antesttntnnioon in UNet to weaken the weight of identity keywords in prompts, such as "sks" in the prompt "a photo of sks person," in the cross-attention map. This reduces the connection between the prompt and pixel distribution of the generated images (Xu et al., 2024; Liu et al., 2024a; Lo et al., 2024). However, both approaches depend on UNet gradients and specific prompts, leading to poor transferability and high computational costs. The third targets the VAE module, which operates independently of the diffusion model. These methods either maximize the divergence between the latent distributions of clean images and adversarial examples (Salman et al., 2023; Shan et al., 2023; Liang & Wu, 2023) or jointly attack the latent mean and variance (Li et al., 2024). Although these methods can generate distorted images with varying levels of noise, they fail to completely erase identity semantic features.
Our methods achieve state-of-the-art performance on two facial image datasets. They not only generate pure-noise images with completely erased identity se- mantics, but also offer better transferability. lower com- putational cost, and faster attack speeds.
2. Related Work
2.1. Diffusion Models
Diffusion models consist of two main phases: the diffusion process and the reverse diffusion process, Given an image Tu. the diffusion process progressively adds noise to the data distribution based on a predefined noise schedule { ) As noise accumulates, the image Fr becomes increasingly noisy. When T is sufficiently large, the image ultimately transforms into pure Gaussian noise. This process is for- mally described as follows:
Ze = 1 + VBa = VāLzo + vI āie
(1)
where ag = 1 - Bz, ā. =[]=-1 a and (0, is the noise sampled from normal distribution.
The reverse process aims to predict the noise e, added at the time step i with a denoising model co c0(x Through this multi-step denoising process, the model gradually recovers the input image zg. Consequently, the training goal of the reverse process is minimize the error between the estimated noise with prompt and the true noise.
3. Method
(o, m0) = (2.,,t..e-W ||e 6a c0( 2) (2)
3.1. In-deep Analysis of VAE Encoder
Adversarial attacks for diffusion models predominantly rely on Projected Gradient Descent (PGD) (Madry et al., 2018) to iteratively update the perturbations. PGD optimizes the perturbation ő added to the input x. aiming to maximize a
2.2. Personalized Systhesis
DreamBooth (Ruiz et al., 2023) personalizes text-to-image models for target concept by learning from a few refer- ence images with a customized prompt like "a photo of sks person", ", where "sks" " represents the learned concept. It uses generic prompt Cp- "a photo of person". ", and a prior preservation loss with a hyperparameter À to balance per- sonalization and generic information, addressing overfitting and text-shifting. The training combines two objectives:
loss function L under the constraint ||01|p- S The PGD update rule at step t with step size is given as:
(OL) 08
))
8++1 =Ilz, 8t + a sign
(4)
(0, x0)= = -E (0.1) ||e - c0(2:41, c)1l
In PGD, sign a5 determines the update direction of per- turbation Therefore, when analyzing the effect of pertur- bation 5 on the variance a², we decompose 85 using the
(0C)
+ llll' - c0(2/+1: (..,t'cc)))) .p)l
3

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
chain rule follows:
where M., represents the influence of the i-th perturbation component in d on the j-th variance component in o². Using the chain rule, the gradient of the loss L with respect to the i-th perturbation ö; can be expressed as:
OC 0c 0a²
(5)
06
d 06
The term Oa directly reflects the relationship between the perturbation matrix 6 and the change in variance matrix o². Aligning with its sign direction offers the most straightfor- ward path to increasing variance. Then, we further decom- pose this term into the following chain form with the hidden state Ø- f(x')= = f(x + 6) of VAE encoder hidden layer:
do
ac 06,
OL 003 06;
do3
Oc dog
Σ
Σ
Mas
(11)
Building upon Equation 4, we observe that the update direc- tion of the i-th perturbation component in the perturbation matrix is determined by sign (66
(02) and the linear combina-
0C ac
2
ac
0a²
tion of two matrices M..-
(6)
05
08
0o2
a5
The VAE encoder is nonlinear, making it difficult to directly analyze the effect of 5 on the hidden state Φ. However, when ō is small, the relationship can be locally approximated as linear. Specifically, for small perturbations ||6|| S 6, the first-order Taylor expansion of the VAE encoder (xx at x provides:
3.2. Laplace Approximation is Local Optimum
From Equations 5 and 10, observe that the sign of dof represents the direction in which the i-th perturbation di- rectly affects the increase of the j-th variance component. This allows the optimization process to be simplified by 0C ensuring a direct re- assigning a constant gradient to 007
as,
- + f())))))()+++++++++))))())))))+) Jj -,
f(z)+((+)))))
lationship with variance growth. Based on this insight, we propose Laplace-based (LA) loss function defined as:
The Jacobian matrix , of f with respect to the input cap- tures the first-order partial derivatives of each hidden state Φe with respect to each input pixel 85. Intuitively, Jy pro- vides local linear mapping, describing how small changes on perturbations å in image space propagate to the hidden state 6 in latent space. The product Jy 8 represents the locally linearized effect of perturbations on each feature component of the hidden state matrix Φb. Given the small magnitude of 6, its effect on the hidden state in latent space can be approximated as:
102 14
Laplace
(12)
b
where fr is the target mean (usually 0), and b is scaling factor (usually 1). The gradient of this loss function with respect to o² is constant value t. This constant gradient ensures that the optimization direction is solely determined
by as, . which reflects the inherent relationship between the perturbation and the variance growth direction. Specifically, & uniformly scales the gradient across all dimensions, pre- serving the alignment of optimization with variance growth. Consequently, the LA loss ensures consistent alignment with the variance growth direction, enabling each PGD step to achieve locally optimal update.
Dok 06,
Re f0.0
(8)
The Jacobian matrix Ju represents the mapping rela- tionship between the i-th perturbation component in the perturbation matrix 8 and the k-th hidden state component in the hidden state matrix o. Similarly, the mapping from the hidden state matrix é to the variance matrix o² is gov- erned by another neural network. This mapping can be linearized using the Jacobian matrix Jg, which quantifies how changes in the k-th component of Φ affect the j-th vari- ance component o3. The Jacobian matrix Jy characterizes this relationship as follows:
3.3. Limitations of Previous Work
Adversarial attacks for variance manipulation rely on loss function, which dictates the gradient direction ac rectly impacts optimization. In this section, we analyze two common methods for variance optimization: Mean Squared Error (MSE) (Liang & Wu, 2023) and PID (Li et al., 2024), revealing their limitations. The MSE loss is defined as:
de and di-
00²
J
dor 9(k5)
LMSE = [o3 ddtenn)
(13)
where J Jan., quantifies the influence of the k-th hidden state de on the j-th variance component o3 in the latent space. do² Consequently, 30 can be approximated as the product of two Jacobian matrices:
,
where o3 is the j-th variance component in the latent space, and 03 represents the variance of clean latent distribu- tions. Its gradient with respect to o3 is:
Mr 06, k
0o²
Jou.. -J TTan
Σ
(10)
OLMSE 2 20² 0 clenn) 0o7
(14)
4

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
MSE
PID
nents due to its larger gradient values lower variances. This ensures that the gradient sign @Crin aligns with the variance growth direction M.j. However, PID is less re- sponsive to large variance components. Once small vari- ances are optimized, the variance distribution becomes more uniform, which reduces the optimization space and leads to slow convergence. As illustrated in Figure 2, although PID effectively increases variance step 100, its growth rate is significantly slower compared to our proposed LA Loss.
of of Iatentt code
o' of lanent code
day
35
Stags
Stope LA(ours)
LE(ours)
d- of istnt code
of lssent coce
24
1.25
3.4. Lagrange Entropy Loss
3.50
3.25
0
To overcome the limitations of MSE and PID, we propose the Lagrange Entropy (LE) Loss, which integrates tropy term and a Lagrange constraint to balance variance optimization:
-1///
0.55
Steps
22eps
Figure 2. Variance growth trends of latent code within 100 steps. MSE is trapped in a local optimum, with variance remaining at the 1e-6 level. PID increases variance but at very slow rate. In contrast, our methods, LA and LE, rapidly amplify variance to the 1c5 level within just 30 50 steps.
LLE = oofgoo3)) + A
Σ 0
(17)
The entropy term promotes the growth of small variance components and the Lagrange constraint ensures overall variance remains balanced. À = 0.1 controls the trade-off between the two terms, and = sets the target variance. The gradient of TLE with respect to r² is expressed as:
MSE loss maximizes the squared distance between a3 and Plean" However, it suffers from notable limitations. When variance components are perturbed, the initial variance dis- tribution becomes uneven. In the early stages of optimiza- tion, MSE mistakenly prioritizes the few large variance components while neglecting the majority of small variance components. This behavior causes MSE to prioritize large variances, disrupting the optimization process. Specifically, the product of &Cggg and Mr, can result in negative doy
Σ c )
δLLE - (1og(03) + 11 + 22 Oo7
(18)
This gradient ensures that small variance components are optimized more strongly at the beginning, driven by the entropy term. As the distribution grows, the Lagrange term maintains sufficient optimization space for larger variances. As shown in Figure 2, LE loss enables efficient variance growth. Both LE and LA losses can completely remove identity semantics within 30-50 steps, significantly outper- forming the 900-step requirement PID.
value for certain perturbations. This misalignment reverses the gradient direction, leading to ineffective variance growth. This issue can be formally expressed as:
OLMSE, sign(> 06:
ÔLMSE dog
sign(
Σ j
MMi)) + sign() Σ
Mag)
4. Experiments
(15)
As shown in Figure 2, at an optimization step of 100, MSE only increases the variance to 3 x 10-6, which remains nearly identical to d This indicates that MSE falls into local trap at the start of optimization. It fails to effectively amplify the larger variance components it targets and neglects to expand the smaller variance components that require growth. For PID, the loss of variance and gradient are defined as:
4.1. Experimental Setup
We compare LA and LE with several representa- tive methods on the CelebA-HQ (Karras et al.. 2018) and VGGFace2 (Cao et al., 2018) datasets. We select 50 differ- ent identities from each dataset and use 4 reference images per identity as training data. We apply the PGD (Madry et al., 2018) update the perturbations, setting the pertur- bation budget 77 to 0.05 and the step size a to 1/255 for all methods. Baseline methods use default settings, and Stable Diffusion v1.5 serves as the base model.
Datasets:
LLID Σ (logo} -log 22tean)2
(16)
dLPID 0o7
2 (log o3 log o o.
Our goal is to prevent diffusion mod- els from learning core concepts of training images, generat- ing pure noise with erased facial semantics. We generate 100
Evaluation Metrics:
PID prioritizes the optimization of small variance compo-
5

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Table Comparing the performance of method with basclines against DreamBooth (Ruiz al., 2023) on CelchA-HQ and VGGFace2.
The best result under each metric is marked with bold. The prompt used is "a phoio of sks person."
CelebA-HO
VGGFace2
Method
ISM 4
FDFR
Brisque
LPIPS f
ISM 4
FDFR
Brisque
LPIPS
No Defense
0.608
0.041
17.896
0.662
0.638
0.025
18.192
0.724
AdvDM (Liang al., 2023)
0.424
0.307
24.215
0.798
0.142
0.944
47.862
0.868
ASPL (Van Le et al., 2023)
0.406
0.287
24.419
0.805
0.158
0.906
46.142
0.865
Mist (Liang & Wu, 2023)
0.249
0.169
13.981
0.707
0.246
0.257
18.324
0.756
MetaCloak (Liu et al., 2024b)
0.593
0.051
36.325
0.712
0.525
0.059
36.771
0.747
SimAC (Wang et al., 2024)
0.253
0,865
51.059
0,823
0.196
0.981
51.874
0,836
DisDiff (Liu et al., 2024a)
0.605
0.116
29.361
0.695
0.263
0.902
43.623
0.758
SDS- (Xue et al., 2023)
0.655
0.005
38.519
0.743
0.591
0.002
37.325
0.781
PID al., 2024)
0.069
0.938
85.533
0,899
0.046
0.968
86.946
0.945
LF(ours)
1
155.804
1.021
0
1
154.494
1.028
LA(ours)
1
155.845
0.959
0
1
155.845
1.031
AdDM
ANP.
Mist
MetaClonk
SimaC!
DisDia
PID
LE(ours)
(.Aours)
Ci
Figure Visualization our methods baselines against DreamBooth on CelebA-HQ VGGFace2. The first third rows show reference images, while the second and fourth rows display generated images from tne-tuned models.
images for each fine-tuned model and evaluate them using four metries. Face Detection Failure Rate (FDFR) mea- sures whether the RetinaFace detector (Deng et al., 2020) recognizes the images as faces; higher FDFR indicates bet- ter protection. For detected faces, Identity Score Matching (ISM) calculates the distance between facial embeddings of generated and reference images; smaller ISM means weaker identity correlation. Brisque measures image natu- ralness; higher scores indicate greater deviation and distor- tion. LPIPS quantifies perceptual differences; higher scores reflect stronger semantic erasure.
CelebA-HQ and VGGFace2. They achieve ISM of 0 and FDFR of 1 on both datasets, completely removing identity semantics. Figure further demonstrates that our methods generate pure noise images, while baselines retain partial semantics. Higher Brisque and LPIPS scores (155,804 and 1.021) confirm significant image degradation, making the images highly unnatural and unrecognizable.
Cross-model Transferability against LoRA (Hu et al., 2021): To evaluate the transferability of our methods across different diffusion model architectures, we use adversar- ial examples generated on SD1.5 as training data for other models. We test two similar models (SD2.1 and SDXL) and two with significantly different architectures (SD3.5 and FLUX. 1-dev). Tables 2 and 3 demonstrate that our methods achieve superior transferability across diverse dif-
4.2. Main Results
Results againt DreamBooth (Ruiz et al., 2023): Table and Figure 3 show that our methods outperform baselines
6

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Table Comparing the transferability of our method with other approaches against LoRA (Hu al.. 2021) on CelebA-HQ The best
result under each metric is marked with bold.
SD2.1
SDXL
Method
ISM +
FDFR f
Brisque f
LPIPS 1
ISM 4
FDFR
Brisque f
LPIPS f
No Defense
0.729
0.073
16.409
0.669
0.791
0.001
13,483
0.515
AdvDM (Liang et al., 2023)
0.532
0.313
39.369
0.704
0.765
0.019
18.419
0.539
ASPL (Van Le et al., 2023)
0.519
0.331
39.226
0.709
0.766
0.002
20.589
0.524
Mist (Liang Wu, 2023)
0.179
0.231
18.097
0.677
0.583
0.126
24.143
0.622
MetaCloak (Liu et al., 2024b)
0.635
0,087
41.381
0.699
0.738
0.002
22.766
0.517
SimAC (Wang et al., 2024)
0.401
0.642
41.409
0.733
0.746
0.018
10.459
0.546
DisDiff (Liu et al., 2024a)
0.627
0.166
40.127
0.709
0.782
0
17.366
0.519
SDS- (Xue et al., 2023)
0.673
0.016
52.108
0.711
0.732
0
8. 8.103
0.569
PID (Li et al., 2024)
0.089
0.887
91.461
0,949
0.602
0
17.848
0.545
LE(ours) LA(ours)
1
151.089
0.947
0.171
0.649
126.369
0.893
1
154.724
0.955
0.178
0.401
102.815
0.822
Table Comparing the transferability of method with other approaches against LoRA (Hu al., 2021) CelehA-HQ. SD3.5 and
L.UXX1-dew uses more advanced text encoder and replaces the UNet with MM-DiT as the denoising network. The best performance under each metric is marked with bold.
SD3.5
FLUX. 1-dev
Method
ISM 1
FDFR
Brisque
LPIPS
ISM
FDFR
Brisque
LPIPS
No Defense
0.587
0.001
0.699
0.589
0.705
0.001
7.722
0.598
AdvDM (Liang al., 2023)
0.532
0.002
11.755
0.612
0.739
0.009
13.268
0.651
ASPL (Van Le et al., 2023)
0.543
0.001
11.541
0.621
0.743
0.004
12.991
0.641
Mist (Liang & Wu, 2023)
0.456
0.003
18.712
0.629
0.736
0.005
25.182
0.587
MetaCloak (Liu et al., 2024b)
0.469
0
21.861
0,599
0.727
0.007
23.027
0.618
SimAC (Wang et al., 2024)
0.495
0
9,251
0.606
0.735
0.004
3.601
0.645
DisDiff (Liu et al., 2024a)
0.535
0.003
10.252
0.601
0.708
0.021
13.488
0.639
SDS- (Xue al., 2023)
0.499
0
46.701
0.617
0.741
0.003
24.258
0.609
PID (Li al., 2024)
0.484
0.013
15.434
0.605
0.729
0.001
15.326
0.596
LE(ours) LA(ours)
0.217
0.241
65.903
0.803
0.257
0.204
58.423
0.828
0.235
0.214
71.873
0.793
0.282
0.269
52.554
0.845
fusion model architectures. On S and SDXL., meth- ods consistently outperform baselines, achieving the lowest ISM scores (e.g.. O on SD2.1 and 0.171 on SDXL) and the highest FDFR values (e.g., on SD2.1 and 0.649 on SDXL). This indicates their effectiveness in completely eras- ing identity semantics while maintaining strong robustness. On models with more advanced architectures like SD3.5 and FL.UX. which utilize MM-DiT the denoising network, methods consistently demonstrate strong per- formance, achieving significantly lower ISM scores (e.g., 0.211 on SD3.5) and higher Brisque values (c.g., 58.423 on FLUXX.1-dev) compared baselines. These results highlight the adaptability our methods handling both traditional UNet-based and advanced denoising architectures, ensuring consistent semantic erasure under various model structures.
only 4.5 GB of GPU memory and completes protection in 8 seconds. In contrast, other methods demand significantly more resources and time. For example, ASPL and SimAC consume over 24 GB of GPU memory, while MetaCloak takes 1843 seconds per image. These results demonstrate the efficiency and practicality our approach, making accessible to users with standard setups.
4.3. Ablation Experiments
In this section, we present the major ablation experiments. Additional ablation results and visualizations are pro- vided in the appendix C and D.
Attack Step: Table 5 shows that our methods generate pure noise images with completely erased identity semantics within 30 steps. After 50 steps, the protection behavior changes. The images transition from pure noise to chaotic scene-like visuals, still retaining their ability to erase identity semantics but showing decline in metric performance.
Time and GPU memory consumption: Considering the limited GPU resources of individual users, we compare the time and GPU memory required by different methods to protect a single image. Table 4 shows that our method uses

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Table Comparison of the time and GPU memory consumption incurred by our method and other approaches to protect x12x image on A100 GPU.
Table 7. Comparison of defense effectiveness and visual quality with different noise budgets CelebA HQ. "*" is default.
Defense Quality
Visual Quality
Method
Time/s 4
GPU/MB
IMS &
Brisque t
PSNR t
SSIM t
AdvDM (Liang al., 2023)
18.63
8278.63
4/255
0.631
22.385
14.244
0.414
ASPL (Van Le et al., 2023)
189.95
34366.92
8/255
122.266
13.771
0.361
Mist (Liang & Wu, 2023)
18.81
8278.63
0.05
0
155,804
13.664
0.313
MetaCloak (Liu et al., 2024b)
1843.47
16955.00
16/255
155,845
12.301
0.271
SimAC (Wang et al., 2024)
124.57
38640.00
DisDiff (Liu et al., 2024a)
65.54
25960.50
SDS- (Xue al., 2023)
18.61
8278.63
Table 8. Comparing the robustness of methods against image
PID (Li al., 2024)
241.31
4581.93
preprocessing and adaptive attack on CelebA-HQQ
LE(ours) LA(ours)
7.34
4469.80
Method
IMS +
FDFR t Brisque t
LPIPS
8.47
4469.80
Clean
0.608
0.041
17.896
0.662
Crop
0.356
0.743
79.369
0.879
Gaussin Filter
0.369
0.697
79.607
0.913
Table Comparison of defense effectiveness with different itera-
JPEG Comp.
0.451
0.358
62.541
0.821
tion steps CelebA-HQ. "*" is default.
DuftPure
0.435
0.244
46.880
0.804
Step
IMS +
FDFR
Brisque t
LPIPS
GrIDPure
0.456
0.231
49.071
0.769
20
0.549
0.322
16.186
0.808
Fixed o
0.503
0.255
37.920
0.823
25
0.068
0.965
123.411
1,001
Clipped
0.512
0.205
39.468
0.795
30
0
1
155.845
1.021
50
0
1
155.692
1.019
75
0.022
0.993
58.244
0.933
100
0.019
0.997
33,288
0.912
1) = 0.05 as the default noise budget.
Image Preprocessing and Adaptive Attack: We evaluate the robustness of our methods against image preprocessing and adaptive attacks. For preprocessing, we test traditional methods (Cropping, Gaussian filtering, and JPEG compres- sion), and advanced purification techniques (DiftPure (Nie et al., 2022) and GrIDPure (Zhao et al., 2024)). For adap- tive attacks, we simulate scenarios where attackers fix or clip latent variance during fine-tuning bypass protection. Table 8 shows that methods still demonstrate notable robustness in these cases. They reduce face detection in generated images (higher FDFR) and lower identity-relatee similarity (lower IMS). Additionally, the higher Brisque and LPIPS scores indicate decline in image quality. Although purification and bypassing techniques affect performance, they cannot fully remove our perturbations.
Table Ablation study LE loss terms in CelebA-HQ.
Centropy
Liagrange
IMS
FDFR
Brisque
LPIPS f
X
0.61
0.04
17.89
0.66
V 0.68
0.03
17.62
0.72
X
0.69
0.02
23.82
0.71
0
1
155.81
1.02
Therefore, we terminate the optimization at 30 steps. The appendix C.1 provides detailed analysis and visualizations of this behavior.
Module of LE: We conduct an ablation experiment on the proposed LE method by testing its performance when the Lagrangian term the entropy term is removed. As shown in Table 6, using either loss term alone fails to achieve the ideal attack effectiveness. This result demonstrates that the LE loss function is not a simple combination of multiple loss terms but is carefully designed to ensure that the gradient direction aligns with the variance growth direction.
5. Conclusion
In this paper, we discuss the relationship between the vari- ance of VAE encoders and the effectiveness of adversarial attacks in disrupting generative diffusion models. We iden- tify variance as key to image distortion and propose two novel loss functions, LA and LE, overcome limitations of existing methods. These loss functions completely erase identity semantics, generating pure-noise images resistant to misuse. Our methods also excel in transferability and efficiency, requiring fewer resources and achieving faster protection, making them practical for rapidly evolving mod- els and resource-constraing devices.
Noise Budget 7): We evaluate the impact of the noise budget 77 the performance of LE the CelebA-HQ dataset. Ta- ble 7 shows that 8/255 is sufficient to fully erase identity semantics and produce low-quality images. Increasing 17 to 0.05 raises the Brisque score, suggesting further deviation from natural image characteristics, but without significant visual degradation of adversarial examples. Thus, we set
8

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Acknowledgments
Scaling rectified flow transformers for high-resolution image synthesis. In ICML,, 2024.
The authors would like to thank the support of Guangdong Provincial Key Laboratory of Information Security Tech- nology (Grant No. No..003311 12 12060002)) Guangdong Provin- cial Key Laboratory of Intelligent Information Processing and Shenzhen Key Laboratory of Media Security (Grant
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2022.
No.202331212000076)
Goodfellow, J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. ICLR, 2015.
Impact Statement
Our work aims to prevent unauthorized images from be- ing exploited by malicious users to generate fake images through personalized generative techniques. While existing approaches attempt to address this issue, they suffer from several limitations, including slow protection speed, high computational resource requirements, and poor transferabil- ity. These drawbacks make them impractical for real-world applications where models undergo rapid iterations or oper- ate in resource-constrainee environments. Moreover, most existing methods fail to provide robust identity protection, as they still produce images that retain identity semantics, leaving them vulnerable to misuse. In contrast, our methods effectively remove identity semantics by generating pure noise images, ensuring strong protection against unautho- rized identity synthesis while maintaining computational ef- ficiency and transferability across different generative mod- approaches also
Higgins, E. Making pictures of Trump get- ting arrested while waiting for Trump's arrest. attps:/ttwiteer.cmm/fL..... statuss16377276111233888 2023-03-21.
2023, Accessed:
Ho, J., Jain, A., and Abbeel, P. Denoising diffusiio proba- bilistic models. In NeurlPS, pp. 1840-6851, 2020.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR, 2021.
Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
els. Beyond identity protection, applicable to copyright protection in the artistic domain, increasingly critical issue given the rapid advancement of personalized generative techniques. As Al models become more adept at learning and replicating artistic styles, the risk of unauthorized reproduction and commercialization of artworks grows. Our methods provide proactive defense against such risks by preventing Al models from adapting to protected artistic content. By offering robust defense mechanism against the unauthorized use of both identity and artistic content, our work contributes to the development of responsible AI, ensuring ethical and secure deployment of generative models.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, Multi-concept customization of text-to-image diffusion. In CVPR, pp. 1931-1941, 2023.
Labs, B. E. Flux. https: /gtt/gittt/tt github.com.. black forest- -aabsaaabsssus 2023.
Li, A., Mo, Y. Li, M., and Wang, Y. PID: Prompt- independent data protection against latent diffusion mod- els. In ICML, 2024.
Liang. C. and Wu, X. Mist: Towards improved adver- sarial examples for diffusion models. arxiv preprint areXiv:2305.12683 2023.
References
Liang. C., Wu, X., Hua, Y., Zhang, J., Xue, Y.. Song, T., Xue, Z., Ma, R., and Guan, H. Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples. In ICML, pp. 20763-20786, 2023,
Cao, Q., Shen, I.,, Xie, W., Parkhi, O. M., and Zisserman, A. VGGFace2: dataset for recognising faces across pose and age. In FG, pp. 67-74, 2018.
Liu, Y., An. J., Zhang, W., Wu, D., Gu, J., Lin, Z., and Wang, W. Disrupting Diffusion: Token-level attention erasure attack against diffusion-based customization. In ACM MM, 2024a.
Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In SP, pp. 39 57, 2017.
Deng, J., Guo, J., Ververas, E., Kotsia, I., and Zafeiriou, S. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR, pp. 5203-5212, 2020.
Liu, Y., Fan, C., Dai, Y., Chen, X., Zhou, P., and Sun, L. MetaCloak: Preventing unauthorized subject-driven text- to- to-image diffusion based synthesis via Meta-learning. In CVPR, pp. 24219 2422222 2024b.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Lo, L., Yeo, C. Y., Shuai, H.-H., and Cheng, W.-H. Distrac- tion is all you need: Memory-efficient image immuniza- tion against diffusion-based image editing. In CVPR, pp. 24462 24471, 2024.
Wang. E,, Tan, Z., Wei, T., Yue, Y., and Huang, Q. SimAC: A simple anti-customization method for protecting face privacy against text-to-image synthesis of diffusion mod- els. In CVPR, pp. 12047-12056, 2024.
Madry. A., Makelov, A.. Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
Xu, J., Lu, Y., Li, Y., Lu, S., Wang, D., and Wei, X. Perturb- ing attention gives you imaging perturbations that efficiently fool customized diffusion models. In CVPR, pp. 25534-2454,, 2024.
bang for the buck: Subtle
Nesterov, Y. Introductory lectures on convex optimization: basic course, volume 87. Springer Science & Business Media, 2013.
Xue, H., Liang, C., Wu, X., and Chen, Toward effective protection against diffusion-basel mimicry through score distillation. In ICLR, 2023.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabiliisicc models. In ICML, pp. 8162-8171, 2021.
Ye, H., Zhang. J., Liu, S., Han, X., and Yang. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arxiv:2308.00 06721, 2023,
Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. Diffusion models for adversarial purifi- cation. In ICML, pp. 16805-16827, 2022.
Zhang. L., Rao, A., and Agrawala, M. Adding conditional control text-to-image diffusion models. ICCV, pp. 3836-3844 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In CVPR, pp. 4195-4205, 2023.
Zhao, Z., Duan, J., Xu, K., Wang, C., Zhang, R., Du, Z., Guo, Q. and Hu, X. Can protective perturbation safe- guard personal data from being exploited by stable diffu- sion? CVPR, pp. 24398 24407, 2024.
Podell, D., English, Z.,, Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Im- proving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. DreamBooth: Fine tuning text-to-image
diffusion models for subject-driven generation. CVPR, pp. 22500-22510, 2023.
Saleh, B. and Elgammal, A. Large-scale classification of fine-art paintings: Learning the right metric the right feature. arXiv preprint
2015.
Salman, H., Khaddaj, A., Leclere, G., Ilyas, A., and Madry, A. Raising the cost malicious Al-powered image edit- ing. In ICML, pp. 29894-29918, 2023.
Shan, S., Cryan, J., Wenger, E., Zheng, H., Hanocka, R., and Zhao, B. Y. Glaze: Protecting artists from style mimicry by text-to-image models. In USENIX Security, pp. 2187-2204, 2023.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
Van Le, T., Phung, H., Nguyen, T. H., Dao, Q., Tran, N. N., and Tran, A. Anti-DheamBooth: Protecting users from personalized text-to-image synthesis. In ICCV, pp. 2116 2127, 2023.
10

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
A. Deep Analysis of Our Work
A.1. Rationale for Neglecting Higher-Order Terms under Small Perturbations
In adversarial scenarios, we often examine how small perturbation E I" propagates through a deep network (c.g., the VAE encoder in Stable Diffusion) and affects the final loss. A common strategy linearize the network around the clean input x, effectively discarding higher-order (nonlinear) terms. In the following, we provide step-by-step justification for ignoring these higher-order corrections when ||ö]| S is sufficiently small.
Second-Order Taylor Expansion. Let f R + Ih denote the VAE encoder, mapping an input x to a latent representation f(x). A second-order Taylor expansion around x yields:
f(x + f(x)+ (z(x) 2 Hy 06)8, 8, O50Ss,
(19)
where (x) (x) is the Jacobian matrix, and (-))) is the Hessian. The remainder term 1 ()))) 8 represents second-order (and beyond) corrections.
3-Smoothness Assumption. A standard assumption in optimization is that is 3-smooth (or has Lipschitz continuous gradients) within local neighborhood of (Nesterov, 2013). Concretely,
[V] ff(o)]l < 3 ||u - vll, Yu,uEN(u
(20)
Equivalently, the Hessian operator norm satisfies remainder in (19) by
[=)]| S B. Under this assumption, we can bound the second-order
11 5 H x 08) .||
s 8 2 |8112.
(21)
Hence, if ||0|| S is small, the contribution from higher-order terms scales as 0(2), which is typically negligible compared the first-order term (ee).
Influence on the Loss Gradient.
adversarial settings, the primary concern is often V. V L(f(( +))) By the chain rule,
vaf(x+
(22)
With the 3-smooth assumption, show the discrepancy between the exact gradient and its first-order approximation is bounded by O((1))) (Goodfellow et al., 2015). Therefore, in sufficiently small neighborhood, ignoring higher-order terms does not significantly alter the gradient direction that guides adversarial perturbations.
Consistency with Iterative Attacks. Projected Gradient Descent (PGD) (Madry et al., 2018) updates 8 in small increments,
+ ( + a (oL[f(x+8)] +8))
(23)
ensuring that the input + 8 remains within the local region | S Such small-step iterative methods naturally align with a local linear approximation, as they do not abruptly jump outside the domain where 3-smoothness is valid. Hence, the ignored second-order terms remain bounded throughout the attack process.
Empirical Observations. Many adversarial methods, including PGD (Madry et al., 2018) and C&W (Carlini & Wagner, 2017), rely almost exclusively on first-order information yet demonstrate highly effective attacks in small-|jo|| regimes. Empirical comparisons further suggest that the angle between the exact gradient and the first-order approximation remains small (Goodfellow et al., 2015), reinforcing the idea that higher-order terms can be safely neglected for small perturbations.
Conclusion. Under the 3-smoothmess hypothesis and for perturbations constrained by || < e, the second-order remainder in (19) contributes only Oce²), hence can be dismissed relative to the Occ) first-order effect. Consequently, linearizing the VAE encoder (or other neural components) is both a common heuristic and a theoretically supported practice in adversarial analysis.

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
A.2. Jacobian Computation: Practical Challenges and Mitigation
In our theoretical derivations (see Eqs.8-10), we decompose how a perturbation å propagates through the VAE encoder using Jacobian matrices. However, explicitly computing these Jacobians in practice can face several challenges:
High Dimensionality and Computation Costs. For inputs of size H x W x 3 (e.g., color images) and a nigh-dimensional latent space in the VAE, the Jacobian matrix can be extremely large. Storing or multiplying such matrices directly would require substantial memory and computation. When either the input resolution or the latent-space dimension grows, the overall Jacobian can easily scale to millions of parameters.
No Need for Exact Elementwise Computation. In our formulation, the purpose of these Jacobians is primarily theoretical to show how a perturbation flows through successive layers and eventually affects the variance term o² and the final loss. We do not advocate explicit elementwise computation or storage of such high-dimensional matrices actual implementations. Instead, we mainly rely on the chain structure of the Jacobian product to reveal how each layer's local derivatives combine. In practice, automatic differentiation typically yields the end-to-end gradient without needing to expand the intermediate Jacobians. Moreover, demonstrated in Table 4, our method only requires approximately 4.5 GB of GPU memory on an NVIDIA A100 to protect single 512x512 image in about 8 seconds, underscoring that we do not need to hold the entire Jacobian in memory to achieve efficient and feasible runtime.
Gradient Stability (See Section C.1). Deep networks are known to exhibit exploding or vanishing gradients, especially when certain activation functions saturate or become inactive. Although this phenomenon is not strictly caused by explicitly handling the Jacobian, it can affect the numerical stability of Jacobian- based products. Moreover, gradient masking can sometimes occur when the network's gradient pathways are obstructed or improperly managed (e.g., due to overly aggressive regularization or non-differentiable structures). To mitigate these issues, we employ an early-stopping strategy during the optimization steps (see Section C.I). This early-stopping procedure helps avoid practical pitfalls such exploding or vanishing gradients and reduces the risk of artificially masking the true gradient signals.
Summary. In short, the Jacobians in Eqs. 8-10 primarily serve an analytical role: they illustrate how perturbations impact each stage of the encoder in chain-of-influence manner. These do not imply that one must explicitly compute store the entire Jacobian in a real-world scenario. Our experimental results indicate that our method requires only about 4.5 GB of GPU memory on an NVIDIA A100 to protect a single 512 image in roughly 8 seconds, underscoring that it is unnecessary to hold the full Jacobian in memory to achieve efficient and feasible runtime performance.
A.3. Illustrative Example for Gradient Sign Reversal
Before we delve into how the Laplace Approximation (LA) maintains a consistent variance- variance-growth direction, let us first consider a concrete numerical example to illustrate how the gradient direction (as computed via Eq. 1 can be influenced by the loss derivatives d0g exceed the clean variance a² Jelan consider a simple three-dimensional scenario:
in MSE, PID, LA and LE. To illustrate how each loss behaves when certain variances already
o² = [ot o2. a
[0.001, 0.001, 0.006], 2e < {0.001, 0.006).
(24)
Furthermore, let the Jacobian for single perturbation dimension 6 be
0a² Mij - [1, 1, -1].
(25)
06,
Note that o3 is notably larger than deaa and also has a negative sensitivity -1. The emergence of such an extreme variance component like oj 0.006 can be attributed to the influence of the initial perturbation. Initially, the variance matrix is uniformly distributed. However, the perturbation disrupts this balance, causing the variance distribution to become uneven.
(1) MSE Loss.
We define
ÒLMSE 2(03 - do?
MSE
Σ
E03
02 eeean)
(26)
12

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Substituting of 0.001, oj 0.001, o3 0.006 and assuming Le much smaller (effectively zero compared to these values), yields partial derivatives 2 x 0.001, 0.001, 0.006 =0.002, 0.002, 0.012. Hence,
OLMSE 06;
[2(0] cleen)) (M.g) 0.002 0.002 1 + 0.012 (-1) -0.008, Σ
(27)
which is negative. Although each oz term is larger than 2aran the third component's combination of larger partial derivative (0.012) and negative Jacobian factor (-1) dominates, reversing the overall gradient sign.
Therefore, MSE prioritizes optimizing large variance components (e.g.. 0.006) while neglecting smaller variance components (e.g., 0.001). As a result, the gradient sign of MSE is often misaligned with the direction of increasing variance Mij. This misalignment causes MSE fall into a local optimization trap, where fails effectively increase either its targeted large variance components or the smaller variance components that require growth.
(2) PID Loss.
Define
dLPID doz
2 (log oî -log odlean) = of
CPID =
Σ (log o3 - llog o detean)
(28)
For o3 0.006 > clean the term log(oz) - log(o is positive, ensuring that @Crrp > 0. Thus, PID maintains a 80g
positive contribution to 86;
OC unless the negative Jacobian factor is extremely large. This means that PID correctly prioritizes
small variance components during the initial optimization, where gradients are larger for smaller variances due to the 1/03 factor.
However, as small variance components are optimized and their values grow, the variance matrix becomes more uniform. This uniformity reduces the optimization space, as the differences between variance components shrink. Additionally, the PID gradient 0Crn weakens over time because the dj term diminishes as oî increases. Consequently, PID requires significantly more steps achieve sufficiently large variance. This slow convergence is reflected in its original work, where PID requires 900 steps remove most identity semantics, compared the 30-50 steps required by our proposed methods.
dof
(3) LA (Laplace Approx.) Loss.
The Laplace Approximation (LA) Loss is defined as:
022
ml
QLLA 0o²
LLA
(29)
b'
where μ is typically set to 0, and b is scaling factor, typically set to 1. Importantly, the gradient with respect to of is constant and independent of the variance values, meaning the optimization direction is determined solely by the mapping relationship Mi.-
For the example where Mij = [1,, -1], the contribution to 86, becomes:
0LLA 08,
Σ
OLLA doz
Mij
1 M.J
1
Ma
1 (1+1- b'
(30)
8Cz4 is guaranteed to remain positive since > 0. Unlike MSE PID, LA loss maintains consistent alignment Here, 88, with the variance growth direction, as the sign of the gradient depends only on the structure of Mij. and there are no cancellations caused by varying magnitudes of o3 or ac
de
Moreover, this constant gradient ensures simultancous and balanced growth of variance across all dimensions. Unlike PID, which prioritizes small variances but slows down as variance grows, MSE, which prioritizes large variances but may reverse gradient direction, LA Loss provides consistent updates in the direction of increasing variance for all components. This makes LA Loss robust and efficient, avoiding the pitfalls of gradient misalignment or diminishing optimization space.
In addition to its consistent direction, LA Loss maintains large optimization space throughout the process. The constant @LIA ensures equal optimization for all variance components, regardless of their initial magnitudes. Unlike gradient 00
13

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
PID, where the gradient diminishes as variance grows, LA Loss sustains fixed optimization strength across both small and large variances, preventing stagnation bias toward specific components. This balanced optimization accelerates variance growth by enabling simultancous updates across all components. As result, LA Loss rapidly amplifies variance and removes identity semantics within 30-50 steps, significantly outperforming PID's 900-step optimization.
(4) LE Loss
Recall that our Lagrange Entropy (LE) Loss is given by
CLE(O?) = Σ } -f log(o)) + 2 >E
(31)
When c c= 1,
fI.x(09)= = 03 (oooo))=-oooos(((( + X E of (Σ -1)
(32)
whose partial derivative reads
dLLE
(log(o)) +1] [log(of)+1] 2X Σ o -1)
(33)
003
We can interpret the first part -art-(lon (log(o3) + as entropy-driven term acting on each dimension, while 2> of acts global Lagrange penalty controlling the sum of variances.
1)
Note on logarithms. Throughout this paper (and in our code), log(-) refers to the natural logarithm In(-).
Case 1: When o < 1, the entropy term dominates (positive) while the Lagrange penalty is negative. If o < 1, then log(o)) < so - [log(o]) + 1] is significantly positive quantity. For example, if 03 0.01, log(0.01) 4.605 and -llog(0.01)))) +1] x +3.605, giving strong boost to that dimension. Meanwhile, because Σ, o3 - < 0, the Lagrange term 2X(3, of - 1) is negative, partially offsetting the entropy push. However, while the total variance is still well below most dimensions have of < 1, multiple positive entropy terms outweigh this negative penalty overall. Thus, the net gradient remains positive, continuing to increase of without flipping sign or stalling.
Case 2: When Σ, 03 > 1, the Lagrange penalty dominates (positive) while the entropy term can be negative. For dimensions whose of > 1, log(()) + is positive, so - [[og((f] (a}) + 1] becomes negative, preventing any single dimension from growing unboundedly. On the other hand, if oz large positive penalty can override the negative entropy factor. For example. if log(o)) ) + +1= = 1.693 (for oī = 2), then the negative part is - 1.693, but if 2 of 1) +3, the sum is still +1.307, ensuring net positive gradient. As result, even large variances do not cause the update direction to flip to negative. The global Lagrange term ensures a consistent push if the total variance is meant to exceed I.
> 0, 2X(3, oj - 1) is a positive addition. A sufficiently
Implications for Optimization Space and Speed.
No severe compression: A common pitfall in PID is that @Cecz x 1. diminishes severely for large a3. slowing 80 0f
down further variance increases. In contrast, LE's entropy part -02 2 varies only logarithmically with o². As of grows from 0.1 to 1 or even 5, log(oz) changes moderately, so (log(a)) collapse near-zero magnitude. Furthermore, the Lagrange term XE of 1)2 adds a finite amount, but does not drastically shrink the net gradient. Thus, LE avoids the late-stage gradient exhaustion seen in PID and also avoids MSE's extreme sign flips caused by large outlier variances.
) + +1] remains in a range that does not
dcj Minimal risk of sign inversion: MSE can create large partial derivatives that, if multiplied by negative a6 flip the total sign. In LE Loss, the interplay of negative entropy term (for big of) and a positive Lagrange term often cancels out in favor of mild positive net gradient.
LA-like speed: LA uses a constant gradient independent of o3. quickly boosting variances in 30-50 steps. LE, with the two-phase dominance (entropy vs. Lagrange), similarly avoids any late-stage slowdown or early-stage stalling. Thus, it can achieve variance expansion nearly as swiftly as LA.
14

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Conclusion. In this biree-dimensiona example. MSE yield negative overall gradient (despite of > o deean due the mismatch with the negative Jacobian in the third dimension. PID does avoid sign flips but loses momentum for larger variances, thus requiring many iterations to further increase o². LA, by contrast, keeps constant gradient that remains aligned with the "inerease-variance direction at every step. Meanwhile, LE combines the advantages of LA's stable gradient with an additional entropy and Lagrange-based balancing. It achieves near-constant or steadily positive updates while regulating extreme growth in any single variance dimension. Hence, LE can match LA's rapid speed of variance expansion and avoid both MSE's sign flips and PID's late-stage slowdown.
A.4. Why Only Attack Variance Rather Than Mean?
PID (Li et al., 2024)) observes that attacking the mean if of the VAE encoder primarily introduces noisy artifacts or textures into the generated images, Although these artifacts become more pronounced with additional attack steps, they do not effectively remove identity or semantic information. In contrast, attacking the variance o² directly impacts the diffusion model's ability extract semantic concepts. As documented by PID,, increasing the variance optimization steps (from 0 to 300, 600, even 900 steps) gradually erase the identity semantics in the generated image until it is fully obfuscated.
0C and ignore the mean. The mean Hence, to remove high-level semantics (e.g., identity), it is sufficient to focus on 00 perturbation adds little to semantic destruction besides additional noise-like artifacts. Therefore, in our chain-rule analysis, we primarily consider how o² evolves under adversarial perturbations 6, rather than factoring the mean term.
Table Comparing the performance of method with baselines against DreamBooth (Ruiz al., 2023) CelebA-HQ and VGGFace2. The best result under each metric marked with bold. The prompt used is "a dslr portrait of sks person."
CelebA -HQ
VGGFace2
Method
ISM 4
FDFR
Brisque
LPIPS
ISM
FDFR
Brisque +
LPIPS
No Defense
0.419
0.085
7.506
0.748
0.459
0.048
9.496
0.769
AdvDM (Liang al., 2023)
0.319
0.067
11.461
0.796
0.263
0.139
15.288
0.821
ASPL (Van Le et al., 2023)
0.316
0.064
11.381
0.802
0.289
0.113
14.195
0.819
Mist (Liang & Wu, 2023)
0,071
0.095
13.338
0.864
0.153
0.078
17.497
0.857
MetaCloak (Liu et al., 2024b)
0.358
0.117
25.51
0.781
0.377
0.062
20.102
0.803
SimAC (Wang et al., 2024)
0.328
0.298
18.143
0.776
0.384
0,263
19.618
0.787
DisDiff (Liu et al., 2024a)
0.396
0.049
48.672
0.749
0.376
0.254
17.863
0.821
SDS- (Liu al., 2024a)
0.378
0.028
9.435
0.805
0.414
0.033
37.508
0.825
PID (Li al., 2024)
0.077
0.631
35.317
0.899
0.077
0.791
48.318
0.928
LE(ours) LA(ours)
0.031
0.918
103.004
0.968
0.044
0.849
91.221
0.967
0.034
0.827
96.181
0.959
0.056
0.867
86.236
0.974
Table 10. Comparison of our method with other approaches against Textual Inversion and IP-Atttter on CelehA-HQ. The best-performing defense under each metric marked with bold.
Textual Inversion
IP-Adapter
Method
ISM $
FDFR
Brisque
LPIPS t
ISM
FDFR
Brisque 1
LPIPS f
No Defense
0.567
0.014
21.432
0.512
0.341
0.025
18.436
0.689
AdvDM (Liang al., 2023)
0.292
0.358
43.309
0.818
0.319
0.021
40.057
0.768
ASPL (Van Le et al., 2023)
0.332
0.268
40.235
0.812
0.328
0.017
40.941
0.768
Mist (Liang & Wu, 2023)
0.106
0.795
28.514
0.743
0.311
0.026
32.229
0.834
MetaCloak (Liu et al., 2024b)
0.231
0.221
42.588
0.755
0.305
0.028
37.003
0.785
SimAC (Wang et al., 2024)
0.122
0.792
67.545
0,843
0.317
0.028
41.338
0.753
DisDiff (Liu et al., 2024a)
0.345
0.251
35.038
0.770
0.319
0.018
21.866
0.747
SDS- (Xue et al., 2023)
0.452
0.057
44.240
0.764
0.296
0.022
29.968
0.806
PID (Li al., 2024)
0.054
0.938
82.314
0.926
0.296
0.018
21.568
0.819
LE(ours)
0.017
0.988
89.537
0.958
0.078
0.248
47.937
0.875
LA(ours)
0.014
0.958
87.349
0.943
0.091
0.232
45.796
0.893
15

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
B. More Quantitative Results
B.1. Quantitative Results for mismatch prompt against DreamBooth
We simulate a scenario where a user, after training a DreamBooth model, employs an inference prompt "a dsir portrait of sks person," " which differs from the training prompt "a photo of sks person." The specific results are presented in Table and Figure 6. We observe that when the inference prompt does match the training prompt, the identity similarity metric (ISM) of the baselines decreases. Meanwhile, the noise in the generated images is significantly reduced, reflected in the Brisque and LPIPS scores of the baselines, which become similar to those of the No Defense method. In contrast, our approaches consistently produce pure noise images, demonstrating greater robustness.
B.2. Quantitative Results for Textual Inversion and IP-Adapter
We report the results of using Textual Inversion (Gal al., 2022) and IP-Adapter (Ye et al., 2023) as personalization algorithms on the CelebA-HQ dataset in Table 10. Our methods outperform other methods in resisting Textual Inversion and IP-Aeereer. For Textual Inversion, we consistently maintain extremely low ISM scores and high FDFR scores, demonstrating that the generated images contain almost no recognizable facial semantics. When applied to Adapterr only our methods achieve an ISM score below 0.1 and an FDFR score above 0.2. These results confirm that our methods are well-suited to resist various mainstream personalization generation algorithms.
C. More Ablation Experiments
C.1. Attack step
Gradient Explosion and Unstable Oscillations. We conduct an ablation study on the number of optimization steps for LE and LA and observe a notable phenomenon in Figure 4(a): &CLe experiences a gradient explosion after 30 steps, d8
whereas aCis begins oscillate unstably around 50 steps. We hypothesize that these instabilities arise from the product of the two Jacobians (Jy mapping 5 to hidden feature & and Jy mapping ø to g²), amplified by the nonlincarities in the network. Once certain activation functions saturate or shift during high-dimensional updates, the gradients can grow or fluctuate dramatically as the number of steps increases. Figure 4(b) illustrates the generated images at various optimization steps. Between 30 and 75 steps, both LE and LA produce nearly pure noise. When the step count exceeds 100, the latent distribution drifts further, resulting in chaotic "scene-like" patterns. As reported in Table 5, while such images exhibit degraded metrics they nonetheless fully remove identity semantics, thus providing effective protection.
C.2. Uncontrolled Settings:
Our methods introduce carefully crafted perturbations to images to prevent diffusion models from learning their features, and these perturbations are applied all images in our experiments. In this section, we assume that malicious users can access some clean images and mix them with the protected ones as the training set. Table 11 demonstrates that even under different protection ratios, our methods remain effective.
Table Comparison between perturbed images and clean images at different ratios on CelebA-HQ.
Perturbed
Clean
IMS |
FDFR
Brisque
LPIPS 1
0
4
0.608
0.041
17.896
0.662
1
3
0.536
0.148
26.449
0.771
2
2
0.481
0.191
44.075
0.821
3
1
0.227
0.916
123.387
0.972
0
1
155.804
1.021
C.3. On WikiArt
We further evaluate the effectiveness of our methods in protecting artistic works. We select five different styles from the WikiArt dataset (Saleh& Elgammal, 2015): Baroque, Cubism, Expressionism, Fauvism, and Romanticism. For each style, we use four paintings as the training set for DreamBooth, with the training prompt set to "a photo of sks picture". As shown Figure 13, both our LA and LE methods successfully generate pure noise images, effectively preventing the imitation of
16

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
L
08
Delta Gradient Trends
1e12
Delta Gradient Trends
Delta Gradient Mean
Delta Gradient Mean
Dcta Gradient Moggttui
Delta Cradient Ragnitude
Steps
Steps
(a) Trend of for LE and LA
Step 20
Step 25
Step 30
Step 50
Step 75
Step 100
LE
LA-
Visualization generated images under different optimization steps for LE Loss.
Figure 4. (a) Trend of 2C
35 for LE and LA as optimization steps increase. The gradient of LE explodes after step 30, while the gradient of LA becomes unstable after step 50. (b) Adversarial effects of LE and LA different steps. Since both LE and LA generate pure noise images by step 30, we apply an early-stopping strategy to prevent gradient issues and reduce unnecessary computational overhead.
17

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
artistic styles. This demonstrates that our approach be extended beyond identity protection and applied to safeguarding artistic creations from unauthorized style replication.
D. More Visualization
D.1. Visualization of Transferability
We present visualizations in Figures 7-10, demonstrating the transferability of adversarial examples generated by our methods and baselines on Stable Diffusion v1.5 to other models. Specifically, we fine-tune LoRA using these adversarial examples as the training set on Stable Diffusion 2.1 (SD2.1), SDXL (Podell al., 2024), Stable Diffusion 3.5 Medium (SD3.5) (Esser al., 2024), and FLUX.. 1-dev (Labs, 2023), and analyze the resulting images. As shown in Figure since SD2.1 shares nearly identical architecture with SD1.5, both our methods and baselines maintain strong protective effects. The baselines introduce distortions and partially remove semantic content, while our method completely eliminates identity semantics, generating pure noise images. However, the protection effectiveness decreases with SDXL, SD3.5, and FLUXX. 1ede due to significant architectural differences from SD1.5.
Architectural Differences:
SDXL vs. SD1.5: Unlike SD1.5, which uses single UNet, SDXL adopts a two-stage UNet architecture with base UNet followed by refiner UNet for improved image quality. Additionally, SDXL employs larger text encoder (CLIP ViT-L/14) and improved conditioning mechanisms, making it more resistant to perturbations designed for SD1.5.
SD3.5 and FLUX.1-dev: These models replace the traditional UNet with more advanced Multi-Modaa Diffusion Transformer (MM-DiT) architecture, which significantly alters the way latent representations are processed. MM-DiT improves efficiency, handles multimodal inputs, and leverages self-attention across broader spatial contexts, reducing the impact of attacks targeting UNet structures.
Due to these changes, methods that rely UNet gradient information (e.g., AdvDM (Liang al., 2023), ASPL (Van Le et al., 2023), MetaCloak (Liu et al., 2024b), SimAC (Wang et al., 2024), SDS- (Xue et al., 2023)) cross-attention manipulation (e.g., DisDiff (Liu et al., 2024a)) largely fail to maintain protection, with even noticeable noise artifacts disappearing. On the other hand, Mist (Liang & Wu, 2023) and PID (Li et al., 2024), which attack the VAE module, retain some protective effects by introducing visible noise distortions. However, the generated images still preserve identity semantics, limiting their effectiveness.
In contrast, our methods continue eliminate most identity semantics, demonstrating superior transferability and practical applicability across different model architectures.
D.2. Against CaatrollNet-bassee Image Edit
We evaluate our methods and baselines in resisting ControlNet-based (Zhang et al., 2023) image editing. Specifically, we used ControlNet v1.1 with Depth Map, SoftEdge, OpenPose, Normal Map, and Segmentation conditional guidance for image editing. The visualization results presented in Figure 12. As observed, baselines continue to add varying levels of noise to the generated images, whereas our method completely removes identity semantics, producing pure noise images that cannot be used for unauthorized purposes.
18

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
AdoDM
ASPL
MetaClouk
SimAC
DisDif
PID
LEfomrs)
LA(ours)
Figure 5, More visualization of our methods and baselines against DrothBooth on CelebA HQ and VGGFace2. The odd numberebd rows show reference images, while the even numbered rows display generated images. The inference prompt used "a photo of sks person." " All baselines add different levels of noise to the generated images, while only our methods completely remove identity semantics and generate pure noise images.
19

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
AdeDM
ASPL
MetaClouk
SimAC
DisDif
PID
LEfours)
LA(ours)
Figure 6. More visualization of our methods and haselines against DreamBooth on CelebA-HQ and VGGFace2. The add-numhered rows show reference images, while the even- numbered rows display generated images. The inference prompt used is "a dlsr portrait of sks person." " The performance of baselines is affected by the mismatch between the inference prompt and the training prompt, while our methods still completely remove identity semantics and generate pure noise images. viewd zoom.
20

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
AdeDM
ASPL
MetaCCookk
SunAC
DuDidl'
PID
LBfous)
LAfours)
NE
H
EIL
Figure 7. More visualization of our methods and baselines against LoRA on CelebA-HQ. The odd-nummered rows show reference images, while the even-numbered rows display generated images. Adversarial examples generated on SD1.5 are transferred to SD2 the training set. The noise originally added by the baselines is largely removed, indicating poor transferability. In contrast, our method still generates images are unrelated identity semanties, demonstrating better transferability. vievd zoom.
21

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
AdvDM
ASPL
MotaCliokk
StnAC
DuDDil'
SDS-
PTD
Lifoar))
LAfous)
3
NET
H
Figure 8. More visualization of our methods and baselines against LoRA CelebA-HQ. The odd-number ned-numbreed rows show reference images, while nembbrdeee rows display generated images. Adversarial examples generated on SDI.5 are transferred to SDXL as the training set. The noise originally added by the baselines is largely removed, indicating poor transfcrability. In contrast, our method still generates images that are unrelated to identity semantics, demonstrating better transferability. Best viewd zoom.
22

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
AdvDM
ASPL
MotaCliokk
StnAC
DuDDill
SDS-
PDD
Lifoar))
LAfous)
Hf
M A
08
Figure 9. More visualization of our methods and baselines against LoRA CelebA-HQ. The odd-numberge rows show reference images, while the nen-nuumereed rows display generated images. Adversarial examples generated on SD1.5 transferred to SD3.5 Medium as the training set. The noise originally added by the baselines is largely removed, indicating poor transferability. In contrast, our method generates images that are unrelated to identity semantics, demonstrating better transferability. Best viewd zoom.
23

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
AdvDM
ASPL
MetaCCookk
SunAC
DuDiil'
PID
LEfous)
LAfours)
EET
Figure 10. More visualization of our methods and baselines against LoRA on CelebA-HQ. The odd-numbered rows show reference images, while the even nen-nrubeeeed rows display generated images. Adversarial examples generated on SD1.5 are transferred FLUX.I-dev as the training set. The noise originally added by the baselines is largely removed, indicating poor transfcrabilitty In contrast, our method still generates images that are unrelated to identity semantics, demonstrating better transferability. Best viewd zoom.
24

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Ad-DM
ASPL
MetaClonk
SimAAC
DisDiff
SDS.
PID
L
L-oois
NE
H
Figure More visualization of our methods and baselines against Textual Inversion on CelebA-HQ. The odd-numbered rows show reference images, while the even-numbered rows display generated images. All baselines add different levels of noise to the generated images, while only our methods completely remove identity semanties and generate pure noise images.
25

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
AdvDM
ASPL
Mit
MetaCloak
SimAC
DuDiit'
SDS-
PID
LEuuus
LA(ou))
Relerence
Imsgo
Depth
Genentad
Imago
SofFddge
Geuerated
Image
Openpooe
Generated
Image
Normal Map
Genentted
Iniage
Segmentation
Genvratedd
Image
Figure Visualization our methods baselines against ControlNet-based Image Editing (Zhang al., 2023) on CelebA-HO.
We evaluate five image editing applications: Depth Map, SoftEdge, Openpose, Normal Map. and Segmentation. The prompt used is "a man". baselines different levels noise the generated images, while only our method completely removes identity semantics and generates noise images.
26

---

Variance Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
Clean Image
Clean Generated
LE Generated
LA Generated
Baroque
Cubism
Expression
Fauvism
Romanticism
Figure 13, Visualization of our methods and baselines against DreamBooth on WikiArt. We experiment with five different artistic styles: Baroque, Cubism, Expressionism, Fauvism, and Romanticism. Our method consistently generates pure noise images, effectively protecting artistic styles from unauthorized imitation.
27


Reviews:
[{"review_id": "9dvcS9XPFm", "reviewer": "Reviewer_WD7W", "summary": "The paper introduces LA and LE loss functions to enhance semantic erasure in customized diffusion models, addressing privacy concerns by completely removing identity-related features. It identifies variance in VAE latent codes as key to image distortion and uses optimized variance expansion for effective erasure. Experiments on CelebA-HQ and VGGFace2 show state-of-the-art identity removal, 30x faster optimization, and strong transferability across diffusion models. The method is computationally efficient.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Advantages of Variance-based Attack and Better Transferability**\n\n **1.Model Architecture**\n\nEarlier diffusion models (e.g., SD1.5, SD2.1) use U-Net backbones, enabling effective attacks based on U-Net gradients or cross-attention. However, newer models like SD3.5 and FLUX.1 adopt Transformer-based backbones such as MM-DiT, which differ greatly in gradient behaviors. As a result, prior gradient-based attacks often fail. Despite this, all Latent Diffusion Models share a common pipeline: a VAE encodes the input image to a latent space, followed by diffusion. Our method perturbs the input image to corrupt the VAE output, an essential bottleneck, ensuring transferability across different architectures.\n\n **2.Attack VAE Mean $\\mu$**\n\nVAE applies the reparameterization trick to make sampling differentiable:\n\n$$\nz=\\mu(x)+\\sqrt{\\sigma^2(x)}\\odot \\epsilon,\\epsilon \\sim \\mathcal{N}(0,I).\n$$\nPerturbing the mean yields:\n\n$$\nz'=\\mu(x)+\\delta_\\mu+\\sqrt{\\sigma^2(x)}\\odot\\epsilon, \\epsilon \\sim \\mathcal{N}(0,I).\n$$\nThe corresponding probability density function (PDF) becomes:\n\n$$\np(z')=\\frac{1}{\\sqrt{2\\pi \\sigma^2(x)}}exp(-\\frac{(z'-(\\mu(x)+\\delta_\\mu))^2}{2\\sigma^2(x)}).\n$$\nAttacking the mean shifts the center of the latent distribution without altering the shape of its PDF. Since the variance remains unchanged, the distribution stays compact. Due to the translation-invariance of convolutional backbones, the semantic structure is partly preserved. Visually, this effect is mostly limited to local texture distortion, appearing as noise artifacts rather than semantic erasure.\n\n**3.Attack VAE Variance $\\sigma^2$:**\n\nPerturbing the variance gives:\n$$\nz=\\mu(x)+\\sqrt{\\sigma^2(x)+\\delta_{\\sigma^2}}\\odot \\epsilon,\\epsilon \\sim \\mathcal{N}(0,I)\n$$\nThe PDF becomes:\n$$\np(z')=\\frac{1}{\\sqrt{2\\pi(\\sigma^2(x)+\\delta_{\\sigma^2}})}exp(-\\frac{(z'-\\mu(x)^2)}{2(\\sigma^2(x)+\\delta_\\sigma^2)}).\n$$\nHere, **$\\mu(x)$ represents the mean of the original latent distribution**, which corresponds to the dense, semantically meaningful region learned during training.\n\nAs variance grows large, the 1D Gaussian distribution of each latent dimension flattens into a wide and low curve that is nearly parallel to the x-axis. This flattening means that the probability density near the original  mean $\\mu (x)$ drops quickly, while the overall sampling interval expands considerably. As a result, the latent samples are no longer concentrated near the dense semantic center of the distribution but become spread out over a vast latent space.\n\nFrom a high-dimensional perspective, this effect is amplified. Given a $d$-dimensional latent space, the expected squared distance between the latent samples $z'$ and mean  $\\mu (x)$ is:\n$$\n\\mathbb{E}[\\|| z' - \\mu(x) \\||^2] = \\sum_{i=1}^d \\left( \\sigma_i^2(x) + \\delta_{\\sigma^2,i} \\right)\n$$\nAs variance grows, latent samples $z'$ are pushed away from the high-density region centered around $\\mu(x)$. These samples fall outside the data manifold and become unrecognizable to the decoder's prior knowledge. Consequently, the decoder fails to map them to any meaningful structure and instead produces pure noise images.\n\n**Q2: Why Stronger Erasure is Necessary**\n\nExisting methods add noise artifacts that make the image appear unnatural, but still preserve recognizable facial structure. Such noise artifacts added on preserved facial structures often induce a sense of visual eeriness and discomfort. Moreover, this kind of incomplete protection can also be seen as a form of visual uglification of the victim’s appearance, which could potentially be exploited by malicious users.  In contrast, our method aims for complete semantic erasure, providing stronger privacy protection.\n\n **Q3: Higher Resolution and Video tasks**\n\nThank you for the insightful question. We conducted tests with higher-resolution inputs (e.g., 1024×1024). When preserving the adversarial example at full 1024×1024 resolution, the runtime increases from approximately 8 seconds to 19 seconds. \n\nThis is primarily due to the increased computational cost from larger spatial dimensions in both the VAE and the diffusion backbone, which affects all latent-space or diffusion-based attacks. Nevertheless, our method remains highly efficient and consistently outperforms existing approaches in both runtime and GPU usage, even at higher resolutions.\n\nFor video data, our approach can be extended to video generation models using 3D-VAEs or spatiotemporal latent spaces. In this context, variance-based perturbations may disrupt temporal consistency or causal dependencies across frames, offering a promising direction for future adversarial research in video diffusion models.\n\n\n\n**Q4: Gradient Explosion and Unstable Oscillations**\n\nWe appreciate your attention to this important point. Due to character limit, we provide a detailed response to a similar question raised by **Reviewer vD1g Q1**. We kindly invite you to refer to that reply for further clarification.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The proposed LA and LE loss functions effectively address the semantic erasure problem in diffusion models by leveraging variance expansion in VAE latent codes. The methodology is well-grounded in theoretical analysis and overcomes limitations of prior approaches. The evaluation criteria are appropriate, using CelebA-HQ and VGGFace2 datasets, and metrics such as Identity Score Matching (ISM), Face Detection Failure Rate (FDFR), Brisque, and LPIPS, which provide a comprehensive assessment of identity removal effectiveness.", "theoretical_claims": "Please refer previous section.", "experimental_designs_or_analyses": "Yes, I have reviewed the soundness and validity of the experimental designs and analyses.", "relation_to_broader_scientific_literature": "This paper builds on prior work in adversarial attacks on diffusion models and semantic erasure techniques, particularly targeting identity removal in personalized generative models. Previous methods, such as PID and Mist, attempted to erase identity semantics but were limited by slow convergence, poor transferability, and high computational costs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Existing methods may not be optimal, but they seem sufficient to make people doubt the authenticity of a generated image. For instance, in Figure 3, results from SimAC and PID are not as effective as those from your proposed method, but they still make the image look unnatural enough for a viewer to recognize it as manipulated. Could you elaborate on your argument regarding why stronger erasure is necessary beyond this level of distortion?\n2. Given that your proposed methods rapidly expand variance, is there a potential risk of gradient instability or numerical issues during optimization?\n3. Your method demonstrates efficiency in terms of GPU usage and runtime. Could you comment on how this computational efficiency scales to larger images or video data, given the rapidly increasing usage of diffusion models in these domains?\n4. While your method shows strong transferability across different model architectures, could you discuss any theoretical insights explaining why variance-based attacks generalize so effectively across diverse diffusion models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "Js2P41Mt7g", "reviewer": "Reviewer_93PG", "summary": "This paper proposes two novel loss functions, i.e., Laplace Loss (LA) and Lagrange Entropy Loss (LE), which used for adversarial attacks aimed at disrupting Latent Diffusion Models (LDMs). The key insight is identifying the variance of the VAE latent code as critical for effectively erasing identity semantics in generated images. Experimental results show that the proposed methods achieve good performance compared to existing techniques, effectively producing pure-noise images with completely erased identity semantics. Additionally, these methods demonstrate better transferability and reduced computational requirements.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: Attack and Defense Scenario**\n\nThank you for raising this point.  Our method is designed for a practical adversarial setting involving a victim (User A) and an attacker (User B):\n\n- **Defense Phase**: User A wishes to share photos online but wants to prevent misuse by personalization techniques like DreamBooth or LoRA. Before uploading, User A applies our protection method to the images, which introduces subtle perturbations on photos.\n\n\n- **Attack Phase**: User B collects these protected images and tries to train a personalized diffusion model to generate fake content involving User A. However, since the images have been preemptively protected, the model fails to capture meaningful identity information and generates pure noise output.\n\n### **Q2: Trade-off Between Identity Erasure and Visual Quality**\n\nThank you for the thoughtful comment. We address this trade-off in Table 7 of the main paper, where we compare performance under different perturbation budgets. Our method achieves complete identity erasure (e.g., ISM = 0) even with a small perturbation of 8/255, outperforming baselines that require much larger distortions (e.g., 0.05) to reach comparable effectiveness. This indicates that our method is both effective and visually practical.\n\nIn real-world deployment, users can easily balance protection strength and visual quality by adjusting the perturbation budget and number of optimization steps. For instance, limiting the perturbation to 8/255 and stopping at 30 steps provides strong identity removal while preserving reasonable image quality.\n\n### **Q3: Different Prompt** \n\nThank you for raising  this important concern. As shown in **Appendix C.1 (Table 9)** and **Appendix E (Figure 6)**, we evaluate our method using an alternative prompt (“**a dslr portrait of sks person**”) in a DreamBooth personalization setting. The results show that our approach remains highly effective under prompt mismatch, consistently producing pure-noise outputs. This confirms that our method is **prompt-agnostic**.\n\nUnlike prior works such as Anti-DreamBooth, AdvDM, SimAC, and MetaCloak, which require prompt-specific gradient information during optimization, our method does not bind perturbations to any particular prompt. Instead, it directly targets the VAE encoder, which operates before the prompt is introduced. Since prompts influence only the denoising stage (e.g., U-Net or MM-DiT), they do not affect our gradient path, ensuring independence from specific prompt conditions.\n\nMoreover, our method demonstrates strong performance against **LoRA-based personalization (Appendix E, Figure 7)**, where arbitrary prompts can be used during generation, as well as in **ControlNet-based image editing (Appendix E, Figure 28)**, where we apply a new prompt (“**a man**”) to modify the gender of the reference image. These results further validate the prompt-agnostic nature of our approach and highlight its robustness and practicality in diverse real-world scenarios.\n\n### **Q4: Metric**\n\nThank you for the question. Our choice of ISM, FDFR, and BRISQUE, aligns with Anti-DreamBooth, a seminal work in identity erasure. These metrics provide intuitive and reliable evaluations for our task:\n\n- FDFR (Face Detection Failure Rate) measures whether a face is undetectable and is mathematically equivalent to $1-FDS$. We report FDFR instead of FDS because it more intuitively reflects successful cases of identity erasure in privacy protection.\n- ISM evaluates identity similarity between the original and generated images using a face recognition model, directly reflecting whether identity semantics remain.\n- LPIPS is  a perceptual metric to estimate visual degradation. Compared to traditional metrics such as FID or IQS, LPIPS better correlates with human perception, especially in assessing localized texture changes or structural distortions.\n\n### **Q5: Adversarial Training on VAE**\n\nAdversarial training on the VAE may theoretically enhance robustness, but it faces two key challenges:\n\n**1. High Cost and Training Difficulty**\n Adversarial training is computationally expensive and time-consuming. This is especially problematic for VAEs, which are harder to train than classifiers due to issues like posterior collapse and poor convergence with limited or adversarial data. For typical users with limited resources, this makes adversarial training inefficient and impractical.\n\n**2. Stronger Attacks via Proxy Optimization**\n As shown in ASPL, alternating optimization using the VAE as a proxy can significantly boost attack strength. This increases the difficulty of robust training and further destabilizes the learning process.\n\nThese two issues suggest that while adversarial training is a theoretically viable defense, it is both computationally costly and technically nontrivial in practice, especially in low-resource environments.", "claims_and_evidence": "The claims made regarding the efficiency and efficacy of the proposed loss functions (LA and LE) are supported by experimental evidence. State-of-the-art performance metrics (ISM, FDFR, Brisque, LPIPS) demonstrate strong identity erasure capabilities compared to baseline solutions. However, some minor issues remain:\n\n[Issue 1] Clarification of the attack scenario is required. It is suggested to clearly state the training and testing process since not all the readers understand the process. For example, the customization of DM usually require the binding of a specific token to the concept.\n[Issue 2] Evaluation of the protected image quality beyond distortion metrics could strengthen evidence supporting the practical usability of the proposed methods.", "methods_and_evaluation_criteria": "The methods proposed are appropriate for the stated goal of semantic erasure in generative diffusion models. However, the choice of metrics differs from related works without explicit justification.", "theoretical_claims": "The paper provides thorough theoretical insights into the proposed loss functions, and the derivations appear sound upon careful inspection.", "experimental_designs_or_analyses": "The experimental designs are sound, clearly comparing proposed methods to multiple baselines. However, the design is limited by testing only one type of prompt. It is suggested to test different kinds of prompts, as well as the prompts that will be used in practical scenarios.", "relation_to_broader_scientific_literature": "This paper posits its contributions within the context of existing literature, highlighting improvements over current methods in adversarial attacks for LDMs. It effectively identifies gaps in previous approaches (inefficient optimization, misalignment of gradient signs) and clearly differentiates its contributions.", "essential_references_not_discussed": "The related works section appears comprehensive.", "other_strengths_and_weaknesses": "**Strengths**:\n\nS1. Clear Theoretical Justification: The proposed novel loss functions (Laplace-based Loss and Lagrange Entropy-based Loss) are deeply grounded in theoretical analysis, providing a robust rationale for variance manipulation as a mechanism for semantic erasure.\n\nS2. Computational Efficiency: The methods achieve substantial speed improvements and require significantly fewer computational resources, making them practical even in resource-constrained settings.\n\nS3. Empirical Performance: The experiments demonstrate state-of-the-art performance across several datasets and diffusion model architectures, including advanced versions such as SD3.5 and FLUX.1-dev, validating the effectiveness and generalizability of the approach.\n\n(4) High Transferability: Unlike prior methods, the proposed techniques demonstrate consistently high transferability across diverse diffusion model architectures, increasing their utility in various real-world scenarios.\n\nWeaknesses:\n\nW1. Limited Variety in Prompt Selection: The experimental evaluations rely heavily on a single prompt structure (e.g., \"a photo of a sks person\"), restricting the assessment of the generalizability of the attack. Expanding tests to diverse prompt types would better demonstrate the broader applicability of the proposed approach, for example, a photo of a sks person dancing/eating on the train.\n\nW2. Introduction of New Metrics Without Comprehensive Justification: The authors introduce new evaluation metrics, such as Identity Score Matching (ISM) and Face Detection Failure Rate (FDFR), without thoroughly justifying why established metrics (e.g., standard face recognition scores or perceptual quality measures) were insufficient. This lack of detailed explanation may limit the perceived validity and broader acceptance of the evaluation criteria.\n\nW3. The results of adversarial training are encouraged.", "other_comments_or_suggestions": "LINE 196: Replace \"by another neural network\" with \"by another neural network $g$\".\nTable 6: Alignment issues with check and cross symbols should be addressed for clarity.", "questions_for_authors": "Q1. Can you provide a more explicit clarification regarding why only a single type of prompt was used, and discuss potential impacts on the generalization and robustness of your approach?\n\nQ2. Could you elaborate on the rationale for adopting ISM, FDFR, and LPIPS as evaluation metrics over previously established metrics like FDS, FID, and IQS?\n\nQ3. What if we do the adversarial training on VAE?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "BhL7FrFFbw", "reviewer": "Reviewer_36nD", "summary": "This paper protects images from malicious editing by attacking diffusion models. The authors design two loss functions, LA and LE, to attack the image variance after VAE encoding, demonstrating stronger attack effectiveness compared to other methods.\n\n## update after rebuttal\nAuthors' rebuttal have solved my concerns, and I will adjust my rating based on other reviewers' scores.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Experiments of Mean Attack**\n\n**Table 1: Effectiveness of Attacking Mean vs. Variance**\n\n| Method| ISM ↓ | FDFR ↑ | Brisque ↑| LPIPS ↑|\n| -------------------- | ----- | ------ | ----------- | --------- |\n| LA_Mean_30step| 0.276 | 0.598  | 29.801| 0.855|\n| LA_Mean_50step| 0.234 | 0.703  | 31.714| 0.861|\n| LA_Mean_100step| 0.204 | 0.781  | 33.219| 0.871|\n| LA_Mix_100step| 0.231 | 0.772| 32.594| 0.872|\n| LA_Var_30step (ours) | **0** | **1**  | **155.845** | **0.959** |\n\nThank you for your thoughtful comment. We address this concern both theoretically (see our response to Reviewer WD7W, Q1) and empirically in **Table 1**. The results clearly show that attacking the mean alone yields significantly weaker erasure performance, even with more optimization steps. For example, **LA_Mean_100step** achieves ISM = 0.204, whereas our **LA_Var_30step** achieves **ISM = 0** and **FDFR = 1** with much fewer steps. Furthermore, combining mean and variance attacks (**LA_Mix_100step**) limits the growth of variance and results in less effective erasure than attacking variance alone. These consistent trends demonstrate that variance-based attacks are more effective and efficient, both in theory and in practice.\n\n **Q2: Ambiguity about Table 8**\n\n**Table 2: Robustness of different methods against JPEG Compression and GrIDPure**\n\n| Attack Method | No Image Preprocessing | No Image Preprocessing | No Image Preprocessing | No Image Preprocessing | JPEG Compression | JPEG Compression | JPEG Compression | JPEG Compression | GrIDPure| GrIDPure| GrIDPure   | GrIDPure  |\n| ------------- | ---------------------- | ---------------------- | :--------------------: | ---------------------- | ---------------- | ---------------- | ---------------- | ---------------- | --------- | --------- | ---------- | --------- |\n|| ISM ↓| FDFR ↑|Brisque ↑| LPIPS ↑| ISM ↓| FDFR ↑| Brisque ↑| LPIPS ↑| ISM ↓| FDFR ↑| Brisque ↑  | LPIPS ↑|\n| Clean Image   | 0.608| 0.041|17.896| 0.662| -| -| -| -| -| -| -| -|\n| AdvDM| 0.424| 0.307|24.215| 0.798| 0.659| 0.031| 30.269| 0.771| 0.601| 0.031| 20.609| 0.705|\n| ASPL| 0.406| 0.287|24.419| 0.805| 0.668| 0.042| 31.592| 0.769| 0.574| 0.022| 20.061| 0.707|\n| Mist| 0.249| 0.169|13.981| 0.707| 0.629| 0.014| 33.686| 0.807| 0.541| 0.057| 20.149| 0.773|\n| MetaCloak| 0.593| 0.051|36.325| 0.712| 0.592| 0.075| 53.622| 0.813| 0.593| 0.047| 27.871| 0.745|\n| SimAC| 0.253| 0.865|51.059| 0.823| 0.632| 0.032| 32.955| 0.759| 0.548| 0.091| 24.967| 0.699|\n| DisDiff| 0.605| 0.116|29.361| 0.695| 0.672| 0.035| 29.217| 0.753| 0.618| 0.032| 19.651| 0.693|\n| SDS-| 0.655| 0.005|38.519| 0.743| 0.684| 0.022| 33.053| 0.818| 0.624| 0.047| 23.174| 0.729|\n| PID| 0.069| 0.938|85.533| 0.899| 0.473| 0.255| 40.435| 0.804| 0.473| 0.189| 43.878| **0.773** |\n| **LE (ours)** | **0**| **1**|**155.804**| **0.947**| **0.451**| **0.358**| **62.541**| **0.821**| **0.456** | **0.231** | **49.071** | 0.769     |\n| **LA (ours)** | **0**| **1**|**155.845**| **0.959**| **0.447**| **0.342**| **63.572**| 0.791| **0.449** | **0.242** | **49.729** | 0.761     |\n\nThank you for the question, and we apologize for the confusion. In Table 8, “Clean” refers to the original image without adversarial perturbation.  To better illustrate the robustness of our method, we provide **Table 2**, which evaluates robustness performance under both **traditional image preprocessing (JPEG Compression)** and **diffusion-based purification (GrIDPure)**.\n\nIn this table:\n\n- “Clean Image” denotes the unperturbed original image\n-  “No Image Preprocessing” indicates adversarially protected images before any additional post-processing.  \n\nAs shown, our methods consistently achieve stronger robustness than most baselines and deliver performance comparable to or better than PID under both JPEG compression and GrIDPure purification. This demonstrates the effectiveness and reliability of our approaches under common real-world applications.\n\n **Q3：LA vs LE, which is better?**\n\nThank you for the feedback. Both LA and LE losses are designed to align perturbations with the direction of latent variance growth, and they share the same core objective. While their formulations differ slightly, they are conceptually equivalent in guiding effective attacks.\n\nAs shown in **Tables 1, 2, and 3 of the main paper**, both losses consistently outperform existing baselines across different evaluation settings. Table 4 further demonstrates that our method improves attack efficiency by nearly 30× compared to the previous SOTA method PID. In addition, Table 7 shows that our loss functions achieve complete identity erasure (e.g., ISM = 0, Brisque = 122.266) even with a small perturbation budget of 8/255, while baselines require much larger perturbations (e.g., 0.05) to achieve inferior results.\n\nThese results suggest that both losses are effective and efficient, and users can choose either based on implementation preference or training stability.", "claims_and_evidence": "Although the authors emphasize the importance of variance, there is no experiment demonstrating whether attacking the mean also has the same attack effect.", "methods_and_evaluation_criteria": "What does “clean” in Table 8 mean? Does it refer to the performance without any defense method applied, or the performance without any attack? Both should be included in the table to better illustrate the impact of the defense methods on the attacks.", "theoretical_claims": "In my opinion, there are no major issues with the theoretical analysis, but I am not an expert in this area, so I suggest considering the opinions of other reviewers as well.", "experimental_designs_or_analyses": "The relationship between the two proposed losses is not clearly illustrated, and the experiments do not explicitly clarify which loss is more advantageous for specific scenarios.", "relation_to_broader_scientific_literature": "The proposed method is an improvement based on previous VAE methods, such as PID.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths :\n1. The proposed methods show better attack performance compared to other approaches.\n2. The design of the loss functions has a certain level of interpretability.\n\nWeakness see the above", "other_comments_or_suggestions": "No additional comments", "questions_for_authors": "No additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vEXII9bgRX", "reviewer": "Reviewer_vD1g", "summary": "The paper proposes a novel adversarial attack method, leveraging variance manipulation to efficiently and consistently erase identity semantics from images generated by diffusion models, such as Stable Diffusion. The authors introduce two main approaches, Laplace-based and Lagrange Entropy, to address limitations in existing methods like MSE and PID . Their techniques focus on optimizing the variance components of the latent space, enabling more effective and efficient semantic erasure in generated images. Additionally, they demonstrate the transferability of their methods across various diffusion models and against different personalization techniques, showcasing the robustness of their approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: Gradient Explosion and Unstable Oscillations**\n\nThank you for raising this important point.  We acknowledge that rapidly increasing latent variance can lead to gradient explosion and numerical instability. As discussed in **Appendix D.1 and Fig. 4**, the gradient of the LE loss, $\\tfrac{\\partial \\mathcal{L}_{LE}}{\\partial \\delta}$, explodes around 30 optimization steps, while the LA gradient begins oscillating around step 50. These issues are caused by the nonlinearity of the model and the Jacobian terms, which become unstable as the latent distribution expands and certain activations saturate in high-dimensional space.\n\nInterestingly, **this instability is not entirely detrimental**. In fact, it reveals a clear and interpretable progression in the generated images over the course of optimization:\n\n-  **At step 20**, the variance increases slightly. Most latent dimensions remain compact, and only a few begin to expand. The generated image still retains a recognizable facial structure with mild noise and texture distortions.\n-  **Around step 25**, variance expansion becomes more pronounced. The image quality starts to degrade  with strong noise artifacts appearing across the face. However, some identity-related features are still visible. At this point, the visual result looks similar to outputs from methods like SimAC and PID—unnatural and distorted, but not fully erased.\n-  **Between steps 30 and 50**, the variance increases rapidly across many dimensions, causing the latent distribution to flatten and spread widely. This pushes the sampled latent codes far from the dense semantic region covered by the VAE’s training distribution. Since these codes fall outside the decoder’s learned prior, they are interpreted as random noise, leading to outputs with no recognizable facial structure or semantic content.\n-  **At step 75**,  variance grows at different rates across dimensions. Some dimensions saturate while others continue expanding, leading to imbalanced latent structures. The outputs exhibit chaotic combinations of textures and colors, forming a new kind of “random state” in appearance.\n-  **Beyond step 100**, further gradient explosion or sign flipping may occur. In some cases, the decoder maps these extreme latent codes to scene-like artifacts (e.g., indoor layouts). These are not restored content but hallucinated patterns caused by nonlinear decoding from noise.\n\nThis behavior highlights a key advantage of our method: **Controllability**. Users can stop the optimization early (e.g., at step 25–30) to avoid instability while still achieving strong identity removal with reasonable image quality. Alternatively, further optimization gradually strengthens the erasure effect, progressing from facial distortion to complete identity removal and, eventually, to synthetic scene-like hallucinations. This allows users to tailor the erasure strength to their specific needs, offering a practical balance between robustness and flexibility.\n\n\n\n### **Q2. Trade-off Between Identity Erasure and Visual Quality**\n\nThank you for the thoughtful suggestion. We address the trade-off between identity erasure and visual quality in **Table 7 of the main paper**, where we compare our method under different perturbation budgets. Notably, our approach achieves complete identity removal (ISM = 0) even with a small perturbation of 8/255, while baseline methods require significantly larger perturbations (e.g., 0.05) to achieve similar or weaker results. This demonstrates that our method is not only effective but also visually practical.\n\nIn addition, a key strength of our approach is its controllability. Users can flexibly adjust the perturbation size and the number of optimization steps to balance protection strength and image quality. For instance, stopping at around 30 steps with 8/255 perturbation yields strong identity erasure while maintaining acceptable visual quality, which we believe makes the method more adaptable for real-world applications. \n\n\n\n### **Q3: More Experiments on other datasets**\n\nThank you for this valuable suggestion. **As shown in Fig. 13 at the end of the appendix**, we have extended our evaluation to artistic-style images. Our method remains highly effective in this setting, producing pure noise outputs and demonstrating strong semantic removal. These results indicate that our approach is not limited to facial datasets and can generalize well to a broader range of image types. We sincerely appreciate your encouragement to further validate the robustness of our method.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The paper proposes two novel loss functions—LA and LE—focused on optimizing the variance in the latent space of diffusion models. The LA loss ensures the gradient is aligned with the variance growth direction, allowing for efficient local optimization. The LE loss integrates entropy and a Lagrange constraint to balance optimization, promoting variance growth in smaller components and preventing slow convergence as variance becomes more uniform. These two losses outperform traditional methods like MSE and PID in both speed and effectiveness.", "theoretical_claims": "The paper claims that manipulating the variance of the latent codes is the key to erasing identity semantics in generated images. While the authors theoretically justify their approach using gradient flow analysis, the detailed reasons why variance manipulation is superior to other adversarial strategies could be elaborated further.", "experimental_designs_or_analyses": "The experiments are well-designed, covering a range of datasets and comparing the proposed methods to a variety of baselines. However, the study could benefit from a broader set of experiments that test the methods on more diverse datasets, including non-human subjects or artistic images, to assess the method’s generalizability. Additionally, more discussion is needed on the visual quality of generated images and how the methods balance identity erasure with image naturalness.", "relation_to_broader_scientific_literature": "The paper is well-positioned in the context of adversarial attacks and privacy protection for generative models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "# Strength\n\n- The idea of manipulating variance components in latent space to achieve semantic erasure is an innovative approach. It provides an alternative to traditional adversarial techniques that focus on perturbing pixels or the mean of the latent space. This allows for more targeted attacks on identity information.\n- The methods proposed (LA and LE) are computationally efficient, using significantly less memory and processing time compared to existing methods like MetaCloak and SimAC. For example, LE requires only 8 seconds and 4.5 GB of GPU memory, which is impressive considering the task at hand. Additionally, the high transferability of the methods across models like SD1.5, SD2.1, SDXL, and others is a valuable contribution to the fieldxperimental Design and Evaluation\n\n# Weakness\n- The mathematical models in Section 3 offer a detailed explanation of the loss functions (LA and LE), but the implications of these formulations on the stability of the optimization process, especially under high-dimensional settings, could be better discussed . While the authors addrestical challenges (e.g., Jacobian computation), more clarity on potential issues like gradient vanishing/explosion in high-dimensional latent spaces could enhance the robustness of their claims .\n- The authors present a variety of experimental settings, including comparisons with multiple state-of-the-art methods. However, the experiments could benefit from a deeper analysis of the real-world implications of their method. For example, further discussion is needed on the trade-offs between the extent of identity erasure and the visual quality of generated images. Although the images achieve high semantic erasure, they may not always meet the quality standards expected in all applications.\n- The experimental evaluation also primarily focuses on face images (CelebA-HQ, VGGFace2). It would be beneficial to extend these tests to a broader range of image types and tasks (e.g., artistic generation or non-human subjects) to test the generalizability of the proposed method.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by four experts in the field and finally received all positive scores: Weak accept, Accept, Weak accept, and Weak accept.
The major concerns of the reviewers are: 
1.	why variance-based attacks are superior to other adversarial strategies,
2.	additional experimental results on other datasets and other settings.
The authors address the above concerns during the discussion period. Hence, I make the decision to accept the paper.
Decision: Accept (poster)