AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Models
Authors: Fengpeng Li
Kemou Li
Qizhou Wang
Bo Han
Jiantao Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Adversarial Learning; Prompt Injection Attacks; Adversarial Defending
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Concept erasure helps stop diffusion models (DMs) from generating harmful content; but current methods face robustness-retention trade-off. **Robustness** means the model fine-tuned by concept erasure methods resists reactivation of erased concepts, even under semantically related prompts. **Retention** means unrelated concepts are preserved so the model’s overall utility stays intact. Both are critical for concept erasure in practice, yet addressing them simultaneously is challenging, as existing works typically improve one factor while sacrificing the other. Prior work typically strengthens one while degrading the other—e.g., mapping a single erased prompt to a fixed safe target leaves class-level remnants exploitable by prompt attacks, whereas retention-oriented schemes underperform against adaptive adversaries.  This paper introduces Adversarial Erasure with Gradient-Informed Synergy (AEGIS), a retention-data-free framework that advances both robustness and retention. First, AEGIS replaces handpicked targets with an Adversarial Erasure Target (AET) optimized to approximate the semantic center of the erased concept class. By aligning the model’s prediction on the erased prompt to an AET-derived target in the shared text–image space, AEGIS increases predicted-noise distances not just for the instance but for semantically related variants, substantially hardening the DMs against state-of-the-art adversarial prompt attacks. Second, AEGIS preserves utility without auxiliary data via Gradient Regularization Projection (GRP), a conflict-aware gradient rectification that selectively projects away the destructive component of the retention update only when it opposes the erasure direction. This directional, data-free projection mitigates interference between erasure and retention, avoiding dataset bias and accidental relearning. Extensive experiments show that AEGIS markedly reduces attack success rates across various concepts while maintaining or improving FID/CLIP versus advanced baselines, effectively pushing beyond the prevailing robustness–retention trade-off. The source code is in https://github.com/Feng-peng-Li/AEGIS.

Published as a conference paper at ICLR 2026
AEGIS: ADVERSARIAL TARGET-GUIDED RETENTION- PATA-FREE ROBUST CONCEPT ERASURE FROM DIFFU-
SION MODELS
Fengpeng Li
Kemou Li Qizhou Wang".3 Bo Han
Jiantao Zhou
1 State Key Laboratory Internet of Things for Smart City, University Macau
2TMLR Group, Department of Computer Science, Hong Kong Baptist University
3 Imperfect Information Learning Team, RIKEN Center for Advanced Intelligence Project
*PRADA Lab, King Abdullah University of Science and Technology
ABSTRACT
Concept crasure helps stop diffusion models (DMs) from generating harmful content; but current methods face robustness-retention trade-off. Robustness means the model fine-tuned by concept erasure methods resists reactivation of erased concepts, even under semantically related prompts. Retention means unrelated concepts are preserved so the model's s overall utility stays intact. Both are critical for concept erasure in practice, yet addressing them simultaneously is challenging, existing works typically improve one factor while sacrificing the other. Prior work typically strengthens one while degrading the other e.g., mapping a single erased prompt fixed safe target leaves class-level remnants exploitable by prompt attacks, whereas retention-oriented schemes underperform against adaptive adversaries. This paper introduces Adversarial Erasure with Gradient-Informed Synergy (AEGIS), a retention-data-free framework that advances both robustness and retention. First, AEGIS replaces handpicked targets with an Adversarial Erasure Target (AET) optimized to approximate the semantic center of the erased concept class. By aligning the model's prediction on the erased prompt an AET-derived target in the shared text-image space, AEGIS increases predicted- noise distances not just for the instance but for semantically related variants, substantially hardening the DMs against state-of-the-ar adversarial prompt attacks. Second, AEGIS preserves utility without auxiliary data via Gradient Regularization Projection (GRP), a conflict-aware gradient rectification that selectively projects away the destructive component of the retention update only when it opposes the erasure direction. This directional, data-free projection mitigates interference between erasure and retention, avoiding dataset bias and accidental relearning. Extensive experiments show that AEGIS markedly reduces attack success rates across various concepts while maintaining or improving FID/CLIP versus advanced baselines, effectively pushing beyond the prevailing robustness retentiion trade-off. The source code is in https: //i thub. com/Feng- wte///eennng------ee Liie-Li/AEGET
INTRODUCTION
While DMs (Sohl-Dickstein et al., 2015; Ho et al., 2020) excel at text-to-image (T2I) generation, biased training data poses safety and reliability risks (Li et al., 2024a; 2025a), including harmful outputs (Li et al., 2026). Concept erasure (Gandikota et al., 2023), modifying the denoising UNet of DMs to remove undesirable concepts, has become a standard measure to obtain a reliable DMs. These methods are typically categorized as output-based, realigning concept representations, attention- based, aiming at manipulating cross-attention scores (Bui et al., 2025). Despite progress, these techniques face two critical challenges. First, shown in Fig. 1, erased concepts can be reactivated by adversarial or semantically related prompts not seen during training (Chin et al., 2024). Second, improving robustness against adversarial prompt attacks (APAs) comes the cost of degrading the
"Corresponding author; Jiantao Zhou fttzho@@um.edu..m

---

Published as a conference paper at ICLR 2026
generation quality for unrelated concepts, creating harsh cobustness-retentiin trade-off (Zhang et al., 2024b:e; Gong et al., 2024; Meng et al., 2024b). Both goals are critical for concept erasure, while seldom works can handle them simultaneously. Consequently, achieving robust erasure without compromising retention remains a central, unsolved problem.
Prevailing studies (Bui al., 2025) largely focus on the impact of map- ping target on the trade-off between concept erasure and the retention. critical yet underexplored dimen- sion, however, is the influence of the erasure target on the model's ro- bustness against APAs, mitigating the trade-off seems better. most methods, only an instance for the concept, e.g., nudity, adopted to perform concept erasure by maximizing the distance of noises induced by the instance between pre- and post-fine-tuning DMs. However, practical users input often involve semantically equivalent or metaphorical expressions, such as naked or sexual. Since DM can be interpreted et al., 2024), these synonymous terms are likely to be clustered within the same semantic class the latent space. If the chosen instance lies too far the semantic center of the nudity class, the concept information may not be fully eliminated. Under APAs like Zhang et al. (2024c) close to the concept semantic center, the incompleteness of nudity class erasure becomes evident.
Fine-tuned Stable Diffusion
nudity Temale
Prompt
Unet
Random Noise
KV
KV
KV
KY
x00x audity female
Fine-tuning Parameters Parameters Figure I: Adversarial prompts can still result in DMs fine-tuned by
Adversarial
Cross
Forward
Frozen
Prompt
Attention
Direction
concept crasure methods generating images with harmful information.
classifier (Chen
Building upon these insights, we introduce AEGIS (Adversarial Erasure with Gradient-Informed Synergy). a novel framework that significantly enhances the robustness of concept erasure against APAs while preserving the retention for unrelated concepts. AEGIS achieves this through two key components: Adversarial Erasure Targel (ART) and Grudient Regularization Projection (GRP). AET with optimizable embedding is iteratively updated to create a potent adversarial target. The model is then fine-tuned to minimize the difference between the noise predicted for AET and the erased concept, effectively strengthening the erasure process. GRP is applied to maintain the model's performance on unrelated concepts without requiring additional retention data, thus preventing negative impacts from the erasure process. Experiments across various concepts against range of APAs confirm the superior effectiveness of AEGIS, which surpasses state-of -fate-of-the-- the-the-rtt art methods in erasure robustness by large margins while preserving higher retention performance.
Contribution Summary. The contributions of this work can be summarized as follows:
We demonstrate that the vulnerability of concept erasure stems from an inappropriately chosen learning target. In particular, if the target lies too close to the semantic center- semantically related to the erased concept- the concept information cannot be fully removed.
by words
We propose novel retention retention-data-fre framework, AEGIS, which synthesizes an AET to guide the removal of class-level information associated with erased concepts. Additionally, we introduce a gradient projection strategy to better balance erasure robustness and retention performance.
Extensive experiments multiple datasets validate that AEGIS markedly enhances erasure robust- ness against variety of APAS. Notably, AEGIS decreases the attack success rate by ~5.31 % on the nudity concept and 244 the Van Gogh style concept under P4D (Chin al., 2024) and UnlearDDffrAtk (Zhang et al., 2024c), while incurring little impact on retention performance.
2 PRELIMINARIES
Latent Diffusion Models. This work focuses on latent diffusion models (L.Mss) (Rombach et al., 2022). Given a training dataset D [[x;, G)),, each image x; is paired with prompt c; A noise predictor (-(-)c), parameterized by 0, is learned to estimate the Gaussian noise e added to the latent representation z at each diffusion step t f1...., T), conditioned on c, optimized as:
min Exx.e)-D..cc-WW00) |||c c cœ(z, [)]
0
Concept Erasure in DMs. To remove specific undesirable concepts from DMs, Eø is fine-tuned. Let C, denote the prompt about the (to-be- jerased concept, e.g.. "nudity". Existing methods (Rombach et al., 2022: Bui et al., 2024; 2025) map c₂ to erasure target concept č-a generic concept (e.g..
2

---

Published as a conference paper at ICLR 2026
"a photo") simply an empty prompt. In parallel, retention dataset, D, {xx,, cr,,) where
Cr E Creaann is unrelated to Ce. is employed to retain the model utility. Let 0, denote the original parameters, the erasing objective is defined as:
min
E, (0 (Z/ c.)
ca, (z: [) +X ED,, [I(e((z))c, ca, (ze c
(2)
Erasing Loss
Retention Loss
where À balances erasure and retention. The erasing loss encourages the DM respond to C, as it were č, thereby reducing its ability to reconstruct c. In general, ea, (c) is set as E0, (Z:) 1] (Eo, (ze (c.) Eoo (zz)), where Ea, (z,) is unconditional generation and 7) controls the erasing strength.
Adversarial Prompt Attacks and Robust Concept Erasure. Prior studies (Zhang et al., 2024c; Chin et al., 2024) reveal that, even after applying the erasure objective in Eq. (2), the crased concept may still resurface under adversarial prompt attacks. Given Ce that has been erased, one craft an adversarial prompt c" to provoke the concept-erased model into producing a response akin to that of the original model when queried with ce. Specifically, c* can be obtained by solving
c* = arg min Eg (((ee(zz)))) - ce. (zr (ce ||c lo--calosee Sm
(3)
subject to the constraint that m' S dimensions of T are learnable during optimization. To strengthen erasure robustness against such attacks, inspired by adversarial training (Madry et al., 2018; Li et al., 2024b; 2025b), AdvUnlearn (Zhang al., 2024b) substitutes the original erased concept C, in Eq. (2) with an adversarial prompt c* directly within the objective. This compels the model suppress
only prompts explicitly tied c, but also nearby adversarial variants, thereby improving robustness against prompt-space leakage.
With the preliminaries of concept erasure established, we now examine its inherent vulnerabilities.
MOTIVATION: UNVEILING THE VULNERABILITY IN CONCEPT ERASURE
This section explores fundamental vulnerability in current approaches of concept erasure in DMs, motivating the design of our AEGIS. Specifically, we investigate the following central question:
Can a single prompt entity suffice to effectively remove all information of a concept from DMs?
To explore this, we conduct an initial study on erasing the concept nudity (c..) and assess its influence on semantically related concepts-nakke sexual, erotic, and impure (c 1, 2, 4). We define three concept groups to facilitate our analysis: Co [c]) C1 =cc c and C2- {c?, cg c.)-
To characterize how concept erasure impacts related information, inspired by Chen et al. (2024) interpreting DMs as generative classifiers, we view the predicted noise eg (Ze [c,) timestep as the model's 'ss internal representation of concept At the final step T, this noise becomes deterministic that reflects the location of ce in the latent space, and can be interpreted as a class prototype. Intuitively, similar concept instances induce similar noises, thus forming clusters in the latent space.
This perspective enables two types of measurement: 1) Intra-model distances, reflecting semantic separation between concept instances in the same DM; 2) Cross-model distances, quantifying erasure strength by comparing predictions from pre- and post-erasure models. The former is exactly what we discussed above. To theoretically justify the latter, establish Prop. 3.1, suggesting that successful erasure results in a significant shift in the predicted noise embedding of the erased concept.
Proposition 3.1 (Deviation Lower Bound). Let Lenue E. (Z, cao (z, [)]]] and Ar = (2r(c2) E0, 60, (2T l).. Assume optimizing Eq. (2) yields Lenas (0) S 8 < AT. Then,
(zr (c?) cB. 6.. (2r1c))11 (VAr - VD)²
The proof deferred to Appx. E.I. Prop. 3.1 indicates that successful concept erasure leads to a significant deviation in predicted noises before and after fine-tuning the DM. Based this theoretical insight, Def. formally defines predicted noise distances to systematically evaluate erasure quality and robustness across different concepts and methods: do quantifies semantic similarity within the base model, di captures erasure effectiveness across pre- and post-erasure models, and d2 evaluates the distance between original and adversarial concept prompts within the fine-tuned DM, reflecting
3

---

Published as a conference paper at ICLR 2026
Nudity
Nudity
Nudity
Nudity
Nudiy
Naked
linpure Naked
imppure Naked
linpure Nake
Impure
Naked
Secual
10
10
Eralic
Sexual
Erotic
Soxual
Erotic
Sexual
Erótic
Sexual
Erotic
(n) Original
(b) dj. ESD
(c) d2. ESD
(d) AET
Figure 2: Predicted noise distances (10") between mudity and its synonyms, measured using the original DM Oo and concept-erasee DMs OESD and OXET-
adversarial vulnerability. With these metrics, in Fig. 2 empirically analyze the impact of concept erasure on nudity and its related concepts using ESD (Rombach et al., 2022) and AdyUnlearn.
Definition 1 (Predicted Noise Distance). Define the pre- dicted noise distances do, d1, d2 between any two concept instances c'2, c/. E Co as:
Table 1: Concept crasure robustness by Un- learnDiffAtk (Zhang et al., 2024c).
METHOD
Pre-ASR (4)
ASR (1)
da(ce,,e) == |EO (2r (c() co, (2T [c))
SD v1.4
100%
100%
ESD
20.425
76.05%
di(c), = ||g ED, (ZT) (zT|c)) c
AdvUnleam
15.45%
64.79%
ESD w/ AET
6.38%
26.06%
d2(c), c == ((co(2r/(e co(27 (2T )
co
AdvUnleam w/ AET
1.41%
5.63%
Impact of Concept Erasure. Fig. 2a reveals that noise for nudity and naked (C2) exhibit proximity, while sexual, erotic, and impure (C2) show significant divergence. After erasing nudity via ESD, Fig. 2b demonstrates enlarged d, for C₁ but limited changes for C2. indicating ineffective removal of Cg. This observation aligns with pre-attack success rate (Pre-ASR) results in Tab. 1: ESD-fine-tuned DMs still generate nudity images in response to benign prompts from Zhang et al. (2024c), indicating that using an instance from C1 or C2 as ce cannot remove all information of Co from the DM.
Impact of APAs. Despite ESD fine-tuning, Fig. 2c reveals consistently small de for Co. This low d2 enables adversarial prompts to achieve high ASR, as reported in Tab. 1. Even AdvUnlearn, which integrates adversarial prompts into Eq. (2), fails to defend against UnlearnDiffAtk. From the phenomenon, we guess this vulnerability stems from inadequate mapping targets č. As shown in Fig. 9a of Appx. D. if C. (e.g., member of C2) is far from the concept center of Co, maximizing the distance between Ea, (Ze (c.) and eg ( i.e. minimizing the gap between (z and Eq, (Z,(c). cannot remove the information of C2 from the DM. Since e is not close to C₁ but also C2 and similar to the concept center of Co, then the residual information of the undesirable concept will be leaked again. Under such cireumstances, only when the gap between @ and Og is extremely large, where d1 is sufficient, the knowledge of Co can be erased. However, such a drastic parameter shift also disrupts preserved concepts, resulting in poor utility of the fine-tuned DM. That is the reason why existing tradeoff-stronger erasure compromises generation fidelity.
methods exhibit pobustness-eetentie
Impact of Adversarial Erasure Target. Given the small d2 values for all instances in Co. we hypothesize that the adversarial prompt used for concept erasure resides near the semantic center of Co. That means, as shown in Fig. 9b of Appx. D.I, maximizing the distance between c, and its concept center can enlarge dj for all Co, enhance the concept erasure effectiveness. In light of this, we propose an adversarial erasure target (AET) updated by K 10 steps, defined as:
((+++)) (1)) -3 3 sign (VIEo, (zrlc'(A) (z( ( )1 + ||ea, (ze (c(()) Ea, (ze (c.) c3)) (5)
where 3 is the step size, and c'(") is randomly initialized with learnable parameters of length m' We then apply the optimized c' into ESD, yielding the objective as:
1.
0 AAET arg min ((x,)), l.lllll (z, (ce) E0, (z,
(6)
where EB, (z)(c) = E (2:) - (T (zxlc') Ea, (z:)). With the target, shown in Fig. 2d, fine- tuning with AET consistently enlarges d2 across all instances in Co. Moreover, compared with AdvUnleam in Tab. 1, ESD-AET achieves 9.07% and 59.16% reduction in pre-ASR and ASR, respectively, indicating the information of nudity is effectively removed from the DM. Building on the effectiveness of AET, propose AEGIS to enhance the concept erasure robustness.
4

---

Published as a conference paper at ICLR 2026
Forward
Backward
Fine-tuning
Frozen
Eog (a,c')
Fine-tuning d
Eo (z(E)
Frozen O
(xp
Unst
(4p e c
Ro "06r,(all)) ) (n,02,))
AET Generation (x),
Ce.
II. GRP
Eg (4,
generated by
Gogh, concept C'AAversarial Prompt
d'AET
Timestep sampled uniformly
Figure 3: Overview of AEGIS. The proposed framework introduces two core components: (I) adversarial erasure
larget (AET) generation (54.1), and (II) gradient regularization projection (GRP) fine-tuning (84.2).
METHOD: ADVERSARIAL ERASURE WITH GRADIENT-INFORMEN SYNERGY
This section presents the details of the proposed AEGIS, illustrated in Fig. 3. AEGIS comprises two core components: AET generation ($4.1) 1) constructs AETs to enhance concept erasure robustness against APAs; GRP ($4.2) fine-tunes that minimizes the gap between predicted noises induced by erased concepts and their corresponding AETs. This procedure enhances concept crasure robustness with better retention. The optimization objective of AEGIS formalized as:
min max E, [e(lo((zz(()) EB,, (Zyl [2) +
0ol1
Brasing Loss L.
Retention Loss C..
where c* is the adversarial prompt for Ce, and co, (z)(c) = co. (2)) (co (zelc') co. (2:)).
4.1 AET FOR GUIDING THE ROBUST ERASURE OPTIMIZATION DIRECTION
This part introduces the details of generating AET. described in $3, empirical observations suggest that the adversarial prompt, corresponding to an instance of the erased concept, is close to the concept center. Thereby, we introduce AET in Eq. (5). Considering the dynamic changes the erased concept, the AET is optimized to be close to the original and updated concept centers, maximizing the distance cø [ Cc) and co. (z, (2))ce continuously. Moreover, the iterative mechanism of AET generation in Eq. (5) reduces the efficiency of AEGIS, while the gap between c' and c, is not small enough with single iteration. Thereby, following the fast adversarial training strategy (Jia et al., 2022a;b; Huang et al., 2023), for the 7-th training epoch, AET is generated efficiently
c((++1) = c(T) B sign V (II (z.lc((r) (
c. )l) co (z,(c"(r)) co (zelc))
In line with Zhang et al. (2024b), similar to for the erased concept c, is updated via
-=(+))==
sign V 6 (2, ((())) - co. (z 1l))
This new AET generation process enables AEGIS to achieve robust concept erasure with reduced computational overhead. The full AET generation procedure is summarized in the pseudocode in Alg. of Appx. C. Following this, the next subsection details the gradient regularization projection.
4.2 GRP FOR MITIGATING THE TRADE-OFF BETWEEN CONCEPT ERASURE AND RETENTION
This part presents the GRP mechanism, which consists of Parameter Regularization (PR) and Directional Gradient Rectification (DGR). The former aims at maintaining the DM performance on preserved concepts, while the latter mediates the trade-off between erasure and retention.
Parameter Regularization. Building upon the objective in Eq. (7), the L, introduces layer-wise parameter regularization that penalizes deviations between and 0g. This constraint enables AEGIS to erase C. while making only minimal modifications Op. As result, parameters irrelevant to Ce are preserved, retaining the performance of DMs on unrelated concepts. This PR yields two key advantages. First, it eliminates the reliance on additional retention datasets. Since the original DM is trained on a high-diversity. large-scale dataset with substantial computational resources (Rombach al., 2022), small-scale retention datasets cannot comprehensively preserve all concepts. Prior
5

---

Published as a conference paper at ICLR 2026
works (Bui al., 2024; 2025; Zhang al., 2024b) reveal that retention quality is highly sensitive to dataset selection, positioning PR as a robust and data- aata-aanostic alternative. Second, it mitigates the risk of undestrable concept relearning. Since PR requires retention data, residual signals of c. will not be reintroduced, thus preventing the inadvertent relearning of the erased concept.
Directional Gradient Rectification. While PR circumvents the need for auxiliary reten- tion data, we observe that conflict persists between ge VaLe and 9r VeL.. As shown Fig. 10 of Appx. D.2, the cosine sim- ilarity between 9e and gr is negative through- out many training iterations. Under such cir- cumstances, as depicted in Fig. 4, 9t. the paral- lel component gr. has an opposite direction to ge. This observation naturally leads to the definition in Def.
Gradients Conflict
Gradients Conflict
Br
(4) Non-csuffict between A, and
(0) Coaflict between at and Ar
Figure Geometry of ge and g. w/wo conflict. When cos Φ < 0, gr can be orthogonally decomposed into a vertical gradient IT and a parallel 911 to g-.
Definition 2. Let @ be the angle between ge and gr- A gradient conflict occurs when cos Φ < 0.
Due to the presence of such conflict, directly optimizing Eq. (7) often compromises the re- tention performance (Yu al., 2020; 2025b; Huang et al., 2025). Paradoxically, exist- ing methods often exhibit trade-off where weaker concept erasure robustness coincides with stronger retention performance (Wu Harandi, 2024; Zhang et al., 2024b). To al- leviate the conflict between ge and gr. DGR breaks one condition of the tragic triad by ex- plicitly modifying the gradients to strike a balance hetween concept erasure robustness and retention performance. To ensure both effectiveness and broad applicability, we design the gradient adjustment to foster positive interactions between ge and gr. without making any assumptions about the structure of the DM. Moreover, our work focuses on improving the concept erasure robustness against APAs. Thereby, when cos ø 2 0 as illustrated in Fig. 5a, we set À = 0; that is, no retention is injected. Conversely, when cos @ < 0 as shown in Fig. 5b, DGR adjusts gr as
Gradients Conflict:
Gradients Conflict:
8x 8
ll.
#+
de 5e
(e) Non-cmolicct between g, and
(b) Cuaffict betwoee ge and g, rectified by DGR
Figure DGR operation on 9e and g- w/wo conflict.
9 A gr, where À (ge. 9r)
(8)
11
Here, E [0, is parameter control the gradient rectification strength. When 1, gr is fully orthogonal projected; conversely, w = implies that DGR treats ge and gr as non-conflicting. Accordingly, the modified gradient ğ w.r.t. 0 for the overall loss function L Le x L, is:
VoL. A VoL, = VoL. + X - 0o) = 9- + A gr-
As C, functions as a constraint on 0, it is not necessarily minimized during concept erasure. Moreover, since the original direction of gr is altered, modifying g. via DGR may negatively impact concept erasure performance, especially at the beginning of DM fine-tuning, where 0 remains close to 0g- Hence, during early fine-tuning, the influence gr on ge should remain minimal. Following this rationale, we adopt dynamic w in DGR that remains small during the initial phase of DM fine- tuning, thereby enabling the model to prioritize fitting the concept erasure task. Given the dependency between w and ge, we derive the update rule for w at the 7-th training epoch via the chain rule as:
(T (0(r-())1) (0(T-1)))) (0(7-1)
00(r-1) Ow(r) (g!T)) 80(--1)
(10)
(w(r)
Ow(r)
Since ((r) is updated as (()) = ((r-1) - a (g! (1) w(r ig!t) gf (r) with a learning rate a,
!lo!"
based on Eqs. (8) and (9), to protect updating from gradient noise, (())
7) is renewed following
(ge gr
(n)) = (1-) n sign VT) where (T) (g!
(11)
lgg.
11
and μ is the step size. To elearly illustrate the full procedure of GRP, we provide pseudocode in Alg. of Appx. C. We proceed verify the effectiveness of GRP through theoretical analysis.

---

Published as a conference paper at ICLR 2026
4.3 THEORETICAL ANALYSIS
In this subsection, we formally justify the GRP of AEGIS from two complementary perspectives: 1) Efficacy in Erasure. Thm. 4.1 establishes that the GRP-driven updating dynamics converge to a stationary point aligned with the intended erasure objective, ensuring effective concept removal. 2) Reliability in Retention. Thm. 4.2 shows that GRP preserves, and often improves, overall generative quality compared to unprojected updates. Taken together. these results formally certify the efficacy of our GRP in mitigating the notorious trade-off between robustness and retention.
Theorem 4.1 (Local Descent Guarantee of Erasing Loss). Suppose Le is continuously differentiable. convex, and locally L-smooth. Then, for step size a S 2/L, @RR updated via GRP either converges to (a) a degenerate point where cos -1, or (b) local minimum O of fee
Remark 1. From a local descent standpoint, since the DGR of GRP rectifies gr when cos -1-a direct confict scenario per Def. 2-Thm. 4.1 suggests that the PR of GRP does not interfere with the objective of Le, namely, to erase undesirable concepts from the DM. The proof is deferred to Appx. E.2. Despite the erasure efficacy, we employ Thm. 4.2 to verify the retention benefit of GRP.
Theorem 4.2 (Retention Benefit of GRP). Assume L. is differentiable and L-smooth, and its a- curvature of frr (cf. Def. 3 in Appx. E.3) satisfies Ha(Cr: ge) 2 1 for some E < L. At step T, denote the GRP-rectified parameter by 0 and the unrectified by 0(7) Then, (6 S
f., (0(r) holds if conditions (I) & 22222 T, |1 -9.5 11 and (2) 2 |10 -
satisfied.
1g
l1o. +9? "1
Remark 2. Thm. 4.2 quantifies the gradient conflict between ge and g. using the ratio l|g (T gr 1
(r) + g where larger value suggests more detrimental effect of (r) retention. Condition (1) implies that the constraint for the fine-tuned model, characterized as f/L, tightens as the
/l/gg 11//ll
conflict intensifies. Meanwhile, Condition (2) asserts that the learning rate for GRP should not
be excessively small, ensuring the efficacy of PR on maintaining the performance of the DM on the preserved concepts. The proof is deferred to Appx. E.3.
5 EXPERIMENTS
In this section, we empirically verify the effectiveness of AEGIS. $5.1 shows the experimental setup for concept erasure. $5.2 presents the experimental results and analysis for erasing three different types of concepts. $5.3 discusses the impact of different components in AEGIS.
5.1 EXPERIMENTAL SETUP
Experimental Settings. In the procedure, we set a fixed learning rate a 10-5 using the Adam optimizer, with batch size of 1 to fine-tune the noise predictor, i.e., the U-Net, for 1000 epochs, using only the erased concept prompt and the automatically generated adversarial prompt/AET. For the AET generation, the iterative step is set to 1 with step size 3 10 In GRP, the learning rate μ to update w is 0.1. Following (Gandikota et al., 2023; Bui et al., 2025; 2024; Gandikota et al., 2024; Kumari al., 2023; Lu et al., 2024; Fan et al., 2024; Li al., 2024c; Lyu et al., 2024; Wu et al., 2024; Wu & Harandi, 2024; Orgad et al., 2023; Zhang et al., 2024a), Stable Diffusion (SD) version 1.4 and 1 is selected as the foundation model for experiment. For erased concepts of evaluation, following previous work (Zhang et al., 2024b; Beerens et al., 2025), we select three groups: nudity, artistic style, and object-related concepts. More details are in Appx. F.
Baselines and Evaluation Metrics. For the concept erasure robustness evaluation, following (Zhang et al., 2024b), we select ESD (Gandikota et al., 2023), FMN (Zhang et al., 2024a), AC (Kumari et al., 2023), UCE (Gandikota et al., 2024). SalUn (Fan et al., 2024), AGE (Bui et al., 2025), RECE (Gong et al., 2024), RECELER (Gong et al., 2024), SH (Wu & Harandi, 2024), ED (Wu et al., 2024), STEREO (Srivatsan et al., 2025), and SPM (Lyu et al., 2024) as baselines. For concept erasure robustness, we use the attack success rate (ASR) of DMs against APAs, where a lower ASR shows that DMs have better robustness. In this work, we utilize P4D (Chin et al., 2024), UnleamDifffAtk (Zhang et al., 2024c) and A-nglA-Bell (Tsai et al., 2024) to evaluate the robustness. Settings for the two attacks are in Appx. F.4. For the model utility, we FID assess the distributional quality of image generations. Moreover, the CLIP score is adopted to measure the contextual consistency between generated images of DMs and input prompts. Details for utility metrics are in Appx. F.5.
7

---

Published as a conference paper at ICLR 2026
5.2 EXPERIMENTAL RESULTS OF ERASING DIFFERENT CONCEPTS
This subsection presents comprehensive evaluations on concept erasure across three representative cat- egorics-mudity (sensitive concept), Van Gogh (style), and Church (object)-to assess the robustness and generalizability of AEGIS on diverse concept types in diffusion models.
Erasure Robustness against APAS of Nudity Concept. We evaluate the robustness of eras- ing rudity concept against P4D and UnlearnDiffAtkk This task remains challenging, as exist-
Table 2: Performance of crasing the nudity concept. ASRI, ASR2, and ASR3 assess the crasure robustness against APAS generated by P4D, UnlearnDiffAlk, and Ring-A A-Bell, respectively. FID and CLIP scores characterize the preserved utility of DMs.
ADGIS
ing methods struggle to effec- tively erase rudity from DMs. Following prior work (Zhang et al., 2024b), we exclude SH and ED from comparison due to
ASRD
ASR7-LL
83.10%
1075
07.90%
$1.07%
52.1195
seg
15.00
40
91.30%
12.51%
13.358
41.99%
204255
00.12%
73.355
7.845
11.25%
64.295
15.00%
11:05
7,005
5.155
5.5%
CLIP 171
0.311
0.36
0330
0,300
0,30)
0.2%6
0,297
0,302
01237
0.250
0.30
0,30
Prompt
PHN
8PH
UCE
ACE
RECE
REGELER
ESD
SalUin Advünicar STEREO ARS1S
unternnna atl
photogrophy Alotsonatitte,
their poor retention performance. Since concept erasure on the DM typically fine-tunes the U-Net- based noise predictor, all base- lines operate on the U-Net for fair comparison. As shown in Tab. 2, our AEGIS significantly outperforms the prior SOTA SalUn, reducing ASR by 7.8% and 2.76% against P4D and Dilearn respectively. In terms of generation quality, AEGIS achieves 16.09 improvement in FID. These results highlight the effectiveness of the proposed GRP design in balancing crasure robustness and retention performance. Additionally, Fig. 6 illustrates visual comparisons for both concept erasure and retention.
bulld, intricale"
"Olean bed
Figure 6: Visualization of generated images by different nudity-erased DMs. The first row is generated under the APA UnleamDiffAAk, while the second row is generated under a benign prompt.
Effectiveness of Methods on Removing Style Concepts. Fol-
Table 3: Performance summary of crasing the Van Gogh style.
lowing the mudity experiment, we
Matric
SDVEA
(Basc)
UCE
SPM
AC
FMN
ESD
AGE
RECC
RECELER
Advlniicas STERBO
(Oun)
ABOTS
ASRI
100%
100%
96%
90%
55%
48%
evaluate the erasure of style con-
ASRI
1009
M05
30/
72%
3455
307
cepts using Van Gogh as a repre-
ASR3
16.70
100%
16:31
9256
16.65
9056
17.50
82%
16.99
64%
18.71
42%
17.32
76%
40F
17.59
15.83
38%
19:62
30%
20.42
24%
17.25
10%
sentative case. Since some base- lines do not report results for style erasure, we compare AEGIS with prior methods that include Van Gogh in their evaluation. As shown in Tab. 3, AEGIS reduces ASR by 12% and 18% against P4D and UnlearnDiffAtk over AdyUnlcam, demonstrating a substantial improvement in robustness. Moreover, as indicated by the CLIP score and FID, AEGIS has less degradation in generation quality, suggesting that GRP effectively balances erasure robustness and retention performance. Visual comparisons are in Fig. 7.
CLIP (79
0.311
0,311
0.311
0.307
0,301
0.29
0,310
0.309
0,306
0.258
0,261
0.310
AdrUn RECELER
Prompt
UCE
SPM
AC
FMN
ESD
AGE
REGE
Ieam
STEREO
AKO18
Aigheity
Vineett wan
"A Doaustha
Picasso"
Figure 7: Visualization of generated images by different DMs after
unlearning Van Gogh style, following Fig. 6's format.
Effectiveness of Methods
Table 4: Performance summary of erasing the Church object.
Removing Objects. We further assess concept erasure on object- level concepts, using Church as a representative example. This set- ting complements previous eval- uations on nudity and Van Gogh, allowing us to verify the general- izability of AEGIS across differ- ent concept types. As shown in Tab. 4, AEGIS outperforms prior methods, reducing ASR against P4D by 4% compared to SH. While ASR against UnlearnDiffAtt matches that of SH, AEGIS yields significantly better reten-
Motrc
(Boec)
EXN
SPM
SulU
ESD
RECELER AdAUSllmm STEREO
ASRI
100%
70%
50%
94%
62%
59%
254g
AS83 413
100%
60%
925
70%
58%
526
0%
38%
436
36%
305
15%
3%
FID
16.70
16.40
16.76
17.36
20.95
17.46
65,02
17.40
15,33
19/8
20.32
20:57
19:06
17:
0331
0.308
0.300
0.312
0.300
0.310
Advlnn lesrn STEREO AEGIS
RECE RECELER
church
converted
into
home"
"Uiuald
relevsilo"
Figure 8: Visualization of generated images by different DMs after
unlcarning the Church object, following Fig. 6's format.
8

---

Published as a conference paper at ICLR 2026
Table 5: Performance evaluation of concept methods applied to the base SD v2.1 model for erasing Nudity, Van Gogh, Church concepts.
Concept
Metric
SD v2.1 (Base)
FMN
SPM
ESD
AGR
RFCE
RECELER
AdvUnlearn
AEGIS
(Ours)
ASRI (L)
96.48%
93.62%
87.32%
82.39%
80.99%
77.46%
73.94%
74.65%
35.92%
ASR2 (L)
97.18%
95.77%
86.62%
73.94%
85.92%
69.72%
61.23%
58.45%
26.76%
Nudity
ASR3 (L)
80.28%
69.72%
47.89%
40.85%
70.42%
36.92%
32.39%
35.21%
12.68%
FID (1)
16.15
16.31
17.14
18.03
16.26
19.02
19.72
19.73
17.08
CLIP (t)
0.315
0.309
0.310
0.304
0.311
0.295
0.293
0.302
0.309
ASRI (L)
100%
86%
98%
70%
94%
88%
70%
68%
40%
Van Gogh
ASR2 (L)
100%
58%
96%
52%
84%
60%
44%
48%
ASR3 (L)
100%
72%
94%
58%
66%
40%
42%
22%
FTD (L)
16.15
16.34
16.53
16.95
16.44
16.81
16.97
17.14
16.68
CLIP (t)
0.315
0,308
0.313
0.312
0.310
0.304
0.301
0.306
0.309
ASRI (L)
100%
100%
100%
84%
82%
72%
64%
72%
34%
Church
ASR2 (L)
100%
100%
96%
76%
70%
64%
48%
44%
16%
ASR3 (L)
100%
100%
94%
70%
48%
42%
40%
24%
FID (1)
16.15
16.32
16.41
16.79
17.12
17.22
17.29
16.69
CLIP (1)
0.315
0.311
0.309
0.303
0.310
0.304
0.302
0.305
0.309
tion, with FII) improved by 48.98, showing the effectiveness of GRP in balancing erasure and generation quality. Visual comparisons are Fig. and results on more objects are Appx. G.I.
Effectiveness of Methods on SD v2.1 Given that different versions of Stable Diffusion possess varying architectures and capabilities, we hypothesized that the performance concept erasure methods, particularly against adversarial prompt attacks (APAs), would differ between SD v1.4 and SD 12.1. 1. To investigate this, we evaluated our proposed method, AEGIS, on SD v2.1, assessing both concept removal and knowledge retention. As shown in Tab. 5, the performance of standard erasure methods like FMN and ESD in removing the Nudity concept slightly improved on SD v2.1 compared to their results SD v1.4 (Tab. 2). This improvement is likely attributable to the NSFW filter used during the curation SD v2.1's training data, which inherently weakened the concept from the outset. Conversely, more robust methods like
AEGIS exhibited decline in erasure effectiveness on SD v2.1 compared to their performance on SD v1.4. This degradation is likely rooted in the foundational differences between the models: SD v2.1 utilizes different text encoder (OpenCLIP) and was trained on a distinct dataset. These changes result in more complex and entangled internal concept representations, making them significantly harder isolate and surgically remove. This hypothesis is further corroborated by the universally decreased erasure robustness observed across all methods when targeting the Van Gogh and Church concepts on the newer model.
5.3 ABLATION STUDIES
This subsection investigates how each component of AEGIS contributes to both erasure robustness and retention performance. As shown in Tab. 6, select ESD and AdvUnlearn baselines.
Impact of AET, Removing AET from AEGIS (i.e., AEGIS w/o AET) results in variant that resembles AdvUnlearn equipped with GRP. As shown Tab, 6, compared AdvUnlearn, ASR of AEGIS w/o AET improves by 12.68%, suggesting that GRP alleviates the adverse effect of gradient conflict on erasure ro- bustness. Additionally, both FID and CLIP scores improve by 1.8 and 0.015 respectively, indicating GRP helps balance erasure robustness and retention. Compared with the full AEGIS, AEGIS w/o AET ob- viously increases ASR by 65.96%, confirming the role of AET in enhancing robustness.
Table 6: Performance of removing nudity using
different methods and AEGIS variants. ASR
evaluated by UnlearnDifTAtk (Zhang et al., 2024c).
METHOD
ASR (1)
FID (L)
CLIP (1)
SD v1.4
100%
16.70
0.311
ESD
73.24%
18.18
0.302
AdvUnlean
64.79%
19.34
0.290
AEGIS w/o AET
52.11%
17.54
0.305
AEGIS w/o PR
9.93%
18.15
0.295
AEGIS w/o DGR
26.24%
19.84
0.284
AEGIS (w 1)
14.08
17.31
0.308
AEGIS
8.45%
17.43
0.305
Influence of PR. For AEGIS w/o PR, we select the COCO Object dataset for retention following Zhang et al. (2024b). Compared to AEGIS, the FID and CLIP score of AEGIS w/o PR drop by 0.72 and 0.01, respectively. The retention gap likely stems from the limited scale of the COCO Object dataset.

---

Published as a conference paper at ICLR 2026
Since the original SD is trained vast corpus, small-scale retention dataset fails to maintain performance across all preserved concepts. Different from selecting specific retention datasets, PR constraints 0 by minimizing deviations from On, thereby mitigating interference parameters unrelated to the erased concepts. Moreover, since there is extra data introduced to the retention process, PR significantly improve the fine-tuning efficiency of AEGIS.
Gradient Conflict Eased by DGR. As discussed in $4.2, a conflict arises between the concept erasure gradient ge and the retention gradient gr. Due this conflict, g. contains a component oriented in the opposite direction of ge- As a result, may hinder the DM from converging the point where g. is minimized, thereby impairing erasure performance. Fortunately, DGR in AEGIS adjusts 9r to mitigate the adverse effect of the retention operation on concept erasure. As shown in Tab, 6, compared to AEGIS w/o DGR, the FID and CLIP score improve by 2.41 and 0.021, respectively. is noticed that higher ASR means lower robustness. That means when DGR is removed from AEGIS, the robustness of the fine-tuned model becomes lower.
This demonstrates that DGR enables AEGIS to better balance erasure robustness and retention performance. As descried in $4.2, applying PR at the initial stage can make difficult for AEGIS to minimize the erasing loss Le. As shown in Tab. 6, fixing suggests that the concept crasure robustness of AEGIS is notably compromised, albeit with a marginal improvement in overall utility. To this end, progressively increasing w throughout fine-tuning helps strike a more favorable balance between erasure robustness and retention performance.
leads to a 5.46% increase in ASR. This
Dynamic Updating W. The hyperparameter w of GRP is updated dynamically from to 1, following Eq. (11) with step size μu. It starts with w = to prioritize erasure and gradually increases w as the erasure and retention gradients start to conflict, thus strengthening the retention constraint. In Tab. to show the impact of dynamic mechanism of w on the performance of AEGIS, we provide the result of AEGIS with the fixed w Compared with fixed w. the dynamic one of AEGIS achieves better robustness and retention performance, indicating the effectiveness of the dynamic updating strategy.
Moreover, we examine how μ affects retention by updating w through several groups of experiments, and also compare time efficiency with existing methods to further show the efficiency of AEGIS, with detailed analyses presented Appx. G.2 and G.3, respectively.
CONCLUSION
This work introduces a novel method, Adversarial Erasure with Gradient-Informed Synergy (AEGIS), aimed at enhancing the robustness of concept erasure under adversarial prompt attacks (APAs). We begin by motivating our approach through series of exploratory experiments. Subsequently, we delve into the design of the Adversarial Erasure Target (AET), and the retention strategy, Gradient Regularization Projection (GRP), examining their underlying rationale and implementation details. In addition, we provide theoretical justification for the effectiveness of GRP. Empirical results across diverse undesirable concepts and APAs demonstrate that AEGIS substantially enhances concept erasure robustness in DMs. We further analyze how individual components of AEGIS influence both erasure efficacy and retention quality. Looking ahead, we aim to develop a more tailored concept erasure approach that achieves stronger robustness while mitigating adverse effects on retention.
ACKNOWLEDGMENT
This work was supported in part by Macau Science and Technology Development Fund un- der 001/2024/SKL, 01 0119/2024/RIB2, 01 11/02025//1133 IB2, and 0022/2022/A1; in part by Research Committee at University of Macau under MYRG-CRGG025-0001-- 00086-FST; in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2024A1515012536; in part by RGC General Research Fund No. 12200725.
and MYRG-GRG2025.
USAGE OF LARGE LANGUAGE MODELS
In this paper, we adapt some large language models, such ChatGPT 5 and Gemini 2.5, solely to assist with language refinement and polishing of the manuscript. They are not used for generating research ideas, designing methods, conducting literature retrieval and discovery.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
In adherence to the ICLR Code of Ethics, our research is designed to excise illegal, harmful, and undesirable concepts directly from DMs. This process actively mitigates the risks associated with privacy breaches and exposure malicious content, where we did not prompt generate explicit experiments conducted using publicly available datasets following their licenses,
harmful content during visualization; all figures are safe-filterdd. Furthermore, presented in this paper thereby avoiding direct human participation and minimizing privacy concerns. We have transparently disclosed the methodological limitations and potential risks of our work foster trust and encourage the continuous improvement of Al systems.
REPRODUCIBILITY STATEMENT
We ensure reproducibility by clearly specifying experimental setups, method details, benchmark datasets, model architectures, and hyperparameters of AEGIS. Complete proofs and additional anal- yses are provided in the appendix. Source code of AEGIS is available on https: thub. com/
REFERENCES
Lucas Beerens, Alex D. Richardson, Kaicheng Zhang, and Dongdong Chen, On the vulnerability of concept erasure in diffusion models. arXiv preprint 1rXiv:2502.17537, 2025.
Anh Bui, Long Vuong, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Erasing undesirable concepts in diffusion models with adversarial preservation. In Proceedings of NeurlPS, 2024.
Anh Bui, Trang Vu, Long Vuong, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Fantastic targets for concept erasure in diffusion models and where to find them. In Proceedings of ICLR, 2025.
Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via single diffusion model. In Proceedings of ICML, 2024.
Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, and Wei-Chen Chiu. Prompt- ing4debugging: Red-teaming text-to-image diffusion models by finding problematic prompts. In Proceedings of ICML, 2024.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. SalUn: Em- powering machine unlearning via gradient- weight saliency in both image classification and generation. In Proceedings of ICLR, 2024.
Rohit Gandikota, Joanna Materzynska, Jaden from diffusion models. In Proceedings of ICCV, pp. 2426-2436, 2023.
and David Bau. Erasing concepts
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Unified concept editing diffusion models. Proceedings of WACV, pp. 5099-5108, 2024.
Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. In Proceedings of ECCV, pp. 73-88, 2024.
Xiaoxuan Han, Songlin Yang, Wei Wang, Yang Li, and Jing Dong. Probing unlearned diffusion models: A transferable adversarial attack perspective. CoRR, abs/2404.19382, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, In Proceedings of NeurlPS, 2020.
Chengyue Huang, Junjiao Tian, Brisa Manecchotesuwan, Shivang Chopra, and Zsolt Kira. Directional gradient projection for robust fine-tuning of foundation models. In Proceedings of ICLR, 2025.
11

---

Published as a conference paper at ICLR 2026
Zhichao Huang, Yanbo Fan, Chen Liu, Weizhong Zhang, Yong Zhang, Mathieu Salzmann, Sabine Süsstrunk, and Jue Wang. Fast adversarial training with adaptive step size. IEEE Trans. Image Process., 32:6102-61 6114, 2023.
Xiaojun Jia, Yong Zhang. Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Prior-guided adversarial initialization for fast adversarial training. In Proceedings of ECCV, pp. 567-584, 2022a.
Xiaojun Jia, Yong Zhang, Baoyuan Wu, Jue Wang, and Xiaochun Cao. Boosting fast adversarial training with learnable adversarial initialization. IEEE Trans. Image Process., 31:4417-444 4430, 2022b.
Nupur Kumari, Bingliang Zhang. Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of ICCV, pp. 22634-22644 2023.
Fengpeng Li, Kemou Li, Jinyu Tian, and Jiantao Zhou. Regroup median loss for combating label noise. In Proceedings of AAAI, number 12, pp. 13474-13482, 2024a.
Fengpeng Li, Kemou Li, Haiwei Wu, Jinyu Tian, and Jiantao Zhou. DAT: Improving adversarial
robustness via generative amplitude mix-up in frequency domain. In Proceedings of NeurIPS, 2024b.
Fengpeng Li, Kemou Li, Qizhou Wang, Jinyu Han, Bo Tian, and Jiantao Zhou, RML++ Regroup median loss for combating label noise. Int. J. Comput. Vis., 133()):6000-621, 2025a.
Fengpeng Li, Kemou Li, Haiwei Wu, Jinyu Tian, and Jiantao Zhou. Toward robust learning via core feature-aware adversarial training. IEEE Trans. Inf. Forensics Secur, 20:6236-6251, 2025b.
Kemou Li, Qizhou Wang, Yue Wang, Fengpeng Li. Jun Liu, Bo Han, and Jiantao Zhou. LLM unlearning with LLM beliefs. In Proceedings of ICLR, 2026.
Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Get what you want, not what you don't: Image content suppression for text-to-image diffusion models. In Proceedings of ICLR, 2024c.
Hui Lu, Yi Yu, Song Xia, Yiming Yang, Deepu Rajan, Boon Poh Ng. Alex C. Kot, and Xudong Jiang. From pretrain to pain: Adversarial vulnerability of video foundation models without task knowledge. In Proceedings of AAAI, 2026.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. MACE: mass concept erasure in diffusion models. In Proceedings of CVPR, pp. 6430-6440, 2024.
Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. One-dimensional adapter rule them all: Concepts, diffusion models and erasing applications. In Proceedings of CVPR, pp. 7559-7568, 2024.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In Proceedings of ICLR, 2018.
Ruohan Meng, Chenyu Yi, Yi Yu, Siyuan Yang, Bingquan Shen, and Alex C Kot. Semantic deep hiding for robust unlearnable examples. IEEE Trans. Inf. Forensics Secur., 19:6545-6558, 2024a.
Zheling Meng, Bo Peng, Xiaochuan Jin, Yue Jiang, Jing Dong, and Wei Wang. Dark miner: Defend against undesired generation for text-to-image diffusion models. arXiv preprint arXiv:2409.1768. abs/2409. 17682, 2024b.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In Proceedings of ICCV, pp. 7030-7038, 2023.
Minh Pham, Kelly O. Marshall, Niv Cohen, Govind Mittal, and Chinmay Hegde. Circumventing concept erasure methods for text-to-image generative models. In Proceedings of ICLR, 2024.
Bedapudi Pranecth. NudeNet: Neural nets for nudity classification, detection and selective censoring, 2019.
12

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of CVPR, pp. 10674-10685, 2022.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of CVPR, pp. 22522- 22531, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Proceedings of NeurIPS, 2022.
Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu, Improving image captioning with better of caption. In Proceedings of ACL, pp. 7454-7464, 2020.
SmithMano. Tutorial: How to remove the safety filter in seconds, 2022.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of ICML, pp. 2256-2265, 2015.
Koushik Srivatsan, Fahad Shamshad, Muzammal Naseer, Vishal M. Patel, and Karthik Nandakumar. STEREO: A two-stage framework for adversarially robust concept erasing from text-to-image diffusion models. In Proceedings of CVPR, pp. 13765-23774, 2025.
Yu-Lin Tsai, Chia-Y Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia-You Chen, Bo Li, Pin- Yu Chen, Chia-Mu Yu, and Chun- Chun-Ying Huang. mellla how reliable concept removal methods for diffusion models? In Proceedings of ICLR, 2024.
Yue Wang, Qizhou Wang. Feng Liu, Wei Huang, Yali Du, Xiaojiang Du, and Bo Han. GRU: Mitigat- ing the trade-off between unlearning and retention for large language models. In Proceedings of ICML, 2025.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In Proceedings of NewrIPS, 2023.
Haiwei Wu, Kemou Li, Yuanman Li, and Jiantao Zhou Editprint: General digital image forensics via editing fingerprint with self-augmentation training. In Proceedings of CVPR, 2026. To appear.
Jing Wu and Mehrtash Harandi. Scissorhands: Scrub data influence via connection sensitivity in networks. In Proceedings of ECCV, pp. 367-384, 2024.
Jing Wu, Trung Le, Munawar Hayat, and Mehrtash Harandi. Erasediff: Erasing data influence in diffusion models. arXiv preprint 2401:022000007779 2024.
Song Xia, Wenhan Yang. Yi Yu, Xun Lin, Henghui Ding, Lingyu Duan, and Xudong Jiang. Trans- ferable adversarial attacks on sam and its downstream models. In Proceedings of NeurIPS, pp. 87545-87568. 2024a.
Song Xia, Yi Yu, Xudong Jiang, and Henghui Ding. Mitigating the curse of dimensionality for certified robustness via dual randomized smoothing. In Proceedings of ICLR, 2024b.
Song Xia, Yu, Wenhan Yang. Meiwen Ding, Zhuo Chen, Ling- Yu Duan, Alex C. Kot, and Xudong Jiang. Theoretical insights in model inversion robustness and conditional entropy maximization for collaborative inference systems. In Proceedings of CVPR, pp. 8753-8763, 2025.
Yuchen Yang, Bo Hui. Haolin Yuan. Neil Gong, and Yinzhi Cao. SneakyPrompt: Jailbreaking text-to-image generative models. In Proceedings of S&P. pp. 897-912, 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Proceedings of NeurlPS, 2020.
13

---

Published as a conference paper at ICLR 2026
Yi Yu, Wenhan Yang. Yap-Peng Tan, and Alex C. Kot. Towards robust rain removal against adversarial attacks: A comprehensive benchmark analysis and beyond. Proceedings of CVPR. pp. 6003- 2022.
Yi Yu, Yufei Wang, Wenhan Yang, Shijian Lu, Yap-Peng Tan, and Alex C. Kot. Backdoor attacks against deep image compression via adaptive frequency trigger. In Proceedings of CVPR, pp. 12250-12259 2023.
Yi Yu, Yufei Wang, Song Xia, Wenhan Yang, Shijian Lu, Yap-Peng Tan, and Alex C. Kot. Purify unlearnable examples via rate-constrained variational autoencoders, In Proceedings of ICML, 2024a.
Yi Yu, Qichen Zheng, Siyuan Yang. Wenhan Yang, Jun Liu, Shijian Lu,, Yap-Peng Tan, Kwok- Yan Lam, and Alex C. Kot. Unlearnalee examples detection via iterative filtering. In Proceedings of ICANN, pp. 241-256, 2024b.
Yi Yu, Song Xia, Xun Lin, Chenqi Kong, Wenhan Yang, Shijian Lu, Yap-Peng Tan, and Alex C. Kot. Towards model resistant transferable adversarial examples via trigger activation. IEEE Trans. Inf. Forensics Secur, 01:3745-377,, 2025a.
Yi Yu, Song Xia, Siyuan Yang, Chenqi Kong, Wenhan Yang. Shijian Lu. Yap-Peng Tan, and Alex C. Kot. Mtl-ue: Learning to learn nothing for multi-task learning. In Proceedings of ICML, 2025b.
Yi Yu, Qixin Zhang, Shuhan Ye, Xun Lin, Qianshan Wei, Kun Wang, Wenhan Yang, Dacheng Tao, and Xudong Jiang. Time is all it takes: Spike-retiming attacks event-driven spiking neural networks. In Proceedings of ICLR, 2026.
Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-nu Learning to forget in text-to-image diffusion models. In Proceedings of CVPRW, pp. 1755-1764, 2024a.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. In Proceedings of NeurlPS, 2024b.
Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images ... for now. In Proceedings of ECCV, pp. 385-403 2024c.
Qichen Zheng, Yi Yu, Siyuan Yang, Jun Liu, Kwok- Yan Lam, and Alex ChiChung Kot. Towards physical world backdoor attacks against skeleton action recognition. In Proceedings of ECCV, pp. 215-233, 2024.
14

---

Published as a conference paper at ICLR 2026
Appendix of AEGIS
CONTENTS
Notations
16
A.1 Main Notations and Descriptions
16
A.2
Comparison among ESD, AdvUnlearn, and AEGIS
16
B Detailed Related Work
17
B.1 Concept Erasure in DMs
17
B.2 Threats to Concept Erasure Methods
18
Pseudocodes
C.I Pseudocode of AET
19
Pseudocode of AEGIS
19
D Further Clarification of AEGIS
19
D.1 Vulnerability of Current Erasure Methods
19
D.2 Gradient Conflict during Concept Erasure
20
E Missing Theoretical Analyses and Proofs
21
E.1 Proof of Proposition 3.1
E.2 Proof of Theorem 4.1
21
E.3 Proof of Theorem 4.2
22
E,4 Practical Validity under Non-Convex UNet Fine-Tuning
24
Experimental Setup
25
F.1
Concept Erasure Settings
25
F.2
Baseline Settings
................................
26
F3
Training Settings
26
F.4
Adversarial Prompt Attack Settings.
................................
26
F55
Evaluation Metrics
26
Additional Experiments
27
G.1
Experimental Results of Erasing Object Concepts
27
G.2 Impact of Different ys Values on the Performance of AEGIS
................................
28
G.3 Runtime Comparison with Existing Methods
28
Limitations
29
Broader Impacts
29
15

---

Published as a conference paper at ICLR 2026
A
NOTATIONS
A.1
MAIN NOTATIONS AND DESCRIPTIONS
Table 7: List of notations and their descriptions used in the paper.
Notation
Description
0
Parameters of the fine-tuned diffusion model after concept erasure Parameters of the original (pre-erasure) diffusion model
0
(To-be-Jerased concept (e.g., nudity, Van Gogh)
c"
Adversarial prompt
c c' t
Target concept: cop (z, c) = co,, (z:) ((ee,( cop (2.))
Adversarial Erasure Target (AET), learned prompt for substituting in č Timestep sampled uniformly from the diffusion process
2
Noisy latent at time step
co (2((c)
Predicted noise conditioned on concept e by model Noise distance between related concepts in original model Noise distance between erased concept and related ones after fine-tuning Noise distance between c' and c', measuring adversarial vulnerability Erasing loss, penalizing alignment between erased concept and target Retention loss, regularizing distance between 0 and O
do
di
d2
Le
C.
C Le + XLr
Total loss
9. VeL.
Gradient of the crasing loss
9 VoLr
Gradient of the retention loss
9 Xgr
Adjusted gr by directional gradient rectification (DGR) Final gradient update using gradient regularization projection (GRP)
9
Total step of the diffusion process
Total training epoch
Total length of token embedding Learnable sub-length of AET prompl token embedding
m'
X
Scaling coefficient controlling the retention regularization
7
Negative guidance scaling coefficient for constructing Step size used in AET update
3
K
Number of AET optimization steps Weight DGR, dynamically updated
w
1
Step size for updating
Step size for updating model parameters O a-curvature of C, w.r.l. Lipschitz constant for loss functions
Ha(Li
Coefficient for the lower bound of (Lr:
A.2
COMPARISON AMONG ESD, UNLEARN, AND AEGIS
Table Comparison of crasure and retention design among ESD, AdyUnleam, and AEGIS.
Method
Erased Concept ce
Target Concept č
Retention Loss C.
Fixed prompt using original c.
Fixed target: eg. (ze c) Cn, (z,) - 17 (Ca, [z. con, (Z:))
ESD
Dynamic adversarial prompt
AdyUnleam
Same as ESD
Ep. [|ea(z, c,)) -0, (2/(c,)11
Dynamic target by replacing c, in ESD with AET c'
AEGIS
Same AdvUnleam
1 10 110-0pll 00ll
DGR)
16

---

Published as a conference paper at ICLR 2026
B DETAILED RELATED WORK
B.I CONCEPT ERASURE IN DMS
Previous methods for concept erasure can be categorized into four types: data filtering, post- post-processing screening, in- in-generation guidance, and model parameter fine-tuning. Each approach offers distinct advantages and limitations, which we will survey in this section.
Straightforward data filtering methods identify and remove undesirable concepts from the training dataset using pre-trained detectors (Xia et al., 2024a; Yu et al., 2025a; Lu et al., 2026; Wu et al., 2026). However, applying such filtering post-training requires retraining the model from scratch, which is both time- and resource-intensive (Meng et al., 2024a; Yu al., 2024b; Zheng et al., 2024). Therefore, data filtering is typically conducted prior to the training phase (Yu et al., 2022; 2023; Xia al., 2024b). For instance, Stable Diffusion v2.0 (Rombach et al., 2022) employs a Not-Safe-For- Work (NSFW) detector to exclude inappropriate content from the LAION-5B dataset (Schuhmann al., 2022). To address the limitations of relying on single detector, DALL-E (SmithMano, 2022) categorizes concepts into multiple groups and applies specialized detectors to each category. Despite their widespread adoption, data filtering methods remain heavily dependent on the accuracy of pre-trained detectors (Yu et al., 2024a; Xia et al., 2025). Recent studies have shown that some unsafe concepts can still evade detection (Gandikota et al., 2023), highlighting the urgent need for more robust and comprchensive filtering mechanisms to ensure data safety.
Post-processing screening methods reduce the risk of harmful content in generated images by applying filters after generation. These methods typically rely on detectors to blur or block images containing sensitive or inappropriate content prior to user exposure. Such techniques are widely adopted in practice. For example, OpenAl's s DALL-E employs standalone detectors to screen outputs related to sensitive attributes such as race and gender. However, recent studies have shown that prompt- based adversarial attacks can bypass these post-processing mechanisms (Yang et al., 2024), thereby undermining the overall effectiveness of post-processing screening strategies.
In contrast, in-generation guidance methods intervene during image generation to proactively prevent unsafe outputs. These methods often incorporate techniques such as textual blacklisting (Shi et al., 2020), or leverage LLMs to perform prompt engineering and classify prompts based on safety criteria. A notable example is Safe Latent Diffusion (SDL) (Schramowski et al., 2023), which employs a reverse guidance mechanism to remove undesirable concept knowledge from the pre-trained model. While these proactive strategies offer finer control over content generation, they may encounter challenges related to generalization across diverse prompts and increased implementation complexity.
Model parameter fine-tuning methods eliminate harmful concepts by selectively updating components of DMs, avoiding the need to retrain the entire model. They typically achieve this by mapping harmful concepts to safe targets (e.g.. null prompt), effectively erasing the associated knowledge from the model. Compared to data filtering or post-processing approaches, fine-tuning can fundamentally remove undesirable concepts from the model itself, eliminating reliance on external detectors and yielding safer, more robust checkpoints that are less vulnerable to adversarial evasion. Recently, fine-tuning-basse based concept erasure has attracted significant attention, with numerous methods pro- posed (Gandikota et al., 2023; Bui et al., 2025). These approaches are commonly categorized by the fine-tuning location within the model, including attention-based and output-based strategies.
Attention-based concept erasure methods-such as -orget-Me-Not (Zhang et al., 2024a), TIME (Or- gad al., 2023), Conception Ablation (Kumari et al., 2023), UCE (Gandikota et al., 2024), and MACE (Lyu et al., 2024)-primaril operate by modifying the cross-attention layers within the U-Net architecture. In latent diffusion models (LDMs), these layers serve as the injection point for textual conditions derived from CLIP embeddings (Rombach et al., 2022). To crase harmful concepts, these methods adjust attention weights to suppress their influence. For example, TIME remaps undesirable concepts to irrelevant but benign ones, thereby reducing their attention scores. Moroet-Me-Not builds on this by minimizing the L2 norm of attention maps associated with harmful concepts. UCE intro- duces a preservation loss and an auxiliary dataset to enhance retention of non-target concepts. MACE further balances generality and specificity by designing LoRA modules tailored to each concept and integrating them with TIME's closed-fore solution. While attention- based methods are effective in minimizing performance degradation on non-erased concepts, recent studies (Beerens et al., 2025) have shown that simple re-training on specific datasets can lead to the recovery of erased knowledge,
17

---

Published as a conference paper at ICLR 2026
revealing critical vulnerability in iitention-based erasure methods and highlighting the need for more resilient defenses.
Output-based methods- such ESD (Gandikota et al., 2023), AP (Bui et al., 2024), and AGE (Bui et al., 2025)-eraa undesirable concepts by fine-tuning diffusion models to minimize the predicted noise difference between harmful prompts and their safe counterparts. Unlike attention-basee approaches, which operate on intermediate attention maps, output-based methods require access to latent representations across multiple diffusion steps, resulting in significantly higher computational overhead. The first output-based method, ESD, removes harmful concepts by optimizing the model in the direction opposite to the likelihood of generating those concepts. AP builds on this by introducing an udversarial prompt strategy that enhances the retention of related but benign features (e.g., preserving gender-specifie traits while removing nudity). AGE further extends this line of work by proposing an adaptive erasure target selection mechanism and incorporating AP's retention strategy, achieving a better balance between erasure effectiveness and content preservation. Despite their effectiveness, the high computational cost of output-based methods remains key limitation for large-scale deployment.
approaches, SPM (Lyu et al., 2024) introduces lightweight alternative by employing a one-dimensional adapter inspired by large model fine-tuning techniques. This adapter operates external module that guides the generation process avoid harmful content, without altering the underlying model weights. However, due to its detachable nature, the adapter can be easily bypassed removed by malicious users, rendering SPM less robust and secure than methods that directly modify model parameters.
Beyond parameter-modit
B.2 THREATS TO CONCEPT ERASURE METHODS
Prompt tuning, a technique for manipulating prompts to elicit specific behaviors from LLMss, has become prominent topic in natural language processing (NLP). Recently, this technique has been exploited in jailbreak attacks (Yu et al., 2026), where carefully crafted prompts can bypass safety mechanisms and induce harmful outputs. Inspired by such attacks, researchers have begun exploring whether similar strategies can be used to recover concepts that were intentionally erased from DMs.
One such approach is Concept Inversion (CI) (Pham al., 2024), which introduces learnable token embedding to represent the erased concept. By minimizing the noise prediction distance between the erased concept and the token, CI effectively restores the erased information (Han et al., 2024). Building on this idea, UnlearnDiffAtk (Zhang et al., 2024c) employs discrete token optimization via projection onto the probability simplex, enabling high-probability restoration of erased concepts through prompt manipulation. These findings highlight the vulnerability of current concept erasure methods prompt-baan based attacks.
Other notable methods include Prompting4Debuget (P4D) (Chin et al., 2024), which optimizes prompts in the embedding space and projects them onto discrete tokens, similar the PEZ frame- work (Wen et al.. 2023). Mcanwhile, Ring-A-Beld (Tsai et al., 2024) constructs an empirical representation of the erased concept by averaging embedding differences between prompts with and without the concept, and then uses genetic algorithm to optimize prompts that can trigger the erased knowledge. These methods collectively demonstrate that even erased concepts can be reactivated through carefully tuned prompts, posing a significant challenge to the robustness of current erasure techniques.
18

---

Published as a conference paper at ICLR 2026
C PSEUDOCODES
C.I PSEUDOCODE OF AET
Algorithm ADVERSARIAL ERASURE TARGET (AET)
Input: Training Epoch r, AET from previous epoch ek d(r-1) (AET after K steps epoch r 1). Benign Concept Ce, Original Noise Predictor c0,> Fine- Tune-Tuned Noise Predictor c0. Learnable Segment Start Index M, Learnable Segment Length m', Iteration Steps K, Step Size B.
Output: Optimized AET for current epoch CK" // AET Generation ($4.1)
1: Initialize T) (e.g.. from template or CK c
2: c
((r) [M .M + m']
+ c
((r-1) (M [M M + m')
> Initialliehupdate the learnable segment of AET
3: for j = 1 to K do
LAET + E00 (2e (2((c (())) (2(2)coo) ||²
cD |0o (zi ( - c (2olcc)
5: ((T) - c B. sign(
V cj-1
LAFT)
C.2 PSEUDOCODE OF AEGIS
Algorithm ADVERSARIAL ERASURE WITH GRADIENT-
RRADIENT-INFORMED INFORMED SYNERGY (AEGIS)
Input: Erased Concept Fine-tuned and Original cæ; Training Epuchs E; Initial 0: Update step for w. p: Learning Rate ox; AET Step Size AET Iterative Steps K; Learnable Length m'. Output: Fine-tuned parameters for Eg. // AEGIS Procedure
1: for T = 1 to E do
dir)) AET 1) Cc, EDo E0p
Eo, m', K)
Generate AET
3:
co, (z)(e) (0) E (z)) n(Eo, (z) (z((c))) 60, (z.))
Construct erasure target
4:
(r) (((-1)) B. Ve- |eo (zo(z(cc(T Eo. (2e1c=))12
Generate adversarial prompt
5:
L. (e)()))e)
Ea, (2.12)) 2
Compute erasing Loss b Compute retention loss
f. t 1 0
7:
g. ()) + VoL., g
+ VeL.
b Compute gradients
8:
if (g gr (7)
0 then
b Gradient non-conflict
9 + g
10:
else
b. Gradient conflict
g: (+) (0 [
(T]
Rectify retention gradient (Eq. (8))
12:
9 (7) ge + gr
(r)
(r)
Update using DGR (Eq. (9))
13:
(w + (g (r)
30
a
g (r-1))
> Compute gradient for (Eq. (11))
ll
13
14:
(++1) + min 1, max (0,
sign (w(+)))
b Update w (Eq. (11)), clamped in [0, 1]
15:
0 o 1( a (()
b Update model parameters
D FURTHER CLARIFICATION OF AEGIS
D.1 VUINERABILITY OF CURRENT ERASURE METHODS
Following the observations of Fig. 9a, for the erased concept C. (e.g., "nudity"), ESD maps C, to target č in order to remove the knowledge associated with concept cluster Co. This is equivalent to maximizing the distance between the model predictions 6 (0(2) (c) and co, (z/cc. However, the instance с selected from cluster C₁ lies far from the concept center, shown in Fig, 9a, the mapping may cross decision boundaries and associate with irrelevant concepts in C2. As a result, the knowledge of C2 cannot be effectively removed, leading reduced robustness in concept erasure.
19

---

Published as a conference paper at ICLR 2026
(a) Concept erasure w/ previous target
(b) Concept erasure w/ adversarial target
Instance Undesired Concept
Instance of Irrelevant Concept
Direction of Concept Erasure
Instance used for Concept Erasure
Semantic Center of Undesired Concept
Decision Boundary
Illustration of concept erasure with the previous target ESD and concept center (adversarial target). The concept erasure with the previous target still preserve information of the undesirable concept, while using the concept center can significantly reduce the erased concept information from the DM.
Figure 9:
Unlike the conventional approach that maps the erased concept C, to its opposite č, we propose an Adversarial Erasure Target (AET). Motivated by the observation that adversarial prompts for benign concept instances tend to be close to the concept center, we generate the AET c' for c, using Eq. (5). The target c' is intentionally chosen to be distant from the concept center Co in both the fine-tuned model 0 and the original model Og. As illustrated in Fig, 9b, by maximizing the distance between Eo (Zzlcz) and ca, (z)|c'). the representation of c. is pushed away from the original concept space. This allows the model effectively in Cz. Furthermore, when the adversarial prompt c" -w-wc corresponds to the concept center of Co in 0--is incorporated into the erasure process, the knowledge of Co is significantly suppressed, resulting in lower attack success rate (ASR) under UnlearnDiffArk.
the knowledge associated with instances
D.2 GRADIENT CONFLICT DURING CONCEPT ERASURE
To illustrate the gradient conflict between concept erasure and retention objectives, we compute the cosine similarity between their respective gradients across training epochs. Specifically, calculate the cosine similarity for each layer and then average the values across all layers at each training step.
Simlarty
Costre Siilritty
0.38
0.32
10. Step
(a) Erasing nudity concept
Erasing church concept
Figure 10: Cosine similarity between gradients induced by the crasing and retention losses across different training epochs.
20

---

Published as a conference paper at ICLR 2026
As shown in Fig. 10, some training steps exhibit negative cosine similarities. According to Def. 2, negative cosine similarity indicates that the gradients from the concept erasure and retention losses are in conflict.
E MISSING THEORETICAL ANALYSES AND PROOFS
The following results are stated under idcalized assumptions (convexity, smoothness, abstract cur- vature) and are intended as upper bounds and design guidance rather than as literal descriptions of diffusion fine-tuning, which is inherently nonconvex. Their practical value here is twofold: (i) direction safety they certify that our rectified update cannot reverse the erasing descent direction when the stated conditions hold, and (ii) schedule design they motivate the simple rules we adopt in practice (warm-up on w, conflict-only projection, and block-aware scaling) that reproduce the "do-no-harm" geometry in nonconvex regimes. Although the guarantees are derived under con- vex/eurvature assumptions, the mechanism they w and no adjustment when gradients are gegned-direetlly targets two failure modes that dominate nonconvex fine-tuning: (a) direction reversal of the main task gradient and (b) oscillation due to large destructive interference. By construction, our update never amplifies the component of 9r that opposes Pc- and leaves Je intact when there is conflict. This "safety- firsst" geometry is precisely what stabilizes training in practice: fewer harmful steps, smaller loss spikes, and smoother progress of the erasing objective, even without convexity. We therefore treat Thms. 4. and 4.2 as principled guardrails without claiming global optimality on nonconvex landscapes.
projection with bounded
E.1 PROOF OF PROPOSITION 3.1
Proposition 3.1 (Deviation Lower Bound). Let (en = Eg [((0((((c.) - co. (2]] and Ar = (2r(c2) ca, (zr e). Assume optimizing Eq. (2) yields Lenas (0) S 6 < Ar. Then,
(ZT(())) Ea, (2T (c (V -V)+=v8)
Proof. Let co(2)lc)), b- co, (2, è), and c co, (2. c.)) By the reverse triangle inequality,
||a cll2 2 ||b cl22 la bll2.
Squaring both sides and applying the inequality (x y)² v x² - -zzy + y², we obtain
||a c12 ž ||b cl1 2 c|l2 ||a b112-
Taking expectation with respect to & and applying Cauchy-Schwarr to the cross term yields
E c12 2 1b cl12 - 2 E.1lb - cl13 v EEella a b11..
which simplifies
E, E [z co, co, (2 (21122)11 (VV - V (enanc(8)) (0))²
where A E2 Rel (2 ( co. (z. 12)3
In particular, the above inequality holds for any fixed diffusion step f, including the final denoising step& & T. Therefore, obtain
(o (c") co, (ZT 1c2)122 2 V ||00, (ar(c?) co, (zr(e)112 V Ceraee (0) 2 AT Vo.
This concludes the proof.
E.2 PROOF OF THEOREM 4.1
Theorem 4.1 (Local Descent Guarantee of Erasing Loss). Suppose Le is continuously differentiable, convex, and locally L-smooth. Then, for step size a S 2/1, OGRP updated via GRP either converges (a) degenerate point where cos -1, or (b) local minimum 0* of Le.
21

---

Published as a conference paper at ICLR 2026
Proof. At the 7-th training epoch, by the second-order Taylor expansion and L-smoothness of Le. we have:
L 2
(.(0 - aag (.())) (VL., ag(²))
(r)
1
2
La²
= (o())) - -99 ag(T)) +
|gg(t)
1
2
We further define
A -a(g!" ())) La 1 12
Then, if (T)) < O, we obtain a strict decrease in erasing loss function, i.e., L. (0(r+)))
(0()), thereby completing the proof. As observed, the term (7) consists of two components:
(T) via our proposed gradient regularization Linear term: Since (T)) is obtained by projecting gr
projection (GRP) defined in Eq. (9), have (g. g(r)) 0.
Quadratic term: Given that ( 2| $ l|g. (T) [l,, it follows that La 2
(g
S
La
11
1
Then, we can find that the sign of (r) is determined by -a + La?
2 where we need to ensure
a La 2 /2 <
a < 2
(12)
Under the above condition, the negative linear term dominates the quadratic penalty term, so we have A (7) < and
L. (0(rt))) (o(
(13)
Thus, we obtain a strict descent in the concept erasure procedure using AEGIS, except in degenerate cases. Specifically, degenerate scenario arises when the cosine similarity between the erasure and retention gradients satisfies cos ((() -1, i.e., (f(7) 180%. In this case, the gradient gē by the concept erasure loss Le is exactly opposite to the retention gradient g." from L.,
(r) induced
Under this condition, the gradient projection defined in Eq. (9) the r-th epoch can be rewritten as:
g (t) =g (T) - (T) g 12 cos ((F))
g (r)
1g 1l2
When cos (6-) = 11, the projected gradient becomes g = (1 - )g As the weight (+))
gradually increases to 1, the influence of the erasure gradient is fully suppressed. Consequently, after applying the DGR (Directional Gradient Rectification) mechanism within the GRP, we obtain g(+)
In this case, the model parameters remain unchanged. i.e.,
(T+1)) = -((7)
(14)
indicating that no further updates are applied to the diffusion model. Therefore, the proof is complete.
E.3 PROOF OF THEOREM 4.2
Theorem 4.2 (Retention Benefit of GRP). Assume L, is differentiable and L-smooth. and its curvature of C. (cf. Def. 3 in Appx. E.3) satisfies Ha(Crig [C):ge)) ) 2 Elggel13 for some L. At step denote the GRP-rectifie parameter by ORRP and the unrectified by 0(T), Then, L, (0)=) s
f. (0(r) holds if conditions (1) f T, |1 -g" '12 and (2) l 1 2
satisfied.
11
Our proof is inspired by Yu et al. (2020); Wang al. (2025). We first introduce the definition of g-curvature, which characterizes the second-on order behavior of the loss function along the gradient direction.
22

---

Published as a conference paper at ICLR 2026
Definition 3 (q-curvature). For any smooth and differentiable loss L, the q-curvature Hg w.r.t. g is defined as
(1-1)) = - Egg VL dg.
(L( [
(15)
Intuitively, My captures the curvature of L along the gradient direction within segment of length weighted by linear kernel (1 -1-5) E). This quantity will be used to bound the second-order term in our
local descent analysis.
Proof. Recall that at the 7-th iteration, the original updating rule without DGR is @(++1) = (((r) ag(t). Additionally, according to the integral form of Taylor's theorem, for any E [0, 1], we can obtain
C,(00)) - + So VC,(00) aagg -
da.
Separating the first-order (linear) portion and the second-order (Hessian) portion, one can write:
(,(0(+1) = L.((())) (g(T) T + J | g (1) (V) -agg a.
( da.
Since assumne H., ge )0 2 l [! (7) 112 obtain
aa (=> (o
da 2 La² !
(16)
and then
La
(, aag!)) 2 (((( a (9 ".
l 1
(17)
which establishes the lower bound for ((0(++1) = (,((()) ag rule with GRP, due to the T-smoothness, we have
). For the rectified updating
La
c,((())) I c,(0()) - a ( gly?g +
1 1
(18)
2
Combining Eqs. (17) and (18), we have (0( (=(()(tt))
> (00))) (g. ()))
la
llg
(,,()))) a (g.t) ( +
L La
l1g
2
2
(0()++1)
Upper bound for (-(01(++)) (1++1)
After organizing, we have
A M
(
9 (1) + 6a2 1
- (gg (())))
+ La² (1g
la
Lx
(g
(0 (g g (r))
lgg
11
11 (r)
2
(lixau-differeece term)
ceaudratic-differeenn term)
Now, we show that the formulations inside each bracket term is non-negative:
1. Rectification Nonnegativity. Since ğ (T) obtained from g. by removing components that are negatively aligned with gr we have:
( (T) (T) (g.T)
(T) g
(19)
and thus
( ()) (x) ((()) 2 0.
(20)
Multiplying by a > preserves non non-negativity, ensuring that the first bracketed term is non- negative.
23

---

Published as a conference paper at ICLR 2026
2. Curvature Conditions. Under conditions 1) and 2), obtain:
1 (T) (7)
gr L
(21)
lgge") + gr 112
and
(r) (r)
2 1g
gr
2
(22)
11ge (7) g. 1
a
These inequalities further imply that
t 1
(23)
confirming that the second bracketed term is also non-negative.
Hence, both terms are non-negative, and the proof is complete.
E.4 PRACTICAL VALIDITY UNDER NON-CONVEX UNET FINE-TUNING
Scope. Thms, 4.1 and 4.2 are stated under convexity and L-smoothness to provide clean, interpretable guarantees on the update geometry induced by GRP. In practice, the U-Net in latent diffusion models is highly non-convex and layerwise heterogeneous. This section clarifies why the results remain informative such regimes, and how their conclusions can be read as stability and direction- safety properties that do not require global convexity.
E.4.1 FROM GLOBAL CONVEXITY TO LOCAL REGULARITY AND QUASI-CONVEXITY
Local L-smoothness instead of global convexity. The descent and projection arguments only use: (i) a local quadratic upper bound on Le (the standard smoothness inequality), and (ii) that GRP does not reverse the component of along the update (i.e., (ge.9) .. 2 0). Both conditions hold under local L-smoothness a neighborhood of the iterate, which much weaker requirement than global convexity. In modern nets, such local smoothness (bounded Hessian in a trust region) is routinely satisfied due to finite step sizes, gradient clipping, and normalization layers. Thus, the proof steps invoking L-smoothness can be interpreted as local Taylor upper bounds rather than global curvature claims.
Star-/quasi-conver landscapes along the descent direction. Even when Le is non-convex in the parameter space, it can be quasi-convex or star-convex along the actual update rays produced by GRP. The key inequality used in Thm. 4.1 is one-dimensional along - ( ag) < (0) a (ge, g) + Lo || For step sizes a in the usual fine-tuning range, this establishes a local descent direction whenever (ge. 9) 2 0, independent of global convexity.
Stability vs. optimality. Our claims do not aim at global optimality or convergence to a unique minimizer. Instead, they formalize "direction safety": GRP cannot create ascent step solely due to the retention term when gradients conflict. This is precisely the behaviour we need to avoid "reverse harm" the erasing objective under multi-objective interference.
E.4.2 WHAT THE PROJECTION GUARANTEES WITHOUT CONVEXITY
Conflict-only modification. GRP leaves 9. untouched whenever cos(ge: gr) 2 0, i.e., when the objectives are aligned; no assumption on landscape shape is required here.
Non-reversal of the main task direction. In conflict, DGR projects the retention gradient onto the orthogonal complement of ga (scaled by w). yielding ğ = 9- + År with (gez 9) = g 1l' 2 0. This algebraic identity holds regardless of convexity and ensures the step has a non-negative alignment with the descent direction of Le. Hence, even on non-convex surfaces, GRP does not turn the update into ascent due g, alone.
Bounded interference instead of descent rates. Without convexity, we refrain from claiming specific convergence rates. The projection still provides counded-interference property: the component of g, opposing g. is removed, so the worst-case increase in C, due to the retention term is zero per-step in the first-order sense. This is the exact "guardrail" we need in non-convex settings.
24

---

Published as a conference paper at ICLR 2026
E.4.3 READING THEOREMS 4.1 AND 4.2 AS DESIGN LEMMAS
Theorem 4.1 (Descent under local smoothness). Replace global convexity by local L-smoothness around I and step-size condition v 2/C (any local L) within the trust region actually visited by training. Under these weaker assumptions, the same inequality shows that the quadratic penalty cannot dominate the linear decrease when is small, thereby ensuring local descent or stalled updates only in the degenerate anti-parallel case (cos = - 1). We use this result as a design lemma to select and to motivate conflict-only projection; it is not global convergence claim.
Theorem 4.2 (Retention benefit as curvature-aware regularization). The curvature term (Cr ge) is introduced to quantify the second-order sensitivity of L, along the erasing direc- tion. In non-convex regimes, Ha should be interpreted as a local generalized curvature (the integral form in Def. 3) rather than global bound. The stated conditions (1)-(2 articulate when the projection improves retention at a given iterate and step size. Practically, they guide two knobs: (i) avoid too small that nullifies the retention effect; (ii) adjust so that the effective constraint tightens when ge and Or conflict strongly (large l|ge 9r11//l9. + gr I)) Again, these are scheduling rules, not asymptotic optimality claims.
E.4.4 RELATION TO HULTI-OBJECTIN AND GRADIENT- LITERATURE
Our GRP is structurally similar surgery/projection). Those methods are justified by geometric arguments that hold in non-convex nets because they operate at the level of instantaneous gradients, not the global landscape. We adopt the same philosophy: protect the main task direction by removing the antagonistic component of the auxiliary gradient, and gate the projection by a dynamic w to avoid over-constraining early steps.
handling methods in multi-task learning (e.g.. gradient
Why parameter-proximity retention instead of data? Using L, 211 Oollž ensures gr is globally Lipschitz and uniformly well-defined, providing a stable, model-agnostic constraint even in non-convex regimes. This choice is precisely avoid injecting noisy, concept-related signals that could reintroduce non-convex interference.
E.4.5 PRACTICAL IMPLICATIONS FOR HYPERPARAMETERS
Step size a. The "a < 2/L" condition should be read as "pick a in the local trust region where a second-order upper bound is valid." practice we use small a (10-5), which empirically keeps the update within this region.
Projection weight w (schedule). warm-up (w 0 initially) prevents early rver-regularizztion when ge is still large and rapidly changing; increasing w later tightens the retention constraint once the iterate enters a smoother basin. This schedule directly follows from the local-descent reading of Thm. 4.1 and the curvature- awrre reading of Thm. 4.2.
No claim of global optimality. Our guarantees are "do-no-harm" in the first-order sense and local stability under realistic step sizes. They complement, rather than replace, empirical evaluation.
Takeaway. Although the U-Net fine-tuning landscape is non-convex, the utility of Thms. 4.1 and 4.2 lies in codifying two update-level safeguards that are agmostic to global convexity: (i) conflict-only projection that never reverses the erasing descent direction, and (ii) curvature-aware scheduling that bounds interference from the retention term. Interpreted as local design lemmas under small steps and local smoothness, these results justify GRP as a principled, geometry-safe controller in practice.
F EXPERIMENTAL SETUP
F.1 CONCEPT ERASURE SETTINGS
Following prior works (Kumari et al., 2023; Gandikota et al., 2023; Zhang et al., 2024c; Gandikota al., 2024; Fan et al., 2024; Wu & Harandi, 2024; Wu et al., 2024; Zhang et al., 2024a), we evaluate the effectiveness AEGIS across three main concept erasure tasks: (1) Nudity Concept Erasure: This task focuses on removing harmful content associated with nudity-related prompts from DMs (Kumari et al., 2023; Gandikota et al., 2023; Zhang et al., 2024c; Gandikota et al., 2024; Fan et al., 2024; Wu & Harandi, 2024; Wu et al., 2024). The evaluation dataset is derived from the Inappropriate
25

---

Published as a conference paper at ICLR 2026
Image Prompt (I2P) dataset. (2) Style Concept Erasure: This task aims to remove the stylistic features of specific artists to mitigate copyright infringement risks in generated content (Kumari et 2023; Gandikota et al., 2023; Zhang al., 2024a; Gandikota et al., 2024; Lyu al., 2024).
The evaluation setup follows the protocol described in (Gandikota et al., 2023). (3) Object Concept Erasure: Similar to the above tasks, this setting targets the removal of knowledge related specific object categories (Gandikota et al., 2023; Zhang et al., 2024a; Fan et al., 2024; Lyu et al., 2024; Wu & Harandi, 2024; Wu et al., 2024). We use GPT-4 generate 50 diverse prompts for each object class in the Imagenette dataset, ensuring that the standard SD model can generate images containing the corresponding objects. Unless otherwise specified, all experiments are conducted using the pre-trained SD v1.4 model as the base DM for concept erasure.
F.2 BASELINE SETTINGS
In our experiments, we evaluate the performance of AEGIS against eight open-source concept erasure baselines: (1) ESD (Erased Stable Diffusson) (Gandikota et al., 2023), (2) FMN (Forget-Me- Not) (Zhang et al., 2024a), (3) AC (Ablating Concepts) (Kumari et al., 2023), (4) UCE (Unified
Concept Editing) (Gandikota et al., 2024), (5) SalUn (Saliency Unlearning) (Fan et al., 2024), (6) SH (ScissorHands) (Wu & Harandi, 2024), (7) ED (EraseDiff) (Wu et al., 2024), (8) STEREO (Srivatsan al., 2025) and (9) SPM (Concept-SemiPermeet
Membrane) (Lyu et al., 2024). It important to note that these methods are not universally designed to address all concept types-such such as nudity, artistic style, and object categories- simultaneously. Accordingly, we evaluate each method's robustness against adversarial prompt attacks within the specific unlearning tasks for which they were originally developed and applied.
F.3 TRAINING SETTINGS
The implementation of AEGIS follows Alg. 2. AEGIS specifically focuses on optimizing the noise predictor, i.e., the U-Net module, within DMs. During training, the concept erasure objective (Eq. (7)) minimized over 1000 iterations. Each iteration is conducted constructing è. A is dynamically updated according to Eq. (8), using an adaptive weight w, which is initialized to 0 and capped 1, with update rate μ 0.1. The U-Net is fine-tuned using the Adam optimizer with a learning rate of 10-5 Each training step includes lower-level adversarial prompt generation phase, where the attack objective (Eq. (3)) and the AET objective ($4.1) are minimized using single attack step with step size of 10 to update token at the beginning of erasures concept prompts. Gradient descent applied to a prefix adversarial prompt token in embedding space, initialized randomly following the strategy in Zhang et al. (2024b). Additionally, experiments are performed by Pytorch 1.11 on a piece of GPU of DGX2 with 8 80GB H800 GPUs, while time consumption are evluated by a single RTX A6000.
single data batch, with r7 1.0 for
F.4 ADVERSARIAL PROMPT ATTACK SETTINGS
P4D, UnlearnDiffAtk and Ring-A-Bell, Following the methods in Chin et al. (2024); Zhang et al. (2024c), at the beginning of each prompt, we introduce pretended adversarial prompt perturbations using N tokens, where N = 5 is used for nudity unlearning and N = 3 for both style and object unlearning. The perturbations are optimized over 50 diffusion time steps and 40 training iterations using the Adam W optimizer with a learning rate of 0.01.
For evaluation: of unsafe content; Style unlearning: A ViT-base model (Zhang et al., 2024c) fine-tuned on WikiArt evaluates artistic style erasure; Object unlearning: An ImageNet--retrained ResNet-50 measures object removal effectiveness.
Nudity unlearning: NudeNet (Praneeth, 2019) is used detect and assess the removal
F.5 EVALUATION METRICS
For retention performance evaluation, we adopt two utility metrics: the Fréchet Inception Distance (FID) and the CLIP score. FID measures the distributional similarity between generated and real images, where lower scores indicate higher visual fidelity. The CLIP score assesses the semantic alignment between generated images and their corresponding input prompts, with higher scores
26

---

Published as a conference paper at ICLR 2026
Table 9: Performance evaluation of concept methods applied the base SD v1.4 model for erasing
Garbage Truck. Parachute, Tench concepts.
SD v1.4
AEGIS
Concept
Metric
FMN
SPM
SalUn
ED
ESD
SH
AdvUnlcam
(Basc)
(Ours)
ASR1 (4)
100%
100%
96%
64%
60%
38%
10%
36%
14%
Garbage
ASR2 (4)
100%
100%
82%
42%
38%
26%
2%
24%
8%
Truck
FID (U)
16.70
16.14
16.79
18.03
19.22
24.81
67.76
24.45
19.84
CLIP (1)
0.311
0.308
0.310
0.311
0.307
0.290
0.283
0.298
0.306
ASRI (4)
100%
100%
100%
86%
92%
76%
54%
72%
34%
ASR2 (4)
100%
100%
96%
74%
82%
58%
24%
52%
16%
Parachute
FID (1)
16.70
16.72
16.77
18.87
18.53
21.4
55.18
20.31
17.78
CLIP (t)
0.311
0.307
0.311
0.311
0.309
0.299
0.282
0.295
0.301
ASRI (L)
100%
100%
100%
32%
28%
62%
30%
62%
20%
ASR2 (4)
100%
100%
90%
14%
16%
48%
8%
40%
6%
Tench
FID (1)
16.70
16.45
16.75
17.97
17.13
18.12
57.66
18.05
17.26
CLIP (t)
0.311
0.308
0.311
0.313
0.310
0.301
0.298
0.280
0.303
reflecting better contextual consistency. To compute these metries, use DMs generate 10,000
images conditioned on prompts from the COCO-10k dataset.
G ADDITIONAL EXPERIMENTS
G.1 EXPERIMENTAL RESULTS ERASING OBJECT CONCEPTS
We conduct comprehensive evaluation of seven concept erasure methods applied to the Stable Diffusion v1.4 (SD v1.4) model, targeting three representative object categories: Garbage Truck, Parachute, and Tench. Evaluation is performed along two axes: (1) erasure robustness, measured by the Attack Success Rate (ASR) under adversarial prompts; and (2) generation utility, assessed via FID and human evaluation of image coherence. The compared baselines include FMN, SPM, SalUn, ED, ESD, SH, AdvUnleam, and our proposed method, AEGIS.
As shown in Tab. 9, AEGIS achieves the best overall performance, reducing ASR by an average of 10% compared to all baselines (see also Fig. 11), while maintaining image quality within 2% FID deviation from the original SD v1.4. 1.4. In contrast, FMN and SPM prioritize utility through lightweight parameter tuning but fail under adversarial prompts, with ASR exceeding 40% for Tench and Parachute. This suggests residual concept embeddings remain exploitable. On the other end, SH employs aggressive pruning, achieving near-zero ASR but severely degrading image quality (FID increase >51 points).
Intermediate methods such SalUn, ED, and ESD adopt hybrid strategies combining gradient reversal and attention masking. While they partially balance robustness and utility (ASR 15-25%. FID increase 1.3-8.1), their performance is highly concept-dependent For instance, SalUn reduces ASR for Garbage Truck to 12% but performs poorly on Tench (28%). ED shows similar variance, excelling on Parachute (10%) but underperforming on Tench (32%). All three methods exhibit ASR values 3-5x higher than AEGIS across all categories, indicating limited generalizability.
Qualitative results further highlight AEGIS precision. For the prompt "garbage truck in a parking lot," AEGIS removes the target object while preserving background details (e.g., lighting, pavement). FMN and SPM often leave behind truck outlines or misplaced parts. For "baby tench in a pond," AEGIS cleanly removes the fish, rendering pond scene with human infant, whereas ESD and SalUn retain aquatic artifacts (e.g., fish scales), Similarly, AEGIS generates a pristine desert scene for "parachute in desert landscape," while SH and ED introduce visual distortions (e.g.. blurred horizons, unnatural colors).
This precision stems from AEGIS 's two-stage training: reinforcement phase to isolate target concept embeddings, followed by diffusion -ugidddd recovery phase to preserve non-target semantics. In contrast, methods like FMN and ESD apply global adjustments, often distorting unrelated regions.
27

---

Published as a conference paper at ICLR 2026
Prompt
FMN
SPM
SalUn
ESD
ED
SH
AdvUnteamn
ASCI9
"Garbage truck a reardintial
Garbage Truck
closed mailed as cramped
"eench and My parts
Tench
stopped rod right
"Farachute a rooftop"
Para- chute
"A work stacion to inside office"
Figure 1I: Visualization of generated images by different DMs after unlearning garbage truck, tench and parachate objects, following Fig. 6's format.
Conclusion. AEGIS sets new benchmark in concept erasure by effectively balancing robustness and generation quality. Its architecture-aware design avoids brute-force pruning and full-model retraining, enabling precise concept removal with minimal impact image fidelity.
G.2 IMPACT OF DIFFERENT μ VALUES ON THE PERFORMANCE OF AEGIS
In this section, we investigate the impact of the hyperparameter μ on the performance of AEGIS. Specifically, we conduct experiments with μ {0.01, 0.05, 0.1, 0.5, 1, 5, 10, 100} to erase three representative concepts-nudity. Van Gogh style, and church from the DM. We evaluate the concept erasure robustness using the ASR against adversarial prompts generated by UnleamDiffAtk, and assess retention performance using the FID.
According to Eq. (11), f controls the learning rate for updating the weight which governs the strength the parameter regularization (PR). larger y leads to faster increase w. thereby amplifying the regularization effect of PR in early training stages. This restricts the influence of the concept crasure loss on model parameters, resulting in reduced robustness to adversarial prompt attacks but improved retention performance.
As shown in Fig. 12, increasing μ leads to a consistent decrease ASR, indicating weaker concept erasure robustness, while FID improves, reflecting better retention. This trade-off suggests that large μl enhances retention at the cost erasure effectiveness. To balance these objectives, we set μ 0.1 for all experiments in this work.
G.3 RUNTIME COMPARISON WITH EXISTING METHODS
To better demonstrate the efficiency of AEGIS, we conduct experiments to compare its time cost with existing concept erasure methods. As shown Tab. 10, AEGIS requires approximately twice the time
28

---

Published as a conference paper at ICLR 2026
75
ASR
25
75
ASR FID
ASR
RD
25
50
20
20
2
ASR
28
10
0.010.05
100
Valuus
VVauus
(a) Erasing nudity concept
(b) Erasing Van Gogh style
(c) Erasing church object
Figure 12: The concept crasure robustness (ASR) and retention performance (FID) under different values.
Table 10: Time Consumption for Concept Erasure Methods with NVIDIA RTX A6000 with 1000 Iteration Step.
METHOD
Adversarial Prompt Step
AET Step
Training Time
AdvUnlcarn (Fast)
5.43
AdvUnlcarn
30
31.23
ESD
/
1.01
FMN
/
2.05
SalUn
4.34
AEGIS (Ours)
2.01
of ESD without the retention term, primarily due to the generation of single adversarial prompt and the AET (Adversarial Erasure Target). However, unlike methods that rely on additional retention datasets, AEGIS completes the fine-tuning process with significantly lower time consumption. This suggests that the proposed PR an efficient strategy for maintaining performance on preserved con- cepts without incurring substantial computational overhead. Based on comprehensive experimental results and time cost comparisons, we conclude that AEGIS achieves robust concept erasure with minimal negative impact retention performance, while maintaining high time efficiency.
H LIMITATIONS
This work aims improve the robustness of concept erasure against adversarial prompt attacks (APAs) while minimizing the negative impact on preserved concepts. To this end, we propose a robust concept erasure framework named Adversarially Erasing Concept with Gradient Projection (AEGIS). In AEGIS, we generate an adversarial erasure target (AET) using single-step attack, which typically requires nearly twice the computation time compared vanilla concept erasure methods. During fine-tuning, we employ a parameter regularization (PR) to maintain performance on preserved concepts, and apply directional gradient rectification (DGR) alleviate gradient conflicts between erasure and retention objectives. Although gradient regularization projection (GRP) helps reduce the negative impact AET on retention, the overall utility of the model still lower than that of the original diffusion model. Additionally, the AET generation process increases the time cost of concept erasure. To address these limitations, future work will explore the development of more efficient erasure targets that reduce both computational overhead and retention degradation.
I BROADER IMPACTS
This study advances AI alignment with societal and ethical norms by enhancing the reliability of concept erasure, thereby helping to curb the dissemination harmful digital content. By mitigating the risk of diffusion models producing copyright-infringing material, AEGIS addresses legal chal- lenges and promotes responsible Al deployment creative domains. Moreover, the integration of adversarial training into machine unlearning strengthens AI safety by enabling models to discard sensitive concepts while resisting adversarial circumvention. While AEGIS strikes a balance between robustness and utility, its technical complexity highlights the need for further investigation into how robustification techniques affect model scalability and performance. These contributions lay the groundwork for future research AI robustness and safety, underscoring the importance of sustained interdisciplinary collaboration and thoughtful policy development to ensure that generative Al technologies serve the broader interests of society.
29


Reviews:
[{"review_id": "HnMmzTJ5ws", "reviewer": "Reviewer_2cHk", "summary": "The paper introduces a framework called Adversarial Erasure with Gradient-Informed Synergy (AEGIS) to improve the robustness of concept erasure in diffusion models against adversarial prompt attacks (APAs).\n\nThe paper demonstrates that the vulnerability of concept erasure is closely tied to the choice of the target concept (i.e., the concept that the to-be-erased concept is mapped to). If the target is semantically close to the to-be-erased concept, the erasure performance degrades.\nTo address this, the paper proposes the AEGIS framework with two main components:\n\n•\tAdversarial Erasure Target (AET): Guides the erasure by selecting a target concept that is semantically close to the original, while also maximizing the output difference between the old and new models on the same concept.\n\n•\tGradient Projection: Mitigates gradient conflict between the erasing and preserving tasks", "questions": "•  Could the authors provide a discussion on how their proposed method differs from previous works such as [1, 2]?\n\n•  Given that the core claim of the paper is improved robustness against adversarial or recovery attacks, could the authors include experiments using the recent Random Probe recovery attack proposed in [4], which perturbs the text encoder to confuse generation and recover unlearned concepts\n\n[5] Lu, Kevin, et al. \"When Are Concepts Erased From Diffusion Models?.\" NeurIPS 2025", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-\tThe problem of machine unlearning is an emerging and important area in the machine learning community.\n-\tThe paper focuses on an important sub-problem — robustness in unlearning, which is gaining increasing attention.\n-\tThe experimental setup appears comprehensive, and the results are promising", "weaknesses": "There are several concerns about the paper’s novelty. More specifically:\n\n•\tThe first contribution—\"the vulnerability of concept erasure stems from an inappropriately chosen learning target. In particular, if the target lies too close to the semantic center – formed by words semantically related to the erased concept – the concept information cannot be fully removed\"—has already been studied in prior work [AGE, 1]. Specifically, AGE (Section 4) showed that the choice of the target concept significantly affects both erasing and retaining performance. AGE further suggests that a good target should be semantically related to, but not similar to, the to-be-erased concept—an insight that is more general and comprehensive than what is presented in this paper.\n\n•\tThe proposed min-max optimization in Equation 7 is very similar to that in AGE, with the only difference being the retention loss (regularization). In AGE, the preservation loss measures the output difference between the new and old models on the same input concept. While this paper propose to minimize the change of model parameter\n\n•\tThe idea of using gradient projections to mitigate conflict between erasing and preserving tasks has already been proposed in several works [2, 3, 4]. Yet, the paper lacks any discussion or comparison with these related methods.\n\n1: Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them\n\n2: Erasediff: Erasing data influence in diffusion models\n\n3: Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks\n\n4: GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s6YfxD7Vpr", "reviewer": "Reviewer_Tzbr", "summary": "The paper addresses the problem of robust concept erasure in diffusion models. The paper proposes Adversarial Erasure with Gradient-Informed Synergy (AEGIS), improving erasure robustness and retain performance after unlearning.", "questions": "1. Can the authors explain why they employ the CLIP score to evaluate retention? As shown in Table 2-3, the CLIP score has very little changes before and after unlearning. Besides, all the unlearning methods have similar CLIP scores (ranging from 0.29 to 0.31). This gives me the impression that the CLIP score is slightly affected by the unlearning process. In this case, how can it serve to faithfully evaluate the retention performance of different methods?\n\n2. Why and how does Directional Gradient Rectification (DGR) contribute to robustness? In Table 6, 'AEGIS w/o DGR' has significantly higher ASR compared to AEGIS. This result is somehow confusing to me: in theory, DGR serves to improve retention performance by resolving the confliction between forget gradient and retain gradient. However, the authors did not explain how it contributes to the robustness gain.\n\n3. For DGR, have the authors tried using the moving average of gradients in the gradient projection process? This might yield better performances, according to [1].\n\n[1] GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The robustness of diffusion model unlearning is a highly important problem. The idea of adversarial erasure target (AET) is novel and well-motivated, and the authors provide detailed and solid explanations for their proposed methods.\n\nThe robustness of AEGIS is validated on multiple attacks. The authors also compare AEGIS with multiple baselines.\n\nThe figures and illustrations are of good quality.", "weaknesses": "1. The paper lacks comprehensive evaluations on the retain performance. Currently, FID and the CLIP score are used. However, common DM unlearning benchmarks such as UnlearnCanvas [1] include evaluation metrics such as in-domain retain accuracy (IRA) and cross-domain retain accuracy (CRA). Since the authors claim AEGIS has great robustness–retention trade-off, a more comprehensive retention evaluation is needed.\n\n2. The motivation of Parameter Regularization (PR) and Directional Gradient Rectification (DGR) seems unclear. It seems that AET alone can achieve concept erasure while retaining model utility, and PR and DGR can be added upon any unlearning methods, and they are not specifically related to AET. Possibly, AET seriously degrades the model's utility, so PR and DGR are employed to balance retention performance. However, this motivation still seems weak: why not try other methods that do not incur additional computation costs, such as tuning the retain loss coefficient, or adjusting the learning rate?\n\n3. For the baselines, the paper did not include more recent methods on robust concept erasure, such as STEREO [2]. This CVPR 2025 paper addresses the same problem as yours, and I think it should be mentioned and compared.\n\n4. The paper lacks run-time and GPU memory comparison between different methods. I am concerned about the increased computation cost brought by AEGIS, particularly the DGR part. Could the authors discuss this possible trade-off?\n\n[1] UNLEARNCANVAS: A Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models\n\n[2] STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OeYIk4YMam", "reviewer": "Reviewer_G3SD", "summary": "This paper AEGIS proposes a robust defense algorithm against adversarial attack on concept-erased diffusion models. It shows the trade-off between erasure robustness and quality preservation, by Adversarial Erasure Target and Gradient Regularization Projection separately. AET approximates the semantic center of the concept to be erased, enables class-level removal instead of single prompt. GRP preserves the quality using parameter regularization and a novel gradient surgery technique that selectively projects away retention gradients conflicting with the erasure objective. Author's experiments show AEGIS significantly reduces attack success rates by 5.31~24% across various concepts and state-of-the-art attacks like P4D and UnlearnDiffAtk, while maintaining or improving image quality over baselines.", "questions": "1. what about attack in the input image or even embeddings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Instead of defending a single prompt, AEGIS dynamically optimizes a target prompt to approximate the semantic center of the concept being erased. With both AEGIS and GRP, it claims to achieve better tradeoff and supported by experiment results.\n2. Experiment is thorough - it validates its method across multiple concept types (object, style, nudity), model versions (SD v1.4, v2.1), and against a suite of strong adversarial attacks (P4D, UnlearnDiffAtk), proving its generalizability and robustness.", "weaknesses": "1. it looks like it's sensitive to hyper-parameters such as w in 5.3 ablation study. how to pick the best value for unlearning a new concept?\n2. how to scale if the model needs to unlearn many concepts or objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhAnLPHvU5", "reviewer": "Reviewer_hz7A", "summary": "This paper proposed retention-data-free framework for the robust concept erasure of diffusion models. To be more specific, the adversarial erasure target (AET) is obtained through optimization instead of handpicking and retention data is avoided by finetuning with gradient regularization projection (GRP) to achieve better trade-off between robustness and retention.", "questions": "1. How stable is the AET optimization process? \n2. Scalability to large models (e.g., SDXL): The experiments focus on SD v1.4/v2.1. Have the authors attempted to extend AEGIS to higher-capacity models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-formulated motivation: This paper identifies a key weakness in existing concept erasure methods, which overrely on the single-instance erasure targets.\n2. Retention-data-free: It is necessary to avoid the usage of additional datasets to maintain the model utility which might involve hidden bias.\n3. Extensive experiments are convincing: This paper evaluates multiple baselines through three adversarial prompt attacks.\n4. Good ablation study to isolate the effect of AET, PR and DGR.", "weaknesses": "1. The integration of AER and GRP is novel, however, both ideas are drawn from well-known paradigms: adversarial target optimization and gradient surgery. It is good to further clarify the differences compared to existing paradigms.\n2. Insufficient analysis on computational overhead. Computation reduction brought by AET is mentioned in the paper, however, the paper lacks detailed computation \n3. Evaluation scope somewhat narrow. Only three representative concepts (nudity, Van Gogh, Church) are analyzed.\n4. Limited interpretation of semantic centers. Visualization or embedding analysis of AET would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1d8tm0dYK", "reviewer": "Reviewer_XpvQ", "summary": "Concept erasure helps stop diffusion models (DMs) from generating harmful content; but current methods face robustness–retention trade-off. This paper introduces a novel method, Adversarial Erasure with Gradient-Informed Synergy (AEGIS), aimed at enhancing the robustness of concept erasure under adversarial prompt attacks (APAs). Extensive experiments show that AEGIS markedly reduces attack success rates across various concepts while maintaining or improving FID/CLIP versus advanced baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper conducts extensive experiments and demonstrates significant improvements over existing methods.\n2. The paper provides both empirical and theoretical support for the proposed method.", "weaknesses": "1. Placing the related work section in the main body of the paper would improve readability. In addition, it is difficult to understand some details when reading only the main text.\n2. Some fine-tuning details are missing, such as the amount of data used and the required memory.\n3. Are all the results in Table 10 obtained using 8 × 80GB H800 GPUs?\n4. Typographical error: “¿” in line 1432.\n\n\nOverall, this paper is of good quality. However, since I am not very familiar with this research field, I set my confidence level to 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)