Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate
Authors: Byung Hyun Lee
Sungjin Lim
Seunggyu Lee
Dong Un Kang
Se Young Chun
Venue: ICLR 2025 Poster (2025)
Keywords: Concept erasing
Generative model
Diffusion model
Primary area: generative models
TL;DR: We propose Concept Pinpoint Eraser (CPE) to selectively erase target concepts while preserving diverse remaining concepts, achieving superior erasing performance and robustness with attention anchoring loss and adversarial training.

Abstract: Remarkable progress in text-to-image diffusion models has brought a major concern about potentially generating images on inappropriate or trademarked concepts. Concept erasing has been investigated with the goals of deleting target concepts in diffusion models while preserving other concepts with minimal distortion. To achieve these goals, recent concept erasing methods usually fine-tune the cross-attention layers of diffusion models. In this work, we first show that merely updating the cross-attention layers in diffusion models, which is mathematically equivalent to adding linear modules to weights, may not be able to preserve diverse remaining concepts. Then, we propose a novel framework, dubbed Concept Pinpoint Eraser (CPE), by adding nonlinear Residual Attention Gates (ResAGs) that selectively erase (or cut) target concepts while safeguarding remaining concepts from broad distributions by employing an attention anchoring loss to prevent the forgetting. Moreover, we adversarially train CPE with ResAG and learnable text embeddings in an iterative manner to maximize erasing performance and enhance robustness against adversarial attacks. Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. Code is available at https://github.com/Hyun1A/CPE.

Published as a conference paper at ICLR 2025
CONCEPT PINPOINT ERASER FOR TEXI-TO-IMAGE DIFFUSION MODELS VIA RESIDUAL ATTENTION GATE
Byung Hyun Leeh", Sungjin Lim Seunggyu Lee', Dong Un Kang', Se Young Chun 12.31
DDpaatmeent of ECE, IIPII, 'INMC, Seoul National University (Idlqudgus756, sjin.im, leeseunggyu, qkrtnskfk223, sychun) @snu.ac.kr
ABSTRACT
Remarkable progress in text-to-image diffusion models has brought major con- cern about potentially generating images on inappropriate or trademarked con- cepts. Concept crasing has been investigated with the goals of deleting target con- cepts in diffusion models while preserving other concepts with minimal distor- tion. To achieve these goals, recent concept erasing methods usually fine-tune the cross-attention layers of diffusion models. In this work, we first show that merely updating the cross- coos-attentionn layers in diffusion models, which is mathematically equivalent to adding linear modules to weights, may not be able to preserve di- verse remaining concepts. Then, propose novel framework, dubbed Concept Pinpoint Eraser (CPE), by adding nonlinear Residual Attention Gates (ResAGs) that selectively erase (or cut) target concepts while safeguarding remaining con- cepts from broad distributions by employing an attention anchoring loss to prevent the forgetting. Moreover, we adversarially train CPE with ResAG and learnable text embeddings in an iterative manner to maximize erasing performance and en- hance robustness against adversarial attacks. Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. Code is available at https://gituu..com/Ht A/CPE.
INTRODUCTION
Large-scale text-to-image (T2I) diffusion models have achieved remarkable success in yielding exquisite images faithfully reflecting given texts (Dhariwal & Nichol, 2021; Nichol et al., 2022; Saharia et al., 2022; Rombach et al., 2022a; Chang et al., 2023; Ramesh et al., 2022; Lu al., 2023; Zhang et al., 2023b), but also brought a major risk on potentially creating images including copy- righted, offensive and outdated concepts (Ramesh et al., 2023; Somepalli et al., 2023). To alleviate this risk, previous works for concept erasing sug- gested various approaches such as training with curated datasets (Rombach, 2022), post-generation filtering (Rando et al., 2022; Laborde, 2020) or inference guiding (Schramowski et al., 2023). Un- fortunately, they usually require enormous computation resources (Rombach, 2022), may introduce new biases (Dixon et al., 2018) potentially allow for circumventing filters and guidance (Rando al., 2022). As an alternative solution, recent fine-tuning approaches have achieved promising re- sults on concept erasing while preserving remaining concepts (Gandikota et al., 2023; Kumari et al., 2023; Heng Soh, 2023; Fan al., 2024: Lyu et al., 2024). Especially, solely tuning pross-attention (CA) layers within diffusion model has further improved the erasing effectiveness. (Orgad et al.,
al., 2022; Rando et al., 2022; Schramowski
2023; Gandikota et al., 2024; Lu et al., 2024; Huang et al., 2023; Gong et al., 2024).
However, the effectiveness of only updating the CA layers in concept erasing has not been well stud- ied for the issue of preserving remaining concepts from diverse domains. Hcre, we first investigate the effecctivnees of merely updating of the CA layers, which is mathematically equivalent to adding linear modules, for concept erasing and show that it could deteriorate the distribution of remaining concepts. Then, propose nonlinear additive modules called Residual Attention Gates (ResAGs)
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
Baseline 1
Baseline 2
To crase
To presarve
Proj.
5DvIA
Proj
LoRA CA fine-tuning
Attack
CA fine- taning
based on LoRA
Proj
CPE (Oun))
Frase
A etivated
ResAG
Suppressud
Vreseiee
CA fine- tuning based on ResAG
Concept alterated (b) Qualitative results crasing "Claude Monet" artistic style
Concept preserved
Concept forgot
(a) Comparison of fine tuning approaches
(a) Comparison of fine-tuning approaches for concept erasing. Previous methods could affect both on target and remaining concepts as they merely fine-tunes CA layers. In contrast, our method, CPE, can adatively transmit the change for target concepts to erase while successfully suppressing it for remaining concepts, by using the proposed ResAGs. (b) Qualitative results on erasing "Claude Monet" artistic style, comparing with baseline (Gandikota et al., 2023).
Figure 1:
by employing the mechanism of attention gates (Bahdanau al., 2015; Vaswani et al., 2017: An- derson et al., 2018; 8; Oktay et al., 2022). As shown in Figure 1, ResAGs can selectively transmit the change of the CA layer output only for the target concepts to erase. To improve preservation ability of ResAG, we propose an attention anchoring loss that blocks the alteration of the CA layer out- put for remaining concepts. We also improve robustness to attack prompts by adversarially training methods, includ-
ResAG and learnable text embeddings in iterative manner. By combining ing ResAGs, attention anchoring loss and robust training strategy, we dub the overall framework Concept Pinpoint Eraser (CPE). Extensive experiments demonstrated CPE delete the target concepts while successfully preserving the various remaining concepts.
Here contributions: 1) We theoretically showed that updating the CA layers may not retain wide-ranging remaining concepts. 2) We propose our framework CPE, consisting of ResAGs, atten- tion anchoring loss, and robust training strategy, that robustly deletes target concepts while main- taining various remaining concepts. 3) Extensive experiments showed CPE robustly erased target concepts while preserving diverse remaining concepts, outperforming baselines by large margins.
RELATED WORKS
Safe T2I image generation. The risks of generating inappropriate content by T2I diffusion mod- els, like Stable Diffusion (SD), have been extensively explored (Abid et al., 2021; Birhane Prabhu, 2021; Hutson, 2021; Rombach al., 2022b; Rombach, 2022). One approach for safe generation is data censoring (Nichol et al., 2022; Ramesh et al., 2022), as training datasets could contain un- desirable content (Schuhmann et al., 2021: 2022). However, retraining resource-intensive may introduce biases, or fails to completely remove unwanted content (O' connor, 2022; Dixon et al., 2018). Inference-guiding methods (Schramowski et al., 2023) leverage model priors to steer gener- ation towards safe concepts, but they can be bypassed by disabling them (Rando et al., 2022).
Fine-tuning for erasing concepts. Beyond these limitations, fine-tuning approaches have been explored to climinate target concepts from models for safe release. FMN (Zhang et al., 2023a) efficiently erased target concepts by re-steering CA layers. AbiCon (Kumari al., 2023) and ESD (Gandikota et al., 2023) aligned target concept distributions with a surrogate concept, showing that CA layer fine-tuning is effective for erasing target concepts. TIME (Orgad et al., 2023) updated projections in CA layers with closed-form solution, allowing for quick computation. Meanwhile, recent red-teaming tools showed that many fine-tuning approaches are vulnerable to adversarial prompts to regenerate the erased concepts (Rando et al.. 2022: Tsai et al., 2024; Zhang et al., 2023c). RECE (Gong et al., 2024) and Receler (Huang et al., 2023) improved robustness against the attack prompts by iterative learning schemes where erasing and adversarial attack are alternately trained.
2

---

Published as a conference paper at ICLR 2025
Forgetting remaining concepts. Despite their effectiveness in erasing target concepts, pre- venting the forgetting of remaining concepts is still required. Selective Amnesia (Heng & Soh, 2023) added regularization loss for remaining concepts, inspired by continual learning (Kirkpatrick et al., 2017: Lee et al., 2023; 2024). UCE (Gandikota et al., 2024) adopted TIME's approach (Orgad et al., 2023), proposing closed-form solution. Based on Parameter- Frrameter-Efffcientt Fine-Tuning (PEFT) (Zhou et al., 2022; Yao et al., 2023; Vaswani et al., 2017), SPM (Lyu et al., 2024) applies one-dimensional LoRA (Hu et al., 2021) the layers of diffusion models and proposed an anchoring loss for distant concepts. This effectively prevented forgetting of distant concepts even when sequentially erasing target concepts. MACE (Lu al., 2024), which has similar closed form of UCE, proposed using LoRA for erasing each target concept and introduced loss integrating LoRAs from multiple target concepts, enabling the massive concept erasing while mitigating forgetting of remaining concepts.
3 CONCEPT PINPOINT ERASER VIA RESIDUAL ATTENTION GATE
In this section, we demonstrate mere fine-tuning of cross-attention (CA) layer could challenge reten- tion of diverse remaining concepts. We then propose Concept Pinpoint Eraser (CPE), to effectively erase target concepts while maintaining various remaining concepts. As components of CPE, we de- scribe a residual attention gate, an attention anchoring loss, and introduce robust training strategy.
3.1 ONLY FINE-TUNING CA LAYERS MAY NOT PRESERVE REMAINING CONCEPTS
We first define CA layer (Chen et al., 2021; Wang et al., 2024). We denote the linear projections of a H head CA layer in [-th block of model as 0 - U (With w W w where the elements are linear projections of h-th head for query, key, value, and output. respectively. For simplicity, we omit the notations Raxm be a text embedding where d is its feature dimension and m is the length of token sequence.
and subscripts (g. k, u, if there is no confusion. Let E
Definition 1 (CA Layer). Let (-)) be softmax along the column. Then, a H-head CA layer, pa- rameterized with 0 U q1 Wh. Wh. Wh) with a query token E Rdh is defined as:
H
W (W
T(z, E) = Σ
W.
9
(1)
m
A=1
where Wh
R # w E +R#x xd wh E # and Wh Rd
To investigate the challenge of retention remaining concepts by only updating the CA layers, assume that the change in a CA layer output can induce the change in the diffusion model output from the original output. Under this assumption, we first investigate the upper bound of the variation of a CA layer output when linear projections in the CA layers change. Then, we demonstrate there is limitations in tightening the upper bound for remaining concepts by deriving their expected upper bound of the change in the CA layer output, potentially leading to forgetting in diffusion model output. Proofs of the following theorems are provided in Appendix B.
For simplicity. we consider cases where W and W u are updated as previous works (Gandikota al., 2024: Lu et al., 2024). Then, we derive the upper bound for the change of output from CA layers due to variations in W & and Wh. v. The upper bound is derived as below:
Theorem 1. Let Wh. and Wh be the updated weights of W and Wh. Assume that || S M and ||2||0 < M2. Denoting the Lipschitz constant of the linear transforms W as Lww
|| E; WA W T(z, E; W Whll S C (2)
where AW = W W. Cf
Maym mym
Lwgwy and C = Lwg.
Let Eus and Egem be the text embeddings of target and remaining concept, respectively. Theorem implies | WWE! be sufficiently large for the target to ensure their distribution shifts. Otherwise, the CA layer output for the target will not change enough. Meanwhile, large upper bound in Equation
3

---

Published as a conference paper at ICLR 2025
Pmjection in CA Layer
Text
Projection in CA Layer
Text Embedding
Embedding
E
W
Wx
E
ResAG
RexAG
Shared Layers
ResAG;
U2
ReshG
U U!
ResAG
max. of gate values along concepts
(a) Architecture of ResAG for single concept
(b) Merging multiple ResAGs for inference
Figure 2: (a) Architecture of ResAG module in CA layers for selectively erasing target concept while preserving remaining concepts. (b) To erase multiple targets during inference, we merge mul- tiple ResAGs by only adding the ResAG of the target with the highest gate value for each token.
(2) cannot guarantee the change of CA layer output for remaining concepts. Therefore, minimizing | WWErn for remaining concepts is desirable to suppress the change of their CA layer output. Our question is then how much we can suppress for arbitrary | W.emlll ll. For this, we analyze the expectation of || A WE under the Gaussian mixture model for text embeddings, explored by previous works TforresCarrasqquls
et al., 2002b;a; Viroli McLachlan, 2019; Deng et al., 2023).
Theorem 2. Let Erem = (e1.'.. ,,, ,.ml where is token in a text embedding. Suppose tokens of the embeddings for remaining concepts follow mixed Gaussian distribution, that is, p(e;)= Di (e pi,, d²I) and E T. = 1. With pt, = [4] H²,
prr), we show that:
R
R
E [| F 1|:]] Callw
AW C==
T
Czll [| from Equation (3) highlights the dilemma of only updating CA layers; increasing [AW] y to erase targets raises the upper bound for remaining concepts. Once AW is fixed, the upper bound remains as long as we cannot suppress of. If the modes of Erem are not in the null space of AW, the bound even worsens. It implies that fine-tuning the CA layer to erase the targets may not preserve the remaining concepts, leading to alterations in the diffusion model's output.
3.2 RESIDUAL ATTENTION GATE (RESAG)
From the above observations, propose to add nonlinear module that can selectively erase target concept. Suppose we can detect the concept from which E is sampled. Let f (EE = V, m be embedding-dependent projection adaptive to the concepts of samples. Then,
show that: Corollary 1. If we use (E) instead of AWE, Equation (3) for Erem can be modified to:
R V
Eg Eean AW [| |AAW P
Σ
#WW V.. 1.
(4)
Equation (4) implies that if ((E) can detect the concepts of embeddings and find V, such that |VV is suppressed to zero for remaining concepts while it is large for target concepts, we can expect effective erasure of target concepts while minimizing the change of remaining concepts. To further clarify this, consider the following simple example.
Proposition 1. Let (EE) (EE) where (EE) 2 0, I is an identity matrix, and D is the distribu- tion of embeddings for target concepts. Suppose that (EE) can classify the distribution from which embeddings of target concepts are sampled or
1. E~ Drarr O. otherwise.
a(E)
=
(5)
Then, Equation (4) becomes zero for Exem suppressing EE [[AWE, V ((Eem)) ||
4

---

Published as a conference paper at ICLR 2025
Unfortunately, finding such (E) is infeasible. Moreover, since concept is not merely an indepen- dent token in a text embedding but token sequence of text prompts, f(E) should be designed based on the relationships among tokens in the embedding. For instance, if 'Bill Clinton" is a target concept and 'Bill Murray' is a remaining concept, ((E) should be activated for the former while suppress- ing the latter even if they contain the same token for 'Bill.. It requires to consider the following elements to design V (E): 1) understanding relationships between concepts within text embeddings, 2) accurately identifying embeddings with target concepts, and 3) efficiently and effectively altering outputs for target concepts while suppressing changes for other concepts. In the next section, we propose a module and loss to meet these stringent requirements.
To design f(E). we are inspired by the mechanism of attention gates (Bahdanau et al., 2015; An- derson et al., 2018; Li et al., 2020; Oktay et al., 2022). Attention gates selectively focus on the relationship between embeddings and dynamically adjust their learned importance. Then, it can fil- ter out irrelevant details, that meets the requirements for f(E). Specifically, for each target concept we learn separate attention gate module fe and handle multiple concepts by merging the learned modules during inference. For a target concept e, we design its structure as shown in Figure 2. (a):
(
(E( A.S(v. A. -0
(6)
Vm
where U1,es U E Raxd V. E Rd for $1 < S()) E Rmxm is a diagonal matrix whose diagonal is the sigmoid for the input. Here, A. is the attention of embeddings focusing on concept c and 5() ) is the gate value for the reorganized embeddings. By this structure, (E) can distinguish target and remaining concepts and selectively modifies for a target concept. We also note that U1,e+ U2.0. and v are shared across all CA layers, since detecting a target concept in text embedding independent of them. It allows to erase the target concept with very few parameters. For AW... c. we define a low-rank matrix AW, = UT, U3.e (Hu et al., 2021) where U3.c, U4.c E R and 82 < d, which has shown to be sufficient for editing concepts in diffusion models (Lyu et al., 2024; Lu et al., 2024). Then, the change of projection output is U (E), Since it is added in residual manner to the original projection output, we call this Residual Attention Gate (ResAG).
For multiple concepts erasing with ResAGs learned for each target concept, we propose a simple method to merge them into one module shown in Figure 2 (b). Denote the residual attention gate for concept e as (E) = Ua, and its i-th token as (E). Then, we add rf (E) to the original projection for i-th token where e* = arg max.(5(vv) (S(vTEA.))) EA.)a)- That is, we only add the ResAG of the target concept with the highest gate value for each token.
3.3 LOss DESIGN FOR CPE
The loss for our CPE is primarily designed to directly train on the output of key/value projections within CA layers. Since we jointly train both the key and value projections with the proposed loss in the same way, we omit the subscript notation for key/value on projection W without confusion.
Erasing loss. Let Esur be the embedding of a surrogate concept, which we want to generate instead of the target concept (Gandikota et al., 2023; Lu et al., 2024), , Ear and & be the sets of text embeddings containing target and surrogate concepts respectively. We then train ResAG to make the projection output of Ega in Ear similar to the output of Esur in Exurs using the following crasing loss:
Lera Esur) E.E,.. (WE Rear (Eaar) (WE w/W (E
(7)
where larger 1) means more intense erasure toward the surrogate concept (Gandikota et al., 2023; Huang et al., 2023) and REt is for erasing target concept. While the previous works applied the loss to the output of the diffusion model, we directly use it to the key/value projection output.
Attention anchoring loss. To prevent undesirable degradation on remaining concepts by erasing loss, we propose an attention anchoring loss. Specifically, we minimize the upper bound (2) for predefined anchor concepts. We note that A in the upper bound (2) can be easily replaced by the the proposed module ResAG. Defining Eune as a set of text embeddings containing the anchor concepts, we minimize the upper bound for embeddings of anchor concepts E in E
Lau (Eanc) EE,.. R Panc)](p
(8)
5

---

Published as a conference paper at ICLR 2025
Algorithm Framework of training ResAG in CPE
1: Input: W of key or value in CA layer, sets of text embeddings Emrs Extrs and Eme.. ResAG for the target concept Rur of learnable adversarial embeddings N, # of initial and later train- ing iterations for the ResAG Ti and 72, # of training iterations for the learnable adversarial embeddings T3, 7) for erasing loss, and for attention anchoring loss. B
2: Initialize Eadv {E) adv E
3: for <- 1,2 ... ,S do
4: if then
5: for t + 1,2, ................................ T3 do
6: end for
Endy t Updateg (W, Rur, Eoer Endv, Eanc) by Eq. (9)
7:
8:
end if
9:
T + Ti if = else T2
for < 1,2.... do
11:
if 8 = then
12:
Rar + Update fir (W, Ruar Ear Esur, Eane, 17, A) by Eq. (10) without middle term
13:
else
14:
RAE + Update R. (W, Runi Eur Eady: Exur: Eaney 2) by Eq. (10).
15:
end if
16:
end for
17:
end for
18:
Output: ResAG for a target concept, Rar-
We selected several anchor concepts as in previous works (Gandikota et al., 2024; Lu et al., 2024; Gong al., 2024), slightly modifying their approaches. Specifically, for certain domain of target concepts, we utilize a large language model (Liu et al., 2023) to extract a large number of concepts similar the target concepts. constructing an anchor concept pool. Then for erasing target concept, we select only a few concepts from the anchor concept pool that are most similar to the target concept and use them anchor concepts. Additionally, since using it alone is vulnerable to being over-fit to the anchors, we inject noise into a pair of the anchors and randomly interpolate them for diversity, inspired by Zhang et al. (2017) and Lim et al. (2022). For more details on the anchor sampling methods with experiments, please refer to Appendix C.I and D.3.
3.4 ROBUST TRAINING FOR CPE
While ResAG can minimize the effects on remaining concepts by performing pinpoint crasure on the target concept, directly training ResAGs is challenging since the nonlincarity of ResAGs may hinder broad-area crasure of the target concept with the limited number of instances of the target. This could compromise the performance of ResAG overall erasure and robustness prompt attacks. To prevent this, we design an iterative adversarial learning strategy. For the trained ResAG, train adversarial embeddings to be added Eaa regenerate images containing target concepts. Then, the ResAG is retrained for defense against the adversarial embeddings. We call this Robust erasure via Adversarial Residual Embeddings(RARE))
Let Eadr = EE E2..... ... EE,, }o be set of learnable adversarial embeddings. With E,, = Ema + ER we train Endv while freezing the ResAG for the target concept, Rar, using the following loss:
min Ess
M Eg., [ REn (E,,) 7==1
WE
N
That is, we train the learnable embeddings make the output with ResAG similar to the original projection output of Emr After training Exdv, we retrain ResAG to block the adversarial embeddings. Then, the final loss for robust erasure target concept is as follows:
N
min Cer (Eurr E + Bay
Σ
Cema + E Exar) + X. (Eanc);
(10)
N
==1
where Eur + E (Enr Ed Ema E E and the middle term is omitted for the initial crasing stage. We train the ResAG with Equation (10) during the erasing stage and Exty with Equation (9)

---

Published as a conference paper at ICLR 2025
4
FMN
ESD-u
BSD-x
UCE
MACE
CPE Ours
Erase
Erasing calabrity:
Andrew Carfiedd CarierC
official photo"
Preserve
Preses ving celebrity: AAn image capturing
Kearnu orrvs public event
Preserving
artwork Alberto Seveso
Preserving character: "An image of Minnie
Figure 3: Qualitative results of our CPE and baselines multiple concepts erasing. We erased 50 celebrities at once. shows that CPE successfully preserves both similar and dissimilar concepts.
Table 1: Quantitative results celebrities erasure. We used CS and GCD accuracy (ACC) for target celebrities. We measured CS and FID for COCO-30K, or KID for the other remaining concepts.
Turget Concepts
Remaining Concepts
Method
Celebrities
100 Celebrities
100 Artistic Styles
64 Characters
COCO-30K
CS L ACC(%)
CS
ACC(%)
KID x 100)
CS
XID(x100))
CS
KID(x 100)
CS f FID
FMN iZhang al., 2023a)
27.62
ESD-x (Gandikota et al., 2023)
24.41
7.30
26.23
10.39
2.66
26.65
1.20
25.82
0.91
29.55
14.40
(Gandikota
2023)
UCE (Gandikota et al., 2024)
17.73
0.09
24.76
34.42
1.43
20.41
5.39
19.53
3,31
20.13
97.09
2024)
CPE (Ours)
20.79
0.37
34.82
88.26
0.08
29.01
0.01
29.27
0.02
31.29
14.13
(Rombach et 2022b)
during the adversarial stage, repeating over multiple stages. The overall procedure training ResAG for target concept in our proposed framework is presented in Algorithm 1.
By integrating all components, we name our proposed framework Concept Pinpoint Eraser (CPE). Since CPE directly trains ResAG with the projection output, it is highly cost-efficient. For instance, erasing single concept requires additional parameters less than 0.01% % of SD v1.4 (Rombach al., 2022b). Detailed information its efficiency is provided in Appendix C.3.
4 EXPERIMENTS
We first experimented multiple concept erasing tasks: celebrities erasure and artistic styles era- sure. To show that our method consistently prevents the forgetting various remaining concepts, we considered four domains to preserve: celebrities. artistic styles, characters, and COCO-30K cap- tions. Next, we conducted experiments against adversarial prompt attacks using recently proposed red-teaming tools. We compared with five recent basclines for celebrities and artistic styles erasure: FMN (Zhang et al.. 2023a), ESD-x (Gandikota et al., 2023), ESD-u (Gandikota et al., 2023), UCE (Gandikota et al., 2024), and MACE compared
removal of explicit contents and evaluated the robustness
(Lu et al., 2024). Additionally, leam (Zhang et al., 2024) for explicit content erasure and robustness to adversarial prompts. We fine-tuned them on SD v1.4 (Rombach et al., 2022b) and generated images by DDIM with 50 steps.
CPE with RECE (Gong et al., 2024) and AdvUn-
For quantitative evaluation, we measured CLIP score (CS) (Hessel et al., 2021) for target and remain- ing concepts, where lower CS for target concepts indicates more effective erasure and higher CS

---

Published as a conference paper at ICLR 2025
v1.4
FMN
ESD-u
BSD-x
MACE
Ouur
Erase
Frastin ariistic style Image in the style of Claude Monet
Preserve
Preserving artistic styl: A famous artwork by Albert Lynch
Preserving celebrity: 'An oil painting Of Amanda Seyfried
Figure 4: Qualitative results on artistic styles erasure. We erased 100 artistic styles at once. It shows that CPE successfully erases the target artistic styles while preserving diverse remaining concepts.
Table Quantitative results atistic styles erasure. We measured CS for target artistic styles. We measured CS and FID for COCO-30K, or KID for the other remaining concepts.
Target Concepts
Remaining Concepts
Method
100 Artistic Styles
100
Artistic Styles
100 Colebritics
64 Characters
COC0-30K
KID
ACC(%) 1 x10))))0)
KIDI
CS
FMN (Zhang et al,, 2023a)
28.21
28.90
0.30
34.01
M6.46
0.16
28.41
0.14
31.31
13.99
ESD-x (Gandikota al., 2023)
20.89
21.21
81.41
0.81
1.13
29.52
15.19
ESD-u (Gandikota al. 2023)
19.66
19.55
7.37
20.77
29.28
10.69
22.21
4.52
27.76
17.07
UCE (Gandikota et 2024)
21.31
25.70
1.86
22.04
3.71
3.30
19.71
3.00
19.17
77.72
MACE (La al., 2024)
22.59
28.58
0.25
26.87
10.79
1.06
24.56
0.75
29.51
12.71
(Ours)
20.67
28.95
34.81
89.80
29.14
0.01
31.26
14.20
SDv1.4 (Rombuch et 2022b)
29.63
28.96
34.83
90.86
29.14
31.34
14.04
for remaining concepts means better preservation. Especially for celebrities, used the GIPHY Celebrity Detector (Hat) et al., 2024) to measure the top-1 GCD accuracy (ACC) of the generated celebrity images. For accuracy, lower values are better for target celebrities while higher values are better for remaining celebrities. We also evaluated the Frechet Inception Distance (FID) (Heusel al., 2017) for COCO-30K captions. For the other remaining concepts, we used the Kernel Inception Distance (KID) instead of FID, since KID is known to be more stable and reliable with a smaller number of samples. For KID and FID, lower values are better for remaining concepts. For more details on the implementation details, please refer to Appendix D.
4.1 CELEBRITIES ERASURE
We selected 50 celebrities the targets from the list of celebrities provided by Lu al. (2024), consisting of 200 celebrities, and generated 1250 images using 5 prompt templates with 5 random seeds. For remaining concepts, considered three domains: 100 celebrities and 100 artistic styles from (Lu et al., 2024), 64 characters from Word2Vec by Church (2017). We generated 25 images using 5 prompt templates with 5 random seeds, resulting in 2,500, 2,500, and 1,600 images for each remaining domain. We also used COCO-30K as remaining concepts. We set the rank of modules in ResAG as (81, $2) (16, 1) and (1,, A) (0.3, 1.0 x 10a) for erasing and attention anchoring loss. For robustness to adversarial prompts, we trained 16 residual adversarial embeddings over 5 stages.
From Figure 5, CPE and all baselines effectively erased target concepts in qualitative results. but CPE was the only method that successfully prevented forgetting of all remaining concepts. The baselines showed visual degradation for at least one of the remaining concepts. From Table CPE and UCE demonstrated remarkable erasure performance lower than 0.5%. However, UCE showed generally low preservation performance for remaining con- cepts. While FMN well retained remaining concepts, its erasing effectiveness was lower compared to other baselines. MACE used remaining celebrities and COCO captions as anchors and resulted in excellent preservation performance for these domains, but forgetting occurred for the other domains. In contrast, CPE showed the most minimal effect on the remaining concepts and outperformed on all remaining concepts by large margin, while ensuring impressive erasing capability.
target celebrities, with GCD accuracy
8

---

Published as a conference paper at ICLR 2025
BSD
MACE
CPE (Ours)
ESD MACE
CPE (Ours)
Wilhooa attock
UnleamDiif
ottractive cesign, geinting
valleyabs hoczdattractree
denis wrightieldworrcc
affondability affond @lar processes
Figure 5: Qualitative results of CPE and baselines robustness to adversarial attacks by Unlearn- Diff (Zhang et al., 2023c). It shows that CPE successfully defends against adversarial attack prompts.
Table 3:
Results of detected number of explicit contents using NudeNet detector on I2P and preser- vation performance on MS-COCO 30K with CS, FID.
Number of mudity detected on 12P (Detected Quanlity)
COCO-30K
Method
Armpits
Belly
Breasts Genitalia (F) Breasts (M) Genitlia
FMN (Zhang al., 2023a)
43
117
12
59
155
17
19
2
424
30.39 13.52
ESD-x x (Gandikota et al., 2023)
59
73
12
39
100
4
30
8
315
30.69
14.41
ESD-u
(Gundikota et al., 2023)
2
35
30.21 15.10
UCE (Gandikota al., 2024)
29
7
35
5
11
30.85
14.07
MACE (Iu al., 2024)
17
19
2
39
16
0
7
111
29.41
13.42
RECE (Gong al., 2024)
31
3
89
30.95
CPE (Ours)
8
8
40
31.19 13.89
SD vl.4 (Romhach et al., 2022b)
148
170
29
63
266
18
42
7
743
31.34 14.04
SD v2.1 (Rombuch, 2022)
105
159
177
57
2
31.53
14.87
4.2 ARTISTIC STYLES ERASURE
For artistic styles erasure, we selected 100 target artistic styles following (L.u et al., 2024). We used the same remaining concepts in Section 4.1. We set ($1, 52) = (16, 1) and (17, A) = (0.5, 1.0 x 10*), and we trained 16 residual adversarial embeddings for robustness to adversarial prompts over 10 stages. From Figure 3, methods except FMN effectively erased the target concepts in the qualita- tive results. Notably, all baselines showed visual alterations forgetting for remaining concepts in Figure 3. contrast, CPE generated visually indistinguishable images of remaining concepts from the original images. Table shows results for artistic style erasure that is consistent with those from celebrities erasure experiments. In this case, ESD-u achieved the lowest CS for target artistic styles, but resulted degradation in the generation remaining concepts. Meanwhile, CPE demonstrated competitive erasing effectiveness to ESD-u with the most minimal impact on remaining concepts.
4.3 EXPLICIT CONTENTS ERASURE
We generate images from I2P prompts (Schramowski et al., 2023) consisting of 4,703 ordinary prompts without inappropriate words which bypass to generate offensive contents. To erase explicit concepts, we removed four keywords following Lu et al. (2024): 'nudity', 'naked', 'erotic', and 'sex- ual', To measure the frequency of explicit contents, we employed the NudeNet detector (Bedapudi, 2022), setting its detection threshold to 0.6 (Lu et al., 2024). For preservation performance,
used COCO-30K. For erasing stage, we used (s), = (64, 4) and (17 A) = (3.0, 1.0 x 10*), and we trained 64 adversarial embeddings for adversarial learning stage number of explicit contents detected by the NudeNer detector. CPE resulted in the fewest detected explicit contents, recording less than half of the result of the second-best method. terms of FID, FMN and MACE showed better results, but FMN showed low efficacy on erasing explicit contents and MACE used COCO as anchors. In contrast, although we didn't 't use COCO-30K captions as anchors, CPE successfully maintained the concepts from them and achieved the best CS.
20 stages. Table 3 shows the
4.4 ROBUST ERASURE LEARNED ATTACK PROMPTS
We utilized Ring-A-Bell(Tsai et al., 2024) and UnleamDiffiZhanm et al., 2023c) as the red-teaming tools to verify the robustness our CPE. We considered Van Gogh and I2P prompts as the domains for the attack experiments. We also selected celebrities among 50 celebrities used in Section 4.1 to evaluate robustness on celebrities erasure. We used attack success rate (ASR) as the evaluation metric to measure the ratio of regenerated images containing the target concepts by the red-teaming

---

Published as a conference paper at ICLR 2025
Table 4: Comparison of CPE and baselines on robust concept erasure against adversarial attack
prompts: Ring-A-Bell (Tsai et al., 2024) and UnlearnDiff (Zhang et al., 2023c).
6 Celebrities
I2P Nudity
Artistic Style (Vincent van Gogh)
Method
Ring-A- Bell
Ring-A A-Bell
UnlearDDfff
Ring-A A-e---Bii Bell
UnleamDiff
FMN (Zhang et al., 2023a)
58.06
80.85
97.89
14
56
ESD (Gandikkta et al., 21123)
30.48
61,70
76.05
32
UCE (Gandikota et al. 2024)
35,46
79.55
MACE (Lu et al.. 2024)
29.33
4.26
66. 66.90
24
RECE (Gong al., 2024)
13.38
65.46
28
64
AdvUnleam (Zhang et al., 2024)
21.13
CPE (Ours)
1.14
0.00
30.28
Table 5: Ablation study on the effect of the components of ResAG. The row with bold represents the selected configurations. We can see all components are crucial for deletion and preservation.
Components of
Target Concepts
Remaining Concepts
ResAG
fex
RARE
Rank (51)
50 Celebritics
100 Colebritics
Artistic Styles
COCO-IK
CS
ACC
f
ACC
KID(x
CS
KID(x 100)
1
KID(x 100).
16
22.38
9.83
34.01
83.96
0.34
25.64
1.83
29.56
0.23
16
0.09
v
16
21.75
2.23
34.83
88.31
0.08
29.02
0.01
31.27
0.05
16
20.79
0.37
21.34
0.74
34.43
87.63
0.10
28.
0.03
31.06
0.07
20.73
64
20.76
0.31
34.81
88.41
0.07
29.03
0.01
31.27
0.04
20.82
29.05
SDVL4 (Rombuch el al.,, 20226)
34.49
91.35
34,83
90.86
28.96
31.34
tools. From Table 4, CPE demonstrated robust erasure of target concepts competitive to recent robust methods (Gong al., 2024: Zhang et al., 2024). In particular, CPE successfully defended attacks by BinglA-Bell on explicit contents erasure, recording zero ASR. It also defended all attack prompts by Ring-A-Bell and UnlearnDiff on erasure of Van Gogh's artistic style, verifying its robustness.
4.5 ABLATION STUDY
For ablation study, we measured KID on COCO-1K which is randomly selected 1,000 captions from COCO dataset instead of FID on COCO-30K. For ablating ResAGs, we only trained U3 and U4. We studied on celebrities erasure with the same setup in Section 4.1. From Table 5, the performance dropped when the components were removed, verifying their importance. Removing ResAGs led to significant degradation in both deletion and preservation. Training ResAG without Can slightly improved erasure performance but severely impaired preservation capability on remaining concepts. Removing RARE demonstrated its role in enhancing target concept removal with minimal impact on the remaining concepts. We also measured performance variations based on the rank $1 of the attention in ResAG. Even with s CPE showed SOTA-level erasing performance compared to baselines, while maintaining strong protection of nearly all remaining concepts. Although increasing the rank from 16 improved performance, the enhancement was marginal for higher ranks. suggests that the low-rank property is sufficient to capture the target concepts without affecting the remaining concepts. For additional ablation studies on 82, 1), A, the architecture of ResAGs, and RARE in robustness evaluation, please refer to Appendix E.
5 CONCLUSION
In this work, we theoretically demonstrated that only fine-tuning CA layers for concept erasing in diffusion models could have challenge in preserving remaining concepts. As one solution for this, we proposed our framework, Concept Pinpoint Eraser (CPE), simple and effective approach to erase target concepts while maintaining diverse remaining concepts. We designed lightweight modules called the Residual Attention Gates (ResAGs), adaptively adjusting the CA layer outputs dependent on text embeddings. To robustly erase target concepts without forgetting on remaining concepts, ResAG is trained with attention anchoring loss under adversarial learning scheme. Through extensive experiments on erasure of celebrities, artistic styles, and explicit concepts, CPE ensures the robust deletion of target concepts and protection of diverse remaining concepts.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by Institute of Information & communications Technology Plan- ning & Evaluation (ITTP) grant funded by the Korea government(MSIT) [NO.RS-2021-I121 1343, Artificial Intelligence Graduate School Program (Seoul National University)] and by the National Research Foundation of Korea(NRF) grant funded by the Korea government (No. NRF- 2022R I A4A 10305797, NRI-2022M3CI A30920221 Also, the authors acknowledged the finan- cial support from the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University.
REFERENCES
Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. AAAI/ACM on AI, Ethics, and Society, 2021.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. CVPR, 2018.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.
Praneeth Bedapudi. Nudenet: Neural nets for nudity detection and censoring. nudenet www.in tech/, 2022.
https://
Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? 2021.
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Patrick Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. /CML, 2023.
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. ICCV, 2021.
Kenneth Ward Church. Word2vec. Natural Language Engineering, 2017.
Jingeheng Deng, Liang Pang, Huawei Shen, and Xueqi Cheng. Regavae: A retrieval-augmentee gaussian mixture variational auto-encoder for language modeling. EMNLP, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models heat gans on image synthesis. NeurlPS, 2021.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. AAAI/ACM on AI, Ethics, and Society, 2018.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based mod- els. NewrIPS, 2020.
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Neurips, 2021.
Bradley Efron. Tweedie's s formula and selection bias. Journal of the American Statistical Associa- fion, 166(49))1602216114 2011.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Em- powering machine unlearning via gradient- based weight saliency in both image classification and generation. ICLR, 2024.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. ICCV, 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau. Unified concept editing in diffusion models. WACV, 2024.

---

Published as a conference paper at ICLR 2025
Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu YunGang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. arXiv preprint arXiv:2407.1238. 2024.
Nick Hasty, Ihor Kroosh, Dmitry Voitekh, and Dmytro Korduban, Giphy celebrity detector. https:
//githuke cww/ wtt.coit////aaec.. ce wwwebeeeeeeeeeeeeeeeeeeeeeeeeeee detection-ose 2024.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. Neur/PS, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. reference-free evaluation metric for image captioning. EMNLP, 2021.
Clipscore: A
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2021.
Chi-Pin Huang, Kai-Po Chang, Chung-Tine Tsai, lung-Hsuan Lai, and Yu-Chiang Frank Wang. Receler: Reliable concept erasing text-to-image diffusion models via lightweight erasers. arXiv preprint arXiv:2311.17717,222 2023.
Matthew Hutson. Robo-writers: the rise and risks of Language-generatin ai. Nature, 591(7848):
22-25, 2021.
Surea Proxima Centauri B, Erratica, and Stephen Young. Image synthesis style studies. https: / www. a aiartapps
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. ICCV, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkkk--Baiikaa,,.aa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Nupur Kumari, Bingliang Zhang, Sheng-Yu Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. ICCV,2023.
Gant Laborde. Nsfw detection machine learning model. 2020.
com/ GantMan//////
Byung Hyun Lee, Okchul Jung, Jonghyun Choi, and Se Young Chun. Online continual learning on hierarchical label expansion. ICCV, 2023.
Byung Hyun Lee, Min-hwan Oh, and Se Young Chun. Doubly perturbed task free continual learning. AAAI, 2024.
Yang Li, Guodong Long, Tao Shen, Tianyi Zhou, Lina Yao, Huan Huo, and Jing Jiang. Self-attention enhanced selective gate with entity-aware embedding for distantly supervised relation extraction. AAAI, 2020.
Soon Hoe Lim, Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W Mahoney. Noisy feature mixup. ICLR, 2022.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
12

---

Published as a conference paper at ICLR 2025
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2016.
Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Tf-icon: Diffusion-based training-free cross- domain image composition. ICCV, 2023.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: erasure in diffusion models. CVPR, 2024.
Mass concept
Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. One-dimensional adapter to rule them all: Concepts, diffusion models and erasing applications. CVPR, 2024.
A. M. Mathai and Serge Provost. Quadratic Forms Random Variables: Theory and Applica- tions, volume 126 of Statistics: Textbooks and Monographs. Marcel Dekker, New York, 1992. ISBN 9780824786915.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML,, 2022.
Ryan O' O'connor. Stable diffusion vs 2 what you need to know. btst:p::/www.assemb...e.g com/blog/et sttp://sallleeeee wwwasaaaaaflssasaeeeeeeeeetaaale diffusiffuuioooooo wwwooonn---v-eeee 2hatt-----------
wwt.:wwwsseee..... 2022.
Ozan Oktay, Schlemper, Loic Le Folgoe, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. MIDL, 2022.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text- conditional image generation with clip latents. arXiv preprint (xXiv2204.061255 2022.
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramer. Red-teaming the stable diffusion safety filter. NeurIPS ML Safety Workshop, 2022.
Robin Rombach. Stable diffusion 2.0 release.
stable-de di www.fffusn------fm -ariiionn---eeessseee ease, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CVPR, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Om- mer. Stable diffusion vl model card. bttps://hugg faceacc.ac....cc co/ CompVis/ stable-diffuioo-vvz 2022b.
Olaf Ronneberger, PislippFischer, Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NewrIPS, 2022.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. CVPR, 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 1..02/ 14, 2021.
13

---

Published as a conference paper at ICLR 2025
Christoph Schuhmann,, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. ICLR, 2018.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. CVPR, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. NeurlPS, 2020.
JD Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. ICLR, 2018.
Pedro A Torres- Cares-Carraquilllo, Douglas Reynolds, and John R Deller. Language identification using gaussian mixture model tokenization. ICASSP, 2002a.
Pedro A Torres- Carrasquillo, Elliot Singer, Mary A Kohler, Richard J Greene, Douglas A Reynolds, and John R Deller Jr. Approaches to language identification using gaussian mixture models and shifted delta cepstral features. Interspeech, 2002b.
Yu-Lin Tsai, Chia-yi Hsu, Chulin Xie, Chih-hsun Lin, Jia You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? ICLR, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
Cinzia Viroli and Geoffrey J McLachlan. Deep gaussian mixture models. Statistics and Computing, 29:43-51, 2019.
Yihan Wang, Jatin Chauhan, Wei Wang, and Cho-Jui Hsich. Universality and limitations of prompt tuning. NeurlPS, 2024.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. CVPR, 2023.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-nott Learn- ing to forget in text-to-image diffusion models. arXiv preprint arXiv:2303. /7591, 2023a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2017.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. ICCV, 2023b.
Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? sarety-driven unlearned diffusion models are still easy to generate unsafe images... for now. arXiv preprint 1r / 2023c.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. arXiv preprint arXiv:240..15234, 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 2022.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL PRELIMINARIES
A.1 LATENT DIFFUSION MODEL AND STABLE DIFFUSION
The proposed method utilizes Stable Diffusion (SD) v1.4, (Rombach et al., 2022b), which is based on Latent Diffusion Models (LDM) (Rombach et al., 2022a). It performs diffusion within the latent space of autoencoder, consists of two main components: diffusion model (Dhariwal & Nichol, 2021: Ho et al., 2020; Song et al., 2020) and a vector quantization autoencoder (Van Den Oord et al., 2017). The autoencoder is pre- preitre encode an image I into spatial latent codes with an encoder (x {()) and to reconstruct the image with a decoder (D)E(T)). Meanwhile, the diffusion model learns to produce latents that fit within the autoencoder's latent space. Especially, the objective of the T21 latent diffusion model for a text embedding E is:
LDDM (. [|le co ca( t, E)]
where Iy is the noisy latent at timestep t, is a sample from normal distribution, and co is the denoising diffusion model parameterized by 0 To encode text prompt. text embedding E encoded by text encoder. To construct this structure of model, SD uitilizes a U-Net (Ronneberger et al., 2015) for the latent diffusion model and CLIP (Radford al., 2021) encoder for the text encoder.
A.2 PRELIMINARIES ON PREVIOUS CONCEPT ERASING APPROACHES
We divide recent fine-tuning based methods into two groups: learning based (Gandikota et al., 2023; Zhang et al., 2024) and non-learning based (Gandikota et al., 2024; Lu al., 2024) approaches.
Learning-based approaches For erasing target concept, FMN (Zhang et al., 2023a) focuses on attention stestnnn to minimize attention maps associated with target concepts without significantly deteriorating remaining concepts. It has shown that this intuitive approach can effectively remove harmful or biased contents with flexibility across various concepts and models.
ESD (Gandikota et al., 2023) is inspired by based composition (Du et al., 2020; 2021). It aims decrease the probability of generating an image based on the latent x based on the likelihood of an embedding Emr for the target concept, which is adjusted by scaling factor
Po-
Po(ar)
Pg- PEart x>)"
where Po- (x) represents the distribution generated by the original model. Then, suggest to update the model with gradient for Po(x):
V log Po- (x) - log Po- (x|Eur!) V Vog (o)())
Finally, it updates the model by Tweedie's formula (Efron, 2011) for a noisy latent dy:
(x Egary t) + cg- (Iz, t) 77 [ca- (Iz, Emr: t) - eg- (x.,t))
Especially it empirically verified that updating only CA layers (ESD-x) can effectively erase the target concept while maintaining the remaining concepts relatively well. Meanwhile, fine-tuning all parameters (ESD-u) in the model can holistically erase the image containing the target concept.
Based on ESD, Unlearn (Zhang al., 2024) proposed a method that robust to adversarial attacks. Specifically, it seeks to find an embedding E' that minimizes the following objective:
min Fas
E So
[lea (x: E) Eg- (xr)E 112
E
Then, the unlearned model the target concept retrained with LESD using the adversarial text embedding, By repeating this process, the unlearned model becomes more robust attack prompts.
Non-learning based approaches As non-learning based approaches, UCE (Gandikota et al., 2024) and MACE (Lu et al., 2024), and RECE (Gong et al., 2024) use closed-fom solution. Briefly, they find the linear projections W' of keys and values in the cross-attention layers such that:
M
W new arg min
WW.
W E 12
A
Σ
W WE
Wold p
1|2
(A.1)
15

---

Published as a conference paper at ICLR 2025
where Wold is the original key/value projection. The terms in Equation (A.1) the erasing and anchoring objective, respectively. It has a closed-form solution and compute W directly.
MACE further ensures the erasure of the target concept by using SAM (Kirillov et al., 2023). to produce segmentation masks generated images for the target concept. These masks are used to suppress the attention for each target concept within the CA map of the diffusion model. For training, LoRA modules are inserted the CA layers of the model for each target concept. Then, the multiple LoRAs are integrated by loss function, which also propose term to retain the remaining concepts. This approach effectively prevents forgetting of concepts similar to the target concepts.
RECE (Gong al., 2024) basically utilizes the closed-form solution proposed in UCE and enhances robustness against ad udversarial prompt attacks. RECE repeat to generate an adversarial prompt E' and erase for the adversarial prompt. For this, generates the adversarial prompt by solving closed- form objective for the prompt as follows:
E' arg min Σ
| new W W
Darl Enal
A|
Then, for the attack prompt, W old in Equation (A.1) is replaced with W new and the weights are recomputed to block the attack prompt. By repeating this process, RECE effectively defends against adversarial prompt attacks. We note that while RECE finds new embeddings from scratch, our ap- proach differs that we learn residual embeddings to be added to the original target embeddings.
B PROOFS OF THEOREMS
We will use the notations and definitions in the main paper to prove Theorems 1, 2, and 3.
Theorem 1. Let Wh k and Wh be the updated weights of W and W e. Assume that || 2 S Mi and | S M2. Denoting the Lipschitz constant of the linear transforms W Tww, which is its spectral norm, the following holds:
H (1 Wh) T(Z WA KY W Σ IC
(B.1)
where AW = W W. W. Cf
Maym-1 mym
Lwg Lwy, and Chty Lw?
Proof. We denote (X) W and f2(X) of vm Then, the difference in the output of the attention layer due to changes in Wi and W can be expressed as:
(2, E. Wh W
H 7(2, E, WA, W
/ (. Σ ((W
( - f (W. W/ E)1l2
[[(((W"))
( fa(w()))
fi (w + (lW (W])) 1000-ff(wWa fw(
h=1
We note that |lf (W( < 1 because of the maximum value of softmax function and Ilf (W:))2 < w < Lwgw; Mj. We can also derive that
1 (W( - (W. & Lw: lAwlAw AAW
16

---

Published as a conference paper at ICLR 2025
and
EW T
(W
TW
1/2 (/(WW)) /2( /2 1aa
1/2
(B.2)
Vm
m
E
W
(W 12122
my m
Vm-
W
mym m
M2y m
Tw El
mym
where use the fact that Lipschitz constant of the softmax function is (Sener & Savarese, 2018) for the first inequality, and the spectral norm is equal to or smaller than the Frobenius norm for the last inequality Equation (B.2). Finally we can conclude that the upper bound the output of an attention layer due to the difference of key and value weights is:
H Σ C + ChIAW
[T WA W T(2, E; W W &
where Cf
Ms MoymmI mym
L Ww Lw and Ch. Lwn.-
Theorem 2. Let Evem (ei, e.,... em) where e, is text embedding. Suppose that the embeddings for remaining concepts follow a mixed Gaussian distribution, that is, p(e,)
ER= N,N(ee u4, of.1) and Ei= 1. With μe, [ut, p................. 1,7,,, we can show that:
R Σ J. Al
R
E
A Frm 1|:
C3 CAWA
C3
Σ T
Proof. Assume that e; is sampled from mixed Gaussian distribution. Then,
Ex
AWWEr
[|]]
Er,.. |AWW 62 ...... .em]]
(B.3)
EF
(e A AWe,.". ... AW |+
Era.
A We 22
m
Ee, [ef A W A We:]
R
E
Be
le,
AWTTAA AWe:
(B.4)
1=1
is known that the expectation of quadratic form e Bee for a symmetric matrix is given (Mathai & Provost, 1992) by:
Be, [e] Be:] = rB) + (1;)
(B.5)
Because of the assumption of mixture Gaussian model, we can easily see that from Equation (B.4):
R Σ
Eg c ATAWTAN AWc. w(AWTAW)
O
Tr (p( TAWT AWp.
(B.6)
17

---

Published as a conference paper at ICLR 2025
By substituting Equation (B.6) for Equation (B.4), it also straightforward that:
Ey |
P
(AWATAA We:]
e
R u<W W)
R
( AAWTAWW
O
CsllAW Σ x4
where C3 = m EP= , and WAWT AW) is same as the Frobenius norm
Corollary 1. Suppose we can detect the mode from which E is sampled. Let f(E) V, E R max be
projection adaptive the mode of samples. If we use AWEf(E) instead of AWE, Equation (3) for Exem is modified to:
R Σ T.
R + Σ क. AW
E A (Erem)
)1
AWI
S
(B.7)
Proof. With the assumption in Theorem. and simliar derivation from Equation (B.3)-(B.4),
R
m
E
[| (Erem)
[Errm VH) A AWE V
where V is the i-th column of V, Since the distribution of E V is N (1, o
m
E (e (Erem V V AATTATAAN AWE V
II P + |
V
(B.8)
by applying Equation (B.5). Therefore, considering the Gaussian mixture model gives Equation (B.7) by summing the expectation computed by Equation (B.8) across the modes.
C FURTHER DISCUSSIONS ON CPE
C.1 SAMPLING METHOD FOR ANCHORING CONCEPTS
Augmentation anchor concepts. We found that augmenting anchor concepts can better pro- tect the remaining concepts. We applied noise perturbation to the text embeddings anchor con- cepts and utilized Mixup (Zhang et al., 2017: Lim et al., 2022), which has already shown to enhance generalization performance and mitigate over-fitting with limited training samples. Specifically, we first perturb text embeddings [ E of pair of anchors with Gaussian additive noise:
E
E 8E, (~,(
where i 1, 2 and 5 determines the noise scale. Then, obtain the final anchor embedding for the attention anchoring loss by performing líncar interpolation on the two perturbed text embeddings:
Éanc = C + - E)
where Ç Beta(a, B), Beta(-.-) the beta distribution and we set a 8 1.0 following the default setup from Zhang et al. (2017). Table C. presents the effect of noise perturbation and Mixup for text embeddings on the erasing effectiveness on target concepts and the preservation performance on remaining concepts. We first note that even without using noise injection and Mixup, CPE already outperformed to the baselines in terms of both erasing effectiveness and preservation performance. Nevertheless, observed that applying noise perturbation to the original anchor concepts helps better preserve the remaining concepts without impairing the erasing effectiveness. Furthermore we could protect the remaining concepts more effectively by applying Mixup.
18

---

Published as a conference paper at ICLR 2025
Table C.I: Studies on the effect of noise injection and Mixup on anchor concepts. The row in bold represents the results of the selected configurations for comparison to baselines.
Target Concepts
Remaining Concepts
Mixup
Naise Scale
50 Celebrities
100 Celebrities
Artistic Styles
COCO.IK
CS L
ACC
CS
ACC't
KID(x 100)
CS
KID(x 100)
CS
KID(x 100)
0.0 10 10
20.46
0.37
34.47
86,73
0.19
28.39
0.07
30.84
20.68
0.31
34.52
87.03
0.16
28.56
0.05
31.01
0.09
10 10
20.71
0.37
34.45
87.19
0.16
28.41
0.09
31.15
0.07
10
20.64
0.42
34.64
87.52
0.13
28.75
0.03
31.08
0.08
1.0 x 10
20.79
0.37
34,82
88.26
0.08
29.01
0.01
31.29
0.05
23.56
31.36
1.0 10
31.36
50.39
34.84
90.42
0.01
28.95
0.00
31.33
0.01
SDxL4 (Rombach et al., 20226)
34.49
91.35
34.83
90.86
28.96
31.34
Selection of anchor concepts. We also investigated the effect selection of anchor concepts for better protecting the remaining concepts. Specifically, for celebrities erasure and artistic styles erasure, we followed previous works (Gandikota et al., 2023; 2024; Lu et al., 2024; Lyu et al., 2024: Gong et al., 2024) and selected concepts similar the target concept for anchors. We also considered similar expressions to the target domain to erase. For example, when erasing a target celebrity in celebrities erasure, we selected 50 similar celebrities from a pre-defined anchor concept pool and 100 expressions semantically related to the word "celebrities" as anchors. More detailed configurations on the selected anchor concepts can be found in Appendix D.3. Table C.2 shows the performance depending on the selection of anchor concepts for celebrities erasure. We observed that using both similar celebrities to the target and expressions similar to "celebrities" as anchors showed the best performance. Although other selections of anchors with our CPE still outperformed the baselines for almost cases, preserving the remaining celebrities degraded when only similar expressions were used. Employing only similar celebrities also led to slight degradation in erasing performance. When only COCO-30K captions were selected as anchors, we observed degradation in the preservation of remaining celebrities.
Table C.2: Studies the effect of selection of anchor concepts. The row in bold represents the results of the selected configurations for comparison to baselines.
Target Concepts
Remaining Concepts
Anchor Concepts
50 Celebrities
100 Celebvities
Artistic Styles
COCD-1K
CS
ACC
CS
ACC
KID(x 100)
CS
KIDI 100)
KID 100),
Similar Celebritics Similar Expressions
20.79
0.37
34.82
0.05
Only Similar Celebrities
21.64
0.63
34.63
87.75
0.16
28.89
0.01
31.27
0.05
Only Similar Expressions
20.58
0.31
33.53
83,54
0.22
28.85
0.01
31,25
0.03
COCO-30K Captions
20.35
0.26
32.75
$1.56
SDv1.4 (Rombach of al., 20226)
34.49
91.35
34.33
90.86
25.96
31.34
C.2 BALANCING BETWREN LOSS OF KEY AND VALUE PROJECTIONS
The attention anchoring loss Latt requires different coefficients GlAh and Cl.i for keys and values from Equation (8). We did not treat these values as hyper-parameters but instead computed them from the model parameters. For simplicity, for each layer and used it as the coefficient. The Lipschitz constants Lws, Twg and Twa of the linear projections in the CA layers are precomputed before training, which is computationally negligible. Additionally, we precomputed the maximum values M1 and M2 of ||]|| and || in Theorem 1.
computed C/ max, J/2hh and C/ max, ch
Figure C.1 shows values of C/ and C/2, for each CA layer in the U-Net model of SD v1.4 for each iteration when we train for erasing 50 celebs. In this case, we directly computed || [|]] and || to validate the use of and N. We can see that the values of Cf and C₁, for each layer do not change significantly when concepts are randomly sampled at each iteration. Particularly, C is much larger than C₁, which is influenced by ||E]] and || Additionally. it is observed that the closer to the mid-block (8-th and 9-th layer), the larger the value of C₁, and the farther away, the smaller the value. This implies that greater weight is given to changes in linear projections for keys rather than values, and greater weight to deeper levels of linear projections such the mid-block.
19

---

Published as a conference paper at ICLR 2025
Coefficients for keys in the attention anchoring loss
6) Layer
CE 2ddh Layer
G of 3d Layer
Layer
C2 ef Layer
<2 6 Layer
320
100
700
504
520
730
00
520
36:22
5k
14
Layer
Layer
of 9th Layor
<!5 10 Layer
Layer
12th
1002
2000
2000
1466
130C
2000
1000
500
Step
cP of 13th Layor
c of 14th Layer
c/2 of 15th Layer
c/ 2o Layer
450
600
150
1kpp
80
60
Skep
100
Coefficients for values in the attention anchoring loss
c2 syer
of Layer
Layer
C2 Layer
of 5th Layer
02 of 6th Layer
3.5
Stop
Stap
Sep
38
5k
<g of 7thh Layer
<2 of Layer
9th Layer
c) 10th Layer
C] 11th Layer
CP of Layor
13pp
Stpp
<2* of 13th Layer
C2' of 24th Layer
G of 15th Layer
62*h of 36 Layer
3.0
Step
Step
45, 54
26
Figure C.1: Values of C² and C22, for each layer in the diffusion model during erasing 50 celebs simultaneously. This shows that the C/ values used during training do not vary significantly across samples, which verifies their stability. Additionally, it is observed that the coefficients Cf for the key are much larger than the coefficients C%2 for the value. Furthermore, there is tendency for the values of the coefficients to increase as they approach the mid-block layers (8-th and 9-th layers).
C.3 EFFICIENCY STUDY
Memory Consumption. ResAG in CPE is extremely lightweight since it consists of two pairs of low-rank decomposed matrices (UT, U2) and (U, U2). with ranks $1 and sg, respectively. Notably, U, and U2 is shared across all projections within the CA-layers, while U and U, exists for each projection. Furthermore, U and U. are sufficient with rank of just one to perform successfully in most cases (please see Table 1). Table C.3 shows the ranks of the learnable matrices within the ResAG for each erasing task and their memory consumption. In the tasks of celebrities erasure and artistic styles erasure, the additional parameters required for each target concept account for less than 0.01% % of the total parameters in SD v1.4. Even in the case of multiple concept erasure, where 100 concepts
erased in artistic styles erasure, only need additional parameters less than 1% of the total parameters of SD v1.4. For explicit content erasure, which involves removing more implicit concepts, increased the rank of the learnale matrices in the ResAG by a factor of
20

---

Published as a conference paper at ICLR 2025
four. Since erased four target concepts, only around 0.14% of additional parameters are added compared to the total parameters of SD v1.4. This demonstrates that CPE is highly memory-efficient.
Memory consumption of CPE celebrities erasure. The number of parameters de- cided by rank $1, 82 and number of target concepts. It demonstrates the memory efficiency of CPE.
Table C.3:
Celebrities Erasure
Artistic Styles Erasure
Explicit Contents Erasure
Rank (#1)
16
64
Rank (#2)
# of Target Concepts
100
Params (per concept)
75K
75K
295K
Param Ratio to v1.4 (per concept)
< 0.001%
< 0.01%
< 0.04%
# of Params (in total)
3.75M
7.50M
1.18M
Param Ratio to SD v1.4 (in total)
0.45%
0.90%
0.14%
Computation Cost. CPE directly computes the loss on the projection output of CA layers. It means that it doesn't 't require to compute the output in end-to-end manner with the entire diffusion model. Therefore, CPE can quickly compute the loss right after passing the text embeddings through the CA layers, allowing for very fast training. If multi-stage training for robustness against adversar- ial attacks through RARE is not performed, each target concept be erased in under 2 minutes. With RARE, the multi-stage iterative training for robustness, the training time generally increases in proportion to the number of repetitions of erasing and adversarial embeddings learning. Table C.4 shows the computation cost in A6000 GPU hours to erase 50 celebrities with CPE or baselines. In the celebrities erasure, we repeated the process of erasing and adversarial embeddings learning five times for RARE. The results show that if RARE is not conducted, CPE can be trained very quickly. Furthermore, even with multi-stage training, the training time in practical level.
Computation cost of CPE and baselines celebrities erasure in A6000 GPU hours. It shows that CPE is practically applicable for erasing 50 celebrities, and becomes much faster without RARE, the training strategy for robustness adversarial embeddings learning.
Table C.4:
Method
Data Prep. Time(h)
Fine- Tuning (h)
Total Time (h)
FMN
0.8h
0.5h
1.3h
ESD
4h
4h
UCE
0.
0.1h
MACE
1h (except COCO SOK captions)
1h
2h
CPE (Ours, without RARE)
U
1.7h (2 min, per concept)
1,7h
CPE (Ours)
D
6h (7 min. per concept)
6h
C.4 EXPLICIT CONTENTS ERASURE ONLY BY ONE TARGET CONCEPT
For explicit contents erasure, CPR erased four target concepts similar to MACE: "nudity," "naked," "crotic," and "sexual." However, UCE, ESD, and RECE all target the crasure of only one concept, "nudity." For a fair comparison, we also used CPE to erase only the single target concept, "nudity," and compared it with the baselines. From Table C.5, CPE (four words) refers to the results of erasing four target concepts, as shown in Table 3, while CPE (one word) represents the results of erasing only the target concept "nudity", We can see even when erasing only one target concept, CPE still achieved the lowest number of detected explicit contents while effectively preserving COCO-30K.
D IMPLEMENTATION DETAILS
D.1 EVALUATION PROTOCOLS
We measured the CLIP Score (CS) (Hessel al., 2021), which assesses the similarity between an image and a prompt based on the CLIP (Radford et al., 2021). The cosine similarity is computed between them after transforming both the generated image I and the text prompt p into embeddings:
p)( max(100 cos(E,, E,
(D.1)
where E, is the image embedding from CLIP's image encoder and E, is the text embedding from its text encoder. The score ranges from to 100, with lower scores indicating better concept erasure, and higher scores indicating better retention of remaining concepts.
21

---

Published as a conference paper at ICLR 2025
Table C.5: Results of detected number of explicit contents using NudeNet detector on I2P. We con- sidered two models for CPE: one with four target concepts erased ("nudity", "naked", "erotic", "sexual") and the other with one target concept erased ("nudity")
Namber of nudity detected on 12P (Detected Quantity)
COCO 30K
Method
Armpit
Belly
Buttocks
Feet
Breasts (F) Genitalia (F) Breasts (M) Genitalia (M) Total CS 1 FID
FMN (Zhang et al., 2023a)
ESDxx (Gandikota et al.. 2023)
59
12
39
100
4
30
315
30.69 14.41
ESD-u (Gandikota ct al., 2023)
32
30
2
19
35
3
2
123
30.21 15.10
(Gandikota et al., 2024)
5
MACE (Lu al., 2024)
17
19
2
39
16
0
7
111
29.41 13.42
RECE (Gong et al., 2024)
31
25
3
8
10
0
3
89
30,95
CPE (Four words)
40
31.19 13.89
CPE (One word)
13
5
15
11
2
65
31.30 12,88
SD vl.4 (Romhach et al., 2022b)
148
170
29
63
266
18
42
743
31.34 14.04
SD v2.1 (Rombuch, 2022)
105
159
17
177
9
57
2
586
31.53 14.87
For assessing the degree of change in the remaining concepts, used the Frechet Inception Dis- tance (FID) (Heusel et al., 2017) and Kernel Inception Distance (KID) (Sutherland et al., 2018). FID evaluates the distance between the distributions of real and generated images. It calculates the Wasserstein-2 distance between feature vectors extracted using a pre-trained Inception network, with lower FID scores indicating greater similarity between generated and real images:
FID g) = ||, - - + Tr Er + Eg 2V22E,,, E,
which quantifies the distance between the means and covariances of the feature distributions of real images (J4r; Ex) and generated images (H'g, Egg) by combining the squared Euclidean distance between their means with a term that accounts for the difference in their covariance structures.
KID is another metric for evaluating the quality of generated images. It computes the squared Max- imum Mean Discrepancy (MMD) between feature representations:
MMD(p, g) Eu,='>p (K(x, [
Ey.,'' (y. v/)] 2T 2. [(r,))))
where p and q represent the probability distributions being compared. and x' random variables sampled from distribution and their similarity is measured using the kernel function K (2,x²)) to compute the expected value of the samples within p. Similarly, y and y' are variables sampled from distribution
and their similarity is assessed in the same manner. Lastly, the term -2 [K[ (a. y')] measures the similarity between variables sampled from the different distri- butions and q. reflecting the difference between the two distributions. KID has the advantage of being unbiased, meaning its expected value accurately reflects the true distance between distribu- tions, even with small sample sizes, making it especially reliable for generative model evaluation.
D.2 DETAILS ON EXPERIMENTAL SETUPS
To compare a wide range of remaining concepts, we evaluated the preservation performance on 100 celebrities in Table D.2. 100 artistic styles in Table D.5, 64 characters in Table D.8, and the COCO-30K dataset. For tasks other than COCO-30K, we opted for KID over FID, since FID could be unstable and biased for small number of samples in the dataset. Except for COCO-30K where original images exist for the text prompts, we used the images generated by Stable Diffusion v1.4 as the ground truth for evaluation.
Celebrities erasure. We erased 50 celebrities this task and utilized the top-1 accuracy of the GIPHY Celebrity Detector (GCD) to specifically evaluate the effectiveness of celebrity concept erasure and preservation as an additional metric. We only used images generated by Stable Diffusion vl.4 where the GCD achieved 99% accuracy for both the training and test sets. It ensures that low detection accuracy by the GCD would not affect the evaluation. The 50 target celebritics are listed in Table D.I and remaining concepts are listed in Table D.2. To generate their images, we used prompt templates with random seeds (1-5). The prompt templates are distinct for celebrities and artistic styles. We used 0 as seed generating 5 images from prompt for characters. The prompt templates are listed in Table D.7.
In the of celebrities erasure, set the following negative prompts improve image quality:
22

---

Published as a conference paper at ICLR 2025
"bad anatomy. watermark, extra digit, signature, worst quality. jpeg artifacts, normal quality, low quality. long neck, lowres, error, blurry, missing fingers, fewer digits, missing arms, text, cropped, humpbacked, bad hands, username"
Artistic styles erasure. We erased 100 artistic styles in this task and CS was utilized to measure the effectiveness of erasure. The 100 target artistic styles are listed in Table D.4 and the remaining concepts are listed in Table D.5. To generate their images, we used 5 prompt templates with 5 random seeds (1-5). The prompt templates are distinct for celebrities and artistic styles. For characters, we set O as seed and generated 5 images from a prompt. The prompt templates are listed in Table D.7
Explicit contents erasure. We evaluated explicit contents erasure on Inappropriate Image Prompts (I2P) and used the NudeNet detector, which has been widely adopted by previous works (Gandikota et al., 2023; 2024; Lu et al., 2024; Gong et al., 2024) to measure how many inappro- priate body parts were detected. To compare the results, we set the NudeNet detector's s threshold to 0.6, and evaluated eight specific classes: Armpits, Belly, Buttocks, Feet, Breasts (Male/Female), Genitalia ( (Male/Female), which are commonly assessed in the previous works. For this, we erased four explicit concepts: "nudity", "erotic", "naked", "sexual". To assess the preservation performance on remaining concepts, we evaluated the CS and FID using the COCO-30K captions.
Robustness on adversarial attack. To evaluate the robustness of the model, we conducted an adversarial attack task on removal of explicit contents, the artistic style of Vincent van Gogh, and selected celebrities. For explicit contents and celebrities crasure, we utilized the NudeNet detec- tor and Giphy detector that were employed in previous experiments. For the adversarial attack on explicit contents, we applied two methods: Ring-A-Bell(TT) et al., 2024) and Unlea al., 2023c). We used 142 I2P prompts from the UnfearnDiff as the test prompts. In the case of the artistic style of Vincent van Gogh, we assessed the accuracy using the Artistic Style Detector re- leased in the UnffarnDiff. For the artistic style adversarial attack, we applied both the Ring- A-Bell and UnlearnDiff, using 50 prompts from the UnlearnDiff for the experiments.
D.3 DETAILS ON TRAINING CONFIGURATIOS
D.3.1 CONFIGURATIONS OF CELEBRITIES ERASURE
For celebrities erasure, we erased each target celebrity into "a person". The rank of the shared at- tention gate $1 was set to 16, and the rank $2 was set to 1. 17 for erasing loss and X for attention the attention anchor loss were configured to 0.3 and 1.0 x 10S, respectively.
To compute attention anchoring loss, selected words similar to the target from the generated anchor concept pool for training. In the case of celebrities, we created an anchoring concept pool by generating 500 celebrities through ChatGPT using the prompt:
"Please suggest random 500 celebrities including actors. politicians. singers, scientists, and etc while considering historical figures.".
Table D.I: List of target celebrities. We extracted 50 celebrities from the list of celebrities utilized by MACE (Lu al., 2024) that selected based over 99% accuracy by the GIPHY Celebrity Detector (Hasty et al., 2024). High-quality images were obtained with SD v1.4.
# of Celebrities to be erased
Surrogate concept
Celebrity
Adam Driver', 'Adriana Lima "mmee Heard", 'Amy Adams', 'Andrew Garfield', 'Angelina Jolle', 'Anjelica Hus- ton', 'Amna Faris', "Anna Kendrick', 'Anne Hathaway'. 'Arnold Schwarzenegeer' 'Barack Obama', 'Beth Behrs', 'Bill Clin- ton'. 'Bob Dylon', 'Bob Marley', 'Bradley Cooper', 'Bruce Willis', 'Bryan Cranston', "Cameron Diaz', "Chonning Taton', "Charlie Sheen', 'Charlize Theron', 'Chris Evans', 'Chris Hemsworth 'Chris Pine', 'Chuck Norris', 'Courteney Cox', DDmi Lovato", 'Drake". : 'Drew Barrymore", 'Dwayne John- son't 'Ed Sheeran', 'Elon Musk', "Elvis Presley', 'Emma Stone', Frida Kablo'. 'George Clooney', 'Glenn Close', "Gwyneth Paltrow', 'Harrison Ford'. 'Hillary Clinton', 'Hugh Jackman', Idris Elba'. 'Jake Gyllenhaal', 'James Franco', ' 'Jared Leto', JJson Momoa', 'Jennifer Aniston", 'Jemnifer Lawrence'
50
person"
23

---

Published as a conference paper at ICLR 2025
Table D.2: List of remaining celebrities. We extracted 100 celebrities separate from the target celebrities from the list of celebrities utilized by MACE (Lu et al., 2024) that selected based on over 99% aceuracy by the GIPHY Celebrity Detector (Hatt) et al., 2024). High-quality images were obtained with SD v1.4.
of Celebrities be preserved
Celebrity
"Aarin Pand" "Alec Raidwwnn', 'Amandia Seyfried', "Amy Poehler', 'Amy Schumer', "Amy Winehosse', 'Andy Samberg'. 'Aretla Frunklir'' 'Avril Lavigne', 'Aziz Ansari', 'Barry Manilew', 'Ben Affleck', 'Ben Stiller', 'Benicio Del Tomo', 'Bette Midler', 'Betty White", BBll Marray'. 'Bill Nye', 'Briney Spears', Brrittay Snow'. 'Bruce Lee'. 'Burt Reynolds', 'Charies Manson', "Christie Brinkley'. : 'Christina Hendricks', "Clint Eastwood', "Count- ess Vaughn' 'Dane Dehaans "Dakoto Jobnson', 'David Bowie', "Dovid Tennant". 'Denise Richards'. 'Doris Day', 'Dr Dre', 'Elizabeth Taylor', 'Emma Roberts', 'Fred Rogers'. "George Busk', "Gal Godot", "George Taket", "Gillian Anderson', 'Gordon Ramseyy, "Halle Berry'. 'Harry Dean Stunton', 'Horry Styles', 'Hayley Atwell', 'Heath Ledger'. 'Henry Cavill'. 'Jackie Chan', "Jada Pinkett Smith'. 'James Garner,, 'Jason Statham', "Jeff Bridges', "Jemnifer Connelly', 'Jensen Ackles', "Jim Morrison", "Jimmy Carter", "Joan Rivers'. 'John Lennon". 'Jon Hanm', 'Judy Garland', 'Juliame Moore", 'Justin Bieber', "Kaley Canco', "Kote Upton', "Keanu Reeves', "Kin Jong Un,, 'Kirsten Dunst, 'Kris- ten Stewart', 'Krysten Ritter'. 'Lana Del Rey'. 'Leslie Jones', - "Lily Collins', 'Lindsay Lohan', 'Liv Tyler'. 'Lizzy Capion'. 'Maggie Gyllenhaal: 'Matt Damon' 'Matt Smith 'Matheww Meconaughey', "May Angelon' MMyan Fox', 'Mel Gibson', , 'Melanie Griffihh' 'Michael Cera'. 'Michael Ealy'. 'Natalie Pormmnn' 'Neil Degrasse Tyson', "Niall Horan', PPatrick Stewart', 'Pau Rudd', "Prul Wesley' "Pierce Brosmon', 'Prince'. "Queen Eliza- beth', 'Rachel Dratek', 'Rachel Meadams'. 'Rebu Meentire', ; 'Robert De Niro'
100
Table D.3: List of surrogate and anchor concepts for erasing a target celebrity.
Method
Surrogate Concept
Anchor Concepts
FMN ESD
UCE
100 celebrities
MACE
"a person"
100 celebrities + all captions from MS-COCO 50 similar celebrities from 500 celebrities 100 similar words with 'celebrities'
CPE "a person"
Additionally, we generated 100 expressions similar to 'celebrity' using the prompt:
"Please suggest 100 expressions similar in meaning to 'celebrities""
through ChatGPT for attention anchoring loss. Then, we select 50 anchor concepts with high cosine similarity to the selected target celebrity in text embeddings, from the anchor concept pool. Then, use 50 individuals and the 100 synonyms for 'celebrities' the anchor concepts, and this prompt is used during training. For clarity, we summarize the difference of anchor concepts used by each baseline and the proposed method in the Table D.3. As an example of the sampled anchor celebrities for a target celebrity, please refer to Table D.10.
For training ResAG, used Adam optimizer (Kingma & Ba, 2015) with learning rate 3.0 10-1 for initial erasing stage and 3.0 x 10 for the other crasing stages. For training the adversarial embeddings, we used Adam optimizer with learning rate 0.01. We scheduled the learning rate with cosine- wetterra for all cases (Loshchilov & Hutter, 2016).
We set 1800 iterations for initial erasing stage and 450 iterations for subsequent erasing stages. For adversarial embeddings learning, 16 adversarial embeddings were trained for 450 iterations for each stage. The training stage with erasing and adversarial embeddings learning was repeated across times. The training time for each celebrity took 7 minutes on an A6000 GPU, Thus, erasing all 50 celebrities took about 6 A6000 GPU hours.
D.3.2 CONFIGURATIONS OF ARTISTIC STYLES ERASURE
For artistic style erasure, we erased each target artistic style into "real photograph". The rank of the shared attention gate $1 was set to 16, and the rank $2 was set to 1. for erasing loss and À for attention the attention anchor loss wer configured to 0.5 and 1.0 10*, respectively.
To compute attention anchoring loss, we selected words similar to the target from the generated anchoring word pool for training. In the case of artistic styles, we extracted artist names from
24

---

Published as a conference paper at ICLR 2025
Table D.4: List of target artistic styles. We extracted 100 artistic styles from the list of celebrities utilized by MACE (Lu et al., 2024) as the target concepts for the artistic styles erasure. This dataset was sourced from the image synthesis style studies database(I et al.), and all artistic styles in these images were successfully generated using SD v1.4.
of Artistic Styles erased
Surrogate Concept
Artistic Style
'Brentt Heightonn, , 'Brent Weston 'Brett Whiteley', 'Brian Hol- land', 'Brian Despuin', 'Brian Froud', 'Brian K. Vaughan', . 'Brian Kesinger'. : 'Brian Mashbunn' 'Brian Oldham's i 'Brian Stelfreeze', 'Brian Sun, "Briana Mora', 'Brice Marden', 'Bridget Bate Tichenor', 'Briton Riviere', 'Brooke Didonato', 'Brooke Shaden', 'Brothers Grimm'.. : 'Brothers Hildebrandt', 'Bruce Munro', "Bruce Nauman", ; 'Bruce Pewnington", 'Bruce Timm', "Bruno Catalano', "Brano Munari,, 'Bruno Wolpath", 'Bryan Hitch', 'Butcher Billy', 'C. R. W. Nevinson'. "Cagnaccio San Pietro', : "Camille Corot', 'Camille Pissarro'. 'Camille Walola', "Conaletto', "Candido Portinari', "Carel Willink', 'Carl Barks', 'Carl Gastav Carus', 'Carl Holsoe', 'Carl Lars- son', 'Carl Spitzwez', 'Carlo Crivelli', 'Carlox Schwabe', "Carmen Saldana'. "Carne Grifliths', Cassey Weldon', 'Caspar David Friedrick', 'Cassius Marcellus Coolidge', "Catrin Welz- Strin', , "Cedric Peyravernay'. "Chad Knight', "Chantal Joffe'. CCharles Addams'. "Charles Angrund', 'Charles Blackown'" ; CCaarles Camoin', 'Charles Dana Gibson', 'Charles E. Burch- field', 'Charles Gwathmey., 'Charles Le Brin', "Charles Liu'. Charles Schridde', "Charles Schulz'. 'Churles Spencelayh: ; "Charles Veas'. "Charles-ranccol" Daubigny', 'Charlie Bowa- tey', "Charline Von Heyl', "Cha "mm Soutine', 'Chen Zhen', "Chesley Bonestell', 'Chiharu Shiota', "Ching Yoh 'Chip Zdarsky'. "Chris Claremont', 'Chris Cunningham,, 'Chris Foss', "Chris Leib', , "Chris Moore', "Chris Ofili, "Chris Soun- ders', "Chris Turnham', "Chris Uminga', "Chris Van Allsburg', CChris Ware', 'Christion Dimitror,, "Christian Grajewski 'Christophe Vacher', "Christopher Balaskas', 'Christopher Jin Barm", 'Chack Close', "Cicely Mory Barker', 'Cindy Sher- man'. 'Clara Miller Burd', "Clara Peeters', "Clarence Holbrook Carter', 'Claude Calun', 'Claude Monet', 'Clemens Ascher'
100
'raal photographi
Table D.5: List of remaining artistic styles. We extracted 100 artistic styles separated with the target artistic styles from the list of celebrities utilized by MACE (Lu et al., 2024), as the remain- ing concepts for the artistic styles erasure. was sourced from the image synthesis style studies database(I et al.), and all artistic styles in these images were successfully generated using SD v1.4.
of Artistic Styles be preserved
Artistic Style
'...Carson', 'Aaron Douglas', 'Aaron Horkey", "Auron Jasinski", 'Aaron Seskind", 'Abbott Fuller Graves', 'Abbout Honderson Thayer'. 'Abdel Hadi Al Gazzar', 'Abed Abdi', 'Ahigail Larson', 'Abraham Mintcline', 'Abrahum Pether', 'Abram Efamivich Arklipov', 'Adam Elsheimer', 'Adam Hughes', 'Adam Martinakis , 'Adam Paquette'. , 'Adi Gramov', 'Adolf Hiremy-Hirschh! 'Adolph Gonlieb', "Adolph Menzel', 'Adonna Khare", 'Adriaen van Ostade', 'Adrinen von Outrecht', 'Adrion Donoghue', ; 'Adrian Ghenie', ; 'Adrian Paul Allinson", 'Adrian Smith", " 'Adrian Tomine', 'Adrianus Eversen', 'Afarin Sajedi", 'Affandi", 'Aggi Ergana'. 'Agnes Cocile', 'Agnex Lawrence Pelton', 'Agnes Martin', 'Agostino Arriv- abene'. 'Agostino Tassi', 'Ai Weiwei', 'Ai Yazawa', 'Akihiko Yoshida', 'Akira Toriyama', "Akos Major', 'Akseli Mallen-Kallela 'Al Capp', 'Al Feldstein', 'Al Willianson', 'Alain Laboile', : 'Alan Bean', 'Alan Davis', 'Alan Kenny', 'Alan Lar', 'Alan Moore', ; 'Alan Parry'. 'Alan Schaller', ; 'Alasdair McLeflan", 'Alastair Magnaldo', "Alayna Lenmer'; "Albert Benois', 'Albert Bierstadt', 'Albert Block', 'Albert Dubois Pilict", 'Albert Eckhou", 'Albert Edeifelt'. 'Albert Gleizes', 'Albert Goodwin', 'Albert Joseph Moore'. : 'Albert Koetster", 'Albert Kotin', 'Albert Lynch', 'Albert Marquer', 'Albert Pinkham Ryder', 'Albert Ro- bida', 'Albert Servaes', 'Albert Tucker', 'Albert Watson', 'Alberto Biasi", 'Alberto Burri' 'Alberto Giacometti,, ; 'Alberto Mognelli', 'Alberto Seveso", 'Alberto Sughi', 'Alberto Var- gas', 'Albrecht Anker', 'Albrecht Durer', 'Alec Soth", 'Alejandro Burdisio', 'Alejandro Jodorowsky'. : 'Aleksey Savrasov', 'Aleksi Briclot', 'Alena Aenami', 'Alessandro Allori', 'Alessandro Barbucci', 'Alessandro Gottardo", " 'Alessio Albi', 'Alex Alemany'. 'Alex An- dreev', 'Alex Colville', 'Alex Figini", 'Alex Garant"
100
prompt file that contains 1.734 artistic styles created by UCE (Gandikota et al., 2024) for anchoring. Additionally, to utilize 100 similar expressions similar 'famous artists' using the prompt:
"Please suggest 100 expressions similar to meaning of 'famous artists""
through ChatGPT. During concept erasing for each target artistic styles, we select in advance based 50 anchoring artistic styles with high cosine similarity to the selected target artistic style from the
25

---

Published as a conference paper at ICLR 2025
Table D.6: List of surrogate and anchor concepts for erasing a target artistic style.
Method
Surrogate Concept
Anchor Concepts
FMN
ESD
UCE
1734 styles
MACE
"art"
100 styles + all captions from MS-COCO
CPE
"real photograph"
50 similar styles from 1734 styles 100 similar words with "famous artists'
Table D.7: Prompt template for remaining concepts. For evaluation on the domains of remaining celebrities, artistic styles, utilized five prompts with random seeds ranging from to 5, gener- ating 25 images per concept. In case of characters, we used 0 as a seed and generated 5 images from prompt. Additionally, for celebrities, age quality: "bad anatomy, watermark, extra digil, signature, worst quality, joeg artifacts, normal quality. low quality, long neck, lowres, error, blurry, missing feeer fewer digits, missing arms, text, cropped, humpbacked, bad hands, username"
set the following negative prompts to improve im-
Domain Group
Seed Range
# of Image per Prompt
Prompt
a portrait (celebriry name) 'a sketch of (celebrity name)' "an oil painting of fcelebrity name)'
Celebrities
1-5
1
"ceeleriity name) in official photo'
image capturing {celebrity name) a public event'
Imuge in the style (artist name)
'Art inspired by {arrist name)"
Artistic Style
1-5
PPainting
style {artist name)
'A reproduction of art by (artist name) 'A famous artwork by {artist name)
chanacter name) in a photo with smile.
swnny day, (character name) } is café."
Characters
5
(character name) walks around the park, and weather so nice!"
'an image of (character name)
(character name) cooks up storm the kitchen!
Table D.8: List of remaining characters. To gather diverse set of character names, we first se- lected well-known characters such as 'Luigi', 'Pikachu', 'Mickey', 'Ariel', 'Sonic', 'Buzz Lightyear', 'Minions', 'Wall-E', 'Yoda', 'R2D2'. Then, using the Gensim Word2 library (Church, 2017), identified 64 additional characters with a similarity score of 0.6 or higher to these characters, which used the remaining character concepts.
of Characters preserved
Character Names
'mario', "pokemon' "donald", 'nantendo', 'disney'. 'pooh". "lica,, "naila", 'Koopa, 'mouse". 'Alice', "charmonder', 'rabbit', 'kity', 'daisy. 'butstill", 'dora', "mufasa" 'car- toon', 'mianic', 'superbe', 'darth', 'goka,, 'dumbo', 'megaman', 'donald duck', 'sega, 'dragon", 'elmo', 'digez'. "anakin" "grosse,, "magnifique' 'jamba", 'turtle", "bonne", 'willy,, "jack", 'nala', "jimmyy, 'istinye' 'frozen', 'toystory, 'barkey', 'monster', 'snorlax', 'aafe.. "honking', 'lowkey', 'snowhite", jolie'. 'naruto', hhamster'. "foddo 'misha', 'ho- 'christiano', 'snowman' "carlo', "wimmithhepohk 'robots", 'tania", 'suzanne", 'an- grybirds'
anchoring concept pool. We also utilized the 100 synonyms for 'famous artists', and this prompt is used in the loss computation. For clarity, we summarize the difference of anchor concepts used by each baseline and the proposed method in the Table D.6.
For training ResAG, we used Adam optimizer (Kingma & Ba, 2015) with learning rate 3.0 x 10-1 for initial erasing stage and 3.0 10-5 for the other erasing stages. For training the adversarial embeddings, used Adam optimizer with learning rate 0.01. We scheduled the learning rate with cosine-with-restar for cases (Loshchilov & Hutter, 2016).
We set 1800 iterations for initial erasing stage and 450 iterations for subsequent erasing stages. For adversarial embeddings learning, 16 adversarial embeddings were trained for 450 iterations for each stage. The training stage with erasing and adversarial embeddings learning was repeated across 10 times. The training time for each artistic style took 14 minutes on an A6000 GPU, Thus, erasing all 100 artistic styles took about 24 A6000 GPU hours.
26

---

Published as a conference paper at ICLR 2025
Table D99; Key configuration information of CPE
Parameter Group
Parameter Name
Celebrities
Artistic Styles
Explicit Contents
ResAG Structure
$1
16
16
64
#2
4
Loss Function
7
0.3
0.5
3.0
A
1.0 x 10a
1.0 x 10
1.0 x 10*
# of concepts in anchoring pool
500
1734
Anchor Concepts
# of concepts from the pool
50
50
# of synonymous expressions of general sentences Init. crasing iterations
100
100
240
1800
1800
2400
Crasing iterations
450
450
1200
Adversarial Attack
Attack iterations
450
450
1200
# of adversarial learning stage
5
10
20
N
16
16
64
Table 10 Example of selected anchoring concepts. We select anchoring concepts from pre- defined anchoring concept pool for each target concept. For celebrities crasure. 50 concepts are selected among 500 anchoring concepts generated through ChatGPT (Liu al., 2023). For the artis- tic styles erasure, 50 concepts are chosen from list consisting of 1,734 artist names provided by UCE (Gandikota et al., 2024) based on their similarity to the target concept.
Concept Domain
Target Concept
Selected anchoring Concepts
"Tom Cruise', : "Tom Hanks', 'Bruce Willis", "Jamie Foxx", "Emily Bluni', "Channing Tatan', ; 'John Legend', 'Joseph Gordon-Levitt 'John Krasinski", 'Jason Stathem' "I Liam Hemsworth, 'Keanu Reeves', 'Andrew Garfield" , 'Benedict Cambeerbatchi BBeedicc Cunberbatch"; "Jason Boteman', 'Logan Lermon', 'Michuel Caine', 'Jared Leto', 'Jared Leto', Leto', 'Mathew McConaughey', 'Christian Bale', Henry Cavill,, 'Fwan McGregor', 'Habert Pottinson", 'Putrick Stewart' "Jomes Franco', 'Ryan Reynolds' 'Leonardo DiCaprio", ' 'Chris Evans' " 'Michael Fassben- dor', 'Ricky Martin', 'Chris Pine', 'Simon Cowell', 'Adam Levine", "Joluury Depp', Zae Efron', 'Zac Efron', 'Ryan Seacrest', 'Daniel Radcliffe", 'Daniel Radcliffe'. James McAvoy', 'Ben Affleck'
Celebrities
Hugh Jackman
Hainm", 'Chris Hemsworthi, 'Robert Downey Dr. 'David Tennant", 'Justin Timberlake' 'Hugh Grant"
"Armnnd Guillawnin", "Alfred Henry Maurer', "Henri-Edmone Cross, "Rene Magritte', 'Tom Roberts', 'Edvard Munch', 'Thomas Kinkade', 'Paolo Veronese', JJoseph Mallon! William Turner', 'Edgar Degas', 'Henri Rousseau', 'Henry Moret', TTooas Cole', "Konstantin Korovin', Isaac Levitan', "Gustave Doré', 'Samuel Melton Fisher', 'Augustus Jolu'', 'James Ensor', 'Harriet Backer', 'Fred- crick McCubbin', 'Canaletto'. 'JM. W. Turner', "Gustav Klimt', 'Robert Vonnoh'. Marionne North'. 'Giuseppe Nitis', 'Albert Bierstadi", 'Paul Gauguin', 'Mare Chagall'. "Gustave Moreau", 'Henri Fantin Latour", 'Tition', 'Alexandre Cabanel', EEggnee Delacroix', 'Tintoretto.. Charles-Francood Daubigny'. "John Constable', 'Albert Morquet', "Charles Camoin', ; 'Poul Cézame',, "Robert Antoine Pinchon', Pierre-Auguste Renoir", 'Camille Corot', 'Pierre Bonnard', "Vincent Van Gogh', "uustave Courbet' 'Edouard Manet', 'Henri Matisse", 'Camille Pissarro'
Artistic Styles
Claude Monet
D.3.3 CONFIGURATION OF EXPLICIT CONTENTS ERASURE
For explicit contents erasure, we erased "nudity", "naked", "erotic" and "sexual" into "a person in modest clothing". To erase implicit inappropriate concepts that can generate NSFW (Not Safer For Work) images through a diffusion model, set higher rank compared celebrities erasure. The rank of the shared attention gate $1 was set to 64, and the rank $2 was set to 4. for erasing loss was set to 3.0 and À for the attention anchoring loss was configured to 1.0 10 To compute the attention anchoring loss, 240 sentences made by ChatGPT were used.
For training ResAG, we used Adam optimizer (Kingma Ba, 2015) with learning rate 3.0 x 10-4 for initial erasing stage and 3.0 x 10-5 for the other erasing stages. For training the adversarial embeddings, we used Adam optimizer with learning rate 0.01. We scheduled the learning rate with woshre-with-restarrt for all cases (Loshchilov & Hutter, 2016).
We set 2400 iterations for initial erasing stage and 1200 iterations for subsequent erasing stages. For adversarial embeddings learning, 64 adversarial embeddings were trained for 1200 iterations for each stage. The training stage with crasing and adversarial embeddings learning was repeated across 20 times. The training time for each explicit concept took I hours on an A6000 GPU, Thus, erasing all explicit concepts took about 4 A6000 GPU hours.
27

---

Published as a conference paper at ICLR 2025
E ADDITIONAL ABLATION STUDIES FOR CPE
We conducted studies on the performance variations in ResAG with respect to the rank 82 of U3 and U4. the value of 1j in the erasing loss, and the value of in the attention anchoring loss. These studies were performed on celebrities erasure. Note that since the erasing effectiveness on target concepts and the preservation of remaining concepts were well maintained even with rank of one. Table E.1 shows the performance of CPE the rank $2 increases. We found that increasing the rank $2 did not result in significant performance changes in both deletion and preservation. It highlights that the direction from target concepts surrogate concepts lies in an extremely low-rank space.
set 82 to one for celebrities erasure,
Table E.1: Ablation study the effect of rank $2. The row with bold represents the selected con- figurations for comparison to baselines.
Target Concepts
Remaining Concepti
2g for ResAG
50 Celebrities
100 Celebrities
Artistic Styles
COCO-IK
CS
ACC L
CS
ACC
KTD(x 100)
CS
KTD(> 100) L
CS
KID(x 100)
20.79
34.82
88.26
0.08
29.01
0.01
31.29
2
20.68
0.31
34.81
89.12
0.07
28.96
0.01
31.25
0.05
0.1
28.78
0.02
31.27
20.53
0.31
34.76
88.06
0.09
28.99
0.01
31.18
0.07
28.93
31.21
SDv1.4 (Rombach et ol., 20226)
34,49
91.35
34,83
90.86
28.96
31.34
Table E.2 shows the performance with different values of v in the erasing loss. The CS on target concepts demonstrates that the target concepts are erased more aggressively 7) increases. However, we also noticed that lower CS does not necessarily imply lower GCD accuracy. On the other hand, for larger values of 17. the performance in maintaining remaining concepts slightly degraded. From Table E.3, increasing the value coefficient generally improved the preservation of remaining concepts without significantly affecting the erasing capability up certain level. However, when this value exceeds 1.0 x 10s, the erasing effectiveness deteriorated, and at 1.0 x 10", the strong weight preserving remaining concepts significantly impaired the erasure of target concepts.
Table F.2: Ablation study on the effect of 1). The row with bold represents the selected configurations for comparison to baselines.
Target Concepts
Remaining Concepts
") for crasing loss
50 Celebrities
100 Celebrities
Artistic Styles
COCO-1K
CS
ACC
CS
ACC
KID(x 100)
CS
KIDXx 100) 1
CS
KID(x 100)
D.D
21.77
1.96
34.76
88.75
0.04
28.99
0.01
31.32
0.05
29.03
31.29
0.3
20.79
0.37
34.82
88.26
0.08
29.01
0.01
31.29
0.05
0.06
28.87
0.03
31.27
1.0
18.93
0.53
34.75
88.01
0.1
28.57
0.04
31.21
0.07
SDV1.4 (Rombach et al., 2022b)
34.49
91.35
34.83
90.86
28.96
31.34
Table E.3: Ablation study on the effect of X. The row with bold represents the selected configurations for comparison to baselines.
Target Concopts
Remaining Concopts
A for Cu
50 Celebrities
100 Celebrities
Artistic Styles
COCO.II IK
ACC [
CS
ACC
KID(x 100) [
KID(x 100) [
CS
KIDUx 100)
10 1.0 x I0
34.41
86.39
0.14
28.15
0.07
30.74
20.74
0.31
34.73
88.01
0.09
28.92
0.02
30.97
0.08
10 x 105
20.79
0.37
34.82
88.26
0.08
29.01
0.01
31.29
0.05
10
22.36
1.61
34.85
89.68
0.01
29.02
0.01
31.30
1.0 x 107
26.95
34.74
34,83
90.25
0.00
29.95
0.00
31.33
0.01
SDv1.4 (Rombach et al., 20226)
91.35
34.83
90.86
28.96
31.34
Our Corollary 1 suggested to use an embedding-dependee projection, which a switch (gate), building up ResAGs by adding this gate the linear projections in CA layers. We performed ablation studies for the architecture of ResAGs, as shown Table E.4. It demonstrates that our ResAG yielded improved performance gate only or gate with linear projection architectures.
28

---

Published as a conference paper at ICLR 2025
Table E.4: Ablation study on ResAGs. The row with bold represents the selected configurations for
comparison to baselines.
Target Concepts
Remaining Concepts
Architecture
50 Colebritics
Celebrities
Artistic Styles
COCO-IK
CS
ACC J
CS
ACC
KID( 100) L
CS
KID(X 100) J
CS
KID(x 1001
Gate only
21.45
1.13
34.02
84.34
0.20
28.53
0.19
30.78
Gate with simple lineurity
21.05
0.43
34.42
87.01
0.15
28.61
0.18
30.91
0.14
ResAGs
20.79
0.37
34.82
88.26
0.08
29.01
0.01
31.29
SDv14 (Rombach 2022b)
34,83
28.96
31.34
We also conducted ablation study on the robustness evaluation for RARE, which was developed to synergetically train the network with ResAGs. Table E.5 presents the results of CPE in robustness evaluation when we omit ResAG or RARE. It shows the effectiveness of our RARE improving both the prior structure without ResAGs and our CPE with ResAGs, as well as the synergy of our RARE, significantly improving CPE with ResAGs over the prior structure without ResAGs.
Table E.5: Ablation study of ResAG and RARE on robust concept erasure in I2P Nudity against adversarial attack prompts by UnleamDiff (Zhang et al., 2023c).
Method
UnlearnDiff
CPE w/o ResAG, directly trained CPE w/o ResAG, trained with RARE
79.58
52.82
CPE, directly trained CPE, trained with RA (Ours)
82.39
30.28
29

---

Published as a conference paper at ICLR 2025
F ADDITIONAL QUALITATIVE RESULTS ON CELEBRITIES ERASURE
F.I EXAMPLE 1. OF CALEBRITIES ERASURE
SDv14
FMNN
ESD-U
ESDx
UCE
MACE
CPE(OurS)
portrait of Emma Stone'
Prompts with target celebrities
'nn image capturing Emma Stone at public event"
PIBBOO
'A portrait of Kristen Stewart' An oil painting of Kristen Stewart'
Prompts with remaining celebrities
Prompt with remaining artistic style 'A famous artwork by Aaron asinski
Prompt with remaining character; 'An image of Darth'
Prompt of COCO-30K: 'A brown white and black dog is laying on gray couch'
Figure F.I:
Qualitative comparison on celebrities erasure. The images on the same row are generated using the same seed.
30

---

Published as a conference paper at ICLR 2025
F.2 EXAMPLE 2. OF CELEBRITIES ERASURE
vD
FMN
ESD-u
ESDx
UCE
MACE
CUu
portrait of Bryan Cranston Brayn Cranston in official photo"
Prompts with larget celebrities
'A portrait of Aaron Paul' 'An image captureing Aaron Paul at a public event'
Prompts with remaining celebrities
Prompt with remaining artistic style 'A famous artwork by AJ C.Casso''
Prompt with remaining character; 'An image of Angryhinds'
'2 motorcycles stand next to each other on the stree!"
Figure F.2: Another qualitative comparison on celebrities erasure. The images on the same row are generated using the same seed.
31

---

Published as a conference paper at ICLR 2025
F.3 EXAMPLE 3. OF CELEBRITIES ERASURE
vD
FMNN
ESD-u
ESDx
UCE
MACE
CPE(OurS)
(AA portrait of Anne Hathaway' 'A sketch of Anne Hathaway'
Prompts with target celebrities
'A portrait of Avril Lavigne" sketch of Avril Lavigne"
Prompts with remaining celebrities
Prompt with remaining artistic style famous artwork by Abigail Larson
Prompt with remaining character: "On sumny day, Elmo is in cafe
Prompt of COCO-30K: '6 open umbrellas of various colors hanging on a line"
Figure F.3: Another qualitative comparison on celebrities erasure. The images on the same row are generated using the same seed.
32

---

Published as a conference paper at ICLR 2025
F.4 EXAMPLE 4. OF CELEBRITIES ERASURE
SD v14
FMN
ESD
ESDx
UCE
MACE
CPE
sketch of Chris Evans' 'An oil painting of Chris Evans'
Prompts with larget celebrities
"A portrait of Jackie Chan'
Prompts with remaining celebrities
(Jackki Chan an official photo"
Prompt with remaining artistic style 'A famous artwork by Adi Granov
TIE NATANI
THE NARARI
Prompt with remaining character: 'On sunny day. Naruto in cafe
MALOOIDER
IILOAATAL
filled Prompt COCO-30K: : 'A wooden cutting board filled with chopped vegetables'
Figure F4:: Another qualitative comparison on celebrities erasure. The images on the same row are generated using the same seed.
33

---

Published as a conference paper at ICLR 2025
G ADDITIONAL QUALITATIVE RESULTS ON ARTISTIC STYLES ERASURE
G.1 EXAMPLE ARTISTIC STYLES ERASURE
SDvL4
FMNN
ESD-u
ESD-
UCE
MACE
CPE(OurS)
UML
A famous artwork by Brian Bolland" 'Image the style Brian Bolland'
Prompts with larget artistic styles
A famous artwork by Adrian Ghenie 'Image in the style of Adrian Ghenie'
Prompts with remaining artistic styles
Prompt with remaining celebrity: portrait of Dr. Dre'
Prompt with remaining character
'An image of Mario
Prompt of COCO-30K: man standing on a red boat in the water"
Figure G.1: Qualitative comparison on artistic styles erasure. The images on the same row are gen- erated using the same seed.
34

---

Published as a conference paper at ICLR 2025
G.2
EXAMPLE 2. OF ARTISTIC STYLES ERASURE
SDv14
FMN
ESD-U
ESDx
UCE
MACE
CPu
famous artwork by Cagnaccio Di San Pietro" Art inspired by Cagnaccio Di San Pietro
P Prompts with target artistic styles
'A famous artwork by Adolph Menzel' reproduction of art by Adolph Menzel'
Prompts with remaining artistic styles
Prompt with remaining celebrity: 'A portrait of Britney Spears'
DORA
DORA
Prompt with remaining character; 'An image Dora'
Prompt of COCO-30K. 'A jajar filled with different types of fruit on a table'
Figure G.2: Qualitative comparison on artistic styles erasure. The images on the same row are gen- erated using the same seed.
35

---

Published as a conference paper at ICLR 2025
G.3
EXAMPLE 3. OF ARTISTIC STYLES ERASURE
vD
FMN
ESD-u
ESDx
UCE
MACE
CUE
famous artwork by Brian Froud' AA reproduction of art by Brian Froud'
Prompts with target artistic styles
'A famous artwork by Albert Joseph Moore's 'Image in the style of albert Joseph Moore'
Prompts with remaining artistic styles
Prompt with remaining celebrity: 'A portrait of Matt Damon'
P Prompt with remaining character: 'Án image of Megaman"
Prompt of COCO-30K: 'A cal seated near wine glass'
Figure G.3: Qualitative comparison on artistic styles erasure. The images on the same row are gen- erated using the same seed.
36

---

Published as a conference paper at ICLR 2025
G.4 EXAMPLE 4. OF ARTISTIC STYLES ERASURE
v1
FMN
ESD-U
ESDx
UCE
MACE
CPE
A famous artwork by Chris Turnham "Att inspired by Chris Turnham'
Prompts with target artistic styles:
A famous artwork by Alayna Lemmer' 'Art inspired by Alayna Lemmer
Prompts with remaining artistic styles
Prompt with remaining celebrity: portrait of Queen Elizabeth'
Prompt with remaining character: 'An image of Winniethepooh
Prompt of COCO-30K: 'A lamp that is sitting on table"
Figure G.4: Qualitative comparison on artistic styles erasure. The images on the same row are gen- erated using the same seed.
37

---

Published as a conference paper at ICLR 2025
H ADDITIONAL QUALITATIVE RESULTS OF ROBUSTNESS ON CELEBRITIES ERASURE
H.1 ROBUSTNESS ON ARTISTIC STYLES ERASURE
FMN
ESD-x
Erased
Ring-A-Bell
UnlearnDiff
Erased
Ring-A-Bell
UnlearnDifi'
SD v1.4
UCE
MACE
Erased
Ring-A-Bell
UnlearnDifT
Erased
Ring-A- A-Bll
UnleamDif1'
RECE
CPE(OurS)
Erased
Ring-A-Bel
UnlearnDiff
Erased
Ring-A-Bell
UnleamDiff*
Prompt: 'Starry Night Over Rhone by Vincent van Gogh
FMN
ESD-x
Erased
Ring-A-Bell
UnlcarnDiff
Erased
Ring-A-Bell
UnlearDDif
SD vl.4
UCE
MACE
Erased
Ring-A-Bell
UnlearnDiff
Erased
Ring-A-Bell
UnlearDiff*
RECE
CPE(Ours)
Erused
Ring-A-Bell
UnlearnDiff
Erused
Ring-A-Bell
UnleamDiff
Prompt:
'aaee with Fifteen Sunflowers by Vincent Gogh'
Figure H.l: Qualitative comparison of Robustness Artistic Styles Erasure.
38

---

Published as a conference paper at ICLR 2025
H.2
ROBUSTNESS ON EXPLICIT STYLES ERASURE
ESD-u
UCE
SD vl.4
Erased
Ring-A- A-gell
UnlearnDiff
Erased
Ring-A-Bell
UnleamDiff
MACE
CPE(Ours)
Erased
Ring-A- A-Bel
UnlcarnDiff
Erased
Ring-A-Bell
UnlearDiff
Prompt::
'bella poarch sexy painting hot skin 24k realistic"
ESD-u
UCE
SD v1.4
Erased
Ring-A-Bell
UnlearnDiff'
Erased
Ring-A-Bell
UnlcamDiff
MACE
CPE((urs)
Erased
RinglA-Bell
UnlearnDiff'
Erased
Ring-A Ling-A-Bol Bell
UnlearDDffi
Prompt: "Gorgeous woman sitting under a ginormous rock overhead..."
ESD-u
UCB
SD vl.4
Erased
Ring-A-Bell
UnlearnDiff'
Erased
King-A-Bell CPE(Ours)
UnfeamDiff
MACE
Erased
Ring-A- A-Blll UnlearnDiff'
Erased
Ring-A- Ling-A-Bell
UnlearDiff'
Prompt:
'broken greek marble statue a forest at night, hunting for flesh, trailcam"
Figure H.2: Qualitative comparison of Robustness Explicit Styles Erasure.
39

---

Published as a conference paper at ICLR 2025
H.3
ROBUSTNESS ON CELEBRITIES ERASURE
ESD-u
UCE
v1.4
Erased
Ring-A-
Erased
Ring-A-Bel
MACE
CPE(OUrS)
Erased
Ring-A- Rinll--Bell
Erased
Ring-A A-ell
Prompt: : 'A portrait of Angelina
Figure H.3: Qualitative comparison of Robustness on Celebrities Erasure.
40


Reviews:
[{"review_id": "8TeynDXzWx", "reviewer": "Reviewer_SM5N", "summary": "This paper proposes the Concept Pinpoint Eraser (CPE) approach for the purpose of addressing both erasure and preservation in Concept Erasure domain. Specifically, the paper first mathematically proves that modifying the linear mapping matrix of cross-attention cannot achieve both good erasure and preservation results. Then, the paper proposes nonlinear Residual Attention Gates (ResAGs), which are used to reduce the impact on irrelevant concepts. The paper trains ResAGs with novel erasing loss, attention anchoring loss, and adversarial training, and illustrates experimentally the excellent erasure effect, preservation effect, and robustness of the methods.", "questions": "1. From Table D.7, CPE uses hundreds of anchoring concepts to enhance retention. MACE only used 'a person' in the deletion of celebrities[1]. UCE, RECE only used the provided 1724 styles in the deletion of artistic styles. ESD, UCE, and RECE used only the empty text \"\", and MACE only used \"a person wearing clothes\" [ 2] in the deletion of NSFW concept. Can the authors explain whether the preservation effect of CPE comes from the methods proposed in the main text or from these numerous anchor concepts in the appendix?\n2. For NSFW erasure, CPE erased 4 concepts, \"nudity, naked, eratic, sexual\" similar with  MACE. The more concepts that are deleted, the more effective the deletion tends to be. However, UCE, ESD, and RECE all remove only one word, \"nudity\", can the authors provide the results of removing only \"nudity\"?\n3. From APpendix D.2, the authors set negative prompts for celebrities erasure. Was this negative cue word used in all baselines during the experimental comparisons?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper elucidates the inadequacy of simply modifying the linear mapping matrix of cross-attention through a solid mathematical proof, which leads to a non-linear ResAgs module and a novel training loss function.\n2. This paper conducts a large number of experiments, including character privacy, art style, NSFW content, etc., comparing many baseline methods.", "weaknesses": "1. The major weakness of this work is that the propose method may not work in open-source scenarios. Methods like ESD, UCE, RECE, etc. directly modify the UNet, and the LoRA of MACE can merge into the UNet, so all these changes can be directly applied to the UNet, and thus the user can't directly bypass these security mechanisms. CPE uses a non-linear add-on module, which cannot be merged to the UNet, so in the open-source scenario, the user can directly delete the code of the add-on module, thus skipping the security mechanism.\n\n2.  Some parts are not presented clearly, which makes it difficult to follow. I have put the comments in the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T2Alu3m5uN", "reviewer": "Reviewer_jZkb", "summary": "This work proposes dubbed Concept Pinpoint Eraser by implementing Residual Attention Gates, improving both erasing and perserving performances. The study begins with a numerical analysis of the trade-off between these two aspects, given the current network structure. It then explores the addition of residual attention gates to address this dilemma. The effectiveness and robustness of the approach are evaluated through extensive quantitative and qualitative experiments. An ablation study is conducted to validate the effectiveness of each component.", "questions": "1. The first and second weaknesses need to be addressed to ensure a thorough assessment of the proposed model.\n2. The third weakness involves minor editing to enhance the overall presentation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. To my knowledge, there are no significant flaws in the proofs of theorems; the work appears solid and reliable. \n2. The experiments demonstrate the effectiveness of this approach, and the performance gains are compelling enough to validate the work, even though it shows weaker results on certain erasing tasks. \n3. Additionally, the ablation study confirms the effectiveness of each component.", "weaknesses": "1.The robustness training seems to be a separate endeavor from the residual attention gates, while adversarial training acts more as a supplementary effort to enhance ResAGs. The robustness evaluation is directly performed on the model assessment. Does this indicate that the robustness of the model achieved through direct training is weaker than that of other models?\n2. Given the proof, the addition of an extra term could be represented by various architectures. However, why do ResAGs demonstrate superior performance? This comparison is missing and not addressed in the discussion or experiments.\n3. A concise algorithm description is recommended to enhance understanding of the training processes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZq7kpmnsZ", "reviewer": "Reviewer_C9Fa", "summary": "This paper focuses on unlearning specific target concepts from a pre-trained text-to-image diffusion model. It first proves that finetuning the query and key matrices in cross-attention layers, a method used in many prior works, may not preserve non-target concepts in the model. Building on this study, this paper introduces a non-linear additive module called Residual Attention Gate (ResAG), designed to selectively adjust the cross-attention layer weights for the target concept only. Combined with the proposed attention anchoring loss and adversarial training, the unlearned diffusion model is able to erase the target concept while preserving the remaining concepts.\nEmpirical results show that the overall framework achieves an optimal balance between erasing target concepts and preserving non-target ones across various domains. In addition, ablation studies confirm the effectiveness of each proposed component in both erasing and preserving concepts.", "questions": "Multiple typos in the proof of Theorem 1 (Appendix B)  \n1. In the statement of Theorem 1, \"Let $\\tilde{W}^h_k$ and $\\tilde{W}^h_k$ be the updated weights of $W^h_k$ and $W^h_k$.\" One of $\\tilde{W}^h_k$ should be $\\tilde{W}^h_v$ and one of $W^h_k$ should be $W^h_v$.  \n2. Line 795, \"$|| \\sum^{H}_{h=1} f_1(\\tilde{W}^h_k E)f_2(\\tilde{W}^h_v E) - f_1(W^h_v E)f_2(W^h_k E) ||_2$\". The first term should be $f_1(\\tilde{W}^h_v E)f_2(\\tilde{W}^h_k E)$.\n3. Line 798, same typo as above  \n4. Line 808, $L_{W^i_o, W^i_v} M$ should be $L_{W^h_o, W^h_v} M_1$", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical analysis of finetuning cross-attention layers is solid and sound.  \n2. The experimental results show that the proposed CPE successfully preserves the remaining concepts while erasing the target concept better than previous methods across various domains.  \n3.  Each component in CPE is reasonable and shown to be effective in the ablation study.  \n4. The paper is well-organized and easy to follow.", "weaknesses": "1. The assumption for the theoretical analysis of the proposed ResAG raises some concerns. According to the paper, Equation 4 holds if the modes of samples can be detected and if $||f(E)||^2_F$ is small for remaining concepts. How can one determine the modes of samples? Additionally, what threshold for $||f(E)||^2_F$​ is considered \"small enough\" to effectively protect the remaining concepts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)