Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them
Authors: Anh Tuan Bui
Thuy-Trang Vu
Long Tung Vuong
Trung Le
Paul Montague
Tamas Abraham
Junae Kim
Dinh Phung
Venue: ICLR 2025 Poster (2025)
Keywords: Trustworthy Generative AI
Diffusion Models
Machine Unlearning
Erasing Concepts
Primary area: generative models

Abstract: Concept erasure has emerged as a promising technique for mitigating the risk of harmful content generation in diffusion models by selectively unlearning undesirable concepts. The common principle of previous works to remove a specific concept is to map it to a fixed generic concept, such as a neutral concept or just an empty text prompt. In this paper, we demonstrate that this fixed-target strategy is suboptimal, as it fails to account for the impact of erasing one concept on the others. To address this limitation, we model the concept space as a graph and empirically analyze the effects of erasing one concept on the remaining concepts. Our analysis uncovers intriguing geometric properties of the concept space, where the influence of erasing a concept is confined to a local region. Building on this insight, we propose the Adaptive Guided Erasure (AGE) method, which dynamically selects optimal target concepts tailored to each undesirable concept, minimizing unintended side effects. Experimental results show that AGE significantly outperforms state-of-the-art erasure methods on preserving unrelated concepts while maintaining effective erasure performance. Our code is published at {https://github.com/tuananhbui89/Adaptive-Guided-Erasure}.

Published as a conference paper at ICLR 2025
FANTASTIC TARGETS FOR CONCEPT ERASURE IN DIFFUSION MODELS AND WHERE TO FIND THEM
Anh Bui', Thuy-Trang Vul, Long Vuong", Trung Le", Paul Montague", Tamas Abraham² Junae Kim?, and Dinh Phung!
Monash University
Defence Science and Technology Group, Australia
ABSTRACT
Concept erasure has emerged promising technique for mitigating the risk of harmful content generation in diffusion models by selectively unlearning unde- sirable concepts. The common principle of previous works to remove specific concept is to map it to fixed generic concept, such as a neutral concept or just an empty text prompt. In this paper, we demonstrate that this fixed-target strat- egy is suboptimal, as it fails to account for the impact of erasing one concept on the others. To address this limitation, we model the concept space as graph and empirically analyze the effects of erasing one concept on the remaining con- cepts. Our analysis uncovers intriguing geometric properties the concept space, where the influence of erasing concept is confined to local region. Building on this insight, we propose the Adaptive Guided Erasure (AGE) method, which dynamically selects optimal target concepts tailored each undesirable concept, minimizing unintended side effects. Experimental results show that AGE signifi- cantly outperforms state-of-the-art erasure methods on preserving unrelated con- cepts while maintaining effective erasure performance. Our code is published at https://gttuu..com//eee http://w...ouuunaaau//////dpppoo -Gunpdiiii--iidd-----e -rasuuee
INTRODUCTION
The widespread accessibility of text-to-image generation models has introduced significant risks such as the generation harmful content, copyright infringements, and biases due to the exposure of models undesirable concepts during training. Initial attempts to address these concerns focus on dataset filtering during the training phase (StabilityAl 2022). post- post-generating filtering (Rando) eet al. 2022) or inference guiding (Schramowski et al.) 2023b). While dataset filtering often requires costly retraining, post-processing and inference-based methods can be easily bypassed (Yang et al. 2024 Recently, concept erasure (Bui et al.] 2024b a Gandikota et al.) 2023 Orgad et 2023 Zhang et (023 Kumari et al., (023), which aims to directly remove the concept from the model's parameters without the need for complete retraining, has emerged as a promising alternative.
output-based and attention-based. Output-based methods aim to neutralize the output associated with undesirable concepts (Bui et al. 2024b a Gandikota et al. 2023 Wu et al. 2024), while attention sheettoon-based meth- ods modify the attention scores of these concepts
Concept erasure methods can be broadly categorized into two approaches:
cross attention layers of the model (Zhang et al. 2023 Orgad et al. 22033 Gandikota et al.) 2024 Lu et al. 2024 Lyu al. [024).. Despite their fixed, generic target, such "a photo" or empty text prompt. Although these methods have shown suc- cess in crasing undesirable concepts, they do not consider how the choice target concepts affects both the effectiveness of erasure and the preservation of benign concepts.
differences, both approaches share a common principle: mapping undesirable concepts
To address this limitation, we first model the concept space as a graph, where each node represents concept, and the edge weights denote the impact of erasing one concept another. We perform an empirical analysis using a newly curated evaluation dataset, NetFive, to understand the structure of the concept space. Our findings suggest that the concept impact space has geometric structure,

---

Published as a conference paper at ICLR 2025
characterized by a key property: locality the impact of erasing one concept is localized in the concept space, i..., affects strongly only those concepts that are close to the erased one.
Furthermore, explore different types of target Soncepts-Synonny concepts, and semantically distant concepts- ences both erasure performance and the preservation of benign concepts. At the end, identify the ideal target concept as the most affected by the change of the model parameters when the corre- sponding concept is erased, but must not be its synonym.
semantically related local find that the choice of target significantly influ-
Building on these insights, we propose the Adaptive Guided Erasure (AGE) method, which automat- ically selects the optimal target concept for each erasure query by solving a minimax optimization problem. To further improve the richness of the target concept, we model it as a learned mixture of multiple single concepts, allowing us to search for the optimal target in continuous rather than discrete space. We evaluate the proposed AGE method on various erasure tasks, including object removal, Not-Safe-For Work (NSFW) attribute erasure, and artistic style removal. Experimental re- sults show that AGE significantly outperforms state-of-the-art concept erasure methods, achieving near-perfect preservation of benign concepts while effectively erasing undesirable ones.
Our main contributions can be summarized as follows. 1 We present novel empirical evaluation of the structure and geometric properties of the concept space, which provides new insights into concept erasure, such as the locality of the impact of erasing one concept another. @ We analyze the impact of target concept selection on both erasure effectiveness and the preservation of benign concepts, identifying
key properties of desirable target concepts, i.e, closely related but not synonyms to the to-be-erased concept. Motivated by the analysis, we propose a novel adaptive method for selecting the optimal target concept for each erasure query satisfying the two key prop- erties. 4 Finally, we conduct extensive experiments demonstrating the effectiveness of our method and its ability to preserve benign concepts while erasing undesirable ones.
2 BACKGROUND
Latent Diffusion Models. Diffusion models, recent class of generative models, have shown impressive results generating high-resolution images (Ho et al., 2020, Rombach et al.] [022] Ramesh et al. 2021: 2022). In nutshell, training diffusion model involves two processes:
forward diffusion process where noise is gradually added to the input image xo ~ Pdata. and a reverse denoising diffusion process where the model tries to predict noise c, which is added in the forward process. The model is trained by minimizing the difference between the true noise c and the predicted noise (222,, t) diffusion step t parameterized by the denoising model f. With an intuition that semantic information that controls the main concept of an image can be represented in low-dimensional space, |Robbach al. (2022) proposed a diffusion process operating on the latent space of pre-trained encoder & which compresses the input data He into oow-dimeniional latent representation 20 Exx). The objective function of the latent diffusion model as follows:
L = E (er ||c co 1)
Concept Erasing, Given a textual description in a set of undesirable concepts c. E E, the con- cept erasing problem aims to remove this concept from a pretrained text-to-image diffusion model 7()), t), typically via finetuning to obtain a benign output Eg/ (2t, T(c), t) from sanitized model Ce' parameterized by 0'_ For the remainder of this paper, because all outputs are from the same time step & and latent vector 24, we omit the time step and latent vector 24 for the sake of simplicity, i... (c(r(c)) and car (T(c)). While proposed in different forms, previous works (Gandikota al.
|023] Orgad et al. 2023: Gandikota et al.) 2024) share a common principle to map a to-be-erased concept Ce to a target neutral concept oz. The target concept o can be either a generic concept, such as "a photo" or null concept, such as an empty text prompt. It is typically predefined and fixed for all undesirable concepts c. E E. To preserve the model's 's performance on other concepts, an additional term T.2 is added to ensure that the output of the neutral concept remains unchanged. More specifically, the erasing objective can be formulated as follows:
min ER,ER [Ilor (F(ce) en ca(T(c))) + (T()) -c((:)))))((((())) co(T(ce)) 8'
(2)
2

---

Published as a conference paper at ICLR 2025
While this naive principle is simple and effective in erasing the specific concept, choosing fixed neutral concept as the target concept for all concepts to be erased is obviously not optimal (i.e., G Cn Vc, E). Intuitively. remapping the visual concept "English Springer" to "Dog" will likely cause a less drastic change on the model's parameter compared to remapping it to a neutral concept such "A photo"
leading a better preservation performance on other concepts. In the following section, we will provide series of evidence to support this intuition which leads us to a principled approach on choosing an optimal target concept for each concept be erased.
QUANTIFYING IMPACT OF ERASING CONCEPTS
Netfive Dataset. Two main challenges in evaluating erasing method are: (1) How verify whether a concept is present in the generated images or not? (2) How to ensure the evaluation is diverse enough cover the output space of the model which is of infinite possibilities? To tackle these two challenges, we propose an evaluation dataset called NetFive, which consists of 25 concepts from the ImageNet dataset, for which we can leverage the pre-trained classification model verify the presence of the concepts in the generated images. We also ensure the diversity of the evaluation by generating 500 samples for each concept. More specifically, we choose a total of five subsets of concepts, each subset contains one anchor concept, e.g.. "English Springer" and four related concepts, ranked by their closeness to the anchor concept as follows: (the details and rationale of choosing the concepts are provided in Appendix C.T
Dog: English springer, Clumber spaniel, English setter, Blenheim spaniel, Border collie.
Vehicle: Garbage truck, Moving van, Fire engine, Ambulance, School bus.
Instrument: French horn, Basson, Trombone, Oboe, Saxophone.
Building: Church, Monastery, Bell cote, Dome, Library.
Equipment: Cassette Player, Polaroid camera, Loudspeaker, Typewriter keyboard, Projector.
Metric to measure the generation capability of the model. Because of intentionally choosing all concepts from the ImageNet dataset, we can leverage the pre-trained classification model detect the presence of the concept in the generated image, More specifically, given a model Gg. we generate k 500 images with the input description c, and then measure using the metrics:
Detection Score (DS-1/DS-5): # of samples that classified the concept in top-1 top-5 predictions/ / & (i.e., top-k accuracy).
Confident Score (CS-I/CS-5): Average confident score w.r.t. the concept c in top-1 or top-5 predictions, otherwise, the confident score is set to be O. This metric indicates how good the model Gø is when generating the concept c. A higher score means that the concept c more likely appeared in the generated images.
For all metrics, higher values indicate better generation capability of the model concept In the rest of the paper, we use the Detection Score (DS-5) as the main metric, while results for other metrics provided in Appendix C.2
We present Go(c;) and Ga, (cz) ( as the generation capability of the original model and the sanitized model on the same query concept ey after erasing concept Ce, respectively. With this, can mea- sure the impact of crasing concept on the generation capability of concept ej by computing the difference between Go(c;) and Ge (cz), i.e., A(ce; Gj) Golcy) G G.,(C)).
3.1 TARGETING TO A GENERIC CONCEPT
We first analyze the impact of choosing generic concept such as an empty " as the target concept for crasure. The top subfigure in Figure shows a sample analysis of the impact of erasing concept "English Springer" to the generation capability of all NetFive concepts. Each column corresponds to one concept e, in the NetFive dataset. The blue bar represents the generation capability G.. (c₂) of the sanitized model Go(c,) of the original model on concept ej. The red bar, therefore, represents the gap of generation capability between the sanitized model and the original model, i.e., A(ce) Go(c;) Ga, (c))
concept cj. while the total height of the stack is the generation capability
3

---

Published as a conference paper at ICLR 2025
EEngish Springar"
H67
Netric ASS
Seewr
Botte Cole
Lartug
French
Toyle Dvrk
Spriet
grver cader
endive connes Spewiter
Figure I: Analysis of the impact of choosing empty concept as the target concept for erasure. Com- plete details are provided in Figure 9
The higher the total height of the stack, the higher the generation capability of the original model on concept cj. while the higher the red bar, the higher the difference between the generation capability of the sanitized model and the original model. We also provide the confidence score of the classifier when predicting the concept from the generated images from the original model (the green line) and the sanitized model (the orange line).
Sample Analysis. It can be seen from the top subfigure in Figure I that the concept ca "English Springer" has been successfully removed from the sanitized model as evidenced by Ge, ("English Springer") = 0. can also be seen that all closely related concepts to concept ca "English Springer" are affected by the erasure of this concept, evidenced by the significant drop in the generation capability G.. (cj) < Go(c,), as well as the corresponding confidence scores. In contrast, the other concepts that are not closely related to the concept Ce less affected, evi- denced by the unchanged generation capability Ge. (c;) Rs Go(cz). Interestingly, the two abnormal concepts that not related to the ro-be-erased concept ce but are also affected, "Bell Cote" and "Oboe". These two concepts have low generation capability even before the erasure of concept Ce. i.e., Go(cz) 60%, compared to other concepts which have Go(c) Rs 100%.
Systematic Analysis. Figure shows the impact of erasing one concept all the other concepts in the NetFive dataset, where each row corresponds erasing concept, and the first row cor- responds to erasing the concept "English Springer" as shown in the top subfigure. Each cell in the matrix represents the drop of the generation capability of concept cj after erasing concept ce, i.e., ( Cj) = Gu(c,) Ge, (cy). ). A deeper red color indicates a larger drop of the generation capabil- ity of concept e, after erasing concept c.. The matrix can be interpreted concept graph, showing the impact of one concept on other concepts.
There several intriguing observations that be made from Figure
4

---

Published as a conference paper at ICLR 2025
Metric: & DS
Spriger- Spriager
English Spriger dumbbrr Spon
Engish
Erplin Spriger Garaage Trucen
100
English French
Exgish Sprige
Cartagg Trure Moving
20
GathageToo Tusk Trxc
Darsage french
Garbage Truck Dhuck
Geraace Trick
Fronch-om
french flom Sesscos
-20
Frenct Hom Hudial indtureett Hor
french Norm Cossette tyor Trench
Engiih peo
English Setter
goold cons
Certer Wir The School Hort
Bick
Libvary Couserte Player poseres Carone Livil Sporonte Koydired Projector
Figure 2:
Analysis the impact of choosing a specific concept as the target concept for erasure. Full details are provided in Figure [2
Locality:
The concept graph is sparse and localized, which means that the impact of eras- ing one concept does not strongly spread to all the other concepts but only local concepts that are semantically related the erased concept Ce.
The concept graph is asymmetric such that the impact of erasing concept ce on concept Cg not the same as the impact erasing concept cj on concept Ge-
Asymmetry:
Abnormal:
The two abnormal concepts "Bell Cote" and "Oboe", which have low genera- tion capability to begin with, are sensitive to the erasure of any concept.
In the case erasing exclusive concepts, such as "Taylor Swift". "Van Gogh", "gun", and "nudity" as shown in the last four rows of the figure, the impact of erasing these concepts to all NetFive con- cepts are also limited, except for the two abnormal concepts, which supports the above observations on the concept graph.
Results with Stable Diffusion version 2.1 Figure 13 shows the impact of choosing the empty concept as the target concept for erasure with Stable Diffusion v2.1 It can be seen that our earlier observations are still valid, except that the abnormal concepts which have low generation capability and are sensitive to the erasure of any concept are now "Bell Cote" and "Projector". This consistency indicates the generalization of the observations on the concept graph of different models, trained on different datasets and settings.
3.2 TARGETING TO SPECIFIC CONCEPT
We now compare different strategies of choosing the target concept for erasure to see how the choice of the target concept affects the preservation of other concepts. each subset, we choose erase same concept ce but with seven different target concepts c₁ in the following order: (1*t) a synonym of c.. (2nd)3rd) two semantically related concepts but not synonyms, (4*h) a general. upper-level concept that covers the subset, (e.g. "Dog" for the concept "English Springer") (5th 6th) two se- mantically unrelated concepts, and (7th) an empty concept. The explanation of finding a synonym to an anchor concept is provided in Appendix C.3
can be seen from Figure 2 that the preservation performance highly depends on the choice of target concept. More specifically, there are several notable observations:
Locality: Regardless of the choice of the target concept, the impact of crasing one concept is still sparse and localized.
5

---

Published as a conference paper at ICLR 2025
The two abnormal concepts "Bell Cote" and "Oboe" are still sensitive to the erasure any concept regardless of the choice the target concept.
Abnormal:
Synonym XXX: Mapping synonym of the anchor concept leads a minimal change as evidenced by the lowest (ce. Cÿ) for all cj. However, it also the least effective in erasing the undesirable concept Ce.
All Unrelated Concepts XX: Mapping to semantically unrelated concepts demonstrates the similar performance as the generic concept " as evidenced by the similar A(ce; Gy) between the three last rows (5th 7th) in each subset.
General concepts X: While choosing general concept as the target concept is intuitive and reasonable, it does not necessary lead to good preservation performance. For example, "English Springer" + "Dog" or "French Horn" "Musical Instrument Horn" still cause drop on related concepts in their respective subsets. Moreover, there are also small drops on erasing performance compared to other strategies, shown by DS-5 of 83%, 91%, and 78% when mapping "Garbage Truck" to "Truck", "French Horn" to "Musical Instrument Horn", and "Cassette Player" to "Audio Device", respectively. The two observations indicate that choosing general concept is not an optimal strategy.
In-class /: The highest preservation performance which is consistently observed in all five subsets is achieved when the target concept a closely related concept to ce. For example, "English Springer" + "Clumber spaniel" or "Garbage Truck" - "Moving Van" or "School Bus" the "Dog" and "Vehicle" subsets.
PROPOSED METHOD: ADAPTIVE GUIDED ERASURE
Motivated by the observations in Section propose to select the target concept for erasure adaptively for each query concept to mitigate the side effects crasing undesirable concepts in diffusion models. More specifically, the ideal target concept should satisfy the following properties:
It should not be a synonym the query concept that resembles a similar visual appearance (c.g., "nudity" "naked" "nude", "Garbage Truck" to "Waste Collection Vehicle"). This ensures that the erasure performance on the query concept remains effective.
It should be closely related to the query concept but identical (e.g.. "English Springer" to "Clumber Spaniel", or "Garbage Truck" to "Moving Van"). This helps preserve the model's generation capabilities on other concepts. As suggested by the locality property of the concept graph, changes in the model's output can be used to identify these locally related concepts.
Although is possible to manually select the ideal target concept for each query concept based on the above properties, this approach is not scalable for a large erasing set E. Therefore, propose an optimization-bant based approach to automatically and adaptively find the optimal target concept for each query concept. Specifically, aim to solve the following optimization problem:
min E max [leor (T(c)) co(((e))) (r(()) + +Alle (T(a))
8' EE
where À is trade-off hyperparameter and C is the search space of target concepts Ct-
Minimizing the objective L1 w.r.t. 6' ensures that the output of the sanitized model for the query concept c, is close the output the original model but for the target concept c,, which serves the purpose of erasing the undesirable concept c.. Meanwhile, minimizing the objective L2 w.r.t. 0' ensures that the output of the two models remain similar for the same input concept e,, preserving the model's 's capability on the remaining concepts. In summary, the outer minimization problem opti- mizes the sanitized model parameters to simultaneously erase undesirable concepts and preserve the model's functionality for other concepts.
On the other hand, maximizing the objective L, w.r.t c ensures that the solution of is not a synonym of the query concept Ge, while maximizing the objective La2 w.r.t. a finds sensitive

---

Published as a conference paper at ICLR 2025
concept the change of the model's parameter 8 + 6'. This helps identify the most related concept to the query concept Ce, as suggested by the locality property the concept graph in Section 3 By maximizing both L1 and L2 w.r.t Cr, ensure that the solution of satisfies both key properties of the ideal target concept. We provide empirical evidence Section 5.2 and Appendix D.6 showing that the intermediate value of the solution ej from the optimization problem equation igns with the above analysis.
Optimization Details. Since the concept space C discrete and finite, the straightforward ap- proach is to enumerate all the concepts in C and select the that maximizes the total loss in each optimization step the outer minimization problem. However, this approach is computationally prohibitive for large C. Moreover, some concepts can be complex and may not be interpreted single concept in the concept space C. To address this, we formulate the target concept as a com- bination of multiple concepts the concept space C, i... T(G) G( e Tc. where G is the operator, RRC is learnable variable, and Te is the textual embedding matrix of the entire concept space C. We choose the Gumbel-Softmax operator with a temperature less than I to ensure that the target concept is combination few main concepts rather than mixture of all concepts in C. The optimization problem equation 3 can be rewritten as follows:
min CEEE
max
leo (T(c - (G Te)ll +1 (G(() @ Tc) - ee lc)
La
(4)
in which the objective of the inner-max transformed find the continuous weight instead of the discrete concept This trick allows us to increase the richness and expressiveness of the target concept as well as the optimization efficiency. We provide further details Appendix
5 EXPERIMENTS
In this section, we demonstrate the effectiveness of our method erasing various concepts from the foundation model, including object-related concepts, artistic styles, and NSFW attributes. We compare our approach with the state-of-the-art erasure methods including AP (Bui et al. 2024b), ESD (Gandikota et al. (022)), UCE (Gandikota al. 2024), CA (Kumari al. 2023). and MACE (Lu et al. (2024). Our experiments follow the same setup as in Bui et al. (2024b); Gandikota et al. 2023] 2024). Specifically, we use Stable Diffusion (SD) version 1.4 as the foundation model, and fine-tune the model for 1000 steps with batch size of 1, using the Adam optimizer with learning rate 10-5
Further implementation details and analyses are provided in the appendix, including qualitative re- sults (Section D an examination the impact of vocabularies (Section[D.) and hyperparameters (Section D.3 as well an analysis of the search for the optimal target concepts (Section (.6).. We strongly recommend referring to the appendix for a deeper understanding of our method and exper- iments.
5.1 ERASING OBIECT-RELATE CONCEPTS
Setting. In this experiment, we assess our method's ability to erase object-related concepts, such as "Dog" or "Cat". from a foundational model. We utilize the Imagenette dataset subset of Tma- geNet (Deng et al. 2009), which contains 10 easily recognizable classes as suggested in (Gandikotal ict al. (023). We conduct four different erasing tasks, each involving the simultancous crasure of five classes while preserving the other five, generating 500 images per class.
Metrics. Erasing performance is measured using the Erasing Success Rate (ESR-K), which cal- culates the percentage of generated images where the "to-be-crased" classes are not detected in the top-k predictions. Preserving performance is evaluated using the Preserving Success Rate (PSR-k)
hhaps///tthhb.ccm/aaaimmgggee

---

Published as a conference paper at ICLR 2025
to measure how well "to-be-preserved" classes are detected, along with FID and CLIP scores on the COCO 30K validation set to assess preservation of common concepts. For example, in Table [ the average PSR-5 of the original SD model is 97.6%, which means that 97.6% of the generated images contain the object-related concepts in the top-5 predictions, indicating the strong ability to generate object-related concepts accurately of the foundation model.
Results. In term of erasure, all baselines perform well, with the lowest ESR-1 and ESR-5 scores being 95.5% and 88.9% respectively. indicating that only small proportion of the generated images retain the object-related concepts. The UCE method achieves perfect 100% ESR-1 and ESR-5, while our method reaches 98.1 1% ESR-1 and 95.7% ESR-5, slightly below AP, CA, and MACE.
However, when comes to concept preservation, the baselines, particularly UCE, perform poorly, with the most recent SOTA baseline AP showing the best PSR-1 and PSR-5 scores at 55.2% and 79.9%, respectively. In contrast, our method significantly outperforms the baselines, achieving 73.6% PSR-1 and 95.6% PSR-5, far exceeding AP and approaching the original SD model's per- formance. Our method also achieves the best FID score 16.1 and CLIP score 26.0, surpassing the second-best AP method by 0.2 and 0.1 points, respectively. These results demonstrate that our method not only effectively crases unwanted concepts but also excels in preserving other important concepts, closely matching the performance of the foundation model.
Table 1: Erasing object-related concepts. The best erasing results are highlighted in bold, and the second-best erasing results are highlighted in underline.
Method
ESR-1T
ESR-5t
PSR-1f
PSR-5t
FID
CLIP
SD
22.0 11.6
2.4 1.4
78.0 11.6
97.6 +: 1.4
16.1
26.4
ESD
95.5 + 0.8
88.9 1.0
41.2 = 12.9
56.1 + 12.4
17.9
24.5
UCE
100 0.0
100 0.0
2.6
4.0
19.1
21.4
CA
0.3
96.8
9.77
66.5
16.6
25.8
MACE
99.3 19.30.3
97.6 + 1.2
47.4 12.0
72.8 + 10.5
16.9
24.9
AP
98.6 1.1
96.1 2.7
55.2
10.0
+
16.3
26.1
Ours
98.1 + 1.1
95.7 + 2.5
73.6 + 9.8
95.6 + 1.1
16.1
26.0
5.2 ERASING NSFW CONCEPTS
Setting. In this experiment, we focus unlearning Not-Safe-For- Forr Work (NSFW) attributes like "nudity" from the model's capability. We follow the same setting as in (Gandikota et al. [0023) focusing exclusively on fine-tuning the non-cross-attention modules. To generate NSFW images, we employ I2P prompts (Schramowski et al. 2023b) and generate a dataset comprising 4703 images with attributes encompassing sexual, violent. and racist content.
Metrics. We utilize the detector PPannet 2019) which can accurately detect several types of exposed body parts to recognize the pres-
Table 2: Evaluation on the nudity erasure setting.
ence of the nudity concept in the generated im- ages. The detector (Praneet 2019) provides multi-label predictions with associated confi- dence scores, allowing us adjust the thresh- old and control the trade-off between the num-
NER-0.31
NER-0.5,
NER-0.71
NER-088
FDD
13.84
9.27
4.74
1.68
20.76
UCE
6,87
3.42
0.68
0.21
15.98
ESD
5.32
2.36
0.74
0.23
17.
AP
3.64
1.70
0.40
0.06
15.52
Ours
5.06
1.53
0.32
0.04
14.20
ber of detected body parts and the confidence of the detection, i.e., the higher the threshold, the fewer the number of detected body parts. Erasing performance is measured using the Nudity Exposure Rate (NER-k), which measures the ratio of images with any exposed body parts detected by the detector (Pranect 2019) over the total number of generated images with confidence score greater than the threshold k. For example, in Table 2 with the threshold set at 0.5, the NER score for the CA model stands at 9.27%, indicating that 9.27% of the generated images contain signs of nudity concept from the detector's perspective. Preserving performance evaluated using FID score on COCO 30K validation set.
8

---

Published as a conference paper at ICLR 2025
HOT
Figure 4: Top/a: Intermediate results of the search process, with images generated from the most sensitive concepts ot found by our method and C. at the same optimization step. Bottom/b: Similarity between nudity attributes and keywords.
Results. Table shows the NER score across thresholds ranging from 0.3 to 0.8. With k 0.5, CA, ESD, UCE, and AP achieve 9.27%, 2.36%, 3.42%, and 1.70% NER, respectively, while our method achieves 1.53% NER, the lowest among the baselines, indicating the highest erasing performance. This result remains consistent across different thresholds, emphasizing the robustness of our method in erasing NSFW content. In term of preserving performance, our method achieves the best FID score of 14.20, a significant improvement over the second and third-best AP and UCE methods at 15.52 and 15.98, respectively, indicating that our method can simultaneously erase concept while preserving other concepts effectively.
Figure 3 provides detailed statistics of differ- ent exposed body parts the generated images. It can be seen that in the original SD model, among all the body parts, the female breast is the most detected body part in the generated images, accounting for more than 320 images out of the total 4703 images. Both baselines, ESD and UCE, as well as our method, achieve significant reduction in the number of detected body parts, with our method achieving the low- est number among the baselines. Our method also achieves the lowest number of detected body parts for the most sensitive body parts, only surpassing the baselines for feet which is a less sensitive body part.
Ampts
Mel sond
Contule
Pagis [stsove))
Figure 3: Number of exposed body parts counted in all generated images with threshold 0.5.
Searching for the optimal target concepts. We investigate the search for optimal target concepts e by visualizing the generated images from c found by our method and the corresponding output of at the same optimization step shown in Figure 4.. Additionally, we present the similarity scores between the nudity attributes detected by the detector and C. as in Figure 4p. Firstly, the to-be-erased concept e, ("nudity"") is closely related the sensitive body parts such "breasts" and "genitalia", explaining why removing the "nudity" concept also eliminates these sensitive attributes. On the other hand, as discussed Section 4 the target concept e; designed be locally aligned with but not exactly the same as C.. In our results, all intermediate concepts C₁ such as "Model", "Drawing"" and "Toy" are highly correlated with the "nudity" concept satisifying the first condition. However, while being highly correlated with less sensitive parts such as "Feet", they are less corre- lated with more sensitive ones like "breasts" and "genitalia", meeting the second condition. It is worth recall that in Equation 3 ez serves retained concept to preserve the model's capabilities. Therefore, the strong correlation between and "Feet", and its weaker connection with other sensi- tive parts, explains the interesting advantage of our method that it can still retain the "Feet" concept while successfully crasing others, as observed in Figure We provide more results and analyses in A Appendix D.6

---

Published as a conference paper at ICLR 2025
5.3 ERASING ARTISTIC CONCEPTS
Setting. In this experiment, evaluate our method's ability to erase artistic style concepts. We focus on five well-known artists with highly recognizable styles that are commonly mimicked by text-to-image generative models, including "Kelly Mckernan", "Thomas Kinkade", "Tyler Edlin", "Kilian Eng", and "Ajin Demi Human" as in (Gandikota et al.] 2023). The experiment involves five tasks, each aiming to erase one artist's style while preserving the others.
Metries. A major challenge in this setting is the lack of a reliable detector for identifying the presence of artistic styles in generated images. time-consumingg not scalable, and more importantly. easily biased. To overcome this, we utilize the CLIP score
is avoided due the high cost,
P to measure the alignment between the generated images and the textual prompts, which has been shown to be effective in similar tasks (Gandikota et al., 2023). To enhance the correctness, we make use of a list of long textual prompts that are designed exclusively for each artist (credits to (Gandikota et al. (023)), combined with 5 seeds per prompt to generate 200 images for each artist across all methods. This approach allows the use of the CLIP score as a more meaningful measurement to evaluate the erasing and preserving performance. We also use LPIPS (Zhang al., 2018) to measure the distortion in generated images by the original SD model and editing methods, where a low LPIPS score indicates less distortion between two sets of images. However, as LPIPS is designed for quantitative comparison between two outputs, it might not be as good as CLIP score in order to verify the presence of specific concept in the generated images, and should be used as a complementary metric to CLIP score.
Results. Figure 5 presents the results of the artistic style erasure task. In term of CLIP score, it can be seen that our method along with AP and UCE lie in the Pareto frontier (with the ideal point being the top-left corner), with clear trade-off between erasing and preserving performance. For example, AP achieves bet- ter erasing performance than method with a CLIP score of 21.57 and 22.44, respectively, but method outperforms AP in preserv- ing performance with CLIP score of 30.45 and 33.13, respectively. However, in term of LPIPS score, our method clearly achieves the best trade-off between erasing and preserving performance, with the erasing score of 0.80 and preserving score of 0.44, while AP achieves the erasing score of 0.78 and preserving score of 0.47.
C50
LCE
PACE
Oun
21.-
CUP
UCE
PS (To Reean)
Ours
245
dbll JAPS
MACE
281
CLP (To-Eras)) 1 LIPSS (To-Gruae)
Figure 5: CLIP and LPIPS scores of the artistic style erasure task. ()) LPIPS at x-axis is scaled by 34 for better visualization. Full results are in Table 6
CONCLUSION
In this paper, we introduced the Adaptive
Guided Erasure (AGE) method, a novel approach for concept crasure that addresses the limita- tions of existing fixed-target methods. By modeling the concept space graph and analyzing its geometric properties, we demonstrated that selecting a locally related target concept can mini- mize unintended side effects. AGE adapts target selection through minimax optimization, further enriched by representing targets as mixtures of single concepts. Our experiments show that AGE outperforms state-of-the-an methods, effectively erasing undesirable concepts while preserving be- nign ones across various tasks. Besides the method, we also provided the first comprehensive study of the concept space structure, providing new intriguing insights that shed light on the concept space geometry. We believe that these insights will inspire future research on understanding and manipu- lating the concept space for various applications.
hhtprr/iiiiiininilllicicccceeeen score.html
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The Commonwealth of Australia (represented by the Advanced Strategic Capabilities Accelera- tor) supports this research through a Defence Science Partnerships agreement. Dinh Phung fur- ther acknowledges the support from the Australian Research Council (ARC) Discovery Project DP250100262. The authors would like to thank the anonymous reviewers for their insightful feed- back and valuable suggestions, which have significantly enhanced the quality of this work.
REFERENCES
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04248. 2017.
Anh Bui, Trung Le, He Zhao, Paul Montague, Olivier de Tamas Abraham, and Dinh Phung. Improving adversarial robustness by enforcing local and global compactness. In European Con- ference Computer Vision, pp. 209-223. Springer, 2020.
Anh Bui, Trung Le, He Zhao, Paul Montague, Seyit Camtepe, and Dinh Phung. Understanding and achieving efficient robustness with adversarial supervised contrastive learning. arXiv preprint arXiv:2101.10022 2021.
Anh Bui, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Hiding and recovering knowledge in text-to-image diffusion models via learnable prompts. arXiv preprint arXiv:2403. 12326, 2024a.
Anh Bui, Long Vuong, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Erasing undesirable concepts in diffusion models with adversarial preservation. NeurIPS, 2024b.
Tuan Anh Bui, Trung Le, Quan Tran, He Zhao, and Dinh Phung. A unified wasserstein distributional robustness framework for adversarial training. arXiv preprint arXiv:2202. 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 2009. doi: 10. 109/C//R.2009 2009556800...
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0166 01618, 2022.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5111-5120, 2024.
Rohit Gandikota et al. Erasing concepts from diffusion models. ICCV, 2023.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint orXiv:1412.653 2014.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, (66680066811,, 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 20:::2/060000 006.0.06...... 2021.
Nupur Kumari, Bingliang Zhang, Sheng-Yu Yu Wang. Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22691-22702. 2023.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept crasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -440- 2024.

---

Published as a conference paper at ICLR 2025
Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. One-dimensione adapter to rule them all: Concepts diffusion models and erasing applications. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7559-7568, 2024.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In IEEE International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 7030-7038. IEEE, 2023. doi: 10. 10.100//CCV5110022223..6666
Minh Pham, Kelly O Marshall, and Chinmay Hegde. Circumventing concept erasure methods for text-to-image generative models. arXiv preprint arXiv:2308.0 2023.
Bedapudi Praneet. Nudenet: Neural nets for nudity classification, detection and selective censorin. 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204006125, 1(2):, 2022.
Javier Rando et al. Red-teaming the stable diffusion safety filter. NeurIPS Workshop MLSW, 2022.
Dana Rao.
Responsible innovation in the age of generative ai, 2023.
URL
htttp::/bloq.aaeeeo http://ee.oo//e/ooblsss////////0
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. pp. 12522-225533 April 2023a.
Patrick Schramowski et al. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In CVPR, 2023b.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URLhttps: /tpplttts///peeeevveeww wwwpreveneeeeeeeeeeppppeeeee..em .ew wttt:/frummii--ie
4VMSMcY
Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Improving image captioning with better use of caption. pp. 7454-7464, Online, July 2020. Association for Computational Linguis- tics. doi: 10.18653/v 1t00://022....mmmm..6..... URL
acl-main.66 664
SmithMano. Tutorial: How to remove the safety filter in 5 seconds, 2022.
StabilityAI. Stable diffusion 2.0 release. 2022. URL http:://stabllty.a///1l
Jing Wu, Trung Le, Munawar Hayat, and Mehrtash Harandi. Erasediff: Erasing data influence in diffusion models. arXiv preprint arXiv:2401. 055799, 2024.
Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Jailbreaking text-to-image generative models. In 2024 IEEE Symposium Security and Privacy (SP), pp- 123-123. IEEE Computer Society, 2024.
Eric Zhang et al. Forget-me-not: Learning to forget in eotttto-image diffusion models. arXiv preprint arXiv:2303. 17591, 2023.
12

---

Published as a conference paper at ICLR 2025
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
13

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Related Work
14
B Further Details on the Adaptive Guided Erasure Method
16
Experimental Settings
NetFive Datasel
17
Metric to measure the generation capability of the model
18
Finding synonym to an anchor concept
19
Further Analysis
20
D.1
Measuring the impact of erasing the generation capability of other concepts
20
D22
Impact of choosing vocabularies
22
D.3
Hyperparameter analysis)
30
D.4
Evaluation on Scalability
30
D.5
Mixture concepts)
31
D.6
Searching for the optimal target concept
33
D.7
Impact of crasing synonyms
34
D.8
Qualitative results
34
A RELATED WORK
Existing methods for removing or unlearning undesirable concepts from text-to-image generative models can be categorized into four main categories: (1) pre-processing dataset filtering, (2) post- processing screening, (3) model fine-tuning, and (4) in-generation guidance. Each approach has its advantages and drawbacks. While pre- pre-processing and post-processing methods are effective, they may not address the underlying tendencies of models to generate inappropriate content (SmithMano) 2022 Fine-tuning approaches provide more direct solution but often struggle with maintaining model performance while removing specific concepts. In-generation guidance offers flexibility but can be resource-intensiv and requires careful tuning.
Pre-processing dataset filtering is the most straightforward method, involving mechanisms to de- tect and remove or flag undesirable content the training data. This generally requires a pre-trained detector to identify objectionable content within both the original dataset and any newly added data. However, the primary drawback of this approach is that it necessitates retraining the model from scratch, which is computationally expensive and impractical for dynamic erasure requests. Exam- ples of this include Stable Diffusson v2.0 (StabilityAl (022). which uses an NSFW detector to filter the LAION-5B dataset (Schuhmann al. 2022). and Adobe Firefly, which trained on licensed and public-domain content to ensure commercial safety (Rao 2023). DALL-E (Shi et al., 2020)) further refines this by subdividing the NSFW concept into specific categories and applying individ- ualized detectors. Despite significant efforts in curating training data, pre- pre-processing methods often leave models inadequately sanitized, as shown in Gandikota al. 2023).
Post-processing screening methods identify inappropriate content in generated images after in- ference. Images flagged by detectors are either blurred or blocked before being shown to users. This approach is employed by organizations such as OpenAl (DALL-E), Stability (Stable Diffusion), and Midjourney Inc. DALL-E 3, for instance, enhances this approach by training standalone de- and gender. While post-processine processing is highly effective,
tectors for various concepts such can be vulnerable to adversarial attacks. As demonstrated in Yang et al. (2024), adversaries can use techniques like Boundary Attack (Brendel et al. 2017) to circumvent such filters.
14

---

Published as a conference paper at ICLR 2025
In-generation guidance methods intervene directly during the image generation process, often using techniques such as handcrafted textual blacklisting (Shi et al. 2020 or employing large language models for prompt engineering and safety classification. Safe Latent Diffusion (SLD) (Schramowski et al.] 2023a) takes an alternative approach by leveraging inappropriate knowledge encoded in pre-trained models for reverse guidance during generation.
Model parameter fine-tuning offers more scalable solution by fine-tuning pre- pre-trained founda- tion models to remove unwanted concepts without retraining from scratch. This approach modifies the model's parameters, making it unnecessary to detectors in post-pracessing. Consequently, releasing a sanitized model is as simple as providing a new checkpoint, which is less vulnerable to adversarial evasion. Fine-tumi Fine-tuning methods have received considerable attention, with notable works including TIME (Orgad et al. 2023). ESD (Gandikota et al.] (023), Concept Ablation (Kumari al. 2023), UCE (Gandikota al. 2024), Forget-Me-Not (Zhang et al. 2023). and MACE (Lu et al. 2024). However, these models remain susceptible to inversion attacks (Pham et al. (023)),
where adversaries can use textual inversion techniques (Gal et al., 2022) learn and regenerate the removed concepts using the sanitized model.
Within fine-tuning, there are two main branches of concept erasing techniques: (1) Attention-based, and (2) Output-based or optimization- based.
Attention-baser methods (Zhang al., 2023) Orgad al.] 2023 Kumari et al., 2023 Gandikota
et al. 2024 Lu et al., 2024) focus on modifying the attention mechanisms within models to remove undesirable concepts. In Latent Diffusion Models (LDMs), for instance, the textual conditions are embedded via pre-trained CLIP model and injected into the cross-attention layers of the UNet model (Rombach et al. [2022] Ramesh et al.) 2022). To remove an unwanted concept, the attention mechanism between the textual condition and visual information flow is altered. For example, in TIME (Orgad et al. 2023), the authors propose the following optimization:
min Σ W C.EE
WI ce - + -IIWW - W1
(5)
where W represents the original cross-attention weights, W' the fine-tuned weights, C. the em- bedding of the unwanted concept, and of the targeted vector. Setting of = Wc,, where C2 is the embedding of a desired concept. allows the model to project the unwanted concept toward a more acceptable one. Follow-up works (Zhang et al. [023 Gandikota et al. 2024 Lu et al.. 2024) share this principle. Specifically, Forget-Me-Not (Zhang et al., 2023) introduces an attention resteering method that minimizes the 1.2 norm of the attention maps related to the unwanted concept. UCE
Gandikota et al. 2024) extends TIME by proposing a preservation term that allows the retention of certain concepts while erasing others. MACE (Lu et al. 2024) improves the generality and speci- ficity of concept erasure by employing LoRA modules (Hu et al. 2021) for each individual concept, combining them with the closed-form solution from TIME (Orgad et al., 2023). This category has two main advantages: (1) the Tikhonov regularization form of the objective function alaloo for a closed-form solution, demonstrated in (Orgad et al.) 2023). and (2) it operates solely on textual embeddings, making it faster than optimization-basee methods.
Output-based methods (Gandikota et al. 2023; Wu et al. 2024) focus on optimizing the output image by minimizing the difference between the predicted noise Eg' (,r,t,c, and the target noise <0(2 t, c.). Unlike irrentioo-based methods, this approach requires intermediate images 2 sampled at various time steps during the diffusion process. While this method is computationally more expensive, it generally yields superior erasure results by directly optimizing the image, ensuring the removal of unwanted concepts (Gandikota et al.) 2023). Our proposed method belongs to this category, but our insights into optimal target concepts are generalizable across both categories.
A recent addition to the field, SPM (Lyu et al., 2024). introduces one-dimensional adapters that, when combined with pre-trained LDMs, prevent the generation of images containing unwanted con- cepts. SPM introduces a new diffusion process é E Ca O,Mc)). where Mc, is adapter model trained to remove the undesirable concept ce. While these adapters can be shared and reused across different models, the original model 0 remains unchanged, allowing malicious users move the adapter and generate harmful content. Thus, SPM is less robust and practical compared to the other approaches discussed.
15

---

Published as a conference paper at ICLR 2025
Ad Adversarial Concepts. The definition of adversarial concepts- those most affected by changes in model parameters when a concept is erased- was first introduced in (Bui et al. 2024b). Building on empirical observations that erasing different concepts impacts the remaining ones in various ways and that preserving a neutral concept alone is insufficient to maintain the model's 's capabilities, the authors propose method for preserving sensitive concepts while erasing undesirable ones. This ap- proach is similar to the ad adversarial training principle (Goodfellow et al. [014] Bui et al., 2022] |021 2020) which has been successfully applied to improve the robustness of models against adversarial attacks. Our work is inspired by the idea of adversarial concepts but with several key differences: ! Instead of focusing on the to-be-presers pe-preeeeeed concept, we investigate the impact of the target concept on the remaining concepts, offering new insights into the selection and importance of the target con- cept in the erasure problem. @ We propose an improved optimization method that more accurately identifies adversarial concepts. explicitly excluding those identical to the to-he-erased concept-an aspect not considered in (Bui et al. 2024b). @ For the first time, we introduce the concept a concept graph and its properties, providing deeper understanding of concept relationships in the latent space.
B FURTHER DETAILS ON THE ADAPTIVE GUIDED ERASURE METHOD
In this section, we provide more details on the proposed adaptive guided erasure method.
Algorithm.
We first recall the central optimization problem in Equation equation 3
min 8'
max
-a Tc)ll +A (G(() @ Tc) - e (G( @ Tc)ll
C
L2
where T-) is the textual embedding function that embeds a concept into a textual embedding, G(.) is the Sumbel-Softmax operator, and @ denotes the element product. Te is the textual embedding matrix of the entire concept space C, i.c., Tc = (T(CI). ((c)). (T(c)) that can be pre-computed by the fixed textual encoderr (.) in the foundation model.
The algorithm to solve the min-max optimization problem is provided in Algorithm T More specifically, given the foundation model 0, concept space C, and the erasing set E, precom- pute the embedding matrix Tc. Since we erase multiple concepts simultaneously, each concept ce has an associated optimal target concept c.. Therefore, we maintain dictionary D store the weights of the optimal target concepts for all concepts in the erasing set E throughout the op- timization process. Initially, the weights π are uniformly distributed across all concepts in C, i.e. D(ce] = [L/[C], 1/IC), 1//CI,,-.,,/III.. 1//[[]]
During each iteration, we first sample a concept c. from the erasing set E. Then, we retrieve the previous target concept x of ce from the dictionary D. By doing so, we can ensure the learning process is stable and continuous from previous learning process. We then compute the gradient of the loss function w.r.t. the weight и and update the weight It can be done iteratively for Niter times, however, in our experiment, we just simply set Nitr to be for simplicity. After having the updated weight ,, we update the model parameters by performing a gradient descent step, as well as the dictionary D.
detailed analysis on the impact of hyperparameters is provided Section D.3 In all experiments, simply set the trade-off > = 1.0. the temperature y 0.1, learning rate 1) 0.001, and Nuer = 1. Visualizations of the mixture of concepts can be found in Section D.5 and the process of searching for optimal target concepts is discussed in Section D.6
Limitation. A crucial aspect of our method is the concept space C, which is used to search for the optimal target concept. As discussed earlier, we use the Gumbel-Softme trick, which requires feedings the model with the embedding matrix Te of all concepts in the concept space C. However, this requires a large computational cost, especially when the concept space C is large. To mitigate the issue, we use small set of concepts C.,, which contains the most A closest concepts to the
16

---

Published as a conference paper at ICLR 2025
Algorithm 1 Adaptive Guided Erasure Fine-tuning
Input: Oc, E, A, Searching hyperparameters: Nier Learning rate:
Output: 0
4- O, 0g +- 0
Dic,] + 1//IC), . Vc, E E
Init the dictionary
while Not Converged do
Ce ~ E
// Find the target concept & w.r.t. current Ge
+- D(cr]
Retrieve the previous target concept
for to Niter do
π - x + VV (1(5,0,0%) + AL2(T,,,66))
> Inner Max
end for D 4-
b Update the dictionary
Update model parameters
0241 Og aVe, LI(T,0,0,, + AL2(T,0,002)
Outer min
1: #- k-1
end while
concept c, in the original concept space C for each concept ce reduce the computational cost. We simply choose k = 100 for all experiments. The similarity between concepts is measured in textual embedding space r(-) by cosine similarity. We provide the analysis on the impact of choosing concept space C., in Section
Additionally, demonstrated in Section D.5 the mixture of concepts is not a simple linear com- bination of the concepts. Some transformations, like from English springer" to French Horn", are smooth, while others are not. This limits the potential of fully leveraging the concept mixture. We leave further exploration of smoother concept mixtures for future work.
C EXPERIMENTAL SETTINGS
C.I NETFIVE DATASET
Two main challenges in evaluating an erasing method are: (1) How to verify whether concept present in the generated images or not? (2) How to ensure the evaluation is diverse enough to cover the output space of the model which is of infinite possibilities? To tackle the two challenges, we propose a evaluation dataset called NetFive, which consists of 25 concepts from the ImageNet dataset, for which concepts in the generated images. We also ensure the diversity of the evaluation by generating 500 samples for each concept More specifically, we choose total five subsets of concepts, each subset contains one anchor concept, e.g., "English Springer" and four related concepts, ranked by their relative closeness to the anchor concept.
leverage the pre-trained classification model verify the presence of the
To choose the related concepts, we use the original SD model to generate 500 samples for each concept. We then use the pre-trained ResNet50 model classify the generated samples. We then choose related concepts from top-10 10 most frequently appeared concepts in the output. In the end, we choose the following subsets of concepts:
Dog Subset:
English springer, Clumber spaniel (315), English setter (160), Blenheim spaniel (45), Border collie (39)
Vehicle Subset: Garbage truck. Moving van (244), Fire engine (188), Ambulance (97),
School bus (33)
Instrument Subset: French hor,, Basson (327), Trombone (210), Oboe (199), Saxophone (38)
Building Subset: Church, Monastery (445), Bell cote (274), Dome (134), Library (17)
17

---

Published as a conference paper at ICLR 2025
Ingln Coltegr
Engin Geen
Figure 6: Sample images generated by the SD model from the NetFive dataset. Each row corre- sponds to subset of concepts.
Equipment Subset: Cassette Player, Polaroid camera (78), Loudspeaker (51), Typewriter keyboard (19), Projector (9)
Each subset contains one anchor concept, i.e., English springer, and four related concepts. ranked by the frequency of appearance of that concept in the top-10 predictions. For example, in 500 samples with the input description "English Springer", there are 315 samples that classified "Clumber spaniel" in the top-10 predictions. The larger the frequency, the more related the concept is to the anchor concept. We provide sample images generated by the SD model in Figure 6
is worth noting that, while most of the concepts in the dataset have high generation capability, i.e., the total height of the stack is almost 100% shown in Figure I there are two concepts "Bell Cote" and "Oboe" that have low generation capability, with their total height of the stack around 60-80% as shown in Figure Beside Figure 6 Figure 7 for reference.
we also provide a failed sample from the "Oboe" concept in
C.2 METRIC TO MEASURE THE GENERATION CAPABILITY OF THE MODEL
In image generative models, while common metrics such as FID and Inception Score are used to evaluate the quality of the generated images, there is no direct metric to evaluate the generation capability of the model a specific concept. One of the main tasks in evaluating the generation capability is to detect the presence of the concept in the generated image. In our experiment, we intentionally choose all concepts from the ImageNet dataset, which allows trained classification model to detect the presence of the concept in the generated image. Thus, we can indirectly evaluate the generation capability of the model specific concept.
leverage the pre-
More specifically, given model Gg, we generate k 500 images with the input description cj, and then measure the two metrics:
Detection Score (DS-1/DS-5): of samples that are classified as the concept Cj in top-1 or top-5 predictions / k. This metric indicates how many samples can classified as the concept ej by the pre-trained classification model.
18

---

Published as a conference paper at ICLR 2025
Tnyla Collggr
Conker
Enple Geber
Figure 7: Sample images generated by the SD model from the NetFive dataset. Each row corre- sponds to subset of concepts. Failed sample from the "Oboe" concept showing instead of an oboe.
Confident Score (CS-1/CS-5)) Average confident score w.r.t. the concept Gj in top-1 or top-5 predictions if the concept Cj is detected in top-5 predictions, otherwise, the confident score is set to be O. This metric indicates how confident the model is when generating the concept Cj. A higher score means that the concept Cj is more likely to appear in the generated images.
It is worth noting that the confident score is designed to prefer a model that can generate more low-confident samples over model that generates fewer high-confident samples. Additionally, the top-5 predictions are more reliable than the strict top-1 1 predictions, because the top-5 predictions can tolerate some errors in the model's prediction, e.g., dog breed can be easily misclassified as another dog breed.
We present Go(c;) and Gz, (cz) as the generation capability of the original model and the sanitized model on the same query concept 6j after erasing concept ce, respectively. With this, we can mea- sure the impact of erasing concept e, on the generation capability of concept ej by computing the difference between Go(c;) and Go. ic,), A(ce, cj) = Go(c;) Ge, (cj).
C.3 FINDING A SYNONYM TO AN ANCHOR CONCEPT
In this section, we evaluate the erasing capability on the synonyms of object-related concepts, e.g., 'Church'. More specifically, we first utilize set of tools including ChatGPT, Dictio- nary/Thesaurus.com and Google image search to find the best synonyms for each target concept. To verify that these synonyms are indeed resembling the target concept, we then use the original model to generate images from the synonyms (e.g., 'a photo of Chapel'), and use the ResNet-50 model to classify the generated images. We then only keep the synonyms that have the top-5 accuracy higher than 50% to ensure that they are indeed gemeration-similar to the target concept. For some concepts we could not find any good synonyms, such as 'Golf ball' or 'Chain saw', except for some minor variations. We provide (top-1 I and top-5) accuracy of the synonyms as below, well those numbers of target concepts, the higher the accuracy, the more similar the synonyms are to the target concept.
19

---

Published as a conference paper at ICLR 2025
Cassette player (5.4;96.6): tape player (10.0;95.0), cassette deck (0.0:95.0), cassette recorder (2.5:92.5), tape deck (5.0;100.0)
Church (84.4;100.0): chapel (80.0:100.0), cathedral (50.0;100.0), minster (87.5;100.0), basilica (32.5;100.0)
Garbage truck (83.2:99.2): trash truck (87.5:97.5), refuse truck (80.0;100.0), waste col-
lection vehicle (97.5;100.0), sanitation truck (47.5:100.0)
Parachute (95.2:99.2): skydiving chute (93.9;100.0) paraglider (100.0:100.0)
French horn (99.6;100.0): brass horn (32.5;95.0). double horn (37.5;65.0), German horn (22.5:80.0)
Chain saw (76.4;89.0): chainsaw (92.0;96.0), power saw (26.0;58.0)
Gas pump (69.0;97.0); fuel dispenser (50.0:97.5), petrol pump (85.0;100.0), fuel pump (47.5;65.0), gasoline pump (70.5;100.0), service station pump (47.5;87.5)
Tench (76.0;98.0): eyprinus tinca (60.0:95.0), eyprinus zeelt (52.5;100.0)
English springer (92.4:97.8): springer spaniel 10..0,00.0), springing spaniel (60.0:90.0),
welsh springer spaniel (2.5:72.5), English cocker spaniel (32.5:77.5)
Golf ball (98.2;99.2); golfing ball (99.0:99.0)
D FURTHER ANALYSIS
D.1 MEASURING THE IMPACT OF ERASING ON THE GENERATION CAPABILITY OF OTHER CONCEPTS
We would like to provide additional results with different metrics follows while all results can be found in the project repository.
Targeting generic concept, with DS-1 metric (Figure 8) DS-5 metric (Figure 9. CS-1 metric (Figure 10), CS-5 metric (Figure I1..
Targeting to a specific concept with DS-5 metric (Figure 12
Targeting to a generic concept on Stable Diffusion version with DS-5 metric (Figure
Targeting to a specific concept on Stable Diffusion version with DS-5 metric (Figure 14
Observations based on DS-5 metric. To make the reading easier, we would like to recall the observations in Section which based on the DS-5 metric.
First, when mapping to generic concept, we can observe the following properties of the concept graph:
(Locality) The concept graph sparse and localized, which means that the impact of eras- ing one concept does not strongly spread all the other concepts but only a few concepts that are related to the erased concept ce.
(Asymmetry) The concept graph is asymmetric such that the impact of erasing concept Ge concept Cj is the same as the impact of erasing concept Gj on concept ce. Mathe- matically, A(c;, (5) * A(c,,
(Abnormal) The two abnormal concepts "Bell Cote" and "Oboe", which have low genera- tion capability begin with, are sensitive to the erasure of any concept.
In the case erasing exclusive concepts, such as "Taylor Swift", "Van Gogh", "gun", and "nudity" shown in the last four rows of the figure, the impact of erasing these concepts all NetFive cepts are also limited, except for the two abnormal concepts, which supports the above observations on the concept graph.
When mapping to a specific concept, we can observe the following properties that guide the choice of the target concept for erasure:
20

---

Published as a conference paper at ICLR 2025
(Locality) Regardless of the choice of the target concept, the impact of erasing concept is still sparse and localized.
(Abnormal) The two abnormal concepts "Bell Cote" and "Oboe" are still sensitive to the erasure of any concept regardless of the choice of the target concept.
(Synonym XXX) Mapping to a synonym of the anchor concept leads a minimal change as evidenced by the lowest (ce. oÿ) for all cj. However, it also the least effective in erasing the undesirable concept Ce-
(All Unrelated Concepts XX) Mapping to semantically unrelated concepts demonstrates the similar performance the generic concept as evidenced by the similar (ce,, 0/) between the three last rows (5th 7th in each subset.
(General concepts X) While choosing a general concept the target concept is intuitive and reasonable, it does not necessary lead to good preservation performance. For example, "English Springer" - "Dog" or "French Horn" "Musical Instrument Horn" still cause drop on related concepts in their respective subsets. Moreover, there are also small drops on erasing performance compared to other strategies, shown by DS-5 of 83%, 91%, and 78% when mapping "Garbage Truck" to "Truck", "French Horn" to "Musical Instrument Horn", and "Cassette Player" "Audio Device", respectively. The two observations indicate that choosing a general concept is not an optimal strategy.
(In-class /) The highest preservation performance which is consistently observed in all five subsets achieved when the target concept a closely related concept to G.- For example, "English Springer" + "Clumber spaniel" "Garbage Truck" - "Moving Van" or "School Bus" in the "Dog" and "Vehicle" subsets.
14 show the impact of choosing the empty concept and a specific concept the target concept for erasure with Stable Diffusion version 2. can be seen that the above observations are still valid, except for the abnormal concepts which have low generation capability and are sensitive to the erasure of any concept ("Bell Cote" and "Projector"). This consistency indicates the generalization of the observations on the concept graph of different models, trained on different datasets and settings.
Results with Stable Diffusion version 2. Figures 13
In addition, we conduct four different experiments, each involving the simultaneous erasure five classes from the Imagenette dataset while preserving the remaining five classes, generating 500 images per class. While we can successfully deploy the ESD method on Stable Diffusion 1, we are unable to do the same for the UCE method because implementation issue.
It can be seen from Table 3 that our method achieves significant improvements over the ESD method in both erasure and preservation performance, with gain of 2.5% in ESR-5 and 8% in PSR-5. This indicates the generalizability of our method on different generative models.
Method
ESR-11
ESR-51
PSR-17
PSR-5t
SD
18.28 + 8.97
1.80 + 0.64
$1.72
98.20 + 0..4
ESD
91.99 + 6.35
87.83 7.79
5..24
75.45 6.43
AGE
92.75 = 6.96
90.27 # 8.73
62.45 + 13.30
83.43 9.71
Table 3: Results of the experiment with Stable Diffusion version 2.
Why Use the DS-5 Metric? The CS- and CS-5 metrics measure the change confidence scores associated with query concept 0j in the generated images, whereas the DS-1 and DS-5 metrics assess the change in detection status (presence or absence) of the query concept. Therefore, the CS-1 and CS-5 metrics are more sensitive to the change of the image content, and should be treated as a complementary analysis to the detection metrics.
Due to the high degree of class similarity in datasets like ImageNet, where, for instance, one dog breed may easily be misclassified as another, the DS-5 metric more reliable than the DS-1 metric for verifying the presence of concept in the generated images. The DS-5 metric's focus de- tection across the top five predictions reduces the risk of misclassification, making it a more robust choice in such scenarios.
21

---

Published as a conference paper at ICLR 2025
Observations Based on Other Metries B illlssratee the impact of using the empty concept as the target for crasure, as measured by the DS-1 metric. While the results are not as pronounced as those seen with the DS-5 metric, similar patterns can be observed on the concept graph. For example, closely related concepts still show higher sensitivity to the erasure of concept, as indicated by larger differences in the DS-1 metric (more red color values exceeding 60%). The abnormal concepts such as "Oboc" remain sensitive to the erasure of any concept, with the smallest differences still exceeding 30%.
Similar trends are observed with the CS-1 metric, as shown in Figure 10, and with the CS-5 metric Figure IT] However, draw definitive conclusions, we emphasize that the impact of erasing concept does not strongly propagate to all other concepts but tends to affect only local concepts that are semantically closely related to the erased concept Ce. Nonetheless, a weak impact on other concepts can still be detected using complementary metrics.
D.2 IMPACT OF CHOOSING VOCABULARIES
Object-related settings Our method requires a concept space predefined by users to search for the optimal target concept. In this experiment, assess the impact of choosing different vocabu- laries for this task. The vocabularies compared include:
ImageNet: A list containing all 1000 classes from the ImageNet dataset.
Oxford: list the 3000 most common English words
CLIP: The CLIP token vocabulary, consisting of 49,408 tokens. While comprehensive, the
CLIP vocabulary includes significant number of nonsensical tokens, making it challeng- ing to use.
ChatGPT: vocabulary generated by ChatGPT using the following prompt: "Provide k=100 objects that resemble a "Cassette Player' but are not 'Cassette Player,' and export the list CSV file."
To reduce computational costs, we filter each original dictionary to set of k: 100 concepts most similar to the target concept c., forming the concept space Co.- Results indicate that the ChatGPT- generated vocabulary achieves the best preservation performance, with an impressive 96.49% top-5 accuracy- nearly matching the original model's performance. However, this vocabulary underper- forms in the erasing task, as reflected by its lower ESR-1 and ESR-5 scores. Both the Oxford and CLIP vocabularies demonstrate similar erasing and preservation performance. Finally, the ImageNet vocabulary strikes the best balance between erasing and preserving, achieving the highest ESR-1 and ESR-5 scores, while maintaining respectable PSR-1 (80.84%) and PSR-5 (94.4%) scores, only slightly below the original model's performance. Consequently, we select the ImageNet vocabu- lary the default vocabulary for all object-related experiments in the main paper. We provide the intermediate results of the erasing process for different vocabularies in Section
Table 4: Impact of choosing vocabularies object-related settings.
Vocab SD
ESR-11
ESR-51
PSR-11
PSR-5t
26.44
1.00
82.40
96.20
ESD
95.48
88.88
41.32
56.12
UCE
100.00
100.00
21.96
38.04
ImageNet
97.08
93.48
80.84
94.4
Oxford
93.48
87.68
66.88
85.40
CLIP
93.40
84.96
69.96
87.56
ChatGPT 83.60
41.84
80.92
96.49
Artistic style settings For artistic style settings, we use the same vocabularies in the object- related settings, except we exclude ImageNet. For the ChatGPT vocabulary, we modify the prompt to focus artistic style concepts rather than object concepts. The prompts used are: "Provide k=100
http://wwwwwoooooeeccmommtoooooe hww.:////oo.ttttacnnnnnnnnnooooo english/oxford30000
22

---

Published as a conference paper at ICLR 2025
Erase: "English Springer" Target:
100
80
D.6
Confidentt Score
40
102
Metric: DS-1
English
24
20
Clumber Spaniel
31
Engltsh
Bernain Spariel
Border Colle
Garsage Truck
Moving Van
Fire Engine
Ambulance
Schnol Rux
French Ilom
Basscon
Trembone
Okuv
Saxephone
Church
Monastery
Bel Cote
Dome
Utrrar
Cassette Player
Palarid Camna
Inucipeaber
Typewiiar Keyboard
Projector
Gogn
Taylor
m.dit
:
5
building
audio
English Springer Claniter Spantel
English Sefter
Blenbeim Spanial
Maving Hin
Fire Enghie
Amtiulannce
sickool Bus
Barmoon
Probboon
Oloer
Sananhone
Charrt
Masasstey
Done
Cansette Player
Library
Poserred Comera
Tpeeenter Kiy Graaaa Projector
Loudspekker
Bonser
darbage
French
Figure 8: Analysis of the impact of choosing empty concept as the target concept for erasure with DS-1 metric.
23

---

Published as a conference paper at ICLR 2025
Erase: "English Springer" Target:
100
80
D.6
Confidentt Score
40
102
0.0
Metric: DS-5
English
Clumber Spaniel
Engltsh
Bernain Spariel
Border Colle
21
Garzage Truck
Moving Van
H
Fire Engine
16
Ambulance
Schnol Rux
French Ioo
Basscon
Trembone
Okov
Saxephone
Church
Monastery
Bel Cote
Dome
Utraar
Cassette Player
Palarid Camna
Inucipeaker
Typewiiar Keyboard
Projector
Gogn
Taylor
m.dit
J
dog
building
audio
2
English Springer Clantter Spontal
Engiish Sefter
Blenbeim Spanial
Maving Han
Fire Enghie
Amtiulance
sickool Bus
Barmooo
Oloer
Sanantonn
Churrt
Masuastery
Cote
Dons
Cansette Player
Library
Postrred Comera
Tyeeenter Kiy inaanar
Loudspekker
Projector
Bonser
darbage
French
Figure 9: Analysis of the impact of choosing the empty concept as the target concept for erasure with DS-5 metric.
24

---

Published as a conference paper at ICLR 2025
Erase: "English Springer" Target:
100
80
D.6
Confidennt Score
40-
20
0.0
0
Metric: CS-1
English
0.95
0.03
0.0
0.09
E.0
631
3.55
5.22
9.50
0.01
0.30
0.02
0.90
0:33
3:85
0.00
0.11
Clumber Spaniel
0.73
0.07
0.01
8.11
5.5
580
0.12
0
0.04
0.0
1..91
0:28
3.54
3.90
0.94
Engitsh Setter
1114
010
660
6:12
39C
D15
010
0.91
0:31
3.99
Bernain Spariel
443
0.11
0.62
M60
648
5.44
6:22
2.0 0.50 M65
616
0400
0.02
0:08
3.77
3.00
0.37
Border Colle
8.3
413
0.13
627
6.22
6.2
3.06
4000
0.29
0.5
0.9
0.69
642
631
3.9
3.0 0.39 0.66
0.6
+3.0
5.03
6:22
3.00
2.0
0.10
Garzage Truck
807
008
006
040
660
2.2
213
0.39
0.66
0.6
6.36
6.0
6:33
3.99
313 0.13
0.01 0.32
0.0
6.02
622
331
300
0.10
Moving Van
802
006
4.53
008
043
0:38
8.24
381
0.25
0.00
0.00
0.08
6.00
6:3
3.33
3.00 0.10
0.06
0.04
0.04
0.81
6.28
3.30
3.0
0.10
Fire Engine
806
000
4.42
4.11
660
0:25
3.09
333
0.4
0.00
0.66
032
0.01
624
812
3.00
0.45
0.5
0.08
0.60
0.90
6:25
3.99
3.04
0.13
Ambulance
806
000
4:53
042
042
0:5
934
312
0.47
0.00
0.44
0.08
0.03
034
3.99
3.11 0.11
0.01
0.30
0.00
0.53
0.33
3:87
3.00
0.77
Schnol Rux
806
4:2
8.9
637
114
5.99
0.77
0.00
0.02
0.08
0.69
633
3.86
314
0.12
0/06
0.04
0.08
8.43
6.39
3.99
3.8
0.32
renchllloms
806
008
8:3
000
600
1.54
0.00
0.75
0.0
030
0.29
0.3
035
1.97
3.00
0.12
0.01
0333
0.09
0.53
0:33
0.30
7.00
0.11
Basscon
4:22
5.5
0.01
0.35
0.04
037
0.29
8.32
635
5.97
2.80
0.14
0.06
0.10
0.64
8.65
0:32
3.99
3.8
0.21
Trembone
2.81
0.29
0.2
0.34
6.32
6.35
2.11
300
0.10
0066
0.00
4..0
6.22
3.29
3.20
Okov
0.44
0.77
0.60
0.15
0.00
387
03.4
030
43.00
0.14
2.50
3.84
15.00
Saxophone
0.34
0.32
3.97
0.50
0.81
0.24
8.30
Chunch
0.25
0.04
0.00
4.01
3.06
3.40
0.59
0.00
0.08
0.53
8.55
Monastery
0.32
4.90
0.1
3.09
0.40
0.06
6.21
8.44
Bel Cote
0.8
0.39
Dome
Ubrrar
Cassette Player
Palarid Camma
Inudipeaber
Typewilir Keyboard
Projector
Gogn
Taylor
m.dit
dog
004
0.55
6.31
6.55
2333
3.50
0.06
building
0.30
0.06
0.4
0.34
6.0
6.2
3.09
2.56
0.12
065
0.55
0.11
640
0.2
2.9
2.00
audio
0.25
0.44
0.32
4.02
6:4
2.87
247
0.50
0.66
047
0.1
4.8L
6.24
8:25
2.0
0.24
English Springer
Clantter Spartall
English Sefter
Blenbedm Spanial
Bonser Callle
darbage Sruch
Maving Hin
Fire Enghie
Artiulannce
sichool Bis
Freech Ham
Barmooo
Oloer
Qhirfi
Masusstery
Dons
Cassette Player
Library
Poserred Comera
Tyeeenter Kiy iosaad
Leudspekker
Projector
Figure 10: Analysis of the impact of choosing the empty concept as the target concept for erasure with CS-1 metric.
25

---

Published as a conference paper at ICLR 2025
Erase: "English Springer" Target:
100
80
D.6
Confidennt Score
40-
20
0.0
0
Metric: CS-5
English
0.04
0
0.03
0.04
0.02
632
3:50
3.22
0.99
0.00
0.08
0.02
0.90
0:32
3:85
0.00
0.11
Clumber Spaniel
0
0.00
6.62
8.22
5.5
380
0.10
0
0.09
0.0
0..91
0:27
3.54
3.91
0.99
Engitsh Setter
0:7
64:
0.14
5.4
30H
000
D10
010
0:03
0:31
0.99
0.04
11:91
Bernain Spariel
0.11
0.62
M60
040
5.44
6:24
235
3.0
0.99 M65
046
0.00
-0.11
0:08 3.57
3.01
0.00
Border Colle
8.3
0.15
626
6.22
6C44
8.55
4000
0.00
0.5
0.0
0.69
642
632
3.7
3.0 0.30 0.66 0.6
+3.0
5.03
6:21
3.90
3.2
0.11
Garzage Truck
807
000
006
040
666
0.29
2.23
3.5G
0.30
0.6
0.68
6.36
6.65
6.44
3.93
313 0.13 0.06 0.11
0.0
6.02
0:24
3.11
3.0
0.19
Moving Van
802
000
4.53
4.89
0.0
4:2
8.00
3.1L
0.23
0.55
0.00
0.07
6.22
6:3
8.82
3,7 0.50 0.06 0.01
0.04
6.60
6.2
3.30
3.0
0.11
Fire Engine
888
006
4.41
4.11 600
0.4
3.09
333
0.40
0.2
0.66
032
642
634
311
3.00 0.44
0.06
0.06
0.60
6.00
6.44
3.90
2.00
0.44
Ambulance
806
000
4:53
040
640
8:4
313
311
0.47
0.10
0.00
0.08
0.64
034
3.55
3.50 0.00
0.00
0.08
0.00
0.62
0.22
3.87
300
0.00
Schnol Rux
A08
4:2
8.9
640
635
915
5.99
0.9
0.00
0.01
0.67
0.69
633
8.85
312 0.11 0/08
0.64
0.00
8.62
6:37
3.89
300
0.35
Tenchnlloms
808
008
8:3
002 600
047
5.5
0.00
0.72
0.02
036
0.29
0.37
0:39
3.97
2.99
0.11
0.08
0.30
0.09
0.53
0:33 931
7.1
0.12
Basscon
4:5
911
045
5.4
0.11
0.32
0.04
028
0.30
2.38
6.58
5.97
207
0.12
0.08
0.10
0.64
8.45
0:31
3.99
3.8
Trembone
2.11
0.30
0.1
0.35
6.18
6.36
2.31
300
0.30
0006
0.00
6.02
6.22
8.03
3.20
Okov
0.54
0.29
3.04
0.16
0.35
0..
14.8
03.4
030
0.02
0.44
2.50
3.8
Saxophone
000
6.38
0.38
3.97
0.30
43.00
0.24
3.30
Church
0.24
0.04
0.00
4.90
3.55
3.40
0.17
0.11
0.08
0.00
6.08
8.55
Monastery
0.02
0.32
6.0
3.00
0.41
6.21
8.44
Bel Cote
0.77
0.55
0.39
8.39
Dome
Ubrrar
Cassette Player
Palarid Camma
Inudipeaber
Typewilir Keyboard
Projector
Gogn
Taylor
m.dit
dog
0.44
6.36
6.36
233
3.50
0.10
building
0.27
0.7
0.4
0.15
040
6.2
3.09
2.56
0.12
0.00
0.5
0.11
6.0
6.5
audio
0.24
0.2
0.38
E32
6:5
2.87
2,7
0.29
0:4
C46
4.8L
0.2
1:36
2.0
0.24
English Springer
Clantter Spartall
English Sefter
Blenbedm Spanial
Bonser Callle
darbage Sruch
Maving Hin
Fire Enghie
Artiulannce
sichool Bis
Freech Ham
Barmoon
Oloer
Qhurfi
Masusstery
Dons
Cansette Player
Library
Poserred Comera
Tyeeenter Kay insaad
Leudspekker
Projector
Bell
Figure 11: Analysis of the impact of choosing the empty concept as the target concept for erasure with CS-5 metric.
26

---

Published as a conference paper at ICLR 2025
Erase: "English Springer" Target: "Springer Spaniel"
100
80
-0.6
Conrident Score
60
D5-5
0.2
20
English Springer Springer Spaniel
23
12
English Springer
Clumber
English Soringer Border Colle
300
Engiish Sprinee-
Endlin Springer Garbage Trace
Engish Engish-Spinnge Springer
Waste Collection Vericle
Gerbage Track Moving Van
Garaage Thurk
Sehool
Garbage Trace
Garbage Truck: French Horn
Garbage
Garbage Truck
French forn Basscon
Secophone
French Hom
Musical Inst uneen Horn
French
French Hom
Cassette Raye
frenchs Hom
Church Coapel
Church i Monustery
Church Bu iding
Cassette Payyr
Church Englih Springe
Church
Grosdiie Player Tape Baye
Cassetta Polaroid
Cassette Payer Prjeecto
Consette Device
Cassette Payer Eng Ish Springe
Cassette Garbage Truce
Payer
Engiish Springne
Cumbber Spanie
Erglish Serter
Blenheinn Spansel
Benter Costie
dlvingg Voen
rire Engliee
Scrieot Bus
French Horm
Womboore
Oharch
Mansttery
Bell Cole
Donte
Cassetto Player
Library
Pistersi Camerr
Tyt den Aiy shoard
Loudtspokkrr
Projecoor
bugie
Figure 12: Analysis of the impact of choosing a specific concept as the target concept for erasure with DS-5 metric.
27

---

Published as a conference paper at ICLR 2025
Erase: "English Springer" Target:
100
80
Confident Score
04
40
Metric: DS-5
English Springer
Clumber Spaniel
Engitsh
Bernain Spariel
Border Colle
Garzage Truck
Moving Van
Fire Engine
H
Ambulance
H
Schnol Rux
fenm
Basscon
Trembone
Okuv
Saxephone
Church
16
Monastery
Bel Cote
Dome
Utrrar
Cassette Player
Palaridd Camma
Inucipeaker
Typewilar Kuyboard
Projector
Gogn
Taylor
m.dit
2
building
audio
22
English Springer
Clantter Spontel
English Setter
Blenbein Spanial
Bonser Collie
Moving Han
Fire Enghie
Antislance
scciool Bus
Barmooo
Chirrt
Masuastery
Cote
Dons
Cansette Player
Library
Poserred Comera
Tyeeenter Kiy inaaaa Projector
Loudspekker
darbage
French
Figure 13: Analysis of the impact of choosing the empty concept as the target concept for erasure with Stable Diffusion version 2 with DS-5 metric.
28

---

Published as a conference paper at ICLR 2025
Erase: "English Springer" Target: "Springer Spaniel"
100
80
Confident Score
DS-5
0.22
0.0
Metric: DS-5
English Springer Spaniel
English Springer
Clumber Spaniel
English Border Colle
Engiih Springr Dog
English Springe
bage
Garbage Truck Moving Van
Garbage
Garbage Truck Track
Garbage
French Hom
Hor
Bross Hor
French Bassooh
French Hom
Saxophone
french Hom
Musica Instrument
Trench Hom
Church Chapel
Church Monaetery
Church Library
Church
Church
Cassette Player
Tape Rayer
Cassette Plyyer
Polamid Camera
Cossette Mayar
Audio Device
Engiist Springer
Clumber Sakmner
Engilihe Senter
Bleshelkr Stariee
Border Collee
Gutlage Trook
Moving vaen
fire Engoie
Aeibociance
Sctorll Biss
Frenchh Horn
Blsscon
Rranbatee
Saxaphone
Charcs
Honaastry
Cote
Doms
Caseette Player
Library
Pateroid Camera
Tytwer der Aryanarrd Projectos
Leuttgroktor
Figure 14: Analysis of the impact of choosing a specific concept as the target concept for erasure with Stable Diffusion version 2 with DS-5 metric.
29

---

Published as a conference paper at ICLR 2025
keywords or phrases that resemble 'painting,' and export the list to CSV file" and "Provide k=100 artistic concepts similar to the 'Kelly McKernan' art style, and export the list to a CSV file." The complete list of concepts is available in the corresponding repository. As shown in Table 5 the ChatGPT vocabulary achieves comparable erasing performance to the Oxford vocabulary, while outperforming it in preservation. Based these results, select the ChatGPT vocabulary the default for all artistic style experiments in the main paper.
Table 5: Impact of choosing vocabularies on artistic style settings.
To Erase
To Retain
CLIP
LPIPSt
CLIPT
LPIPS.
ESD
23.56 t 4.73
0.72 0.11
29.63 + 3.57
0.49 + 0.13
CA
27.79 + 4.67
0.82 = 0.07
29.85 + 3.78
0.76 + 0.07
UCE
24.17 + 4.73
0.74
30.89 + 3.56
0.40 = 0.13
Oxford
22.03
5.24
0.79
29.97 3.54
0.49 0.14
ChatGPT
22.44 5.03
0.80 0.12
30.45 3.35
0.44 0.13
Table 6: Erasing artistic style concepts. The best and second-best results are highlighted in bold and underline, respectively.
To Erase
Retain
CLIP
+
LPIPST
CLIP
LPIPS.
ESD
23.56 + 4.73
0.72 + 0.11
29.63 + 3.57
0.49 + 0.13
CA
27.79 + 4.67
0.82 - 0.07
29.85 - 3.78
0.76 0.07
UCE
24.47 1 4.73
0.100
30.89 1 3.56
0.40 0.133
MACE
27.96 1 4.22
0.60
0.100
31.52 2.91
0.25 0.122
AP
21.57
5.46
0.78+
0.78
+ 0.10
30.13 3.44
3.44
0.47
22.44 + 5.03
0.80 0.12
30.45 - 3.35
0.44 0.13
D.3 HYPERPARAMETER ANALYSIS
In this experiment, we analyze the impact of the trade-off hyperparameter À and the temperature in our method. For all other experiments, fixed X 1.0 and 7 = 0.1 We conduct experiment with object-related setting, erasing five concepts "Cassette Player", "Church", "Garbage Truck", "Parachute", and "French Horn" simultaneously. We vary and within the range of 0.01 to 2.
It can be seen from Figure 15a that the erasing performance decreases as À increases, while the preserving performance improves monotonically. This aligns with theoretical analysis in Section where À controls the balance between the erasure loss L1 and the preserving loss L2 in Equation equation
In Figure 15b
we observe that as increases, the preserving performance declines. though there is no clear trend for erasing performance. It worth reminding that y the temperature parameter in the Gumbel-
operator, which controls the discreteness of the mixed weight G((r) r in Equation equation 4 i.e., the lower 7 is, the closer (()) the true one-hot vector.
D.4 EVALUATION ON SCALABILITY
We conduct additional experiment demonstrateeevvllutt the scalability of our proposed method. More specifically, we erase 25 concepts from the NetFive dataset, simultaneously and collect additional other concepts from the ImageNet dataset to form a set of 100 concepts for evaluation. In the preservation set of 75 concepts, we intentionally include 25 concepts that are semantically similar to the 25 concepts being erased and 50 other concepts that are semantically un- related. We use the ImageNet hierarchy from this website https: abservablehe com/ (embostoce imagenet hierarchy and Google search to find these visually semantically similar concepts.
Table shows the breakdown the concepts (to- co-bbe-rased and pprserveddssimilee used the experiment.
The results of the experiment are shown in Table We investigate three different vocabularies for the concept space C in the experiment including the ImageNet (AGE-I), Oxford-3K (AGE-O), and
30

---

Published as a conference paper at ICLR 2025
E5R-1
ESR-5
PSR-1
P5R-5
ESR-1
E5I-5
PSR-1
P5R-5
97
98-
Prosery Success Rato
Brosing Success Rate
96
80
05
Erasing Success
94
88
Preservin Success
94
70
60
92
50
40
90
70
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1,75
2.00
(b) Y
Figure 15: Ablation study on the trade-off hyperparameter X and the temperature hyperparameter y.
Super-Category
To-be-erased
To-be-preserved
Dog
English Springer, Clumber Spaniel,
Chihuahua, Tibetan Mastiff,
English Setter, Blenheim Spaniel,
Red Fox, White Wolf,
Border Collie
Hyena
Vehicle
Garbage Truck, Moving Van,
Moped, Model T,
Fire Engine, Ambulance,
Golf Cart, Tractor,
School Bus
Forklift
Music Instrument
French Horn, Bassoon, Trombone, Oboe,
Organ, Grand Piano,
Guitar, Drum,
Saxophone
Cello
Building
Church, Monastery,
Boathouse, Greenhouse,
Bell Cote, Dome,
Cinema, Bookshop.
Library
Restaurant
Device
Cassette Player, Polaroid Camera, Loudspeaker, Typewriter Keyboard, Projector
Cellular Telephone, Laptop,
Television, Desktop Computer,
iPod
Table 7: Breakdown of concepts used in scalability experiment
the manually crafted vocabulary (AGE-M), where we leverage the knowledge of the to-be-erased concepts to generate the vocabulary, i.c. which words are semantically related the to-be-erased concepts, like "dog", "car", "instrument", etc. This manually crafted vocabulary is similar to the Oxford-3K but much smaller in size. We compare the performance of AGE with the ESD and UCE methods.
Compared to the baseline methods, it can be seen that UCE totally fails when the number of concepts increases. Our AGE method with Oxford-3K vocabulary (AGE-O) outperforms the ESD method in both erasure and preservation performance. Compared among the three vocabularies, can be seen that while AGE-O outperforms other vocabularies in erasure performance, it has significant performance drop in preservation performance. The AGE method with manually crafted vocabulary (AGE-M) achieves the best trade-off between erasure and preservation performance, which has small drop erasure performance compared to ESD but a much better preservation performance, which is consistent with analysis in Section 3.2
D.S MIXTURE OF CONCEPTS
Why mixture of concepts? Given that the concept space C is discrete and finite, natural approach to solving Equation equation 3 would be to enumerate all the concepts in and select the most sensitive concept & that maximizes the total loss each optimization step of the outer minimization
31

---

Published as a conference paper at ICLR 2025
Concept
SD
ESD
UCE
AGE-I
AGE-O
AGE-M
Erased Concepts
dog
99.4
11.6
0.0
35.8
11.8
7.6
truck
99.4
23.4
0.0
14.4
9.8
6.4
inst.
98.6
10.8
0.0
28.8
12.8
16.0
build.
91.2
38.8
0.0
61.2
53.0
74.8
elect.
95.0
23.8
0.0
31.6
11.8
52.0
Similar Concepts
dog
100.0
94.4
0.0
99.8
92.2
99.2
truck
100.0
79.4
inst.
98.8
33.4
0.0
80.4
34.8
84.8
build.
97.4
77.0
0.0
90.6
86,6
95.2
elect.
92.0
21.6
0.0
69.6
35.6
80.0
General Concepts
99.8
98.2
0.0
99.8
98.4
100.0
bird
99.6
0.2
rept.
95.6
83.2
0.0
94.4
83.8
89,2
insect
78.6
66.4
0.0
75.8
65.0
74.4
fish
93.8
73.0
0.0
95.8
64.6
92.8
veh.
99.8
82.8
craft
99.2
96.6
furn.
96.8
64.6
0.4
83.0
72.0
80.4
fruit
100.0
81.6
0.0
99.8
83.8
99.2
obj.
100.0
74.4
0.2
94.8
76.8
95.6
ESR-1
24.2
90.6
100.0
84.7
91.2
86.1
ESR-5
3.3
78.3
100.0
65.6
80.2
68.6
PSR-1
83.1
56.3
57.1
73.5
PSR-5
96.8
72.1
0.1
91.8
73.9
91.8
Table 8: Comparison of different methods on concept crasure.
problem. However, this method is computationally impractical due to the large number of concepts C. Moreover, many concepts are inherently complex, often composed multiple attributes, and cannot always be interpreted singular, isolated concepts within the space C.
Since the concepts are represented textually and the output of Latent Diffusion Models controlled by textual prompts, the most intuitive way create mixture of concepts, such as combining and 02, is through textual template. For example, using a prompt like "a photo of a ei and c2" ensures that both concepts appear in the generated image. However, this approach has significant limitation: it does not provide gradients, preventing us from using standard backpropagation to learn and fine-tune the target concept ca.
To address this issue, we employ the Gumbel-Softmax operator to approximate a mixture of con- cepts. We set the temperature to value below 1, ensuring that the resulting target concept is combination of a few dominant concepts, rather than an indiscriminate mixture of all the concepts in C.
Visualization To illustrate the effect of concept mixtures as described in Equation equation 4 we visualize the generated images g(=r, the image generation function (the diffusion backward process), 27 the initial noise input, and is the textual encoder. We fix с₁ "English Springer" and vary from 0 to I with different 02 values to simulate the mixture of concepts G(TT) C in Equation equation Intuitively, we expect the generated images to gradually transition from concept C1 concept C2 increases.
(c)(cc)) + aT(c2)) in Figure 16 where g() represents
In cases like "Church," "French Horn." and "Garbage Truck," this gradual transformation is indeed observable, where the image transitions smoothly from "English Springer" to the target concept as increases. However, for other concepts like "Border Collie" and "Golf Ball," the transformation less smooth. For instance, the image shifts abruptly from "English Springer" "Border Collie"
32

---

Published as a conference paper at ICLR 2025
Enghit
0CT
01
Cof
Figure 16: Visualization of the output images g(zo, (1 - (c) + (e(()2)) with different a and 02-
6-90100
D-TN""
Drviiri
Intermediate results of the search process, with images generated from the most sensitive concepts found by method and e, at the same optimization step. Bottom/b: Similarity between nudity attributes and keywords. The figure repeated for reading convenience.
Figure 17: Top/a:
when a increases from 0.4 to 0.5, or from "English Springer" to "Golf Ball" as a changes from 0.4 to 0.7.
This phenomenon suggests that mixing concepts is merely a linear interpolation between two target concepts, but is influenced by the intrinsic nature of the concepts C₁ and
D.6 SEARCHING FOR THE OPTIMAL TARGET CONCEPT
Erasing Nudity Concept We investigate the search for optimal target concepts e by visualizing the generated images from c, found by our method and the corresponding output of at the same optimization step as shown in Figure 17p. Additionally, we present the similarity scores between the nudity attributes detected by the detector and e as in Figure 17p. Firstly, the to-be-erased concept c. ("nudity"") is closely related to the sensitive body parts such as "breasts" and "genitalia", explaining why removing the "nudity" concept also eliminates these sensitive attributes. On the other hand, as discussed in Section 4. the target concept is designed to be locally aligned with but not exactly the same as Ce. In our results, all intermediate concepts e such as "Model". "Drawing". and "Toy" are highly correlated with the "nudity" concept satisifying the first condition. However, while being highly correlated with less sensitive parts such as "Feet", they are less correlated with more sensitive ones like "breasts" and "genitalia", meeting the second condition. It is worth recall that Equation S Ca serves as retained concept to preserve the model's capabilities. Therefore, the strong correlation between c and "Feet", and its weaker connection with other sensitive parts, explains the interesting advantage of our method that it can still retain the "Feet" concept while successfully erasing others, observed in Figure
33

---

Published as a conference paper at ICLR 2025
Erasing Object-Related Concepts To further illustrate how method operates across different settings, we provide intermediate results of the search process in Figures 18 19 These experiments follow the same setup in Section 5.1 exploring the effeett of using various vocabularies including CLIP, Oxford, ChatGPT, and ImageNet, as introduced in Section D.2
In each subfigure, the first row depicts images generated from the most sensitive concepts iden- tified by our method, while the second row shows the corresponding to-be-erased concepts Ge- Im- portantly, all images are generated from the same initial noise input 27, resulting in similar back- grounds, while still featuring the relevant concepts and Ce.
The results reveal that different vocabularies lead to distinct erasing outcomes.
can be seen that different vocabularies lead to different erasing effects. For instance, when us- ing ChatGPT's vocabulary, as shown in Figure [8c| the intermediate concepts or identified include ["Chapel," "Altar room," "Shrine"], which are semantically similar to the to-be-erased concept "Church." This close semantic similarity results in weaker erasing effect, where the "Church" concept is not fully removed in the generated images. This phenomenon is also evident in other object-related concepts, such as "Garbage Truck" as shown Figure 19c This result consistent with the low erasing performance score observed in Table as discussed in Section D.2
In contrast, using ImageNet as the vocabulary, as shown in Figures 18d and 19d leads to more effective erasure. Initially, the target concepts are closely related to the to-be-erased concepts ce. but they gradually shift to less related concepts. This results stronger erasing effect, as outlined in Section D.2
D.7 IMPACT OF ERASING ON SYNONYMS
Given the list of 'valid' synonyms as introduced in Section C3 we generated images using the sanitized models from four object-related settings, where each setting corresponds to erasing five Imagenette concepts simultancously. described in Section 5.1 We then evaluated the sanitized models using the ESR-1, ESR-5, PSR-1, and PSR-5 metrics. Specifically, if an image generated from synonym of to- to-be-erased concept classified that concept, the model has failed to erase it, and vice versa. The results are shown in Table
First, the SD-org and SD-syn represent the results of the original model using the original concepts and their synonyms, respectively. Notably, the PSR-1 score for SD-syn is only 58.5%, indicating that out of 100 images generated from synonyms of concept like "Trash Truck," only 58.5% classified as "Garbage Truck" by the ResNet-50 model. This is significantly lower than the 78% PSR-1 of the original concept (SD-org). However, for PSR-5, SD-syn achieves 92.5%, which is only 5% lower than SD-org. This suggests that while the images generated from synonyms may not be recognized as the target concept in the top-1 prediction, they are still often classified correctly within the top-5. With this understanding, we analyzed the performance of erasure methods in handling synonyms of both the to-be-erased and oo-be-erreerred concepts.
The results show that while UCE and ESD maintain strong erasure performance for the to-be-erased concepts, they struggle to preserve the synonyms of the to-be-preserved concepts. Conversely, CA achieves better preservation but at the cost of reduced erasure performance. MACE strikes better balance between erasing and preserving, making it the most effective baseline. Our method outper- forms MACE in preserving performance by large margin, achieving the best results overall, though MACE still leads in erasure effecctiveness.
D.8 QUALITATIVE RESULTS
This section provides qualitative examples to further highlight the effectiveness of our approach in comparison to the baselines. Due to internal policies regarding sensitive content, we are only able to display results from two settings: erasing object-related concepts and erasing artistic concepts.
Erasing Relect-Related Concepts Figures 21
22 and 23 show the results of erasing object-
related concepts using ESD, UCE, and our method, respectively. Figure shows the generated
34

---

Published as a conference paper at ICLR 2025
(a) CLIP
(b) Oxford
(c) ChatGPT
(d) ImageNet
Figure 18: Intermediate results of the search process. The first row generated from the most sensitive concepts found by our method, and the second row is generated from the corresponding to-be-erased concepts c, "Church". Each column represents different fine-tuning steps in increasing order. Each subfigure represents for different vocabularies.
35

---

Published as a conference paper at ICLR 2025
(a) CLIP
(b) Oxford
(c) ChatGPT
(d) ImageNet
Figure 19: Intermediate results of the search process. The first row generated from the most sensitive concepts found by our method, and the second row is generated from the corresponding to-be-erased concepts ce "Garbage Truck". Each column represents different fine-tuning steps in increasing order. Each subfigure represents for different vocabularies.
36

---

Published as a conference paper at ICLR 2025
Table Impact of erasing on synonyms
Method
ESR,-1 -1
ESR,-5
PSR,-11
PSR, 55
SD-org
22.0 11.6
2.4 1.4
78.0 11.6
97.6 1.4
SD-syn
41.5 8.2
7.5 +
58.5 +
92.5
2.3
ESD
91.
81.6
3.3
6.6
84.1
67.4
7.44
1.9
UCE
9.1
99.2
19.5 4.4
0.6
MACE
98.1 0.9
84.7
41.4 10.3
73.3 3.1
Ours
78.8 + 6.3
62.3 + 7.2
55.4 + 8.3
90.0 + 2.5
images from the original SD model. Each column represents different random seeds, and each row displays the generated images from either the to-be-erased objects or the to-be-preserved objects.
that the original SD model generate all objects effectively. When erasing objects using ESD (Figure 2T). the model maintains the quality of the preserved objects, but also generates objects that should have been erased, such the "Church" in the second row. This aligns with the quantitative results in Table [ where ESD achieves the lowest erasing performance.
From Figure 20
When using UCE (Figure 22 the model effectively erases the objects shown in rows 1-5, but the quality of the preserved objects is significantly degraded, such as "tench" and "English springer" in the 8th and 9th rows. This is consistent with the quantitative results Table [ where UCE achieves the highest erasing performance but the lowest preservation performance.
In contrast, our method (Figure 23 effectively erases the objects while maintaining the quality of the preserved objects.
Erasing Artistic Concepts Figures 24 25 show the results of crasing artistic style concepts using method compared to the baselines. Each column represents the erasure of specific artist, except the first column, which represents the generated images from the original SD model. Each row displays the generated images from the same prompt but with different artists. The ideal erasure should result in changes in the diagonal pictures (marked by red box) compared to the first column, while the off-diagonal pictures should remain the same. The results demonstrate that method effectively erases the artistic style concepts while maintaining the quality of the remaining concepts.
37

---

Published as a conference paper at ICLR 2025
prompt-0
prompt-1
prompt-2
prompt-3
prompt-4
prompt-5
prompt-6
prompt-7
prompt-8
prompt-9
Figure 20: Generated images from the original model. Five first to-be-erased objects (marked by red text) and the rest are to-be-preserved objects. Each column represents different random seeds.
38

---

Published as a conference paper at ICLR 2025
prompt-0
prompt-1
prompt-2
prompt-3
prompt-4
prompt-5
prompt-6
prompt-7
prompt-8
prompt-9
Macace
Figure 21: Erasing objects using ESD. Five first rows are to-be-crased objects (marked by red text) and the rest are he-be-preserved objects. Each column represents different random seeds.
39

---

Published as a conference paper at ICLR 2025
prompt-0
prompt-1
prompt-2
prompt-3
prompt-4
prompt-5
prompt-6
prompt-7
prompt-8
prompt-9
Figure 22: Erasing objects using UCE, Five first rows are to-be-erased objects (marked by red text) and the rest are presereed objects. Each column represents different random seeds.
40

---

Published as a conference paper at ICLR 2025
prompt-0
prompt-1
prompt-2
prompt-3
prompt-4
prompt-5
prompt-6
prompt-7
prompt-8
prompt-9
Figure 23: Erasing objects using our method. Five first rows are to-be-erased objects (marked by text) and the are co-be-preservee objects. Each column represents different random seeds.
41

---

Published as a conference paper at ICLR 2025
Ours
(b) ESD
Netde
(c) UCE
(d) CA
Figure 24: Frasing artistic style concepts. Fach column represents the erasure specific artist, except the first column which represents the generated images from the original SD model. Each row represents the generated images from the same prompt but with different artists. The ideal erasure should result in the change in the diagonal pictures (marked by a red box) compared to the first column, while the off dia-ona pictures should remain the same.
42

---

Published as a conference paper at ICLR 2025
Ours
(b) ESD
Natde
(c) UCE
(d) CA
Erasing artistic style concepts (continued). Each column represents the erasure of specific artist, except the first column which represents the generated images from the original SD model. Each row represents the generated images from the same prompt but with different artists. The ideal erasure should result in the change in the diagonal pictures (marked by red box) com- pared to the first column, while the off- off-oiigonal pictures should remain the same.
Figure 25:
43


Reviews:
[{"review_id": "08YKXn7b7S", "reviewer": "Reviewer_TuUt", "summary": "The paper \"Optimal Targets for Concept Erasure in Diffusion Models and Where to Find Them\" introduces a novel approach to concept erasure in diffusion models, aimed at mitigating the generation of harmful content by selectively unlearning undesirable concepts. The authors critique the existing fixed-target strategy, which maps undesirable concepts to a generic target, as suboptimal due to its failure to consider the impact on other concepts. Instead, they propose modeling the concept space as a graph to analyze the effects of erasing one concept on others, revealing that the impact is localized.\nThe paper's key contributions include:\nEmpirical Evaluation of Concept Space: The authors present a novel empirical evaluation of the concept space's structure and geometric properties, highlighting the locality of the impact of erasing one concept on another.\nAnalysis of Target Concept Selection: They analyze how the choice of target concepts affects erasure effectiveness and the preservation of benign concepts, identifying that optimal targets should be closely related but not synonyms to the concept being erased.\nAdaptive Guided Erasure (AGE) Method: Based on their analysis, the authors propose the AGE method, which dynamically selects optimal target concepts for each undesirable concept using a minimax optimization problem. This method models target concepts as a learned mixture of multiple single concepts, allowing for a continuous search space.\nExperimental Validation: The paper demonstrates the effectiveness of AGE through extensive experiments on various erasure tasks, including object removal, NSFW attribute erasure, and artistic style removal. AGE significantly outperforms state-of-the-art methods in preserving unrelated concepts while effectively erasing undesirable ones.\nThe authors also introduce the NetFive dataset for evaluating erasure methods and provide metrics for assessing generation capability. Their findings suggest that the concept space is sparse and localized, with the impact of erasing a concept being asymmetric and affecting only semantically related concepts. The paper concludes that AGE offers a superior balance between erasing undesirable concepts and preserving benign ones, supported by a comprehensive study of the concept space's structure.", "questions": "How does the method ensure that the erasure of a concept does not inadvertently affect semantically related but benign concepts?\nHave you tested the AGE method on other types of diffusion models or generative models? If so, what were the results, and if not, what are the anticipated challenges?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality#\nThe paper presents a novel approach to concept erasure in diffusion models by introducing the Adaptive Guided Erasure (AGE) method. This method departs from the traditional fixed-target strategy by dynamically selecting optimal target concepts, which is a significant innovation in the field. The modeling of the concept space as a graph to understand the localized impact of concept erasure is a creative and original contribution. This approach not only addresses the limitations of existing methods but also provides new insights into the geometric properties of the concept space.\nQuality\nThe quality of the research is high, as evidenced by the thorough empirical analysis and the development of the NetFive dataset for evaluation. The authors provide a comprehensive set of experiments across various tasks, demonstrating the effectiveness of the AGE method. The use of a minimax optimization problem to select target concepts is well-justified and effectively implemented. The paper also includes detailed metrics and comparisons with state-of-the-art methods, which strengthen the validity of the results.\nClarity\nThe paper is clearly written and well-structured, making it accessible to readers with a background in machine learning and diffusion models. The authors provide a clear explanation of the problem, the limitations of existing methods, and the rationale behind their proposed approach. The use of figures and tables to illustrate the results and the impact of different target concepts enhances the clarity of the presentation. Additionally, the inclusion of appendices with further details and analyses supports the main text and provides a deeper understanding of the methodology.\nSignificance\nThe significance of the paper lies in its potential to improve the safety and reliability of diffusion models by effectively erasing undesirable concepts while preserving benign ones. The insights into the concept space's structure and the introduction of the AGE method could inspire future research in concept manipulation and erasure. The paper's contributions are relevant to a wide range of applications, including content moderation, bias reduction, and intellectual property protection in generative models. By addressing a critical limitation of existing methods, this work has the potential to significantly impact the development and deployment of safer AI systems.\nIn summary, the paper is a strong contribution to the field, offering original insights and a high-quality, well-executed methodology with significant implications for the future of diffusion models and concept erasure.", "weaknesses": "arget Concept Selection: The paper could benefit from a more detailed exploration of the target concept selection process, including specific examples and potential challenges.\nScalability: The scalability of the minimax optimization approach for large concept spaces is not fully addressed. Discussing computational complexity and optimization strategies would be helpful.\nGeneralization: The method's applicability to different types of diffusion or generative models is not thoroughly explored. Additional experiments or discussions on this aspect could enhance the paper's impact.\nEvaluation Metrics: A more comprehensive discussion on the choice and limitations of the evaluation metrics used would strengthen the validation of the method's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K9MSR4755g", "reviewer": "Reviewer_eC2n", "summary": "This paper considers the erasure of undesirable concepts for generative models, namely diffusion models. The proposed approach, AGE, Adaptive Guided Erasure, models the concept space as a graph and locally selects a related target concept to minimize unintended side effects. Furthermore, the approach performs very well empirically. The authors demonstrate its use in several different settings.", "questions": "How does the approach scale with the number of concepts? \nHow should we think how the granularity of concepts impacts the proposed method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Concept erasure for reducing harmful content creation is clearly a highly important and impactful research area.\n\nThe proposed approach has several strengths:\n* Clever and intuitive knowledge graph-based approach\n* Clear and well motivated storyline for the proposed objective\n* Wide variety of empirical experiments", "weaknesses": "I think that the paper could be improved by the following:\n\n* The general techniques and ideas appear as not very complex or novel (rather an application of related ideas, e.g. classic graph-based approaches) to new problems. On some level, the depth of empirical analysis makes up for this. However, the reader is left feeling as though there could have been more methodological innovation in the work. \n* The presentation of results could be a bit clearer to show more of where gains come from. For instance, in Table 1, it seems no other method comes close to the proposed one. I would love to better understand why this is?\n\nMinor:\n* typo 116: concept such as “A photo” or “ ”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yhb5OJIF4K", "reviewer": "Reviewer_BfS9", "summary": "This paper introduces the AGE method, aimed at enhancing the effectiveness of concept erasure within text-to-image diffusion models. AGE dynamically selects an optimal target concept that minimizes interference with unrelated concepts. The method models the concept space as a graph, discovering that erasure effects are localized. This insight leads to AGE's selection of closely related but non-synonymous targets for each undesirable concept, reducing unintended impacts on other model functionalities.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. AGE introduces an adaptive erasure approach that refines concept targeting by leveraging graph-based insights about concept space structure.\n2. The method minimizes unintended impacts on unrelated concepts, which addresses a notable limitation in previous fixed-target erasure methods.\n3. The findings are validated across different models, reinforcing AGE’s potential adaptability to various generative tasks and model architectures.", "weaknesses": "1. The optimization procedure for selecting target concepts may be computationally demanding for models with large concept spaces, which could limit AGE’s scalability in practice.\n2. The approach’s effectiveness relies on the accuracy of the concept graph’s structure. Any inaccuracies in capturing semantic relationships may affect erasure outcomes. Is there any discussion regarding this?\n3. Are there any human evaluations of artistic style?\n4. The proposed method doesn't achieve the SOTA, like Table 3. Any detailed discussion about this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)