Concept Siever : Towards Controllable Erasure of Concepts from Diffusion Models without Side-effect
Authors: Aakash Kumar Singh
Priyam Dey
Sribhav Srivatsa
Venkatesh Babu Radhakrishnan
Venue: Accepted by TMLR ()

Abstract: Diffusion models' unprecedented success with image generation can largely be attributed to their large-scale pretraining on massive datasets. Yet, the necessity of forgetting specific concepts for regulatory or copyright compliance poses a critical challenge. Existing approaches in concept forgetting, although reasonably successful in forgetting a given concept, frequently fail to preserve generation quality or demand extensive domain expertise for preservation. To alleviate such issues, we introduce Concept Siever, an end-to-end framework for targeted concept removal within pre-trained text-to-image diffusion models. The foundation of Concept Siever rests on \textit{two key innovations}: First, an automatic technique to create paired dataset of target concept and its negations by utilizing the diffusion model’s latent space. A key property of these pairs is that they differ only in the target concept, enabling forgetting with \textit{minimal side effects} and \textit{without requiring domain expertise}. Second, we present Concept Sieve, a localization method for identifying and isolating the model components most responsible to the target concept. By retraining only these localized components on our paired dataset for a target concept, Concept Siever accurately removes the concept with \textit{negligible side-effects, preserving neighboring and unrelated concepts}. Moreover, given the subjective nature of forgetting a concept like nudity, we propose Concept Sieve which provides a \texit{fine-grained control over the forgetting strength at inference time}, catering to diverse deployment needs without any need of finetuning. We report state-of-the-art performance on the I2P benchmark, surpassing previous domain-agnostic methods by over $33\%$ while showing superior structure preservation. We validate our results through extensive quantitative and qualitative evaluation along with a user study.

Published in Transactions on Machine Learning Research (10/2025)
Concept Siever Towards Controllable Erasure of Concepts from Diffusion Models without Side-effects
Aakash Kumar Singh" Vision and AI Lab, Department of Computational and Data Sciences, ISSe Bangalore
Priyam Dey
Sribhav Srivatsal
R. Venkatesh Babu
Reviewed on OpenReview: https://opeurevviv..eeee/ wttp://www...e/rurrmr////o//////
WARNING: This paper contains ezamples of NSFW (Not Safe For Work) content generated by teat-to-image diffusion models.
Inference Time Control (Ours)
Forgotten Concept "Artistic Style of Van Gogh"
Structure of the image preserved
X
Incrensing Strength of Forgetiing
Identity Van Gogh preserved
SD
AdvUnlearn (NeurIPS 24)
GLoCE (CVPR 25)
Ours
Forgotten Concept "Snoopy"
Structure of the image preserved
CHARPERON ROY II
Related Concept Preserved?
CHARLE
(Churlie Brown)
Base SD
MACE (CVPR 24)
GLoCE (CVPR 25)
Ours
Forgotten Concept "Nudity
Nearby Concept "Pezson"
We roduce Concept Siever, concept forgetting framework for diffusion models that provides surgical, concept-level control. By generating a precise vector in weight space, Concept Siever removes a target concept while preserving neighboring ones, overcoming limitations like structure or related concept distortion of state-of-the-an methods like Adversarial Unlearn ÇZhang et al. 2025), MACE (Lu et al. 2024] and GLoCE (Lee et al. 2225). Above we show that Concept Siever can forget Van Gogh's 's artistic style while retaining his identity, and can generate "Charlie Brown" without the unprompted appearance of "Snoopy", A key innovation of our framework is offering fine-grade fine-grained control over the forgetting strength at inference time, allowing users to dynamically adjust its effect without any retraining.
Figure 1:
*Correspondence Aakash Kumar Singh <aakashxumariiiec.co in>, Priyam Dey <priyandeedlissc.ac.tt. Venkatesh Babu <venkydiisc. inz. Wookk done during internship Vision and AT Lab, TSe Bangalore.

---

Published in Transactions on Machine Learning Research (10/2025)
Abstract
Diffusion models' unprecedented success with image generation can largely be attributed to their large-scale pretraining on massive datasets. Yet, the necessity of forgetting spe- cific concepts for regulatory or copyright compliance poses critical challenge. Existing approaches in concept forgetting, although reasonably successful in forgetting a given con- cept, frequently fail to preserve generation quality or demand extensive domain expertise for preservation. To alleviate such issues, we introduce Concept Siever, an end-to-end frame- work for targeted concept removal within pre-trained text-to-image diffusion models. The foundation of Concept Siever rests on two key innovations: First, an automatic technique to create paired dataset of target concept and its negations by utilizing the diffusion model's latent space. key property of these pairs is that they differ only in the target concept, enabling forgetting with minimal side effects and without requiring domain expertise. Sec- ond, we present Concept. Sieve, localization method for identifying and isolating the model components most responsible to the target concept. By retraining only these localized com- ponents on our paired dataset for a target concept, Concept Siever accurately removes the concept with negligible side-effects, preserving neighboring and unrelated concepts. More- over, given the subjective nature of forgetting a concept like mudity, we propose Concept Sieve which provides a fine-grained control over the forgetting strength at inference time, catering to diverse deployment needs without any need of finetuning. We report state-of- the-art performance the T2P benchmark, surpassing previous domain-agnostic methods by over 33% while showing superior structure preservation. We validate our results through extensive quantitative and qualitative evaluation along with user study.
1 Introduction
Modern large-scale cext-to-image (T2I) diffiusion models (Rombach et al. 2022 Ramesh et al. 2021 Saharia et al. 2 are trained on vast amount of internet-scraped data (Schuhmann et al. 2022). raising concerns on data privacy, copyright issues and inappropriate generations like NSFW (not safe for work) images. This has led to development of data protection acts to regulate their usage. For example, established under Article 17 of GDPR (GDPR.eu, 2018), the "Right to be Forgotten" allows individuals to request for deletion of their personal data when it's no longer necessary or if the consent has been withdrawn. Therefore, the ability to forgel has come up as an important requirement, wherein model needs to unlearn some particular dala it has seen during its training in order to align with the regulatory requirements. Given that these models are trained at large-scale, a post-hoc intervention becomes an important necessity to remove specific content when demanded. Towards achieving this, research in Concept Forgetting is gaining traction, where one aims to develop mechanisms to identify and remove a particular concept from a model. Doing so effectively can significantly improve these models' reliability, safety, and ethical compliance.
Early concept- foreept-forgetting methods (Gandikota et al. 2023 Zhang et al. 2024a) have successfully been able to forget particular concept in a model by designing specialized techniques, but they often suffer from poor specificily, where the model's generation capability for neighboring or unrelated concepts get severely impacted. Resolving this inherent trade-off between forgetting efficacy and preserving specificity is the central challenge of concept forgetting. Recent methods (Zhang et al., 2025 Heng & Soh 2024 Lu et al. 2024 Gandikota et al. 2024) aim to improve specificity by performing an additional step of preservation using a preservation set, which consists of neighboring or unrelated concepts. The nature of this preservation set dictates the degree to which one can improve specificity of the model. For instance, methods like MACE [Lu et al., [2024), UCE (Gandikota et al.) (2024) and GLoCE (Lee et al.] (0225) typically use a domain- specific preservation set often manually curated - to ensure high specificity. These "concept-aware" sets contain semantically similar examples to the target; for example, to forget one celebrity, the preservation set may include other celebrities who share the common attributes like nationality, ethnicity, or frequent
Although there is universally agreed-upon definition of what is concept, but generally speaking, a concept can take the form of object within an image, like kettle or a human face, to an abstract form, like an artistic style of painter.
2

---

Published in Transactions on Machine Learning Research (10/2025)
GLoCE's efficacy on I2P Bench
GLoCE's Specificity on Celeb identity
Table 1:
Comparison of preservation set cu- ration. Compared to other methods, Concept Siever satisfies all the desired properties: it au- tomates the process of preservation set curation, not requiring any domain-specific knowledge to do so, while also being concepl-aware.
Original Sel
Original Set
Celese)
1D raplacad
B
20 replacad
Set C-
Generic Celebs
Professions
300
Humans
80 (b) Specificity (Acc %6 1
(a) Number of Unsale Images|
Method
Automated Domain Awareness? Data Curationl' Agnostic?
Concept
UCE (WACV'20)
Sensitivity towards preservation set. Meth- ods that rely on domain knowledge (here GLoCE is shown) are often highly sensitive to the exact composition of the preser- vation set, with small progressive changes to it results in sharp drop in performance. See sub-section "Limitatious of Domain- specific preservation sets" in See for discussion on this.
Figure 2:
MACE (CVPR(2)
RECE (BOCV"24)
GLoCE (CVPR'25)
SA (NeuiPS5233
x
X
FMN (CVPR-W"24)
x
x
AdvUnlear (NearIPS:24)
Coucept Siever (Ouzs
collaborations with the forgotten celebrity. On the other hand, domain-agnostic methods that use generic sets naturally suffer from poor specificity due to the inherent limitations of their preservation set.
Limitations of Domain-specific preservation sets: One of the key limitations of methods using domain- specific sets is their reliance on domain expertise, which is often difficult to obtain in practice for users seeking to forget a concept. We demonstrate this issue for GLoCE (Lee al.. 2025). a state-of-the-an domain- speciffc preservation method, on two forgetting tasks: (a) forgetting NSFW content, and (b) forgetting the celebrity identity of "Brad Pitt". In Figure aa,,, we plot the no. of generated NSFW images by GLoCE when the preservation set shifts from its original, very specific and obscure prompts (e.g., black modest clothes, termed "Original" in the plot) to more standard ones (e.g., person in modest clothing, Set C). The observed sharp rise in the generation of NSFW images demonstrates the challenge in designing a perfect preservation set for any given concept. Further, methods that depend on such curated sets often exhibit high sensitivity to the exact set composition. We show an example of this in the case of forgetting celebrity identity of "Brad Pitt" in Figure Bb), where progressive replacement of the carefully curated celebrity set with increasingly generic celebritics leads to significant drop in specificity of the model, highlighting the issue of sensitivity. Another key limitation of such dopain-specific sets is that they can never be ezhauslive, i..., il is practically impossible to curate a set that guarantees the preservation of all other concepts. We illustrate a simple example of this in the top-left panel of Fig-1, where forgetting " "Artistic Style" of Van Gogh results in crasure of the identity of the painter itself for state-of-the-arl preservation- based methods like AdvUnlearn (Zhang el al. 2025 and
GLoCE (Lee et al. 2025 demonstrating the inherent limitation and incompleteness of preservation sets.
Unintended side-effects of forgetting algorithms: Specificity is not just inffuenced by the quality of preservation sets but also by the design of the forgetting algorithm itself. We show multiple examples of this in Figure I For instance, in the top-left panel, forgetting "Artistic Style" of Van Gogh using methods like AdvUnlearn and GLoCE results in distortion of the entire structure of the image. A similar effect can also be seen in the bottom-left panel of the same figure, where forgetting "Snoopy" completely alters the structure of the scene. Apart from structure distortion, we also observe that such a forgetting also impacts related concepts like "Charlic Brown" (see bottom row of bottom-left panel), even though they are made available in the preservation set. Such side-effects are undesirable, as scenes often contain multiple concepts, and forgetting should ideally target only the specified concept without affecting the rest of the image.
Subjectivity in concept forgetting: Finally, we note that the notion of forgetting a concept such as nudity is inherently subjective and deeply influenced by cultural context. We explain this using the example of Figure (right panel), where we progressively downgrade the level of nudity in the same NSFW image. Now depending on one's socio-cultural background, an individual may perceive the fourth image in the sequence as depicting nudity, while another may not even consider the second image to be inappropriate. Consequently, to provide the flexibility of control during inference time over the degree of forgetting of a concept is not only desirable but also crucial for accommodating the diverse sensitivities of different groups.
We note that this issue occurs state-of-theeart method like GLoCE even after preserving hand picked set of 100 neighhoring concepts provided by MACE.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Desiderata of concept forgetting: To summarize our discussion till now, a good concept-forgetting method should possess the following desiderata: a) good specificily, i.e., to ensure the forgetting is targeted and specific, not impacting any other generation capabilities of the model, b) good gemerality, ie., the ability to ensure effective forgetting of a given concept in all possible contexts, c) domain- aggnsiii,, i.c., the ability to forget concepts without the need of specifie preservation sets, which not only bring with them their own set of limitations as discussed above, but are also not scalable, and d) inference-time controllability to provide a user-level control over the degree of forgetting at inference time to cater to diverse requirements.
In this work, we present Concept Siever, novel concept-forgetting framework that fulfills all these desider- ata for forgetting concepts in Text-to-Image (T2I) diffusion models, achieving effective forgetting in just a few minutes with only the model and a set of prompts. Our approach introduces an automated pipeline for generating paired datasets of concept and concept-negated images (Stage I in Figure 3 by leveraging the model's latent space. key property of these pairs is that they differ only in the target concept, enabling for- getting without side effects and without requiring domain expertise. We utilize this dataset to train a Concept Sieve (Stage-II in Figure 3 and Section 3.1) which accuralely identifies and isolates the components of the diffusion model responsible for the target concept generation (we quantify this in Section 3.2) This targeted isolation ensures specific forgetting while preserving performance on unrelated concepts. By controlling the strength of this Concept Sieve using a single hyperparameter À at inference time (Section 3.2). we natively provide fine-grained, user-defined controllability over the strength of forgetting. Therefore, by combining all the above components, Concept Siever naturally delivers the specificity, generality, dommin-agnosticcimn, and controllability that define a good amcept-forgetting method. We illustrate this in Figure [ where Concept Siever preserves the overall image structure during forgetting (first row of left panels), minimizes impact neighboring related concepts through its enhanced specificity (second row, left pancls), and provides fine-grained user-control over NSFW content at inference time (right panel).
We empirically demonstrate the utility and effectiveness of our approach on the popular 12P benchmark of NSFW images (Schramowski et al., 2023), where we achiove a significant improvement of over 33% over the current stale-of-the-ar lorgetting methods like AdvUnlearn (Zhang et al., 2025). We also showcase our forgetting results on the diverse concepts of celebrity identity as well as artistic style. To summarize:
We introduce Concept Siever, a novel end-to-end framework for concept forgetting in T2T diffusson models, that operates without relying on domsin-specific knowledge or external models.
We propose an novel automated mothod for generating paired datasots of concopts and their negations by leveraging the diffusion model's latent space corresponding to the concept to be forgotten.
We also propose Concept Sieve, an accurate localization method to identify and isolate components of the model most relevant to the target concept, enabling precise and effective forgetting.
Our framework, by construction, facilitates fine-grained inference-ting time control of the strength of forget- ting without requiring any additional model finctuning.
We achieve state-of-the-arl results on the I2P benchmark (Schramowski et al. 2023 with a significant improvement of over 33% over prior concept-orrgeting methods. Our approach also demonstrates superior structure preservation when compared to all fine-tuning based baselines. We validate this claim through thorough quantitative and qualitative evaluation, along with user study.
2 Related Work
Concept Forgetting. Recent works in concept forgetting (Zhang et al., 2024a Kumari et al. 2023; Schramowski et al. 2023) addresses the challenge of preservation mainly via two paradigms: those requir- ing expert-designed, domain-specifie prompts to preserve related concepts, and those that operate without domain knowledge. Methods like GLoCE (Lee et al. 2025), MACE (Lu et al. 2024) and UCE (Gandikota et al., 2024) belong to the first category, relying heavily on carefully curated preservation sets tailored to each concept. In contrast, methods like Selective Amnesia fall into the second category, using generative replay with a general set of prompts for retraining after forgetting a concept using the technique of Elastic Weight Consolidation (Kirkpatrick et al.] [2017). While Selective Amnesia achieves good forgetting, it suffers from significant concept leakage (Heng & Soh 2024) due to its use of concept-indepeeddett regularization.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Different from these methods, which rely on manually curated preservation sets, and thus are inherently limited by the quality and the incompleteness of these sets, we completely eliminate such need and instead fully automate this generation process by leveraging the latent space of a diffusion model conditioned on the target concept. Another line of work (RECE (Gong ct al.) 2024). AdvUnlearn (Zhang et al.) 2025
try to enhance the robustness of the forgetting process by iterating through adversarial cycles: first they generate prompt attacks (e.g., via gradient- text perturbations (Gong ct al. 2024)) to expose residual concept traces, then apply methods like UCE (Gandikota et al. 2024) and ESD (Gandikota et al. 2023) to enable successful forgetting of the concept. While this co- co-training paradigm improves resistance to malicious regeneration attempts, it often comes at the cost of reduced specificity. In contrast, our approach effectively retains specificity by leveraging (1) automated concept-aware preservation set, and (2) Concept Sieve, which performs precise identification of the model layers responsible for generating the concept.
Model Editing. Model editing (Yao al., 2023) has gained significant traction in large language models (LLMs) as a cost-effective alternative (Lu et al. 2024 Orgad et al.) 2023) full model fine-tuning, which often requires substantial computational resources and extensive datasets. Task vectors (Tharco et all, )023), promising model editing approach, deliver strong results with smaller number of training epochs. Their simple arithmetic properties make them composable and suitable for interpolation, enabling flexible adjustments to models (Sanh et al.) 2021 Wortsman et al. 2022a b). But despite their efficacy, task vectors often disrupt model behavior due to their interaction with unrelated portions of the dataset. Additionally, training the entire model can also unintentionally alter its broader functionality. We address these challenges by proposing a sparse and localized technique of identifying target concept within the model components (Concept Sieve, Section 3.2 ensuring precise edits without compromising the model's overall integrity.
Efficient Finetuning Methods. Parameter-Efficien Fine-Tunnng (PEFT) (Houlsby et al. 2019) refers to techniques designed to reduce the number of trainable parameters, making model training more efficient and cost-effective. A common approach in PEFT methods involves learning parameter changes, AW, rather than
the full parameter sct, W (Hu et al. 2021 Kopiczko et al. 2023 Liu et al. 2024]. Among these, adapters based on low-rank decomposition of linear layers have gained popularity. While these methods have made model fine-tuning faster, more accessible, and less computationally demanding, they often become entangled with undesired concepts present in the training data. We address such issucs in our proposed method by training a secondary adapter concept-negated data to capture the collective influence of such undesired concepts, thereby enabling a complete disentanglemne of their effects. Subtracting this adapter from the concept adapter yields a well-isolated and disentangled representation direction of the target concept.
3 Proposed Framework: Concept Siever
In this section, we present our proposed framework, shown in Figure 3 At high-level, our approach isolates the signals that capture the concept to be forgotten from the diffusion model. The only input that is required for this isolation is the name of the concept. The first step is to create a set of paired data {ar a;"",, where of is an image of the concept to be forgotten and æ.th is another image with the same characteristics (pose, styling, background etc.) as of, but with the concept perturbed. We propose a novel automated way for creating such paired data from just the name of the concept, which we explain in Section 3.1 Next, we learn a vector in the weight space, which when added to the pre-trained weights, makes the model forget the specific concept of interest. We call this vector as Concept Sieve, explained in Section 3.2 It essentially acts as filter in the weight space that inhibits the concept to be forgotten, while allowing other concepts to pass through. This motivates us to name our methodology as Concept Siever. Concept Sieve, being a vector, allows us to scale it appropriately to have fine-grained inference time control over the strength of forgetting, a feature unique to our novel approach.
3.1 Automated curation of Concept-negated dataset
The core hypothesis of Concept Siever is the ability to isolate the changes in weight space related to the specific concept to be forgotten, while minimizing sidc-effocts on neighboring concopts. Towards this, we first create a dataset of image pairs with and without the concept. A key requirement is that there should be minimal changes in these image pairs except for the concept to be forgotten. To achieve that, we
5

---

Published in Transactions on Machine Learning Research (10/2025)
Stage-l: Concept-specific Automated
Stage-Il: Training Concept Sieve on the curated dataset
Stage-Ili: Forgetting using Concept Sieve
Dataset Curation
UNet
Prompt1,
Brad Fitt
photo
Original UNet
Brad
Block
Block
Block
Boue
PromptN]
Concept- negated Image
Concept
Train All loyers Bose UNet
Select Top-K Loyers
Retroin Selected
Concept DORA
negoted DORA
Bose UNee
photo
No Activation Perturbation
of Brad
Brad
Concept Image
Concapt- negoted DORA
DORA
Modified UNet
CLIP Text Encoder
We generate paired datasets by perturbing CLIP text embeddings at concept-specifie token positions, creating concept and concept-negated dataset preserving other attributes. Stage II: We train separate DoRA adapters on each dataset and compute their difference as our Concept Sieve (Eq. 4 Stage-III: When applied to the original model with scaling factor X, this vector acts filter that selectively removes the target concept while preserving related concepts. It enables precise control over lorgetting strength through vector scaling and layer-wise significance scores.
Figure 3: Concept Siever Framework: Stage=I:
first curate the concepl dataset. Given & concept e, say the celebrity Brad Pitt, obtain prompts with the concepts using LLMs, using predefined standard templates like "A photo of Brad Pitt", "Brad Pitt on red carpet". Images generated from Stable Diffusion with these prompts inputs constitute the concept dataset D.. To automatically create the equivalent concept-negated dataset D Den, we take the text embeddings corresponding to the concept phrase within the sentence (i.e., embeddings of only the phrase "Brad Pitt" "A photo of Brad Pitt") and add Gaussian noise to it:
E n ~ N(,, if S
otherwise
where Es are penultimate layer embeddings from CLIP RRdford et al.] 2021) text encoder, 2.2h are corrupted latents, and S the subset of indices that are related to the concept be forgotten. S 8 (4 for generating the corrupted embedding for the prompt "A photo of Brad Pitt". This simple automated approach shown schematically in Stage-1 of Figure 3 is able meet our requirements of the two datasets mentioned above. We attribute this localization property to the cntepoolationfriendly semantic latent space (Bhalla et al. 2024) of CLIP. The noise variance can be varied further lo obtain varying degree of separation between the concept and concept-negated dataset. We test different levels noise variance and the layer number of CLIP from which the embedding extracted in Figure of the supplementary.
3.2 Learning the Concept Sieve
The diffusion model Φe of Stable diffusson [Rombach et al. 2022) operates in the latent space of images. de contains a set encoder and decoder blocks each with self-attention, cross-attention and convolutional layers. Let 1 denote the number of such layers in the diffusion model. We hypothesize that there exists Concept Sieve in the weight space, which when removed from 0 would yield model 0" devoid of the specific concept to be forgotten with minimal side-effects to other concepts:
0" = XT
(2)
The above equation assumess the hypothesis of linear separability of concepts in weight space, which is well- supported in the editing literature (Ortiz-Jimenez et al. 2024). and we provide a brief discussion on the same
We the term "automated" refer the fact that most existing methods rely "mamually-curtedd and explicit" preservation sets, regardless of their speclfispe.ifiictt. contrast, our construction of concept-negated dataset émplicitly preservation without requiring any manual curation the list of concepts be preserved.

---

Published in Transactions on Machine Learning Research (10/2025)
in Appendix G.1 of the supplementary to justify its usage in this work. Therefore, model vector that points from model trained with forgotten concept, to that of model trained without the same in the weight manifold, scaled with the hyper-parameter A. For this, we finetune the base diffusion parameters on the concept and concept-negated dataset. Let w' 6 be the weight in the 8th layer of the diffusion model. For finetuning, we adopt parameter-efficient strategy like DoRA (Liu el al. 2024), where we choose decompose w into its magnitude m and direction V components as following (Salimans Kingma 2016):
V |V = =|wllle-
w
w = m
|
(3)
Then, the direction components V are further decomposed into A and B and trained via LoRA (Hu et al. 2021), along with m to obtain the modified weight w' (trainable parameters are underlined),
V+AV w I AV
AB I + AB
V
(4)
When the model is finetuned on datapoints from the concept dataset, we will obtain w/, and when finetuned on concept-negated dataset, we will obtain wen- We train both these models independently with the same configuration but with different datasets De and Den. We define the Concept Sieve T follows:
T= (w] w Vimi
(5)
The objective function C for this fine-tuning stage is the MSE loss between 2, and 2',, where 2, is the encoded latent representation of the image Fe at time step t, and z't computed as mentioned in Equation (
below:
/ do (2411,, 1, 0o) V000(2441,4,,,000
(6)
Note that Equation 6 can also be interpreted as lincar approximation to Equation 22, and y is the text conditioning.
Benefits: Concept Sieve 7 offers key advantages to the problem setup of concept forgetting with minimal side-cffects: Firstly, helps pin-point the specific layer of the diffusion model that has the most affinity towards a task. This score can be computed as follows: Stayer |mm - men ll / d, where me and men are the magnitude vectors of the DoRA parameters that we learn in Equation (4) and d is dimension of the layer, while training concept dataset and concept negated dataset. Empirically we find that if we identify the most important layer(s) using the score Stayer and finetune just those layer(s), we can reliably boost the sieving ability of our approach. Further, we can analyze the direction vectors inside V matrix to further prune those columns that maximally effoct the crasure of the concopt. This provides an extra degree of in inference-time control without any fine-tuning. We refer to this as Column Masking. Its score can be computed as follows:
Ui of mm m
Scotumn
(7)
where o² is the direction and m² is the magnitude of 7th column vector in V and respectively.
4 Experiments and Results
We evaluate Concopt Siever on three datascts: I2P Benchmark which consists of NSFW content (Shhramowski al. 2023), Celebrity Identity (Heng Soh 2024 and Artistic Style Gannikooa el al.
"This equation is inspired from Taylor's first degree approximation, and applicable setup
steer the
model towards targeted forgetting gently JOrtiz-Jimenez al. 2024). Also that Va forward paass Jacobian operation evaluated the initial checkpoint lg. The implementation adapted from this repositary.

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Forgetting Explicit Content. Middle panel reports the no. of images flagged as unsafe by NudeNet, with total count reported in the last column. The last pancl reports CLIP and FID scores for MSCOCO-30K (Lin et al., 2014) measuring the model's textual fidelity and image quality. Baselines in violet needs domain specific data. for training and not directly comparable. Concept Siever obtaín state-of-the- art results in NSFW content forgetting with good preservation quality, In the bottom two rows, we show how scaling down the Concept Sieve with varying À provides flexible trade-off between preservation quality (last panel) and forgetting efficacy (middle panel). * means the results are adopted from the original paper.
Results of NudeNet Detection I2P (# of images classified nude 4)
MS-COCO 30K
Method
Armpits Belly Buttocks Feet Breasts(F) Genitalia (reasts(M) Genitalia(M) Total
CLIP
FID 4
SD v1.4
148
170
29
63
266
18
42
7
743
31.34
14.04
SD v2.1
105
159
17
60
177
9
57
2
586
31.53
14.87
UCE
29
29
182
30.85
14.07
MACE
17
39
31.34
13.42
RECE
8
30.95
14.17
GLoCE*
14
30.95
13.39
SA
72
77
19
25
83
16
0
292
FMN
43
117
12
59
155
17
19
2
424
30.39
13.52
AdyUnlearn*
8
2
1
24
29.30
15.03
Ours (X-10)
8
3
12
0
2
31
30.34
14.26
Ours (A=12.5) 12.5)
1
0
2
16
29.73
14.51
We evaluate against. seven baselines, out of which four of them depend explicit preservation of domain-specific concepts GLoCE (Lee et al.] 2025 UCE (Gandikota et al. 2024), MACE (Lu et al. 2024), RECE (Gong et al. 2024), and the other three which are domain-agnosuic Forget Me Not (FMN) (Zhang et al.) 2024a), Selective Amnesia (Heng & Soh 2024 and AdvUnlearn (Zhang et al. 2025). As stated carlier, our method belongs to the latter category, where we not require access to any domain knowledge to curate the preservation set. Different from the preservation set, we curate the concept dataset using 40-50 text. prompts that are either human-designed or generated by large language model (LLM)), each explicitly referencing the concept be forgotten. Following previous works (Heng & Soh 2024 Gandikota al. 2024: Gong al. 2024 Zhang al. 2025), conduct all our experiments using Stable Diffusion v1.4 (Rombach 2022 as the base dilluusion model. We run the DDIM (Song el al. 2020 sampler for 50 time sleps.
2023
et al.
4.1
Forgetting Explicit Content
We follow MACE's evaluation protocol for benchmarking on I2P dataset for forgetting NSFW content. Detailed evaluation procedure is provided Appendix J.T] The results are presented in Table 2 where we can observe that Concept Siever sets a new state-of-the-art performance in removing NSFW content, improving over prior domain-agnostic SOTA methods like AdvUnlearn (Zhang et al. 2025 by significant margin of over 33%. We further demonstrate fine-graine grained inference-ting control over the NSFW content in Figure [ (right-pancl) by varying the À parameter (Equation (2)) Such control can cater to diverse sensitivities of different sub-groups at once, without the need of any model finetuning. Moreover, our approach also maintains comparable or better semantic knowledge and image quality compared to existing domain-agnostic baselines as demonstrated by the CLIP and FID scores the same table. We also include SD v2.1 as bascline in Table 2 is the SD version finc-tuned on training data with NSFW content filtered out.
4.2 Forgetting Celebrity Identity
Following Selective Amnesia (Heng & Soh, 2024), we aim to forget the identity of the actor "Brad Pitt" from Stable Diffusion, and evaluate the results using the standard GCD classifier (Heng Soh 2024 (lower value of this metric implics better forgotting). More details on the evaluation procodure and metrics can be found in Appendix J.2 The results are shown in Table 3 where we report superior Top-1 Accuracy results in the forgetting efficacy (see "Forgetting Effficacy" panel, lower is better here, implying better forgetting) compared
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Forgetting Celebrity Identity. Results on the efficacy forgetting are reported the middle panel of the table ("Forgetting Efficacy"), while the last panel reports specificity using GCD accuracy (f) for 100 unseen celebrities and other metrics. Best results are shown in bold, with next best results underlined. Baselines which are not domain-agnostic (x) are not directly comparable. Concept Siever demonstrate state-of-the-art results among the domsin-agnostie methods. Last three rows shows our efficacy-speciffiity control by varying the percentage of column masking (xx% CM) (see Section 4.4 and Appendix K.4.
Concept: Celeb Identity
Forgetting Efficacy
Specificity (Other Celebrities)
Domain Agnostic?
Top-1 Acc (%) +
#imgs w/o faces +
Top-1 Acc (%)
#imgs w/o faces 4
Method
LPIPS +
SD vl.4
94.74
96.61
21
UCE
X
0.00
8
95.46
32
0.347
MACE
X 0.83
78.09
0.543
RECE
X 0.00
23
52.05
83
0.347
GLoCE
X
2.00
96.45
0.002
SA FMN
20.00
88.42
0.563
V
18.75
10
64.30
63
0.545
Ours (40% CM)
V
15.04
92.23
29
0.127
Ours (45% CM)
4.05
86.83
32
0.140
Ours (70% CM)
0.82
75.86
0.178
Base Model
Domain Specific Methods
Domain Agnostic Methods
Forgotten Concept
Brad Pitt
Neighboring Concepts
Tom Hanks
Chris Evans
SDv14
MACE
GLoCE
FMN
SA
Ours
Figure 4: Forgetting Celebrity Identity. Post Forgetting "Brad Pitt" (top row), Concept Siever (last column) exhibits significant improvement over Selective Amnesia (SA). Note that methods like MACE and UCE generate images without faces for neighboring concepts (see bottom panel).
to all the existing domain-agnostic baselines. We also report two more results ours corresponding to the different column masking fractions to demonstrate better specifieity-effieccy trade-off.
To assess the specificity, we report Top-1 Accuracy, number of images where the model does not generate any face after forgetting (higher number implies poor specificity), and LPIPS metric for 30 other celebrities. The results of these metrics also shown in the same table (Table 3h. As one can observe, Concept Siever causes minimal impact on the model's ability to generate faces (imgs w/o faces 1), or any other image characteristies (LPIPS 1). We also present qualitative results in Figure where we observe the same trend better preservation capabilities Concept Siever compared to baseline approaches. This is facilitated by precise layer localization of our framework which we discuss detail Appendix F.TI. Additionally, the "concopt awareness" of our automated preservation set also allows to effectively preserve neighboring concepts despite being domain-agnostic. Additional qualitative result comparisons are provided in Appendix 0.1

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Forgetting Artistic style. We show superior performance of Concept Siever compared to state- of-the-art domain-agnostic methods shown by the columns "Structure Preservation" and "Style LPIPS Dif- ference" for forgetting, and by "Other Concepts" panel for preserving other concepts (specificity). We report specificity by calculating LPIPS 2500 images related to 100 artists, and efficacy by evaluating on 250 images on the forgotten concept i.e. "Artistic Style" of Van Gogh. Methods with domsin-knowleddes preserve 100 concepts. Best results are shown in bold, with next best results underlined. See 4.3 for more details.
Forgotten Concept: Artistic Style of Van Gogh
Other Concepts
Method
Domain Agnostic?
Naive LPIPS
Structure LPIPS
LPIPS 4
UCE
0.718
0.626
0.271
MACE
0.707
0.603
0.437
RECE GLoCE
0.736
0.636
0.398
0.722
0.522
0.007
SA
0.750
0.818
0.685
FMN
0.727
0.605
0.467
Ady Unlearn
0.745
0,645
0,500
Ours
0.645
0.442
0.228
Forgotten Concept
Base Model
Domain Specific Methods
Domain Agnostic Methods
Artistic style of Van Gogh
Related Concept
Identity Gogh
SRARSSIC SYAME
10 4
Artistic
Style
Briana Mora
SD
MACE
UCE
GLcCE
SAA
AdvLin
Ours
Figure 5: Forgetting Artistic Style. After forgetting the artistic style of "Van Gogh" (first row), Among the baselines, Concept Siever show better preservation of image content in both forgotten and rest of the styles, while maximally preserving the artistic style for the neighboring artist "Briana Mora".
4.3 Forgetting Artistic Style
Following ESD (Gandikota et al. 2023), we evaluate the forgetting of the "Artistic Style" of Van Gogh. For methods requiring explicit preservation (GLoCE, UCE, RECE, MACE), we preserve 100 different artistic styles for fairness. Standard evaluation for artistic style uses LPIPS measure between the original and forgotten images (we term this "Naive LPIPS"); however, this metric can be misleading, as large structural distortions undesirable per our desiderata mentioned earlier will also contribute to higher LPIPS score, which is incorrect. Therefore, we separately quantify structure preservation by observing that structure (or content) in image is primarily represented in the low-frequency components, and therefore to quantify this, we blur both the original and forgotten SD images using Gaussian noise across varying strengths, identify the blur range whore high-level structure remains intact, and compute the average LPIPS between the corresponding blurred images over this range, reflecting the overall degree of structure preservation.
The results are shown in Table with the structure preservation result reported under the column "Structure LPIPS From the table, we note that Concept Siever demonstrates best preservation quality (last panel)
AdvUnlearn is omitted from the celebrity identity as official implementation led catastrophic forgetting, causing the model to generate random noise retained concepts.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Forgotten Concept: Brad Pitt
Neighboring Concepts
Forgetting Efficacy
Tom
Hanks
Chris Evans
10
20
30
40
50
60
70
10%
40 %
50 %
70 %
% of column updated (Column Masking)
% of column updated (Column Masking)
Figure 6: Inference-time Control using Column Masking. By slowly increasing the number columns per layers (loft-to-right), we perform successful fine-grained forgetting of Brad Pitt (reflected by efficacy score), with minor impact in neighboring concepts (right plot).
along with superior structure prescrvation among methods that do not explicitly preserve the neighboring concepts. We also present qualitative results in Figure P where Concept Siever not only excels at preserving the neighboring concepts but also the original structure of the forgotten image, while successfully forgetting the style. To further validate this, we perform user study with 33 participants on 22 generated samples, asking them to pick the method that gives the best crasure of artistic style while preserving the image structure and other concepts. Results show that users prefer our method ~ 73% of the time compared to the baselines, validating our superior preservation efficacy. This can be attributed to the novel concept-negated dataset as well as the column masking feature of our framework, which precisely identifies the model layers responsible for the target concept, thereby incurring minimal impact on other concepts.
4.4 Inference-time Control over Forgetting strength
As mentioned before, significant advantage of our framework is the ability to provide fino-grained continuous control over the strength of forgetting at inference time, allowing users to seamlessly navigate the efficacy- specificity trade-off without any retraining overhead. This control is achieved through two complementary mechanisms: (1) scale of the steering vector X (Equation 2 and (2) column masking (Equation 7 Our primary mechanism for control is the learned stecring vector т, which shifts the model's weights to crase a concept. The intensity of this shift. can be continuously modulated by X: By adjusting À al inference time, user can smoothly transition from the original model behavior (A = 0) complete concept erasure. We demonstrate this capability qualitatively for NSFW content in Figure 1 (right-panel).
We also leverage column masking (Section 3.2 as complementary second axis of control. This technique allows us to define the scope of the intervention by selecting a specific percentage of the most relevant model parameters to update. To demonstrate this control, we conduct an experiment on the concept of celebrity identity "Brad Pitt", where we progressively increase the percentage updated columns from 10% to 70%. As illustrated in Figure 6. increasing the column mask percentage strengthens forgetting efficacy, successfully leading to a corresponding reduction in the classifier's softmax score for the target identity. However, expected, this increased efficacy produces collateral effects specificity: At higher masking percentages, we observe a noticeable degradation in the generated identities of neighboring concepts (e-g., "Tom Hanks" and "Chris Evans"), highlighting the direct trade-off that can be managed through this control. Finally, joint study analyzing the combined effects of the steering vector scale X, and the column masking percentage for the artistic style domain is provided in the supplementary material (Appendix 0.2 and Appendix M..
5 Targeted Forgetting: Top-K Layer Selection and Column Masking
One of our core hypothesis is that sparse, targeted edits to the model would allow us to preserve its behavior for other concepts. Therefore, to identify these components, leverage our automated data curation
11

---

Published in Transactions on Machine Learning Research (10/2025)
Forgotten
Ablation Concept Sieve Refinement
0.5
LPIPS
0.3
Others
0.2
0.1
0.0
Base
Single
Complete Data
+top-K
Column Masking
Single Image
Complete
+top-K +Column
Model
Image
Layers
Data
Layers
Masking
Figure Concept Sieve Refinement. First we show result using just one image in De and Den which rosults in high leakage with good forgetting. Then use complete D. and Dan (our approach), which leads to better preservation. Performing (op-K layer selection, followed by column masking (CM) further improves identity preservation and reduces leakage. Notice the re-emergence of identity of the Artist after localizing the weights (last two columns). We also report LPIPS for neighboring concepts in the bar chart on the right.
method to guide the editing process (Section 3.1.. While our data curation method does preserve the image semantics well, some distortions still occur. However, we note that these distortions are unique to each image, whereas the concept-related differences remain consistent across the dataset. Therefore, by training multiple images, reduce the leakage significantly. We can further stem the leakage by reducing the no. of selected layers while performing top-K layer selection for finetuning, as well using column masking (CM) (see Stage-II in Figure 3 To demonstrate this, we perform an ablation on these components and show their results in Figure 7 It is evident from the last two columns of Figure 7 that localizing top layers (top-K) and using column masking (CM) further preserves the image structure and content. Our further analysis on the localized layers show that attention layers are more relevant for the concept than other layers
(see Appendix F.1
6 Conclusion
Concept Siever presents an effective and flexible way to forget concepts in text-to-image diffusion models without requiring any extra guidance or domain specific knowledge. Through an extensive evaluation, we demonstrate the effectiveness of our method in preserving text fidelity and image quality, reducing concept leakage, and providing an active inference-time control to trade-off specificity with forgetting quality. Concept Siever makes T21 diffusion models easily accessible larger audience by giving them control over generative capabilities of stable diffusion by following the goals of safe AI.
7 Broader Impact Statement
While our proposed framework effectively removes harmful, explicit or copyrighted content through the concept sieve, acknowledge its potential for dual use- sse-specifiaally, the possibility of reversing the sieve to amplify the concepts which were supposed to be removed. This concern, however, is not just unique to our approach but applies broadly to the field of concept forgetting and model editing, and our method also falls within this scope. That said, we advocate responsible use our technology, and leave the development of techniques to detect whether images have had concepts suppressed or reinforced as future work.
lckmowwedggentt: The authors would like to thank the reviewers and the Action Editor for their valuable feedback which has significantly improved the quality of this manuscript. We grateful to Joseph KJ and Srikrishua Karanam for their valuable guidance lowards conceptualization and development of this work. We also extend our sincere thanks Kinshuk Vasisht for the helpful discussions and for providing code for identifying concept tokens.
12

---

Published in Transactions on Machine Learning Research (10/2025)
References
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). arXiv preprint arXiv:2402.106: 2024.
Zi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, and Wei-Chen Chiu. Promptingldehugging: Red-teaming text-to-image diffusion models by finding problematic prompts. arXiv preprint arXiv:2309.061 036.5 2023.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 2259-3269. PML.R 2020.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from dilfuusio models. Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2426-2436, 2023.
Rohit Gandikota, Hadas Orgad, Yonalan Belinkov, Joamna Materzyúska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5111-5120, 2024.
GDPR.eu. Right erasure ('right to be forgotten'), 2018. URL httpa://gdpr-info wwwareeaaaaeeeeee 17-t--17gd/// Accessed: 2025-03-07.
Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. In European Conference on Computer Vision, pp. 73-88. Springer, 2024.
Nick Hasty, Ihor Kroosh, Dinitry Voitekh, and Dmytro Korduban. Giphy celebrity detector. https://github. com/ Siphhr/aeee--dttcction-- 2019. Accessed: March 04, 2025.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36, 2024.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09665, 2021.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hanmaneh Hajishirzi, and Ali Farhadi. Editing models with lask arithmelic. In The Eleventh International Conference on Learning Representations, 2023.
URL httts:// nwtttfouummiieeeeeeeeeee fot//forrmmddddt00fff8--- jrjj
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Bhabska-Barwinskee et al. Overcoming catastrophic forgetting neural networks. Proceedings of the national academy of sciences. 114(13)33521-3555 3526, 2017.
Dawid J Kopiezko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaplation. arXiv preprint arXiv:2310. 11454, 2023.
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22691-22700 2023.
Byung Hyun Lee, Sungjin Lim, and Se Young Chun. Localized concept erasure for text-to-image diffusion models using training-free gated low-rank adaptation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18596-18606, 2025.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft. coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Shin-Yang Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang-Tie Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposedd low-rank adaptation. arXiv preprint arXiv:2402.009: 09353, 2024.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffussin models. In Proceedings of the IEEE/CV /CF Conference on Compuler Vision and Pattern Recognition, pp. 6430-64400 6440, 2024.
Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7559-7568, 2024.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7053-7061, 2023.
Guillermo Ortiz-Jimenez, Alessanro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nlya Sutskever. Zero-shot text-to-image generation. In International conference machine learning, pp. 8821 8831. Pmir, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:364199 36494, 2022.
Tim Salimans and Durk P Kingma, Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Autoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the TEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22522-225311 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. arXiv:2010.02502, 2020.
Denoising diffusion implicit models.
arXiv preprint
Yu-Lin Tsai, Chia-Yi Han, Chulin Xie, Chih-Haun Lin, Jia- Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? arXiv preprint arXiv:2310.1001 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebeoca Roelofs, Raphael Gontijo-Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple line-tuned models improves accuracy without increasing inference time. International conference on machine learning, pp. 23965-2333 23998. PMLR, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hanmanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, el al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022b.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang, Editing large language models: Problems, methods, and opportamilies. arXiv prepriut urXiv:2305.13172 2023.
Gong Zhang, Kai Wang, Xingqian Xu, Zhangvang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 1755-1764, 2024a.
Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy generate unsafe images... now. European Conference Computer Vision, pp. 383-403. Springer, 2024b.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. Advances in Neural Information Processing Systems, 37:36748 36776, 2025.
15


Reviews:
[{"review_id": "xZ2yr8rtee", "reviewer": "Reviewer_LwxK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\n\nThe authors proposed a principled and scalable method for removing concepts from Stable diffusion model. The authors proposed a automated method to curate image text pairs with or without certain concept, specifically, by generating the image twice with target concept token intact or masked out. This paired dataset is used to finetune the model twice, resulting in two DoRA weight vector, then the difference of the weights is used to change the base model, resulting a change in model behavior, focused on removing of the concept. \n\nThe authors performed extensive experiment to validate the efficiency and specificity of this method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "### Strength\n\n- The general conceptual framework for thinking about diffusion forgetting is interesting! esp. the specificity and generality tradeoff.\n- The automated paired dataset creation is very interesting, and this paradigm should be broadly useful for T2I in general.\n- Using contrast between finetuning LORA to measure importance of layer and edit model is interesting! It’s like an advanced form of gradient attribution.\n- Figures are very well made and illustrative. The experimental validation is also thorough.\n- The analysis of cross attention layer and and layer attribution is interesting!", "audience_yn": "Yes", "audience_explanation": "Yes, editting diffusion model is generally interesting question, and pratically relevant to usage of them in reality. The idea in the method (esp. auto data curation) is quite general and should be known by more people.", "requested_changes": "### Weakness\n\n- The notation and formalism in the method part Sec 3.2 is not clear, and could be improved by a lot.\n    - As an example $z^\\prime=\\phi(x) + \\tau \\circ \\nabla \\phi (x)$.\n        - in stable diffusion, there is VAE, and UNet, and the UNet has (at least) three inputs, latent $x$, time step $t$ and the prompt conditioning $y$. the notation authors used is a bit unclear, and I recommend making this explicit.\n        - Similarly $\\nabla$ is not clear which derivative the authors are taking, it’s better to denote it’s specifically gradient or Jacobian towards $w$.\n        - also should it be $\\phi(x)$ or $\\phi(z)$?\n    - For Eq. 3 and 4, the authors should mention that this decomposition is applied to matrix like parameters (also mention which matrix parameter do they include e.g. attention?) The equation do not make sense for vector parameters…\n    - In Eq. 3, why $\\|.\\|_c$? what does c denote?\n- The loss function seems important, but it’s not a separate equation. It should be written as a oneline equation and note the definition of z’ there.\n- Further, for Eq. 3, since it involves gradient to parameter in the loss, does the optimization involves 2nd order gradient? which parameter is trained in this final finetuning? it’s also not clear from the text… is it just the LORA weights or all weights?\n- The Figure 8B insight about ft effect on QKVO weights are super interesting! Can you quantify this finding more? I can only see one example, it seems that fine tuning QK is pretty good, kind of removing the style but not the content…\n\n### Questions\n\n- Why Stage 1 corrupted the penultimate layer of CLIP instead of last layer? What’s the rationale or intuition behind it? (I know the authors mentioned that they tuned such parameter and validate it).\n- Since the CLIP embedding is contextual and information flows everywhere, is there fear that even if you messed up with the specific tokens correspond to the concept, but other tokens already contains certain information of the ablated concept?\n- The training method is very efficient, why? Is it tricky to tune how many gradient steps are used to train the two LORA? Is there risk of under train or over train the LORA?", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gviiBJA8XF", "reviewer": "Reviewer_Km7W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Concept Siever as a new model editing-based framework to unlearn specific concepts from pre-trained text-to-image diffusion models. The method's primary contribution is an automated, domain-agnostic pipeline that does not require manually curated sets of images or prompts to generate a “concept-negated” dataset and preserve related concepts. This is achieved by adding Gaussian noise to the CLIP text embeddings of the target concept, which is hypothesized to disrupt the concept while preserving all other image attributes. Then, by fine-tuning two separate low-rank adapters on the original and negated datasets and subtracting their weights, the method derives a “sieve” vector that, when applied to the base model, erases the target concept.\n\nStrengths:\n1. The work targets aa relevant and challenging problem of concept erasure, which has significant implications for AI safety, copyright, and privacy.\n2. The method achieves state-of-the-art results on several benchmarks.\n3. The idea of generating a dataset via embedding perturbation is interesting and appears to be reasonable to the model editing toolkit.\n4. The framework provides practical fine-grained control over the concept erasure process, which could be valuable for real-world deployment.\n\nWeaknesses\n1. The paper claims to solve the problem of reliance on “preservation sets”. However, what it does fundamentally is to automate the creation of an implicit preservation set, thus (likely) inheriting the original problem’s limitations rather than solving them.\n2. The overall framework relies on a (unproven) assumption that concepts are linearly separable and disentangled within the model's latent and weight spaces. However, the paper provides no direct evidence to support this premise.\n3. Some of the core claims are not well supported by the method and empirical studies. See below for details.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper successfully supports its claims of empirical efficacy, which is commendable. Yet it fails to provide convincing evidence for other fundamental claims, and some are even contradicted by its own results.\n\nFirst, the paper’s primary motivation is to overcome the limitations of preservation sets (brittleness, incompleteness, domain expertise). It claims to solve this by eliminating the need for them. However, this appears to be a mischaracterization - the concept-negated dataset ($\\mathcal{D}_{\\rm cn}$) is an implicit preservation set that serves the exact same purpose: providing a large corpus of examples to constrain the model's update and preserve desired attributes. In this sense, the paper has not eliminated the preservation set; it has automated its creation. So, the method is still limited by the quality and completeness of this automated set, and it is difficult to confirm the core problem of ensuring specificity has well been addressed.\n\nSecond, the overall framework relies on a critical assumption that concepts are sufficiently disentangled in CLIP's latent space, such that adding noise to the \"Van Gogh\" token does not corrupt semantically related concepts like \"Post-Impressionism\" or \"thick brushstrokes.\" The paper provides no theoretical or empirical evidence to validate this assumption. If this assumption does not hold, the observed side-effects, e.g., degradation of related concepts, are not minor flaws but the reasonable consequence of the flawed premise: concept entanglement causes the sieve vector $\\tau$ to inadvertently target more than just the intended concept.\n\nThird, some core claims are contradicted by the paper’s own results. For example, the claim of “erasure without side-effects” is demonstrably false. Table 3 shows a clear drop in specificity (top-1 acc for other celebrities from 96.6% to 92.2%).\n\nIn summary, the paper successfully proves it has built a high-performing tool but fails to substantiate the claims.", "audience_yn": "Yes", "audience_explanation": "The paper’s findings would be of interest to a broad segment of the TMLR audience. The work's value is primarily the empirical results and its introduction of a practical technique. The communities that might find this work interesting include: AI safety practitioners, generative model and model editing researchers.", "requested_changes": "1. The authors should revise the manuscript to accurately frame their contribution. The narrative should shift from “solving the preservation set problem” to “proposing a new method to automate the creation of an implicit preservation set”. This requires major revision to many places including abstract, intro, and conclusion to better reflect the method is a new heuristic, not yet a fundamental solution to the specificity problem.\n2. The authors should add a dedicated discussion of the method’s assumption: the linear subtractability and functional disentanglement of concepts in the latent space. They should acknowledge this is a strong assumption and explicitly connect it to the literature and the empirically observed side-effects.\n3. To provide evidence for the claim that non-target attributes are preserved, the authors should conduct an analysis comparing the statistical properties of the concept dataset and the concept-negated dataset. For example, the authors might consider to use off-the-shelf classifiers to show that the distribution of key attributes (e.g., pose, background type, secondary objects) are unchanged.\n4. The paper is believed to be more robust if i includes a qualitative analysis of failure modes that result from concept entanglement. For instance, does forgetting a concept like \"lion\" also damage the generation of \"tiger\" or \"cat\"? This would be providing insights into whether ther are limitations when we rely on the implicit structure of the CLIP embedding space.", "broader_impact_concerns": "The work is well-motivated from an AI safety perspective, aiming to provide tools for removing harmful or copyrighted content. The potential for dual-use (e.g., for malicious censorship) is a general concern for the entire field of model editing and is not specific to this paper. The paper does not require a separate Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b8mLSKszIt", "reviewer": "Reviewer_Y7R6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary of contribution\nThis work proposes an end-to-end concept-forgetting model for white-box text-to-image latent diffusion models called Concept Siever that does not require a domain-specific preservation set per concept, while enabling strong concept unlearning and preservation of untargeted concepts simultaneously. The proposed latent perturbation-based data generation protocol and the controllability of test-time forgetting strength bring practical benefits towards desired generative models.\n\n# Strengths\n* The proposed method achieves strong concept forgetting performance without requiring domain-specific real data for forgetting and remaining concepts.\n* Dynamic control for the strength of forgetting during inference time enables flexible applications for various needs of users.\n\n# Weaknesses\n* Missing validation on the linearly separable weight hypothesis (the main working condition of the proposed Concept Sieve)\n  * The authors formulate the Concept Sieve adapted new model weight as a linear combination of the original weight vector and concept sieve weight vector. \n  * However, they do not discuss the realism of this hypothesis, nor how the violation of this hypothesis potentially affects the quality of forgetting.\n  * For example, in the weight interpolation literature, there is a well-known hypothesis, '_linear mode connectivity_' [1,2], to support the validity of the linear weight merging-based approach. There is also a line of work that claims weight space linear separability does not naturally hold [3], and controlling the separability can affect the quality of weight vector arithmetic. The author can make use this kind of discussion to add rigor to their justification on methodology.\n* Computational overhead\n  * The proposed protocol requires data generation (hundreds of, and one thousand images) from a diffusion model, which requires somewhat huge computation.\n  * It requires per-concept model training as well, which induces non-trivial resources. \n  * If users curated a dataset using the proposed pipeline, they may prefer to use a training-free method, e.g., GLoCE, on top of it, rather than utilizing the proposed training-heavy method concept sieve.\n* Overclaims\n  * The authors introduce their data generation technique as an **automated method** in the abstract. \n    * However, it requires hand-made text prompts for each concept to generate from them (the authors said they used 40-50 prompts for this). Although we can generate them with LLMs, it still requires prompt specification to LLMs, and obviously, this is hard to be seen as a fully automated method.\n    * Besides, the variance of the Gaussian noise and the layer index for perturbation should also be tuned. As shown in the Appendix, those two hyperparameters affect the overall quality of generations significantly.\n  * The authors emphasize that their data generation protocol ensures that the pair of concept and negated-concept images only differ in terms of the target concept while achieving minimal changes to other semantics. However, this claim was not validated, and **there is no theoretical guarantee that all the pairs from multiple concepts have only a difference in the target concept.**\n* Presentation issue \n  * [`Performance reporting inconsistency`] \n    * The authors include AdvUnlearn in Table 2 and Table 4 but omit it in Table 3, which makes the generality of the claim \"state-of-the-art among domain-agnostic methods\" questionable. \n    * In Table 4, the authors provide a new metric, 'structured LPIPS', as an alternative to standard LPIPS, but they only provide it on the forgetting concept evaluation, while not showing structured LPIPS for the remaining concepts.\n  * [`Missing formulation of structured LPIPS`] Although the 'structured LPIPS' is a new metric proposed by the authors to justify their claim on the semantic content preservation capability of the model, they do not provide a formal formulation for the metric anywhere, which makes reader hard to figure out the validity of the design of structured LPIPS.\n\n---\n\n> Reference\n\n1. Linear Mode Connectivity and the Lottery Ticket Hypothesis, Frankle et al. 2020\n2. Robust fine-tuning of zero-shot models, Wortsman et al. 2022\n3. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models, Ortiz-Jimenez et al. 2023", "claims_and_evidence_yn": "Yes", "claims_explanation": "As mentioned in the \"Summary Of Contributions\" section, there are a couple of issues with the validation of the authors' claim:\n1. the claim on the automated data curation pipeline => practically not automated.\n2. the claim on the generated pairs of concept/negated-concept images \"only differ in the target concept\" is not rigorously validated by empirical or theoretical evidence.\n3. tha claim on \"achieving the state-of-the-art among the domain-agnostic methods\" is not quite convincing, given inconsistent reporting of the previous SOTA method AdvUnlearn", "audience_yn": "Yes", "audience_explanation": "Although I brought up some weaknesses of this work, the TMLR's audience may find some interesting insights and practical benefits from Concept Siever, given that it doesn't require domain-specific real data (though it requires some computational budget), inference-time dynamic control on the forgetting strength, and the promise of a task weight vector arithmetic-based concept forgetting approach.", "requested_changes": "[Requesting items to secure my recommendation for acceptance]\n1. Weakening the claims on the data-generating pipeline -- non-automated and no guarantee on 'only differ in target concept'.\n2. Comparing the computational cost and wall-clock runtime of Concept Siever to baseline methods at least in the Supplementary material, and guide readers to that section who may wonder about it.\n\n[Requesting items to simply strengthen the work in my view]\n1. Adding a brief discussion on the validity of the linear separability of the diffusion model weight to make the justification of the method stronger.\n2. Consistently reporting the most competitive baseline, AdvUnlearn.\n3. Adding formulation of the structured LPIPS or adding implementation details for that metric.", "broader_impact_concerns": "Including this work, all the works related to concept-forgetting have potential risks that can be maliciously utilized by adversarial users. Simply using the reverse direction of the learned concept sieve can amplify the sensitive concepts of a model, so I think the author should discuss this potential misuse in a separate broader impact section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZynM8HDPft", "reviewer": "Reviewer_kSZQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the problem of concept erasure in diffusion models. The authors propose Concept Siever, an end-to-end framework that automatically constructs concept/anti-concept pairs by perturbing CLIP embeddings of the target concept tokens. Two DoRA adapters are then trained on these pairs, and their difference defines a \"forgetting direction\" that can be applied at inference time with a controllable strength parameter $\\lambda$. Experiments on NSFW content removal (I2P), celebrity identity suppression, and artistic style forgetting demonstrate significant improvements over domain-agnostic baselines while maintaining image quality and semantic alignment. Additional user studies and qualitative results highlight controllable forgetting strength and structure preservation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the submission are mostly supported by evidence, but not fully convincing. On the positive side, the paper provides quantitative results on the I2P benchmark and compares against several baselines, with clear improvements in NSFW unlaerning while maintaining CLIP scores and FID. The qualitative examples are compelling, and the ablation studies on the controllable parameter $\\lambda$ and masking strategies demonstrate that the method works as intended. However, the evidence has important gaps: the evaluation relies almost entirely on a single NSFW detector (NudeNet), which may bias the results, and no robustness checks with alternative detectors are presented. The experiments are also limited to Stable Diffusion 1.4, raising doubts about scalability to newer backbones. Finally, the assumption that Gaussian noise perturbation yields valid anti-concept samples is not validated with direct analysis. Overall, the evidence is clear but not fully convincing.", "audience_yn": "Yes", "audience_explanation": "I believe audiences working on diffusion models and concept unlearning would be interested in it.", "requested_changes": "- Provide stronger validation of the NSFW unlearning performance beyond a single NudeNet detector. At minimum, include evaluations with multiple independent classifiers or human annotation to rule out detector-specific bias.\n- Demonstrate the method’s generalization beyond Stable Diffusion 1.4 by including experiments on at least one newer backbone such as Stable Diffusion 2.1 or SDXL, showing whether the approach scales without major modification.\n- Justify the anti-concept construction strategy with empirical evidence. For example, evaluate whether perturbing embeddings with Gaussian noise consistently removes only the intended concept across paraphrased or synonymous prompts.", "broader_impact_concerns": "The submission does not include an ethics or broader impact statement, which is concerning given the sensitivity of the problem it addresses. Concept forgetting has clear societal and regulatory implications, particularly in contexts such as NSFW suppression, privacy protection, and copyright compliance. The paper also include unsafe images for demonstration (e.g., nude person), but no disclaimer or warning is presented. I suggest the authors include a Broader Impact Statement that discuss the potential ethical concerns and how to mitigate them.", "additional_comments": "- How robust is the proposed method to adversarial prompts (e.g., those generated by MMA-Diffusion [1])?\n\nRef:\n\n[1]: Yang et al., MMA-Diffusion: MultiModal Attack on Diffusion Models. CVPR 2024.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)