SDErasure: Concept-Specific Trajectory Shifting for Concept Erasure via Adaptive Diffusion Classifier
Authors: Fengyuan Miao
Shancheng Fang
Lingyun Yu
Yadong Qu
Yuhao Sun
Xiaorui Wang
Hongtao Xie
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Model
AIGC Safety
Concept Erasure
Primary area: generative models

Abstract: Concept erasure methods have proven effective in mitigating the potential for text‑to‑image diffusion models to produce harmful content. Nevertheless, prevailing methods based on post fine-tuning introduce substantial disruption to the original model’s parameter distribution and suffer from excessive model intrusiveness in two dimensions. (1) Images generated under erased concepts are perceptually aberrant. (2) Images generated under unrelated concepts exhibit pronounced quality degradation. We attribute these limitations to applying a uniform strategy to erase diverse concepts, failing to account for concept-specific generative mechanisms. Through rigorous experimentation and analysis, we identify that the generative process of each concept hinges on a narrow subset of critical timesteps. This insight motivates a targeted intervention strategy that enables precise and minimally invasive concept erasure. Therefore, we introduce $\textbf{SDErasure}$, a novel training framework for concept-specific erasure via adaptive trajectory shifting. First, a Step Selection algorithm that utilizes a diffusion classifier is proposed to guide the model in pinpointing the key timesteps associated with the undesired concept’s generation. Second, a Score Rematching loss is introduced to align the model’s predicted score function with that of anchor concepts, extending its applicability to both anchor-free erasing and anchor-based altering. Third, a Quality Regulation consisting of early-preserve loss and concept-retain loss is introduced to maintain the model's generative quality along two dimensions. Empirical results demonstrate that SDErasure achieves state-of-the-art concept erasure performance, reducing FID from 9.51 to 6.74 while effectively eliminating the target concept.

Published as a conference paper at ICLR 2026
SDERASURE: CONCEPT-SPECIFIC TRAJECTORY CONCEPT ERASURE VIA ADAPTIVE
SHIFTING FOR
DIFFUSION CLASSIFIER
Fengyuan Miao
Shancheng Fang"* Lingyun Yu
Yadong Qu
Yuhao Sun'
Xiaorui Wang
Hongtao Xie
University of Science and Technology of China 'Metastone Technology
Shenzhen University
(mfy123, eqqyd, syh3327) .ut.uuutcc....... edu en fangsclszu.edu (htxie, puly)austc.edu.. harzywxr@outlook.
ABSTRACT
Concept erasure methods have proven effective in mitigating the potential for text-to-image diffusion models to produce harmful content. Nevertheless, pre- vailing methods based post fine-tuning introduce substantial disruption to the original model's ' parameter distribution and suffer from excessive model intrusive- ness in two dimensions. (1) Images generated under erased concepts are perceptu- ally aberrant. (2) Images generated under unrelated concepts exhibit pronounced quality degradation. We attribute these limitations to applying a uniform strategy to erase diverse concepts, failing to account for concept-specific generative mech- anisms. Through experimentation and analysis, process of each concept hinges on a narrow subset of critical timesteps. This in- sight motivates a targeted intervention strategy that enables precise and minimally invasive concept erasure. Therefore, we introduce SDErasure, novel framework for cencept-speciffe erasure via adaptive trajectory shifting. First, a Step Selection algorithm that utilizes diffusion classifier is proposed to guide the model in pin- pointing the key timesteps associated with the undesired concept's generation. Second, Score Rematching loss is introduced to align the model's predicted score function with that of anchor concepts, extending its applicability to both anchor-free erasing and anchor- based altering. Third, Quality Regulation con- sisting of carly-preserve loss and concept-retain loss is introduced to maintain the model's generative quality along two dimensions. Empirical results demonstrate that SDErasure achieves state-of-the-ar concept erasure performance, reducing FID from 9.51 6.74 while effectively eliminating the target concept.
identify that the generative
INTRODUCTION
Large-scale text-to image models (Ramesh et al. 2022
Nichol et al. 2021 Ding et al.
2022
Chang
al., 2023 Rombach al. 2022 Saharia al. 2022) have seen rapid advances. However, training
on uncurated datasets that include NSFW content (Schramowski et al. 2023 Zhhag et al.
2024c
copyrighted artworks (Jiang et al. 2023). , and public-figure faces (Mirsky Lee 2021 Verdo-
liva. 2020 Sun et al., 2024) leads models to memorize and reproduce undesirable content (Carlini
ct al. 2023 Somepalli et al. 2023). Consequently, concept erasure is essential for preventing in-
appropriate outputs by removing undesired concepts, such that related prompts longer produce corresponding visual content and instead yield semantically unrelated outputs.
While training-free methods such |Schramowski al. (2023) and Meng et al. (2025) offer cost-efficient concept removal, they are vulnerable to bypassing when the source code is openly available. Prevailing training-based methods (Gandikota et al.) 2023 2024 /hhng al. 2024a Lu
et al. 2024 Gong et al. 2024) perform targeted fine-tuning to the model weights to achieve more
CCorresponding author.

---

Published as a conference paper at ICLR 2026
target
steps to choose
original trajectory
new (lacc))orylaat)
new (mjiccoryymmidde)
injeecory(eaaly)
target
step separability score
(d)
Different in Detaily Different in Concept
Different in Votal
Figure I: Illustration of erasure at different denoising timesteps on different concepts. For "cat", erasing at all steps (c) or early steps (d) changes the overall image structure, indicating over-erasure. In contrast, erasing at late steps (a) only modifies image details, indicating mnder-erasure. Erasing at middle steps (b) achieves the desired performance. For other concepts such as "Airplane" and "Flon Musk", adaptively selecting the denoising timesteps necessary to achieve the best performance.
comprchensive concept crasure. For instance, ESD (Gandikota et al. 2023) mitigates target con- cepts by adopting a negative guidance loss. However, training-based methods introduce substantial disruption to the original model's parameter distribution, resulting in marked decline in generation quality. Thus, the core challenge lies in balancing effective concept erasure with maintaining overall generation performance along two dimensions: (1) Related-concept quality measures the visual quality of images generated under the target concept, ensuring they remain high-quality even when the concept is erased. (2) Unrelated-concept fidelity quantifies the model's ability to maintain image fidelity when generating samples from unrelated concepts.
Recent studies have systematically analyzed the denoising trajectories of diffusson models (Meng) cet al., 202T] Raya & Ambrogioni 2023 Li et al. 20255), revealing that different concepts emerge distinet critical timesteps during the generation process. In the concept erasure task, identify that structural concepts (e.g.. airplane, church) are determined during the high-noise phases, while fine-grained semantic details (e.g., facial identities, artist styles) emerge in later, low-noise phases. Our preliminary experiments identify that restricting fine-tuning to a carefully chosen subset of de- noising timesteps substantially enhances erasure efficacy. These observations highlight the impor- tance of executing targeted trajectory shifts precisely at the timesteps when each concept emerges, thereby enabling more effective and efficient concept removal. As illustrated in Figure 11 we apply concept erasure to "cat" by fine-tuning at late (a), middle (b), early (d), and all (c) timesteps, respec- tively. Only the middle-stage tuning successfully removes the cat while preserving overall structure, whereas early and all-step tuning either distort structure leave residual artifacts, and late-stage tuning fails to fully erase the concept. Moreover, optimal erasure timing varies by concept type: structural concepts like "Airplane" require early-step intervention, while fine-grained concepts like "Elon Musk" benefit from later-step erasure to preserve semantic integrity, as shown in Figure []]
While fine-tuning at designated timesteps yields performance gains, the requirement to manually determine distinct timesteps for each concept severely limits the practicality and generalizability of this method. Therefore, propose novel training framework, SDErasure, adaptively determine optimal erasure timesteps. First, we introduce the Step Selection algorithm. At each timestep, the model predicts a noise conditioned both the target concept (to be erased) and the anchor concept (a replacement concept that steers the model away from the target). A diffusion classifier is then used compute the classification probability for each prediction, yielding a Step Separability Score (SSScore) which are then normalized across timesteps. Timesteps with higher scores indicate greater divergence between the denoising trajectories of the target and anchor concepts. By modifying the predicted noise at critical denoising steps, we shift the trajectory of the target concept away from concept-relevant regions, achieving precise erasure while minimizing interference with other data manifolds. Accordingly, Score Rematching loss is introduced to shift the denoising trajectory of the target concept toward that of the anchor concept. The Score Rematching loss supports anchor- based concept altering and degenerates into anchor-free free erasing when the anchor is set to empty. To prevent excessive intrusiveness in the model, a Quality Regulation employed with two compo- nents. The early-preserve term preserves noise predictions for the target concept on early timesteps,
2

---

Published as a conference paper at ICLR 2026
avoiding hazardous trajectory shifts and preserving related-concept quality. The concept-retain term preserves predictions for unrelated concepts, maintaining unrelated-concept fidelity.
SDErasure removes concepts by targeting critical denoising steps, preserving overall generation quality. SSScore locates these steps, and Score Rematching redirects the denoising trajectory away from the target concept. Quality Regulation ensures structural and conceptual fidelity. SDErasure outperforms prior methods with substantially lower FID (decreased from 9.51 to 6.74). Our contri- butions can be summarized follows:
We analyze and experimentally demonstrate why prior concept-erasure methods struggle to preserve a diffusion model's generative capabilities, and we show that fine-tuning at carefully selected timesteps alone achieve effective concept removal.
A novel Step Selection algorithm and SSScore are introduced to pinpoint critical denoising timesteps for adaptive and targeted erasure. Compared to existing methods, our algorithm substantially improves generation quality.
We design Score Rematching loss and a Quality Regulation consisting of two components to enhance erasure efficiency and generative quality along two dimensions.
RELATED WORK
Existing concept-erasure methods can be broadly grouped into three categories, each represented by seminal works. First, inference-time guidance methods, exemplified by SLD (Schramowski et al., 2023), avoid fine-tuning model weights by introducing reverse guidance during sampling to prevent undesired content generation. Although these strategies maintain the model's generative quality, they exhibit limited effectiveness in removing the target concept. Second, training-basee meth- ods alter model weights directly. Representative approaches include ESD and FMN (Zhang al., 2024a). The former employs a negative guidance loss, while the latter uses an attention attention-map map-based loss. While these methods achieve
comprehensive erasure, fine-tuning model weights often leads to a noticeable degradation in overall generation quality. Third, closed-form editing methods, such UCE (Gandikota et al.) (2044) directly manipulate cross-attention matrices by aligning the target concept with unrelated concepts. This strategy typically yields strong erasure performance in many scenarios but still suffers from a clear decline in the model's 's generative fidelity.
As research on concept erasure advances, more works have begun to address the balance between erasure and preservation of general generative capabilities. Several studies (Gong al.. 2024 Lu eet al. 20244 Zhang et al.) 2024b; Li et al.] [025c] Sun et al., (025) investigate this balance from various perspectives. However, these approaches apply a uniform erasure strategy to all target con- cepts, neglecting the unique characteristics of individual concepts. Erasing-nythinn (Gao et al.) 2025) addresses flow-based transformer architectures by employing a bi-level optimization strategy that combines LoRA fine-tuning with attention map regularization. Dark Miner (Meng et al.) 2024) proposes an iterative framework consisting of mining, verifying, and circumventing stages to adap- tively identify and erase latent embeddings responsible for undesirable generation. Co-erasing (Li eet al., 2025a) bridges the text-image modality gap by integrating self-generated images as visual supervision within collaborative framework. EraseBench (Amara al., 2025) is introduced to evaluate the robustness and side effects of crasure techniques. ANT (Li et al. (0225) provides valu- able insights into denoising trajectories and proposes crasing target concepts during the mid-to-late stages of denoising to avoid uncontrolled trajectory deviations. However, their approach overlooks the specificity of different concept generation processes, which limits ANT's overall performance. In contrast, our method adaptively selects critical timesteps for each concept to perform targeted and precise erasure, while minimizing interference with the remaining generation process. This strategy effectively removes the concept and preserves the model's ability to generate high-quality images.
METHOD
3.1 DIFFUSION CLASSIFIER PRELIMINARIES
Building on the generative capabilities DDPMs (Ho et al. 2020), a diffusion based classifier LLi al. 2023) can be derived by applying Bayes" theorem the model's approximate likelihood.
3

---

Published as a conference paper at ICLR 2026
c'atr" 'og'']
15
Erasure Training
Inpar image
KV
KV
Cor (Ap "cat", .
t=7
(= 0
Sumple N(O, !)
SSScore A?
Early Preserving
Sample (1.T)
Frozen Diffusiio Model
Early stage X
For (it "dog".
Figure 2: Overview of the proposed framework (Step Selection). Images of the target concept at various noise levels are fed into the model together with text prompts containing both the target and anchor concept. Under each textual condition, the model predicts the noise and computes the MSE loss against the true noise, yielding SSScore for each diffusion timestep. Timesteps with scores above the threshold are selected for erasure fine-tuning, while early timesteps with relatively low 50) are used compute the early-preserve loss.
scores (c.g.. 45
Since the exact conditional likelihood (No c) is intractable to compute, diffusion models opti- mize the Evidence Lower Bound (ELBO). The log- likelihood can be approximated by the negative expected noise prediction error across all timesteps:
(x = -E,, [||c co(x,, t, +C, +C,
where C is constant independent of c. By substituting this approximation into Bayes' theorem, and given class labels {c:),=1 with a uniform prior p(c,) 1/N, the posterior probability can be formulated as:
p(c:) (x ci)
expf - Er, llle cx ,, t, c:) c
po(e: xo) = Σjp(c) G))
(2)
E, exp E,, (x t, |)]]]
3.2 SINGLE STEP ERASURE
The analysis begins with timestet erasure to assess its efficacy across concept categories. The original ESD (Gandikota al. 2023) loss defined as:
Lesd E, [1lea(Es, c,t) + Go- Ep- (Ze, 1)
(3)
where t (I...., T) 77 controls the strength concept erasure, Go- represents the classifier- free guidance (Ho & Salimans 2022) of the original model. cp denotes the noise prediction of the fine-tuned model and 6g- represents the prediction of the original model. single step erasure, target concepts categorized high-frequency concepts (e.g., facial features) and low- wwwfireuenccy concepts (e.g., objects). Fine-tuning performed different denoising phases, with late-stage steps (e.g., & 15) for tigh-frequency concepts and early-stage steps (e.g., te 35) for low- low-frequency concepts. Our training objective is formally defined as:
L (2 C) + Go- eg- (e-, 1) 1= l le-
Concepts associated with smaller timesteps are generally easier to erase and incur less invasive- ness to the model. This is because high-frequency concepts tend to concentrate their own low- dimensional manifolds (e.g., People-Man-Elon Musk), so erasing such concepts does not interfere with other regions of the latent space. In contrast, existing methods perform erasure uniformly across all timesteps. resulting in excessive interference with the model.
3.3 STEP SELECTION
Despite performance gains from concept-specific timestep fine-tuning, the requirement to manually identify distinct timesteps for each concept greatly hinders the method's practicality and generaliz- ability. To address this limitation, an adaptive timestep selection strategy is introduced, leveraging the insight that diffusion models function effective classifiers even individual denoising steps (Li et al.] (023). As illustrated in Figure| 2 at each denoising step, a confidence score is computed quantify the classifier's certainty. Steps with higher scores are indicative of greater
4

---

Published as a conference paper at ICLR 2026
divergence between the diffusion trajectories of the target and anchor concepts. By concentrating our erasure training exclusively on these high-confidence steps, we focus the model's capacity on the critical phases of the diffusion process, yiclding more efficient and effective concept removal.
For the denoising trajectory of the target concept x = {IT, 21, Tt, mo we aim to identify the intermediate state xy at which the generation process for the target concept maximally diverges from that of the anchor. We analyze this divergence from two complementary theoretical perspec- tives: geometric trajectory and probabilistic likelihood.
Score Function Divergence. First, we characterize the direction of gen- eration using the score function, defined as the gradient of the log-density with respect to the data. At each timestep t, the generative directions conditioned on the target concept and the anchor concept ca are given by:
Geometric Motivation:
(c) V log a)),
(a) = V Po ca).
(5)
To quantify the divergence between these directions, we leverage the relationship between the score function and the noise prediction network diffusion models (Song & Ermon 22019] Ho et al.) 2020). Specifically, the score function can be approximated by the scaled noise prediction:
1
Vz, log Po c) Vi- 0₁
(o(,,,t,))
(6)
This equivalence allows us to reformulate the gradient divergence directly in terms of the model's noise prediction errors. By substituting Equation (6) into the objective of matching the true data score, the discrepancy for the target and anchor concepts can be measured as:
(2,, - 2 cla) = ea t, ca) -
Geometrically, a large difference between c/a) and - implies that the vector field guiding the generation of the target concept points in significantly different direction than that of the anchor, indicating a critical bifurcation in their trajectories.
Probabilistic Quantification: Instantancous Posterior. While the geometric perspective high- lights the qualitative nature of trajectory shifts, it lacks a normalized metric for automated selection. To strictly quantify this divergence, we interpret these losses through the lens of diffusion classi- fier (Li et al. 2023). The noise prediction error approximates the negative log- lig-liieddhood of the
data: log p(x, c) - (c) C. Consequently, the term exp(-ct.) serves as proxy for the unnormalized instantaneous likelihood. Based on Bayes' theorem, we introduce the SSScore as the instantaneous posterior probability of the target concept:
p(c, xx)
exp(- (c)
S. p(c: 21) p(co xx)
(8)
(xp(-c(0) exp(-L(a))
Probabilistically, a high S, signifies that the model distinguishes the target from the anchor with high confidence. This statistical certainty serves as quantifiable indicator that the target concept's trajectory has successfully decoupled from the anchor's. This identifies timestep t as critical intervention point to perform precise erasure with minimal collateral damage the shared generative manifold.
Accordingly, we employ a threshold-basee strategy where all timesteps satisfying S, >> are iden- tified as critical intervention points for fine-tuning. The pseudocode of the Step Selection algorithm can be found in Appendix B The discussion and experimental results in Appendix [ further confirm that SSScore successfully captures concept-specific characteristies.
By adaptively identifying highly discrininative timesteps, our strategy facilitates precise and min- imal model interventions, enhancing erasure performance while preserving overall image fidelity. Moreover, SSScore computation is a one-time pre-processing step rather than a repeated cost during training inference.
3.4 TRAJECTORY SHIFT FINE-TUNING
Figure g provides an overview of the training objective. The training follows a self-contrastive fine-tuning scheme.
5

---

Published as a conference paper at ICLR 2026
Score Rematching. The primary objective is to rematch the noise predictions under the target concept to those under the anchor concept. This is formalized as the following alignment loss:
Lu (ate, Cr, t) ca. (Ie, Cas t)ll2
To further enhance erasure effectiivness, negative guidance term is introduced:
(22, Ct2 Ca. = EØ- (2',, Ca. t) cØ (Z)e Cas t)
(10)
which captures the conceptual trajectory shift between G and Ca under the original model. This guidance is then used to steer the noise prediction away from the target concept and ward the anchor, enabling more effective and directed rematching. The final score rematch- ing loss is defined as:
Noisy latents
Self-contmsiive Fine-tuning
Score rematching Ay
SSocore
KV
Parfy-preserve: 4
C. = (1E,
Farly stage
- Ico- (Fz, Ca, 1) - 17 ( Cr Ca2 f)l
(11)
Concept-retain: 4
available in Ap- pendix E ca is set to empty to achieve anchor- free erasure. The erasure strength is flexibly controlled by hyperparameter
Detailed derivations
Inclevant concepts
Figure 3: relevant concept. Three groups of noisy latents are sampled for optimization: (1) latents with SSScore À are employed for Ce to suppress the target concept, (2) early-stage latents em- ployed for Lp to maintain related-concept qual- ity, (3) latents from irrelevant concepts em- ployed for C, to preserve unrelated-concept fi- delity. The training follows self-contrastive fine- tuning scheme, where the original model provides supervision for the fine-tuned model.
Training objective. C represents ir-
Quality Regulation. Empirical observations indicate that during the early denoising steps (e.g., 45 t 50), different concepts tend to converge as the diffusion process primarily directs samples toward the natural image man- ifold. Intervening at this stage disrupts the general trajectory, leading to substantial qual- ity degradation and -uttr---iistibutioon artifacts. Therefore, an early-preserve loss introduced:
Cp (Ee(xx,,,, ég- (It-, Ct. 1"
(12)
where t" represents early denoising steps. Rather than intervening in early steps, Lp enables target- ing later steps where concept-specific details are more prominently manifested.
To further preserve model's ability to retain unrelated concepts, a concept-retain loss is introduced:
C, || ( Gr,4) () <9- (xt Cr, l)
(13)
where denotes the concept to be protected. The concept-retain loss enforces consistency in noise predictions before and after erasure, preserving the semantic integrity of c,-
The final optimization objective then combines L. with the regulation terms:
C. + 31 C, 32 fep.
(14)
where 81 and 8n are hyperparameters that balance the importance of quality regulation against primary loss Le.
EXPERIMENTS
4.1 EVALUATION METRICS
A comprehensive evaluation of SDErasure is conducted across four distinct concept erasure tasks: object erasure, celebrity erasure, artistic style erasure, and explicit content erasure. SDErasure compared against range of state-of-the-aa baselines. including ESD (Gandikota et al. 2023
FMN (Zhang et al. 2024a), UCE (Gandikota et al. 2024). MACE (Lu et 20.4). RECE (Gong
al. 2024), SPEED (Li al. 2025c) and ANT (Li al. 2025b). The evaluation focuses three

---

Published as a conference paper at ICLR 2026
key properties (Lu al. 2024): efficacy reflects the ability to suppress generation of target concepts, specificity measures the preservation of concepts unrelated to the erasure target, and generality indicates robustmess synonyms and paraphrased prompts.
For efficacy, the CLIP Score (CS) (Radiord et al. 202T) is computed between the generated images and their corresponding text prompts. A lower CLIP Score indicates reduced alignment between the images and the erased concept, reflecting stronger concept removal.
For specificity, evaluation is conducted along two dimensions: (1) The model's ability to generate semantically adjacent concepts is assessed through its performance in producing alternative faces in the celebrity erasure task. This ability is further quantified by the LPIPS score (Zhang al.) 2018). which measures the perceptual similarity between image pairs. Lower LPIPS scores indicate smaller impact on the generation of semantically related content. (2) The preservation of unrelated concepts is evaluated on MSCOCO-300 (Lin et al. 2014) using CLIP Score (higher is better) for semantic alignment and FID (lower is better) for image quality and diversity.
For generality, the study focuses on object erasure, leveraging standardized synonyms (e.g., "car" vs. "automobile") to evaluate robustness against paraphrased prompts.
See Appendix H for more implementation details ]nd for additional experiments including multi- concept erasure.
4.2 OBJECT ERASURE
Evaluation Setup. Each erasure method is applied to ten individually fine-tuned models the CIFAR-10 dataset (Krizhevsky Hinton 2009), with each model targeting a specific object class.
To evaluate efficacy, 200 images are generated using the prompt "a photo of the (erased class)" and classified with CLIP, lower classification accuracy indicates more comprehensive erasure. Specificity is assessed by generating 200 images for each of the nine remaining classes using the prompt "a photo of the (unaltered class)." where higher accuracy reflects better preservation of non-target concepts. For generality, three synonyms selected per erased class to generate 200 images each using the prompt "a photo of the (synonymous class)" and reduced classification accuracy on these prompts suggests stronger generalization.
The overall erasure capability is measured by the harmonic mean Hg of the three metrics:
3
Ho (1 Accx)_1+++++ (Acc,) -lc.)--la (1 (1-Acc,) Accy)-I
(15)
where (1) Acc, represents the accuracy for the erased class. (2) Acc, represents the accuracy for unaltered classes. (3) Acc, represents the accuracy for synonyms of the erased class. Higher Ho indicates better overall concept erasure performance.
Analysis and Discussion. Table summarizes the erasure results for the first four classes in the CIFAR-10 10 dataset, along with the average performance across all classes. Since SPEED (Li et al.) 2025ch not applicable this task, it is not included. The results for the remaining six classes are provided in Appendix J.2 By training at optimal timesteps, SDErasure achieves the highest harmonic mean most categories, demonstrating its superior overall effectiveness. Notably, com- pared to our ESD baseline, SDErasure exhibits a significant advantage in maintaining high-quality outputs. Qualitative results are provided in Appendix J.4
4.3 CELEBRITY ERASURE
Evaluation setup. Efficacy is evaluated by generating 100 images from prompts containing the target concept and measuring their CLIP similarity to the prompts. Specificity assessed by gener- ating 100 images from semantically related prompts, measuring the LPIPS distance between images before and after crasure, and evaluating the CLIP similarity between post-erasure images and their associated prompts. As part of specificity, overall generation quality is quantified using FID scores on the MSCOCÓ -00k dataset.
Analysis and discussion. Table 2 compares methods target concept erasure, neighboring and unrelated concept generation. SDErasure consistently achieves significantly lower FID scores, in- dicating better preservation of general generative ability. SDErasure also yields the lowest LPPPS
7

---

Published as a conference paper at ICLR 2026
Table I: Evaluation of Category Erasure on CIFAR-10. Results are reported for the first four cate- gories and the average across all ten categories. All values are expressed as percentages (%).. The original model's performance included for reference.
Mahhed
Airplane
Automelsic
Bind
Cit
Average
Act. |
Acc.
Are.
H,
AAtry
Acc.
Acc:
R,
A U
Ansi
Act;
Art.
Acc,
F1,
Ans, L
Ace.
Ani;
FL
vl
96.06
08.92
95.05
195.75
98.95
75.01
99:72
98.51
95.45
98.93
98.60
99:05
98.63
98.63
63.64
FMN
96.76
98.32
94.15
6.12
95.08
96.86
79.45
11:44
99:46
99.13
96.75
1.38
94.89
97:97
95.71
6.83
96.55
97.82
91.52
6.45
UCE
40.32
98.79
49.81
64.09
4.73
99.02
37:25
82.12
10.71
98.3
15:07
90.18
2.35
98.00
2.58
97.70
14.53
98.52
26.41
83.52
ESD
7.38
85.48
5.92
90.57
30.29
91.02
32.12
74,88
13:17
86.17
20.65
83.98
11.77
91.45
13.5
88.68
13,65
88.53
18.05
$4.53
MACE
9.06
95.39
10:33
92.01
6.97
95.18
14.23
91.15
9.88
97:45
15.48
90.39
2.22
98.85
3.91
97.56
7.03
96.72
10.91
92.78
RECE
15.01
98.76
30.59
82.65
56.13
99.04
42.39
69.50
7.95
98.01
22.50
88.17
15.76
98.51
18.82
87.36
17.58
98.45
23.96
84.26
12.25
12.93
98.57
98.58
Ours
4.37
96.09
2.77
98.80
9.72
97.82
89.37
0.49
98.56
97.78
2.71
95.35
Table Evaluation of Celebrity Concept Erasure. The column with italic column headers (CS and FID) represents the performance of erasing the target concept. The remaining columns report the generation quality for other identities after the target concept has been erased.
Erasing Elon Musk
Erasing Taytor Switt
Erasing Donald Trump
Method
Elox Musk
Taylor Swift
Donald Trump
Flon Maskk
Taylor Swift
Donald Trump
Elon Musk
Taylor Swift
Dowald Trump
CSL
FID
CST
LPIPS
CST
LPIPS
CSt
LPIPS
CS}
FID
CSt
LPIPS
CSt
LPIPS]
CST
LPtPS
CSL
FID_
FMN
19.79
14.15
24.64
0.565
22:55
0.651
25.13
0.608
21.85
14.31
22.38
0.655
25.47
0.610
24.40
0.568
17.90
14.12
UCE
17.39
12,60
24.70
0.474
23.72
0.594
25.96
0.491
14.96
11.85
23.74
0,546
26.34
0.507
24.93
0.474
16.73
12.41
ESD
17.90
24.43
0.481
20.18
0.642
34.55
0,509
15.50
0.591
24.16
0.499
14.70
MACE
15.34
12.85
0.422
23.54
0.535
25:55
0.526
18.30
12.86
23.55
26.12
0.483
24.48
0.844
15.95
12.52
RBCE
16.67
13.20
24.90
0.476
23.80
0.603
26.32
0.526
15.65
12.03
24.08
0.570
25.45
0.567
24.00
0.689
15.34
18.34
SPEED
19.37
13.92
24.31
0.464
24.16
0.517
27.00
0.364
22.83
13.41
24.06
0.42
24:23
0.577
22.97
0.654
17.90
21.76
21.77
23.57
16.20
12.24
Ours
16.62
7.60
24.79
0.275
23.93
0.357
26.57
0.239
16.91
6.49
23.87
0.252
26.91
0.343
24.67
0.252
17.57
7.72
scores for neighboring concepts, suggesting minimal unintended trajectory shifts. As shown in Fig- u existing methods (first row) for erasing "Elon Musk" lead to degradation in the generation of semantically unrelated attributes such as structure (first column), color (second column), and pose (third column). In contrast, our SDErasure method (second row) enables both anchor-free erasure (first and second column) and anchor-based altering (third column), despite not achieving the low- est CLIP similarity for erased concepts, SDErasure effectively removes the target while retaining low- frw-frequency attributes such as color and pose. Additional qualitative results are provided in Ap- pendix 0.4
4.4 ARTISTIC STYLE ERASURE
Evaluation setup. This section evaluates SDErasure and baselines on erasing artistic styles. Efficacy measured by generating 100 images per artistic style using the prompt "a painting in the style of fartistic style)", followed by computing the average CLIP similarity between the images and their prompts. Specificity is assessed via FID and CLIP scores calculated on the MSCOCO-30 dataset.
Analysis and discussion. Table presents comparison of efficacy and specificity across different methods. While ESD demonstrates strong erasure capability, it substantially degrades the model's generative performance. SPEED and ANT better maintain overall generation quality but fail to sufficienlly remove the target concepts some cases. In comparison, SDErasure achieves superior trade-off between efficacy and specificity. This advantage stems from training at concept-sensitive timesteps and applying quality regularization.
4.5 EXPLICIT CONTENT ERASURE
Evaluation Sctup. The model is fine-tuned to erase inappropriate concepts such as "Nu- dity". To assess efficacy, images are generated using the Inappropriate Image Prompt (i2p) dataset (Schramowski et al.] 2023), with NudeNet (Bedapudi) 2019) employed at confidence threshold of 0.6 to detect sensitive content. Specificity is evaluated by measuring FID and CLIP scores MSCOCO-306 dataset.
8

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation of Artistic Style Erasure. The table presents the results of erasing the tar- get style and the model's generation quality post- erasure.
ESD
MACE
RECE
Previous SOTAs
Gogh
Previous methods crasing "Elon Musk
Method
MSCOCO
MSCOCO
MSCOCO
(S(1)
CSL1)
CSL))
FIDU) CLIPA))
FID(J CLIPKT)
FID()) CLINT)
Ouuss
FMN
22.36
13.85
30.33
21.24
14.63
30.88
25.23
14:25
30.87
UCB
21.22
14.33
31.33
22.84
7.65
31.15
23.17
14.44
30.73
13.60
30.50
21.10
30.79
13.51
MACE
23.49
12.28
31,04
23.77
12.08
31.01
21.70
12.28
30.95
"Eier-Mfuni
"Cn
"Dog"
RECE
22.78
8.05
51.08
24.01
7.67
31.16
22,30
8.65
31.17
Figure 4: Visual comparison of different era- methods. SDErasure effectively preserves semantically unrelated information while erasing target concepts.
7.49
ANT
21.4
7.75
31.16
22.03
8.94
31.12
23,42
7.8
31.09
Ours
21.12
7.02
31.12
20.98
9.06
31.13
20.78
8.09
31.17
Table 4: Evaluation Sensitive Content Erasure. The left side reports the number of sensitive instances detected by the NudeNet detector on the i2p dataset. The right side presents CLIP and FID scores on the MSCÓCO-304 dataset, with results from the original model included for reference.
Method
NudeNet Detection on 12P (Count)
Total
MSCOC0-30K
ARMPITS
BELLY
BUTTOCKS
BREAST(M)
BREAST(E)
GEN(F)
GEN(M)
FID.
FMN
43
117
12
59
19
155
17
424
15.81
30.51
17.70
ESD-u
33
31
24
14
121
21.10
30.07
RECE
22
27
2
69
19.16
30.60
SPEED
ANT
5
23
41.25
29.23
Ours
12
16.92
30.84
SDL4
148
170
29
63
42
266
18
7 743
31.16
Analysis and Discussion. As demonstrated Table 4 while ANT exhibits the lowest incidence of sensitive content across all body-part categories, this performance is achieved at a significant cost to generation quality. SDErasure achieves the second-lowed sensitive content generation while preserving high-fidelity output, obtaining competitive scores of 16.92 FID and 30.84 CLIP similarity on MSCOCO-308 These results substantiate that SDErasure successfully reconciles content safety with generation quality, underscoring its practical utility for responsible diffusion-based synthesis.
4.6 ABLATION STUDY
Choice of Step Selection threshold The threshold is varied from 0 to with a step size of 0.1, yielding 11 experimental configurations. For each setting, erasure effectiveness (CLIP) and generation quality (FID) are evaluated across a range of concepts, and the results are averaged. The overall trends are illustrated Figure Specifically, À represents the baseline that samples all timesteps uniformly for training. As can be seen, this strategy yields inferior erasure and generation performance compared our proposed approach, especially in terms of generation quality. When = 1, only the timestep with the highest SSScore is selected. This setting also underperforms compared selecting set of timesteps with an appropriate threshold. The results suggest that choosing À in the range of [0.5, 0.8) achieves favorable trade-off between erasure effectiveness and generation capability. In particular, ison, fine-tuning 5 randomly selected timesteps leads to significantly higher FID scores, further demonstrating the advantage the score-guided selection strategy.
0.8 shows the best overall performance. As a compar-
Impact of Core Training Components. This section evaluates the impact of incorporating the quality regulation Cp and L,. Four experiments are conducted involving batch erasure of 20 celebri- ties, with results reported as the average CLIP and FID scores. The results are shown in Table 5 Compared to the baseline without regularization, the proposed method substantially enhances the post-erasure generation quality. Specifically, the FID score decreases from 8.05 to 6.65, represent-

---

Published as a conference paper at ICLR 2026
CLiPe is beiter)
17.00
10
26.75
Table 5: Impact of Regularization Loss on model's generative ability. The CLIP scores of the original model are included for reference.
16.50
26.25
.08
16.00
15.75
0.4
0.3
Setting
CLIP
MSCOCO-300 30k FID!
15.10
6.5
7.0
7,5
(lower better)
0.0
0.5
9.0
9.5
Baseline (no regularization)
10.27
8.05
+ Lp
10.83
7,39
Figure 5: Impact of Step Selection threshold The yellow dots and their corresponding num- bers represent the experimental results under each À setting, while the red dot indicates the result of randomly selecting 5 timesteps.
+
10.47
7.06
Ours(+ Cg Lx)
10.91
6.65
SD1.4
29.25
Table 6: Comparison of anchor selection strategies for concept crasure. Random anchors are ran- domly selected from ("sky", "ground", "forest", "street"). Empty represents anchor-free operation. Ours employs heuristic selection rules for anchor assignment.
Anchox Type
Airgiane
Automabile
Bird
Cat
Average
Ace, 1 Acc, 1 Acc, 1 H. Apc,
Ace, 7 Aoc, I1,
Acc,
Aec, 1 Acc, 1 Hl.1 |Ac, I
Act,
Arc,
H,
|Asc,
Apc,
Acc,
IT,1
SD v1.4
96.06
98.92
95.08
95.75
98.95
75.91
99.72
98.51
95.45
98.93
98.60
99.05
98.63
98.63
83.64
random
15.65
96.61
12.98
8900
11.16
93.87
24.14
35.89
4.11
90.83
6.95
95.14
13.52
88.58
10.72
38.10
10.12
92.96
11.70
90.11
empty
18.79
90.52
12.76
87.25
14.71
98.36
18.95
87.64
11.4
93.6
20.11
86.09
7.37
97.97
17.96
90:38
14.09
98.84
15.93
88.17
Ours
4.37
95.32
2.66
96.09
2.77
98.00
11.54
94.60
9.72
97.83
18:49
89.37
0.49
98.56
4.63
97.78
2.71
98.00
8.71
95.33
ing 17.4% improvement. Mcanwhile, the CLIP score shows a modest increase from 10.27 to
10.91 (6.2%). Notably, the CLIP score remains well below that of the original model, indicating ef- fective erasure. These results indicate that the introduction of quality regulation contributes notably to improving overall performance.
For object erasure tasks where structural layout is established in early timesteps, we employ a heuristic anchor selection strategy. Specifically, we select anchor concepts that share high visual and structural similarity with the target (e.g., pairing "Cat" with "Dog", or "Automobile" with "Truck"), ensuring the crased trajectory transitions smoothly to a proximate natural image manifold. To systematically investigate the influence of anchor selection, compare three strategies on CIFAR-10: (1) Ours (heuristic), (2) Random (selecting generic concepts like "Sky" "Forest"), and (3) Empty (anchor-free).
Impact of Anchor choice.
Quantitative results in Table show that heuristic method achieves the best overall performance, confirming that structurally aligned anchors minimize disturbance to the generative process. When using a sub-optimal anchor (random), our method still achieves performance superior to all baselines except MACE. Although slightly lower than MACE, it is important note that MACE requires images along with SAM-generated masks as training data, whereas our method attains comparable effectiveness using only a single prompt. The Empty setting remains highly effective, significantly outperforming most baselines. This demonstrates that while our heuristic anchor enhances quality for structural concepts, the core robustness of SDErasure stems from the precise Step Selection and Quality Regulation. Further details are provided in Appendix 1.2
CONCLUSION
In this work, we investigate the denoising dynamics of diffusion models and demonstrate that con- cept erasure does not need to be applied uniformly across all timesteps. Instead, we show that selective erasure at specific timesteps can be both effective and efficient. Building on this insight, we propose SDErasure, a novel framework designed for adaptive and concept-specific erasure. First, Step Selection is proposed to pinpoint concept-specific key timesteps, while Score Rematching fur- ther improves alignment between the erased and anchor concepts. To preserve the model's gener- ative capacity, we also introduce a quality regulation consisting of the early-preserve loss and the concept-retain loss. Experimental results demonstrate that the proposed method not only achieves effective concept erasure but also excels two key aspects: enhancing related-concept quality and preserving unrelated-concept fidelity, thereby setting new state-of-the-art in concept erasure.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All datasets used in this work are publicly available. Our study is dedicated to addressing the safety concerns of generative models. To demonstrate the effectiveness of different erasure methods, the paper unavoidably involves explicit content. However, we ensure that all such content has been properly obfuscated (e.g., blurred or masked) to prevent misuse and to minimize potential ethical risks.
REPRODUCIBILITY STATEMENT
We have made extensive efforts ensure the reproducibility of our work. Detailed theoretical derivations and algorithmic pscudocode are provided in Appendix B and E The hyperparameter configurations and training setups documented in Section H For the experimental studies, de- (celebrity erasure), Sec- tion HS (artistic style erasure), and Section H.4 (NSFW concept erasure). All code required to reproduce our results is included in the supplementary material.
tailed procedures are provided Section H.T (object erasure), Section H.2
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (62425114, 62121002, U23B2028, 62472395), and the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM103)
REFERENCES
Ibtihel Amara, Ahmed Imtiaz Humayun, Ivana Kajic, Zarana Parekh, Natalic Harris, Sarah Young, Chirag Nagpal, Najoung Kim, Junfeng Hc, Cristina Nader Vasconcelos, et al. Erasebench: Un- derstanding the ripple effects of concept erasure techniques. arXiv preprint arXiv:2501.09833, 2025.
P Bedapudi. Nudenet: Neural nets for nudity classification, detection and selective censoring, 2019.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5222225 3270, 2023.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- cration via masked generative transformers. arXiv preprint W.Xiv:2301.0070. 2023.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances Neural Information Processing Systems, 35:16890-16902, 2022.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaud Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 2426-24222222 2436, 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision, pp. 5111-5120, 2024.
Daiheng Gao, Shilin Lu, Wenbo Zhou, Jiaming Chu, Jie Zhang, Mengxi Jia, Bang Zhang, Zhaoxin Fan, and Weiming Zhang. Eraseanything: Enabling concept erasure in rectified flow transformers. In Forty-second International Conference on Machine Learning. 2025.
Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. In European Conference on Computer Vision, pp. 73-88. Springer, 2024.
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho and Tim Salimans. 17Xi::2207.22598, 2022.
Classifier-fre
diffusion
guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851- 2020.
Harry H. Jiang. Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru. Ai art and its impact artists. In Proceedings of the 2023 AAAI/ACM Conference on Al, Ethics, and Sociery, AIES '23, pp. 363-33 374,, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702310. doi: 10.1145/ 3600211.360466
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https: hottps://www... www.wc..tee... wew.ntoorttoooooooo edd//
f 2009 TR. pdf Technical Report.
-kriz/learnig features-tt--eeee
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your dif- fusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pp. 2206-2217, 2023.
Feiran Li, Qianqian Xu, Shilong Bao, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. One image is worth a thousand words: A usability preservable text-image collaborative erasing frame- work. arXiv preprint arXiv:2505. 111, 2025a.
Leyang Li, Shilin Lu, Yan Ren, and Adams Wai-Kin Kong. Set you straight: Auto-steering denoising trajectories to sidestep unwanted concepts. arXiv preprint arXiv:2504. 12782, 2025b.
Ouxiang Li, Yuan Wang, Xinting Hu, Houcheng Jiang. Tao Liang, Yanbin Hao, Guojun Ma, and Fuli Feng. Speed: Scalable, precise, and efficient concept erasure for diffusion models. arXiv preprint arXiv:2503.07392 2025c.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6430-6440, 2024.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 2021.
Zheling Meng. Bo Peng, Xiaochuan Jin, Yue Jiang, Jing Dong, and Wei Wang. Dark miner: Defend against undesired generation for text-to-image diffusion models. arXiv preprint arXiv:2409. 17682, 2024.
Zheling Meng. Bo Peng, Xiaochuan Jin, Yueming Lyu, Wei Wang. Jing Dong, and Tieniu Tan. Concept corrector: Erase concepts on the fly for text-to-image diffusion models. arXiv preprint arXiv:2502.166368 2025.
Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: survey, ACM computing surveys (CSUR), 54(1):1-41,00
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PmLR, 2021.
12

---

Published as a conference paper at ICLR 2026
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 1(2):3, 2022.
Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion mod- els. Advances in Neural Information Processing Systems, 36:66377-66389, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10693 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 35:36479-36435 2022.
Patrick Schramowski, Manuel Brack, Bjöm Deiseroth, and Kristian Kersting, Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12522-22535 2023.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6048-6058, 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yuhao Sun, Lingyun Yu, Hongtao Xie, Jiaming Li, and Yongdong Zhang. Diffam: Diffusion- based adversarial makeup transfer for facial privacy protection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 245994, 2024.
Yuhao Sun, Yihua Zhang, Gaowen Liu, Hongtao Xie, and Sijia Liu. Invisible watermarks, visible gains: Steering machine unlearning with bi-level watermarking design. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2417-2423 2025.
Luisa Verdoliva. Media forensics and deepfakes: an overview. IEEE journal of selected topics in signal processing, 14(5)910-932, 2020.
Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-now Learn- ing to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1755-1764, 2024a.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept era- sure in diffusion models. Advances in neural information processing systems, 37:36748-36770 2024b.
Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate not? safety-driven unleamed diffusion models are still easy to generate unsafe images... for now. In European Conference on Computer Vision, pp. 385-403. Springer, 2024c.
13

---

Published as a conference paper at ICLR 2026
A PRELIMINARIES
A.1 DIFFUSION MODEL.
Denoising Diffusion Probabilistic Models (DDPMs) establish generative framework wherein data sample X₂ is progressively perturbed into standard Gaussian noise over 7 discrete timesteps via predefined forward diffusion process. The model is trained to learn reverse denoising trajectory that reconstructs the original data by approximating the time-reversed dynamics of this process:
p= xi) (xi po(x,,,), t), d.
(16)
The network typically predicts the mean μ4g. often parameterized via noise estimate and op- tionally the variance of. In practice, of is usually fixed as Be, and the model learns predict the noise term:
[o(x,,t) Voi
X
Be
co(xx:t)) t)).
(17)
Training is conducted by minimizing simplified form of the variational bound:
Sample Ex,, e, |le (x,, +)13
(18)
where X, Var + VI- a, This loss is derived from the Evidence Lower Bound (ELBO), a standard objective in variational inference. In the DDPM framework, the simplified loss Lsimple corresponds to optimizing reweighted version of the ELBO. The model learns to predict the noise added at each step of the forward process.
During sampling, the process begins with xy AO I) and recursively applies pe from T to 1 to generate Xy- To improve conditional fidelity, classifier-free guidance combines conditional and unconditional score estimates:
(19) where & is the guidance scale and e denotes conditioning embedding. This simple yet powerful mechanism allows trade-off between sample diversity and adherence to the conditioning signal.
ég = co (x,, t) + s(co(x,,t, t, - co(x,, f)):
A.2 SCORE FUNCTION
In score-based diffusion, the forward process is described by the SDE: dx f(x, t)dt + g(f)dw,
(20)
with marginal density (x().. The score function is defined as:
s(x, t) Vxlog (x)).
(21)
which points toward regions of higher data density at each timestep. We approximate this with a neural network sa 1) trained via denoising score matching:
(DSM(0) Ex,x,.x, (x(t))so(x,, t) Vx, pxg((((x))x))
(22)
In Diffusion Models, given Gaussian distribution NN the gradient of the log- probability density represented as:
Vx logg(x) = Vx 20 (x 11)2
1
1
(23)
recall that q (xe x1) N Vaxxo (1 (1- --a)) I) and therefore. So (x,,t) t) ~ Vx, log (x,) = Ey(xo) Vx, log 9 (x: xxo]] ca (xr,t)
= By(xx)
/1 ai
(xc:
(24)
v1 Crt
In summary, the noise predictor co(xe, t) relates to the score via:
se (x - (x),,tt)
(25)
de
This correspondence enables Diffusion models to leverage the score function implicitly through cp.
14

---

Published as a conference paper at ICLR 2026
Algorithm Step Selection Based Diffusion Classifier
Input: Image z, target concept ct2 anchor concept Ca
Parameter: Timestep set 7 {1, , 49), iteration count K, threshold A
Output: Relevant timestep set
1: Initialize T <
2: for each t T do
3: Initialize SSScores f-
4: for k 1 to K do
5: 2t - add_noise(r;,e)
6: 6t c0(xe,,, ct)
7:
Sa + ea(t,, C.
(xp(-((x-e)))
8:
S. f
9: Append St to SSScores
10: end for
11: Compute S, + mean (SSScores)
end for
13: Normalize { to [0,
for each t do
15: S; > À then
Add to T 17: end if
18:
end for
19: return T
B PSEUDOCODE OF STEP SELECTION
The pseudocode for the Step Selection algorithm is presented Algorithm At each timestep, the model predicts noise conditioned on both the target concept (to be erased) and the anchor concept (a guiding replacement concept that redirects generation). diffusion classifier employed com- pute the classification probability for each prediction, based which the Step Separability Score (SSScore) computed and normalized across timesteps. Timesteps with higher scores correspond to greater divergence between the denoising trajectories of the target and anchor concepts.
C LIMITATIONS
For object erasure tasks, SDErasure incorporates an anchor concept during training. As with previ- ous anchor-base based approaches, performance may vary slightly with different anchor choices. Addi- tional heuristics and ablations, including human-selected, random, and empty anchors, are presented in Section
D TIME COST OF SSSCORE
The Step Selection (SSScore calculation) is one-time pre-processing step, not recurring cost during the main training loop or at inference time.
The overhead of this step minimal. As detailed Section H.l for object erasure on CIFAR- 10, the entire erasure process takes only about 20 minutes a single RTX 4090, with the Step Selection algorithm consuming only less than minutes of that time. By pinpointing the few critical timesteps, we avoid fine-tuning across all timesteps, which makes our overall training process approximately 75% faster than previous srajectory-awann methods like ANT. Thus, the slight pre- computation overhead is far outweighed by the significant gain in training efficiency.
ACCELERATION STRATEGIES
Furthermore, this pre-computation step can be significantly accelerated, particularly for large-scale applications. We propose two effective strategies.
15

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of Generation Results for Different Artist Styles using Sparse Sampling for SSScore calculation. 'original" refers to sampling at every timestep. 'sparse sampling (k)" refers to sampling every & steps.
Concept
Method
CLI 4
FiDeoco
CLIP Poo0 t
Time cost
original
21.17
7.02
31.17
161.94s
Van Gogh
sparse sampling (3)
20.43
7.62
31.12
57.17s
sparse sampling (5)
18.85
8.44
31.13
34.25s
original
20.98
9.06
31.13
164.628
Picasso
sparse sampling (3)
16.77
9.98
31.08
57.55s
sparse sampling (5)
18.02
9.25
31.06
33.88s
original
20.78
8.09
31.17
164.02s
Monet
sparse sampling (3)
18.60
8.40
31.11
57.37s
sparse sampling (5)
20.22
7.87
31.12
33.79s
original
20.98
8.06
31.16
163.538
Average
sparse sampling (3)
18.60
8.67
31.10
57.36s
sparse sampling (5)
19.03
8.52
31.10
33.97s
SPARSE SAMPLING
As shown in Figure 6 the SSScore distribution across timesteps is relatively smooth and continuous. This suggests that calculating the score at every single timestep unnecessary. We can employ sparse sampling (e.g., computing scores every 3 or 5 steps) and use interpolation to estimate the full curve.
To validate this, we conducted additional experiments to explore the influence of sparse sampling on model performance for artist style erasure. The experimental results are shown in Table 7 CLIPtargeet represents the similarity between the target concept and its corresponding prompt, indi- cating the effectiveness of the erasure. FID coco and CLIPcoco measure the generative performance of the crasure model on the MSCOCO dataset.
As shown in Table 7 sparse sampling performs comparably to the original strategy (sampling every step). On average, it produces slightly stronger erasure (lower target CLIP) and marginally weaker general-purpose purpose generation (slightly higher FID). Importantly, sparse sampling at 5-step intervals reduces the pre-computation time cost by ~80 % (from 163.5s 34s), demonstrating clear prac- tical value for accelerating the process.
CONCEPT CLUSTERING FOR MULTI-CONCHE ERASURE
For large-scale multi-concept erasure, second acceleration strategy be employed. Our key observation is that structural concepts primarily emerge in the early-mid denoising stages, whereas fine-grained concepts (like faces or styles) appear in the mid-late stages.
Based on this, we can first use a Large Language Model (LLM) to cluster the concepts to be removed (e.g., into "structural" vs. "fine-grained" groups). Concepts within the same cluster then share single SSScore calculation. This dramatically reduces the pre-computation cost from O ((Nooncepts) (Nalustere). In fact, the multi-concept crasure results in Table 12 of the appendix were trained using such shared set of SSScore and optimal steps, showing that this approach achieves both high efficiency and accuracy.
E DERIVATION OF SCORE REMATCHING LOSS
In image generation, the score function serves as vector field that guides samples toward regions of high data density, thereby acting as the fundamental mechanism underlying the generative process.
The primary objective of the proposed method is to align the predicted score function of the fine- tuned model, conditioned on the target concept, with the original model's score function under an anchor concept:
80(#1, Ct 1) x- sp- (2te, Cas
(26)
16

---

Published as a conference paper at ICLR 2026
Table 8: The synonyms and anchor concepts for the object classes in the CIFAR-10 dataset.
Object Classes
Airplane
Automobile
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Aircraft
Car
Avian
Feline
Hart
Canine
Amphihian
Equine
Vessel
Lorry
Synonyms
Plane
Vehicle
Fowl
Kitty
Stag
Pooch
Anuran
Steed
Boat
Rig
Motorcar
Winged Creature
Housecat
Tadpole
Mount
Watercraft
Hauler
Anchors
Automobile
Truck
Cat
Dog
Horse
Cat
Cat
Deer
Aisplane
Ship
where 9 denotes the parameters of the fine-tuned model, 8" denotes the parameters of the original model, denotes the target concept, and Ca denotes the anchor concept.
Furthermore, the fine-tuned model is encouraged to deviate from the original model's score function under the target concept. To achieve this, negative guidance term is introduced into Equation (26
(T Ce, t) + se- (xt, Ca. t) ( Ca,
(27)
where the guidance is defined as:
Ts (2 Ct2 Ca: 1) (2t, Ct, (Ite Cay l).
(28)
Applying the relation between Eo and so in Equation 25 we obtain the following training objective for Stable Diffusion:
co( t ca- Ca, Cast)
na(ae, Cts Cas t),
(29)
with
Cas 8) = ca (1,) Eg. (Jez Ca:
(30)
F SSSCORE ANALYSIS AND VALIDATION
To demonstrate the effectiveness SSScore and its relationship with concept variation, compute the SSScore for Van Gogh Style and Picasso Style across different timesteps and vi- sualize the results as bar charts in Figure 6 Van Gogh Style typically features rich brush- strokes and rapid color transitions, with pre- dominance of fieguffryeency detail informa- tion. Correspondingly, SSScore identifies the critical timesteps for Van Gogh Style genera- tion in the middle-to-late stages of denoising, which precisely aligns with the period when figh-frequency details are synthesized. In con- trast, Picasso Style is characterized by large color blocks and a prevalence low-frequency structural information, resulting in higher SSS- core values during the early-to-middle stages of denoising. This experiment validates that SSScore can adaptively select critical timesteps for concept generation based the distinctive characteristics of different concepts, thereby demonstrating the effectiveness of our Step Se- lection mechanism.
5SScore
0.6
0.4
Van Gogh Style
70
Skage
1.0
0.8
55500t
0-4
Picasso Style
000
20
Steps
Figure 6: SSScores for Van Gogh Style and Pi- casso Style, with the anchor concept set to empty in both cases.
In Figure 7 present bar charts of SSScores for a broader range of concepts, including objects, celebrities, artistic styles, and nudity. Consistent with our earlier observations, structural concepts such objects and nudity exhibit high SSScores primarily in the early mid denoising stages, whereas fine- trae-grained concepts such as human faces and artistic styles tend to peak in the mid to late stages. These results further substantiate the effectiveness of SSScore.
17

---

Published as a conference paper at ICLR 2026
To verify that the identified timesteps indeed optimal, conduct an ablation study by evenly dividing all denoising steps into ten groups. Group corresponds to steps [45, 49] and Group 10 to steps [1, 5]. We erase the Picasso Style using each group of timesteps and evaluate both the erasure performance (CLIP) and the generation quality (FID). The results in Figure g show that Group (steps [25, 30) achieves the best erasure performance, which coincides with the peak of the SSScore, confirming that the timesteps selected by SSScore are optimal.
Cotes
535001
S35nm
50
30
10
70
Stapp
Stopa
Stage
nutornhille
Monet Style
Music
58 00
535em
50
30
90
40
10
Stope
Sieps
Staps
Figure 7: SSScores for different categories of concepts.
CLIP Daveer batur)
FID (lonee la botze)
30
20
10
Group
Step
Figure 8: Erasure performance of the Picasso Style across different timestep groups, together with the corresponding SSScore bar charts. The best performance occurs in Group 5 (timesteps 25-30), which aligns with the SSScore peak.
G GENERALIZABILITY OF STEP SELECTION
In this section, study whether the Step Selection procedure can still identify the correct critical timesteps when the total number of diffusion timesteps changes. Using Van Gogh and Picasso as examples, visualize SSScore bar charts for total timesteps of T 50 (default), T 100, and T 200; the results are shown in Figure
The plots indicate that the SSScore profile remains consistent in shape when T changes. This obser- vation suggests that SSScore identifies the relative generative phase (i.e., the effective noise level) rather than absolute timestep index. Consequently. the Step Selection strategy exhibits strong generalizability across different discretizations of the diffusion timeline.
H IMPLEMENTATION DETAILS
Hyperparameters. In practice, K is set to 20, 17 is set to 1, 31 and 3g are set to 0.1 to ensure that the model continues to perform well on the main task. The selection fine-tuned parameters is adapted to the characteristics of different erasure scenarios. Specifically, W, in the cross-attention
18

---

Published as a conference paper at ICLR 2026
Picasso
Stapp
Skgg
Vau Goggh
Stopa
Strps
Figure 9: SSScores when the total number of timesteps changes. The left column corresponds to total time step of 50, the middle column corresponds to a total time step of 100, and the right column corresponds to total time step of 200.
modules is tuned for removing concepts such celebrities and artistic styles, while the entire cross- attention modules are updated for object-level erasure. For global concepts, such as explicit content, all attention layers are selected achieve broader semantic removal.
H.1 OBJECT ERASURE DETAILS
To evaluate the generalization capability of concept crasure, this work adopts the MACE protocol by selecting three synonyms for each of the ten CIFAR- 10 classes, detailed in Table 8 For each class, a separate SDErasure model is trained using diverse prompts generated via the DeepSeck-v3 API to enhance robustness against prompt variation. A Step Selection threshold of to ensure stability across different anchor concepts. Fine-tuning is conducted on Stable Diffusion v1.4 for 200 steps with a learning rate of 2e-5. The entire training process is memory-efficient, requiring only 14GB of GPU memory 20 minutes, including around 3 minutes for Step Selection. Compared to previous trajectory-awee aware method ANT, our approach reduces training time by upproximately 75%.
0.5 employed
single RTX 4090, and completes within approximately
For each target concept, a distinct CIFAR-I0 class is designated as the anchor to evaluate the ef- fectiveness of concept alteration. The full experimental configuration is summarized in Table & To compare different erasure methods, the fine-tuned models are evaluated along three key dimensions:
Efficacy: For the erased target concept, 200 images are generated using the prompt "a
photo the (erased class)."
Generality: For the three synonym concepts of the erased target, 200 images are generated each using the prompt "a photo of the {synonymous class}," for total of 600 images.
Specificity: For the nine remaining concepts, 200 images are generated each using the prompt "a photo of the (unaltered class)." " for a total of 1,800 images.
H.2 CELEBRITY ERASURE DETAILS
The Step Selection threshold is set to X = 0.8. Fine-tuning performed in an anchor-free setting, where the anchor is omitted. The training is conducted for 300 steps with a learning rate of le-4. To evaluate the crasure effectiveness for cach celebrity identity, the performance is assessed along three dimensions:
Efficacy: 100 images are generated with the prompt "a photo of (erased celebrity)."
Specificity: 100 images generated with prompts "a photo of {neighbor celebrity)" for other celebrities and compute the LPIPS score against the original model's outputs. Moreover, FID and CLIP scores are evaluated 3k subset of MSCOCO-30k.
19

---

Published as a conference paper at ICLR 2026
Table CLIP and FID scores for erasing in- dividual celebrity identities using SDErasure. Each row reports the results for a specific celebrity. The bottom two rows summarize the average performance across all identities and the corresponding CLIP score of the original SD1.4 .4 model for comparison.
CLIP.
FIDJ
Name
CLIP)
Anne Hathaway
10.79
6.59
Andy Samberg
10.35
6.40
Adam Driver
7.96
6.66
Alec Raldwin
9.49
6.71
Amy
Lavigne
Aaron Paul
4.00
6.53
Arnll Schwarzenegger
13.15
7.14
Amanda Seyfried
11.53
6.52
Andrew Gurield
14.51
6.41
Amy
Amy
Anna Faris
13:46
6.96
Amher Heard
12.28
6.19
(a)
(b)
(c)
Adriana Lima
9.26
7.65
Aretha Frunklin
11.39
6.65
Amy
Anjelica Huston
Figure 10: Visualization of Erasure with
Angelina Jolie
16.72
6.67
Anna Kendrick
7.52
6.43
Generic Anchors. (a) Erasing "airplane" to
"sky". (b) Erasing "automobile" "forest", (c)
Average
CLI 10.91
6,65
Erasing "bird" to "street".
SDL4
CLIP: 28.56
FID:
H.3 ARTIST STYLE ERASURE DETAILS
All training hyperparameters are the same in Celebrity Erasure. The erasure performance is assessed along two dimensions:
Efficacy: 100 images are generated with the prompt "a painting in the style of ferased style}."
Specificity: FID and CLIP scores are evaluated on a 3k subset MSCOCO-30K.
H.4 EXPLICIT CONCEPT ERASURE DETAILS
The Step Selection threshold is set to À 0.8. The anchor is set to "a person wearing clothes". The training runs for 200 steps with a learning rate of 2e-5. The sensitive-content erasure performance assessed along dimensions:
Efficacy: NudeNet is applied to count sensitive outputs on the i2p dataset.
Specificity: FID and CLIP scores are evaluated on a 3k subset MSCOCO-30K.
1 CHOICES OF ANCHOR CONCEPT
Overall, for the erasure of celebrities and artistic styles, we recommend employing an anchor training approach by setting the anchor concept to empty. This is because for such fine-grained semantic details, SSScore tends to select timesteps in the middle-to-late stages of the diffusion pro- cess. During these stages, fine-tuning has minimal impact on the overall image structure, allowing effective erasure to be achieved without the need for anchoring. In contrast, for structural concepts such objects, SSScore typically identifies timesteps in the early-to-middie stages, where employ- ing anchor helps ensure that the generated images remain within the natural image manifold. We provide detailed discussions on the anchor selection rationale the following Sections I.1 and
I.1 CELEBRITY AND STYLE ERASURE
By default, setting the anchor concept to empty prompt is sufficient for effective concept erasure, while preserving semantically unrelated attributes such as structure and pose. However, when finer control over the generation process is desired, specific anchor concept be provided steer the
20

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of anchor selection strategies for concept erasure. Results of the remaining six classes are reported. Empty represents anchor-free operation. Ours employs heuristic selection rules for anchor assignment.
Auchoc Type
Dear
Frog
Hoss
Ship
Truck
Adty
Asa,
Ass,
R.
Ass,
Asc,
Anyg
N.T
Aoa,
Ass,
Ass, 4 H.
Ass:
Augg
Ass,
Aos,
Ast,
H.
Ass,
Asc.
R,
70.12
99.74
98.62
99:53
99.78
95.00
nasion
15.49
92.
11.19
87.45
9:35
91.01
11.56
89.08
11.32
9687
15.18
89-21
1.9
$5.00
0.58
90.14
6.75
98.2
12.19
91.38
2.64
96.18
2.30
97.6
mmpy
11:11
90
3.8
8.99
12.04
12.14
19.71
31.36
Table 11: Evaluation of Category Erasure on CIFAR-10. Results for the remaining six cate- gories reported. Acc, represents the accuracy for the erased class. Acc, represents the accuracy for unaltered classes. Acc, represents the accuracy for synonyms of the erased class. Higher Ho indicates better overall concept erasure performance. All values expressed percentages (%) The original model's performance is included for reference.
Ship
Trnck
Method
Acc,
Acc, AAc, L. Ho.1 Acc, L.A Aog H.T LAD. A00 Acc, LoA,, Aec, il, LAv (.000 5%
Acc, Aoc, Aoc,
FMN
9895
94.13 60.24 3.04 97.64 98.12 96.95
91.60 94.59
97.97
96.75
95.37
UCF
1188
8.94 92.94 19.22 98
89.00
85.53
12.70
6.13
50.00
70.13
ESD
18.14
73811
693
82.17 27.03 89.75 28.52
77.24
12:32
88.05
7.62 89.32
17.69
82,23
64.73
18.28 94.32
15.95
86.33
25.11
65.35
21.47 78.98
MACE
13,47
97.71 6.88 92.48 11,07 96.77 10.56
91.47 11.45
97.25
1308
90.83
4.89
97.48 7.85 94.86 8.58 98.56 14.40 91.56
7.29
99.38
9.38 93.79
RFCR
9.62
98.27
7.51 93:60 19.72 08.5 36.78 83.55 11.69 08.43
10.2
91.96
9.98
98.10
10
14.61
90087 21 92. 98.41 25.61 $0.90
90.42,
98.43
40.00 72.48
ANT
3,61
98.46
2.5 97.44 17.22 98.56 20.12 96.34
23.99 98.42 10.8
86.39
12.93
98.47
11.05 91.02 13.74 98.35 21.39 $7.00
22.04
98.47
21.93 8382
Ours
98.29
97.27
1.03 98.17
91.06
7.30
5,26
SD
9987 98.49
70.02
98.74 98
98.25
99.03 98.49
92.04
99.78
98.50
45.74
99.64 98.63 63.16
98.89
98.60
95.00
resulting image. For example, when erasing the concept "Taylor Swift", using an anchor such as "a young woman" helps guide the model toward more targeted and semantically aligned replacements.
1.2 OBJECT ERASURE
For concept crasure involving specific object categories, we recommend selecting anchor concepts that share similar visual characteristics with the target. For example, "cat" and "dog". or "deer" and "horse" exhibit comparable appearances and thus of structural object categories, such as "automobile", an anchor concept like "truck" can provide a structurally coherent reference. However, some concepts may lack suitable anchors with the afore- mentioned characteristics. In such cases, generic concepts such as "sky", "forest" or "street" can be used anchors provide broad semantic guidance shown in Figure 10
as effective anchor pairs. In the
1.3 QUANTITATIVE RESULTS ON ANCHOR SELECTION
Table TO presents quantitative results comparing random selection of generic concepts as anchors (random), anchor-free operation (empty), and heuristic anchor selection (ours). Random anchor selection achieves effective erasure performance, but the semantic distance between generic and target concepts adversely affects generation quality, as evidenced by lower Acc, of 92.96 compared to 95.84 and 98.00 for the other two configurations. Setting the anchor to empty also demonstrates satisfactory erasure capabilities, though inferior to heuristic selection. In summary, selection strategy achieves optimal crasure performance, while both anchor- free and random anchor approaches demonstrate competitive advantages over existing methods.
proposed
21

---

Published as a conference paper at ICLR 2026
CLIP similarity scores for 20 con- cepts after multi-concept erasure using single SDErasure model. The bottom two rows report the average performance and the baseline CLIP score the original SD1.4 model.
Table 12:
Table 13: Detailed correspondence between era- sure categories, concepts, and their visualiza- tions.
Category
Concept
Figure Index
Nudity Erasure
Nudity
Figure
Concept
CLIP
Concept
CLIP
Celebrity Erasure
Elon Musk
Figure
12
Aaron Paul
19.28
Andrew Garfield
Donald Trump
Figure
Adam Driver
17.01
Andy Samberg
20.01
Taylor Swift
Figure
Adriana Lima
17.79
Angelina Jolie
17.28
Object Erasure
Airplane
Figure
13
Alcc Baldwin
15.49
Anjelica Huston
Automobile
Figure
Amanda Seyfried
15.18
Anna Faris
21.16
Bird
Figure
Amber Heard
14,88
Anna Kendrick
16.25
Cat
Figure
16
Amy Adams
15.77
Hathaway
Deer
Figure
17
Amy Pochler
17.61
Aretha Franklin
18.28
Dog
Figure
18
Amy Schumer
16.14
Arnold Schwarzenegger
19.32
Frog
Figure
19
Amy Winehouse
19.11
Avril Lavigne
Horse
Figure
Ship
Figure
Average
CLIP: 17.46
FID: 8.82
Truck
Figure
CLIIP:
FID:
Table 14: Comparison of concept crasure methods on multi-concept crasure task. The CLIP score of the original model is provided as reference.
Method A Average CLIP
FID 4
UCE
23.24
16.87
MACE
21.35
14.90
ANT
21.74
14.18
Ours
17.46
8.82
Original
28.56
J ADDITIONAL EXPERIMENTS
J.1 ADDITIONAL CELEBRITY ERASURE
To evaluate the generalizability of the proposed method, additional experiments are conducted on broader of celebrity identities. As shown in Table 9 CLIP scores consistently decrease after erasure, indicating effective concept removal, while FID scores remain low (around 6.5), suggesting preserved image quality. These results demonstrate that the method scales well diverse concepts.
J.2 ERASING THE CIFAR- 10 CLASSES
Table II presents the complete results of erasing the remaining six object categories in the CIFAR- 10 dataset. SDErasure consistently achieves the highest harmonic mean, attributed to its concept- specific training strategy that adaptively shifts the denoising trajectories toward anchor concepts at critical timesteps.
J.3 MULTI-CONCEPT ERASURE
To assess the model's capacity for multi-concept erasure, 20 concepts are removed simultaneously within single model. Table [2] presents the CLIP scores for each concept and the overall FID after erasure. Table [4]peeentss the performance of different methods on multi-concept erasure tasks under the same experimental setup. While CLIP similarities increase compared to single-concept settings, they remain notably lower than those of the original model. Additionally, although erasing 20 concepts simultancously, the FID score only modestly increases (from 6.65 to 8.82), and still outperforms previous methods as shown in Table [4] These results highlight the effectiveness of SDErasure in multi-concept erasure scenarios. This advantage stems from the design of SDErasure,
22

---

Published as a conference paper at ICLR 2026
SD1.4
FMN
ESD
UCE
MACE
RECE
SPEED
ANT
Ours
Masks added by authors publication
Figure 11: Qualitative results on nudity erasure. The images on the same row are generated using the same random seed.
which focuses fine-tuning low-noise timesteps when erasing celebrity concepts. Such strategy mitigates interference among multiple concepts during simultaneous erasure, thereby maintaining erasure effectiveness while substantially improving the model's generative performance.
J.4 ADDITIONAL QUALITATIVE RESULTS
In this section, we provide comprehensive qualitative results across a variety of concept erasure scenarios. Table 13 summarizes the mapping between each experiment and its corresponding visu- alization results.
K
USE LARGE LANGUAGE MODELS
The authors acknowledge the use of large language models (LLMs) in the preparation of this manuscript. Specifically, LLMs were employed for language editing and writing refinement pur- poses, including but not limited to grammar correction, sentence restructuring, and stylistic im- provements to enhance the clarity and readability of the text. All technical content, experimental design, data analysis, results interpretation, and scientific conclusions presented in this work entirely the authors' own contributions. The use of LLMs was limited to linguistic assistance and did not involve the generation of research ideas, methodological approaches, or substantive content. All factual claims and technical details have been independently verified by the authors.
23

---

Published as a conference paper at ICLR 2026
501.4
FMN
ESD
UCE
MACE
RECE
SPEED
ANT
Ours
Target concept: "Euo"
pbl
pbl
Irrelevant concept: "Thylor Swift" and "Donald Trump"
SD1.4
FMN
ESD
UCE
MACE
RECE
SPEED
ANT
Ours
Target concept: DDomaldl Trump"
concept: "Flon Musk" and "Taylor Switt"
SD1.4
FMN
ESD
UCE
MACE
RECE
SPEED
ANT
Ours
Target concept: "Taylor-Swipp Swift"
Irrelevant concept: "Elon Musk" and "Donald Trump"
Figure 12: Qualitative results celebrity erasure. The images the same generated using the same random seed.
24

---

Published as a conference paper at ICLR 2026
SDI.4
ANT
Target concept: "a photo airplane"
Synonymous concept: photo of the aireraf""
Inclevant concopt: "a photo of the bird"
Figure 13: Qualitative results on airplane erasure. The images on the same row are generated using the same random seed.
25

---

Published as a conference paper at ICLR 2026
SD1.4
FMN
ANT
Target concept: "a photo the bird"
Synonymous concept: photo the aviun"
Inrelevant concept: "a photo of the airplane"
Figure 14: Qualitative results on automobile erasure. The images on the same row are generated using the same random seed.
26

---

Published as a conference paper at ICLR 2026
SD1.4
FMN
UCE
ESD
MACE
RECE
ANT
Ours
Target concept: photo bird"
H
Synonymous concept: photo of the avian'
Irrelevant concept: 'a photo of the airplane"
Figure 15: Qualitative results on bird erasure. The images on the same row are generated using the
same random seed.
27

---

Published as a conference paper at ICLR 2026
SD1.4
FMN
RECE
ANT
Target concept: photo cat"
Synonymeus concept: photo of the feline"
Inrelevant concept: "a photo of the airplane"
Figure 16: Qualitative results on cat erasure. The images the same row are generated using the same random seed.
28

---

Published as a conference paper at ICLR 2026
SD1.4
ANT
Target concept: "a photo deer"
Synonymous concept: photo of the doe"
Inrlleant concept: "a photo of the airplane"
Figure 17: Qualitative results on deer erasure. The images on the same row generated using the
same random seed.
29

---

Published as a conference paper at ICLR 2026
SD1.4
FMN
RECE
ANT
Target concept: "a photo of the dog"
Synonymous concept: "a photo of the canine"
Inrelevant concept: "a photo of the airplane"
Figure 18: Qualitative results on dog erasure. The images on the same row are generated using the
same random seed.
30

---

Published as a conference paper at ICLR 2026
SDI.4
FMN
UCE
ESD
MACE
RECE
ANT
Ours
Turget concept: "a photo of the frog"
Synonymous concupt: "a photo of the amphibian"
Inrelevant concept: "a photo airplane"
Figure 19: Qualitative results on frog erasure, The images on the same row are generated using the same random seed.
31

---

Published as a conference paper at ICLR 2026
SD1.4
ANT
Turget concept: "a photo of the horse"
Synonymous concept: "a photo of the equine"
Inrelevant concept: "a photo of the airplane"
Figure 20: Qualitative results on horse erasure. The images on the same row are generated using the same random seed.
32

---

Published as a conference paper at ICLR 2026
SD1.4
ANT
Target concept: "a photo of the ship"
Synonymous concept: "n photo of the bout"
Inrelevant concept: "a photo of the airplane"
Figure 21: Qualitative results on ship erasure. The images on the same row generated using the same random seed.
33

---

Published as a conference paper at ICLR 2026
ANT
Turget concept: photo truck"
LIR VK
Synonymous concept: "u photo of the hauler"
Inrelevatt concept: "a photo of the airplane"
Figure 22: Qualitative results on truck erasure. The images the same row are generated using the same random seed.
34


Reviews:
[{"review_id": "bw8j63OJ6V", "reviewer": "Reviewer_DjSd", "summary": "This paper introduces SDErasure, a concept erasure method for diffusion models that adaptively targets concept-specific timesteps based on a diffusion classifier. It proposes a Step Separability Score (SSScore) to select critical steps, and introduces Score Rematching and Quality Regulation to balance erasure efficacy and generation quality. The method outperforms prior work across multiple erasure tasks.", "questions": "1. Since the method evaluates SSScore at every single timestep using multiple forward passes, this may become a bottleneck for large-scale or multi-concept erasure tasks. Can this process be approximated or accelerated further?\n2. How robust is the method when anchors are suboptimal, or when no clear anchor exists? Can the method adaptively learn anchors instead of relying on heuristic rules? \n\nI expect the authors to response to these concerns. I will raise my rating if they are well-addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is built on clear observations that different types of concepts emerge at different stages of the denoising process, and addresses this with a principled, targeted erasure strategy.\n2. The proposed SSScore + Step Selection mechanism allows the model to automatically identify key timesteps for each concept, improving erasure precision without requiring heuristic or manual selection. Specifically, I praise the usage of the diffusion classifier to guide the training process.\n3. The experiments are extensive, covering multiple concept types, metrics (efficacy, specificity, generality), and include ablations, multi-concept erasure, and comparisons with strong baselines.", "weaknesses": "1. The SSScore must be computed at every timestep via multiple forward passes using a diffusion classifier. Although this is only done once per concept, the cost is still high, especially when scaling to many concepts or large batches.\n\n2. Although anchor-free erasure is supported, in many cases selecting a suitable anchor is important for quality preservation. The method relies on heuristic selection rules, and the anchor choice significantly affects performance. However, there seems no effective anchor-selection method now (though mentioned in the limitations, but I believe this is really important in this method).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hi7C0KdyVH", "reviewer": "Reviewer_tk2R", "summary": "This paper proposes a fine-grained loss for concept erasure. It targets key timesteps critical for each concept, which are automatically identified using a diffusion classifier. By incorporating constraints to preserve untargeted concepts, the method achieves effective concept erasure in experiments. The authors evaluate the approach against multiple baselines.", "questions": "1. Comparison with other methods in multi-concept settings.\n2. Analysis of error predictions in step selection.\n3. Assessment of computational efficiency.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The fine-grained loss across different timesteps for each concept is noteworthy, and the corresponding early-preserve loss mitigates the instability issues of fine-tuning.\n2. The selection of critical timesteps demonstrates potential for effective concept erasure while preserving untargeted concepts.\n3. The authors compare their method with recent attacks, such as SPEED and ANT, showing that it achieves superior performance.", "weaknesses": "1. Appendix G.3 presents the performance of the proposed method in multi-concept settings. However, comparisons with other methods in these settings should be included to better demonstrate the scalability of the approach.\n\n2. Does the step selection lead to error prediction? This could help determine whether all concepts follow the same principles during step selection.\n\n3. The SSScore is calculated at each timestep. How does this impact computational efficiency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5W8F9vPbII", "reviewer": "Reviewer_drbX", "summary": "This paper presents SDErasure, a new framework for concept-specific erasure in text-to-image (T2I) diffusion models. Rather than modifying the full model, SDErasure focuses on selectively fine-tuning critical denoising timesteps, which are identified using a Step Selection algorithm. This approach allows the model to effectively erase target concepts with minimal disruption to unrelated content. By introducing two key technique: Score Rematching Loss and Quality Regulation, the method achieves a strong balance between effective concept erasure and preserving generative quality. Experimental results show that SDErasure outperforms existing methods in various scenarios, especially when erasing semantically meaningful or fine-grained concepts.", "questions": "1. How sensitive is the method to the choice of threshold in the Step Selection algorithm? Does this threshold need to be manually tuned for each concept, or can it be set universally? Is there any theoretical or empirical basis for choosing an optimal value?\n\n2. Structural concepts (e.g., human pose or object layout) are often deeply embedded in the generation process. Could the Quality Regulation mechanism interfere with the goal of erasing such structural concepts, especially if they strongly influence generation quality?\n\n3. For multi-concept erasure, if both structural and fine-grained concepts need to be erased simultaneously, will the method still work effectively?\n\n4. SDErasure effectively preserves the generative quality of unrelated concepts while removing the target concept. Can you provide a more detailed analysis of how Quality Regulation and Score Rematching loss impact generative quality, respectively?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper makes a key observation: only a subset of denoising timesteps are critical for concept representation in diffusion models. By fine-tuning these specific steps instead of the whole model, the method is both efficient and targeted, reducing unnecessary side effects.\n2. Quality Regulation is an important contribution. It explicitly addresses the trade-off between removing unwanted concepts and maintaining high-quality generations. This helps prevent issues like over-erasure or degradation of unrelated content.\n3. The use of Score Rematching Loss aligns the denoising trajectory after erasure with the expected distribution.", "weaknesses": "1. The method’s effectiveness appears to be sensitive to anchor concept selection. This dependence might limit the method's generalizability across concepts that lack well-defined or semantically close anchors. A more robust or anchor-independent strategy would improve usability.\n\n2. While the Step Selection algorithm is central to the method, the paper does not provide a theoretical justification for why the selected timesteps are optimal or what properties make certain steps more important. This makes it harder to understand or predict the method's behavior.\n\n3. Although the method performs well on many tasks, performance gains are inconsistent. In some cases, improvements are small or even absent. The paper could benefit from a more in-depth analysis of failure cases, including explanations of when and why the method struggles.\n\n4. Several related works on concept erasure and safety in T2I models are not cited or discussed. Notably missing are:\n\n[1] Eraseanything: Enabling concept erasure in rectified flow transformers\n\n[2] Dark miner: Defend against unsafe generation for text-to-image diffusion models\n\n[3] One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Framework\n\n[4] Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Concepts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50Tb0hsyTQ", "reviewer": "Reviewer_DdAx", "summary": "This paper observes that different concepts show distinct generative processes, with critical timesteps varying across concepts. Building on this insight, the paper introduces SDErasure, a novel concept erasure framework for text-to-image diffusion models. First, it proposes a step selection strategy to identify the critical timesteps most relevant to the target concept. It then introduces a score rematching loss to steer the generative process away from the target concept and toward an anchor concept.\nTo preserve the model’s generative capability, two additional regularization losses are proposed: the early-preserve loss, which enforces consistent noise predictions for target concepts at early steps, and the concept-retain loss, which maintains prediction consistency for non-target concepts after erasure.\nComparisons with prior concept erasure methods across four different domains (objects, celebrities, artistic styles, and sensitive content) demonstrate the effectiveness of SDErasure.", "questions": "See the weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of adaptively selecting critical timesteps for concept erasure is novel and well-motivated.\n- The proposed framework is efficient and effective, achieving strong performance in erasing target concepts while preserving non-target ones in all settings.\n- The paper is well-written and easy to follow.", "weaknesses": "- The paper lacks an ablation study on the timestep selection strategy. It is unclear how the identified timesteps are verified to be optimal. For example, in Figure 6, the critical timesteps for erasing the Van-Gogh-style concept are found in the mid-to-late stages, but the truly optimal timesteps might be in the early-to-middle stages.\n- The generalizability of the step selection strategy remains uncertain. If the total number of timesteps changes (e.g., from 50 to 200), can the same strategy still identify the correct critical timesteps?\n- Since SSScore plays a crucial role in SDErasure, it would be valuable to include additional analyses similar to Figure 6. For example, what are the SSScore distributions for different concept categories, such as objects and celebrities? Do they follow consistent patterns, e.g., critical timesteps for objects appearing in mid stages and for celebrities in mid-to-late stages?\n- In the multi-concept erasure experiments (Table 10), several established methods, such as UCE, MACE, and RECE, are omitted. Including these baselines would provide a more complete evaluation.\n- (minor) In Appendix E, what do the hyper-parameters $\\alpha$ and $\\beta$ represent? Are they typos for $\\beta_1$​ and $\\beta_2$​ in Equation (14)?\n- (minor) What values are used for the hyper-parameter $\\eta$ in Equation (11) and for the iteration count $K$ in the pseudocode?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)