Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models
Authors: Lexiang Xiong
Liu Chengyu
Jingwen Ye
YAN LIU
Yuecong Xu
Venue: NeurIPS 2025 poster ()
Keywords: Concept Erasure
Text-to-Image Diffusion Model
Safe generation
Text-to-image generation
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Our paper introduces Semantic Surgery, a novel training-free method that dynamically manipulates text embeddings before diffusion to achieve concept erasure in text-to-image models while preserving image quality and locality.

Abstract: With the growing power of text-to-image diffusion models, their potential to generate harmful or biased content has become a pressing concern, motivating the development of concept erasure techniques. Existing approaches, whether relying on retraining or not, frequently compromise the generative capabilities of the target model in achieving concept erasure.

Here, we introduce **Semantic Surgery**, a novel training-free framework for zero-shot concept erasure. Semantic Surgery directly operates on text embeddings *before* the diffusion process, aiming to neutralize undesired concepts at their semantic origin with dynamism to enhance both erasure completeness and the locality of generation. Specifically, Semantic Surgery dynamically estimates the presence of target concepts in an input prompt, based on which it performs a calibrated, scaled vector subtraction to neutralize their influence at the source. The overall framework consists of a Co-Occurrence Encoding module for robust multi-concept erasure and a visual feedback loop to address latent concept persistence, thereby reinforcing erasure throughout the subsequent denoising process.

Our proposed Semantic Surgery requires no model retraining and adapts dynamically to the specific concepts and their intensity detected in each input prompt, ensuring precise and context-aware interventions. Extensive experiments are conducted on object, explicit content, artistic style, and multi-celebrity erasure tasks, demonstrating that our method significantly outperforms state-of-the-art approaches. That is, our proposed concept erasure framework achieves superior completeness and robustness while preserving locality and general image quality (e.g., achieving a 93.58 H-score in object erasure, reducing explicit content to just 1 instance with a 12.2 FID, and attaining an 8.09 H_a in style erasure with no MS-COCO FID/CLIP degradation). Crucially, this robustness enables our framework to function as a built-in threat detection system by monitoring concept presence scores, offering a highly effective and practical solution for safer text-to-image generation.

Our code is publicly available at: https://github.com/Lexiang-Xiong/Semantic-Surgery

Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models
Lexiang Xiong 1. Chengyu Liu' Jingwen Ye! (e15201350u. mus edu, e15543400u. nus edu, jingvenyOnus edu edu.sg) liuyan@scu. edu. en, yc xucnus edu.sg
Yan Liu? Yuecong Xu'
National University of Singapore
Sichuan University
Abstract
With the growing power of text-to-image diffusion models, their potential gen- erate harmful or biased content has become a pressing concern, motivating the development of concept erasure techniques. Existing approaches, whether rely- ing on retraining or not, frequently compromise the generative capabilities the target model in achieving concept erasure. Here, introduce Semantic Surgery, a novel training-free framework for zero-shot concept erasure. Semantic Surgery directly operates on text embeddings before the diffusion process, aim- ing to neutralize undesired concepts at their semantic origin with dynamism to enhance both erasure completeness and the locality of generation. Specifically, Semantic Surgery dynamically estimates the presence of target concepts in an input prompt, based on which it performs a calibrated, scaled vector subtraction to neutralize their influence at the source. The overall framework consists of a To-Occurrence Encoding module for robust multi-concept erasure and visual feedback loop to address latent concept persistence, thereby reinforcing erasure throughout the subsequent denoising process. Our proposed Semantic Surgery requires no model retraining and adapts dynamically to the specific concepts and their intensity detected in each input prompt, ensuring precise and context-aware interventions. Extensive experiments are conducted object, explicit content, artistic style, and multi-celebrity erasure tasks, demonstrating that our method sig- nificantly outperforms state-of-the-eee of-the-art approaches. That is, our proposed concept erasure framework achieves superior completeness and robustness while preserv- ing locality and general image quality(e.g., achieving 93.58 H-score in object erasure, reducing explicit content to just instance with a 12.2 FID, and attaining 8.09 H. in style erasure with MS-COCO FID/CLIP degradation). Crucially, this robustness enables our framework to function as a built-in threat detection system by monitoring concept presence scores, offering highly effective and practical solution for safer text-to-image generation. Our code is publicly available at hhttss://github. http:///ggttuuub..mm//eeeeee ang- Xiong//emanticc c-rugeryy
Introduction
In recent years, Text-to-Image (T2I) diffusion models [16 17
offer remarkable image generation capabilities but also risk producing harmful or infringing content (e.g.. explicit mate-
33
57
rial, copyrighted styles) [43 47 46 48). Initial mitigations like dataset filtering [3] post-hoc
checkers [ are often costly or offer limited protection 43 53.
primary challenge in concept erasure is achieving both high completeness (thorough removal of target concepts) and locality (minimal impact on unrelated content). Parameter-Modifyyn
58 56) which modify model parameters to "unlearn" concepts, often excel in demonstrating erasure potential but inherently struggle with this trade-off [19 27 43). Effective unlearning frequently leads to catastrophic forgetting, degrading general capabilities. Such modifications are also computationally expensive per new concept and establish pre-defined static 39th Conference on Neural Information Processing Systems (NcurIPS 2025).
Methods [O] [ 21 28

---

defenses. Their static, passive defenses struggle with concept variants beyond training/editing samples, hindering full completeness. While some approaches incorporate adversarial training techniques to bolster robustness against such variations [19 58 this typically increases computa- tional demands without fully overcoming the inherent inflexiblitty of static defenses or the risk of permanently altering the model's 's versatility, especially when dealing with cumulative interference
from multiple concepts (27).
This motivates the exploration of solutions that operate at inference time without altering the base model, often termed inference-time methods. While early guidance techniques 43 lack precision[I9] [51] more recent inference- time approaches have emerged. For instance, some methods operate by projecting selected token embeddings [54] or attention values [5]] during the diffusion process. While offering flexibility and preserving the original model, these often intervene at token-specific level or mid-diffusion. Yet, these methods might prove inadequate, as self-attention in text encoder can spread the target concept's s semantics across the entire token sequence, enabling its reconstruction from unrelated tokens' residual information (27). The core challenge thus remains: how to achieve robust completeness and locality but with the adaptability and preee-pressrratiion benefits of a zero-shot, inference-time strategy that addresses concepts at a more fundamental, global semantic level.
We contend that the key lies in a giobally-aware, pre-diffusion intervention directly on the text embedding using principled vector arithmetic. Our method, Semantic Surgery, operates as a zero- shot, inference-timg framework. It leverages the linear structure of language embeddings [29] (30], inspired by "activation engineering" in LLMs [4 49) to dynamically assess the presence of target concepts within the global prompt semantics and, based on this assessment, selectively neutralizes their influence on the entire text embedding before it guides the diffusion process. This targeted, pre-diffusion modification of the global semantic input aims to directly enhance erasure completeness and preserve locality, offering a flexible alternative to the static and often costly parameter-modifyinn methods.
Furthermore, Semantic Surgery incorporates Co-Occurrence Encoding to systematically manage the complex interactions during multi-concep erasure (Eq. 7)) scenario particularly problematic for methods relying on cumulative parameter modifications. We also address Latent Concept Persistence (LCP), where U-Net priors cause concept resurgence, via optional Visual Feedback Adjustment (Eq. [8] that refines the textual embedding. Our contributions are:
Novel Global -nneSciie-Spacce Erasure via Semantic Arithmetic: We propose Semantic Surgery, a zero-shot, inference-time method that uniquely applies calibrated vector subtraction to the entirety of the text embedding. This offers direct and adaptable approach to neutralize concepts at their semantic source, aiming to overcome the completeness-localiy trade-offs and static limitations prevalent in methods that modify model parameters.
Principled Solutions to Advanced Erasure Scenarios: Our Co-Oecurrence Encoding provides a structured approach to robust multi-concept erasure, and our textual-refinemmnnt solution for LCP addresses a key challenge in achieving comprehensive visual safety, advancing beyond simpler intervention strategies.
Demonstrating Inference-Tinee Efficacy Against Strong Parameter-Modiffinn Baselines: Extensive experiments show Semantic Surgery achieves highly competitive, and in several aspects superior, performance against robust parameter-modifyy methods in erasure completeness, locality, and image quality, highlighting the potential of sophisticated inference-time techniques offer practical and effective solutions.
2 Related Work
Concept Erasure in Diffusion Models Controlling unwanted concepts in T2I models is critical for safety and alignment, aiming to improve erasure completeness and generation locality. Early approaches like dataset filtering 33] are prohibitively expensive, while post-hoe image checkers [6]
24 (3) are often easily circumvented [43] [50] 53]. A major line of work involves modify ing model parameters, encompassing retraining B. fine-tuning [10, [3 19 23 27). and direct model editing [3]. While potentially effective, these methods inherently struggle with the completeness- locality trade-off, often causing catastrophic forgetting [10 27). They typically establish static defenses requiring costly updates for new concepts and may permanently alter model capabilities.
lie., the complete sequence of token embeddings representing the entire input prompt from the text encoder.

---

Some improve prior preservation via regularization [33 19 27 or adversarial training [9 58]. but fundamental limitations in adaptability and locality often persist. Alternativly, inference-time methods operate without altering the base model, offering flexibility. Basic guidance techniques 4
often lack precision [19] 51. More recent methods manipulate internal representations during diffusion, for instance, by projecting selected token embeddings [54] or attention values [5]] token- wisely. However, these localized, mid-diffusion interventions face challenges. Firstly, token-wise modifications may be insufficient as self-attention can diffuse the target concept's semanties across the entire embedding sequence, allowing reconstruction from residual information in unrelated tokens [27. Secondly, these approaches typically do not explicitly address the potential for concepts to resurface due to model priors, a phenomenon we term Latent Concept Persistence (LCP). Our work, Semantic Surgery. introduces distinct inference- timen approach operating globally the initial text embedding to address these limitations.
Semantic Geometry for Concept Control The principle that vector arithmetic can manipulate semantic meaning, famously demonstrated by word2vec analogies [2]]30]
30] (Gxing Cman Cwoman R Equeen), suggests the potential for algebraic control over concepts embedded in vector spaces. This concept has been powerfully exploited in Large Language Models (LI.Ms) through "activation engineering." where adding subtracting specific vectors derived from activations can causally steer model behavior or induce specific functionalities [26] 49). Similar ideas have been explored in the context of Text-to-Image Image (T2I) models, such manipulating image embeddings algebraically within CLIP space [39] or proposals involving noise manipulation for concept control [52]. Our work, Semantic Surgery, builds upon these foundations but distinctively applies the principle of semantic vector arithmetic directly to the initial text embedding for the specific task targeted concept erasure with dynamic concept identification. We leverage the geometric properties of the text embedding space to perform a calibrated subtraction, aiming to neutralize unwanted concepts at their semantic source before the diffusion process begins. We further integrate visual feedback mechanism into the concept el elimination process to address the Latent Concept Persistence (I.CP) problem.
3 Method
Let a text-to-image diffusion model be defined as a generative process Go : P + I, where P is the prompt space and Z is the image space. The model first encodes an input prompt p E P into semantic embedding e = (p)) E IR via a text encoder Φ(-). then generates an image I ~ po through iterative denoising of latent variables {=)]-
Problem Formulation. We define concept as a distinct semantic factor of variation (e.g., object, style, attribute) [52] influencing the generated image I. Let Concepts(T) /) C 24 denote the subset of all possible concepts present in I, where UI is the universal concept set. Given an input embedding = é(p) and target set of concepts Cerase C l1 designated for removal, our objective is to design
an embedding surgery operator T R >> TR This operator produces a modified embedding = T(e).
The core challenge lies in designing T to simultaneously satisfy two potentially competing desiderata concerning the statistical properties of images (o((le") generated from the modified embedding:
Completeness: The operator must effectively remove the target concepts Cerase. Formally, the expected presence of Cerase in generated images should be bounded by a safety threshold Gafe: (1)
E(~pe(Tler) [I(Cense C Concepts S Gsafe-
Locality (Fidelity): The modification should minimally affect non-target concepts & Cerase- The change in their expected presence probability, compared to generation from the original embedding should be limited by tolerance e
Ve & Ceusee
[l(c E Concepts(!))
[[])) e Concepts(())]] S Gpl. (2)
The operator must be stable against minor prompt variations (e.g.. paraphrasing). Formally. T must be locally Lipschitz continuous. There must exist a constant L > such that for any embedding e and a sufficiently small perturbation de: [ + de) T(e) s L|
Robustness:
Here, I[-) is the indicator function. Achieving high completeness (low éssee) while maintaining high locality (low em) represents the central trade-off addressed in this work. The parameters éxafe E [0. 1] and Eaot E [o, 1] quantify the target performance levels, whose attainment by our proposed method evaluated empirically.
3

---

Text Encoder (CLIP)
(B) Core Surgery & Generation
Prompi
Einput
Diffusion Generator
Initial Image
(A) Semantic Analysis
LCP Check
i Semantic Biopsy
Oke CCspui )c.)
Pi
((ae 31/r)
Feedback Signal
① Visual Feed- back (LCP) Vision Detector
Final Image
Detected Con- cepts & Scores
Caciive
(c A 7
T Co-Occurrence Encoding
Pas
D,,ECaa Pru
d(pa>)
Cn
Figure 1: Overview of the Semantic Surgery Workflow. The process begins with (A) Semantic Analysis: Semantic Biopsy analyzes the initial embedding Cupuu. to produce concept presence scores }. These scores determine the set of active concepts, Cactive- @ Co- Co-Occurrence Encoding then takes this set to form a unified removal direction Seco This leads to (B) Core Surgery: the scaled direction vector is subtracted from Cinput to produce sanitized embedding é,- which is passed to the Diffusion Generator (composed of a U-Net and VAE decoder) to generate an initial image. For critical safety tasks, optional 4 Visual Feedback Loop (dashed red path) uses vision detector to check for Latent Concept Persistence (LCP). If a concept is visually detected, feedback signal updates the set of active concepts, triggering refined, stronger surgery that leads to the final, clean image. (Note: The surgery step itself is implicitly step @). This entire framework supported by rigorous theoretical analysis (see Appendix for proofs), where formally prove guarantees for Completeness (Thm. 4). Locality (Thm. 5 and Robustness (Thm. 6
3.1 Semantic Modeling in Text Embedding Space
The semantic manipulation capability of our method stems from the intrinsic linear structure of CLIP's joint text-image embedding space (38). This structure allows us to formalize concept removal through geometric operations that preserve semantic integrity.
Linear Analogy Basis for Single-Concepy Erasure. The foundational insight originates from CLIP's ability to encode semantic relationships as vector displacements
language model(30) For example, the analogy of" p("king") "(" man") > o(" (("queen")) - of woman") demonstrates that semantic transformations can be modeled through vector arithmetic. Formally, for concept pairs (a, b) and (c, d) sharing analogous relationships (e.g., king man queen woman), their embeddings satisfy:
(la) - ((b) = @(c) -(d)..
establishing the linear structure essential for semantic manipulation.
Let e, "("") denote the neutral reference embedding. Given an input prompt embedding Cinput oi (Pinpuu and target concept e with embedding Cerase = (bPersee), generalize Eq. (4) for concept removal, we introduce binary presence indicator {0, }} that explicitly encodes whether the target concept exists in the input. This allows unified representation of both concept-containing (p 1) and concept-free (p 0) scenarios. Specifically, the analogical projection becomes:
Cinput Peerase RJ Φ (Piuputtc))
pens
(5)
where Plaput' denotes the concept-free prompt.
Rearranging terms yields the semantic surgery operator that maps Cipput to the concept-removed space: (6)
impput
Cinput
p Cgrase
en)
Adams which projects the input embedding Cinput into the semantic subspace excluding concept e, satisfying e'mpu Re of c.)
Co-Occurrence Encoding for Multi Conti-CCnneet Erasure. When extending single-concept removal to multiple targets {c, 1 , a naive approach would linearly superimpose individual concept offsets

---

Σi= Pc, Aec, following Eq. '6's paradigm. However, this strategy fails to account for semantic overlaps which will lead to uncontrollable semantic elimination. For instance, removing both "gull" and "sparrows" via separate offsets would excessively diminish avian features due to shared bird semantics. A qualitative comparison in Appendix (Fig. B visually demonstrates the superiority of this approach over naive vector summation, which leads to significant image degradation. The core solution lies in CLIP's capacity to model composite semantics. We construct the co-crasure direction through existence-ordered composition:
Aew pPp..) - En. Pcu ① (1pe) E ))
where + denotes the concept concatenation operator of existing concepts (p. 1 cases). For simplicity, we define it as comma-separated string concatenation. The composite prompt Pro leverages CLIP's contextual embeddings [38] resolve semantic overlaps through phrase-level interaction, avoiding redundant subtraction of shared components.
Let Pjoint [0, 1] denote the joint presence of all concepts selected in Peo-
(Am==1) [1 if Vc, S.
Pjoint
8
otherwise
LEES
where S C Censs is the concept subset actually selected in Poo- The final operation preserves the form: Cmpul = Cinput Pjoint Aepo.
3.2 Semantic Surgery
Projection Decomposition. The theoretical foundation stems from the geometric relationship in Eq. 66). We formalize concept intensity estimation through: Theorem 1 (Concept Presence Projection). For input embedding Eiuput and concept direction Aeersse the presence intensity P satisfies:
(Eimpurs AGerase) (Clapure A Aeense
(10)
|e 1?
where imut represents the ideal sanitized embedding, which is generally unobservable during inference and serves as a theoretical construct for this geometric interpretation.
Corollary 3.1 (Angular Formulation). For tz-normalized encoders with ||Cmpuell =s (/Cppuelll let k e ||||||
Defining az cos( (eos(eapuas Aeenss) and a' cos((e Denase), we derive: p = k((a aa)
(11)
Semantic Biopsy. Since the sanitized embedding <mpuu is unobservable during inference, the residual similarity a' cannot be directly computed. The core objective of semantic biopsy is to estimate the concept intensity P solely from the observable a.- As established by our empirical findings (Assumption B.1), the value of a₂ itself is highly discriminative of whether the prompt implies concept presence or absence. We leverage this discriminability to calibrate an estimator p(a..) that maps a, to a probabilistic presence score:
Assumption 3.1 (Statistical Se-aaarrabillty). Let D1 denote the distribution of (p( values when prompt p contains the target concept, and Do when p does not contain the concept. There exist threshold B E R, separation margin
0, and small tail probability geep E (0,1/2) such that:
Po.~D, (0ke. 2 3 + e]21 8uep and Pa.-Da lo. S
8sep-
(12)
This assumption, supported by empirical observations (Fig. 0 Appendix E.2
states that with high
probability, ar values from the two classes are separated by at least 2e.
Theorem 2 (Sigmoid Calibration with High Confidence), Under Assumption 3.1
for any target error
bound Jeep& E (0, 1/2), if we choose the sigmoid steepness parameter as
7= logit(1 Berr)
(13)
In
Bana
Oerr
then the calibrated estimator p(a.) = o sigmonid
A
satisfies:
!f Orc ~ 'D1 (concept present) and ac 2 3 + e, with probability 2 1 - Ssep' (p(ac) - 1 S deer-
af ~ Do (concept absent) and ac < B - with probability 2 - S lø (x)) - 01 < 8err-
This means that for inputs falling outside the uncertainty region (B - e, B + t), which happens with probability at least 1 - Ssep for each class, the estimator is Berr-close to the ideal binary presence.
5

---

Practical Parameter Estimation. The practical estimation of the decision threshold 3 and sensitiv- ity parameter y is crucial for calibration. We determine 3 empirically by analyzing the separability of ar distributions and fine-tune y to balance classification sharpness and robustness. A detailed discussion of this process is provided Appendix D
Semantic Operation. Given the set of calibrated presence scores (Á:} } for each concept E Cerase (estimated via semantic biopsy, see Section (.2), we proceed with the multi-concepl removal. First, we determine the subset of actively present concepts by applying threshold T:
Caetive = {cr E Cerase j 2 T}.
(14)
If Cactive is non-empty, we construct the corresponding co-occurrence prompt Pco B e, Pe. (optionally ordering concepts by descending P:) and compute its joint semantic direction Arco ((pca) - e, as defined in Eq. 7
To modulate the removal strength, we approximate the joint presence factor Pjoin (from Eq. 9 using the maximum estimated score among the active concepts:
Pjointt max {Ø:}.
(15)
CC The final semantic surgery operation is then performed using this approximated strength and the joint direction:
énpar Einput Pjoint If Cactive is empty, the operation reduces to the identity, limput handles single concepts as a special case where [Cactive| 3
(16)
Cioput- This formulation naturally
3.3 Visual Feedback for Latent Concept Persistence (LCP) Mitigation
Latent Concept Persistence (LCP). Despite the initial semantic surgery (Eq. (16)) yiclding e where target concepts Cerase are idcally absent from its direct semantics, these concepts may still be generated due to the U-Net's visual priors being triggered by other concepts Cim in ég (e.g.. "road" implying "trees"). The LCP risk, R(c), quantifies this unintended visual resurgence for a set of visually detected persistent concepts Cuurget C Cerase (see Appendix | for formal definition).
LCP-Aware Concept Activation and Final Surgery via Visual Feedback. To mitigate LCP,
images generated using é,, are evaluated by vision detector D, yielding visual presence scores {p!m)
for C E Ceraee Concepts with Pim 2 Tvis form the visually active set Cyis, which constitutes our LCP target set Curget The final set of concepts for removal, C C.em Uvis (where Crem is from initial semantic biopsy, Sec. 3.2. determines the LCP-specific co-occurrence prompt P.. = + ej Pey, from which the direction Aet. is derived. The joint removal strength, Pjuin is determined by the maximum effective confidence among concepts in C* For Cj E C", its effective score is β
Cj E Csem (original semantic biopsy score), or Awaspfe) c, E Cvis (visually detected score scaled by a factor Xvis E (0, 1)). If is in both sets, its effective score is the maximum of these two.
Pjoint max (effective_ (iciive_ccor((e)),, or if C* is empty.
(17)
eyEC-
The final, LCP-mitigated embedding Enaa is then computed by applying a comprehensive surgery to the original Cinpus:
6'mma = Cinput
P jjoin Ae:..
(18)
Theoretical Support for Risk Reduction. The reduction in LCP risk via Eq. (18) is theoretically
supported. By constructing Aet to include visually persistent concepts Cangett our surgery direction aims to counteract the U-Net's LCP-inducing pathways. Under standard assumptions of directional alignment and local L-smoothness of the LCP risk function R(e) (see Appendix FF. applying the surgery with an appropriate strength Pioion provably bounds the post-surgery risk (Theorem 3. This leads to risk reduction of the form R(( < R(E)) Ci + ((())o))) )²), where C₁ 0. The parameters Avis: Tsem, Tvis are tuned empirically to achieve a practical Poont that balances risk reduction and fidelity (details in Appendix [F3)].
The computational overhead of this optional two-pass mechanism is analyzed in Appendix B where we show that its average inference time remains comparable to other training-free methods on practical safety benchmarks.
4 Experiments
We evaluate Semantic Surgery on diverse concept erasure tasks, focusing on completeness (Eq. locality (Eq. (2)), and robustness. Robustness, crucial for practical completeness, measures crasure

---

efficacy against paraphrased prompts unseen during configuration. We compare Semantic Surgery against state-of-the-art (SOTA) methods across these dimensions.
4.1 Experimental Setup
Tasks and Baselines. We evaluate Semantic Surgery on five diverse erasure challenges. The first four are standard benchmarks: Object Erasure (CIFAR-I0 classes (22)). Explicit Content
Removal (I2P dataset [43], and both Artistic Style and Conceet Celebrity Erasure 227. To
specifically address a critical aspect of security, our fifth evaluation focuses on Robustness Against Adversarial Attacks, where we test our method's s resilience against both black-box (RAB
(50)
and white-box (UnlearnDiffArk (59) adversarial prompts. All experiments use Stable Diffusion v1.4 42 Our method is compared against state-of wfae-off-the-art parameter-modiffing (e.g.. MACE p7)
Receler 19] and inference-time (SLD (43), SAFREE (54)) baselines. Detailed dataset provenances, evaluation metrics, and baseline configurations are provided in Appendix A.1
Implementation Details. Our method's key hyperparameters are set as 7 = 0.02 and == 0.5, with sask-specific thresholds J3. The optional LCP feedback loop is enabled for object and explicit content erasure. To ensure unbiased evaluation in the object erasure task, we use an independent OWL-ViT detector (31 for final performance measurement. Further details on all hyperparameter choices and LCP configuration can be found in Appendix A.2
We compare Semantic Surgery against two classes of methods. Our primary comparison is against strong marameter-modifyy ESD (-u and -x) [10]. UCE [II), AC [23), Receler [19], and MACE [27]. To provide a more comprehensive assessment, time methods, SLD
baselines, which represent the state-of-the the in erasure performance:
also include a targeted comparison against contemporary inference- 43 and SAFREE (54). specifically on the critical I2P safety benchmark where most paramount. For tasks adopting the MACE setup (Artistic Style, Multi-Concept Celebrity Erasure), results for MACE and other common baselines are taken directly from MACE (27) to ensure fair comparison with their reported optimal performance, Other baseline results are reproduced using official code and recommended settings.
the practical benefits of fast, training-free free deployment
4.2 Object Erasure
Experimental Setup. We evaluate object erasure the 10 categories of the CIFAR-10 10 dataset For each category, a model is configured to erase that specific target concept. We measure Efficacy (Accg), Robustness (AceR), and Locality (Accz). Accg is the percentage of successful erasures for simple prompts (e.g.. "A photo of [class)"). ACer measures erasure success for paraphrased prompts (e.g.. "A sleek jetliner soaring through clear skies" for "airplane"), generated via ChatGPT as per Receler [19]. Acc, is the generation accuracy for the nine non-target classes using their paraphrased prompts. Detection is performed by OWL-ViT B1. Overall performance is summarized by the harmonic mean (H) of successful erasure rates (as percentages) and locality: H 3/ (1/(100 Accg) + 1/0000 Aceg) + 1/AccL), where higher is better.
[2].
Result Analysis. Table [ shows Semantic Surgery achieves SOTA performance in object erasure, with the highest average H-score (93.58), +4.84 points over Receler. A key advantage of our method its exceptional robustness. With an average ACer of just 2.00, Semantic Surgery is 5x more effective at resisting paraphrased prompts than Receler (10.00) and nearly 7x more effective than MACE (13.80). This resilience stems from our global, pre-diffusion embedding manipulation, which addresses concepts at fundamental semantic level. While MACE achieves slightly better efficacy on simple prompts (Acog of 0.40 vs. our 1.50), this small gap is far outweighed by our substantial gains in robustness. Furthermore, our method achieves the best locality (Acc, of 85.56) among all high-performing methods, confirming that precise, dynamic intervention minimizes impact on unrelated concepts.
4.3 Explicit Content Removal
Experimental Setup. We evaluate the removal of "nudity" and related concepts using the 4,703 prompts from the I2P dataset [43). To achieve comprehensive safety against range of prompts,
we target a group of four related concepts: "nude", "naked", "sexual", and "erotic". We count the
instances of generated images classified as containing nude content by NudeNet [6] (threshold 0.6).
We also report FID (36) and CLIP scores
(38] on the MS-COCO 30k validation set 25 to assess image quality and prompt alignment for general, non-sensitive prompts.
7

---

Table 1: Full evaluation of object erasure all CIFAR-10 classes. Aceg: Efficacy (lower is better), Aceg: Robustness (lower is better), Accy: Locality (higher better), H: Harmonic Mean (higher is
better). Full results for all 10 classes are in Appendix H
Classes
Metric
SD1.4
ESD-x
ESD-u
AC
UCE
Receler
MACE
Ours
airplane
Accel
100.00
30.00
12.00
2,00
10.00
4.00
0.00
2.00
Acent
70.00
62.00
24.00
18.00
34.00
6.00
10.00
4,00
Accz
89.11
87.89
86.44
87.44
90.78
87.33
85.56
89.1
Hf
57.72
83.13
88.67
80.48
92.29
91.47
94.21
automobile
Accæl
96.00
30.00
24.00
0.00
0.00
3.00
0.00
0.00
AcoRl
84.00
74.00
64.00
24.00
54.00
18.00
18.00
4.00
Accz
87.56
87.44
88.22
83.78
85.11
84.00
83.11
79.78
H1
46.74
57.39
85,48
68.98
87.19
87.65
91.04
bird
Accæl
87.56
11.00
10.00
0.00
4.00
1.00
0.00
0.00
AccRl
100.00
84.00
50.00
82.00
62.00
26.00
24.00
2.00
Accz f
90.00
84.56
78.00
87.44
85.67
80.22
74.89
86.89
Hf
35.06
68.29
38.97
61.98
83.15
82.17
94.60
cat
Accgl
97.00
18.00
4.00
0.00
1.00
0.00
0.00
1.00
Acent
98.00
46.00
26.00
42,00
0.00
18.00
0.00
Accz
86.00
84.33
78.44
86.11
83.56
80.22
83.33
86.00
Hf
70.47
81.79
77.21
91.72
92.41
87.73
94.55
Avg
Accel
99.10
22.20
12.50
3.30
2.30
2.50
0.40
1.50
Acend 87.20
63.20
39.40
47.60
28.20
10.00
13.80
2,00
Accz 87.33
85.49
81.87
85.53
85.50
81.58
79.09
85.56
Hf
56.03
73.72
70.00
81.90
88.74
87.13
93.58
Table 2: Explicit content (nudity) removal on the I2P dataset. Columns list NudeNet-dettcted instances per category and total. FID and CLIP scores on MS-COCO 30k assess general image quality. Baselines are grouped by type. FID/CLIP scores for SLD and SAFREE were reported as the primary comparison is on safety efficacy.
Method
Armpits
Belly
Buttocks
Feel
Breasts (F) Genitalia (F)
Breasts (M)
Genitalia (M)
Total
FID
CLIP 1
SD v1.4 <1..iiOriga v2. (Filtered)
149
172
28
66
267
19
43
751
14.04
31.34
87
159
18
69
133
43
515
14.87
31,53
Panameter Parameter-Modfjjing
AC
51
%
27
56
15
215
14.13
31.37
ESD4s
11
13
13
3
3
55
15.1
30.21
ESD-x
94
101
19
41
123
22
410
14.41
30.69
Receler
48
24
17
159
14.1
31,02
MACE
31
17
26
30
13
0
123
13.42
29.41
Methous
SAFREE
11
22
15
15
82
Ours
12.2
30.75
Result Analysis. Table demonstrates Semantic Surgery's 's state-of-the-art performance in sup- pressing nudity. Our method reduces the total count of sensitive instances to just 1. This represents new benchmark for inference-time methods, showing over a 98% reduction compared to SAFREE (82 instances) and also significantly outperforming strong purameter-modifyle (123) and ESD-u (55). Notably, this near-perfed near-perfect erasure is achieved while also improving general image quality, evidenced by an excellent FID score of 12.2 (surpassing SD v1.4's 14.04) and main- taining a competitive CLIPP score (30.75). The visual feedhack mechanism was instrumental for this level of completeness, analyzed in Appx.
baselines like MACE
4.4 Artistic Style Erasure
Experimental Setup. To assess the erasure of nuanced attributes, focus on removing artistic styles. This task evaluates the model's 's ability to neutralize specific stylistic influences while preserving general semantic content and other visual characteristics. Following the methodology of MACE [27]. we utilize the Image Synthesis Style Studies Database [20] to curate set of 200 distinct artists. This set is divided into an "erasure group" of 100 artists, whose styles are targeted for removal, and "retention group" of 100 artists, whose styles should remain generatable. For each artist, prompts are formulated as "Image in the style of [artist name)".
Erasure performance is quantified using CLiP-based metrics, consistent with MACE:

---

CLIP. (Efficacy): The average CLIP similarity [8 (values scaled by 100 as per common practice, lower indicates better erasure) between prompts of artists in the erasure group and the images generated in response to these prompts.
CLIP, (Specificity/L.oclity)) The average CLIP similarity for artists in the retention group, measuring the preservation of non-target styles (higher indicates better locality).
H. (Harmonic Balance): An overall score calculated as Ha - CLIP, CLIP,, where a higher Ha signifies superior balance between effective erasure and style preservation.
To evaluate the impact on general image generation capabilities, also report FID scores (36 and
CLIP similarity on 30,000 captions from the MS-COCO validation set [25 (denoted FID-30K and CLIP-30K).
Table Assessment of Erasing 100 Artistic Styles. CLIP, (efficacy) and FID-30K should be lower. CLIP, (specificity/llcaitt)) H. (overall balance), and CLIP- --0K- should be higher. Best results for crasure methods are bolded. SD v1.4 provides baseline for original model performance.
Method
CLIPet
CLIP,t
Hat
FID-30K
CLIP-30K
AC [23
29.26
28.54
-0.72
14.08
31.29
UCE [1]]
21.35
26.32
4.97
77.72
19.17
ESD-x
10
20.89
21.21
0.32
15.19
29.52
ESD-u 10
19.66
19.55
-0.11
17.07
27.76
Receler
19
23.25
23.17
-0.08
16.55
30.24
MACE
27
22.59
28.58
5.99
12.71
29.51
Ours
20.75
28.84
8.09
14.04
31.34
SD v1.4 (Original)
29.63
28.90
N/A
14.04
31.34
Result Analysis. Table 3 shows Semantic Surgery excels at erasing 100 artistic styles. It achieves the highest H. (8.09). >2 points over MACE (H1, = 5.99). This stems from top-tier specificity (CLIP, = 28. 88) nearly matching SD v1.4 (0) while retaining strong efficacy (CLIP, -20.75 75). ESD-u has better CLIP, (19.66) but poor CLIP, (19.55) and negative Ha, indicating indiscriminate style damage. Crucially, Semantic Surgery shows no degradation in general image quality: FID-30K (14.04) and CLIP-30K (31.34) match the original SD v1.4. This contrasts with other methods like UCE (FID-30K = 77.72, severe degradation) or MACE FFD-30K 12.71). Qualitative examples in Appx.
4.5 Multi-Concept Celebrity Erasure
Experimental Setup. We adopt MACE's setup 27 for celebrity crasure: a dataset of 200 celebrities
(100 "erasure group". 100 "retention group"). We erase 5, 10, and all 100 celebrities from the crasure group. Efficacy (Accuracye- Fig. 2.. lower is better) is Top-1 GIPHY Celebrity Detector (GCD) 14
accuracy on erased celebrities. Locality (Accuracy,. Fig. 2b. higher is better) is Top-1 GCD accuracy
for retained celebrities. Overall performance (H, = 2/ Accuracy.) + +/AAcuurcyy.) Fig. 2
higher is better). General image quality (FID-30K, Fig. 2 and semantic alignment (CLIP-30K.
Fig. 2)) on MS-COCO 30k are also assessed. Baseline data in Fig. is adapted from MACE (27
with Receler data generated by as it was not included in their original comparison.
Result Analysis. Figure illustrates Semantic Surgery's 's superior performance in multi-celebrity erasure. Our method achieves near-perfect erasure efficacy (Accuracy. As 0.005 even for 100 concepts, Fig. 2 and maintains high locality (Accuracy: 2 0.936 for 100 concepts, Fig. 2p).
closely matching SD1.4's original specificity across all scales. This leads to a consistently high overall H. score (Fig. 2c). When erasing 100 celebrities, Semantic Surgery achieves H. RJ 0.965, significantly outperforming MACE (H, 0.892). UCE (Hl, x 0.554), Receler (Hl, 0.441), and other baselines. MACE [27] notes that methods like AC and SLD-M show limited effectiveness, reflected in their low He scores. Crucially, Semantic Surgery maintains excellent general image quality and semantic alignment (Fig. [il-e.. FID-30K remains stable around 14.45 and CLIP-30K around 31.343, comparable SDI.4 (FID Rs 14.060, CLIP As 31.326) even when erasing 100 celebrities. This contrasts with UCE, which, as MACE 277 also observed, sees its FID sharply increase and CLIP score drastically drop when erasing 10+ concepts. Our approach effectively handles challenging scenarios by preciscly targeting semanties.

---

3.
22
04
02
10
160
5
300
165
100
16
100
Completaness
[c] Overall
Image
Semanti Alignmant
Figure 2: Multi-Concepp Celebrity Erasure. (a) Completeness (Accuracye: lower better). (b) Locality (Accuracys, higher is better). (c) Overall (He, higher is better). (d) Image Quality (FID-30K on MS-COCO, lower better). (e) Semantic Alignment (CLIPP-3KK on MS-COCO, higher is better).
X-axis: # of Erased Celebrities. Baseline data adapted from MACE [7]: Receler data generated by us.
4.6 Robustness Against Adversarial Attacks
To assess adversarial robustness, we evaluated our method against black-box (RAB, 380 prompts) (50
and white-box (UnlearnDiffArk) [59] attacks. As shown in Table Semantic Surgery demonstrates superior resilience. It achieves a state-of-the-art Attack Success Rate (ASR) of 1.05 % against RAB attacks, a statistically significant improvement over the best baseline (MACE, 3.95%; p=0.0089). Remarkably, it achieved 0.0 % ASR against the white-box attack. We attribute this robustness to our Semantic Biopsy mechanism, which acts as an effective semantic gate. A detailed setup and analysis, including discussion on how our method doubles as threat detection system, are provided in Appendix[]
Table Robustness against adversarial attacks. Our method demonstrates superior resilience to both black-box (RAB) and white-box (UnlearnDiffAtt) attacks. The difference in ASR the 380-prompt RAB test between our method and the next-best baseline (MACE) is statistically significant (Fisher's Exact Test, p=0.0089).
Type
Method
(ASR)
SLD
78.68%
SAFREE
Black Box (RAB, 380 prompts)
MACE
3.95%
Ours (88, LCP)
1.05 %
Whito-Box (UnlcamDiffAAk)
Ours (SS, LCP)
0.0%
4.7 Qualitative Analysis.
Beyond quantitative metrics, we provide extensive qualitative results in Appendix which visually demonstrate the nuances of our method's performance. For instance, F shhowcases side-by-side comparisons on object, style, and explicit content erasure, illustrating effective concept removal while preserving scene context. Furthermore, Figure 3 in Appendix C provides striking visual confirmation of Co-Occurrence Encoding's superiority over naive vector subtraction in multi-concept scenarios. These examples visually corroborate quantitative findings and highlight strong compltteness-locaity balance.
5 Conclusion
We introduced Semantic Surgery, novel zero-shot, inference-ting framework enabling robust concept erasure via calibrated vector subtraction directly on the global text embedding, guided by dynamic concept presence assessment. This pre-diffusion strategy, enhanced by Co-Occurrence En- coding for multi-concept scenarios and a Visual Feedback Adjustment for Latent Concept Persistence, fundamentally targets superior completeness and locality. Our extensive experiments against strong contemporary baselines confirm Semantic Surgery achieves state-of-the-all efficacy and robustness while preserving content quality across diverse benchmarks, often by a significant margin. Offering a precise, adaptable, and model-agnostic solution, Semantic Surgery marks key advancement in safer, more controllable text-to-image generation, with future work poised for cross-modal extension and deeper semantic disentanglement.
10

---

Acknowledgments and Disclosure of Funding
We extend our sincere gratitude to the anonymous reviewers for their insightful and constructive feed- back, which substantially improved this work. Their suggestions were instrumental in strengthening our theoretical analysis by better connecting our framework to the core objectives of completeness, locality, and robustness. On the experimental front, we are especially grateful for the guidance that led us to conduct crucial new evaluations, including a more rigorous, unbiased assessment of object erasure, comprehensive comparisons against additional inference-tinge baselines, and extensive testing against both black- box and white-box adversarial attacks. The deep dive into adversarial robustness, prompted by their feedback, uncovered important new findings regarding method's inherent resilience and its secondary function as a threat detection system. Their collective input has significantly enhanced the clarity, rigor, and overall quality of the paper.
References
[1] Safety checker nested in stable diffusion.
https://hugiingaccc.cco//////// srablee-ddffuussoonaaeeeyyccooo 2022.
[2] Stable diffusion. http::///./gggaooooooo/////////a 2022.
[3] Stable diffusion v2.0. 2022.
[4] Stable diffusion v2.1.
hutps://bugianfaaacccossss111111
stable-diffusiion22 2022.
[5] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, NG Aidan, Lukasz, and P Illia. attention all you need", advances in neural information processing systems. NeurIPS Proceedings, 2017.
[6] Praneeth Bedapudi. Nudenet: Neural nets for nudity detection and censoring, 2022. URL https://github. com/housA-secch/uueeeell
[7] Anh Tuan Bui, Long Tung Vuong, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Erasing undesirable concepts in diffusion models with adversarial preservation. In The Thihth- Annual Conference on Neural Information Processing Systems.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[9] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
[10] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426-2436, 2023.
[11] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 5111-5120, 2024.
[12] Alex Gittens, Dimitris Achlioptas, and Michael W Mahoney. Skip-gram- zipf+ uniform= vector additivity. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 69-76, 2017.
[13] Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. In European Conference Computer Vision, pages 73-88. Springer, 2024.
[14] Nick Hasty, Ihor Kroosh, Dmitry Voitekh, and Dmytro Korduban. Giphy celebrity detector. Giphyyceleb-ddeection-sss, 2019.
[15] Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36: 177991199744 2023.
11

---

[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[17] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications.
[18] Chia Yi Hsu, Yu Lin Tsai, Chulin Xie, Chih Hsun Lin, Jia You Chen, Bo Li, Pin Yu Chen, Chia Mu Yu, and Chun Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? In 12th International Conference on Learning Representations, ICLR 2024, 2024.
[19] Chi-Pin Huang, Kai-Po Chang, Chung- Ting Tsai, Yung-Hsuar Lai, Fu-En Yang, and Yu- Chiang Frank Wang. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. In European Conference on Computer Vision, pages 360-376. Springer, 2024.
[20] Surea I, Proxima Centauri B, Erratica, and Stephen Young. Image synthesis style studies. httpps//wwet www.:waaappppp..oo comp ai art- https//maaasstttsssi--tt--lleeee studies
[21] Changhoon Kim, Kyle Min, and Yezhou Yang. Race: Robust adversarial concept erasure for secure text-to-image diffusion model. In European Conference on Computer Vision, pages 461-478. Springer, 2024.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Cifar-10 dataset. https: httpp:///ww/ /ttp:://wwww..oo hooppp/ edu/ kriz/cifaa html 2009, Accessed: 2025-05-13.
[23] Nupur Kumari, Bingliang Zhang, Sheng-Yo Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22691 2023.
[24] Gant Laborde. Deep nn for nsfw detection. https: // g wwt.:/ocon/aaaaannnnnaaa nswl model
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[26] Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference Machine Learning, pages 32287-32307, 2024.
[27] Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[28] Mengyao Lyu, Yuhong Yang. Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. One-dimensional adapter rule them all: Concepts, diffusion models and erasing applications. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7569-7568. IEEE, 2024.
[29] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[30] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-751,
2013.
[31] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, al. Simple open- open-vocabulary object detection. In European conference on computer vision, pages 728-755. Springer, 2022.
12

---

[32] Andrew Ng and Landing AI. Agentic object detection.
https://landinn ai/
2025.
[33] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabílistic models. In International conference on machine learning, pages 5162-81711 PMLR, 2021.
[34] Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models.
[35] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. Forty-first International Conference on Machine Learning.
[36] Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On aliased resizing and surprising subtleties gan evaluation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11410-11420 2022.
[37] Jeffrey Pennington, Richard Socher, and Christopher D Manning, Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1332-554,, 2014.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-878877 8763. PmLR, 2021.
[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.
[40] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. In NeurlPS Workshop on Machine Learning Safety. ETH Zurich, 2022.
[41] Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504-15522, 2024.
[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[43] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1-522-22531,22 2023.
[44] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278 22222422222992...22 2022.
[45] Christoph Schuhmann, Robert Kaczmarczyk. Aran Komatsuzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis. 440om-400m: Open dataset of clip-filtered 400 million image-text pairs. NeurlPS Workshop Datacentric AI, number FZJ-2022-00923. Jülich Supercomputing Center, 2021.
[46] Christoph Schuhmann, Richard Vencu, Gentec Data, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, Juelich Supercomputing Center JSC, et al. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.
[47] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. Glaze: protecting artists from style mimicry by text-to-image models. In Proceedings of the 32nd USENIX Conference Security Symposimm, pages 2187-2204, 2023,
13

---

[48] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffu- sion art or digital forgery? investigating data replication diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023.
[49] Eric Todd, Millicent Li, Arnab Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Func- tion vectors in large language models. In International Conference on Learning Representations. ICLR, 2024.
[50] Yu-Lin Tsai, Chia-Y Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia You Chen, Bo Li, Pin- Yu Chen, Chia-Mu Yu, and Chun-Ying for diffusion models? In The Twelfth International Conferencee on Learning Representations.
Ying Huang. Ring-a-bell! how reliable are concept removal methods
[51] Yuan Wang, Ouxiang Li, Tingting Mu, Yanbin Hao, Kuien Liu, Xiang Wang, and Xiangnan He. Precise, fast, and low-cost concept erasure in value space: Orthogonal complement matters. arXiv preprint arXiv:2412.066
2024.
[52] Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text- controlled generative models. Advances in Neural Information Processing Systems, 36:35331- 35349, 2023.
[53] Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Tsung- Yi Ho, Nan Xu, and Qiang Xu. Mma- diffusion: Multimodal attack diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7737-7746, 2024.
[54] Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, and Mohit Bansal. Safree: Training-free and adaptive guard for safe text-to-image and video generation. In The Thirteenth International Conference on Learning Representations.
[55] Chulin Xie Chih-Hsun Lin Jia-You Chen Bo Li Pin- Yu Chen Chia-Mu Yu Chun- Ying Huang Yu-Lin Tsai* Chia-Y Yi Hsu* Ring-a-bell! how reliable are concept removal methods for diffusion models? In The Twelfth International Conference on Learning Representations, 2024.
[56] Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me- not: Learning to forget in text-to-image image diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1755-1764, 2024.
[57] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3847, 2023.
[58] Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure diffusion models. Advances in Neural Information Processing Systems, 37:36748-36776, 2024.
[59] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now. In ECCV (57), 2024.
[60] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International conference on machine learning, pages 12979-12990. PMLR, 2021.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA]
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes]" " is generally preferable to "[No]" it is perfectly acceptable to answer " "[No]" provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often
nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found. IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction state the core contributions (Semantic Surgery framework, Co-Occurrence Encoding, LCP mitigation) and claims of superior performance in completeness, locality, and robustness, which are supported by theoretical discussions (Sec 3 Appendix E F) and extensive experiments 4).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification: A dedicated "Limitations" section is provided in Appendix K discussing aspects such as dependence on text encoder quality, hyperparameter sensitivity, reliance on visual detectors for LCP, and computational overhead of the visual feedback loop.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical results (Theorem T Corollary, Theorem 2 Theorem 3 are pre-
sented with their assumptions, Proofs and detailed derivations are provided in Appendix E (covering Thm I, Corollary, Thm assumptions) and Appendix (covering Thm 3).
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper details the experimental setup in fectionfficludiing base models, datasets (with citations), evaluation metrics, specific hyperparameter values used for our method (7, 3, T, Avi), and baseline configurations. Further details on prompt generation and full CIFAR- results are in Appendix E.2 and
Justification:
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-sonnee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have included the code and prompt datasets used for experiments in the supplementary material, ensuring that the experiments can be easily reproduced by following the provided instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwwaiiiciguiieeeieeeeeeeeeeeeeee
ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section (Experimental Setup) specifies the base model (Stable Diffusion vI.4), sampler (DDIM, 50 steps), datasets (CIFAR-10, I2P, Image Synthesis Style Studies, MACE celbbrryy dataset, MS-COCO 30k) and their splits/usage per prior work or standard practice. Hyperparameters for Semantic Surgery (7, 3, T, Avis) are explicitly stated. The process for choosing 3 is described in 3ec Appendix E.2 so no optimizer details are needed for
Our method is training-free,,
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our adversarial robustness evaluation on 380 prompts includes a Fisher's 's Exact Test to confirm the statistical significance (p=0.0089) of our method's superiority. This is presented in Section 4.6 and detailed Appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eттor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
18

---

Answer: [Yes]
Justification: We provide detailed inference time analysis in Appendix B (Table 7), comparing our method's computational overhead against baselines and analyzing the cost of the optional LCP module.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httts::/neuurpps. cttpps.cc//pubiape http:/p.......lllec.hcccsscslldd
Answer: [Yes]
The research aims to improve Al safety by removing undesirable content, aligning with ethical AI development. We have reviewed the NeurlPS Code of Ethics and believe our work conforms to it. Broader impacts, including potential misuse, are discussed in Appendix
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: dedicated "Broader Impact" section is provided in Appendix discussing potential positive impacts (harmful content reduction, IP protection, bias mitigation) and negative impacts (over-erasure, adversarial attacks, false sense of security), along with potential mitigation strategies.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
19

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper introduces a method for concept erasure, which is itself a form of safeguard. We do not release models datasets that pose high risk of misuse. The method operates on existing models and aims to make them safer.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets used (CIFAR-10. 12P, MS-COCO, Image Synthesis Style Studies, MACE celebrity dataset) are publicly available and are cited with references to their original publications Section 4 The base model (Stable Diffusion v1.4) is also cited. We assume adherence to their respective licenses as per standard academic practice. Specific license names are listed for each dataset within the paper but can be found via the provided citations.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
The paper does not introduce or release new datasets, models, or other such assets. The primary contribution is novel method.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The research presented in this paper does not involve crowdsourcing experi- ments or direct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects directly, therefore IRB approval not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
21

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: ChatGPT was used generate paraphrased prompts for the object erasure task (Section 4 specifically subsection Object Erasure), which is a component our
experimental methodology for evaluating robustness. This usage is explicitly mentioned.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy battps://neurre /etprrppps..ccoooffeeeeeee///222 for what should or should not be described.
22

---

A Detailed Experimental Setup
This section provides a comprehensive overview of the experimental configurations, complementing the summary in the main paper (Sec. 4).
A.1 Tasks, Datasets, and Metries
We evaluate our method across four distinct concept erasure challenges:
Object Erasure: We test the removal of common nouns using the 10 object classes from the CIFAR-10 dataset [22]. Following Receler [19], we use simple prompts (e.g.. "A photo of (class)") ") for efficacy (Acog) and paraphrased prompts generated via ChatGPT for robustness (AccR). Locality (Acer)) is the generation accuracy for the nine non-target classes. Object presence is determined by an independent OWL-ViT detector (31 for unbiased final evaluation.
This safety-critical task focuses on eliminating nudity, We use the 4,703 prompts from the Inappropriate Image Prompt (I2P) dataset [43] Efficacy is the total count of images flagged by NudeNet [6]. General image quality is assessed via FID 33 and CLIP
Explicit Content Removal:
scores 38 on MS-COCO 30k (25
Attribute/Style Erasure:
We target the removal of artistic styles using the setup from MACE [27
and the Image Synthesis Style Studies database [20]. Performance is measured with CLIP-based metries: CLIP, (efficacy, lower better), CLIP, (specificity, higher is better), and the harmonic balance H. CLIP, CLIP..
We assess scalability by crasing multiple celebrities, following MACE [27] We use a celebrity dataset to erase subsets of 5, 10, and 100 celebrities. Efficacy and locality are measured by the GCD detector [14] accuracy on erased and retained groups, respectively, summarized by harmonic mean Hę-
Multi-Concept Erasure:
Robustness Against Adversarial Attacks:
To specifically evaluate security resilience, we test our method against adversarial prompts. We measure the Attack Success Rate (ASR) using two prominent benchmarks: the black-box, model-agnostic Ring-A-Bell (RAB) attack (50) on an expanded set of 380 prompts. and the white-box, optimization-based UnleamDiffArt (59] This
task directly evaluates the method's ability to withstand sophisticated circumvention attempts.
A.2 Hyperparameter Settings and Visual Feedback Configuration
Key hyperparameters and the use of the Latent Concept Persistence (LCP) mitigation module varied across experiments, summarized in Table The LCP module, when active, performs second- stage inference if visual concepts targeted for erasure are detected in the first-stage output. All experiments use Stable Diffusion v1.4 42 with the DDIM sampler (50 steps).
Table 5: Summary of key hyperparameter settings and LCP visual feedback configuration across experiments.
Experiment Tisk
e (Decision Threshold)
(Stoepness)
(Activation Threshold)
Visual Feedback (LCP)
Ava (if (CLLC active)
CIFAR 10 Object Erasure
-0.12
0.02 (Global)
0.5 (Global)
Yes (AOD (22
1.0
I2P Explicit Content Removal
-0.06
0.02 (Global)
0.5 (Global)
Yes (NudeNet
1.0
Artistic Style Erasure
(Global)
Multi Concept Celebrity Erusure
-0.28
0.02 (Global)
0.5 (Global)
No
NIA
Adversaria Robusrness
-0.06
0.02 (Global)
0.5 (Globbl)
No
NIA
A.3
Visual Feedback Implementation Details
The visual feedback (LCP mitigation) module was employed for CIFAR- 10 object erasure and I2P explicit content removal. If a targeted concept was detected above a predefined threshold in the initial generation, feedback reinforced the erasure in a second pass.
For the I2P task, NudeNet [6] identifies exposed body parts (e.g., BUTTOCKS, EXPOSED, FEMALE FEMALE_BREAST BREAST EXPOSED). If the maximum detection score for any such element exceeded the
threshold (e.g., 0.6), the erasure strength for all four targeted abstract concepts ("nude", "naked", "erotic", "sexual") was simultaneously increased in the second-stage -tage generation, amplified by Avis-
A.4 Dataset Details
The datasets used for our experiments were sourced as follows:
CIFAR-10 Object Erasure: We utilized the simple prompts and paraphrased prompts from the
10 task setup of Receler [9]
23

---

I2P Explicit Content Removal: This task used the standard I2P (Imagen Prompt Dataset) a common benchmark for evaluating safety in text-to-image models.
43
Artistic Style Erasure: The list of artists targeted for style erasure was adopted from the experimental setup of MACE [7], facilitating direct comparison.
Multi-Concept Celebrity Erasure: Similarly, the set of celebrities for the multi concept erasure task was also sourced from MACE [27] ensure fair comparability with prior work.
Using established dataset configurations where possible aids in reproducibility and allows for more direct comparisons with existing methods.
B Ablation Study of Visual Feedback
We conduct an ablation study to evaluate the impact of the Visual Feedback Adjustment module (Section 3.3). Table shows the performance of Semantic Surgery with and without this component on the CIFAR-10 object crasure task (averaged over all classes) and the I2P explicit content removal task.
Table 6: Ablation study of the Visual Feedback Adjustment module. "Visual Feedback': / indicates the module used, x indicates it is not. For CIFAR-10, 10, metrics are averages over 10 classes. H, for CIFAR-10 10 corresponds the H score defined in Sec 4.2. "Total' for I2P refers the total count of nude images generated.
Task
Config
Visual Feedback
CIFARIO
I2P
AccE
Acen
Aec1
Ha
Total
1
4
6.4
87.38
92.18
47
2
1.5
2.0
85.56
93.58
The results demonstrate the sigmificant benefit of the visual feedback loop. On CIFAR-10, incorporat- ing visual feedback improves erasure efficacy (Aceg from 4.0 to 1.5) and robustness (Aceæ from 6.4 to 2.0) with only marginal decrease in locality (Ace,, from 87.38 to 85.56), leading to an overall improvement in the score from 92. 18 to 93.58. This slight drop in locality primarily attributable to imperfections in the visual detector used for feedback. Specifically, we observed that the AOD detector occasionally misclassifies related but distinet concepts, for instance, identifying "truck" as "automobile." Such false positives incorrectly trigger second-stage erasure surgery, which can inadvertently affect the generation of non-target concepts and thus slightly reduce the measured locality. This observation underscores the importance of employing high-precision detector within the feedback loop to maximize its benefits while minimizing non-local effects.
The impact is nude images are generated. With visual feedback, this number plummets to just 1, highlighting the module's ' critical role in addressing Latent Concept Persistence (LCP) for sensitive concepts that may not be fully neutralized by semantic manipulation alone. The LCP scaling factor Avis (referred to as for this)
pronounced for explicit content removal (I2P). Without visual feedback. 47
1.0 for these experiments, balancing aggressive LCP correction with fidelity. For celebrity and style erasure tasks, the visual feedback loop was not employed due minimal observed LCP risk (celebrity) or lack of reliable detector (style), as noted in Section 4
A in the main text if Eq. was intended
Inference Time Analysis. To quantify the computational overhead of the LCP visual feedback loop. we conducted a detailed timing analysis. The feedback loop is an on-demand, two-pass process; second generation is triggered only if the detector identifies a persistent concept. Table benchmarks our method's 's inference time per image against baselines.
The analysis reveals three key points: (1) The core Semantic Surgery framework (LCP disabled) adds negligible overhead (3.21s vs. 3.11s for the baseline). (2) The "doubled time" scenario is worst-case, not the norm. (3) a practical, high-security application like the I2P task, the average inference time (4.09s) is comparable to other faining-free methods like SLD and SAFREE, while providing vastly superior erasure completeness (as shown in main paper, Table This highlights that the LCP module offers a highly effective and efficient trade-off for critical safety needs.

---

Table 7: Inference time analysis image (50 steps) on single NVIDIA RTX4090 GPU. The
average LCP time for Semantic Surgery on the I2P task demonstrates its practical efficiency.
Scenario
Time per Image (seconds)
Baseline SDv1.4
3.11 s
Inference-Time Methods
SLD
4.07
3. SAFREE
3.96 s
Ours (LCP disabled)
3.21 s
Semantic Surgery with LCP
5. Ours (LCP worst-case, always 2-pass)
6.43
Ours (LCP average on 12P task)
4.09 s
C Qualitative Analysis of Co-Occurrence Encoding
Our main paper (Sec. 3.1) introduces Co-Occurrence Encoding (Eq. 6) for robust multi-concept erasure. Figure 3 qualitativell compares it against a "Naive Approach" (summing individual erasure vectors) for erasing "dog" and "cat" from "dog and cat playing together."
Co-Occurrence Encoding successfully removes both target animals while preserving the "playing together" action, often substituting them with plausible alternatives like children, thus maintaining the scene's 's narrative. In contrast, the Naive Approach significantly degrades image quality and semantic coherence, leading to muddled imagery. This demonstrates Co-Oecurrence Encoding's advantage in neutralizing targets while protecting contextual integrity and image quality, unlike the naive method's tendency to over-erase.
D Experimental Analysis of Hyperparameter Sensitivity
Practical Parameter Estimation. Theorem 2
provides the theoretical basis for calibration using ag- Practical application requires determining the parameters 33 and oy for the estimator p(ac) =
sigmoid (a. B)/7).
Decision Threshold (9): This parameter corresponds to the threshold 3 described in Assump- tion 3.1 is empirically determined by analyzing the distributions of a, values for concept- present and concept-absent prompts to find value that effectively separates them (see Ap- pendix E.2 and Figure 6). This 3 optimally centers the sigmoid's ' decision boundary based on the observed data.
Sensitivity Tuning (y): The parameter controls the steepness of the sigmoid function around B. While Theorem provides a theoretical construction for y (Eq. (3) to achieve a specific error bound 6ery given observed separation margin in practice, ? often fine-tuned empirically. This empirical tuning balances the sharpness of classification (smaller y for more decisive output near (3) with robustness to variations in a₂ near the decision boundary and the desired level of confidence in the output ý. It typically involves evaluating performance on a validation set.
We analyzed sensitivity to hyperparameters (sigmoid steepness, main paper, Eq. 12) and B (concept presence threshold, main paper, Sec. 3.2) CIFAR-10 10 object erasure (average over 10 classes, no visual feedback for this test). Defaults: y = 0.02 (global), task-dependent , (e.g., -0.12 for CIFAR-10). For y sensitivity, B was fixed near optimal value -0.06).
Impact of y (Sigmoid Steepness): Figure 4 shows metrics (Accg, AceR, ACeL, HH)) y varies (log scale: 0.02 to 1.0). Stability across this range indicates low sensitivity to 7, justifying our global choice of y 0.02.
Impact of 3 (Concept Presence Threshold): Figure 5 shows 3's impact. 3 is crucial for erasure activation. H. indicates an optimal 3 range (around -0. to -0.06 here) balaneing effective erasure (Aceg R 0) with high Aceg and Acez, This supports selecting 3 based on a, distributions (main paper, Sec. 3.2).
These analyses confirm that while 3 requires careful selection, robustness to variations enhances practicality.
Impact of (Concept Activation Threshold): The threshold r determines the minimum confi- dence score å required to activate the crasure for concept c₁. To evaluate its impact, conducted
25

---

Co-Occurrence Encoding
Naive Approach
Original
Prompt: "dog and cat playing together" Erased Concepts: ["dog", "cat"]
Figure 3: Qualitative comparison for multi-concept erasure ("dog". "cat"). Our Co-Occurrence Encoding (center) preserves semantics (e.g., "playing together" with children) while the Naive Approach (right) degrades image quality compared to the Original (left).
an ablation study on the CIFAR-10 10 object erasure task (averaged over 10 classes, without visual feedback), with 3 and fixed at their default values. Table 8 presents the results.
Table Ablation study for the concept activation threshold on CIFAR- object erasure. Perfor- mance is stable across wide range of values.
ACcE +
ACCR &
Acc1 t
H-score 1
0.95
4.5
24.2
88.40
85.77
0.5 (Default)
4.0
6.4
87.38
92.18
0.05
4.0
3.5
84.90
92.14
The results show that the model's performance, particularly the overall H-score, is highly stable for within a wide range of (0.05, 0.5]. A very high threshold (e.g., 0.95) can degrade robustness (Acog) as it may fail to activate erasure for weaker semantic cues, while a very low threshold (e.g., 0.05) can slightly harm locality (Accr,) by being overly sensitive. Our chosen default of - 0.5 provides a strong balance, and the overall stability indicates that the method not overly sensitive this hyperparameter.
26

---

Impact of Object Erasure (CIFAR10) Performance Metrics
100
Metric Score (%)
Metrics
Acce
Acc,
40
R: (Overall Performake
Value (Log Scale)
Figure 4: Impact of y (Sigmoid Steepness, log scale) on Object Erasure (CIFAR-10) Performance Metrics. Metries show high stability across tested y values.
Impact of f on Object (rasure(CIFAR10) Performance Metrics
80
Metric Score (%)
Metrics
Acca
Acc.
He Performance)
23
Value
Figure 5: Impact of 3 (Concept Presence Threshold) on Object Erasure (CIFAR-10) Performance Metrics. An optimal range balances erasure effectiveness with semantic preservation.
E
Theoretical Framework and Empirical Support for Semantic Surgery
E.1
Geometric Interpretation of Ideal Concept Removal (Contextual Background)
This section outlines the geometric relationship for an idealized concept removal, as expressed in Theorem [ and its Corollary. While our primary calibration mechanism (Theorem 2 relies the empirical discriminability of a, values rather than directly inverting these equations, this geometric view provides context for why aa
indicator of concept presence.
Theorem I (Concept Presence Projection). For input embedding Cinput and concept direction
Cerase, the presence intensity p (as defined in Eq. 6 satisfies:
(Cinput Aeerase) (Cinpat (erasse
(19)
A A
where Eypgu represents the ideal sanitized embedding Cinpat Peeaas.
27

---

Proof. From Eq. (6) in the main text, Cnput = Cinput parerase Rearranging gives paeerase
Cinput Cimpu: Taking the inner product of both sides with A Perase CpAcerase A = (Cinnut
Capurs Cerase) Since p is scalar, pfAeerase ACerase) (Cinuu Acewse) (Cmmut Aerase) Given
(Aeerase; Aerase A Cerase 112, and assuming Ceruse o O, we solve for p. yielding Eq. (9). L
Corollary (Angular Formulation). For le fo-normalized encoders where can be assumed that |/emput|| |I R |ppall (c.g., if p is small or re-normalization is applied), let k |/einpur / Aersce ||-
Defining Oee cos coos(Cmpen (Cmput, ACerss) ) and a' cos(( (Cppus ACcrsse: then:
p = ki(ac a0)
(20)
Proof. From Theorem
= ((Ciaput: Aecrsse - (Cmmpu (eerase)) Ferase 11? eenseell
Using (u, v)
|ell cos(u, u): = (l |ll l/ac | Gerase ||a Aerase 1.2 = (lleinpal Oc
Empal maaullaa)) (le)//leerrreellll Il- If eimpat ~
1 empuulll then P 2 |einpur (ake ax) a')/ll Demsell ||- Defining
k (enpul ||/|| Ceeasell || gives Eq.
20
Relevance Calibration: These results illustrate that, ideally, a higher are (when Cinput contains the concept, so p > 0 and a' is for a sanitized state) compared to a lower (when Cinput itself is like a sanitized state, so p and a a') is indicative of concept presence. Assumption 3.1 directly captures this empirically by observing the separation of a. values based on whether the input prompt implies concept presence or absence.
E.2 Empirical Support for a -e-aearrabllity (Assumption 3.1
Assumption 3.1 (Statistical ag- -e-Separability), which posits that (c(p) = cos(é(p), Aerass) values for
prompts containing target concept are well-separated from those for prompts not containing it, is central to our calibration method. We provide strong empirical support for this assumption through a systematic prompt generation and evaluation process.
Systematic Prompt Generation. To rigorously test the separability, we designed a 'RobustPrompt- Generator' (details of its configuration and generation logic are available in our supplementary code).
This generator takes a configuration for an "anchor concept" (c.g. an object like "chair", a modifier like "red". style like "Van Gogh", or sensitive content like "nude") and systematically generates three categories of prompts, each with N = 100 unique instances:
Related Prompts (P (Present:: These prompts directly include the anchor concept (e.g.. "a wooden chair in studio photography"). Non-anchor components like scenes, other modifiers, and styles are varied randomly from predefined pools to ensure diversity.
Variant Prompts (P ::rianss): These prompts replace the anchor concept with one of its pre-defined synonyms or close semantic variants (e.g.. "an ergonomic seat 3D render" if "seat" a variant of "chair"). Other components are varied similarly to related prompts. This group helps assess the robustness of Og to linguistic variations of the target concept.
Unrelated Prompts (Pabsent): These prompts replace the anchor concept with a pre-defined unrelated concept from a similar category (e.g., replacing "chair" with "lamp" if both are objects, or "red" with "blue" if both are colors), while attempting keep other contextual components (scenes, styles, non-anchor modifiers/objects) consistent with those used for the related prompts where applicable, or varied from similar pools. This creates challenging negative samples that are contextually similar but semantically distinct regarding the anchor concept.
For each generated prompt p. we compute its embedding d(p) and then (p(p) = cos (d(),, Acanchor) where Aaanhoo is the pre-computed semantic direction for the specific anchor concept.
Observed a. Distributions and Separability. The distributions of aa values obtained from these prompt sets consistently demonstrate strong separability. Figure @lllustrate typical outcomes for four diverse anchor concepts: "Object (Chair)", "Explicit Content (Nude)", "Style (Van Gogh)", and "Modifier (Red)".
As shown in Figure 6
The ae values for Passeat (blue distributions, p 0) consistently cluster in lower range of ac values.
The or values for Ppresem (red distributions, p = consistently cluster in a significantly higher range.
28

---

B-1 COMPEE
2-1 ECONEDT SRAMPS
2-1 COPONE MIHNS
-24
03
Object (Chair)
Explicit Content (Nude)
boneot refereo
$10.1.0
3.
Style (Van Gogh)
Modifier (Red)
Figure 6: Illustrative a, distributions for different anchor concepts, demonstrating empirical separa- bility. For each subplot: The red distribution (p = 1, concept present) corresponds to ae values from P Prresent (direct inclusion of the anchor concept). The yellow distribution = 1, concept variants) corresponds to a₂ values from Pvariants. The blue distribution (p = O, concept absent) corresponds to ae values from Patsent (unrelated prompts where the anchor concept is replaced). The green dashed line indicates an empirically chosen threshold /8 that effectively separates the "concept absent" (blue) distribution from the "concept present" (red and yellow) distributions. The clear separation between the blue distributions and the red/yellow distributions across diverse concept types strongly supports Assumption 3.1 The "concept variants" (yellow) generally align well with the "concept present" (red) distributions, indicating robustness to linguistic variations.
The a values for Peariants (yellow distributions, also considered = for the purpose of concept presence) generally overlap substantially with the P Preseen distributions, indicating that our anchhur captures the core semantics despite linguistic variations.
Crucially, for each anchor concept, there is clear separation or minimal overlap between the Patsent (blue) distribution and the combined P Ppeeeent/ Pvariants (red/yellow) distributions.
This observed separability allows for the robust determination of threshold 3 and margin c as defined in Assumption 3.T For instance, the green dashed lines Figure represent empirically chosen 8 values. The margin can then be identified as the distance from to the edge of the high-density regions of the separated distributions, and the tail probabilities óeep are observed to be very small. This systematic empirical validation across diverse concept types and prompt structures provides strong backing for our theoretical framework.
E.3 Proof of Theorem 2 (Sigmoid Calibration)
Theorem 2. Under Assumption 3.1 for any target error bound derr E (0, choosing y = e/logit(1 derr) ensures p(ax)) sigmoid((a,, - 8)/>) satisfies: If Dj and a. 2 - (p(a.)) - 1| < Serr. If a. ~ Do and a, 3 - (ā(a.) of < serr.
Proof. Let (xx) = 1/(1 + x) be the sigmoid function. The logit function is logit(p) = (n(p/(1 p)). We want (XX) >
derr for positive classification, which implics X > logit(1 derr). We want (X) < derr for negative classification, which implies < logit(6err). Note that logit(derr) lloittl derr).
C The parameter is chosen as 7 = logit(1 ) (0, 1/2). we have 7 0.
Since e > 0 and logit(1 serr) > 0 for Seer E
29

---

Case 1: Concept is present (a, D1) and a 2 8+c. The input to the sigmoid is Xpres = (ac-8)/7 Since a, 3 2 e, we have pres 2 e/7. Substituting the chosen ?.
xres </logit(1 Berr) logit(1 derr)
As (-) is monotonically increasing: p(aco) = o (Xess) 2 ollogt(( Serr)) = 8err
Thus, 1- ((a.) S derr. Since p(ac) 1, this implies p((a)) - 11 S serr. This occurs with probability 2 1 6zep for ag ~ I. Case 2: Concept absent (a, Du) and de I B-c. The input the sigmoid is Xabs (ac- B)/7 Since a₂ - I -, we have Xats I - e/7,. Substituting the chosen y/:
-
Xabs I e/logit(1- - Serr) llggt((1 Serr) = logit(
As (-)) is monotonically increasing:
p(a²) o (Xabs) S a (1logi(( = derr
Thus, lā(a.) - 0] S serr- This occurs with probability 2 - deep for ag Do. This proves that for inputs falling into the high- sigh-conideenc regions (i.e., outside (B c, B + e)), the estimator p(a,)) is -lose to the ideal binary value. Assumption enseurs regions are indeed where most samples from D, and Do lie.
ensures these high-confidence
Remark on the uncertainty region (B e, 3 + e): The theorem guarantees performance for a, values outside this uncertainty margin. Inside this margin, the value of p(a:e) will smoothly transition between approximately derr and 1 Serr. Assumption 3.1 states that the probability of an ae from either class Du or D2 falling into the *other* class's side of this margin (i.e., a Do having Di having < 3 + e) is at most öeep If 8sep is small, misclassification or uncertain estimations are infrequent. The practical choice of B aims to minimize errors within this region based on the observed distributions.
>
F
Theoretical Analysis of Latent Concept Persistence (LCP) Mitigation
This appendix provides a formal theoretical framework for the LCP mitigation strategy described in based on standard assumptions from optimization theory.
Section 3.3
F. Formal LCP Risk Definition and Assumptions
Latent Concept Persistence (LCP) occurs when concepts Cersse intended for removal and presump- tively absent from the direct semantics of an initial surgically modified embedding & (output of Eq. (16), are nonetheless generated. This is attributed to the U-Net's 's visual priors being triggered by other, semantically distinct concepts Cam present in or implied by 2. Let Cuage S. Cerase be the subset of concepts visually detected as persistent from images generated by é', We define the LCP risk function R R - [0, 1] for an embedding e (which is assumed to be semantically sanitized w.r.t. Curget but may still cause their visual generation) as:
R(e) (X)p)(()e) max [l E Concepts(!))
(21)
cCCurge
This measures the probability of at least one concept from Cunger appearing. For analysis, we assume Ree) or a suitable smooth surrogate differentiable.
The final ICC--mitigated surgery (Eq. (18) is E Cimput Pett Ae where we map notation:
co Cmput (the embedding to which the full surgery is applied).
Emma (the embedding after the LCP-aware surgery).
Pat =
Poonn (the effective removal strength from Eq. (I7).
Aesug =
deto (the joint removal direction for C* = Csem Grrrget.
The risk we aim to reduce is that observed at R(E)).
We make the following standard assumptions:
(A1) Directional Alignment: The chosen surgery direction Aesary is well-aligned with the
negative gradient of the risk function R evaluated at co Cipput That is, there exists constant > 0 such that: (VR(eo), Aearg) 2 M/R. (R(en) All
(22)
This implies that subtracting De from ep moves the embedding in direction that tends to reduce R. The construction of Aesurg based on Cunges (visually persistent concepts) strongly supports this alignment for mitigating LCP.
30

---

(A2) L-Smoothness: The gradient (R( is Lipschitz continuous with constant L 2 0: [VR(e.) VR(e,)l & Lile, eall Vea: Cb.
(23)
F.2 LCP Risk Reduction Theorem
Theorem 3 (LCP Risk Bound via Directional Surgery). Let eo = Cipuut and e1 = Enmnt eo e Under Assumptions (A1) and (22) the post-surgery risk R(e1) bounded by:
R(e1) & R(eo)) Pe (R(ee) mu L(Pen)
| 1|?
(24)
2
For risk reduction (ie., R(e)) < R(ea)), the effective step size Pog must satisfy Pett < 2yr (R
L L
Proof. This result is a standard consequence of L-smoothness in optimization. For an L-smooth function R, and update e 6g + òc, Taylor's theorem with remainder bound (or descent lemma) gives:
L
R(e1)= - R(eo) + (VR(eo), 6e) + ||ōe||..
(25)
In our LCP mitigation surgery, the update is de = -o = -Per -e Substituting this into Eq. (5
L
R(e1) s R((e)) + (VR(eo)) Pott A +
3 - Pett A essurg |?
2
R(ee) - Purt (R(co), Aesug) + [
| g 11
Now, we apply the Directional Alignment condition (Assumption (AI)) (VR(co), A >
mR (R(ee) Aesurg |- Since this term is preceded by a negative sign in Eq. (??), we have:
- Pat VR(eo) , Assug) - | (Aleggg sirglll ll. Substituting this back gives:
+ [(pen)²
R(ei) < R(eo) PentiR (R((e)) Ael
A 1|
This completes the proof of Eq. (24). For R(e1) to be less than R(eo), require the sum of the second and third terms on the right- hand side to be negative:
PR (R( Acl sung || + L(pett)
||A <
2
Since Per > and assuming | > O, we can divide by Petl Aesurgl
Ipen
mye VR(ee)]]
2 |esurgl <0
21RR. (R(ee)]
Pett L.I
This provides the condition on Per for guaranteed risk reduction relative to R(co).
Interpretation for LCP Mitigation Effect on RŒ))) The theorem formally bounds R (E(mm))) relative to R(Cmpu)) The practical goal is reduce the LCP risk R(²)) that was observed after the initial semantic-only surgery. The final surgery (Eq. (88) is specifically designed based on Curget (visually persistent concepts from é(). Thus, Aesurg and Pet chosen to counteract the factors leading to R(()) > O. While the direct proof applies to the change from Cinput: the informed construction of the surgery ensures that e is moved region of lower LCP risk compared to the state e. The simplified statement in the main text, R( ( S R(()) Cip" + ((p))) heuristically captures this intended consequence.
F3 Connection to Method Implementation and Empirical Validation
The directional aligument (A!) supported by constructing Aesung (our Aet.) to explicitly include visually persistent concepts Curgen L-smoothness (A2) is common modeling assumption for deep learning systems. The step size Penr (our Pjoint is effectively controlled by the empirical tuning of Ayis and the estimated concept presence scores. Qualitative examples demonstrating successful LCP removal provide visual confirmation of the method's 's efficacy.
The example in Figurry7cleearly demonstrates that even when a concept (like "tree") is not semantically cued by the prompt ("a road") and might be missed by semantic-ome analysis, LCP can cause its generation due to strong model priors. Our visual feedback mechanism effectively identifies such instances, and the subsequent LCP-aware surgery successfully removes the unintended concept, aligning with the theoretical prediction of risk reduction. This highlights the importance of the visual feedback loop in achieving comprchensive concept erasure.
31

---

Semantic Biopsy Only
Add Visual Feedback Adjustment
Prompt: road Ky
Semantic Not Related
Prompt: road
Visually Detected
Erased Concept: tree
Erased Concept: tree
Concept Not Detected!
Concept Detected Through Feedback!
Figure 7: Qualitative illustration of LCP mitigation for the target concept "tree". (Left Semantic Biopsy Only): The input prompt is "a road", and the "tree" concept is targeted for erasure. Despite the prompt not semantically implying "tree", the initial semantic surgery (ε²) fails prevent trees from appearing, due to the U-Ner's strong prior associating roads with trees. Semantic biopsy alone does not detect this LCP issue as the prompt itself is "clean". (Right Add Visual Feedback Adjustment): After generating the image on the left, visual feedback detects the persistent "tree" concept. Our LCP-aware refined surgery (resulting in é/mma) then incorporates this visual information. The subsequent image generation successfully removes the trees, yielding a scene consistent with only "a road". This visual improvement directly corroborates the risk reduction predicted by Theorem
G Theoretical Guarantees for Semantic Surgery
This appendix provides the formal theoretical framework supporting Semantic Surgery, linking our method to the desiderata of Completeness, Locality, and Robustness defined in the main paper (Sec. 3
G.1 Core Theoretical Framework and Assumptions
Our analysis relies on three foundational principles, which are empirically validated in our work.
Assumption G.1 (Statistical Separability). For any concept c, the distribution of cosine similarities 0g cos( (p)), Aee) for prompts containing the concept (D₂) is statistically separable from the distribution for prompts not containing the concept (Do). This formally stated in the main paper's
Assumption 3.1 and empirically validated in Appendix E.2 This property enables high-accuracy
concept detection via our sigmoid calibration (Theorem ensuring lp probability.
Pideatl der with high
Assumption G.2 (Bounded Linear Erasure Error). Vector subtraction provides bounded approx- imation of ideal semantic removal. For an embedding e, its ideal counterpart without concept c, Φ(p c), and the presence p. there exists small, empirically bounded error
[|(e paee) o(p e)]l S E.
This is grounded the linear structure of CLIP's embedding space 130 and is validated by the high efficacy (Aceg) in our experiments.
Assumption G.3 (Lipschitz Continuity of the Generative Process). Small perturbations in the
embedding space lead to bounded changes in the probability of generating a concept. Let ge(e)= (t [[[c Concepts([))) [])]. We assume ge(e) ) is locally Lipschitz continuous with constant Xe:
(gc(er) ge(e2) I xelle1 eq
32

---

G.2 Proof of Theoretical Guarantees
G.2.1
Completeness (Erasure Guarantee)
Theorem 4 (Completeness). Let e' Eipuus e be the modified embedding for a concept E Cenue Under Assumptions G.1 G.3 for a chosen safety threshold Esafer if parameters are set such that Ae ( + {) < Esafe then:
ge(e') S safe + O(6sep).
Proof. Let Gideal
ó(p be the ideal embedding without concept c. The error between
modified embedding e' and the ideal one is ||e' eideal = [einput ( - Gideall- By applying
the triangle inequality and adding/subrracting p paeerase (where p the true presence):
Gideal
= =1 (Cinpus
p Perase Pideal) + (p (1pp-))aaaaae (
< (eeinuu D Gideal l| + 11(p D) Censell
v + 1p 01 - e (by Assumption G.2).
From Assumption G.1 and Theorem 2 from the main paper, we know that P 0 < sen with
probability 2 1 -sep- Thus, l|e" Eideall S 5
+ derl Aeensel. By Assumption G.3 (Lipschitz
continuity of gc):
19a(e') ge (e(dcaa)) S Aclle' eldeall S Ae + dewll Acarssll).
By construction, the ideal embedding Cideal does not contain concept c, so ge(Eidea))
This gives:
ge(e') I >o(E + Sen |
If we choose parameters such that the right-hand side is less than <aafes the bound holds. The term (fxep) accounts for the small probability that our presence estimation ý falls outside the & error bound. The LCP mitigation module (main paper, Sec. 3.3) further reduces any residual risk empirically.
G.2.2 Locality (Fidelity Guarantee)
Theorem 5 (Locality). For any non-target concept d & Cerases the change in its generation probability is bounded. With probability at least 1
ōxep:
Iga(c') ga(e)] = O,
and with probability at most Swp. the impact is bounded by negligible value k: [ga(ee) - ga(e)) S K, where k= Xa
Proof. Let de = c' c -Poin Cc be the surgery vector applied to the input embedding e. The proof proceeds by considering two cases based on whether the surgery is activated.
Case 1: Surgery is not activated (ōe = 0). This occurs when no target concepts are detected in the input prompt, i.e., Psoint < T (where T is the activation threshold). From Assumption G.T and Theorem 2 the probability of false positive detection for any single concept is low. Thus, the probability of incorrectly activating the surgery when no target concepts are present bounded by deep Consequently, with a high probability of at least 1 - 6sep. no surgery is applied. In this case,
= e, which means = 0, and the impact on any non-target concept is zero: 19a(e') ga(e)] O.
Case 2: Surgery is activated (5e + 0). This occurs with a low probability of at most Seep when no target concept is present, when target concept is correctly detected. By Assumption G.3 (Lipschitz continuity), the change in the generation probability of concept d is bounded by:
Iga(ee) ga(e)) S >aale' - ell = >zalōeel.-
We can bound the norm of the surgery vector. Since Pjoine E [0, 1] and letting Mc. = | be the norm of the co-occurrence direction vector (which is bounded for given set of concepts), have:
||ōe|| I/pjiint < M
Therefore, the maximum impact in this case is bounded by K k= A
While this provides a formal bound, the practical impact is typically much smaller. This is because the surgery vector Aec constructed from target concepts and is thus nearly orthogonal the semantic direction Aea of distinct, unrelated concept d. This pear-orthogonality ensures that the vector subtraction primarily affects semantic components related to Cerse, minimizing collateral impact on d.
Conclusion. Combining both cases, the expected impact on a non-target concept is negligible. The most common scenario (with probability 2 deep for concept-absent prompts) results in zero impact. In the less frequent case of an incorrect activation, the impact bounded by a small constant k. This proves the locality of our method.
33

---

G.2.3 Robustness (Resistance to Paraphrasing)
Theorem 6 (Robustness Lipschitz Continuity). The Semantic Surgery operator T(e) Ace) (c) is locally Lipschitz continuous. That is, there exists a constant L > 0 such that for any embedding and a sufficiently small perturbation de:
(e + òe) - T(e)ll < L|jōell.
Proof. Let 61 = F. and e2 = e de. We need to bound (e()) - -Tei)ll-
(iT(e2) T(e))] = (((2 - Ac (e2)) (62) (ei (ca)))
= (e2 - e1) - (Ā(e2) e - (co))))
< l ei || + +1(((22) Ac (e2) )(ei) (e)))
The first term is ||ōe|]. We focus on the second term by adding and subtracting p (en)(())) (lā(ez) (e)) - ((ei) (ec((e)) S (ā(e2) ((ei) ) (e))) + +((e))]] Aeco (e2 Aen
Let's bound each component:
1. p(e) = ((coo(e,, Aee) - 9)/7) is composition of locally Lipschitz functions. The cosine similarity cos(e, v) is locally Lipschitz w.r.t. e, and the sigmoid a is globally Lipschitz. Thus, p(e) is locally Lipschitz with some constant Lp. so (Ā(e2) ((ei)l S Lolle2 eill Lp]ldell.
2. The concept direction norm ( )l is bounded by some constant M.
3. The co-occurrence direction (e) is also locally Lipschitz. This is because it depends on which concepts exceed the threshold 7, but within a neighborhood where the active set of concepts does not change, A is constant. At the boundaries, it has jump discontinuities, but we consider a neighborhood where it stable. For simplicity, assume it is locally Lipschitz with constant Leo, so (e2)) - (e))l S 6ell
4. The term [ is bounded by 1.
Substituting these bounds back:
IT((e2)
- T(e:)ll < ||ōe|| + ( + (Z)(Lcoll)el)) (1 + ML, Lel)]
Letting 1= Mp + Len. we have shown that locally Lipschitz continuous, which proves the theorem. This directly guarantees that small changes in input embeddings lead to proportionally small changes in the output, ensuring robustness.
H Additional Evaluation Results on CIFAR10 Classes
This section provides the complete evaluation results for object erasure across all 10 CIFAR-10 classes, supplementing the selected results presented in the main paper [Table The metrics Acce (Efficacy), Acen (Robustness), Ace,, (Locality), and H (Harmonic Mean) are defined in Section 4 As shown in Table 9 Semantic Surgery consistently demonstrates strong performance across all classes, achieving the highest average H score.
I Detailed Analysis of Adversarial Robustness
To address the critical concern of adversarial robustness, we evaluated Semantic Surgery against both black-box and white-box adversarial prompt attacks, which are designed to circumvent erasure mechanisms and regenerate forbidden concepts [50 (5).
Experimental Setup. We conducted two sets of experiments. (1) Black-Box Attack: We used
the Ring-A-Bell (RAB) benchmark [50], a model-agnostic attack that generates adversarial prompts through linguistic manipulation. To ensure statistical significance, expanded the test set to 380 adversarial prompts targeting object erasure. We compared our method against both training-free (SLD, SAFREE) and modiffyin (MACE, Receler) baselines. (2) White-Boy Attack: We
adapted the Unlearn LineearDiff'Ark framework [59], an optimization-basae attack that directly manipulates embeddings in model-aware manner, test our method's resilience against gradient-based search for vulnerabilities. For both setups, we measure the Attack Success Rate (ASR), where lower is better. Our core method (Semantic Surgery without LCP) was used for these tests.
Detailed Analysis. As shown in Table in the main paper, Semantic Surgery exhibits of the-art adversarial robustness. Against the large-scale black-box RAB attack, our method achieves an ASR of just 1.05%. %. a 3.7x reduction compared to the strongest baseline, MACE (3.95%). A one-sided Fisher's Exact Test on this 380-sample set confirms this difference highly statistically significant (p=0.0089).
34

---

Table 9: Full evaluation of object erasure all CIFAR-10 classes. Aceg: Efficacy (lower is better), Aceg: Robustness (lower is better), Accy: Locality (higher better), H: Harmonic Mean (higher is better).
Classes
Metric
SD1.4
ESD-x
ESD-u
AC
UCE
Receler
MACE
Ours
airplane
Accg-
100.00
30.00
12.00
2.00
10.00
4.00
0.00
2.00
Accal
70.00
62.00
24.00
18.00
34.00
6.00
10.00
4.00
A
89.11
87.89
86.44
87.44
90.78
87.33
85.56
89.11
Hi
57.72
83.13
88.67
80.48
92.29
91.47
94.21
automobile
Acegl 96.00
30.00
24.00
0.00
0.00
3.00
0.00
0.00
Accat
84.00
74.00
64.00
24.00
54.00
18.00
18.00
4.00
Accr
87.56
87.44
88.22
83.78
85.11
84.00
83.11
79.78
111
46.74
57.39
85,48
68.98
87.19
87.65
91.04
Accel
87.56
11.00
10.00
0.00
4.00
1.00
0.00
0.00
Accal
100.00
84.00
50.00
82,00
62.00
26.00
24.00
2.00
A
90.00
84.56
78.00
87.44
85.67
80.22
74.89
86.89
Hf
35.06
68.29
38.97
61.98
83.15
82.17
94.60
cat
Accel
97.00
18.00
4,00
0.00
1.00
0.00
0.00
1.00
Accal
98.00
46.00
26.00
42.00
6.00
0.00
18.00
0.00
1
86.00
84.33
78.44
86.
83.56
80.22
83.33
86.00
Hf
70.47
81.79
77.21
91.72
92.41
87.73
94.55
deer
Accel
99.00
7.00
6.00
6.00
2.00
0.00
0.00
1.00
AceRl
96.00
60.00
32.00
68.00
2.00
0.00
0.00
0.00
Accrt
86.22
82.78
71.11
82.89
84.78
72.22
72.22
84.44
Hf
62.72
76.13
55.60
93.16
88.64
88.64
93.92
dog
Accel
99.00
32.00
14.00
0.00
1.00
0.00
0.00
2.00
AccRI
92.00
72.00
40.00
66.00
30.00
2.00
16.00
Acez 86.67
83.22
77.56
84.89
83.67
75.78
78.89
86,44
IIf
48.05
72.84
58.60
82.56
89.82
86.75
94.42
frog
Accel
100.00
12.00
3.00
0,00
1,00
0.00
0,00
4,00
Accal
88.00
34.00
20.00
24.00
4.00
0.00
2.00
2.00
Acct f
87.11
85.22
80.89
85.44
87.11
82.89
70.00
86.89
Hf
78.43
85.30
86.05
93.76
93.56
86.98
93.37
horse
Accel
100.00
9,00
0.00
0.00
0.00
0.00
Aceal
96.00
62.00
48.00
44.00
18.00
4.00
6.00
0.00
Accz
86.22
84.78
83.11
86.67
81.56
80.22
75.56
82.22
H
60.64
71.00
76.15
87.07
91.24
88.56
93.28
ship
Accel 100.00
40.00
30.00
25,00
4.00
17.00
4,00
3,00
AccRl
70.00
64.00
46.00
58.00
46.00
32.00
18.00
6.00
Acez
89.11
88.56
87.78
86.33
88.56
89.11
89.11
89.11
Ht
53.82
67.88
61.57
74.58
79.00
88.67
93.26
truck
Accel 100.00
13.00
0.00
0.00
0.00
2.00
Accal
88.00
74.00
44.00
50.00
26.00
12.00
26.00
2.00
Accz 87.11
86.11
87.11
84.33
84.22
83.78
78.22
84.67
Hf
46.61
73.47
71.67
84.78
90.09
82.65
93.11
Avg
Accgl 99.10
22.20
12.50
3.30
2.30
2.50
0.40
1.50
Accal
87.20
63.20
39.40
47.60
28.20
10.00
13.80
2.00
Accz 87.33
85.49
81.87
85.53
85.50
81.58
79.09
85.56
H
56.03
73.72
70.00
81.90
88.74
87.13
93.58
Even more notably, our method achieved 0.0% ASR against the white-box UnlearDiffAAlk We hypothesize this exceptional resilience stems from our method's s core "binary gate" mechanism. The Semantic Biopsy step (Sec. (.22 creates sharp decision boundary based on cosine similarity (az). An adversarial attack faces difficult optimization challenge: it must craft embedding that contains the target concept visually but remains on the "concept-absent" side of the decision threshold semantically, a contradictory objective that proved insurmountable for the attacks tested.
Interestingly, we observed that while the white-box attack failed to regenerate the concept, it often produced images with significant quality degradation. This reveals secondary benefit: our framework
35

---

acts as a built-in threat detection system. An attacker's attempt to find a bypass consistently triggers high concept presence score (p), even the final image corrupted. By monitoring high-p flags for erased concepts at the pre-generation stage, a system can proactively detect and log adversarial probing attempts without needing to analyze final visual output.
SD1.4
ESD
CA
Recelor
UCE
MACE
Ours
T
Figure 8: Qualitative comparison of Completeness On CIFAR10. automobile)
Qualitative Results
This section provides qualitative examples illustrating the performance of Semantic Surgery. Figure shows side-by-side comparisons for various erasure tasks including object (automobile, bird, ship), explicit content (Nude, showing safe alternative), style (Van Gogh). These examples demonstrate the method's ability to effectively remove the target concept while preserving the overall image
36

---

structure and non- non-target elements. Figure 10 and Figure 8 specifically highlights the locality and the completeness of our method on CIFAR- 1- object erasure. The samples are randomly chosen from the erasure group and the remaining group. As our method successfully remove the targeted concepts while preserving most of the unrelated semantics remaining the image, our method succeed in eliminating the targeted concepts with accurate semantic surgery. Figure 11 provides examples of artistic style crasure. The generated images successfully remove the characteristic stylistic features of the target artist while retaining the subject matter described in the prompt. Figure 12 demonstrates celebrity erasure. Images generated for erased celebrities do not depict the individuals, while images for retained celebrities generated correctly. These qualitative results visually corroborate the quantitative findings presented in the main paper, showcasing the effectiveness and precision of Semantic Surgery.
Van Gogh
automobile
bird
ship
nudity
SD1.4
ESD
UCE
JALA
Receler
Ours
Figure 9: Qualitative comparison across different concept erasure tasks.
K Limitations and Future Work
While Semantic Surgery demonstrates strong performance, it has several limitations that open avenues for future work:
Dependence Text Encoder's Linear Structure: The efficacy of approach is funda- mentally conditioned on the text encoder (e.g., CLIP) exhibiting near-linear structure where concepts are semantically separable. This assumption, while holding for many common concepts, may be less effective for: (a) future, potentially non-linear encoder architectures, or (b) highly entangled, abstract, or metaphorical concepts (e.g.. feeling of nostalgia") where clean vector representation is ill-defined.
Although our method shows strong locality, is not entirely free from noc-l effects (as seen in the Acc, metric). We identify three primary sources for this:
Sources of Non-Local Effects:
Imperfect Semantic Thresholding:
The decision boundary defined by 3 may not be perfectly sharp for every prompt-concept pair. In ambiguous cases, our method might perform partial semantic operation, inadvertently affecting related but untargeted semantics.
Visual Detector Errors (in LCP): This significant factor. For the optional LCP module, false positives from the vision detector can trigger unnecessary second-stage erasure,
37

---

SD1.4
ESD
CA
Receler
UCE
MACE
Ours
Figure 10: Qualitative comparison of locality on CIFAR-10.
SDI.4
Ours
SDL4
Ours
Artist in erasing group
Artist retention group
Figure 11: Qualitative comparison Artist Removal.
38

---

II AP
SD1.4
Ours
SD144
Ours
Erased group
Other group
Qualitative comparison on Celebrity Removal. The target celebrity is erased from the generated image, while non-target celebrities general scenes are generated correctly.
Figure 12:
harming locality. For example, when erasing "automobile," the AOD detector sometimes misclassifies "trucks" as "automobiles," " causing our LCP module to incorrectly trigger stronger erasure on prompts that should have generated "truck." " This highlights the critical dependence of the LCP's locality on the detector's precision.
Inherent Model Stochasticity: Some locality degradation is attributable to the inherent stochas- ticity of the base generative model itself, which can fail to generate concepts perfectly aligned with the prompt even without any intervention.
The performance of our optional LCP module is inherently tied to the availability and accuracy of an external visual detector. The lack of reliable detectors for abstract concepts (e.g., complex artistic styles) limits the module's applicability to more concrete, visually verifiable concepts. Furthermore, the detector's performance creates trade-off: false negatives can lead to incomplete erasure, while false positives can harm locality by triggering incorrect feedback, discussed above.
Detector Dependency for LCP Module:
Scalability Newer Models: Our experiments were conducted on Stable Diffusion v1.4 to ensure a fair and direct comparison with the extensive list of prior works in concept erasure [10 27 19], which almost exclusively use as standard benchmark. Validating the performance and adapting the calibration of Semantic Surgery for newer, more powerful models like SDXL or SD3 is crucial next step for future work.
Future work could focus on developing adaptive hyperparameter tuning mechanisms, exploring LCP mitigation strategies that are less reliant on external detectors, and investigating methods to further improve concept disentanglement in the embedding space for even more precise multi-concept erasure.
L Broader Impact
Semantic Surgery offers a promising approach for enhancing the safety and controllability of text-to- image diffusion models. Potential Positive Societal Impacts:
The primary positive impact is the ability to effectively remove undesirable concepts such as explicit content (as demonstrated by near-perfect nudity removal), hate speech related symbols properly defined as concepts), or violent imagery. This can make generative AI tools safer for wider public use and reduce the proliferation of harmful synthetic media.
Harmful Content Reduction:
Copyright and Intellectual Property Protection: The method can be used to erase copyrighted artistic styles, characters, or specific celebrity likenesses, helping to mitigate infringement con- cerns associated with generative models. This could foster more ethical use of Al in creative industries.
Bias Mitigation: biases (e.g., stereotypical associations) can be identified and represented semantic concepts, Semantic Surgery could potentially be used to neutralize these biases in
39

---

generated images, leading to fairer and more equitable Al systems. For example, removing concept that model over-associates with particular demographic.
Enhanced User Control and Customization: Beyond safety, the ability precisely remove concepts gives users finer-grained control over image generation, allowing for more creative and specific outputs by excluding unwanted elements.
Reduced Need for Costly Retraining: As training free method, it offers more agile and cost-effective way to update safety protocols or adapt to new content restrictions compared to retraining large diffusion models.
Potential Negative Societal Impacts and Misuse:
Over-Erasure and Censorship Concerns: If not carefully calibrated or if applied too broadly, concept erasure techniques could lead to over-erasure, stifling creative expression uninten- tionally removing benign content that is semantically close to a target concept. Defining what constitutes an "undesirable" concept can be subjective and culturally dependent, raising concerns about who decides what is erased and the potential for censorship.
Adversarial Attacks and Circumvention: Like any safety mechanism, Semantic Surgery might be susceptible adversarial attacks designed bypass its erasure capabilities. Malicious actors could attempt to craft prompts or manipulate embeddings ways that circumvent the concept detection or neutralization process.
While effective, erasure method is likely to be 100% foolproof against all possible prompts or concept variations. Over-reliance on such tools without acknowledging their limitations could lead to false sense of security regarding the safety of generative models.
False Sense of Security:
Impact on Artistic Expression or Fair Use: While useful for IP protection, aggressive erasure of artistic styles could also limit transformative uses or artistic exploration that might fall under fair use or be part of legitimate artistic critique parody.
The development of erasure techniques might contribute to "arms race" where methods generate problematic content and methods to block it continually evolve, requiring ongoing research and adaptation.
Arms Race in Generative AI Safety:
Mitigation Strategies: To mitigate potential negative impacts, several strategies can be considered:
Transparency and User Control: Users should be aware when concept erasure is being applied and, where appropriate, have some control over its application intensity.
Careful Policy and Guideline Development: The definition of concepts to be erased should be guided by clear, ethical, and transparent policies, ideally developed with community input.
Continuously test the system against adversarial attacks and diverse prompts to identify and address vulnerabilities.
Robustness Testing and Red Teaming:
Combining with Other Safety Measures: Semantic Surgery should be seen as one layer in a multi-faceted approach to AI safety, complemented by dataset filtering, model alignment techniques, and post-hoe detection.
Research into Explainability: Further research into why certain concepts are detected missed can help improve the precision and fairness of the erasure process.
Overall, Semantic Surgery is a tool with significant potential for positive impact, but like all powerful technologies, its deployment requires careful consideration of ethical implications and potential misuse.
40


Reviews:
[{"review_id": "1Q1wnIQHOL", "reviewer": "Reviewer_moxY", "summary": "This paper introduces a training-free, zero-shot framework for concept erasure in T2I diffusion models.  Instead of editing model weights, it intervenes once, before generation, by:\n1. estimating the strength with which a target concept is present in the CLIP text-embedding of a prompt,\n2. subtracting a calibrated “concept vector” from the entire embedding (global rather than token-wise),\n3. extending this subtraction with a Co-Occurrence Encoding to handle many concepts jointly, \n4. applying an optional Visual Feedback Loop that re-checks generated images for concept leakage and readjusts the embedding to mitigate latent concept persistence.", "questions": "check weakness", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Solid technical formulation with clear geometric intuition and a calibrated sigmoid estimator.\n2.  Novel combination of global embedding subtraction, dynamic intensity estimation, and visual feedback, which can distinguish itself from token-projection methods.\n\n\nWeakness:\n1. Appendix contains essential calibration details; a shorter main-text description would help.\n2. Co-Occurrence Encoding uses simple string concatenation, which might conflate semantically overlapping concepts.\n3. Missing common robustness evaluation through adversarial prompt attacks (e.g., UnlearndiffAtk [1])\n\n[1]  To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images … For Now, ECCV 2024.", "was_revised": true, "final_justification": "All of my concerns have been fully addressed. I appreciate the authors’ thorough response and the additional experiments, which strengthen the paper. After carefully reading the other reviews and the rebuttal, I believe this work makes a meaningful contribution to the field. I have therefore decided to raise my rating to a strong accept.", "rebuttal": "We sincerely thank the reviewer for the positive feedback on our technical formulation and novelty. We also appreciate the insightful comments on potential weaknesses, which have pushed us to conduct several new, comprehensive experiments. We are pleased to report that the new results not only address your concerns but also reveal surprising new strengths of our method.\n\n### Weakness 1: Essential calibration details in the Appendix.\nThank you for the suggestion. We agree that moving key calibration details to the main text would improve readability. In the final version, we will condense our method section and move a concise summary of the key calibration details from the appendix to the main text, ensuring the paper is more self-contained and easier to read.\n\n### Weakness 2: Possibility of conflating semantically overlapping concepts with Co-Occurrence Encoding.\nWe strongly agree with the reviewer that handling semantic overlaps is indeed crucial. This is precisely why we designed the Co-Occurrence Encoding. As demonstrated in our **Supplementary material (Fig. 1)**, our comma-concatenation approach effectively preserves the semantic context (e.g., 'playing together'), while a naive vector summation leads to significant image degradation. This qualitative result demonstrates that our method successfully avoids the conflation issue in multi-concept erasure.\n\nThe effectiveness of comma-separated concatenation stems from the powerful compositional understanding of the pre-trained CLIP text encoder. CLIP was trained on vast amounts of web text, where concepts listed with commas typically represent a collection of distinct entities within a shared context. When CLIP processes a string such as \"dog, cat, playing together\", it generates a composite embedding that captures the individual concepts (\"dog\", \"cat\") while preserving the shared context (\"playing together\").\n\n### Weakness 3: Missing common robustness evaluation through adversarial prompt attacks.\nWe sincerely thank the reviewer for this critical feedback and agree that evaluating adversarial robustness is an important aspect. Our initial work focused on establishing a general-purpose framework for high-fidelity zero-shot concept erasure under standard conditions, aiming to first achieve state-of-the-art performance in completeness and locality, similar to other general-purpose methods including MACE (CVPR 2024) and AdaVD (CVPR 2025). This was the primary reason for not including adversarial evaluations in the initial submission.\n\nHowever, we fully acknowledge the importance of this dimension. To address this valid concern, we have conducted a new, targeted experiment to probe our method's robustness. For this study, we chose the **Ring-A-Bell (RAB)** benchmark [1]. Similar to the UnlearndiffAtk framework you mentioned, RAB represents a state-of-the-art approach for generating adversarial prompts. We selected RAB for two principled reasons: (1) Its prompt-based nature makes it model-agnostic, ensuring a fair, end-to-end comparison across all baselines; and (2) its focus on linguistic manipulation provides a direct stress test of our core semantic-level intervention.\n\nWe created a test set of 95 adversarial prompts from RAB to evaluate the Attack Success Rate (ASR). The results are as follows:\n\n**New Adversarial Robustness Study (ASR on 95 RAB prompts):**\n| Method                 | ASR (↓)  |\n|------------------------|----------|\n| SLD                    | 69.47%   |\n| SAFREE                 | 53.68%   |\n| MACE                   | 7.37%    |\n| Receler                | 6.32%    |\n| **Ours (without detector)** | **0.0%**|\n\nThe results show that on this test set, our method achieved a **0.0%** Attack Success Rate. It is particularly noteworthy that this result was achieved using our core semantic module alone, without relying on the LCP visual feedback loop. This demonstrates the powerful, inherent resilience of our semantic-level manipulation against this type of prompt-based attack. \n\nWhile we acknowledge this is not an exhaustive study covering all attack vectors, this strong initial evidence suggests our core mechanism possesses significant advantages. We have added this new experiment and discussion to the final paper. We are grateful for your suggestion, as it has helped us uncover this important strength.\n\nWe hope our responses and the new experimental results have fully addressed your concerns.\n\n[1] Tsai et al. \"Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models?.\" The Twelfth International Conference on Learning Representations. 2024."}, {"review_id": "PbwzFz1stc", "reviewer": "Reviewer_EaRh", "summary": "This work introduces an innovative inference-time approach to tackle the problem of semantic concept erasure from generative models while preserving the original generative quality of the models. The authors recognize that this is a difficult problem that requires three objectives to be satisfied: erasure completeness to ensure the concept is adequately removed with no traces, erasure locality to ensure that the model's generations are minimally affected for all other concepts, and robustness to different variations of the concept that need to be adequately handled too. Their proposed approach builds on the key insight from previous works that the text embedding space for various LLMs and CLIP is linear which means that embeddings can be manipulated using vector arithmetic. They do a deep dive into the theory behind this including statistical experiments. They then exploit this property to modify the prompt's text embedding to remove the desired concepts, including support for multi-concept erasure, using cosine similarity to detect the degree of presence of these concepts in the prompt and subtracting the concepts' embedding accordingly. They further introduce an optional visual feedback mechanism to better handle concepts that may still be generated due to latent concept persistence in the generative model itself. Their approach offers the advantage of being fast and minimally intrusive (as it does not modify the generative model or its process in any way) while being highly effective. They conduct extensive experiments to demonstrate the efficacy of their approach compared to previous methods.", "questions": "1. DId you use the AOD detector for both, LCP and model evaluation? If yes, this introduces a bias in the evaluation as the same model is used to give feedback within the method. If this is the case, I suggest using different detectors during LCP and model evaluation, re-running and sharing the updated results.\n2. Would be good if the weaknesses mentioned above could be addressed.", "limitations": "Not sufficiently discussed.\n1. The method is limited to generative models where the text embedding space adheres to linear operations/manipulations.\n2. The method while better than previous methods, still does suffer from non-local effects, i.e., the generation of other related concepts is affected.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. Addresses an important and difficult problem.\n2. The approach is not necessarily restricted to a single generative model and in theory can be applied to almost any generative model, at least models that use the CLIP encoder.\n3. Very clearly outline the various assumptions, statistical findings and related theoretical concepts for their approach.\n4. Discuss the various challenges of this problem (erasure completeness, locality and robustness) and specifically conduct evaluations for each of these different aspects to determine the efficacy of the proposed approach and other baselines for each sub-problem.\n5. Comprehensive comparisons done against other methods on a variety of tasks (object erasure, explicit content erasure, style erasure) with relevant metrics.\n6. Propose a simple yet powerful approach building on top of previous insights on the linearity of CLIP space. The method is not restricted to single concept erasure and can easily be extended to multiple concepts. Moreover, since the approach is an inference-time approach without any training and/or fine-tuning, it is relatively quick and inexpensive to run.\n\n**Weaknesses:**\n1. No visuals in the main paper to show examples of the generated images after applying their method. I did notice that they included some visuals in the supplementary but would have been nice to have a few in the main text too.\n2. Using SD v1.4 which is fairly outdated now. Would have been better to try the method on more recent and better models such as SD v2.\n3. Not much discussion done on the limitations of the approach.", "was_revised": true, "final_justification": "This work introduces a simple yet effective inference-time approach for semantic concept erasure in generative models, utilizing the linearity of CLIP’s text embedding space. The authors clearly outline the theoretical concepts, design considerations, and challenges (completeness, locality and robustness), and validate their method with comprehensive experiments and comparisons to strong baselines. The approach is fast, model-agnostic in theory, and supports multi-concept erasure without retraining. While most of my concerns were addressed adequately in the rebuttal, the scalability to other better and more recent generative models has not been concretely demonstrated empirically. Thus, taking everything into consideration, I have decided to keep my score unchanged.", "rebuttal": "We are deeply grateful to the reviewer for their strong support and for the detailed, insightful feedback. Your positive assessment of our work is incredibly encouraging. We have carefully considered all your suggestions, especially the critical point about evaluation bias, and have conducted new experiments to address them. We believe these additions have made our paper significantly stronger.\n\n### Weakness 1: Lack of visualization and discussion on limitations\nThank you for these valuable suggestions. \n\nFor Visuals:\nYou are correct that qualitative examples are very important. While we included several in the appendix due to space constraints, we agree that a key figure should be in the main text. To give you a preview of what we will add, our appendix already contains extensive qualitative results: \n- **Figure 3** demonstrates the effectiveness of our LCP module in mitigating latent concept persistence, correcting errors caused by implicit concept association. \n\n- **Figures 4 & 6** provide qualitative evidence for erasure Completeness and Locality on the CIFAR-10 benchmark. \n\n- **Figures 5, 7, & 8** showcase the versatility of our method across diverse tasks, including object erasure, multi-celebrity erasure, and artistic style removal.\n\n- In our supplementary materials, **Figure 1**  gives a qualitative comparison between our Co-Occurrence Encoding and Naive Approach, showcasing the effectiveness of our proposed Co-Occurrence Encoding mechanism on multi-concept erasure. \n\nTo better showcase this in the main paper, we have designed a new, consolidated figure. The design features the most compelling examples, mostly from Figure 5, to provide an at-a-glance demonstration of our method's wide-ranging capabilities. We have prepared this new figure for inclusion in our final revised version.\n\nFor discussion on limitations: To demonstrate our commitment to a balanced presentation, here is the detailed summary of the expanded limitations section we will add to the paper. We believe a thorough analysis of these points is crucial for contextualizing our work and guiding future research.\n\n***\n\n### **Limitations and Future Work**\n\nOur method, while effective, has several limitations that open avenues for future work:\n\n**1. Reliance on Encoder's Linear Structure:**\nOur approach's applicability is fundamentally conditioned on the text encoder itself exhibiting a near-linear structure where concepts are semantically separable. This assumption, while holding for powerful encoders like CLIP and many common concepts, may not be universally true. It might be less effective for: **(a)** future, potentially non-linear encoder architectures, or **(b)** highly entangled, abstract, or metaphorical concepts where a clean vector representation is ill-defined.\n\n**2. Sources of Non-local Effects on Locality (ACC_L):**\nAlthough our method shows strong locality, it is not entirely free from non-local effects, as reflected by the ACC_L metric. We identify three primary sources for this:\n*   **Imperfect Semantic Thresholding:** The decision boundary defined by our `β` parameter, while effective, may not be perfectly sharp for every conceivable prompt-concept pair. In ambiguous cases, our method might perform a semantic operation on a prompt where the concept is only weakly present, thus affecting related but untargeted semantics and impacting locality.\n*   **Visual Detector Errors (in LCP):** This is a significant factor. For the optional LCP module, false positives from the vision detector can trigger an unnecessary second-stage erasure, which can inadvertently alter related visual elements. We observed a concrete example of this during our experiments: when erasing the concept \"automobile\", the AOD detector used in our LCP loop frequently misclassifies \"trucks\" as \"automobiles\". This causes our LCP module to incorrectly trigger a stronger erasure on prompts that should have generated a \"truck\", leading to a noticeable drop in locality for the \"truck\" class. This highlights the critical dependence of the LCP's locality on the detector's precision.\n*   **Inherent Model Stochasticity:** Some locality degradation is attributable to the inherent stochasticity of the base generative model itself. As seen in our baseline results (SDv1.4 with No Erase), the model sometimes fails to generate concepts perfectly aligned with the prompt, which contributes to a baseline level of locality \"noise\".\n\n**3. Detector Dependency for LCP Module:**\nThe performance of our optional LCP module is inherently tied to the availability and accuracy of an external vision detector. The lack of reliable detectors for abstract concepts (e.g., 'loneliness', 'artistic style') limits the module's applicability to more concrete, visually verifiable concepts. Furthermore, the detector's performance directly creates a trade-off: **false negatives** (missing a concept) can lead to incomplete erasure, while **false positives** (detecting something that isn't there) can harm locality by triggering incorrect feedback, as discussed in point 2.\n\n***\n\nWe have already included this detailed discussion in our paper to provide a clearer context for our work and to chart clear paths for future improvements in the field.\n\n### Weakness 2: SD v1.4 may be outdated.\nWe thank the reviewer for his idea and agree that this is a thoughtful point. We chose SD v1.4 primarily to ensure a fair and direct comparison with the extensive list of prior works in concept erasure (e.g., ESD, MACE, Receler), which almost exclusively use SD v1.4 as the standard benchmark model. This allows for a clear assessment of our method's advancement over the existing literature. We agree that testing on newer models like SDXL or SD 3 is an important next step, which we would include in our discussion of our method's current limitation.\n\n### Question 1: Did we use the AOD detector for both LCP and model evaluation?\nThis is an extremely insightful and critical question. We sincerely thank you for identifying this potential evaluation bias in our original submission. We initially did use the same AOD detector for both the LCP feedback and the final evaluation. \n\nWe agree that this setup is not ideal and may introduce a bias. To rectify this, we have conducted a new, more rigorous evaluation. While keeping AOD as the feedback detector within our LCP loop (to maintain the method's original mechanism), we now use a completely independent and powerful open-vocabulary detector, OWL-ViT[1] , as the unbiased \"referee\". To ensure a fair comparison under this new, stricter protocol, we re-evaluated our method and a representative set of strong and foundational baselines.\n\n**New Evaluation on CIFAR-10 (Avg, Evaluated by the independent OWL-ViT)**:\n| Method             | ACC_E (↓) | ACC_R (↓) | ACC_L (↑) | H-score (↑) |\n|--------------------|-----------|-----------|-----------|-------------|\n| SDv1.4 (No Erase)  | 99.10     | 87.20     | 87.30     | --          |\n| ESD-u    | 12.50     | 39.40     | 82.25     | 74.83       |\n| MACE         | 0.4       | 13.8      | 78.62     | 87.31       |\n| Receler      | 2.50      | 10.00     | 72.69     | 85.41       |\n| Ours (This work)   | 1.50      | 2.00      | 85.51     | 93.60       |\n\nThis new, unbiased evaluation reveals several key observations and findings:\n1.  **Maintained Superiority:** Our method maintains its significant lead in the overall H-score (93.60) over the strongest baselines, confirming our reported superiority is robust and not an artifact of evaluation bias.\n2.  **Dominance in Robustness:** As pointed out by another reviewer, a key advantage of our method is its robustness. This is starkly evident here, with our method achieving an ACC_R of 2.00, an order of magnitude better than the next best parameter-modifying method (Receler at 10.00).\n3.  **Unbiased High Performance:** The results confirm our method's exceptional balance between Completeness (ACC_E) and Robustness (ACC_R) without overly sacrificing Locality (ACC_L).\n\nWe are deeply grateful for this crucial suggestion, as it has substantially improved the scientific rigor of our work. We will replace the original evaluation tables with these new results in the final paper.\n\n### Paper Formatting Concerns\n1. Typo \"desiderata\": Thank you for pointing out the typo and we will correct it in the final version. We have also gone through the paper and make sure typos are corrected before the final version.\n\nOnce again, thank you for your strong endorsement and for providing feedback that has been instrumental in strengthening our paper. We hope our responses and the new, more rigorous experiments have further solidified your confidence in our work.\n\n[1] Minderer, Matthias, et al. \"Simple open-vocabulary object detection.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022."}, {"review_id": "a0zERQc1MG", "reviewer": "Reviewer_KFQr", "summary": "This paper proposes a test-time concept erasure method that leverages calibration to modify the text embedding before the diffusion process begins. To enhance performance when handling multiple concepts, the method incorporates a visual feedback loop. The effectiveness of the approach is validated across various datasets and application scenarios.", "questions": "In addition to the issues mentioned above, I have a few specific questions regarding the method details:\n\n1. In Line 169, how can concatenating a comma-separated string serve as an effective operator for concept merging? Concatenation increases text length and thus affects position embeddings, which may alter the behavior of the model. \n2. In Line 222, why is the maximum estimated score among the active concepts used, rather than an average?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is conceptually interesting and supported by sufficient theoretical analysis.\n2. The overall performance is strong—especially in multi-concept and style erasure tasks—making the approach largely convincing.\n\nWeaknesses:\n\nI believe the primary weaknesses lie in the method presentation and the interpretation of certain experimental results.\n\n1. In Sec. 3, the paper introduces formal definitions of Completeness and Locality within the problem statement. However, this part might be better placed in a separate section rather than within the method description. Additionally, since the evaluation involves three dimensions (the third being Robustness), this metric should also be formally defined. Most importantly, the connection between these formal definitions and the proposed method is not made clear. The paper would benefit from explicitly showing how the method guarantees or improves Completenessand Locality, rather than simply stating their definitions without further integration.\n2. The experimental setup in Sec. 4.2 is unclear. Stable Diffusion v1.4 typically operates on high-resolution images, whereas CIFAR-10 images are only 32×32 pixels. It’s unclear how SD v1.4 is applied in this setting. If fine-tuning is involved, it should be clearly stated. Furthermore, if fine-tuning is indeed performed, it might be more appropriate to use higher-resolution datasets such as those used in DreamBooth.\n3. The interpretation of results in Sec. 4.2 could be misleading. Table 1 does demonstrate strong overall performance, but upon closer inspection, the key advantage of the proposed method seems to lie in its high robustness. This may be due to the fact that the method modifies text embeddings directly, making it less sensitive to rephrased prompts. However, the paper does not discuss this potential strength. Instead, it introduces a combined metric, H, which merges all three metrics. This aggregation may obscure the actual source of improvement and reduce interpretability for the reader.\n4. The method involves several hyper-parameters, which could significantly impact performance. However, this aspect is not discussed in the main text. A sensitivity analysis or ablation study would greatly help readers understand the robustness and generalizability of the method.\n\nMinor Weaknesses:\n1. The core idea of the method—starting from linear operations on text embeddings—is not conceptually complicated. It would be helpful to include a concise explanation or visual diagram to make the intuition more accessible to readers.", "was_revised": true, "final_justification": "The paper presents an interesting method on an important topic, with solid experimental validation. My main concern was the lack of coherence in the writing, especially in the connection between the formal definitions and the method. The rebuttal has addressed this by providing the missing connections and clarifications. Considering these improvements, I recommend accepting the paper.", "rebuttal": "We sincerely thank the reviewer for appreciating the conceptual novelty and strong performance of our work. We are also very grateful for the detailed, constructive feedback on improving the paper's presentation and experimental interpretation. We have carefully addressed all your points and conducted new experiments, which we detail below.\n\n### Weakness 1: Connection between formal definitions and the method.\nThis is an excellent point regarding the paper's structure and logical flow. We agree completely that the connection between our problem formulation and methodology needs to be stronger and clearer. To address the three specific issues you raised, we have implemented the following targeted revisions:\n\n**1. Placement and Completeness of Definitions:**\nWe followed your suggestion and move the problem statement into its own dedicated **\"3. Problem Formulation\"** section. In this section, we formally defined the three key objectives, building upon the definitions already in our paper. Here is our revised and expanded formulation:\n\n*   **Completeness:** As defined in our paper (Eq. 1), the operator must effectively remove the target concepts `C_erase`. Intuitively, this means the probability of `C_erase` appearing in a generated image `I'` should be below a safety threshold `ϵ_safe`.\n*   **Locality (Fidelity):** As defined in our paper (Eq. 2), the modification should minimally affect non-target concepts. This means the change in the presence probability of any non-target concept `c` should be limited by a tolerance `ϵ_tol`.\n*   **Robustness (New Definition):** Building on this, we formally defined **Robustness** as maintaining **Completeness** even when the prompt `p` is paraphrased or adversarially modified to `p_adv`. This frames Robustness as a crucial stress test for the erasure's reliability.\n\n**2. Explicit Link to Method:**\nTo bridge the gap between these definitions and our method, we added concise, explicit statements:\n\n*   **In the Methodology Section:** We added a short \"bridging\" paragraph after the new Problem Formulation. It will state: *\"To achieve these objectives, Semantic Surgery is designed with two core principles. We tackle **Completeness** and **Locality** through a precisely **calibrated vector subtraction** that neutralizes semantics at their source. To ensure **Robustness**, our method operates on the **global text embedding** and \"LCP Visual Feedback\", making it inherently resilient to token-level paraphrasing, unlike methods that target specific words.\"*\n\n*   **In the Experiments Section:** When introducing the evaluation metrics (ACC_E, ACC_R, ACC_L), we explicitly **\"called back\"** to these intuitive definitions, reinforcing the link right where the results are presented.\n\nWe believe this structured approach—with complete, upfront definitions and explicit links in both the methodology and experiments—will create a much clearer and more coherent narrative. Thank you for this deeply insightful suggestion; it has been instrumental in helping us refine the core logical structure of our paper.\n\n### Weakness 2: Unclear experimental setup for CIFAR-10.\nThank you for asking for this clarification. We apologize for the ambiguity. We followed the standard evaluation protocol established by prior works in this area (e.g., ESD, MACE, Receler). To be explicit:\n- We do not use the 32x32 CIFAR-10 images for any training or fine-tuning. Our method is entirely training-free.\n- We use the CIFAR-10 class names (e.g., \"dog\", \"airplane\") to form text prompts, such as \"a photo of a {class}\".\n- These prompts are used to generate high-resolution images with the standard SD v1.4 model and the concept-eliminated SD v1.4 models(with different erasing methods).\n- Finally, these generated images are evaluated by an object detector to calculate the metrics.\nWe have revised Section 4.2 to make this standard process presented in a clearer manner.\n\n### Weakness 3: Interpretation of results could be misleading (H-score vs. Robustness).\nThis is a great observation, which we fully agree. A key strength of our method lies in its superior robustness (ACCR). Your point is further validated by our new adversarial robustness experiments conducted and presented in our response to reviewer moxY. In these tests against the challenging RAB benchmark, our method significantly outperformed all baselines.\n\nNew Adversarial Robustness Study (ASR on RAB prompts):\n| Method                           | ASR (↓)  |\n|----------------------------------|----------|\n| SLD                              | 69.47%   |\n| SAFREE                           | 53.68%   |\n| MACE                             | 7.37%    |\n| Receler                          | 6.32%    |\n| Ours (Semantic Surgery, no LCP)  | **0.0%** |\n\nIt is noteworthy that this remarkable 0.0% Attack Success Rate was achieved using our core semantic module alone, without the LCP detector.\n\nIn the final version, following your suggestion, we have revised our results analysis. We first highlighted the significant gains in both standard robustness (ACCR) and this newly-demonstrated adversarial robustness. We then presented the H-score as an overall summary metric, providing a much clearer picture of our method's distinct advantages.\n\n### Weakness 4 & Minor Weakness 1: Hyper-parameter discussion and lack of a visual diagram.\nWe thank you for these important points.\n- Hyper-parameters: We apologize for not highlighting these analyses sufficiently in the main part of the submitted version. We have conducted extensive studies on our hyperparameters. The analysis for β and γ can be found in the **Supplementary material (Figures 2 & 3)**. We further conducted a new ablation study on the threshold τ, which confirmed our method's stability (H-score > 92.0 for τ in [0.05, 0.5] with a fixed beta(-0.12) and gamma(0.02)). We have created a new subsection in the main paper to summarize these crucial findings.\n- Visual Diagram: We agree that a clear visual diagram is essential. We have prepared a new figure that illustrates the entire workflow of Semantic Surgery. This has been added to Section 3 in the final revised version to make our method's intuition much more accessible.\n\n### Question 1. How can comma-separated string concatenation be an effective operator?\n\nThis is a good question about the underlying mechanism. The effectiveness of comma-separated concatenation stems from the powerful compositional understanding of the pre-trained CLIP text encoder. CLIP was trained on vast amounts of web text, where concepts listed with commas typically represent a collection of distinct entities within a shared context. When CLIP processes a string like \"dog, cat, playing together\", it generates a composite embedding that captures the individual concepts (\"dog\", \"cat\") while preserving the shared context (\"playing together\"). Our qualitative results in the **Supplementary material (Figure 1)** provide strong empirical evidence for this: our method successfully removes both animals while keeping the \"playing\" scene intact, whereas a naive arithmetic approach fails. This demonstrates that leveraging CLIP's inherent compositional ability is a simple yet highly effective strategy.\n\n### Question 2. Why use the maximum estimated score (p_joint = max{p_i}) rather than an average?\n\nThis is another great question about our design choice. We chose to use the maximum score based on a conservative, 'safety-first' principle. In multi-concept erasure, if even one concept is detected with very high confidence (a high ρ_i), it implies a strong user intent for erasure related to that entire conceptual group. An average score could be diluted by other, lower-confidence concepts in the prompt, potentially leading to an under-powered erasure strength and incomplete removal of the most prominent target concept. The max operator ensures that our erasure strength is always driven by the strongest signal of user intent, prioritizing erasure completeness, which is often crucial in safety-related applications.\n\nWe hope our detailed responses, supported by the new experiments, have fully addressed your concerns. We are very grateful for your meticulous review, which has provided invaluable guidance for improving the clarity and depth of our paper."}, {"review_id": "qaKSErB8q1", "reviewer": "Reviewer_x355", "summary": "The paper proposes a novel training-free method for removing undesired concepts on the text-embedding level space to improve completeness-locality trade-off in concept erasing for diffusion models. Unlike finetuning methods that retrain or edit the model, the proposed method, Semantic Surgery, operates purely at inference time by manipulating the text embedding before it enters the diffusion pipeline. The key idea is to perform a calibrated vector subtraction that neutralizes the semantic influence of the target concept. To achieve this, Semantic Biopsy estimates how strongly a concept is represented in the prompt compared the text embeddings of target concepts, which is measured as a confidence score through a sigmoid function. For multi-concept erasure, Co-Occurrence Encoding constructs a joint semantic direction for concept removal while avoiding excessive overlap or semantic distortion. To address Latent Concept Persistence (LCP), they introduce a Visual Feedback Loop using detectors like NudeNet or AOD to identify and suppress residual traces of the target concept in generated images. Experiments across four tasks like object erasure, explicit content removal, artistic style erasure, and multi-celebrity erasure demonstrate the state-of-the-art performance on the trade-off between completeness for erasing target concepts and locality for preserving remaining concepts.", "questions": "Strengths and Weaknessses include questions", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n1. The paper proposes a simple yet novel approach for concept erasing based on arithmetic operations on text embeddings. This method is training-free and does not require any computational cost for training to achieve concept erasure.\n2. In situations where the presence of target concepts in the prompt is unknown, the authors introduce a novel measure called concept presence intensity $\\rho$, which estimates whether the concepts exist. They also provide a rigorous theoretical derivation to support an effective estimation method for this measure.\n3. The proposed method demonstrates clear performance improvements on existing extensive concept erasing benchmarks, including objects, explicit content, artistic styles, and celebrities. In particular, the results on I2P benchmarks and multi-concept celebrity erasure tasks exhibit an impressive balance in the completeness-locality trade-off.\n\n** **\n\n**Weaknesses**\n1. Without any visual guidance for the explanation of the proposed method, the presentation appears to omit core explanations. For instance, the distinctions between $e _{input}$, $e _{erase}$, $e^{'} _{input}$, $e^{'} _{final}$ need to be described more clearly. In particular, it is stated that the vision detector $\\mathcal{D}$ is used to detect the LCP-specific occurrence of target concepts in the generated image. Does this mean that sequential generations of the image from the sample prompts are required? If so, it could significantly increase inference time, which is not well reported in this work. Additionally, the process for construction of $C _{vis}$ should be described more concretely.\n2. The method is an inference-time concept erasure but it is never compared with baselines of a similar nature, such as SLD or SAFREE. To validate the strength of the approach as a training-free method, comparisons with these baselines could be included.\n3. One crucial concern about this work is the robustness of the proposed method against adversarial attacks. Red-teaming tools such as RAB [1] and UnlearnDiffAttack [2] have shown that several concept erasure approaches in diffusion models are vulnerable to adversarial prompt attacks, allowing erased concepts to be easily regenerated. Recent works on safety in T2I diffusion models—such as Receler, CPE [3], and AdvUnlearn—highlight the importance of addressing this issue. The lack of any such consideration in this work raises concerns regarding the method’s robustness to adversarial scenarios.\n4. Ablation studies are lacking: Although the proposed method includes several important components, the paper does not provide corresponding ablation analyses. Key elements that should be ablated include:\n    1. The use of Eq. (9) for calibrated presence estimation vs. naive vector addition $\\sum _{i=1} ^n \\rho _{c _{i}} \\Delta e _{c _{i}}$.\n    2. The impact of visual feedback for LCP mitigation.\n    3. The influence of the threshold parameter $\\tau$ on the method’s performance, which should be explored and reported.\n\n\nI am happy to raise my score if my concerns could be well addressed.\n\n**References**\n\n[1] Tsai et al. \"Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models?.\" *The Twelfth International Conference on Learning Representations*. 2024.\n\n[2] Zhang et al. \"To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2024.\n\n[3] Lee et al. \"Concept pinpoint eraser for text-to-image diffusion models via residual attention gate.\" *The Thirteenth International Conference on Learning Representations*. 2025.", "was_revised": true, "final_justification": "This work introduces a simple yet novel training-fee method for concept erasing using arithmetic operations on text embeddings. The authors provided a new metric called concept presence intensity ($\\rho$), along with a rigorous theoretical foundation for its estimation. The method shows clear performance gains on a wide range of concept erasing benchmarks. Although initial submission raised concerns about clarity on presentation, insufficient baselines, lack of experiments on robustness, and ablation studies, the authors addressed most of the concerns by the rebuttal. Therefore, the reviewer raises the score from 3 to 4.", "rebuttal": "We sincerely thank you for the exceptionally detailed and highly constructive review. Your comments have provided a clear roadmap for improving our paper, and we are especially grateful for your offer to raise your score. We are pleased to report that we have conducted **four new sets of experiments** during the rebuttal period that specifically address every one of your concerns. We are confident that these new results and clarifications will resolve your questions.\n\n\n### Weakness 1: Clarity, LCP process, inference time, and C_vis construction.\nThank you for these detailed questions. We apologize for the lack of clarity in our initial submission and have worked to significantly improve the presentation.\n\n**For Visual Explanation:** We have prepared a new diagram that visually explains our entire workflow. This has been included in Section 3 of the final paper to make our method much more intuitive.\n\n**For LCP Process & Detailed Inference Time Analysis:** \nTo clarify, the LCP feedback loop is an on-demand, two-pass process. A second, time-consuming pass is **only triggered if** residual concepts are detected. To quantify this overhead, we have **conducted a new, detailed timing experiment**:\n\n| Scenario                           | Time per Image |\n|------------------------------------|----------------|\n| 1. Baseline SDv1.4                 | 3.11 s         |\n| 2. SLD                             | 4.07 s         |\n| 3. SAFREE                          | 3.96 s         |\n| 4. Ours (LCP disabled)             | 3.21 s         |\n| 5. Ours (LCP worst-case, Pass 2)   | 6.43 s         |\n| 6. Ours (LCP avg on I2P)           | **4.09 s**     |\n\nThis analysis reveals three key points: \n\n(1) Our core method adds negligible overhead (3.21s vs 3.11s). \n\n(2) The \"doubled time\" is a worst-case scenario.\n\n(3) In a practical safety task (I2P), our average time (4.09s) is comparable to other training-free methods like SLD and SAFREE, while achieving vastly superior safety (as shown in our new I2P comparison). The LCP module is an optional(only in high demand when security requirements are extremely high), highly efficient trade-off for critical safety needs. \n\nThis complete timing analysis has been prepared for inclusion in the final paper.\n\n**For C_vis Construction**: To improve clarity, we have expanded the description in Section 3.3. The revised text now explicitly states that C_vis is formed by selecting concepts {c_i} where the vision detector's confidence score surpasses a threshold τ_vis, and we emphasize that this is an efficient, single-batch operation. This clarification will be present in the final revised version.\n\n\n### Weakness 2: Missing comparison with baselines like SLD or SAFREE.\nThis is a very important point, and we thank you for raising it. Our initial experiments focused on comparing against strong **parameter-modifying** baselines (like MACE and Receler). The rationale was based on the general understanding in the field that inference-time methods, while flexible, typically do not match the performance of methods that directly edit model weights. Our primary goal was therefore attempting to demonstrate that our inference-time approach could not only compete with but **surpass** these powerful, heavyweight baselines, which would be a significant finding. However, to validate your point and provide a complete picture of our method's standing among its direct, training-free peers, we have conducted a **new, head-to-head comparison with SLD and SAFREE on the challenging I2P safety task.**\nThe results, broken down by category, are as follows:\n\n**New Comparison on I2P (Unsafe Instances Count ↓):**\n| Method | Armpits | Belly | Buttocks | Feet | Breasts (F) | Genitalia (F) | Breasts (M) | Genitalia (M) | Total |\n|--------|---------|-------|----------|------|-------------|---------------|-------------|---------------|-------|\n| SAFREE | 11      | 22    | 5        | 9    | 15          | 4             | 15          | 1             | 82    |\n| SLD    | 18      | 48    | 7        | 4    | 57          | 15            | 0           | 0             | 149   |\n| Ours   | 0       | 0     | 1        | 0    | 0           | 0             | 0           | 0             | 1     |\n\nThis detailed breakdown shows that our method achieves near-perfect erasure across almost all categories, reducing the total number of unsafe generations by **over 98%** compared to the next best training-free method, SAFREE(from 82 instances to 1). This confirms that our method holds a substantial advantage within its category on safety-critical tasks. We have already drafted the analysis for this new comparison and will ensure the table and discussion are included in the final paper.\n\n\n### Weakness 3: Lack of robustness evaluation against adversarial attacks.\nWe agree that adversarial robustness is a critical concern. Our initial work focused on establishing a general-purpose framework for high-fidelity erasure under standard conditions, similar to other general-purpose methods like MACE (CVPR 2024) and AdaVD (CVPR 2025) that didn't add the adversarial experiment as well. However, to address your valid concern, we conducted a **new, targeted experiment** using the **Ring-A-Bell (RAB) benchmark** you referenced. We created a test set of 95 adversarial prompts using methodology provided by ring-a-bell to evaluate the Attack Success Rate (ASR).\nThe results of this preliminary study were highly encouraging:\n\n**New Adversarial Robustness Study (ASR on 95 RAB prompts):**\n| Method                          | ASR (↓)  |\n|---------------------------------|----------|\n| SLD                             | 69.47%   |\n| MACE                            | 7.37%    |\n| Receler                         | 6.32%    |\n| Ours (Semantic Surgery, no LCP) | **0.0%** |\n\nIt is noteworthy that this 0.0% Attack Success Rate was achieved by our core semantic module alone, without the LCP detector. While not an exhaustive study, this strong initial evidence suggests our approach has inherent advantages in resisting prompt-level attacks. We have prepared this new experiment and its analysis for inclusion in the final revised version.\n\n\n### Weakness 4: Ablation studies are lacking.\nThank you for pushing for a more thorough analysis. We agree this is essential and address each point:\n\n\na. Calibrated vs. Naive Addition: This comparison is provided qualitatively in the **supplementary material (Figure 1)**, which clearly shows naive addition fails catastrophically in multi-concept scenarios. We will highlight this in the main text.\n\n\nb. Impact of LCP: This ablation is presented in the **main paper's appendix (Table 5)**. It shows the LCP module is critical for safety, reducing unsafe images on I2P from 47 to 1.\n\n\nc. Influence of threshold τ: You are correct, this was an omission. We apologize. We have now conducted a new ablation study on τ of Object Erasure(CIFAR10). The results are as follows:\n\n\n|   τ   | Acc_e (↓) | Acc_r (↓) | Acc_l (↑) | H-score (↑) |\n|-------|-----------|-----------|-----------|-------------|\n| 0.95  | 4.5       | 24.2      | 88.40     | 85.77       |\n| 0.5   | 4.0       | 6.4       | 87.38     | **92.18**   |\n| 0.05  | 4.0       | 3.5       | 84.90     | **92.14**   |\n\nThe results show that performance is highly stable within a wide range [0.05, 0.5], confirming the robustness of our choice. We have prepared this new ablation study and its analysis for inclusion in the final revised version.\n\nWe hope our detailed responses, backed by these extensive new experiments, have fully addressed your concerns. We are deeply grateful for your insightful guidance and look forward to your further feedback."}]

Author rebuttal:

Meta-review: This paper focuses on concept erasure in T2I models, and proposes a training-free zero-shot method for this. The proposed approach, Semantic Surgery, is designed to erase the target concept, while minimally effecting the generations for other concepts. They aim to achieve this by modifying the text embedding through linear concept removal, subtracting our the direction associated with the target concept from the text embedding. They additionally extending this with a co-occurrence encoding to handle multiple concepts simultaneously. Even when the target concept is removed from the text embedding, the T2I model's U-Net prior can often re-introduce the concept back into the final image. The authors counteract this by introducing a Visual Feedback Loop to iteratively remove the target by analysis the generated image. 

The proposed approach is theoretically backed, and is based on clear geometric intuition. The Visual Feedback Loop co-occurrence encoding in particular make the approach more than an incremental step beyond existing linear token embedding debiasing approaches, and solve challenges particular to the T2I setting (namely, the fact that the U-Net's prior can reintroduce the removed concept). The proposed approach is sound and does not seem to be overengineered. Empirical results compare against a reasonable set of baselines on a variety of tasks.  

Reviewers identified several weaknesses during the initial review phase. Primarily, concerns centered around missing robustness evaluation through adversarial prompt attacks [moxY, x355], the fact that evaluation did not include state-of-the-art diffusion models [EaRh], some issues regarding clarity and details of the experimental setup [KFQr, x355], the number of hyperparameters potentially leading to a brittle system [KFQr], lack of comparison with prior methods such as SLD or SAFREE [x355], and limited ablation results [x355]. 

The authors addressed most of these concerns to an adequate level during the rebuttal phase, particularly by adding additional compared methods, new results on an adversarial prompt benchmark dataset (Ring-A-Bell), and expanded ablation results. After the rebuttal, all reviewers lean towards acceptance and have indicated their concerns have been mostly addressed. 

The primary concern remaining after the rebuttal is that SD v1.4 is very outdated by this point, and that the approach's improvement over baselines may not be as impressive on more modern T2I models such as SD2 or SD3. Additionally, while the authors updated results on the adversarial prompt benchmark are appreciated, only one benchmark has been added. The new results are thus promising, but not definitive. 

Despite this, the reviewers and I see this as a strong submission and valuable research that should be disseminated to the broader research community. I thus recommend acceptance. I also urge the authors to include a discussion on the use of the older SD v1.4 model and lack of results on newer SD models into their limitation section.
Decision: Accept (poster)